research findings and conclusions

How to Write a Conclusion for Research Papers (with Examples)

The conclusion of a research paper is a crucial section that plays a significant role in the overall impact and effectiveness of your research paper. However, this is also the section that typically receives less attention compared to the introduction and the body of the paper. The conclusion serves to provide a concise summary of the key findings, their significance, their implications, and a sense of closure to the study. Discussing how can the findings be applied in real-world scenarios or inform policy, practice, or decision-making is especially valuable to practitioners and policymakers. The research paper conclusion also provides researchers with clear insights and valuable information for their own work, which they can then build on and contribute to the advancement of knowledge in the field.

The research paper conclusion should explain the significance of your findings within the broader context of your field. It restates how your results contribute to the existing body of knowledge and whether they confirm or challenge existing theories or hypotheses. Also, by identifying unanswered questions or areas requiring further investigation, your awareness of the broader research landscape can be demonstrated.

Remember to tailor the research paper conclusion to the specific needs and interests of your intended audience, which may include researchers, practitioners, policymakers, or a combination of these.

Table of Contents

What is a conclusion in a research paper, summarizing conclusion, editorial conclusion, externalizing conclusion, importance of a good research paper conclusion, how to write a conclusion for your research paper, research paper conclusion examples.

• How to write a research paper conclusion with Paperpal? 

Frequently Asked Questions

A conclusion in a research paper is the final section where you summarize and wrap up your research, presenting the key findings and insights derived from your study. The research paper conclusion is not the place to introduce new information or data that was not discussed in the main body of the paper. When working on how to conclude a research paper, remember to stick to summarizing and interpreting existing content. The research paper conclusion serves the following purposes: 1

• Warn readers of the possible consequences of not attending to the problem.
• Recommend specific course(s) of action.
• Restate key ideas to drive home the ultimate point of your research paper.
• Provide a “take-home” message that you want the readers to remember about your study.

Types of conclusions for research papers

In research papers, the conclusion provides closure to the reader. The type of research paper conclusion you choose depends on the nature of your study, your goals, and your target audience. I provide you with three common types of conclusions:

A summarizing conclusion is the most common type of conclusion in research papers. It involves summarizing the main points, reiterating the research question, and restating the significance of the findings. This common type of research paper conclusion is used across different disciplines.

An editorial conclusion is less common but can be used in research papers that are focused on proposing or advocating for a particular viewpoint or policy. It involves presenting a strong editorial or opinion based on the research findings and offering recommendations or calls to action.

An externalizing conclusion is a type of conclusion that extends the research beyond the scope of the paper by suggesting potential future research directions or discussing the broader implications of the findings. This type of conclusion is often used in more theoretical or exploratory research papers.

Align your conclusion’s tone with the rest of your research paper. Start Writing with Paperpal Now!  

The conclusion in a research paper serves several important purposes:

• Offers Implications and Recommendations : Your research paper conclusion is an excellent place to discuss the broader implications of your research and suggest potential areas for further study. It’s also an opportunity to offer practical recommendations based on your findings.
• Provides Closure : A good research paper conclusion provides a sense of closure to your paper. It should leave the reader with a feeling that they have reached the end of a well-structured and thought-provoking research project.
• Leaves a Lasting Impression : Writing a well-crafted research paper conclusion leaves a lasting impression on your readers. It’s your final opportunity to leave them with a new idea, a call to action, or a memorable quote.

Writing a strong conclusion for your research paper is essential to leave a lasting impression on your readers. Here’s a step-by-step process to help you create and know what to put in the conclusion of a research paper: 2

• Research Statement : Begin your research paper conclusion by restating your research statement. This reminds the reader of the main point you’ve been trying to prove throughout your paper. Keep it concise and clear.
• Key Points : Summarize the main arguments and key points you’ve made in your paper. Avoid introducing new information in the research paper conclusion. Instead, provide a concise overview of what you’ve discussed in the body of your paper.
• Address the Research Questions : If your research paper is based on specific research questions or hypotheses, briefly address whether you’ve answered them or achieved your research goals. Discuss the significance of your findings in this context.
• Significance : Highlight the importance of your research and its relevance in the broader context. Explain why your findings matter and how they contribute to the existing knowledge in your field.
• Implications : Explore the practical or theoretical implications of your research. How might your findings impact future research, policy, or real-world applications? Consider the “so what?” question.
• Future Research : Offer suggestions for future research in your area. What questions or aspects remain unanswered or warrant further investigation? This shows that your work opens the door for future exploration.
• Closing Thought : Conclude your research paper conclusion with a thought-provoking or memorable statement. This can leave a lasting impression on your readers and wrap up your paper effectively. Avoid introducing new information or arguments here.
• Proofread and Revise : Carefully proofread your conclusion for grammar, spelling, and clarity. Ensure that your ideas flow smoothly and that your conclusion is coherent and well-structured.

Write your research paper conclusion 2x faster with Paperpal. Try it now!

Remember that a well-crafted research paper conclusion is a reflection of the strength of your research and your ability to communicate its significance effectively. It should leave a lasting impression on your readers and tie together all the threads of your paper. Now you know how to start the conclusion of a research paper and what elements to include to make it impactful, let’s look at a research paper conclusion sample.

How to write a research paper conclusion with Paperpal?

A research paper conclusion is not just a summary of your study, but a synthesis of the key findings that ties the research together and places it in a broader context. A research paper conclusion should be concise, typically around one paragraph in length. However, some complex topics may require a longer conclusion to ensure the reader is left with a clear understanding of the study’s significance. Paperpal, an AI writing assistant trusted by over 800,000 academics globally, can help you write a well-structured conclusion for your research paper. 

• Sign Up or Log In: Create a new Paperpal account or login with your details.  
• Navigate to Features : Once logged in, head over to the features’ side navigation pane. Click on Templates and you’ll find a suite of generative AI features to help you write better, faster.  
• Generate an outline: Under Templates, select ‘Outlines’. Choose ‘Research article’ as your document type.  
• Select your section: Since you’re focusing on the conclusion, select this section when prompted.  
• Choose your field of study: Identifying your field of study allows Paperpal to provide more targeted suggestions, ensuring the relevance of your conclusion to your specific area of research. 
• Provide a brief description of your study: Enter details about your research topic and findings. This information helps Paperpal generate a tailored outline that aligns with your paper’s content. 
• Generate the conclusion outline: After entering all necessary details, click on ‘generate’. Paperpal will then create a structured outline for your conclusion, to help you start writing and build upon the outline.  
• Write your conclusion: Use the generated outline to build your conclusion. The outline serves as a guide, ensuring you cover all critical aspects of a strong conclusion, from summarizing key findings to highlighting the research’s implications. 
• Refine and enhance: Paperpal’s ‘Make Academic’ feature can be particularly useful in the final stages. Select any paragraph of your conclusion and use this feature to elevate the academic tone, ensuring your writing is aligned to the academic journal standards. 

By following these steps, Paperpal not only simplifies the process of writing a research paper conclusion but also ensures it is impactful, concise, and aligned with academic standards. Sign up with Paperpal today and write your research paper conclusion 2x faster .  

The research paper conclusion is a crucial part of your paper as it provides the final opportunity to leave a strong impression on your readers. In the research paper conclusion, summarize the main points of your research paper by restating your research statement, highlighting the most important findings, addressing the research questions or objectives, explaining the broader context of the study, discussing the significance of your findings, providing recommendations if applicable, and emphasizing the takeaway message. The main purpose of the conclusion is to remind the reader of the main point or argument of your paper and to provide a clear and concise summary of the key findings and their implications. All these elements should feature on your list of what to put in the conclusion of a research paper to create a strong final statement for your work.

A strong conclusion is a critical component of a research paper, as it provides an opportunity to wrap up your arguments, reiterate your main points, and leave a lasting impression on your readers. Here are the key elements of a strong research paper conclusion: 1. Conciseness : A research paper conclusion should be concise and to the point. It should not introduce new information or ideas that were not discussed in the body of the paper. 2. Summarization : The research paper conclusion should be comprehensive enough to give the reader a clear understanding of the research’s main contributions. 3 . Relevance : Ensure that the information included in the research paper conclusion is directly relevant to the research paper’s main topic and objectives; avoid unnecessary details. 4 . Connection to the Introduction : A well-structured research paper conclusion often revisits the key points made in the introduction and shows how the research has addressed the initial questions or objectives. 5. Emphasis : Highlight the significance and implications of your research. Why is your study important? What are the broader implications or applications of your findings? 6 . Call to Action : Include a call to action or a recommendation for future research or action based on your findings.

The length of a research paper conclusion can vary depending on several factors, including the overall length of the paper, the complexity of the research, and the specific journal requirements. While there is no strict rule for the length of a conclusion, but it’s generally advisable to keep it relatively short. A typical research paper conclusion might be around 5-10% of the paper’s total length. For example, if your paper is 10 pages long, the conclusion might be roughly half a page to one page in length.

In general, you do not need to include citations in the research paper conclusion. Citations are typically reserved for the body of the paper to support your arguments and provide evidence for your claims. However, there may be some exceptions to this rule: 1. If you are drawing a direct quote or paraphrasing a specific source in your research paper conclusion, you should include a citation to give proper credit to the original author. 2. If your conclusion refers to or discusses specific research, data, or sources that are crucial to the overall argument, citations can be included to reinforce your conclusion’s validity.

The conclusion of a research paper serves several important purposes: 1. Summarize the Key Points 2. Reinforce the Main Argument 3. Provide Closure 4. Offer Insights or Implications 5. Engage the Reader. 6. Reflect on Limitations

Remember that the primary purpose of the research paper conclusion is to leave a lasting impression on the reader, reinforcing the key points and providing closure to your research. It’s often the last part of the paper that the reader will see, so it should be strong and well-crafted.

• Makar, G., Foltz, C., Lendner, M., & Vaccaro, A. R. (2018). How to write effective discussion and conclusion sections. Clinical spine surgery, 31(8), 345-346.
• Bunton, D. (2005). The structure of PhD conclusion chapters.  Journal of English for academic purposes ,  4 (3), 207-224.

Paperpal is a comprehensive AI writing toolkit that helps students and researchers achieve 2x the writing in half the time. It leverages 21+ years of STM experience and insights from millions of research articles to provide in-depth academic writing, language editing, and submission readiness support to help you write better, faster.  

Get accurate academic translations, rewriting support, grammar checks, vocabulary suggestions, and generative AI assistance that delivers human precision at machine speed. Try for free or upgrade to Paperpal Prime starting at US$19 a month to access premium features, including consistency, plagiarism, and 30+ submission readiness checks to help you succeed.  

Experience the future of academic writing – Sign up to Paperpal and start writing for free!  

Related Reads:

• 5 Reasons for Rejection After Peer Review
• Ethical Research Practices For Research with Human Subjects

7 Ways to Improve Your Academic Writing Process

• Paraphrasing in Academic Writing: Answering Top Author Queries

Preflight For Editorial Desk: The Perfect Hybrid (AI + Human) Assistance Against Compromised Manuscripts

You may also like, what are journal guidelines on using generative ai..., quillbot review: features, pricing, and free alternatives, what is an academic paper types and elements , should you use ai tools like chatgpt for..., publish research papers: 9 steps for successful publications , what are the different types of research papers, how to make translating academic papers less challenging, self-plagiarism in research: what it is and how..., 6 tips for post-doc researchers to take their..., presenting research data effectively through tables and figures.

• USC Libraries
• Research Guides

Organizing Your Social Sciences Research Paper

• 9. The Conclusion
• Purpose of Guide
• Design Flaws to Avoid
• Independent and Dependent Variables
• Glossary of Research Terms
• Reading Research Effectively
• Narrowing a Topic Idea
• Broadening a Topic Idea
• Extending the Timeliness of a Topic Idea
• Academic Writing Style
• Choosing a Title
• Making an Outline
• Paragraph Development
• Research Process Video Series
• Executive Summary
• The C.A.R.S. Model
• Background Information
• The Research Problem/Question
• Theoretical Framework
• Citation Tracking
• Content Alert Services
• Evaluating Sources
• Primary Sources
• Secondary Sources
• Tiertiary Sources
• Scholarly vs. Popular Publications
• Qualitative Methods
• Quantitative Methods
• Insiderness
• Using Non-Textual Elements
• Limitations of the Study
• Common Grammar Mistakes
• Writing Concisely
• Avoiding Plagiarism
• Footnotes or Endnotes?
• Further Readings
• Generative AI and Writing
• USC Libraries Tutorials and Other Guides
• Bibliography

The conclusion is intended to help the reader understand why your research should matter to them after they have finished reading the paper. A conclusion is not merely a summary of the main topics covered or a re-statement of your research problem, but a synthesis of key points derived from the findings of your study and, if applicable, where you recommend new areas for future research. For most college-level research papers, two or three well-developed paragraphs is sufficient for a conclusion, although in some cases, more paragraphs may be required in describing the key findings and their significance.

Conclusions. The Writing Center. University of North Carolina; Conclusions. The Writing Lab and The OWL. Purdue University.

Importance of a Good Conclusion

A well-written conclusion provides you with important opportunities to demonstrate to the reader your understanding of the research problem. These include:

• Presenting the last word on the issues you raised in your paper . Just as the introduction gives a first impression to your reader, the conclusion offers a chance to leave a lasting impression. Do this, for example, by highlighting key findings in your analysis that advance new understanding about the research problem, that are unusual or unexpected, or that have important implications applied to practice.
• Summarizing your thoughts and conveying the larger significance of your study . The conclusion is an opportunity to succinctly re-emphasize  your answer to the "So What?" question by placing the study within the context of how your research advances past research about the topic.
• Identifying how a gap in the literature has been addressed . The conclusion can be where you describe how a previously identified gap in the literature [first identified in your literature review section] has been addressed by your research and why this contribution is significant.
• Demonstrating the importance of your ideas . Don't be shy. The conclusion offers an opportunity to elaborate on the impact and significance of your findings. This is particularly important if your study approached examining the research problem from an unusual or innovative perspective.
• Introducing possible new or expanded ways of thinking about the research problem . This does not refer to introducing new information [which should be avoided], but to offer new insight and creative approaches for framing or contextualizing the research problem based on the results of your study.

Bunton, David. “The Structure of PhD Conclusion Chapters.” Journal of English for Academic Purposes 4 (July 2005): 207–224; Conclusions. The Writing Center. University of North Carolina; Kretchmer, Paul. Twelve Steps to Writing an Effective Conclusion. San Francisco Edit, 2003-2008; Conclusions. The Writing Lab and The OWL. Purdue University; Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8.

Structure and Writing Style

I.  General Rules

The general function of your paper's conclusion is to restate the main argument . It reminds the reader of the strengths of your main argument(s) and reiterates the most important evidence supporting those argument(s). Do this by clearly summarizing the context, background, and necessity of pursuing the research problem you investigated in relation to an issue, controversy, or a gap found in the literature. However, make sure that your conclusion is not simply a repetitive summary of the findings. This reduces the impact of the argument(s) you have developed in your paper.

When writing the conclusion to your paper, follow these general rules:

• Present your conclusions in clear, concise language. Re-state the purpose of your study, then describe how your findings differ or support those of other studies and why [i.e., what were the unique, new, or crucial contributions your study made to the overall research about your topic?].
• Do not simply reiterate your findings or the discussion of your results. Provide a synthesis of arguments presented in the paper to show how these converge to address the research problem and the overall objectives of your study.
• Indicate opportunities for future research if you haven't already done so in the discussion section of your paper. Highlighting the need for further research provides the reader with evidence that you have an in-depth awareness of the research problem but that further investigations should take place beyond the scope of your investigation.

Consider the following points to help ensure your conclusion is presented well:

• If the argument or purpose of your paper is complex, you may need to summarize the argument for your reader.
• If, prior to your conclusion, you have not yet explained the significance of your findings or if you are proceeding inductively, use the end of your paper to describe your main points and explain their significance.
• Move from a detailed to a general level of consideration that returns the topic to the context provided by the introduction or within a new context that emerges from the data [this is opposite of the introduction, which begins with general discussion of the context and ends with a detailed description of the research problem]. 

The conclusion also provides a place for you to persuasively and succinctly restate the research problem, given that the reader has now been presented with all the information about the topic . Depending on the discipline you are writing in, the concluding paragraph may contain your reflections on the evidence presented. However, the nature of being introspective about the research you have conducted will depend on the topic and whether your professor wants you to express your observations in this way. If asked to think introspectively about the topics, do not delve into idle speculation. Being introspective means looking within yourself as an author to try and understand an issue more deeply, not to guess at possible outcomes or make up scenarios not supported by the evidence.

II.  Developing a Compelling Conclusion

Although an effective conclusion needs to be clear and succinct, it does not need to be written passively or lack a compelling narrative. Strategies to help you move beyond merely summarizing the key points of your research paper may include any of the following:

• If your essay deals with a critical, contemporary problem, warn readers of the possible consequences of not attending to the problem proactively.
• Recommend a specific course or courses of action that, if adopted, could address a specific problem in practice or in the development of new knowledge leading to positive change.
• Cite a relevant quotation or expert opinion already noted in your paper in order to lend authority and support to the conclusion(s) you have reached [a good source would be from your literature review].
• Explain the consequences of your research in a way that elicits action or demonstrates urgency in seeking change.
• Restate a key statistic, fact, or visual image to emphasize the most important finding of your paper.
• If your discipline encourages personal reflection, illustrate your concluding point by drawing from your own life experiences.
• Return to an anecdote, an example, or a quotation that you presented in your introduction, but add further insight derived from the findings of your study; use your interpretation of results from your study to recast it in new or important ways.
• Provide a "take-home" message in the form of a succinct, declarative statement that you want the reader to remember about your study.

III. Problems to Avoid

Failure to be concise Your conclusion section should be concise and to the point. Conclusions that are too lengthy often have unnecessary information in them. The conclusion is not the place for details about your methodology or results. Although you should give a summary of what was learned from your research, this summary should be relatively brief, since the emphasis in the conclusion is on the implications, evaluations, insights, and other forms of analysis that you make. Strategies for writing concisely can be found here .

Failure to comment on larger, more significant issues In the introduction, your task was to move from the general [the field of study] to the specific [the research problem]. However, in the conclusion, your task is to move from a specific discussion [your research problem] back to a general discussion framed around the implications and significance of your findings [i.e., how your research contributes new understanding or fills an important gap in the literature]. In short, the conclusion is where you should place your research within a larger context [visualize your paper as an hourglass--start with a broad introduction and review of the literature, move to the specific analysis and discussion, conclude with a broad summary of the study's implications and significance].

Failure to reveal problems and negative results Negative aspects of the research process should never be ignored. These are problems, deficiencies, or challenges encountered during your study. They should be summarized as a way of qualifying your overall conclusions. If you encountered negative or unintended results [i.e., findings that are validated outside the research context in which they were generated], you must report them in the results section and discuss their implications in the discussion section of your paper. In the conclusion, use negative results as an opportunity to explain their possible significance and/or how they may form the basis for future research.

Failure to provide a clear summary of what was learned In order to be able to discuss how your research fits within your field of study [and possibly the world at large], you need to summarize briefly and succinctly how it contributes to new knowledge or a new understanding about the research problem. This element of your conclusion may be only a few sentences long.

Failure to match the objectives of your research Often research objectives in the social and behavioral sciences change while the research is being carried out. This is not a problem unless you forget to go back and refine the original objectives in your introduction. As these changes emerge they must be documented so that they accurately reflect what you were trying to accomplish in your research [not what you thought you might accomplish when you began].

Resist the urge to apologize If you've immersed yourself in studying the research problem, you presumably should know a good deal about it [perhaps even more than your professor!]. Nevertheless, by the time you have finished writing, you may be having some doubts about what you have produced. Repress those doubts! Don't undermine your authority as a researcher by saying something like, "This is just one approach to examining this problem; there may be other, much better approaches that...." The overall tone of your conclusion should convey confidence to the reader about the study's validity and realiability.

Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8; Concluding Paragraphs. College Writing Center at Meramec. St. Louis Community College; Conclusions. The Writing Center. University of North Carolina; Conclusions. The Writing Lab and The OWL. Purdue University; Freedman, Leora  and Jerry Plotnick. Introductions and Conclusions. The Lab Report. University College Writing Centre. University of Toronto; Leibensperger, Summer. Draft Your Conclusion. Academic Center, the University of Houston-Victoria, 2003; Make Your Last Words Count. The Writer’s Handbook. Writing Center. University of Wisconsin Madison; Miquel, Fuster-Marquez and Carmen Gregori-Signes. “Chapter Six: ‘Last but Not Least:’ Writing the Conclusion of Your Paper.” In Writing an Applied Linguistics Thesis or Dissertation: A Guide to Presenting Empirical Research . John Bitchener, editor. (Basingstoke,UK: Palgrave Macmillan, 2010), pp. 93-105; Tips for Writing a Good Conclusion. Writing@CSU. Colorado State University; Kretchmer, Paul. Twelve Steps to Writing an Effective Conclusion. San Francisco Edit, 2003-2008; Writing Conclusions. Writing Tutorial Services, Center for Innovative Teaching and Learning. Indiana University; Writing: Considering Structure and Organization. Institute for Writing Rhetoric. Dartmouth College.

Writing Tip

Don't Belabor the Obvious!

Avoid phrases like "in conclusion...," "in summary...," or "in closing...." These phrases can be useful, even welcome, in oral presentations. But readers can see by the tell-tale section heading and number of pages remaining that they are reaching the end of your paper. You'll irritate your readers if you belabor the obvious.

Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8.

Another Writing Tip

New Insight, Not New Information!

Don't surprise the reader with new information in your conclusion that was never referenced anywhere else in the paper. This why the conclusion rarely has citations to sources. If you have new information to present, add it to the discussion or other appropriate section of the paper. Note that, although no new information is introduced, the conclusion, along with the discussion section, is where you offer your most "original" contributions in the paper; the conclusion is where you describe the value of your research, demonstrate that you understand the material that you’ve presented, and position your findings within the larger context of scholarship on the topic, including describing how your research contributes new insights to that scholarship.

Assan, Joseph. "Writing the Conclusion Chapter: The Good, the Bad and the Missing." Liverpool: Development Studies Association (2009): 1-8; Conclusions. The Writing Center. University of North Carolina.

• << Previous: Limitations of the Study
• Next: Appendices >>
• Last Updated: Apr 3, 2024 10:04 AM
• URL: https://libguides.usc.edu/writingguide

When you choose to publish with PLOS, your research makes an impact. Make your work accessible to all, without restrictions, and accelerate scientific discovery with options like preprints and published peer review that make your work more Open.

• PLOS Biology
• PLOS Climate
• PLOS Complex Systems
• PLOS Computational Biology
• PLOS Digital Health
• PLOS Genetics
• PLOS Global Public Health
• PLOS Medicine
• PLOS Mental Health
• PLOS Neglected Tropical Diseases
• PLOS Pathogens
• PLOS Sustainability and Transformation
• PLOS Collections
• How to Write Discussions and Conclusions

The discussion section contains the results and outcomes of a study. An effective discussion informs readers what can be learned from your experiment and provides context for the results.

What makes an effective discussion?

When you’re ready to write your discussion, you’ve already introduced the purpose of your study and provided an in-depth description of the methodology. The discussion informs readers about the larger implications of your study based on the results. Highlighting these implications while not overstating the findings can be challenging, especially when you’re submitting to a journal that selects articles based on novelty or potential impact. Regardless of what journal you are submitting to, the discussion section always serves the same purpose: concluding what your study results actually mean.

A successful discussion section puts your findings in context. It should include:

• the results of your research,
• a discussion of related research, and
• a comparison between your results and initial hypothesis.

Tip: Not all journals share the same naming conventions.

You can apply the advice in this article to the conclusion, results or discussion sections of your manuscript.

Our Early Career Researcher community tells us that the conclusion is often considered the most difficult aspect of a manuscript to write. To help, this guide provides questions to ask yourself, a basic structure to model your discussion off of and examples from published manuscripts. 

Questions to ask yourself:

• Was my hypothesis correct?
• If my hypothesis is partially correct or entirely different, what can be learned from the results? 
• How do the conclusions reshape or add onto the existing knowledge in the field? What does previous research say about the topic? 
• Why are the results important or relevant to your audience? Do they add further evidence to a scientific consensus or disprove prior studies? 
• How can future research build on these observations? What are the key experiments that must be done? 
• What is the “take-home” message you want your reader to leave with?

How to structure a discussion

Trying to fit a complete discussion into a single paragraph can add unnecessary stress to the writing process. If possible, you’ll want to give yourself two or three paragraphs to give the reader a comprehensive understanding of your study as a whole. Here’s one way to structure an effective discussion:

Writing Tips

While the above sections can help you brainstorm and structure your discussion, there are many common mistakes that writers revert to when having difficulties with their paper. Writing a discussion can be a delicate balance between summarizing your results, providing proper context for your research and avoiding introducing new information. Remember that your paper should be both confident and honest about the results! 

• Read the journal’s guidelines on the discussion and conclusion sections. If possible, learn about the guidelines before writing the discussion to ensure you’re writing to meet their expectations. 
• Begin with a clear statement of the principal findings. This will reinforce the main take-away for the reader and set up the rest of the discussion. 
• Explain why the outcomes of your study are important to the reader. Discuss the implications of your findings realistically based on previous literature, highlighting both the strengths and limitations of the research. 
• State whether the results prove or disprove your hypothesis. If your hypothesis was disproved, what might be the reasons? 
• Introduce new or expanded ways to think about the research question. Indicate what next steps can be taken to further pursue any unresolved questions. 
• If dealing with a contemporary or ongoing problem, such as climate change, discuss possible consequences if the problem is avoided. 
• Be concise. Adding unnecessary detail can distract from the main findings. 

Don’t

• Rewrite your abstract. Statements with “we investigated” or “we studied” generally do not belong in the discussion. 
• Include new arguments or evidence not previously discussed. Necessary information and evidence should be introduced in the main body of the paper. 
• Apologize. Even if your research contains significant limitations, don’t undermine your authority by including statements that doubt your methodology or execution. 
• Shy away from speaking on limitations or negative results. Including limitations and negative results will give readers a complete understanding of the presented research. Potential limitations include sources of potential bias, threats to internal or external validity, barriers to implementing an intervention and other issues inherent to the study design. 
• Overstate the importance of your findings. Making grand statements about how a study will fully resolve large questions can lead readers to doubt the success of the research. 

Snippets of Effective Discussions:

Consumer-based actions to reduce plastic pollution in rivers: A multi-criteria decision analysis approach

Identifying reliable indicators of fitness in polar bears

• How to Write a Great Title
• How to Write an Abstract
• How to Write Your Methods
• How to Report Statistics
• How to Edit Your Work

The contents of the Peer Review Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

The contents of the Writing Center are also available as a live, interactive training session, complete with slides, talking points, and activities. …

There’s a lot to consider when deciding where to submit your work. Learn how to choose a journal that will help your study reach its audience, while reflecting your values as a researcher…

Join thousands of product people at Insight Out Conf on April 11. Register free.

Insights hub solutions

Analyze data

Uncover deep customer insights with fast, powerful features, store insights, curate and manage insights in one searchable platform, scale research, unlock the potential of customer insights at enterprise scale.

Featured reads

Inspiration

Three things to look forward to at Insight Out

Tips and tricks

Make magic with your customer data in Dovetail

Four ways Dovetail helps Product Managers master continuous product discovery

Events and videos

© Dovetail Research Pty. Ltd.

How to write a strong conclusion for your research paper

Last updated

17 February 2024

Reviewed by

Writing a research paper is a chance to share your knowledge and hypothesis. It's an opportunity to demonstrate your many hours of research and prove your ability to write convincingly.

Ideally, by the end of your research paper, you'll have brought your readers on a journey to reach the conclusions you've pre-determined. However, if you don't stick the landing with a good conclusion, you'll risk losing your reader’s trust.

Writing a strong conclusion for your research paper involves a few important steps, including restating the thesis and summing up everything properly.

Find out what to include and what to avoid, so you can effectively demonstrate your understanding of the topic and prove your expertise.

• Why is a good conclusion important?

A good conclusion can cement your paper in the reader’s mind. Making a strong impression in your introduction can draw your readers in, but it's the conclusion that will inspire them.

• What to include in a research paper conclusion

There are a few specifics you should include in your research paper conclusion. Offer your readers some sense of urgency or consequence by pointing out why they should care about the topic you have covered. Discuss any common problems associated with your topic and provide suggestions as to how these problems can be solved or addressed.

The conclusion should include a restatement of your initial thesis. Thesis statements are strengthened after you’ve presented supporting evidence (as you will have done in the paper), so make a point to reintroduce it at the end.

Finally, recap the main points of your research paper, highlighting the key takeaways you want readers to remember. If you've made multiple points throughout the paper, refer to the ones with the strongest supporting evidence.

• Steps for writing a research paper conclusion

Many writers find the conclusion the most challenging part of any research project . By following these three steps, you'll be prepared to write a conclusion that is effective and concise.

• Step 1: Restate the problem

Always begin by restating the research problem in the conclusion of a research paper. This serves to remind the reader of your hypothesis and refresh them on the main point of the paper. 

When restating the problem, take care to avoid using exactly the same words you employed earlier in the paper.

• Step 2: Sum up the paper

After you've restated the problem, sum up the paper by revealing your overall findings. The method for this differs slightly, depending on whether you're crafting an argumentative paper or an empirical paper.

Argumentative paper: Restate your thesis and arguments

Argumentative papers involve introducing a thesis statement early on. In crafting the conclusion for an argumentative paper, always restate the thesis, outlining the way you've developed it throughout the entire paper.

It might be appropriate to mention any counterarguments in the conclusion, so you can demonstrate how your thesis is correct or how the data best supports your main points.

Empirical paper: Summarize research findings

Empirical papers break down a series of research questions. In your conclusion, discuss the findings your research revealed, including any information that surprised you.

Be clear about the conclusions you reached, and explain whether or not you expected to arrive at these particular ones.

• Step 3: Discuss the implications of your research

Argumentative papers and empirical papers also differ in this part of a research paper conclusion. Here are some tips on crafting conclusions for argumentative and empirical papers.

Argumentative paper: Powerful closing statement

In an argumentative paper, you'll have spent a great deal of time expressing the opinions you formed after doing a significant amount of research. Make a strong closing statement in your argumentative paper's conclusion to share the significance of your work.

You can outline the next steps through a bold call to action, or restate how powerful your ideas turned out to be.

Empirical paper: Directions for future research

Empirical papers are broader in scope. They usually cover a variety of aspects and can include several points of view.

To write a good conclusion for an empirical paper, suggest the type of research that could be done in the future, including methods for further investigation or outlining ways other researchers might proceed.

If you feel your research had any limitations, even if they were outside your control, you could mention these in your conclusion.

After you finish outlining your conclusion, ask someone to read it and offer feedback. In any research project you're especially close to, it can be hard to identify problem areas. Having a close friend or someone whose opinion you value read the research paper and provide honest feedback can be invaluable. Take note of any suggested edits and consider incorporating them into your paper if they make sense.

• Things to avoid in a research paper conclusion

Keep these aspects to avoid in mind as you're writing your conclusion and refer to them after you've created an outline.

Dry summary

Writing a memorable, succinct conclusion is arguably more important than a strong introduction. Take care to avoid just rephrasing your main points, and don't fall into the trap of repeating dry facts or citations.

You can provide a new perspective for your readers to think about or contextualize your research. Either way, make the conclusion vibrant and interesting, rather than a rote recitation of your research paper’s highlights.

Clichéd or generic phrasing

Your research paper conclusion should feel fresh and inspiring. Avoid generic phrases like "to sum up" or "in conclusion." These phrases tend to be overused, especially in an academic context and might turn your readers off.

The conclusion also isn't the time to introduce colloquial phrases or informal language. Retain a professional, confident tone consistent throughout your paper’s conclusion so it feels exciting and bold.

New data or evidence

While you should present strong data throughout your paper, the conclusion isn't the place to introduce new evidence. This is because readers are engaged in actively learning as they read through the body of your paper.

By the time they reach the conclusion, they will have formed an opinion one way or the other (hopefully in your favor!). Introducing new evidence in the conclusion will only serve to surprise or frustrate your reader.

Ignoring contradictory evidence

If your research reveals contradictory evidence, don't ignore it in the conclusion. This will damage your credibility as an expert and might even serve to highlight the contradictions.

Be as transparent as possible and admit to any shortcomings in your research, but don't dwell on them for too long.

Ambiguous or unclear resolutions

The point of a research paper conclusion is to provide closure and bring all your ideas together. You should wrap up any arguments you introduced in the paper and tie up any loose ends, while demonstrating why your research and data are strong.

Use direct language in your conclusion and avoid ambiguity. Even if some of the data and sources you cite are inconclusive or contradictory, note this in your conclusion to come across as confident and trustworthy.

• Examples of research paper conclusions

Your research paper should provide a compelling close to the paper as a whole, highlighting your research and hard work. While the conclusion should represent your unique style, these examples offer a starting point:

Ultimately, the data we examined all point to the same conclusion: Encouraging a good work-life balance improves employee productivity and benefits the company overall. The research suggests that when employees feel their personal lives are valued and respected by their employers, they are more likely to be productive when at work. In addition, company turnover tends to be reduced when employees have a balance between their personal and professional lives. While additional research is required to establish ways companies can support employees in creating a stronger work-life balance, it's clear the need is there.

Social media is a primary method of communication among young people. As we've seen in the data presented, most young people in high school use a variety of social media applications at least every hour, including Instagram and Facebook. While social media is an avenue for connection with peers, research increasingly suggests that social media use correlates with body image issues. Young girls with lower self-esteem tend to use social media more often than those who don't log onto social media apps every day. As new applications continue to gain popularity, and as more high school students are given smartphones, more research will be required to measure the effects of prolonged social media use.

What are the different kinds of research paper conclusions?

There are no formal types of research paper conclusions. Ultimately, the conclusion depends on the outline of your paper and the type of research you’re presenting. While some experts note that research papers can end with a new perspective or commentary, most papers should conclude with a combination of both. The most important aspect of a good research paper conclusion is that it accurately represents the body of the paper.

Can I present new arguments in my research paper conclusion?

Research paper conclusions are not the place to introduce new data or arguments. The body of your paper is where you should share research and insights, where the reader is actively absorbing the content. By the time a reader reaches the conclusion of the research paper, they should have formed their opinion. Introducing new arguments in the conclusion can take a reader by surprise, and not in a positive way. It might also serve to frustrate readers.

How long should a research paper conclusion be?

There's no set length for a research paper conclusion. However, it's a good idea not to run on too long, since conclusions are supposed to be succinct. A good rule of thumb is to keep your conclusion around 5 to 10 percent of the paper's total length. If your paper is 10 pages, try to keep your conclusion under one page.

What should I include in a research paper conclusion?

A good research paper conclusion should always include a sense of urgency, so the reader can see how and why the topic should matter to them. You can also note some recommended actions to help fix the problem and some obstacles they might encounter. A conclusion should also remind the reader of the thesis statement, along with the main points you covered in the paper. At the end of the conclusion, add a powerful closing statement that helps cement the paper in the mind of the reader.

Get started today

Go from raw data to valuable insights with a flexible research platform

Editor’s picks

Last updated: 21 December 2023

Last updated: 16 December 2023

Last updated: 6 October 2023

Last updated: 17 February 2024

Last updated: 5 March 2024

Last updated: 19 November 2023

Last updated: 15 February 2024

Last updated: 11 March 2024

Last updated: 12 December 2023

Last updated: 6 March 2024

Last updated: 10 April 2023

Last updated: 20 December 2023

Latest articles

Related topics, log in or sign up.

Get started for free

How To Write The Conclusion Chapter

The what, why & how explained simply (with examples).

By: Jenna Crossley (PhD Cand). Reviewed By: Dr. Eunice Rautenbach | September 2021

So, you’ve wrapped up your results and discussion chapters, and you’re finally on the home stretch – the conclusion chapter . In this post, we’ll discuss everything you need to know to craft a high-quality conclusion chapter for your dissertation or thesis project.

Overview: Dissertation Conclusion Chapter

• What the thesis/dissertation conclusion chapter is
• What to include in your conclusion chapter
• How to structure and write up your conclusion chapter
• A few tips  to help you ace the chapter

What exactly is the conclusion chapter?

The conclusion chapter is typically the final major chapter of a dissertation or thesis. As such, it serves as a concluding summary of your research findings and wraps up the document. While some publications such as journal articles and research reports combine the discussion and conclusion sections, these are typically separate chapters in a dissertation or thesis. As always, be sure to check what your university’s structural preference is before you start writing up these chapters.

So, what’s the difference between the discussion and the conclusion chapter?

Well, the two chapters are quite similar , as they both discuss the key findings of the study. However, the conclusion chapter is typically more general and high-level in nature. In your discussion chapter, you’ll typically discuss the intricate details of your study, but in your conclusion chapter, you’ll take a   broader perspective, reporting on the main research outcomes and how these addressed your research aim (or aims) .

A core function of the conclusion chapter is to synthesise all major points covered in your study and to tell the reader what they should take away from your work. Basically, you need to tell them what you found , why it’s valuable , how it can be applied , and what further research can be done.

Whatever you do, don’t just copy and paste what you’ve written in your discussion chapter! The conclusion chapter should not be a simple rehash of the discussion chapter. While the two chapters are similar, they have distinctly different functions.  

What should I include in the conclusion chapter?

To understand what needs to go into your conclusion chapter, it’s useful to understand what the chapter needs to achieve. In general, a good dissertation conclusion chapter should achieve the following:

• Summarise the key findings of the study
• Explicitly answer the research question(s) and address the research aims
• Inform the reader of the study’s main contributions
• Discuss any limitations or weaknesses of the study
• Present recommendations for future research

Therefore, your conclusion chapter needs to cover these core components. Importantly, you need to be careful not to include any new findings or data points. Your conclusion chapter should be based purely on data and analysis findings that you’ve already presented in the earlier chapters. If there’s a new point you want to introduce, you’ll need to go back to your results and discussion chapters to weave the foundation in there.

In many cases, readers will jump from the introduction chapter directly to the conclusions chapter to get a quick overview of the study’s purpose and key findings. Therefore, when you write up your conclusion chapter, it’s useful to assume that the reader hasn’t consumed the inner chapters of your dissertation or thesis. In other words, craft your conclusion chapter such that there’s a strong connection and smooth flow between the introduction and conclusion chapters, even though they’re on opposite ends of your document.

Need a helping hand?

How to write the conclusion chapter

Now that you have a clearer view of what the conclusion chapter is about, let’s break down the structure of this chapter so that you can get writing. Keep in mind that this is merely a typical structure – it’s not set in stone or universal. Some universities will prefer that you cover some of these points in the discussion chapter , or that you cover the points at different levels in different chapters.

Step 1: Craft a brief introduction section

As with all chapters in your dissertation or thesis, the conclusions chapter needs to start with a brief introduction. In this introductory section, you’ll want to tell the reader what they can expect to find in the chapter, and in what order . Here’s an example of what this might look like:

This chapter will conclude the study by summarising the key research findings in relation to the research aims and questions and discussing the value and contribution thereof. It will also review the limitations of the study and propose opportunities for future research.

Importantly, the objective here is just to give the reader a taste of what’s to come (a roadmap of sorts), not a summary of the chapter. So, keep it short and sweet – a paragraph or two should be ample.

Step 2: Discuss the overall findings in relation to the research aims

The next step in writing your conclusions chapter is to discuss the overall findings of your study , as they relate to the research aims and research questions . You would have likely covered similar ground in the discussion chapter, so it’s important to zoom out a little bit here and focus on the broader findings – specifically, how these help address the research aims .

In practical terms, it’s useful to start this section by reminding your reader of your research aims and research questions, so that the findings are well contextualised. In this section, phrases such as, “This study aimed to…” and “the results indicate that…” will likely come in handy. For example, you could say something like the following:

This study aimed to investigate the feeding habits of the naked mole-rat. The results indicate that naked mole rats feed on underground roots and tubers. Further findings show that these creatures eat only a part of the plant, leaving essential parts to ensure long-term food stability.

Be careful not to make overly bold claims here. Avoid claims such as “this study proves that” or “the findings disprove existing the existing theory”. It’s seldom the case that a single study can prove or disprove something. Typically, this is achieved by a broader body of research, not a single study – especially not a dissertation or thesis which will inherently have significant and limitations. We’ll discuss those limitations a little later.

Step 3: Discuss how your study contributes to the field

Next, you’ll need to discuss how your research has contributed to the field – both in terms of theory and practice . This involves talking about what you achieved in your study, highlighting why this is important and valuable, and how it can be used or applied.

In this section you’ll want to:

• Mention any research outputs created as a result of your study (e.g., articles, publications, etc.)
• Inform the reader on just how your research solves your research problem , and why that matters
• Reflect on gaps in the existing research and discuss how your study contributes towards addressing these gaps
• Discuss your study in relation to relevant theories . For example, does it confirm these theories or constructively challenge them?
• Discuss how your research findings can be applied in the real world . For example, what specific actions can practitioners take, based on your findings?

Be careful to strike a careful balance between being firm but humble in your arguments here. It’s unlikely that your one study will fundamentally change paradigms or shake up the discipline, so making claims to this effect will be frowned upon . At the same time though, you need to present your arguments with confidence, firmly asserting the contribution your research has made, however small that contribution may be. Simply put, you need to keep it balanced .

Step 4: Reflect on the limitations of your study

Now that you’ve pumped your research up, the next step is to critically reflect on the limitations and potential shortcomings of your study. You may have already covered this in the discussion chapter, depending on your university’s structural preferences, so be careful not to repeat yourself unnecessarily.

There are many potential limitations that can apply to any given study. Some common ones include:

• Sampling issues that reduce the generalisability of the findings (e.g., non-probability sampling )
• Insufficient sample size (e.g., not getting enough survey responses ) or limited data access
• Low-resolution data collection or analysis techniques
• Researcher bias or lack of experience
• Lack of access to research equipment
• Time constraints that limit the methodology (e.g. cross-sectional vs longitudinal time horizon)
• Budget constraints that limit various aspects of the study

Discussing the limitations of your research may feel self-defeating (no one wants to highlight their weaknesses, right), but it’s a critical component of high-quality research. It’s important to appreciate that all studies have limitations (even well-funded studies by expert researchers) – therefore acknowledging these limitations adds credibility to your research by showing that you understand the limitations of your research design .

That being said, keep an eye on your wording and make sure that you don’t undermine your research . It’s important to strike a balance between recognising the limitations, but also highlighting the value of your research despite those limitations. Show the reader that you understand the limitations, that these were justified given your constraints, and that you know how they can be improved upon – this will get you marks.

Next, you’ll need to make recommendations for future studies. This will largely be built on the limitations you just discussed. For example, if one of your study’s weaknesses was related to a specific data collection or analysis method, you can make a recommendation that future researchers undertake similar research using a more sophisticated method.

Another potential source of future research recommendations is any data points or analysis findings that were interesting or surprising , but not directly related to your study’s research aims and research questions. So, if you observed anything that “stood out” in your analysis, but you didn’t explore it in your discussion (due to a lack of relevance to your research aims), you can earmark that for further exploration in this section.

Essentially, this section is an opportunity to outline how other researchers can build on your study to take the research further and help develop the body of knowledge. So, think carefully about the new questions that your study has raised, and clearly outline these for future researchers to pick up on.

Step 6: Wrap up with a closing summary

Quick tips for a top-notch conclusion chapter

Now that we’ve covered the what , why and how of the conclusion chapter, here are some quick tips and suggestions to help you craft a rock-solid conclusion.

• Don’t ramble . The conclusion chapter usually consumes 5-7% of the total word count (although this will vary between universities), so you need to be concise. Edit this chapter thoroughly with a focus on brevity and clarity.
• Be very careful about the claims you make in terms of your study’s contribution. Nothing will make the marker’s eyes roll back faster than exaggerated or unfounded claims. Be humble but firm in your claim-making.
• Use clear and simple language that can be easily understood by an intelligent layman. Remember that not every reader will be an expert in your field, so it’s important to make your writing accessible. Bear in mind that no one knows your research better than you do, so it’s important to spell things out clearly for readers.

Hopefully, this post has given you some direction and confidence to take on the conclusion chapter of your dissertation or thesis with confidence. If you’re still feeling a little shaky and need a helping hand, consider booking a free initial consultation with a friendly Grad Coach to discuss how we can help you with hands-on, private coaching.

Psst… there’s more (for free)

This post is part of our dissertation mini-course, which covers everything you need to get started with your dissertation, thesis or research project. 

You Might Also Like:

17 Comments

Really you team are doing great!

Your guide on writing the concluding chapter of a research is really informative especially to the beginners who really do not know where to start. Im now ready to start. Keep it up guys

Really your team are doing great!

Very helpful guidelines, timely saved. Thanks so much for the tips.

This post was very helpful and informative. Thank you team.

A very enjoyable, understandable and crisp presentation on how to write a conclusion chapter. I thoroughly enjoyed it. Thanks Jenna.

This was a very helpful article which really gave me practical pointers for my concluding chapter. Keep doing what you are doing! It meant a lot to me to be able to have this guide. Thank you so much.

Nice content dealing with the conclusion chapter, it’s a relief after the streneous task of completing discussion part.Thanks for valuable guidance

Thanks for your guidance

I get all my doubts clarified regarding the conclusion chapter. It’s really amazing. Many thanks.

Very helpful tips. Thanks so much for the guidance

Thank you very much for this piece. It offers a very helpful starting point in writing the conclusion chapter of my thesis.

It’s awesome! Most useful and timely too. Thanks a million times

Bundle of thanks for your guidance. It was greatly helpful.

Wonderful, clear, practical guidance. So grateful to read this as I conclude my research. Thank you.

Submit a Comment Cancel reply

Your email address will not be published. Required fields are marked *

Save my name, email, and website in this browser for the next time I comment.

• Print Friendly

• Privacy Policy

Buy Me a Coffee

Home » Research Results Section – Writing Guide and Examples

Research Results Section – Writing Guide and Examples

Table of Contents

Research Results

Research results refer to the findings and conclusions derived from a systematic investigation or study conducted to answer a specific question or hypothesis. These results are typically presented in a written report or paper and can include various forms of data such as numerical data, qualitative data, statistics, charts, graphs, and visual aids.

Results Section in Research

The results section of the research paper presents the findings of the study. It is the part of the paper where the researcher reports the data collected during the study and analyzes it to draw conclusions.

In the results section, the researcher should describe the data that was collected, the statistical analysis performed, and the findings of the study. It is important to be objective and not interpret the data in this section. Instead, the researcher should report the data as accurately and objectively as possible.

Structure of Research Results Section

The structure of the research results section can vary depending on the type of research conducted, but in general, it should contain the following components:

• Introduction: The introduction should provide an overview of the study, its aims, and its research questions. It should also briefly explain the methodology used to conduct the study.
• Data presentation : This section presents the data collected during the study. It may include tables, graphs, or other visual aids to help readers better understand the data. The data presented should be organized in a logical and coherent way, with headings and subheadings used to help guide the reader.
• Data analysis: In this section, the data presented in the previous section are analyzed and interpreted. The statistical tests used to analyze the data should be clearly explained, and the results of the tests should be presented in a way that is easy to understand.
• Discussion of results : This section should provide an interpretation of the results of the study, including a discussion of any unexpected findings. The discussion should also address the study’s research questions and explain how the results contribute to the field of study.
• Limitations: This section should acknowledge any limitations of the study, such as sample size, data collection methods, or other factors that may have influenced the results.
• Conclusions: The conclusions should summarize the main findings of the study and provide a final interpretation of the results. The conclusions should also address the study’s research questions and explain how the results contribute to the field of study.
• Recommendations : This section may provide recommendations for future research based on the study’s findings. It may also suggest practical applications for the study’s results in real-world settings.

Outline of Research Results Section

The following is an outline of the key components typically included in the Results section:

I. Introduction

• A brief overview of the research objectives and hypotheses
• A statement of the research question

II. Descriptive statistics

• Summary statistics (e.g., mean, standard deviation) for each variable analyzed
• Frequencies and percentages for categorical variables

III. Inferential statistics

• Results of statistical analyses, including tests of hypotheses
• Tables or figures to display statistical results

IV. Effect sizes and confidence intervals

• Effect sizes (e.g., Cohen’s d, odds ratio) to quantify the strength of the relationship between variables
• Confidence intervals to estimate the range of plausible values for the effect size

V. Subgroup analyses

• Results of analyses that examined differences between subgroups (e.g., by gender, age, treatment group)

VI. Limitations and assumptions

• Discussion of any limitations of the study and potential sources of bias
• Assumptions made in the statistical analyses

VII. Conclusions

• A summary of the key findings and their implications
• A statement of whether the hypotheses were supported or not
• Suggestions for future research

Example of Research Results Section

An Example of a Research Results Section could be:

• This study sought to examine the relationship between sleep quality and academic performance in college students.
• Hypothesis : College students who report better sleep quality will have higher GPAs than those who report poor sleep quality.
• Methodology : Participants completed a survey about their sleep habits and academic performance.

II. Participants

• Participants were college students (N=200) from a mid-sized public university in the United States.
• The sample was evenly split by gender (50% female, 50% male) and predominantly white (85%).
• Participants were recruited through flyers and online advertisements.

III. Results

• Participants who reported better sleep quality had significantly higher GPAs (M=3.5, SD=0.5) than those who reported poor sleep quality (M=2.9, SD=0.6).
• See Table 1 for a summary of the results.
• Participants who reported consistent sleep schedules had higher GPAs than those with irregular sleep schedules.

IV. Discussion

• The results support the hypothesis that better sleep quality is associated with higher academic performance in college students.
• These findings have implications for college students, as prioritizing sleep could lead to better academic outcomes.
• Limitations of the study include self-reported data and the lack of control for other variables that could impact academic performance.

V. Conclusion

• College students who prioritize sleep may see a positive impact on their academic performance.
• These findings highlight the importance of sleep in academic success.
• Future research could explore interventions to improve sleep quality in college students.

Example of Research Results in Research Paper :

Our study aimed to compare the performance of three different machine learning algorithms (Random Forest, Support Vector Machine, and Neural Network) in predicting customer churn in a telecommunications company. We collected a dataset of 10,000 customer records, with 20 predictor variables and a binary churn outcome variable.

Our analysis revealed that all three algorithms performed well in predicting customer churn, with an overall accuracy of 85%. However, the Random Forest algorithm showed the highest accuracy (88%), followed by the Support Vector Machine (86%) and the Neural Network (84%).

Furthermore, we found that the most important predictor variables for customer churn were monthly charges, contract type, and tenure. Random Forest identified monthly charges as the most important variable, while Support Vector Machine and Neural Network identified contract type as the most important.

Overall, our results suggest that machine learning algorithms can be effective in predicting customer churn in a telecommunications company, and that Random Forest is the most accurate algorithm for this task.

Example 3 :

Title : The Impact of Social Media on Body Image and Self-Esteem

Abstract : This study aimed to investigate the relationship between social media use, body image, and self-esteem among young adults. A total of 200 participants were recruited from a university and completed self-report measures of social media use, body image satisfaction, and self-esteem.

Results: The results showed that social media use was significantly associated with body image dissatisfaction and lower self-esteem. Specifically, participants who reported spending more time on social media platforms had lower levels of body image satisfaction and self-esteem compared to those who reported less social media use. Moreover, the study found that comparing oneself to others on social media was a significant predictor of body image dissatisfaction and lower self-esteem.

Conclusion : These results suggest that social media use can have negative effects on body image satisfaction and self-esteem among young adults. It is important for individuals to be mindful of their social media use and to recognize the potential negative impact it can have on their mental health. Furthermore, interventions aimed at promoting positive body image and self-esteem should take into account the role of social media in shaping these attitudes and behaviors.

Importance of Research Results

Research results are important for several reasons, including:

• Advancing knowledge: Research results can contribute to the advancement of knowledge in a particular field, whether it be in science, technology, medicine, social sciences, or humanities.
• Developing theories: Research results can help to develop or modify existing theories and create new ones.
• Improving practices: Research results can inform and improve practices in various fields, such as education, healthcare, business, and public policy.
• Identifying problems and solutions: Research results can identify problems and provide solutions to complex issues in society, including issues related to health, environment, social justice, and economics.
• Validating claims : Research results can validate or refute claims made by individuals or groups in society, such as politicians, corporations, or activists.
• Providing evidence: Research results can provide evidence to support decision-making, policy-making, and resource allocation in various fields.

How to Write Results in A Research Paper

Here are some general guidelines on how to write results in a research paper:

• Organize the results section: Start by organizing the results section in a logical and coherent manner. Divide the section into subsections if necessary, based on the research questions or hypotheses.
• Present the findings: Present the findings in a clear and concise manner. Use tables, graphs, and figures to illustrate the data and make the presentation more engaging.
• Describe the data: Describe the data in detail, including the sample size, response rate, and any missing data. Provide relevant descriptive statistics such as means, standard deviations, and ranges.
• Interpret the findings: Interpret the findings in light of the research questions or hypotheses. Discuss the implications of the findings and the extent to which they support or contradict existing theories or previous research.
• Discuss the limitations : Discuss the limitations of the study, including any potential sources of bias or confounding factors that may have affected the results.
• Compare the results : Compare the results with those of previous studies or theoretical predictions. Discuss any similarities, differences, or inconsistencies.
• Avoid redundancy: Avoid repeating information that has already been presented in the introduction or methods sections. Instead, focus on presenting new and relevant information.
• Be objective: Be objective in presenting the results, avoiding any personal biases or interpretations.

When to Write Research Results

Here are situations When to Write Research Results”

• After conducting research on the chosen topic and obtaining relevant data, organize the findings in a structured format that accurately represents the information gathered.
• Once the data has been analyzed and interpreted, and conclusions have been drawn, begin the writing process.
• Before starting to write, ensure that the research results adhere to the guidelines and requirements of the intended audience, such as a scientific journal or academic conference.
• Begin by writing an abstract that briefly summarizes the research question, methodology, findings, and conclusions.
• Follow the abstract with an introduction that provides context for the research, explains its significance, and outlines the research question and objectives.
• The next section should be a literature review that provides an overview of existing research on the topic and highlights the gaps in knowledge that the current research seeks to address.
• The methodology section should provide a detailed explanation of the research design, including the sample size, data collection methods, and analytical techniques used.
• Present the research results in a clear and concise manner, using graphs, tables, and figures to illustrate the findings.
• Discuss the implications of the research results, including how they contribute to the existing body of knowledge on the topic and what further research is needed.
• Conclude the paper by summarizing the main findings, reiterating the significance of the research, and offering suggestions for future research.

Purpose of Research Results

The purposes of Research Results are as follows:

• Informing policy and practice: Research results can provide evidence-based information to inform policy decisions, such as in the fields of healthcare, education, and environmental regulation. They can also inform best practices in fields such as business, engineering, and social work.
• Addressing societal problems : Research results can be used to help address societal problems, such as reducing poverty, improving public health, and promoting social justice.
• Generating economic benefits : Research results can lead to the development of new products, services, and technologies that can create economic value and improve quality of life.
• Supporting academic and professional development : Research results can be used to support academic and professional development by providing opportunities for students, researchers, and practitioners to learn about new findings and methodologies in their field.
• Enhancing public understanding: Research results can help to educate the public about important issues and promote scientific literacy, leading to more informed decision-making and better public policy.
• Evaluating interventions: Research results can be used to evaluate the effectiveness of interventions, such as treatments, educational programs, and social policies. This can help to identify areas where improvements are needed and guide future interventions.
• Contributing to scientific progress: Research results can contribute to the advancement of science by providing new insights and discoveries that can lead to new theories, methods, and techniques.
• Informing decision-making : Research results can provide decision-makers with the information they need to make informed decisions. This can include decision-making at the individual, organizational, or governmental levels.
• Fostering collaboration : Research results can facilitate collaboration between researchers and practitioners, leading to new partnerships, interdisciplinary approaches, and innovative solutions to complex problems.

Advantages of Research Results

Some Advantages of Research Results are as follows:

• Improved decision-making: Research results can help inform decision-making in various fields, including medicine, business, and government. For example, research on the effectiveness of different treatments for a particular disease can help doctors make informed decisions about the best course of treatment for their patients.
• Innovation : Research results can lead to the development of new technologies, products, and services. For example, research on renewable energy sources can lead to the development of new and more efficient ways to harness renewable energy.
• Economic benefits: Research results can stimulate economic growth by providing new opportunities for businesses and entrepreneurs. For example, research on new materials or manufacturing techniques can lead to the development of new products and processes that can create new jobs and boost economic activity.
• Improved quality of life: Research results can contribute to improving the quality of life for individuals and society as a whole. For example, research on the causes of a particular disease can lead to the development of new treatments and cures, improving the health and well-being of millions of people.

About the author

Muhammad Hassan

Researcher, Academic Writer, Web developer

You may also like

How to Cite Research Paper – All Formats and...

Data Collection – Methods Types and Examples

Delimitations in Research – Types, Examples and...

Research Paper Format – Types, Examples and...

Research Process – Steps, Examples and Tips

Research Design – Types, Methods and Examples

• Affiliate Program

• UNITED STATES
• 台灣 (TAIWAN)
• TÜRKIYE (TURKEY)
• Academic Editing Services
• - Research Paper
• - Journal Manuscript
• - Dissertation
• - College & University Assignments
• Admissions Editing Services
• - Application Essay
• - Personal Statement
• - Recommendation Letter
• - Cover Letter
• - CV/Resume
• Business Editing Services
• - Business Documents
• - Report & Brochure
• - Website & Blog
• Writer Editing Services
• - Script & Screenplay
• Our Editors
• Client Reviews
• Editing & Proofreading Prices
• Wordvice Points
• Partner Discount
• Plagiarism Checker
• APA Citation Generator
• MLA Citation Generator
• Chicago Citation Generator
• Vancouver Citation Generator
• - APA Style
• - MLA Style
• - Chicago Style
• - Vancouver Style
• Writing & Editing Guide
• Academic Resources
• Admissions Resources

How to Write a Research Paper Conclusion Section

What is a conclusion in a research paper?

The conclusion in a research paper is the final paragraph or two in a research paper. In scientific papers, the conclusion usually follows the Discussion section , summarizing the importance of the findings and reminding the reader why the work presented in the paper is relevant.

However, it can be a bit confusing to distinguish the conclusion section/paragraph from a summary or a repetition of your findings, your own opinion, or the statement of the implications of your work. In fact, the conclusion should contain a bit of all of these other parts but go beyond it—but not too far beyond! 

The structure and content of the conclusion section can also vary depending on whether you are writing a research manuscript or an essay. This article will explain how to write a good conclusion section, what exactly it should (and should not) contain, how it should be structured, and what you should avoid when writing it.  

Table of Contents:

What does a good conclusion section do, what to include in a research paper conclusion.

• Conclusion in an Essay
• Research Paper Conclusion 
• Conclusion Paragraph Outline and Example
• What Not to Do When Writing a Conclusion

The conclusion of a research paper has several key objectives. It should:

• Restate your research problem addressed in the introduction section
• Summarize your main arguments, important findings, and broader implications
• Synthesize key takeaways from your study

The specific content in the conclusion depends on whether your paper presents the results of original scientific research or constructs an argument through engagement with previously published sources.

You presented your general field of study to the reader in the introduction section, by moving from general information (the background of your work, often combined with a literature review ) to the rationale of your study and then to the specific problem or topic you addressed, formulated in the form of the statement of the problem in research or the thesis statement in an essay.

In the conclusion section, in contrast, your task is to move from your specific findings or arguments back to a more general depiction of how your research contributes to the readers’ understanding of a certain concept or helps solve a practical problem, or fills an important gap in the literature. The content of your conclusion section depends on the type of research you are doing and what type of paper you are writing. But whatever the outcome of your work is, the conclusion is where you briefly summarize it and place it within a larger context. It could be called the “take-home message” of the entire paper.

What to summarize in the conclusion

Your conclusion section needs to contain a very brief summary of your work , a very brief summary of the main findings of your work, and a mention of anything else that seems relevant when you now look at your work from a bigger perspective, even if it was not initially listed as one of your main research questions. This could be a limitation, for example, a problem with the design of your experiment that either needs to be considered when drawing any conclusions or that led you to ask a different question and therefore draw different conclusions at the end of your study (compared to when you started out).

Once you have reminded the reader of what you did and what you found, you need to go beyond that and also provide either your own opinion on why your work is relevant (and for whom, and how) or theoretical or practical implications of the study , or make a specific call for action if there is one to be made.   

How to Write an Essay Conclusion

Academic essays follow quite different structures than their counterparts in STEM and the natural sciences. Humanities papers often have conclusion sections that are much longer and contain more detail than scientific papers. There are three main types of academic essay conclusions.

Summarizing conclusion

The most typical conclusion at the end of an analytical/explanatory/argumentative essay is a summarizing conclusion . This is, as the name suggests, a clear summary of the main points of your topic and thesis. Since you might have gone through a number of different arguments or subtopics in the main part of your essay, you need to remind the reader again what those were, how they fit into each other, and how they helped you develop or corroborate your hypothesis.

For an essay that analyzes how recruiters can hire the best candidates in the shortest time or on “how starving yourself will increase your lifespan, according to science”, a summary of all the points you discussed might be all you need. Note that you should not exactly repeat what you said earlier, but rather highlight the essential details and present those to your reader in a different way. 

Externalizing conclusion

If you think that just reminding the reader of your main points is not enough, you can opt for an externalizing conclusion instead, that presents new points that were not presented in the paper so far. These new points can be additional facts and information or they can be ideas that are relevant to the topic and have not been mentioned before.

Such a conclusion can stimulate your readers to think about your topic or the implications of your analysis in a whole new way. For example, at the end of a historical analysis of a specific event or development, you could direct your reader’s attention to some current events that were not the topic of your essay but that provide a different context for your findings.

Editorial conclusion

In an editorial conclusion , another common type of conclusion that you will find at the end of papers and essays, you do not add new information but instead present your own experiences or opinions on the topic to round everything up. What makes this type of conclusion interesting is that you can choose to agree or disagree with the information you presented in your paper so far. For example, if you have collected and analyzed information on how a specific diet helps people lose weight, you can nevertheless have your doubts on the sustainability of that diet or its practicability in real life—if such arguments were not included in your original thesis and have therefore not been covered in the main part of your paper, the conclusion section is the place where you can get your opinion across.    

How to Conclude an Empirical Research Paper

An empirical research paper is usually more concise and succinct than an essay, because, if it is written well, it focuses on one specific question, describes the method that was used to answer that one question, describes and explains the results, and guides the reader in a logical way from the introduction to the discussion without going on tangents or digging into not absolutely relevant topics.

Summarize the findings

In a scientific paper, you should include a summary of the findings. Don’t go into great detail here (you will have presented your in-depth  results  and  discussion  already), but do clearly express the answers to the  research questions  you investigated.

Describe your main findings, even if they weren’t necessarily the ones anticipated, and explain the conclusion they led you to. Explain these findings in as few words as possible.

Instead of beginning with “ In conclusion, in this study, we investigated the effect of stress on the brain using fMRI …”, you should try to find a way to incorporate the repetition of the essential (and only the essential) details into the summary of the key points. “ The findings of this fMRI study on the effect of stress on the brain suggest that …” or “ While it has been known for a long time that stress has an effect on the brain, the findings of this fMRI study show that, surprisingly… ” would be better ways to start a conclusion. 

You should also not bring up new ideas or present new facts in the conclusion of a research paper, but stick to the background information you have presented earlier, to the findings you have already discussed, and the limitations and implications you have already described. The one thing you can add here is a practical recommendation that you haven’t clearly stated before—but even that one needs to follow logically from everything you have already discussed in the discussion section.

Discuss the implications

After summing up your key arguments or findings, conclude the paper by stating the broader implications of the research , whether in methods , approach, or findings. Express practical or theoretical takeaways from your paper. This often looks like a “call to action” or a final “sales pitch” that puts an exclamation point on your paper.

If your research topic is more theoretical in nature, your closing statement should express the significance of your argument—for example, in proposing a new understanding of a topic or laying the groundwork for future research.

Future research example

Future research into education standards should focus on establishing a more detailed picture of how novel pedagogical approaches impact young people’s ability to absorb new and difficult concepts. Moreover, observational studies are needed to gain more insight into how specific teaching models affect the retention of relationships and facts—for instance, how inquiry-based learning and its emphasis on lateral thinking can be used as a jumping-off point for more holistic classroom approaches.

Research Conclusion Example and Outline

Let’s revisit the study on the effect of stress on the brain we mentioned before and see what the common structure for a conclusion paragraph looks like, in three steps. Following these simple steps will make it easy for you to wrap everything up in one short paragraph that contains all the essential information: 

One: Short summary of what you did, but integrated into the summary of your findings:

While it has been known for a long time that stress has an effect on the brain, the findings of this fMRI study in 25 university students going through mid-term exams show that, surprisingly, one’s attitude to the experienced stress significantly modulates the brain’s response to it. 

Note that you don’t need to repeat any methodological or technical details here—the reader has been presented with all of these before, they have read your results section and the discussion of your results, and even (hopefully!) a discussion of the limitations and strengths of your paper. The only thing you need to remind them of here is the essential outcome of your work. 

Two: Add implications, and don’t forget to specify who this might be relevant for: 

Students could be considered a specific subsample of the general population, but earlier research shows that the effect that exam stress has on their physical and mental health is comparable to the effects of other types of stress on individuals of other ages and occupations. Further research into practical ways of modulating not only one’s mental stress response but potentially also one’s brain activity (e.g., via neurofeedback training) are warranted.

This is a “research implication”, and it is nicely combined with a mention of a potential limitation of the study (the student sample) that turns out not to be a limitation after all (because earlier research suggests we can generalize to other populations). If there already is a lot of research on neurofeedback for stress control, by the way, then this should have been discussed in your discussion section earlier and you wouldn’t say such studies are “warranted” here but rather specify how your findings could inspire specific future experiments or how they should be implemented in existing applications. 

Three: The most important thing is that your conclusion paragraph accurately reflects the content of your paper. Compare it to your research paper title , your research paper abstract , and to your journal submission cover letter , in case you already have one—if these do not all tell the same story, then you need to go back to your paper, start again from the introduction section, and find out where you lost the logical thread. As always, consistency is key.    

Problems to Avoid When Writing a Conclusion 

• Do not suddenly introduce new information that has never been mentioned before (unless you are writing an essay and opting for an externalizing conclusion, see above). The conclusion section is not where you want to surprise your readers, but the take-home message of what you have already presented.
• Do not simply copy your abstract, the conclusion section of your abstract, or the first sentence of your introduction, and put it at the end of the discussion section. Even if these parts of your paper cover the same points, they should not be identical.
• Do not start the conclusion with “In conclusion”. If it has its own section heading, that is redundant, and if it is the last paragraph of the discussion section, it is inelegant and also not really necessary. The reader expects you to wrap your work up in the last paragraph, so you don’t have to announce that. Just look at the above example to see how to start a conclusion in a natural way.
• Do not forget what your research objectives were and how you initially formulated the statement of the problem in your introduction section. If your story/approach/conclusions changed because of methodological issues or information you were not aware of when you started, then make sure you go back to the beginning and adapt your entire story (not just the ending). 

Consider Receiving Academic Editing Services

When you have arrived at the conclusion of your paper, you might want to head over to Wordvice AI’s AI Writing Assistant to receive a free grammar check for any academic content. 

After drafting, you can also receive English editing and proofreading services , including paper editing services for your journal manuscript. If you need advice on how to write the other parts of your research paper , or on how to make a research paper outline if you are struggling with putting everything you did together, then head over to the Wordvice academic resources pages , where we have a lot more articles and videos for you.

• U.S. Locations
• UMGC Europe
• Learn Online
• Find Answers
• 855-655-8682
• Current Students

Online Guide to Writing and Research

The research process, explore more of umgc.

• Online Guide to Writing

Planning and Writing a Research Paper

Draw Conclusions

As a writer, you are presenting your viewpoint, opinions, evidence, etc. for others to review, so you must take on this task with maturity, courage and thoughtfulness.  Remember, you are adding to the discourse community with every research paper that you write.  This is a privilege and an opportunity to share your point of view with the world at large in an academic setting.

Because research generates further research, the conclusions you draw from your research are important. As a researcher, you depend on the integrity of the research that precedes your own efforts, and researchers depend on each other to draw valid conclusions. 

To test the validity of your conclusions, you will have to review both the content of your paper and the way in which you arrived at the content. You may ask yourself questions, such as the ones presented below, to detect any weak areas in your paper, so you can then make those areas stronger.  Notice that some of the questions relate to your process, others to your sources, and others to how you arrived at your conclusions.

Checklist for Evaluating Your Conclusions

Key takeaways.

• Because research generates further research, the conclusions you draw from your research are important.
• To test the validity of your conclusions, you will have to review both the content of your paper and the way in which you arrived at the content.

Mailing Address: 3501 University Blvd. East, Adelphi, MD 20783 This work is licensed under a  Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License . © 2022 UMGC. All links to external sites were verified at the time of publication. UMGC is not responsible for the validity or integrity of information located at external sites.

Table of Contents: Online Guide to Writing

Chapter 1: College Writing

How Does College Writing Differ from Workplace Writing?

What Is College Writing?

Why So Much Emphasis on Writing?

Chapter 2: The Writing Process

Doing Exploratory Research

Getting from Notes to Your Draft

Introduction

Prewriting - Techniques to Get Started - Mining Your Intuition

Prewriting: Targeting Your Audience

Prewriting: Techniques to Get Started

Prewriting: Understanding Your Assignment

Rewriting: Being Your Own Critic

Rewriting: Creating a Revision Strategy

Rewriting: Getting Feedback

Rewriting: The Final Draft

Techniques to Get Started - Outlining

Techniques to Get Started - Using Systematic Techniques

Thesis Statement and Controlling Idea

Writing: Getting from Notes to Your Draft - Freewriting

Writing: Getting from Notes to Your Draft - Summarizing Your Ideas

Writing: Outlining What You Will Write

Chapter 3: Thinking Strategies

A Word About Style, Voice, and Tone

A Word About Style, Voice, and Tone: Style Through Vocabulary and Diction

Critical Strategies and Writing

Critical Strategies and Writing: Analysis

Critical Strategies and Writing: Evaluation

Critical Strategies and Writing: Persuasion

Critical Strategies and Writing: Synthesis

Developing a Paper Using Strategies

Kinds of Assignments You Will Write

Patterns for Presenting Information

Patterns for Presenting Information: Critiques

Patterns for Presenting Information: Discussing Raw Data

Patterns for Presenting Information: General-to-Specific Pattern

Patterns for Presenting Information: Problem-Cause-Solution Pattern

Patterns for Presenting Information: Specific-to-General Pattern

Patterns for Presenting Information: Summaries and Abstracts

Supporting with Research and Examples

Writing Essay Examinations

Writing Essay Examinations: Make Your Answer Relevant and Complete

Writing Essay Examinations: Organize Thinking Before Writing

Writing Essay Examinations: Read and Understand the Question

Chapter 4: The Research Process

Planning and Writing a Research Paper: Ask a Research Question

Planning and Writing a Research Paper: Cite Sources

Planning and Writing a Research Paper: Collect Evidence

Planning and Writing a Research Paper: Decide Your Point of View, or Role, for Your Research

Planning and Writing a Research Paper: Draw Conclusions

Planning and Writing a Research Paper: Find a Topic and Get an Overview

Planning and Writing a Research Paper: Manage Your Resources

Planning and Writing a Research Paper: Outline

Planning and Writing a Research Paper: Survey the Literature

Planning and Writing a Research Paper: Work Your Sources into Your Research Writing

Research Resources: Where Are Research Resources Found? - Human Resources

Research Resources: What Are Research Resources?

Research Resources: Where Are Research Resources Found?

Research Resources: Where Are Research Resources Found? - Electronic Resources

Research Resources: Where Are Research Resources Found? - Print Resources

Structuring the Research Paper: Formal Research Structure

Structuring the Research Paper: Informal Research Structure

The Nature of Research

The Research Assignment: How Should Research Sources Be Evaluated?

The Research Assignment: When Is Research Needed?

The Research Assignment: Why Perform Research?

Chapter 5: Academic Integrity

Academic Integrity

Giving Credit to Sources

Giving Credit to Sources: Copyright Laws

Giving Credit to Sources: Documentation

Giving Credit to Sources: Style Guides

Integrating Sources

Practicing Academic Integrity

Practicing Academic Integrity: Keeping Accurate Records

Practicing Academic Integrity: Managing Source Material

Practicing Academic Integrity: Managing Source Material - Paraphrasing Your Source

Practicing Academic Integrity: Managing Source Material - Quoting Your Source

Practicing Academic Integrity: Managing Source Material - Summarizing Your Sources

Types of Documentation

Types of Documentation: Bibliographies and Source Lists

Types of Documentation: Citing World Wide Web Sources

Types of Documentation: In-Text or Parenthetical Citations

Types of Documentation: In-Text or Parenthetical Citations - APA Style

Types of Documentation: In-Text or Parenthetical Citations - CSE/CBE Style

Types of Documentation: In-Text or Parenthetical Citations - Chicago Style

Types of Documentation: In-Text or Parenthetical Citations - MLA Style

Types of Documentation: Note Citations

Chapter 6: Using Library Resources

Finding Library Resources

Chapter 7: Assessing Your Writing

How Is Writing Graded?

How Is Writing Graded?: A General Assessment Tool

The Draft Stage

The Draft Stage: The First Draft

The Draft Stage: The Revision Process and the Final Draft

The Draft Stage: Using Feedback

The Research Stage

Using Assessment to Improve Your Writing

Chapter 8: Other Frequently Assigned Papers

Reviews and Reaction Papers: Article and Book Reviews

Reviews and Reaction Papers: Reaction Papers

Writing Arguments

Writing Arguments: Adapting the Argument Structure

Writing Arguments: Purposes of Argument

Writing Arguments: References to Consult for Writing Arguments

Writing Arguments: Steps to Writing an Argument - Anticipate Active Opposition

Writing Arguments: Steps to Writing an Argument - Determine Your Organization

Writing Arguments: Steps to Writing an Argument - Develop Your Argument

Writing Arguments: Steps to Writing an Argument - Introduce Your Argument

Writing Arguments: Steps to Writing an Argument - State Your Thesis or Proposition

Writing Arguments: Steps to Writing an Argument - Write Your Conclusion

Writing Arguments: Types of Argument

Appendix A: Books to Help Improve Your Writing

Dictionaries

General Style Manuals

Researching on the Internet

Special Style Manuals

Writing Handbooks

Appendix B: Collaborative Writing and Peer Reviewing

Collaborative Writing: Assignments to Accompany the Group Project

Collaborative Writing: Informal Progress Report

Collaborative Writing: Issues to Resolve

Collaborative Writing: Methodology

Collaborative Writing: Peer Evaluation

Collaborative Writing: Tasks of Collaborative Writing Group Members

Collaborative Writing: Writing Plan

General Introduction

Peer Reviewing

Appendix C: Developing an Improvement Plan

Working with Your Instructor’s Comments and Grades

Appendix D: Writing Plan and Project Schedule

Devising a Writing Project Plan and Schedule

Reviewing Your Plan with Others

By using our website you agree to our use of cookies. Learn more about how we use cookies by reading our  Privacy Policy .

Jump to navigation

Cochrane Training

Chapter 15: interpreting results and drawing conclusions.

Holger J Schünemann, Gunn E Vist, Julian PT Higgins, Nancy Santesso, Jonathan J Deeks, Paul Glasziou, Elie A Akl, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group

Key Points:

• This chapter provides guidance on interpreting the results of synthesis in order to communicate the conclusions of the review effectively.
• Methods are presented for computing, presenting and interpreting relative and absolute effects for dichotomous outcome data, including the number needed to treat (NNT).
• For continuous outcome measures, review authors can present summary results for studies using natural units of measurement or as minimal important differences when all studies use the same scale. When studies measure the same construct but with different scales, review authors will need to find a way to interpret the standardized mean difference, or to use an alternative effect measure for the meta-analysis such as the ratio of means.
• Review authors should not describe results as ‘statistically significant’, ‘not statistically significant’ or ‘non-significant’ or unduly rely on thresholds for P values, but report the confidence interval together with the exact P value.
• Review authors should not make recommendations about healthcare decisions, but they can – after describing the certainty of evidence and the balance of benefits and harms – highlight different actions that might be consistent with particular patterns of values and preferences and other factors that determine a decision such as cost.

Cite this chapter as: Schünemann HJ, Vist GE, Higgins JPT, Santesso N, Deeks JJ, Glasziou P, Akl EA, Guyatt GH. Chapter 15: Interpreting results and drawing conclusions. In: Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, Welch VA (editors). Cochrane Handbook for Systematic Reviews of Interventions version 6.4 (updated August 2023). Cochrane, 2023. Available from www.training.cochrane.org/handbook .

15.1 Introduction

The purpose of Cochrane Reviews is to facilitate healthcare decisions by patients and the general public, clinicians, guideline developers, administrators and policy makers. They also inform future research. A clear statement of findings, a considered discussion and a clear presentation of the authors’ conclusions are, therefore, important parts of the review. In particular, the following issues can help people make better informed decisions and increase the usability of Cochrane Reviews:

• information on all important outcomes, including adverse outcomes;
• the certainty of the evidence for each of these outcomes, as it applies to specific populations and specific interventions; and
• clarification of the manner in which particular values and preferences may bear on the desirable and undesirable consequences of the intervention.

A ‘Summary of findings’ table, described in Chapter 14 , Section 14.1 , provides key pieces of information about health benefits and harms in a quick and accessible format. It is highly desirable that review authors include a ‘Summary of findings’ table in Cochrane Reviews alongside a sufficient description of the studies and meta-analyses to support its contents. This description includes the rating of the certainty of evidence, also called the quality of the evidence or confidence in the estimates of the effects, which is expected in all Cochrane Reviews.

‘Summary of findings’ tables are usually supported by full evidence profiles which include the detailed ratings of the evidence (Guyatt et al 2011a, Guyatt et al 2013a, Guyatt et al 2013b, Santesso et al 2016). The Discussion section of the text of the review provides space to reflect and consider the implications of these aspects of the review’s findings. Cochrane Reviews include five standard subheadings to ensure the Discussion section places the review in an appropriate context: ‘Summary of main results (benefits and harms)’; ‘Potential biases in the review process’; ‘Overall completeness and applicability of evidence’; ‘Certainty of the evidence’; and ‘Agreements and disagreements with other studies or reviews’. Following the Discussion, the Authors’ conclusions section is divided into two standard subsections: ‘Implications for practice’ and ‘Implications for research’. The assessment of the certainty of evidence facilitates a structured description of the implications for practice and research.

Because Cochrane Reviews have an international audience, the Discussion and Authors’ conclusions should, so far as possible, assume a broad international perspective and provide guidance for how the results could be applied in different settings, rather than being restricted to specific national or local circumstances. Cultural differences and economic differences may both play an important role in determining the best course of action based on the results of a Cochrane Review. Furthermore, individuals within societies have widely varying values and preferences regarding health states, and use of societal resources to achieve particular health states. For all these reasons, and because information that goes beyond that included in a Cochrane Review is required to make fully informed decisions, different people will often make different decisions based on the same evidence presented in a review.

Thus, review authors should avoid specific recommendations that inevitably depend on assumptions about available resources, values and preferences, and other factors such as equity considerations, feasibility and acceptability of an intervention. The purpose of the review should be to present information and aid interpretation rather than to offer recommendations. The discussion and conclusions should help people understand the implications of the evidence in relation to practical decisions and apply the results to their specific situation. Review authors can aid this understanding of the implications by laying out different scenarios that describe certain value structures.

In this chapter, we address first one of the key aspects of interpreting findings that is also fundamental in completing a ‘Summary of findings’ table: the certainty of evidence related to each of the outcomes. We then provide a more detailed consideration of issues around applicability and around interpretation of numerical results, and provide suggestions for presenting authors’ conclusions.

15.2 Issues of indirectness and applicability

15.2.1 the role of the review author.

“A leap of faith is always required when applying any study findings to the population at large” or to a specific person. “In making that jump, one must always strike a balance between making justifiable broad generalizations and being too conservative in one’s conclusions” (Friedman et al 1985). In addition to issues about risk of bias and other domains determining the certainty of evidence, this leap of faith is related to how well the identified body of evidence matches the posed PICO ( Population, Intervention, Comparator(s) and Outcome ) question. As to the population, no individual can be entirely matched to the population included in research studies. At the time of decision, there will always be differences between the study population and the person or population to whom the evidence is applied; sometimes these differences are slight, sometimes large.

The terms applicability, generalizability, external validity and transferability are related, sometimes used interchangeably and have in common that they lack a clear and consistent definition in the classic epidemiological literature (Schünemann et al 2013). However, all of the terms describe one overarching theme: whether or not available research evidence can be directly used to answer the health and healthcare question at hand, ideally supported by a judgement about the degree of confidence in this use (Schünemann et al 2013). GRADE’s certainty domains include a judgement about ‘indirectness’ to describe all of these aspects including the concept of direct versus indirect comparisons of different interventions (Atkins et al 2004, Guyatt et al 2008, Guyatt et al 2011b).

To address adequately the extent to which a review is relevant for the purpose to which it is being put, there are certain things the review author must do, and certain things the user of the review must do to assess the degree of indirectness. Cochrane and the GRADE Working Group suggest using a very structured framework to address indirectness. We discuss here and in Chapter 14 what the review author can do to help the user. Cochrane Review authors must be extremely clear on the population, intervention and outcomes that they intend to address. Chapter 14, Section 14.1.2 , also emphasizes a crucial step: the specification of all patient-important outcomes relevant to the intervention strategies under comparison.

In considering whether the effect of an intervention applies equally to all participants, and whether different variations on the intervention have similar effects, review authors need to make a priori hypotheses about possible effect modifiers, and then examine those hypotheses (see Chapter 10, Section 10.10 and Section 10.11 ). If they find apparent subgroup effects, they must ultimately decide whether or not these effects are credible (Sun et al 2012). Differences between subgroups, particularly those that correspond to differences between studies, should be interpreted cautiously. Some chance variation between subgroups is inevitable so, unless there is good reason to believe that there is an interaction, review authors should not assume that the subgroup effect exists. If, despite due caution, review authors judge subgroup effects in terms of relative effect estimates as credible (i.e. the effects differ credibly), they should conduct separate meta-analyses for the relevant subgroups, and produce separate ‘Summary of findings’ tables for those subgroups.

The user of the review will be challenged with ‘individualization’ of the findings, whether they seek to apply the findings to an individual patient or a policy decision in a specific context. For example, even if relative effects are similar across subgroups, absolute effects will differ according to baseline risk. Review authors can help provide this information by identifying identifiable groups of people with varying baseline risks in the ‘Summary of findings’ tables, as discussed in Chapter 14, Section 14.1.3 . Users can then identify their specific case or population as belonging to a particular risk group, if relevant, and assess their likely magnitude of benefit or harm accordingly. A description of the identifying prognostic or baseline risk factors in a brief scenario (e.g. age or gender) will help users of a review further.

Another decision users must make is whether their individual case or population of interest is so different from those included in the studies that they cannot use the results of the systematic review and meta-analysis at all. Rather than rigidly applying the inclusion and exclusion criteria of studies, it is better to ask whether or not there are compelling reasons why the evidence should not be applied to a particular patient. Review authors can sometimes help decision makers by identifying important variation where divergence might limit the applicability of results (Rothwell 2005, Schünemann et al 2006, Guyatt et al 2011b, Schünemann et al 2013), including biologic and cultural variation, and variation in adherence to an intervention.

In addressing these issues, review authors cannot be aware of, or address, the myriad of differences in circumstances around the world. They can, however, address differences of known importance to many people and, importantly, they should avoid assuming that other people’s circumstances are the same as their own in discussing the results and drawing conclusions.

15.2.2 Biological variation

Issues of biological variation that may affect the applicability of a result to a reader or population include divergence in pathophysiology (e.g. biological differences between women and men that may affect responsiveness to an intervention) and divergence in a causative agent (e.g. for infectious diseases such as malaria, which may be caused by several different parasites). The discussion of the results in the review should make clear whether the included studies addressed all or only some of these groups, and whether any important subgroup effects were found.

15.2.3 Variation in context

Some interventions, particularly non-pharmacological interventions, may work in some contexts but not in others; the situation has been described as program by context interaction (Hawe et al 2004). Contextual factors might pertain to the host organization in which an intervention is offered, such as the expertise, experience and morale of the staff expected to carry out the intervention, the competing priorities for the clinician’s or staff’s attention, the local resources such as service and facilities made available to the program and the status or importance given to the program by the host organization. Broader context issues might include aspects of the system within which the host organization operates, such as the fee or payment structure for healthcare providers and the local insurance system. Some interventions, in particular complex interventions (see Chapter 17 ), can be only partially implemented in some contexts, and this requires judgements about indirectness of the intervention and its components for readers in that context (Schünemann 2013).

Contextual factors may also pertain to the characteristics of the target group or population, such as cultural and linguistic diversity, socio-economic position, rural/urban setting. These factors may mean that a particular style of care or relationship evolves between service providers and consumers that may or may not match the values and technology of the program.

For many years these aspects have been acknowledged when decision makers have argued that results of evidence reviews from other countries do not apply in their own country or setting. Whilst some programmes/interventions have been successfully transferred from one context to another, others have not (Resnicow et al 1993, Lumley et al 2004, Coleman et al 2015). Review authors should be cautious when making generalizations from one context to another. They should report on the presence (or otherwise) of context-related information in intervention studies, where this information is available.

15.2.4 Variation in adherence

Variation in the adherence of the recipients and providers of care can limit the certainty in the applicability of results. Predictable differences in adherence can be due to divergence in how recipients of care perceive the intervention (e.g. the importance of side effects), economic conditions or attitudes that make some forms of care inaccessible in some settings, such as in low-income countries (Dans et al 2007). It should not be assumed that high levels of adherence in closely monitored randomized trials will translate into similar levels of adherence in normal practice.

15.2.5 Variation in values and preferences

Decisions about healthcare management strategies and options involve trading off health benefits and harms. The right choice may differ for people with different values and preferences (i.e. the importance people place on the outcomes and interventions), and it is important that decision makers ensure that decisions are consistent with a patient or population’s values and preferences. The importance placed on outcomes, together with other factors, will influence whether the recipients of care will or will not accept an option that is offered (Alonso-Coello et al 2016) and, thus, can be one factor influencing adherence. In Section 15.6 , we describe how the review author can help this process and the limits of supporting decision making based on intervention reviews.

15.3 Interpreting results of statistical analyses

15.3.1 confidence intervals.

Results for both individual studies and meta-analyses are reported with a point estimate together with an associated confidence interval. For example, ‘The odds ratio was 0.75 with a 95% confidence interval of 0.70 to 0.80’. The point estimate (0.75) is the best estimate of the magnitude and direction of the experimental intervention’s effect compared with the comparator intervention. The confidence interval describes the uncertainty inherent in any estimate, and describes a range of values within which we can be reasonably sure that the true effect actually lies. If the confidence interval is relatively narrow (e.g. 0.70 to 0.80), the effect size is known precisely. If the interval is wider (e.g. 0.60 to 0.93) the uncertainty is greater, although there may still be enough precision to make decisions about the utility of the intervention. Intervals that are very wide (e.g. 0.50 to 1.10) indicate that we have little knowledge about the effect and this imprecision affects our certainty in the evidence, and that further information would be needed before we could draw a more certain conclusion.

A 95% confidence interval is often interpreted as indicating a range within which we can be 95% certain that the true effect lies. This statement is a loose interpretation, but is useful as a rough guide. The strictly correct interpretation of a confidence interval is based on the hypothetical notion of considering the results that would be obtained if the study were repeated many times. If a study were repeated infinitely often, and on each occasion a 95% confidence interval calculated, then 95% of these intervals would contain the true effect (see Section 15.3.3 for further explanation).

The width of the confidence interval for an individual study depends to a large extent on the sample size. Larger studies tend to give more precise estimates of effects (and hence have narrower confidence intervals) than smaller studies. For continuous outcomes, precision depends also on the variability in the outcome measurements (i.e. how widely individual results vary between people in the study, measured as the standard deviation); for dichotomous outcomes it depends on the risk of the event (more frequent events allow more precision, and narrower confidence intervals), and for time-to-event outcomes it also depends on the number of events observed. All these quantities are used in computation of the standard errors of effect estimates from which the confidence interval is derived.

The width of a confidence interval for a meta-analysis depends on the precision of the individual study estimates and on the number of studies combined. In addition, for random-effects models, precision will decrease with increasing heterogeneity and confidence intervals will widen correspondingly (see Chapter 10, Section 10.10.4 ). As more studies are added to a meta-analysis the width of the confidence interval usually decreases. However, if the additional studies increase the heterogeneity in the meta-analysis and a random-effects model is used, it is possible that the confidence interval width will increase.

Confidence intervals and point estimates have different interpretations in fixed-effect and random-effects models. While the fixed-effect estimate and its confidence interval address the question ‘what is the best (single) estimate of the effect?’, the random-effects estimate assumes there to be a distribution of effects, and the estimate and its confidence interval address the question ‘what is the best estimate of the average effect?’ A confidence interval may be reported for any level of confidence (although they are most commonly reported for 95%, and sometimes 90% or 99%). For example, the odds ratio of 0.80 could be reported with an 80% confidence interval of 0.73 to 0.88; a 90% interval of 0.72 to 0.89; and a 95% interval of 0.70 to 0.92. As the confidence level increases, the confidence interval widens.

There is logical correspondence between the confidence interval and the P value (see Section 15.3.3 ). The 95% confidence interval for an effect will exclude the null value (such as an odds ratio of 1.0 or a risk difference of 0) if and only if the test of significance yields a P value of less than 0.05. If the P value is exactly 0.05, then either the upper or lower limit of the 95% confidence interval will be at the null value. Similarly, the 99% confidence interval will exclude the null if and only if the test of significance yields a P value of less than 0.01.

Together, the point estimate and confidence interval provide information to assess the effects of the intervention on the outcome. For example, suppose that we are evaluating an intervention that reduces the risk of an event and we decide that it would be useful only if it reduced the risk of an event from 30% by at least 5 percentage points to 25% (these values will depend on the specific clinical scenario and outcomes, including the anticipated harms). If the meta-analysis yielded an effect estimate of a reduction of 10 percentage points with a tight 95% confidence interval, say, from 7% to 13%, we would be able to conclude that the intervention was useful since both the point estimate and the entire range of the interval exceed our criterion of a reduction of 5% for net health benefit. However, if the meta-analysis reported the same risk reduction of 10% but with a wider interval, say, from 2% to 18%, although we would still conclude that our best estimate of the intervention effect is that it provides net benefit, we could not be so confident as we still entertain the possibility that the effect could be between 2% and 5%. If the confidence interval was wider still, and included the null value of a difference of 0%, we would still consider the possibility that the intervention has no effect on the outcome whatsoever, and would need to be even more sceptical in our conclusions.

Review authors may use the same general approach to conclude that an intervention is not useful. Continuing with the above example where the criterion for an important difference that should be achieved to provide more benefit than harm is a 5% risk difference, an effect estimate of 2% with a 95% confidence interval of 1% to 4% suggests that the intervention does not provide net health benefit.

15.3.2 P values and statistical significance

A P value is the standard result of a statistical test, and is the probability of obtaining the observed effect (or larger) under a ‘null hypothesis’. In the context of Cochrane Reviews there are two commonly used statistical tests. The first is a test of overall effect (a Z-test), and its null hypothesis is that there is no overall effect of the experimental intervention compared with the comparator on the outcome of interest. The second is the (Chi 2 ) test for heterogeneity, and its null hypothesis is that there are no differences in the intervention effects across studies.

A P value that is very small indicates that the observed effect is very unlikely to have arisen purely by chance, and therefore provides evidence against the null hypothesis. It has been common practice to interpret a P value by examining whether it is smaller than particular threshold values. In particular, P values less than 0.05 are often reported as ‘statistically significant’, and interpreted as being small enough to justify rejection of the null hypothesis. However, the 0.05 threshold is an arbitrary one that became commonly used in medical and psychological research largely because P values were determined by comparing the test statistic against tabulations of specific percentage points of statistical distributions. If review authors decide to present a P value with the results of a meta-analysis, they should report a precise P value (as calculated by most statistical software), together with the 95% confidence interval. Review authors should not describe results as ‘statistically significant’, ‘not statistically significant’ or ‘non-significant’ or unduly rely on thresholds for P values , but report the confidence interval together with the exact P value (see MECIR Box 15.3.a ).

We discuss interpretation of the test for heterogeneity in Chapter 10, Section 10.10.2 ; the remainder of this section refers mainly to tests for an overall effect. For tests of an overall effect, the computation of P involves both the effect estimate and precision of the effect estimate (driven largely by sample size). As precision increases, the range of plausible effects that could occur by chance is reduced. Correspondingly, the statistical significance of an effect of a particular magnitude will usually be greater (the P value will be smaller) in a larger study than in a smaller study.

P values are commonly misinterpreted in two ways. First, a moderate or large P value (e.g. greater than 0.05) may be misinterpreted as evidence that the intervention has no effect on the outcome. There is an important difference between this statement and the correct interpretation that there is a high probability that the observed effect on the outcome is due to chance alone. To avoid such a misinterpretation, review authors should always examine the effect estimate and its 95% confidence interval.

The second misinterpretation is to assume that a result with a small P value for the summary effect estimate implies that an experimental intervention has an important benefit. Such a misinterpretation is more likely to occur in large studies and meta-analyses that accumulate data over dozens of studies and thousands of participants. The P value addresses the question of whether the experimental intervention effect is precisely nil; it does not examine whether the effect is of a magnitude of importance to potential recipients of the intervention. In a large study, a small P value may represent the detection of a trivial effect that may not lead to net health benefit when compared with the potential harms (i.e. harmful effects on other important outcomes). Again, inspection of the point estimate and confidence interval helps correct interpretations (see Section 15.3.1 ).

MECIR Box 15.3.a Relevant expectations for conduct of intervention reviews

15.3.3 Relation between confidence intervals, statistical significance and certainty of evidence

The confidence interval (and imprecision) is only one domain that influences overall uncertainty about effect estimates. Uncertainty resulting from imprecision (i.e. statistical uncertainty) may be no less important than uncertainty from indirectness, or any other GRADE domain, in the context of decision making (Schünemann 2016). Thus, the extent to which interpretations of the confidence interval described in Sections 15.3.1 and 15.3.2 correspond to conclusions about overall certainty of the evidence for the outcome of interest depends on these other domains. If there are no concerns about other domains that determine the certainty of the evidence (i.e. risk of bias, inconsistency, indirectness or publication bias), then the interpretation in Sections 15.3.1 and 15.3.2 . about the relation of the confidence interval to the true effect may be carried forward to the overall certainty. However, if there are concerns about the other domains that affect the certainty of the evidence, the interpretation about the true effect needs to be seen in the context of further uncertainty resulting from those concerns.

For example, nine randomized controlled trials in almost 6000 cancer patients indicated that the administration of heparin reduces the risk of venous thromboembolism (VTE), with a risk ratio of 43% (95% CI 19% to 60%) (Akl et al 2011a). For patients with a plausible baseline risk of approximately 4.6% per year, this relative effect suggests that heparin leads to an absolute risk reduction of 20 fewer VTEs (95% CI 9 fewer to 27 fewer) per 1000 people per year (Akl et al 2011a). Now consider that the review authors or those applying the evidence in a guideline have lowered the certainty in the evidence as a result of indirectness. While the confidence intervals would remain unchanged, the certainty in that confidence interval and in the point estimate as reflecting the truth for the question of interest will be lowered. In fact, the certainty range will have unknown width so there will be unknown likelihood of a result within that range because of this indirectness. The lower the certainty in the evidence, the less we know about the width of the certainty range, although methods for quantifying risk of bias and understanding potential direction of bias may offer insight when lowered certainty is due to risk of bias. Nevertheless, decision makers must consider this uncertainty, and must do so in relation to the effect measure that is being evaluated (e.g. a relative or absolute measure). We will describe the impact on interpretations for dichotomous outcomes in Section 15.4 .

15.4 Interpreting results from dichotomous outcomes (including numbers needed to treat)

15.4.1 relative and absolute risk reductions.

Clinicians may be more inclined to prescribe an intervention that reduces the relative risk of death by 25% than one that reduces the risk of death by 1 percentage point, although both presentations of the evidence may relate to the same benefit (i.e. a reduction in risk from 4% to 3%). The former refers to the relative reduction in risk and the latter to the absolute reduction in risk. As described in Chapter 6, Section 6.4.1 , there are several measures for comparing dichotomous outcomes in two groups. Meta-analyses are usually undertaken using risk ratios (RR), odds ratios (OR) or risk differences (RD), but there are several alternative ways of expressing results.

Relative risk reduction (RRR) is a convenient way of re-expressing a risk ratio as a percentage reduction:

For example, a risk ratio of 0.75 translates to a relative risk reduction of 25%, as in the example above.

The risk difference is often referred to as the absolute risk reduction (ARR) or absolute risk increase (ARI), and may be presented as a percentage (e.g. 1%), as a decimal (e.g. 0.01), or as account (e.g. 10 out of 1000). We consider different choices for presenting absolute effects in Section 15.4.3 . We then describe computations for obtaining these numbers from the results of individual studies and of meta-analyses in Section 15.4.4 .

15.4.2 Number needed to treat (NNT)

The number needed to treat (NNT) is a common alternative way of presenting information on the effect of an intervention. The NNT is defined as the expected number of people who need to receive the experimental rather than the comparator intervention for one additional person to either incur or avoid an event (depending on the direction of the result) in a given time frame. Thus, for example, an NNT of 10 can be interpreted as ‘it is expected that one additional (or less) person will incur an event for every 10 participants receiving the experimental intervention rather than comparator over a given time frame’. It is important to be clear that:

• since the NNT is derived from the risk difference, it is still a comparative measure of effect (experimental versus a specific comparator) and not a general property of a single intervention; and
• the NNT gives an ‘expected value’. For example, NNT = 10 does not imply that one additional event will occur in each and every group of 10 people.

NNTs can be computed for both beneficial and detrimental events, and for interventions that cause both improvements and deteriorations in outcomes. In all instances NNTs are expressed as positive whole numbers. Some authors use the term ‘number needed to harm’ (NNH) when an intervention leads to an adverse outcome, or a decrease in a positive outcome, rather than improvement. However, this phrase can be misleading (most notably, it can easily be read to imply the number of people who will experience a harmful outcome if given the intervention), and it is strongly recommended that ‘number needed to harm’ and ‘NNH’ are avoided. The preferred alternative is to use phrases such as ‘number needed to treat for an additional beneficial outcome’ (NNTB) and ‘number needed to treat for an additional harmful outcome’ (NNTH) to indicate direction of effect.

As NNTs refer to events, their interpretation needs to be worded carefully when the binary outcome is a dichotomization of a scale-based outcome. For example, if the outcome is pain measured on a ‘none, mild, moderate or severe’ scale it may have been dichotomized as ‘none or mild’ versus ‘moderate or severe’. It would be inappropriate for an NNT from these data to be referred to as an ‘NNT for pain’. It is an ‘NNT for moderate or severe pain’.

We consider different choices for presenting absolute effects in Section 15.4.3 . We then describe computations for obtaining these numbers from the results of individual studies and of meta-analyses in Section 15.4.4 .

15.4.3 Expressing risk differences

Users of reviews are liable to be influenced by the choice of statistical presentations of the evidence. Hoffrage and colleagues suggest that physicians’ inferences about statistical outcomes are more appropriate when they deal with ‘natural frequencies’ – whole numbers of people, both treated and untreated (e.g. treatment results in a drop from 20 out of 1000 to 10 out of 1000 women having breast cancer) – than when effects are presented as percentages (e.g. 1% absolute reduction in breast cancer risk) (Hoffrage et al 2000). Probabilities may be more difficult to understand than frequencies, particularly when events are rare. While standardization may be important in improving the presentation of research evidence (and participation in healthcare decisions), current evidence suggests that the presentation of natural frequencies for expressing differences in absolute risk is best understood by consumers of healthcare information (Akl et al 2011b). This evidence provides the rationale for presenting absolute risks in ‘Summary of findings’ tables as numbers of people with events per 1000 people receiving the intervention (see Chapter 14 ).

RRs and RRRs remain crucial because relative effects tend to be substantially more stable across risk groups than absolute effects (see Chapter 10, Section 10.4.3 ). Review authors can use their own data to study this consistency (Cates 1999, Smeeth et al 1999). Risk differences from studies are least likely to be consistent across baseline event rates; thus, they are rarely appropriate for computing numbers needed to treat in systematic reviews. If a relative effect measure (OR or RR) is chosen for meta-analysis, then a comparator group risk needs to be specified as part of the calculation of an RD or NNT. In addition, if there are several different groups of participants with different levels of risk, it is crucial to express absolute benefit for each clinically identifiable risk group, clarifying the time period to which this applies. Studies in patients with differing severity of disease, or studies with different lengths of follow-up will almost certainly have different comparator group risks. In these cases, different comparator group risks lead to different RDs and NNTs (except when the intervention has no effect). A recommended approach is to re-express an odds ratio or a risk ratio as a variety of RD or NNTs across a range of assumed comparator risks (ACRs) (McQuay and Moore 1997, Smeeth et al 1999). Review authors should bear these considerations in mind not only when constructing their ‘Summary of findings’ table, but also in the text of their review.

For example, a review of oral anticoagulants to prevent stroke presented information to users by describing absolute benefits for various baseline risks (Aguilar and Hart 2005, Aguilar et al 2007). They presented their principal findings as “The inherent risk of stroke should be considered in the decision to use oral anticoagulants in atrial fibrillation patients, selecting those who stand to benefit most for this therapy” (Aguilar and Hart 2005). Among high-risk atrial fibrillation patients with prior stroke or transient ischaemic attack who have stroke rates of about 12% (120 per 1000) per year, warfarin prevents about 70 strokes yearly per 1000 patients, whereas for low-risk atrial fibrillation patients (with a stroke rate of about 2% per year or 20 per 1000), warfarin prevents only 12 strokes. This presentation helps users to understand the important impact that typical baseline risks have on the absolute benefit that they can expect.

15.4.4 Computations

Direct computation of risk difference (RD) or a number needed to treat (NNT) depends on the summary statistic (odds ratio, risk ratio or risk differences) available from the study or meta-analysis. When expressing results of meta-analyses, review authors should use, in the computations, whatever statistic they determined to be the most appropriate summary for meta-analysis (see Chapter 10, Section 10.4.3 ). Here we present calculations to obtain RD as a reduction in the number of participants per 1000. For example, a risk difference of –0.133 corresponds to 133 fewer participants with the event per 1000.

RDs and NNTs should not be computed from the aggregated total numbers of participants and events across the trials. This approach ignores the randomization within studies, and may produce seriously misleading results if there is unbalanced randomization in any of the studies. Using the pooled result of a meta-analysis is more appropriate. When computing NNTs, the values obtained are by convention always rounded up to the next whole number.

15.4.4.1 Computing NNT from a risk difference (RD)

A NNT may be computed from a risk difference as

where the vertical bars (‘absolute value of’) in the denominator indicate that any minus sign should be ignored. It is convention to round the NNT up to the nearest whole number. For example, if the risk difference is –0.12 the NNT is 9; if the risk difference is –0.22 the NNT is 5. Cochrane Review authors should qualify the NNT as referring to benefit (improvement) or harm by denoting the NNT as NNTB or NNTH. Note that this approach, although feasible, should be used only for the results of a meta-analysis of risk differences. In most cases meta-analyses will be undertaken using a relative measure of effect (RR or OR), and those statistics should be used to calculate the NNT (see Section 15.4.4.2 and 15.4.4.3 ).

15.4.4.2 Computing risk differences or NNT from a risk ratio

To aid interpretation of the results of a meta-analysis of risk ratios, review authors may compute an absolute risk reduction or NNT. In order to do this, an assumed comparator risk (ACR) (otherwise known as a baseline risk, or risk that the outcome of interest would occur with the comparator intervention) is required. It will usually be appropriate to do this for a range of different ACRs. The computation proceeds as follows:

As an example, suppose the risk ratio is RR = 0.92, and an ACR = 0.3 (300 per 1000) is assumed. Then the effect on risk is 24 fewer per 1000:

The NNT is 42:

15.4.4.3 Computing risk differences or NNT from an odds ratio

Review authors may wish to compute a risk difference or NNT from the results of a meta-analysis of odds ratios. In order to do this, an ACR is required. It will usually be appropriate to do this for a range of different ACRs. The computation proceeds as follows:

As an example, suppose the odds ratio is OR = 0.73, and a comparator risk of ACR = 0.3 is assumed. Then the effect on risk is 62 fewer per 1000:

The NNT is 17:

15.4.4.4 Computing risk ratio from an odds ratio

Because risk ratios are easier to interpret than odds ratios, but odds ratios have favourable mathematical properties, a review author may decide to undertake a meta-analysis based on odds ratios, but to express the result as a summary risk ratio (or relative risk reduction). This requires an ACR. Then

It will often be reasonable to perform this transformation using the median comparator group risk from the studies in the meta-analysis.

15.4.4.5 Computing confidence limits

Confidence limits for RDs and NNTs may be calculated by applying the above formulae to the upper and lower confidence limits for the summary statistic (RD, RR or OR) (Altman 1998). Note that this confidence interval does not incorporate uncertainty around the ACR.

If the 95% confidence interval of OR or RR includes the value 1, one of the confidence limits will indicate benefit and the other harm. Thus, appropriate use of the words ‘fewer’ and ‘more’ is required for each limit when presenting results in terms of events. For NNTs, the two confidence limits should be labelled as NNTB and NNTH to indicate the direction of effect in each case. The confidence interval for the NNT will include a ‘discontinuity’, because increasingly smaller risk differences that approach zero will lead to NNTs approaching infinity. Thus, the confidence interval will include both an infinitely large NNTB and an infinitely large NNTH.

15.5 Interpreting results from continuous outcomes (including standardized mean differences)

15.5.1 meta-analyses with continuous outcomes.

Review authors should describe in the study protocol how they plan to interpret results for continuous outcomes. When outcomes are continuous, review authors have a number of options to present summary results. These options differ if studies report the same measure that is familiar to the target audiences, studies report the same or very similar measures that are less familiar to the target audiences, or studies report different measures.

15.5.2 Meta-analyses with continuous outcomes using the same measure

If all studies have used the same familiar units, for instance, results are expressed as durations of events, such as symptoms for conditions including diarrhoea, sore throat, otitis media, influenza or duration of hospitalization, a meta-analysis may generate a summary estimate in those units, as a difference in mean response (see, for instance, the row summarizing results for duration of diarrhoea in Chapter 14, Figure 14.1.b and the row summarizing oedema in Chapter 14, Figure 14.1.a ). For such outcomes, the ‘Summary of findings’ table should include a difference of means between the two interventions. However, when units of such outcomes may be difficult to interpret, particularly when they relate to rating scales (again, see the oedema row of Chapter 14, Figure 14.1.a ). ‘Summary of findings’ tables should include the minimum and maximum of the scale of measurement, and the direction. Knowledge of the smallest change in instrument score that patients perceive is important – the minimal important difference (MID) – and can greatly facilitate the interpretation of results (Guyatt et al 1998, Schünemann and Guyatt 2005). Knowing the MID allows review authors and users to place results in context. Review authors should state the MID – if known – in the Comments column of their ‘Summary of findings’ table. For example, the chronic respiratory questionnaire has possible scores in health-related quality of life ranging from 1 to 7 and 0.5 represents a well-established MID (Jaeschke et al 1989, Schünemann et al 2005).

15.5.3 Meta-analyses with continuous outcomes using different measures

When studies have used different instruments to measure the same construct, a standardized mean difference (SMD) may be used in meta-analysis for combining continuous data. Without guidance, clinicians and patients may have little idea how to interpret results presented as SMDs. Review authors should therefore consider issues of interpretability when planning their analysis at the protocol stage and should consider whether there will be suitable ways to re-express the SMD or whether alternative effect measures, such as a ratio of means, or possibly as minimal important difference units (Guyatt et al 2013b) should be used. Table 15.5.a and the following sections describe these options.

Table 15.5.a Approaches and their implications to presenting results of continuous variables when primary studies have used different instruments to measure the same construct. Adapted from Guyatt et al (2013b)

15.5.3.1 Presenting and interpreting SMDs using generic effect size estimates

The SMD expresses the intervention effect in standard units rather than the original units of measurement. The SMD is the difference in mean effects between the experimental and comparator groups divided by the pooled standard deviation of participants’ outcomes, or external SDs when studies are very small (see Chapter 6, Section 6.5.1.2 ). The value of a SMD thus depends on both the size of the effect (the difference between means) and the standard deviation of the outcomes (the inherent variability among participants or based on an external SD).

If review authors use the SMD, they might choose to present the results directly as SMDs (row 1a, Table 15.5.a and Table 15.5.b ). However, absolute values of the intervention and comparison groups are typically not useful because studies have used different measurement instruments with different units. Guiding rules for interpreting SMDs (or ‘Cohen’s effect sizes’) exist, and have arisen mainly from researchers in the social sciences (Cohen 1988). One example is as follows: 0.2 represents a small effect, 0.5 a moderate effect and 0.8 a large effect (Cohen 1988). Variations exist (e.g. <0.40=small, 0.40 to 0.70=moderate, >0.70=large). Review authors might consider including such a guiding rule in interpreting the SMD in the text of the review, and in summary versions such as the Comments column of a ‘Summary of findings’ table. However, some methodologists believe that such interpretations are problematic because patient importance of a finding is context-dependent and not amenable to generic statements.

15.5.3.2 Re-expressing SMDs using a familiar instrument

The second possibility for interpreting the SMD is to express it in the units of one or more of the specific measurement instruments used by the included studies (row 1b, Table 15.5.a and Table 15.5.b ). The approach is to calculate an absolute difference in means by multiplying the SMD by an estimate of the SD associated with the most familiar instrument. To obtain this SD, a reasonable option is to calculate a weighted average across all intervention groups of all studies that used the selected instrument (preferably a pre-intervention or post-intervention SD as discussed in Chapter 10, Section 10.5.2 ). To better reflect among-person variation in practice, or to use an instrument not represented in the meta-analysis, it may be preferable to use a standard deviation from a representative observational study. The summary effect is thus re-expressed in the original units of that particular instrument and the clinical relevance and impact of the intervention effect can be interpreted using that familiar instrument.

The same approach of re-expressing the results for a familiar instrument can also be used for other standardized effect measures such as when standardizing by MIDs (Guyatt et al 2013b): see Section 15.5.3.5 .

Table 15.5.b Application of approaches when studies have used different measures: effects of dexamethasone for pain after laparoscopic cholecystectomy (Karanicolas et al 2008). Reproduced with permission of Wolters Kluwer

1 Certainty rated according to GRADE from very low to high certainty. 2 Substantial unexplained heterogeneity in study results. 3 Imprecision due to wide confidence intervals. 4 The 20% comes from the proportion in the control group requiring rescue analgesia. 5 Crude (arithmetic) means of the post-operative pain mean responses across all five trials when transformed to a 100-point scale.

15.5.3.3 Re-expressing SMDs through dichotomization and transformation to relative and absolute measures

A third approach (row 1c, Table 15.5.a and Table 15.5.b ) relies on converting the continuous measure into a dichotomy and thus allows calculation of relative and absolute effects on a binary scale. A transformation of a SMD to a (log) odds ratio is available, based on the assumption that an underlying continuous variable has a logistic distribution with equal standard deviation in the two intervention groups, as discussed in Chapter 10, Section 10.6  (Furukawa 1999, Guyatt et al 2013b). The assumption is unlikely to hold exactly and the results must be regarded as an approximation. The log odds ratio is estimated as

(or approximately 1.81✕SMD). The resulting odds ratio can then be presented as normal, and in a ‘Summary of findings’ table, combined with an assumed comparator group risk to be expressed as an absolute risk difference. The comparator group risk in this case would refer to the proportion of people who have achieved a specific value of the continuous outcome. In randomized trials this can be interpreted as the proportion who have improved by some (specified) amount (responders), for instance by 5 points on a 0 to 100 scale. Table 15.5.c shows some illustrative results from this method. The risk differences can then be converted to NNTs or to people per thousand using methods described in Section 15.4.4 .

Table 15.5.c Risk difference derived for specific SMDs for various given ‘proportions improved’ in the comparator group (Furukawa 1999, Guyatt et al 2013b). Reproduced with permission of Elsevier 

15.5.3.4 Ratio of means

A more frequently used approach is based on calculation of a ratio of means between the intervention and comparator groups (Friedrich et al 2008) as discussed in Chapter 6, Section 6.5.1.3 . Interpretational advantages of this approach include the ability to pool studies with outcomes expressed in different units directly, to avoid the vulnerability of heterogeneous populations that limits approaches that rely on SD units, and for ease of clinical interpretation (row 2, Table 15.5.a and Table 15.5.b ). This method is currently designed for post-intervention scores only. However, it is possible to calculate a ratio of change scores if both intervention and comparator groups change in the same direction in each relevant study, and this ratio may sometimes be informative.

Limitations to this approach include its limited applicability to change scores (since it is unlikely that both intervention and comparator group changes are in the same direction in all studies) and the possibility of misleading results if the comparator group mean is very small, in which case even a modest difference from the intervention group will yield a large and therefore misleading ratio of means. It also requires that separate ratios of means be calculated for each included study, and then entered into a generic inverse variance meta-analysis (see Chapter 10, Section 10.3 ).

The ratio of means approach illustrated in Table 15.5.b suggests a relative reduction in pain of only 13%, meaning that those receiving steroids have a pain severity 87% of those in the comparator group, an effect that might be considered modest.

15.5.3.5 Presenting continuous results as minimally important difference units

To express results in MID units, review authors have two options. First, they can be combined across studies in the same way as the SMD, but instead of dividing the mean difference of each study by its SD, review authors divide by the MID associated with that outcome (Johnston et al 2010, Guyatt et al 2013b). Instead of SD units, the pooled results represent MID units (row 3, Table 15.5.a and Table 15.5.b ), and may be more easily interpretable. This approach avoids the problem of varying SDs across studies that may distort estimates of effect in approaches that rely on the SMD. The approach, however, relies on having well-established MIDs. The approach is also risky in that a difference less than the MID may be interpreted as trivial when a substantial proportion of patients may have achieved an important benefit.

The other approach makes a simple conversion (not shown in Table 15.5.b ), before undertaking the meta-analysis, of the means and SDs from each study to means and SDs on the scale of a particular familiar instrument whose MID is known. For example, one can rescale the mean and SD of other chronic respiratory disease instruments (e.g. rescaling a 0 to 100 score of an instrument) to a the 1 to 7 score in Chronic Respiratory Disease Questionnaire (CRQ) units (by assuming 0 equals 1 and 100 equals 7 on the CRQ). Given the MID of the CRQ of 0.5, a mean difference in change of 0.71 after rescaling of all studies suggests a substantial effect of the intervention (Guyatt et al 2013b). This approach, presenting in units of the most familiar instrument, may be the most desirable when the target audiences have extensive experience with that instrument, particularly if the MID is well established.

15.6 Drawing conclusions

15.6.1 conclusions sections of a cochrane review.

Authors’ conclusions in a Cochrane Review are divided into implications for practice and implications for research. While Cochrane Reviews about interventions can provide meaningful information and guidance for practice, decisions about the desirable and undesirable consequences of healthcare options require evidence and judgements for criteria that most Cochrane Reviews do not provide (Alonso-Coello et al 2016). In describing the implications for practice and the development of recommendations, however, review authors may consider the certainty of the evidence, the balance of benefits and harms, and assumed values and preferences.

15.6.2 Implications for practice

Drawing conclusions about the practical usefulness of an intervention entails making trade-offs, either implicitly or explicitly, between the estimated benefits, harms and the values and preferences. Making such trade-offs, and thus making specific recommendations for an action in a specific context, goes beyond a Cochrane Review and requires additional evidence and informed judgements that most Cochrane Reviews do not provide (Alonso-Coello et al 2016). Such judgements are typically the domain of clinical practice guideline developers for which Cochrane Reviews will provide crucial information (Graham et al 2011, Schünemann et al 2014, Zhang et al 2018a). Thus, authors of Cochrane Reviews should not make recommendations.

If review authors feel compelled to lay out actions that clinicians and patients could take, they should – after describing the certainty of evidence and the balance of benefits and harms – highlight different actions that might be consistent with particular patterns of values and preferences. Other factors that might influence a decision should also be highlighted, including any known factors that would be expected to modify the effects of the intervention, the baseline risk or status of the patient, costs and who bears those costs, and the availability of resources. Review authors should ensure they consider all patient-important outcomes, including those for which limited data may be available. In the context of public health reviews the focus may be on population-important outcomes as the target may be an entire (non-diseased) population and include outcomes that are not measured in the population receiving an intervention (e.g. a reduction of transmission of infections from those receiving an intervention). This process implies a high level of explicitness in judgements about values or preferences attached to different outcomes and the certainty of the related evidence (Zhang et al 2018b, Zhang et al 2018c); this and a full cost-effectiveness analysis is beyond the scope of most Cochrane Reviews (although they might well be used for such analyses; see Chapter 20 ).

A review on the use of anticoagulation in cancer patients to increase survival (Akl et al 2011a) provides an example for laying out clinical implications for situations where there are important trade-offs between desirable and undesirable effects of the intervention: “The decision for a patient with cancer to start heparin therapy for survival benefit should balance the benefits and downsides and integrate the patient’s values and preferences. Patients with a high preference for a potential survival prolongation, limited aversion to potential bleeding, and who do not consider heparin (both UFH or LMWH) therapy a burden may opt to use heparin, while those with aversion to bleeding may not.”

15.6.3 Implications for research

The second category for authors’ conclusions in a Cochrane Review is implications for research. To help people make well-informed decisions about future healthcare research, the ‘Implications for research’ section should comment on the need for further research, and the nature of the further research that would be most desirable. It is helpful to consider the population, intervention, comparison and outcomes that could be addressed, or addressed more effectively in the future, in the context of the certainty of the evidence in the current review (Brown et al 2006):

• P (Population): diagnosis, disease stage, comorbidity, risk factor, sex, age, ethnic group, specific inclusion or exclusion criteria, clinical setting;
• I (Intervention): type, frequency, dose, duration, prognostic factor;
• C (Comparison): placebo, routine care, alternative treatment/management;
• O (Outcome): which clinical or patient-related outcomes will the researcher need to measure, improve, influence or accomplish? Which methods of measurement should be used?

While Cochrane Review authors will find the PICO domains helpful, the domains of the GRADE certainty framework further support understanding and describing what additional research will improve the certainty in the available evidence. Note that as the certainty of the evidence is likely to vary by outcome, these implications will be specific to certain outcomes in the review. Table 15.6.a shows how review authors may be aided in their interpretation of the body of evidence and drawing conclusions about future research and practice.

Table 15.6.a Implications for research and practice suggested by individual GRADE domains

The review of compression stockings for prevention of deep vein thrombosis (DVT) in airline passengers described in Chapter 14 provides an example where there is some convincing evidence of a benefit of the intervention: “This review shows that the question of the effects on symptomless DVT of wearing versus not wearing compression stockings in the types of people studied in these trials should now be regarded as answered. Further research may be justified to investigate the relative effects of different strengths of stockings or of stockings compared to other preventative strategies. Further randomised trials to address the remaining uncertainty about the effects of wearing versus not wearing compression stockings on outcomes such as death, pulmonary embolism and symptomatic DVT would need to be large.” (Clarke et al 2016).

A review of therapeutic touch for anxiety disorder provides an example of the implications for research when no eligible studies had been found: “This review highlights the need for randomized controlled trials to evaluate the effectiveness of therapeutic touch in reducing anxiety symptoms in people diagnosed with anxiety disorders. Future trials need to be rigorous in design and delivery, with subsequent reporting to include high quality descriptions of all aspects of methodology to enable appraisal and interpretation of results.” (Robinson et al 2007).

15.6.4 Reaching conclusions

A common mistake is to confuse ‘no evidence of an effect’ with ‘evidence of no effect’. When the confidence intervals are too wide (e.g. including no effect), it is wrong to claim that the experimental intervention has ‘no effect’ or is ‘no different’ from the comparator intervention. Review authors may also incorrectly ‘positively’ frame results for some effects but not others. For example, when the effect estimate is positive for a beneficial outcome but confidence intervals are wide, review authors may describe the effect as promising. However, when the effect estimate is negative for an outcome that is considered harmful but the confidence intervals include no effect, review authors report no effect. Another mistake is to frame the conclusion in wishful terms. For example, review authors might write, “there were too few people in the analysis to detect a reduction in mortality” when the included studies showed a reduction or even increase in mortality that was not ‘statistically significant’. One way of avoiding errors such as these is to consider the results blinded; that is, consider how the results would be presented and framed in the conclusions if the direction of the results was reversed. If the confidence interval for the estimate of the difference in the effects of the interventions overlaps with no effect, the analysis is compatible with both a true beneficial effect and a true harmful effect. If one of the possibilities is mentioned in the conclusion, the other possibility should be mentioned as well. Table 15.6.b suggests narrative statements for drawing conclusions based on the effect estimate from the meta-analysis and the certainty of the evidence.

Table 15.6.b Suggested narrative statements for phrasing conclusions

Another common mistake is to reach conclusions that go beyond the evidence. Often this is done implicitly, without referring to the additional information or judgements that are used in reaching conclusions about the implications of a review for practice. Even when additional information and explicit judgements support conclusions about the implications of a review for practice, review authors rarely conduct systematic reviews of the additional information. Furthermore, implications for practice are often dependent on specific circumstances and values that must be taken into consideration. As we have noted, review authors should always be cautious when drawing conclusions about implications for practice and they should not make recommendations.

15.7 Chapter information

Authors: Holger J Schünemann, Gunn E Vist, Julian PT Higgins, Nancy Santesso, Jonathan J Deeks, Paul Glasziou, Elie Akl, Gordon H Guyatt; on behalf of the Cochrane GRADEing Methods Group

Acknowledgements: Andrew Oxman, Jonathan Sterne, Michael Borenstein and Rob Scholten contributed text to earlier versions of this chapter.

Funding: This work was in part supported by funding from the Michael G DeGroote Cochrane Canada Centre and the Ontario Ministry of Health. JJD receives support from the National Institute for Health Research (NIHR) Birmingham Biomedical Research Centre at the University Hospitals Birmingham NHS Foundation Trust and the University of Birmingham. JPTH receives support from the NIHR Biomedical Research Centre at University Hospitals Bristol NHS Foundation Trust and the University of Bristol. The views expressed are those of the author(s) and not necessarily those of the NHS, the NIHR or the Department of Health.

15.8 References

Aguilar MI, Hart R. Oral anticoagulants for preventing stroke in patients with non-valvular atrial fibrillation and no previous history of stroke or transient ischemic attacks. Cochrane Database of Systematic Reviews 2005; 3 : CD001927.

Aguilar MI, Hart R, Pearce LA. Oral anticoagulants versus antiplatelet therapy for preventing stroke in patients with non-valvular atrial fibrillation and no history of stroke or transient ischemic attacks. Cochrane Database of Systematic Reviews 2007; 3 : CD006186.

Akl EA, Gunukula S, Barba M, Yosuico VE, van Doormaal FF, Kuipers S, Middeldorp S, Dickinson HO, Bryant A, Schünemann H. Parenteral anticoagulation in patients with cancer who have no therapeutic or prophylactic indication for anticoagulation. Cochrane Database of Systematic Reviews 2011a; 1 : CD006652.

Akl EA, Oxman AD, Herrin J, Vist GE, Terrenato I, Sperati F, Costiniuk C, Blank D, Schünemann H. Using alternative statistical formats for presenting risks and risk reductions. Cochrane Database of Systematic Reviews 2011b; 3 : CD006776.

Alonso-Coello P, Schünemann HJ, Moberg J, Brignardello-Petersen R, Akl EA, Davoli M, Treweek S, Mustafa RA, Rada G, Rosenbaum S, Morelli A, Guyatt GH, Oxman AD, Group GW. GRADE Evidence to Decision (EtD) frameworks: a systematic and transparent approach to making well informed healthcare choices. 1: Introduction. BMJ 2016; 353 : i2016.

Altman DG. Confidence intervals for the number needed to treat. BMJ 1998; 317 : 1309-1312.

Atkins D, Best D, Briss PA, Eccles M, Falck-Ytter Y, Flottorp S, Guyatt GH, Harbour RT, Haugh MC, Henry D, Hill S, Jaeschke R, Leng G, Liberati A, Magrini N, Mason J, Middleton P, Mrukowicz J, O'Connell D, Oxman AD, Phillips B, Schünemann HJ, Edejer TT, Varonen H, Vist GE, Williams JW, Jr., Zaza S. Grading quality of evidence and strength of recommendations. BMJ 2004; 328 : 1490.

Brown P, Brunnhuber K, Chalkidou K, Chalmers I, Clarke M, Fenton M, Forbes C, Glanville J, Hicks NJ, Moody J, Twaddle S, Timimi H, Young P. How to formulate research recommendations. BMJ 2006; 333 : 804-806.

Cates C. Confidence intervals for the number needed to treat: Pooling numbers needed to treat may not be reliable. BMJ 1999; 318 : 1764-1765.

Clarke MJ, Broderick C, Hopewell S, Juszczak E, Eisinga A. Compression stockings for preventing deep vein thrombosis in airline passengers. Cochrane Database of Systematic Reviews 2016; 9 : CD004002.

Cohen J. Statistical Power Analysis in the Behavioral Sciences . 2nd edition ed. Hillsdale (NJ): Lawrence Erlbaum Associates, Inc.; 1988.

Coleman T, Chamberlain C, Davey MA, Cooper SE, Leonardi-Bee J. Pharmacological interventions for promoting smoking cessation during pregnancy. Cochrane Database of Systematic Reviews 2015; 12 : CD010078.

Dans AM, Dans L, Oxman AD, Robinson V, Acuin J, Tugwell P, Dennis R, Kang D. Assessing equity in clinical practice guidelines. Journal of Clinical Epidemiology 2007; 60 : 540-546.

Friedman LM, Furberg CD, DeMets DL. Fundamentals of Clinical Trials . 2nd edition ed. Littleton (MA): John Wright PSG, Inc.; 1985.

Friedrich JO, Adhikari NK, Beyene J. The ratio of means method as an alternative to mean differences for analyzing continuous outcome variables in meta-analysis: a simulation study. BMC Medical Research Methodology 2008; 8 : 32.

Furukawa T. From effect size into number needed to treat. Lancet 1999; 353 : 1680.

Graham R, Mancher M, Wolman DM, Greenfield S, Steinberg E. Committee on Standards for Developing Trustworthy Clinical Practice Guidelines, Board on Health Care Services: Clinical Practice Guidelines We Can Trust. Washington, DC: National Academies Press; 2011.

Guyatt G, Oxman AD, Akl EA, Kunz R, Vist G, Brozek J, Norris S, Falck-Ytter Y, Glasziou P, DeBeer H, Jaeschke R, Rind D, Meerpohl J, Dahm P, Schünemann HJ. GRADE guidelines: 1. Introduction-GRADE evidence profiles and summary of findings tables. Journal of Clinical Epidemiology 2011a; 64 : 383-394.

Guyatt GH, Juniper EF, Walter SD, Griffith LE, Goldstein RS. Interpreting treatment effects in randomised trials. BMJ 1998; 316 : 690-693.

Guyatt GH, Oxman AD, Vist GE, Kunz R, Falck-Ytter Y, Alonso-Coello P, Schünemann HJ. GRADE: an emerging consensus on rating quality of evidence and strength of recommendations. BMJ 2008; 336 : 924-926.

Guyatt GH, Oxman AD, Kunz R, Woodcock J, Brozek J, Helfand M, Alonso-Coello P, Falck-Ytter Y, Jaeschke R, Vist G, Akl EA, Post PN, Norris S, Meerpohl J, Shukla VK, Nasser M, Schünemann HJ. GRADE guidelines: 8. Rating the quality of evidence--indirectness. Journal of Clinical Epidemiology 2011b; 64 : 1303-1310.

Guyatt GH, Oxman AD, Santesso N, Helfand M, Vist G, Kunz R, Brozek J, Norris S, Meerpohl J, Djulbegovic B, Alonso-Coello P, Post PN, Busse JW, Glasziou P, Christensen R, Schünemann HJ. GRADE guidelines: 12. Preparing summary of findings tables-binary outcomes. Journal of Clinical Epidemiology 2013a; 66 : 158-172.

Guyatt GH, Thorlund K, Oxman AD, Walter SD, Patrick D, Furukawa TA, Johnston BC, Karanicolas P, Akl EA, Vist G, Kunz R, Brozek J, Kupper LL, Martin SL, Meerpohl JJ, Alonso-Coello P, Christensen R, Schünemann HJ. GRADE guidelines: 13. Preparing summary of findings tables and evidence profiles-continuous outcomes. Journal of Clinical Epidemiology 2013b; 66 : 173-183.

Hawe P, Shiell A, Riley T, Gold L. Methods for exploring implementation variation and local context within a cluster randomised community intervention trial. Journal of Epidemiology and Community Health 2004; 58 : 788-793.

Hoffrage U, Lindsey S, Hertwig R, Gigerenzer G. Medicine. Communicating statistical information. Science 2000; 290 : 2261-2262.

Jaeschke R, Singer J, Guyatt GH. Measurement of health status. Ascertaining the minimal clinically important difference. Controlled Clinical Trials 1989; 10 : 407-415.

Johnston B, Thorlund K, Schünemann H, Xie F, Murad M, Montori V, Guyatt G. Improving the interpretation of health-related quality of life evidence in meta-analysis: The application of minimal important difference units. . Health Outcomes and Qualithy of Life 2010; 11 : 116.

Karanicolas PJ, Smith SE, Kanbur B, Davies E, Guyatt GH. The impact of prophylactic dexamethasone on nausea and vomiting after laparoscopic cholecystectomy: a systematic review and meta-analysis. Annals of Surgery 2008; 248 : 751-762.

Lumley J, Oliver SS, Chamberlain C, Oakley L. Interventions for promoting smoking cessation during pregnancy. Cochrane Database of Systematic Reviews 2004; 4 : CD001055.

McQuay HJ, Moore RA. Using numerical results from systematic reviews in clinical practice. Annals of Internal Medicine 1997; 126 : 712-720.

Resnicow K, Cross D, Wynder E. The Know Your Body program: a review of evaluation studies. Bulletin of the New York Academy of Medicine 1993; 70 : 188-207.

Robinson J, Biley FC, Dolk H. Therapeutic touch for anxiety disorders. Cochrane Database of Systematic Reviews 2007; 3 : CD006240.

Rothwell PM. External validity of randomised controlled trials: "to whom do the results of this trial apply?". Lancet 2005; 365 : 82-93.

Santesso N, Carrasco-Labra A, Langendam M, Brignardello-Petersen R, Mustafa RA, Heus P, Lasserson T, Opiyo N, Kunnamo I, Sinclair D, Garner P, Treweek S, Tovey D, Akl EA, Tugwell P, Brozek JL, Guyatt G, Schünemann HJ. Improving GRADE evidence tables part 3: detailed guidance for explanatory footnotes supports creating and understanding GRADE certainty in the evidence judgments. Journal of Clinical Epidemiology 2016; 74 : 28-39.

Schünemann HJ, Puhan M, Goldstein R, Jaeschke R, Guyatt GH. Measurement properties and interpretability of the Chronic respiratory disease questionnaire (CRQ). COPD: Journal of Chronic Obstructive Pulmonary Disease 2005; 2 : 81-89.

Schünemann HJ, Guyatt GH. Commentary--goodbye M(C)ID! Hello MID, where do you come from? Health Services Research 2005; 40 : 593-597.

Schünemann HJ, Fretheim A, Oxman AD. Improving the use of research evidence in guideline development: 13. Applicability, transferability and adaptation. Health Research Policy and Systems 2006; 4 : 25.

Schünemann HJ. Methodological idiosyncracies, frameworks and challenges of non-pharmaceutical and non-technical treatment interventions. Zeitschrift für Evidenz, Fortbildung und Qualität im Gesundheitswesen 2013; 107 : 214-220.

Schünemann HJ, Tugwell P, Reeves BC, Akl EA, Santesso N, Spencer FA, Shea B, Wells G, Helfand M. Non-randomized studies as a source of complementary, sequential or replacement evidence for randomized controlled trials in systematic reviews on the effects of interventions. Research Synthesis Methods 2013; 4 : 49-62.

Schünemann HJ, Wiercioch W, Etxeandia I, Falavigna M, Santesso N, Mustafa R, Ventresca M, Brignardello-Petersen R, Laisaar KT, Kowalski S, Baldeh T, Zhang Y, Raid U, Neumann I, Norris SL, Thornton J, Harbour R, Treweek S, Guyatt G, Alonso-Coello P, Reinap M, Brozek J, Oxman A, Akl EA. Guidelines 2.0: systematic development of a comprehensive checklist for a successful guideline enterprise. CMAJ: Canadian Medical Association Journal 2014; 186 : E123-142.

Schünemann HJ. Interpreting GRADE's levels of certainty or quality of the evidence: GRADE for statisticians, considering review information size or less emphasis on imprecision? Journal of Clinical Epidemiology 2016; 75 : 6-15.

Smeeth L, Haines A, Ebrahim S. Numbers needed to treat derived from meta-analyses--sometimes informative, usually misleading. BMJ 1999; 318 : 1548-1551.

Sun X, Briel M, Busse JW, You JJ, Akl EA, Mejza F, Bala MM, Bassler D, Mertz D, Diaz-Granados N, Vandvik PO, Malaga G, Srinathan SK, Dahm P, Johnston BC, Alonso-Coello P, Hassouneh B, Walter SD, Heels-Ansdell D, Bhatnagar N, Altman DG, Guyatt GH. Credibility of claims of subgroup effects in randomised controlled trials: systematic review. BMJ 2012; 344 : e1553.

Zhang Y, Akl EA, Schünemann HJ. Using systematic reviews in guideline development: the GRADE approach. Research Synthesis Methods 2018a: doi: 10.1002/jrsm.1313.

Zhang Y, Alonso-Coello P, Guyatt GH, Yepes-Nunez JJ, Akl EA, Hazlewood G, Pardo-Hernandez H, Etxeandia-Ikobaltzeta I, Qaseem A, Williams JW, Jr., Tugwell P, Flottorp S, Chang Y, Zhang Y, Mustafa RA, Rojas MX, Schünemann HJ. GRADE Guidelines: 19. Assessing the certainty of evidence in the importance of outcomes or values and preferences-Risk of bias and indirectness. Journal of Clinical Epidemiology 2018b: doi: 10.1016/j.jclinepi.2018.1001.1013.

Zhang Y, Alonso Coello P, Guyatt G, Yepes-Nunez JJ, Akl EA, Hazlewood G, Pardo-Hernandez H, Etxeandia-Ikobaltzeta I, Qaseem A, Williams JW, Jr., Tugwell P, Flottorp S, Chang Y, Zhang Y, Mustafa RA, Rojas MX, Xie F, Schünemann HJ. GRADE Guidelines: 20. Assessing the certainty of evidence in the importance of outcomes or values and preferences - Inconsistency, Imprecision, and other Domains. Journal of Clinical Epidemiology 2018c: doi: 10.1016/j.jclinepi.2018.1005.1011.

For permission to re-use material from the Handbook (either academic or commercial), please see here for full details.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Am J Pharm Educ
• v.74(8); 2010 Oct 11

Presenting and Evaluating Qualitative Research

The purpose of this paper is to help authors to think about ways to present qualitative research papers in the American Journal of Pharmaceutical Education . It also discusses methods for reviewers to assess the rigour, quality, and usefulness of qualitative research. Examples of different ways to present data from interviews, observations, and focus groups are included. The paper concludes with guidance for publishing qualitative research and a checklist for authors and reviewers.

INTRODUCTION

Policy and practice decisions, including those in education, increasingly are informed by findings from qualitative as well as quantitative research. Qualitative research is useful to policymakers because it often describes the settings in which policies will be implemented. Qualitative research is also useful to both pharmacy practitioners and pharmacy academics who are involved in researching educational issues in both universities and practice and in developing teaching and learning.

Qualitative research involves the collection, analysis, and interpretation of data that are not easily reduced to numbers. These data relate to the social world and the concepts and behaviors of people within it. Qualitative research can be found in all social sciences and in the applied fields that derive from them, for example, research in health services, nursing, and pharmacy. 1 It looks at X in terms of how X varies in different circumstances rather than how big is X or how many Xs are there? 2 Textbooks often subdivide research into qualitative and quantitative approaches, furthering the common assumption that there are fundamental differences between the 2 approaches. With pharmacy educators who have been trained in the natural and clinical sciences, there is often a tendency to embrace quantitative research, perhaps due to familiarity. A growing consensus is emerging that sees both qualitative and quantitative approaches as useful to answering research questions and understanding the world. Increasingly mixed methods research is being carried out where the researcher explicitly combines the quantitative and qualitative aspects of the study. 3 , 4

Like healthcare, education involves complex human interactions that can rarely be studied or explained in simple terms. Complex educational situations demand complex understanding; thus, the scope of educational research can be extended by the use of qualitative methods. Qualitative research can sometimes provide a better understanding of the nature of educational problems and thus add to insights into teaching and learning in a number of contexts. For example, at the University of Nottingham, we conducted in-depth interviews with pharmacists to determine their perceptions of continuing professional development and who had influenced their learning. We also have used a case study approach using observation of practice and in-depth interviews to explore physiotherapists' views of influences on their leaning in practice. We have conducted in-depth interviews with a variety of stakeholders in Malawi, Africa, to explore the issues surrounding pharmacy academic capacity building. A colleague has interviewed and conducted focus groups with students to explore cultural issues as part of a joint Nottingham-Malaysia pharmacy degree program. Another colleague has interviewed pharmacists and patients regarding their expectations before and after clinic appointments and then observed pharmacist-patient communication in clinics and assessed it using the Calgary Cambridge model in order to develop recommendations for communication skills training. 5 We have also performed documentary analysis on curriculum data to compare pharmacist and nurse supplementary prescribing courses in the United Kingdom.

It is important to choose the most appropriate methods for what is being investigated. Qualitative research is not appropriate to answer every research question and researchers need to think carefully about their objectives. Do they wish to study a particular phenomenon in depth (eg, students' perceptions of studying in a different culture)? Or are they more interested in making standardized comparisons and accounting for variance (eg, examining differences in examination grades after changing the way the content of a module is taught). Clearly a quantitative approach would be more appropriate in the last example. As with any research project, a clear research objective has to be identified to know which methods should be applied.

Types of qualitative data include:

• Audio recordings and transcripts from in-depth or semi-structured interviews
• Structured interview questionnaires containing substantial open comments including a substantial number of responses to open comment items.
• Audio recordings and transcripts from focus group sessions.
• Field notes (notes taken by the researcher while in the field [setting] being studied)
• Video recordings (eg, lecture delivery, class assignments, laboratory performance)
• Case study notes
• Documents (reports, meeting minutes, e-mails)
• Diaries, video diaries
• Observation notes
• Press clippings
• Photographs

RIGOUR IN QUALITATIVE RESEARCH

Qualitative research is often criticized as biased, small scale, anecdotal, and/or lacking rigor; however, when it is carried out properly it is unbiased, in depth, valid, reliable, credible and rigorous. In qualitative research, there needs to be a way of assessing the “extent to which claims are supported by convincing evidence.” 1 Although the terms reliability and validity traditionally have been associated with quantitative research, increasingly they are being seen as important concepts in qualitative research as well. Examining the data for reliability and validity assesses both the objectivity and credibility of the research. Validity relates to the honesty and genuineness of the research data, while reliability relates to the reproducibility and stability of the data.

The validity of research findings refers to the extent to which the findings are an accurate representation of the phenomena they are intended to represent. The reliability of a study refers to the reproducibility of the findings. Validity can be substantiated by a number of techniques including triangulation use of contradictory evidence, respondent validation, and constant comparison. Triangulation is using 2 or more methods to study the same phenomenon. Contradictory evidence, often known as deviant cases, must be sought out, examined, and accounted for in the analysis to ensure that researcher bias does not interfere with or alter their perception of the data and any insights offered. Respondent validation, which is allowing participants to read through the data and analyses and provide feedback on the researchers' interpretations of their responses, provides researchers with a method of checking for inconsistencies, challenges the researchers' assumptions, and provides them with an opportunity to re-analyze their data. The use of constant comparison means that one piece of data (for example, an interview) is compared with previous data and not considered on its own, enabling researchers to treat the data as a whole rather than fragmenting it. Constant comparison also enables the researcher to identify emerging/unanticipated themes within the research project.

STRENGTHS AND LIMITATIONS OF QUALITATIVE RESEARCH

Qualitative researchers have been criticized for overusing interviews and focus groups at the expense of other methods such as ethnography, observation, documentary analysis, case studies, and conversational analysis. Qualitative research has numerous strengths when properly conducted.

Strengths of Qualitative Research

• Issues can be examined in detail and in depth.
• Interviews are not restricted to specific questions and can be guided/redirected by the researcher in real time.
• The research framework and direction can be quickly revised as new information emerges.
• The data based on human experience that is obtained is powerful and sometimes more compelling than quantitative data.
• Subtleties and complexities about the research subjects and/or topic are discovered that are often missed by more positivistic enquiries.
• Data usually are collected from a few cases or individuals so findings cannot be generalized to a larger population. Findings can however be transferable to another setting.

Limitations of Qualitative Research

• Research quality is heavily dependent on the individual skills of the researcher and more easily influenced by the researcher's personal biases and idiosyncrasies.
• Rigor is more difficult to maintain, assess, and demonstrate.
• The volume of data makes analysis and interpretation time consuming.
• It is sometimes not as well understood and accepted as quantitative research within the scientific community
• The researcher's presence during data gathering, which is often unavoidable in qualitative research, can affect the subjects' responses.
• Issues of anonymity and confidentiality can present problems when presenting findings
• Findings can be more difficult and time consuming to characterize in a visual way.

PRESENTATION OF QUALITATIVE RESEARCH FINDINGS

The following extracts are examples of how qualitative data might be presented:

Data From an Interview.

The following is an example of how to present and discuss a quote from an interview.

The researcher should select quotes that are poignant and/or most representative of the research findings. Including large portions of an interview in a research paper is not necessary and often tedious for the reader. The setting and speakers should be established in the text at the end of the quote.

The student describes how he had used deep learning in a dispensing module. He was able to draw on learning from a previous module, “I found that while using the e learning programme I was able to apply the knowledge and skills that I had gained in last year's diseases and goals of treatment module.” (interviewee 22, male)

This is an excerpt from an article on curriculum reform that used interviews 5 :

The first question was, “Without the accreditation mandate, how much of this curriculum reform would have been attempted?” According to respondents, accreditation played a significant role in prompting the broad-based curricular change, and their comments revealed a nuanced view. Most indicated that the change would likely have occurred even without the mandate from the accreditation process: “It reflects where the profession wants to be … training a professional who wants to take on more responsibility.” However, they also commented that “if it were not mandated, it could have been a very difficult road.” Or it “would have happened, but much later.” The change would more likely have been incremental, “evolutionary,” or far more limited in its scope. “Accreditation tipped the balance” was the way one person phrased it. “Nobody got serious until the accrediting body said it would no longer accredit programs that did not change.”

Data From Observations

The following example is some data taken from observation of pharmacist patient consultations using the Calgary Cambridge guide. 6 , 7 The data are first presented and a discussion follows:

Pharmacist: We will soon be starting a stop smoking clinic. Patient: Is the interview over now? Pharmacist: No this is part of it. (Laughs) You can't tell me to bog off (sic) yet. (pause) We will be starting a stop smoking service here, Patient: Yes. Pharmacist: with one-to-one and we will be able to help you or try to help you. If you want it. In this example, the pharmacist has picked up from the patient's reaction to the stop smoking clinic that she is not receptive to advice about giving up smoking at this time; in fact she would rather end the consultation. The pharmacist draws on his prior relationship with the patient and makes use of a joke to lighten the tone. He feels his message is important enough to persevere but he presents the information in a succinct and non-pressurised way. His final comment of “If you want it” is important as this makes it clear that he is not putting any pressure on the patient to take up this offer. This extract shows that some patient cues were picked up, and appropriately dealt with, but this was not the case in all examples.

Data From Focus Groups

This excerpt from a study involving 11 focus groups illustrates how findings are presented using representative quotes from focus group participants. 8

Those pharmacists who were initially familiar with CPD endorsed the model for their peers, and suggested it had made a meaningful difference in the way they viewed their own practice. In virtually all focus groups sessions, pharmacists familiar with and supportive of the CPD paradigm had worked in collaborative practice environments such as hospital pharmacy practice. For these pharmacists, the major advantage of CPD was the linking of workplace learning with continuous education. One pharmacist stated, “It's amazing how much I have to learn every day, when I work as a pharmacist. With [the learning portfolio] it helps to show how much learning we all do, every day. It's kind of satisfying to look it over and see how much you accomplish.” Within many of the learning portfolio-sharing sessions, debates emerged regarding the true value of traditional continuing education and its outcome in changing an individual's practice. While participants appreciated the opportunity for social and professional networking inherent in some forms of traditional CE, most eventually conceded that the academic value of most CE programming was limited by the lack of a systematic process for following-up and implementing new learning in the workplace. “Well it's nice to go to these [continuing education] events, but really, I don't know how useful they are. You go, you sit, you listen, but then, well I at least forget.”

The following is an extract from a focus group (conducted by the author) with first-year pharmacy students about community placements. It illustrates how focus groups provide a chance for participants to discuss issues on which they might disagree.

Interviewer: So you are saying that you would prefer health related placements? Student 1: Not exactly so long as I could be developing my communication skill. Student 2: Yes but I still think the more health related the placement is the more I'll gain from it. Student 3: I disagree because other people related skills are useful and you may learn those from taking part in a community project like building a garden. Interviewer: So would you prefer a mixture of health and non health related community placements?

GUIDANCE FOR PUBLISHING QUALITATIVE RESEARCH

Qualitative research is becoming increasingly accepted and published in pharmacy and medical journals. Some journals and publishers have guidelines for presenting qualitative research, for example, the British Medical Journal 9 and Biomedcentral . 10 Medical Education published a useful series of articles on qualitative research. 11 Some of the important issues that should be considered by authors, reviewers and editors when publishing qualitative research are discussed below.

Introduction.

A good introduction provides a brief overview of the manuscript, including the research question and a statement justifying the research question and the reasons for using qualitative research methods. This section also should provide background information, including relevant literature from pharmacy, medicine, and other health professions, as well as literature from the field of education that addresses similar issues. Any specific educational or research terminology used in the manuscript should be defined in the introduction.

The methods section should clearly state and justify why the particular method, for example, face to face semistructured interviews, was chosen. The method should be outlined and illustrated with examples such as the interview questions, focusing exercises, observation criteria, etc. The criteria for selecting the study participants should then be explained and justified. The way in which the participants were recruited and by whom also must be stated. A brief explanation/description should be included of those who were invited to participate but chose not to. It is important to consider “fair dealing,” ie, whether the research design explicitly incorporates a wide range of different perspectives so that the viewpoint of 1 group is never presented as if it represents the sole truth about any situation. The process by which ethical and or research/institutional governance approval was obtained should be described and cited.

The study sample and the research setting should be described. Sampling differs between qualitative and quantitative studies. In quantitative survey studies, it is important to select probability samples so that statistics can be used to provide generalizations to the population from which the sample was drawn. Qualitative research necessitates having a small sample because of the detailed and intensive work required for the study. So sample sizes are not calculated using mathematical rules and probability statistics are not applied. Instead qualitative researchers should describe their sample in terms of characteristics and relevance to the wider population. Purposive sampling is common in qualitative research. Particular individuals are chosen with characteristics relevant to the study who are thought will be most informative. Purposive sampling also may be used to produce maximum variation within a sample. Participants being chosen based for example, on year of study, gender, place of work, etc. Representative samples also may be used, for example, 20 students from each of 6 schools of pharmacy. Convenience samples involve the researcher choosing those who are either most accessible or most willing to take part. This may be fine for exploratory studies; however, this form of sampling may be biased and unrepresentative of the population in question. Theoretical sampling uses insights gained from previous research to inform sample selection for a new study. The method for gaining informed consent from the participants should be described, as well as how anonymity and confidentiality of subjects were guaranteed. The method of recording, eg, audio or video recording, should be noted, along with procedures used for transcribing the data.

Data Analysis.

A description of how the data were analyzed also should be included. Was computer-aided qualitative data analysis software such as NVivo (QSR International, Cambridge, MA) used? Arrival at “data saturation” or the end of data collection should then be described and justified. A good rule when considering how much information to include is that readers should have been given enough information to be able to carry out similar research themselves.

One of the strengths of qualitative research is the recognition that data must always be understood in relation to the context of their production. 1 The analytical approach taken should be described in detail and theoretically justified in light of the research question. If the analysis was repeated by more than 1 researcher to ensure reliability or trustworthiness, this should be stated and methods of resolving any disagreements clearly described. Some researchers ask participants to check the data. If this was done, it should be fully discussed in the paper.

An adequate account of how the findings were produced should be included A description of how the themes and concepts were derived from the data also should be included. Was an inductive or deductive process used? The analysis should not be limited to just those issues that the researcher thinks are important, anticipated themes, but also consider issues that participants raised, ie, emergent themes. Qualitative researchers must be open regarding the data analysis and provide evidence of their thinking, for example, were alternative explanations for the data considered and dismissed, and if so, why were they dismissed? It also is important to present outlying or negative/deviant cases that did not fit with the central interpretation.

The interpretation should usually be grounded in interviewees or respondents' contributions and may be semi-quantified, if this is possible or appropriate, for example, “Half of the respondents said …” “The majority said …” “Three said…” Readers should be presented with data that enable them to “see what the researcher is talking about.” 1 Sufficient data should be presented to allow the reader to clearly see the relationship between the data and the interpretation of the data. Qualitative data conventionally are presented by using illustrative quotes. Quotes are “raw data” and should be compiled and analyzed, not just listed. There should be an explanation of how the quotes were chosen and how they are labeled. For example, have pseudonyms been given to each respondent or are the respondents identified using codes, and if so, how? It is important for the reader to be able to see that a range of participants have contributed to the data and that not all the quotes are drawn from 1 or 2 individuals. There is a tendency for authors to overuse quotes and for papers to be dominated by a series of long quotes with little analysis or discussion. This should be avoided.

Participants do not always state the truth and may say what they think the interviewer wishes to hear. A good qualitative researcher should not only examine what people say but also consider how they structured their responses and how they talked about the subject being discussed, for example, the person's emotions, tone, nonverbal communication, etc. If the research was triangulated with other qualitative or quantitative data, this should be discussed.

Discussion.

The findings should be presented in the context of any similar previous research and or theories. A discussion of the existing literature and how this present research contributes to the area should be included. A consideration must also be made about how transferrable the research would be to other settings. Any particular strengths and limitations of the research also should be discussed. It is common practice to include some discussion within the results section of qualitative research and follow with a concluding discussion.

The author also should reflect on their own influence on the data, including a consideration of how the researcher(s) may have introduced bias to the results. The researcher should critically examine their own influence on the design and development of the research, as well as on data collection and interpretation of the data, eg, were they an experienced teacher who researched teaching methods? If so, they should discuss how this might have influenced their interpretation of the results.

Conclusion.

The conclusion should summarize the main findings from the study and emphasize what the study adds to knowledge in the area being studied. Mays and Pope suggest the researcher ask the following 3 questions to determine whether the conclusions of a qualitative study are valid 12 : How well does this analysis explain why people behave in the way they do? How comprehensible would this explanation be to a thoughtful participant in the setting? How well does the explanation cohere with what we already know?

CHECKLIST FOR QUALITATIVE PAPERS

This paper establishes criteria for judging the quality of qualitative research. It provides guidance for authors and reviewers to prepare and review qualitative research papers for the American Journal of Pharmaceutical Education . A checklist is provided in Appendix 1 to assist both authors and reviewers of qualitative data.

ACKNOWLEDGEMENTS

Thank you to the 3 reviewers whose ideas helped me to shape this paper.

Appendix 1. Checklist for authors and reviewers of qualitative research.

Introduction

• □ Research question is clearly stated.
• □ Research question is justified and related to the existing knowledge base (empirical research, theory, policy).
• □ Any specific research or educational terminology used later in manuscript is defined.
• □ The process by which ethical and or research/institutional governance approval was obtained is described and cited.
• □ Reason for choosing particular research method is stated.
• □ Criteria for selecting study participants are explained and justified.
• □ Recruitment methods are explicitly stated.
• □ Details of who chose not to participate and why are given.
• □ Study sample and research setting used are described.
• □ Method for gaining informed consent from the participants is described.
• □ Maintenance/Preservation of subject anonymity and confidentiality is described.
• □ Method of recording data (eg, audio or video recording) and procedures for transcribing data are described.
• □ Methods are outlined and examples given (eg, interview guide).
• □ Decision to stop data collection is described and justified.
• □ Data analysis and verification are described, including by whom they were performed.
• □ Methods for identifying/extrapolating themes and concepts from the data are discussed.
• □ Sufficient data are presented to allow a reader to assess whether or not the interpretation is supported by the data.
• □ Outlying or negative/deviant cases that do not fit with the central interpretation are presented.
• □ Transferability of research findings to other settings is discussed.
• □ Findings are presented in the context of any similar previous research and social theories.
• □ Discussion often is incorporated into the results in qualitative papers.
• □ A discussion of the existing literature and how this present research contributes to the area is included.
• □ Any particular strengths and limitations of the research are discussed.
• □ Reflection of the influence of the researcher(s) on the data, including a consideration of how the researcher(s) may have introduced bias to the results is included.

Conclusions

• □ The conclusion states the main finings of the study and emphasizes what the study adds to knowledge in the subject area.

Have a language expert improve your writing

Run a free plagiarism check in 10 minutes, generate accurate citations for free.

• Knowledge Base
• Research paper
• How to Write Recommendations in Research | Examples & Tips

How to Write Recommendations in Research | Examples & Tips

Published on September 15, 2022 by Tegan George . Revised on July 18, 2023.

Recommendations in research are a crucial component of your discussion section and the conclusion of your thesis , dissertation , or research paper .

As you conduct your research and analyze the data you collected , perhaps there are ideas or results that don’t quite fit the scope of your research topic. Or, maybe your results suggest that there are further implications of your results or the causal relationships between previously-studied variables than covered in extant research.

Instantly correct all language mistakes in your text

Upload your document to correct all your mistakes in minutes

Table of contents

What should recommendations look like, building your research recommendation, how should your recommendations be written, recommendation in research example, other interesting articles, frequently asked questions about recommendations.

Recommendations for future research should be:

• Concrete and specific
• Supported with a clear rationale
• Directly connected to your research

Overall, strive to highlight ways other researchers can reproduce or replicate your results to draw further conclusions, and suggest different directions that future research can take, if applicable.

Relatedly, when making these recommendations, avoid:

• Undermining your own work, but rather offer suggestions on how future studies can build upon it
• Suggesting recommendations actually needed to complete your argument, but rather ensure that your research stands alone on its own merits
• Using recommendations as a place for self-criticism, but rather as a natural extension point for your work

Scribbr Citation Checker New

The AI-powered Citation Checker helps you avoid common mistakes such as:

• Missing commas and periods
• Incorrect usage of “et al.”
• Ampersands (&) in narrative citations
• Missing reference entries

There are many different ways to frame recommendations, but the easiest is perhaps to follow the formula of research question   conclusion  recommendation. Here’s an example.

Conclusion An important condition for controlling many social skills is mastering language. If children have a better command of language, they can express themselves better and are better able to understand their peers. Opportunities to practice social skills are thus dependent on the development of language skills.

As a rule of thumb, try to limit yourself to only the most relevant future recommendations: ones that stem directly from your work. While you can have multiple recommendations for each research conclusion, it is also acceptable to have one recommendation that is connected to more than one conclusion.

These recommendations should be targeted at your audience, specifically toward peers or colleagues in your field that work on similar subjects to your paper or dissertation topic . They can flow directly from any limitations you found while conducting your work, offering concrete and actionable possibilities for how future research can build on anything that your own work was unable to address at the time of your writing.

See below for a full research recommendation example that you can use as a template to write your own.

The only proofreading tool specialized in correcting academic writing - try for free!

The academic proofreading tool has been trained on 1000s of academic texts and by native English editors. Making it the most accurate and reliable proofreading tool for students.

Try for free

If you want to know more about AI for academic writing, AI tools, or research bias, make sure to check out some of our other articles with explanations and examples or go directly to our tools!

Research bias

• Survivorship bias
• Self-serving bias
• Availability heuristic
• Halo effect
• Hindsight bias
• Deep learning
• Generative AI
• Machine learning
• Reinforcement learning
• Supervised vs. unsupervised learning

 (AI) Tools

• Grammar Checker
• Paraphrasing Tool
• Text Summarizer
• AI Detector
• Plagiarism Checker
• Citation Generator

While it may be tempting to present new arguments or evidence in your thesis or disseration conclusion , especially if you have a particularly striking argument you’d like to finish your analysis with, you shouldn’t. Theses and dissertations follow a more formal structure than this.

All your findings and arguments should be presented in the body of the text (more specifically in the discussion section and results section .) The conclusion is meant to summarize and reflect on the evidence and arguments you have already presented, not introduce new ones.

The conclusion of your thesis or dissertation should include the following:

• A restatement of your research question
• A summary of your key arguments and/or results
• A short discussion of the implications of your research

For a stronger dissertation conclusion , avoid including:

• Important evidence or analysis that wasn’t mentioned in the discussion section and results section
• Generic concluding phrases (e.g. “In conclusion …”)
• Weak statements that undermine your argument (e.g., “There are good points on both sides of this issue.”)

Your conclusion should leave the reader with a strong, decisive impression of your work.

In a thesis or dissertation, the discussion is an in-depth exploration of the results, going into detail about the meaning of your findings and citing relevant sources to put them in context.

The conclusion is more shorter and more general: it concisely answers your main research question and makes recommendations based on your overall findings.

Cite this Scribbr article

If you want to cite this source, you can copy and paste the citation or click the “Cite this Scribbr article” button to automatically add the citation to our free Citation Generator.

George, T. (2023, July 18). How to Write Recommendations in Research | Examples & Tips. Scribbr. Retrieved April 2, 2024, from https://www.scribbr.com/dissertation/recommendations-in-research/

Is this article helpful?

Tegan George

Other students also liked, how to write a discussion section | tips & examples, how to write a thesis or dissertation conclusion, how to write a results section | tips & examples, unlimited academic ai-proofreading.

✔ Document error-free in 5minutes ✔ Unlimited document corrections ✔ Specialized in correcting academic texts

April 2, 2024

• Critical Gaps in Medical Research Reporting by Online News Media

This study reveals significant shortcomings in reporting medical research by online news outlets. The study found that crucial information such as conflicts of interest, study limitations, and inferential statistics were frequently omitted from news reports. While research conclusions were generally conveyed accurately, underreporting these key elements raises concerns about the transparency and credibility of medical research communication. The findings underscore the need for more comprehensive and balanced reporting to prevent potential misinterpretations and the spread of misinformation.

Citation: Heston TF. Critical Gaps in Medical Research Reporting by Online News Media. Cureus. 2024 Apr 2;16(4):e57457. doi: 10.7759/cureus.57457 .

Recent Posts

• Blockchain and AI: A Synergistic Solution to Climate Change
• Beware online mental health chatbots, specialists warn
• Statistical Fragility in Surveys: A Cautionary Tale
• Two Left Shoes: A Musical Essay on Homelessness
• February 2024
• January 2024
• December 2023
• November 2023
• October 2023
• September 2023
• August 2023

Be boundless

Connect with us:.

© 2024 University of Washington | Seattle, WA

This paper is in the following e-collection/theme issue:

Published on 3.4.2024 in Vol 26 (2024)

Public Discourse, User Reactions, and Conspiracy Theories on the X Platform About HIV Vaccines: Data Mining and Content Analysis

Authors of this article:

Original Paper

• Jueman M Zhang 1 , PhD   ; 
• Yi Wang 2 , PhD   ; 
• Magali Mouton 3   ; 
• Jixuan Zhang 4   ; 
• Molu Shi 5 , PhD  

1 Harrington School of Communication and Media, University of Rhode Island, Kingston, RI, United States

2 Department of Communication, University of Louisville, Louisville, KY, United States

3 School of Rehabilitation Sciences, University of Ottawa, Ottawa, ON, Canada

4 Polk School of Communications, Long Island University, Brooklyn, NY, United States

5 College of Business, University of Louisville, Louisville, KY, United States

Corresponding Author:

Jueman M Zhang, PhD

Harrington School of Communication and Media

University of Rhode Island

10 Ranger Road

Kingston, RI, 02881

United States

Phone: 1 401 874 2110

Email: [email protected]

Background: The initiation of clinical trials for messenger RNA (mRNA) HIV vaccines in early 2022 revived public discussion on HIV vaccines after 3 decades of unsuccessful research. These trials followed the success of mRNA technology in COVID-19 vaccines but unfolded amid intense vaccine debates during the COVID-19 pandemic. It is crucial to gain insights into public discourse and reactions about potential new vaccines, and social media platforms such as X (formerly known as Twitter) provide important channels.

Objective: Drawing from infodemiology and infoveillance research, this study investigated the patterns of public discourse and message-level drivers of user reactions on X regarding HIV vaccines by analyzing posts using machine learning algorithms. We examined how users used different post types to contribute to topics and valence and how these topics and valence influenced like and repost counts. In addition, the study identified salient aspects of HIV vaccines related to COVID-19 and prominent anti–HIV vaccine conspiracy theories through manual coding.

Methods: We collected 36,424 English-language original posts about HIV vaccines on the X platform from January 1, 2022, to December 31, 2022. We used topic modeling and sentiment analysis to uncover latent topics and valence, which were subsequently analyzed across post types in cross-tabulation analyses and integrated into linear regression models to predict user reactions, specifically likes and reposts. Furthermore, we manually coded the 1000 most engaged posts about HIV and COVID-19 to uncover salient aspects of HIV vaccines related to COVID-19 and the 1000 most engaged negative posts to identify prominent anti–HIV vaccine conspiracy theories.

Results: Topic modeling revealed 3 topics: HIV and COVID-19, mRNA HIV vaccine trials, and HIV vaccine and immunity. HIV and COVID-19 underscored the connections between HIV vaccines and COVID-19 vaccines, as evidenced by subtopics about their reciprocal impact on development and various comparisons. The overall valence of the posts was marginally positive. Compared to self-composed posts initiating new conversations, there was a higher proportion of HIV and COVID-19–related and negative posts among quote posts and replies, which contribute to existing conversations. The topic of mRNA HIV vaccine trials, most evident in self-composed posts, increased repost counts. Positive valence increased like and repost counts. Prominent anti–HIV vaccine conspiracy theories often falsely linked HIV vaccines to concurrent COVID-19 and other HIV-related events.

Conclusions: The results highlight COVID-19 as a significant context for public discourse and reactions regarding HIV vaccines from both positive and negative perspectives. The success of mRNA COVID-19 vaccines shed a positive light on HIV vaccines. However, COVID-19 also situated HIV vaccines in a negative context, as observed in some anti–HIV vaccine conspiracy theories misleadingly connecting HIV vaccines with COVID-19. These findings have implications for public health communication strategies concerning HIV vaccines.

Introduction

Vaccination has long been recognized as a crucial preventive measure against diseases and infections, but opposition to vaccines has endured [ 1 ]. HIV vaccination has been regarded as a potential preventive measure to help combat the HIV epidemic in the United States, with research and progress dating back to the mid-1980s but without success thus far [ 2 ]. An estimated 1.2 million people were living with HIV in the United States by the end of 2021, with 36,136 new HIV diagnoses reported in 2021 [ 3 ].

On January 27, 2022, the biotechnology company Moderna announced the initiation of clinical trials for an HIV vaccine using messenger RNA (mRNA) technology [ 4 ]. In March 2022, the National Institutes of Health announced the start of clinical trials for 3 mRNA HIV vaccines [ 5 ]. The mRNA technology had previously been used in the Pfizer-BioNTech and Moderna COVID-19 vaccines, which protected individuals against severe symptoms and fatalities during the pandemic [ 6 ]. Following the successes of mRNA COVID-19 vaccines, which led to the Nobel Prize in Physiology or Medicine being awarded to 2 scientists in October 2023 [ 7 ], researchers have been investigating the potential of mRNA vaccines for various other diseases, including HIV [ 8 , 9 ]. The announcements of clinical trials for mRNA HIV vaccines revived public discussion on the prospect of vaccines to combat HIV [ 9 ] despite >3 decades of unsuccessful research [ 2 ]. Meanwhile, these announcements were made against the backdrop of intense vaccine debates during the COVID-19 pandemic, with misinformation and conspiracy theories fueling vaccine hesitancy [ 10 - 12 ].

The X platform, formerly known as Twitter, has been a significant social media platform and a vital source for text-based public discourse. Posts on X have been studied to understand public discourse about vaccines in general [ 13 - 15 ] and about specific vaccines, such as COVID-19 vaccines in recent years [ 12 , 16 , 17 ]. However, there is a dearth of research about public discourse on HIV vaccines on social media. Given the advancement in mRNA technology in COVID-19 vaccines and heated vaccine debates, it has become especially important to gain insights into public discourse and reactions regarding potential new vaccines.

This study is grounded in the growing field of infodemiology and infoveillance, which investigates the “distribution and determinants of information in an electronic medium,” specifically on the web, by analyzing unstructured text with the aim of informing public health practices or serving surveillance objectives [ 18 ]. In recent infodemiology and infoveillance studies, machine learning algorithms have been increasingly used to examine substantial amounts of social media content, such as posts on X related to COVID-19 vaccines [ 12 , 16 , 17 ] and HIV prevention [ 19 ], to extract insights into public discourse and reactions. These algorithms automatically analyze extensive volumes of posts and capture latent textual information such as topics and sentiments. This study aimed to investigate how users used different post types to contribute original content to topics and valence identified through machine learning algorithms and how these topics and valence affected user reactions on X regarding HIV vaccines. In addition, by manually coding the most engaged posts, similar to an approach used in previous infodemiology and infoveillance research [ 20 ], the study intended to identify salient aspects of HIV vaccines related to COVID-19 as well as prominent anti–HIV vaccine conspiracy theories. Analyzing posts on X about HIV vaccines can shed light on public discourse and information diffusion. These findings have implications for shaping public health communication strategies about HIV vaccines [ 18 ]. Furthermore, the findings may help in understanding the acceptability of the HIV vaccine upon its successful development in comparison with adherence to existing HIV prevention measures. Previous infodemiology and infoveillance research effectively increased the forecast accuracy of COVID-19 vaccine uptake by leveraging insights derived from posts on X [ 21 ].

Literature Review

Public discourse about hiv prevention on x.

Social media platforms have become important channels for HIV communication, enabling the dissemination of and engagement with content encompassing a wide array of issues related to HIV prevention, treatment, coping, and available resources [ 22 , 23 ]. An earlier infodemiology study examined 69,197 posts on the X platform containing the hashtag #HIVPrevention between 2014 and 2019 and categorized these posts into 10 identified topics concerning HIV prevention [ 19 ]. Among them, pre-exposure prophylaxis had the highest representation with 13,895 posts, followed by HIV testing; condoms; harm reduction; gender equity and violence against women; voluntary medical male circumcision; sex work; postexposure prophylaxis; elimination of mother-to-child transmission of HIV; and abstinence, which had the lowest representation with 180 posts. Furthermore, that study suggested a consistency between the volume of posts related to specific HIV prevention measures on X over time and the temporal trends in the uptake of those measures [ 19 ]. It is noteworthy that the topic of HIV vaccines was absent, which suggested minimal public discourse on the topic during these years. This may be associated with the extensive history of unsuccessful research in this area [ 2 ].

Despite the availability of current HIV prevention measures, efforts have been made to develop HIV vaccines, which are considered necessary to bridge the gap between the challenges in adhering to existing HIV prevention measures and the ambitious goal set by United Nations member states to end the HIV epidemic by 2030 [ 24 , 25 ]. The surge in public discussion about HIV vaccines, possibly elicited by the clinical trials for mRNA HIV vaccines [ 9 ], presented an optimal opportunity to investigate public discourse and reactions regarding HIV vaccines. To the best of our knowledge, this is the first study to analyze posts on X about HIV vaccines.

Public Discourse and Post Types on X

On the X platform, public discourse featuring original content can be observed through 3 post types: self-composed posts, quote posts, and replies [ 26 ]. X users can compose a post. They can also create a quote post, which entails reposting a post while adding their comments. In addition, they can reply to a post to share their comments [ 26 ]. While self-composed posts initiate new conversations, quote posts and replies enable users to join existing conversations by contributing their own comments [ 27 ]. The Pew Research Center’s analysis of survey respondents’ posts on X from October 2022 to April 2023 revealed the composition of different types of posts. Regarding the 3 types of posts containing original content, replies accounted for the highest proportion at 40%, followed by self-composed posts at 15% and quote posts at 9%. The remaining 35% were reposts [ 28 ].

Machine learning algorithms have been increasingly used in recent years to identify latent message features, including textual topics and sentiment valence, among vast numbers of social media posts, as exemplified by previous research analyzing posts on X about COVID-19 vaccines [ 12 , 16 , 17 ] and HIV prevention [ 19 ]. However, the patterns of public discourse in social media conversations are unclear. Specifically, there is a scarcity of research on how people contribute their original content to topics and valence related to a public health issue. This study aimed to address this gap by examining the relationship between post types and message features, specifically topics and valence uncovered using machine learning algorithms, with a focus on HIV vaccines as the subject matter. The findings will advance our knowledge of user contributions to social media conversations about HIV vaccines.

Message Features Influencing User Reactions on X

Examining message diffusion on social media has been a multifaceted challenge, especially with vaccines being a contentious issue debated fervently during the COVID-19 pandemic [ 16 ]. Another contribution of this study is to advance this research area by using machine learning to investigate the synergistic impact of content and account features on user reactions regarding a potential new vaccine amid the context of intense vaccine debates.

The extent to which a message results in optimal diffusion on social media can be gauged by user reactions [ 16 , 29 - 31 ]. On X, a user can engage with posts—be it a self-composed post, quote post, or reply—in 2 primary 1-click reactions: liking and reposting [ 26 ]. An X user can like a post to show appreciation for it or repost it to share it publicly. Compared to liking, reposting is a more social behavior [ 16 , 32 ]. Unlike X’s old timeline, which mostly displayed posts from accounts that a user followed, its current “For you” timeline also shows posts that those accounts have engaged with along with other posts recommended based on user reactions [ 33 ]. The nature of promoting posts based on user reactions makes it more important to investigate the factors that influence user reactions.

This study investigated 2 categories of message-level features that, according to previous research, can drive user interactions: content features in terms of topics and valence and account features in terms of user verification and follower count. Post topics affect likes and reposts on X [ 16 , 30 , 34 ]. Previous research on COVID-19 vaccine posts on X has indicated that posts containing useful information garner more likes and reposts [ 16 ]. This is likely because information utility fills people’s knowledge gaps and serves their utilitarian needs in the face of health risks [ 16 , 32 , 34 - 36 ]. In addition, previous studies have suggested that the novelty of useful information further facilitates sharing of digital health information [ 32 , 36 ], such as updates about COVID-19 vaccine development [ 12 ]. Given the initial success of mRNA technology in COVID-19 vaccines, mRNA HIV vaccine candidates may possess the inherent features of prospective usefulness and ongoing novelty. As a result, posts presenting pertinent information have the potential to generate more likes and reposts. Meanwhile, the announcements of clinical trials for mRNA HIV vaccines were made amidst intense vaccine debates during the COVID-19 pandemic [ 12 ]. Previous research has shown that perceived controversiality in health information increases viewership but not sharing on social media [ 32 ]. In the context of the heated controversy surrounding vaccines, it is crucial to understand user reactions to new potential vaccines.

In addition to post topics, post valence can play a role in user reactions [ 34 ]. Past research has generally revealed that there are more positive than negative posts on X about vaccines in general [ 13 - 15 ] and, more recently, about COVID-19 vaccines in particular [ 12 , 16 , 17 ]. However, the influence of post valence on user reactions remains unclear. One study on COVID-19 vaccines showed that positive posts on X received more likes but not more reposts [ 16 ]. Another study on vaccines regardless of their type revealed that antivaccine posts garnered more reposts than provaccine posts on X [ 13 ]. A psychological rationale supporting the social transmission of positive content is the motivation of individuals to present themselves positively and shape their self-identity [ 35 , 37 ]. In comparison, social transmission of negative content can be attributed to the idea that certain negative content triggers activation, which drives user reactions [ 35 ].

Furthermore, previous research has shown that account features such as verification status and follower count affect user reactions on social media [ 13 , 16 , 34 ]. Given the vast amounts of information available in the digital age, the authenticity of user accounts becomes crucial in the diffusion of health information. One study revealed that account verification enhanced the number of likes and reposts for posts about COVID-19 vaccines on X [ 16 ]. Another study indicated that follower counts increased the number of reposts for posts about vaccines on X regardless of vaccine type [ 13 ].

Conspiracy Theories

A conspiracy theory refers to the belief that a coalition of powerholders forms secret agreements with malevolent intentions [ 38 , 39 ]. It differs from other types of misinformation by hypothesizing a pattern in which people, objects, or events are interconnected in a causal manner [ 39 ]. Previous research has revealed conspiracy theories as a salient theme in antivaccine discourse on social media, along with other themes such as side effects and inefficacy [ 40 , 41 ]. For HIV vaccines, conspiracy theories are crucial in understanding public discourse against them given the limited information about side effects and inefficacy until future success. An additional contribution of this study is the identification of prominent anti–HIV vaccine conspiracy theories through manual coding of the most engaged with negative posts.

Antivaccine conspiracy theories contribute to vaccine hesitancy [ 42 - 44 ], as observed recently with COVID-19 vaccines [ 10 , 11 ]. Understanding the themes and reasoning behind antivaccine conspiracy theories will provide vital implications for deploying evidence-based and logic-driven strategies to counter them [ 45 - 47 ]. A systematic review of antivaccine discourse on social media from 2015 to 2019 revealed pre–COVID-19 conspiracy theories [ 41 ]. These theories claimed that powerholders promoted vaccines for self-serving interests, including hiding vaccine side effects for financial gain and controlling society and the population [ 40 , 41 ]. During the COVID-19 pandemic, antivaccine conspiracy theories thrived on social media. Some theories claimed that the pandemic was invented for pharmaceutical companies’ profit from vaccines [ 44 ], whereas others linked mRNA COVID-19 vaccines to infertility and population control [ 10 , 11 , 44 , 48 , 49 ]. Another conspiracy theory claimed that Bill Gates and the US government aimed to implant trackable microchips into people through mass vaccination [ 11 , 27 , 49 ]. This aligns with conspiracy theories from earlier years. In particular, the Big Pharma conspiracy theory claims that pharmaceutical companies, together with politicians and other powerholders, conspire against the public interest [ 50 ]. The New World Order conspiracy theory alleges that a power elite with a globalization agenda colludes to rule the world [ 51 ]. Conspiracy theories have also linked other vaccines, such as poliovirus vaccines in the past [ 52 , 53 ] and COVID-19 vaccines in recent years, to HIV infection [ 54 , 55 ]. These conspiracy theories were based on the claims that alleged vaccines contained HIV.

Research Questions

To understand public discourse and reactions surrounding HIV vaccines on the X platform, we put forward the following research questions (RQs):

• What are the topics of the posts about HIV vaccines? (RQ 1)
• What is the valence of the posts about HIV vaccines? (RQ 2)
• How do topics and valence vary across different types of posts? (RQ 3)
• How do content features (topics and valence) and account features (verification status and follower count) affect 1-click reactions in terms of likes and reposts, respectively? (RQ 4)
• What are the prominent anti–HIV vaccine conspiracy theories that receive the most reactions? (RQ 5)

Data Source

We collected English-language original posts about HIV vaccines on the X platform from January 1, 2022, to December 31, 2022, using Netlytic [ 56 ]. The selected time frame began in January 2022 with the initiation of mRNA HIV vaccine clinical trials fueling public discussion and concluded in December 2022, a significant month for HIV and AIDS awareness marked by World AIDS Day on the first day of the month. Posts, excluding reposts, that contained both keywords (case insensitive)—“HIV” and “vaccine”—were extracted, resulting in a total of 36,424 posts across 365 days. Posts were collected weekly. Posts published from the last ending time point to at least 24 hours before each collection time point were included in the data set, allowing for a substantial reaction time.

The unit of analysis was a post. For each post, automated extraction produced data for user reactions (the number of likes and reposts) as well as account features (account verification status and follower count). All 36,424 posts underwent topic modeling using latent Dirichlet allocation (LDA) to identify latent topics, as well as sentiment analysis using Valence Aware Dictionary and Sentiment Reasoner (VADER) to access valence. LDA generated topic-specific loadings and identified the dominant topic for each post. VADER generated a valence compound score for each post, which was also categorized as positive, neutral, or negative based on standard VADER classification values.

LDA revealed 3 topics. As the topic of HIV and COVID-19 dominated in a large proportion of posts, we manually coded the 1000 most engaged posts containing the words “HIV” and “COVID” to uncover the salient aspects of HIV vaccines related to COVID-19. To develop coding for subtopics, 2 researchers initially reviewed and coded the top 200 posts with the most reactions. Subtopics were categorized by adapting existing categories from the literature [ 16 , 34 ] and integrating newly identified subtopics from the posts. The Scott π was 0.80 for categorizing subtopics. Subsequently, each researcher independently coded half of the remaining 800 posts.

We then conducted cross-tabulation analyses among all posts to examine the distribution of topics and valence among different types of posts. Furthermore, we conducted linear regression analyses among all posts to assess the influence of content and account features on these 1-click reactions. Of all 36,424 posts, 19,284 (52.94%) received ≥1 like, and 9155 (25.13%) received ≥1 repost. We added a constant value of 1 to all data points for likes and reposts before applying the natural logarithm. This was done to include posts with 0 likes or reposts and to mitigate the skewness of the data distribution.

Of the 28,439 posts that received likes or reposts, 6176 (21.72%) were negative. We manually coded the top 1000 negative posts with the most reactions to uncover prominent anti–HIV vaccine conspiracy theories. To develop coding for conspiracy theories, 2 researchers initially reviewed and coded the top 200 negative posts that received the most reactions. Posts containing conspiracy theories were identified based on expressions of postulated causal connections between people, objects, or events with malevolent intent [ 38 , 39 ]. Conspiracy theories were then classified based on the existing ones from the literature [ 50 , 51 ] and the emerging ones observed in the posts. Coding discrepancies were resolved through a further review of questionable posts and refinement of the conspiracy theories following the approach used in previous social media content analyses [ 40 , 57 ]. The procedure identified conspiracy theories and established intercoder reliability. The Scott π was 0.83 for identifying conspiracy theories and 0.81 for categorizing them. Each researcher then independently coded half of the remaining 800 negative posts.

User Reactions

One-click reactions were measured by the number of likes and reposts, which were automatically extracted. Because a small number of posts garnered significant 1-click reactions, the distribution of likes and reposts was right skewed. To reduce right skewness, we used the natural logarithm of the number of likes and reposts in linear regression analyses, as done in previous research [ 16 , 30 , 34 ].

Post Topics

All posts underwent topic modeling using LDA [ 58 ]. Topic modeling is a commonly used unsupervised learning method that generates a probabilistic model for a corpus of text data [ 59 ]. As a widely used topic model [ 59 ], LDA has been applied to discover topics within rich sources of digital health information, such as electronic health records [ 60 ], reviews on the web [ 61 ], and posts on X [ 16 , 34 ].

LDA relies on 2 matrices to define the underlying topical structure: the word-topic matrix and the document-topic matrix [ 62 ]. In this study, a post was considered a document. The general idea is that a post is represented by a Dirichlet distribution of latent topics, with each latent topic being represented by a Dirichlet distribution of words [ 59 ]. In the word-topic matrix, where the rows represent words and the columns represent topics, each element reveals the conditional probability of a word appearing within a topic [ 62 ]. A topic can be interpreted by examining a list of the most probable words ranked by their frequencies within a given topic using 3 to 30 words [ 63 ]. In the document-topic matrix, where rows represent posts and columns represent topics, each element reveals the conditional probability of a topic underlying a post [ 62 ]. In other words, it reveals the topic-specific loadings for each post.

When interpreting each topic, we reviewed the word-topic matrix as well as sample posts with high topic-specific loadings and significant reactions. LDA generated topic-specific loadings for each post ranging from 0 to 1, with values closer to 1 indicating a higher probability of a topic being associated with a post. Furthermore, LDA determined the dominant topic for each post by selecting the topic with the highest topic-specific loading among all topics. In the cross-tabulation analysis examining the distribution of topics across post types, the dominant topic for each post was entered for analysis. In the linear regression models assessing message-level drivers of user reactions, topic-specific loadings for each post were entered as topic values following previous research [ 16 , 34 ].

Post Valence

We used VADER to analyze the sentiment valence of each post. VADER is a rule-based model specifically attuned for assessing sentiments expressed in social media text [ 64 ]. VADER generated a compound valence score for each post ranging from –1 to 1, with a value of –1 indicating the most negative sentiment and a value of 1 indicating the most positive sentiment [ 65 ]. The standard VADER compound value thresholds for classifying valence categories are as follows: 0.05 to 1 for positive, −0.05 to 0.05 for neutral, and −0.05 to −1 for negative [ 65 ]. In the cross-tabulation analysis examining the distribution of valence among post types, the valence category for each post was entered for analysis. In the linear regression models assessing message-level drivers of user reactions, the VADER compound valence score for each post was used.

This study collected original posts excluding reposts. For each original post, it was automatically extracted whether it was a self-composed post, a quote post with comments, or a reply.

In total, 2 researchers manually coded the top 1000 out of 6176 negative posts with the highest total number of likes and reposts to uncover highly engaged conspiracy theories. They distinguished conspiracy theories from other types of negative information, particularly other types of misinformation, by recognizing the presence of a hypothesized pattern of causal connections between people, objects, or events for malicious intent [ 38 , 39 ]. Conspiracy theories were then categorized based on the existing ones from the literature and the emerging ones observed in the posts.

As an example, consider a post paraphrased as follows:

Image using condoms consistently, only to contract HIV from a COVID vaccine.

It was posted on February 9, 2022, and received 783 likes and 296 reposts. This post was not coded as displaying a conspiracy theory as it only presented misinformation suggesting that COVID-19 vaccines caused HIV. In comparison, another post was paraphrased as follows:

The COVID vaccine contained a spike protein derived from HIV. I was banned from saying this and ridiculed for months. Also, pharmacies stock up HIV self-tests.

It was posted on February 8, 2022, with 147 likes and 48 reposts. This post was coded as displaying a conspiracy theory. It was further classified within the category of conspiracy theories linked to COVID-19 vaccines containing, causing, or increasing HIV. This post suggested a hypothesized pattern of maliciously intended causal connections between the claim that the COVID-19 vaccine contained HIV and the stocking of HIV self-tests in pharmacies. As another example, a post was paraphrased as follows:

Scientists uncover a “highly virulent” strain of HIV in the Netherlands.

It was posted on February 12, 2022, and received 11 likes and 11 reposts. This post conveyed negative information but did not present a conspiracy theory. In comparison, another post was paraphrased as follows:

By coincidence again, the development of a new mRNA HIV vaccine began just before the emergence of the new HIV strain.

It was posted on February 8, 2022, and received 102 likes and 4 reposts. This post was coded as presenting a conspiracy theory and further classified into the category of conspiracy theories linked to the identification of a new highly virulent HIV strain. This post emphasized the speculative timing of the discovery of the new highly virulent HIV strain occurring shortly after the announcement of the development of a new mRNA HIV vaccine.

Account Features

For each post, the posting account’s verification status and follower count were automatically extracted.

Data Analysis

We used cross-tabulation analyses to investigate the distribution of topics and valence across different post types, in which the dominant topic and valence category for each post were entered, respectively, alongside the post type. We used linear regression models to examine the message-level drivers of user reactions among posts that received likes or reposts. In the linear regression models, a constant value of 1 was added to all data points of like and repost counts. The natural log-transformed values for each post were then regressed on 3 topic-specific loadings generated from LDA, the valence compound score generated from VADER, and 2 autoextracted account features—account verification status and follower count. The “plus one” technique was used to include posts that received 0 likes or reposts and to address the skewness of the data distribution.

Ethical Considerations

Following Long Island University’s institutional review board determination process, an institutional review board review was deemed unnecessary for this study, which collected and analyzed publicly available social media data. All referenced posts were paraphrased to avoid association with any particular user on the X platform.

RQ 1 asked about the topics present in all the posts. We trained a topic model using LDA exploring topic numbers ranging from 2 to 20. The optimal number of topics ( k ) was selected considering both the coherence score ( C v ) and the topic model visualization in a Python library called pyLDAvis , as done in previous research [ 16 , 66 ]. C v is a metric that reflects the semantic coherence of topics by evaluating the word co-occurrence likelihood within topics [ 67 ]. A higher C v indicates a better classification achieved by the topic model. In this study, the model with 2 topics ( k =2) yielded the highest C v (0.42), whereas the model with 3 topics ( k =3) yielded the second highest C v (0.35). The pyLDAvis chart depicts each topic as a circle. Overlapping areas between circles suggest similarities in topics. Thus, a chart without overlapping circles is preferable for k . The pyLDAvis chart for this study showed that, when the value of k was 2 or 3, the circles did not overlap. However, when k reached 4, the circles began to overlap, and overlapping circles persisted for values of k ranging from 4 to 20. Between the k values of 2 and 3, we opted for a model comprising 3 topics ( k =3) considering that a smaller number of topics tends to result in overly broad meanings for each topic [ 68 ].

Table 1 summarizes the 3 topics and lists their representative posts. Each topic was interpreted by examining the top 10 probable words ranked by frequency, along with sample posts exhibiting high topic-specific loadings and 1-click reactions. Topic 1 was HIV and COVID-19, covering 78% of the tokens [ 69 ] and dominating in 92.46% (33,678/36,424) of the posts. Topic 2 was mRNA HIV vaccine trials, covering 14% of the tokens and dominating in 5.91% (2151/36,424) of the posts. Topic 3 was HIV vaccine and immunity, covering 8% of the tokens and dominating in 1.63% (595/36,424) of the posts.

Figure 1 illustrates the daily numbers of original posts about HIV vaccines throughout 2022, in total and categorized into 3 topics. Moderna’s announcement of clinical trials for its first mRNA HIV vaccine on January 27, 2022, likely triggered the initial surge, culminating in a daily peak when the number of posts reached 805 on January 29, 2022. The daily number of posts about mRNA HIV vaccine trials (topic 2) in the week following Moderna’s announcement was higher than on other days throughout the year. Nevertheless, even during that week, there were higher daily numbers of posts about HIV and COVID-19 (topic 1), which remained dominant among the 3 topics during the entire year. The year’s second and highest daily peak occurred on February 8, 2022, recording a total of 1603 posts, most of which focused on HIV and COVID-19 (topic 1). This could be attributed to the emergence of new HIV-related events in early February 2022, including the promotion of HIV tests by public figures [ 64 ] and the discovery of a new highly virulent HIV strain [ 65 ]. The third highest daily peak, comprising 1085 posts, occurred on May 18, 2022, which has marked HIV Vaccine Awareness Day since 1998. Most of the posts centered on HIV and COVID-19 (topic 1). The remainder of the year did not reach such high peaks, with the largest daily volume of 205 posts occurring on December 2, 2022, the day following World AIDS Day, observed since 1988. Similar to previous daily peaks, most of the posts revolved around HIV and COVID-19 (topic 1).

The results revealed the dominance of HIV and COVID-19 (topic 1) in 92.46% (33,678/36,424) of the posts, with HIV as the most frequent word and COVID as the fourth most frequent word. To gain a deeper understanding of salient aspects of HIV vaccines related to COVID-19, we manually coded the top 1000 posts with the highest total number of likes and reposts that contained both HIV and COVID . Table 2 summarizes the subtopics and their representative posts with like and repost counts.

The first major subtopic, comprising 24% (240/1000) of the posts, focused on the reciprocal influence of HIV vaccines and COVID-19 vaccines on each other’s development. Years of HIV vaccine research facilitated the rapid development of mRNA COVID-19 vaccines, and the success of COVID-19 vaccines might accelerate the development of mRNA HIV vaccines. The second major subtopic, comprising 17.6% (176/1000) of the posts, involved comparisons between HIV and COVID-19 in various aspects. Specifically, the development speed of HIV vaccines compared to COVID-19 vaccines was a major point of comparison. In addition, some posts questioned whether potential HIV vaccines could be comparable to COVID-19 vaccines in terms of cost and accessibility during rollout. Others raised concerns about efficacy, safety, and inequality for both vaccines. The third major subtopic, comprising 26.5% (265/1000) of the posts, connected COVID-19 vaccines with HIV. One issue discussed was whether COVID-19 vaccines contained, caused, or increased HIV. Another issue raised was distinguishing between HIV symptoms and COVID-19 vaccine side effects, such as a fabricated condition called VAIDS , short for vaccine-acquired immunodeficiency syndrome. The fourth major subtopic, comprising 13.6% (136/1000) of the posts, featured conspiracy theories that presented hypothesized patterns linking COVID-19, HIV, and their vaccines with malicious intent. Prominent conspiracy theories in this subtopic included connecting misinformation that COVID-19 vaccines contain, cause, or increase HIV with the ongoing development of HIV vaccines; associating HIV and AIDS symptoms with side effects of COVID-19 vaccines; and claiming that COVID-19 originated from unsuccessful HIV vaccine research. As this study also manually coded the 1000 most engaged negative posts to identify prominent conspiracy theories, additional results pertaining to conspiracy theories will be discussed further in another subsection. The remaining posts related to HIV and COVID-19 included those that generally mentioned research on them or made connections without specifying details.

a mRNA: messenger RNA.

a The reaction count is the total number of likes and reposts.

b PrEP: pre-exposure prophylaxis.

c VAIDS: vaccine-acquired immunodeficiency syndrome.

d The categories labeled as “other” contain various topics. Thus, no representative post is displayed.

RQ 2 asked about the sentiment valence present in all the posts. According to the standard VADER classification values, valence is categorized by compound scores as follows: positive (0.05 to 1), neutral (−0.05 to 0.05), and negative (−0.05 to −1) [ 65 ]. On average, all posts had a marginally positive score of 0.053. HIV and COVID-19 (topic 1) had a slightly positive average score of 0.055. The mRNA HIV vaccine trials (topic 2) had a neutral average score of 0.040, leaning toward the positive side. HIV vaccine and immunity (topic 3) had a more neutral average score of −0.0008. Moreover, 42.78% (15,584/36,424) of the posts were positive, 25.64% (9338/36,424) of the posts were neutral, and 31.58% (11,502/36,424) of the posts were negative.

Topics and Valence Across Post Types

Of the 36,424 posts, 18,580 (51.01%) were replies, making up over half of the overall count. Self-composed posts totaled 41.6% (15,151/36,424), whereas the remaining 7.39% (2693/36,424) were quote posts. RQ 3 asked about the distribution of topics and valence among the 3 post types. As Table 3 shows, the distribution of topics varied by post type (N=36,424, χ 2 4 =2511.4, P <.001). Of the self-composed posts, 85.36% (12,933/15,151) focused on HIV and COVID-19 (topic 1) and 13.21% (2001/15,151) focused on mRNA HIV vaccine trials (topic 2). In comparison, quote posts and replies exhibited a different pattern, in each case >97% of posts centering on HIV and COVID-19 (topic 1; 2616/2693, 97.14% and 18,129/18,580, 97.57%, respectively).

As Table 4 shows, the distribution of valence also varied by post type (N=36,424, χ 2 4 =911.7, P <.001). The proportion of positive posts was slightly higher among self-composed posts at 44.95% (6810/15,151) compared to replies at 41.09% (7634/18,580) and quote posts at 42.33% (1140/2693). Self-composed posts had a smaller proportion of negative posts at 23.56% (3570/15,151) compared to replies at 37.64% (6994/18,580) and quote posts at 34.83% (938/2693). The proportion of neutral posts was larger for self-composed posts at 31.49% (4771/15,151) compared to quote posts at 22.84% (615/2693) and replies at 21.27% (3952/18,580).

Regarding the distribution of topics and valence among the 3 types of posts, quote posts and replies displayed similarities, whereas self-composed posts diverged. Compared to self-composed posts, which initiate new conversations, there was a higher proportion of HIV and COVID-19-related posts (topic 1) and a greater proportion of negative posts among quote posts and replies, which contribute to existing conversations.

a N=36,424, χ 2 4 =2511.4, P <.001.

b mRNA: messenger RNA.

a N=36,424, X 2 4 =911.7, P <.001.

Content and Account Features Influencing User Reactions

RQ 4 asked about the influence of content and account features on likes and reposts.

Liking is more common than reposting. While 52.94% (19,284/36,424) of posts received an average of 24.83 likes, ranging from 1 to 102,843, a total of 25.13% (9155/36,424) posts received an average of 11.38 reposts, ranging from 1 to 10,572. Table 5 reveals the influence of content features (topics and valence) and account features (verification status and follower count) on the natural log-transformed number of likes and reposts. Both linear regression models were significant at P <.001. The adjusted  R 2 was 0.072 for the like model and 0.090 for the repost model.

Among the 3 topics identified using LDA, HIV and COVID-19 (topic 1) did not affect like counts but decreased repost counts. In comparison, mRNA HIV vaccine trials (topic 2) decreased like counts while increasing repost counts. Positive valence increased like and repost counts. Account verification status and follower count increased like and repost counts.

a The natural logarithm, ln (Y i +1), was calculated on like and repost counts. This transformation was conducted to include posts receiving 0 likes and reposts, as well as to account for the skewness of the data distribution.

b F (model significance): P <.001; adjusted R 2 =0.072.

c F (model significance): P <.001; adjusted R 2 =0.090.

d mRNA: messenger RNA.

e The models excluded topic 3 on HIV vaccine and immunity to address multicollinearity issues arising from its correlations with topics 1 and 2. The reported standard β for topic 3 represents a possible β value if it had been included in the models.

Posts With Most Reactions

Table 6 summarizes posts ranked within the top 5 for the number of likes and reposts presented in chronological order. It is worth noting that all posts in the top 5 for likes and reposts were self-composed. One particular post, which garnered the most likes (n=102,843) and reposts (n=10,572), expressed the incredible feeling of witnessing the development of an HIV vaccine within our lifetimes. It was posted by an unverified account on January 28, 2022, the day after Moderna’s announcement of clinical trials for its first mRNA HIV vaccine.

a Ranks beyond the fifth were not indicated.

Anti–HIV Vaccine Conspiracy Theories

RQ 5 asked about prominent anti–HIV vaccine conspiracy theories. Of the 1000 negative posts that received the most reactions, 227 (22.7%) contained conspiracy theories. As Table 7 shows, we classified these prominent anti–HIV vaccine conspiracy theories into 4 categories and presented their representative posts and the number of reactions.

The first category, comprising 44.9% (102/227) of the posts, formulated conspiracy theories by connecting COVID-19, COVID-19 vaccines, HIV, and HIV vaccines. For instance, 52.9% (54/102) of these posts connected the misinformation regarding COVID-19 vaccines containing, causing, or increasing HIV with the ongoing efforts to develop HIV vaccines. This misinformation may have arisen from past occurrences resurfacing following Moderna’s initiation of its mRNA HIV vaccine trials. One incident occurred at the end of 2020, when an Australian COVID-19 vaccine, which used a small fragment of protein from HIV to clamp SARS-CoV-2’s spike proteins, was abandoned due to false HIV-positive results [ 70 ]. Another incident occurred in October 2020, when 4 researchers sent a letter to a medical journal expressing concerns about the potential increased risk of HIV acquisition among men receiving COVID-19 vaccines using adenovirus type-5 vectors without supporting data from COVID-19 vaccines [ 71 ]. The misinformation typically interpreted the incidents out of context and generally suggested that COVID-19 vaccines contained, caused, or increased HIV without specifying details. In addition, there were conspiracy theories linking HIV and AIDS to COVID-19 vaccine side effects, including a fabricated condition known as VAIDS. VAIDS falsely suggests that COVID-19 vaccines caused immune deficiency [ 72 ]. Furthermore, there were claims that COVID-19 originated from unsuccessful HIV vaccine research.

The second category, comprising 38.3% (87/227) of the posts, suggested that the alignment of concurrent events with Moderna’s start of mRNA HIV vaccine trials in late January 2022 was intentional to manipulate the market for HIV vaccines. These events included the rising HIV discussion and fear; promotion of HIV tests by public figures [ 73 ]; the discovery of a new highly virulent HIV strain [ 74 ]; and the passing away of HIV researchers, including Luc Montagnier, codiscoverer of HIV with an antivaccine stance during the COVID-19 pandemic [ 75 ], all occurring in early February 2022.

The third category, with 11.5% (26/227) of the posts, revealed conspiracy theories based on the distrust of powerholders [ 76 ]. Some posts extended existing conspiracy theories, such as the Big Pharma conspiracy theory [ 50 ] and the New World Order conspiracy theory [ 51 ], into the context of HIV vaccines, emphasizing the intent of powerholders, including major pharmaceutical companies and governments, behind vaccine promotion for financial profits and society control. Other posts created conspiracy theories about the government’s research on HIV vaccines. The remaining posts generally stated that HIV vaccines were a scam. The final category comprised the remaining 5.3% (12/227) of the posts with other conspiracy theories.

It is worth noting that, of the 227 posts containing conspiracy theories, 39 (17.2%) were posted by accounts that had already been suspended at the time of manual coding. For these posts, the X platform displays the following message—“This post is from a suspended account”—and the content of the post is not visible. The X platform suspends accounts that violate its rules [ 77 ]. However, specific details of the violations are not accessible on the platform. The invisibility of these posts halted their spread when the suspension was enacted. For our manual coding of these posts, we used the text obtained during the data collection process.

b The posts were from suspended accounts.

d The categories labeled as “other” contain various conspiracy theories. Thus, no representative post is displayed.

Principal Findings

This study investigated the patterns of public discourse and the message-level drivers of user reactions on the X platform regarding HIV vaccines through the analysis of posts using machine learning algorithms. We examined the distribution of topics and valence across different post types and assessed the influence of content features (topics and valence) and account features (account verification status and follower count) on like and repost counts. In addition, we manually coded the 1000 most engaged posts about HIV and COVID-19 to understand the salient aspects of HIV vaccines related to COVID-19 and the 1000 most engaged negative posts to identify prominent anti–HIV vaccine conspiracy theories.

The results revealed that COVID-19 plays a substantial role as a context for public discourse and reactions regarding HIV vaccines. Of the 3 topics identified using LDA, the leading topic was HIV and COVID-19, covering 78% of tokens and dominating in 92.46% (33,678/36,424) of the posts. Furthermore, on each of the top 4 days with the highest post counts, most of the posts were about HIV and COVID-19. This comprehensive topic included important subtopics that linked HIV vaccines with COVID-19 vaccines, as demonstrated through the manual coding of the 1000 most engaged posts about HIV and COVID-19. These subtopics encompassed the reciprocal influence of HIV vaccines and COVID-19 vaccines in advancing each other’s development; comparisons in their development speed; inquiries about the possible alignment of HIV vaccines with COVID-19 vaccines in terms of cost and accessibility during distribution; and concerns about efficacy, safety, and equality for both vaccines.

COVID-19 positioned HIV vaccines in both a positive and negative context. On the one hand, the success of mRNA technology in COVID-19 vaccines [ 6 ] potentially cast mRNA HIV vaccines in a positive light. The topic of HIV and COVID-19 had a marginally positive valence score of 0.055. Moreover, 3 (60%) out of the 5 most liked posts and 2 (40%) out of the 5 most reposted posts expressed excitement about advancements in HIV vaccines that were based on the experience with COVID-19 vaccines. On the other hand, antivaccine discourse, including conspiracy theories, heated up during the COVID-19 pandemic [ 10 , 11 , 27 , 44 , 48 , 49 ], which posed challenges to HIV vaccines. Of the 1000 most engaged posts about HIV and COVID-19, a total of 136 (13.6%) featured conspiracy theories. Of the 1000 most engaged negative posts, 227 (22.7%) contained conspiracy theories, with 102 (44.9%) of them revolving around HIV and COVID-19. For instance, a prominent conspiracy theory connected the misinformation about COVID-19 vaccines containing, causing, or increasing HIV infection [ 55 ] with the initiation of clinical trials for mRNA HIV vaccines [ 4 , 5 ], implying a malevolent intent behind the deliberate connection. The results indicate that conspiracy theories tend to elicit an approach-oriented response, as evidenced by people engaging in liking and reposting, as opposed to an avoidance-oriented approach [ 39 ]. This underscores the need to intensify efforts to counter conspiracy theories in public health communication about HIV vaccines.

According to a study conducted by the Pew Research Center, irrespective of the subject matter, replies constituted the largest portion of original posts on X, followed by self-composed and quote posts [ 28 ]. Specifically, the number of replies was 3 times greater than that of self-composed posts. In this study, although replies constituted slightly more than half (18,580/36,424, 51.01%) of the posts, it is worth noting that the subject of HIV vaccines elicited a higher proportion of self-composed posts at 41.6% (15,151/36,424). Specifically, the number of replies was 23% higher than that of self-composed posts. Moreover, the topic of mRNA vaccine trials was most evident in self-composed posts compared to replies and quote posts. In comparison, there was a higher proportion of focus on the topic of HIV and COVID-19 and a greater proportion of negative posts among quote posts and replies, which contribute to existing conversations. This suggests that users were more likely to initiate new conversations rather than joining existing conversations about mRNA HIV vaccines. In contrast, they were more likely to join existing conversations rather than starting new conversations about HIV and COVID-19. In addition, users were less likely to initiate new conversations negatively but more likely to contribute negatively to existing ones.

As the primary topic, HIV and COVID-19 had no impact on like counts but had a negative impact on repost counts. In comparison, the topic of mRNA HIV vaccine trials had a negative impact on like counts and a positive impact on repost counts. The results should be interpreted while considering that, as revealed in previous research [ 16 , 34 ] and this study, most posts on the X platform are unlikely to receive likes and even less likely to receive reposts. In this study, among the total of 36,424 posts, approximately half (n=19,284, 52.94%) received likes, and approximately one-quarter (n=9155, 25.13%) received reposts. To include all posts and mitigate the data distribution skewness in the linear regression analysis, we applied the “plus one” technique. This involved adding a constant value of 1 to all like and repost data points before taking the natural logarithm. Although most posts were not liked or reposted, it is noteworthy that the topic of mRNA HIV vaccines led to an increase in repost counts, highlighting its positive influence on social sharing. In addition, 2 (40%) out of the 5 most reposted posts were about mRNA HIV vaccine trials. These results correspond to the findings of previous research that suggested the diffusion of novel useful information [ 12 , 16 , 32 , 36 ].

The overall valence of the posts about HIV vaccines was marginally positive. The positivity aligns with the positive sentiment found in posts on X about vaccines in general [ 13 - 15 ] and COVID-19 vaccines in particular [ 12 , 16 , 17 ]. However, the positivity about HIV vaccines was not apparent as the average score of 0.053 placed it on the edge of the neutral range, which goes from −0.05 to 0.05 according to the standard VADER classification values. Positive sentiment had a favorable impact on like and repost counts, partially consistent with findings of previous research on COVID-19 vaccines [ 16 ]. The post that achieved the most likes conveyed the incredible feeling of witnessing the development of an HIV vaccine in our lifetimes. This could be attributed to the psychological rationale that social transmission of positive content fulfills people’s motivation to present a positive image [ 35 , 37 ]. In alignment with the findings of previous research [ 13 , 16 , 34 ], account verification status and follower count increased like and repost counts.

This study has implications for public health communication related to HIV vaccines and potentially other vaccines. Given the massive scale of the COVID-19 vaccination campaign, it is understandable that people will draw comparisons with other vaccines. Topic modeling identified HIV and COVID-19 as the primary topic, and manual coding revealed various intertwined aspects. Leveraging the advantages observed in the COVID-19 vaccine campaign, such as its widespread accessibility, could be valuable. Furthermore, addressing common concerns such as efficacy, safety, and inequality could also prove beneficial.

In the case of HIV vaccines, it is essential to tackle concerns associated with COVID-19 vaccines, especially those related to HIV vaccines. A major subtopic of HIV and COVID-19 involved suspicions about COVID-19 vaccines containing, causing, or increasing HIV. Another major subtopic was the confusion between HIV symptoms and the alleged side effects of COVID-19 vaccines, such as VAIDS. Misinformation concerning both subtopics has been woven into conspiracy theories, further complicating this situation. To combat misinformation and conspiracies that have these elements, efforts could focus on promoting evidence-based factual information [ 45 - 47 ].

Another notable technique in the conspiracy theories was linking concurrent COVID-19 and other HIV-related events in unsubstantiated relationships to create false perceptions, suggesting that these events were intentional to manipulate the market for HIV vaccines. These HIV-related events included rising HIV discussion and fear, promotion of HIV tests by public figures [ 73 ], the discovery of a new highly virulent HIV strain [ 74 ], and the passing away of HIV researchers, all occurring in early February 2022. These findings suggest that refuting false connections among such concurrent events can be an effective strategy to counter these conspiracy theories [ 45 - 47 ]. These occurrences, frequently entwined within conspiracy theories, could be specifically addressed in public health communication efforts.

Limitations

This study has several limitations. Because we used autoidentified content features (topics and valence) and autoextracted account features (verification status and follower count) in the regression models to predict the autoextracted number of user reactions (likes and reposts), the results were mostly limited to the examined autoidentified and autoextracted factors. For instance, political polarization, which manifested in a wide range of issues, including response to vaccines [ 78 ], could be a factor worth investigating in future studies. Furthermore, manual coding of conspiracy theories revealed a prevalent technique of twisting concurrent events into false relationships. This underscores the significance of refuting unfounded associations among these incidents to counter such conspiracy theories. It will be interesting for future research to assess the impact of this technique on user reactions to conspiracy theories. These findings could provide further insights into public health communication strategies to combat conspiracy theories.

Conclusions

The results highlight COVID-19 as a significant backdrop for public discourse and reactions on the X platform regarding HIV vaccines. COVID-19 situated HIV vaccines in both a positive and negative context. The success of mRNA COVID-19 vaccines shed a positive light on HIV vaccines. However, COVID-19 also situated HIV vaccines in a negative context, as evident in anti–HIV vaccine conspiracy theories falsely linking HIV vaccines to COVID-19. The findings provide implications for public health communication strategies concerning HIV vaccines.

Acknowledgments

This study was supported in part by the College of Arts and Sciences and the Harrington School of Communication and Media at the University of Rhode Island. The authors express their appreciation for the support. The funders had no role in study design, data collection, data analysis, data interpretation, or writing of the report.

Data Availability

The data sets collected and analyzed during this study are available from the corresponding author upon request.

Conflicts of Interest

None declared.

• Hussain A, Ali S, Ahmed M, Hussain S. The anti-vaccination movement: a regression in modern medicine. Cureus. Jul 03, 2018;10(7):e2919. [ FREE Full text ] [ CrossRef ] [ Medline ]
• History of HIV vaccine research. National Institute of Allergy and Infectious Diseases. URL: https://www.niaid.nih.gov/diseases-conditions/hiv-vaccine-research-history [accessed 2023-09-20]
• HIV surveillance report. Centers for Disease Control and Prevention. URL: https://www.cdc.gov/hiv/library/reports/hiv-surveillance/vol-34/index.html [accessed 2023-09-20]
• IAVI and Moderna launch trial of HIV vaccine antigens delivered through mRNA technology. Moderna. Jan 27, 2022. URL: https:/​/investors.​modernatx.com/​news/​news-details/​2022/​IAVI-and-Moderna-Launch-Trial-of-HIV-Vaccine-Antigens-Delivered-Through-mRNA-Technology/​default.​aspx [accessed 2023-09-20]
• NIH launches clinical trial of three mRNA HIV vaccines. National Institutes of Health. Mar 14, 2022. URL: https://www.nih.gov/news-events/news-releases/nih-launches-clinical-trial-three-mrna-hiv-vaccines [accessed 2024-09-20]
• Tenforde MW, Self WH, Gaglani M, Ginde AA, Douin DJ, Talbot HK, et al. Effectiveness of mRNA vaccination in preventing COVID-19-associated invasive mechanical ventilation and death - United States, March 2021-January 2022. MMWR Morb Mortal Wkly Rep. Mar 25, 2022;71(12):459-465. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Press release. Nobel Prize Outreach. URL: https://www.nobelprize.org/prizes/medicine/2023/press-release/ [accessed 2023-10-04]
• May M. After COVID-19 successes, researchers push to develop mRNA vaccines for other diseases. Nat Med. Jun 2021;27(6):930-932. [ FREE Full text ] [ CrossRef ] [ Medline ]
• mRNA vaccine technology: a promising idea for fighting HIV. National Institute of Health. Feb 20, 2023. URL: https://covid1nih.gov/news-and-stories/mrna-vaccine-technology-promising-idea-fighting-hiv [accessed 2023-09-20]
• Abbasi J. Widespread misinformation about infertility continues to create COVID-19 vaccine hesitancy. JAMA. Mar 15, 2022;327(11):1013-1015. [ CrossRef ] [ Medline ]
• Islam MS, Kamal AH, Kabir A, Southern DL, Khan SH, Hasan SM, et al. COVID-19 vaccine rumors and conspiracy theories: the need for cognitive inoculation against misinformation to improve vaccine adherence. PLoS One. May 12, 2021;16(5):e0251605. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Lyu JC, Han EL, Luli GK. COVID-19 vaccine-related discussion on Twitter: topic modeling and sentiment analysis. J Med Internet Res. Jun 29, 2021;23(6):e24435. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Blankenship EB, Goff ME, Yin J, Tse ZT, Fu KW, Liang H, et al. Sentiment, contents, and retweets: a study of two vaccine-related Twitter datasets. Perm J. Sep 1, 2018;22:17-138. [ CrossRef ]
• Gunaratne K, Coomes EA, Haghbayan H. Temporal trends in anti-vaccine discourse on Twitter. Vaccine. Aug 14, 2019;37(35):4867-4871. [ CrossRef ] [ Medline ]
• Love B, Himelboim I, Holton A, Stewart K. Twitter as a source of vaccination information: content drivers and what they are saying. Am J Infect Control. Jun 2013;41(6):568-570. [ CrossRef ] [ Medline ]
• Zhang J, Wang Y, Shi M, Wang X. Factors driving the popularity and virality of COVID-19 vaccine discourse on Twitter: text mining and data visualization study. JMIR Public Health Surveill. Dec 03, 2021;7(12):e32814. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hussain A, Tahir A, Hussain Z, Sheikh Z, Gogate M, Dashtipour K, et al. Artificial intelligence-enabled analysis of public attitudes on Facebook and Twitter toward COVID-19 vaccines in the United Kingdom and the United States: observational study. J Med Internet Res. Apr 05, 2021;23(4):e26627. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Eysenbach G. Infodemiology and infoveillance: framework for an emerging set of public health informatics methods to analyze search, communication and publication behavior on the internet. J Med Internet Res. Mar 27, 2009;11(1):e11. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Burgess R, Feliciano JT, Lizbinski L, Ransome Y. Trends and characteristics of #HIVPrevention Tweets posted between 2014 and 2019: retrospective infodemiology study. JMIR Public Health Surveill. Aug 11, 2022;8(8):e35937. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Tran HT, Lu SH, Tran HT, Nguyen BV. Social media insights during the COVID-19 pandemic: infodemiology study using big data. JMIR Med Inform. Jul 16, 2021;9(7):e27116. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Sigalo N, Awasthi N, Abrar SM, Frias-Martinez V. Using COVID-19 vaccine attitudes on Twitter to improve vaccine uptake forecast models in the United States: infodemiology study of Tweets. JMIR Infodemiology. Aug 21, 2023;3:e43703. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Taggart T, Grewe ME, Conserve DF, Gliwa C, Roman Isler M. Social media and HIV: a systematic review of uses of social media in HIV communication. J Med Internet Res. Nov 02, 2015;17(11):e248. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chan MP, Morales A, Zlotorzynska M, Sullivan P, Sanchez T, Zhai C, et al. Estimating the influence of Twitter on pre-exposure prophylaxis use and HIV testing as a function of rates of men who have sex with men in the United States. AIDS. May 01, 2021;35(Suppl 1):S101-S109. [ CrossRef ] [ Medline ]
• The world needs an HIV vaccine. International AIDS Vaccine Initiative. URL: https://www.iavi.org/wp-content/uploads/2024/02/iavi_fact_sheet_need-for-hiv-vaccine.pdf [accessed 2024-03-15]
• AIDS and the sustainable development goals. United Nations Programme on HIV/AIDS. URL: https://www.unaids.org/en/AIDS_SDGs [accessed 2023-09-20]
• Using X. Twitter. URL: https://help.twitter.com/en/using-x [accessed 2024-09-20]
• Trevisan M, Vassio L, Giordano D. Debate on online social networks at the time of COVID-19: an Italian case study. Online Soc Netw Media. May 2021;23:100136. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chapekis A, Smith A. How U.S. adults on Twitter use the site in the Elon Musk era. Pew Research Center. May 17, 2023. URL: https:/​/www.​pewresearch.org/​short-reads/​2023/​05/​17/​how-us-adults-on-twitter-use-the-site-in-the-elon-musk-era/​ [accessed 2023-09-20]
• Lee J, Hong IB. Predicting positive user responses to social media advertising: the roles of emotional appeal, informativeness, and creativity. Int J Inf Manage. Jun 2016;36(3):360-373. [ CrossRef ]
• Pancer E, Poole M. The popularity and virality of political social media: hashtags, mentions, and links predict likes and retweets of 2016 U.S. presidential nominees’ tweets. Soc Influ. 2016;11(4):259-270. [ CrossRef ]
• Boulianne S, Larsson AO. Engagement with candidate posts on Twitter, Instagram, and Facebook during the 2019 election. New Media Soc. 2023;25(1):119-140. [ CrossRef ]
• Kim HS. Attracting views and going viral: how message features and news-sharing channels affect health news diffusion. J Commun. Jun 01, 2015;65(3):512-534. [ FREE Full text ] [ CrossRef ] [ Medline ]
• About your For you timeline on X. Twitter. URL: https://help.twitter.com/en/using-twitter/twitter-timeline [accessed 2023-09-20]
• Nanath K, Joy G. Leveraging Twitter data to analyze the virality of Covid-19 tweets: a text mining approach. Behav Inf Technol. 2023;42(2):196-214. [ CrossRef ]
• Berger J, Milkman KL. What makes online content viral? J Market Res. 2012;49(2):192-205. [ CrossRef ]
• Tellis GJ, MacInnis DJ, Tirunillai S, Zhang Y. What drives virality (sharing) of online digital content? the critical role of information, emotion, and brand prominence. J Market. 2019;83(4):1-20. [ CrossRef ]
• Wojnicki AC, Godes D. Word-of-mouth as self-enhancement. SSRN J. Jun 16, 2006. [ CrossRef ]
• Pummerer L, Böhm R, Lilleholt L, Winter K, Zettler I, Sassenberg K. Conspiracy theories and their societal effects during the COVID-19 pandemic. Soc Psychol Pers Sci. 2022;13(1):49-59. [ CrossRef ]
• van Prooijen JW, van Vugt M. Conspiracy theories: evolved functions and psychological mechanisms. Perspect Psychol Sci. Nov 2018;13(6):770-788. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Massey PM, Kearney MD, Hauer MK, Selvan P, Koku E, Leader AE. Dimensions of misinformation about the HPV vaccine on Instagram: content and network analysis of social media characteristics. J Med Internet Res. Dec 03, 2020;22(12):e21451. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wawrzuta D, Jaworski M, Gotlib J, Panczyk M. Characteristics of antivaccine messages on social media: systematic review. J Med Internet Res. Jun 04, 2021;23(6):e24564. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garett R, Young SD. Online misinformation and vaccine hesitancy. Transl Behav Med. Dec 14, 2021;11(12):2194-2199. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jolley D, Douglas KM. The effects of anti-vaccine conspiracy theories on vaccination intentions. PLoS One. Feb 20, 2014;9(2):e89177. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ullah I, Khan KS, Tahir MJ, Ahmed A, Harapan H. Myths and conspiracy theories on vaccines and COVID-19: potential effect on global vaccine refusals. Vacunas. 2021;22(2):93-97. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chan MP, Jones CR, Hall Jamieson K, Albarracín D. Debunking: a meta-analysis of the psychological efficacy of messages countering misinformation. Psychol Sci. Nov 2017;28(11):1531-1546. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ecker UK, Lewandowsky S, Cook J, Schmid P, Fazio LK, Brashier N, et al. The psychological drivers of misinformation belief and its resistance to correction. Nat Rev Psychol. Jan 12, 2022;1:13-29. [ CrossRef ]
• Jolley D, Douglas KM. Prevention is better than cure: addressing anti‐vaccine conspiracy theories. J Appl Soc Psychol. Aug 2017;47(8):459-469. [ CrossRef ]
• Dotto C. Vaccine infertility claims on YouTube sweep across fringe platforms. First Draft News. Mar 24, 2021. URL: https://firstdraftnews.org/articles/vaccine-infertility-claims-youtube-fringe/ [accessed 2023-09-20]
• Hamei L, Lopes L, Kirzinger A, Sparks G, Stokes M, Brodie M. KFF COVID-19 vaccine monitor: media and misinformation. Kaiser Family Foundation. Nov 8, 2021. URL: https:/​/www.​kff.org/​coronavirus-covid-19/​poll-finding/​kff-covid-19-vaccine-monitor-media-and-misinformation/​ [accessed 2023-09-20]
• Blaskiewicz R. The Big Pharma conspiracy theory. Med Writ. 2013;22(4):259-261. [ CrossRef ]
• Spark A. Conjuring order: the new world order and conspiracy theories of globalization. Sociol Rev. 2000;48(2_suppl):46-62. [ CrossRef ]
• Aylward RB, Heymann DL. Can we capitalize on the virtues of vaccines? insights from the polio eradication initiative. Am J Public Health. May 2005;95(5):773-777. [ CrossRef ] [ Medline ]
• Jegede AS. What led to the Nigerian boycott of the polio vaccination campaign? PLoS Med. Mar 2007;4(3):e73. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fact check: an Australian vaccine trial did not give trial participants HIV. Reuters. Dec 18, 2020. URL: https://www.reuters.com/article/idUSKBN28R2WM/ [accessed 2023-09-20]
• COVID-19 vaccines do not cause HIV or AIDS. Reuters. Feb 21, 2022. URL: https:/​/www.​reuters.com/​article/​factcheck-vaccines-hiv/​fact-check-covid-19-vaccines-do-not-cause-hiv-or-aids-idUSL1N2UW10H [accessed 2023-09-20]
• Gruzd A. Netlytic: software for automated text and social network analysis. Netlytic. 2016. URL: https://netlytic.org/home/?page_id=49 [accessed 2023-09-20]
• Massey PM, Leader A, Yom-Tov E, Budenz A, Fisher K, Klassen AC. Applying multiple data collection tools to quantify human papillomavirus vaccine communication on Twitter. J Med Internet Res. Dec 05, 2016;18(12):e318. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Blei DM, Ng AY, Jordan MI. Latent Dirichlet allocation. J Mach Learn Res. 2003;3:993-1022. [ FREE Full text ]
• Allahyari M, Pouriyeh S, Assefi M, Safaei S, Trippe ED, Gutierrez JB, et al. Text summarization techniques: a brief survey. Int J Adv Comput Sci Appl. 2017;8(10):397-405. [ CrossRef ]
• Fohner AE, Greene JD, Lawson BL, Chen JH, Kipnis P, Escobar GJ, et al. Assessing clinical heterogeneity in sepsis through treatment patterns and machine learning. J Am Med Inform Assoc. Dec 01, 2019;26(12):1466-1477. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wallace BC, Paul MJ, Sarkar U, Trikalinos TA, Dredze M. A large-scale quantitative analysis of latent factors and sentiment in online doctor reviews. J Am Med Inform Assoc. 2014;21(6):1098-1103. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Maier D, Waldherr A, Miltner P, Wiedemann G, Niekler A, Keinert A, et al. Applying LDA topic modeling in communication research: toward a valid and reliable methodology. Commun Methods Meas. 2018;12(2-3):93-118. [ CrossRef ]
• Sievert C, Shirley K. LDAvis: a method for visualizing and interpreting topics. In: Proceedings of the Workshop on Interactive Language Learning, Visualization, and Interfaces. 2014. Presented at: ILLBI 2014; June 27, 2014; Baltimore, MD. [ CrossRef ]
• Hutto C, Gilbert E. VADER: a parsimonious rule-based model for sentiment analysis of social media text. Proc Int AAAI Conf Web Soc Media. May 16, 2014;8(1):216-225. [ CrossRef ]
• VADER-sentiment-analysis. GitHub. URL: https://github.com/cjhutto/vaderSentiment [accessed 2023-09-20]
• Feng Y, Chen H, Kong Q. An expert with whom i can identify: the role of narratives in influencer marketing. Int J Advert. 2021;40(7):972-993. [ CrossRef ]
• Röder M, Both A, Hinneburg A. Exploring the space of topic coherence measures. In: Proceedings of the Eighth ACM International Conference on Web Search and Data Mining. 2015. Presented at: WSDM '15; February 2-6, 2015; Shanghai, China. [ CrossRef ]
• Gan J, Qi Y. Selection of the optimal number of topics for LDA topic model-taking patent policy analysis as an example. Entropy (Basel). Oct 03, 2021;23(10):1301. [ FREE Full text ] [ CrossRef ] [ Medline ]
• API documentation. pyLDAvis. URL: https://pyldavis.readthedocs.io/en/latest/modules/API.html [accessed 2023-09-20]
• Horizon special: the vaccine. British Broadcasting Corporation. 2021. URL: https://www.bbc.co.uk/programmes/m000x2tf [accessed 2023-09-20]
• Buchbinder SP, McElrath MJ, Dieffenbach C, Corey L. Use of adenovirus type-5 vectored vaccines: a cautionary tale. Lancet. Oct 31, 2020;396(10260):e68-e69. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kelety J. COVID-19 vaccines don’t contain HIV. AP News. Jan 30, 2023. URL: https://apnews.com/article/fact-check-coronavirus-vaccine-hiv-185375755407 [accessed 2023-09-20]
• Davies C. Prince Harry: get tested for HIV to protect others in same way as for Covid. Guardian News and Media Limited. Feb 10, 2022. URL: https:/​/www.​theguardian.com/​uk-news/​2022/​feb/​10/​prince-harry-get-tested-for-hiv-to-protect-others-in-same-way-as-covid-diana-princess-of-wales [accessed 2024-03-06]
• Wymant C, Bezemer D, Blanquart F, Ferretti L, Gall A, Hall M, et al. A highly virulent variant of HIV-1 circulating in the Netherlands. Science. Feb 04, 2022;375(6580):540-545. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Epstein RH. Luc Montagnier, nobel-winning co-discoverer of H.I.V., dies at 89. The New York Times. Feb 10, 2022. URL: https://www.nytimes.com/2022/02/10/science/luc-montagnier-dead.html [accessed 2023-09-20]
• Bunting H, Gaskell J, Stoker G. Trust, mistrust and distrust: a gendered perspective on meanings and measurements. Front Polit Sci. Jul 6, 2021;3(642129):1-15. [ CrossRef ]
• About suspended accounts. Twitter. URL: https://help.twitter.com/en/managing-your-account/suspended-twitter-accounts [accessed 2023-09-20]
• Fridman A, Gershon R, Gneezy A. COVID-19 and vaccine hesitancy: a longitudinal study. PLoS One. Apr 16, 2021;16(4):e0250123. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by G Eysenbach; submitted 04.10.23; peer-reviewed by X Ma, J Zhang; comments to author 18.10.23; revised version received 08.11.23; accepted 28.02.24; published 03.04.24.

©Jueman M Zhang, Yi Wang, Magali Mouton, Jixuan Zhang, Molu Shi. Originally published in the Journal of Medical Internet Research (https://www.jmir.org), 03.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in the Journal of Medical Internet Research, is properly cited. The complete bibliographic information, a link to the original publication on https://www.jmir.org/, as well as this copyright and license information must be included.

Snapsolve any problem by taking a picture. Try it in the Numerade app?