literature review human service

• For Students
• For Faculty
• Interlibrary Loan
• Request an Item from Everett Library
• Study Rooms & Labs
• Archives & Special Collections
• Accessibility Guide
• Citing Your Sources
• Off-Campus Access
• Online Tutorials
• Library 101
• Library Staff
• Library Hours
• Library Diversity Statement
• Friends of the Library
• History of Everett Library
• Give to the Library
• Library Mission & Vision
• Library Policies

Service Alert

Human Service Studies

• What are Scholarly Articles?
• Find Articles
• Evaluate Sources

What is a Literature Review?

What does a lit review do, step 1: define/refine topic, step 2: determine approach, step 3: research effectively, step 4: analyze and evaluate, step 5: read critically, step 6: take great notes and be organized, step 7: draft your written product, step 8: edit and refine, ask for help, get lit: the literature review video.

• Cite Sources
• Plagiarism This link opens in a new window

A lit review surveys, summarizes, and links information about a given topic. It's a synthesis of information relevant to your work.

Even Better

A  good  lit review assesses this information and distills it for the reader.

The best lit reviews tell a good story.

• Purdue OWL Lit Review Example For a visual example of a lit review, check out this student sample from the Purdue OWL. The lit review begins on p. 6.
• Clarifies understanding of the field
• Explains the rationale for your research
• Places your research within a broader context
• Evaluates the results of previous research
• Defines key concepts and ideas
• Identifies research in related areas that is generalizable or transferable to your topic
• Identifies relevant methodological issues

What's your current knowledge on the topic? 

Do you know what you don't know? 

This is an iterative process, meaning that you're going to come back to defining and refining your topic several times. But it's easier if you begin focused and with some general knowledge already.

Develop a set of questions to be applied to all of the articles.

How will you structure your lit review?

• Thematically?
• NOT by author!

The literature you choose will inform and underpin everything you write, so plan your searches carefully!

This is where I come in. My contact information is on this page for you to use! Let me know what I can do to help.

Make Sure You're...

Using high quality sources

Using a variety of sources

Mining references for sources, looking for repetitions and reviews

Connecting the dots

When you're gathering information for your lit review, you won't use everything you collect. Only use the best sources. 

• What is your research question, and how does this material relate? 
• Are there foundational articles that must be included?
• How is the topic framed? 
• Are there fringe works that should not be included? 
• Is there a central debate that should be acknowledged and addressed?
• Where is the topic headed in the future?

Ask Yourself

• What is the source? 
• What is the claim? 
• What are the conclusions?

Perspective and/or bias

Reasons for publishing

Significance of findings

Is the methodology valid?

Look for flawed reasoning or fallacies, alternative explanations, omissions.

How does it relate to other sources? And what does it mean?

Highlighting is good for skimming.

Margin notes suggest your analysis and connections of the material.

Outlines may be useful for complex or important works.

Spreadsheets help track numerous sources across consistent variables or metrics.

Checklists help track progress and connections.

Summaries (written in your own words) help keep you on track and your sources straight.

Summarize Each Source to:

• Recap the important and most relevant information in that source
• Identify variables
• Identify context/setting
• Identify theories
• Identify findings

Integrate the Literature

• Identify similarities and differences.
• Trace the intellectual progression of the field, including major debates.
• Reflect upon the importance of the body of literature for your research.
• Evaluate the sources and advise the reader on the most pertinent or relevant sources. 

Tell the story, make your case.

Use your own words.

Write to your audience.

Make connections for your readers.

Cite accurately.

Introduction

• History and/or background
• Narrows topic
• Talks about implications
• They help with navigation.
• Start by painting the big picture by theme.
• Use large or important works first.
• Then discuss more detailed studies as needed and appropriate.
• Provide assessment about quality
• Gaps, new directions, information needed?

Let it sit. 

Get outside opinions and assistance. 

Don't rush this step!

When you're ready, continue with the questions for revision below.

Questions for Revision of Writing Style

• Does the writer use headings or paragraph breaks to show distinctions in the groups of studies under consideration?
• Does the writer explain why certain groups of studies (or individual studies) are being reviewed by drawing a clear connection to his or her topic?
• Does the writer make clear which of the studies described are most important?
• Does the writer cover all important areas of research related to his or her topic?
• Does the writer use transitions and summaries to move from one study or set of studies to the next?
• By the end of the literature review, is it clear why the current research is necessary?

Questions for Revision about the Research

• Does the review mentions flaws, gaps, or shortcomings of specific studies or groups of studies?
• Does the author point out areas that have not yet been researched or have not yet been researched sufficiently?
• Does the review demonstrate a change over time or recent developments that make the author’s research relevant now?
• Does the author discuss research methods used to study this topic and/or related topics?
• Does the author clearly state why his or her research is necessary?

Sign up for an appointment with the Writing Center . 

Thank you to the University of Colorado, Colorado Springs School of Public Affairs, Purdue University's Online Writing Lab, the Libraries at Virginia Commonwealth University, and our Queens Writing Center for much of the information on this page. 

• Literature Reviews Made Easy
• The Literature Review
• The Lit and The Review
• Your First Literature Review
• << Previous: Evaluate Sources
• Next: Cite Sources >>
• Last Updated: Sep 15, 2023 1:53 PM
• URL: https://library.queens.edu/humanservice

• Human Services - Statewide
• Ivy Tech Community College Libraries
• Ivy Tech Libraries
• Statewide Guides

Literature Reviews

Human services - statewide: literature reviews.

• Locating Peer-Reviewed Articles
• Treatment Programs
• Additional Resources
• Ask-a-Librarian This link opens in a new window

Research Assistance

Research specialists are available through the Ivy Tech Library network to assist you in locating relevant sources and to answer questions you may have about your research project.

You can  chat with a Librarian live  or submit a question to research specialists through the  Ask-A-Librarian   channel.

You can also visit your local campus Library website  here .

Discover Search

Literature review overview.

The following resources from the Purdue OWL explain the basic purpose and structure of a literature review.  

• Conducting Research Writing: A Literature Review
• Writing in the Social Sciences: Literature Reviews
• Types of APA Papers: Literature Reviews

Research Style Requirements

Always consult your assignment guidelines to determine the research style requirements as provided by the instructor.  Some classes require MLA style, while others require APA style, including Human Services classes.

APA Style Guide

• APA 7th Ed. Paper Format
• APA 7th Ed. References List
• APA 7th Ed. In-Text Citations
• APA 7th Ed. Tables & Figures
• APA 7th Ed. Italics & Quotation Marks
• APA 7th Ed. Capitalization Rules

Connecting with Campus Tutoring Services

Writing tutors at your campus are always willing to assist you with your writing projects, including the literature review.

Click  here  to learn more about campus tutoring or use the self-serve  TracCloud dashboard  in MyIvy to search for campus tutor availabilities or appointments.

• << Previous: Treatment Programs
• Next: Additional Resources >>
• Email a librarian
• Find people
• Library Guides
• Wireless Printing (on campus)

• Mardigian Library
• Subject Guides

HHS 230: Research Methods in Health and Human Services

• Literature Reviews
• Getting started @ Mardigian Library
• Find Journal Articles
• Find Books or eBooks
• How to Request Books and Articles from Other Libraries (ILL)
• Peer Review
• Citation Tools
• Annotated Bibliographies
• Video Tutorials
• Get library help

An Introduction to the Literature Review

This  brief 10-minute video  explains what a literature review is, and techniques for getting started. (It's just a good resource for anyone doing a literature review, even if they aren't a graduate student.)

(VIDEO) Literature Reviews: An Overview for Graduate Students   (from North Carolina State University)

Handout summarizing the Lit Review

• Literature Reviews An overview of the literature review in brief.

Raya's 2019 Presentation about Literature Reviews (Graduate Studies)

• Raya's Presentation about Literature Reviews (2019) Appx. 45-minute presentation that Raya Samet (CEHHS Librarian) made for UM-Dearborn Graduate Studies Professional Development series in 2019.
• Raya's slide deck about Lit Reviews (2019) Slides that correspond to the above presentation.

Synthesis Matrix for comparing and contrasting ideas/theories

A synthesis matrix helps you record the main points of each source and document how sources relate to each other.  After summarizing and evaluating your sources, arrange them in a matrix to help you see how they relate to each other, and apply to each of your themes or variables.  By arranging your sources in a matrix by theme or variable, you can see how your sources relate to each other, and can start thinking about how you weave them together to create a narrative.

• Synthesis matrix template with examples (Excel) This excel synthesis matrix shows you how you could set up your own.
• Blank Synthesis Matrix Template (Word) This simple chart-type synthesis matrix in word is a helpful template to get you started.
• Blank Synthesis Matrix Template (Excel) This blank spreadsheet-style matrix is a nice template to help you get started.
• << Previous: Annotated Bibliographies
• Next: Video Tutorials >>
• Last Updated: Mar 10, 2024 2:54 PM
• URL: https://guides.umd.umich.edu/hhsresearchmethods

Call us at 313-593-5559

Chat with us

Text us: 313-486-5399

Email us your question

• 4901 Evergreen Road Dearborn, MI 48128, USA
• Phone: 313-593-5000
• Maps & Directions
• M+Google Mail
• Emergency Information
• UM-Dearborn Connect
• Wolverine Access

Penn State University Libraries

Hd fs 402: human services seminar.

• Finding Articles
• Finding Books
• Recommended Journals
• Recommended Books
• Finding Organizations
• Other Resources
• IRB and Undergraduate research
• Literature Reviews: Strategies for Writing

Literature Reviews

What is a Literature Review? The literature review is a critical look at the existing research that is significant to the work that you are carrying out. This overview identifies prominent research trends in addition to assessing the overall strengths and weaknesses of the existing research.

Purpose of the Literature Review

• To provide background information about a research topic.
• To establish the importance of a topic.
• To demonstrate familiarity with a topic/problem.
• To “carve out a space” for further work and allow you to position yourself in a scholarly conversation.

Characteristics of an effective literature review In addition to fulfilling the purposes outlined above, an effective literature review provides a critical overview of existing research by

• Outlining important research trends.
• Assessing strengths and weaknesses (of individual studies as well the existing research as a whole).
• Identifying potential gaps in knowledge.
• Establishing a need for current and/or future research projects.

Steps of the Literature Review Process

1) Planning: identify the focus, type, scope and discipline of the review you intend to write. 2) Reading and Research: collect and read current research on your topic. Select only those sources that are most relevant to your project. 3) Analyzing: summarize, synthesize, critique, and compare your sources in order to assess the field of research as a whole. 4) Drafting: develop a thesis or claim to make about the existing research and decide how to organize your material. 5) Revising: revise and finalize the structural, stylistic, and grammatical issues of your paper.

This process is not always a linear process; depending on the size and scope of your literature review, you may find yourself returning to some of these steps repeatedly as you continue to focus your project.

These steps adapted from the full workshop offered by the Graduate Writing Center at Penn State. See the CSU Bakersfield guide here. 

• << Previous: IRB and Undergraduate research
• Last Updated: Dec 5, 2023 9:15 AM
• URL: https://guides.libraries.psu.edu/shenango-hdfs402

Browse Econ Literature

• Working papers
• Software components
• Book chapters
• JEL classification

More features

• Subscribe to new research

RePEc Biblio

Author registration.

• Economics Virtual Seminar Calendar NEW!

How to Undertake an Impactful Literature Review: Understanding Review Approaches and Guidelines for High-impact Systematic Literature Reviews

• Author & abstract
• Related works & more

Corrections

• Amrita Chakraborty
• Arpan Kumar Kar

Suggested Citation

Download full text from publisher.

Follow serials, authors, keywords & more

Public profiles for Economics researchers

Various research rankings in Economics

RePEc Genealogy

Who was a student of whom, using RePEc

Curated articles & papers on economics topics

Upload your paper to be listed on RePEc and IDEAS

New papers by email

Subscribe to new additions to RePEc

EconAcademics

Blog aggregator for economics research

Cases of plagiarism in Economics

About RePEc

Initiative for open bibliographies in Economics

News about RePEc

Questions about IDEAS and RePEc

RePEc volunteers

Participating archives

Publishers indexing in RePEc

Privacy statement

Found an error or omission?

Opportunities to help RePEc

Get papers listed

Have your research listed on RePEc

Open a RePEc archive

Have your institution's/publisher's output listed on RePEc

Get RePEc data

Use data assembled by RePEc

• Open access
• Published: 09 May 2024

Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research

• Jingcheng Du 1 ,
• Ekin Soysal 1 , 3 ,
• Dong Wang 2 ,
• Long He 1 ,
• Bin Lin 1 ,
• Jingqi Wang 1 ,
• Frank J. Manion 1 ,
• Yeran Li 2 ,
• Elise Wu 2 &
• Lixia Yao 2  

BMC Medical Research Methodology volume  24 , Article number:  108 ( 2024 ) Cite this article

121 Accesses

2 Altmetric

Metrics details

Systematic literature reviews (SLRs) are critical for life-science research. However, the manual selection and retrieval of relevant publications can be a time-consuming process. This study aims to (1) develop two disease-specific annotated corpora, one for human papillomavirus (HPV) associated diseases and the other for pneumococcal-associated pediatric diseases (PAPD), and (2) optimize machine- and deep-learning models to facilitate automation of the SLR abstract screening.

This study constructed two disease-specific SLR screening corpora for HPV and PAPD, which contained citation metadata and corresponding abstracts. Performance was evaluated using precision, recall, accuracy, and F1-score of multiple combinations of machine- and deep-learning algorithms and features such as keywords and MeSH terms.

Results and conclusions

The HPV corpus contained 1697 entries, with 538 relevant and 1159 irrelevant articles. The PAPD corpus included 2865 entries, with 711 relevant and 2154 irrelevant articles. Adding additional features beyond title and abstract improved the performance (measured in Accuracy) of machine learning models by 3% for HPV corpus and 2% for PAPD corpus. Transformer-based deep learning models that consistently outperformed conventional machine learning algorithms, highlighting the strength of domain-specific pre-trained language models for SLR abstract screening. This study provides a foundation for the development of more intelligent SLR systems.

Peer Review reports

Introduction

Systematic literature reviews (SLRs) are an essential tool in many areas of health sciences, enabling researchers to understand the current knowledge around a topic and identify future research and development directions. In the field of health economics and outcomes research (HEOR), SLRs play a crucial role in synthesizing evidence around unmet medical needs, comparing treatment options, and preparing the design and execution of future real-world evidence studies. SLRs provide a comprehensive and transparent analysis of available evidence, allowing researchers to make informed decisions and improve patient outcomes.

Conducting a SLR involves synthesizing high-quality evidence from biomedical literature in a transparent and reproducible manner, and seeks to include all available evidence on a given research question, and provides some assessment regarding quality of the evidence [ 1 , 2 ]. To conduct an SLR one or more bibliographic databases are queried based on a given research question and a corresponding set of inclusion and exclusion criteria, resulting in the selection of a relevant set of abstracts. The abstracts are reviewed, further refining the set of articles that are used to address the research question. Finally, appropriate data is systematically extracted from the articles and summarized [ 1 , 3 ].

The current approach to conducting a SLR is through manual review, with data collection, and summary done by domain experts against pre-specified eligibility criteria. This is time-consuming, labor-intensive, expensive, and non-scalable given the current more-than linear growth of the biomedical literature [ 4 ]. Michelson and Reuter estimate that each SLR costs approximately $141,194.80 and that on average major pharmaceutical companies conduct 23.36 SLRs, and major academic centers 177.32 SLRs per year, though the cost may vary based on the scope of different reviews [ 4 ]. Clearly automated methods are needed, both from a cost/time savings perspective, and for the ability to effectively scan and identify increasing amounts of literature, thereby allowing the domain experts to spend more time analyzing the data and gleaning the insights.

One major task of SLR project that involves large amounts of manual effort, is the abstract screening task. For this task, selection criteria are developed and the citation metadata and abstract for articles tentatively meeting these criteria are retrieved from one or more bibliographic databases (e.g., PubMed). The abstracts are then examined in more detail to determine if they are relevant to the research question(s) and should be included or excluded from further consideration. Consequently, the task of determining whether articles are relevant or not based on their titles, abstracts and metadata can be treated as a binary classification task, which can be addressed by natural language processing (NLP). NLP involves recognizing entities and relationships expressed in text and leverages machine-learning (ML) and deep-learning (DL) algorithms together with computational semantics to extract information. The past decade has witnessed significant advances in these areas for biomedical literature mining. A comprehensive review on how NLP techniques in particular are being applied for automatic mining and knowledge extraction from biomedical literature can be found in Zhao et al. [ 5 ].

Materials and methods

The aims of this study were to: (1) identify and develop two disease-specific corpora, one for human papillomavirus (HPV) associated diseases and the other for pneumococcal-associated pediatric diseases suitable for training the ML and DL models underlying the necessary NLP functions; (2) investigate and optimize the performance of the ML and DL models using different sets of features (e.g., keywords, Medical Subject Heading (MeSH) terms [ 6 ]) to facilitate automation of the abstract screening tasks necessary to construct a SLR. Note that these screening corpora can be used as training data to build different NLP models. We intend to freely share these two corpora with the entire scientific community so they can serve as benchmark corpora for future NLP model development in this area.

SLR corpora preparation

Two completed disease-specific SLR studies by Merck & Co., Inc., Rahway, NJ, USA were used as the basis to construct corpora for abstract-level screening. The two SLR studies were both relevant to health economics and outcome research, including one for human papillomavirus (HPV) associated diseases (referred to as the HPV corpus), and one for pneumococcal-associated pediatric diseases (which we refer to as the PAPD corpus). Both of the original SLR studies contained literature from PubMed/MEDLINE and EMBASE. Since we intended for the screening corpora to be released to the community, we only kept citations found from PubMed/MEDLINE in the finalized corpora. Because the original SLR studies did not contain the PubMed ID (PMID) for each article, we matched each article’s citation information (if available) against PubMed and then collected meta-data such as authors, journals, keywords, MeSH terms, publication types, etc., using PubMed Entrez Programming Utilities (E-utilities) Application Programming Interface (API). The detailed description of the two corpora can be seen in Table  1 . Both of the resulting corpora are publicly available at [ https://github.com/Merck/NLP-SLR-corpora ].

Machine learning algorithms

Although deep learning algorithms have demonstrated superior performance on many NLP tasks, conventional machine learning algorithms have certain advantages, such as low computation costs and faster training and prediction speed.

We evaluated four traditional ML-based document classification algorithms, XGBoost [ 7 ], Support Vector Machines (SVM) [ 8 ], Logistic regression (LR) [ 9 ], and Random Forest [ 10 ] on the binary inclusion/exclusion classification task for abstract screening. Salient characteristics of these models are as follows:

XGBoost: Short for “eXtreme Gradient Boosting”, XGBoost is a boosting-based ensemble of algorithms that turn weak learners into strong learners by focusing on where the individual models went wrong. In Gradient Boosting, individual weak models train upon the difference between the prediction and the actual results [ 7 ]. We set max_depth at 3, n_estimators at 150 and learning rate at 0.7.

Support vector machine (SVM): SVM is one of the most robust prediction methods based on statistical learning frameworks. It aims to find a hyperplane in an N-dimensional space (where N = the number of features) that distinctly classifies the data points [ 8 ]. We set C at 100, gamma at 0.005 and kernel as radial basis function.

Logistic regression (LR): LR is a classic statistical model that in its basic form uses a logistic function to model a binary dependent variable [ 9 ]. We set C at 5 and penalty as l2.

Random forest (RF): RF is a machine learning technique that utilizes ensemble learning to combine many decision trees classifiers through bagging or bootstrap aggregating [ 10 ]. We set n_estimators at 100 and max_depth at 14.

These four algorithms were trained for both the HPV screening task and the PAPD screening task using the corresponding training corpus.

For each of the four algorithms, we examined performance using (1) only the baseline feature criteria (title and abstract of each article), and (2) with five additional meta-data features (MeSH, Authors, Keywords, Journal, Publication types.) retrieved from each article using the PubMed E-utilities API. Conventionally, title and abstract are the first information a human reviewer would depend on when making a judgment for inclusion or exclusion of an article. Consequently, we used title and abstract as the baseline features to classify whether an abstract should be included at the abstract screening stage. We further evaluated the performance with additional features that can be retrieved by PubMed E-utilities API, including MeSH terms, authors, journal, keywords and publication type. For baseline evaluation, we concatenated the titles and abstracts and extracted the TF-IDF (term frequency-inverse document frequency) vector for the corpus. TF-IDF evaluates how relevant a word is to a document in a collection of documents. For additional features, we extracted TF-IDF vector using each feature respectively and then concatenated the extracted vectors with title and abstract vector. XGBoost was selected for the feature evaluation process, due to its relatively quick computational running time and robust performance.

Deep learning algorithms

Conventional ML methods rely heavily on manually designed features and suffer from the challenges of data sparsity and poor transportability when applied to new use cases. Deep learning (DL) is a set of machine learning algorithms based on deep neural networks that has advanced performance of text classification along with many other NLP tasks. Transformer-based deep learning models, such as BERT (Bidirectional encoder representations from transformers), have achieved state-of-the-art performance in many NLP tasks [ 11 ]. A Transformer is an emerging architecture of deep learning models designed to handle sequential input data such as natural language by adopting the mechanisms of attention to differentially weigh the significance of each part of the input data [ 12 ]. The BERT model and its variants (which use Transformer as a basic unit) leverage the power of transfer learning by first pre-training the models over 100’s of millions of parameters using large volumes of unlabeled textual data. The resulting model is then fine-tuned for a particular downstream NLP application, such as text classification, named entity recognition, relation extraction, etc. The following three BERT models were evaluated against both the HPV and Pediatric pneumococcal corpus using two sets of features (title and abstract versus adding all additional features into the text). For all BERT models, we used Adam optimizer with weight decay. We set learning rate at 1e-5, batch size at 8 and number of epochs at 20.

BERT base: this is the original BERT model released by Google. The BERT base model was pre-trained on textual data in the general domain, i.e., BooksCorpus (800 M words) and English Wikipedia (2500 M words) [ 11 ].

BioBERT base: as the biomedical language is different from general language, the BERT models trained on general textual data may not work well on biomedical NLP tasks. BioBERT was further pre-trained (based on original BERT models) in the large-scale biomedical corpora, including PubMed abstracts (4.5B words) and PubMed Central Full-text articles (13.5B words) [ 13 ].

PubMedBERT: PubMedBERT was pre-trained from scratch using abstracts from PubMed. This model has achieved state-of-the-art performance on several biomedical NLP tasks on Biomedical Language Understanding and Reasoning Benchmark [ 14 ].

Text pre-processing and libraries that were used

We have removed special characters and common English words as a part of text pre-processing. Default tokenizer from scikit-learn was adopted for tokenization. Scikit-learn was also used for TF-IDF feature extraction and machine learning algorithms implementation. Transformers libraries from Hugging Face were used for deep learning algorithms implementation.

Evaluation datasets were constructed from the HPV and Pediatric pneumococcal corpora and were split into training, validation and testing sets with a ratio of 8:1:1 for the two evaluation tasks: (1) ML algorithms performance assessment; and (2) DL algorithms performance assessment. Models were fitted on the training sets, and model hyperparameters were optimized on the validation sets and the performance were evaluated on the testing sets. The following major metrics are expressed by the noted calculations:

Where True positive is an outcome where the model correctly predicts the positive (e.g., “included” in our tasks) class. Similarly, a True negative is an outcome where the model correctly predicts the negative class (e.g., “excluded” in our tasks). False positive is an outcome where the model incorrectly predicts the positive class, and a False negative is an outcome where the model incorrectly predicts the negative class. We have repeated all experiments five times and reported the mean scores with standard deviation.

Table  2 shows the baseline comparison using different feature combinations for the SLR text classification tasks using XGBoost. As noted, adding additional features in addition to title and abstract was effective in further improving the classification accuracy. Specifically, using all available features for the HPV classification increased accuracy by ? ∼  3% and F1 score by ? ∼  3%; using all available features for Pediatric pneumococcal classification increased accuracy by ? ∼  2% and F1 score by ? ∼  4%. As observed, adding additional features provided a stronger boost in precision, which contributed to the overall performance improvement.

The comparison of the article inclusion/exclusion classification task for four machine learning algorithms with all features is shown in Table  3 . XGBoost achieved the highest accuracy and F-1 scores in both tasks. Table  4 shows the comparison between XGBoost and deep learning algorithms on the classification tasks for each disease. Both XGBoost and deep learning models consistently have achieved higher accuracy scores when using all features as input. Among all models, BioBERT has achieved the highest accuracy at 0.88, compared with XGBoost at 0.86. XGBoost has the highest F1 score at 0.8 and the highest recall score at 0.9 for inclusion prediction.

Discussions and conclusions

Abstract screening is a crucial step in conducting a systematic literature review (SLR), as it helps to identify relevant citations and reduces the effort required for full-text screening and data element extraction. However, screening thousands of abstracts can be a time-consuming and burdensome task for scientific reviewers. In this study, we systematically investigated the use of various machine learning and deep learning algorithms, using different sets of features, to automate abstract screening tasks. We evaluated these algorithms using disease-focused SLR corpora, including one for human papillomavirus (HPV) associated diseases and another for pneumococcal-associated pediatric diseases (PADA). The publicly available corpora used in this study can be used by the scientific community for advanced algorithm development and evaluation. Our findings suggest that machine learning and deep learning algorithms can effectively automate abstract screening tasks, saving valuable time and effort in the SLR process.

Although machine learning and deep learning algorithms trained on the two SLR corpora showed some variations in performance, there were also some consistencies. Firstly, adding additional citation features significantly improved the performance of conventional machine learning algorithms, although the improvement was not as strong in transformer-based deep learning models. This may be because transformer models were mostly pre-trained on abstracts, which do not include additional citation information like MeSH terms, keywords, and journal names. Secondly, when using only title and abstract as input, transformer models consistently outperformed conventional machine learning algorithms, highlighting the strength of subject domain-specific pre-trained language models. When all citation features were combined as input, conventional machine learning algorithms showed comparable performance to deep learning models. Given the much lower computation costs and faster training and prediction time, XGBoost or support vector machines with all citation features could be an excellent choice for developing an abstract screening system.

Some limitations remain for this study. Although we’ve evaluated cutting-edge machine learning and deep learning algorithms on two SLR corpora, we did not conduct much task-specific customization to the learning algorithms, including task-specific feature engineering and rule-based post-processing, which could offer additional benefits to the performance. As the focus of this study is to provide generalizable strategies for employing machine learning to abstract screening tasks, we leave the task-specific customization to future improvement. The corpora we evaluated in this study mainly focus on health economics and outcome research, the generalizability of learning algorithms to another domain will benefit from formal examination.

Extensive studies have shown the superiority of transformer-based deep learning models for many NLP tasks [ 11 , 13 , 14 , 15 , 16 ]. Based on our experiments, however, adding features to the pre-trained language models that have not seen these features before may not significantly boost their performance. It would be interesting to find a better way of encoding additional features to these pre-trained language models to maximize their performance. In addition, transfer learning has proven to be an effective technique to improve the performance on a target task by leveraging annotation data from a source task [ 17 , 18 , 19 ]. Thus, for a new SLR abstract screening task, it would be worthwhile to investigate the use of transfer learning by adapting our (publicly available) corpora to the new target task.

When labeled data is available, supervised machine learning algorithms can be very effective and efficient for article screening. However, as there is increasing need for explainability and transparency in NLP-assisted SLR workflow, supervised machine learning algorithms are facing challenges in explaining why certain papers fail to fulfill the criteria. The recent advances in large language models (LLMs), such as ChatGPT [ 20 ] and Gemini [ 21 ], show remarkable performance on NLP tasks and good potentials in explainablity. Although there are some concerns on the bias and hallucinations that LLMs could bring, it would be worthwhile to evaluate further how LLMs could be applied to SLR tasks and understand the performance of using LLMs to take free-text article screening criteria as the input and provide explainanation for article screening decisions.

Data availability

The annotated corpora underlying this article are available at https://github.com/Merck/NLP-SLR-corpora .

Bullers K, Howard AM, Hanson A, et al. It takes longer than you think: librarian time spent on systematic review tasks. J Med Libr Assoc. 2018;106:198–207. https://doi.org/10.5195/jmla.2018.323 .

Article   PubMed   PubMed Central   Google Scholar  

Carver JC, Hassler E, Hernandes E et al. Identifying Barriers to the Systematic Literature Review Process. In: 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement . 2013. 203–12. https://doi.org/10.1109/ESEM.2013.28 .

Lame G. Systematic literature reviews: an introduction. Proc Des Society: Int Conf Eng Des. 2019;1:1633–42. https://doi.org/10.1017/dsi.2019.169 .

Article   Google Scholar  

Michelson M, Reuter K. The significant cost of systematic reviews and meta-analyses: a call for greater involvement of machine learning to assess the promise of clinical trials. Contemp Clin Trials Commun. 2019;16:100443. https://doi.org/10.1016/j.conctc.2019.100443 .

Recent advances in. biomedical literature mining | Briefings in Bioinformatics | Oxford Academic. https://academic.oup.com/bib/article/22/3/bbaa057/5838460?login=true (accessed 30 May 2022).

Medical Subject Headings - Home Page. https://www.nlm.nih.gov/mesh/meshhome.html (accessed 30 May 2022).

Chen T, Guestrin C, XGBoost:. A Scalable Tree Boosting System. In: Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining . New York, NY, USA: Association for Computing Machinery 2016. 785–94. https://doi.org/10.1145/2939672.2939785 .

Noble WS. What is a support vector machine? Nat Biotechnol. 2006;24:1565–7. https://doi.org/10.1038/nbt1206-1565 .

Article   CAS   PubMed   Google Scholar  

Logistic Regression . https://doi.org/10.1007/978-1-4419-1742-3 (accessed 30 May 2022).

Random forest classifier for remote sensing classification. International Journal of Remote Sensing: Vol 26, No 1. https://www.tandfonline.com/doi/abs/10.1080/01431160412331269698 (accessed 30 May 2022).

Devlin J, Chang M-W, Lee K, et al. BERT: pre-training of Deep Bidirectional transformers for Language understanding. arXiv. 2019. https://doi.org/10.48550/arXiv.1810.04805 .

Vaswani A, Shazeer N, Parmar N et al. Attention is All you Need. In: Advances in Neural Information Processing Systems . Curran Associates, Inc. 2017. https://proceedings.neurips.cc/paper/2017/hash/3f5ee243547dee91fbd053c1c4a845aa-Abstract.html (accessed 30 May 2022).

BioBERT. a pre-trained biomedical language representation model for biomedical text mining | Bioinformatics | Oxford Academic. https://academic.oup.com/bioinformatics/article/36/4/1234/5566506 (accessed 3 Jun 2020).

Gu Y, Tinn R, Cheng H, et al. Domain-specific Language Model Pretraining for Biomedical Natural Language Processing. ACM Trans Comput Healthc. 2021;3(2):1–2. https://doi.org/10.1145/3458754 .

Article   CAS   Google Scholar  

Chen Q, Du J, Allot A, et al. LitMC-BERT: transformer-based multi-label classification of biomedical literature with an application on COVID-19 literature curation. arXiv. 2022. https://doi.org/10.48550/arXiv.2204.08649 .

Chen Q, Allot A, Leaman R, et al. Multi-label classification for biomedical literature: an overview of the BioCreative VII LitCovid Track for COVID-19 literature topic annotations. arXiv. 2022. https://doi.org/10.48550/arXiv.2204.09781 .

Kermany DS, Goldbaum M, Cai W, et al. Identifying Medical diagnoses and Treatable diseases by Image-based deep learning. Cell. 2018;172:1122–e11319. https://doi.org/10.1016/j.cell.2018.02.010 .

Howard J, Ruder S. Universal Language Model fine-tuning for text classification. arXiv. 2018. https://doi.org/10.48550/arXiv.1801.06146 .

Do CB, Ng AY. Transfer learning for text classification. In: Advances in Neural Information Processing Systems . MIT Press. 2005. https://proceedings.neurips.cc/paper/2005/hash/bf2fb7d1825a1df3ca308ad0bf48591e-Abstract.html (accessed 30 May 2022).

Achiam J et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774 (2023).

https:// gemini.google.com/app/a4dcd2e2d7672354 . (accessed 01 Feb 2024).

Download references

Acknowledgements

We thank Dr. Majid Rastegar-Mojarad for conducting some additional experiments during revision.

This research was supported by Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA.

Author information

Authors and affiliations.

Intelligent Medical Objects, Houston, TX, USA

Jingcheng Du, Ekin Soysal, Long He, Bin Lin, Jingqi Wang & Frank J. Manion

Merck & Co., Inc, Rahway, NJ, USA

Dong Wang, Yeran Li, Elise Wu & Lixia Yao

McWilliams School of Biomedical Informatics, University of Texas Health Science Center at Houston, Houston, TX, USA

Ekin Soysal

You can also search for this author in PubMed   Google Scholar

Contributions

Study concept and design: JD and LY Corpus preparation: DW, YL and LY Experiments: JD and ES Draft of the manuscript: JD, DW, FJM and LY Acquisition, analysis, or interpretation of data: JD, ES, DW and LY Critical revision of the manuscript for important intellectual content: JD, ES, DW, LH, BL, JW, FJM, YL, EW, LY Study supervision: LY.

Corresponding author

Correspondence to Lixia Yao .

Ethics declarations

Disclaimers.

The content is the sole responsibility of the authors and does not necessarily represent the official views of Merck & Co., Inc., Rahway, NJ, USA or Intelligent Medical Objects.

Ethics approval and consent to participate

Not applicable.

Consent for publication

Competing interests.

DW is an employee of Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA. EW, YL, and LY were employees of Merck Sharp & Dohme LLC, a subsidiary of Merck & Co., Inc., Rahway, NJ, USA for this work. JD, LH, JW, and FJM are employees of Intelligent Medical Objects. ES was an employee of Intelligent Medical Objects during his contributions, and is currently an employee of EBSCO Information Services. All the other authors declare no competing interest.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Du, J., Soysal, E., Wang, D. et al. Machine learning models for abstract screening task - A systematic literature review application for health economics and outcome research. BMC Med Res Methodol 24 , 108 (2024). https://doi.org/10.1186/s12874-024-02224-3

Download citation

Received : 19 May 2023

Accepted : 18 April 2024

Published : 09 May 2024

DOI : https://doi.org/10.1186/s12874-024-02224-3

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Machine learning
• Deep learning
• Text classification
• Article screening
• Systematic literature review

BMC Medical Research Methodology

ISSN: 1471-2288

• General enquiries: [email protected]

The Federal Register

The daily journal of the united states government, request access.

Due to aggressive automated scraping of FederalRegister.gov and eCFR.gov, programmatic access to these sites is limited to access to our extensive developer APIs.

If you are human user receiving this message, we can add your IP address to a set of IPs that can access FederalRegister.gov & eCFR.gov; complete the CAPTCHA (bot test) below and click "Request Access". This process will be necessary for each IP address you wish to access the site from, requests are valid for approximately one quarter (three months) after which the process may need to be repeated.

An official website of the United States government.

If you want to request a wider IP range, first request access for your current IP, and then use the "Site Feedback" button found in the lower left-hand side to make the request.