clinical research documentation in electronic health records

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

Publications
Account settings
My Bibliography
Collections
Citation manager

Save citation to file

Email citation, add to collections.

Create a new collection
Add to an existing collection

Add to My Bibliography

Your saved search, create a file for external citation management software, your rss feed.

Search in PubMed
Search in NLM Catalog
Add to Search

Improving clinical documentation: introduction of electronic health records in paediatrics

Affiliations.

1 Department of Paediatrics, University Hospitals of Derby and Burton NHS Foundation Trust, Queen's Hospital, Belvedere Road, Burton Upon Trent, UK.
2 Department of Paediatrics, University Hospitals of Derby and Burton NHS Foundation Trust, Queen's Hospital, Belvedere Road, Burton Upon Trent, UK [email protected].
PMID: 33589503
PMCID: PMC7887344
DOI: 10.1136/bmjoq-2020-000918

Medical records are crucial facet of a patient's journey. These provide the clinician with a permanent record of the patient's illness and ongoing medical care, thus enabling informed clinical decisions. In many hospitals, patient medical records are written on paper. However, written notes are liable to misinterpretation due to illegibility and misplacement. This can affect the patient's medical care and has medico-legal implications. Electronic patient records (EPR) have been gradually introduced to replace patient's paper notes with the aim of providing a more reliable record-keeping system. It is perceived that EPR improve the quality and efficiency of patient care. The paediatric department at Queen's Hospital Burton uses a mix of paper notes and computerised medical records. Clinicians primarily use paper notes for admission clerking, ward rounds, ward reviews and outpatient clinic consultations. Laboratory tests, imaging results and prescription requests are executed via the EPR system. Documentation by nurses is also carried out electronically. We aimed to improve and standardise clinical documentation of paediatric admissions and ward round notes by developing electronic proforma for initial paediatric clerking, ward rounds and patient reviews. This quality improvement project improved clinical documentation on the paediatric wards and enhanced patient record-keeping, boosted clinical information-sharing and streamlined patient journey. It fulfilled various generic multidisciplinary record keeping audit tool standards endorsed by the Royal College of Physicians by 100%. We undertook a staff survey to investigate the opinion before and after implementing the electronic health record. Doctors, nurses and healthcare support workers overwhelmingly supported the quality, usefulness, completeness of specified fields and practicality of the electronic records.

Keywords: continuous quality improvement; electronic health records; paediatrics.

PubMed Disclaimer

Conflict of interest statement

Competing interests: None declared.

Overall responses towards EHRs in…

Overall responses towards EHRs in general, before and after EHR implementation. EHR, electronic…

Responses towards the electronic clerking…

Responses towards the electronic clerking proforma and electronic ward round notes before and…

Related information

Linkout - more resources, full text sources.

Europe PubMed Central
PubMed Central

Other Literature Sources

scite Smart Citations
MedlinePlus Consumer Health Information
MedlinePlus Health Information
Citation Manager

NCBI Literature Resources

MeSH PMC Bookshelf Disclaimer

The PubMed wordmark and PubMed logo are registered trademarks of the U.S. Department of Health and Human Services (HHS). Unauthorized use of these marks is strictly prohibited.

Search Menu
Sign in through your institution
Advance articles
Editor's Choice
Focus Issue Archive
Open Access Articles
JAMIA Journal Club
Author Guidelines
Submission Site
Open Access
Call for Papers
About Journal of the American Medical Informatics Association
About the American Medical Informatics Association
Journals Career Network
Editorial Board
Advertising and Corporate Services
Self-Archiving Policy
Dispatch Dates
For Reviewers
Journals on Oxford Academic
Books on Oxford Academic

Article Contents

Introduction, materials and methods, author contributions, supplementary material, data availability statement, conflict of interest statement.

< Previous

Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review

Article contents
Figures & tables
Supplementary Data

Amanda J Moy, Jessica M Schwartz, RuiJun Chen, Shirin Sadri, Eugene Lucas, Kenrick D Cato, Sarah Collins Rossetti, Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review, Journal of the American Medical Informatics Association , Volume 28, Issue 5, May 2021, Pages 998–1008, https://doi.org/10.1093/jamia/ocaa325

Permissions Icon Permissions

Electronic health records (EHRs) are linked with documentation burden resulting in clinician burnout. While clear classifications and validated measures of burnout exist, documentation burden remains ill-defined and inconsistently measured. We aim to conduct a scoping review focused on identifying approaches to documentation burden measurement and their characteristics.

Based on Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) Extension for Scoping Reviews (ScR) guidelines, we conducted a scoping review assessing MEDLINE, Embase, Web of Science, and CINAHL from inception to April 2020 for studies investigating documentation burden among physicians and nurses in ambulatory or inpatient settings. Two reviewers evaluated each potentially relevant study for inclusion/exclusion criteria.

Of the 3482 articles retrieved, 35 studies met inclusion criteria. We identified 15 measurement characteristics, including 7 effort constructs: EHR usage and workload, clinical documentation/review, EHR work after hours and remotely, administrative tasks, cognitively cumbersome work, fragmentation of workflow, and patient interaction. We uncovered 4 time constructs: average time, proportion of time, timeliness of completion, activity rate, and 11 units of analysis. Only 45.0% of studies assessed the impact of EHRs on clinicians and/or patients and 40.0% mentioned clinician burnout .

Standard and validated measures of documentation burden are lacking. While time and effort were the core concepts measured, there appears to be no consensus on the best approach nor degree of rigor to study documentation burden.

Further research is needed to reliably operationalize the concept of documentation burden, explore best practices for measurement, and standardize its use.

Rapid adoption of electronic health records (EHRs) following the passage of the Health Information Technology for Economic and Clinical Health (HITECH) Act has led to advances in both individual- and population-level health. 1 HITECH has improved healthcare quality, patient safety, and diagnostic accuracy through enhanced data management and timely reuse; interoperable systems have facilitated care continuity and monitoring of compliance metrics. 2–5 EHR-facilitated, guideline-based care has been associated with reduced redundancies 6 , 7 and streamlined billing administration. 8

Largely still in its infancy, the implementation of EHRs has also resulted in unintended consequences on clinical practice and healthcare systems, including significant increases in clinician documentation time. 9–13 Extended work hours, time constraints, clerical workload, and disruptions to the patient-provider encounter, have led to a rise in discontent with existing documentation methods in EHR systems. 6 , 14 , 15 This documentation burden has been linked to increases in medical errors, 3 , 9 , 16 threats to patient safety, 3 , 9 , 16 inferior documentation quality, 17 , 18 job attrition, and, ultimately, burnout among nurses and physicians. 3 , 9–11 , 14 , 16–22

In concert with Affordable Care Act (ACA) reimbursement models, Meaningful Use (MU) mandates, and a regulatory-rich environment, EHRs have drastically altered clinical documentation workflow and communication in routine healthcare. 13 , 15 , 23 Physicians have reported willingness to remain out of compliance with EHR incentive programs (eg, MU and the Physician Quality Reporting System 24 ) in favor of mitigating documentation burden (hereinafter referred interchangeably as “burden”). 15 , 25 Still, studies consistently demonstrate that physicians spend twice as much time on electronic documentation and clerical tasks as compared to time providing direct patient care. 14 , 26–30 Similarly, nurses devote more than half of their shift time to EHR data entry and retrieval 19 , 20 and report reduced direct patient contact. 31 , 32

While researchers have discussed the challenges of burden and its implications for clinician burnout due to EHRs over the past decade, 5 , 15 , 33 limited attention has been paid to discriminating the antecedent concept of burden (defined as a duty, responsibility, etc, that causes worry, difficulty, or hard work), 34 from burnout (defined as long-term work-related stress reaction marked by emotional exhaustion, depersonalization, and a lack of sense of personal accomplishment). 35 , 36 Clinician burnout has been well-documented and widely quantified using surveys and psychological measurements throughout peer-reviewed literature. 37–40 Yet, to our best knowledge, there is a lack of consensus on approaches to measure burden. 15 , 37 , 41–45

While EHR dissatisfaction has been extensively studied and some clinician activity metrics have been proposed, 46 few empirically-based readily-available solutions to reduce burden exist. 11 Interventions to assuage burden have ranged from the utilization of scribes and remote transcription services 27 to text summarization and dictation software. 16 , 47 In March 2020, the Department of Health and Human Services (HHS) released a report outlining 3 primary goals to reduce EHR-related clinician burdens that influence care: reduce the time and effort clinicians require to document health information, reduce the effort required to meet regulatory requirements, and improve EHR ease of use. 48 Evaluating the impact of interventions that target these goals will necessitate standardized, quantitative measurements.

The purpose of this scoping review is to assess the state of science, identify gaps in knowledge, and synthesize characteristics of documentation burden measurement among physicians and nurses using EHRs.

We conducted a scoping review using the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA) extension for Scoping Reviews (ScR) guidelines. 49 A scoping review fit our objective to describe the breadth of methods used to measure documentation burden. 49

Search strategy and selection criteria

We systematically searched the MEDLINE, Embase, Web of Science, and CINAHL databases for all English-language studies published in peer-reviewed journals and conference proceedings, investigating documentation burden among physicians and/or nurses in ambulatory and/or inpatient settings from inception to April 20, 2020. We evaluated all relevant literature identified through in-text references among eligible studies. Burden is not specifically represented in Medical Subject Headings (MeSH); therefore, we explored both keyword and MeSH terms for 2 burden-related concepts outlined in the HHS report Strategy on Reducing Burden Relating to the Use of Health IT and EHRs 48 documentation: (a) effort, and (b) time. We also focused our search on: (a) the EHR and (b) physicians or nurses. The finalized search strategy is summarized in Table 1 .

Summary of search terms and query employed to each academic literature database in our review

Concept .	Search Strings .	Operator .
documentation time	(“Task Performance and Analysis”[Mesh]) OR (“Costs and Cost Analysis”[Mesh]) OR (“Time Factors”[Mesh]) OR (“Process Assessment, Health Care”[Mesh]) OR (“time ”) OR (“Measure ”) OR (“measurement”) OR (“quantify”) OR (“quanti ”) OR (“metric”)	AND
documentation effort	(“Documentation ”[Mesh]) OR (“documentation ”) OR (“note ”) or (“unstructured data”) OR (“narrative”) OR (“Burnout, Professional”[Mesh]) OR (“Cognition ”[Mesh]) OR (“Cognitive load”) OR (“Burnout”) OR (“burden”)	AND
EHR	(“Electronic Health Records ”[Mesh]) OR (“electronic health record ”) OR (“electronic medical record ”) OR (“EHR”) OR (“EMR”) OR (“computerized medical record ”)	AND
physicians/nurses	(“Physicians”[Mesh]) OR (“Nurses”[Mesh]) OR (“nurse ”) OR (“physician ”)

Concept .	Search Strings .	Operator .
documentation time	(“Task Performance and Analysis”[Mesh]) OR (“Costs and Cost Analysis”[Mesh]) OR (“Time Factors”[Mesh]) OR (“Process Assessment, Health Care”[Mesh]) OR (“time ”) OR (“Measure ”) OR (“measurement”) OR (“quantify”) OR (“quanti ”) OR (“metric”)	AND
documentation effort	(“Documentation ”[Mesh]) OR (“documentation ”) OR (“note ”) or (“unstructured data”) OR (“narrative”) OR (“Burnout, Professional”[Mesh]) OR (“Cognition ”[Mesh]) OR (“Cognitive load”) OR (“Burnout”) OR (“burden”)	AND
EHR	(“Electronic Health Records ”[Mesh]) OR (“electronic health record ”) OR (“electronic medical record ”) OR (“EHR”) OR (“EMR”) OR (“computerized medical record ”)	AND
physicians/nurses	(“Physicians”[Mesh]) OR (“Nurses”[Mesh]) OR (“nurse ”) OR (“physician ”)

Designates wildcard search.

Study selection and selection criteria

We selected inclusion and exclusion parameters a priori, and iteratively modified them to exclude studies involving niche clinical systems and those strictly comparing to paper-based documentation ( Table 2 ). We included all peer-reviewed primary studies that focused on EHR utilization with an objective time or effort measure 48 (eg, EHR usage logs, which report time stamped documentation events) in the review.

Inclusion and exclusion criteria

Inclusion .	Exclusion .

Inclusion .	Exclusion .

The term “physicians” encompassed attending physicians, fellows, resident physicians, and interns; “nurses” referred to registered nurses. We focused on physicians and nurses given our aim of identifying interprofessional measurements of documentation burden. We excluded studies comparing EHR documentation to paper-based systems if they were not focused on measuring burden, but rather on EHR implementation evaluation.

After removing duplicates, 2 reviewers (AJM and JMS, AJM and RC, AJM and SS, or AJM and EL) independently screened article titles and abstracts for relevance using Covidence. 50 Two authors (with a third serving as a tiebreaker) reviewed each potentially relevant abstract for eligibility criteria in the full-text. We included full-text articles with concordant decisions by the 2 reviewers in the final analysis; for discordant decisions, all reviewers reexamined and adjudicated until a consensus was reached.

Data extraction and analysis

One author (AJM) performed data charting for all articles meeting full-text inclusion criteria (see online Supplementary Table ), which was reviewed by all authors and discussed. We extracted the following information: publication year, geographic location, time source, unit(s) of analysis, activity, sample size, sample characteristics, EHR system, provider role/specialty, clinical setting, study design and objectives, study type (eg, quantitative or mixed-methods), site type (eg, single or multisite), exposure and outcome measures, analytical and statistical methods, study limitations/bias, and major findings. We reported study limitations and biases such as threats to internal and external validity to appraise rigor. We used the HHS concepts to organize our reporting of measurement characteristics. 48 HHS does not elaborate further on definitions of: (a) time, (b) effort, and (c) outcomes assessed 48 ; therefore, we conducted purposeful thematic analysis to identify proxies and synthesize these 3 recurring concepts. 51 We iteratively combined themes until we achieved a consensus.

Sources of evidence

Our search strategy yielded 3482 potentially relevant manuscripts from MEDLINE (n = 507), Embase (n = 1143), Web of Science (n = 1007), and CINAHL (n = 825). Seven additional manuscripts were identified through in-text references. After eliminating duplicates, 1946 titles/abstracts were screened; of those, 166 were eligible for full-text review. Consensus was achieved for all disagreements concerning the inclusion of full-text articles. Thirty-five studies meeting criteria were summarized in the final analysis ( Figure 1 ).

PRISMA flow diagram for scoping review of eligible studies.

Study characteristics

Studies were conducted in the United States (n = 31), 13 , 14 , 22 , 27 , 28 , 43 , 52–76 Europe (n = 1), 29 and Asia (n = 3). 77–79 Studies included a mix of ambulatory (n = 22) 14 , 22 , 27–29 , 52–54 , 56 , 61–66 , 69 , 71–75 and hospital (ie, inpatient and emergency) settings (n = 11) 13 , 55 , 57 , 59 , 60 , 67 , 68 , 70 , 76 , 78 , 79 with 2 involving both. 58 , 77 A majority of those studies involved single sites (77.1%) and were affiliated with an academic institution/teaching hospital (80.0%). One third used Epic systems (n = 13), 13 , 22 , 27 , 43 , 53 , 54 , 56 , 57 , 64 , 66 , 69 , 72 , 73 followed by multiple/other/unspecified (n = 12), 14 , 28 , 29 , 52 , 63 , 67 , 68 , 75–79 Cerner (n = 6), 58–60 , 65 , 71 , 74 Allscripts (n = 2), 61 , 62 and Eclipsys (n = 2). 55 , 70

Articles were published between 2010 and 2020 with 2018 (n = 8) 13 , 14 , 29 , 56 , 57 , 63 , 67 , 76 and 2019 (n = 8) 22,54,58,69,72,74,6066 representing the highest volumes. Range of study sample sizes was expansive among the studies (4 ≤ n ≤ 154 719). Most studies exclusively focused on physicians (n = 25) 13–2942 as compared to nurses (n = 5) 58 , 67 , 76–78 or an interprofessional sample of providers (n = 5). 22 , 55 , 56 , 69 , 73 Clinician specialties were heterogeneous; over half the studies involved single specialties (general [n = 11], 14 , 27 , 52 , 53 , 61 , 62 , 64 , 71–73 , 75 emergency [n = 2], 57 , 79 intensivist [n = 2], 67 , 70 other [(n = 5] 13 , 54 , 56 , 74 , 78 ), while the remaining were multiple subspecialties (n = 13) 9 , 22 , 28 , 43 , 58–60 , 63 , 65 , 66 , 69 , 76 , 77 or unspecified (n = 2). 55 , 68 Across all studies, most involved general medicine (n = 17) 14 , 22 , 27 , 28 , 43 , 52 , 53 , 59–66 , 69 , 71–73 , 75–77 followed by surgical subspecialties (n = 8), 13 , 29 , 58 , 59 , 66 , 74 , 77 , 78 intensive care (n = 6), 58 , 59 , 67 , 70 , 76 and emergency medicine (n = 4) 57 , 58 , 60 , 79 ; 10 included other subspecialties. 22 , 28 , 29 , 54 , 56 , 58 , 60 , 63 , 65 , 66

Thirty were strictly quantitative studies. While purely qualitative studies were excluded, 5 studies employed mixed methods 28 , 52 , 55 , 61 , 62 (see online Supplementary Table ). Study designs varied, including time-and-motion (TM [n = 5] 28 , 61 , 62 , 67 , 68 ), validation of TM (n = 2), 27 , 70 cohort (n = 15), 13 , 27 , 43 , 54 , 55 , 57 , 59 , 60 , 63 , 64 , 66 , 72 , 74 , 75 , 77 experimental/quasi-experimental (n = 8), 14 , 22 , 29 , 53 , 56 , 58 , 78 , 79 and cross-sectional studies (n = 4). 69 , 71 , 73 , 76 Eight studies evaluated an intervention, 14 , 22 , 52 , 53 , 56 , 58 , 75 , 78 including scribes (n = 3), 14 , 52 , 53 documentation redesign (n = 3), 58 , 75 , 78 or EHR training programs (n = 2) 22 , 56 ; the remaining were descriptive studies on EHR activities and usage (n = 27)—2 of which involved the implementation of new EHR systems. 29 , 79

A diversity of analytical methods was employed. Most studies to which statistical testing were relevant (n = 23) applied parametric (n = 19) as opposed to non-parametric methods (n = 12). Qualitative methods employed in the mixed-methods studies involved informal interviews, 62 social network analysis, 55 thematic analysis, 62 focus groups, 52 and self-reported diary. 28 Few studies addressed validity or reliability of measurements in their studies (n = 11) 22 , 52 , 53 , 59 , 60 , 63 , 64 , 67 , 69 , 73 , 78 ; 2 examined interobserver reliability, 28 , 68 2 employed TM approaches to validate novel analytical methods to examine workflow 70 and the use of EHR usage logs to estimate workload, 27 2 examined correlations between self-reported and objective EHR usage log times, 22 , 73 and 1 employed video recording timers to validate EHR usage log times. 58

Characterization of effort

Seven overarching effort constructs emerged ( Table 3 ): (a) general workload such as overall EHR usage (n = 4) 53 , 56 , 68 , 69 ; (b) clinical documentation/review (n = 15) 28 , 29 , 55 , 57–61 , 67 , 72 , 75–79 ; (c) excess workload including EHR usage after hours (n = 15) 13 , 22 , 27 , 52–54 , 59 , 63–66 , 69 , 71 , 73 , 74 and remote access (n = 1) 72 ; (d) administrative tasks, such as inbox management (n = 2) 69 , 73 ; (e) cognitively cumbersome work, such as multitasking (n = 3) 61 , 62 , 68 ; (f) fragmentation of EHR workflow (n = 1) 70 ; and (g) patient interaction/in-person visits (n = 7). 14 , 28 , 29 , 43 , 53 , 62 , 68 Several terms were employed referring to EHR usage afterhours including “work after work,” 66 “pajama time,” 66 and “Clinician Logged-In Outside Clinic” (CLOC) time. 22 For example, Cox et al proposed the “amount of EHR usage taking place after scheduled duty hours” specifically for surgical residents. 13

Identified measurement characteristics from study findings

Documentation Burden Concepts .	Measurement Constructs .
Effort	EHR usage and workload
	Clinical documentation/review
	EHR work afterhours and remotely
	Administrative tasks (eg, inbox management)
	Cognitively cumbersome work (eg, multitasking)
	Fragmentation of workflow
	Patient interaction
Time	Average time spent
	Proportion or percentage of time spent
	Binary of timeliness of completion (eg, documenting within shift or policy time frame)
	Activity rate
Units of analysis	Clinically-oriented units of analysis	Temporally-oriented units of analysis
	Encounter	Seconds
	Encounter	Minutes
	Provider	Minutes
	Patient	Seconds
	Patient	Minutes
	Event/Task	Seconds
		Minutes
		Hours
		Shifts
		Days
		Weeks
		Months

Documentation Burden Concepts .	Measurement Constructs .
Effort	EHR usage and workload
	Clinical documentation/review
	EHR work afterhours and remotely
	Administrative tasks (eg, inbox management)
	Cognitively cumbersome work (eg, multitasking)
	Fragmentation of workflow
	Patient interaction
Time	Average time spent
	Proportion or percentage of time spent
	Binary of timeliness of completion (eg, documenting within shift or policy time frame)
	Activity rate
Units of analysis	Clinically-oriented units of analysis	Temporally-oriented units of analysis
	Encounter	Seconds
	Encounter	Minutes
	Provider	Minutes
	Patient	Seconds
	Patient	Minutes
	Event/Task	Seconds
		Minutes
		Hours
		Shifts
		Days
		Weeks
		Months

Note: constructs and units are not intended to be comprehensive of all possibilities but rather reflect content identified in scoping review.

Measurement of time

Time spent documenting was assessed in all studies and was measured using 3 key data collection strategies: EHR usage logs (n = 28), 13 , 14 , 22 , 27 , 43 , 53–60 , 63–67 , 69 , 71–79 activity capture applications (n = 8), 27–29 , 52 , 61 , 62 , 68 , 80 and video recordings (n = 1). 58 Few studies triangulated these data through multiple data collection strategies (n = 2). 27 , 58 Time constructs identified ( Table 3 ) include (a) average time spent (n = 20), 22 , 27 , 29 , 43 , 54 , 55 , 57 , 59–61 , 63–67 , 69 , 71–73 , 78 (b) proportion or percentage of time spent (n = 10), 13 , 28 , 53 , 56 , 62 , 68 , 70 , 72 , 74 , 75 (c) binary of timeliness of completion (n = 1), 77 and (d) activity rate (n = 2). 61 , 76 Units of analysis varied within and across studies ( Table 3 ), including time reported per: (a) encounter (n = 5), 54 , 60 , 65 , 67 , 69 (b) provider (n = 2), 14 , 73 (c) patient (n = 3), 57 , 59 , 78 or (d) event/task (n = 28). 13 , 14 , 22 , 27–29 , 43 , 53–56 , 58 , 61–64 , 66 , 68 , 70–79 Units of analysis also included average hours per day, per week, or per month (n = 6) 22 , 29 , 43 , 63 , 71 , 72 and average minutes per day, per week, per shift, or per clinical full-time equivalent per week (n = 7). 27 , 55 , 61 , 64 , 66 , 73 , 78 We have organized these units of analysis into 2 levels for combination in individual measures: (a) a clinically oriented unit of analysis, such as “per encounter,” and (b) a temporally oriented unit of analysis, such as “per hour” (see Table 3 ). Operationalization of a shift and “active versus idle” time in the EHR also varied. Among the 15 studies that examined shifts, 13 , 22 , 27 , 54 , 56 , 57 , 59 , 64 , 65 , 68 , 69 , 71 , 73 , 74 , 76 9 distinct shift times were identified with 6:00 am–6:00 pm (n = 4), 13 , 65 , 71 , 74 7:00 am–7:00 pm (n = 3), 69 , 73 , 76 and 8:00 am–6:00 pm (n = 2) 22 , 27 representing the most frequently reported intervals. Meanwhile, only half the studies employing EHR usage logs explicitly operationalized active versus idle time in the EHR to account for the time a clinician is logged in but not actively using the system. However, determination of “active and idle” time were measured at different levels of granularity (ie, complete system time-out [n = 3] 13 , 43 , 73 vs “active versus idle” between tasks [n = 11] 22 , 27 , 56 , 59 , 60 , 64 , 65 , 69 , 71 , 72 , 74 ). “Active versus idle” activity time was largely vendor defined (n = 7), 22 , 59 , 60 , 65 , 69 , 71 , 74 relied on mouse clicks and keystrokes (n = 5), 59 , 60 , 65 , 71 , 74 and/or idle time between 30 seconds and 10 minutes of length (n = 5). 27 , 56 , 64 , 69 , 72

Outcome assessment

Less than half the studies assessed the impact of documentation burden on clinicians and/or patients (n = 16). Among those studies, authors referenced the temporal relationship between burden and burnout at a higher proportion (68.8%) compared to those that did not extend beyond measuring time and effort alone (50.0%). Outcomes measured included clinical process measures [n = 8 (ie, treatment time, encounter closure, length of stay) 14 , 54 , 57 , 69 , 79 ], clinician (n = 7) 14 , 22 , 52 , 53 , 75 , 78 , 79 and patient satisfaction (n = 4), 14 , 52 , 53 , 63 burnout/stress (n = 5), 22 , 64 , 69 , 73 , 75 patient census/mortality (n = 2), 59 response to messages (n = 1), 22 and team interactions (n = 1). 55 Primary predictors and outcomes of interest are summarized in the online Supplementary Table .

Limitations and biases reported

Two limitations were ubiquitous across included studies ( Table 4 ): (a) threats to generalizability due to constraints in sample size (n = 19), 14 , 28 , 29 , 52–54 , 57 , 59–64 , 67 , 69 , 70 , 73 , 74 , 79 study setting (n = 21), 22 , 28 , 52–55 , 59–62 , 64 , 68–70 , 72–74 , 76–79 patient population, 57 , 77 EHR system (n = 6), 58 , 60 , 61 , 70 , 75 , 78 , 81 activity type, 76 clinician role or seniority, 57 , 59 , 61 , 69 , 70 early adoption, 43 and/or subspecialty; 62 , 64 , 70 and, (b) measurement error including the inability of logs to distinguish between “idle and active” time (n = 6), 27 , 43 , 55 , 64 , 73 , 80 uncertainty regarding the definition of “afterhours,” 59 , 73 incomplete measurement of tasks (n = 15), 13 , 27 , 29 , 43 , 56–58 , 65 , 68–71 , 76 , 78 , 80 imprecision of time capture, 27 , 43 , 55 , 73 , 80 information bias (n = 10), 27–29 , 56 , 61 , 68 , 70 and validity of measures. 53 , 54 , 64

Study limitations identified in the review

Author (Year) .	generalizability .	small sample size .	selection bias .	response bias .	measurement error .	misclassification .	information bias .	no data triangulation .	confounding .	self-reported data .
Adler-Milstein et al (2020)	•	•			•	•				•
Ahn et al (2016)	•
Anderson et al (2020)	•				•				•
Arndt et al (2017)	•		•			•	•
Aziz et al (2019)	•	•			•	•				•
Carlson et al (2015)	•		•	•	•				•	•
Collins et al (2018)	•				•			•
Cox et al (2018)	•				•	•		•	•
DiAngi et al (2019)	•		•	•	•					•
Earls et al (2017)	•	•	•		•		•	•	•	•
Gidwani et al (2017)	•	•			•			•		•
Goldstein et al (2019)	•	•			•			•	•
Hripcsak et al (2011)	•				•			•
Hsieh et al (2016)	•				•					•
Inokuchi et al (2015)	•	•	•	•						•
Joukes et al (2018)	•	•	•		•		•		•
Kadish et al (2018)	•			•	•		•	•		•
Kannampallil et al (2018)	•	•	•		•	•			•
Karp et al (2019)	•				•
Krawiec et al (2019)	•	•			•			•
Krawiec et al (2020)	•	•			•				•
Mamykina et al (2012)	•	•			•		•
Mamykina et al (2016)	•	•	•		•		•		•
Marmor et al (2018)	•	•			•			•	•	•
Micek et al (2020)	•	•		•	•	•				•
Mishra et al (2018)		•	•				•	•		•
Overhage et al (2020)	•				•	•			•
Saag et al (2019)								•
Sinsky et al (2016)	•	•	•		•		•		•	•
Smith et al (2018)	•	•			•
Tai-Seale et al (2017)	•		•		•	•
Tipping et al (2010)	•				•		•
Tran et al (2019)	•	•			•				•	•
Wang et al (2019)	•				•
Zheng et al (2010)	•	•	•		•		•

Author (Year) .	generalizability .	small sample size .	selection bias .	response bias .	measurement error .	misclassification .	information bias .	no data triangulation .	confounding .	self-reported data .
Adler-Milstein et al (2020)	•	•			•	•				•
Ahn et al (2016)	•
Anderson et al (2020)	•				•				•
Arndt et al (2017)	•		•			•	•
Aziz et al (2019)	•	•			•	•				•
Carlson et al (2015)	•		•	•	•				•	•
Collins et al (2018)	•				•			•
Cox et al (2018)	•				•	•		•	•
DiAngi et al (2019)	•		•	•	•					•
Earls et al (2017)	•	•	•		•		•	•	•	•
Gidwani et al (2017)	•	•			•			•		•
Goldstein et al (2019)	•	•			•			•	•
Hripcsak et al (2011)	•				•			•
Hsieh et al (2016)	•				•					•
Inokuchi et al (2015)	•	•	•	•						•
Joukes et al (2018)	•	•	•		•		•		•
Kadish et al (2018)	•			•	•		•	•		•
Kannampallil et al (2018)	•	•	•		•	•			•
Karp et al (2019)	•				•
Krawiec et al (2019)	•	•			•			•
Krawiec et al (2020)	•	•			•				•
Mamykina et al (2012)	•	•			•		•
Mamykina et al (2016)	•	•	•		•		•		•
Marmor et al (2018)	•	•			•			•	•	•
Micek et al (2020)	•	•		•	•	•				•
Mishra et al (2018)		•	•				•	•		•
Overhage et al (2020)	•				•	•			•
Saag et al (2019)								•
Sinsky et al (2016)	•	•	•		•		•		•	•
Smith et al (2018)	•	•			•
Tai-Seale et al (2017)	•		•		•	•
Tipping et al (2010)	•				•		•
Tran et al (2019)	•	•			•				•	•
Wang et al (2019)	•				•
Zheng et al (2010)	•	•	•		•		•

Six studies cited selection bias derived from both the presence of self-selection and voluntary participation among high-performing subjects 27 , 28 and the presence of low response. 22 , 56 , 64 , 75 Eleven studies noted a lack of data triangulation, such as combining log data with direct observations, encounter information or qualitative data to offer contextual information corresponding to types of EHR interfaces used (eg, remote, inpatient, outpatient) for login timestamps, direct patient care, and other data. 13 , 14 , 52–56 , 60 , 63 , 66 , 76 Twelve studies identified the presence of potential confounding. 13 , 28 , 29 , 52 , 54 , 57 , 59 , 62 , 63 , 65 , 69 , 71 , 75

In this scoping review, we identified 35 studies that explored the measurement of documentation burden among physicians and nurses, underlining the overall paucity of research in the domain. As may be expected, all 35 studies were published post-HITECH Act. Seven effort constructs, 4 time constructs, and 11 units of analysis were uncovered. Our effort constructs—except workflow fragmentation and cognitively cumbersome work (eg, multitasking)—largely align with “proposed core EHR use measures (for practice efficiency)” published by Sinsky and colleagues which indicates burden may be quantified through existing metrics. 46 Generated with expert stakeholders, Sinsky’s core measures include total EHR time, work outside of work, time on documentation, time on prescriptions, inbox time, teamwork for orders, and undivided attention to patients. 46 Further efforts should examine these measures for validity and reliability. Fewer than half (n = 16) of the studies investigated the impact of burden on clinicians and/or patients. Methodologies varied across study design, suggesting there is no current consensus regarding best approach or standard to study burden, although it is possible an ensemble of methods coupled with the triangulation of multiple data sources will emerge as a best practice.

Historically, TM studies have been considered the gold standard for quantifying the effects of computer systems on task-based clinical workflow and duration. 82 , 83 Despite yielding valid results, 70 , 84 , 85 TM studies are costly and time-consuming to perform 83 and engage only a handful of participants per study. In addition to concerns regarding the generalizability of TM studies, prior research has identified widespread methodological inconsistencies in their design and conduct as well as in their quantitative analyses and reporting of results, making it difficult to synthesize findings across studies. 70 , 86 Readily accessible and scalable, and less subject to the Hawthorne effect, evidence may suggest that analyzing EHR usage logs is a more feasible alternative as these data were used in the overwhelming majority of included studies (80.0%). Nevertheless, research on the use of EHR usage logs to evaluate clinical activity has revealed a dearth of validation, cross-study analyses, and, most critically, defined terminology (eg, access log, audit log) and measures. 46 , 87 These inconsistencies parallel those found in TM studies, as described above. TM studies provide valuable contextual information on time and sequence of activities performed which can be triangulated with EHR usage logs to better understand burden in the context of clinical workflows. In recognizing that all methods have strengths and weaknesses, we anticipate that future work will identify the methods of measurement and triangulation of data that best align with different research objectives related to burden.

One major finding of this review was the absence of quantitative studies assessing the reliability and validity of time and effort measures. Of the 35 studies included, only 1 study intended to develop a measure of burden (ie, EHR usage outside shift), 13 while 2 studies individually employed TM studies to empirically validate proposed measures of workflow and the use of EHR usage log data in characterizing workload. 27 , 70 Interobserver reliability was reported in only 2 studies. 28 , 68 As described above, previous studies on quantifying physician EHR activity through EHR usage logs have noted similar challenges. 87 The lack of studies developing and validating burden measures confirms that limited efforts have been dedicated to formally and objectively quantifying and measuring burden, despite increasing references to it in public policy and lay literature. Researchers have often used unstandardized proxies to quantify burden which elucidates why no objective proxies exist. 6 , 13 , 14 , 25 Reinforcing the absence of empirical validation studies, there is a lack of an agreed-upon definition for burden and a plethora of definitions throughout the literature. 6 , 13 , 14 , 25 , 28 , 43 , 47 , 52 , 88–90 We found that many related—but different—concepts were used in the context of studies quantifying time and effort, such as workload, 27 , 78 workflow, 13 , 74 work disruption, 75 efficiency, 22 , 52 cognitive burden, 56 usability, 74 and productivity, among others. 69 In contrast, burnout is identifiable in controlled vocabularies including, the International Classification of Diseases (ICD), in addition to the Diagnostic and Statistical Manual for Mental Disorders (DSM) and MeSH. 91 , 92 Furthermore, validated measures of burnout, such as the Maslach Burnout Inventory and the Mini Z burnout survey are often applied, 69 , 73 whereas no known analog for burden is currently available. Likewise, in a literature review conducted on the impact of EHRs on documentation time among physicians and nurses, Poissant and colleagues suggested that a lack of research evaluating EHR time efficiency is likely associated with the poverty of rigorous methods accurately capturing time. 12 We found that generalizability and measurement error issues were partially driven by the use of distinct EHR systems with some instances of proprietary and opaque vendor-defined time metrics for shift and active EHR time. 73 There was also imprecision in time capture among EHR usage log studies. Reported elsewhere in the literature, EHR usage logs have exhibited unreliable degrees of accuracy for both clinician activity and time durations captured. 87 Intended for troubleshooting technical problems and HIPAA compliance, EHR usage logs originate from many interconnected information systems and sources (eg, devices). 93 Vendor-defined time metrics may not be generalizable between, or within, institutions or provide precise estimates in real-world settings. Therefore, given the value in measuring clinician EHR time, researchers should explore novel algorithmic methods to validate these metrics and EHR usage log data. For example, Dziorny and colleagues developed an automated algorithm to quantify shift duration among physicians in an inpatient setting and internally validated it against scheduled shift-time. 83 Likewise, DiAngi et al proposed the “calculated EHR time outside of clinic” (CLOC) metric for ambulatory settings to measure after clinic hours using EHR usage logs and were able to correlate their findings with self-reported time spent in the EHR after clinic hours. 22

The HHS Report— Strategy on Reducing Burden Relating to the Use of Health IT and EHRs —aims to evaluate the clinical impact of burden (ie, time and effort ) on clinicians and/or patients; 48 however, fewer than half the studies reviewed investigated an outcome of interest (n = 16). Of those studies (note: outcomes were not mutually exclusive), the majority examined clinician satisfaction and burnout (n = 12), while only half examined clinical process measures as an end goal. Half evaluated patient satisfaction and health indicators. Research questions and study objectives were widespread across included studies.

In this review, scribes represented 1 of 3 areas of study concerning proposed interventions to mitigate burden (n = 3); 14 , 52 , 53 however, associated costs and high turnover rates among scribes suggest that this solution may not be broadly feasible or sustainable. 47 In the context of reducing documentation burden, implementing and measuring the impact of scribes does not solve the higher-level information processing issues that informatics research should be investigating (eg, reduction in data entry requirements, improvement of system usability) and possibly diverts resources away from more sophisticated biomedical informatics approaches. Other identified interventions, such as training on EHR use (n = 2) 22 , 56 and documentation redesign (n = 3) 58 , 75 , 78 also have their strengths and weaknesses. Training may represent a lower cost method of mitigating burden than scribes, while documentation redesign may be more costly but likely more effective at solving information processing and usability concerns. Moreover, lack of standardized measures leads to the inability to conduct comparative effectiveness studies on design modifications within EHR systems 15 or across distinct burden-alleviating interventions.

In summary, our findings identified distinct, but not necessarily comprehensive, characteristics of measuring burden: 7 effort constructs, 4 time constructs , and 11 units of analysis (see Table 3 ).

Limitations

While this study sought to investigate literature on the operationalization of documentation burden and the development and/or validation of quantitative burden measures, research in this domain has not yet matured. Despite employing broad search terms and queries, the majority of the literature retrieved did not detail how to conceptualize and/or measure burden. We extracted manuscripts using keywords, as extant MeSH terms were unable to capture the phenomenon of study interest; in fact, no term for burden used in this specific context exists. It is conceivable that some articles were not captured because: (a) our keywords were limited, and/or (b) our queries were not sufficiently broad or narrow.

Future directions

Future research should build upon existing burden evidence, focusing on strengthening objectivity and generalizability. Proposed quantitative measures of burden such as the after scheduled duty hours measure described by Cox and colleagues should undergo rigorous testing and validation across settings and specialties. 13 Additionally, HHS links time and effort concepts to clinical impact; 48 therefore, research should directly connect measurement of these concepts with specific outcome measures to be able to accurately evaluate documentation burden over time. This remains a difficult undertaking as studies have shown that neither burden nor task value in the clinical context are identical across all EHR interactions or across different roles and specialties. 10 , 20 , 25 Examining tradeoffs between specific tasks within the EHR, Rao and colleagues discovered that EHR functionalities are not equally burdensome. 25 They also found that settings are not equally burdensome, reporting that shift-based work may be associated with less burden and that ambulatory clinical documentation is rated equally valuable and burdensome. 25 Perceptions of distinct documentation types among nurses have also been studied, yet no objective criteria have been established to evaluate value. 19 We found that only 1 study investigating EHR work afterhours (ie, “pajama time”) included nurses. 77 While “pajama time” connotes remotely accessing the EHR from home to document, few inpatient nurses do so given the immediacy of their documentation. Thus, data entry rates may be more suitable for measuring nurse burden. 76 Because physicians working in general medicine were most represented in our findings, future work should be dedicated to characterizing and measuring burden among understudied professions and settings (ie, nurses and subspecialties).

However, promisingly, burden measures identified were not strictly unique to individual professions and workflows, supporting the opportunity for defining interprofessional measures of burden in future work. We propose that burden be examined as a global composite measure, indicative of magnitude and directionality, consistent with the characteristics uncovered in this review. This would require: (a) developing a universally agreed-upon inventory for key EHR tasks and activities weighted for relative value according to burden (ie, a taxonomy) that could be linked to clinical outcomes such as “quality, financial or professional satisfaction” 15 , 27 , 28 ; and (b) quantifying the relationship between “pain points” and specific features in the EHR with more granularity. This furnishes the examination of task value , as indicated by task relationship with burden, a high priority area for future research. Such research would allow the identification of tasks that are of high burden but low value so that EHR design and intervention efforts may target the elimination or mitigation of these tasks.

Documentation burden among interprofessional clinical roles remains understudied and under-measured in both inpatient and ambulatory settings. This review suggests that concrete, validated measures of burden in research are lacking, which pales in comparison to burnout literature. 36 Moreover, this review demonstrates that the existing evidence is imprecise and fragmentary. While there is a multitude of measures for both effort and time among the included studies, the majority lack generalizability across study setting, patient population, EHR system, activity type, role, and subspecialty. In the absence of standardization, these studies additionally run the risk of measurement error including misclassification of idle and active time, completeness of task measurement, and precision of time capture. Hence, it would be prudent to further explore easily accessible, scalable alternatives, such as EHR usage log data. Targeting burden to evaluate the impact of quality improvement strategies and interventions requires quantifiable measures that are comparable and consistent across time, settings, professions, and contexts. We propose that burden should be examined as a global composite measure based on task value, consistent with burden measurement characteristics uncovered in this review. Further research is needed to reliably operationalize and standardize the concept of burden and to explore how it is best measured across clinical settings.

This study was supported by the US National Library of Medicine of the National Institutes of Health (NIH) under the training fellowship award 5T15LM007079 and the National Institute for Nursing Research (NINR) under grant numbers 1R01NR016941 and 5T32NR007969.

AJM and SCR conceptualized the scope of this review. AJM, JMS, RC, SS, and EL conducted the initial and full-text screenings. AJM drafted the manuscript with significant revisions and feedback from JMS, RC, KDC, and SCR.

Supplementary material is available at Journal of the American Medical Informatics Association online.

The data underlying this article are available in the article and in its online supplementary material .

None declared.

Hripcsak G , Bloomrosen M , FlatelyBrennan P , et al. Health data use, stewardship, and governance: ongoing gaps and challenges: a report from AMIA’s 2012 health policy meeting . J Am Med Inform Assoc 2014 ; 21 ( 2 ): 204 – 11 .

Google Scholar

Paterick ZR , Patel NJ , Paterick TE. Unintended consequences of the electronic medical record on physicians in training and their mentors . Postgrad Med J 2018 ; 94 ( 1117 ): 659 – 61 .

Coiera E , Ash J , Berg M. The unintended consequences of health information technology revisited . Yearb Med Inform 2016 ; 25 ( 01 ): 163 – 9 .

Palabindala V , Pamarthy A , Jonnalagadda NR. Adoption of electronic health records and barriers . J Community Hosp Intern Med Perspect 2016 ; 6 ( 5 ): 32643 .

Khairat S , Burke G , Archambault H , et al. Focus section on health IT usability: perceived burden of EHRs on physicians at different stages of their career . Appl Clin Inform 2018 ; 09 ( 02 ): 336 – 47 .

Howard J , Clark EC , Friedman A , et al. Electronic health record impact on work burden in small, unaffiliated, community-based primary care practices . J Gen Intern Med 2013 ; 28 ( 1 ): 107 – 13 .

Chaudhry B , Wang J , Wu S , et al. Systematic review: impact of health information technology on quality, efficiency, and costs of medical care . Ann Intern Med 2006 ; 144 ( 10 ): 742 – 52 .

Hillestad R , Bigelow J , Bower A , et al. Can electronic medical record systems transform health care? Potential health benefits, savings, and costs . Health Aff 2005 ; 24 ( 5 ): 1103 – 17 .

Ash JS , Berg M , Coiera E. Some unintended consequences of information technology in health care: the nature of patient care information system-related errors . J Am Med Inform Assoc 2003 ; 11 ( 2 ): 104 – 12 .

DesRoches CM , Campbell EG , Rao SR , et al. Electronic health records in ambulatory care—a national survey of physicians . N Engl J Med 2008 ; 359 ( 1 ): 50 – 60 .

DiAngi YT , Longhurst CA , Payne TH. Taming the EHR (electronic health record): there is hope . J Fam Med 2016 ; 3 ( 6 ): 1072 .

Poissant L , Pereira J , Tamblyn R , et al. The impact of electronic health records on time efficiency of physicians and nurses: a systematic review . J Am Med Inform Assoc 2005 ; 12 ( 5 ): 505 – 16 .

Cox ML , Farjat AE , Risoli TJ , et al. Documenting or operating: where is time spent in general surgery residency? J Surg Educ 2018 ; 75 ( 6 ): e97 – 106 .

Mishra P , Kiang JC , Grant RW. Association of medical scribes in primary care with physician workflow and patient experience . JAMA Intern Med 2018 ; 178 ( 11 ): 1467 – 72 .

Shanafelt TD , Dyrbye LN , Sinsky C , et al. Relationship between clerical burden and characteristics of the electronic environment with physician burnout and professional satisfaction . Mayo Clin Proc 2016 ; 91 ( 7 ): 836 – 48 .

Kaufman DR , Sheehan B , Stetson P , et al. Natural language processing-enabled and conventional data capture methods for input to electronic health records: a comparative usability study . JMIR Med Inform 2016 ; 4 ( 4 ): e35 .

Ollapally V. EHRs add to surgeons’ administrative burdens: the ACS responds. 2018 . http://bulletin.facs.org/2018/09/ehrs-add-to-surgeons-administrative-burdens-the-acs-responds/ Accessed February 6, 2019.

Dykstra RH , Ash JS , Campbell E , et al. Persistent paper: the myth of going paperless . AMIA Annu Symp Proc 2009 ; 2009 : 158 – 62 .

Olivares Bøgeskov B , Grimshaw-Aagaard SLS. Essential task or meaningless burden? Nurses’ perceptions of the value of documentation . Nord J Nurs Res 2019 ; 39 ( 1 ): 9 – 19 .

Kossman SP , Scheidenhelm SL. Nurses’ perceptions of the impact of electronic health records on work and patient outcomes . Comput Inform Nurs 2008 ; 26 ( 2 ): 69 – 77 .

Shanafelt TD , Dyrbye LN , West CP. Addressing physician burnout: the way forward . Jama 2017 ; 317 ( 9 ): 901 – 2 .

DiAngi YT , Stevens LA , Halpern-Felsher B , et al. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers’ perceived control over their workload in the EHR . JAMIA Open 2019 ; 2 ( 2 ): 222 – 30 .

Goldenberg JN. The breadth and burden of data collection in clinical practice . Neurol Clin Pract 2016 ; Feb; 6 ( 1 ): 81 – 6 .

Center for Medicare and Medicaid Services. An introduction to the Medicare EHR Incentive Program for eligible professionals. 2014 ; https://www.cms.gov/Regulations-and-Guidance/Legislation/EHRIncentivePrograms/Downloads/EHR_Medicare_Stg1_BegGuide.pdf Accessed August 14, 2019.

Rao SK , Kimball AB , Lehrhoff SR , et al. The impact of administrative burden on academic physicians . Acad Med 2017 ; 92 ( 2 ): 237 – 43 .

Colicchio TK , Cimino JJ. Clinicians’ reasoning as reflected in electronic clinical note-entry and reading/retrieval: a systematic review and qualitative synthesis . J Am Med Inform Assoc 2019 ; 26 ( 2 ): 172 – 84 .

Arndt BG , Beasley JW , Watkinson MD , et al. Tethered to the EHR: primary care physician workload assessment using EHR event log data and time-motion observations . Ann Fam Med 2017 ; 15 ( 5 ): 419 – 26 .

Sinsky C , Colligan L , Li L , et al. Allocation of physician time in ambulatory practice: a time and motion study in 4 specialties . Ann Intern Med 2016 ; 165 ( 11 ): 753 – 60 .

Joukes E , Abu-Hanna A , Cornet R , et al. Time spent on dedicated patient care and documentation tasks before and after the introduction of a structured and standardized electronic health record . Appl Clin Inform 2018 ; 09 ( 01 ): 46 – 53 .

Wang SJ , Middleton B , Prosser LA , et al. A cost-benefit analysis of electronic medical records in primary care . Am J Med 2003 ; 114 ( 5 ): 397 – 403 .

Wang N , Yu P , Hailey D. The quality of paper-based versus electronic nursing care plan in Australian aged care homes: a documentation audit study . Int J Med Inform 2015 ; 84 ( 8 ): 561 – 9 .

Lavander P , Meriläinen M , Turkki L. Working time use and division of labour among nurses and health-care workers in hospitals: a systematic review . J Nurs Manag 2016 ; 24 ( 8 ): 1027 – 40 .

Schulte F , Fry E. Death by a thousand clicks: where electronic health records went wrong. Kaiser Health News. 2019 . https://khn.org/news/death-by-a-thousand-clicks/ Accessed June 16, 2020.

Oxford English Dictionary. 2019 . https://www.oed.com/view/Entry/24885? rskey=DnyH6B&result=1&isAdvanced=false#eid Accessed July 16, 2019

Agency for Healthcare Research and Quality. Physician burnout. 2017 . https://www.ahrq.gov/professionals/clinicians-providers/ahrq-works/burnout/index.html Accessed July 16, 2019

West CP , Dyrbye LN , Erwin PJ , et al. Interventions to prevent and reduce physician burnout: a systematic review and meta-analysis . Lancet 2016 ; 388 ( 10057 ): 2272 – 81 .

Heinemann LV , Heinemann T. Burnout research: emergence and scientific investigation of a contested diagnosis . SAGE Open 2017 ; 7 ( 1 ): 215824401769715 . https://doi.org/10.1177/2158244017697154

Dyrbye LN , Shanafelt TD , Johnson PO , Johnson LA , Satele D , West CP. A cross-sectional study exploring the relationship between burnout, absenteeism, and job performance among American nurses . BMC Nurs 2019 ; 18 ( 1 ): 57. https://doi.org/10.1186/s12912-019-0382-7

Dyrbye LN , Shanafelt TD , Sinsky CA , et al. Burnout among health care professionals: a call to explore and address this underrecognized threat to safe, high-quality care . NAM Perspectives 2017 ; 7 ( 7 ). 10.31478/201707b

Moss M , Good VS , Gozal D , et al. An official critical care societies collaborative statement—burnout syndrome in critical care health-care professionals: a call for action . Crit Care Med 2016 ; 44 ( 7 ): 1414 – 21 .

Babbott S , Manwell LB , Brown R , et al. Electronic medical records and physician stress in primary care: results from the MEMO Study . J Am Med Inform Assoc 2014 ; 21 ( e1 ): e100-6 – e106 .

Meigs SL , Solomon M. Electronic health record use a bitter pill for many physicians . Perspect Heal Inf Manag 2016 ; 13 : 1 – 17 .

Tai-Seale M , Olson CW , Li J , et al. Electronic health record logs indicate that physicians split time evenly between seeing patients and desktop medicine . Health Aff 2017 ; 36 ( 4 ): 655 – 62 .

Maslach C , Jackson SE. The measurement of experienced burnout . J Organiz Behav 1981 ; 2 ( 2 ): 99 – 113 .

Olson K , Sinsky C , Rinne ST , et al. Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory . Stress Health 2019 ; 35 ( 2 ): 157 – 75 .

Sinsky CA , Rule A , Cohen G , et al. Metrics for assessing physician activity using electronic health record log data . J Am Med Inform Assoc 2020 ; 27 ( 4 ): 639 – 43 .

Dela Cruz JE , Shabosky JC , Albrecht M , et al. Typed versus voice recognition for data entry in electronic health records: emergency physician time use and interruptions . West J Emerg Med 2014 ; 15 ( 4 ): 541 – 7 .

Office of the National Coordinator for Health Information Technology. Strategy on reducing burden relating to the use of health IT and EHRs. 2020 . https://www.healthit.gov/topic/usability-and-provider-burden/strategy-reducing-burden-relating-use-health-it-and-ehrs Accessed April 2, 2020.

Tricco AC , Lillie E , Zarin W , et al. PRISMA extension for scoping reviews (PRISMA-ScR): Checklist and explanation . Ann Intern Med 2018 ; 169 ( 7 ): 467 – 73 .

Veritas Health Innovation. Covidence Systematic Review Software . Melbourne, Australia : Veritas Health Innovation ; 2018 .

Google Preview

Nowell LS , Norris JM , White DE , et al. Thematic analysis: striving to meet the trustworthiness criteria . Int J Qual Methods 2017 ; 16 ( 1 ): 160940691773384 .

Earls ST , Savageau JA , Begley S , et al. Can scribes boost FPs’ efficiency and job satisfaction? J Fam Pract 2017 ; 66 ( 4 ): 206 – 14 .

Gidwani R , Nguyen C , Kofoed A , et al. Impact of scribes on physician satisfaction, patient satisfaction, and charting efficiency: a randomized controlled trial . Ann Fam Med 2017 ; 15 ( 5 ): 427 – 33 .

Goldstein IH , Hwang T , Gowrisankaran S , et al. Changes in electronic health record use time and documentation over the course of a decade . Ophthalmology 2019 ; 126 ( 6 ): 783 – 91 .

Hripcsak G , Vawdrey DK , Fred MR , et al. Use of electronic clinical documentation: time spent and team interactions . J Am Med Inform Assoc 2011 ; 18 ( 2 ): 112 – 7 .

Kadish SS , Mayer EL , Jackman DM , et al. Implementation to optimization: a tailored, data-driven approach to improve provider efficiency and confidence in use of the electronic medical record . J OncolPract 2018 ; 14 ( 7 ): e421 – ee428 .

Kannampallil TG , Denton CA , Shapiro JS , et al. Efficiency of emergency physicians: insights from an observational study using EHR log files . Appl Clin Inform 2018 ; 9 ( 1 ): 99 – 104 .

Karp EL , Freeman R , Simpson KN , et al. Changes in efficiency and quality of nursing electronic health record documentation after implementation of an admission patient history essential data set . Comput Inform Nurs 2019 ; 37 ( 5 ): 260 – 5 .

Krawiec C , Stetter C , Kong L , et al. Impact of patient census and admission mortality on pediatric intensive care unit attending electronic health record activity: a preliminary study . Appl Clin Inform 2020 ; 11 ( 02 ): 226 – 34 .

Krawiec C , Marker C , Stetter C , et al. Tracking resident pre-rounding electronic health record usage . Int J Health Care Qual Assur 2019 ; 32 ( 3 ): 611 – 20 .

Mamykina L , Vawdrey DK , Stetson PD , et al. Clinical documentation: composition or synthesis? J Am Med Inform Assoc 2012 ; 19 ( 6 ): 1025 – 31 .

Mamykina L , Vawdrey DK , Hripcsak G. How do residents spend their shift time? A time and motion study with a particular focus on the use of computers . Acad Med 2016 ; 91 ( 6 ): 827 – 32 .

Marmor RA , Clay B , Millen M , et al. The impact of physician EHR usage on patient satisfaction . Appl Clin Inform 2018 ; ( 01 ): 011 – 4 . doi: 10.1055/s-0037-1620263

Micek MA , Arndt B , Tuan W-J , et al. Physician burnout and timing of electronic health record use . ACI Open 2020 ; 04 ( 01 ): e1 – e8 .

Overhage JM , McCallie D. Physician time spent using the electronic health record during outpatient encounters a descriptive study . Ann Intern Med 2020 ; 172 ( 3 ): 169 – 74 .

Saag HS , Shah K , Jones SA , et al. Pajama time: working after work in the electronic health record . J Gen Intern Med 2019 ; 34 ( 9 ): 1695 – 6 .

Smith B , Sreeramakavacham S , Kim JH , et al. Improving computerized charting in an intensive care unit. In: proceedings of the Digital Human Modeling: Applications in Safety, Ergonomics, and Risk Management ; July 15-20, 2018; Las Vegas, NV. New York (USA): Springer International Publishing; 2018: 537 – 46 .

Tipping MD , Forth VE , O'Leary KJ , et al. Where did the day go? A time-motion study of hospitalists . J Hosp Med 2010 ; 5 ( 6 ): 323 – 8 .

Tran B , Lenhart A , Ross R , et al. Burnout and EHR use among academic primary care physicians with varied clinical workloads . AMIA Jt Summits Transl Sci Proceedings AMIA Jt Summits Transl Sci 2019 ; 2019 : 136 – 44 .

Zheng K , Haftel HM , Hirschl RB , et al. Quantifying the impact of health IT implementations on clinical workflow: a new methodological perspective . J Am Med Inf Assoc 2010 ; 17 ( 4 ): 454 – 61 .

Anderson J , Leubner J , Brown SR. EHR overtime: an analysis of time spent after hours by family physicians . Fam Med 2020 ; 52 ( 2 ): 135 – 7 .

Wang JK , Ouyang D , Hom J , et al. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data . PLoS One 2019 ; 14 ( 2 ): e0205379 .

Adler-Milstein J , Zhao W , Willard-Grace R , et al. Electronic health records and burnout: time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians . J Am Med Informatics Assoc 2020 ; 27 ( 4 ): 531 – 8 .

Aziz F , Talhelm L , Keefer J , et al. Vascular surgery residents spend one fifth of their time on electronic health records after duty hours . J Vasc Surg 2019 ; 69 ( 5 ): 1574 – 9 .

Carlson KL , McFadden SE , Barkin S. Improving documentation timeliness: a “brighter future” for the electronic medical record in resident clinics . Acad Med 2015 ; 90 ( 12 ): 1641 – 5 .

Collins S , Couture B , Kang MJ , et al. Quantifying and visualizing nursing flowsheet documentation burden in acute and critical care . AMIA. Annu Symp Proc AMIA Symp 2018 ; 2018 : 348 – 57 .

Ahn M , Choi M , Kim Y. Factors associated with the timeliness of electronic nursing documentation . Healthc Inform Res 2016 ; 22 ( 4 ): 270 – 6 .

Hsieh H-Y , Henker R , Ren D , et al. Improving effectiveness and satisfaction of an electronic charting system in Taiwan . Clin Nurse Spec 2016 ; 30 ( 6 ): E1 – 6 .

Inokuchi R , Sato H , Iwagami M , et al. Impact of a new medical record system for emergency departments designed to accelerate clinical documentation: a crossover study . Med (United States) 2015 ; 94 ( 26 ): e856.

Zheng K , Haftel HMM , Hirschl RBB , et al. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data . PLoS One 2018 ; 9 ( 6 ): 531 – 8 .

Goldstein IH , Hwang T , Gowrisankaran S , et al. Changes in electronic health record use time over a decade of use . Investig Ophthalmol Vis Sci 2019 ; 126 (6): 783 – 91 .

Lopetegui M , Yen P-Y , Lai A , et al. Time motion studies in healthcare: what are we talking about? J Biomed Inform 2014 ; 49 : 292 – 9 .

Dziorny A , Orenstein E , Lindell R , et al. Automatic detection of front-line clinician hospital shifts: a novel use of electronic health record timestamp data . Appl Clin Inform 2019 ; 10 ( 01 ): 28 – 37 .

Ampt A , Westbrook J , Creswick N , et al. A comparison of self-reported and observational work sampling techniques for measuring time in nursing tasks . J Health Serv Res Policy 2007 ; 12 ( 1 ): 18 – 24 .

Burke TA , McKee JR , Wilson HC , et al. A comparison of time-and-motion and self-reporting methods of work measurement . J Nurs Adm 2000 ; 30 ( 3 ): 118 – 25 .

Zheng K , Guo MH , Hanauer DA. Using the time and motion method to study clinical work processes and workflow: methodological inconsistencies and a call for standardized research . J Am Med Inform Assoc 2011 ; 18 ( 5 ): 704 – 10 .

Rule A , Chiang MF , Hribar MR. Using electronic health record audit logs to study clinical activity: a systematic review of aims, measures, and methods . J Am Med Inform Assoc 2020 ; 27 ( 3 ): 480 – 90 . doi: 10.1093/jamia/ocz196.

Koshy S , Feustel PJ , Hong M , et al. Scribes in an ambulatory urology practice: patient and physician satisfaction . J Urol 2010 ; 184 ( 1 ): 258 – 62 .

Park SY , Lee SY , Chen Y. The effects of EMR deployment on doctors’ work practices: a qualitative study in the emergency department of a teaching hospital . Int J Med Inform 2012 ; 81 ( 3 ): 204 – 17 .

Senathirajah Y , Kaufman DR , Bakken SR. User-composable electronic health record improves efficiency of clinician data viewing for patient case appraisal: a mixed-methods study . eGEMs 2016 ; 4 ( 1 ): 7 .

Korczak D , Huber B , Kister C. Differential diagnostic of the burnout syndrome . GMS Health Technol Assess 2010 ; 6 : Doc09 .

Hall LH , Johnson J , Watt I , et al. Healthcare staff wellbeing, burnout, and patient safety: a systematic review . PLoS One 2016 ; 11 ( 7 ): e0159015 – 12 .

Wickramage C , Sahama T , Fidge C. Anatomy of log files: implications for information accountability measures. In: proceedings of the 2016 IEEE 18th International Conference on e-Health Networking, Applications and Services (Healthcom); September 14–17, 2016; Munich, Germany. New Jersey (USA): IEEE Press; 2016 : 1 – 6 .

Supplementary data

Month:	Total Views:
January 2021	1,136
February 2021	385
March 2021	373
April 2021	490
May 2021	537
June 2021	408
July 2021	360
August 2021	383
September 2021	450
October 2021	472
November 2021	502
December 2021	346
January 2022	487
February 2022	631
March 2022	593
April 2022	569
May 2022	480
June 2022	420
July 2022	424
August 2022	384
September 2022	478
October 2022	615
November 2022	508
December 2022	367
January 2023	523
February 2023	676
March 2023	849
April 2023	715
May 2023	749
June 2023	606
July 2023	701
August 2023	661
September 2023	865
October 2023	920
November 2023	929
December 2023	671
January 2024	910
February 2024	925
March 2024	1,218
April 2024	1,031
May 2024	915
June 2024	858
July 2024	1,046

Email alerts

Citing articles via.

Recommend to your Library

Affiliations

Online ISSN 1527-974X
Copyright © 2024 American Medical Informatics Association
About Oxford Academic
Publish journals with us
University press partners
What we publish
New features
Open access
Institutional account management
Rights and permissions
Get help with access
Accessibility
Advertising
Media enquiries
Oxford University Press
Oxford Languages
University of Oxford

Oxford University Press is a department of the University of Oxford. It furthers the University's objective of excellence in research, scholarship, and education by publishing worldwide

Copyright © 2024 Oxford University Press
Cookie settings
Cookie policy
Privacy policy
Legal notice

This Feature Is Available To Subscribers Only

This PDF is available to Subscribers Only

For full access to this pdf, sign in to an existing account, or purchase an annual subscription.

Toward Reliable Symptom Coding in Electronic Health Records for Symptom Assessment and Research: Identification and Categorization of International Classification of Diseases, Ninth Revision, Clinical Modification Symptom Codes

CIN Computers Informatics Nursing
This person is not on ResearchGate, or hasn't claimed this research yet.

University of Texas Health Science Center at Houston

University of Utah

Discover the world's research

25+ million members
160+ million publication pages
2.3+ billion citations

No full-text available

To read the full-text of this research, you can request a copy directly from the authors.

Hannah Christina Puhr
Rainer Puhr
D. Kuchling
Aysegül Ilhan-Mutlu
J PAIN SYMPTOM MANAG

Thomas J. Hoffman
Sonya L. Heath
Christine Miaskowski

Margaret Bale
Rashmee U. Shah

Riccardo Miotto
Joel T. Dudley

SUPPORT CARE CANCER

Debasish Tripathy

BRAZ J MED BIOL RES

Robert J. Carroll

AIDS Res Ther

Katherina Pugliese
Daniela Francisci

HEALTH QUAL LIFE OUT
Kit N Simpson

Suzanne B. Henry RN DNSc FAAN

Jillian Inouye RN PhD
P.A. Grootenhuis

Meagan Whisenant
Xueying Wang

ARCH PHYS MED REHAB

Areej El-Jawahri

Yichi Zhang

Katherine P. Liao

J AM MED INFORM ASSN

Kevin T. Nead

Theresa A. Koleck

Philip E. Bourne

Donna J Cartwright
NAT BIOTECHNOL

Dan M. Roden

Alexandra A García
VALUE HEALTH

HEALTH SERV RES
L Duncan Saunders
William A Ghali
Semin Oncol Nurs
Verna A. Rhodes
Phyllis M. Watson
J CLIN EPIDEMIOL
Amy C. Justice

J CLIN ONCOL

K Mitchell Russell
Recruit researchers
Join for free
Login Email Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google Welcome back! Please log in. Email · Hint Tip: Most researchers use their institutional email address as their ResearchGate login Password Forgot password? Keep me logged in Log in or Continue with Google No account? Sign up

clinical research documentation in electronic health records

Nurse documentation and the electronic health record

Use the nursing process to take advantage of ehrs’ capabilities and optimize patient care..

Takeaways:

If not used properly, the electronic health record (EHR) can create communication gaps.
The nursing process can be applied to electronic documentation to avoid workarounds and close gaps in communication.
Effective use the EHR can improve patient safety and care outcomes.

Clinical documentation supports patient care, improves clinical outcomes, and enhances interprofessional communication. When you document your assessments, plans, and actions, you rely on nursing practice standards, organizational policies, meaningful use directives, and a variety of quality criteria.

Proper documentation protects patients and your license

Standardizing handoff communication

Nursing informatics: The EHR and beyond

Electronic health records (EHRs) support that documentation with data that help you enhance patient safety, evaluate care quality, maximize efficiency, and measure staffing needs. And they serve as a standard form of documentation that can be shared by everyone on the healthcare team. However, when not used appropriately, EHRs can reduce nurses’ use of their critical-thinking skills, increase reliance on workarounds to bypass forms, and lead to errors and lost documentation. How can nurses take advantage of the benefits inherent in EHRs and eliminate some of the frustrations?

Confirming suspicions

With that question in mind, the Nurse Practice Council (NPC) explored the prevalence of docu- mentation gaps in our organization, St. Joseph’s University Medical Center (including St. Joseph’s Children’s Hospital), which has received American Nurses Credentialing Center’s (ANCC) Magnet ® recognition four consecutive times. A close look at our quality department’s reports of near misses validated our suspicions on a range of issues, including human errors in recording heights and weights, missed vital sign trends, and generally poor handoff communication. The new workflow was affecting critical thinking and clinical judgment.

We took our concerns to the NPC where members described feelings of being torn between the priority of patient care and the chores of documentation. Nurses by nature are adaptive, so many resorted to workarounds, completing only mandatory elements, which led to less-than-ideal documentation. They told us that they were frustrated and dissatisfied with the EHR. Through collaboration with the NPC Informatics and Evidence Based Practice Committees, we explored how to improve nursing documentation by re-introducing the nursing process.

Identifying the problems

Over a period of 3 months, we retrospectively audited patient records from the medical-surgical area for baseline nursing documentation; data elements were analyzed for care decisions and patient safety. The initial work helped identify a number of design gaps, including fields that nurses weren’t required to complete but were essential for quality care. This deficiency was promptly fixed and was an easy win.

After all of the problems were identified, we chose nurse champions who were trained to continue chart audits and proper documentation, using the nursing process model on a larger scale. Training included workshops for proper EHR documentation techniques, record audits, case scenarios, and reflective feedback using Gibbs’ reflective cycle, a tool for helping people learn from situations. (For more information about Gibbs’ reflective cycle, see resources.eln.io/gibbs-reflective-cycle-model-1988/ .) With our goal of integrating the nursing process with the EHR, we adopted the American Nurses Association’s definition of the nursing process as “an assertive, problem-solving approach to the identi- fication and treatment of patient problems.” (See Make the connection .) And to give the nurses a tool to help develop patient-centered care plans, we adopted the plan-do-study-act change model. (See On the map .)

documentation electronic health record connection

Reviewing the outcomes

One year later, the project has expanded to many avenues of nursing, including RN orientation, preceptor classes, and individual unit education. Subsequent auditing (3, 6, and 9 months after education) shows improved documentation in areas with significant effect on patient care and safety, including these 3-month results:

admission medication reconciliation—from 52% to 70%
isolation indication—39% to 100%
plan of care appropriate for patient’s chief com- plaint—83% to 100%
plan of care related to patient comorbidities— 30% to 87%
education level—4% to 17%
safe patient handling—4% to 13%.
discharge planning—4% to 17%.

As with every change project, leadership commitment is key. Our nursing leaders were supportive of the project and the proposed solutions. However, we encountered some challenges (and developed some solutions), including:

Limited resources to spearhead change on a large scale— The NPC designed a tool to integrate the nursing process in our existing EHR.
Inability to reach all users and cover all specialties— The nursing process tool was disseminated through each NPC representative to their respective specialties to be used as a guide in EHR documentation.
Barriers to measuring the impact of change on patient outcomes and financial returns— Work has begun to develop a shorter and better way of auditing real-time documentation and evaluating nurses’ awareness and knowledge.

Staying in charge

This project empowered our NPC members to evaluate their documentation practices and reflect on what they learned from the audits, quality reports, and data mining. It enabled them to look to their future practices in clinical documentation and follow through with the nursing process. The EHR documentation review and tools have become part of the curriculum for the nursing preceptor workshops and our new hire orientation.

The authors work at St. Joseph’s University Medical Center in Paterson, New Jersey. Janet Pagulayan the nursing informatics coordinator. Salim Eltair is a nursing informatics systems manager. Kathy Faber is a clinical nurse leader and co-chair of EBP Nursing Practice Committee.

Selected references

Affordable Care Act (ACA). 2010. healthcare.gov/glossary/affordable- care-act Agency for Healthcare Research and Quality (AHRQ). Slide set: National quality strategy overview. 2017. ahrq.gov/workingforquality/nqs/overview.htm

American EMRs Association (ANA). Nursing: Scope and standards of practice . Silver Spring, MD: 2010; EMRsbooks.org nursingworld.org/~4af4f2/globalassets/docs/ana/ethics/principles-of-nursing-documentation.pdf

Beck SL, Weiss ME, Ryan-Wenger N, et al. Measuring nurses’ impact on health care quality: Progress, challenges, and future directions. Med Care . 2013;51(4 Suppl 2):S15-22.

Bowman S. Impact of electronic health record systems on information integrity: Quality and safety implications. Perspect Health Inf Manag . 2013;10:1c.

Conn J. Joint Commission puts focus on EHR, patient safety. Modern Healthcare . July 3, 2013. modernhealthcare.com/article/20130703/blog/307039936

Gibbs G. Learning by Doing: A Guide to Teaching and Learning Methods . Oxford: Further Education Unit; 1988.

Hendrich A, Chow MP, Skierczynski BA, Lu Z. A 36-hospital time and motion study: How do medical-surgical nurses spend their time? Perm J . 2008;12(3):25-34.

Joint Commission, The. Sentinel event alert 54: Safe use of health information technology. March 31, 2015. jointcommission.org/assets/1/6/SEA_54_HIT_4_26_16.pdf

Lavin MA, Harper E, Barr N. Health information technology, patient safety, and professional nursing care documentation in acute care settings. Online J Issues Nurs . 2015;20(2):6.

Mamykina L, Vawdrey DK, Stetson PD, Zheng K, Hripcsak G. Clinical documentation: Composition or synthesis? J Am Med Inform Assoc . 2012;19(6):1025-31.

Nykänen P, Kaipio J, Kuusisto A. Evaluation of the national nursing model and four nursing documentation systems in Finland—Lessons learned and directions for the future. Int J Med Inform . 2012;81(8):507-20.

Phillips W, Fleming D. Ethical concerns in the use of electronic medical records. Mo Med . 2009;106(5):328-33.

Nicol JS, Dosser I. Understanding reflective practice. Nurs Stand . 2016;30(36):34-42.

Sittig DF, Singh H. Toward more proactive approaches to safety in the electronic health record era. Jt Comm J Qual Patient Saf . 2017; 43(10):540-7.

Sockolow PS, Liao C, Chittams JL, Bowles KH. Evaluating the impact of electronic health records on nurse clinical process at two commu- nity health sites. NI. 2012:381.

U.S. Department of Health & Human Services. Health Information Technology for Economic and Clinical Health (HITECH) Act. 2009. hhs.gov/hipaa/for-professionals/special-topics/hitech-act-enforcement- interim-final-rule/index.html

3 Comments .

A great way of improving the use of EHR is to give proper education and training to give efficiency and productivity to nurses who handle patients every day. Thanks for the well-researched article. Keep it up.

Great way to take charge of things and bring the change that was required. Congrats to everyone who was involved.

using EHR in documentation ,has it really improved the work of the nurses or otherwise.

Comments are closed.

NurseLine Newsletter

First Name *
Last Name *
Hidden Referrer

*By submitting your e-mail, you are opting in to receiving information from Healthcom Media and Affiliates. The details, including your email address/mobile number, may be used to keep you informed about future products and services.

Test Your Knowledge

Oncology nursing: Clinical assistant onboarding

Medication safety and pediatric health

Confidence is in

Engaging nurses in scholarly work

Sepsis navigators and hospital outcomes

Electronic peer review and CLABSI rates

Tips for thriving–not just surviving–on the night shift

Cesarean birth reduction: EBP in action

Discharge hospitality center

LPNs in modified care delivery models

Patient care assistant training

Magnet4Europe

Innovation at the frontline of nursing

Partnering for nursing advancement

Journey of a novice Magnet® program director

Measuring Documentation Burden in Healthcare

Systematic Review
Published: 29 July 2024

Cite this article

M. Hassan Murad MD, MPH ORCID: orcid.org/0000-0001-5502-5975 1 ,
Brianna E. Vaa Stelling MD, MHPE 2 ,
Colin P. West MD, PhD 3 ,
Bashar Hasan MD 1 ,
Suvyaktha Simha B.A. 1 ,
Samer Saadi MD 1 ,
Mohammed Firwana MBBS 1 ,
Kelly E. Viola MPS 1 ,
Larry J. Prokop MLIS 4 ,
Tarek Nayfeh MD 1 &
Zhen Wang PhD 1

11 Accesses

Explore all metrics

The enactment of the Health Information Technology for Economic and Clinical Health Act and the wide adoption of electronic health record (EHR) systems have ushered in increasing documentation burden, frequently cited as a key factor affecting the work experience of healthcare professionals and a contributor to burnout. This systematic review aims to identify and characterize measures of documentation burden.

We integrated discussions with Key Informants and a comprehensive search of the literature, including MEDLINE, Embase, Scopus, and gray literature published between 2010 and 2023. Data were narratively and thematically synthesized.

We identified 135 articles about measuring documentation burden. We classified measures into 11 categories: overall time spent in EHR, activities related to clinical documentation, inbox management, time spent in clinical review, time spent in orders, work outside work/after hours, administrative tasks (billing and insurance related), fragmentation of workflow, measures of efficiency, EHR activity rate, and usability. The most common source of data for most measures was EHR usage logs. Direct tracking such as through time–motion analysis was fairly uncommon. Measures were developed and applied across various settings and populations, with physicians and nurses in the USA being the most frequently represented healthcare professionals. Evidence of validity of these measures was limited and incomplete. Data on the appropriateness of measures in terms of scalability, feasibility, or equity across various contexts were limited. The physician perspective was the most robustly captured and prominently focused on increased stress and burnout.

Numerous measures for documentation burden are available and have been tested in a variety of settings and contexts. However, most are one-dimensional, do not capture various domains of this construct, and lack robust validity evidence. This report serves as a call to action highlighting an urgent need for measure development that represents diverse clinical contexts and support future interventions.

This is a preview of subscription content, log in via an institution to check access.

Access this article

Subscribe and save.

Get 10 units per month
Download Article/Chapter or eBook
1 Unit = 1 Article or 1 Chapter
Cancel anytime

Price includes VAT (Russian Federation)

Instant access to the full article PDF.

Rent this article via DeepDyve

Institutional subscriptions

Temporal Associations Between EHR-Derived Workload, Burnout, and Errors: a Prospective Cohort Study

Frontline Perspectives on Physician Burnout and Strategies to Improve Well-Being: Interviews with Physicians and Health System Leaders

The Electronic Health Record

Shanafelt TD, West CP, Dyrbye LN, et al. Changes in Burnout and Satisfaction With Work-Life Integration in Physicians During the First 2 Years of the COVID-19 Pandemic. Mayo Clin Proc . 2022;97(12):2248-2258. https://doi.org/10.1016/j.mayocp.2022.09.002

Article CAS PubMed Google Scholar

National Academies of Sciences E, Medicine, National Academy of M, Committee on Systems Approaches to Improve Patient Care by Supporting Clinician W-B. Taking Action Against Clinician Burnout: A Systems Approach to Professional Well-Being. National Academies Press; 2019.

The Office of the National Coordinator for Health Information Technology. Strategy on Reducing Regulatory and Administrative Burden Relating to the Use of Health IT and EHRs . 2020. https://www.healthit.gov/sites/default/files/page/2020-02/BurdenReport_0.pdf . Accessed 24 July 2024.

Association AMI. AMIA 25x5: Reducing Documentation Burden. https://amia.org/about-amia/amia-25x5 . Accessed 24 July 2024.

Shanafelt TD, Dyrbye LN, Sinsky C, et al. Relationship Between Clerical Burden and Characteristics of the Electronic Environment With Physician Burnout and Professional Satisfaction. Journal Article. Mayo Clin Proc . 2016;91(7):836-48. https://doi.org/10.1016/j.mayocp.2016.05.007

Article PubMed Google Scholar

Moy AJ, Schwartz JM, Chen R, et al. Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review. Journal Article Research Support, N.I.H., Extramural Review. J Am Med Inform Assoc . 23 2021;28(5):998-1008. https://doi.org/10.1093/jamia/ocaa325

Article PubMed PubMed Central Google Scholar

Melnick ER, Dyrbye LN, Sinsky CA, et al. The Association Between Perceived Electronic Health Record Usability and Professional Burnout Among US Physicians. Mayo Clin Proc . 2020;95(3):476-487. https://doi.org/10.1016/j.mayocp.2019.09.024

Arndt BG, Beasley JW, Watkinson MD, et al. Tethered to the EHR: Primary Care Physician Workload Assessment Using EHR Event Log Data and Time-Motion Observations. Evaluation Study Journal Article. Ann Fam Med . Sep 2017;15(5):419-426. https://doi.org/10.1370/afm.2121

Dyrbye LN, Gordon J, O’Horo J, et al. Relationships Between EHR-Based Audit Log Data and Physician Burnout and Clinical Practice Process Measures. Mayo Clin Proc . 2023;98(3):398-409. https://doi.org/10.1016/j.mayocp.2022.10.027

Gaffney A, Woolhandler S, Cai C, et al. Medical Documentation Burden Among US Office-Based Physicians in 2019: A National Study. Letter. JAMA Intern Med . 2022;182(5):564-566. https://doi.org/10.1001/jamainternmed.2022.0372

Article Google Scholar

Sinsky CA, Biddison LD, Mallick A, et al. Organizational Evidence-Based and Promising Practices for Improving Clinician Well-Being. NAM Perspect . 2020;2020: https://doi.org/10.31478/202011a . https://doi.org/10.31478/202011a

Cutrona SL, Fouayzi H, Burns L, et al. Primary Care Providers' Opening of Time-Sensitive Alerts Sent to Commercial Electronic Health Record InBaskets. J Gen Intern Med . 2017;32(11):1210-1219. https://doi.org/10.1007/s11606-017-4146-3

Powell L, Sittig DF, Chrouser K, Singh H. Assessment of Health Information Technology-Related Outpatient Diagnostic Delays in the US Veterans Affairs Health Care System: A Qualitative Study of Aggregated Root Cause Analysis Data. JAMA Netw Open . 2020;3(6):e206752. https://doi.org/10.1001/jamanetworkopen.2020.6752

Marmor RA, Clay B, Millen M, Savides TJ, Longhurst CA. The Impact of Physician EHR Usage on Patient Satisfaction. Letter. Appl Clin Inform . Jan 2018;9(1):11-14. https://doi.org/10.1055/s-0037-1620263

Nguyen OT, Jenkins NJ, Khanna N, et al. A systematic review of contributing factors of and solutions to electronic health record-related impacts on physician well-being. Journal Article Systematic Review. J Am Med Inform Assoc . Apr 23 2021;28(5):974-984. https://doi.org/10.1093/jamia/ocaa339

Wilson LL, Fulton M. Risk management: how doctors, hospitals and MDOs can limit the costs of malpractice litigation. Med J Aust . 2000;172(2):77-80.

Albano GD, Bertozzi G, Maglietta F, et al. Medical Records Quality as Prevention Tool for Healthcare-Associated Infections (HAIs) Related Litigation: a Case Series. Curr Pharm Biotechnol . 2019;20(8):653-657. https://doi.org/10.2174/1389201020666190408102221

Teichman PG. Documentation tips for reducing malpractice risk. Fam Pract Manag . 2000;7(3):29-33.

CAS PubMed Google Scholar

Gettinger A, Zayas-Caban T. HITECH to 21st century cures: clinician burden and evolving health IT policy. Journal Article. Journal of the American Medical Informatics Association . 2021;28(5):1022-1025. https://doi.org/10.1093/jamia/ocaa330

DesRoches CM, Charles D, Furukawa MF, et al. Adoption Of Electronic Health Records Grows Rapidly, But Fewer Than Half Of US Hospitals Had At Least A Basic System In 2012. Health Affairs . 2013;32(8):1478-1485. https://doi.org/10.1377/hlthaff.2013.0308

Henry J, Pylypchuk Y, Searcy T, Patel V. Adoption of electronic health record systems among US non-federal acute care hospitals: 2008–2015. ONC data brief . 2016;35(35):2008-15.

Google Scholar

Association AM. Joy in Medicine ™: Health System Recognition Program. https://www.ama-assn.org/practice-management/physician-health/joy-medicine-health-system-recognition-program . Accessed 24 July 2024.

Research Protocol: Documentation Burden. Effective Health Care Program, Agency for Healthcare Research and Quality, Rockville, MD. https://effectivehealthcare.ahrq.gov/products/documentation-burden/protocol . Accessed 24 July 2024.

Messick S. Validity. In R. L. Linn (Ed.), Educational measurement 3rd ed. American Council on education and Macmillan . 1989:13–104.

Cook DA, Beckman TJ. Current concepts in validity and reliability for psychometric instruments: theory and application. Am J Med . 2006;119(2):166 e7-16. https://doi.org/10.1016/j.amjmed.2005.10.036

Anderson J, Leubner J, Brown SR. EHR Overtime: An Analysis of Time Spent After Hours by Family Physicians. Family Medicine . 2020;52(2):135-137. https://doi.org/10.22454/fammed.2020.942762

Aziz F, Talhelm L, Keefer J, Krawiec C. Vascular surgery residents spend one fifth of their time on electronic health records after duty hours. Journal of Vascular Surgery . 2019;69(5):1574-1579. https://doi.org/10.1016/j.jvs.2018.08.173

Bliven B, Bragg M, Long B. Medical Device Connectivity Case Study. Journal of Clinical Engineering . 2016;41(2):E1-E11. https://doi.org/10.1097/jce.0000000000000144

Cox ML, Farjat AE, Risoli TJ, et al. Documenting or Operating: Where Is Time Spent in General Surgery Residency? Journal Article. J Surg Educ . 2018;75(6):e97-e106. https://doi.org/10.1016/j.jsurg.2018.10.010

Cross DA, Holmgren AJ, Apathy NC. The role of organizations in shaping physician use of electronic health records. Journal Article. Health Serv Res . 2023;12:12. https://doi.org/10.1111/1475-6773.14203

Dibbs RP, Appel R, Smerica AM, Davies LW, Ferry AM, Buchanan EP. Inefficiencies of electronic medical record use by surgical healthcare providers. Health Policy and Technology . 2022;11(1)100597. https://doi.org/10.1016/j.hlpt.2022.100597

Holmgren AJ, Lindeman B, Ford EW. Resident Physician Experience and Duration of Electronic Health Record Use. Journal Article. Appl Clin Inform . 2021;12(4):721-728. https://doi.org/10.1055/s-0041-1732403

Holmgren AJ, Rotenstein L, Downing NL, Bates DW, Schulman K. Association between state-level malpractice environment and clinician electronic health record (EHR) time. Journal Article. J Am Med Inform Assoc . 2022;29(6):1069-1077. https://doi.org/10.1093/jamia/ocac034

Jhaveri P, Abdulahad D, Fogel B, et al. Impact of Scribe Intervention on Documentation in an Outpatient Pediatric Primary Care Practice. Journal Article Observational Study. Acad Pediatr . 2022;22(2):289-295. https://doi.org/10.1016/j.acap.2021.05.004

Kadish SS, Mayer EL, Jackman DM, et al. Implementation to Optimization: A Tailored, Data-Driven Approach to Improve Provider Efficiency and Confidence in Use of the Electronic Medical Record. Journal Article. J Oncol Pract . 2018;14(7):e421-e428. https://doi.org/10.1200/JOP.18.00093

Kannampallil TG, Denton CA, Shapiro JS, Patel VL. Efficiency of Emergency Physicians: Insights from an Observational Study using EHR Log Files. Applied clinical informatics . 2018;9(1):99-104. https://doi.org/10.1055/s-0037-1621705

Khan AR, Rosenthal CD, Ternes K, Sing RF, Sachdev G. Time Spent by Intensive Care Unit Nurses on the Electronic Health Record. Journal Article Observational Study. Crit Care Nurse . 2022;42(5):44-50. https://doi.org/10.4037/ccn2022518

Lo B, Sequeira L, Karunaithas A, Strudwick G, Jankowicz D, Tajirian T. The Impact of the COVID-19 Pandemic on Physician Electronic Health Record Use and Burden at a Canadian Mental Health Hospital. Journal Article. Stud Health Technol Inform . 2022;295:157-160. https://doi.org/10.3233/SHTI220685

Loszko A, Watson M, Khan A, et al. Acute Care Surgeons Spend More Time than General Surgeons on the Electronic Health Record (EHR). Journal Article. Am Surg . 2023;89(5):1497-1503. https://doi.org/10.1177/00031348211061102

Lou SS, Lew D, Harford DR, et al. Temporal Associations Between EHR-Derived Workload, Burnout, and Errors: a Prospective Cohort Study. Journal Article Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. J Gen Intern Med . 2022;37(9):2165-2172. https://doi.org/10.1007/s11606-022-07620-3

Micek MA, Arndt B, Baltus JJ, et al. The effect of remote scribes on primary care physicians' wellness, EHR satisfaction, and EHR use. Journal Article. Healthc (Amst) . 2022;10(4):100663. https://doi.org/10.1016/j.hjdsi.2022.100663

Mosquera MJ, Ward HB, Holland C, Boland R, Torous J. Using objective clinical metrics to understand the relationship between the electronic health record and physician well-being: observational pilot study. Journal Article. BJPsych Open . 2021;7(5):e174. https://doi.org/10.1192/bjo.2021.993

Nguyen OT, Turner K, Apathy NC, et al. Primary care physicians' electronic health record proficiency and efficiency behaviors and time interacting with electronic health records: a quantile regression analysis. Journal Article Research Support, N.I.H., Extramural. J Am Med Inform Assoc . 2022;29(3):461-471. https://doi.org/10.1093/jamia/ocab272

Nguyen O, Turner K, Apathy N, et al. Time Spent in the Electronic Health Record by Neurologists: Analyzing daily use inside and outside the clinic using EPIC Signal data. Conference Abstract. Neurology . 2022;98(18)

Ong SY, Moore Jeffery M, Williams B, O’Connell RT, Goldstein R, Melnick ER. How a Virtual Scribe Program Improves Physicians’ EHR Experience, Documentation Time, and Note Quality. Article. NEJM Catal Inno Care Del . 2021;2(12) https://doi.org/10.1056/cat.21.0294

Rittenberg E, Liebman JB, Rexrode KM. Primary Care Physician Gender and Electronic Health Record Workload. Journal Article Research Support, Non-U.S. Gov't. J Gen Intern Med . 2022;37(13):3295-3301. https://doi.org/10.1007/s11606-021-07298-z

Rotenstein LS, Holmgren AJ, Healey MJ, et al. Association Between Electronic Health Record Time and Quality of Care Metrics in Primary Care. Journal Article. JAMA Netw Open . 2022;5(10):e2237086. https://doi.org/10.1001/jamanetworkopen.2022.37086

Rotenstein LS, Apathy N, Holmgren AJ, Bates DW. Physician Note Composition Patterns and Time on the EHR Across Specialty Types: a National, Cross-sectional Study. Journal Article. J Gen Intern Med . 2023;38(5):1119-1126. https://doi.org/10.1007/s11606-022-07834-5

Ruan E, Beiser M, Lu V, et al. Physician Electronic Health Record Usage as Affected by the COVID-19 Pandemic. Journal Article. Appl Clin Inform . 2022;13(4):785-793. https://doi.org/10.1055/a-1877-2745

Sinsky C, Colligan L, Li L, et al. Allocation of Physician Time in Ambulatory Practice: A Time and Motion Study in 4 Specialties. Annals of Internal Medicine . 2016;165(11):753. https://doi.org/10.7326/m16-0961

Tai-Seale M, Olson CW, Li J, et al. Electronic Health Record Logs Indicate That Physicians Split Time Evenly Between Seeing Patients And Desktop Medicine. Health Aff (Millwood) . 2017;36(4):655-662. https://doi.org/10.1377/hlthaff.2016.0811

Tran B, Lenhart A, Ross R, Dorr DA. Burnout and EHR use among academic primary care physicians with varied clinical workloads. AMIA Jt Summits Transl Sci Proc . 2019;2019:136-144.

PubMed PubMed Central Google Scholar

Verma G, Ivanov A, Benn F, et al. Analyses of electronic health records utilization in a large community hospital. Journal Article. PLoS ONE . 2020;15(7):e0233004. https://doi.org/10.1371/journal.pone.0233004

Wang JK, Ouyang D, Hom J, Chi J, Chen JH. Characterizing electronic health record usage patterns of inpatient medicine residents using event log data. Journal Article Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. PLoS ONE . 2019;14(2):e0205379. https://doi.org/10.1371/journal.pone.0205379

Article CAS PubMed PubMed Central Google Scholar

Tang K, Labagnara K, Babar M, et al. Electronic Health Record Usage Patterns Across Surgical Subspecialties. Journal Article. Applied Clinical Informatics . 2023;18:18. https://doi.org/10.1055/a-2194-1061

Sim J, Mani K, Fazzari M, et al. Using K-Means Clustering to Identify Physician Clusters by Electronic Health Record Burden and Efficiency. Journal Article. Telemed J E Health . 2023;21:21. https://doi.org/10.1089/tmj.2023.0167

de Hoop T, Neumuth T. Evaluating Electronic Health Record Limitations and Time Expenditure in a German Medical Center. Journal Article. Appl Clin Inform . 2021;12(5):1082-1090. https://doi.org/10.1055/s-0041-1739519

Ebbers T, Kool RB, Smeele LE, Takes RP, van den Broek GB, Dirven R. Quantifying the Electronic Health Record Burden in Head and Neck Cancer Care. Journal Article. Appl Clin Inform . 2022;13(4):857-864. https://doi.org/10.1055/s-0042-1756422

Ehrler F, Wu DTY, Ducloux P, Blondon K. A mobile application to support bedside nurse documentation and care: a time and motion study. Journal Article. JAMIA open . 2021;4(3):ooab046. https://doi.org/10.1093/jamiaopen/ooab046

Lindsay MR, Lytle K. Implementing Best Practices to Redesign Workflow and Optimize Nursing Documentation in the Electronic Health Record. Journal Article. Appl Clin Inform . 2022;13(3):711-719. https://doi.org/10.1055/a-1868-6431

Ho VT, Sgroi MD, Chandra V, Asch SM, Chen JH, Lee JT. Utilizing remote access for electronic medical records reduces overall electronic medical record time for vascular surgery residents. Journal Article. J Vasc Surg . 2023;77(6):1797-1802. https://doi.org/10.1016/j.jvs.2023.01.198

Patel R, Yang Y, Lin FC, et al. Descriptive Analysis of Documentation Time for the National Developmental-Behavioral Pediatric Physician Workforce Using a Commercial Electronic Health Record System. Journal Article. J Dev Behav Pediatr . 2023;44(5):e365-e369. https://doi.org/10.1097/DBP.0000000000001185

Berg GM, Shupsky T, Morales K. Resident Indentified Violations of Usability Heuristic Principles in Local Electronic Health Records. Journal Article. Kans J Med . 2020;13:84-89.

PubMed Google Scholar

Congelosi PD, Eid MA, Sorensen MJ. Surgical Providers' Perceptions of the Patient Portal: Before and After the 21st Century Cures Act. Journal Article. J Surg Res . 2023;289:234-240. https://doi.org/10.1016/j.jss.2023.03.007

Meltzer EC, Vorseth KS, Croghan IT, et al. Use of the Electronic Health Record During Clinical Encounters: An Experience Survey. Journal Article Research Support, Non-U.S. Gov't. Ann Fam Med . 2022;20(4):312-318. https://doi.org/10.1370/afm.2826

Mishra P, Kiang JC, Grant RW. Association of Medical Scribes in Primary Care With Physician Workflow and Patient Experience. Journal Article. JAMA Internal Medicine . 2018;178(11):1467-1472. https://doi.org/10.1001/jamainternmed.2018.3956

Tajirian T, Stergiopoulos V, Strudwick G, et al. The Influence of Electronic Health Record Use on Physician Burnout: Cross-Sectional Survey. Journal Article. J Med Internet Res . 2020;22(7):e19274. https://doi.org/10.2196/19274

Ahn M, Choi M, Kim Y. Factors Associated with the Timeliness of Electronic Nursing Documentation. Healthc Inform Res . 2016;22(4):270-276. https://doi.org/10.4258/hir.2016.22.4.270

Apathy NC, Rotenstein L, Bates DW, Holmgren AJ. Documentation dynamics: Note composition, burden, and physician efficiency. Journal Article Research Support, U.S. Gov't, P.H.S. Health Serv Res . 2023;58(3):674-685. https://doi.org/10.1111/1475-6773.14097

Apathy NC, Hare AJ, Fendrich S, Cross DA. I had not time to make it shorter: an exploratory analysis of how physicians reduce note length and time in notes. Journal Article Research Support, N.I.H., Extramural Research Support, U.S. Gov't, P.H.S. J Am Med Inform Assoc . 2023;30(2):355-360. https://doi.org/10.1093/jamia/ocac211

Bartek B, Lou SS, Kannampallil T. Measuring the cognitive effort associated with task switching in routine EHR-based tasks. Journal Article. J Biomed Inform . 2023;141:104349. https://doi.org/10.1016/j.jbi.2023.104349

Edwards A, Kanner L, Tewar S, Pesce L, Leyser M. The Value of Adding Scribe Services to 2 Distinct Pediatric Subspecialties in the Era of the Electronic Medical Record. Journal Article. Clin Pediatr (Phila) . May 16 2023:99228231174849. https://doi.org/10.1177/00099228231174849

Holmgren AJ, Apathy NC. Assessing the impact of patient access to clinical notes on clinician EHR documentation. Journal Article. J Am Med Inform Assoc . 2022;29(10):1733-1736. https://doi.org/10.1093/jamia/ocac120

Horn JJ, Doucette JN, Sweeney NL. An Essential Clinical Dataset Intervention for Nursing Documentation of a Pediatric Admission History Database. Journal Article. J Pediatr Nurs . 2021;59:110-114. https://doi.org/10.1016/j.pedn.2021.03.022

Hripcsak G, Vawdrey DK, Fred MR, Bostwick SB. Use of electronic clinical documentation: time spent and team interactions. Journal of the American Medical Informatics Association : JAMIA . 2011;18(2):112-117. https://doi.org/10.1136/jamia.2010.008441

Hsieh H-Y, Henker R, Ren D, et al. Improving Effectiveness and Satisfaction of an Electronic Charting System in Taiwan. Journal Article. Clin Nurse Spec . 2016;30(6):E1-E6. https://doi.org/10.1097/NUR.0000000000000250

Karp EL, Freeman R, Simpson KN, Simpson AN. Changes in Efficiency and Quality of Nursing Electronic Health Record Documentation After Implementation of an Admission Patient History Essential Data Set. CIN: Computers, Informatics, Nursing . 2019;37(5):260-265. https://doi.org/10.1097/cin.0000000000000516

Krawiec C, Stetter C, Kong L, Haidet P. Impact of Patient Census and Admission Mortality on Pediatric Intensive Care Unit Attending Electronic Health Record Activity: A Preliminary Study. Applied clinical informatics . 2020;11(2):226-234. https://doi.org/10.1055/s-0040-1705108

Lam C, Shumaker K, Butt M, Leiphart P, Miller JJ, Anderson BE. Impact of medical scribes on physician and patient satisfaction in dermatology. Journal Article. Arch Dermatol Res . 2022;314(1):71-76. https://doi.org/10.1007/s00403-021-02206-1

Mani K, Canarick J, Ruan E, Liu J, Kitsis E, Jariwala SP. Effect of Telemedicine and the COVID-19 Pandemic on Medical Trainees' Usage of the Electronic Health Record in the Outpatient Setting. Journal Article. Appl Clin Inform . 2023;14(2):309-320. https://doi.org/10.1055/a-2031-9437

Moy AJ, Aaron L, Cato KD, et al. Characterizing Multitasking and Workflow Fragmentation in Electronic Health Records among Emergency Department Clinicians: Using Time-Motion Data to Understand Documentation Burden. Journal Article Research Support, N.I.H., Extramural. Appl Clin Inform . 2021;12(5):1002-1013. https://doi.org/10.1055/s-0041-1736625

Munyisia EN, Yu P, Hailey D. The impact of an electronic nursing documentation system on efficiency of documentation by caregivers in a residential aged care facility. J Clin Nurs . 2012;21(19-20):2940-8. https://doi.org/10.1111/j.1365-2702.2012.04157.x

Overhage JM, McCallie D. Physician Time Spent Using the Electronic Health Record During Outpatient Encounters. Annals of Internal Medicine . 2020;173(7):594-595. https://doi.org/10.7326/l20-0278

Phillips T, Baur K. Nursing Praxis for Reducing Documentation Burden Within Nursing Admission Assessments. Journal Article. Comput Inform Nurs . 2021;39(11):627-633. https://doi.org/10.1097/CIN.0000000000000776

Sutton DE, Fogel JR, Giard AS, Gulker LA, Ivory CH, Rosa AM. Defining an Essential Clinical Dataset for Admission Patient History to Reduce Nursing Documentation Burden. Journal Article. Appl Clin Inform . 2020;11(3):464-473. https://doi.org/10.1055/s-0040-1713634

Vogel M, Kaisers W, Wassmuth R, Mayatepek E. Analysis of Documentation Speed Using Web-Based Medical Speech Recognition Technology: Randomized Controlled Trial. Journal Article Randomized Controlled Trial. J Med Internet Res . 2015;17(11):e247. https://doi.org/10.2196/jmir.5072

Baxter SL, Gali HE, Chiang MF, et al. Promoting Quality Face-to-Face Communication during Ophthalmology Encounters in the Electronic Health Record Era. Journal Article Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. Appl Clin Inform . 2020;11(1):130-141. https://doi.org/10.1055/s-0040-1701255

Chen AJ, Baxter SL, Gali HE, et al. Evaluation of Electronic Health Record Implementation in an Academic Oculoplastics Practice. Journal Article. Ophthal Plast Reconstr Surg . 2020;36(3):277-283. https://doi.org/10.1097/IOP.0000000000001531

Dela Cruz JE, Shabosky JC, Albrecht M, et al. Typed versus voice recognition for data entry in electronic health records: emergency physician time use and interruptions. Journal Article Multicenter Study Observational Study. West J Emerg Med . 2014;15(4):541-7. https://doi.org/10.5811/westjem.2014.3.19658

Earls ST, Savageau JA, Begley S, Saver BG, Sullivan K, Chuman A. Can scribes boost FPs' efficiency and job satisfaction? Journal Article. J . 2017;66(4):206-214.

Feely K, Edbrooke L, Bower W, et al. Allied health professionals' experiences and lessons learned in response to a big bang electronic medical record implementation: A prospective observational study. Observational Study Journal Article. Int J Med Inf . 2023;176:105094. https://doi.org/10.1016/j.ijmedinf.2023.105094

Gali HE, Baxter SL, Lander L, et al. Impact of Electronic Health Record Implementation on Ophthalmology Trainee Time Expenditures. Journal Article. J Acad Ophthalmol (2017) . 2019;11(2):e65-e72. https://doi.org/10.1055/s-0039-3401986

Heaton HA, Wang R, Farrell KJ, et al. Time Motion Analysis: Impact of Scribes on Provider Time Management. Journal Article Observational Study. J Emerg Med . 2018;55(1):135-140. https://doi.org/10.1016/j.jemermed.2018.04.018

Lilly CM, Cucchi E, Marshall N, Katz A. Battling Intensivist Burnout: A Role for Workload Management. Review. Chest . 2019;156(5):1001-1007. https://doi.org/10.1016/j.chest.2019.04.103

Mamykina L, Vawdrey DK, Stetson PD, Zheng K, Hripcsak G. Clinical documentation: composition or synthesis? J Am Med Inform Assoc . 2012;19(6):1025-31. https://doi.org/10.1136/amiajnl-2012-000901

Mamykina L, Vawdrey DK, Hripcsak G. How Do Residents Spend Their Shift Time? A Time and Motion Study With a Particular Focus on the Use of Computers. Acad Med . 2016;91(6):827-832. https://doi.org/10.1097/ACM.0000000000001148

Raney L, McManaman J, Elsaid M, et al. Multisite Quality Improvement Initiative to Repair Incomplete Electronic Medical Record Documentation As One of Many Causes of Provider Burnout. Journal Article. JCO Oncol Pract . 2020;16(11):e1412-e1416. https://doi.org/10.1200/OP.20.00294

Shuaib W, Hilmi J, Caballero J, et al. Impact of a scribe program on patient throughput, physician productivity, and patient satisfaction in a community-based emergency department. Journal Article. Health Inform J . Mar 2019;25(1):216-224. https://doi.org/10.1177/1460458217704255

Collins S, Couture B, Kang MJ, et al. Quantifying and Visualizing Nursing Flowsheet Documentation Burden in Acute and Critical Care. Journal Article Research Support, N.I.H., Extramural. AMIA Annu Symp Proc . 2018;2018:348-357.

Ludley A, Ting A, Malik D, Sivanadarajah N. Observational analysis of documentation burden and data duplication in trauma patient pathways at a major trauma centre. Observational Study Journal Article. BMJ open qual . 2023;12(2):04. https://doi.org/10.1136/bmjoq-2022-002084

Hilliard RW, Haskell J, Gardner RL. Are specific elements of electronic health record use associated with clinician burnout more than others? Journal Article. J Am Med Inform Assoc . 2020;27(9):1401-1410. https://doi.org/10.1093/jamia/ocaa092

Kumah-Crystal YA, Stein PM, Chen Q, et al. Before-Visit Questionnaire: A Tool to Augment Communication and Decrease Provider Documentation Burden in Pediatric Diabetes. Journal Article Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. Appl Clin Inform . 2021;12(5):969-978. https://doi.org/10.1055/s-0041-1736223

Nguyen OT, Hanna K, Merlo LJ, et al. Early Performance of the Patients Over Paperwork Initiative among Family Medicine Physicians. Journal Article Research Support, N.I.H., Extramural Research Support, Non-U.S. Gov't. South Med J . 03 2023;116(3):255-263. https://doi.org/10.14423/SMJ.0000000000001526

Parker R, Smith G. Sex differences in provider documentation as a source of burnout: A single-institution cross-sectional study. J Am Acad Dermatol . Nov 2021;85(5):1317-1318. https://doi.org/10.1016/j.jaad.2020.10.020

Perotte R, Hajicharalambous C, Sugalski G, Underwood JP. Characterization of Electronic Health Record Documentation Shortcuts: Does the use of dotphrases increase efficiency in the Emergency Department? Journal Article. AMIA Annu Symp Proc . 2021;2021:969-978.

Alissa R, Hipp JA, Webb K. Saving Time for Patient Care by Optimizing Physician Note Templates: A Pilot Study. Journal Article. Front . 2021;3:772356. https://doi.org/10.3389/fdgth.2021.772356

Arora A, Garg A, Rizvi M, Desai N. National survey of pediatric care providers: Assessing time and impact of coding and documentation in physician practice. Pediatrics Conference: National Conference on Education . 2018;141(1) https://doi.org/10.1177/0009922818774341

Carlson KL, McFadden SE, Barkin S. Improving Documentation Timeliness: A "Brighter Future" for the Electronic Medical Record in Resident Clinics. Journal Article Observational Study Research Support, Non-U.S. Gov't. Academic Medicine . 2015;90(12):1641-5. https://doi.org/10.1097/ACM.0000000000000792

Gardner RL, Cooper E, Haskell J, et al. Physician stress and burnout: the impact of health information technology. Journal Article Research Support, Non-U.S. Gov't. J Am Med Inform Assoc . 2019;26(2):106-114. https://doi.org/10.1093/jamia/ocy145

Gesner E, Dykes PC, Zhang L, Gazarian P. Documentation Burden in Nursing and Its Role in Clinician Burnout Syndrome. Journal Article. Appl Clin Inform . 2022;13(5):983-990. https://doi.org/10.1055/s-0042-1757157

Harris DA, Haskell J, Cooper E, Crouse N, Gardner R. Estimating the association between burnout and electronic health record-related stress among advanced practice registered nurses. Journal Article. Appl Nurs Res . 2018;43:36-41. https://doi.org/10.1016/j.apnr.2018.06.014

Kroth PJ, Morioka-Douglas N, Veres S, et al. The electronic elephant in the room: Physicians and the electronic health record. Journal Article. JAMIA open . 2018;1(1):49-56. https://doi.org/10.1093/jamiaopen/ooy016

Linzer M, Poplau S, Babbott S, et al. Worklife and Wellness in Academic General Internal Medicine: Results from a National Survey. Journal Article Research Support, N.I.H., Extramural. J Gen Intern Med . 2016;31(9):1004-10. https://doi.org/10.1007/s11606-016-3720-4

Olson K, Sinsky C, Rinne ST, et al. Cross-sectional survey of workplace stressors associated with physician burnout measured by the Mini-Z and the Maslach Burnout Inventory. Journal Article. Stress health . 2019;35(2):157-175. https://doi.org/10.1002/smi.2849

Rassolian M, Peterson LE, Fang B, et al. Workplace Factors Associated With Burnout of Family Physicians. Letter. JAMA Intern Med . 2017;177(7):1036-1038. https://doi.org/10.1001/jamainternmed.2017.1391

Rotenstein LS, Apathy N, Landon B, Bates DW. Assessment of Satisfaction With the Electronic Health Record Among Physicians in Physician-Owned vs Non-Physician-Owned Practices. Journal Article. JAMA Netw Open . 2022;5(4):e228301. https://doi.org/10.1001/jamanetworkopen.2022.8301

Taylor KA, McQuilkin D, Hughes RG. Medical Scribe Impact on Patient and Provider Experience. Journal Article. Mil Med . 2019;184(9-10):388-393. https://doi.org/10.1093/milmed/usz030

Yuan CM, Little DJ, Marks ES, et al. The Electronic Medical Record and Nephrology Fellowship Education in the United States: An Opinion Survey. Journal Article. Clin J Am Soc Nephrol . 2020;15(7):949-956. https://doi.org/10.2215/CJN.14191119

Ferguson K, Fraser M, Tuna M, Bruntz C, Dahrouge S. The Impact of an Electronic Portal on Patient Encounters in Primary Care: Interrupted Time-Series Analysis. Journal Article. JMIR Med Inform . 2023;11:e43567. https://doi.org/10.2196/43567

Kesler K, Wynn M, Pugely AJ. Time and Clerical Burden Posed by the Current Electronic Health Record for Orthopaedic Surgeons. Journal Article. J Am Acad Orthop Surg . 2022;30(1):e34-e43. https://doi.org/10.5435/JAAOS-D-21-00094

Adler-Milstein J, Zhao W, Willard-Grace R, Knox M, Grumbach K. Electronic health records and burnout: Time spent on the electronic health record after hours and message volume associated with exhaustion but not with cynicism among primary care clinicians. Journal of the American Medical Informatics Association : JAMIA . 2020;27(4):531-538. https://doi.org/10.1093/jamia/ocz220

Benson M, Gopal D, Pfau P. Electronic Health Record Work Demands for Gastroenterology and Hepatology Providers: A Prospective Use Analysis and Survey Study. Journal Article. Dig Dis Sci . 2023;68(4):1218-1225. https://doi.org/10.1007/s10620-022-07691-6

DiAngi YT, Stevens LA, Halpern-Felsher B, Pageler NM, Lee TC. Electronic health record (EHR) training program identifies a new tool to quantify the EHR time burden and improves providers' perceived control over their workload in the EHR. JAMIA open . 2019;2(2):222-230. https://doi.org/10.1093/jamiaopen/ooz003

Gilman EA, Aakre C, Meyers A, et al. Development of a Visit Facilitator Role to Assist Physicians in an Ambulatory Consultative Medical Practice. Journal Article. Mayo Clin Proc Innov Qual Outcomes . 2023;7(3):187-193. https://doi.org/10.1016/j.mayocpiqo.2023.04.003

Attipoe S, Hoffman J, Rust S, et al. Characterization of Electronic Health Record Use Outside Scheduled Clinic Hours Among Primary Care Pediatricians: Retrospective Descriptive Task Analysis of Electronic Health Record Access Log Data. Journal Article. JMIR Med Inform . 2022;10(5):e34787. https://doi.org/10.2196/34787

Beiser M, Lu V, Paul S, et al. Electronic Health Record Usage Patterns: Assessing Telemedicine's Impact on the Provider Experience During the COVID-19 Pandemic. Journal Article. Telemed J E Health . 2021;27(8):934-938. https://doi.org/10.1089/tmj.2020.0490

De Groot K, De Veer AJE, Munster AM, Francke AL, Paans W. Nursing documentation and its relationship with perceived nursing workload: a mixed-methods study among community nurses. Journal Article. BMC Nurs . 28 2022;21(1):34. https://doi.org/10.1186/s12912-022-00811-7

Frintner MP, Kaelber DC, Kirkendall ES, Lourie EM, Somberg CA, Lehmann CU. The Effect of Electronic Health Record Burden on Pediatricians' Work-Life Balance and Career Satisfaction. Journal Article Research Support, Non-U.S. Gov't. Appl Clin Inform . 2021;12(3):697-707. https://doi.org/10.1055/s-0041-1732402

Peccoralo LA, Kaplan CA, Pietrzak RH, Charney DS, Ripp JA. The impact of time spent on the electronic health record after work and of clerical work on burnout among clinical faculty. Journal Article Research Support, Non-U.S. Gov't. J Am Med Inform Assoc . 2021;28(5):938-947. https://doi.org/10.1093/jamia/ocaa349

Moy AJ, Schwartz JM, Elias J, et al. Time-motion examination of electronic health record utilization and clinician workflows indicate frequent task switching and documentation burden. Journal Article Research Support, N.I.H., Extramural. AMIA Annu Symp Proc . 2020;2020:886-895.

Chaparro JD, Hussain C, Lee JA, Hehmeyer J, Nguyen M, Hoffman J. Reducing Interruptive Alert Burden Using Quality Improvement Methodology. Journal Article. Appl Clin Inform . 2020;11(1):46-58. https://doi.org/10.1055/s-0039-3402757

Fouquet SD, Fitzmaurice L, Chan YR, Palmer EM. Doctors documenting: an ethnographic and informatics approach to understanding attending physician documentation in the pediatric emergency department. Journal Article. J Am Med Inform Assoc . 2021;28(2):239-248. https://doi.org/10.1093/jamia/ocaa252

Sockolow PS, Liao C, Chittams JL, Bowles KH. Evaluating the impact of electronic health records on nurse clinical process at two community health sites. Journal Article. Ni 2012 (2012) . 2012;2012:381.

Gidwani R, Nguyen C, Kofoed A, et al. Impact of Scribes on Physician Satisfaction, Patient Satisfaction, and Charting Efficiency: A Randomized Controlled Trial. Journal Article Randomized Controlled Trial Research Support, Non-U.S. Gov't. Ann Fam Med . 2017;15(5):427-433. https://doi.org/10.1370/afm.2122

Florig ST, Corby S, Rosson NT, et al. Chart Completion Time of Attending Physicians While Using Medical Scribes. Journal Article Research Support, U.S. Gov't, P.H.S. AMIA Annu Symp Proc . 2021;2021:457-465.

Goldstein IH, Hwang T, Gowrisankaran S, Bales R, Chiang MF, Hribar MR. Changes in Electronic Health Record Use Time and Documentation over the Course of a Decade. Ophthalmology . 2019;126(6):783-791. https://doi.org/10.1016/j.ophtha.2019.01.011

Zallman L, Altman W, Chu L, et al. Do Medical Scribes Help Primary Care Providers Respond More Quickly to Out-of-Visit Tasks? Journal Article Research Support, Non-U.S. Gov't. J Am Board Fam Med . 2021;34(1):70-77. https://doi.org/10.3122/jabfm.2021.01.200330

Li T, Yu L, Zhou L, Wang P. Using less keystrokes to achieve high top-1 accuracy in Chinese clinical text entry. Journal Article. Digit Health . 2023;9:20552076231179027. https://doi.org/10.1177/20552076231179027

Windle JR, Windle TA, Shamavu KY, et al. Roadmap to a more useful and usable electronic health record. Journal Article. Cardiovasc Digit Health J . 2021;2(6):301-311. https://doi.org/10.1016/j.cvdhj.2021.09.007

Tell A, Westenhofer J, Harth V, Mache S. Stressors, Resources, and Strain Associated with Digitization Processes of Medical Staff Working in Neurosurgical and Vascular Surgical Hospital Wards: A Multimethod Study. Journal Article. Healthcare (Basel) . 2023;11(14):09. https://doi.org/10.3390/healthcare11141988

Baugh JJ, Monette DL, Takayesu JK, Raja AS, Yun BJ. Documentation Displaces Teaching in an Academic Emergency Department. Journal Article. West J Emerg Med . 2020;21(4):974-977. https://doi.org/10.5811/westjem.2020.5.46962

AlQahtani M, AlShaibani W, AlAmri E, Edward D, Khandekar R. Electronic Health Record-Related Stress Among Nurses: Determinants and Solutions. Journal Article. Telemed J E Health . 2021;27(5):544-550. https://doi.org/10.1089/tmj.2020.0059

Brown JA, Cooper AL, Albrecht MA. Development and content validation of the Burden of Documentation for Nurses and Midwives (BurDoNsaM) survey. J Adv Nurs . 2020;76(5):1273-1281. https://doi.org/10.1111/jan.14320

Arndt BG, Micek MA, Rule A, Shafer CM, Baltus JJ, Sinsky CA. More Tethered to the EHR: EHR Workload Trends Among Academic Primary Care Physicians, 2019-2023. Ann Fam Med . 2024;22(1):12-18. https://doi.org/10.1370/afm.3047

Cohen GR, Boi J, Johnson C, Brown L, Patel V. Measuring time clinicians spend using EHRs in the inpatient setting: a national, mixed-methods study. Journal Article Research Support, Non-U.S. Gov't. J Am Med Inform Assoc . 2021;28(8):1676-1682. https://doi.org/10.1093/jamia/ocab042

Baxter SL, Apathy NC, Cross DA, Sinsky C, Hribar MR. Measures of electronic health record use in outpatient settings across vendors. Journal Article Research Support, N.I.H., Extramural. Journal of the American Medical Informatics Association . 2021;28(5):955-959. https://doi.org/10.1093/jamia/ocaa266

Download references

Acknowledgements:

This report is based on research conducted by the Mayo Clinic Evidence-based Practice Center (EPC) under contract from the Agency for Healthcare Research and Quality (AHRQ), Rockville, MD (Contract No. 75Q80120D00005/75Q80123F32005). The authors gratefully acknowledge Task Order Officers Angela Carr, D.Soc.Sci., M.H.A., R.N, and Suchitra Iyer, Ph.D., from the Agency for Healthcare Research and Quality for their contributions to this project.

Author information

Authors and affiliations.

Mayo Clinic Evidence-Based Practice Center, Mayo Clinic Robert D. and Patricia E. Kern Center for the Science of Healthcare Delivery, Mayo Clinic, Rochester, MN, USA

M. Hassan Murad MD, MPH, Bashar Hasan MD, Suvyaktha Simha B.A., Samer Saadi MD, Mohammed Firwana MBBS, Kelly E. Viola MPS, Tarek Nayfeh MD & Zhen Wang PhD

Division of Community Internal Medicine, Department of Medicine, Mayo Clinic, Rochester, MN, USA

Brianna E. Vaa Stelling MD, MHPE

Division of General Internal Medicine, Department of Medicine, Mayo Clinic, Rochester, MN, USA

Colin P. West MD, PhD

Mayo Clinic Libraries, Mayo Clinic, Rochester, MN, USA

Larry J. Prokop MLIS

You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to M. Hassan Murad MD, MPH .

Ethics declarations

Conflict of interest:.

The authors declare that they do not have a conflict of interest.

Disclaimer:

The findings and conclusions in this document are those of the authors, who are responsible for its contents; the findings and conclusions do not necessarily represent the views of AHRQ or the U.S. Department of Health and Human Services. Therefore, no statement in this report should be construed as an official position of AHRQ or of the U.S. Department of Health and Human Services.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Below is the link to the electronic supplementary material.

Supplementary file1 (DOCX 478 KB)

Rights and permissions.

Springer Nature or its licensor (e.g. a society or other partner) holds exclusive rights to this article under a publishing agreement with the author(s) or other rightsholder(s); author self-archiving of the accepted manuscript version of this article is solely governed by the terms of such publishing agreement and applicable law.

Reprints and permissions

About this article

Murad, M.H., Vaa Stelling, B.E., West, C.P. et al. Measuring Documentation Burden in Healthcare. J GEN INTERN MED (2024). https://doi.org/10.1007/s11606-024-08956-8

Download citation

Received : 23 May 2024

Accepted : 17 July 2024

Published : 29 July 2024

DOI : https://doi.org/10.1007/s11606-024-08956-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

documentation burden
systematic review
healthcare professionals
satisfaction
electronic health record
Find a journal
Publish with us
Track your research

A .gov website belongs to an official government organization in the United States.

A lock ( ) or https:// means you've safely connected to the .gov website. Share sensitive information only on official, secure websites.

National Electronic Health Records Survey
Participants
About the Data
Results and Publications

National Center for Health Statistics

NEHRS is the National Electronic Health Records Survey.
NEHRS is a source of data about electronic health record systems and health information exchange.
NEHRS collects data directly from U.S. office-based physicians and their practices.

Image of the National Electronic Health Records Survey identifier

Selected for NEHRS?‎

The National Electronic Health Records Survey (NEHRS) is an annual survey that measures the progress U.S. physicians and their offices have made in adopting electronic health records (EHRs). The survey also monitors health information exchange, including interoperability, and the burden associated with EHRs.

Health information exchange is the sharing of health record information between and with different providers and facilities. Electronic health records systems are interoperable when they can share usable information directly from system to system.

NEHRS began in 2008. It initially was an annual mail supplement to the National Ambulatory Medical Care Survey (NAMCS) . In 2010, NEHRS expanded to measure EHR adoption rates across all 50 U.S. states and the District of Columbia. NEHRS became a standalone survey, conducted separately from NAMCS, in 2012.

The Office of the National Coordinator for Health Information Technology sponsors NEHRS. The National Center for Health Statistics (NCHS) works with RTI International, a non-profit research organization, to conduct NEHRS.

What's collected

NEHRS collects data about electronic health record use from office-based physicians practicing in the United States. NEHRS does not collect data from physicians who are employed by the federal government. NEHRS also does not collect data from anesthesiologists, radiologists, and pathologists.

Currently, NCHS collects NEHRS data through a web survey and paper questionnaire. Previously, NCHS also collected NEHRS data by telephone.

Data and documentation

NCHS publishes NEHRS reports and summary web tables to describe how many physicians have adopted EHRs and how quickly physicians are adopting EHRs. In producing these resources, NEHRS staff prioritizes protecting the privacy of providers and practices.

Findings and reports‎

Public use and restricted data files.

Public use data files (PUFs) are available to download. All potentially identifiable information about providers has been removed from PUFs to ensure the confidentiality of respondents.

To access restricted NEHRS data, investigators must submit requests in the Standard Application Process portal. Restricted data files are available through NCHS's Research Data Center (RDC) for a fee. The RDC provides instructions for preparing and submitting an application to access restricted data.

Find data documentation for these datasets at Questionnaires, Data Sets, and Documentation .

How the data are used

Results from NEHRS provide data to track the adoption and use of EHR systems. Researchers can use NEHRS data to produce state and national estimates of adoption, capabilities, and burden.

NEHRS also monitors physicians' progress toward meeting the policy goals of the Health Information Technology for Economic and Clinical Health (HITECH) Act.

In recent years, survey data have been used to—

Measure the sharing of patient health information
Track participation in the Centers for Medicare and Medicaid Services' Promoting Interoperability Programs
Analyze trends in reports such as Health, United States and Healthy People
Advance federal, public, and international research

What's new from NEHRS

Public use file national weighted estimates: National Electronic Health Records Survey, 2021 (01/2023)

Visit these websites to learn more about current healthcare provider surveys, and the survey that previously collected data about electronic health records.

National Health Care Surveys Program

National Ambulatory Medical Care Survey

NCHS collects, analyzes, and disseminates timely, relevant, and accurate health data and statistics to inform the public and guide program and policy decisions to improve our nation’s health.

Solutions Solutions
Who We Serve Who We Serve
Resources Resources
Company Company

An all-in-one solution that improves clinical effectiveness, patient experience, and financial performance.

A connected, tailorable EHR solution that empowers clinicians.

RCM software and services that boost efficiency and profitability.

Communication and care tools that improve patient relationships.

Technology that helps improve patient outcomes and reduce costs.

Expert consulting and support to help improve performance and ROI.

Easily connect to athenaOne and extend its capabilities.

Tailor your athenahealth experience with industry-leading partners.

Enterprise RCM software that maximizes collections with less work.

Provider Organizations Startup Practices Small Medical Practices Medium to Large Medical Groups Enterprises & Health Systems Federally Qualified Health Centers Member-centric Care Organizations
Medical Specialties Behavioral Health Orthopedics Pediatrics Primary Care Urgent Care Women's Health View All Specialties
Other Organizations Healthcare Payers Labs and Imaging Centers
Partners Marketplace Partners Developers
Thought Leadership Our Insights
Resources Case Studies
About athenahealth Who We Are Leadership athenaGives
Join Us Contact Us Careers
In the News Newsroom Press Releases

Select a topic

Healthcare Technology
Patient Experience
Financial Performance
Practice Management
Clinical Trends
Value-Based Care
Partner Content

Optimizing revenue cycle management for value-based care success

July 30, 2024

A member of a clinical staff smiles, while using a successful value-based care system.

In recent years, the U.S. healthcare industry has been steadily shifting to incorporate more alternative payment models that prioritize and reward care outcomes, reduced costs, and improved patient access. These alternative models—known generally as value-based care—aim to increase the quality of care patients receive, while simultaneously driving down costs.

The rise of value-based care is driven, in part, by the distressing legacy of the more common fee-for-service model—today, the U.S. spends more per capita on healthcare than any other developed country, while achieving the worst health outcomes among developed countries. 1 Charging payers and patients for services rendered, rather than care outcomes, has helped create this troublesome imbalance.

Thankfully, value-based care programs are growing in popularity. McKinsey estimates that the number of people covered under value-based care models will increase by 109% between 2022 and 2027, from roughly 43 million people to roughly 90 million people. 2 This rising trend should help improve care outcomes while driving healthcare costs down, a benefit to both patients and healthcare providers.

Practices interested in incorporating value-based care programs into their revenue models should be aware of certain revenue cycle management (RCM) complexities unique to these alternative models and consider how well suited their HIT tools are to help manage those complexities.

Let’s take a closer look at the RCM and healthcare IT considerations behind this emerging model.

Healthcare organizations that use mix of payment models feel more financially secure

Value-based care is a relatively new model, and many healthcare organizations are still evaluating if, and how, they’ll incorporate value-based programs into their business strategies. Some, like those that accept Medicare alongside insurance payers, may be delivering value-based care without thinking about it as such. Most are choosing to test the viability of these programs in a measured way. Today, there are relatively few practices using value-based care programs exclusively; instead, most are adding value-based programs alongside more traditional fee-for-service models.

This approach is showing positive returns: according to recent research , physicians using a mix of payment models feel that their practices are more financially secure . That’s good news: it means healthcare organizations have an opportunity to branch into value-based care and test its viability and diversify revenue streams, all without assuming too much financial risk in the short term.

Value-based care success hinges on effective use of an integrated healthcare IT platform

Data and connectivity are essential components in managing and monitoring value-based care programs and capturing maximum reimbursements. According to HFMA , “By embracing change and investing in technology that supports automated revenue cycle processes and advanced data analytics, providers will be better able to manage the intersection of VBC and revenue cycle management.” 3

Automation isn’t the only consideration: utilizing a healthcare IT platform that integrates electronic health records and patient engagement capabilities alongside revenue cycle management tools and services is also important due to the unique reporting requirements for value-based care programs. For example, a particular value-based care program may require reporting on the quality and effectiveness of the care provided, the costs associated with the services provided, and specific patient engagement metrics, among others. A consolidated practice management platform should be able to report on all these requirements efficiently and accurately.

Interoperability is also key: a connected healthcare IT platform that can send and receive information with other platforms is crucial in order improve care coordination, a top goal of VBC.

Next, let’s look at the top technology-enabled capabilities healthcare organizations should leverage for revenue cycle success with value-based care programs.

Top 5 revenue cycle management considerations for value-based care success

Here are some of the most important technology-enabled revenue cycle management considerations for healthcare organizations looking to incorporate value-based care programs.

1. Accurate and complete clinical documentation

Proper documentation is crucial for value-based care reimbursement. Providers need to ensure that all relevant patient information, including diagnoses, procedures, and outcomes, are accurately and completely documented within electronic health records to support appropriate medical coding and billing.

While keeping up with clinical documentation can be burdensome , there are AI-enabled documentation tools available today that can help providers increase the accuracy and efficiency of documentation while also removing a large portion of the workload typically managed by physicians and clinical staff.

2. Integrated platforms for electronic health records and medical billing

Providers should utilize a healthcare IT platform that combines electronic health record (EHR) and billing capabilities into a single platform. This integration allows for seamless capture of clinical and financial data, ensuring accurate and complete coding and billing for value-based care services.

Some technology providers also offer revenue cycle management services for processes like billing, which can provide even better operational efficiency and accuracy.

3. Population health and risk management

Managing value-based care programs often involves risk stratification and population health management to identify high-risk patients and provide targeted interventions for patient-centered care .

The right platform, consolidating electronic health records and revenue cycle management capabilities, should support these efforts by facilitating the identification and tracking of high-risk patients and ensuring appropriate billing and reimbursement for the associated services.

4. Accurate and compliant medical coding

Accurate and compliant medical coding is essential for value-based care reimbursement. Providers should take special care to ensure that their coding practices align with the specific requirements of value-based care programs, such as Hierarchical Condition Category (HCC) coding for risk adjustment.

Similar to the closely related medical billing function, some technology providers offer services to take on much of the medical coding work, including providing expertise around evolving compliance issues related to value-based care contracts .

5. Quality reporting and performance monitoring

Organizations that want to leverage value-based care need a healthcare IT platform that offers robust quality reporting and performance monitoring capabilities. These capabilities enable practices to track and report on various quality metrics required for value-based care programs, such as MIPS , ACO, and PCMH. The platform should also provide real-time visibility into performance metrics, allowing practices to identify areas for improvement and optimize reimbursement.

It's also important to remember that payers have unique and evolving requirements for quality and performance, so look for a healthcare IT partner with both broad and in-depth experience with a variety of payers.

Leverage athenaOne ® to optimize revenue cycle management for value-based care

athenaOne ® is a comprehensive, integrated healthcare IT platform with robust capabilities around revenue cycle management , electronic health records , and patient engagement . The athenaOne platform and associated services enable healthcare organizations to thrive while operating under whatever mix of payment models suit their practice best, including value-based care programs. The platform is highly interoperable: athenaOne customers have access to more than 165,000 clinical integrations with labs, imaging centers, pharmacies, patient record sharing networks, and other entities. 4

athenaOne value-based care capabilities

athenaOne is built with the capabilities needed to achieve success with value-based care programs, including:

Innovative tools for clinical documentation to help you efficiently produce accurate and complete documentation
Population health management reports to help you identify high-risk patients and provide targeted interventions
Automation and services for accurate, compliant, and scalable medical coding, resulting in better rates of reimbursement
Revenue cycle analytics and reporting to identify opportunities and optimize program performance
Compliance and regulatory support to provide guidance on evolving regulatory requirements

The future of value-based care in the US

Many healthcare organizations are still in the early stages of incorporating value-based care programs into their businesses and operating models. These programs offer benefits to both providers and patients that may well be worth the effort required to get them established and successful.

A knowledgeable and experienced technology and industry partner is a must-have for organizations wanting to succeed with value-based care models. Consider a platform like athenaOne to provide the unique capabilities and expertise needed to find success with value-based care.

Read the articles below to learn more about how athenaOne helps healthcare organizations find great financial success.

The Commonwealth Fund, U.S. Health Care from a Global Perspective, 2022: Accelerating Spending, Worsening Outcomes, Jan. 2023; https://www.commonwealthfund.org/publications/issue-briefs/2023/jan/us-health-care-global-perspective-2022
McKinsey & Company, What to expect in US healthcare in 2024 and beyond, Jan. 2024; https://www.mckinsey.com/industries/healthcare/our-insights/what-to-expect-in-us-healthcare-in-2024-and-beyond
HFMA, Bridging the gap: Integrating value-based care into revenue cycle management, May 2024; https://www.hfma.org/revenue-cycle/bridging-the-gap-integrating-value-based-care-into-revenue-cycle-management/
Based on athenahealth data as of Jun. 2024

The crucial role of interoperability in driving value-based care transformation

Transforming healthcare delivery and enhancing the patient experience

Why a shift to value-based care needs to include a shift in organizational values and technology

Transitioning to a “data-sharing ecosystem” requires smarter, not harder work across the continuum

Getting started with value-based care

Laying the groundwork for VBC success

Open access
Published: 24 July 2024

Strategies used to detect and mitigate system-related errors over time: A qualitative study in an Australian health district

Madaline Kinlay 1 ,
Wu Yi Zheng 2 ,
Rosemary Burke 3 ,
Ilona Juraskova 4 ,
Lai Mun Ho 3 ,
Hannah Turton 3 ,
Jason Trinh 3 &
Melissa T. Baysari 1

BMC Health Services Research volume 24 , Article number: 839 ( 2024 ) Cite this article

131 Accesses

Metrics details

Electronic medical record (EMR) systems provide timely access to clinical information and have been shown to improve medication safety. However, EMRs can also create opportunities for error, including system-related errors or errors that were unlikely or not possible with the use of paper medication charts. This study aimed to determine the detection and mitigation strategies adopted by a health district in Australia to target system-related errors and to explore stakeholder views on strategies needed to curb future system-related errors from emerging.

A qualitative descriptive study design was used comprising semi-structured interviews. Data were collected from three hospitals within a health district in Sydney, Australia, between September 2020 and May 2021. Interviews were conducted with EMR users and other key stakeholders (e.g. clinical informatics team members). Participants were asked to reflect on how system-related errors changed over time, and to describe approaches taken by their organisation to detect and mitigate these errors. Thematic analysis was conducted iteratively using a general inductive approach, where codes were assigned as themes emerged from the data.

Interviews were conducted with 25 stakeholders. Participants reported that most system-related errors were detected by front-line clinicians. Following error detection, clinicians either reported system-related errors directly to the clinical informatics team or submitted reports to the incident information management system. System-related errors were also reported to be detected via reports run within the EMR, or during organisational processes such as incident investigations or system enhancement projects. EMR redesign was the main approach described by participants for mitigating system-related errors, however other strategies, like regular user education and minimising the use of hybrid systems, were also reported.

Conclusions

Initial detection of system-related errors relies heavily on front-line clinicians, however other organisational strategies that are proactive and layered can improve the systemic detection, investigation, and management of errors. Together with EMR design changes, complementary error mitigation strategies, including targeted staff education, can support safe EMR use and development.

Peer Review reports

An electronic medical record (EMR) provides access to longitudinal patient data and clinical information in a timely and convenient manner, [ 1 ] while allowing clinicians to prescribe, review and administer medications on a single digital platform, often with the assistance of clinical decision support. Although the use of EMR systems results in fewer medication errors, [ 2 ] they can also create new system-related errors; errors that were highly unlikely or not possible with the use of paper medication charts (e.g. a doctor selecting the wrong dose from a drop-down menu). Previous research has identified the types and factors contributing to system-related errors, [ 3 , 4 , 5 ] as well as their prevalence [ 6 ], but the detection of these errors can be challenging in both a clinical and research context. Research investigating the types and rates of system-related errors at two hospitals revealed that of the 493 system-related errors that were discovered, only 13% were detected by hospital staff prior to the study [ 4 ]. Further, the rate of system-related errors varies between studies, ranging from 1.2 to 34.8% of all errors [ 7 ] with this rate dependent on the detection method employed [ 6 ].

To our knowledge, there has been no research that has specifically examined how system-related errors are detected by the organisations impacted by them. While the first step in reducing system-related errors is error detection, another important component of error management is learning from previous errors and improving on processes and systems [ 8 , 9 ]. Our previous work has described EMR system enhancements made to target system-related errors, [ 10 ] however research on how system-related errors are rectified or managed once error detection has occurred is in its infancy. Therefore, the current study asked the following research questions: (1) what are the detection and mitigation strategies adopted by a health district to target system-related errors? and (2) what are stakeholder views on strategies needed to curb future system-related errors from emerging?

This study formed part of a larger qualitative research project examining stakeholder understanding and experiences of system-related errors [ 11 ]. The research was conducted at three hospitals in Sydney, Australia, that used the same commercial EMR system (Cerner Millennium ® ). The system had been in place for different durations at each site (14 years, 4 years and 2 years) and roll-out strategies varied in length and approach.

Recruitment and data collection

Participants included any hospital employee who dealt with the EMR directly or indirectly, including end-users (i.e., doctors, nurses, pharmacists), clinical informatics team members (e.g. system trainers), members of relevant committees (e.g. medicine safety committee) and department directors. A clinical informatics pharmacist at each site identified individuals who they believed were knowledgeable about the EMR or had relevant roles, and the research team invited these potential participants to take part via email. This technique was combined with snowball sampling, where participants were asked to propose additional staff members for inclusion. In total, 45 email invitations were distributed.

Semi-structured interviews were conducted either by video conference or in-person at the hospital. Interviews were in two parts. In Part 1, reported elsewhere, [ 11 ] participants were asked to describe common system-related errors and factors contributing to them. In Part 2, reported here, participants were asked to reflect on how system-related errors changed over time, and to describe detection and mitigation strategies their organisation had adopted. Separate interview guides were created for end-users and for individuals who supported EMR use (see the Additional file 1 and 2 ). Interview guides were developed by a multi-disciplinary team, including clinicians, and those with extensive knowledge of the EMR. Participants had the option to contact the researcher with any additional questions or comments following the interview. The lead investigator (MK), a student completing interviews as part of her doctoral degree, obtained written consent from participants and conducted all interviews. The interviewer was not known to participants before interviews commenced. Interviews were audio-recorded, transcribed verbatim and de-identified. Data collection ceased upon reaching thematic saturation across the overall dataset [ 12 ].

Data analysis

Interviews were thematically analysed using a general inductive approach, where codes were assigned as themes emerged from the data [ 13 ]. Three researchers (MK, MB and WYZ) independently coded data from individual interviews into themes and met at regular intervals to discuss categories and resolve discrepancies. Data from the two different interview groups (end-users and individuals who supported EMR use) were analysed together, but general participant identifiers (users/EMR team) were maintained to allow any differences in the two groups to be identified. After agreeing upon a coding framework, researchers coded the remaining interviews and undertook a final review to discuss ambiguities, inconsistencies and confirm major themes and subthemes. Themes were checked by multi-disciplinary members of the research team, including clinicians and EMR experts, who confirmed face validity.

This project was approved by the district’s Human Research Ethics Committee (HREC reference number: 2020/ETH00198). All participants provided informed written consent to participate, including to be audio-recorded.

Participant demographics

Interviews were conducted with 25 stakeholders, comprising 15 clinicians (end users of the EMR) and 10 staff from the EMR implementation and support team. Participant demographics appear in Table 1 (see [1] for more detailed demographics). Interviews occurred between September 2020 and May 2021 and took an average of 35 min, ranging from 9 to 55 min. No differences emerged in the results from end-users and individuals who supported EMR use, and therefore themes for these groups are presented together. Note that CI preceding a participant code (e.g. CIDR vs. DR) indicates the quotation relates to a clinical informatics (EMR) expert, not end-user.

An overview of the themes and subthemes, along with corresponding codes and quotations from interviews, is presented in Table 2 .

Detection of system-related errors

Participants described several methods by which system-related errors were detected by the hospital sites (see Fig. 1 ).

Flowchart depicting the process by which system-related errors are detected and mitigated by hospital staff, based on the themes extracted from interviews with key stakeholders. SRE = System-related error, IIMS = Incident information management system, EMR = Electronic medication record

Detection of system-related errors by clinicians

Detection by front-line clinicians was the primary method of system-related error detection reported by participants. Specifically, participants explained that pharmacists identified system-related errors during medication review or reconciliation, and nurses detected system-related errors when completing routine checks prior to administering medications. ‘All orders get verified by a pharmacist , so that pharmacist might intervene if they recognise that an error has occurred by reviewing the order. And nursing staff will also check orders and before administering medications , and they may recognise one of these system errors.’ (CIDR2).

However, some participants noted that detecting system-related errors was often difficult for nurses as it required them to discern the intended prescription from the recorded prescription.

Organisational processes in place to detect system-related errors

One of the most frequent organisational strategies highlighted by participants to complement clinicians’ detection of system-related errors was clinicians reporting potential system-related errors to the clinical informatics team, who then ascertained whether the error was in fact system-related. Clinical informatics team members noted that system-related errors were difficult to detect without clinician input, and investigations into system-related errors were often dependent on clinicians bringing potential cases to their attention. ‘Frankly speaking , you don’t have anything that can alert you […] It requires a lot of clinicians reporting these issues back to me , for me to be able to know these things are happening on the ward.’ (CIPH2).

Participants also explained that system-related errors could be detected via the Incident Information Management System (IIMS); the organisation’s voluntary reporting system for clinical, work health and safety, and security events. ‘So , at a high level they can be reported through our incident monitoring system.’ (PH3).

However, interviewees also noted that this detection strategy relied upon clinicians identifying and proactively self-reporting system-related errors. ‘In terms of how we found out about them , incident reporting is something I think we are hoping to be more and more proactive about.’ (CIDR2).

Another method reportedly used by clinical informatics staff to detect system-related errors was the generation of specific reports within the EMR, such as a monthly report of pharmacy interventions to identify reports that cited the involvement of an EMR system issue. These reports displayed trends in error types and were viewed as useful for determining whether specific system-related errors occurred regularly and what factors could be contributing to error occurrence. ‘I will run reports on the EMR to see whether there is a consistent pattern that is happening across the facility. […] Identifying patterns , identifying whether it’s a prescribing issue or whether it’s a nursing workflow issues , or whether it is actually an EMR issue.’ (CIPH2).

Some participants reported that errors were detected by a clinician or project team during inquiries into adverse patient events or during EMR system enhancements when intensive testing sometimes uncovered system-related errors. For instance, when creating a new cancer module in the EMR, project team members discovered that chemotherapy prescriptions did not display all the necessary order components to the user.

Management and mitigation of system-related errors over time

Participants described various approaches to manage and reduce system-related errors, including EMR design changes and organisational strategies (see Fig. 1 ).

EMR design changes to mitigate system-related errors

Participants explained that after clinicians escalated concerns to the clinical informatics team and a system-related error was confirmed, the EMR system design was modified, if this was deemed to be essential and possible. Modification of the EMR system design could occur when the clinical informatics team recognised a patient safety or workflow benefit from the change and the system was able to be altered (i.e. no system configuration limitations). ‘Where we have found people making mistakes , we’ve been able to implement some actions to circumvent them.’ (PH3).

Looking forward, participants stated that over time they would expect fewer system-related errors, attributing this reduction to the fact that errors had been identified and rectified.

‘Because , one , we are better aware of how to design the system to reduce the likelihood of some of these errors.’ (CIDR2).

Participants provided specific examples of system redesign to target system-related errors (see Table 3 ). A frequently reported category of system redesign was the addition of alerts for specific processes and medications, such as high-risk medications. Improved visibility and clarity of information in the EMR was another strategy reported by participants to mitigate system-related errors. Participants also described a more intuitive and consistent system. References were made to incorporating human factors design principles into the EMR and ensuring the system aligns with workflow. For example, one doctor suggested that the system become more user-friendly when adjusting doses and times, while a pharmacist proposed that the system provide more clarity of the job role required so that clinicians know which tasks to attend to on the system (i.e., checking off a box is only for nurses).

Although EMR design changes were said to decrease system-related errors, participants highlighted that it was possible for these system functionality changes to result in new types of errors over time.

‘As we continue to change it and change the workflows , we will get different errors’ (CIPH3).

Participants also noted that some current system-related errors would remain, citing constraints in the system build, preventing design changes that could resolve errors and therefore requiring other strategies to manage these system-related errors. ‘There’s always going to be the [errors] that we can’t resolve , in that we can’t change the way the system is built’ (CINU1).

Organisational strategies to mitigate system-related errors

The most frequently reported organisational strategy employed to minimise system-related errors was education, either to an individual user, a group of clinicians, or hospital-wide. Providing individual feedback or training was said to occur in response to a specific incident, usually in cases where unfamiliarity with the EMR was believed to have contributed to the error. When system-related errors were more widespread, occurring across a particular cohort, ward or hospital, participants explained that education was delivered more broadly.

‘Once [nurses] have flagged the problem to the helpdesk , the supervisor or whoever’s in charge , […] they will try to find the problem and then give us advice on what to do next.’ (NU7).

Participants referred to examples where system functionality or configuration was unable to be changed after identification of a system-related error, and so staff education and training focused on safely bypassing system limitations or constraints so that work could continue.

Although education was viewed to be an effective strategy for reducing system-related errors, some participants reported the challenge of system-related errors persisting due to staff turnover and the employment of new clinicians.

‘Because its constantly new staff coming in , they then don’t know the messages that have been sent out last year… They tend to make the same mistake again at some point or another.’ (PH3).

However, participants explained that with more widespread EMR use in the future, users would become more familiar and confident with the system, and fewer system-related errors would result. Despite this, new errors were reported to also arise when users take more shortcuts or workarounds as they become more familiar with the system. For example, a clinical informatics pharmacist described clinicians exporting information from previous admissions into the patient’s current medication chart without consulting the patient.

‘You’re seeing different types of errors where prescribers are very comfortable now with using information from previous admissions but forgetting that they also need to talk to patient and get updated information … When you’re familiar with the system , you kind of take certain shortcuts.’ (CIPH2).

Some clinical informatics team members noted that raising issues with the chief executive or chief information officer was another organisational strategy used to mitigate system-related errors, particularly when system-related errors were likely to be occurring at other hospital sites and system changes at a broader level were necessary.

Finally, minimising the use of hybrid systems (i.e., paper and electronic systems, dual electronic systems), was mentioned by some participants as another strategy to reduce system-related errors. However, participants also noted that as users become less familiar with paper-based medication charts, new errors may arise when clinicians are required to use paper charts during EMR downtime. ‘Some of the new , younger generation , they find it difficult to use as a paper form , when a downtime happens.’ (NU5).

Interviews uncovered detection and mitigation strategies implemented by a health district to target system-related errors, including existing and potential methods required to prevent future errors from occurring. Participants explained that initial detection of system-related errors was highly dependent on clinicians identifying errors. Once error detection occurred, participants highlighted that clinicians either directly reported these errors to the clinical informatics team or submitted an IIMS report for escalation. EMR redesign was described as the main approach for error reduction, however other organisational strategies, like regular user education and minimising the use of hybrid systems were also reported.

It is noteworthy that many of the reported approaches for system-related error detection put the onus on clinicians to identify and subsequently report errors. Although verbal and incident reporting by clinicians are conventional methods of error detection, irrespective of EMR involvement, [ 14 ] system-related errors are challenging for clinicians to recognise and may go unnoticed unless they lead to an error (i.e. medication error) or adverse patient event [ 15 ]. Clinicians’ reliance on the EMR system for care delivery is growing due to an increase in automation and system guidance, [ 16 , 17 ] influencing their ability to recognise a system-related error. Additionally, the complexity of the EMR system, [ 18 ] unfamiliarity with the EMR, and distraction caused by competing priorities [ 19 ] can all hinder detection of system-related errors.

In addition to difficulties in error detection, challenges associated with reporting of system-related errors are also likely. Clinicians may not report system-related errors if they fear individual blame or punishment, [ 20 ] or are unsupported in their efforts to improve patient safety [ 21 , 22 ]. Factors driving under-reporting of incidents are likely to also be at play in reporting of system-related errors to clinical informatics teams, including a perception of low value of reporting if reports are not used to identify error patterns and prevent future incidents [ 23 ].

Implementing a systematic feedback process, where clinicians are informed of changes to EMR systems or processes that result from reporting, would increase the perceived value, confidence and motivation to report system-related errors. The challenges associated with clinician detection and reporting of system-related errors highlight the importance of utilising complementary strategies to detect these errors. We found that system enhancement projects, as well as EMR reports, were other proactive methods of detection, though reported less often. Combining reactive front-line detection with proactive clinical surveillance and monitoring is likely to ensure system-related errors are promptly identified and investigated [ 15 ].

EMR design changes were the most common approach suggested by participants to reduce system related errors, with many believing EMR redesign would result in fewer system-related errors. However, an unintended consequence of modifying system configuration was the generation of different system-related errors, and several participants stated that certain errors would persist as constraints in the EMR system build limited design alterations. While incremental design changes are necessary for maintenance and development of the EMR system, [ 24 ] the possibility of design changes resulting in the emergence of different system-related errors reinforces the importance of testing environments that simulate real-life EMR situations prior to the go-live of any modifications [ 25 ].

Education, either one-on-one, to a particular cohort, or hospital wide, was another mitigation strategy we identified to reduce system-related errors. Despite the reported benefits of education, participants noted that staff turnover and the employment of new staff could contribute to an increase in errors. By regularly updating training material and providing periodic, targeted education (e.g. as part of onboarding new staff), this would ensure new staff are aware of the most up-to-date material and minimise the risk of medication errors [ 26 ]. Participants indicated that as more staff become proficient in using EMRs, there are likely be fewer system-related errors, but potentially larger numbers of workarounds, with previous research supporting this latter suggestion [ 27 , 28 ]. Although workarounds can compromise patient safety and quality of care, [ 29 ] comprehensive training about EMR risks and ongoing support for EMR users, can reduce clinicians’ use of workarounds [ 3 ].

Strengths and limitations

Qualitative research methods allowed the authors to conduct, for the first time, an in-depth investigation of detection and mitigation strategies, however this research did not measure how often system-related errors were detected or the effectiveness of improvement methods. Additionally, interviews were conducted with clinicians and key stakeholders in one Local Health District, across only three hospitals, and therefore results may not be generalisable to other settings and the detection and mitigation approaches identified may not be exhaustive.

To our knowledge, this is the first study to examine how system-related errors are detected by organisations and adds to the growing body of evidence exploring error mitigation. Front-line clinicians play a critical role in system-related error detection, however other organisational approaches, such as system enhancement projects, improve systemic error detection, investigation, and management. Organisations must take a proactive approach to error identification and ensure detection processes are layered. Although EMR design changes were highlighted as important for error reduction, changes were not always possible. Complementary strategies, such as targeted staff education, can support safe use of the EMR and its ongoing development.

Data availability

The qualitative data collected from participants for this study are not available.

Abbreviations

Electronic medical record

Medication Administration Record

Incident Information Management System

Clinical Informatics

Fennelly O, Cunningham C, Grogan L, Cronin H, O’Shea C, Roche M, et al. Successfully implementing a national electronic health record: a rapid umbrella review. Int J Med Inf. 2020;144:104281.

Article Google Scholar

Ammenwerth E, Schnell-Inderst P, Machan C, Siebert U. The Effect of Electronic prescribing on medication errors and adverse drug events: a systematic review. J Am Med Inf Assoc. 2008;15(5):585–600.

Brown CL, Mulcaster HL, Triffitt KL, Sittig DF, Ash JS, Reygate K, et al. A systematic review of the types and causes of prescribing errors generated from using computerized provider order entry systems in primary and secondary care. J Am Med Inf Assoc. 2017;24(2):432–40.

Westbrook J, Baysari M, Li L, Burke R, Richardson K, Day R. The safety of electronic prescribing: manifestations, mechanisms, and rates of system-related errors associated with two commercial systems in hospitals. J Am Med Inf Assoc. 2013;20(6):1159–67.

Tolley CL, Forde NE, Coffey KL, Sittig DF, Ash JS, Husband AK, et al. Factors contributing to medication errors made when using computerized order entry in pediatrics: a systematic review. J Am Med Inf Assoc. 2018;25(5):575–84.

Korb-Savoldelli V, Boussadi A, Durieux P, Sabatier B. Prevalence of computerized physician order entry systems–related medication prescription errors: a systematic review. Int J Med Inf. 2018;111:112–22.

Gates PJ, Hardie RA, Raban MZ, Li L, Westbrook JI. How effective are electronic medication systems in reducing medication error rates and associated harm among hospital inpatients? A systematic review and meta-analysis. J Am Med Inf Assoc. 2021;28(1):167–76.

Vincent C. Patient safety. Wiley; 2011.

Montesi G, Lechi A. Prevention of medication errors: detection and audit. Br J Clin Pharmacol. 2009;67(6):651–5.

Article PubMed PubMed Central Google Scholar

Kinlay M, Ho LMR, Zheng WY, Burke R, Juraskova I, Moles R, et al. Electronic Medication Management Systems: Analysis of Enhancements to reduce errors and improve Workflow. Appl Clin Inf. 2021;12(5):1049–60.

Kinlay M, Zheng WY, Burke R, Juraskova I, Ho LMR, Turton H, et al. Stakeholder perspectives of system-related errors: types, contributing factors, and consequences. Int J Med Inf. 2022;165:104821.

Glaser BG, Strauss AL. Discovery of grounded theory: strategies for qualitative research. Routledge; 2017.

Bradley EH, Curry LA, Devers KJ. Qualitative data analysis for health services research: developing taxonomy, themes, and theory. Health Serv Res. 2007;42(4):1758–72.

Wolf ZR, Hughes RG. Error reporting and disclosure. Rockville: Agency for Healthcare Research and Quality (US); 2008.

Google Scholar

Thomas EJ, Petersen LA. Measuring errors and adverse events in health care. J Gen Intern Med. 2003;18(1):61–7.

Parasuraman R, Manzey DH. Complacency and bias in human use of automation: an attentional integration. Hum Factors. 2010;52(3):381–410.

Article PubMed Google Scholar

Sittig DF, Wright A, Ash J, Singh H. New Unintended Adverse Consequences of Electronic Health Records. Yearb Med Inf. 2016(1):7–12.

Reason J. The contribution of latent human failures to the breakdown of complex systems. Philos Trans R Soc Lond B Biol Sci. 1990;327(1241):475–84.

Article CAS PubMed Google Scholar

Moray N. The role of attention in the detection of errors and the diagnosis of failures in Man-Machine systems. In: Rasmussen J, Rouse WB, editors. Human detection and diagnosis of system failures. Boston, MA: Springer US; 1981. pp. 185–98.

Chapter Google Scholar

Lawton R, Parker D. Barriers to incident reporting in a healthcare system. Qual Saf Health Care. 2002;11(1):15–8.

Article CAS PubMed PubMed Central Google Scholar

Stump LS. Re-engineering the medication error-reporting process: removing the blame and improving the system. Am J Health Syst Pharm. 2000;57(Suppl 4):S10–7.

Meaney ME. Error reduction, patient safety and institutional ethics committees. J Law Med Ethics. 2004;32(2):358–64.

Pfeiffer Y, Manser T, Wehner T. Conceptualising barriers to incident reporting: a psychological framework. Qual Saf Health Care. 2010;19(6):e60–e.

CAS PubMed Google Scholar

Bennett KH, Rajlich VT, Wilde N. Software Evolution and the staged model of the Software lifecycle. In: Zelkowitz MV, editor. Advances in computers. Volume 56. Elsevier; 2002. pp. 1–54.

Wright A, Aaron S, Sittig DF. Testing electronic health records in the production environment: an essential step in the journey to a safe and effective health care system. J Am Med Inf Assoc. 2016;24(1):188–92.

Leape LL. Error in Medicine. JAMA. 1994;272(23):1851–7.

Zheng K, Abraham J, Novak L, Reynolds T, Gettinger A. A survey of the literature on unintended consequences associated with health information technology: 2014–2015. Yearb Med Inf. 2016;25(01):13–29.

Koppel R, Wetterneck T, Telles JL, Karsh B-T. Workarounds to Barcode Medication Administration Systems: their occurrences, causes, and threats to Patient Safety. J Am Med Inf Assoc. 2008;15(4):408–23.

Cresswell KM, Mozaffar H, Lee L, Williams R, Sheikh A. Workarounds to hospital electronic prescribing systems: a qualitative study in English hospitals. BMJ Qual Saf. 2016;26(7):519–21.

Download references

Acknowledgements

Not applicable.

This work was supported by an Australian Government Research Training Program (RTP) Scholarship to MK.

Author information

Authors and affiliations.

Biomedical Informatics and Digital Health, School of Medical Sciences, Faculty of Medicine and Health, The University of Sydney, Sydney, NSW, 2006, Australia

Madaline Kinlay & Melissa T. Baysari

Black Dog Institute, Sydney, Australia

Wu Yi Zheng

Pharmacy Services, Sydney Local Health District, Sydney, Australia

Rosemary Burke, Lai Mun Ho, Hannah Turton & Jason Trinh

School of Psychology, Faculty of Science, The University of Sydney, Sydney, Australia

Ilona Juraskova

You can also search for this author in PubMed Google Scholar

Contributions

MK, MB and WYS designed the study. RB, LMH, HT and JT assisted in the recruitment of participants. MK analysed the data, with assistance from MB and WYZ. All authors assisted in interpreting results and writing the manuscript. All authors read and approved the final manuscript.

Corresponding author

Correspondence to Melissa T. Baysari .

Ethics declarations

Ethics approval and consent to participate.

This project was approved by the Sydney Local Health District Human Research Ethics Committee (HREC reference number: 2020/ETH00198). All participants provided informed written consent to participate.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Kinlay, M., Zheng, W.Y., Burke, R. et al. Strategies used to detect and mitigate system-related errors over time: A qualitative study in an Australian health district. BMC Health Serv Res 24 , 839 (2024). https://doi.org/10.1186/s12913-024-11309-0

Download citation

Received : 28 February 2024

Accepted : 12 July 2024

Published : 24 July 2024

DOI : https://doi.org/10.1186/s12913-024-11309-0

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

System-related errors
Patient safety
Error detection
Error prevention

BMC Health Services Research

ISSN: 1472-6963

General enquiries: [email protected]

Attitudes of Mental Health Service Users Toward Storage and Use of Electronic Health Records

Information & authors, metrics & citations, view options, conclusions:, benefits and concerns regarding ehr use, ehrs and service users’ involvement, service users’ voice on ehr storage and sharing, participants, questionnaire, procedure and data collection, analysis and interpretation, demographic characteristics.

Characteristic	N	%
Age in years
0–15	2	1
16–24	4	2
25–34	43	17
35–44	86	34
45–54	83	33
55–64	25	10
65–74	10	4
Gender
Women	179	71
Men	69	27
Other	5	2
Employment
Employed	160	63
Unemployed	93	37
Education level
University, ≥4 years	99	39
University, <4 years	68	27
Nonuniversity	86	34
Ancestry, Norwegian
Yes	241	95
Other	12	5

Awareness of Health Authorities Storing and Sharing Health Records

Participants’ attitudes toward data storage.

Participants’ Attitudes Toward Data Sharing

Participants’ opinions about reasons for sharing ehrs, comments of interest.

“Historically, there has been little information for service users about what, how, and which health information is recorded, including access. Lack of info [and] access can create uncertainty for some.” “ What is stored, who has access, and how will they use it?”

“I have nothing against health authorities sharing my health information, as long as it’s anonymized.” “I do not trust that they are able to anonymize efficiently.”

“I am not certain what my information will be used for. . . . [I am concerned that, first], sensitive information will be used in relation to employment. [Second], misunderstandings arise among health care professionals due to previous illness[es] that are no longer relevant. [Third], sensitive information . . . [is] used against me in insurance cases or similar legal matters.” “Mental health can be stigmatized and misunderstood.” “Old, outdated records are used in current cases, which can have largely negative consequences.”

Conclusions

Information, published in.

Child psychiatry
Adolescents
Electronic health record
General psychiatry
Mental health systems
Service users

Affiliations

Funding information, export citations.

If you have the appropriate software installed, you can download article citation data to the citation manager of your choice. Simply select your manager software from the list below and click Download. For more information or tips please see 'Downloading to a citation manager' in the Help menu .

Format

Citation style
Style

To download the citation to this article, select your reference manager software.

There are no citations for this item

View options

Already a subscriber? Access your subscription through your login credentials or your institution for full access to this article.

Purchase Options

Purchase this article to access the full text.

PPV Articles - Psychiatric Services

Not a subscriber?

Subscribe Now / Learn More

PsychiatryOnline subscription options offer access to the DSM-5-TR ® library, books, journals, CME, and patient resources. This all-in-one virtual library provides psychiatrists and mental health professionals with key resources for diagnosis, treatment, research, and professional development.

Need more help? PsychiatryOnline Customer Service may be reached by emailing [email protected] or by calling 800-368-5777 (in the U.S.) or 703-907-7322 (outside the U.S.).

Share article link

Copying failed.

Next article, request username.

Can't sign in? Forgot your username? Enter your email address below and we will send you your username

If the address matches an existing account you will receive an email with instructions to retrieve your username

Create a new account

Change password, password changed successfully.

Your password has been changed

Reset password

Can't sign in? Forgot your password?

Enter your email address below and we will send you the reset instructions

If the address matches an existing account you will receive an email with instructions to reset your password.

Your Phone has been verified

As described within the American Psychiatric Association (APA)'s Privacy Policy and Terms of Use , this website utilizes cookies, including for the purpose of offering an optimal online experience and services tailored to your preferences. Please read the entire Privacy Policy and Terms of Use. By closing this message, browsing this website, continuing the navigation, or otherwise continuing to use the APA's websites, you confirm that you understand and accept the terms of the Privacy Policy and Terms of Use, including the utilization of cookies.

Advanced search

Advanced Search

Primary care health professionals’ approach to clinical coding: a qualitative interview study

Find this author on Google Scholar
Find this author on PubMed
Search for this author on this site
ORCID record for Aled Huw Davies
For correspondence: [email protected]
ORCID record for Haroon Ahmed

Background: Clinical coding allows for structured and standardised recording of data in patients electronic healthcare records. How clinical and non-clinical staff in general practice approach clinical coding is poorly understood. Aim: To explore primary care staff’s experiences and views on clinical coding. Design and setting: Qualitative, semi-structured interview study with primary care staff across Wales in 2023. Method: All general practices within Wales were invited to participate via NHS Health Boards. Semi-structured interview questions guided the interview. Audio-recorded data were transcribed and analysed using reflexive thematic analysis. Results: 19 participants from general practices across Wales were interviewed. Six themes were identified: ‘the daily task of coding’, ‘making coding easier’, ’coding challenges’, ‘what and when to code?’, ‘motivation to code’ and ‘coding through COVID’. Conclusion: This study demonstrates the complexity of clinical coding in primary care. Clinical and non-clinical staff spoke of systems that lacked intuitiveness, and the challenges of multi-morbidity and time pressures when coding in clinical situations. These challenges are likely to be exacerbated in socio-economically deprived areas, leading to under-reporting of disease in these areas. Challenges of clinical coding may lead to implications for data quality, particularly the validity of research findings generated from studies reliant on clinical coding from primary care. There are also consequences for patient care. Participants cared about coding quality and wanted a better way of using coding. There is a need to explore technological and non-technological solutions, such as artificial intelligence, training and education, to unburden people using clinical coding in primary care.

Received January 17, 2024.
Accepted May 13, 2024.
Copyright © 2024, The Authors

This article is Open Access: CC BY license ( https://creativecommons.org/licenses/by/4.0/ )

Online First

Thank you for recommending British Journal of General Practice.

NOTE: We only request your email address so that the person to whom you are recommending the page knows that you wanted them to see it, and that it is not junk mail. We do not capture any email address.

Citation Manager Formats

EndNote (tagged)
EndNote 8 (xml)
RefWorks Tagged
Ref Manager

Tweet Widget
Facebook Like
Google Plus One

Jump to section

Electronic health records to facilitate clinical research

Martin r. cowie.

1 National Heart and Lung Institute, Imperial College London, Royal Brompton Hospital, Sydney Street, London, SW3 6HP UK

Juuso I. Blomster

2 Astra Zeneca R&D, Molndal, Sweden

3 University of Turku, Turku, Finland

Lesley H. Curtis

4 Duke Clinical Research Institute, Durham, NC USA

Sylvie Duclaux

5 Servier, Paris, France

6 Robertson Centre for Biostatistics, University of Glasgow, Glasgow, UK

Fleur Fritz

7 University of Münster, Münster, Germany

Samantha Goldman

8 Daiichi-Sankyo, London, UK

Salim Janmohamed

9 GlaxoSmithKline, Stockley Park, UK

Jörg Kreuzer

10 Boehringer-Ingelheim, Pharma GmbH & Co KG, Ingelheim, Germany

Mark Leenay

11 Optum International, London, UK

Alexander Michel

12 Bayer Pharma, Berlin, Germany

13 Pfizer Ltd., Surrey, UK

Jill P. Pell

14 Institute of Health and Wellbeing, University of Glasgow, Glasgow, UK

Mary Ross Southworth

15 Food and Drug Administration, Silver Spring, MD USA

Wendy Gattis Stough

16 Campbell University College of Pharmacy and Health Sciences, Campbell, NC USA

Martin Thoenes

17 Edwards LifeSciences, Nyon, Switzerland

Faiez Zannad

18 INSERM, Centre d’Investigation Clinique 9501 and Unité 961, Centre Hospitalier Universitaire, Nancy, France

19 Department of Cardiology, Nancy University, Université de Lorraine, Nancy, France

Andrew Zalewski

20 Glaxo Smith Kline, King of Prussia, Pennsylvania, USA

Electronic health records (EHRs) provide opportunities to enhance patient care, embed performance measures in clinical practice, and facilitate clinical research. Concerns have been raised about the increasing recruitment challenges in trials, burdensome and obtrusive data collection, and uncertain generalizability of the results. Leveraging electronic health records to counterbalance these trends is an area of intense interest. The initial applications of electronic health records, as the primary data source is envisioned for observational studies, embedded pragmatic or post-marketing registry-based randomized studies, or comparative effectiveness studies. Advancing this approach to randomized clinical trials, electronic health records may potentially be used to assess study feasibility, to facilitate patient recruitment, and streamline data collection at baseline and follow-up. Ensuring data security and privacy, overcoming the challenges associated with linking diverse systems and maintaining infrastructure for repeat use of high quality data, are some of the challenges associated with using electronic health records in clinical research. Collaboration between academia, industry, regulatory bodies, policy makers, patients, and electronic health record vendors is critical for the greater use of electronic health records in clinical research. This manuscript identifies the key steps required to advance the role of electronic health records in cardiovascular clinical research.

Introduction

Electronic health records (EHRs) provide opportunities to enhance patient care, to embed performance measures in clinical practice, and to improve the identification and recruitment of eligible patients and healthcare providers in clinical research. On a macroeconomic scale, EHRs (by enabling pragmatic clinical trials) may assist in the assessment of whether new treatments or innovation in healthcare delivery result in improved outcomes or healthcare savings.

Concerns have been raised about the current state of cardiovascular clinical research: the increasing recruitment challenges; burdensome data collection; and uncertain generalizability to clinical practice [ 1 ]. These factors add to the increasing costs of clinical research [ 2 ] and are thought to contribute to declining investment in the field [ 1 ].

The Cardiovascular Round Table (CRT) of the European Society of Cardiology (ESC) convened a two-day workshop among international experts in cardiovascular clinical research and health informatics to explore how EHRs could advance cardiovascular clinical research. This paper summarizes the key insights and discussions from the workshop, acknowledges the barriers to EHR implementation in clinical research, and identifies practical solutions for engaging stakeholders (i.e., academia, industry, regulatory bodies, policy makers, patients, and EHR vendors) in the implementation of EHRs in clinical research.

Overview of electronic health records

Broadly defined, EHRs represent longitudinal data (in electronic format) that are collected during routine delivery of health care [ 3 ]. EHRs generally contain demographic, vital statistics, administrative, claims (medical and pharmacy), clinical, and patient-centered (e.g., originating from health-related quality-of-life instruments, home-monitoring devices, and frailty or caregiver assessments) data. The scope of an EHR varies widely across the world. Systems originating primarily as billing systems were not designed to support clinical work flow. Moving forward, EHR should be designed to optimize diagnosis and clinical care, which will enhance their relevance for clinical research. The EHR may reflect single components of care (e.g., primary care, emergency department, and intensive care unit) or data from an integrated hospital-wide or inter-hospital linked system [ 4 ]. EHRs may also change over time, reflecting evolving technology capabilities or external influences (e.g., changes in type of data collected related to coding or reimbursement practices).

EHRs emerged largely as a means to improve healthcare quality [ 5 – 7 ] and to capture billing data. EHRs may potentially be used to assess study feasibility, facilitate patient recruitment, streamline data collection, or conduct entirely EHR-based observational, embedded pragmatic, or post-marketing randomized registry studies, or comparative effectiveness studies. The various applications of EHRs for observational studies, safety surveillance, clinical research, and regulatory purposes are shown in Table 1 [ 3 , 8 – 10 ].

Table 1

Electronic health records in research

Type	Example	Status
Observational studies	Health utilization Drug utilization Epidemiology (incidence/prevalence) Natural history Risk factors	Widely used and accepted
Safety surveillance	Traditional post-marketing safety surveillance	Widely used and accepted
	Active surveillance (e.g., Sentinel )	Emerging
Clinical research	Hypothesis generation	Accepted
	Feasibility assessments	Accepted
	Performance improvement, guideline adherence	Accepted
	Patient recruitment	Emerging
	Comparative effectiveness, health technology assessments	Emerging
	Pragmatic trials (e.g. PROBE design)	Emerging
	Point of care randomization	Emerging
	Registry randomized trials to test new interventions	Emerging
	Source data to populate eCRF (eliminating or minimizing need for data extraction/data entry)	Emerging/potential
	Endpoint or SAE ascertainment	Emerging/potential
Regulatory	Safety surveillance, pharmacovigilance	Accepted
	New indications or marketing authorization	Potential

a Sentinel is the United States Food and Drug Administration’s national electronic system to proactively monitor medical product safety post-marketing, through rapidly and securely accessing data from large amounts of electronic healthcare records, insurance claims, and registries, from a diverse group of data partners [ 24 ]

PROBE prospective randomized open blinded endpoint, eCRF electronic case report form, SAE serious adverse event

Electronic health records for research applications

Epidemiologic and observational research.

EHR data have been used to support observational studies, either as stand-alone data or following linkage to primary research data or other administrative data sets [ 3 , 11 – 14 ]. For example, the initial Euro Heart Survey [ 15 ] and subsequent Eurobservational Research Program (EORP) [ 16 ], the American College of Cardiology National Cardiovascular Data Registry (ACC-NCDR) [ 14 ], National Registry of Myocardial Infarction (NRMI), and American Heart Association Get With the Guidelines (AHA GWTG) [ 17 ] represent clinical data (collected from health records into an electronic case report form [eCRF] designed for the specific registry) on the management of patients across a spectrum of different cardiovascular diseases. However, modern EHR systems can minimize or eliminate the need for duplicate data collection (i.e., in a separate registry-specific eCRF), are capable of integrating large amounts of medical information accumulated throughout the patient’s life, enabling longitudinal study of diseases using the existing informatics infrastructure [ 18 ]. For example, EHR systems increasingly house imaging data which provide more detailed disease characterization than previously available in most observational data sets. In some countries (e.g., Farr Institute in Scotland [ 19 ]), the EHR can be linked, at an individual level, to other data sets, including general population health and lifestyle surveys, disease registries, and data collected by other sectors (e.g., education, housing, social care, and criminal justice). EHR data support a wide range of epidemiological research on the natural history of disease, drug utilization, and safety, as well as health services research.

Safety surveillance and regulatory uses

Active post-marketing safety surveillance and signal detection are important, emerging applications for EHRs, because they can provide realistic rates of events (unlike spontaneous event reports) and information on real-world use of drugs [ 20 ]. The EU-ADR project linked 8 databases in four European countries (Denmark, Italy, The Netherlands, United Kingdom) to enable analysis of select target adverse drug events [ 21 ]. The European Medicines Agency (EMA) coordinates the European Network of Centres for Pharmacoepidemiology and Pharmacovigilance (ENCePP) which aims to conduct post-marketing risk assessment using various EHR sources [ 22 , 23 ]. In the United States, the Food and Drug Administration (FDA) uses EHR data from several different sources (e.g., Sentinel and Mini-Sentinel System [ 24 ], Centers for Medicare and Medicaid Services [CMS], Veterans Affairs, Department of Defense, Substance Abuse and Mental Health Services Administration) to support post-marketing safety investigations [ 25 ].

Prospective clinical research

National patient registries that contain data extracted from the EHR are an accepted modality to assess guideline adherence and the effectiveness of performance improvement initiatives [ 26 – 33 ]. However, the use of EHRs for prospective clinical research is still limited, despite the fact that data collected for routine medical care overlap considerably with data collected for research. The most straightforward and generally accepted application for EHR is assessing trial feasibility and facilitating patient recruitment, and EHRs are currently used for this purpose in some centers. Using EHR technology to generate lists of patients who might be eligible for research is recognized as an option to meet meaningful use standards for EHR in the United States [ 6 ]. However, incomplete data may prohibit screening for the complete list of eligibility criteria [ 34 ], but EHRs may facilitate pre-screening of patients by age, gender, and diagnosis, particularly for exclusion of ineligible patients, and reduce the overall screening burden in clinical trials [ 35 ]. A second, and more complex, step involves the reuse of information collected in EHRs for routine clinical care as source data for research. Using EHRs as the source for demographic information, co-morbidities, and concomitant medications has several advantages over separately recording these data into an eCRF. Transcription errors may be reduced, since EHR data are entered by providers directly involved in a patient’s care as opposed to secondary eCRF entry by study personnel. The eCRF may be a redundant and costly step in a clinical trial, since local health records (electronic or paper) are used to verify source data entered into the eCRF. Finally, EHRs might enhance patient safety and reduce timelines if real-time EHR systems are used in clinical trials, in contrast to delays encountered with manual data entry into an eCRF. The EHR may facilitate implementation of remote data monitoring, which has the potential to greatly reduce clinical trial costs. The Innovative Medicine Initiative (IMI) Electronic Health Records for Clinical Research (EHR4CR, http://www.ehr4cr.eu ) project is one example, where tools and processes are being developed to facilitate reuse of EHR data for clinical research purposes. Systems to assess protocol feasibility and identify eligible patients for recruitment have been implemented, and efforts to link EHRs with clinical research electronic data collection are ongoing [ 36 ].

A shift towards pragmatic trials has been proposed as a mechanism to improve clinical trial efficiency [ 37 ]. Most of the data in a pragmatic trial are collected in the context of routine clinical care, which reduce trial-specific clinic visits and assessments, and should also reduce costs [ 38 ]. This concept is being applied in the National Institutes of Health (NIH) Health Care Systems Research Collaboratory. Trials conducted within the NIH Collaboratory aim to answer questions related to care delivery and the EHR contains relevant data for this purpose. Studies may have additional data collection modules if variables not routinely captured in the EHR are needed for a specific study. Similarly, the Patient-Centered Outcomes Research Institute (PCORI) has launched PCORnet, a research network that uses a common data platform alongside the existing EHR to conduct observational and interventional comparative effectiveness research [ 9 , 39 , 40 ].

The integration of EHRs in the conventional randomized controlled trials intended to support a new indication is more complex. EHRs may be an alternative to eCRFs when data collection is focused and limited to critical variables that are consistently collected in routine clinical care. Regulatory feedback indicates that while a new indication for a marketed drug might be achieved through EHRs, first marketing authorization using data entirely from EHRs would most likely not be possible with current systems until validation studies are performed and reviewed by regulatory agencies. The EHR could also be used to collect serious adverse events (SAE) that result in hospitalization, or to collect endpoints that do not necessarily require blinded adjudication (e.g., death), although the utility of EHRs for this purpose is dependent on the type of endpoint, whether it can reliably be identified in the EHR, and the timeliness of EHR data availability. Events that are coded for reimbursement (e.g., hospitalizations, MI) or new diagnoses, where disease-specific therapy is initiated (e.g., initiation of glucose lowering drugs to define new onset diabetes) tend to be more reliable. The reliability of endpoint collection varies by region and depends on the extent of linkage between different databases.

Challenges to using electronic health records in clinical trials and steps toward solutions

Challenges to using EHRs in clinical trials have been identified, related to data quality and validation, complete data capture, heterogeneity between systems, and developing a working knowledge across systems (Table 2 ). Ongoing projects, such as those conducted within the NIH Collaboratory and PCORnet [ 39 , 41 ] in the United States or the Farr Institute of Health Informatics Research in Scotland, have demonstrated the feasibility of using EHRs for aspects of clinical research, particularly comparative effectiveness. The success of these endeavors is connected to careful planning by a multi-stakeholder group committed to patient privacy, data security, fair governance, robust data infrastructure, and quality science from the outset. The next hurdle is to adapt the accrued knowledge for application to a broader base of clinical trials.

Table 2

Challenges of using electronic health records in research

Problem	Example	Potential Solutions
Data quality and validation	Selecting measurement of interest for a clinical trial when multiple measurements are available (e.g., laboratory data) Inaccurate information in EHRs Coding errors	Specific parameters (e.g., using date or time windows) stated in protocol or operating procedures for extracting data from EHR into eCRF Use codes linked to reimbursement, which have greater likelihood of reliability Stakeholder collaboration to develop validation methodology Stakeholder collaboration to contribute data for EHR validation studies
Complete data capture	Clinical endpoints SAEs Problematic in multiple-payer systems Death	Develop standards for data sharing and privacy Explore linking EHRs to national death registries
Heterogeneity among systems	Multiple different vendors within a given country or region Inconfigurable systems Lack of flexible architecture Lack of common data fields, data definitions, and difficulty with data mapping Incomplete data capture Missing fields of interest (i.e. relevant to some diseases but not others) Inability to link systems (i.e. different patient identifiers)	Commit resources to harmonization efforts Form working group with representation from all stakeholders to develop consensus agreement on a common set of data variables to be included in all systems
System knowledge	Inadequate understanding of database and its structure Researchers may not understand limitations of database	Transparency Develop and maintain data standards and operations manuals Report strengths, limitations, and nuances of databases in primary manuscripts Informatics training for investigators

EHR electronic health record, SAE serious adverse event

Data quality and validation

Data quality and validation are key factors in determining whether EHRs might be suitable data sources in clinical trials. Concerns about coding inaccuracies or bias introduced by selection of codes driven by billing incentives rather than clinical care may be diminished when healthcare providers enter data directly into the EHRs or when EHRs are used throughout all areas of the health-system, but such systems have not yet been widely implemented [ 42 ]. Excessive or busy workloads may also contribute to errors in clinician data entry [ 43 ]. Indeed, errors in EHRs have been reported [ 43 – 45 ].

Complete data capture is also a critical aspect of using EHRs for clinical research, particularly if EHRs are used for endpoint ascertainment or SAE collection. Complete data capture can be a major barrier in regions, where patients receive care from different providers or hospitals operating in different EHR systems that are not linked.

Consistent, validated methods for assessing data quality and completeness have not yet been adopted [ 46 ], but validation is a critical factor for the regulatory acceptance of EHR data. Proposed validation approaches include using both an eCRF and EHRs in a study in parallel and comparing results using the two data collection methods. This approach will require collaborative efforts to embed EHR substudies in large cardiovascular studies conducted by several sponsors. Assessing selected outcomes of interest from several EHR-based trials to compare different methodologies with an agreed statistical framework will be required to gauge precision of data collection via EHRs. A hybrid approach has also been proposed, where the EHR is used to identify study endpoints (e.g., death, hospitalization, myocardial infarction, and cancer), followed by adjudication and validation of EHR findings using clinical data (e.g., electrocardiogram and laboratory data).

Validity should be defined a priori and should be specific to the endpoints of interest as well as relevant to the country or healthcare system. Validation studies should aim to assess both the consistency between EHR data and standard data collection methods, and also how identified differences influence a study’s results. Proposed uses of EHRs for registration trials and methods for their validation will likely be considered by regulatory agencies on a case-by-case basis, because of the limited experience with EHRs for this purpose at the current time. Collaboration among industry sponsors to share cumulative experiences with EHR validation studies might lead to faster acceptance by regulatory authorities.

The ESC-CRT recommends that initial efforts to integrate EHRs in clinical trials focus on a few efficacy endpoints of interest, preferably objective endpoints (e.g., all-cause or cause-specific mortality) that are less susceptible to bias or subjective interpretation. As noted above, mortality may be incompletely captured in EHRs, particularly if patients die outside of the hospital, or at another institution using a non-integrated EHR. Thus, methods to supplement endpoint ascertainment in the EHR may be necessary if data completeness is uncertain. Standardized endpoint definitions based on the EHR should be included in the study protocol and analysis plan. A narrow set of data elements for auditing should be prospectively defined to ensure the required variables which are contained in the EHR.

Early interaction between sponsors, clinical investigators, and regulators is recommended to enable robust designs for clinical trials aiming to use EHRs for endpoint ascertainment. Plans to translate Good Clinical Practice into an EHR facilitated research environment should be described. Gaps in personnel training and education should be identified and specific actions to address training deficiencies should be communicated to regulators and in place prior to the start of the trial.

Timely access to electronic health record data

The potential for delays in data access is an important consideration when EHRs are used in clinical trials. EHRs may contain data originally collected as free text that was later coded for the EHR. Thus, coded information may not be available for patient identification/recruitment during the admission. Similarly, coding may occur weeks or months after discharge. In nationally integrated systems, data availability may also be delayed. These delays may be critical depending on the purpose of data extracted from the EHR (e.g., SAE reporting, source data, or endpoints in a time-sensitive study).

Heterogeneity between systems

Patients may be treated by multiple healthcare providers who operate independently of one another. Such patients may have more than one EHR, and these EHRs may not be linked. This heterogeneity adds to the complexity of using EHRs for clinical trials, since data coordinating centres have to develop processes for interacting or extracting data from any number of different systems. Differences in quality [ 47 ], non-standardized terminology, incomplete data capture, issues related to data sharing and data privacy, lack of common data fields, and the inability of systems to be configured to communicate with each other may also be problematic. Achieving agreement on a minimum set of common data fields to enable cross communication between systems would be a major step forward towards enabling EHRs to be used in clinical trials across centers and regions [ 48 , 49 ].

Data security and privacy

Privacy issues and information governance are among the most complex aspects of implementing EHRs for clinical research, in part because attitudes and regulations related to data privacy vary markedly around the world. Data security and appropriate use are high priorities, but access should not be restricted to the extent that the data are of limited usefulness. Access to EHR data by regulatory agencies will be necessary for auditing purposes in registration trials. Distributed analyses have the advantage of allowing data to remain with the individual site and under its control [ 39 , 41 ].

Pre-trial planning is critical to anticipate data security issues and to develop optimal standards and infrastructure. For pivotal registration trials, patients should be informed during the consent process about how their EHRs will be used and by whom. Modified approaches to obtaining informed consent for comparative effectiveness research studies of commonly used clinical practices or interventions may be possible [ 50 ]. A general upfront consent stating that EHR data may be used for research is a proactive step that may minimize later barriers to data access, although revision of existing legislation or ethics board rules may be needed to allow this approach. Patients and the public should be recognized as important stakeholders, and they can be advocates for clinical research using EHRs and improve the quality of EHR-based research if they are educated and engaged in the process and the purpose and procedures for EHR use are transparent. Developing optimal procedures for ensuring patients that are informed and protected, balanced with minimizing barriers to research is a major consideration as EHR-based research advances.

System capabilities

EHRs for use in clinical research need a flexible architecture to accommodate studies of different interventions or disease states. EHR systems may be capable of matching eligibility criteria to relevant data fields and flagging potential trial subjects to investigators. Patient questionnaires and surveys can be linked to EHRs to provide additional context to clinical data. Pre-population of eCRFs has been proposed as a potential role for EHRs, but the proportion of fields in an EHR that can be mapped to an eCRF varies substantially across systems.

EHRs may be more suitable for pragmatic trials where data collection mirrors those variables collected in routine clinical care. Whether regulators would require collection of additional elements to support a new drug or new indication depends on the drug, intended indication, patient population, and potential safety concerns.

Sustainability

The sustainability of EHRs in clinical research will largely depend on the materialization of their promised efficiencies. Programs like the NIH Collaboratory [ 41 ] and PCORnet [ 39 , 41 ], and randomized registry trials [ 51 , 52 ] are demonstrating the feasibility of these more efficient approaches to clinical research. The sustainability of using EHRs for pivotal registration clinical trials will depend on regulatory acceptance of the approach and whether the efficiencies support a business case for their use.

Role of stakeholders

To make the vision of EHRs in clinical trials a reality, stakeholders should collaborate and contribute to the advancement of EHRs for research. Professional bodies, such as the ESC, can play a major role in the training and education of researchers and the public about the potential value of EHR. Clinical trialists and industry must be committed to advancing validation methodology [ 53 ]. Investigators should develop, conduct, and promote institutional EHR trials that change clinical practice; such experience may encourage EHR trial adoption by industry and the agencies. Development of core or minimal data sets could streamline the process, reduce redundancy and heterogeneity, and decrease start-up time for future EHR-based clinical trials. These and other stakeholder contributions are outlined in Table 3 .

Table 3

Role and influence of stakeholders in advancing the use of electronic health records in clinical research

Stakeholder	Contribution
Professional societies	Training and education Global platform for education at annual meetings or congresses Leverage industry support Public education to foster public support Transform EORP into a prospective trial instrument; generate support from industry who may use this resource for future trials Develop data standards (CARDS-revisited) Organize working groups charged with generating common EHR templates or data sets, or achieving agreement on minimum standards Lobby regulatory agencies and industry sponsors
Clinical trialists and industry	Engage other collaborators (e.g., ethicists, CROs, academic CROs, information governance, registries, IT providers, EHR companies, patient advocacy groups, data protection/security experts, legislators/agencies, public funders, legal experts, treating physicians, hospital administrators) Pilot the evaluation of EHR versus conventional non-EHR trials Pilot trials to compare event collection using EHRs versus usual eCRF Conduct actual EHR trials, initially in smaller countries, adapting the approach based on lessons learned, then applying to larger settings Adopt EHRs on an experimental basis for feasibility assessments and patient recruitment Lobby other stakeholders to collaborate towards developing robust methodology to incorporate EHRs in clinical trials Educate professionals and the public about potential value of EHRs
Regulatory	Work with industry to identify appropriate ways to incorporate EHR data prospectively into study designs
EHR vendors	Invest in building research capabilities on EHR platforms

CARDS cardiology audit and registration data standards, CRO contract research organization, eCRF electronic case report form, IT information technology, EHR electronic health record, EORP European Observational Research Program

Electronic health records are a promising resource to improve the efficiency of clinical trials and to capitalize on novel research approaches. EHRs are useful data sources to support comparative effectiveness research and new trial designs that may answer relevant clinical questions as well as improve efficiency and reduce the cost of cardiovascular clinical research. Initial experience with EHRs has been encouraging, and accruing knowledge will continue to transform the application of EHRs for clinical research. The pace of technology has produced unprecedented analytic capabilities, but these must be pursued with appropriate measures in place to manage security, privacy, and ensure adequacy of informed consent. Ongoing programs have implemented creative solutions for these issues using distributed analyses to allow organizations to retain data control and by engaging patient stakeholders. Whether EHRs can be successfully applied to the conventional drug development in pivotal, registration trials remains to be seen and will depend on demonstration of data quality and validity, as well as realization of expected efficiencies.

Acknowledgments

This paper was generated from discussions during a cardiovascular round table (CRT) Workshop organized on 23–24 April 2015 by the European Society of Cardiology (ESC). The CRT is a strategic forum for high-level dialogues between academia, regulators, industry, and ESC leadership to identify and discuss key strategic issues for the future of cardiovascular health in Europe and other parts of the world. We acknowledge Colin Freer for his participation in the meeting. This article reflects the views of the authors and should not be construed to represent FDA’s views or policies. The opinions expressed in this paper are those of the authors and cannot be interpreted as the opinion of any of the organizations that employ the authors. MRC’s salary is supported by the National Institute for Health Research (NIHR) Cardiovascular Biomedical Research Unit at the Royal Brompton Hospital, London, UK.

Conflict of interest

Martin R. Cowie: Research grants from ResMed, Boston Scientific, and Bayer; personal fees from ResMed, Boston Scientific, Bayer, Servier, Novartis, St. Jude Medical, and Pfizer. Juuso Blomster: Astra Zeneca employee. Lesley Curtis: Funding from FDA for work with the Mini-Sentinel program and from PCORI for work with the PCORnet program. Sylvie Duclaux: None. Ian Ford: None. Fleur Fritz: None. Samantha Goldman: None. Salim Janmohamed: GSK employee and shareholder. Jörg Kreuzer: Employee of Boehringer-Ingelheim. Mark Leenay: Employee of Optum. Alexander Michel: Bayer employee and shareholder. Seleen Ong: Employee of Pfizer. Jill Pell: None. Mary Ross Southworth: None. Wendy Gattis Stough: Consultant to European Society of Cardiology, Heart Failure Association of the European Society of Cardiology, European Drug Development Hub, Relypsa, CHU Nancy, Heart Failure Society of America, Overcome, Stealth BioTherapeutics, Covis Pharmaceuticals, University of Gottingen, and University of North Carolina. Martin Thoenes: Employee of Edwards Lifesciences. Faiez Zannad: Personal fees from Boston Scientific, Servier, Pfizer, Novartis, Takeda, Janssen, Resmed, Eli Lilly, CVRx, AstraZeneca, Merck, Stealth Peptides, Relypsa, ZS Pharma, Air Liquide, Quantum Genomics, Bayer for Steering Committee, Advisory Board, or DSMB member. Andrew Zalewski: Employee of GSK.

Environment
Science & Technology
Business & Industry
Health & Public Welfare
Topics (CFR Indexing Terms)
Public Inspection
Presidential Documents
Document Search
Advanced Document Search
Public Inspection Search
Reader Aids Home
Office of the Federal Register Announcements
Using FederalRegister.Gov
Understanding the Federal Register
Recent Site Updates
Federal Register & CFR Statistics
Videos & Tutorials
Developer Resources
Government Policy and OFR Procedures
Congressional Review
My Clipboard
My Comments
My Subscriptions
Sign In / Sign Up
Site Feedback
Search the Federal Register

The Federal Register

The daily journal of the united states government.

Legal Status

This site displays a prototype of a “Web 2.0” version of the daily Federal Register. It is not an official legal edition of the Federal Register, and does not replace the official print version or the official electronic version on GPO’s govinfo.gov.

The documents posted on this site are XML renditions of published Federal Register documents. Each document posted on the site includes a link to the corresponding official PDF file on govinfo.gov. This prototype edition of the daily Federal Register on FederalRegister.gov will remain an unofficial informational resource until the Administrative Committee of the Federal Register (ACFR) issues a regulation granting it official legal status. For complete information about, and access to, our official publications and services, go to About the Federal Register on NARA's archives.gov.

The OFR/GPO partnership is committed to presenting accurate and reliable regulatory information on FederalRegister.gov with the objective of establishing the XML-based Federal Register as an ACFR-sanctioned publication in the future. While every effort has been made to ensure that the material on FederalRegister.gov is accurately displayed, consistent with the official SGML-based PDF version on govinfo.gov, those relying on it for legal research should verify their results against an official edition of the Federal Register. Until the ACFR grants it official status, the XML rendition of the daily Federal Register on FederalRegister.gov does not provide legal notice to the public or judicial notice to the courts.

Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products; Guidance for Industry; Availability

A Notice by the Food and Drug Administration on 07/25/2024

Document Details

Information about this document as published in the Federal Register .

Document Statistics

Enhanced content.

Relevant information about this document from Regulations.gov provides additional context. This information is not part of the official Federal Register document.

Published Document

This document has been published in the Federal Register . Use the PDF linked in the document sidebar for the official electronic format.

Enhanced Content - Table of Contents

This table of contents is a navigational tool, processed from the headings within the legal text of Federal Register documents. This repetition of headings to form internal navigation links has no substantive legal effect.

Electronic Submissions

Written/paper submissions, for further information contact:, supplementary information:, i. background, ii. paperwork reduction act of 1995, iii. electronic access, enhanced content - submit public comment.

This feature is not available for this document.

Enhanced Content - Read Public Comments

Enhanced content - sharing.

Email this document to a friend

Enhanced Content - Document Print View

Print this document

Enhanced Content - Document Tools

These tools are designed to help you understand the official document better and aid in comparing the online edition to the print edition.

These markup elements allow the user to see how the document follows the Document Drafting Handbook that agencies use to create their documents. These can be useful for better understanding how a document is structured but are not part of the published document itself.

Enhanced Content - Developer Tools

This document is available in the following developer friendly formats:.

JSON: Normalized attributes and metadata
XML: Original full text XML
MODS: Government Publishing Office metadata

More information and documentation can be found in our developer tools pages .

Official Content

View printed version (PDF)

This PDF is the current document as it appeared on Public Inspection on 07/24/2024 at 8:45 am. It was viewed 0 times while on Public Inspection.

If you are using public inspection listings for legal research, you should verify the contents of the documents against a final, official edition of the Federal Register. Only official editions of the Federal Register provide legal notice of publication to the public and judicial notice to the courts under 44 U.S.C. 1503 & 1507 . Learn more here .

Food and Drug Administration, HHS.

Notice of availability.

The Food and Drug Administration (FDA or Agency) is announcing the availability of a final guidance for industry entitled “Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products.” FDA is issuing this guidance as part of its Real-World Evidence (RWE) program and to satisfy, in part, the mandate under the Federal Food, Drug, and Cosmetic Act (FD&C Act) to issue guidance about the use of RWE in regulatory decision making. This guidance is intended to provide sponsors and other interested parties with considerations when proposing to use electronic health records (EHRs) or medical claims data in clinical studies to support a regulatory decision for effectiveness or safety. This guidance finalizes the draft guidance of the same title issued on September 30, 2021.

The announcement of the guidance is published in the Federal Register on July 25, 2024.

You may submit either electronic or written comments on Agency guidances at any time as follows:

Submit electronic comments in the following way:

Federal eRulemaking Portal: https://www.regulations.gov . Follow the instructions for submitting comments. Comments submitted electronically, including attachments, to https://www.regulations.gov will be posted to the docket unchanged. Because your comment will be made public, you are solely responsible for ensuring that your comment does not include any confidential information that you or a third party may not wish to be posted, such as medical information, your or anyone else's Social Security number, or confidential business information, such as a manufacturing process. Please note that if you include your name, contact information, or other information that identifies you in the body of your comments, that information will be posted on https://www.regulations.gov .
If you want to submit a comment with confidential information that you do not wish to be made available to the public, submit the comment as a written/paper submission and in the manner detailed (see “Written/Paper Submissions” and “Instructions”).

Submit written/paper submissions as follows:

Mail/Hand Delivery/Courier (for written/paper submissions): Dockets Management Staff (HFA-305), Food and Drug Administration, 5630 Fishers Lane, Rm. 1061, Rockville, MD 20852.
For written/paper comments submitted to the Dockets Management Staff, FDA will post your comment, as well as any attachments, except for information submitted, marked and identified, as confidential, if submitted as detailed in “Instructions.”

Instructions: All submissions received must include the Docket No. FDA-2020-D-2307 for “Real-World Data: Start Printed Page 60431 Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products.” Received comments will be placed in the docket and, except for those submitted as “Confidential Submissions,” publicly viewable at https://www.regulations.gov or at the Dockets Management Staff between 9 a.m. and 4 p.m., Monday through Friday, 240-402-7500.

Confidential Submissions—To submit a comment with confidential information that you do not wish to be made publicly available, submit your comments only as a written/paper submission. You should submit two copies total. One copy will include the information you claim to be confidential with a heading or cover note that states “THIS DOCUMENT CONTAINS CONFIDENTIAL INFORMATION.” The Agency will review this copy, including the claimed confidential information, in its consideration of comments. The second copy, which will have the claimed confidential information redacted/blacked out, will be available for public viewing and posted on https://www.regulations.gov . Submit both copies to the Dockets Management Staff. If you do not wish your name and contact information to be made publicly available, you can provide this information on the cover sheet and not in the body of your comments and you must identify this information as “confidential.” Any information marked as “confidential” will not be disclosed except in accordance with 21 CFR 10.20 and other applicable disclosure law. For more information about FDA's posting of comments to public dockets, see 80 FR 56469 , September 18, 2015, or access the information at: https://www.govinfo.gov/content/pkg/FR-2015-09-18/pdf/2015-23389.pdf .

Docket: For access to the docket to read background documents or the electronic and written/paper comments received, go to https://www.regulations.gov and insert the docket number, found in brackets in the heading of this document, into the “Search” box and follow the prompts and/or go to the Dockets Management Staff, 5630 Fishers Lane, Rm. 1061, Rockville, MD 20852, 240-402-7500.

You may submit comments on any guidance at any time (see 21 CFR 10.115(g)(5) ).

Submit written requests for single copies of this guidance to the Division of Drug Information, Center for Drug Evaluation and Research, Food and Drug Administration, 10001 New Hampshire Ave., Hillandale Building, 4th Floor, Silver Spring, MD 20993-0002 or to the Office of Communication, Outreach and Development, Center for Biologics Evaluation and Research (CBER), Food and Drug Administration, 10903 New Hampshire Ave., Bldg. 71, Rm. 3128, Silver Spring, MD 20993-0002. Send one self-addressed adhesive label to assist that office in processing your requests. See the SUPPLEMENTARY INFORMATION section for electronic access to the guidance document.

Dianne Paraoan, Center for Drug Evaluation and Research, Food and Drug Administration, 10903 New Hampshire Ave., Bldg. 51, Rm. 3226, Silver Spring, MD 20993-0002, 301-796-3161; or James Myers, Center for Biologics Evaluation and Research, Food and Drug Administration, 10903 New Hampshire Ave., Bldg. 71, Rm. 7301, Silver Spring, MD 20993-0002, 240-402-7911.

FDA is announcing the availability of a guidance for industry entitled “Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products.” This guidance discusses the following topics related to the potential use of EHRs and medical claims in clinical studies to support regulatory decisions: selection of data sources that appropriately address the study question and sufficiently capture study populations, exposure, outcomes of interest, and key covariates; development and validation of definitions for study design elements ( e.g., exposure, outcomes, covariates); and data traceability and quality during data accrual, data curation, and incorporation into the final study-specific dataset.

Section 3022 of the 21st Century Cures Act (Cures Act) of 2016 amended the FD&C Act to add section 505F, Utilizing Real World Evidence ( 21 U.S.C. 355g ), which requires FDA to issue guidance about the use of RWE in regulatory decision making. In addition, under the Prescription Drug User Fee Amendments of 2017 (PDUFA VI), FDA committed to publish draft guidance on how RWE can contribute to the assessment of safety and effectiveness in regulatory submissions. In 2018, FDA created an RWE Framework and RWE Program to evaluate the potential use of RWE to help support the approval of a new indication for a drug already approved under the FD&C Act or to help support or satisfy postapproval study requirements. In late 2021, FDA utilized the program to issue draft guidances outlining considerations for the use of real-world data and RWE in regulatory decision making to, among other things, help satisfy the Cures Act mandate and the PDUFA VI commitment.

This guidance finalizes the draft guidance of the same title issued on September 30, 2021 ( 86 FR 54219 ). FDA considered comments received on the draft guidance as the guidance was finalized. Changes from the draft to the final guidance include: (1) clarifying that the selection of study variables for validation and the extent of effort required for validation depends on the necessary level of certainty and the implication of potential misclassification on study inference; (2) noting that choice of a reference standard for validation may vary by the study design and question, variable of interest, and the necessary level of certainty; (3) recommending the use of quantitative approaches, such as quantitative bias analyses, either a priori for feasibility assessment, or to facilitate interpretation of study results, or for both purposes, to demonstrate whether and how misclassification, if present, might impact study findings; and (4) removing defined terms that are generally understood and transferring other relevant definitions from a glossary to the text. In addition, editorial changes were made to improve clarity.

This guidance is being issued consistent with FDA's good guidance practices regulation ( 21 CFR 10.115 ). The guidance represents the current thinking of FDA on “Real-World Data: Assessing Electronic Health Records and Medical Claims Data To Support Regulatory Decision-Making for Drug and Biological Products.” It does not establish any rights for any person and is not binding on FDA or the public. You can use an alternative approach if it satisfies the requirements of the applicable statutes and regulations.

While this guidance contains no collection of information, it does refer to previously approved FDA collections of information. The previously approved collections of information are subject to review by the Office of Management and Budget (OMB) under the Paperwork Reduction Act of 1995 (PRA) ( 44 U.S.C. 3501-3521 ). The collections of information in 21 CFR part 11 have been approved under OMB control number 0910-0303; the collections of information in 21 CFR part 312 have been approved under OMB control number 0910-0014; the collections of information in 21 CFR part 314 have been approved under OMB control number 0910-0001; and the collections Start Printed Page 60432 of information in 21 CFR part 601 have been approved under OMB control number 0910-0338.

Persons with access to the internet may obtain the guidance at https://www.fda.gov/drugs/guidance-compliance-regulatory-information/guidances-drugs , https://www.fda.gov/vaccines-blood-biologics/guidance-compliance-regulatory-information-biologics/biologics-guidances , https://www.fda.gov/regulatory-information/search-fda-guidance-documents , or https://www.regulations.gov .

Dated: July 22, 2024.

Lauren K. Roth,

Associate Commissioner for Policy.

[ FR Doc. 2024-16338 Filed 7-24-24; 8:45 am]

BILLING CODE 4164-01-P

Executive Orders

Reader Aids

Information.

About This Site
Accessibility
No Fear Act
Continuity Information

Large language models for accurate disease detection in electronic health records

Find this author on Google Scholar
Find this author on PubMed
Search for this author on this site
ORCID record for Nils Bürgisser
For correspondence: [email protected]
ORCID record for Clement P. Buclin
ORCID record for Kim Lauper
ORCID record for Delphine S. Courvoisier
ORCID record for Denis Mongin
Info/History
Supplementary material
Preview PDF

Importance The use of large language models (LLMs) in medicine is increasing, with potential applications in electronic health records (EHR) to create patient cohorts or identify patients who meet clinical trial recruitment criteria. However, significant barriers remain, including the extensive computer resources required, lack of performance evaluation, and challenges in implementation.

Objective This study aims to propose and test a framework to detect disease diagnosis using a recent light LLM on French-language EHR documents. Specifically, it focuses on detecting gout (“goutte” in French), a ubiquitous French term that have multiple meanings beyond the disease. The study will compare the performance of the LLM-based framework with traditional natural language processing techniques and test its dependence on the parameter used.

Design The framework was developed using a training and testing set of 700 paragraphs assessing “gout”, issued from a random selection of retrospective EHR documents. All paragraphs were manually reviewed and classified by two health-care professionals (HCP) into disease (true gout) and non-disease (gold standard). The LLM’s accuracy was tested using few-shot and chain-of-thought prompting and compared to a regular expression (regex)-based method, focusing on the effects of model parameters and prompt structure. The framework was further validated on 600 paragraphs assessing “Calcium Pyrophosphate Deposition Disease (CPPD)”.

Setting The documents were sampled from the electronic health-records of a tertiary university hospital in Geneva, Switzerland.

Participants Adults over 18 years of age.

Exposure Meta’s Llama 3 8B LLM or traditional method, against a gold standard.

Main Outcomes and Measures Positive and negative predictive value, as well as accuracy of tested models.

Results The LLM-based algorithm outperformed the regex method, achieving a 92.7% [88.7-95.4%] positive predictive value, a 96.6% [94.6-97.8%] negative predictive value, and an accuracy of 95.4% [93.6-96.7%] for gout. In the validation set on CPPD, accuracy was 94.1% [90.2-97.6%]. The LLM framework performed well over a wide range of parameter values.

Conclusions and Relevance LLMs were able to accurately detect disease diagnoses from EHRs, even in non-English languages. They could facilitate creating large disease registries in any language, improving disease care assessment and patient recruitment for clinical trials.

Question How accurate and efficient are large language models (LLMs) in detecting diseases from unstructured electronic health records (EHR) text compared to traditional natural language processing techniques?

Findings This study proposes a framework based on Meta’s Llama 3 8B, a recent public LLM, outperforming traditional natural language processing techniques in detecting gout and calcium pyrophosphate deposition disease in unstructured text. It achieves high positive and negative predictive values and accuracy. Performance was robust over a wide range of parameters.

Meaning The proposed framework can ease the use of LLMs in effectively detecting disease in EHR data for various clinical applications.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This project was funded by the Private Foundation of the Geneva University Hospitals, a not-for-profit foundation.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

This study involves human participants and the creation and use of the register for quality improvement programs has been approved by the Geneva ethics commission (CCER 2023-00129). The need for consent was waived by the Geneva Ethics Committee because this study qualifies as a quality improvement initiative.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

↵ * co-last authors

Data Availability

All prompts and code have been made available at the following gitlab repository: https://gitlab.unige.ch/goutte/llm_detection_of_diagnosis . Due to medical confidentiality, we are unable to share the sentences and document data. However, if authorization is obtained from the ethics committee, we may be able to provide access to the data

https://gitlab.unige.ch/goutte/llm_detection_of_diagnosis

View the discussion thread.

Supplementary Material

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Citation Manager Formats

EndNote (tagged)
EndNote 8 (xml)
RefWorks Tagged
Ref Manager
Tweet Widget
Facebook Like
Google Plus One

Subject Area

Rheumatology
Addiction Medicine (335)
Allergy and Immunology (658)
Anesthesia (177)
Cardiovascular Medicine (2565)
Dentistry and Oral Medicine (310)
Dermatology (217)
Emergency Medicine (389)
Endocrinology (including Diabetes Mellitus and Metabolic Disease) (907)
Epidemiology (12070)
Forensic Medicine (10)
Gastroenterology (741)
Genetic and Genomic Medicine (3983)
Geriatric Medicine (375)
Health Economics (666)
Health Informatics (2572)
Health Policy (992)
Health Systems and Quality Improvement (955)
Hematology (357)
HIV/AIDS (824)
Infectious Diseases (except HIV/AIDS) (13567)
Intensive Care and Critical Care Medicine (783)
Medical Education (394)
Medical Ethics (106)
Nephrology (422)
Neurology (3747)
Nursing (206)
Nutrition (558)
Obstetrics and Gynecology (717)
Occupational and Environmental Health (686)
Oncology (1948)
Ophthalmology (565)
Orthopedics (233)
Otolaryngology (300)
Pain Medicine (246)
Palliative Medicine (72)
Pathology (469)
Pediatrics (1088)
Pharmacology and Therapeutics (453)
Primary Care Research (442)
Psychiatry and Clinical Psychology (3346)
Public and Global Health (6411)
Radiology and Imaging (1354)
Rehabilitation Medicine and Physical Therapy (793)
Respiratory Medicine (857)
Rheumatology (394)
Sexual and Reproductive Health (395)
Sports Medicine (336)
Surgery (431)
Toxicology (51)
Transplantation (184)
Urology (162)

Open access
Published: 29 July 2024

Creating a health informatics data resource for hearing health research

Nishchay Mehta 1 , 2 ,
Baptiste Briot Ribeyre 1 , 3 ,
Lilia Dimitrov 1 , 2 ,
Louise J. English 1 , 3 ,
Colleen Ewart 4 ,
Antje Heinrich 5 , 6 ,
Nikhil Joshi 1 , 2 ,
Kevin J. Munro 5 , 6 ,
Gail Roadknight 8 ,
Luis Romao 1 , 3 ,
Anne Gm Schilder 1 , 2 ,
Ruth V. Spriggs 9 , 10 ,
Ruth Norris 5 , 11 ,
Talisa Ross 1 , 2 , 7 &
George Tilston 5 , 11

BMC Medical Informatics and Decision Making volume 24 , Article number: 209 ( 2024 ) Cite this article

63 Accesses

1 Altmetric

Metrics details

The National Institute of Health and Social Care Research (NIHR) Health Informatics Collaborative (HIC) for Hearing Health has been established in the UK to curate routinely collected hearing health data to address research questions. This study defines priority research areas, outlines its aims, governance structure and demonstrates how hearing health data have been integrated into a common data model using pure tone audiometry (PTA) as a case study.

After identifying key research aims in hearing health, the governance structure for the NIHR HIC for Hearing Health is described. The Observational Medical Outcomes Partnership (OMOP) was chosen as our common data model to provide a case study example.

The NIHR HIC Hearing Health theme have developed a data architecture outlying the flow of data from all of the various siloed electronic patient record systems to allow the effective linkage of data from electronic patient record systems to research systems. Using PTAs as an example, OMOPification of hearing health data successfully collated a rich breadth of datapoints across multiple centres.

This study identified priority research areas where routinely collected hearing health data could be useful. It demonstrates integration and standardisation of such data into a common data model from multiple centres. By describing the process of data sharing across the HIC, we hope to invite more centres to contribute and utilise data to address research questions in hearing health. This national initiative has the power to transform UK hearing research and hearing care using routinely collected clinical data.

Peer Review reports

An estimated 12 million UK adults have hearing loss. This is set to rise to 14.2 million by 2035 [ 1 , 2 ]. The total cost in the UK of untreated, disabling hearing loss is estimated at £25.5 billion annually [ 3 ]. Hearing loss affects functioning, communication [ 4 ], social interactions [ 5 ] and employment opportunities [ 6 ].

Despite hearing loss ranking third for Years Lived with Disability [ 7 ] and being the commonest sensory disorder [ 8 ], it receives less than 1% of UK research funding [ 9 ]. Research funding spent for hearing loss per individual is only £1, compared to £11 for sight loss. Efforts are underway to raise awareness and increase the budget for hearing research [ 10 ].

This article highlights how the National Institute of Health and Social Care Research (NIHR) Health Informatics Collaborative (HIC) for Hearing Health [ 11 ] has been established. We first list priority research areas where data could be useful, then outline the NIHR HIC Hearing Health’s aims, governance structure and study adoption processes. We demonstrate how hearing health data have been integrated into a common data model, using pure tone audiometry (PTA) as a case study. Finally, we detail how health data from contributing centres are ingested and stored.

Priority areas for data driven hearing research

Most estimates on the prevalence of hearing loss are from the 1980s [ 12 ]. There is a need for up-to-date data on the burden of hearing loss, causes, risk factors and predictors of progression so that new and effective treatments can be developed.

Hearing loss is unequally distributed, with people from lower socioeconomic and ethnic minority backgrounds being at higher risk [ 13 , 14 ]. Since these groups are less likely to seek interventions and participate in research, true risk may be underestimated, calling for novel approaches to include data from these groups.

Hearing loss has recently been linked to other chronic conditions such as dementia [ 15 ], diabetes [ 16 ] and falls [ 17 ]. Data-driven approaches could disentangle these associations and provide possible models of causation.

Hearing devices are the most common treatment for adults with hearing loss. The NHS is the largest purchaser of hearing aids worldwide, procuring 1.2 million annually. However, most people with aidable hearing loss never receive a hearing aid [ 18 ] and those that do may not always use them. Identifying patient, disease and device characteristics that predict who is most unlikely to be offered or use hearing devices would help devise strategies to improve uptake and usage.

Recent insights in genetic and molecular mechanisms causing hearing loss [ 19 ] have allowed detection of therapeutic targets and development of therapies aimed at protecting or restoring hearing [ 20 ]. These highly targeted treatments call for large scale geno- and phenotyping efforts to improve patient selection for upcoming clinical trials.

National Institute of Health and Social Care Research Health Informatics Collaborative.

The UK offers a unique infrastructure for data-driven research because 80% of all healthcare is provided by the National Health Service (NHS). This creates an unparalleled flow of routine health data across diverse ethnic and socio-economic groups [ 21 ]. To standardise and combine data across NHS providers, NIHR established its HIC; a collaboration between NHS trusts and their partner universities, hosted by the Biomedical Research Centres (BRCs) [ 22 ]. The NIHR HIC brings together clinical, scientific, and informatics expertise to support the establishment and maintenance of catalogued, comparable, and comprehensive flows of patient data at each Trust, and to create a governance framework for data sharing and re-use across the trusts and partner organisations.

NIHR HIC hearing heath theme

Recognising clinical need and opportunities for hearing health, the NIHR HIC Hearing Health theme was established in 2022. It aims to bring together the following routinely collected hearing health data and repurpose them for research. We interviewed NHS England, local commissioning groups, NHS procurement and NHS genomics to better understand hearing data collected through NHS organisations across England. Parameters included number of individuals undergoing hearing tests as part of the newborn hearing screen, volume of routine hearing consultations, number of audiometric assessments, number of hearing devices fitted and amount of requests for hearing panels. Based on responses received, the following was obtained:

New-born hearing screen

This national programme uses automated oto-acoustic emissions (OAE) at birth and auditory brainstem response (ABR) for those who fail or have specific risk factors. Since 2017, the UK has had between 680,000 and 750,000 births annually and a coverage of the new-born hearing screen of > 95%. This equates to hearing data of over three million patients, of which 80,000 failed the initial OAE screen and had ABR, and an estimated 5,000 who had confirmed permanent hearing loss.

Routine hearing loss consultations

Across the UK, 355,000 new consultations by audiologists and ENT surgeons are undertaken for hearing loss annually. These provide data on demographics, hearing symptoms, risk factors and interventions. Data are documented in electronic hospital records.

Audiometric assessments

Audiometric assessments inform the diagnosis of hearing loss and effects of interventions. Each of the 106 UK Clinical Commissioning Groups commissions 10,000–30,000 assessments annually, totalling 10 million hearing tests over five years, stored on NHS hospital or audiology clinic servers in codified format.

Hearing devices

The NHS is the largest procurer of hearing aids in the world, fitting new hearing aids for the first time on 355,000 adults annually. Additionally, 1,000 NHS patients receive cochlear implants annually, with over 12,000 NHS patients with cochlear implants so far. Data on the provision, maintenance, and use of these devices, as well as user, environmental and impedance data are stored on NHS servers.

Genetic testing for hearing loss

Since 2021, all children and adults in the UK with potential inherited causes of bilateral sensorineural hearing loss are eligible for genetic testing. There has been an increase in the number of hearing loss gene panels requested, with nearly 1000 requested in 2022.

NIHR HIC hearing health theme key topics

Between 2nd December 2020 and 16th January 2021, a stakeholder consultation took place online using a newly developed questionnaire on the platform Select Surveys (Appendix 1). It was aimed at clinicians, academic scientists and industry partners. They were asked to list any questions they would like to see formally investigated relating to diagnosis/assessment, treatment/ intervention and follow-up support. A total of 74 stakeholders (34 clinicians, 33 academic scientists, 7 industry partners) responded. The responses were condensed into four key topics:

Exploration of effects of known and novel risk factors such as disease clustering for hearing loss.

Identification of genetic causes of hearing loss.

Definition of hearing loss sub-types.

Optimisation of benefit from individualised treatment strategies.

NIHR HIC hearing health theme governance

The NIHR HIC Hearing Health theme is co-led by the founding BRCs: University College London Hospital Trust (UCLH), Nottingham University Hospitals Trust (NUH) and Manchester Foundation Trust (MFT), and their academic partners. Patients are key to decision-making processes. Specifically, regulation around the national data opt-out was directly informed by patient representatives. Patient public involvement (PPI) support is offered to all researchers submitting study requests to ensure that their research question is in line with patient priorities, and that their research study includes proportionate and meaningful PPI.

Data contribution and management

All NHS hearing health providers are encouraged to join the NIHR HIC Hearing Health theme as contributing centres, and contribute their locally stored, de-identified datasets to the central data repository, which is stored in a secure server at University College London (UCL).

The NIHR HIC Hearing Health theme does not allow any data to be removed once it has arrived in the data repository. All researchers at contributing centres are welcome to submit a study request for approval by the steering committee. Once the request is approved, the study is added to the Hearing Health HIC’s portfolio, and researchers are allowed to undertake research within the secure server at UCL.

Further detail is available here, which describes the framework through which we created this resource ( https://github.com/uclh-criu/hic-hearing-health-docs ) and specific code that relates to a licensed version of their electronic patient record can be accessed upon request.

The protocol for the collection and management of data was approved by Central Bristol Research Ethics Committee (Reference Number: 21/SW/0139).

NIHR data sharing agreement

All NHS hearing health providers can contribute data to the NIHR HIC Hearing Health theme under the NIHR data sharing framework. This covers a range of data and research collaborations and must be signed by all contributing centres. The NIHR HIC data sharing framework addresses common requirements and considerations regarding data sharing between centres, contractual responsibilities, confidentiality, intellectual property and a publications policy. This general agreement will underpin individual agreements for research collaborations with third party academic, clinical and industry partners. Any collaboration with industry partners requires additional agreements, with additional governance checks by participating sites. Industry partners will only be allowed to participate in research by collaborating with a contributing centre.

The HIC has applications to other international healthcare systems where existing relationships between hospitals and academic institutes exist, however given that it focuses on the UK based on the NIHR data sharing agreement, this is a limitation of this paper that its application is not directly transferrable and inevitably obstacles may be encountered in other counties based on local frameworks.

Study adoption

The process that allows hearing health researchers from contributing centres to undertake research on the NIHR HIC Hearing Health theme’s dataset is summarised in Fig. 1 .

Process of requesting approval to use NIHR HIC Hearing Health data for research

Researchers from contributing centres can request to undertake an analysis on existing data, or can request new data fields to be extracted from contributing centres to be added to the central data repository for the purpose of their study.

The steering committee reviews study requests from researchers at contributing centres. At the quarterly meeting, each proposal is reviewed by the contributing centres and PPI group, and a decision is made, through consensus amongst the steering committee, as to whether the study should be adopted, rejected, or sent back for further refinement. Studies may be rejected if they do not align with the key topics described above or are unachievable.

For studies that are to be adopted by the NIHR HIC Hearing Health theme, each centre is given the opportunity to submit their de-identified hospital data for pooled analysis. This offers each centre continued autonomy over their local data, irrespective of where the data are housed. This policy has been introduced following patient and key stakeholder input into anxieties over loss of autonomy over local data.

Researchers from the approved study centre will be onboarded to the central data repository and given access to the environment to analyse an excerpt of the dataset that is relevant to their study questions. Whilst the results of their analyses can be extracted from the secure environment, no raw data will be sent out.

Development of a common data model

Hearing healthcare data are stored across multiple platforms on local servers. Since 2002, all providers of NHS hearing aids have moved to patient management systems as part of the Department of Health and Social Care Modernising Hearing Aid Services; however, these management systems preceded electronic patient records and are not integrated with Trust electronic care records or with each other. No healthcare recording system currently exists that collects and stores all NHS hearing healthcare information within a single database. Each hospital has its own data flow and software specific databases. This is because a multitude of proprietary audiology-specific hard- and software products are used.

The variety of electronic patient record systems means that assessment and management data are stored across multiple software databases:

Diagnostic hearing tests are undertaken using locally purchased proprietary hardware, each with its own software and data model (assessment).

Hearing aid fitting is undertaken through manufacturer specific hardware, each with its own software and data model (management).

Patient management system stores patient data.

Linkage software communicates between the above software systems that stores its own versions of datasets.

The integration of assessment and management data is undertaken by a third-party software called NOAH, developed by Hearing Instrument Manufacturers’ Software Association (HIMSA). NOAH’s primary function is to provide a unified system managing data collected during a diagnostic hearing test, which can then be used by manufacturer-specific hearing aid fitting systems. NOAH software is built into patient management software, which captures appointment level data and allows a single portal to link assessment and management.

Locally, databases exist for diagnostic hearing tests, hearing aid fitting software, NOAH and clinical management systems, each storing part of the dataset (see Fig. 2 ). In addition to hearing-related information, general operational data around patient referrals, waiting times and staffing capacity are stored on bespoke databases. Medical information, including surgical interventions is captured on general hospital medical records, whilst imaging data is stored on Picture Archiving and Communication Systems (PACS).

Local databases for auditory assessment (green) and hearing aid fitting (blue) and patient management (orange) all need to be combined in a meaningful way

Combining all these different types of data in a meaningful way is challenging. The NIHR HIC Hearing Health theme have developed a data architecture outlying the flow of data from all the various siloed electronic patient record systems to allow the effective linkage of data from electronic patient record to research systems.

Data extraction, linkage and standardisation, within contributing centres.

Whilst each software package allows patient by patient data review and occasionally extraction, very few allow wholesale extraction. Using Open Database Connectivity (ODBC), an application programming interface, we have been able to extract all diagnostic hearing tests and hearing device fitting data stored on audiology patient management software, which generally are all based on SQL databases.

The architecture of the database, as well as data structure within each database, varies between software packages. We pursued manufacturers to release internal database architecture and data keys. This information was used to identify and decode key data fields, such as patient identifiers and clinical fields such as hearing test results.

Using probabilistic linkage algorithms, based on national ID, hospital ID, name, date of birth, we linked hearing data to hospital medical records. We prioritised demographic data from patient medical records if they conflicted with patient management software. Patients who signed up for the national data opt-out were removed from the cohort.

The formats in which demographics, diagnoses, and treatments are stored within separate databases within and across hospitals do not always match. Therefore, we opted to convert all data into a model using common data terminologies. We chose the Observational Medical Outcomes Partnership (OMOP) as our common data model. This is an international data model that enables the capture of information (e.g., encounters, patients, providers, diagnoses, drugs, devices, measurements and procedures) in the same way across different institutions. Its usefulness has been demonstrated in multiple health themes [ 23 ]. This model is coding language agnostic and maps across multiple vocabularies. Additionally, OMOP does not require a specific technology. It can be realised in any relational database, such as Oracle, SQL Server etc. or as SAS analytical datasets.

The NIHR HIC Hearing Health OMOP data model outlines the structure of the dataset and the associations between the data fields. Local vocabularies are mapped onto standardised OMOP vocabularies and labelled with OMOP domains.

To provide details on how we standardised data with OMOP, we have used hearing test data as an example.

A case study: omopification of hearing test data

The challenge.

PTA is the standard test of hearing [ 24 ]. It measures the lowest level (in Decibels) that a pure tone can be reliably heard at multiple sound frequencies in each ear (Hertz). The sound can be presented to the ear canal (air conduction), or onto the bone behind the ear (bone conduction). The non-tested ear can be deliberately presented with noise (masking) to prevent it from hearing sound presented to the test ear.

There are already several PTA-related concepts imported from various clinical vocabularies into the OMOP framework (Table 1 ). However, there is no architecture that inter-relates these codes. As such, new classifications and an inter-relational architecture were required.

The solution

We mapped test outcomes from PTA to OMOP by having a [Procedure_Occurrence] record for each test performed. The most appropriate OMOP procedure concept (from the Systematized Nomenclature of Medicine - SNOMED - vocabulary) was used in each case (e.g. 4091134 = Pure tone audiometry, or 4091877 = Soundfield audiometry). Multiple records from the [Observation] table were then related to the [Procedure_Occurrence], to describe results for each frequency (and ear) tested. Concepts such as masking level, air conduction or bone conduction could then be related to the [Observation], whereas concepts such as headphone (worn over the ear) or inserts (worn in the ear) could be related to the [Procedure Occurrence]. Fact relationship tables were used to communicate these relations [ 25 ]. These tables allow relationships to be defined between concepts from the same table or different tables. Figure 3 shows the structure of how codes were linked using fact relationships, using a few common examples. Whilst codes exist for most variables, some local codes were developed when no concept was previously available, for example to assign noise status (see Fig. 3 ).

Examples of linking structure of codes using fact relationships

Data pipeline for collection and integration of data, between contributing centres

The process of data flow from contributing centres to the central data repository is summarised in Fig. 4 . The HIC Data Alchemist platform has been designed to import data from contributing centres within the NIHR HIC Hearing Health theme into a centralised OMOP database. This platform was established to manage data provided in its most raw form, as CSV files compliant with RFC 4180, placing as little burden as possible on individual hospitals. Data is provided in data bundles, each of a different data type, allowing the contributing centre to progressively build their data contribution. Each data bundle focuses on a different goal. This stepwise process facilitates incremental development and feedback. To progress to the next data bundle, the previous data bundle must be completed to a specific standard. Compliance will be measured strictly, as any issues at an early stage may create a risk for further data bundles.

The different stages of data standardisation and integration from different types of data across a variety of platforms and providers within the field of hearing health research. There are two main steps:1. Step 1: generate local view of the uploaded CSV files for the sites to inspect and ensures it has been possible to load data into the database2. Step 2: updates and merges site’s existing data with newly ingested data, ensuring there are valid foreign keys (a system of connecting different tables of data), duplicates have been removed and deletions and updates have been processed to obtain the latest local version of a site’s data

Integration

Each data bundle undergoes a data ingestion process that runs from handling raw CSV files to integration within the multisite database (see Fig. 4 ).

Further details about the HIC Data Alchemist can be found at https://uclh-criu.github.io/hic-hearing-health-docs/ .

Data is stored in the UCL Data Safe Haven (DSH). This is an ISO27001 certified and NHS Data Security Protection Toolkit compliant Trusted Research Environment (TRE). Processing, analysis and storage capabilities exceed what is available to hospital-based researchers and include a full High Performance Computing cluster, cutting-edge GPU computing nodes for AI research, end-user environments with the latest analytical software and the facility to host any database or application server on enterprise-grade virtual machine infrastructure.

The NIHR HIC Hearing Health theme Research Database has been developed and is in the process of OMOPification. Data ingestion is underway. The data collected fall into the following categories: (a) basic information (e.g., demographics, hospital visits, death, discharge and study sites), (b) laboratory data, (c) treatments, (d) diagnoses, (e) hearing test data, (f) device data and (g) other clinical information.

An NIHR HIC Hearing Health theme has been established, to bring together routinely collected hearing health data, to address urgent research questions in an efficient and cost-effective way. After identifying priority research areas where these data could be useful, the NIHR HIC Hearing Health’s aims, governance structure and study adoption processes were identified. Key hearing health data can be integrated into a common data model. Health data from contributing centres are ingested and stored on a research ready database. Data across a variety of platforms and providers within the field of hearing health research can be standardised and integrated. By detailing the process of sharing data and submitting research proposals we hope to inspire more hearing-health researchers and NHS trusts to contribute to the database and use the wealth of its data to address urgent questions in hearing-health research. This national initiative has the power to transform UK hearing research and hearing care using routinely collected clinical data.

Data availability

All the raw data (including participants’ voice files and the texts of the interviews) will be confidential and will not be able to share publicly. However, the codes that emerged during the current study are available from the corresponding author upon reasonable request.

Royal National Institute for the Deaf. Facts and figures - RNID. https://rnid.org.uk/about-us/research-and-policy/facts-and-figures/ (accessed 12 Apr 2023).

Office of National Statistics. 2011 censuses UK comparability report. https://www.ons.gov.uk/census/planningforcensus2021/ukcensusdata (accessed 12 Apr 2023).

Shield B. Hearing Loss-Numbers and Costs EVALUATION OF THE SOCIAL AND ECONOMIC COSTS OF HEARING IMPAIRMENT A report for Hear-It AISBL. 2019.

Demorest ME, Erdman SA. Relationships among behavioral, environmental, and affective communication variables: a canonical analysis of the CPHI. J Speech Hear Disorders. 1989;54:180–8. https://doi.org/10.1044/jshd.5402.180 .

Article CAS Google Scholar

Meadow-Orlans KP. Social and psychological effects of hearing loss in adulthood: a literature review. Adjustment Adult Hear loss 1985;:35–57.

Shan A, Ting JS, Price C, et al. Hearing loss and employment: a systematic review of the association between hearing loss and employment among adults. J Laryngol Otol. 2020;134:387–97. https://doi.org/10.1017/S0022215120001012 .

Article CAS PubMed Google Scholar

Haile L, Orji A, Briant P, et al. Updates on hearing from the global burden of Disease Study. Innov Aging. 2020;4:808–808. https://doi.org/10.1093/geroni/igaa057.2934 .

Article PubMed Central Google Scholar

Raviv D, Dror AA, Avraham KB. Hearing loss: a common disorder caused by many rare alleles. Ann N Y Acad Sci. 2010;1214:168–79. https://doi.org/10.1111/j.1749-6632.2010.05868.x .

Article CAS PubMed PubMed Central Google Scholar

The Lancet. Calling for research into deafness. Lancet. 2007;370:1738. https://doi.org/10.1016/S0140-6736(07)61729-2 .

Article Google Scholar

Wilson BS, Tucci DL, O’Donoghue GM, et al. A Lancet Commission to address the global burden of hearing loss. Lancet. 2019;393:2106–8. https://doi.org/10.1016/S0140-6736(19)30484-2 .

Article PubMed Google Scholar

Health Informatics Collaborative – National Institute for Health Research. https://hic.nihr.ac.uk/ (accessed 12 Apr 2023).

Davis AC. The prevalence of hearing impairment and reported hearing disability among adults in Great Britain. 1389. https://academic.oup.com/ije/article/18/4/911/683421 .

Tsimpida D, Kontopantelis E, Ashcroft DM, et al. Regional patterns and trends of hearing loss in England: evidence from the English longitudinal study of ageing (ELSA) and implications for health policy. BMC Geriatr. 2020;20:536. https://doi.org/10.1186/s12877-020-01945-6 .

Article PubMed PubMed Central Google Scholar

Sawyer CS, Armitage CJ, Munro KJ, et al. Correlates of hearing Aid Use in UK adults. Ear Hear. 2019;40:1061–8. https://doi.org/10.1097/AUD.0000000000000695 .

Lin FR, Albert M. Hearing loss and dementia – who is listening? Aging Ment Health. 2014;18:671–3. https://doi.org/10.1080/13607863.2014.915924 .

Samocha-Bonet D, Wu B, Ryugo DK. Diabetes mellitus and hearing loss: a review. Ageing Res Rev. 2021;71:101423. https://doi.org/10.1016/j.arr.2021.101423 .

Lin FR. Hearing loss and Falls among older adults in the United States. Arch Intern Med. 2012;172:369. https://doi.org/10.1001/archinternmed.2011.728 .

Chien W. Prevalence of hearing Aid Use among older adults in the United States. Arch Intern Med. 2012;172:292. https://doi.org/10.1001/archinternmed.2011.1408 .

Melgar-Rojas P, Alvarado JC, Fuentes-Santamaría V et al. Cellular mechanisms of Age-related hearing loss. 2015. 305–33. https://doi.org/10.1007/978-3-319-13473-4_15 .

Schilder AGM, Su MP, Mandavia R, et al. Early phase trials of novel hearing therapeutics: avenues and opportunities. Hear Res. 2019;380:175–86. https://doi.org/10.1016/j.heares.2019.07.003 .

Rai T, Dixon S, Ziebland S. Shifting research culture to address the mismatch between where trials recruit and where populations with the most disease live: a qualitative study. BMC Med Res Methodol. 2021;21:80. https://doi.org/10.1186/s12874-021-01268-z .

Experimental medicine | NIHR. https://www.nihr.ac.uk/explore-nihr/support/experimental-medicine.htm (accessed 12 Apr 2023).

Reinecke I, Zoch M, Reich C et al. The Usage of OHDSI OMOP – A Scoping Review. 2021. https://doi.org/10.3233/SHTI210546 .

ISO 8253-1. 2010(en), Acoustics — Audiometric test methods — Part 1: Pure-tone air and bone conduction audiometry. https://www.iso.org/obp/ui/#iso:std:iso:8253:-1:ed-2:v1:en (accessed 12 Apr 2023).

documentation:cdm:single. -page [Observational Health Data Sciences and Informatics]. https://www.ohdsi.org/web/wiki/doku.php?id=documentation:cdm:single-page (accessed 12 Apr 2023).

Download references

Acknowledgements

NIHR BRC colleagues at University of Manchester, University College London and University of Nottingham.

Not applicable.

Author information

Authors and affiliations.

NIHR University College London Hospitals Biomedical Research Centre, London, UK

Nishchay Mehta, Baptiste Briot Ribeyre, Lilia Dimitrov, Louise J. English, Nikhil Joshi, Luis Romao, Anne Gm Schilder & Talisa Ross

Royal National ENT Hospital, UCLH Trust, London, UK

Nishchay Mehta, Lilia Dimitrov, Nikhil Joshi, Anne Gm Schilder & Talisa Ross

Institute of Health Informatics, Faculty of Population Health Sciences, University College London, London, UK

Baptiste Briot Ribeyre, Louise J. English & Luis Romao

NIHR Health Informatics Collaborative Hearing Health, Patient and Public Engagement Group, London, UK

Colleen Ewart

NIHR Manchester Biomedical Research Centre, Manchester, UK

Antje Heinrich, Kevin J. Munro, Ruth Norris & George Tilston

Manchester Centre for Audiology and Deafness (ManCAD), School of Health Sciences, The University of Manchester, Manchester, UK

Antje Heinrich & Kevin J. Munro

Nottingham Audiology Services, Nottingham University Hospitals, Nottingham, UK

Talisa Ross

Oxford University Hospitals NHS Foundation Trust: Oxford, Oxfordshire, UK

Gail Roadknight

NIHR Biomedical Research Centre, School of Medicine, University of Nottingham, Nottingham, UK

Ruth V. Spriggs

Mental Health and Clinical Neurosciences, School of Medicine, University of Nottingham, Nottingham, UK

Centre for Health Informatics, School of Health Sciences, The University of Manchester, Manchester, UK

Ruth Norris & George Tilston

You can also search for this author in PubMed Google Scholar

Contributions

Planning – NM, BBR, LD, LJE, CE, AH, NJ, KM, GR, LR, AS, RVS, RN, TR, GTConception and design– NM, BBR, LD, LJE, CE, AH, NJ, KM, GR, LR, AS, RVS, RN, TR, GTAcquisition of data– NM, BBR, LD, LJE, CE, AH, NJ, KM, GR, LR, AS, RVS, RN, TR, GTAnalysis– NM, BBR, LD, LJE, CE, AH, NJ, KM, GR, LR, AS, RVS, RN, TR, GTWrite up– NM, BBR, LD, LJE, CE, AH, NJ, KM, GR, LR, AS, RVS, RN, TR, GT.

Corresponding author

Correspondence to Talisa Ross .

Ethics declarations

Ethics approval and consent to participate.

The protocol for the collection and management of data was approved by Central Bristol Research Ethics Committee (Reference Number: 21/SW/0139) and carried out according to the Declaration of Helsinki. The lack of need for informed consent was confirmed by the ethics committee due to the lack of identifiable patient data being included.

Consent for publication

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Electronic supplementary material

Below is the link to the electronic supplementary material.

Supplementary Material 1

Rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License, which permits any non-commercial use, sharing, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if you modified the licensed material. You do not have permission under this licence to share adapted material derived from this article or parts of it.The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder.To view a copy of this licence, visit http://creativecommons.org/licenses/by-nc-nd/4.0/ .

Reprints and permissions

About this article

Cite this article.

Mehta, N., Ribeyre, B.B., Dimitrov, L. et al. Creating a health informatics data resource for hearing health research. BMC Med Inform Decis Mak 24 , 209 (2024). https://doi.org/10.1186/s12911-024-02589-x

Download citation

Received : 19 November 2023

Accepted : 25 June 2024

Published : 29 July 2024

DOI : https://doi.org/10.1186/s12911-024-02589-x

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Data sharing
Hearing health

BMC Medical Informatics and Decision Making

ISSN: 1472-6947

General enquiries: [email protected]

Reconsider Machine Learning Method for Variable Selection and Validation with High Dimensional Data

Event sponsored by:

IMAGES

Nurse documentation and the electronic health record
Electronic Health Record (EHR) Implementation Checklist
Nurse documentation and the electronic health record
Electronic Health Record (EHR) and use in clinical trials
CODE-EHR best-practice framework for the use of structured electronic
Why is Medical Documentation with EHRs Important in Practices?

VIDEO

Electronic Medical Records are used to review medical history. clinicalresearch
Electronic Health Records Management Using Blockchain
The Mayo Clinic Approach to Managing Its EHR System
Medical Documentation in the EHR Age
Medical records and data driven healthcare
Electronic Systems, Electronic Records, and Electronic Signatures Webinar

COMMENTS

Clinical Documentation in Electronic Health Record Systems: Analysis of Patient Record Review During Outpatient Ophthalmology Visits
Introduction. Electronic health record systems (EHRs) have become an integral part of healthcare within the past decade. As of 2015, 87% of all office-based providers in the United States had an EHR. 1 Adoption within Non-Federal Acute Care Hospitals is almost universal, with over 96% possessing a certified EHR. 2 While clinicians recognize the benefits of EHR usage 3-5, they have concerns ...
Improving clinical documentation: introduction of electronic health
Background. Clinical documentation is the process of generating a structured record of a patient's medical care. This record of medical information assists in optimising patient care, enables communication between health professionals involved in the patient's care, supplies data for audits and research as well as serves as potential evidence for medico-legal cases.
Unlocking the Potential of Electronic Health Records for Health Research
Electronic Health Records (EHRs) are systemized collections of patient health information and documentation, collected in real-time, and stored in a digital format [ 1 ]. EHRs were originally designed to facilitate clinical decision-making regarding health care delivery for individual patients, and to improve the quality of care. EHRs have seen ...
Integration of clinical research documentation in electronic health records
Although regulations and guidelines define both source documentation and the medical record, integration of research documentation in the electronic health record is not clearly defined. At minimum, the signed informed consent (s), investigational drug or device usage, and research team contact information should be documented within the ...
Integration of Clinical Research Documentation in Electronic Health Records
The electronic medical record (EMR) and electronic health record (EHR) are platforms utilized within both clinical care and clinical research settings to collect data ultimately used by a clinical ...
How to Maintain Excellent Clinical Documentation
Electronic health record systems have become common in the workplace. Research has shown that the EHR can increase efficiency, improve patient safety, and facilitate improved access to patient records [4, 5].Spending time up front to customize your EHR platform can streamline your workflow and allow you to more efficiently document your clinical care [].
Improving clinical documentation: introduction of electronic health
Electronic patient records (EPR) have been gradually introduced to replace patient's paper notes with the aim of providing a more reliable record-keeping system. It is perceived that EPR improve the quality and efficiency of patient care. The paediatric department at Queen's Hospital Burton uses a mix of paper notes and computerised medical ...
Measurement of clinical documentation burden among physicians and
INTRODUCTION. Rapid adoption of electronic health records (EHRs) following the passage of the Health Information Technology for Economic and Clinical Health (HITECH) Act has led to advances in both individual- and population-level health. 1 HITECH has improved healthcare quality, patient safety, and diagnostic accuracy through enhanced data management and timely reuse; interoperable systems ...
Clinical documentation integrity: Its role in health data integrity
Clinical documentation has been defined as "information that is recorded about a person's care. The primary purpose of clinical documentation is to facilitate, safe, high quality and continuous care. . .and is stored within a health record" (Australian Commission on Safety and Quality in Healthcare, 2023).It needs to accurately reflect clinical events and decision-making for purposes of ...
Electronic Documentation in Pediatrics: The Rationale and Functionality
Clinical documentation has dramatically changed since the implementation and use of electronic health records and electronic provider documentation. The purpose of this report is to review these changes and promote the development of standards and best practices for electronic documentation for pediatric patients. In this report, we evaluate the unique aspects of clinical documentation for ...
The Importance and Use of Electronic Health Records in Clinical Research
The electronic medical record (EMR) and electronic health record (EHR) are platforms utilized within both clinical care and clinical research settings to collect data ultimately used by a clinical research information system (CRIS) to support clinical research. This chapter will review the concepts of the EMR, EHR, CRIS; the architecture of an ...
Toward Reliable Symptom Coding in Electronic Health Records for Symptom
To date, symptom documentation has mostly relied on clinical notes in electronic health records or patient-reported outcomes using disease-specific symptom inventories.
Use of Electronic Health Record Data in Clinical Investigations
Print. Docket Number: FDA-2016-D-1224. Issued by: Center for Drug Evaluation and Research. Center for Biologics Evaluation and Research. Center for Devices and Radiological Health. Procedural ...
PDF Use of Electronic Health Record Data in Clinical Investigations
SCOPE. The recommendations outlined in this guidance apply to the use of EHR data in: Prospective clinical investigations of human drugs and biological products, medical devices, and combination ...
Nurse documentation and the electronic health record
Takeaways: If not used properly, the electronic health record (EHR) can create communication gaps. The nursing process can be applied to electronic documentation to avoid workarounds and close gaps in communication. Effective use the EHR can improve patient safety and care outcomes. Clinical documentation supports patient care, improves ...
Effects of Electronic Health Record Implementation and Barriers to
1. Introduction. In the early 1990s, a trend in the shift from paper-based health records to electronic records started; this was in response to advances in technology as well as the advocacy of the Institute of Medicine in the United States [1,2].As a result of the inadequacies of paper-based health records gradually becoming evident to the healthcare industry [], electronic records have ...
Electronic health record
Sample view of an electronic health record. An electronic health record (EHR) is the systematized collection of patient and population electronically stored health information in a digital format. These records can be shared across different health care settings. Records are shared through network-connected, enterprise-wide information systems or other information networks and exchanges.
Measuring Documentation Burden in Healthcare
Background The enactment of the Health Information Technology for Economic and Clinical Health Act and the wide adoption of electronic health record (EHR) systems have ushered in increasing documentation burden, frequently cited as a key factor affecting the work experience of healthcare professionals and a contributor to burnout. This systematic review aims to identify and characterize ...
PDF Improving clinical documentation: introduction of electronic health
Improving clinical documentation: introduction of electronic health records in paediatrics Justin Koh, Mansoor Ahmed To cite: Koh J, Ahmed M. Improving clinical documentation: introduction of electronic health records in paediatrics. BMJ Open Quality 2021;10:e000918. doi:10.1136/ bmjoq-2020-000918 Received 4 February 2020 Revised 25 January 2021
National Electronic Health Records Survey
The National Electronic Health Records Survey (NEHRS) is an annual survey that measures the progress U.S. physicians and their offices have made in adopting electronic health records (EHRs). The survey also monitors health information exchange, including interoperability, and the burden associated with EHRs.
TGH adopts Nuance's AI-powered clinical documentation tech
Earlier this year, the technology was integrated into the Epic electronic health record system. As per studies cited by TGH, DAX Copilot slashes documentation time by 50%.
Optimize RCM for value-based care
1. Accurate and complete clinical documentation. Proper documentation is crucial for value-based care reimbursement. Providers need to ensure that all relevant patient information, including diagnoses, procedures, and outcomes, are accurately and completely documented within electronic health records to support appropriate medical coding and ...
Strategies used to detect and mitigate system-related errors over time
An electronic medical record (EMR) provides access to longitudinal patient data and clinical information in a timely and convenient manner, [] while allowing clinicians to prescribe, review and administer medications on a single digital platform, often with the assistance of clinical decision support.Although the use of EMR systems results in fewer medication errors, [] they can also create ...
Attitudes of Mental Health Service Users Toward Storage and Use of
Objective: Electronic health records (EHRs) are used for both clinical practice and research. Because mental health service users' views are underrepresented in perspectives on EHR use, the authors examined service users' awareness, attitudes, and opinions about EHR data storage and sharing. Methods: A mixed-methods, cross-sectional design was used to examine attitudes of 253 Norwegian ...
Primary care health professionals' approach to clinical coding: a
Background: Clinical coding allows for structured and standardised recording of data in patients electronic healthcare records. How clinical and non-clinical staff in general practice approach clinical coding is poorly understood. Aim: To explore primary care staff's experiences and views on clinical coding. Design and setting: Qualitative, semi-structured interview study with primary care ...
Electronic health records to facilitate clinical research
Abstract. Electronic health records (EHRs) provide opportunities to enhance patient care, embed performance measures in clinical practice, and facilitate clinical research. Concerns have been raised about the increasing recruitment challenges in trials, burdensome and obtrusive data collection, and uncertain generalizability of the results.
Real-World Data: Assessing Electronic Health Records and Medical Claims
This guidance is intended to provide sponsors and other interested parties with considerations when proposing to use electronic health records (EHRs) or medical claims data in clinical studies to support a regulatory decision for effectiveness or safety. This guidance finalizes the draft guidance of the same title issued on September 30, 2021 ...
Large language models for accurate disease detection in electronic
Importance: The use of large language models (LLMs) in medicine is increasing, with potential applications in electronic health records (EHR) to create patient cohorts or identify patients who meet clinical trial recruitment criteria. However, significant barriers remain, including the extensive computer resources required, lack of performance evaluation, and challenges in implementation.
Creating a health informatics data resource for hearing health research
The National Institute of Health and Social Care Research (NIHR) Health Informatics Collaborative (HIC) for Hearing Health has been established in the UK to curate routinely collected hearing health data to address research questions. This study defines priority research areas, outlines its aims, governance structure and demonstrates how hearing health data have been integrated into a common ...
Reconsider Machine Learning Method for Variable Selection and
Machine learning is also a popular topic in clinical medicine to implement analysis on electronic health records and medical image data, which traditional statistics model is not adequate for. However, we realize that machine learning is not panacea and its defects such as loss of interpretability and excess selection may restrict its application.

Save citation to file

Add to My Bibliography

Improving clinical documentation: introduction of electronic health records in paediatrics

Conflict of interest statement

Similar articles

Related information

Other Literature Sources

Article Contents

Measurement of clinical documentation burden among physicians and nurses using electronic health records: a scoping review

Search strategy and selection criteria

Study selection and selection criteria

Data extraction and analysis

Sources of evidence

Study characteristics

Characterization of effort

Measurement of time

Outcome assessment

Limitations and biases reported

Limitations

Future directions

Supplementary data

Email alerts

Affiliations

This Feature Is Available To Subscribers Only

Toward Reliable Symptom Coding in Electronic Health Records for Symptom Assessment and Research: Identification and Categorization of International Classification of Diseases, Ninth Revision, Clinical Modification Symptom Codes

No full-text available

Nurse documentation and the electronic health record

Confirming suspicions

Identifying the problems

Reviewing the outcomes

Staying in charge

Selected references

3 Comments .

NurseLine Newsletter

Test Your Knowledge

Oncology nursing: Clinical assistant onboarding

Medication safety and pediatric health

Confidence is in

Engaging nurses in scholarly work

Sepsis navigators and hospital outcomes

Electronic peer review and CLABSI rates

Tips for thriving–not just surviving–on the night shift

Cesarean birth reduction: EBP in action

Discharge hospitality center

LPNs in modified care delivery models

Patient care assistant training

Magnet4Europe

Innovation at the frontline of nursing

Partnering for nursing advancement

Journey of a novice Magnet® program director

Measuring Documentation Burden in Healthcare

Cite this article

Access this article

Similar content being viewed by others

Temporal Associations Between EHR-Derived Workload, Burnout, and Errors: a Prospective Cohort Study

Frontline Perspectives on Physician Burnout and Strategies to Improve Well-Being: Interviews with Physicians and Health System Leaders

The Electronic Health Record

Acknowledgements:

Author information

Corresponding author

Ethics declarations

Disclaimer:

Additional information

Supplementary Information

Supplementary file1 (DOCX 478 KB)

About this article

Share this article

National Center for Health Statistics

Selected for NEHRS?‎

What's collected

Data and documentation

Findings and reports‎

How the data are used

What's new from NEHRS

Select a topic

Optimizing revenue cycle management for value-based care success

Healthcare organizations that use mix of payment models feel more financially secure

Value-based care success hinges on effective use of an integrated healthcare IT platform

Top 5 revenue cycle management considerations for value-based care success

1. Accurate and complete clinical documentation