research review journal impact factor

• All subject areas
• Agricultural and Biological Sciences
• Arts and Humanities
• Biochemistry, Genetics and Molecular Biology
• Business, Management and Accounting
• Chemical Engineering
• Computer Science
• Decision Sciences
• Earth and Planetary Sciences
• Economics, Econometrics and Finance
• Engineering
• Environmental Science
• Health Professions
• Immunology and Microbiology
• Materials Science
• Mathematics
• Multidisciplinary
• Neuroscience
• Pharmacology, Toxicology and Pharmaceutics
• Physics and Astronomy
• Social Sciences
• All subject categories
• Acoustics and Ultrasonics
• Advanced and Specialized Nursing
• Aerospace Engineering
• Agricultural and Biological Sciences (miscellaneous)
• Agronomy and Crop Science
• Algebra and Number Theory
• Analytical Chemistry
• Anesthesiology and Pain Medicine
• Animal Science and Zoology
• Anthropology
• Applied Mathematics
• Applied Microbiology and Biotechnology
• Applied Psychology
• Aquatic Science
• Archeology (arts and humanities)
• Architecture
• Artificial Intelligence
• Arts and Humanities (miscellaneous)
• Assessment and Diagnosis
• Astronomy and Astrophysics
• Atmospheric Science
• Atomic and Molecular Physics, and Optics
• Automotive Engineering
• Behavioral Neuroscience
• Biochemistry
• Biochemistry, Genetics and Molecular Biology (miscellaneous)
• Biochemistry (medical)
• Bioengineering
• Biological Psychiatry
• Biomaterials
• Biomedical Engineering
• Biotechnology
• Building and Construction
• Business and International Management
• Business, Management and Accounting (miscellaneous)
• Cancer Research
• Cardiology and Cardiovascular Medicine
• Care Planning
• Cell Biology
• Cellular and Molecular Neuroscience
• Ceramics and Composites
• Chemical Engineering (miscellaneous)
• Chemical Health and Safety
• Chemistry (miscellaneous)
• Chiropractics
• Civil and Structural Engineering
• Clinical Biochemistry
• Clinical Psychology
• Cognitive Neuroscience
• Colloid and Surface Chemistry
• Communication
• Community and Home Care
• Complementary and Alternative Medicine
• Complementary and Manual Therapy
• Computational Mathematics
• Computational Mechanics
• Computational Theory and Mathematics
• Computer Graphics and Computer-Aided Design
• Computer Networks and Communications
• Computer Science Applications
• Computer Science (miscellaneous)
• Computer Vision and Pattern Recognition
• Computers in Earth Sciences
• Condensed Matter Physics
• Conservation
• Control and Optimization
• Control and Systems Engineering
• Critical Care and Intensive Care Medicine
• Critical Care Nursing
• Cultural Studies
• Decision Sciences (miscellaneous)
• Dental Assisting
• Dental Hygiene
• Dentistry (miscellaneous)
• Dermatology
• Development
• Developmental and Educational Psychology
• Developmental Biology
• Developmental Neuroscience
• Discrete Mathematics and Combinatorics
• Drug Discovery
• Drug Guides
• Earth and Planetary Sciences (miscellaneous)
• Earth-Surface Processes
• Ecological Modeling
• Ecology, Evolution, Behavior and Systematics
• Economic Geology
• Economics and Econometrics
• Economics, Econometrics and Finance (miscellaneous)
• Electrical and Electronic Engineering
• Electrochemistry
• Electronic, Optical and Magnetic Materials
• Emergency Medical Services
• Emergency Medicine
• Emergency Nursing
• Endocrine and Autonomic Systems
• Endocrinology
• Endocrinology, Diabetes and Metabolism
• Energy Engineering and Power Technology
• Energy (miscellaneous)
• Engineering (miscellaneous)
• Environmental Chemistry
• Environmental Engineering
• Environmental Science (miscellaneous)
• Epidemiology
• Experimental and Cognitive Psychology
• Family Practice
• Filtration and Separation
• Fluid Flow and Transfer Processes
• Food Animals
• Food Science
• Fuel Technology
• Fundamentals and Skills
• Gastroenterology
• Gender Studies
• Genetics (clinical)
• Geochemistry and Petrology
• Geography, Planning and Development
• Geometry and Topology
• Geotechnical Engineering and Engineering Geology
• Geriatrics and Gerontology
• Gerontology
• Global and Planetary Change
• Hardware and Architecture
• Health Informatics
• Health Information Management
• Health Policy
• Health Professions (miscellaneous)
• Health (social science)
• Health, Toxicology and Mutagenesis
• History and Philosophy of Science
• Horticulture
• Human Factors and Ergonomics
• Human-Computer Interaction
• Immunology and Allergy
• Immunology and Microbiology (miscellaneous)
• Industrial and Manufacturing Engineering
• Industrial Relations
• Infectious Diseases
• Information Systems
• Information Systems and Management
• Inorganic Chemistry
• Insect Science
• Instrumentation
• Internal Medicine
• Issues, Ethics and Legal Aspects
• Leadership and Management
• Library and Information Sciences
• Life-span and Life-course Studies
• Linguistics and Language
• Literature and Literary Theory
• LPN and LVN
• Management Information Systems
• Management, Monitoring, Policy and Law
• Management of Technology and Innovation
• Management Science and Operations Research
• Materials Chemistry
• Materials Science (miscellaneous)
• Maternity and Midwifery
• Mathematical Physics
• Mathematics (miscellaneous)
• Mechanical Engineering
• Mechanics of Materials
• Media Technology
• Medical and Surgical Nursing
• Medical Assisting and Transcription
• Medical Laboratory Technology
• Medical Terminology
• Medicine (miscellaneous)
• Metals and Alloys
• Microbiology
• Microbiology (medical)
• Modeling and Simulation
• Molecular Biology
• Molecular Medicine
• Nanoscience and Nanotechnology
• Nature and Landscape Conservation
• Neurology (clinical)
• Neuropsychology and Physiological Psychology
• Neuroscience (miscellaneous)
• Nuclear and High Energy Physics
• Nuclear Energy and Engineering
• Numerical Analysis
• Nurse Assisting
• Nursing (miscellaneous)
• Nutrition and Dietetics
• Obstetrics and Gynecology
• Occupational Therapy
• Ocean Engineering
• Oceanography
• Oncology (nursing)
• Ophthalmology
• Oral Surgery
• Organic Chemistry
• Organizational Behavior and Human Resource Management
• Orthodontics
• Orthopedics and Sports Medicine
• Otorhinolaryngology
• Paleontology
• Parasitology
• Pathology and Forensic Medicine
• Pathophysiology
• Pediatrics, Perinatology and Child Health
• Periodontics
• Pharmaceutical Science
• Pharmacology
• Pharmacology (medical)
• Pharmacology (nursing)
• Pharmacology, Toxicology and Pharmaceutics (miscellaneous)
• Physical and Theoretical Chemistry
• Physical Therapy, Sports Therapy and Rehabilitation
• Physics and Astronomy (miscellaneous)
• Physiology (medical)
• Plant Science
• Political Science and International Relations
• Polymers and Plastics
• Process Chemistry and Technology
• Psychiatry and Mental Health
• Psychology (miscellaneous)
• Public Administration
• Public Health, Environmental and Occupational Health
• Pulmonary and Respiratory Medicine
• Radiological and Ultrasound Technology
• Radiology, Nuclear Medicine and Imaging
• Rehabilitation
• Religious Studies
• Renewable Energy, Sustainability and the Environment
• Reproductive Medicine
• Research and Theory
• Respiratory Care
• Review and Exam Preparation
• Reviews and References (medical)
• Rheumatology
• Safety Research
• Safety, Risk, Reliability and Quality
• Sensory Systems
• Signal Processing
• Small Animals
• Social Psychology
• Social Sciences (miscellaneous)
• Social Work
• Sociology and Political Science
• Soil Science
• Space and Planetary Science
• Spectroscopy
• Speech and Hearing
• Sports Science
• Statistical and Nonlinear Physics
• Statistics and Probability
• Statistics, Probability and Uncertainty
• Strategy and Management
• Stratigraphy
• Structural Biology
• Surfaces and Interfaces
• Surfaces, Coatings and Films
• Theoretical Computer Science
• Tourism, Leisure and Hospitality Management
• Transplantation
• Transportation
• Urban Studies
• Veterinary (miscellaneous)
• Visual Arts and Performing Arts
• Waste Management and Disposal
• Water Science and Technology
• All regions / countries
• Asiatic Region
• Eastern Europe
• Latin America
• Middle East
• Northern America
• Pacific Region
• Western Europe
• ARAB COUNTRIES
• IBEROAMERICA
• NORDIC COUNTRIES
• Afghanistan
• Bosnia and Herzegovina
• Brunei Darussalam
• Czech Republic
• Dominican Republic
• Netherlands
• New Caledonia
• New Zealand
• Papua New Guinea
• Philippines
• Puerto Rico
• Russian Federation
• Saudi Arabia
• South Africa
• South Korea
• Switzerland
• Syrian Arab Republic
• Trinidad and Tobago
• United Arab Emirates
• United Kingdom
• United States
• Vatican City State
• Book Series
• Conferences and Proceedings
• Trade Journals

• Citable Docs. (3years)
• Total Cites (3years)

-->

	Title	Type
1		journal	106.094 Q1	211	49	124	4844	35427	89	381.89	98.86	43.95
2		journal	37.044 Q1	39	3	13	897	955	13	100.11	299.00	27.78
3		journal	35.910 Q1	508	123	336	11462	13599	153	34.50	93.19	29.41
4		journal	30.448 Q1	306	47	136	3645	2240	136	11.14	77.55	26.67
5		journal	26.837 Q1	505	105	304	10805	10951	163	31.23	102.90	44.33
6		journal	24.342 Q1	892	439	1496	32820	53447	1207	31.30	74.76	40.19
7		journal	22.399 Q1	391	239	731	8584	13091	153	19.72	35.92	34.15
8		journal	22.344 Q1	359	95	353	6242	4811	351	10.33	65.71	23.89
9		journal	21.836 Q1	184	117	335	8842	13775	196	31.17	75.57	26.86
10		journal	21.048 Q1	217	127	400	9888	10807	183	28.36	77.86	38.85
11		journal	20.544 Q1	1184	1388	4522	14603	107246	1824	21.69	10.52	38.26
12		journal	19.139 Q1	352	83	227	5043	1938	221	7.00	60.76	16.91
13		journal	19.045 Q1	630	595	1363	16478	36243	729	27.23	27.69	43.99
14		journal	18.663 Q1	71	0	19	0	963	19	0.00	0.00	0.00
15		journal	18.587 Q1	23	11	16	0	802	16	47.57	0.00	81.69
16		journal	18.530 Q1	215	83	261	4493	2531	258	7.04	54.13	17.80
17		journal	18.509 Q1	1391	3770	8037	74917	160102	3840	19.40	19.87	38.12
18		journal	18.117 Q1	511	485	1066	13393	17008	461	13.24	27.61	35.19
19		journal	17.828 Q1	833	271	851	115878	50519	819	49.76	427.59	30.50
20		journal	17.701 Q1	223	75	273	3371	1946	268	6.24	44.95	13.84
21		journal	17.654 Q1	234	108	410	6448	4495	409	8.04	59.70	16.43
22		journal	17.507 Q1	398	178	590	11546	12604	360	19.83	64.87	41.91
23		journal	17.497 Q1	229	229	609	6629	16808	379	26.18	28.95	29.53
24		journal	17.300 Q1	639	336	654	13672	13100	504	19.88	40.69	37.01
25		journal	16.061 Q1	388	36	103	14097	4303	99	42.66	391.58	14.94
26		journal	16.009 Q1	467	169	540	11148	13815	304	23.17	65.96	36.44
27		journal	15.966 Q1	264	102	252	19168	11266	244	38.64	187.92	24.30
28		journal	15.827 Q1	140	106	297	4359	4041	62	12.99	41.12	41.35
29		journal	15.620 Q1	328	23	84	4178	2696	83	27.02	181.65	40.68
30		journal	14.943 Q1	115	16	42	403	896	41	24.10	25.19	77.78
31		journal	14.796 Q1	388	400	978	11477	15900	588	17.52	28.69	33.83
32		journal	14.780 Q1	123	0	13	0	374	11	12.56	0.00	0.00
33		journal	14.707 Q1	32	46	35	4815	2160	34	61.71	104.67	36.44
34		journal	14.618 Q1	160	70	247	587	5353	230	21.11	8.39	58.79
35		journal	14.605 Q1	109	23	72	5797	1938	70	14.90	252.04	45.57
36		journal	14.577 Q1	419	262	637	10044	17562	466	27.42	38.34	28.93
37		journal	14.293 Q1	421	123	346	10202	6211	207	17.40	82.94	32.86
38		journal	14.231 Q1	558	306	834	9499	20730	593	24.08	31.04	24.85
39		journal	14.175 Q1	210	28	92	3163	1260	86	10.59	112.96	42.59
40		journal	13.942 Q1	294	144	670	5180	12698	362	18.81	35.97	39.02
41		book series	13.670 Q1	210	14	42	3772	1271	39	23.96	269.43	26.09
42		journal	13.655 Q1	311	89	563	4857	6315	559	11.14	54.57	23.11
43		journal	13.609 Q1	165	93	250	5332	1699	250	6.02	57.33	15.88
44		journal	13.578 Q1	455	233	688	15608	13409	550	16.89	66.99	40.35
45		journal	13.315 Q1	136	180	471	6682	12109	368	24.27	37.12	26.28
46		journal	13.080 Q1	260	243	827	1865	14374	679	16.58	7.67	62.53
47		journal	12.511 Q1	635	252	983	61439	44032	979	38.71	243.81	32.40
48		journal	12.324 Q1	81	55	137	2838	862	137	6.20	51.60	17.36
49		journal	12.294 Q1	46	62	154	8174	4174	86	27.10	131.84	31.14
50		journal	12.288 Q1	44	60	79	4858	3323	78	42.06	80.97	33.06

Follow us on @ScimagoJR Scimago Lab , Copyright 2007-2024. Data Source: Scopus®

Cookie settings

Cookie Policy

Legal Notice

Privacy Policy

What is a Good Impact Factor for a Journal?

• Research Process

A good journal impact factor (IF) is often the main consideration for researchers when they’re looking for a place to publish their work. Many researchers assume that a high impact factor indicates a more prestigious journal.

Updated on February 22, 2024

A good journal impact factor (IF) is often the main consideration for researchers when they’re looking for a place to publish their work. Many researchers assume that a high impact factor indicates a more prestigious journal. And that means more recognition for the manuscript author(s).

So, by that logic, the higher the impact factor, the better the journal, right?

Well, it’s not that simple.

In principle, a higher IF is better than a lower IF, but there are many conditions, variations, and other issues to consider.

There’s no single determinant of what makes a good journal impact factor. It depends on the field of research, and what you mean by “good.” What is “good” for a breakthrough immunology study may not apply as “good” for an incremental regional economics study.

Using impact factors in the academic world to rank journals remains controversial. The San Francisco Declaration on Research Assessment (DORA) , for example, tried to tackle the issue of over-reliance on journal IFs when evaluating published research.

Yet researchers continue to associate a good IF with better quality research. So, until DORA or others develop a better solution, we’re stuck with the IF, simplistic as it may be.

Read on to increase your understanding of impact factors and learn what’s a good one for your research.

First, what’s an impact factor?

A journal impact factor is a metric that assesses the citation rate of articles published in a particular journal over a specific time – that’s usually 2 years (see below).

For example, an IF of 3 means that published articles have been cited on average 3 times during the previous 2 years.

How impact factors are calculated

The IF for a particular year is calculated as the ratio of the total times the journal’s articles were cited in the previous 2 years to the total citable items it published in those 2 years.

For example, in 2018, Nature had an IF of 43.070. That's a good journal impact factor. This is calculated as follows:

(Adapted from https://clarivate.libguides.com/jcr )

(Source: 2018 Journal Citation Reports)

Clarivate Analytics annually computes IFs for journals indexed in Web of Science. These scores are then collectively published in the Journal Citation Reports (JCR) database.

Clarivate publishes two different JCR databases every year. The Science Citation Index (SCI) is for the STEM (science, technology, engineering, and mathematics) disciplines, and the Social Science Citation Index (SSCI) is for, you guessed it, the social sciences. These are the only acceptable and reputable sources for impact factors out there: If your journal is using another index, then beware – it could be predatory .

Types of impact factors and metrics

In addition to the 2-year impact factor, Clarivate offers metrics for short-, medium-, and long-term analysis of a JCR journal’s performance. These metrics include:

• Immediacy index – Average number of times an article is cited during the same year it’s published.
• Citing and cited half-life – Median age of citations produced and received by a journal, respectively, during the current JCR year.
• 5-year impact factor – Average number of times articles published in a journal during the past 5 years have been cited in the current JCR year.
• Eigenfactor Score (ES) – Similar to the 5-year impact factor; differences are that (ES) eliminates self-citations and considers the importance of citations received by a journal.
• Article influence score – Derived from the ES; measures the average influence of each article published in a journal.

Journal ranking within a specific subject category can also be indicated by quartiles. Many universities around the world prefer the use of these metrics rather than raw IF for selecting journals. Four quartiles rank journals from highest to lowest based on the impact factor: Q1, Q2, Q3, and Q4.

Q1 comprises the most (statistically) prestigious journals within the subject category; i.e., the top 25% of the journals on the list. Q2 journals fall in the 25%–50% group, Q3 journals in the 50%–75% group, and finally, Q4 in the 75%–100% group.

Numbers = status, and many authors, or their institutions, insist on publication in a Q1 or at least a Q2 journal.

An alternative ranking system, and one that is free to access, Scimago Journal & Country Rank , also uses quartiles. Be sure not to confuse the two.

OK, back to the main question: What’s a good impact factor?

As mentioned, separate JCR databases are published for STEM and social sciences.

Discrepancies in fields

The main reason is that there are wide discrepancies in impact factor scores across different research fields. Some of the likely causes for these discrepancies are:

• Differences in citation behavior in different research fields; e.g., review articles tend to attract more citations than research articles, and the tendency to cite books in the social sciences.
• Differences in types of research; e.g., interdisciplinary and basic research attract more citations than intradisciplinary and applied research.
• Differences in field coverage by JCR; e.g., more in-depth coverage of STEM fields compared with humanities and the social sciences.

To put things into perspective, data prepared by SCI Journal for the 2018/2019 journal impact factor rankings are shown in this table.

Source: scijournal.org

The table illustrates where a journal subject area ranks in the four classes: top 80%, top 60%, top 40%, and top 20%. Outliers were removed for the sake of cleaner data.

STEM impact factors

The data demonstrate how journals for subject areas show a range of impact factors.

Large fields, such as the life sciences, generally have higher IFs. They get cited more, so that makes sense. For example, it's natural that a study on a vaccine breakthrough will lead to more citations than a study on community development.

This obviously skews the use of impact factors when assessing researchers across a university. We can’t all keep up with the biologists!

For example, the top-ranked journal by Clarivate in 2019 was CA: A Cancer Journal for Clinicians, which had a remarkably high IF of 292.278. The New England Journal of Medicine , which has long been a prestigious journal, came in second with a high IF of 74.699. Those are amazingly good journal impact factors.

Conversely, reputable journals in smaller fields, such as mathematics, tend to have lower impact factors than the natural sciences.

For example, the 2019 JCR impact factors for respectable mathematical journals such as Inventiones Mathematicae and Duke Mathematical Journal were 2.986 and 2.194, respectively.

Social sciences impact factors

As for the social sciences, you could simply argue they’re “less popular” than the natural sciences. So, their reputable journals also tend to have lower impact factors.

For example, well-regarded journals such as the American Journal of Sociology and the British Journal of Sociology had JCR 2019 impact factors of 3.232 and 2.908, respectively. Those are good impact factors for the social sciences but would look rather low for STEM unless it was a regional or niche topic.

Therefore, what may be seen as an excellent impact factor in mathematics and the social sciences may be viewed as way below average in the life sciences.

That’s not to say the social sciences are less important. It just means they’re comparably researched and cited less.

Niche and specialized impact factors

The comparison doesn’t end there. Another aspect that shouldn’t be overlooked is the research subfield. Journals in, say, a physics subfield such as astronomy have a different impact factor than journals in fluid dynamics.

For example, two well-known astrophysics journals, MNRAS and Astrophysical Journal , had JCR 2019 IFs of 5.356 and 5.745, respectively. Meanwhile, two respectable journals in fluid dynamics, Journal of Fluid Mechanics and Physics of Fluids had JCR 2019 IFs of 3.354 and 3.514, respectively.

They may not be in The New England Journal of Medicine territory, but all are reputable journals.

So, all things considered, what is a good journal impact factor?

You have to look at the bigger picture here because there’s a lot more to consider than the single numeric representation.

If you’re in a field/subfield with high-impact-factor journals, it’s only logical that the cutoff for a good IF will also be high. And, of course, it’ll be lower for a field/subfield with lower impact factor journals.

Impact factor statistics should, thus, be interpreted relatively and with caution, because the scores represented are not absolute.

A good impact factor is either, in short, what your institution or you say it is. Otherwise, it’s one that's sufficient to connote prestige while still being a good forum for your research to be read and cited.

Let’s look at a few of the other factors, apart from IF, to help you choose your target journal.

Pros and cons of using impact factor to judge a journal’s quality

The impact factor was used initially to rank journals, which will then help you decide on which one to which to submit your research. Some of the pros of an IF include:

• Easily accessible
• Gives a general picture of a journal’s prestige and reputation
• Is pretty good for comparison within a field if not across fields
• Appeals to people and institutions that like rankings and numbers

Despite its popularity, the impact factor is clearly a flawed metric, and its use to judge if a journal is good is criticized. That’s what we’ve seen with DORA, among plenty of others.

In addition to the previously mentioned shortcoming of not being able to use the impact factor for comparing journals across fields, other cons include:

• Ambiguous description of what “citable items” are
• Lack of consideration of highly cited papers resulting in skewed citation distributions
• Encouragement of self-citations by journals

The criticism is nothing new (see Kurmis, 2003 , among others). But we’ve got to live with it until there’s something better.

On a personal note, we’re rather tired of seeing the great stress of publishing in a Q1 journal that’s placed on researchers. Especially those from certain economies that pressure their researchers to publish when they’d be better off fostering good, reflective, valuable research.

So here are some other factors with impact, even if they’re not impact factors.

Things other than impact factor to consider when choosing a journal

A good impact factor may be a requirement by your institution. But it shouldn’t be the only aspect you consider when choosing where to publish your manuscript.

Aims and scope

Another key factor is whether the work to be published fits within the aims and scope of the journal.

You can determine this by analyzing the journal’s subjects covered, types of articles published, and peer-review process. Some very targeted journals would welcome your research with open arms.

Target audience

An additional factor to consider is the  target audience.  Who is likely to read and cite the article? Where do these researchers publish? This can facilitate the shortlisting of some journals.

Other factors, apart from IF, for choosing your target journal

Other tips for choosing a journal include:

• Find journals that publish research that’s similar to yours, especially if it’s quite specialized
• Read as many published articles as you can in your target journal
• Go through your list of references to see which journals have the most citations
• Find out where fellow researchers and colleagues publish

Well-known publishers like Springer and Elsevier also list factors for choosing a journal.

Scimago as an alternative

Scimago Journal Rank (SJR) , as mentioned above, is a useful portal that scores and ranks journals, which are indexed in Elsevier’s Scopus database, based on citation data.

The SJR indicator (PDF) not only measures the citations received from a journal but also the importance or prestige of the journal where these citations come from. It can be used to view journal rankings by subject category and compare journals within the same field.

And here’s a great scholarly article with useful references that provide information on how to identify and avoid submitting to predatory journals.

Conclusion on what’s a good impact factor, especially for you and your research

What makes a good impact factor boils down to the field of research and the host of arbiters of “good.” Highly reputable journals may have low impact factors not because they lack credibility, but because they’re in specialized/niche fields with low citations.

The interpretation of what is a “good” journal impact factor varies. Possibly due to the ambitious nature of some researchers or the ignorance of others.

An IF can indeed serve as a starting point during decision-making, but, if possible, and if you don’t have to meet some arbitrary target, more emphasis should be paid to publishing high-quality research.

The prevailing mindset should be that a journal stands to benefit more from the good-quality research it publishes rather than the other way around.

The AJE Team

See our "Privacy Policy"

Find the right journal for your work and with your desired impact factor

AJE will compile a report of three Journal Recommendations specifically for your scientific manuscript. This way, you can get pursue the impact factor you want and/or pursue which journal is most suitable or is likely to publish your work faster.

• Skip to Guides Search
• Skip to breadcrumb
• Skip to main content
• Skip to footer
• Skip to chat link
• Report accessibility issues and get help
• Go to Penn Libraries Home
• Go to Franklin catalog

Research Impact and Citation Analysis

• Compile Departmental Publications
• Curate Researcher Publications
• Discover Potential Collaborators

Compare Journals

• Explore a Document's Impact
• Calculate H-Index

Counterpoints

How do metrics like journal impact factor drive editorial choices, and even prompt manipulative behavior on the part of publishers? See Gaye Tuchman's 2012 article on Inside Higher Ed , "Commodifying the Academic Self."

Limitations   |  Comparing Journal Impact Factor   |  Counterpoints

Limitations

Journal Impact Factor is only one metric by which journals can be compared. What are some other ways to determine a journal's impact, quality, or relevance?

How many patents cite a journal? You can find this out at Lens.org, a free agglomeration database which harvests bibliographic data from PubMed, Crossref, and other sources. Or how about public policy documents, government reports, and statutes?

Interested in reviewing journal quality? One way is to browse the journal's contents. Review the subject matter, and note its relevance to your work. Pick a sampling of articles and study their research methods sections -- do the methods seem rigorous? You can also check the journal's editorial policies around rigor. Does the journal employ statistical reviewers, if relevant? 

Cabell's Journalytics and Predatory Reports give a number of potential factors to review while evaluating journals. Does a journal falsely claim to be indexed in an academic database? Is there a clearly stated peer review policy on the journal website? Does one managing editor appear to be the editor for dozens of journals? Is the editorial board listed?

All these and more are questions a researcher should ask when comparing journal impact, quality, and relevance.

Comparing Journal Impact Factor

In this tutorial, we’ll be using Journal Citation Reports (JCR) from Clarivate. JCR has two metrics of interest, journal impact factor (JIF) and the journal citation indicator (JCI). 

Journal Impact Factor (JIF) is a Clarivate metric. In any given year, the impact factor of a journal is the average number of citations received per paper published in that journal during the two preceding years. The impact factor is based on two figures:  the number of citations to a given journal over the previous two years (A) and the number of research articles and review articles published by that journal over the same two-year period (B), so:  A/B = Impact Factor (JCR). There is also a separate 5-year JIF, which applies the same formula for citations to a given journal over the previous five years, rather than two years.

Journal Citation Indicator (JCI) is a three-year average of a field-weighted metric called CNCI, itself a ratio between number of citations to a journal and the number of expected citations to a journal. How the expectations are calculated is a black box which Clarivate does not reveal. The end result of the JCI is a number which is supposed to be comparable across disciplines. If a journal is given a JCI of 1.0, it is exactly the global average for citation impact. If a journal is given 2.0, it’s impact score is twice the average. In other words, Clarivate frames having a higher than 1.0 JCI as desirable. 

There are many other ways to evaluate journal quality other than simply using these metrics – please see the Considerations and Context section for more. For now, we’ll look at how to retrieve and compare JIF.

First, connect to Journal Citation Reports . Sign in with your Clarivate credentials (the same ones used to sign into Web of Science) to create saved Favorites lists.

Starting with specific journals

Go to Journals at the top and search for a journal name, Click Enter when your desired result appears, or click “see 1 result” . For our purposes, this is better than than clicking the name of the actual journal when it appears; clicking it would open a new window with information about the journal itself.

Check off the correct box. Your boxes will stay selected until you consciously choose to deselect them at the bottom. So, if you like, you could just get through your list of up to 50 journals for comparison and add them all to a Favorite list at once.

Once you have selected all the journals you’d like to compare, click the Add to Favorites list button. Make a new list if applicable.

If your journal doesn’t come up by name, try using ISSN instead. Search for the journal online or in Franklin Catalog to find their ISSN. Both print and e-issns should be listed for a journal, so it shouldn’t matter. 

To view the compared list, go to My Favorites, and click on your list.

You should be able to compare attributes between all of the journals on that list, including journal impact factor.

For additional analysis, you can compare four journals at once using the Compare feature at the bottom of the screen, after the selection phase. This provides both JIF and JCI, but also additional visualizations based on OA, quartiles, etc.

Starting with a journal category

If you don’t know what journals you’d like to compare, but just want to compare a number of journals in a given category, go to Categories at the top, drill down to a category of interest, and click on the number of journals to view their comparison. You can also search for categories in the main search box.

You’ll have the option to explore journals within a category in either the ESCI, the Emerging Sources Citation Index, or a more longstanding index, like the SSCI (Social Sciences Citation Index), SCIE (Science Citation Index-Expanded), and the AHCI (Arts and Humanities Citation Index).

Here you can sort based on JCR, JCI, total citations, or %OA gold. You can also change the JCR year using the filters on the left. If you click on Customize toward the top of the table, you can see other metrics, such as article influence score, normalized Eigenfactor, 5-year JIF, JIF without self-cites, and more. You can also customize which metrics you’d like to see on the screen at once.

You can also click on any one journal name to look at all of these metrics with visualizations for that one journal.

• << Previous: Discover Potential Collaborators
• Next: Explore a Document's Impact >>
• Last Updated: Sep 12, 2023 3:44 PM
• URL: https://guides.library.upenn.edu/bibliometrics

RESEARCH REVIEW International Journal of Multidisciplinary

About the journal.

RESEARCH REVIEW International Journal of Multidisciplinary (RRIJM) is an international Double-blind peer-reviewed [refereed] open access online journal. Too often a journal’s decision to publish a paper is dominated by what the editor/s think is interesting and will gain greater readership-both of which are subjective judgments and lead to decisions which are frustrating and delay the publication of your work. RRIJM will rigorously peer-review your submissions and publish all papers that are judged to be technically sound. Judgments about the importance of any particular paper are then made after publication by the readership (who are the most qualified to determine what is of interest to them).

Most conventional journals publish papers from tightly defined subject areas, making it more difficult for readers from other disciplines to read them. RRIJM has no such barriers, which helps your research reach the entire scientific community.

• Title:  RESEARCH REVIEW International Multidisciplinary Research Journal
• ISSN:  2455-3085 (Online)
• Impact Factor: 6.849
• Crossref DOI:  10.31305/rrijm
• Frequency of Publication:  Monthly  [12 issues per year]
• Languages:  English/Hindi/Gujarat  [Multiple Languages]
• Accessibility:  Open Access
• Peer Review Process:  Double Blind Peer Review Process
• Subject:  Multidisciplinary
• Plagiarism Checker:  Turnitin (License)
• Publication Format:  Online
• Contact No.:  +91- 99784 40833
• Email:  [email protected]
• Old Website: https://old.rrjournals.com/
• New Website: https://rrjournals.com/ 
• Address:  15, Kalyan Nagar, Shahpur, Ahmedabad, Gujarat 380001

Key Features of RRIJM

• Journal was listed in  UGC  with  Journal No. 44945 (Till 14-06-2019)
• Journal Publishes online every month
• Online article submission
• Standard peer review process

Current Issue

Research on Green Finance: A Bibliometric Analysis

The criminal law policy for the buyers of pornographic content, factors causing water loss in the distribution system at pt air minum giri menang (perseroda), comparative study of swami vivekananda’s educational philosophy and modern educational systems स्वामी विवेकानंद के शैक्षिक दर्शन और आधुनिक शिक्षा प्रणालियों का तुलनात्मक अध्ययन, spatial analysis of population landscape: a case study of scheduled caste population in basti district, sustainability practices and esg scores: a sectoral study of nifty 50 firms, a study of the effect of weight training on athletes' agility વજન તાલીમ દ્વારા ખેલાડીઓની ચપળતા પર થતી અસરનો અભ્યાસ, religion and socialization: the psycho-sociological role of islam in socializing individuals and the muslim community (ummah), a study of the effect of yoga and step aerobic activities on circulatory endurance યોગ અને સ્ટેપ એરોબિક પ્રવૃત્તિઓ દ્વારા રૂધિરાભિસરણ સહનશક્તિ પર થતી અસરનો અભ્યાસ, from eccentric incarnations to dehumanized stereotypes: a reading of malayalam films showcasing the visually impaired, bridging research and resources: bibliometric analysis of international journal of research in special education as a library service model, analysis of the impact of reform movements on indian nationalism भारतीय राष्ट्रवाद पर सुधार आन्दोलनों के प्रभाव का विश्लेषण, perspectives on gender and inequality, ethical implications of ai in criminal justice: balancing efficiency and due process, geographically weighted panel regression (gwpr) and geographically and temporally weighted regression (gtwr) methods in the influence of factors affecting the minimum wage of each province in indonesia, study of drought using the theory of run and hydrological drought index and its relationship to reservoir storage at the batubai dam and pengga dam, analysis of risk factors for delays in the implementation of the meninting dam construction project, west lombok regency, juridical analysis of implementation of multi-purpose financing agreements at pt. sinarmas mataram branch, a study on the portrayal of mythological hybrid forms in modern indian art, recollecting the history of mysore ganjifa: an art historical perspective, beyond the shadows: unveiling the socio-political contribution of women in colonial rajasthan in india, information.

• For Readers
• For Authors
• For Librarians

Make a Submission

RESEARCH REVIEW International Journal of Multidisciplinary is licensed under a Creative Commons Attribution 4.0 International License .

Click here to go to Old Website

Management Research Review

• Submit your paper
• Author guidelines
• Editorial team
• Indexing & metrics
• Calls for papers & news

Before you start

For queries relating to the status of your paper pre decision, please contact the Editor or Journal Editorial Office. For queries post acceptance, please contact the Supplier Project Manager. These details can be found in the Editorial Team section.

Author responsibilities

Our goal is to provide you with a professional and courteous experience at each stage of the review and publication process. There are also some responsibilities that sit with you as the author. Our expectation is that you will:

• Respond swiftly to any queries during the publication process.
• Be accountable for all aspects of your work. This includes investigating and resolving any questions about accuracy or research integrity .
• Treat communications between you and the journal editor as confidential until an editorial decision has been made.
• Include anyone who has made a substantial and meaningful contribution to the submission (anyone else involved in the paper should be listed in the acknowledgements).
• Exclude anyone who hasn’t contributed to the paper, or who has chosen not to be associated with the research.
• In accordance with COPE’s position statement on AI tools , Large Language Models cannot be credited with authorship as they are incapable of conceptualising a research design without human direction and cannot be accountable for the integrity, originality, and validity of the published work. The author(s) must describe the content created or modified as well as appropriately cite the name and version of the AI tool used; any additional works drawn on by the AI tool should also be appropriately cited and referenced. Standard tools that are used to improve spelling and grammar are not included within the parameters of this guidance. The Editor and Publisher reserve the right to determine whether the use of an AI tool is permissible.
• If your article involves human participants, you must ensure you have considered whether or not you require ethical approval for your research, and include this information as part of your submission. Find out more about informed consent .

Generative AI usage key principles

• Copywriting any part of an article using a generative AI tool/LLM would not be permissible, including the generation of the abstract or the literature review, for as per Emerald’s authorship criteria, the author(s) must be responsible for the work and accountable for its accuracy, integrity, and validity.
• The generation or reporting of results using a generative AI tool/LLM is not permissible, for as per Emerald’s authorship criteria, the author(s) must be responsible for the creation and interpretation of their work and accountable for its accuracy, integrity, and validity.
• The in-text reporting of statistics using a generative AI tool/LLM is not permissible due to concerns over the authenticity, integrity, and validity of the data produced, although the use of such a tool to aid in the analysis of the work would be permissible.
• Copy-editing an article using a generative AI tool/LLM in order to improve its language and readability would be permissible as this mirrors standard tools already employed to improve spelling and grammar, and uses existing author-created material, rather than generating wholly new content, while the author(s) remains responsible for the original work.
• The submission and publication of images created by AI tools or large-scale generative models is not permitted.

Research and publishing ethics

Our editors and employees work hard to ensure the content we publish is ethically sound. To help us achieve that goal, we closely follow the advice laid out in the guidelines and flowcharts on the COPE (Committee on Publication Ethics) website .

We have also developed our research and publishing ethics guidelines . If you haven’t already read these, we urge you to do so – they will help you avoid the most common publishing ethics issues.

A few key points:

• Any manuscript you submit to this journal should be original. That means it should not have been published before in its current, or similar, form. Exceptions to this rule are outlined in our pre-print and conference paper policies .  If any substantial element of your paper has been previously published, you need to declare this to the journal editor upon submission. Please note, the journal editor may use  Crossref Similarity Check  to check on the originality of submissions received. This service compares submissions against a database of 49 million works from 800 scholarly publishers.
• Your work should not have been submitted elsewhere and should not be under consideration by any other publication.
• If you have a conflict of interest, you must declare it upon submission; this allows the editor to decide how they would like to proceed. Read about conflict of interest in our research and publishing ethics guidelines .
• By submitting your work to Emerald, you are guaranteeing that the work is not in infringement of any existing copyright.

Third party copyright permissions

Prior to article submission, you need to ensure you’ve applied for, and received, written permission to use any material in your manuscript that has been created by a third party. Please note, we are unable to publish any article that still has permissions pending. The rights we require are:

• Non-exclusive rights to reproduce the material in the article or book chapter.
• Print and electronic rights.
• Worldwide English-language rights.
• To use the material for the life of the work. That means there should be no time restrictions on its re-use e.g. a one-year licence.

We are a member of the International Association of Scientific, Technical, and Medical Publishers (STM) and participate in the STM permissions guidelines , a reciprocal free exchange of material with other STM publishers.  In some cases, this may mean that you don’t need permission to re-use content. If so, please highlight this at the submission stage.

Please take a few moments to read our guide to publishing permissions  to ensure you have met all the requirements, so that we can process your submission without delay.

Open access submissions and information

All our journals currently offer two open access (OA) publishing paths; gold open access and green open access.

If you would like to, or are required to, make the branded publisher PDF (also known as the version of record) freely available immediately upon publication, you can select the gold open access route once your paper is accepted. 

If you’ve chosen to publish gold open access, this is the point you will be asked to pay the APC (article processing charge) . This varies per journal and can be found on our APC price list or on the editorial system at the point of submission. Your article will be published with a Creative Commons CC BY 4.0 user licence , which outlines how readers can reuse your work.

Alternatively, if you would like to, or are required to, publish open access but your funding doesn’t cover the cost of the APC, you can choose the green open access, or self-archiving, route. As soon as your article is published, you can make the author accepted manuscript (the version accepted for publication) openly available, free from payment and embargo periods.

You can find out more about our open access routes, our APCs and waivers and read our FAQs on our open research page. 

Find out about open

Transparency and Openness Promotion (TOP) Guidelines

We are a signatory of the Transparency and Openness Promotion (TOP) Guidelines , a framework that supports the reproducibility of research through the adoption of transparent research practices. That means we encourage you to:

• Cite and fully reference all data, program code, and other methods in your article.
• Include persistent identifiers, such as a Digital Object Identifier (DOI), in references for datasets and program codes. Persistent identifiers ensure future access to unique published digital objects, such as a piece of text or datasets. Persistent identifiers are assigned to datasets by digital archives, such as institutional repositories and partners in the Data Preservation Alliance for the Social Sciences (Data-PASS).
• Follow appropriate international and national procedures with respect to data protection, rights to privacy and other ethical considerations, whenever you cite data. For further guidance please refer to our  research and publishing ethics guidelines . For an example on how to cite datasets, please refer to the references section below.

Prepare your submission

Manuscript support services.

We are pleased to partner with Editage, a platform that connects you with relevant experts in language support, translation, editing, visuals, consulting, and more. After you’ve agreed a fee, they will work with you to enhance your manuscript and get it submission-ready.

This is an optional service for authors who feel they need a little extra support. It does not guarantee your work will be accepted for review or publication.

Visit Editage

Manuscript requirements

Before you submit your manuscript, it’s important you read and follow the guidelines below. You will also find some useful tips in our structure your journal submission how-to guide.

	Article files should be provided in Microsoft Word format. While you are welcome to submit a PDF of the document alongside the Word file, PDFs alone are not acceptable. LaTeX files can also be used but only if an accompanying PDF document is provided. Acceptable figure file types are listed further below.
	Articles should be between 6000  and 8000 words in length. This includes all text, for example, the structured abstract, references, all text in tables, and figures and appendices.  Please allow 280 words for each figure or table.
	A concisely worded title should be provided.
	The names of all contributing authors should be added to the ScholarOne submission; please list them in the order in which you’d like them to be published. Each contributing author will need their own ScholarOne author account, from which we will extract the following details: (institutional preferred). . We will reproduce it exactly, so any middle names and/or initials they want featured must be included. . This should be where they were based when the research for the paper was conducted. In multi-authored papers, it’s important that ALL authors that have made a significant contribution to the paper are listed. Those who have provided support but have not contributed to the research should be featured in an acknowledgements section. You should never include people who have not contributed to the paper or who don’t want to be associated with the research. Read about our for authorship.
	If you want to include these items, save them in a separate Microsoft Word document and upload the file with your submission. Where they are included, a brief professional biography of not more than 100 words should be supplied for each named author.
	Your article must reference all sources of external research funding in the acknowledgements section. You should describe the role of the funder or financial sponsor in the entire research process, from study design to submission.
	All submissions must include a structured abstract, following the format outlined below. These four sub-headings and their accompanying explanations must always be included: The following three sub-headings are optional and can be included, if applicable: You can find some useful tips in our  how-to guide. The maximum length of your abstract should be 250 words in total, including keywords and article classification (see the sections below).
	Your submission should include up to 12 appropriate and short keywords that capture the principal topics of the paper. Our  how to guide contains some practical guidance on choosing search-engine friendly keywords. Please note, while we will always try to use the keywords you’ve suggested, the in-house editorial team may replace some of them with matching terms to ensure consistency across publications and improve your article’s visibility.
	During the submission process, you will be asked to select a type for your paper; the options are listed below. If you don’t see an exact match, please choose the best fit: You will also be asked to select a category for your paper. The options for this are listed below. If you don’t see an exact match, please choose the best fit:  Reports on any type of research undertaken by the author(s), including:  Covers any paper where content is dependent on the author's opinion and interpretation. This includes journalistic and magazine-style pieces.  Describes and evaluates technical products, processes or services.  Focuses on developing hypotheses and is usually discursive. Covers philosophical discussions and comparative studies of other authors’ work and thinking.  Describes actual interventions or experiences within organizations. It can be subjective and doesn’t generally report on research. Also covers a description of a legal case or a hypothetical case study used as a teaching exercise.  This category should only be used if the main purpose of the paper is to annotate and/or critique the literature in a particular field. It could be a selective bibliography providing advice on information sources, or the paper may aim to cover the main contributors to the development of a topic and explore their different views.  Provides an overview or historical examination of some concept, technique or phenomenon. Papers are likely to be more descriptive or instructional (‘how to’ papers) than discursive.
	Headings must be concise, with a clear indication of the required hierarchy.  The preferred format is for first level headings to be in bold, and subsequent sub-headings to be in medium italics.
	Notes or endnotes should only be used if absolutely necessary. They should be identified in the text by consecutive numbers enclosed in square brackets. These numbers should then be listed, and explained, at the end of the article.
	All figures (charts, diagrams, line drawings, webpages/screenshots, and photographic images) should be submitted electronically. Both colour and black and white files are accepted. There are a few other important points to note:
	Tables should be typed and submitted in a separate file to the main body of the article. The position of each table should be clearly labelled in the main body of the article with corresponding labels clearly shown in the table file. Tables should be numbered consecutively in Roman numerals (e.g. I, II, etc.). Give each table a brief title. Ensure that any superscripts or asterisks are shown next to the relevant items and have explanations displayed as footnotes to the table, figure or plate.
	Where tables, figures, appendices, and other additional content are supplementary to the article but not critical to the reader’s understanding of it, you can choose to host these supplementary files alongside your article on Insight, Emerald’s content-hosting platform (this is Emerald's recommended option as we are able to ensure the data remain accessible), or on an alternative trusted online repository. All supplementary material must be submitted prior to acceptance. Emerald recommends that authors use the following two lists when searching for a suitable and trusted repository:     , you must submit these as separate files alongside your article. Files should be clearly labelled in such a way that makes it clear they are supplementary; Emerald recommends that the file name is descriptive and that it follows the format ‘Supplementary_material_appendix_1’ or ‘Supplementary tables’. All supplementary material must be mentioned at the appropriate moment in the main text of the article; there is no need to include the content of the file only the file name. A link to the supplementary material will be added to the article during production, and the material will be made available alongside the main text of the article at the point of EarlyCite publication. Please note that Emerald will not make any changes to the material; it will not be copy-edited or typeset, and authors will not receive proofs of this content. Emerald therefore strongly recommends that you style all supplementary material ahead of acceptance of the article. Emerald Insight can host the following file types and extensions: , you should ensure that the supplementary material is hosted on the repository ahead of submission, and then include a link only to the repository within the article. It is the responsibility of the submitting author to ensure that the material is free to access and that it remains permanently available. Where an alternative trusted online repository is used, the files hosted should always be presented as read-only; please be aware that such usage risks compromising your anonymity during the review process if the repository contains any information that may enable the reviewer to identify you; as such, we recommend that all links to alternative repositories are reviewed carefully prior to submission. Please note that extensive supplementary material may be subject to peer review; this is at the discretion of the journal Editor and dependent on the content of the material (for example, whether including it would support the reviewer making a decision on the article during the peer review process).
	All references in your manuscript must be formatted using one of the recognised Harvard styles. You are welcome to use the Harvard style Emerald has adopted – we’ve provided a detailed guide below. Want to use a different Harvard style? That’s fine, our typesetters will make any necessary changes to your manuscript if it is accepted. Please ensure you check all your citations for completeness, accuracy and consistency. References to other publications in your text should be written as follows: , 2006) Please note, ‘ ' should always be written in italics. A few other style points. These apply to both the main body of text and your final list of references. At the end of your paper, please supply a reference list in alphabetical order using the style guidelines below. Where a DOI is available, this should be included at the end of the reference.
	Surname, initials (year),  , publisher, place of publication. e.g. Harrow, R. (2005),  , Simon & Schuster, New York, NY.
	Surname, initials (year), "chapter title", editor's surname, initials (Ed.), , publisher, place of publication, page numbers. e.g. Calabrese, F.A. (2005), "The early pathways: theory to practice – a continuum", Stankosky, M. (Ed.),  , Elsevier, New York, NY, pp.15-20.
	Surname, initials (year), "title of article",  , volume issue, page numbers. e.g. Capizzi, M.T. and Ferguson, R. (2005), "Loyalty trends for the twenty-first century",  , Vol. 22 No. 2, pp.72-80.
	Surname, initials (year of publication), "title of paper", in editor’s surname, initials (Ed.),  , publisher, place of publication, page numbers. e.g. Wilde, S. and Cox, C. (2008), “Principal factors contributing to the competitiveness of tourism destinations at varying stages of development”, in Richardson, S., Fredline, L., Patiar A., & Ternel, M. (Ed.s),  , Griffith University, Gold Coast, Qld, pp.115-118.
	Surname, initials (year), "title of paper", paper presented at [name of conference], [date of conference], [place of conference], available at: URL if freely available on the internet (accessed date). e.g. Aumueller, D. (2005), "Semantic authoring and retrieval within a wiki", paper presented at the European Semantic Web Conference (ESWC), 29 May-1 June, Heraklion, Crete, available at: http://dbs.uni-leipzig.de/file/aumueller05wiksar.pdf (accessed 20 February 2007).
	Surname, initials (year), "title of article", working paper [number if available], institution or organization, place of organization, date. e.g. Moizer, P. (2003), "How published academic research can inform policy decisions: the case of mandatory rotation of audit appointments", working paper, Leeds University Business School, University of Leeds, Leeds, 28 March.
	(year), "title of entry", volume, edition, title of encyclopaedia, publisher, place of publication, page numbers. e.g.   (1926), "Psychology of culture contact", Vol. 1, 13th ed., Encyclopaedia Britannica, London and New York, NY, pp.765-771. (for authored entries, please refer to book chapter guidelines above)
	Surname, initials (year), "article title",  , date, page numbers. e.g. Smith, A. (2008), "Money for old rope",  , 21 January, pp.1, 3-4.
	(year), "article title", date, page numbers. e.g.   (2008), "Small change", 2 February, p.7.
	Surname, initials (year), "title of document", unpublished manuscript, collection name, inventory record, name of archive, location of archive. e.g. Litman, S. (1902), "Mechanism & Technique of Commerce", unpublished manuscript, Simon Litman Papers, Record series 9/5/29 Box 3, University of Illinois Archives, Urbana-Champaign, IL.
	If available online, the full URL should be supplied at the end of the reference, as well as the date that the resource was accessed. Surname, initials (year), “title of electronic source”, available at: persistent URL (accessed date month year). e.g. Weida, S. and Stolley, K. (2013), “Developing strong thesis statements”, available at: https://owl.english.purdue.edu/owl/resource/588/1/ (accessed 20 June 2018) Standalone URLs, i.e. those without an author or date, should be included either inside parentheses within the main text, or preferably set as a note (Roman numeral within square brackets within text followed by the full URL address at the end of the paper).
	Surname, initials (year),  , name of data repository, available at: persistent URL, (accessed date month year). e.g. Campbell, A. and Kahn, R.L. (2015),  , ICPSR07218-v4, Inter-university Consortium for Political and Social Research (distributor), Ann Arbor, MI, available at: https://doi.org/10.3886/ICPSR07218.v4 (accessed 20 June 2018)

Submit your manuscript

There are a number of key steps you should follow to ensure a smooth and trouble-free submission.

Double check your manuscript

Before submitting your work, it is your responsibility to check that the manuscript is complete, grammatically correct, and without spelling or typographical errors. A few other important points:

• Give the journal aims and scope a final read. Is your manuscript definitely a good fit? If it isn’t, the editor may decline it without peer review.
• Does your manuscript comply with our research and publishing ethics guidelines ?
• Have you cleared any necessary publishing permissions ?
• Have you followed all the formatting requirements laid out in these author guidelines?
• If you need to refer to your own work, use wording such as ‘previous research has demonstrated’ not ‘our previous research has demonstrated’.
• If you need to refer to your own, currently unpublished work, don’t include this work in the reference list.
• Any acknowledgments or author biographies should be uploaded as separate files.
• Carry out a final check to ensure that no author names appear anywhere in the manuscript. This includes in figures or captions.

You will find a helpful submission checklist on the website Think.Check.Submit .

The submission process

All manuscripts should be submitted through our editorial system by the corresponding author.

The only way to submit to the journal is through the journal’s ScholarOne site as accessed via the Emerald website, and not by email or through any third-party agent/company, journal representative, or website. Submissions should be done directly by the author(s) through the ScholarOne site and not via a third-party proxy on their behalf.

A separate author account is required for each journal you submit to. If this is your first time submitting to this journal, please choose the Create an account or Register now option in the editorial system. If you already have an Emerald login, you are welcome to reuse the existing username and password here.

Please note, the next time you log into the system, you will be asked for your username. This will be the email address you entered when you set up your account.

Don't forget to add your  ORCiD ID during the submission process. It will be embedded in your published article, along with a link to the ORCiD registry allowing others to easily match you with your work.

Don’t have one yet? It only takes a few moments to register for a free ORCiD identifier .

Visit the ScholarOne support centre  for further help and guidance.

What you can expect next

You will receive an automated email from the journal editor, confirming your successful submission. It will provide you with a manuscript number, which will be used in all future correspondence about your submission. If you have any reason to suspect the confirmation email you receive might be fraudulent, please contact the journal editor in the first instance.

Post submission

Review and decision process.

Each submission is checked by the editor. At this stage, they may choose to decline or unsubmit your manuscript if it doesn’t fit the journal aims and scope, or they feel the language/manuscript quality is too low.

If they think it might be suitable for the publication, they will send it to at least two independent referees for double anonymous peer review.  Once these reviewers have provided their feedback, the editor may decide to accept your manuscript, request minor or major revisions, or decline your work.

This journal offers an article transfer service. If the editor decides to decline your manuscript, either before or after peer review, they may offer to transfer it to a more relevant Emerald journal in this field. If you accept, your ScholarOne author account, and the accounts of your co-authors, will automatically transfer to the new journal, along with your manuscript and any accompanying peer review reports. However, you will still need to log in to ScholarOne to complete the submission process using your existing username and password. While accepting a transfer does not guarantee the receiving journal will publish your work, an editor will only suggest a transfer if they feel your article is a good fit with the new title.

While all journals work to different timescales, the goal is that the editor will inform you of their first decision within 60 days.

During this period, we will send you automated updates on the progress of your manuscript via our submission system, or you can log in to check on the current status of your paper.  Each time we contact you, we will quote the manuscript number you were given at the point of submission. If you receive an email that does not match these criteria, it could be fraudulent and we recommend you contact the journal editor in the first instance.

Manuscript transfer service

Emerald’s manuscript transfer service takes the pain out of the submission process if your manuscript doesn’t fit your initial journal choice. Our team of expert Editors from participating journals work together to identify alternative journals that better align with your research, ensuring your work finds the ideal publication home it deserves. Our dedicated team is committed to supporting authors like you in finding the right home for your research.

If a journal is participating in the manuscript transfer program, the Editor has the option to recommend your paper for transfer. If a transfer decision is made by the Editor, you will receive an email with the details of the recommended journal and the option to accept or reject the transfer. It’s always down to you as the author to decide if you’d like to accept. If you do accept, your paper and any reviewer reports will automatically be transferred to the recommended journals. Authors will then confirm resubmissions in the new journal’s ScholarOne system.

Our Manuscript Transfer Service page has more information on the process.

If your submission is accepted

Open access.

Once your paper is accepted, you will have the opportunity to indicate whether you would like to publish your paper via the gold open access route.

If you’ve chosen to publish gold open access, this is the point you will be asked to pay the APC (article processing charge).  This varies per journal and can be found on our APC price list or on the editorial system at the point of submission. Your article will be published with a Creative Commons CC BY 4.0 user licence , which outlines how readers can reuse your work.

For UK journal article authors - if you wish to submit your work accepted by Emerald to REF 2021, you must make a ‘closed deposit’ of your accepted manuscript to your respective institutional repository upon acceptance of your article. Articles accepted for publication after 1st April 2018 should be deposited as soon as possible, but no later than three months after the acceptance date. For further information and guidance, please refer to the REF 2021 website.

All accepted authors are sent an email with a link to a licence form.  This should be checked for accuracy, for example whether contact and affiliation details are up to date and your name is spelled correctly, and then returned to us electronically. If there is a reason why you can’t assign copyright to us, you should discuss this with your journal content editor. You will find their contact details on the editorial team section above.

Proofing and typesetting

Once we have received your completed licence form, the article will pass directly into the production process. We will carry out editorial checks, copyediting, and typesetting and then return proofs to you (if you are the corresponding author) for your review. This is your opportunity to correct any typographical errors, grammatical errors or incorrect author details. We can’t accept requests to rewrite texts at this stage.

When the page proofs are finalised, the fully typeset and proofed version of record is published online. This is referred to as the EarlyCite version. While an EarlyCite article has yet to be assigned to a volume or issue, it does have a digital object identifier (DOI) and is fully citable. It will be compiled into an issue according to the journal’s issue schedule, with papers being added by chronological date of publication.

How to share your paper

Visit our author rights page  to find out how you can reuse and share your work.

To find tips on increasing the visibility of your published paper, read about  how to promote your work .

Correcting inaccuracies in your published paper

Sometimes errors are made during the research, writing and publishing processes. When these issues arise, we have the option of withdrawing the paper or introducing a correction notice. Find out more about our  article withdrawal and correction policies .

Need to make a change to the author list? See our frequently asked questions (FAQs) below.

Frequently asked questions

	The only time we will ever ask you for money to publish in an Emerald journal is if you have chosen to publish via the gold open access route. You will be asked to pay an APC (article-processing charge) once your paper has been accepted (unless it is a sponsored open access journal), and never at submission. At no other time will you be asked to contribute financially towards your article’s publication, processing, or review. If you haven’t chosen gold open access and you receive an email that appears to be from Emerald, the journal, or a third party, asking you for payment to publish, please contact our support team via .
	Please contact the editor for the journal, with a copy of your CV. You will find their contact details on the editorial team tab on this page.
	Typically, papers are added to an issue according to their date of publication. If you would like to know in advance which issue your paper will appear in, please contact the content editor of the journal. You will find their contact details on the editorial team tab on this page. Once your paper has been published in an issue, you will be notified by email.
	Please email the journal editor – you will find their contact details on the editorial team tab on this page. If you ever suspect an email you’ve received from Emerald might not be genuine, you are welcome to verify it with the content editor for the journal, whose contact details can be found on the editorial team tab on this page.
	If you’ve read the aims and scope on the journal landing page and are still unsure whether your paper is suitable for the journal, please email the editor and include your paper's title and structured abstract. They will be able to advise on your manuscript’s suitability. You will find their contact details on the Editorial team tab on this page.
	Authorship and the order in which the authors are listed on the paper should be agreed prior to submission. We have a right first time policy on this and no changes can be made to the list once submitted. If you have made an error in the submission process, please email the Journal Editorial Office who will look into your request – you will find their contact details on the editorial team tab on this page.

• Lerong He State University of New York at Geneseo - USA [email protected]
• Jay J. Janney University of Dayton - USA [email protected]

Editorial Assistant

• Chunghui Kuo Individual researcher - USA [email protected]
• Chloe Campbell Emerald Publishing - UK [email protected]

Journal Editorial Office (For queries related to pre-acceptance)

• Shrushti Gupta Emerald Publishing [email protected]

Supplier Project Manager (For queries related to post-acceptance)

• Nitesh Shetty Emerald Publishing [email protected]

Editorial Advisory Board

• Steven H. Appelbaum John Molson School of Business, Concordia University - Canada
• Elisa Arrigo University of Milano-Bicocca - Italy
• Muhammad Awais Bhatti King Faisal University - Saudi Arabia
• Timothy Bartram RMIT University - Australia
• Gary Chaison Graduate School of Management, Clark University - USA
• Stewart Clegg University of Technology Sydney - Australia
• James J Cordeiro Department of Business Administration & Economics, SUNY at Brockport - USA
• Matteo Cristofaro University of Rome Tor Vegata - Italy
• J. Barton Cunningham University of Victoria - Canada
• Alison Dean Newcastle Business School - Australia
• Behnam Fahimnia University of Sydney - Australia
• Piyali Ghosh Indian Institute of Management Ranchi - India
• Anna Graziano Link Campus University - Italy
• Maria Hayu Agustini Soegijapranata Catholic University - Indonesia
• Barry Hettler Ohio University - USA
• Faizul Huq College of Business, Ohio University - USA
• Peter Jones University of Gloucestershire Business School - UK
• Boris Kabanoff School of Management, Queensland University of Technology - Australia
• Anastasia Katou University of Macedonia - Greece
• Lisa A Keister Duke University - USA
• Sascha Kraus Free University of Bozen-Bolzano - Italy
• Darren Lee-Ross James Cook University - Australia
• Frank Lefley University of Hradec Králové - Czech Republic
• Laura Meade M J Neeley School of Business, Texas Christian University - USA
• Boniface Michael California State University, Sacramento - USA
• Armando Papa University of Salerno - Italy
• R D Pathak School of Social & Economic Development, The University of South Pacific - Fiji
• Ken Peattie Cardiff University - UK
• Rajesh K. Pillania Management Development Institute, Gurgaon - India
• Abdul A. Rasheed University of Texas at Arlington - USA
• Elena Revilla Instituto de Empresa - Spain
• Hazel Rosin York University - Canada
• Kanti Saini NL Dalmia Institute of Management Studies and Research - India
• Joseph Sarkis Worcester Polytechnic Institute, USA
• Tara Shankar Shaw Indian Institute of Technology, Bombay - India
• Clive Smallman University of Western Sydney - Australia
• R P Sundarraj Indian Institute of Technology Madras - India
• Srinivas Talluri Eli Broad Graduate School of Management, Michigan State University - USA
• Vas Taras University of North Carolina at Greensboro - USA
• Greg Teal School of Management, University of Western Sydney - Australia
• Diego Vazquez-Brust University of Portsmouth, Portsmouth Business School - UK
• Karen Yuan Wang School of Management, University of Technology, Sydney - Australia
• Cherrie Jiuhua Zhu Monash University - Australia

Citation metrics

CiteScore 2023

Further information

CiteScore is a simple way of measuring the citation impact of sources, such as journals.

Calculating the CiteScore is based on the number of citations to documents (articles, reviews, conference papers, book chapters, and data papers) by a journal over four years, divided by the number of the same document types indexed in Scopus and published in those same four years.

For more information and methodology visit the Scopus definition

CiteScore Tracker 2024

(updated monthly)

CiteScore Tracker is calculated in the same way as CiteScore, but for the current year rather than previous, complete years.

The CiteScore Tracker calculation is updated every month, as a current indication of a title's performance.

2023 Impact Factor

The Journal Impact Factor is published each year by Clarivate Analytics. It is a measure of the number of times an average paper in a particular journal is cited during the preceding two years.

For more information and methodology see Clarivate Analytics

5-year Impact Factor (2023)

A base of five years may be more appropriate for journals in certain fields because the body of citations may not be large enough to make reasonable comparisons, or it may take longer than two years to publish and distribute leading to a longer period before others cite the work.

Actual value is intentionally only displayed for the most recent year. Earlier values are available in the Journal Citation Reports from Clarivate Analytics .

Publication timeline

Time to first decision

Time to first decision , expressed in days, the "first decision" occurs when the journal’s editorial team reviews the peer reviewers’ comments and recommendations. Based on this feedback, they decide whether to accept, reject, or request revisions for the manuscript.

Data is taken from submissions between 1st June 2023 and 31st May 2024

Acceptance to publication

Acceptance to publication , expressed in days, is the average time between when the journal’s editorial team decide whether to accept, reject, or request revisions for the manuscript and the date of publication in the journal. 

Data is taken from the previous 12 months (Last updated July 2024)

Acceptance rate

The acceptance rate is a measurement of how many manuscripts a journal accepts for publication compared to the total number of manuscripts submitted expressed as a percentage %

Data is taken from submissions between 1st June 2023 and 31st May 2024 .

This figure is the total amount of downloads for all articles published early cite in the last 12 months

(Last updated: July 2024)

This journal is abstracted and indexed by

• British Library
• Business Source Alumni Edition/Complete/Government Edition/Corporate Plus/Elite/Premier
• Cabell's Directory of Publishing Opportunities is Management and Marketing
• Current Abstracts (EBSCO)
• Emerging Sources Citation Index
• OCLC's Electronic Collections Online
• ReadCube Discover
• TOC Premier (EBSCO)

This journal is ranked by

• Association of Business Schools (ABS) Academic Journal Quality Guide
• ANVUR Class A (Italy)
• Australian Business Deans Council (ABDC) Quality Journal List
• BFI (Denmark)
• Federation of Management Societies of China (FMS)
• JourQUAL 2.1 (Germany)
• NSD (Norway)
• Polish Scholarly Bibliography (PBN)
• QUALIS (Brazil)
• The Publication Forum (Finland)

Reviewer information

Peer review process.

This journal engages in a double-anonymous peer review process, which strives to match the expertise of a reviewer with the submitted manuscript. Reviews are completed with evidence of thoughtful engagement with the manuscript, provide constructive feedback, and add value to the overall knowledge and information presented in the manuscript.

The mission of the peer review process is to achieve excellence and rigour in scholarly publications and research.

Our vision is to give voice to professionals in the subject area who contribute unique and diverse scholarly perspectives to the field.

The journal values diverse perspectives from the field and reviewers who provide critical, constructive, and respectful feedback to authors. Reviewers come from a variety of organizations, careers, and backgrounds from around the world.

All invitations to review, abstracts, manuscripts, and reviews should be kept confidential. Reviewers must not share their review or information about the review process with anyone without the agreement of the editors and authors involved, even after publication. This also applies to other reviewers’ “comments to author” which are shared with you on decision.

Resources to guide you through the review process

Discover practical tips and guidance on all aspects of peer review in our reviewers' section. See how being a reviewer could benefit your career, and discover what's involved in shaping a review.

More reviewer information

Thank you to the 2022 Reviewers of Management Research Review

The publishing and editorial teams would like to thank the following, for their invaluable service as 2022 reviewers for this journal. We are very grateful for the contributions made. With their help, the journal has been able to publish such high...

Thank you to the 2021 Reviewers of Management Research Review

The publishing and editorial teams would like to thank the following, for their invaluable service as 2021 reviewers for this journal. We are very grateful for the contributions made. With their help, the journal has been able to publish such high...

Literati awards

Management Research Review - Literati Award Winners 2023

We are pleased to announce our 2023 Literati Award winners. Outstanding Paper From conventional to digital leadership: explo...

Management Research Review - Literati Award Winners 2022 

We are pleased to announce our 2022 Literati Award winners. Outstanding Paper Arresting fake news sharing on...

Management Research Review - Literati Award Winners 2021

We are pleased to announce our 2021 Literati Award winners. Outstanding Paper Virtuous leadership: a source ...

Management Research Review (MRR) publishes high-quality quantitative and qualitative research in the field of general management with a viewpoint to emphasize executive and managerial practice implications.

Aims and scope

Management Research Review (MRR) publishes a wide variety of articles outlining the latest management research. We emphasize management implications from multiple disciplines. We welcome high-quality empirical and theoretical studies, literature reviews, and articles with important tactical implications.

Published 12 times a year, the journal prides itself on quick publication of the very latest research in general management. The key issues featured include:

• Business Ethics and Sustainability
• Corporate Governance
• Entrepreneurship and Small Business Management
• Industrial Relations
• Knowledge and Innovation Management
• International Business
• Human Resource Management
• Management in Practice
• Marketing Management
• Organizational Behavior
• Organizational Theory
• Production and Operations Management
• Strategic Management
• General Management
• Other Management Related Topics

Latest articles

These are the latest articles published in this journal (Last updated: July 2024)

Multilevel study of transformational leadership and work behavior: job autonomy matters in public service

The role of perceived knowledge on key brand community constructs of trust, involvement and engagement, unveiling the potential of perceived authentic leadership to enhance followers' moral intentions: a self-determination theory perspective, top downloaded articles.

These are the most downloaded articles over the last 12 months for this journal (Last updated: July 2024)

Framework for sustainable value creation: a synthesis of fragmented sustainable business model literature

High-performance work systems and firminnovation: themoderating role of digital technology and employee participation. evidence fromeurope, driving sustained work engagement- moderating role of leadership and organizational support for remote work.

These are the top cited articles for this journal, from the last 12 months according to Crossref (Last updated: July 2024)

Start-up sustainability: Does blockchain adoption drives sustainability in start-ups? A systematic literature reviews

The human side of entrepreneurship: an empirical investigation of relationally embedded ties with stakeholders.

This journal is aligned with our responsible management goal

We aim to champion researchers, practitioners, policymakers and organisations who share our goals of contributing to a more ethical, responsible and sustainable way of working.

Related journals

This journal is part of our Business, management & strategy collection. Explore our Business, management & strategy subject area to find out more.  

See all related journals

Management Research: The Journal of the Iberoamerican Academy of Management

Management Research: Journal of the Iberoamerican Academy of Management publishes high-quality research in the field of...

Baltic Journal of Management

Baltic Journal of Management (BJM) publishes multidisciplinary perspectives on contemporary issues and opportunities in...

Journal of Advances in Management Research

Journal of Advances in Management Research (JAMR) publishes research to contribute to the improvement of management...

University Library

Introduction to Impact Factor and Other Research Metrics

• Types of Metrics

Impact Factor

• Identifying Journals
• More Resources

More Information

• About Journal Impact Factor Visit this article published by Thomson Reuters on journal impact factor to learn more about the bibliometric and how it measures importance.
• Annual Reviews Rankings in JCR Visit this page to see how Annual Review Journals currently rank in Journal Citation Reports.
• SCImago Journal and Country Rank The SCImago Journal & Country Rank is a publicly available portal that includes the journals and country scientific indicators developed from the information contained in the Scopus database.

Ask a Librarian

Impact factor,  or Journal Impact Factor,  is a measure of the frequency with which the "average article" published in a given scholarly journal has been cited in a particular year or period and is often used to measure or describe the importance of a particular journal to its field. Impact factor was originally developed by Eugene Garfield, the founder of Institute of Scientific Information, which is now a part of Clarivate Analytics. Journal Impact Factor can be found in the  Journal Citation Reports or the JCR, as it's commonly known. Over the years various organizations have been created similar journal-level metrics, such as  SCImago Journal & Country Rank .

This page describes how to find impact factor in Journal Citation Reports .

Journal Citation Reports

Clarivate Analytics (formerly Institute for Scientific Information (ISI)) ranks, evaluates, and compares journals within subject categories and publishes the results in Journal Citation Reports . Journal Citation Reports provides ranking for journals in science, technology, and the social sciences. For every journal, the database collects and/or calculates information such as:

• citation/article counts
• impact factor
• immediacy index
• cited half-life
• citing half-life
• source data listing
• citing journal listing
• cited journal listing
• publisher information
• subject categories

Find Journal Citation Reports (JCR)

Follow the instructions below to find the Journal Citation Reports using the Library's resources.

• Begin at the Library homepage .
• Click on the orange tab that says "Find Materials," then scroll down until you see a laptop icon with the words "Databases by Subject and A-Z"; click on the icon.
• Type journal citations reports in the search box on the left side of the screen and then click on the magnifying glass to search that title.
• Your result will say "Journal Citation Reports"; click on it. It might ask you to provide your Net ID and password if you are off campus.

Find the Impact Factor

• Once in the database you either search by journal title (if you know which journal you want to see) or browse by category, which will let you view journals by JIF by discipline.
• On the left side you can choose search criteria, like impact factor range, year, and if the journal is open access.
• It is important to choose the right edition based on your subject area, as you won't be able to see specific journals if you choose the wrong one. Once you have finished selecting what to search, click Submit.
• You can't access impact factors from last year because the calculations only happen every two years (i.e. if the current year is 2021 the farthest you can go back is 2020). Most people choose the most current year they can access.
• Journals limited by the subject area, publisher, or geographic region.
• View all journals in order to browse.
• Search for a specific journal if you already know its title

Once you find a journal, the JCR gives you information about the journal, including the journal's abbreviations, how often it is published each year, the publisher, and the ISSN. 

Controversy

Many people have questioned the legitimacy of impact factor. Here are a few reasons why:

• Impact factor focuses purely on the numbers. There is no consideration of qualitative elements that have become important in today's world.
• Impact factor fails to incorporate more recent ways of sharing and using research, including Twitter mentions and posts, citation management downloads, and news and community information.
• Because impact factor is based on citations in only indexed journals , it fails to incorporate statistics from journals that might not be indexed and other sources like conference papers (which are important in the social sciences).
• Basic or summary information is usually cited the most in academia. That means that journals that publish articles with basic information are more likely to have higher impact factors. Journals that publish obscure or innovative information might not have as high of an impact factor.
• Some argue that impact factor is encouraging scholars to stick with mainstream topics and research.
• Scholars don't always have to cite something for it to be influential. Sometimes researchers just read something and it influences them, regardless of if they cite it in a future paper or piece of research.
• The journals in the JCR are mostly published in English. This means that many international sources aren't included in the conversation.
• It has been argued that journals have the ability to skew impact factor for their own journal. Before publishing an author, they will ask the author to cite more articles within their journal so that their impact factor goes up. This is NOT a common occurrence but instead something we should be aware of.
• << Previous: Types of Metrics
• Next: Identifying Journals >>
• Last Updated: Feb 28, 2024 12:49 PM
• URL: https://guides.library.illinois.edu/impact

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• v.21(8); 2023 Aug
• PMC10464996

Relationship between journal impact factor and the thoroughness and helpfulness of peer reviews

Anna severin.

1 Institute of Social and Preventive Medicine, University of Bern, Bern, Switzerland

2 Graduate School for Health Sciences, University of Bern, Bern, Switzerland

Michaela Strinzel

3 Swiss National Science Foundation, Bern, Switzerland

Matthias Egger

4 Population Health Sciences, Bristol Medical School, University of Bristol, Bristol, United Kingdom

Tiago Barros

5 Faculty Opinions, London, United Kingdom

Alexander Sokolov

6 Clarivate, London, United Kingdom

Julia Vilstrup Mouatt

7 University of Auckland, Auckland, New Zealand

Stefan Müller

8 School of Politics and International Relations, University College Dublin, Dublin, Ireland

Associated Data

All relevant data are within the paper and its Supporting Information files. The fine-tuned DistilBERT models, data, and code to verify the reproducibility of all tables and graphs are available at https://doi.org/10.5281/zenodo.8006829 . Publons’ data sharing policy prohibits us from publishing the raw text of the reviews and the annotated sentences.

The Journal Impact Factor is often used as a proxy measure for journal quality, but the empirical evidence is scarce. In particular, it is unclear how peer review characteristics for a journal relate to its impact factor. We analysed 10,000 peer review reports submitted to 1,644 biomedical journals with impact factors ranging from 0.21 to 74.7. Two researchers hand-coded sentences using categories of content related to the thoroughness of the review ( Materials and Methods , Presentation and Reporting , Results and Discussion , Importance and Relevance ) and helpfulness ( Suggestion and Solution , Examples , Praise , Criticism ). We fine-tuned and validated transformer machine learning language models to classify sentences. We then examined the association between the number and percentage of sentences addressing different content categories and 10 groups defined by the Journal Impact Factor . The median length of reviews increased with higher impact factor, from 185 words (group 1) to 387 words (group 10). The percentage of sentences addressing Materials and Methods was greater in the highest Journal Impact Factor journals than in the lowest Journal Impact Factor group. The results for Presentation and Reporting went in the opposite direction, with the highest Journal Impact Factor journals giving less emphasis to such content. For helpfulness, reviews for higher impact factor journals devoted relatively less attention to Suggestion and Solution than lower impact factor journals. In conclusion, peer review in journals with higher impact factors tends to be more thorough, particularly in addressing study methods while giving relatively less emphasis to presentation or suggesting solutions. Differences were modest and variability high, indicating that the Journal Impact Factor is a bad predictor of the quality of peer review of an individual manuscript.

An analysis of the content of 10,000 peer review reports reveals that reports submitted to journals with higher impact factors pay more attention to the materials and methods of a study but less attention to presentation and reporting, whereas journals with low impact factors provide more suggestions, solutions and examples.

Introduction

Peer review is a process of scientific appraisal by which manuscripts submitted for publication in journals are evaluated by experts in the field for originality, rigour, and validity of methods and potential impact [ 1 ]. Peer review is an important scientific contribution and is increasingly visible on databases and researcher profiles [ 2 , 3 ]. In medicine, practitioners rely on sound evidence from clinical research to make a diagnosis or prognosis and choose a therapy. Recent developments, such as the retraction of peer-reviewed COVID-19 publications in prominent medical journals [ 4 ] or the emergence of predatory journals [ 5 , 6 ], have prompted concerns about the rigour and effectiveness of peer review. Despite these concerns, research into the quality of peer review is scarce. Little is known about the determinants and characteristics of high-quality peer review. The confidential nature of many peer review reports and the lack of databases and tools for assessing their quality have hampered larger-scale research on peer review.

The impact factor was originally developed to help libraries make indexing and purchasing decisions for their collections. It is a journal-based metric calculated each year by dividing the number of citations received in that year for papers published in the 2 preceding years by the number of “citable items” published during the 2 preceding years [ 7 ]. The reputation of a journal, its impact factor, and the perceived quality of peer review are among the most common criteria authors use to select journals to publish their work [ 8 – 10 ]. Assuming that citation frequency reflects a journal’s importance in the field, the impact factor is often used as a proxy for journal quality [ 11 ]. It is also used in academic promotion, hiring decisions, and research funding allocation, leading scholars to seek publication in journals with high impact factors [ 12 ].

Despite using the Journal Impact Factor as a proxy for a journal’s quality, empirical research on the impact factor as a measure of journal quality is scarce [ 11 ]. In particular, it is unclear how the peer review characteristics for a journal relate to this metric. We combined human coding of peer review reports and quantitative text analysis to examine the association between peer review characteristics and Journal Impact Factor in the medical and life sciences, based on a sample of 10,000 peer review reports. Specifically, we examined the impact factor’s relationship with the absolute number and the percentages of sentences related to peer review thoroughness and helpfulness.

Characteristics of the study sample

The sample included 5,067 reviews from Essential Science Indicators (ESI) [ 13 ] research field Clinical Medicine, 943 from Environment and Ecology, 942 from Biology and Biochemistry, 733 from Psychiatry and Psychology, 633 from Pharmacology and Toxicology, 576 from Neuroscience and Behaviour, 566 from Molecular Biology and Genetics, 315 from Immunology, and 225 from Microbiology.

Across the 10 groups of journals defined by Journal Impact Factor deciles (1 = lowest, 10 = highest), the median Journal Impact Factor ranged from 1.23 to 8.03, the minimum ranged from 0.21 to 6.51 and the maximum from 1.45 to 74.70 ( Table 1 ). The proportion of reviewers from Asia, Africa, South America, and Australia/Oceania declined when moving from Journal Impact Factor group 1 to group 10. In contrast, there was a trend in the opposite direction for Europe and North America. Information on the continent of affiliation was missing for 43.5% of reviews (4,355). The median length of peer review reports increased by about 202 words from group 1 (median number of words 185) to group 10 (387). S1 File details the 10 journals from each Journal Impact Factor group that provided the highest number of peer review reports, gives the complete list of journals, and shows the distribution of reviews across the 9 ESI disciplines.

	Journal Impact Factor group
	1	2	3	4	5	6	7	8	9	10
Median JIF (range)	1.23 (0.21–1.45)	1.68 (1.46–1.93)	2.07 (1.93–2.22)	2.42 (2.23–2.54)	2.77 (2.54–3.01)	3.26 (3.01–3.55)	3.83 (3.55–4.20)	4.53 (4.21–5.16)	5.67 (5.163–6.5)	8.03 (6.51–74.70)
No. of review reports	1,000	1,000	1,000	1,000	1,000	1,000	1,000	1,000	1,000	1,000
No. of journals	256	224	151	146	183	156	155	129	98	146
No. of reviewers	967	960	969	958	965	973	961	939	970	962
No. of sentences (median; IQR)	9 (4–18)	11 (6–22)	12 (5–22)	13 (6–23)	14 (7–25)	14 (7–25)	16 (8–28)	17 (8–27)	16.5 (9–27)	18 (10–30)
No. of words (median; IQR)	185 (84–359)	232.5 (116–426)	225 (104–419)	256.5 (116–478)	284.5 (146–506)	271 (142–495)	346 (170–581)	344.5 (176–555)	350.5 (195–567)	387 (213–672)
Continent of reviewers’ affiliation
Asia	139	107	163	115	93	135	98	93	80	62
Africa	15	14	18	9	5	14	8	6	5
Europe	119	156	187	190	231	250	268	273	280	241
North America	97	113	105	153	162	151	191	180	166	213
Central/South America	61	42	36	25	38	22	22	20	23	10
Australia/Oceania	50	55	36	46	64	37	26	37	38	52
Missing	519	513	455	462	407	391	387	391	408	422
Gender of reviewer
Female	242	262	261	254	241	211	216	189	260	206
Male	518	516	478	549	548	551	575	584	543	599
Unknown	240	222	261	197	211	238	209	227	197	195

IQR, interquartile range; JIF, Journal Impact Factor.

Continents are ordered by population size.

JIF group defined by deciles (1 = lowest, 10 = highest).

Performance of coders and classifiers

The training of coders resulted in acceptable to good between-coder agreement, with an average Krippendorff’s α across the 8 categories of 0.70. The final analyses included 10,000 review reports, comprising 188,106 sentences, which were submitted by 9,259 reviewers to 1,644 journals. In total, 9,590 unique manuscripts were reviewed.

In the annotated dataset, the most common categories based on human coding were Materials and Methods (coded in 823 sentences or 41.2% out of 2,000 sentences), Suggestion and Solution (638 sentences; 34.2%), and Presentation and Reporting (626 sentences; 31.3%). In contrast, Praise (210; 10.5%) and Importance and Relevance (175; 8.8%) were the least common. On average, the training set had 444 sentences per category, as 1,160 sentences were allocated to more than 1 category. In out-of-sample predictions based on DistilBERT, a transformer model for text classification [ 14 ], precision, recall, and F1 scores (binary averages across both classes [absent/present]) were similar within categories (see S2 File ). The classification was most accurate for Example and Materials and Methods (F1 score 0.71) and least accurate for Criticism (0.57) and Results and Discussion (0.61). The prevalence predicted from the machine learning model was generally close to the human coding: Point estimates did not differ by more than 3 percentage points. Overall, the machine learning classification closely mirrored human coding. Further details are given in S2 File .

Descriptive analysis: Thoroughness and helpfulness of peer review reports

The majority of sentences (107,413 sentences, 57.1%) contributed to more than 1 content category; a minority (23,997 sentences, 12.8%) were not assigned to any category. The average number of sentences addressing each of the 8 content categories in the set of 10,000 reviews ranged from 1.6 sentences on Importance and Relevance to 9.2 sentences on Materials and Methods (upper panel of Fig 1 ). The percentage of sentences addressing each category are shown in the lower panel of Fig 1 . The content categories Materials and Methods (46.7% of sentences), Suggestion and Solution (34.5%), and Presentation and Reporting (30.0%) were most extensively covered. The category Results and Discussion was present in 16.3% of the sentences, and 13.1% were assigned to the category Examples . In contrast, only 8.4% of sentences addressed the Importance and Relevance of the study. Criticism (16.5%) was slightly more common than Praise (14.9%). Most distributions were wide and skewed to the right, with a peak at 0 sentence or 0% corresponding to reviews that did not address the content category ( Fig 1 ).

The number (upper panel) and percentage of sentences (lower panel) in a review allocated to the 8 peer review content categories is shown. A sentence could be allocated to no, one, or several categories. Vertical dashed lines show the average number (upper panel) and average percentage of sentences (lower panel) after aggregating them to the level of reviews. Analysis based on 10,000 review reports. The data underlying this figure can be found in S1 Data .

Fig 2 shows the estimated number of sentences addressing the 8 content categories across the 10 Journal Impact Factor groups. For all categories, the number of sentences increased from Journal Impact Factor groups 1 to 10. However, increases were modest on average, amounting to 2 or fewer additional sentences. The exception was Materials and Methods , where the difference between Journal Impact Factor groups 1 and 10 was 6.5 sentences on average.

A sentence could be allocated to no, one, or several categories. Vertical dashed lines show the average number of sentences after aggregating numbers to the level of reviews. The number of sentences are displayed on a log scale. Analysis based on 10,000 review reports. The data underlying this figure can be found in S2 Data .

Fig 3 shows the estimated percentage of sentences across content categories and Journal Impact Factor groups. Among thoroughness categories, the percentage of sentences addressing Materials and Methods increased from 40.4% to 51.8% from Journal Impact Factor groups 1 to 10. In contrast, attention to Presentation and Reporting declined from 32.9% in group 1 to 25.0% in group 10. No clear trends were evident for Results and Discussion or Importance and Relevance . For helpfulness, the percentage of sentences including Suggestion and Solution declined from 36.9% in group 1 to 30.3% in group 10. The prevalence of sentences providing Examples increased from 11.0% (group 1) to 13.3% (group 10). Praise decreased slightly, whereas Criticism increased slightly when moving from group 1 to group 10. The distributions were broad, even within the groups of journals with similar impact factors.

The percentage of sentences in a review allocated to the 8 peer review quality categories is shown. A sentence could be allocated to no, one, or several categories. Analysis based on 10,000 review reports. Vertical dashed lines show the average prevalence after aggregating prevalences to the level of reviews. The data underlying this figure can be found in S3 Data .

Regression analyses

The association between journal impact factor and the 8 content categories was analysed in 2 regression analyses. The first predicted the number of sentences of each content category across the 10 Journal Impact Factor groups; the second, the changes in the percentage of sentences addressing content categories. All coefficients and standard errors are available from S3 File .

The predicted number of sentences are shown in Fig 4 with their 95% confidence intervals (CI). The results confirm those observed in the descriptive analyses. There was a substantial increase in the number of sentences addressing Materials and Methods from Journal Impact Factor group 1 (6.1 sentences; 95% CI 5.3 to 6.8) to group 10 (12.5 sentences; 95% CI 11.6 to 13.5), for a difference of 6.4 sentences. For the other categories, only small increases were predicted, in line with the descriptive analyses.

Predicted values and 95% confidence intervals are shown. Analysis based on 10,000 review reports. All negative binomial mixed-effects models include random intercepts for the journal name and reviewer ID. The data underlying this figure can be found in S4 Data .

The predicted differences in the percentage of sentences addressing content categories are shown in Fig 5 . Again, the results confirm those observed in the descriptive analyses. The prevalence of sentences on Materials and Methods in the journals with the highest impact factor was higher (+11.0 percentage points; 95% CI + 7.9 to +14.1) than in the group with the lowest impact factor journals. The trend for sentences addressing Presentation and Reporting went in the opposite direction, with reviews submitted to the journals with the highest impact factor giving less emphasis to such content (−7.7 percentage points; 95% CI −10.0 to −5.4). There was slightly less focus on Importance and Relevance in the group of journals with the highest impact factors relative to the group with the lowest impact factors (−1.9 percentage points; 95% CI −3.5 to −0.4) and little evidence of a difference for Results and Discussion (+1.1 percentage points; 95% CI −0.54 to +2.8). Reviews for higher impact factor journals devoted less attention to Suggestion and Solution . The group with the highest Journal Impact Factor had 6.2 percentage points fewer sentences addressing Suggestion and Solution (95% CI −8.5 to −3.8). No substantive differences were observed for Examples (0.3 percentage points; 95% CI −1.7 to +2.3), Praise (1.6 percentage points; 95% CI −0.5 to +3.7), and Criticism (0.5 percentage points; 95% CI −1.0 to +2.0).

Regression coefficients and 95% confidence intervals are shown. Analysis based on 10,000 review reports. All linear mixed-effects models include random intercepts for the journal name and reviewer ID. The data underlying this figure can be found in S5 Data .

Sensitivity analyses

We performed several sensitivity analyses to assess the robustness of findings. In the first, we removed reviews with 0 sentences or 0% in the respective content category, resulting in similar regression coefficients and predicted counts. In the second, the sample was limited to reviews with at least 10 sentences (sentence models) or 200 words (percentage model). The analysis showed that short reviews do not drive associations. In the third sensitivity analysis, the regression models adjusted for additional variables (discipline, career stage of reviewers, and log number of reviews submitted by reviewers). The addition of these variables reduced the sample size from 10,000 to 5,806 reviews because of missing reviewer-level data. Again, the relationships between content categories and journal impact factor persisted. The fourth sensitivity analysis revealed that results were generally similar for male and female reviewers. The fifth showed that the results changed little when replacing the Journal Impact Factor groups with the raw Journal Impact Factor ( S3 File ).

Typical words in content categories

A keyness analysis [ 15 ] extracts typical words for each content category across the full corpus of the 188,106 sentences. The analysis is based on χ 2 tests comparing the frequencies of each word in sentences assigned to a content category and other sentences. Table 2 reports the 50 words appearing more frequently in sentences assigned to the respective content category than in other sentences (according to the DistilBERT classification). The table supports the validity of the classification. Common terms in the thoroughness categories were “data”, “analysis”, “method” ( Materials and Methods ); “please”, “text”, “sentence”, “line”, “figure” ( Presentation and Reporting ); “results”, “discussion”, “findings” ( Results and Discussion ); and “interesting”, “important”, “topic” ( Importance and Relevance ). For helpfulness, common unique words included “please”, “need”, “include ( Suggestion and Solution ); “line”, “page”, “figure” ( Examples ); “interesting”, “good”, “well” ( Praise ); and “however”, “(un)clear”, “mistakes” ( Criticism ).

Results rely on keyness analyses using χ 2 tests for each word, comparing the frequency of words in sentences where a content characteristic was present with sentences (target group) where characteristic was absent (reference group). Table reports the 50 words with the highest χ 2 values per category.

Content category	Words
Materials and Methods	data, methods, analysis, model, patients, method, sample, used, analyses, test, treatment, models, performed, using, criteria, control, experiments, statistical, samples, measures, population, group, parameters, measure, approach, methodology, size, measured, procedure, cohort, groups, variables, scale, controls, design, tests, experiment, experimental, selection, testing, tested, measurements, regression, compared, procedures, measurement, analyzed, trials, score, sampling
Presentation and Reporting	please, text, sentence, line, figure, written, table, section, page, paragraph, figures, references, introduction, tables, english, abstract, language, word, sentences, description, reference, mention, explain, information, detail, specify, reader, clarify, legend, well, needs, lines, described, mentioned, clearly, describe, term, summarize, details, informative, errors, abbreviations, read, well-written, grammar, explained, remove, check, need, clarified
Results and Discussion	results, discussion, findings, conclusions, conclusion, result, outcome, correlation, effect, outcomes, section, finding, interpretation, discussed, correlations, confidence, variance, supported, statistical, regression, significant, implications, discuss, statistically, presented, summarize, main, significance, predictions, analysis, values, deviation, comparison, error, difference, obtained, comparisons, estimates, value, drawn, uncertainty, likelihood, draw, conclude, observed, objective, deviations, discussions, differences, variables
Importance and Relevance	interesting, important, topic, interest, research, contribution, field, novel, importance, work, study, audience, relevance, literature, understanding, paper, useful, future, valuable, insights, knowledge, quality, focus, provides, great, originality, overall, rigor, timely, addresses, approach, clinical, significance, relevant, scientific, implications, usefulness, review, general, insight, context, innovative, readership, area, community, revision, comprehensive, findings, perspective, practical
Suggestion and Solution	please, need, needs, better, suggest, provide, consider, clarify, recommend, helpful, include, must, section, required, needed, discussion, line, revision, table, detail, remove, discuss, explain, sentence, specify, help, check, revise, text, improve, think, reader, added, delete, make, replace, useful, highlight, minor, comment, might, clarified, details, clearer, paragraph, worth, references, information, adding, perhaps
Example	line, page, figure, lines, sentence, paragraph, table, example, replace, delete, legend, remove, please, word, change, line, panel, comma, column, reference, typo, instead, pages, last, page, caption, statement, shown, mean, bottom, sentences, figures, phrase, rephrase, shows, panels, replaced, section, correct, indicate, write, missing, first, figure1, says, confusing, starting, figs, text, meant
Criticism	unclear, clear, however, difficult, confusing, don’t, missing, hard, lack, sure, lacks, seem, seems, understand, little, misleading, doesn’t, enough, vague, confused, incorrect, lacking, unfortunately, somewhat, problematic, insufficient, although, convinced, major, wrong, statement, mistakes, quite, poorly, conclusion, incomplete, questionable, weak, grammatical, inconsistent, errors, sentence, remains, speculative, limited, really, follow, makes, figure, concerns
Praise	interesting, well, good, written, well-written, topic, manuscript, paper, important, interest, excellent, overall, satisfactory, comments, timely, nice, great, valuable, author, work, appreciate, review, provides, publication, comprehensive, contribution, article, study, research, novel, useful, enjoyed, field, concise, sound, impressive, improved, dear, easy, nicely, congratulate, thorough, worthy, addresses, relevant, appreciated, appropriate, presents, designed, adequate

This study used fine-tuned transformer language models to analyse the content of peer review reports and investigate the association of content with the Journal Impact Factor . We found that the impact factor was associated with the characteristics and content of peer review reports and reviewers. The length of reports increased with increasing Journal Impact Factor , with the number of relevant sentences increasing for all content categories, but in particular for Materials and Methods . Expressed as the percentage of sentences addressing a category (and thus standardising for the different lengths of peer review reports), the prevalence of sentences providing suggestions and solutions, examples, or addressing the reporting of the work declined with increasing Journal Impact Factor . Finally, the proportion of reviewers from Asia, Africa, and South America also declined, whereas the proportion of reviewers from Europe and North America increased.

The limitations of the Journal Impact Factor are well documented [ 16 – 18 ], and there is increasing agreement that it should not be used to evaluate the quality of research published in a journal. The San Francisco Declaration on Research Assessment (DORA) calls for the elimination of any journal-based metrics in funding, appointment, and promotion [ 19 ]. DORA is supported by thousands of universities, research institutes and individuals. Our study shows that the peer reviews submitted to journals with higher Journal Impact Factor may be more thorough than those submitted to lower impact journals. Should, therefore, the Journal Impact Factor be rehabilitated and used as a proxy measure for peer review quality? Similar to the distribution of citations in a journal, the length of reports and the prevalence of content related to thoroughness and helpfulness varied widely, within journals and between journals with similar Journal Impact Factor . In other words, the Journal Impact Factor is a poor proxy measure for the thoroughness or helpfulness of peer review authors may expect when submitting their manuscripts.

The increase in the length of peer review reports with increasing Journal Impact Factor might be explained by the fact that reviewers from Europe and North America and reviewers with English as their first language tend to write longer reports and to review for higher impact journals [ 20 ]. Further, high impact factor journals may be more prestigious to review for and can thus afford to recruit more senior scholars. Of note, there is evidence suggesting that the quality of reports decreases with age or years of reviewing [ 21 , 22 ]. Interestingly, several medical journals with high impact factors have recently committed to improving diversity among their reviewers [ 23 – 25 ]. Unfortunately, due to incomplete data, we could not examine the importance of the level of seniority of reviewers. Independently of seniority, reviewers may be brief reviewing for a journal with low impact factor, believing a more superficial review will suffice. On the other hand, brief reviews are not necessarily superficial: The review of a very poor paper may not warrant a long text.

Peer review reports have been hidden for many years, hampering research on their characteristics. Previous studies were based on smaller, selected samples. An early randomised trial evaluating the effect of blinding reviewers to the authors’ identity on the quality of peer review was based on 221 reports submitted to a single journal [ 26 ]. Since then, science has become more open, embracing open access to publications and data and open peer review. Some journals now publish peer reviews and authors’ responses along with the articles [ 27 – 29 ]. Bibliographic databases have also started to publish reviews [ 30 ]. The European Cooperation in Science and Technology (COST) Action on new frontiers of peer review (PEERE), established in 2017 to examine peer review in different areas, was based on data from several hundred Elsevier journals from a wide range of disciplines [ 31 ].

To our knowledge, the Publons database is the largest of peer review reports, and the only one not limited to individual publishers or journals, making it a unique resource for research on peer review. Based on 10,000 peer review reports submitted to medical and life science journals, this is likely the largest study of peer review content ever done. It built on a previous analysis of the characteristics of scholars who review for predatory and legitimate journals [ 32 ]. Other strengths of this study include the careful classification and validation step, based on the coding by hand of 2,000 sentences by trained coders. The performance of the classifiers was high, which is reassuring given that the sentence-level classification tasks deal with imbalanced and sometimes ambiguous categories. Performance is in line with recent studies. For example, a study using an extension of BERT to classify concepts such as nationalism, authoritarianism, and trust reported results for precision and recall similar to the present study [ 33 ]. We trained the algorithm on journals from many disciplines, which should make it applicable to other fields than medicine and the life sciences. Journals and funders could use our approach to analyse the thoroughness and helpfulness of their peer review. Journals could submit their peer review reports to an independent organisation for analysis. The results could help journals improve peer review, give feedback to peer reviewers, inform the training of peer reviewers, and help readers gauge the quality of the journals in their field. Further, such analyses could inform a reviewer credit system that could be used by funders and research institutions.

Our study has several weaknesses. Reviewers may be more likely to submit their review to Publons if they feel it meets general quality criteria. This could have introduced bias if the selection process into Publons’ database depended on the Journal Impact Factor . However, the large number of journals within each Journal Impact Factor group makes it likely that the patterns observed are real and generalizable. We acknowledge that our findings are more reliable for the more common content categories than for the less common. We only examined peer review reports and could not consider the often extensive contributions made by journal editors and editorial staff to improve articles. In other words, although our results provide valuable insights into the peer review process, they give an incomplete picture of the general quality assurance processes of journals. Due to the lack of information in the database, we could not analyse any differences between open (signed) and anonymous peer review reports. Similarly, we could not distinguish between reviews of original research articles and other article types, for example, narrative review articles. Some journals do not consider importance and relevance when assessing submissions, and these journals may have influenced results for this category. We lacked the resources to identify these journals among the over 1,600 outlets included in our study to examine their influence. Finally, we could not assess to what extent the content of peer review reports affected acceptance or rejection of the paper.

Conclusions

This study of peer review characteristics indicates that peer review in journals with higher impact factors tends to be more thorough, particularly in addressing the study’s methods while giving relatively less emphasis to presentation or suggesting solutions. Our findings may have been influenced by differences in reviewer characteristics, quality of submissions, and the attitude of reviewers towards the journals. Differences were modest, and the Journal Impact Factor is therefore a bad predictor of the quality of peer review of an individual manuscript.

Our study was based on peer review reports submitted to Publons from January 24, 2014, to May 23, 2022. Publons (part of Web of Science) is a platform for scholars to track their peer review activities and receive recognition for reviewing [ 34 ]. A total of 2,000 sentences from peer review reports were hand-coded and assigned to none, one, or more than one of 8 content categories related to thoroughness and helpfulness. The transformer model DistilBERT [ 14 , 35 ] was then used to assign the sentences in peer review reports as contributing or not contributing to categories. More details are provided in the Section “Classification and validation” below and S2 File . After validating the classification performance using out-of-sample predictions, the association between the 2019 Journal Impact Factors [ 36 ] and the prevalence of relevant sentences in peer review reports was examined. The sample is limited to review reports submitted to medical and life sciences journals with an impact factor. The analysis took the hierarchical nature of the data into account.

Data sources

As of May 2022, the Publons database contained information on 15 million reviews performed and submitted by more than 1,150,000 scholars for about 55,000 journals and conference proceedings. Reviews can be submitted to Publons in different ways. When scholars review for journals partnering with Publons and wish recognition, Publons receives the review and some meta-data directly from the journal. For other journals, scholars can upload the review and verify it by forwarding the confirmation email from the journal to Publons or by sending a screenshot from the peer review submission system. Publons audits a random subsample of emails and screenshots by contacting editors or journal administrators.

Publons randomly selected English-language peer review reports for the training from a broad spectrum of journals, covering all (ESI) fields [ 37 ] except Physics, Space Science, and Mathematics. Reviews from the latter fields contained many mathematical formulae, which were difficult to categorise. In the next step, a stratified random sample of 10,000 verified prepublication reviews written in English was drawn. First, the Publons database was limited to reviews from medical and life sciences journals based on ESI research fields, resulting in a data set of approximately 5.2 million reviews. The ESI field Multidisciplinary was excluded as these journals publish articles not within the medical and life sciences field (e.g., PLOS ONE , Nature , Science ). Second, these reviews were divided into 10 equal groups based on Journal Impact Factor deciles. Third, 1,000 reviews were selected randomly from each of the 10 groups. Second-round peer review reports were excluded whenever this information was available. The continent of the reviewer’s institutional affiliation, the total number of publications of the reviewer, the start and end year of the reviewers’ publications, and gender were available for a subset of reviews. The gender of reviewers were classified with the gender-guesser Python package (version 0.4.0). Since the data on reviewer characteristics are incomplete and automated gender classification suffers from misclassification, these variables are only included in regression models reported in S3 File .

Classification and validation

Two authors (ASE and MS) were trained in coding sentences. After piloting and refining coding and establishing intercoder reliability, the reviewers labelled 2,000 sentences (1,000 sentences each). They allocated sentences to none, one, or several of 8 content categories. We selected the 8 categories based on prior work, including the Review Quality Instrument and other scales and checklists [ 38 ], and previous studies using text analysis or machine learning to assess student and peer review reports [ 39 – 43 ]. In the manual coding process, the categories were refined, taking into account the ease of operationalising categories and their intercoder reliability. Based on the pilot data, Krippendorff’s α, a measure of reliability in content analysis, was calculated [ 44 ].

The categories describe, first, the Thoroughness of a review, measuring the degree to which a reviewer comments on (1) Materials and Methods (Did the reviewer comment on the methods of the manuscript?); (2) Presentation and Reporting (Did the reviewer comment on the presentation and reporting of the paper?); (3) Results and Discussion (Did the reviewer comment on the results and their interpretation?); and (4) the paper’s Importance and Relevance (Did the reviewer comment on the importance or relevance of the manuscript?). Second, the Helpfulness of a review was examined based on comments on (5) Suggestion and Solution (Did the reviewer provide suggestions for improvement or solutions?); (6) Examples (Did the reviewer give examples to substantiate his or her comments?); (7) Praise (Did the reviewer identify strengths?); and (8) Criticism (Did the reviewer identify problems?). Categories were rated on a binary scale (1 for yes, 0 for no). A sentence could be coded as 1 for multiple categories. S4 File gives further details.

We used the transformer model DistilBERT to predict the absence or presence of the 8 characteristics in each sentence of the peer review reports [ 45 ]. For validation, data were split randomly into a training set of 1,600 sentences and a held-out test set of 400 sentences. Eight DistilBERT models (one for each content categories) were fine-tuned on the set of 1,600 sentences and predicted the categories in the remaining 400 sentences. Performance measures, including precision (i.e., the positive predictive value), recall (i.e., sensitivity), and the F1 score, were calculated. The F1 score is a harmonic mean of precision and recall and an overall measure of accuracy. The F1 score can range between 0 and 1, with higher values indicating better classification performance [ 46 ].

Overall, the classification performance of the fine-tuned DistilBERT language models was high. The average F1 score for the presence of a characteristic was 0.75, ranging from 0.68 ( Praise ) to 0.88 ( Suggestion and Solution ). For most categories, precision and recall were similar, indicating the absence of systematic measurement error. Importance and Relevance and Results and Discussion were the exceptions, with lower recall for characteristics being present. Balanced accuracy (the arithmetic mean of sensitivity and specificity) was also high, ranging from 0.78 to 0.91 (with a mean of 0.83 across the 8 categories). S2 File gives further details.

We compared the percentages of sentences addressing each category between the human annotation dataset and the output from the machine learning model. For the test set of 400 sentences, the percentage of sentences that fall into each of the 8 categories were calculated, separately for the human codings and the DistilBERT predictions. There was a close match between the two: DistilBERT overestimated Importance and Relevance by 3.0 percentage points and underestimated Materials and Methods by 2.3 percentage points. For all other content categories, smaller differences were observed. Having assessed the validity of the classification, the machine learning classifiers were fine-tuned using all 2,000 labelled sentences, and the 8 classifiers were used to predict the presence or absence of content in the full text corpus consisting of 188,106 sentences.

Finally, we identified unique words in each quality category using a “keyness” analysis [ 47 ]. The words retrieved from the keyness analyses reflect typical words used in each content category.

Statistical analysis

The association between peer review characteristics and Journal Impact Factor groups was examined in 2 ways. The analysis of the number of sentences for each category used negative binomial regression models. The analysis of the percentages of sentences addressing content categories relied on linear mixed-effects models. To account for the clustered nature of the data, we include random intercepts for journals and reviewers [ 48 ]. The regression models take the form,

where Y i is the count of sentences addressing a content category (for the negative binomial regression models) or the percentages (for the linear-mixed effects models), while i , β m are the coefficients for the m = 2,…,10 categories of the categorical variable of Journal Impact Factor (with m = 1 as the reference category), and ϵ i is the unobserved error term. The model includes varying intercepts α j [ i ], k [ i ] for J journals and K reviewers. I · denotes the indicator function.

All regression analyses were done in R (version 4.2.1). The fine-tuning of the classifier and sentence-level predictions were done in Python (version 3.8.13). The libraries used for data preparation, text analysis, supervised classification, and regression models were transformers (version 4.20.1) [ 49 ], quanteda (version 3.2.3) and quanteda . textstats (version 0.95) [ 50 ], lme4 (version 1.1.30) [ 51 ], glmmTMB (version 1.1.7) [ 52 ], ggeffects (version 1.1.5) [ 53 ], and tidyverse (version 1.3.2) [ 54 ].

Supporting information

The 10 journals from each journal impact factor group that provided the largest number of peer review reports and all 1,664 journals included in the analysis listed in alphabetical order. The numbers in parentheses represent the JIF and the number of reviews included in the sample.

Further information on the hand-coded set of sentences, the classification approach, and performance provide metrics on the classification performance and show that aggregating the classification closely mirrors human coding of the same set of sentences. All results are out-of-sample predictions, meaning that the data in the held-out test set are not used for training the classifier during validation steps.

All regression tables for the analysis reported in the paper, and plots and regression tables relating to the 5 sensitivity analyses. All sensitivity analyses are conducted for the prevalence-based and sentence-based models.

Coding instructions and examples for each of the 8 characteristics of peer review reports.

Acknowledgments

We are grateful to Anne Jorstad and Gabriel Okasa from the Swiss National Science Foundation (SNSF) data team for valuable comments on an earlier draft of this paper. We would also like to thank Marc Domingo (Publons, part of Web of Science) for help with the sampling procedure.

Abbreviations

CI	confidence interval
COST	European Cooperation in Science and Technology
DORA	San Francisco Declaration on Research Assessment
ESI	Essential Science Indicators

Funding Statement

This study was supported by Swiss National Science Foundation (SNSF) grant 32FP30-189498 to ME, see https://data.snf.ch/grants/grant/189498 ) and internal SNSF resources. The funders had no role in study design, data collection and analysis, decision to publish, or preparation of the manuscript.

Data Availability

• PLoS Biol. 2023 Aug; 21(8): e3002238.

Decision Letter 0

23 Aug 2022

Dear Matthias,

Thank you for submitting your manuscript entitled "Journal Impact Factor and Peer Review Thoroughness and Helpfulness: A Supervised Machine Learning Study" for consideration as a Meta-Research Article by PLOS Biology.

Your manuscript has now been evaluated by the PLOS Biology editorial staff, as well as by an academic editor with relevant expertise, and I'm writing to let you know that we would like to send your submission out for external peer review.

However, before we can send your manuscript to reviewers, we need you to complete your submission by providing the metadata that is required for full assessment. To this end, please login to Editorial Manager where you will find the paper in the 'Submissions Needing Revisions' folder on your homepage. Please click 'Revise Submission' from the Action Links and complete all additional questions in the submission questionnaire.

Once your full submission is complete, your paper will undergo a series of checks in preparation for peer review. After your manuscript has passed the checks it will be sent out for review. To provide the metadata for your submission, please Login to Editorial Manager ( https://www.editorialmanager.com/pbiology ) within two working days, i.e. by Aug 25 2022 11:59PM.

If your manuscript has been previously peer-reviewed at another journal, PLOS Biology is willing to work with those reviews in order to avoid re-starting the process. Submission of the previous reviews is entirely optional and our ability to use them effectively will depend on the willingness of the previous journal to confirm the content of the reports and share the reviewer identities. Please note that we reserve the right to invite additional reviewers if we consider that additional/independent reviewers are needed, although we aim to avoid this as far as possible. In our experience, working with previous reviews does save time.

If you would like us to consider previous reviewer reports, please edit your cover letter to let us know and include the name of the journal where the work was previously considered and the manuscript ID it was given. In addition, please upload a response to the reviews as a 'Prior Peer Review' file type, which should include the reports in full and a point-by-point reply detailing how you have or plan to address the reviewers' concerns.

During the process of completing your manuscript submission, you will be invited to opt-in to posting your pre-review manuscript as a bioRxiv preprint. Visit http://journals.plos.org/plosbiology/s/preprints for full details. If you consent to posting your current manuscript as a preprint, please upload a single Preprint PDF.

Feel free to email us at gro.solp@ygoloibsolp if you have any queries relating to your submission.

Kind regards,

Roland Roberts, PhD

Senior Editor

PLOS Biology

gro.solp@streborr

Decision Letter 1

12 Oct 2022

Thank you for your patience while your manuscript "Journal Impact Factor and Peer Review Thoroughness and Helpfulness: A Supervised Machine Learning Study" was peer-reviewed at PLOS Biology. Your manuscript has been evaluated by the PLOS Biology editors, an Academic Editor with relevant expertise, and by three independent reviewers.

As you will see in the reviewer reports, which can be found at the end of this email, although the reviewers find the work potentially interesting, they have also raised a substantial number of important concerns. Based on their specific comments and following discussion with the Academic Editor, it is clear that a substantial amount of work would be required to meet the criteria for publication in PLOS Biology.

However, given our and the reviewers' interest in your study, we would be open to inviting a comprehensive revision of the study that thoroughly addresses all the reviewers' comments. Given the extent of revision that would be needed, we cannot make a decision about publication until we have seen the revised manuscript and your response to the reviewers' comments. Your revised manuscript would need to be seen by the reviewers again, but please note that we would not engage them unless their main concerns have been addressed.

You'll see that most of reviewer #1's requests are for methodological clarification, but he also wonders whether your results may be skewed by the greater length of the reviews in high-impact journals. He also points out the underappreciated contribution of professional editors (!!). Reviewer #2 is also broadly positive, though regrets the fact that data are from a single source; he recommends that you try to validate with respect to another source, and suggests several substantial additional analyses. Like reviewer #1, he requests some methodological clarifications, mentions some possible confounders, and makes some very helpful textual and interpretational suggestions. Reviewer #3, who is a machine learning expert, is similarly positive, but has a long series of quite severe-sounding criticisms of your methodology and its reporting.

We appreciate that these requests represent a great deal of extra work, and we are willing to relax our standard revision time to allow you 6 months to revise your study. Please email us ( gro.solp@ygoloibsolp ) if you have any questions or concerns, or envision needing a (short) extension.

At this stage, your manuscript remains formally under active consideration at our journal; please notify us by email if you do not intend to submit a revision so that we may withdraw it.

**IMPORTANT - SUBMITTING YOUR REVISION**

Your revisions should address the specific points made by each reviewer. Please submit the following files along with your revised manuscript:

1. A 'Response to Reviewers' file - this should detail your responses to the editorial requests, present a point-by-point response to all of the reviewers' comments, and indicate the changes made to the manuscript.

*NOTE: In your point by point response to the reviewers, please provide the full context of each review. Do not selectively quote paragraphs or sentences to reply to. The entire set of reviewer comments should be present in full and each specific point should be responded to individually, point by point.

You should also cite any additional relevant literature that has been published since the original submission and mention any additional citations in your response.

2. In addition to a clean copy of the manuscript, please also upload a 'track-changes' version of your manuscript that specifies the edits made. This should be uploaded as a "Revised Article with Changes Highlighted " file type.

*Resubmission Checklist*

When you are ready to resubmit your revised manuscript, please refer to this resubmission checklist: https://plos.io/Biology_Checklist

To submit a revised version of your manuscript, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' where you will find your submission record.

Please make sure to read the following important policies and guidelines while preparing your revision:

*Published Peer Review*

Please note while forming your response, if your article is accepted, you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. Please see here for more details:

https://blogs.plos.org/plos/2019/05/plos-journals-now-open-for-published-peer-review/

*PLOS Data Policy*

Please note that as a condition of publication PLOS' data policy ( http://journals.plos.org/plosbiology/s/data-availability ) requires that you make available all data used to draw the conclusions arrived at in your manuscript. If you have not already done so, you must include any data used in your manuscript either in appropriate repositories, within the body of the manuscript, or as supporting information (N.B. this includes any numerical values that were used to generate graphs, histograms etc.). For an example see here: http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001908#s5

*Protocols deposition*

To enhance the reproducibility of your results, we recommend that if applicable you deposit your laboratory protocols in protocols.io, where a protocol can be assigned its own identifier (DOI) such that it can be cited independently in the future. Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols

Thank you again for your submission to our journal. We hope that our editorial process has been constructive thus far, and we welcome your feedback at any time. Please don't hesitate to contact us if you have any questions or comments.

-----------------------------------------

REVIEWERS' COMMENTS:

Reviewer #1:

[identifies himself as Ludo Waltman]

Please find my review at https://ludowaltman.pubpub.org/pub/review-jif-pr/release/1

[the editor has here pasted the text of the review from that location]

This paper presents a large-scale analysis of the content of peer review reports, focusing on different types of comments provided in review reports and the association with the impact factors of journals. The scale of the analysis is impressive. Studies of the content of such a large number of review reports are exceptional. I enjoyed reading the paper, even though I did not find the results presented in the paper to be particularly surprising.

Feedback and suggestions for improvements are provided below.

The methods used by the authors would benefit from a significantly more detailed explanation:

“Scholars can submit their reviews for other journals by either forwarding the review confirmation emails from the journals to Publons or by sending a screenshot of the review from the peer review submission system.”: This sentence is unclear. Review confirmation emails often do not include the review itself, only a brief ‘thank you’ message, so it is not clear to me how a review can be obtained from such a confirmation email. I also do not understand how a review can be obtained from a screenshot. A screenshot may show only part of the review, not the entire review, and there would be a significant technical challenge in converting the screenshot, which is an image, to machine-readable text.

I would like to know whether all reviews are in English or whether there are also reviews in other languages.

Impact factors change over time. New impact factors are calculated each year. The authors need to explain which impact factors they used.

There are many journals that do not have an impact factor. The authors need to explain how these journals were handled.

The authors also need to discuss how reviewers were linked to publication profiles. This is a non-trivial step that needs to be taken to determine the number of publications of a reviewer and the start and end year of the publications of a reviewer. The authors do not explain how this step was taken in their analysis. It is important to provide this information.

“We used a Naïve Bayes algorithm to train the classifier and predict the absence or presence of the eight characteristics in each sentence of the peer review report.”: The machine learning approach used by the authors is explained in just one sentence. A more elaborate explanation is needed. There are lots of machine learning approaches. The authors need to explain why they use Naïve Bayes. They also need to briefly discuss how Naïve Bayes performs the classification task.

Likewise, I would like to see a proper discussion of the statistical model used by the authors. The authors informally explain their statistical approach. I would find it helpful to see a more formal description (in mathematical notation) of the statistical model used by the authors.

“Most distributions were skewed right, with a peak at 0% showing the number of reviews that did not address the content category (Fig 1).”: I do not understand how the peaks at 0% can be explained. Could this be due to problems in the data (e.g., missing or empty review reports)? The authors need to explain this.

“the prevalence of content related to thoroughness and helpfulness varied widely even between journals with similar journal impact factor”: I am not sure whether the word ‘between’ is correct in this sentence. My understanding is that the authors did not distinguish between variation between journals and variation within journals.

“Some journals now publish peer reviews and authors' responses with the articles”: Consider citing the following paper: https://doi.org/10.1007/s11192-020-03488-4 . I also recently published a blog post on this topic: https://www.leidenmadtrics.nl/articles/the-growth-of-open-peer-review .

“Bibliographic databases have also started to publish reviews.”: In addition to Web of Science, I think the work done by Europe PMC needs to be acknowledged as well. See for instance this poster presented at the recent OASPA conference: https://oaspa.org/wp-content/uploads/2022/09/Melissa-Harrison_COASP-2022-poster_V2.pdf .

“peer review in journals with higher impact factors tends to be more thorough in addressing study methods but less helpful in suggesting solutions or providing examples”: I wonder whether this conclusion is justified. Relatively speaking sentences in reviews for higher impact factor journals are indeed more likely to address methods and less likely to suggest solutions or to provide examples. However, as shown by the authors, reviews for higher impact factor journals tend to be substantially longer than reviews for lower impact factor journals. Therefore it seems that the total number of sentences (as opposed to the proportion of sentences) suggesting solutions or providing examples may be higher in reviews for higher impact factor journals than in reviews for lower impact factor journals. If that is indeed the case, it seems to me the conclusion should be that peer review in higher impact factor journals is both more thorough and more helpful.

Finally, I think it needs to be acknowledged that quality assurance processes of journals consist not only of the work done by peer reviewers but also of the work done the editorial staff of journals. This seems important in particular for more prestigious journals, which presumably make more significant investments in editorial quality assurance processes. The results presented in the paper offer valuable insights into peer review processes, but they provide only a partial picture of the overall quality assurance processes of journals.

Reviewer #2:

[identifies himself as Bernd Pulverer]

Severin et al. add to their previous work (ref 23) on analyzing attributes of scholarly referee reports. Peer review is generally regarded as a pivotal component of the scholarly process and as such quantitative analysis is to be welcomed.

The study is based on a large set of about 10,000 referee report across a broad set if biomedical disciplines and uses human annotation to train machine learning based extraction according to 8 pre-identified categories. The study limits itself to analyzing how the 8 referee report attributes compare across 10 Journal Impact Factor (JIF) bins. Regression modelling is applied. Several of the attributes exhibit no trends, others at best very weak trends. That in itself is notable, as for example the minor negative trend of comments on 'Importance and Relevance' vs. JIF is surprising as higher JIF journals tend to instruct referees to comment specifically on these attributes, which for a core part of the selection criteria of such journals. Stronger correlations are reported for the categories 'Materials and Methods' (positive), 'Presentation and Reporting' and 'Suggestion and Solution' (both negative). The authors conclude that referee reports for higher JIF journals may be more 'thorough' but less 'helpful in suggesting solutions and providing examples'. These trends are notable and not predictable - they are also somewhat difficult to rationalize and it is to the authors credit that they don't overinterpret these numbers beyond the conclusions that 'JIF is a bad predictor of peer review' and in fact end the paper with a balanced strength/weakness analysis. The data and approach as reported in detail, but source data for fig 1-3 should be added.

This is an important area of analysis of general interest and the study is thorough. The conclusions are somewhat limited by restricting the analysis to one variable, the JIF, and by limiting the referee report attributes to 8 categories (see below). With the heavy lifting of the human curated training set in hand, it is a pity that the study was not developed beyond the JIF correlations. As such, this specific analysis appears novel and it is based on a large dataset, albeit form a single source.

Major comments:

1) The dataset is large, but limited to one database (Publons). This may well add biases the data, as the authors note themselves. It would have been helpful to expand the analysis to other databases hosting referee reports, such as ORCID, as well as to journals that publish referee reports alongside their papers, such as BMJ, EMBO, eLife and some Nature branded journals. Minimally to test if the reported trends hold up.

2) It would also have been useful to test for another potential bias: open reports vs. closed reports (still the majority): a collaboration with journals that do not publish their reports (and filtering out referees who posted on Publons or ORCID) would have led to an interesting comparison if the trends are identical when peer review is confidential. Since only aggregate data are reported a journal/publisher collaboration should be feasible.

3) The study is based on 8 categories. It is unclear how these categories were chosen and the detail of how the annotators defined them is limited. More importantly, other important attributes are missing, for example number of experimental requests made vs. number of textual requests made, or % of a referee report dedicated to specific points vs. general discussion/subjective points. Expanding the set would add value. As a minor point, it is noted that one category, 'Importance and Relevance' is explicitly excluded from a number of major journals, such as PLOSONE. This could be a confounding factor. I realize that 'multidisciplinary' journals have already been excluded, and maybe this covers all such journals, but please comment.

4) It is unclear if only research papers were analyzed. It is recommended that other peer reviewed papers such as reviews are excluded.

5) The study shows that the JIF does not predict many of the attributes. With the same dataset other variables could be assessed, such as 'subject area' (already defined in the study as ESI research field, in particular clinical research vs. 'basic research'). This is in particular important as baseline JIF is rather different between such categories, which may be a confounding factor in this analysis, but it may also lead to stronger correlations than JIF. Other variables could be category of paper (short report vs. full research paper) or length of paper (report correlation between paper and referee report length). Other interesting areas would be journal name, journal editorial process, referee age or experience, referee gender, referee affiliation. The authors note that referee age could not be analyzed and discuss other variables noting 'adjusting for additional variables strengthened relationships'. It is recommended that this section is expanded and the data added. The referee geography as a function of JIF is reported in Table 1: it would be interesting to correlate this with that of the corresponding authors of the paper refereed, if that is feasible.

6) The 'trends' seen for 'Importance and Relevance' and 'Example' (fig 3) are reported as statistically significant, but they are very small and arguable hard to interpret on the background of complex confounding factors. 'Criticism' shows arguable a similar range of variation and yet is classed as 'no effect'. I would recommend not to emphasize these.

Minor Comments:

1) The very first sentence of the abstract states that JIF is used as a proxy for journal quality and thus peer review quality. First of all, JIF claims to measure 'impact' not quality and this is a distortion, although both may correlate. Also, as stated it is implied that referees select what is published, which is not the case (editors select assisted by referee input). Thus, even assuming editors select for JIF maximization, the JIF to peer review connection is indirect at best.

2) Please explain why the regression analyses were controlled for review length. 'since longer texts …address more categories' seems tenuous as multiple categories can be assigned to each sentence.

3) Discussion, second paragraph: a key outcome of studies such as this is to develop 'referee credit' systems. Processes such as the referee report analysis applied here can be applied to individual referee reports and referees to aid such a system.

4) The 'Typical words' section could be removed as it is covered in S4.

5) Table 2 is of limited value and could be removed or added as a supplementary figure.

Text suggestions:

1) Abstract , line 3: add also no. papers analyzed here.

2) Abstract, line 9: state whole range to avoid confusion: 0.21-74.70, median 1.2-8.0

3) Introduction, lines 4-9: I suggest to remove the claim that peer review is 'particularly critical for the medical sciences'. This is debatable, but the paper is not restricted to the medical sciences (in fact, as noted above, a comparison between medical and biological sciences in this dataset would add considerably).

4) Introduction, second para: the claim 'in the absence of evidence on the quality of peer review ….proxy measures like JIF…' is tenuous at best. JIF is used as a proxy for 'impact' maybe even 'quality' and peer review is a key part of quality assurance but does not in itself define journal selection. In fact, one could highlight two functions: aiding journal selection; improving paper. Please adapt.

5) Introduction, second para: change 'articles published' to 'articles classed as citable (by ISI-Clarivate)'

6) Discussion, line 14: I am not sure the data definitively show high JIF reports are 'more thorough'.

7) Discussion, 3rd para: I disagree with the hypothesis that 'junior referees might be less able to comment on methodology'. All the evidence point the other way, and this is not surprising since ECRs are practitioners. It is fine to pose a hypothesis of course and then to cite evidence again, but this section could also be deleted as it is - unfortunately - not tested here.

8) Discussion 4th para: Ref 19, 20 are cited in support of transparent/open peer review. Nature was actually rather late in adopting this and others, like BMJ group, EmboPress, BMC series and eLife, could be cited.

9) Discussion 5th para: ORCID should be discussed here.

10) Methods: Publons has been part of Clarivate for years not 'now'.

This referee I not an expert in machine learning or statistical analysis and did therefore not assess these aspects of the work in detail.

Reviewer #3:

While this is a highly interesting study, there are several major questions and issues that preclude a favorable assessment at this point.

1) Introduction

It´s relatively well known, at least in my circles, that the impact factor (IF) is misused to assess journal quality and even single paper quality. However, I do not necessarily agree with the notion that this included an overestimation of peer review quality as well. Curiously, the manuscript also does not provide a single reference to back this extension ("re used to assess the quality of journals and, by extension, the quality of peer review."). This is problematic since the premise of the introduction lies upon this idea.

The authors should provide evidence for the notion that IF and peer review quality are linked or have been perceived as linked.

General comment on the methods: The machine learning pipeline is not very well described, also with the added supplement. A supplement should contain no information that is absolutely necessary to understand the methodology.

I strongly suggest a general revision for clarity using standard machine learning terminology and phrasings, and review what is in the main manuscript and the supplement.

One example:

"We divided the sample into five equally sized subsets and ran the cross-validation five times."

This is how cross-validation was explained. While this explanation _could_ in theory mean cross-validation, this definition could also support other data splits which are not cross-validation.

3) Methods / Classification and Validation

In the section that describes the categories, I noted that some points were labeled as "did the reviewers _discuss_" a certain topic, whereas in others the label was "did the reviewers _comment on_" a topic.

To comment on something or to discuss something is a clear qualitative difference. Is there a reason why these phrasings were used?

4) Methods / metrics

The method sections reads as if the authors only calculated PPV, sensitivity, and the F1 score. However, the supplement S2 also describes and shows the accuracy. I assume that the authors calculated an even bigger set of metrics. So please justify why these particular 3 (or 4) metrics were chosen to be presented in the paper (and why others were not).

5) Methods / Results

Generally, the performance of the models is pretty bad. Even the top three that were chosen for further analysis perform pretty badly. Also, the authors have compared only NB and an SVM. If this was a data science project in a bootcamp, the authors would fail it as they kind of stopped after 40% of the work. Especially boosted trees would have been worth exploring as they consistently rank highest in the literature and in competition compared to simpler algorithms. Together with point 6 (below) and proper hyperparameter tuning it is likely that a boosted tree model would lead to better results.

Looking at S2, it becomes pretty clear that there is an imbalance problem, definitely present for the 5 least common categories. I did not find any mention that the authors adjusted their k-fold crossvalidation for imbalanced data. In this case stratified sampling for the k-fold cross validation is the right method which would likely lead to better and more stable results. To assess this the authors should also report the standard deviation of the cross-validation. I would generally also suggest 10 instead of 5 folds when dealing with such a more complicated setup.

How the paper is written, it is very suggestive that the authors believe that the correlation found between high-impact journals and peer-review focusing more on methods is also causal. While the authors to not claim causality (that per definition cannot be shown by ML alone), the phrasings are still very suggestive, e.g. from the abstract: "In conclusion, peer review in journals with higher journal impact factors tends to be more thorough in discussing the methods used...". There is, however, also the mention of a confounding factor. The authors say that reviewers for high impact journals tended to come from a certain geographic (Europe/NA). So, maybe researchers in Europe/NA are trained to be more focused on methods? Given this confounder and the fact that ML is based on correlation, any conclusions drawn from this study should be phrased much more carefully than currently.

8) General comment

Given how the study was based on available peer reviews the categories for very high impact of course contain those high-impact journals but very well known highest-impact journals like new england journal of medicine are not in the top ten. Are such flagship journals even present in the sample? If not that´s a shortcoming and a limitation. The authors should comment on this.

Overall, the methodological shortcomings of the study make it hard for me find the results trustworthy. The ML modelling should be performed completely and according to state-of-the-art and conclusions should only be drawn on data generated by those final models.

Author response to Decision Letter 1

24 Mar 2023

Submitted filename: Authors response.docx

Decision Letter 2

Dear Dr Egger,

Thank you for your patience while we considered your revised manuscript "Journal Impact Factor and Peer Review Thoroughness and Helpfulness: A Supervised Machine Learning Study" for consideration as a Meta-Research Article at PLOS Biology. Your revised study has now been evaluated by the PLOS Biology editors, the Academic Editor, and the original reviewers.

In light of the reviews, which you will find at the end of this email, we are pleased to offer you the opportunity to address the [comments/remaining points] from the reviewers in a revision that we anticipate should not take you very long. We will then assess your revised manuscript and your response to the reviewers' comments with our Academic Editor aiming to avoid further rounds of peer-review, although might need to consult with the reviewers, depending on the nature of the revisions.

IMPORTANT - Please attend to the following:

a) Reviewer #1 raises a potentially important point about your decision to normalise by length, and indeed all three reviewers mention the way that review length was treated in this study. Please address these and the other concerns raised by the reviewers.

b) Please could you change the Title to "Relationship between Journal Impact Factor and the thoroughness and helpfulness of peer reviews"? Normally we would ask you to incorporate the specific finding(s) in the title, but these are somewhat complex and nuanced, and may change in response to the reviewers' comments.

c) Please ensure that you comply with our Data Policy; specifically, we need you to supply the numerical values underlying Figs 1, 2, 3, S1.1, S2.1, S2.2, S2.3, S2.4, and the 4 Figs in “S3 File”, either as a supplementary data file or as a permanent DOI’d deposition. We note that you have plans to deposit the data and R code in the Harvard Metaverse; however, I will need to see this before accepting the paper for publication, and we will also need you to make a permanent DOI’d version (e.g. in Zenodo).

d) Please cite the location of the data clearly in all relevant main and supplementary Figure legends, e.g. “The data underlying this Figure can be found in S1 Data” or “The data underlying this Figure can be found in https://doi.org/10.5281/zenodo.XXXXX”

We expect to receive your revised manuscript within 1 month. Please email us ( gro.solp@ygoloibsolp ) if you have any questions or concerns, or would like to request an extension.

At this stage, your manuscript remains formally under active consideration at our journal; please notify us by email if you do not intend to submit a revision so that we withdraw the manuscript.

*NOTE: In your point-by-point response to the reviewers, please provide the full context of each review. Do not selectively quote paragraphs or sentences to reply to. The entire set of reviewer comments should be present in full and each specific point should be responded to individually.

*Blot and Gel Data Policy*

We require the original, uncropped and minimally adjusted images supporting all blot and gel results reported in an article's figures or Supporting Information files. We will require these files before a manuscript can be accepted so please prepare them now, if you have not already uploaded them. Please carefully read our guidelines for how to prepare and upload this data: https://journals.plos.org/plosbiology/s/figures#loc-blot-and-gel-reporting-requirements

Roli Roberts

----------------------------------------------------------------

I am pleased to see the significant improvements made by the authors to their paper. I have one remaining comment.

The authors conclude that "this study of peer review characteristics indicates that peer review in journals with higher impact factors tends to be more thorough in addressing study methods but less helpful in suggesting solutions or providing examples". As pointed out in my previous review, I don't think this conclusion is warranted. It disregards the fact that review reports in journals with higher impact factors are much longer, on average, than review reports in journals with lower impact factors. The percentage of sentences in review reports that suggest solutions or provide examples is lower for higher impact factor journals than for lower impact factor journals, but the absolute number of sentences suggesting solutions or providing examples is higher, not lower. In my view, the conclusion therefore should be that journals with higher impact factors provide reviews that are more, not less, helpful in suggesting solutions or providing examples.

In their response to my previous report, the authors point out that "our analyses controlled for the length of peer review". This is exactly the problem. All statistics presented in the paper are percentages rather than absolute numbers, so the authors indeed control for the length of a review report. However, length is a relevant factor that, I would argue, one should not necessarily control for. For instance, suppose we have two review reports. One has a length of 100 words, 50 of which are used to provide suggestions. The other has a length of 1000 words, 200 of which are used to provide suggestions. From a relative point of view, the former report is more helpful in providing suggestions (50% vs. 20% of the words are used to provide suggestions), but from an absolute point of view, the latter report is more helpful in providing suggestions (50 vs. 200 words are used to provide suggestions). In my view, the absolute perspective is more relevant. The latter report is the one that will be more helpful for authors to improve their work.

More generally, the fact that review reports in the highest impact factor category are more than twice as long as review reports in the lowest impact factor category is of major importance and, in my view, needs to be emphasized more strongly. It indicates that higher impact factor journals tend to offer more in-depth peer review than lower impact factor journals. This is an important finding that I believe should be mentioned In the abstract and in the concluding section.

Ludo Waltman

PS I published my previous review online. I had hoped to also publish my new review. However, it seems the authors haven't posted their revised paper on a preprint server. I therefore consider the revised paper to be confidential and I won't publish my review.

Ref #2 Re-review:

The authors are to be commended for the thorough responses.

A number of comments:

1) I appreciate the point that many journals with open review processes share these on Publons (now part of Clarivate 'Web of science'). However, Journal with public but unsigned reports less so. Nonetheless, I agree that scraping the literature for non-Publons listed reports in the absence of standardize identifiers is not trivial. I do believe ORCID profiles can point to referee reports from the 'Review URL' field (cf. https://support.orcid.org/hc/en-us/articles/360006971333-Peer-Review ). This study us based on a large dataset and it is certainly reasonable to restrict the study to this dataset as it is unclear if a broader set of input data would alter the conclusions significantly. These points could be discussed.

2) Thank you for pointing out that signed vs. unsigned reports and referee reports on research article vs. reviews was not assessed - that is fine, but I am unclear why signed reports could not be automatically identified and compared to unsigned reports. Note that I had suggested a third comparison between published and unpublished referee reports, but acknowledge that while very interesting, this would be a complex undertaking that can be discussed.

3) - 6) Thank you for the constructive comments and revision

The minor points are addressed, apart from point 3): I would recommend to highlight more clearly that automated analysis of referee reports for quality attributes may inform a referee credit system that could be used objectively and at scale in research assessment by funders and research institutions.

I assessed the responses to ref #1 and #3 and, leaving aside the technical details on statistics and models, which I did not judge, I believe the responses are thorough and the revisions comprehensive leading to a more informative and balanced manuscript. In particular the causality point by ref 3 (no 7) is important and was addressed.

It may be worth emphasizing the referee report length more, both as a correlation with JIF and in the context of the length control applied here (as discussed in ref #1, point 10; ref #2, minor point2).

The re-work of the manuscript was extensive, the authors have addressed all relevant shortcomings very well.

I have only two minor comments, that should be addressed imo before publication.

1) discussion, p. 12

"Our study shows that the peer reviews submitted to journals with higher Journal Impact Factor may be more thorough than those submitted to lower impact journals. Should, therefore, the Journal Impact Factor be rehabilitated, and used as a proxy measure for peer review quality? "

In the following discussion of this question, and also at other parts of the manuscript, there is the implicit assumption that submitted peer reviews are independent of the impact factor and journal, i.e. the same effort is put into providing a review. But of course that is likely not true. That reviews tend to be shorter and less thorough w/r to methodology in lower impact journals can have two additional confounding factors:

a) People are less thorough _because_ it is a low impact journal, believing they do not need to provide a review of as good quality as for a journal with a higher impact factor.

b) People might also be less thorough, when only basing this on the length(!), because the quality does not warrant more text. Let me explain. If I am confronted with an applied AI in healthcare methodology, that is completely not up to the standards, I might just write exactly that and give some examples in bullet points, and suggest rejection. Confronted with a good methodology that has only _some_ major shortcomings, I will likely take the time (and words) to explain these few shortcomings. My point here is that longer -> "more thorough" does not necessary mean more useful or better. The former case does not _warrant_ more text (and it also does not warrant many suggestions. Is it true that I find the former more often in low impact journals? I do not know. I had one of my worst experiences in this regard in the flagship journal of my field (and the paper was accepted despite the fact that I suggested a reject as the only AI methodology expert). But I think this point could still be considered.

I believe that the discussion would improve if these additional points were also discussed as potentially confounding factors and why this topic is very hard to assess.

2) conclusion, p. 14

"This study of peer review characteristics indicates that peer review in journals with higher impact factors tends to be more thorough in addressing study methods but less helpful in suggesting solutions or providing examples."

I believe that this sentence should be followed by something like (authors should modify as they wish): "These differences may also be influenced by differences in geographical reviewer characteristics, quality of submissions, and the attitude of reviewers towards the journals".

Otherwise the conclusion implies, at least to a degree, that higher impact may lead to more thorough reviews.

Author response to Decision Letter 2

13 Jun 2023

Submitted filename: Authors response R3.docx

Decision Letter 3

21 Jun 2023

Thank you for your patience while we considered your revised manuscript "Relationship between Journal Impact Factor and the Thoroughness and Helpfulness of Peer Reviews" for publication as a Meta-Research Article at PLOS Biology. This revised version of your manuscript has been evaluated by the PLOS Biology editors and the Academic Editor.

Based on our Academic Editor's assessment of your revision, we are likely to accept this manuscript for publication, provided you satisfactorily address the following data and other policy-related requests.

IMPORTANT - please attend to the following:

a) Please ensure that you comply with our Data Policy; specifically, we need you to supply the numerical values underlying Figs 1, 2, 3, S1.1, S2.1, S2.2, S2.3, S2.4, and the 4 Figs in “S3 File”, either as a supplementary data file or as a permanent DOI’d deposition. We note that you mention a Zenodo URL ( https://doi.org/10.5281/zenodo.8006829 ); however, this is not accessible, and I will need to see it before accepting the paper for publication.

b) Please cite the location of the data clearly in all relevant main and supplementary Figure legends, e.g. “The data underlying this Figure can be found in https://doi.org/10.5281/zenodo.8006829”

As you address these items, please take this last chance to review your reference list to ensure that it is complete and correct. If you have cited papers that have been retracted, please include the rationale for doing so in the manuscript text, or remove these references and replace them with relevant current references. Any changes to the reference list should be mentioned in the cover letter that accompanies your revised manuscript.

We expect to receive your revised manuscript within two weeks.

To submit your revision, please go to https://www.editorialmanager.com/pbiology/ and log in as an Author. Click the link labelled 'Submissions Needing Revision' to find your submission record. Your revised submission must include the following:

- a cover letter that should detail your responses to any editorial requests, if applicable, and whether changes have been made to the reference list

- a Response to Reviewers file that provides a detailed response to the reviewers' comments (if applicable)

- a track-changes file indicating any changes that you have made to the manuscript.

NOTE: If Supporting Information files are included with your article, note that these are not copyedited and will be published as they are submitted. Please ensure that these files are legible and of high quality (at least 300 dpi) in an easily accessible file format. For this reason, please be aware that any references listed in an SI file will not be indexed. For more information, see our Supporting Information guidelines:

https://journals.plos.org/plosbiology/s/supporting-information

*Published Peer Review History*

Please note that you may have the opportunity to make the peer review history publicly available. The record will include editor decision letters (with reviews) and your responses to reviewer comments. If eligible, we will contact you to opt in or out. Please see here for more details:

Should you, your institution's press office or the journal office choose to press release your paper, please ensure you have opted out of Early Article Posting on the submission form. We ask that you notify us as soon as possible if you or your institution is planning to press release the article.

Please do not hesitate to contact me should you have any questions.

Senior Editor,

gro.solp@streborr ,

------------------------------------------------------------------------

DATA POLICY:

You may be aware of the PLOS Data Policy, which requires that all data be made available without restriction: http://journals.plos.org/plosbiology/s/data-availability . For more information, please also see this editorial: http://dx.doi.org/10.1371/journal.pbio.1001797

Note that we do not require all raw data. Rather, we ask that all individual quantitative observations that underlie the data summarized in the figures and results of your paper be made available in one of the following forms:

1) Supplementary files (e.g., excel). Please ensure that all data files are uploaded as 'Supporting Information' and are invariably referred to (in the manuscript, figure legends, and the Description field when uploading your files) using the following format verbatim: S1 Data, S2 Data, etc. Multiple panels of a single or even several figures can be included as multiple sheets in one excel file that is saved using exactly the following convention: S1_Data.xlsx (using an underscore).

2) Deposition in a publicly available repository. Please also provide the accession code or a reviewer link so that we may view your data before publication.

Regardless of the method selected, please ensure that you provide the individual numerical values that underlie the summary data displayed in the following figure panels as they are essential for readers to assess your analysis and to reproduce it: Figs 1, 2, 3, S1.1, S2.1, S2.2, S2.3, S2.4, and the 4 Figs in “S3 File.” NOTE: the numerical data provided should include all replicates AND the way in which the plotted mean and errors were derived (it should not present only the mean/average values).

IMPORTANT: Please also ensure that figure legends in your manuscript include information on where the underlying data can be found, and ensure your supplemental data file/s has a legend.

Please ensure that your Data Statement in the submission system accurately describes where your data can be found.

DATA NOT SHOWN?

- Please note that per journal policy, we do not allow the mention of "data not shown", "personal communication", "manuscript in preparation" or other references to data that is not publicly available or contained within this manuscript. Please either remove mention of these data or provide figures presenting the results and the data underlying the figure(s).

Decision Letter 4

Thank you for the submission of your revised Meta-Research Article "Relationship between Journal Impact Factor and the Thoroughness and Helpfulness of Peer Reviews" for publication in PLOS Biology. On behalf of my colleagues and the Academic Editor, Ulrich Dirnagl, I'm pleased to say that we can in principle accept your manuscript for publication, provided you address any remaining formatting and reporting issues. These will be detailed in an email you should receive within 2-3 business days from our colleagues in the journal operations team; no action is required from you until then. Please note that we will not be able to formally accept your manuscript and schedule it for publication until you have completed any requested changes.

IMPORTANT: I note that you mention the reviewers ("Ludo Waltman, Bernd Pulverer and an anonymous reviewer") in the Acknowledgements. While we appreciate the sentiment, this is against PLOS policy, so please could you remove this? I will ask my colleagues to include this request in their list of issues to attend to.

Please take a minute to log into Editorial Manager at http://www.editorialmanager.com/pbiology/ , click the "Update My Information" link at the top of the page, and update your user information to ensure an efficient production process.

PRESS: We frequently collaborate with press offices. If your institution or institutions have a press office, please notify them about your upcoming paper at this point, to enable them to help maximise its impact. If the press office is planning to promote your findings, we would be grateful if they could coordinate with gro.solp@sserpygoloib . If you have previously opted in to the early version process, we ask that you notify us immediately of any press plans so that we may opt out on your behalf.

We also ask that you take this opportunity to read our Embargo Policy regarding the discussion, promotion and media coverage of work that is yet to be published by PLOS. As your manuscript is not yet published, it is bound by the conditions of our Embargo Policy. Please be aware that this policy is in place both to ensure that any press coverage of your article is fully substantiated and to provide a direct link between such coverage and the published work. For full details of our Embargo Policy, please visit http://www.plos.org/about/media-inquiries/embargo-policy/ .

Thank you again for choosing PLOS Biology for publication and supporting Open Access publishing. We look forward to publishing your study. 

Sincerely, 

Roland G Roberts, PhD, PhD

Research Impact

In this guide.

• Getting Started
• Your Researcher Identity
• Author Impact
• Article Impact
• Journal Impact
• Broadening Your Impact
• Resources & Classes

Journal impact

Journal level metrics track citation patterns within journals and determine which journals are highly cited. The commonly known journal metric is the Journal Impact Factor (JIF) generated from the Journal Citation Reports (JCR) from Web of Science. It is important to note that metrics can't tell you which are the 'best' journals, but they can help you identify journals that receive more attention on average than others. Whilst publishing in a highly-cited or highly-discussed journal won't guarantee that your paper will be read, cited or shared, it can help raise the profile of your work and boost your CV. Ultimately, however, the decision of where to publish your work depends on many factors that are beyond the scope of metrics.

• Journal Metric Summary Table This document covers the definition, use, pros, and cons of the majority of journal metrics found in citation databases.

Journal level metric overview

Adapted from:

Colosimo, April. " Concept Map ."  Impact Measurements: Journal-level metrics . McGill University. n.d. Web. 27 Aug. 2020. 

Journal level metrics

• Elsevier (Scopus)
• Clarivate Analytics (Web of Science)
• Google Scholar
• CiteScore Calculates the number of citations received by a journal in one year to documents published in the three previous years divided by the number of documents indexed in Scopus in those same three years
• SCImago Journal Rank (SJR) Calculates a ratio of the average number of weighted citations received in a given year by the number of documents published in the journal in the previous three years
• Source Normalized Impact per Paper (SNIP) Measures contextual citation impact by weighting citations based on the total number of citations in a subject field
• Eigenfactor Score Measures the number of times articles from the journal published in the past five years that have been cited in the JCR
• Journal Impact Factor Mostly widely quoted measure for journals is the journal impact factor (JIF) found in the Journal Citation Reports (JCR) of Web of Science; measures how often an article is cited in any year in a journal
• h5-index Measures the h-index for articles published in the last 5 complete years
• h5-median Measures the median number of citations for the articles that make up its h5-index

Journal ranking

• Journal ranking lists
• Predatory journals
• Journal Quality List Compiled and edited by Dr. Anne-Wil Harzing and includes 18 different rankings for over 900 journals
• Journal Citation Reports Provides transparent, publisher-neutral data and statistics to make confident decisions in today's evolving scholarly publishing landscape
• Scimago Journal & Country Rank (SJR) View rankings by subject category
• Eigenfactor Search by ISI category

Publishing your work is central to your career and the advancement of knowledge in your field, so it’s important to choose a trustworthy and credible journal and publisher, and avoid the perils of “predatory journal" or "predatory publisher." “Predatory journals and publishers are entities that prioritize self-interest at the expense of scholarship and are characterized by false or misleading information, deviation from best editorial and publication practices, a lack of transparency, and/or the use of aggressive and indiscriminate solicitation practices” ( Grudniewicz et al., 2019 ). While there are no definitive list of warning signs, the following lists are updated to keep up with the growing predatory journals and publishers. 

• Beall’s list of possible predatory publishers [Blacklist]
• Directory of Open Access Journals (DOAJ) [Whitelist]
• Committee on Publication Ethics (COPE) [Whitelist]
• Open Access Scholarly Publishers Association (OASPA) [Whitelist]
• "Potential, possible, or probable predatory scholarly open-access publishers" from Scholarly Open Access: Critical analysis of scholarly open-access publishing

Find the right journal for your research

Sharing your research is key to the research lifecycle. It progresses your discipline with greater understandings of the world while contributing to your academic career. However, with thousands of journals available, it can be hard to choose a trusted journal and publisher. The following resources can help you find the right journal for you.

• SciRev Provides information on journal response times and review duration based on feedback from individuals.
• Ulrichsweb A periodical directory with detailed information about journals such as publication location, peer-review, etc.
• Think. Check. Submit Provides a checklist and a range of tools and resources to find credible research and publications.

• << Previous: Article Impact
• Next: Broadening Your Impact >>
• Last Updated: Jul 31, 2024 1:54 PM
• URL: https://laneguides.stanford.edu/researchimpact

International Journal of Research and Review

International Journal of Research and Review (IJRR)

Issn: 2349-9788 (online) issn: 2454-2237 (print).

International Journal of Research and Review (E-ISSN: 2349-9788; P-ISSN: 2454-2237)is a double-blind, Indexed peer-reviewed, open access international journal dedicated to promotion of research in multidisciplinary areas. We define Open Access-journals as journals that use a funding model that does not charge readers or their institutions for access. From the BOAI definition of "Open Access" users shall have the right to "read, download, copy, distribute, print, search, or link" to the full texts of articles. The journal publishes original research article from broad areas like Accountancy, Agriculture, Anthropology, Anatomy, Architecture, Arts, Biochemistry, Bioinformatics, Biology, Bioscience, Biostatistics, Biotechnology, Botany, Chemistry, Commerce, Computer Science, Dairy Technology, Dentistry, Ecology, Economics, Education, Engineering, Environmental Science, Food & Nutrition, Forensic Science, Forestry, Geology, Geography, Health Sciences, History, Home Science, Journalism & Mass Communication, Language, Law, Life Science, Literature, Management, Marine Science, Mathematics, Medical Science, Microbiology, Pathology, Paramedical Science, Pharmacy, Philosophy, Physical Education, Physiotherapy, Physics, Political Science, Public Health, Psychology, Science, Social Science, Sociology, Sports Medicine, Statistics, Tourism, Veterinary Science & Animal Husbandry, Yoga, Zoology etc.

The International Journal of Research and Review (IJRR) provides rapid publication of articles in all areas of research.

Frequency: Monthly Language: English

Scope of Journal

International Journal of Research and Review (IJRR) is a double-blind Indexed peer-reviewed open access journal which publishes original articles, reviews and short communications that are not under consideration for publication elsewhere. The journal publishes papers based on original research that are judged by critical reviews, to make a substantial contribution in the field. It aims at rapid publication of high quality research results while maintaining rigorous review process. The Journal welcomes the submission of manuscripts that meet the general criteria of significance and academic excellence. Papers are published approximately one month after acceptance.

IJRR is dedicated to promote high quality research work in multidisciplinary field.

Index Copernicus Value (ICV) for 2022: 100.00; ICV 2021: 100.00; ICV 2020: 100.00; ICV 2019: 96.58; ICV 2018: 95.79; ICV 2017: 85.08; ICV 2016: 67.22; ICV 2015: 73.64; ICV 2014: 52.85

Salient features of our journal.

✵ Journal is indexed with Index Copernicus and other indexing agencies.

✵ Print version (hard copy) with P-ISSN number is available.

✵ Fast Publication Process.

✵ Free e-Certificate of Publication on author's request.

How to Submit Manuscript:

Thesis publication.

Research Articles

Nitin V Vichare, Rakesh Maggon

Smita Deshkar, Niranjan Patil, Pranali Balmiki

Ashish Choudhary, Riyaz Farooq, Aamir Rashid Purra, Fayaz Ahmed Ahangar

Joshi Medhavi H., Desai Devangi S, Singh Lalli M, Joshi Vaibhavi A

Jasvir Singh

Mariya Thomas Chacko, Tima Babu, Jenin Vincent Parambi, Irene T. Ivan, Sheik Haja Sherief

Click Here to Read more...

Anwar Sholeh, Alwi Thamrin Nasution, Radar Radius Tarigan

Arnaldi Fernando, Dairion Gatot, Yan Indra Fajar Sitepu

Iji Clement O, Age Terungwa James

Varsha S. Dhage, Pratibha Chougule

Uzma Khan, Arindam Ghosh

Dr. Uzma Khan, Dr. Arindam Ghosh, Dr. John Pramod

Our Journals

Visit Website

International Journal of Science and Healthcare Research

Galore International Journal of Health Sciences and Research

Galore International Journal of Applied Sciences and Humanities

• Feature Article
• Computational and Systems Biology

Meta-Research: Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations

• Lesley A Schimanski
• Carol Muñoz Nieves
• Lisa Matthias
• Meredith T Niles
• Universidad Nacional Autónoma de México, Mexico ;
• Simon Fraser University, Canada ;
• Freie Universität Berlin, Germany ;
• University of Vermont, United States ;
• Open access
• Copyright information
• Comment Open annotations (there are currently 0 annotations on this page).
• 12,300 views
• 944 downloads
• 169 citations

Share this article

Cite this article.

• Erin C McKiernan
• Juan P Alperin
• Copy to clipboard
• Download BibTeX
• Download .RIS
• Figures and data

Introduction

Conclusions, data availability, decision letter, author response, article and author information.

We analyzed how often and in what ways the Journal Impact Factor (JIF) is currently used in review, promotion, and tenure (RPT) documents of a representative sample of universities from the United States and Canada. 40% of research-intensive institutions and 18% of master’s institutions mentioned the JIF, or closely related terms. Of the institutions that mentioned the JIF, 87% supported its use in at least one of their RPT documents, 13% expressed caution about its use, and none heavily criticized it or prohibited its use. Furthermore, 63% of institutions that mentioned the JIF associated the metric with quality, 40% with impact, importance, or significance, and 20% with prestige, reputation, or status. We conclude that use of the JIF is encouraged in RPT evaluations, especially at research-intensive universities, and that there is work to be done to avoid the potential misuse of metrics like the JIF.

The Journal Impact Factor (JIF) was originally developed to help libraries make indexing and purchasing decisions for their journal collections ( Garfield, 2006 ; Archambault and Larivière, 2009 ; Haustein and Larivière, 2015 ), and the metric’s creator, Eugene Garfield, made it clear that the JIF was not appropriate for evaluating individuals or for assessing the significance of individual articles ( Garfield, 1963 ). However, despite this and the various well-documented limitations of the metric (e.g., Seglen, 1997 ; Moustafa, 2015 ; Brembs et al., 2013 ; The PLOS Medicine Editors, 2006 ; Kurmis, 2003 ; Sugimoto and Larivière, 2018 ; Haustein and Larivière, 2015 ; The Analogue University, 2019 ), over the past few decades the JIF has increasingly been used as a proxy measure to rank journals – and, by extension, the articles and authors published in these journals ( Casadevall and Fang, 2014 ). The association between the JIF, journal prestige, and selectivity is strong, and has led academics to covet publications in journals with high JIFs ( Harley et al., 2010 ). Publishers, in turn, promote their JIF to attract academic authors ( Hecht et al., 1998 ; Sugimoto and Larivière, 2018 ; SpringerNature, 2018 ).

In some academic disciplines, it is considered necessary to have publications in journals with high JIFs to succeed, especially for those on the tenure track (for review see  Schimanski and Alperin, 2018 ). Institutions in some countries financially reward their faculty for publishing in journals with high JIFs ( Fuyuno and Cyranoski, 2006 ; Quan et al., 2017 ), demonstrating an extreme but important example of how this metric may be distorting academic incentives. Even when the incentives are not so clear-cut, faculty still often report intense pressure to publish in these venues ( Harley et al., 2010 ; Walker et al., 2010 ; Tijdink et al., 2016 ). Concerns about the JIF and journals’ perceived prestige may also limit the adoption of open access publishing ( Schroter et al., 2005 ; Swan and Brown, 2004 ; University of California Libraries, 2016 ), indicating how the effects of the JIF permeate to the broader scholarly publishing ecosystem.

This use – and potential misuse – of the JIF to evaluate research and researchers is often raised in broader discussions about the many problems with current academic evaluation systems ( Moher et al., 2018 ). However, while anecdotal information or even formal surveys of faculty are useful for gauging the JIF’s effect on the academic system, there is still a lot we do not know about the extent to which this metric is used in formal academic evaluations. To our knowledge, there have been no studies analyzing the content of university review, promotion, and tenure (RPT) guidelines to determine the extent to which the JIF is being used to evaluate faculty, or in what ways. We therefore sought to answer the following questions: (1) How often is the JIF, and closely related terms, mentioned in RPT documents? (2) Are the JIF mentions supportive or cautionary? (3) What do RPT documents assume the JIF measures?

Document collection

This paper reports a set of findings from a larger study ( Alperin et al., 2019 ) for which we collected documents related to the RPT process from a representative sample of universities in the United States and Canada and many of their academic units. A detailed description of the methods for selecting institutions to include in our sample, how we classified them, how we collected documents, and the analysis approach is included in Alperin et al. (2019) and in the methodological note accompanying the public dataset ( Alperin et al., 2018 ). Briefly, we created a stratified random sample based on the 2015 edition of the Carnegie Classification of Institutions of Higher Education ( Carnegie Foundation for the Advancement of Teaching, 2015 ) and the 2016 edition of the Maclean’s University Rankings ( Rogers Digital Media, 2016 ), which respectively group US and Canadian universities into those focused on doctoral programs (i.e., research intensive; R-type), those that predominantly grant master’s degrees (M-type), and those that focus on undergraduate programs (i.e., baccalaureate; B-type). We used a taxonomy developed by the National Academies in the United States ( The National Academies of Sciences, Engineering and Medicine, 2006 ) to classify the academic units (e.g., department, school, or faculty) within an institution into three major areas: Life Sciences (LS); Physical Sciences and Mathematics (PSM); and Social Sciences and Humanities (SSH). Additional units that could not be classified as belonging to a single area (e.g., a College of Arts and Sciences) were designated as multidisciplinary. The stratified sample was designed to collect documents from enough institutions in each of the R-, M-, and B-type categories to have a statistical power of .8 (assuming a small effect size of .25 of a standard deviation) when making comparisons between categories. An overview of the population of universities by type, the number and percent randomly chosen for our stratified sample, and the number of institutions for which we obtained at least one relevant document can be found in Table 1 . A more detailed table, including institution sub-types, can be found in  Alperin et al. (2019) .

We then used a combination of web searches, crowdsourcing, and targeted emailing to request documents related to the RPT process, including but not limited to collective agreements, faculty handbooks, guidelines, and forms. Some of these documents applied to the institution as a whole, while others applied only to specific academic units. In the end, we obtained 864 documents related to the RPT process of 129 universities, of which 57 were R-type, 39 were M-type, and 33 were B-type institutions. Of the total documents, 370 were institutional-level documents, while the remaining 494 came from 381 academic units within 60 of these universities. Of the 116 units at R-type institutions, 33 (28%) were LS units, 21 (18%) were PSM units, 39 (34%) were SSH units, and 23 (20%) were multidisciplinary units.

Sampling summary of universities from Canada and the United States.

	Number in category	Number sampled	Percent sampled	Number with documents
R-type	350	65	19%	57
M-type	847	50	6%	39
B-type	602	50	8%	33

Document analysis and coding terminology

The RPT documents were loaded into QSR International’s NVivo 12 qualitative data analysis software, where text queries were used to identify documents that mention specific terms. Because the language in RPT documents varies, we first searched all the documents for the words “impact” and “journal”, and read each mention to identify terms that may be referencing the JIF. We classified these terms into three groups: (1) direct references to the JIF as a metric; (2) those that reference journal impact in some way; and (3) indirect but possible references to the JIF. In the first group, we included the terms “impact factor”, “impact score”, “impact metric”, and “impact index”. In the second group, we included the terms “high-impact journal”, “impact of the journal”, and “journal(’s) impact”. The third group contains a larger number and variety of terms, such as “high-ranking journal", “top-tier journal”, and “prestigious journal”. For all terms, we considered both singular and plural equivalents. A map of the terms we found and their grouping into the three categories can be seen in Figure 1 . In our analysis, we looked at only the first two groups of terms, as we considered them to be unambiguously about the JIF (group 1) or sufficiently close to the notion of JIF (group 2). The terms in the third group, however, may or may not refer to the JIF. So while these terms could represent examples of ways in which the idea of the JIF is invoked without being explicit, their mentions were not analyzed further for this study.

Grouping of terms related to the JIF.

Terms found in RPT documents were classified as either: (1) referring directly to the JIF (inner ring); (2) referring in some way to journal impact (middle ring); or (3) indirect but probable references to the JIF. For simplicity, singular versions of each term are shown, but searches included their plural equivalents. Our analysis is based only on those terms found in groups 1 and 2 (the two innermost rings).

The results of each text query for the terms in groups 1 and 2 were placed in an NVivo “node” that contained the text surrounding each of the mentions. We then performed a “matrix coding query” to produce a table with institutions and academic units as rows, terms of interests as columns, and a 1 or a 0 indicating whether the institution or academic unit made mention of the term or not, with the ability to distinguish if the mention appeared in documents that pertain to the whole institution, to one or more academic units, or both. We considered an institution as making mention of a term if the term was present in at least one document from that institution or any of its academic units. More details on this process can be found in  Alperin et al. (2019) .

Qualitative analysis

We also exported the content of each node for a qualitative analysis of the JIF mentions. In some cases, the software extracted complete sentences, while in other cases it pulled only fragments and we retrieved the rest of the text manually to provide better context. Based on a detailed reading of the text, we classified each of the JIF mentions along two dimensions. First, we classified each mention as either: (1) supportive of the JIF’s use in evaluations; (2) cautious , meaning the document expresses some reservations about the use of the JIF in evaluations; or (3) neutral , meaning the mention was neither supportive nor cautious, or not enough information was present in the document to make a judgement. In addition, we read each mention to determine what aspects of research were being measured with the JIF, if specified. Using categories we arrived at inductively, we classified each mention of the JIF as associating the metric with one or more of the following: (i) quality of the research and/or journal; (ii) impact, importance, or significance of the research or publication; (iii) prestige, reputation, or status of the journal or publication; or (iv) left unspecified, meaning the document mentions the JIF, but does not state what the metric is intended to measure. If an institution contained multiple mentions (for example, in two different academic units), it was counted under all the relevant categories.

To arrive at the classification, each mention was independently coded by two of the authors (EM and LM) using the definitions above. After an initial pass, the two coders agreed on all of the classifications for 86% of all mentions. The remaining mentions were independently coded by a third author (LS). In all instances, the third coder agreed with one of the previous two, and this agreement was taken as the final code.

We have shared the data on which this paper is based in two different formats: (1) a spreadsheet with all the JIF-related mentions (including repetitions) extracted from the RPT documents, available as part of the larger public dataset ( Alperin et al., 2018 ), and (2) a text document containing the mentions (minus repetitions), with terms of interest color coded and a qualitative assessment of each quote, available as supplemental information. The main data file in  Alperin et al. (2018) ( scholcommlab-rpt-master-april-2019.tab ) contains two columns for the JIF ( metrics_impact_factor and metrics_high_impact_journals ). A 1 in these columns indicates that at least one document from that institution or any of its academic units contained a JIF term from groups 1 or 2 ( Figure 1 ), respectively, while a 0 indicates no such terms were found in any of the documents for that institution. A set of columns with the prefix if_ similarly contain a 1 if the JIF mention was coded for each category, and a 0 otherwise. We are not able to share the original RPT documents collected for this study, since the copyrights are held by the universities and academic units that created them. However, for publicly available documents, we included Wayback Machine web archive links to them in the shared spreadsheet.

Limitations

Our study covers a broad range of document types that spans an equally diverse range of institutions and academic units. Although we believe the documents analyzed are representative of what is used in practice in RPT evaluations, the diversity of forms and practices means that some documents contain more details than others regarding what is expected of faculty. As a result, the lack of presence of the JIF-related terms may be due to the types of document used at those institutions, and not a lack of interest or focus on using the metric for evaluation.

Along the same lines, we must also recognize that in studying the RPT process through a document-centric approach, our analysis remains limited to what is formalized in the documents themselves. It cannot tell us how RPT committees use the JIF or other citation metrics, if at all, during the process, nor how faculty use these guidelines in preparing their dossiers for review. To this end, we echo the call of  O'Meara (2002) and our own previous study ( Alperin et al., 2019 ) for more work that studies the relationship between RPT guidelines and faculty behaviors, while offering this empirical analysis of RPT documents as foundational evidence.

How often is the JIF mentioned in RPT documents?

While metrics in general are mentioned in RPT documents from 50% of institutions in our sample ( Alperin et al., 2019 ), only 23% mentioned the JIF explicitly or used one of the JIF-related terms (see groups 1 and 2 in Figure 1 ) in their RPT documents. The percentage was higher for R-type institutions (40%) than for either M-type (18%) or B-type (0%) institutions ( Table 2 ). Some mentions were found in the institutional-level documents, while others were found at the level of the academic unit (e.g., college, school, or department). Many of the mentions were from different academic units within the same university. Within the R-type institutions, the percentage of academic units that mentioned JIF-related terms was higher for LS (33%) and PSM (29%) units than for SSH (21%) or multidisciplinary units (17%).

Mentions of the JIF in RPT documents, overall and by institution type.

Note that percentages do not sum to one hundred in any given column, since many institutions had more than one JIF mention that could be classified differently. For example, an institution was marked as having a supportive mention if at least one RPT document from that institution, or any of its academic units, had a supportive mention. The same institution could also be counted under ‘cautious’ if a different academic unit within that institution had such a mention.

		All	R-type	M-type	B-type
How many institutions mention the JIF?	n	129	57	39	33
	JIF mentioned	30 (23%)	23 (40%)	7 (18%)	0 (0%)
Are the JIF mentions supportive or cautionary?	n	30	23	7	0
	supportive	26 (87%)	19 (83%)	7 (100%)	-
	cautious	4 (13%)	3 (13%)	1 (14%)	-
	neutral	5 (17%)	4 (17%)	1 (14%)	-
What do institutions measure with the JIF?	n	30	23	7	0
	quality	19 (63%)	14 (61%)	5 (71%)	-
	impact/importance/significance	12 (40%)	8 (35%)	4 (57%)	-
	prestige/reputation/status	6 (20%)	5 (22%)	1 (14%)	-
	unspecified	23 (77%)	17 (74%)	6 (86%)	-

Are the JIF mentions supportive or cautionary?

The majority of mentions of the JIF were supportive of the metric’s use in evaluations. Overall, 87% of institutions that mentioned the JIF did so supportively in at least one of their RPT documents from our sample ( Table 2 ). Breaking down by institution type, 83% of R-type and 100% of M-type institutions had supportive mentions. In contrast, just 13% of institutions overall had at least one mention which expressed caution about using the JIF in evaluations. Two institutions (University of Central Florida and University of Guelph) had both supportive and cautious mentions of the JIF, but originating from different academic units. Overall, 17% of institutions had at least one neutral mention. Examples of supportive and cautious mentions can be found in the following two sections. Examples of neutral mentions are in the supplemental information.

What do RPT documents assume the JIF measures?

Associating the jif with quality.

The most commonly specified association we observed in these RPT documents was between the JIF and quality, seen in 63% of institutions overall ( Table 2 ). By institution type, 61% of R-type and 71% of M-type institutions in our sample that mention the JIF associate the metric with quality. This association can be seen in the guidelines from the Faculty of Science at the University of Alberta that state: “Of all the criteria listed, the one used most extensively, and generally the most reliable, is the quality and quantity of published work in refereed venues of international stature. Impact factors and/or acceptance rates of refereed venues are useful measures of venue quality…” ( University of Alberta, 2012 ).

While some RPT documents recommend using the JIF to determine the quality of a journal, others suggest that this metric can be used to indicate the quality of individual publications. An example of the latter comes from the Department of Political Science, International Development, and International Affairs at the University of Southern Mississippi: “Consideration will be given to publication quality as measured by the following items (though not exclusive of other quality measures not listed here): journal/press rankings, journal/press reputation in the field, journal impact factors, journal acceptance rates, awards, citations, reviews and/or reprints” ( University of Southern Mississippi, 2016 ).

Other guidelines create their own metrics using the JIF in their calculations and suggest this will incentivize high quality research, as seen in the following example from the Institute of Environmental Sustainability at Loyola University: “For promotion to Professor, the candidate must have an average publication rate of at least one article per year published in peer-reviewed journals in the five-year period preceding the application for promotion. These articles should be regularly cited by other researchers in the field. We will consider both the quality of the journal (as measured by the journal’s impact factor, or JIF) as well as the number of citations of each publication. We will employ the metric: Article Impact Factor (AIF) = (JIF * citations) where “citations” represents the number of citations for the particular publication. Employing this metric, faculty have incentive to publish in the highest quality journals (which will increase the JIF) and simultaneously produce the highest quality research manuscripts, potentially increasing the number of citations, and increasing the AIF” ( Loyola University Chicago, 2015 ).

In sum, there are repeated links made in the sampled RPT documents between the JIF and research, publication, or journal quality.

Associating the JIF with impact, importance, or significance

The second most common specified association we observed in these RPT documents was between the JIF and the impact, importance, or significance of faculty research or publications, found in 40% of institutions in our sample. By institution type, 35% of R-type and 57% of M-type institutions made this association ( Table 2 ). For example, guidelines from the Department of Psychology at Simon Fraser University link the JIF with impact: “The TPC [Tenure and Promotion Committee] may additionally consider metrics such as citation figures, impact factors, or other such measures of the reach and impact of the candidate’s scholarship” ( Simon Fraser University, 2015 ).

Promotion and tenure criteria from the University of Windsor link the JIF to publication importance ( University of Windsor, 2016 ): “Candidates will be encouraged to submit a statement that explains the importance of their publications, which may include factors such as journal impact factors, citation rates, publication in journals with low acceptance rates, high levels of readership, demonstrated importance to their field.”

Guidelines from the Department of History at the University of California, Los Angeles associate the JIF with significance of faculty work: “The [policy on academic personnel]’s concern that the candidate be “continuously and effectively engaged in creative activity of high quality and significance,” should further be demonstrated through other publications that include peer reviewed articles in high impact journals…”.

In all of the above cases, the value of faculty research or individual publications is being evaluated, at least in part, based on the JIF.

Associating the JIF with prestige, reputation, or status

A third set of mentions of the JIF associated the metric with prestige, reputation, or status, typically referring to the publication venue. Overall, 20% of institutions in our sample that mentioned the JIF made such an association. As with other concepts, there was variability by institution type, with 22% of the R-type and 14% of the M-type having at least one instance of this association ( Table 2 ). For example, guidelines from the Department of Sociology at the University of Central Florida link the JIF with prestige: “It is also true that some refereed journal outlets count for more than others. Publication in respected, highly cited journals, that is, counts for more than publication in unranked journals. The top journals in sociology and all other social sciences are ranked in the Thompson/ISI citation data base (which generates the well-known Impact Factors), in the Scopus data base, and in certain other citation data bases. In general, it behooves faculty to be aware of the prestige rankings of the field’s journals and to publish in the highest-ranked journals possible. It is also advisable to include in one’s tenure and promotion file information about the Impact Factors or related metrics for the journals where one’s papers appear” ( University of Central Florida, 2015 ).

Similarly, promotion and tenure forms from the University of Vermont associate the JIF with journal status: “List all works reviewed prior to publication by peers/editorial boards in the field, such as journal articles in refereed journals, juried presentations, books, etc. Indicate up to five of the most important contributions with a double asterisk and briefly explain why these choices have been made. Include a description of the stature of journals and other scholarly venues and how this is known (e.g., impact factors, percentage of submitted work that is accepted, together with an explanation of the interpretation of these measures)” ( University of Vermont, 2016 ).

Overall, these documents show a focus on publication venue and use the JIF as a proxy measure for determining how much individual publications should count in evaluations based on where they are published.

Many mentions do not specify what is measured with the JIF

Lastly, we were left with many instances where the JIF was mentioned without additional information on what it is intended to measure. Such unspecified mentions were found in the RPT documents of 77% of institutions that mentioned the JIF. These correspond to 74% of the R-type institutions and 86% of the M-type institutions with mentions ( Table 2 ). These mentions were often found in research and scholarship sections that ask faculty to list their publications and accompanying information about the publication venues, such as the JIF or journal rank. Some of these documents simply suggest the JIF be included, while others make it a requirement. For example, guidelines from the Russ College of Engineering and Technology at Ohio University request the JIF in the following way: “List relevant peer-reviewed journal and conference papers published over the last five years (or since last promotion or initial appointment, whichever is less) related to pedagogy or other relevant areas of education. Include the journal’s impact factor (or equivalent journal ranking data) and the number of citations of the article(s)” ( Ohio University, 2015 ).

Not all mentions of the JIF support its use

While the majority of the mentions found in our sample of RPT documents were either neutral or supportive of the JIF, we find that 13% of institutions had at least one mention which cautioned against or discouraged use of the JIF in evaluations. We observed varying levels of caution in these mentions. Some do not critique use of the JIF in general, but rather express concern that JIF data are not as relevant for their discipline as for others. For example, criteria for promotion and tenure from the School of Social Work at the University of Central Florida state: “Journal impact factors will not be a primary criteria for the measurement of scholarly activity and prominence as the academic depth and breadth of the profession requires publication in a multitude of journals that may not have high impact factors, especially when compared to the stem [sic] disciplines” ( University of Central Florida, 2014 ).

Similarly, guidelines from the Department of Human Health and Nutritional Sciences at the University of Guelph call the JIF a "problematic" index and discourage its use while again highlighting disciplinary differences: “Discussion of journal quality (by those familiar with the field) may be included in the assessment in addition to consideration of the quality of individual research contributions. However, citation analyses and impact factors are problematic indices, particularly in comparisons across fields, and their use in the review process is not encouraged” ( University of Guelph, 2008 ).

Other guidelines, such as those from the Faculty of Veterinary Medicine at the University of Calgary, caution against relying solely on the JIF as a measure of quality, but still allow it to be considered: “Special consideration is to be given to the quality of the publication and the nature of the authorship. Contributions of the applicant must be clearly documented. The reputation and impact of the journal or other publication format will be considered, but takes secondary consideration to the quality of the publication and the nature of the contributions. Impact factors of journals should not be used as the sole or deciding criteria in assessing quality” ( University of Calgary, 2008 ).

Some RPT documents even seem to show disagreement within evaluation committees on the use of the JIF. For example, a document from the Committee on Academic Personnel at the University of California, San Diego reads: “CAP [Committee on Academic Personnel] welcomes data on journal acceptance rates and impact factors, citation rates and H-index, but some CAP members (as do senior staff of scholarly societies) retain various degrees of skepticism about such measures” ( University of California, San Diego, 2016 ).

None of the RPT documents we analyzed heavily criticize the JIF or prohibit its use in evaluations.

To our knowledge, this is the first study of RPT documents from a representative sample of US and Canadian universities to analyze the use of the JIF in academic evaluations. We found that 40% of R-type and 18% of M-type institutions mentioned the JIF or related terms in their RPT documents. Mentions were largely supportive of JIF use, with 87% of institutions having at least one supportive mention, while just 13% had cautious mentions. The most common specified association we observed in these documents was between the JIF and quality.

How prevalent is the use of the JIF in evaluations?

Mentions of the JIF and related terms in RPT documents are not as ubiquitous as the amount of discussion of current evaluation systems would suggest – 23% of institutions in our sample used these terms explicitly. Sample considerations, including the relatively small total number of institutions included, could be a factor in calculating the prevalence of the use of the JIF. However, given our stratified random sampling approach, we consider our sample to be representative and a good indicator of what would be found in the larger population of U.S. and Canadian universities. Importantly, we note that the results differ depending on institution type, which might suggest that the experiences at R-type universities (where mentions of the JIF were most prevalent) play an outsized role in discussions about evaluation. Furthermore, the analysis we present on the terms in groups 1 and 2 of our coding terminology (see Figure 1 ) may represent only the tip of the iceberg. That is, while we analyzed only those terms that were very closely related to the JIF, we also observed (but did not analyze) terms such as ‘major’, ‘prestigious’, ‘prominent’, ‘highly respected’, ‘highly ranked’, and ‘top tier’ that may be associated with high JIFs in the minds of evaluators. It is impossible to know how RPT committee members interpret such phrases on the basis of the documents alone, but we suspect that some of these additional terms serve to invoke the JIF without explictly naming it. Take the following example from the Department of Anthropology at Boise State University that leaves open for interpretation what measure is used for determining a journal’s status (emphasis added): “The candidate for promotion to associate rank should have a least two publications in upper-tier journals ”.

Such examples do not explicitly mention the JIF (and thus are not counted in our analysis), but do imply the need for some measure for ranking journals. It seems likely, given the ubiquity of the JIF, that some committee members will rely on this metric, at least in part, for such a ranking. In short, counting mentions of a restricted set of terms, as we have done here, is likely an underestimate of the extent of the use of the JIF in RPT processes. However, we believe the in-depth analysis presented herein provides a glimpse into the current use of the JIF and may indicate how faculty are considering the metric in evaluations, particularly with respect to assessments of quality.

The JIF does not measure quality

The association between the JIF and quality was found in 63% of institutions in our sample, but is there evidence that the JIF is a good indicator of quality? Although quality is hard to define, and even harder to measure, there are aspects of methodological rigor which could be considered indicative of quality, such as sample size, experimental design, and reproducibility ( Brembs, 2018 ). What is the relationship between these aspects of a study and the JIF?

Evidence suggests that methodological indicators of quality are not always found in journals with high JIFs. For example, Fraley and Vazire (2014) found that social and personality psychology journals with the highest JIFs tend to publish studies with smaller sample sizes and lower statistical power. Similarly, Munafò et al. (2009) report that higher-ranked journals tend to publish gene-association studies with lower sample sizes and overestimate effect sizes. Analyses of neuroscience and/or psychology studies show either no correlation ( Brembs et al., 2013 ) or a negative correlation ( Szucs and Ioannidis, 2017 ) between statistical power and the JIF.

Several studies have looked at experimental design to assess methodological rigor and quality of a study. Chess and Gagnier (2013) analyzed clinical trial studies for 10 different indicators of quality, including randomization and blinding, and found that less than 1% of studies met all 10 quality criteria, while the JIF of the journals did not significantly predict whether a larger number of quality criteria were met. Barbui et al. (2006) used three different scales that take into account experimental design, bias, randomization, and more to assess quality, and found no clear relationship between the JIF and study quality.

Reproducibility could be used as a measure of quality, since it requires sufficient methodological care and detail. Bustin et al. (2013) analyzed molecular biology studies and found key methodological details lacking, reporting a negative correlation between the JIF and the amount of information provided in the work. Mobley et al. (2013) found that around half of biomedical researchers surveyed reported they were unable to reproduce a published finding, some from journals with a JIF over 20. Prinz et al. (2011) found “that the reproducibility of published data did not significantly correlate with journal impact factors” (pg. 2).

Thus, at least as viewed through the aspects above, there is little to no evidence that the JIF measures research quality. For a more comprehensive review, see Brembs (2018) .

Improving academic evaluation

In the last few years, several proposals and initiatives have challenged the use of the JIF and promoted the responsible use of metrics to improve academic evaluations. These include the Leiden Manifesto ( Hicks et al., 2015 ), the Metric Tide report ( Wilsdon et al., 2015 ), the Next-Generation Metrics report ( Wildson et al., 2017 ), and HuMetricsHSS ( humetricshss.org ). We provide a brief description of some such efforts (for a review, see Moher et al., 2018 ).

Probably the most well-known initiative is the Declaration on Research Assessment (DORA; sfdora.org ). DORA outlines limitations of the JIF, and puts forward a general recommendation to not use the JIF in evaluations, especially as a “surrogate measure of the quality of individual research articles” ( sfdora.org/read ). Particularly relevant to our current research is DORA’s recommendation for institutions to “be explicit about the criteria used to reach hiring, tenure, and promotion decisions, clearly highlighting…that the scientific content of a paper is much more important than publication metrics or the identity of the journal in which it was published.” DORA’s new strategic plan ( DORA Steering Committee, 2018 ) includes spreading awareness of alternatives to the JIF and collecting examples of good evaluation practices ( sfdora.org/good-practices ). To date, DORA has been signed by over 1400 organizations and 14,000 individuals worldwide. None of the institutions in our sample are DORA signatories, but it would be interesting to study how commitment to DORA might be reflected in changes to an institution’s RPT documents and evaluations.

Libraries are leaders in promoting the responsible use of metrics, developing online guides (see, for example,  Duke University Medical Center Library & Archives, 2018 ; University of Illinois at Urbana Champaign Library, 2018 ; University of Surrey Library, 2018 ; University of York Library, 2018 ), and providing in-person advising and training for faculty in publishing and bibliometrics. The Association of College and Research Libraries (ACRL) has developed a Scholarly Communication Toolkit on evaluating journals ( Association of College & Research Libraries, 2018 ), which outlines ways to assess journal quality that go beyond metrics like the JIF. LIBER (Ligue des Bibliothèques Européennes de Recherche) has established a working group which recently recommended increased training in metrics and their responsible uses ( Coombs and Peters, 2017 ). The Measuring your Research Impact (MyRI) project ( myri.conul.ie ) is a joint effort by three Irish academic libraries to provide open educational resources on bibliometrics. The Metrics Toolkit ( www.metrics-toolkit.org ) is a collaborative project by librarians and information professionals to provide “evidence-based information" on traditional and alternative metrics, including use cases.

Overall, our results support the claims of faculty that the JIF features in evaluations of their research, though perhaps less prominently than previously thought, at least with respect to formal RPT guidelines. Importantly, our analysis does not estimate use of the JIF beyond what is found in formal RPT documents, such as faculty members who serve on review committees and pay attention to this metric despite it not being explicitly mentioned in guidelines. Future work will include surveying faculty members, particularly those who have served on RPT committees, to learn more about how they interpret and apply RPT guidelines in evaluations and investigate some of the more subjective issues not addressed in this study.

Our results also raise specific concerns that the JIF is being used to evaluate the quality and significance of research, despite the numerous warnings against such use ( Brembs et al., 2013 ; Brembs, 2018 ; Moustafa, 2015 ; Haustein and Larivière, 2015 ; Sugimoto and Larivière, 2018 ; Seglen, 1997 ; Kurmis, 2003 ; The Analogue University, 2019 ). We hope our work will draw attention to this issue, and that increased educational and outreach efforts, like DORA and the library-led initiatives mentioned above, will help academics make better decisions regarding the use of metrics like the JIF.

The data that support the findings of this study are available in the Harvard Dataverse with the identifier https://doi.org/10.7910/DVN/VY4TJE (Alperin et al., 2018). These data include the list of institutions and academic units for which we have acquired documents along with an indicator of whether terms related to the impact factor were found in the documents for the institution or academic unit, as well as the qualitative coding of each mention reported.

• Muñoz Nieves C
• Schimanski L
• McKiernan EC
• McKiernan E
• Schimanski LA
• Fischman GE
• Google Scholar
• Archambault É
• Larivière V
• Association of College & Research Libraries
• Hellemans J
• Schjerling P
• Beaulieu JF
• Carrasco-Ramiro F
• Ciborowski K
• Cornillie P
• De Brouwer S
• De Ceuninck L
• De Craene J
• De Naeyer H
• De Spiegelaere W
• Ferreira-Teixeira M
• Gonzalo-Flores S
• Heindryckx F
• Karampelias M
• Mudarra-Rubio A
• Pérez-Novo C
• Rodrigues-Santos P
• Schmidt-Bleek K
• Skovgaard K
• Van Acker T
• Van den Broeck W
• Van der Meulen J
• Van Poucke M
• Tshuikina-Wiklander M
• Vandesompele J
• Carnegie Foundation for the Advancement of Teaching
• Casadevall A
• DORA Steering Committee
• Duke University Medical Center Library & Archives
• Cyranoski D
• Earl-Novell S
• Sandberg AA
• de Rijcke S
• Loyola University Chicago
• Ioannidis JPA
• Ohio University
• Asadullah K
• Rogers Digital Media
• Simon Fraser University
• SpringerNature
• Ioannidis JP
• The Analogue University
• The National Academies of Sciences, Engineering and Medicine
• The PLOS Medicine Editors
• Maclaine Pont P
• Smulders YM
• University of Alberta
• University of Calgary
• University of California Libraries
• University of California, San Diego
• University of Central Florida
• University of Guelph
• University of Illinois at Urbana Champaign Library
• University of Southern Mississippi
• University of Surrey Library
• University of Vermont
• University of Windsor
• University of York Library
• Hemmelgarn BR
• Emma Pewsey Reviewing Editor; eLife, United Kingdom
• Peter Rodgers Senior Editor; eLife, United Kingdom
• Björn Brembs Reviewer; FU Berlin, Germany

In the interests of transparency, eLife includes the editorial decision letter and accompanying author responses. A lightly edited version of the letter sent to the authors after peer review is shown, indicating the most substantive concerns; minor comments are not usually included.

Thank you for submitting your article "Use of the Journal Impact Factor in academic review, promotion, and tenure evaluations" for consideration by eLife . Your article has been reviewed by two peer reviewers, and the evaluation has been overseen by myself (Emma Pewsey, Associate Features Editor) and Peter Rodgers (Features Editor). The following individual involved in review of your submission has agreed to reveal his identity: Björn Brembs (Reviewer #1).

The reviewers have discussed the reviews with one another and I have drafted this decision to help you prepare a revised submission.

By ascertaining how often review, promotion and tenure committees refer to the journal impact factor (JIF), this paper provides solid data about an issue that is often discussed using only subjective experience, hearsay and estimates. The paper is well written and easy to read, and is an important contribution to the discourse on organizing the way we go about scholarship.

Essential revisions:

1) In the Materials and methods section, please provide the percentage of respondents, and provide the sample size as a percentage of number of US/Canada institutions total and sampled.

2) An explanation should be added to the Materials and methods section to explain why, with more than 5,000 universities and colleges in the US alone, the sample of 129 universities counts as a "representative" sample, and explain how the numbers were achieved.

3) Please also discuss the size of the sample in the Discussion section.

4) The importance of this study would be significantly greater if JIF usage could be associated with RTP promotion outcome. Without that information the importance of this study is limited to documenting with data what most people already think that they know.

Essential revisions: 1) In the Materials and methods section, please provide the percentage of respondents, and provide the sample size as a percentage of number of US/Canada institutions total and sampled.

To address this concern, we reproduced a simplified version of Table 1 from our previous paper, which gives an overview of the population of universities from the United States and Canada by type, the number and percent randomly chosen for our stratified sample, and the number of institutions for which we obtained at least one relevant document. We added a reference to this table at the end of the first paragraph of the subsection “Document collection”, along with a sentence describing sample considerations (see point 2 below).

As per the table added, we also included a sentence that describes why we consider this sample to be representative. The sentence, added in the first paragraph of the “Document Collection” subsection, reads as follows: “The stratified sample was designed to collect documents from enough institutions in each of the R-, M-, and B-type categories to have a statistical power of. 8 (assuming a small effect size of. 25 of a standard deviation) when making comparisons between disciplines.”

We have added a few sentences about sample size considerations to the Discussion section on the prevalence of the use of the JIF in evaluations.

We appreciate the reviewers’ concerns here. Unfortunately, we have no way of obtaining information on the outcomes of RTP evaluations, especially because this information is often privileged and not publicly documented by departmental committees. However, we believe that our study is still important, even if some of the results do document what people already think they know about RPT evaluations. The information previously available was largely based on small surveys or largely anecdotal. Our study is the first to provide concrete data on how often and in what ways the JIF is used in formal documents governing RPT processes. This information could be valuable to researchers, administrators, and others seeking to understand current issues with RPT processes and how to improve them.

Author details

Erin C McKiernan is in the Departamento de Física, Universidad Nacional Autónoma de México, Mexico City, Mexico

Contribution

Contributed equally with, for correspondence, competing interests.

Lesley A Schimanski is in the Scholarly Communications Lab, Simon Fraser University, Vancouver, Canada

Carol Muñoz Nieves is in the Scholarly Communications Lab, Simon Fraser University, Vancouver, Canada

Lisa Matthias is at the John F Kennedy Institute, Freie Universität Berlin, Berlin, Germany

Meredith T Niles is in the Department of Nutrition and Food Sciences, University of Vermont, Burlington, United States

Juan P Alperin is in the School of Publishing and the Scholarly Communications Lab, Simon Fraser University, Vancouver, Canada

Open Society Foundations (OR2016-29841)

• Juan Pablo Alperin

The funders had no role in study design, data collection and interpretation, or the decision to submit the work for publication.

Acknowledgements

We are grateful to SPARC, the OpenCon community, the DORA Steering Committee (especially Catriona MacCallum and Anna Hatch), Chealsye Bowley, and Abigail Goben for discussions that shaped and improved this work. We also thank Elizabeth Gadd and Erika Mias, who suggested library guides and projects on responsible metrics to highlight in our Discussion.

Publication history

• Received: April 2, 2019
• Accepted: July 10, 2019
• Version of Record published : July 31, 2019

© 2019, McKiernan et al.

This article is distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use and redistribution provided that the original author and source are credited.

Views, downloads and citations are aggregated across all versions of this paper published by eLife.

Download links

Downloads (link to download the article as pdf).

• Article PDF

Open citations (links to open the citations from this article in various online reference manager services)

Cite this article (links to download the citations from this article in formats compatible with various reference manager tools), categories and tags.

• scholarly communications
• academic careers
• impact factor
• higher education
• institutional policy
• Part of Collection

Meta-Research: A Collection of Articles

Further reading.

The study of science itself is a growing field of research.

• Physics of Living Systems

Combining mutation and recombination statistics to infer clonal families in antibody repertoires

B-cell repertoires are characterized by a diverse set of receptors of distinct specificities generated through two processes of somatic diversification: V(D)J recombination and somatic hypermutations. B cell clonal families stem from the same V(D)J recombination event, but differ in their hypermutations. Clonal families identification is key to understanding B-cell repertoire function, evolution and dynamics. We present HILARy (High-precision Inference of Lineages in Antibody Repertoires), an efficient, fast and precise method to identify clonal families from single- or paired-chain repertoire sequencing datasets. HILARy combines probabilistic models that capture the receptor generation and selection statistics with adapted clustering methods to achieve consistently high inference accuracy. It automatically leverages the phylogenetic signal of shared mutations in difficult repertoire subsets. Exploiting the high sensitivity of the method, we find the statistics of evolutionary properties such as the site frequency spectrum and 𝑑𝑁∕𝑑𝑆 ratio do not depend on the junction length. We also identify a broad range of selection pressures spanning two orders of magnitude.

• Cell Biology

circHIPK3 nucleates IGF2BP2 and functions as a competing endogenous RNA

Circular RNAs represent a class of endogenous RNAs that regulate gene expression and influence cell biological decisions with implications for the pathogenesis of several diseases. Here, we disclose a novel gene-regulatory role of circHIPK3 by combining analyses of large genomics datasets and mechanistic cell biological follow-up experiments. Using time-course depletion of circHIPK3 and specific candidate RNA-binding proteins, we identify several perturbed genes by RNA sequencing analyses. Expression-coupled motif analyses identify an 11-mer motif within circHIPK3, which also becomes enriched in genes that are downregulated upon circHIPK3 depletion. By mining eCLIP datasets and combined with RNA immunoprecipitation assays, we demonstrate that the 11-mer motif constitutes a strong binding site for IGF2BP2 in bladder cancer cell lines. Our results suggest that circHIPK3 can sequester IGF2BP2 as a competing endogenous RNA (ceRNA), leading to target mRNA stabilization. As an example of a circHIPK3-regulated gene, we focus on the STAT3 mRNA as a specific substrate of IGF2BP2 and validate that manipulation of circHIPK3 regulates IGF2BP2- STAT3 mRNA binding and, thereby, STAT3 mRNA levels. Surprisingly, absolute copy number quantifications demonstrate that IGF2BP2 outnumbers circHIPK3 by orders of magnitude, which is inconsistent with a simple 1:1 ceRNA hypothesis. Instead, we show that circHIPK3 can nucleate multiple copies of IGF2BP2, potentially via phase separation, to produce IGF2BP2 condensates. Our results support a model where a few cellular circHIPK3 molecules can induce IGF2BP2 condensation, thereby regulating key factors for cell proliferation.

Be the first to read new articles from eLife

Follow us on :

Research and Review Insights

An open access, peer reviewed journal, volume 05, issue 02, item analysis of multiple choice questions (mcqs) from a formative assessment of first year microbiology major students.

Sylvester Chibueze Izah Tamaraukepreye Catherine Odubo Victor Emeka Ajumobi Kelly Ebelakpo Torru

Research Article-Research and Review Insights (RRI)

July 09, 2021

A review on the mechanism of action of favipiravir and hydroxychloroquine in COVID-19

Velichka Pavlova Silvia Hristova Katya Uzunova Toni Vekov

Review Article-Research and Review Insights (RRI)

August 11, 2021

Identifying the impact of COVID-19 isolation practices on persons experiencing homelessness with concurrent disorders

Michael Young Rana Van Tuyl

November 29, 2021

About Journal

Research and Review Insights (RRI) is an open access, peer-reviewed journal which aims to publish research and review articles in Medicine, Medical Sciences, , Epidemiology, Veterinary Sciences, Parasitology, Helminthology, Herpetology, Parasitology, Nematology, Neuroethology, Ornithology, Fine Art, Industrial Relations, Paleozoology, Planktology, Arthropodology, Cetology, Conchology, English, Apiology, Library Science, International Business, Law/Criminology, Life Sciences, Botany, Business Research, Chemistry, Molecular Endocrinology, Immunobiology, Geography, Advertising, Accounting & Finance, Psychology, Health Care, Veterinary Sciences, Apiology, Arachnology, Social Science, Education, Mathematics, Engineering & Technology, Ichthyology, Management, etc.

Basically, it’s a very broad topic where Research and Reviews from different field can be fitted together and most importantly providing a common platform for Researchers from all over the diverse field.

Research and Review Insights (RRI) aims to publish a complete and reliable source of information on discoveries and current developments as original research and review articles in all areas of the field and making them available online without any subscriptions to the researchers worldwide.

The articles can be accessed online immediately on their publication. We are committed to providing authors with a rapid, fair, and constructive external peer-review process, with the goal of providing the majority of first decisions within 4 weeks.

Research and Review Insights welcomes direct submissions from authors: Attach your word file with e-mail and send it to: [email protected]  alternatively to:  [email protected]

Copyright is retained by the authors and articles can be freely used and distributed by others. Articles are distributed under the terms of the Creative Commons Attribution License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published by RRI, is properly cited.

Prof. Yoko Shiba

Department of Chemistry and Biological Sciences Faculty of Science and Engineering Iwate University Japan

Editor-in-Chief

Iwate University, Japan

Regional Editor in Chief

Executive editors, vincenzo naddeo.

Associate Professor in Environmental Engineering Sanitary Environmental Engineering Division (SEED) Department of Civil Engineering University of Salerno Italy

Division of Cardiology Department of Geriatrics The First Affiliated Hospital of Chongqing Medical University The First Youyi Road, Yuzhong District Chongqing China

Editorial Board

Salem bouomrani.

Department of Internal Medicine Military Hospital of Gabes. Beb Bhar Tunisia

Surgeon Jilin University China

Sana Sharifian

Faculty of Marine Science and Technology Hormozgan University Iran

Norbert Nwankwo

Department of Clinical Pharmacy University of Port Harcourt Nigeria

Taha Roodbar Shojaei

Department of Nano-materials and Nanotechnology Institute of Advanced Technology (ITMA) University Putra Malaysia Malaysia

Alexander G. Tonevitsky

Head of Department for translational oncology P.A. Hertsen Moscow Research Oncology Institute Russia

Oriana Trubiani

Department of Medical oral and Biotechnological Sciences "G. D'Annunzio" University of Chieti-Pescara Italy

Marietha Nel

Senior Research Scientist Department of Surgery Faculty of Health Sciences University of the Witwatersrand South Africa

Paul Edward Kaloostian

Department of Neurosurgery University of California at Riverside USA

Laboratory of Protozoology Institute of Evolution & Marine Biodiversity Ocean University of China China

Pratibha Mishra nee Vyas

Microbiology Domain School of Biotechnology and Biosciences Lovely Professional University India

Michael Trimmel

Professor Institute of Environmental Health Medical University of Vienna Austria

Rina Chakrabarti

Aqua Research Lab Department of Zoology University of Delhi India

Ali F. Alghamdi

Chemistry department Faculty of science Taibah University Saudi Arabia

Maria Pina Serra

Department of Biomedical Sciences University of Cagliari Italy

Australian Institute for Bioengineering and Nanotechnology Th e University of Queensland Australia

Mahmoud G Soliman

Faculty of Physics Philipps-Universität Marburg Deutschland

Lucia Musumeci

Department of Cardiovascular Sciences University of Liège Belgium

Laura Mihai

National Institute for Laser Plasma and Radiation Physics (INFLPR) Romania

Mohammad Alghoul

Senior Research Scientist Renewable Energy Program Energy and Building Research Center Kuwait Institute for Scientific Research Kuwait

Mohammed Alasel

Philipps University Institute of Pharmaceutical chemistry Marburg, Germany

Stefania Cantore

School of Medicine University of Bari Aldo Moro Italy

Qingtao Zhou

Department of Geography and Environmental Sustainability University of Oklahoma USA

Amrita Shenoy

Science and Technology Department University of Minnesota Crookston USA

Manuel G. Chávez Angeles

Profesor-investigador Universidad de la Sierra Sur (UNSIS-SUNEO) Guillermo Rojas Mijangos S/N, esquina Av. Universidad Ciudad Universitaria, Miahuatlán de Porfirio Díaz Oaxaca México

Junzhe Xu, M.D.

Associate Professor Department of Psychiatry State University of New York at Buffalo School of Medicine USA

Olga V Akopova

Circulation Department Bogomoletz Institute of Physiology Ukraine

Dr. Amgad Rabie

Head of Clinical Research Department at Dikernis General Hospital Egypt

Volume 3, Issue 1

Neurophysiological manifestation of memory based on wavelet transform of electroencephalogram.

E.A. Yumatov N.A. Karatygin E.N. Dudnik A.I. Filipchenko L.T. Sushkova R.V. Isakov V.A. Al-Haidri O.S Glazachev

Current Issue

Volume 6, issue 1, the dispossessed of egypt part v: the scheme of genetic trumpization for the supremacy of the elite versus the world’s majority plagued with poverty and diseases.

J. H. Wassili Cyril Baradaeus

December 23, 2021

Enhancing effect of streptozotocin-induced insulin deficit on antitumor innate immune defense in rats

Hajto Tibor Péczely László Kuzma Monika Hormay Edina Ollmann Tamás Jaksó Pál Baranyai Lilla Karádi Zoltán

Case Reports-Research and Review Insights (RRI)

May 23, 2022

• June 2022 Issue 1
• December 2021 Issue 2
• August 2021 Issue 1

Inactivated vaccines: a promising old tool against Covid-19

Enrique Iglesias

Commentary-Research and Review Insights (RRI)

December 21, 2020

Can standardized and evidence based plant immunomodulator (rice bran arabinoxylan concentrate) increase the effect of Gemcitabine in a synergistic manner? Case report of a patient with ductal pancreas carcinoma

Tibor Hajto

January 04, 2021

Lupus can be caused by asymptomatic Salmonellosis in Bladder

José Artur Medina

Letter to the Editor-Research and Review Insights (RRI)

February 12, 2021

Microbial density and associated physicochemical properties of palm wine in Wilberforce Island, Nigeria

Glory Richard Felix Okponanabofa Youkparigha Ayobami Omozemoje Aigberua Sylvester Chibueze Izah Edah Oghenetega Morayo

Antibacterial Properties of Tetrapleura tetraptera pod against some pathogens

Esther Benafegha Enaregha Sylvester Chibueze Izah Queen Okiriya

• June 2020 Issue 1

Narrative review of published evidence

Gitana Diana Stukaite

April 06, 2020

Oral health and covid19

Camila Lindoni Azevedo Edgard Michel Crosato Isabella da Cunha Henriques Paulo Sérgio Gomes Henriques

Editorial-Research and Review Insights (RRI)

April 17, 2020

The health effects of electronic cigarette use: An overview

Evangelia Gkoulitou Chrysoula Vakaki Eudoxie Pepelassi

April 20, 2020

Production of micronutrient enriched algae, microorganisms and insects for food and feed: Perspectives and updates

Beatriz Felices Rando Eldon R. Rene

May 29, 2020

• June 2019 Issue 1

Nutritional deficiency cardiomyopathy: A review and pooled analysis of pathophysiology, diagnosis and clinical management

Aref Albakri

Review-Research and Review Insights (RRI)

February 15, 2019

Toxin-induced cardiomyopathy: A review and pooled analysis of pathophysiology, diagnosis and clinical management

February 20, 2019

Fungal cardiomyopathy: A review and pooled analysis of pathophysiology, diagnosis and clinical management

February 27, 2019

Parasitic (Protozoan) cardiomyopathy: A review and pooled analysis of pathophysiology, diagnosis and clinical management

April 30, 2019

Endocrine cardiomyopathy: A review and pooled analysis of pathophysiology, diagnosis and clinical management

May 25, 2019

Neuroinflammation as part of a possible neuroregeneration on a patient with ASIA-A scale using combinatorial biologics: Bioquantine® and allogeneic mesenchymal stem cells

Joel Isaias Osorio Sergei Paylian Ale Ismael Gonzalez Cazares Ira Pastor

Mini-Review-Research and Review Insights (RRI)

June 28, 2019

• December 2018 Issue 4
• September 2018 Issue 3
• July 2018 Issue 2
• March 2018 Issue 1

Evaluating the Effectiveness of Virtual Fracture Clinics for Ankle Injuries

Andrew Robinson Anand Pillai Mohamad Alqubaisi

October 16, 2018

How are fifth metatarsal fractures managed by the virtual fracture clinic

Zeenat Mirza Anand Pillai Mohamad Alqubaisi

The geography of open fractures of the lower limb: Access to trauma care

Georgia Hawthorne Anand Pillai Mohamad Alqubaisi

October 17, 2018

Virtual fracture clinic in the management of foot and ankle fractures

Alqubaisi M Hawarden D Boyle M Robinson A Pillai A

The exploration of enhanced photocurrent with Zn-perylene metal organic frameworks thin film and bodipy via triplet triplet annihilation upconversion

Shargeel Ahmad

November 20, 2018

Analysis of postmenopausal osteoporosis in primary care in the canary islands Spain (ESTOPMAPCA project)

Olmo Quintana V Ramírez T García Santana MI Martin Torres M González Núñez J Orero González A Santos Sancho J Martínez Hernández D Alonso Bilbao JL Sosa Henriquez M

December 28, 2018

p53-dependent cell cycle checkpoint after DNA damage and its relevance to PARP1

Tadashige Nozaki Mitsuko Masutani

July 11, 2018

Comparison of antioxidant properties of Cyathula prostrata Linn and Achyranthes aspera Linn grown in Sri Lanka

D.P.A. Dissanayake S. Sivaganesh M.H.A. Tissera S.M. Handunnetti L.D.A.M. Arawwawala

July 23, 2018

Some determinant factors of antimicrobial susceptibility pattern of plant extracts

Sylvester Chibueze Izah

August 05, 2018

Physical properties of hybrid virus nonorganic complexes TMV-Au

Kasiyanenko VKH Burdeynyy VM

September 27, 2018

The chaotic status of so-called “Colloidal Silver”, that is not a colloid, as well the many ‘Myths’ surrounding Silver’s aqueous vehicle ‘water’ and how best to test it for purity, other hidden properties and related anomalies of this science

September 28, 2018

The dispossessed of Egypt Part II: Beyond the point of fatigue; the export of HCV-4b, mutated diseases and psychological aberrations to the new and old worlds

JH Wassili Cyril Baradaeus

Endocrine involvement in granulomatosis with polyangiitis

Salem Bouomrani Mayada Ben Hamadl Nesrine Regaieg1 Nesrine Belgacem1 Safa Trabelsil

Mini Review-Research and Review Insights (RRI)

April 14, 2018

Basic accountability to stop ill-treatment (BASI): study protocol for a cluster randomized controlled trial in rural tanzania

Redempta Mbatia Jessica Cohen Martin Zuakulu Appolinary Bukuku Shikha Chandarana Eliudi Eliakimu Sisty Moshi Elysia Larson

April 21, 2018

Bioinformatics analysis of lncRNAs differently expressed in patients with schizophrenia

Li-yi Zhang Hui-min Fan Wei Niu Kun-hong Jiang Ming-jun He Ling-ming Kong Sheng-dong Chen

April 24, 2018

Global and National trends of HIV/AIDS

Hikmat Bahadur Raya

May 29, 2018

Objective assessment in continual medical education (CME): Medical objective assessment in system control

Dmytro D Ivanov Yuriy V Voronenko Ozar P Mintser Larysa Yu Babintseva

June 28, 2018

Zoonotic viral Infections in south africa: An overview

Howard Newmana Shareef Abrahamsb

July 15, 2018

The silent biological Hiroshima: GMOs-Induced Darwinian devolution of human race

February 17, 2018

Inflammation and breast cancer

February 17, 2017

Internal Breast Reduction Surgery

Aristides Arellano-Huacuja Anja Arellano-Montalvo Dafne Arellano-Montalvo

March 24, 2017

Serious games to assess mild cognitive impairment: ‘The game is the assessment’

Kyle Leduc-McNiven Benjamin White Huizhe Zheng Robert D McLeod Marcia R Friesen

March 23, 2018

Voice and hyperthyroidism: Subjective voice complaints and alterations of the acoustic parameters of the voice

Gordana Kovacic

March 24, 2018

Brine shrimp cytotoxicity, phytochemical screening and larvicidal activities of plectranthus barbatus extracts

Lawi Y Saria J Kidukuli AW

March 29, 2018

• December 2017 Issue 4
• October 2017 Issue 3
• August 2017 Issue 2
• June 2017 Issue 1

Rescuing Fe0 remediation research from its systemic flaws

Chicgoua Noubactep Suzanne Makota Amitava Bandyopadhyay

November 25, 2017

Why it is impossible to recover periodontal disease areas to the pre-disease stage

José Ricardo Kina

December 14, 2017

Comparison of representative skin pretreatment methods for body surface biopotential measurement

Si Junfeng Yang Jie Mao Shanqing Wang Xinyi Zeng Yan Liu Hongxing

December 16, 2017

Orthognathic surgery for patients with idiopathic condylar resorption: Clinical research

Jeong-Kui Ku Nam-Ki Lee Young-Kyun Kim

December 19, 2017

Effect of pulsed He-Ne laser irradiation on bee honey physicochemical properties

Al humira Elseir Gorashe Ahmmed Ali AS Marouf

December 22, 2017

ArfGAPs; Depletion or Overexpression?

December 26, 2017

Eyelid reconstruction

Vishal Mago

September 16, 2017

“I have collected qualitative data; now what do I do?” approaches to analysing qualitative data

Dr. William Evans Dr. Tom o Connor

September 25, 2017

Can a standardized plant immunomodulator (rice bran arabinoxylan concentrate/MGN-3) increase the effects of MEK and BRAF inhibitors with clinical benefit? Case report of a patient with carcinoma in biliary duct

September 27, 2017

The dispossessed of Egypt, Part I: Anatomy of hepatic-APR to environmental filth and VMAT2’s variant-induced psychosis in aseptic rat models

October 24, 2017

Transformation of hashimoto thyroiditis to Graves’ disease

Stella Pak Damian Valencia Adam Fershko

October 26, 2017

Artificial biofilm thickness and salivary flow effects on fluoride efficacy – A model development study

Frank Lippert Anderson T Hara David Churchley Richard JM Lynch

Three-dimensional gait analysis of cerebral paralysis patients after operation and effect of orthoses

Masaaki Ito

July 06, 2017

The human battle with glioblastoma multiforme: A true challenge

Christ Ordookhanian Samer S. Ghostine Paul E. Kaloostian

Editorial Article-Research and Review Insights (RRI)

August 10, 2017

Identification of carcinoma embryonic antigen release mechanism Carcinoembryonic Antigen (CEA) from the surface of colorectal cancer cells

Shahin Asadi Mahsa Jamali Hamideh Mohammadzadeh

August 17, 2017

Bioinformatics study on translational initiation factor eIF-5a in Panax ginseng

Li Gun Guo Rui Du Ning Xu Fei

Research Article - Research and Review Insights (RRI)

August 23, 2017

Heart rate variability in chronic obstructive pulmonary disease

Golidakis Diamantis Evaggelou Aggelos Peschos Demetrios George Anogeianakis

August 26, 2017

Light deprivation: An efficient way in inducing depression-like behavior animal models

Chanyi Lu Hua-Zhen Lin Yao-Yao Li Yun-Feng Zhang

Mini-Review Article - Research and Review Insights (RRI)

August 30, 2017

Conventional bioprocesses for the removal of gas-phase contaminants

Carol Wambugu Adjei Danso Marvins Jewel Das Eldon R. Rene

June 10, 2017

Improved detection of metastases by color Doppler ultrasonography

Petar Avramovski Maja Avramovska Emilija Sikole

June 12, 2017

Preoperative evaluation, surgical strategy and long-term outcome of anorectal malformations in newborn period: Single center experience

Murat Sanal Oliver Renz Paul Hechenleitner Beatrice Häussler

June 22, 2017

Identifying optimal treatment of common and complex sciatica pain

Christ Ordookhanian Samer S. Ghostine Paul Kaloostian

June 23, 2017

Lymphoscintigraphy and clinic

Lucio Mango Guido Ventroni Sandro Michelini

June 26, 2017

Homo informaticus: Thinking and moral values of humans are shaped by human-computer-interaction

June 29, 2017

Submit Manuscript

Publication charges.

Research and Review Insights is an Open Access journal and we do not charge the end user when accessing a manuscript or any article. This allows the scientific community to view, download, distribution of an article in any medium, provided that the original work is properly cited, under the term of "Creative Commons Attribution License". In line with other open access journals we provide a flat fee submission structure on the acceptance of a peer-reviewed article which covers in part the entirety of the publication pathway (the article processing charge). The process includes our maintenance, submission and peer review systems and international editing, publication and submission to global indexing and tracking organisations and archiving to allow instant access to the whole article and associated supplementary documents. We also have to ensure enough investment to secure a sustainable model which ethically, legally and financially stable.

The publication charges for Research and Review Insights are USD 850.

Frequently Asked Questions (FAQs)

Why are your charges set at these levels?

All articles published in OA Text are open access. Open Access publishing implies that all readers, anywhere in the world, are allowed unrestricted to full text of articles, immediately on publication in OA Text Journals. The Article Publication Charges pay for the editorial and production costs of the journal, for hosting the website, publishing articles online, preparing HTML , PDF and XML versions of the articles and submitting the articles in electronic citation database like CrossRef.

Our financial goals are to:

• Recover capitalization costs;
• Produce sufficient revenue to allow for a sustainable and scalable publishing program, under continuous development;
• Bend the publication-charge cost downward over time.

Who will pay the APC?

Corresponding author or Co-authors has to make the payment on acceptance of the article.

When should I pay?

Corresponding author or the paying institutions should arrange for the payment once they are notified regarding acceptance of the article. APC is exempted for cases in which a wavier agreement has been made in-prior to submission.

* We request an immediate attention towards the payment as the articles will not be published unless the charges have been paid.

How do I pay?

Authors or institutions can make payments by two modes as per their convenience.

• Wire/Bank transfer
• Card payment

Note: No taxes are included in this charge, taxes will be applicable as per the policies of the country of the payee. Additional transaction charges may be levied on the author.

Can I be eligible for wavier on APC?

The waiver request will be considered on a case-by-case basis, and will be provided accordingly.

*The Waiver requests must be made during the submission process and will not be accepted after  processing of the manuscript.

Do I have to pay if my manuscript is rejected?

No, Article processing charges will not be applicable for articles rejected by the Editorial office.

Withdrawal Charges

Are reprints of my article included in the article processing charges (APCs)?

No, Article processing charges (APCs) do not include the charges for the reprints. Reprints facility is optional and should be order separately.

External Databases & Indexes

• Google scholar
• Electronic Journals Library (EZB)
• Scientific Indexing Services (SIS)
• Directory of Research Journals
• SciCurve Open

*Only NIH article.

Pubmed Indexed Articles

Characteristics of online treatment seekers interested in a text messaging intervention for problem drinking: adults 51 and older versus middle-aged and younger adults., liver targeted therapies for hepatocellular carcinoma prior to transplant: contemporary management strategies., a review of the impact of obesity on common gastrointestinal malignancies, brca1 and brca2 mutations and treatment strategies for breast cancer, most viewed, submit your next article.

• Online submission
• Highly indexed and abstracted
• 7 years of successful publishing
• Wider visibility though open access
• Higher impact with wider visibility
• Prompt review
• Privacy Policy
• Terms & conditions

© 2022 Copyright OAT. All rights reserved.

Maintenance work is planned from 21:00 BST on Sunday 18th August 2024 to 21:00 BST on Monday 19th August 2024, and on Thursday 29th August 2024 from 11:00 to 12:00 BST.

During this time the performance of our website may be affected - searches may run slowly, some pages may be temporarily unavailable, and you may be unable to log in or to access content. If this happens, please try refreshing your web browser or try waiting two to three minutes before trying again.

We apologise for any inconvenience this might cause and thank you for your patience.

Chemical Society Reviews  

About Chemical Society Reviews

The home of high impact reviews from across the chemical sciences. Editorial Board Chair: Jennifer Love Impact factor: 40.4 Time to first decision (peer reviewed only): 44.7 days Indexed in MEDLINE

Find an article

Find issues by year (1972 - present), related journals.

• Royal Institute of Chemistry, Reviews (1968-1971)
• Quarterly Reviews, Chemical Society (1947-1971)

Journal information

• About this Journal
• People and Contacts
• Editorial Board
• Subscription Information
• Follow   |    |  

Advertisements

Physical Review D

Covering particles, fields, gravitation, and cosmology.

• Collections
• Editorial Team

2023 Journal Impact Factors

Clarivate Analytics has released the 2023 Journal Citation Reports, which provides journal impact factors and rankings for over 11,000 scholarly journals.

The Physical Review journals continue to hold its world-leading positions among titles publishing high quality, peer-reviewed research in physics and related areas of research. Among the notable highlights are the continued excellence of three premier titles Physical Review Letters , Physical Review X , and Reviews of Modern Physics , as well as another strong score and a new number 1 ranking in its category for PRX Quantum .

APS and the Physical Review journals are extremely grateful to the many members of the scientific community – and especially the journals’ loyal authors, readers, referees, editors, and Editorial Board members – who have supported the Physical Review in maintaining the high quality standards and peer review excellence which are reflected by the above, and other, metrics. Thank you!

All of the updated Impact Factors can be viewed on the metrics page .

Sign up to receive regular email alerts from Physical Review D

• Forgot your username/password?
• Create an account

Article Lookup

Paste a citation or doi, enter a citation.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Review Article
• Published: 09 August 2024

Long COVID science, research and policy

• Ziyad Al-Aly   ORCID: orcid.org/0000-0002-2600-0434 1 , 2 ,
• Hannah Davis   ORCID: orcid.org/0000-0002-1245-2034 3 ,
• Lisa McCorkell   ORCID: orcid.org/0000-0002-3261-6737 3 ,
• Letícia Soares 3 ,
• Sarah Wulf-Hanson 4 ,
• Akiko Iwasaki   ORCID: orcid.org/0000-0002-7824-9856 5 , 6 &
• Eric J. Topol   ORCID: orcid.org/0000-0002-1478-4729 7  

Nature Medicine ( 2024 ) Cite this article

25k Accesses

1920 Altmetric

Metrics details

• Viral infection

Long COVID represents the constellation of post-acute and long-term health effects caused by SARS-CoV-2 infection; it is a complex, multisystem disorder that can affect nearly every organ system and can be severely disabling. The cumulative global incidence of long COVID is around 400 million individuals, which is estimated to have an annual economic impact of approximately $1 trillion—equivalent to about 1% of the global economy. Several mechanistic pathways are implicated in long COVID, including viral persistence, immune dysregulation, mitochondrial dysfunction, complement dysregulation, endothelial inflammation and microbiome dysbiosis. Long COVID can have devastating impacts on individual lives and, due to its complexity and prevalence, it also has major ramifications for health systems and economies, even threatening progress toward achieving the Sustainable Development Goals. Addressing the challenge of long COVID requires an ambitious and coordinated—but so far absent—global research and policy response strategy. In this interdisciplinary review, we provide a synthesis of the state of scientific evidence on long COVID, assess the impacts of long COVID on human health, health systems, the economy and global health metrics, and provide a forward-looking research and policy roadmap.

You have full access to this article via your institution.

Similar content being viewed by others

Long COVID: major findings, mechanisms and recommendations

Insights from an N3C RECOVER EHR-based cohort study characterizing SARS-CoV-2 reinfections and Long COVID

Insights into early recovery from Long COVID—results from the German DigiHero Cohort

Long COVID is best defined as the constellation of post-acute and long-term health effects caused by SARS-CoV-2 infection 1 , 2 , 3 . Long COVID was initially reported by patients who coined the term and, through research and advocacy, drove much of the progress in understanding this condition over the past several years (Fig. 1 ).

The history of long COVID has been defined largely by the patients themselves. In March 2020, as the COVID-19 pandemic began to unfold across the globe, patients with infection-associated chronic conditions presciently warned of the potential emergence of long-term illness after SARS-CoV-2 infection 293 .The first mainstream written personal account of non-recovery from acute COVID-19 was an op-ed by the American journalist Fiona Lowenstein in the New York Times in April 2020 (ref. 294 ). Around the same time, patients began self-organizing, coined the term long COVID 295 and conducted the first known survey—which was subsequently formally published—documenting the breadth of symptomatology experienced by people with long COVID 42 . Considerable activity then ensued, including mainstream media coverage (first by Ed Yong in The Atlantic ) 296 , recognition by national governments (of the United States 281 , Canada 297 , United Kingdom 298 , European Union 299 , Australia 300 and others) and the WHO. Patients continue to lead the way in advocacy and research, which led the US Senate to hold its first-ever hearing on long COVID 275 , 301 . This timeline was curated to provide a brief overview of the history of long COVID, with a focus on the role played by patients and advocates, and does not comprehensively include all events and milestones. ICD, International Classification of Diseases.

Long COVID is a complex, multisystem disorder that affects nearly every organ system, including the cardiovascular system 4 , the nervous system 5 , 6 , 7 , 8 , the endocrine system 9 , 10 , 11 , the immune system 12 , 13 , the reproductive system 14 and the gastrointestinal system 15 . It affects people across the age spectrum (from children 16 , 17 , 18 to older adults 19 , 20 ), people of different race and ethnicities, sex and gender, and baseline health status 21 . Cardinal manifestations include brain fog (or cognitive dysfunction) 7 , fatigue, dysautonomia (which commonly manifests as postural orthostatic tachycardia syndrome (POTS)) 22 and post-exertional malaise 23 . Many of the health effects seen in long COVID are shared across several infection-associated chronic conditions, also called post-acute infection syndromes 23 , 24 , 25 , 26 .

The epidemiology of long COVID is influenced by various factors. The Omicron variant of SARS-CoV-2 is associated with less risk of long COVID than the Delta and pre-Delta variants 27 . Vaccines (before infection) and antivirals (during the acute phase of infection) may reduce the risk of long COVID. Reinfection, on the other hand, is a risk factor for long COVID 28 , 29 ; even if individuals did not experience long COVID after a first SARS-CoV-2 infection, they remain at risk of developing it with subsequent infections 28 , 29 , 30 . Reinfection can trigger de novo long COVID or exacerbate the severity of existing long COVID 28 , 29 . Cumulatively, two infections yield a higher risk of long COVID than one infection and three infections yield a higher risk than two infections 28 , 29 .

A unifying thread of evidence across most studies evaluating the risk of long COVID is the finding that the risk increases as the severity of acute infection increases 3 . People who had severe COVID-19 that necessitated hospitalization exhibit a higher risk of long COVID than those with mild COVID-19. However, because most people around the globe had mild COVID-19, they constitute more than 90% of people with long COVID, despite their lower relative risk compared with that of people with severe COVID-19 (ref. 31 ).

Studies evaluating recovery from long COVID are sparse and inconsistent 32 ; this is largely due to use of various definitions, incomplete accounting for all the manifestations of long COVID and misclassification of remission as ‘recovery’ 33 . However, studies carefully evaluating individual manifestations show that recovery rates are generally low at 1 year 34 , and several studies show only 7–10% fully recovered at 2 years 30 , 33 , 35 , 36 . Furthermore, some manifestations of long COVID, including heart disease, diabetes, myalgic encephalomyelitis and dysautonomia are chronic conditions that last a lifetime 31 , 37 , 38 , 39 . Adding to this are the concerns about the possible emergence of new latent sequelae—that have not yet been characterized—years after the acute infection 37 , 40 , 41 .

The impact of long COVID is not limited to the health and well-being of individual patients and their communities. Owing to its prevalence and the breadth of its clinical manifestations 42 , 43 , 44 , 45 , 46 , it represents a major public health crisis 47 ; it strains health systems and national economies, and threatens progress on global health, including the Sustainable Development Goals (SDGs).

In this interdisciplinary review, we provide a brief synthesis of the current state of scientific evidence on long COVID, including knowns, unknowns and the key controversies. We provide an assessment of the impacts of long COVID on human health, health systems, the economy and global health metrics and, finally, we provide a forward-looking research and policy roadmap that we hope will stimulate global discussion on how to address the challenge of long COVID.

State of the science on long COVID

The global burden.

Estimating the global burden of long COVID presents substantial challenges due to the variability in study designs and populations, follow-up times, choice of control groups (for example, whether studies evaluated people with negative SARS-CoV-2 tests or no known SARS-CoV-2 infection as controls), assessment of baseline health before the infection (to ascertain emergence of a true new health condition) and definitions of what constitutes ‘long COVID' 48 , 49 . Variation in risk estimates also reflects the dynamic nature of the pandemic itself, which gave rise to many variants and subvariants, each yielding potentially different rates of long COVID; the effect of COVID-19 vaccines and use of antivirals in the acute phase, which may reduce the risk of long COVID; and the effect of SARS-CoV-2 reinfections, which contribute additional risk 28 , 29 .

Few countries established surveillance systems to estimate the burden of long COVID at the population level. Data from the US Centers for Disease Control and Prevention (CDC)’s National Health Interview Survey show that in 2022, 6.9% of US adults 50 and 1.3% of children 51 ever had long COVID. Data from the Medical Expenditure Panel Survey—a nationally representative survey of US adults—found that 6.9% of adults had ever had long COVID as of early 2023 (ref. 52 ). Estimates from the CDC’s Household Pulse Survey show that prevalence of current long COVID in US adults was around 6.7% in March 2024 (ref. 53 ). In the United Kingdom, point prevalence estimates from the Office of National Statistics show that 2.9% of the UK population (including children) were experiencing self-reported long COVID in March 2023 (ref. 54 ). Overall, estimates of the burden of long COVID in the general population converge around a point prevalence of 6% to 7% in adults and ~1% in children 50 , 51 , 52 , 53 , 54 .

Also important are estimates of the incidence of long COVID, which can be informed by high-quality meta-analyses of large-scale cohort studies among people infected with SARS-CoV-2. For instance, one analysis pooled results from 54 studies in 22 countries and estimated that approximately 6.2% of symptomatic COVID-19 survivors experience at least one of three common symptom clusters at 3 months after acute infection, across all ages and accounting for different severity levels of the initial infection and pre-COVID health status 31 . This analysis only considers three major symptom clusters in long COVID (fatigue with bodily pain/mood swings, and cognitive and respiratory symptom clusters); however, it sets a conservative benchmark to estimate the global risk of long COVID 31 .

We estimated the global incidence of long COVID on the basis of meta-regression studies that pool together all the available evidence 31 (Fig. 2 ). Incorporating a number of assumptions, including the Institute for Health Metrics and Evaluation’s annual estimates of SARS-CoV-2 infections 31 , 55 , 56 , 57 , 58 , 59 , a proportion symptomatic cases among infections of 65% (ref. 31 ), and a reduction in the risk of long COVID for 2022 and 2023 to account for the putative lower severity of the Omicron variant and the effect of vaccination 60 , we estimated a cumulative global incidence of long COVID by the end of 2023 of approximately 400 million. It is crucial to emphasize that these estimates only represent cases arising from symptomatic infections and are likely to be conservative. The actual incidence of long COVID, including cases from asymptomatic infections 61 or those with a broader range of symptoms, is expected to be higher. Furthermore, the estimates do not account for the added burden of long COVID due to reinfection 29 and the possibility of latent risks (that is, risks that are not yet manifest and may emerge years or decades after infection) 3 , 37 , 41 . The emergence of new variants, changes in public health measures and changes in the effectiveness and uptake of vaccination may also substantially influence these estimates in the future.

We estimated the global incidence of long COVID on the basis of meta-regression estimates that pool together all the available evidence. Considering the Institute for Health Metrics and Evaluation’s annual estimates of SARS-CoV-2 infections 31 , 55 , 56 , 57 , 58 , 59 and assuming the lower risk estimate of 6.2% for long COVID at 3 months after infection 31 , a proportion symptomatic cases among infections of 65% (ref. 31 ), and a reduction in the risk of long COVID for 2022 and 2023 (to account for the combination of the putative lower severity of the Omicron variant and the mildly protective effect of vaccination) 60 , the estimated cumulative global incidence of long COVID was 65 million, 211 million, 337 million and 409 million in 2020, 2021, 2022 and 2023, respectively.

While it is challenging to provide estimates of new cases with high precision, the current evidence makes it compellingly clear that long COVID represents a substantial and ongoing challenge to global health.

Mechanisms of long COVID

The pathophysiological mechanisms of long COVID are still being elucidated 2 , 62 , and it is unlikely that a single mechanism can explain the broad and heterogeneous set of symptoms and diseases spanning various organ systems. Long COVID likely represents a disease with many subtypes; each may have their own risk factors, biological mechanisms and disease trajectory, and may respond differently to treatments 3 . Multiple pathological pathways may be engaged depending on various factors, including prior environmental exposures, genetic makeup, age, sex, prior health, microbiome health, viral characteristics (SARS-CoV-2 variant, viral load), the immune response (which may be influenced by prior infections, vaccines and use of immunosuppressive agents) and medical treatments (antivirals, steroids). All of these drivers likely shape the human host response during the acute phase of SARS-CoV-2 infection and may trigger pathophysiological mechanisms that ultimately produce phenotypes of long COVID.

Several mechanistic pathways have been proposed for long COVID, including viral persistence, immune dysregulation, mitochondrial dysfunction, complement dysregulation, prothrombotic inflammation and microbiome dysbiosis 3 , 7 , 12 , 63 , 64 , 65 , 66 , 67 , 68 , 69 (Fig. 3 ). Viral persistence (either replicating virus or viral RNA or protein fragments)—which may be common 70 —in immune-privileged sites may trigger chronic low-grade inflammation and tissue injury 63 , 71 , 72 , 73 , and may correlate with long COVID symptomatology 72 .

Initial triggers (gray boxes) include viral persistence in tissue reservoirs (or immune-privileged sites) and possible replication of SARS-CoV-2 leading to the generation of viral antigens and RNA, which stimulates adaptive and innate immune cells, respectively. This can lead to immune cell activation, cytokine secretion, T cell exhaustion, antibody secretion against SARS-CoV-2 antigens and complement activation (top yellow box). Innate recognition of viral RNA by myeloid cells can lead to enhanced phagocytosis and cytokine secretion and inflammasome activation (bottom yellow box). These events can trigger autoimmunity (bystander activation or molecular mimicry) and reactivation of dormant herpesviruses (EBV, VZV) and uncoordinated cross-talk between cellular and adaptive immunity. Immune activation can cause downstream pathologies (pink boxes), including mitochondrial dysfunction and impaired energy metabolism; microbiome dysbiosis and translocation and gut nervous system dysregulation; neuronal inflammation, activation of microglia and immune cells with reduced neurogenesis and loss of oligodendrocytes and myelinated axons, possible fusion between neurons and neurons and glial cells and formation of multicellular syncytia, which compromises neuronal activity; dysfunctional hypothalamic–pituitary–adrenal response leading to inappropriately low levels of cortisol; complement activation, endothelial inflammation, platelet activation and red blood cell lysis leading to thromboinflammation and tissue injury. These mechanisms are non-exclusive and may cause inflammation, tissue dysfunction and tissue damage (blue box) leading to clinical manifestations of long COVID.

Studies have demonstrated persistence of the virus in extrapulmonary sites, including the brain and coronary arteries, of individuals with severe COVID-19 (refs. 68 , 74 ). Studies in human and mouse brain organoids showed that SARS-CoV-2 infection induces fusion between neurons and between neurons and glial cells, which may progressively lead to formation of multicellular syncytia compromising neuronal activity 75 . Neuroimaging studies performed in humans 10 months after they ‘recovered’ from mild-to-moderate SARS-CoV-2 infection showed significant alterations (commensurate with 7 ‘years of healthy aging’) of cerebral white matter, including widespread increases of extracellular free water and mean diffusivity (indicative of inflammation) encompassing all brain lobes 76 . Pre- and post-SARS-CoV-2 infection imaging studies showed structural abnormalities and accelerated aging in the brains of people with mild-to-moderate SARS-CoV-2 infection 74 , 77 , 78 . Even in the absence of direct infection in the brain, a transient respiratory infection with SARS-CoV-2 induces prolonged neuroinflammatory responses, activation of microglial cells and impaired neurogenesis 64 , 77 . In addition to neuroinflammation, people with brain fog due to long COVID were shown to have disrupted blood–brain barriers 79 .

Abnormalities in the immune system have been documented in people with long COVID, including increased humoral responses directed against SARS-CoV-2; higher antibody responses against Epstein–Barr virus (EBV) 66 , varicella zoster virus (VZV) 66 and cytomegalovirus 67 (suggesting possible reactivation of herpesviruses 80 ); exhausted T cell responses 12 , 66 ; and uncoordinated cross-talk between the cellular and humoral adaptive immunity 12 , 13 . Autoimmune responses triggered by SARS-CoV-2 infection may underlie long COVID symptoms 81 , 82 . Passive transfer of IgG antibodies from patients with long COVID to healthy mice recapitulated heightened pain sensation and locomotion deficits 82 , 83 .

In the heart, SARS-CoV-2 infects coronary vessels, preferentially targeting coronary artery plaque macrophages and inducing plaque inflammation 68 . Vascular disease in long COVID is likely triggered by complement activation, red blood cell lysis, platelet activation and thromboinflammation—leading to altered coagulation and tissue injury 67 , 84 . Dysfunctional hypothalamic–pituitary–adrenal response with inappropriately low levels of cortisol may mediate some of the symptomatology observed in long COVID (including fatigue, sleep abnormalities and metabolic derangements) 66 , and has been seen in those with persistent respiratory symptoms of long COVID 80 . SARS-CoV-2 infection may lead to reduced intestinal absorption of tryptophan (a serotonin precursor) and subsequently reduced levels of circulating serotonin, which may impair cognition via reduced vagal signaling 85 . SARS-CoV-2 infection may also lead to mitochondrial dysfunction, systemic metabolic abnormalities and abnormal skeletal muscle response to exercise—including exercise-induced myopathy and tissue infiltration of amyloid-containing deposits and leukocytes 65 .

The proposed mechanisms of long COVID share similarities with those of other post-acute infection syndromes, which are beyond the scope of this article and are discussed in detail elsewhere 24 .

Prevention, treatment and care models

Non-pharmaceutical interventions (for example, masking, improved indoor air quality) can reduce the risk of SARS-CoV-2 infection and consequently reduce the risk of long COVID. COVID-19 vaccines may partially reduce the risk of long COVID in adults by 15–70% (mean, ~40%) 86 , 87 , 88 , 89 ; they may also partially reduce the risk of long COVID in children 90 , 91 . In nonhospitalized individuals (mild-to-moderate COVID-19) who have at least one risk factor for the development of severe COVID-19, use of the SARS-CoV-2 antivirals (ritonavir-boosted nirmatrelvir and molnupiravir) in the acute phase may reduce the risk of long COVID 92 , 93 , 94 , 95 , 96 , 97 . However, the effectiveness of these antivirals in reducing risk of long COVID in low-risk groups, including younger individuals with no comorbidities 98 , has not been evaluated. Simnotrelvir—a new SARS-CoV-2 antiviral available in China 99 —resulted in earlier reduction in viral load and faster resolution of acute symptoms (than placebo) 100 , but its effectiveness against long COVID has not yet been evaluated. Exploratory analyses showed that another new SARS-CoV-2 antiviral, ensitrelvir (currently available in Japan), reduced the risk of long COVID when initiated in the acute phase of COVID-19 (refs. 101 , 102 ). Furthermore, metformin (initiated within 7 days of SARS-CoV-2 infection) has been shown to reduce the risk of long COVID in a randomized controlled trial 103 .

Evidence for long COVID treatments is beginning to emerge, but it is still limited. A randomized, double-blind, placebo-controlled trial showed that treatment with a synbiotic preparation (a gut microbiome modulator) alleviated multiple symptoms of long COVID—highlighting the need to further explore microbiome modulators as potential therapeutics in this setting 104 . Another randomized, controlled trial showed that a 15-day course of ritonavir-boosted nirmatrelvir did not reduce the burden of long COVID symptoms in comparison to ritonavir with placebo 105 .

Due to near-total absence of evidence from randomized clinical trials to guide treatment decisions, approaches for the assessment and treatment of respiratory sequelae 106 , cardiovascular complications 107 , fatigue 108 , cognitive symptoms 109 , autonomic dysfunction (including POTS) 110 , 111 , 112 , 113 , 114 and neuropsychiatric impairment 115 , 116 in adults and children 117 are based on evidence of treating similar symptomatology from other conditions—including myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) and Gulf War illness 26 , 118 , 119 , 120 .

Care for people with long COVID varies widely across settings and practitioners 118 , 119 , 121 , 122 , 123 . It is often challenged by lack of widespread recognition and understanding of long COVID among medical professionals, constrained resources and competing demands on healthcare systems still recovering from the shock of the pandemic, lack of standardized care pathways, lack of definitive diagnostic and treatment tools, and a general pervasive pandemic fatigue with an urge to ‘move on’ 124 , 125 . Much of the global burden of long COVID remains undiagnosed, particularly in low-resource settings, and in many instances are erroneously attributed to psychosomatic causes 126 .

Overall, care models for long COVID are evolving, with substantial variability across health systems 127 . While there is still no empirical evidence evaluating comparative effectiveness of long COVID care models 121 , optimal models should be context dependent—based on available resources, expertise and the population being served 121 , 128 .

Impacts of long COVID

In addition to its impact on patients’ daily lives and health outcomes, long COVID has a devastating impact on communities and can have wide-reaching ramifications for health systems, national economies and global health metrics.

Impact on individuals and communities

Long COVID drastically affects patients’ well-being and sense of self, as well as their ability to work, socialize, care for others, manage chores and engage in community activities—which also affects patients’ families, caregivers and their communities 129 . Over three quarters of people with long COVID report a moderate or severe impact on general well-being 130 . The high rates of cognitive and physical symptoms also affect individuals’ identity and sense of self. One in four people with long COVID limit activities outside work in order to continue working 131 . Many patients with long COVID experience social exclusion, isolation and stigma, often from medical providers 43 , 132 , 133 , 134 . These challenges are exacerbated by societal barriers to the inclusion of people with disabilities and chronic illnesses.

Impact on health systems

Because of the large burden of long COVID and its multisystemic effects 135 , it has profound impacts on health systems 136 , 137 . Patients with long COVID frequently require ongoing medical care and multiple specialist consultations to manage their complex symptoms. This increased demand exacerbates existing pressures on health systems, leading to longer wait times, potential delays in essential care and increased costs. In the United States, people with long COVID are more likely to report unmet healthcare needs in the past year because of costs and difficulties finding a clinician and getting an appointment when needed 138 . These issues are exacerbated in low- and middle-income countries 126 , 139 . Furthermore, the lack of standardized diagnostic criteria, treatment protocols and models of care for long COVID adds to the complexity and places additional burdens on healthcare providers 137 , 140 , 141 .

Perhaps the most enduring challenge to health systems lies in the rise in the burden of non-communicable diseases (NCDs; for example, cardiovascular disease and diabetes) as a consequence of SARS-CoV-2 infection 4 , 5 , 9 , 10 , 15 , 136 , 142 , 143 , 144 , 145 , 146 , 147 , 148 , 149 , 150 , 151 , 152 . NCDs are chronic conditions that require lifelong care, impact health system utilization (competing for access and quality of care) and raise healthcare costs 137 .

Impact on economies

Long COVID strains individual financial health 153 and has wide and deep ramifications on national economies 154 , 155 , 156 , 157 , 158 , 159 . In addition to the substantial direct healthcare costs 160 , there is also financial strain on support services and disability benefits. In addition, long COVID affects labor participation, employment and productivity of impacted individuals and their caregivers 129 , 156 , 161 , 162 , 163 —resulting in depleted savings, food and housing insecurity 131 , 164 and negative impact on labor supply, thereby fueling labor shortages 156 . Studies indicate a significant percentage of individuals with long COVID experience a reduced ability to work or may be unable to work at all 165 . A report by the US Brookings Institute estimated that between 2 and 4 million US adults were out of work because of long COVID in 2022 (ref. 165 ). A US Federal Reserve Bank report found that people with long COVID had 10% less likelihood of being employed and worked 25% to 50% fewer hours when employed than uninfected individuals 166 . Survey data from the UK’s Trades Union Congress show that 20% of people with long COVID were not working and that an additional 16% were working reduced hours 167 . An analysis by the European Commission suggested that long COVID had a negative impact on the European labor supply of 0.2–0.3% in 2021 and 0.3–0.5% in 2022 (ref. 168 ).

Quantitative estimates of the total economic impact of long COVID remain preliminary. A study in 2022 estimated the economic cost of three key parameters in the United States, including lost quality of life ($2,195 billion), cost of lost earning ($997 billion) and spending on healthcare ($528 billion), for up to a total cost of $3.7 trillion 154 , 155 —this amounts to $11,000 per capita or 17% of the 2019 gross domestic product (GDP). These economic losses are on par with the global 2008 Great Recession. Assumptions included in these estimates are that burden of disability from long COVID is on par with that of ME/CFS and that long COVID lasts on average for 5 years 155 .

Among OECD (Organization for Economic Co-operation and Development) countries, a preliminary conservative estimate suggested that excluding the direct costs of healthcare, long COVID is likely costing OECD countries as much as $864 billion to $1.04 trillion per year due to reductions in quality of life and labor force participation 169 . A recent analysis by the Economist Impact (a think tank of The Economist) suggested that the economic cost of long COVID in 2024 is expected to be around 0.5% to 2.3% of the GDP of several large economies 170 (Table 1 ). On the basis of all the available data, a conservative estimate of the annual global economic toll of long COVID could be around $1 trillion amounting to 1% of the 2024 global GDP 154 , 155 , 169 , 170 .

Impact on the SDGs

The profound immediate health, social and economic shocks triggered by the COVID-19 pandemic have undermined the ability of many countries to achieve the SDGs by 2030 (ref. 171 ). In addition to the immediate effect of the pandemic, its long tail—in the form of long COVID—presents a more profound and enduring challenge to SDGs than the direct initial disruptions 171 .

Long COVID’s multifaceted impact jeopardizes progress across many SDGs, particularly those aimed at promoting health and economic well-being, and reducing inequalities 172 , 173 , 174 , 175 , 176 , 177 , 178 , 179 . Long COVID can limit access to and quality of healthcare 136 , 137 , reduce labor participation, worsen poverty and hinder economic productivity 169 , and exacerbate existing inequalities 180 , 181 , 182 , 183 . Table 2 lists the impacts of long COVID on several of the SDGs and identifies which collaborative, multi-sectoral partnerships and actions are needed to address these impacts.

The full extent to which long COVID will undermine the SDGs is still evolving and is difficult to fully quantify 174 , 175 ; a deeper understanding of the full scope and scale of this impact is needed.

Research and policy roadmaps

Substantial work lies ahead to address the broad and multifaceted challenges posed by long COVID—including preventing further increase in the number of people with long COVID and addressing the care needs of people already impacted 184 . Responding to these challenges will require coordinated, long-term policy response and visionary research strategies, guided by the principles of health equity and patient centeredness 185 , 186 , 187 . We developed the following research and policy roadmaps on the basis of our assessment of the evidence and policy gaps, as well as our own clinical, research and policy experience and in partnership with patients.

Research roadmap

Biological mechanisms.

Leading mechanistic hypotheses (discussed above) should be examined carefully, particularly for their interactions and potential to guide disease management, trials to test existing drugs and the development of new drugs 3 . Continued investigation (via animal models 188 , 189 or other approaches) of neuroinflammation, immune dysregulation, sex differences 190 , tissue damage and susceptibility features, including genomic 191 , epigenomic 192 , 193 , 194 and other '-omics', is warranted. In evaluating the mechanisms of long COVID, detailed assessment of specific manifestations, for example, understanding the pathophysiology of post-exertional malaise, may yield mechanistic insights that guide clinical management 65 , 195 , 196 .

That SARS-CoV-2 leads to long COVID is unlikely to be a unique property; many other viral agents (including influenza, SARS, Middle East respiratory syndrome, EBV, Dengue, Ebola, Polio, Chikungunya, West Nile virus, Ross River virus, Coxsackie B and VZV) and nonviral agents ( Coxiella burnetii , Borrelia , Giardia lamblia ) also lead to post-acute and long-term health effects 24 , 197 . A deeper understanding of the similarities and distinctions in the biological mechanisms of long COVID and other infection-associated chronic conditions is needed 2 , 3 , 24 , 25 , 26 , 198 , 199 , 200 , 201 , 202 , 203 , 204 , 205 , 206 , 207 , 208 , 209 , 210 , 211 , 212 , 213 , 214 .

Diagnostics

A research agenda is needed to foster the development, testing and validation of more advanced imaging, new blood tests, molecular probes, '-omics' and novel approaches to tissue investigation and analyses—toward better diagnosis of long COVID. Traditional imaging techniques may not reveal abnormalities in long COVID that may be evident in more advanced imaging. For example, new imaging technologies, including magnetic resonance imaging (MRI) with xenon-129 ( 129 XE-MRI) 215 for lungs, diffusion MRI to map glial activity 216 , imaging for glymphatic functioning 217 and arterial spin labeling MRI 218 for cerebral blood flow, have identified abnormalities in long COVID where conventional imaging has not. In a preliminary study, imaging flow cytometry was shown to detect fibrin microclots, which may be more abundant in people with long COVID than controls 219 . Whole-body positron emission tomography imaging using a highly selective radiotracer ([ 18 F]F-AraG) that allows anatomical quantification of activated T lymphocytes, showed increased radiotracer uptake indicative of T cell activation in various anatomic sites (for example, spinal cord, lungs) that were associated with long COVID 220 . These imaging modalities—along with other approaches—should be further investigated for their potential to establish diagnosis of long COVID, to guide trial designs, and for targeted disease management.

Biomarkers are helpful, not only as diagnostics, but also to aid in risk stratification (to guide trials and choice of treatment), determine potential subtypes of disease, and assess severity, prognosis and response to treatment. Candidate biomarkers include immune cell phenotypes, cytokines/chemokines, immunoglobulins, complement and coagulation proteins, acute phase proteins, endocrine markers and markers of neurologic or vascular injury 66 , 67 , 73 , 221 , 222 . Integrated '-omics' analyses 223 , 224 , including genomic, epigenomic, transcriptomic 225 , proteomic 226 , 227 , 228 , metabolomic 229 , lipidomic 230 , and microbiome 231 profiling, may help identify fingerprints for various types of long COVID. However, because of the complexity of long COVID and its diverse manifestations, which likely represent distinct mechanistic pathways, a single or even a panel of laboratory tests may not achieve high-enough performance. Sequela-specific approaches for biomarker discovery may also be productive 221 .

In addition to imaging modalities and biomarkers, harnessing health data from wearable biosensors and other sources may also be useful for diagnosis and to identify triggers and track disease activity.

Epidemiology and clinical course

Studies to understand the incidence, prevalence, severity and trajectory of long COVID over time are critical 35 , 36 , 232 , 233 . Comprehensive understanding of risk factors, including social determinants of health, genetic, environmental, dietary, health behavior (for example, smoking) and other risks of long COVID, is also important.

Research to identify the putative subtypes (or clusters of sequelae) of long COVID has yielded variable results thus far 234 , 235 , 236 , 237 ; greater clarity is needed on putative subtypes and how might they differ in terms of epidemiological features (for example, risk factors), clinical course and potential response to treatment.

Real-world evidence using high-quality data and advanced causal inference approaches (for example, target trial emulation) to evaluate effectiveness of therapeutic interventions will complement evidence generated by randomized trials 238 , 239 . This is particularly relevant in the evaluation of the long-term effects of therapeutic interventions and risks of rare adverse events; trials may have a relatively short follow-up, limiting assessment of long-term outcomes. Moreover, trials may not be adequately powered to detect rare adverse events.

Because long COVID is a new entity (it has been in existence for less than 5 years), longitudinal studies to characterize the long-term health trajectories of people with long COVID—up to 10 years, 20 years and 30 years—are needed, to understand rates and predictors of recovery and relapse of the various manifestations. These long-term studies will also help identify any latent consequences of the disease (that is, impacts that have not yet been realized) and secondary consequences (for example, the downstream health effects that emanate from long COVID). For example, understanding whether people with cognitive dysfunction (or brain fog) are at a higher risk of developing neurodegenerative diseases later in life is critical.

Comparative analyses to understand the post-acute and long-term health consequences of SARS-CoV-2 infection (and reinfection) versus other infections (for example, seasonal and pandemic influenza, respiratory syncytial virus infections and others) is important to enhance our understanding of similarities and differences in their epidemiology and clinical course 197 , 240 , 241 , 242 .

Quantifying the burden of NCDs attributable to long COVID would bring greater clarity to the extent to which billions of SARS-CoV-2 infections around the world may have impacted the global epidemiology of NCDs. The effects of long COVID on global health metrics, including SDGs, should also be periodically evaluated.

Trials to test therapeutics for long COVID

When it comes to clinical trials for long COVID therapeutics, innovation, urgency and scale are all needed 243 , 244 . Long COVID is a complex disease with many manifestations that are likely driven by several different biological mechanisms, and may need different therapeutic approaches. Approaches that reimagine trial design to incorporate the complexities of the disorder and meaningfully incorporate patient input—from trial inception to completion—are needed 244 , 245 . This may include large-scale platform trials with adaptive designs that would test a large battery of potential drug candidates to quickly identify treatments for the various forms of long COVID.

There is a large array of existing drugs that could be readily repurposed and clinically evaluated to address existing hypotheses from viral persistence to immune system dysfunction to vascular damage. Some of these drugs include SARS-CoV-2 antivirals, neutralizing monoclonal antibodies against SARS-CoV-2, non-SARS-CoV-2 antivirals (targeting reactivated EBV and VZV), immunomodulators (for example, JAK–STAT inhibitors, checkpoint inhibitors), anticoagulants, histamine 1 and 2 antagonists, metformin, GLP-1 receptor agonists, SGLT2 inhibitors, microbiome modulators, anti-inflammatory agents, and drugs that improve glymphatic functioning 2 , 62 , 246 . Research agendas must also include development of new antivirals and other new targeted drugs to prepare for the possibility that repurposed drugs may not be sufficiently effective 247 , 248 , 249 . Testing and evaluation of combinations of treatments should also be undertaken when evidence suggests complementary or synergistic mechanisms of action.

Innovation in developing and validating entry criteria and clinical endpoints for long COVID trials is also needed, along with cultivating support for these parameters from stakeholders, including regulators such as the US Food and Drug Administration and European Medicines Agency 250 . Endpoints must include newly developed or improved patient-reported outcomes specific to long COVID and should reflect the often cyclical or relapsing–remitting dynamic of many manifestations of long COVID—with particular focus on tracking post-exertional malaise, a pathophysiological state that impacts all collected data.

Care delivery and health systems research

Research—including comparative analyses—to evaluate the cost and effectiveness of various care pathways in improving quality of care and outcomes in people with long COVID is needed 121 , 127 , 251 . Research to identify and address health inequities and barriers to effective care, especially in low- and middle-income countries, in low-resource settings and in underserved communities, is essential 252 .

Economic impacts

The effect of long COVID on human capital 253 , labor participation, productivity losses (workforce absenteeism, presenteeism and disability) and other economic indicators (including job retention, career advancement and income instability) should be thoroughly evaluated. Research should explore potential disparities in the economic impact of long COVID across various demographic groups, including racial and ethnic minorities, urban and rural communities, socioeconomically marginalized populations, and individuals with preexisting health conditions.

In addition, studies are needed to quantify the direct healthcare costs associated with long COVID. The costs of disability and support systems required to address the needs of people with long COVID should be quantified. The strain that these costs pose on payors (insurance providers and governments) should also be evaluated.

Understanding the economic barriers to healthcare access and affordability for people with long COVID is also important. This includes evaluating out-of-pocket expenses, insurance coverage gaps, and disparities in access to care, rehabilitation services and support services.

Societal impacts

Long COVID affects individual lives and impacts societal well-being. Understanding the effects of long COVID on societies is important, along with understanding the social responses, the perceptions and the genesis and propagation of stigma. Improved knowledge of the social consequences of being affected by long COVID—for example, lost friendships, strained marriages and reduced ability to network—along with the interplay between them and health outcomes, will help to inform supportive interventions. It will also be important to evaluate the burden on caregivers, families and social groups.

Research to develop a deeper understanding of the causes and consequences of misinformation, disinformation and anti-science rhetoric (for example, long COVID denialism) and how to effectively combat them is also needed 254 . Identifying ways to improve science communication, scientific literacy and public trust in science and to bridge the science–policy gap would all help to improve public understanding, as well as the scientific and policy responses to long COVID 254 .

Medical anthropology should also contextualize the response of the science and medicine profession to long COVID within the broader history of medicine. This should include comparative analyses to evaluate and juxtapose the response to long COVID against the responses to the aftereffects of the 1889–1892 flu pandemic and the 1918 flu pandemic and other health crises, including the AIDS crisis in the 1980s 198 , 199 , 201 , 255 , 256 , 257 . Careful anthropologic analysis of how the medical profession approached long COVID as a new disease that emerged in the context of the COVID-19 pandemic is important. It will not only provide historic insights and greater context for our collective response, but also offer insights into how we can optimize responsiveness to emergence of new infection-associated diseases in the future.

Policy roadmap

Given the wide-ranging impact of long COVID on society and the inadequate response thus far, priorities for policy changes are vast. Policies are necessarily dependent on context, resources and various other considerations. The recommendations outlined below are general guidelines that may be adapted to fit the needs of various locales.

Prevention of long COVID

The best way to prevent long COVID is, plainly, to prevent SARS-CoV-2 infection or reinfection in the first place. Masking, especially in high-risk places 258 (for example, healthcare settings), is important—along with isolation guidelines and sick leave policies that permit people with infection to recuperate at home, thereby diminishing the probability of transmission and reducing the risk of long COVID 45 .

Although vaccines may reduce the risk of long COVID, vaccine policies in much of the world restrict vaccine availability to high-risk groups. These policies consider risks of death and hospitalization in the acute phase (which are manifest primarily in older adults and those with comorbidities) and ignore the risk of long COVID. Adding to these policy challenges are the low rates of vaccine uptake in 2023–2024 among eligible populations 259 . Vaccine policies must consider the risk of long COVID, as well as the risk of hospitalization and death during the acute phase of SARS-CoV-2 infection; and strategies to improve vaccine uptake (for example, pairing the COVID-19 vaccine with the annual influenza vaccine and other approaches) should be utilized to achieve wider vaccine coverage and greater protection to populations.

Because SARS-CoV-2 is likely to remain for decades to come, it is important to develop long-term, sustainable prevention solutions. Airborne transmission risk assessment tools, such as the one developed by the World Health Organization (WHO), help inform risk reduction strategies 260 . Ventilation and air filtration systems can play a major role in reducing the risk of infection with airborne pathogens 261 . Calls have been made for mandatory improved air quality standards for public spaces and policies that would support design and equipment of homes to meet these standards 261 . Investment in infrastructure supporting improved indoor air quality will help reduce the risk of SARS-CoV-2 transmission and other airborne pathogens and will ensure greater resilience against future threats from airborne pathogens 262 . Amelioration of indoor air quality also has the added benefit of reducing risk of health effects due to indoor air pollutants 263 , 264 , 265 , thereby improving human health, well-being, productivity and learning 261 , 262 , 266 . Investment in vaccine technologies to develop more durable, variant-proof vaccines that are not rendered ineffective by ongoing mutations of the SARS-CoV-2 virus are important. Vaccine technologies that induce strong mucosal immunity to block SARS-CoV-2 infection and transmission are also needed 267 .

Supporting people with long COVID

Because of the considerable impacts of long COVID on people’s ability to work and care for themselves, it is imperative that an adequate response to the long COVID crisis involves ensuring people have the financial, physical and emotional support 132 . Streamlining of disability benefit processes, as well as increased access to home and community-based services and food and cash assistance is critical. Workplace policies that support individuals with long COVID could include flexible working hours, increased breaks to allow for pacing, the option for remote work, and sick leave policies. Funding should be provided to support patient groups and community-based organizations, which can provide and connect people to critical supports and services.

Access, quality and equity of care

Governments must work to build and expand access to long COVID care, in particular for marginalized communities (for example, rural and indigenous communities). Improving access to care may take various forms in different countries, depending on the structure of the healthcare system and the involvement of national and local governments in financially supporting healthcare services. Adequate coverage of long COVID treatments and rehabilitation services by insurance providers is requisite. Development of quality-of-care metrics for long COVID and policies to monitor and incentivize quality of care should be pursued 121 . As diagnostics and treatments are developed, governments must also ensure equitable access. Shining historical examples include the Brazilian National AIDS Program, which was established in 1996 in response to the HIV/AIDS crisis to ensure free and universal provision of antiretroviral drugs 268 , 269 , and the Ryan White HIV/AIDS Program (based in the United States), which provides outpatient HIV care, treatment and support services to those without health insurance and fills gaps in coverage and cost for those with insurance limitations 270 , 271 , 272 .

Professional education and training

Currently, very few medical schools and health professional training programs include in their curricula any meaningful training about identification and clinical management of infection-associated chronic conditions, including long COVID. A survey of physicians in the United States showed that 78% agree that long COVID is a problem but only about one-quarter feel prepared to address it 273 . Training of healthcare professionals to recognize and manage long COVID effectively must be prioritized. This includes embedding up-to-date information on long COVID and infection-associated chronic conditions into training curricula for health professions, as well as providing regular high-quality continuing education to qualified health providers.

Public health communication

Existing public health education on long COVID has been minimal. A survey in the United States showed that one-third of American adults still had not heard of long COVID as of August 2023 (ref. 274 ), and there remains very low awareness of long COVID in low- and middle-income countries. Through public education campaigns, governments must raise awareness about long COVID and the risk of chronic conditions after infection; combat social stigma across adults and children; and use a harm reduction framework to promote awareness of prevention measures (including vaccination, masking and improved indoor air quality) 258 , 260 , 261 .

Supporting coordinated interdisciplinary research

To achieve the research priorities listed above, governments must substantially increase the amount of funding toward research. In the United States, existing calls for the establishment of a center for infection-associated chronic conditions at the US National Institutes of Health—with a funding request of at least $1 billion per year toward long COVID research and with additional substantial funding for other infection-associated chronic conditions—should be vigorously supported 275 . This proposal would create a coordinating entity to lead a long-term, large-scale interdisciplinary research portfolio to address long COVID research priorities. Other governments should also explore similar proposals.

Policies supporting research should explicitly mandate meaningful patient engagement in research from inception to implementation, and should leverage existing expertise (including scientific, clinical and lived experience) in infection-associated chronic conditions. Furthermore, meaningful efforts must be made to expand the pool of researchers working on infection-associated chronic conditions, by encouraging early career scientists and clinician–researchers to focus on these conditions and providing resources to current experts to lead training and research.

Given the complexity of long COVID and its similarities to other infection-associated chronic conditions, a coordinated approach that integrates research, policy and regulatory efforts across these conditions would reduce duplication of efforts and allow a more comprehensive understanding of the common underlying mechanisms, trial designs and potential treatment strategies.

Policies from funders are needed to mandate meaningful data sharing, which will maximize the utility and pooled insights that can be generated from existing health information. Current open data protocols are insufficient, laden with bureaucratic hurdles and do not allow access to primary data, and consequently do not enable meaningful analyses. Funders must establish data banks (a pioneering exemplar of this is the UK Biobank) for the collection, storage, analysis, retrieval and dissemination of data to make long COVID research more accessible in near real time, all while upholding data privacy and data security standards 276 , 277 .

Building consensus on definitions and clinical endpoints for long COVID

Various interim definitions of long COVID exist 39 , 278 , 279 , 280 , 281 , 282 , 283 , but there is not yet a universal consensus on the most optimal definition—which must be sufficiently nuanced to capture the complexity of the condition and its various manifestations. It is unlikely that a single definition will fit all needs. Consensus definitions that are optimized and empirically tested for various applications, including clinical care, epidemiological surveillance, and research, should be developed. Definitions must necessarily evolve to incorporate new understanding as the evidence base for long COVID grows.

Similarly, developing consensus on clinical endpoints for trials of long COVID is needed. Drug regulatory agencies in consultation with stakeholders, including patients and scientists, should lead in this arena and provide regulatory guidance on clinical endpoints for trials. These endpoints will also have to necessarily evolve as our understanding of long COVID expands.

Building consensus on definitions and clinical endpoints would catalyze progress in this field, remove barriers to entry for the pharmaceutical industry into long COVID trials and facilitate comparative analyses across studies.

Global coordination

The global nature of long COVID necessitates international cooperation in both research and policy. International bodies (for example, the WHO) should facilitate partnership and collaboration among countries across the globe. This collaboration is pivotal to coordinate and synergize efforts across the globe and accelerate progress on the different challenges posed by long COVID.

Professional societies for long COVID

Professional societies (national and global) should be established for long COVID. Because of the multisystemic nature of long COVID (and the other infection-associated chronic conditions), it does not fit neatly under any of the traditional organ-based organizational structures of medical care and research 284 , hence the need for professional home(s) for long COVID and associated conditions. Dedicated professional societies could provide strategic leadership and guidance in the clinical management of long COVID and associated conditions 284 . They could serve as hubs to coordinate education, research and advocacy efforts 284 . These professional societies could play a major role in organizing and hosting national and international conferences, spearheading efforts to provide periodic synthesis of evidence that distills existing research into actionable insights guiding care of people with long COVID. The newly established Clinical Post COVID Society in the United Kingdom may be a promising example of this 284 .

Preparedness for the next pandemic

We must reflect on our collective experience with COVID-19 to enhance resilience and preparedness for future pandemics 285 , 286 , 287 . A major lesson learned from long COVID is that pandemics leave in their wake a long tail of disease and disability 198 . This is not unique to the COVID-19 pandemic 198 ; historical accounts show similar phenomena following previous pandemics 198 , 199 , 255 . Due to climate change, deforestation, human encroachment on animal habitat, increased frequency of travel, a growing livestock industry and other anthropogenic factors, the risk of zoonotic spillover and novel viral sharing among species is likely higher in the twenty-first century than it was in the twentieth century 286 , 288 , 289 , 290 , 291 , 292 . Many of the geographic areas that are most prone to these changes are also projected to have high population density—creating ripe conditions for pandemics 289 , 291 . Future pandemics are likely to also produce long-term disability and disease 198 . Investment in systems to measure the population-level incidence and prevalence of post-acute and chronic disease caused by infectious agents, including SARS-CoV-2, will aid in the characterization of the epidemiology of long COVID and will position us to be better prepared to deal with post-acute and chronic illnesses that will emerge in future pandemics. Incorporating the potential emergence of long-term health effects into initiatives for pandemic preparedness and resilience (for example, the WHO Preparedness and Resilience for Emerging Threats Initiative) is essential to optimize response to the long-term consequences of future pandemics.

Conclusions

Considerable progress has been made in the past several years in characterizing the epidemiology, clinical course and biology of long COVID. But much remains to be done. The scale of long COVID and its far-reaching impacts necessitate a robust and coordinated research and policy response strategy. Addressing the research and care needs of people impacted by long COVID will have broad benefits, potentially unlocking a better understanding of infection-associated chronic illnesses (an ignored area for decades) and optimizing our preparedness for the next pandemic.

Al-Aly, Z., Xie, Y. & Bowe, B. High-dimensional characterization of post-acute sequelae of COVID-19. Nature 594 , 259–264 (2021).

Article   CAS   PubMed   Google Scholar  

Davis, H. E., McCorkell, L., Vogel, J. M. & Topol, E. J. Long COVID: major findings, mechanisms and recommendations. Nat. Rev. Microbiol. 21 , 133–146 (2023).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Al-Aly, Z. & Topol, E. Solving the puzzle of Long COVID. Science 383 , 830–832 (2024).

Xie, Y., Xu, E., Bowe, B. & Al-Aly, Z. Long-term cardiovascular outcomes of COVID-19. Nat. Med. 28 , 583–590 (2022).

Xu, E., Xie, Y. & Al-Aly, Z. Long-term neurologic outcomes of COVID-19. Nat. Med. 28 , 2406–2415 (2022).

Xie, Y., Xu, E. & Al-Aly, Z. Risks of mental health outcomes in people with COVID-19: cohort study. Br. Med. J. 376 , e068993 (2022).

Article   Google Scholar  

Al-Aly, Z. & Rosen, C. J. Long COVID and impaired cognition - more evidence and more work to do. N. Engl. J. Med. 390 , 858–860 (2024).

Article   PubMed   PubMed Central   Google Scholar  

Kim, S. et al. Short- and long-term neuropsychiatric outcomes in long COVID in South Korea and Japan. Nat. Hum. Behav. https://doi.org/10.1038/s41562-024-01895-8 (2024).

Xie, Y. & Al-Aly, Z. Risks and burdens of incident diabetes in long COVID: a cohort study. Lancet Diabetes Endocrinol 10 , 311–321 (2022).

Xu, E., Xie, Y. & Al-Aly, Z. Risks and burdens of incident dyslipidaemia in long COVID: a cohort study. Lancet Diabetes Endocrinol. 11 , 120–128 (2023).

Bowe, B., Xie, Y., Xu, E. & Al-Aly, Z. Kidney outcomes in long COVID. J. Am. Soc. Nephrol. 32 , 2851–2862 (2021).

Yin, K. et al. Long COVID manifests with T cell dysregulation, inflammation and an uncoordinated adaptive immune response to SARS-CoV-2. Nat. Immunol. 25 , 218–225 (2024).

Peluso, M. J., Abdel-Mohsen, M., Henrich, T. J. & Roan, N. R. Systems analysis of innate and adaptive immunity in Long COVID. Semin. Immunol. 72 , 101873 (2024).

Pollack, B. et al. Female reproductive health impacts of Long COVID and associated illnesses including ME/CFS, POTS, and connective tissue disorders: a literature review. Front. Rehabil. Sci. 4 , 1122673 (2023).

Xu, E., Xie, Y. & Al-Aly, Z. Long-term gastrointestinal outcomes of COVID-19. Nat. Commun. 14 , 983 (2023).

Zimmermann, P., Pittet, L. F. & Curtis, N. How common is Long COVID in children and adolescents? Pediatr. Infect. Dis. J. 40 , e482–e487 (2021).

Rao, S. et al. Postacute sequelae of SARS-CoV-2 in children. Pediatrics 153 , e2023062570 (2024).

Article   PubMed   Google Scholar  

Gurdasani, D. et al. Long COVID in children. Lancet Child Adolesc. Health 6 , e2 (2022).

Mansell, V., Hall Dykgraaf, S., Kidd, M. & Goodyear-Smith, F. Long COVID and older people. Lancet Healthy Longev. 3 , e849–e854 (2022).

Fung, K. W., Baye, F., Baik, S. H., Zheng, Z. & McDonald, C. J. Prevalence and characteristics of long COVID in elderly patients: An observational cohort study of over 2 million adults in the US. PLoS Med. 20 , e1004194 (2023).

Xie, Y., Bowe, B. & Al-Aly, Z. Burdens of post-acute sequelae of COVID-19 by severity of acute infection, demographics and health status. Nat. Commun. 12 , 6571 (2021).

Blitshteyn, S. & Whitelaw, S. Postural orthostatic tachycardia syndrome (POTS) and other autonomic disorders after COVID-19 infection: a case series of 20 patients. Immunol. Res. 69 , 205–211 (2021).

Volberding, P. A., Chu, B. X. & Spicer, C. M. Long-Term Health Effects of COVID-19 (The National Academies Press, 2024).

Choutka, J., Jansari, V., Hornig, M. & Iwasaki, A. Unexplained post-acute infection syndromes. Nat. Med. 28 , 911–923 (2022).

Marshall-Gradisnik, S. & Eaton-Fitch, N. Understanding myalgic encephalomyelitis. Science 377 , 1150–1151 (2022).

Komaroff, A. L. & Lipkin, W. I. ME/CFS and Long COVID share similar symptoms and biological abnormalities: road map to the literature. Front. Med. 10 , 1187163 (2023).

Xie, Y., Choi, T. & Ziyad, A.-A. Postacute sequelae of SARS-CoV-2 infection in the pre-delta, delta, and omicron eras. N. Engl. J. Med . https://doi.org/10.1056/NEJMoa2403211 (2024).

Kuang, S. et al. Experiences of Canadians with long-term symptoms following COVID-19. Statistics Canada https://www150.statcan.gc.ca/n1/pub/75-006-x/2023001/article/00015-eng.htm (8 December 2023).

Bowe, B., Xie, Y. & Al-Aly, Z. Acute and postacute sequelae associated with SARS-CoV-2 reinfection. Nat. Med. 28 , 2398–2405 (2022).

Bosworth, M. L. et al. Risk of new-onset long COVID following reinfection with severe acute respiratory syndrome coronavirus 2: a community-based cohort study. Open Forum Infect. Dis. 10 , ofad493 (2023).

Global Burden of Disease Long COVID Collaborators et al. Estimated global proportions of individuals with persistent fatigue, cognitive, and respiratory symptom clusters following symptomatic COVID-19 in 2020 and 2021. J. Am. Med. Assoc. 328 , 1604–1615 (2022).

Graham, F. Daily briefing: answers emerge about long COVID recovery. Nature https://doi.org/10.1038/d41586-023-02190-8 (2023).

Mateu, L. et al. Determinants of the onset and prognosis of the post-COVID-19 condition: a 2-year prospective observational cohort study. Lancet Reg. Health Eur. 33 , 100724 (2023).

Gutiérrez-Canales, L. G. et al. Persistence of COVID-19 symptoms and quality of life at three and twelve months after hospital discharge. Medicina 60 , 944 (2024).

Huang, L. et al. 1-year outcomes in hospital survivors with COVID-19: a longitudinal cohort study. Lancet 398 , 747–758 (2021).

Zhang, H. et al. 3-year outcomes of discharged survivors of COVID-19 following the SARS-CoV-2 omicron (B.1.1.529) wave in 2022 in China: a longitudinal cohort study. Lancet Respir. Med. 12 , 55–66 (2024).

Bowe, B., Xie, Y. & Al-Aly, Z. Postacute sequelae of COVID-19 at 2 years. Nat. Med. 29 , 2347–2357 (2023).

Al-Aly, Z. Diabetes after SARS-CoV-2 infection. Lancet Diabetes Endocrinol. 11 , 11–13 (2023).

Fineberg, H. V., Brown, L., Worku, T. & Goldowitz, I. A Long COVID Definition (National Academies Press, 2024).

Taquet, M. et al. Neurological and psychiatric risk trajectories after SARS-CoV-2 infection: an analysis of 2-year retrospective cohort studies including 1,284,437 patients. Lancet Psychiatry 9 , 815–827 (2022).

Cai, M., Xie, Y., Topol, E. J. & Al-Aly, Z. Three-year outcomes of post-acute sequelae of COVID-19. Nat. Med. 30 , 1564–1573 (2024).

Davis, H. E. et al. Characterizing long COVID in an international cohort: 7 months of symptoms and their impact. EClinicalMedicine 38 , 101019 (2021).

Re’em, Y. et al. Factors associated with psychiatric outcomes and coping in Long COVID. Nat. Ment. Health 1 , 361–372 (2023).

Re’em, Y., Symeonides, M. & McCorkell, L. Serology test results and other important characteristics of patients with persistent COVID-19 symptoms. JAMA Intern. Med. 182 , 576–577 (2022).

Ziauddeen, N. et al. Characteristics and impact of Long Covid: findings from an online survey. PLoS ONE 17 , e0264331 (2022).

O’Brien, K. K. et al. Conceptualising the episodic nature of disability among adults living with Long COVID: a qualitative study. BMJ Glob. Health 8 , e011276 (2023).

Blitshteyn, S. & Verduzco-Gutierrez, M. Long COVID: a major public health issue. Am. J. Phys. Med. Rehabil. https://doi.org/10.1097/PHM.0000000000002486 (2024).

Ledford, H. How common is long COVID? Why studies give different answers. Nature 606 , 852–853 (2022).

Woodrow, M. et al. Systematic review of the prevalence of Long COVID. Open Forum Infect. Dis. 10 , ofad233 (2023).

Adjaye-Gbewonyo, D., Vahratian, A., Perrine, C. G. & Bertolli, J. Long COVID in adults: United States, 2022. NCHS Data Brief 480 , 1–8 (2023).

Google Scholar  

Vahratian, A., Adjaye-Gbewonyo, D., Lin, J. S. & Saydah, S. Long COVID in children: United States, 2022. NCHS Data Brief 479 , 1–6 (2023).

Fang, Z., Ahrnsbrak, R. & Rekito, A. Evidence mounts that about 7% of US adults have had long COVID. J. Am. Med. Assoc 332 , 5–6 (2024).

National Center for Health Statistics. Long COVID: Household Pulse Survey. CDC https://www.cdc.gov/nchs/covid19/pulse/long-covid.htm (accessed 1 July 2024).

UK Office of National Statistics. Prevalence of ongoing symptoms following coronavirus (COVID-19) infection in the UK: 30 March 2023. https://www.ons.gov.uk/peoplepopulationandcommunity/healthandsocialcare/conditionsanddiseases/bulletins/prevalenceofongoingsymptomsfollowingcoronaviruscovid19infectionintheuk/30march2023/ (30 March 2023).

COVID-19 Cumulative Infection Collaborators. Estimating global, regional, and national daily and cumulative infections with SARS-CoV-2 through Nov 14, 2021: a statistical analysis. Lancet 399 , 2351–2380 (2022).

GBD 2021 Demographics Collaborators. Global age-sex-specific mortality, life expectancy, and population estimates in 204 countries and territories and 811 subnational locations, 1950–2021, and the impact of the COVID-19 pandemic: a comprehensive demographic analysis for the Global Burden of Disease Study 2021. Lancet 403 , 1989–2056 (2024).

GBD 2021 Causes of Death Collaborators. Global burden of 288 causes of death and life expectancy decomposition in 204 countries and territories and 811 subnational locations, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet 403 , 2100–2132 (2024).

GBD 2021 Diseases and Injuries Collaborators. Global incidence, prevalence, years lived with disability (YLDs), disability-adjusted life-years (DALYs), and healthy life expectancy (HALE) for 371 diseases and injuries in 204 countries and territories and 811 subnational locations, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet 403 , 2133–2161 (2024).

GBD 2021 Risk Factors Collaborators. Global burden and strength of evidence for 88 risk factors in 204 countries and 811 subnational locations, 1990–2021: a systematic analysis for the Global Burden of Disease Study 2021. Lancet 403 , 2162–2203 (2024).

Antonelli, M., Pujol, J. C., Spector, T. D., Ourselin, S. & Steves, C. J. Risk of long COVID associated with delta versus omicron variants of SARS-CoV-2. Lancet 399 , 2263–2264 (2022).

Ma, Y. et al. Long-term consequences of asymptomatic SARS-CoV-2 infection: a systematic review and meta-analysis. Int. J. Environ. Res. Public Health 20 , 1613 (2023).

Altmann, D. M., Whettlock, E. M., Liu, S., Arachchillage, D. J. & Boyton, R. J. The immunology of long COVID. Nat. Rev. Immunol. 23 , 618–634 (2023).

Proal, A. D. et al. SARS-CoV-2 reservoir in post-acute sequelae of COVID-19 (PASC). Nat. Immunol. 24 , 1616–1627 (2023).

Fernández-Castañeda, A. et al. Mild respiratory COVID can cause multi-lineage neural cell and myelin dysregulation. Cell 185 , 2452–2468 (2022).

Appelman, B. et al. Muscle abnormalities worsen after post-exertional malaise in Long COVID. Nat. Commun. 15 , 17 (2024).

Klein, J. et al. Distinguishing features of long COVID identified through immune profiling. Nature 623 , 139–148 (2023).

Cervia-Hasler, C. et al. Persistent complement dysregulation with signs of thromboinflammation in active Long COVID. Science 383 , eadg7942 (2024).

Eberhardt, N. et al. SARS-CoV-2 infection triggers pro-atherogenic inflammatory responses in human coronary vessels. Nat. Cardiovas. Res. 2 , 899–916 (2023).

Guarnieri, J. W. et al. Core mitochondrial genes are down-regulated during SARS-CoV-2 infection of rodent and human hosts. Sci. Transl. Med. 15 , eabq1533 (2023).

Ghafari, M. et al. Prevalence of persistent SARS-CoV-2 in a large community surveillance study. Nature 626 , 1094–1101 (2024).

Peluso, M. J. et al. Plasma-based antigen persistence in the post-acute phase of COVID-19. Lancet Infect. Dis. 24 , e345–e347 (2024).

Zuo, W. et al. The persistence of SARS-CoV-2 in tissues and its association with long COVID symptoms: a cross-sectional cohort study in China. Lancet Infect. Dis. https://doi.org/10.1016/S1473-3099(24)00171-3 (2024).

Menezes, S. M. et al. Blood transcriptomic analyses reveal persistent SARS-CoV-2 RNA and candidate biomarkers in post-COVID-19 condition. Lancet Microbe https://doi.org/10.1016/S2666-5247(24)00055-7 (2024).

Stein, S. R. et al. SARS-CoV-2 infection and persistence in the human body and brain at autopsy. Nature 612 , 758–763 (2022).

Martínez-Mármol, R. et al. SARS-CoV-2 infection and viral fusogens cause neuronal and glial fusion that compromises neuronal activity. Sci. Adv. 9 , eadg2248 (2023).

Petersen, M. et al. Brain imaging and neuropsychological assessment of individuals recovered from a mild to moderate SARS-CoV-2 infection. Proc. Natl Acad. Sci. USA 120 , e2217232120 (2023).

Monje, M. & Iwasaki, A. The neurobiology of long COVID. Neuron 110 , 3484–3496 (2022).

Douaud, G. et al. SARS-CoV-2 is associated with changes in brain structure in UK Biobank. Nature 604 , 697–707 (2022).

Greene, C. et al. Blood-brain barrier disruption and sustained systemic inflammation in individuals with long COVID-associated cognitive impairment. Nat. Neurosci. 27 , 421–432 (2024).

Su, Y. et al. Multiple early factors anticipate post-acute COVID-19 sequelae. Cell 185 , 881–895 (2022).

Chang, R. et al. Risk of autoimmune diseases in patients with COVID-19: a retrospective cohort study. EClinicalMedicine 56 , 101783 (2023).

Santos Guedes de Sa, K. et al. A causal link between autoantibodies and neurological symptoms in long COVID. Preprint at medRxiv https://doi.org/10.1101/2024.06.18.24309100 (2024).

Chen, H. -J. et al. Transfer of IgG from Long COVID patients induces symptomology in mice. Preprint at bioRxiv https://doi.org/10.1101/2024.05.30.596590 (2024).

Turner, S. et al. Long COVID: pathophysiological factors and abnormalities of coagulation. Trends Endocrinol. Metab. 34 , 321–344 (2023).

Wong, A. C. et al. Serotonin reduction in post-acute sequelae of viral infection. Cell 186 , 4851–4867 (2023).

Català, M. et al. The effectiveness of COVID-19 vaccines to prevent long COVID symptoms: staggered cohort study of data from the UK, Spain, and Estonia. Lancet Respir. Med. 12 , 225–236 (2024).

Trinh, N. T. et al. Effectiveness of COVID-19 vaccines to prevent long COVID: data from Norway. Lancet Respir. Med. 12 , e33–e34 (2024).

Lundberg-Morris, L. et al. COVID-19 vaccine effectiveness against post-COVID-19 condition among 589,722 individuals in Sweden: population based cohort study. Br. Med. J. 383 , e076990 (2023).

Marra, A. R. et al. The effectiveness of COVID-19 vaccine in the prevention of post-COVID conditions: a systematic literature review and meta-analysis of the latest research. Antimicrob. Steward Healthc. Epidemiol. 3 , e168 (2023).

Atchison, C. J. et al. Long-term health impacts of COVID-19 among 242,712 adults in England. Nat. Commun. 14 , 6588 (2023).

Yousaf, A. R. et al. COVID-19 mRNA vaccination reduces the occurrence of post-COVID conditions in U.S. children aged 5-17 years following omicron SARS-CoV-2 infection, July 2021-September 2022. Open Forum Infect. Dis. 10 , ofad500.2466 (2023).

Article   PubMed Central   Google Scholar  

Xie, Y., Bowe, B. & Al-Aly, Z. Molnupiravir and risk of hospital admission or death in adults with COVID-19: emulation of a randomized target trial using electronic health records. Br. Med. J. 380 , e072705 (2023).

Xie, Y., Bowe, B. & Al-Aly, Z. Nirmatrelvir and risk of hospital admission or death in adults with covid-19: emulation of a randomized target trial using electronic health records. Br. Med. J. 381 , e073312 (2023).

Gottlieb, R. L. et al. Early remdesivir to prevent progression to severe COVID-19 in outpatients. N. Engl. J. Med. 386 , 305–315 (2022).

Xie, Y., Choi, T. & Al-Aly, Z. Molnupiravir and risk of post-acute sequelae of COVID-19: cohort study. Br. Med. J. 381 , e074572 (2023).

Xie, Y., Choi, T. & Al-Aly, Z. Association of treatment with nirmatrelvir and the risk of post-COVID-19 condition. JAMA Intern. Med. 183 , 554–564 (2023).

Fung, K. W., Baye, F., Baik, S. H. & McDonald, C. J. Nirmatrelvir and molnupiravir and post–COVID-19 condition in older patients. JAMA Intern. Med. 183 , 1404–1406 (2023).

Hammond, J. et al. Nirmatrelvir for vaccinated or unvaccinated adult outpatients with COVID-19. N. Engl. J. Med. 390 , 1186–1195 (2024).

Jiang, X. et al. Structure-based development and preclinical evaluation of the SARS-CoV-2 3C-like protease inhibitor simnotrelvir. Nat. Commun. 14 , 6463 (2023).

Cao, B. et al. Oral simnotrelvir for adult patients with mild-to-moderate COVID-19. N. Engl. J. Med. 390 , 230–241 (2024).

Antar, A. A. R. & Peluso, M. J. CROI 2023: acute and post-acute COVID-19. Top. Antivir. Med 31 , 493–509 (2023).

PubMed   PubMed Central   Google Scholar  

Yotsuyanagi, H. et al. Prevention of post COVID-19 condition by early treatment with ensitrelvir in the phase 3 SCORPIO-SR trial. Antiviral Res. 229 , 105958 (2024).

Bramante, C. T. et al. Outpatient treatment of COVID-19 and incidence of post-COVID-19 condition over 10 months (COVID-OUT): a multicentre, randomised, quadruple-blind, parallel-group, phase 3 trial. Lancet Infect. Dis. 23 , 1119–1129 (2023).

Lau, R. I. et al. A synbiotic preparation (SIM01) for post-acute COVID-19 syndrome in Hong Kong (RECOVERY): a randomised, double-blind, placebo-controlled trial. Lancet Infect. Dis. 24 , 256–265 (2024).

Geng, L. N. et al. Nirmatrelvir-ritonavir and symptoms in adults with postacute sequelae of SARS-CoV-2 infection: the STOP-PASC randomized clinical trial. JAMA Intern. Med. 7 , e242007 (2024).

Maley, J. H. et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of breathing discomfort and respiratory sequelae in patients with post-acute sequelae of SARS-CoV-2 infection (PASC). PM R. 14 , 77–95 (2022).

Whiteson, J. H. et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of cardiovascular complications in patients with post-acute sequelae of SARS-CoV-2 infection (PASC). PM R. 14 , 855–878 (2022).

Herrera, J. E. et al. Response to letter to the editor regarding ‘multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of fatigue in patients with post-acute sequelae of SARS-CoV-2 infection (PASC)’. PM R. 13 , 1439–1440 (2021).

Fine, J. S. et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of cognitive symptoms in patients with post-acute sequelae of SARS-CoV-2 infection (PASC). PM R. 14 , 96–111 (2022).

Blitshteyn, S. et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of autonomic dysfunction in patients with post-acute sequelae of SARS-CoV-2 infection (PASC). PM R. 14 , 1270–1291 (2022).

Raj, S. R. et al. Canadian Cardiovascular Society Position Statement on postural orthostatic tachycardia syndrome (POTS) and related disorders of chronic orthostatic intolerance. Can. J. Cardiol. 36 , 357–372 (2020).

Espinosa-Gonzalez, A. B. et al. Orthostatic tachycardia after COVID-19. Br. Med. J. 380 , e073488 (2023).

Fedorowski, A. et al. Cardiovascular autonomic dysfunction in post-COVID-19 syndrome: a major health-care burden. Nat. Rev. Cardiol. 21 , 379–395 (2024).

Fedorowski, A. & Sutton, R. Autonomic dysfunction and postural orthostatic tachycardia syndrome in post-acute COVID-19 syndrome. Nat. Rev. Cardiol. 20 , 281–282 (2023).

Cheng, A. L. et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of mental health symptoms in patients with post-acute sequelae of SARS-CoV-2 infection (PASC). PM R. 15 , 1588–1604 (2023).

Melamed, E. et al. Multidisciplinary collaborative consensus guidance statement on the assessment and treatment of neurologic sequelae in patients with post-acute sequelae of SARS-CoV-2 infection (PASC). PM R. 15 , 640–662 (2023).

Malone, L. A. et al. Multi-disciplinary collaborative consensus guidance statement on the assessment and treatment of postacute sequelae of SARS-CoV-2 infection (PASC) in children and adolescents. PM R. 14 , 1241–1269 (2022).

Greenhalgh, T. & Knight, M. Long COVID: a primer for family physicians. Am. Fam. Physician 102 , 716–717 (2020).

PubMed   Google Scholar  

Greenhalgh, T., Sivan, M., Delaney, B., Evans, R. & Milne, R. Long COVID—an update for primary care. Br. Med. J. 378 , e072117 (2022).

Palacio, A., Bast, E., Klimas, N. & Tamariz, L. Lessons learned in implementing a multidisciplinary long COVID clinic. Am. J. Med. https://doi.org/10.1016/j.amjmed.2024.05.020 (2024).

Greenhalgh, T., Darbyshire, J. L., Lee, C., Ladds, E. & Ceolta-Smith, J. What is quality in long covid care? Lessons from a national quality improvement collaborative and multi-site ethnography. BMC Med. 22 , 159 (2024).

Sunkersing, D. et al. What is current care for people with Long COVID in England? A qualitative interview study. BMJ Open 14 , e080967 (2024).

Heightman, M. et al. Post-COVID-19 assessment in a specialist clinical service: a 12-month, single-centre, prospective study in 1325 individuals. BMJ Open Respir. Res. 8 , e001041 (2021).

Clutterbuck, D. et al. Barriers to healthcare access and experiences of stigma: Findings from a coproduced Long Covid case-finding study. Health Expect. 27 , e14037 (2024).

Routen, A. et al. Understanding and tracking the impact of long COVID in the United Kingdom. Nat. Med. 28 , 11–15 (2022).

Ledford, H. Long COVID is a double curse in low-income nations - here’s why. Nature 625 , 20–22 (2024).

Chou, R. et al. Long COVID definitions and models of care: a scoping review. Ann. Intern. Med. https://doi.org/10.7326/M24-0677 (2024).

Turk, F. et al. Pathways to care for Long COVID and for long-term conditions from patients’ and clinicians’ perspective. J. Evid. Based Med. 16 , 435–437 (2023).

Kwon, J. et al. Impact of Long COVID on productivity and informal caregiving. Eur. J. Health. Econ. https://doi.org/10.1007/s10198-023-01653-z (2023).

O’Mahony, L. et al. Impact of Long COVID on health and quality of life. HRB Open Res 5 , 31 (2022).

Karpman, M. et al. Employment and material hardship among adults with long COVID in December 2022. Urban Institute https://www.urban.org/sites/default/files/2023-07/Employment%20and%20Material%20Hardship%20among%20Adults%20with%20Long%20COVID%20in%20December%202022.pdf (July 2023).

McNabb, K. C. et al. ‘It was almost like it’s set up for people to fail’ A qualitative analysis of experiences and unmet supportive needs of people with Long COVID. BMC Public Health 23 , 2131 (2023).

Hossain, M. M. et al. Living with ‘long COVID’: a systematic review and meta-synthesis of qualitative evidence. PLoS ONE 18 , e0281884 (2023).

Au, L., Capotescu, C., Eyal, G. & Finestone, G. Long COVID and medical gaslighting: dismissal, delayed diagnosis, and deferred treatment. SSM Qual. Res. Health 2 , 100167 (2022).

Nalbandian, A. et al. Post-acute COVID-19 syndrome. Nat. Med. 27 , 601–615 (2021).

Al-Aly, Z., Agarwal, A., Alwan, N. & Luyckx, V. A. Long COVID: long-term health outcomes and implications for policy and research. Nat. Rev. Nephrol. 19 , 1–2 (2023).

Katz, G. M. et al. Understanding how post–COVID-19 condition affects adults and health care systems. JAMA Health Forum 4 , e231933 (2023).

Karpman, M., Zuckerman, S. & Morriss, S. Health care access and affordability among US adults aged 18 to 64 years with self-reported post-COVID-19 condition. JAMA Netw. Open 6 , e237455 (2023).

Jassat, W. et al. Long COVID in low-income and middle-income countries: the hidden public health crisis. Lancet 402 , 1115–1117 (2023).

Menges, D. et al. Burden of post-COVID-19 syndrome and implications for healthcare service planning: a population-based cohort study. PLoS ONE 16 , e0254523 (2021).

Greenhalgh, T., Knight, M., A’Court, C., Buxton, M. & Husain, L. Management of post-acute COVID-19 in primary care. Br. Med. J. 370 , m3026 (2020).

Bull-Otterson, L. et al. Post–COVID conditions among adult COVID-19 survivors aged 18–64 and ≥65 years—United States, March 2020–November 2021. MMWR Morb. Mortal. Wkly Rep. 71 , 713–717 (2022).

Article   CAS   PubMed Central   Google Scholar  

Barrett, C. E. et al. Risk for newly diagnosed diabetes >30 days after SARS-CoV-2 infection among persons aged <18 years—United States, March 2020–June 2021. MMWR Morb. Mortal. Wkly Rep . 71 , 59–65 (2022).

Zhang, T. et al. Risk for newly diagnosed diabetes after COVID-19: a systematic review and meta-analysis. BMC Med. 20 , 444 (2022).

Koyama, A. K. et al. Risk of cardiovascular disease after COVID-19 diagnosis among adults with and without diabetes. J. Am. Heart Assoc. 12 , e029696 (2023).

Wang, W., Wang, C. -Y., Wang, S. -I. & Wei, J. C. -C. Long-term cardiovascular outcomes in COVID-19 survivors among non-vaccinated population: a retrospective cohort study from the TriNetX US collaborative networks. EClinicalMedicine 53 , 101619 (2022).

Ayoubkhani, D. et al. Post-Covid syndrome in individuals admitted to hospital with COVID-19: retrospective cohort study. Br. Med. J. 372 , n693 (2021).

Lawson, C. A. et al. Long COVID and cardiovascular disease: a prospective cohort study. Open Heart 11 , e002662 (2024).

Roca-Fernandez, A. et al. Cardiac abnormalities in Long COVID 1-year post-SARS-CoV-2 infection. Open Heart 10 , e002241 (2023).

Dennis, A. et al. Multi-organ impairment and long COVID: a 1-year prospective, longitudinal cohort study. J. R. Soc. Med 116 , 97–112 (2023).

O’Mahoney, L. L. et al. The prevalence and long-term health effects of Long Covid among hospitalised and non-hospitalised populations: a systematic review and meta-analysis. EClinicalMedicine 55 , 101762 (2023).

Al-Aly, Z. & Cao, B. The enduring effect of the COVID-19 pandemic on diabetes. Lancet Diabetes Endocrinol. 12 , 508–510 (2024).

Rhead, R. et al. Long COVID and financial outcomes: evidence from four longitudinal population surveys. J. Epidemiol. Community Health 78 , 458–465 (2024).

Cutler, D. M. The costs of long COVID. JAMA Health Forum 3 , e221809 (2022).

Cutler, D. The economic cost of long COVID: an update. Harvard Kennedy School https://scholar.harvard.edu/sites/scholar.harvard.edu/files/cutler/files/long_covid_update_7-22.pdf (2022).

Reuschke, D. & Houston, D. The impact of Long COVID on the UK workforce. Appl. Econ. Lett. 30 , 2510–2514 (2023).

Gandjour, A. Long COVID: costs for the German economy and health care and pension system. BMC Health Serv. Res. 23 , 641 (2023).

Alexandri, E. et al. The economic burden of Long Covid in the UK. Cambridge Econometric s https://www.camecon.com/wp-content/uploads/2024/03/The-Economic-Burden-of-Long-Covid-in-the-UK_Cambridge-Econometrics_March2024.pdf (March 2024).

Fischer, K., Reade, J. J. & Schmal, W. B. What cannot be cured must be endured: the long-lasting effect of a COVID-19 infection on workplace productivity. Labour Econ. 79 , 102281 (2022).

Wolff Sagy, Y., Feldhamer, I., Brammli-Greenberg, S. & Lavie, G. Estimating the economic burden of long-COVID: the additive cost of healthcare utilisation among COVID-19 recoverees in Israel. BMJ Glob. Health 8 , e012588 (2023).

Ziauddeen, N., Pantelic, M., O’Hara, M. E., Hastie, C. & Alwan, N. A. Impact of long COVID-19 on work: a co-produced survey. Lancet 402 , S98 (2023).

Perlis, R. H. et al. Association of post–COVID-19 condition symptoms and employment status. JAMA Netw. Open 6 , e2256152 (2023).

Ayoubkhani, D. et al. Employment outcomes of people with Long Covid symptoms: community-based cohort study. Eur. J. Public Health 34 , 489–496 (2024).

Packard, S. E. & Susser, E. Association of long COVID with housing insecurity in the United States, 2022–2023. SSM Popul. Health 25 , 101586 (2024).

Bach, K. New data shows Long COVID is keeping as many as 4 million people out of work. Brookings https://www.brookings.edu/articles/new-data-shows-long-covid-is-keeping-as-many-as-4-million-people-out-of-work/ (24 August 2022).

Ham, D. Long-haulers and labor market outcomes. Federal Reserve Bank of Minneapolis https://www.minneapolisfed.org/research/institute-working-papers/long-haulers-and-labor-market-outcomes (7 July 2022).

Trades Union Congress. Workers’ experiences of Long Covid. A TUC report. TUC https://www.tuc.org.uk/research-analysis/reports/workers-experience-long-covid (27 March 2023).

Ramos, S. C., Maldonado, J. E., Vandeplas, A & Ványolós, I. Long COVID: a tentative assessment of its impact on labour market participation and potential economic effects in the EU. In ECONOMIC BRIEF 077 (European Commission Directorate-General for Economic and Financial Affairs, 2024).

Gonzalez, A. E. & Suzuki, E. The impacts of long COVID across OECD countries. OECD Health Working Papers 167 (OECD Publishing, 2024).

Baxa, M. An incomplete picture: understanding the burden of long COVID. The Economist Impact https://impact.economist.com/perspectives/sites/default/files/download/ei264_-_an_incomplete_picture_understanding_the_burden_of_long_covid_v8.pdf (29 April 2024).

Yuan, H. et al. Progress towards the Sustainable Development Goals has been slowed by indirect effects of the COVID-19 pandemic. Commun. Earth Environ. 4 , 184 (2023).

Luyckx, V. A. et al. Sustainable Development Goals relevant to kidney health: an update on progress. Nat. Rev. Nephrol. 17 , 15–32 (2021).

Azar, K. M. J. et al. Disparities in outcomes among COVID-19 patients in a large health care system in California. Health Aff. 39 , 1253–1262 (2020).

Martín-Blanco, C., Zamorano, M., Lizárraga, C. & Molina-Moreno, V. The impact of COVID-19 on the Sustainable Development Goals: achievements and expectations. Int. J. Environ. Res. Public Health 19 , 16266 (2022).

Shuai, C. et al. Quantifying the impacts of COVID-19 on Sustainable Development Goals using machine learning models. Fundam. Res. https://doi.org/10.1016/j.fmre.2022.06.016 (2022).

Luyckx, V. A. & Al-Aly, Z. Impact of COVID-19 on the Sustainable Development Agenda and Global Kidney Health. J. Am. Soc. Nephrol. 34 , 941–943 (2023).

Ameli, M., Shams Esfandabadi, Z., Sadeghi, S., Ranjbari, M. & Zanetti, M. C. COVID-19 and Sustainable Development Goals (SDGs): scenario analysis through fuzzy cognitive map modeling. Gondwana Res. 114 , 138–155 (2023).

Zhang, S., Anser, M. K., Yao-Ping Peng, M. & Chen, C. Visualizing the sustainable development goals and natural resource utilization for green economic recovery after COVID-19 pandemic. Resour. Policy 80 , 103182 (2023).

Zhao, W. et al. Achieving the Sustainable Development Goals in the post-pandemic era. Humanit. Soc. Sci. Commun. 9 , 258 (2022).

Lukkahatai, N., Rodney, T., Ling, C., Daniel, B. & Han, H. R. Long COVID in the context of social determinants of health. Front. Public Health 11 , 1098443 (2023).

Berger, Z., Altiery, D. E. J. V., Assoumou, S. A. & Greenhalgh, T. Long COVID and health inequities: the role of primary care. Milbank Q 99 , 519–541 (2021).

Perry, B. L., Aronson, B. & Pescosolido, B. A. Pandemic precarity: COVID-19 is exposing and exacerbating inequalities in the American heartland. Proc. Natl Acad. Sci. USA 118 , e2020685118 (2021).

Devoto, S. A. Long COVID and chronic pain: overlapping racial inequalities. Disabil. Soc. 38 , 524–529 (2023).

Al-Aly, Z. Prevention of long COVID: progress and challenges. Lancet Infect. Dis. 23 , 776–777 (2023).

Au, L., Capotescu, C., Curi, A., Gonçalves Leonel da Silva, R. & Eyal, G. Long COVID requires a global response centred on equity and dialogue. Glob. Health Action 16 , 2244757 (2023).

McCorkell, L., Assaf, G, S., Davis, H, E., Wei, H. & Akrami, A. Patient-led research collaborative: embedding patients in the Long COVID narrative. Pain Rep. 6 , e913 (2021).

Alwan, N. A. The road to addressing Long Covid. Science 373 , 491–493 (2021).

Couzin-Frankel, J. & Vogel, G. Vaccines may cause rare, Long Covid-like symptoms. Science 375 , 364–366 (2022).

Hofer, U. Animal model of long COVID? Nat. Rev. Microbiol. 20 , 446 (2022).

Silva, J. et al. Sex differences in symptomatology and immune profiles of Long COVID. Preprint at medRxiv https://doi.org/10.1101/2024.02.29.24303568 (2024).

Lammi, V. et al. Genome-wide association study of Long COVID. Preprint at medRxiv https://doi.org/10.1101/2023.06.29.23292056 (2023).

Lee, Y. et al. EWAS of post-COVID-19 patients shows methylation differences in the immune-response associated gene, IFI44L , three months after COVID-19 infection. Sci. Rep. 12 , 11478 (2022).

Ozturkler, Z. & Kalkan, R. A new perspective of COVID-19 infection: an epigenetics point of view. Glob. Med. Genet. 9 , 4–6 (2022).

Wang, R. et al. SARS-CoV-2 restructures host chromatin architecture. Nat. Microbiol. 8 , 679–694 (2023).

Gatmaitan, B. G., Chason, J. L. & Lerner, A. M. Augmentation of the virulence of murine coxsackie-virus B-3 myocardiopathy by exercise. J. Exp. Med. 131 , 1121–1136 (1970).

van Rhijn-Brouwer, F. C. C. et al. Graded exercise therapy should not be recommended for patients with post-exertional malaise. Nat. Rev. Cardiol. 21 , 430–431 (2024).

Xie, Y., Choi, T. & Al-Aly, Z. Long-term outcomes following hospital admission for COVID-19 versus seasonal influenza: a cohort study. Lancet Infect. Dis. 24 , 239–255 (2024).

Spinney, L. Pandemics disable people—the history lesson that policymakers ignore. Nature 602 , 383–385 (2022).

Spinney, L. Pale Rider: The Spanish Flu of 1918 and How it Changed the World (PublicAffairs, 2017).

Bjornevik, K. et al. Longitudinal analysis reveals high prevalence of Epstein–Barr virus associated with multiple sclerosis. Science 375 , 296–301 (2022).

Stefano, G. B. Historical insight into infections and disorders associated with neurological and psychiatric sequelae similar to Long COVID. Med. Sci. Monit. 27 , e931447 (2021).

Vahratian A., Lin, J. -M. S., Bertolli J. & Unger E. R. Myalgic encephalomyelitis/chronic fatigue syndrome in adults: United States, 2021–2022. NCHS Data Brief 488 , 1–6 (2023).

Komaroff, A. L. Advances in understanding the pathophysiology of chronic fatigue syndrome. J. Am. Med. Assoc. 322 , 499–500 (2019).

Paul, B. D., Lemle, M. D., Komaroff, A. L. & Snyder, S. H. Redox imbalance links COVID-19 and myalgic encephalomyelitis/chronic fatigue syndrome. Proc. Natl Acad. Sci. USA 118 , e2024358118 (2021).

Steiner, S. et al. Understanding, diagnosing, and treating myalgic encephalomyelitis/chronic fatigue syndrome - state of the art: Report of the 2nd International Meeting at the Charite Fatigue Center. Autoimmun. Rev. 22 , 103452 (2023).

Arron, H. E. et al. Myalgic encephalomyelitis/chronic fatigue syndrome: the biology of a neglected disease. Front. Immunol. 15 , 1386607 (2024).

Goldenberg, D. L. How to understand the overlap of long COVID, chronic fatigue syndrome/myalgic encephalomyelitis, fibromyalgia and irritable bowel syndromes. Semin. Arthritis Rheum. 67 , 152455 (2024).

Arun, S., Storan, A. & Myers, B. Mast cell activation syndrome and the link with long COVID. Br. J. Hosp. Med. 83 , 1–10 (2022).

Salvucci, F. et al. Antihistamines improve cardiovascular manifestations and other symptoms of long-COVID attributed to mast cell activation. Front. Cardiovasc. Med. 10 , 1202696 (2023).

Sumantri, S. & Rengganis, I. Immunological dysfunction and mast cell activation syndrome in long COVID. Asia Pac. Allergy 13 , 50–53 (2023).

Weinstock, L. B. et al. Mast cell activation symptoms are prevalent in Long-COVID. Int. J. Infect. Dis. 112 , 217–226 (2021).

Chadda, K. R., Blakey, E. E., Huang, C. L. & Jeevaratnam, K. Long COVID-19 and postural orthostatic tachycardia syndrome—is dysautonomia to be blamed? Front. Cardiovasc. Med. 9 , 860198 (2022).

Finsterer, J. Small fiber neuropathy underlying dysautonomia in COVID-19 and in post-SARS-CoV-2 vaccination and long-COVID syndromes. Muscle Nerve 65 , E31–E32 (2022).

Margalit, I. & Yahav, D. The potential role of vagus nerve dysfunction and dysautonomia in long COVID. Clin. Microbiol. Infect. 30 , 423–427 (2024).

Grist, J. T. et al. Lung abnormalities detected with hyperpolarized 129Xe MRI in patients with long COVID. Radiology 305 , 709–717 (2022).

Garcia-Hernandez, R. et al. Mapping microglia and astrocyte activation in vivo using diffusion MRI. Sci. Adv. 8 , eabq2923 (2022).

Jacob, L. et al. Conserved meningeal lymphatic drainage circuits in mice and humans. J. Exp. Med. 219 , e20220035 (2022).

Ajčević, M. et al. Cerebral hypoperfusion in post-COVID-19 cognitively impaired subjects revealed by arterial spin labeling MRI. Sci. Rep. 13 , 5808 (2023).

Turner, S., Laubscher, G. J., Khan, M. A., Kell, D. B. & Pretorius, E. Accelerating discovery: a novel flow cytometric method for detecting fibrin(ogen) amyloid microclots using long COVID as a model. Heliyon 9 , e19605 (2023).

Peluso, M. J. et al. Tissue-based T cell activation and viral RNA persist for up to 2 years after SARS-CoV-2 infection. Sci. Transl. Med. 16 , eadk3295 (2024).

Espín, E. et al. Cellular and molecular biomarkers of long COVID: a scoping review. eBioMedicine 91 , 104552 (2023).

Lai, Y. J. et al. Biomarkers in long COVID-19: a systematic review. Front. Med. 10 , 1085988 (2023).

Su, Y. et al. Multi-omics resolves a sharp disease-state shift between mild and moderate COVID-19. Cell 183 , 1479–1495 (2020).

Li, H. et al. Plasma proteomic and metabolomic characterization of COVID-19 survivors 6 months after discharge. Cell Death Dis. 13 , 235 (2022).

Rusu, E. C. et al. Towards understanding post-COVID-19 condition: a systematic meta-analysis of transcriptomic alterations with sex-specific insights. Comput. Biol. Med. 175 , 108507 (2024).

Gu, X. et al. Probing long COVID through a proteomic lens: a comprehensive two-year longitudinal cohort study of hospitalised survivors. EBioMedicine 98 , 104851 (2023).

Iosef, C. et al. Plasma proteome of Long-COVID patients indicates HIF-mediated vasculo-proliferative disease with impact on brain and heart function. J. Transl. Med. 21 , 377 (2023).

Liang, X., Wang, Y. & Guo, T. Proteomics approaches to long COVID: status and outlooks. Life Med. 2 , 3 (2023).

Saito, S. et al. Metabolomic and immune alterations in long COVID patients with chronic fatigue syndrome. Front. Immunol. 15 , 1341843 (2024).

López-Hernández, Y. et al. Untargeted analysis in post-COVID-19 patients reveals dysregulated lipid pathways two years after recovery. Front. Mol. Biosci. 10 , 1100486 (2023).

Li, Z. et al. The causal role of gut microbiota in susceptibility of Long COVID: a Mendelian randomization study. Front. Microbiol. 15 , 1404673 (2024).

Huang, C. et al. 6-month consequences of COVID-19 in patients discharged from hospital: a cohort study. Lancet 397 , 220–232 (2021).

Al-Aly, Z. Long COVID and its cardiovascular implications: a call to action. Eur. Heart J. 44 , 5001–5003 (2023).

Thaweethai, T. et al. Development of a definition of postacute sequelae of SARS-CoV-2 infection. J. Am. Med. Assoc. 329 , 1934–1946 (2023).

Article   CAS   Google Scholar  

Reese, J. T. et al. Generalisable long COVID subtypes: findings from the NIH N3C and RECOVER programmes. eBioMedicine 87 , 104413 (2023).

Zhang, H. et al. Data-driven identification of post-acute SARS-CoV-2 infection subphenotypes. Nat. Med. 29 , 226–235 (2023).

Liew, F. et al. Large-scale phenotyping of patients with long COVID post-hospitalization reveals mechanistic subtypes of disease. Nat. Immunol. 25 , 607–621 (2024).

Matthews, A. A., Danaei, G., Islam, N. & Kurth, T. Target trial emulation: applying principles of randomised trials to observational studies. Br. Med. J. 378 , e071108 (2022).

Hernán, M. A., Wang, W. & Leaf, D. E. Target trial emulation: a framework for causal inference from observational data. J. Am. Med. Assoc. 328 , 2446–2447 (2022).

Xie, Y., Bowe, B., Maddukuri, G. & Al-Aly, Z. Comparative evaluation of clinical manifestations and risk of death in patients admitted to hospital with COVID-19 and seasonal influenza: cohort study. Br. Med. J. 371 , m4677 (2020).

Xie, Y., Choi, T. & Al-Aly, Z. Risk of death in patients hospitalized for COVID-19 vs seasonal influenza in Fall-Winter 2022–2023. J. Am. Med. Assoc. 329 , 1697–1699 (2023).

Xie, Y., Choi, T. & Al-Aly, Z. Mortality in patients hospitalized for COVID-19 vs influenza in Fall-Winter 2023–2024. J. Am. Med. Assoc. 331 , 1963–1965 (2024).

The Lancet Infectious Diseases. Where are the long COVID trials? Lancet Infect. Dis. 23 , 879 (2023).

Fairbank, R. Long COVID still has no cure—so these patients are turning to research. Nature 628 , 26–28 (2024).

Alwan, N. A. Lessons from Long COVID: working with patients to design better research. Nat. Rev. Immunol. 22 , 201–202 (2022).

Loretelli, C. et al. PD-1 blockade counteracts post-COVID-19 immune abnormalities and stimulates the anti-SARS-CoV-2 immune response. JCI Insight 6 , e146701 (2021).

Service, R. F. Bad news for Paxlovid? Resistance may be coming. Science 377 , 138–139 (2022).

Iketani, S. et al. Multiple pathways for SARS-CoV-2 resistance to nirmatrelvir. Nature 613 , 558–564 (2023).

Duan, Y. et al. Molecular mechanisms of SARS-CoV-2 resistance to nirmatrelvir. Nature 622 , 376–382 (2023).

Buchholz, S., Di Meco, E., Balkowiec-Iskra, E. Z., Sepodes, B. & Cavaleri, M. Generating clinical evidence for treatment and prevention options for long COVID. Nat. Med. https://doi.org/10.1038/s41591-024-03031-5 (2024).

Mohamed, M. O. & Banerjee, A. Long COVID and cardiovascular disease: a learning health system approach. Nat. Rev. Cardiol. 19 , 287–288 (2022).

Alwan, N. A. et al. Long Covid active case finding study protocol: a co-produced community-based pilot within the STIMULATE-ICP study (symptoms, trajectory, inequalities and management: understanding Long-COVID to address and transform existing integrated care pathways). PLoS ONE 18 , e0284297 (2023).

Black, R. E. et al. Health and development from preconception to 20 years of age and human capital. Lancet 399 , 1730–1740 (2022).

Jin, S. L. et al. Social histories of public health misinformation and infodemics: case studies of four pandemics. Lancet Infect. Dis. https://doi.org/10.1016/S1473-3099(24)00105-1 (2024).

Ravenholt, R. T. & Foege, W. 1918 influenza, encephalitis lethargica, parkinsonism. Lancet 320 , 860–864 (1982).

Honigsbaum, M. & Krishnan, L. Taking pandemic sequelae seriously: from the Russian influenza to COVID-19 long-haulers. Lancet 396 , 1389–1391 (2020).

Dehner, G. Howard Phillips, in a time of plague: memories of the ‘Spanish’ flu epidemic of 1918 in South Africa. Soc. Hist. Med. 33 , 343–344 (2019).

Greenhalgh, T. et al. Masks and respirators for prevention of respiratory infections: a state of the science review. Clin. Microbiol Rev. 37 , e0012423 (2024).

Etowa, J. et al. Understanding low vaccine uptake in the context of public health in high-income countries: a scoping review. Vaccines 12 , 269 (2024).

World Health Organization. Indoor airborne risk assessment in the context of SARS-CoV-2: description of airborne transmission mechanism and method to develop a new standardized model for risk assessment. https://iris.who.int/handle/10665/376346 (World Health Organization, 2024).

Morawska, L. et al. Mandating indoor air quality for public buildings. Science 383 , 1418–1420 (2024).

Lewis, D. Indoor air is full of flu and COVID viruses. Will countries clean it up? Nature 615 , 206–208 (2023).

Lunderberg, D. M. et al. Assessing residential PM 2.5 concentrations and infiltration factors with high spatiotemporal resolution using crowdsourced sensors. Proc. Natl Acad. Sci. USA 120 , e2308832120 (2023).

Grippo, A. et al. Indoor air pollution exposure and early childhood development in the Upstate KIDS Study. Environ. Res. 234 , 116528 (2023).

Lee, K. K. et al. Adverse health effects associated with household air pollution: a systematic review, meta-analysis, and burden estimation study. Lancet Glob. Health 8 , e1427–e1434 (2020).

Pulimeno, M., Piscitelli, P., Colazzo, S., Colao, A. & Miani, A. Indoor air quality at school and students’ performance: Recommendations of the UNESCO Chair on Health Education and Sustainable Development & the Italian Society of Environmental Medicine (SIMA). Health Promot. Perspect. 10 , 169–174 (2020).

Topol, E. J. & Iwasaki, A. Operation nasal vaccine—lightning speed to counter COVID-19. Sci. Immunol. 7 , eadd9947 (2022).

Nunn, A. The Politics and History of AIDS Treatment in Brazil (Springer, 2008).

Grangeiro, A. et al. Forty years of the Brazilian response to HIV: reflections on the need for a programmatic shift and policy as a common good. Cad. Saude Publica 39 , e00199423 (2023).

PubMed Central   Google Scholar  

Cheever, L. W., Rich, J. & Adashi, E. Y. The Ryan White HIV/AIDS Program: thirty years of addressing health disparities. J. Health Care Poor Underserved 35 , 726–730 (2024).

Goyal, R. et al. The health equity implications of the Health Resources and Services Administration’s Ryan White HIV/AIDS Program. AIDS 38 , 1025–1032 (2024).

Patel-Larson, A. et al. Looking back to see forward: multidirectional learning between the US Ryan White HIV/AIDS Program and the US President’s Emergency Plan for AIDS Relief. BMJ Glob. Health 8 , e013953 (2024).

de Beaumont Foundation. Poll: Physicians agree Long COVID is a problem but are unprepared to treat it. https://debeaumont.org/wp-content/uploads/2023/03/Long-COVID-Brief.pdf (2023).

National Center for Health Statistics Rapid Surveys System. Long COVID. https://www.cdc.gov/nchs/rss/round1/long-covid.html (accessed 1 July 2024).

Al-Aly, Z. Addressing Long COVID: Advancing Research and Improving Patient Care. US Senate Committee on Health, Education, Labor and Pensions https://www.help.senate.gov/download/al-aly-testimony?download=1 (18 January 2024).

Conroy, M. et al. The advantages of UK Biobank’s open-access strategy for health research. J. Intern. Med. 286 , 389–397 (2019).

Helzlsouer, K. J. & Reedy, J. Data sharing for the public good. J. Natl Cancer Inst. 112 , 867–868 (2020).

Soriano, J. B., Murthy, S., Marshall, J. C., Relan, P. & Diaz, J. V. A clinical case definition of post-COVID-19 condition by a Delphi consensus. Lancet Infect. Dis. 22 , e102–e107 (2022).

US Centers for Disease Control. Long COVID or post-COVID conditions. https://www.cdc.gov/coronavirus/2019-ncov/long-term-effects/ (accessed 1 July 2024).

National Institute for Health and Care Excellence. COVID-19 rapid guideline: managing the long-term effects of COVID-19. NICE https://www.nice.org.uk/guidance/ng188 (accessed 1 July 2024).

US Department of Health and Human Services, Office of the Assistant Secretary for Health. National Research Action Plan on Long COVID. covid.gov https://www.covid.gov/sites/default/files/documents/National-Research-Action-Plan-on-Long-COVID-08012022.pdf (August 2022).

Appel, K. S. et al. Definition of the Post-COVID syndrome using a symptom-based Post-COVID score in a prospective, multi-center, cross-sectoral cohort of the German National Pandemic Cohort Network (NAPKON). Infection https://doi.org/10.1007/s15010-024-02226-9 (2024).

Alwan, N. A. & Johnson, L. Defining long COVID: going back to the start. Med 2 , 501–504 (2021).

Sivan, M. & Heightman, M. A new professional society for post-COVID condition and other post-viral conditions. Adv. Rehabil. Sci. Pract . 13 , https://doi.org/10.1177/27536351241231351 (2024).

National Academies of Sciences, Engineering, and Medicine; Health and Medicine Division; Board on Global Health; Forum on Microbial Threats. Toward a Post-Pandemic World: Lessons from COVID-19 for Now and the Future: Proceedings of a Workshop (National Academies Press, 2022).

To respond to the threat of avian influenza, look back at lessons learned from COVID-19. Nat. Med . 30 , 1507 (2024).

Sachs, J. D. et al. The Lancet Commission on lessons for the future from the COVID-19 pandemic. Lancet 400 , 1224–1280 (2022).

Baker, R. E. et al. Infectious disease in an era of global change. Nat. Rev. Microbiol. 20 , 193–205 (2022).

de Oliveira, T. & Tegally, H. Will climate change amplify epidemics and give rise to pandemics? Science 381 , eadk4500 (2023).

Marani, M., Katul, G. G., Pan, W. K. & Parolari, A. J. Intensity and frequency of extreme novel epidemics. Proc. Natl Acad. Sci. USA 118 , e2105482118 (2021).

Carlson, C. J. et al. Climate change increases cross-species viral transmission risk. Nature 607 , 555–562 (2022).

Neumann, G. & Kawaoka, Y. Which virus will cause the next pandemic? Viruses 15 , 199 (2023).

Vastag, B. & Mazur, B. Researchers warn covid-19 could cause debilitating long-term illness in some patients. The Washington Post https://www.washingtonpost.com/health/could-covid-19-cause-long-term-chronic-fatigue-and-illness-in-some-patients/2020/05/29/bcd5edb2-a02c-11ea-b5c9-570a91917d8d_story.html (30 May 2020).

Lowenstein, F. We need to talk about what coronavirus recoveries look like. The New York Times https://www.nytimes.com/2020/04/13/opinion/coronavirus-recovery.html (13 April 2020).

Callard, F. & Perego, E. How and why patients made Long COVID. Soc. Sci. Med 268 , 113426 (2021).

Young, E. COVID-19 can last for several months. The Atlantic https://www.theatlantic.com/health/archive/2020/06/covid-19-coronavirus-longterm-symptoms-months/612679/ (4 June 2020).

Nemer, M. Post-COVID-19 condition in Canada: what we know, what we don’t know and a framework for action. Office of the Chief Science Advisor of Canada https://science.gc.ca/site/science/sites/default/files/attachments/2023/Post-Covid-Condition_Report-2022.pdf (2022).

Duddy, C. et al. Coronavirus: long COVID. House of Commons Library https://researchbriefings.files.parliament.uk/documents/CBP-9112/CBP-9112.pdf (29 February 2024).

European Commission. Network of expertise on Long COVID. https://health.ec.europa.eu/non-communicable-diseases/expert-group-public-health/network-expertise-long-covid_en (accessed 26 June 2024).

Parliament of Australia. Sick and tired: casting a long shadow. https://www.aph.gov.au/Parliamentary_Business/Committees/House/Health_Aged_Care_and_Sport/LongandrepeatedCOVID/Report (April, 2023).

US Senate Committee on Health, Education, Labor, and Pensions. Chairman Bernie Sanders releases Long COVID moonshot legislative proposal. https://www.help.senate.gov/chair/newsroom/press/news-chairman-bernie-sanders-releases-long-covid-moonshot-legislative-proposal (9 April 2024).

Download references

Author information

Authors and affiliations.

VA St. Louis Health Care System, Saint Louis, MO, USA

Ziyad Al-Aly

Washington University in St. Louis, Saint Louis, MO, USA

Patient-led Research Collaborative, Calabasas, CA, USA

Hannah Davis, Lisa McCorkell & Letícia Soares

University of Washington, Seattle, WA, USA

Sarah Wulf-Hanson

Yale University, New Haven, CT, USA

Akiko Iwasaki

Howard Hughes Medical Institute, Chevy Chase, MD, USA

Scripps Institute, San Diego, California, CA, USA

Eric J. Topol

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Ziyad Al-Aly .

Ethics declarations

Competing interests.

Z.A.-A. reports receiving consultation fees from Pfizer. L.M. reports consulting fees from Evidera. L.S. has received honoraria from consulting for Evidera. A.I. co-founded RIGImmune, Xanadu Bio and PanV and is a member of the board of directors of Roche Holding and Genentech.

Peer review

Peer review information.

Nature Medicine thanks Trisha Greenhalgh, Helen Ward and the other, anonymous, reviewer(s) for their contribution to the peer review of this work. Primary Handling Editor: Karen O’Leary, in collaboration with the Nature Medicine team.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Reprints and permissions

About this article

Cite this article.

Al-Aly, Z., Davis, H., McCorkell, L. et al. Long COVID science, research and policy. Nat Med (2024). https://doi.org/10.1038/s41591-024-03173-6

Download citation

Received : 16 April 2024

Accepted : 02 July 2024

Published : 09 August 2024

DOI : https://doi.org/10.1038/s41591-024-03173-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.