free website for research articles

Open and free content on JSTOR and Artstor

Our partnerships with libraries and publishers help us make content discoverable and freely accessible worldwide

Search open content on JSTOR

Explore our growing collection of Open Access journals

Early Journal Content , articles published prior to the last 95 years in the United States, or prior to the last 143 years if initially published internationally, are freely available to all

Even more content is available when you register to read – millions of articles from nearly 2,000 journals

Thousands of Open Access ebooks are available from top scholarly publishers, including Brill, Cornell University Press, University College of London, and University of California Press – at no cost to libraries or users.

This includes Open Access titles in Spanish:

• Collaboration with El Colegio de México
• Partnership with the Latin American Council of Social Sciences

Images and media

JSTOR hosts a growing number of public collections , including Artstor’s Open Access collections , from museums, archives, libraries, and scholars worldwide.

Research reports

A curated set of more than 34,000 research reports from more than 140 policy institutes selected with faculty, librarian, and expert input.

Resources for librarians

Open content title lists:

• Open Access Journals (xlsx)
• Open Access Books (xlsx)
• JSTOR Early Journal Content (txt)
• Research Reports

Open Access ebook resources for librarians

Library-supported collections

Shared Collections : We have a growing corpus of digital special collections published on JSTOR by our institutional partners.

Reveal Digital : A collaboration with libraries to fund, source, digitize and publish open access primary source collections from under-represented voices.

JSTOR Daily

JSTOR Daily is an online publication that contextualizes current events with scholarship. All of our stories contain links to publicly accessible research on JSTOR. We’re proud to publish articles based in fact and grounded by careful research and to provide free access to that research for all of our readers.

Explore millions of high-quality primary sources and images from around the world, including artworks, maps, photographs, and more.

Explore migration issues through a variety of media types

• Part of The Streets are Talking: Public Forms of Creative Expression from Around the World
• Part of The Journal of Economic Perspectives, Vol. 34, No. 1 (Winter 2020)
• Part of Cato Institute (Aug. 3, 2021)
• Part of University of California Press
• Part of Open: Smithsonian National Museum of African American History & Culture
• Part of Indiana Journal of Global Legal Studies, Vol. 19, No. 1 (Winter 2012)
• Part of R Street Institute (Nov. 1, 2020)
• Part of Leuven University Press
• Part of UN Secretary-General Papers: Ban Ki-moon (2007-2016)
• Part of Perspectives on Terrorism, Vol. 12, No. 4 (August 2018)
• Part of Leveraging Lives: Serbia and Illegal Tunisian Migration to Europe, Carnegie Endowment for International Peace (Mar. 1, 2023)
• Part of UCL Press

Harness the power of visual materials—explore more than 3 million images now on JSTOR.

Enhance your scholarly research with underground newspapers, magazines, and journals.

Explore collections in the arts, sciences, and literature from the world’s leading museums, archives, and scholars.

21 Legit Research Databases for Free Journal Articles in 2024

#scribendiinc

Written by  Scribendi

Has this ever happened to you? While looking for websites for research, you come across a research paper site that claims to connect academics to a peer-reviewed article database for free.

Intrigued, you search for keywords related to your topic, only to discover that you must pay a hefty subscription fee to access the service. After the umpteenth time being duped, you begin to wonder if there's even such a thing as free journal articles.

Subscription fees and paywalls are often the bane of students and academics, especially those at small institutions who don't provide access to many free article directories and repositories.

Whether you're working on an undergraduate paper, a PhD dissertation, or a medical research study, we want to help you find tools to locate and access the information you need to produce well-researched, compelling, and innovative work.

Below, we discuss why peer-reviewed articles are superior and list out the best free article databases to use in 2024.

Download Our Free Research Database Roundup PDF

Why peer-reviewed scholarly journal articles are more authoritative.

Determining what sources are reliable can be challenging. Peer-reviewed scholarly journal articles are the gold standard in academic research. Reputable academic journals have a rigorous peer-review process.

The peer review process provides accountability to the academic community, as well as to the content of the article. The peer review process involves qualified experts in a specific (often very specific) field performing a review of an article's methods and findings to determine things like quality and credibility.

Peer-reviewed articles can be found in peer-reviewed article databases and research databases, and if you know that a database of journals is reliable, that can offer reassurances about the reliability of a free article. Peer review is often double blind, meaning that the author removes all identifying information and, likewise, does not know the identity of the reviewers. This helps reviewers maintain objectivity and impartiality so as to judge an article based on its merit.

Where to Find Peer-Reviewed Articles

Peer-reviewed articles can be found in a variety of research databases. Below is a list of some of the major databases you can use to find peer-reviewed articles and other sources in disciplines spanning the humanities, sciences, and social sciences.

What Are Open Access Journals?

An open access (OA) journal is a journal whose content can be accessed without payment. This provides scholars, students, and researchers with free journal articles. OA journals use alternate methods of funding to cover publication costs so that articles can be published without having to pass those publication costs on to the reader.

Some of these funding models include standard funding methods like advertising, public funding, and author payment models, where the author pays a fee in order to publish in the journal. There are OA journals that have non-peer-reviewed academic content, as well as journals that focus on dissertations, theses, and papers from conferences, but the main focus of OA is peer-reviewed scholarly journal articles.

The internet has certainly made it easier to access research articles and other scholarly publications without needing access to a university library, and OA takes another step in that direction by removing financial barriers to academic content.

Choosing Wisely

Features of legitimate oa journals.

 There are things to look out for when trying to decide if a free publication journal is legitimate:

Mission statement —The mission statement for an OA journal should be available on their website.

Publication history —Is the journal well established? How long has it been available?

Editorial board —Who are the members of the editorial board, and what are their credentials?

Indexing —Can the journal be found in a reliable database?

Peer review —What is the peer review process? Does the journal allow enough time in the process for a reliable assessment of quality?

Impact factor —What is the average number of times the journal is cited over a two-year period?

Features of Illegitimate OA Journals

There are predatory publications that take advantage of the OA format, and they are something to be wary of. Here are some things to look out for:

Contact information —Is contact information provided? Can it be verified?

Turnaround —If the journal makes dubious claims about the amount of time from submission to publication, it is likely unreliable.

Editorial board —Much like determining legitimacy, looking at the editorial board and their credentials can help determine illegitimacy.

Indexing —Can the journal be found in any scholarly databases?

Peer review —Is there a statement about the peer review process? Does it fit what you know about peer review?

How to Find Scholarly Articles

Identify keywords.

Keywords are included in an article by the author. Keywords are an excellent way to find content relevant to your research topic or area of interest. In academic searches, much like you would on a search engine, you can use keywords to navigate through what is available to find exactly what you're looking for.

Authors provide keywords that will help you easily find their article when researching a related topic, often including general terms to accommodate broader searches, as well as some more specific terms for those with a narrower scope. Keywords can be used individually or in combination to refine your scholarly article search.

Narrow Down Results

Sometimes, search results can be overwhelming, and searching for free articles on a journal database is no exception, but there are multiple ways to narrow down your results. A good place to start is discipline.

What category does your topic fall into (psychology, architecture, machine learning, etc.)? You can also narrow down your search with a year range if you're looking for articles that are more recent.

A Boolean search can be incredibly helpful. This entails including terms like AND between two keywords in your search if you need both keywords to be in your results (or, if you are looking to exclude certain keywords, to exclude these words from the results).

Consider Different Avenues

If you're not having luck using keywords in your search for free articles, you may still be able to find what you're looking for by changing your tactics. Casting a wider net sometimes yields positive results, so it may be helpful to try searching by subject if keywords aren't getting you anywhere.

You can search for a specific publisher to see if they have OA publications in the academic journal database. And, if you know more precisely what you're looking for, you can search for the title of the article or the author's name.

Determining the Credibility of Scholarly Sources

Ensuring that sources are both credible and reliable is crucial to academic research. Use these strategies to help evaluate the usefulness of scholarly sources:

• Peer Review : Look for articles that have undergone a rigorous peer-review process. Peer-reviewed articles are typically vetted by experts in the field, ensuring the accuracy of the research findings.

Tip: To determine whether an article has undergone rigorous peer review, review the journal's editorial policies, which are often available on the journal's website. Look for information about the peer-review process, including the criteria for selecting reviewers, the process for handling conflicts of interest, and any transparency measures in place.

• Publisher Reputation : Consider the reputation of the publisher. Established publishers, such as well-known academic journals, are more likely to adhere to high editorial standards and publishing ethics.
• Author Credentials : Evaluate the credentials and expertise of the authors. Check their affiliations, academic credentials, and past publications to assess their authority in the field.
• Citations and References : Examine the citations and references provided in the article. A well-researched article will cite credible sources to support its arguments and findings. Verify the accuracy of the cited sources and ensure they are from reputable sources.
• Publication Date : Consider the publication date of the article. While older articles may still be relevant, particularly in certain fields, it is best to prioritize recent publications for up-to-date research and findings.
• Journal Impact Factor : Assess the journal's impact factor or other metrics that indicate its influence and reputation within the academic community. Higher impact factor journals are generally considered more prestigious and reliable. 

Tip: Journal Citation Reports (JCR), produced by Clarivate Analytics, is a widely used source for impact factor data. You can access JCR through academic libraries or directly from the Clarivate Analytics website if you have a subscription.

• Peer Recommendations : Seek recommendations from peers, mentors, or professors in your field. They can provide valuable insights and guidance on reputable sources and journals within your area of study.
• Cross-Verification : Cross-verify the information presented in the article with other credible sources. Compare findings, methodologies, and conclusions with similar studies to ensure consistency and reliability.

By employing these strategies, researchers can confidently evaluate the credibility and reliability of scholarly sources, ensuring the integrity of their research contributions in an ever-evolving landscape.

The Top 21 Free Online Journal and Research Databases

Navigating OA journals, research article databases, and academic websites trying to find high-quality sources for your research can really make your head spin. What constitutes a reliable database? What is a useful resource for your discipline and research topic? How can you find and access full-text, peer-reviewed articles?

Fortunately, we're here to help. Having covered some of the ins and outs of peer review, OA journals, and how to search for articles, we have compiled a list of the top 21 free online journals and the best research databases. This list of databases is a great resource to help you navigate the wide world of academic research.

These databases provide a variety of free sources, from abstracts and citations to full-text, peer-reviewed OA journals. With databases covering specific areas of research and interdisciplinary databases that provide a variety of material, these are some of our favorite free databases, and they're totally legit!

CORE is a multidisciplinary aggregator of OA research. CORE has the largest collection of OA articles available. It allows users to search more than 219 million OA articles. While most of these link to the full-text article on the original publisher's site, or to a PDF available for download, five million records are hosted directly on CORE.

CORE's mission statement is a simple and straightforward commitment to offering OA articles to anyone, anywhere in the world. They also host communities that are available for researchers to join and an ambassador community to enhance their services globally. In addition to a straightforward keyword search, CORE offers advanced search options to filter results by publication type, year, language, journal, repository, and author.

CORE's user interface is easy to use and navigate. Search results can be sorted based on relevance or recency, and you can search for relevant content directly from the results screen.

Collection : 219,537,133 OA articles

Other Services : Additional services are available from CORE, with extras that are geared toward researchers, repositories, and businesses. There are tools for accessing raw data, including an API that provides direct access to data, datasets that are available for download, and FastSync for syncing data content from the CORE database.

CORE has a recommender plug-in that suggests relevant OA content in the database while conducting a search and a discovery feature that helps you discover OA versions of paywalled articles. Other features include tools for managing content, such as a dashboard for managing repository output and the Repository Edition service to enhance discoverability.

Good Source of Peer-Reviewed Articles : Yes

Advanced Search Options : Language, author, journal, publisher, repository, DOI, year

2. ScienceOpen

Functioning as a research and publishing network, ScienceOpen offers OA to more than 74 million articles in all areas of science. Although you do need to register to view the full text of articles, registration is free. The advanced search function is highly detailed, allowing you to find exactly the research you're looking for.

The Berlin- and Boston-based company was founded in 2013 to "facilitate open and public communications between academics and to allow ideas to be judged on their merit, regardless of where they come from." Search results can be exported for easy integration with reference management systems.

You can also bookmark articles for later research. There are extensive networking options, including your Science Open profile, a forum for interacting with other researchers, the ability to track your usage and citations, and an interactive bibliography. Users have the ability to review articles and provide their knowledge and insight within the community.

Collection : 74,560,631

Other Services : None

Advanced Search Options :   Content type, source, author, journal, discipline

3. Directory of Open Access Journals

A multidisciplinary, community-curated directory, the Directory of Open Access Journals (DOAJ) gives researchers access to high-quality peer-reviewed journals. It has archived more than two million articles from 17,193 journals, allowing you to either browse by subject or search by keyword.

The site was launched in 2003 with the aim of increasing the visibility of OA scholarly journals online. Content on the site covers subjects from science, to law, to fine arts, and everything in between. DOAJ has a commitment to "increase the visibility, accessibility, reputation, usage and impact of quality, peer-reviewed, OA scholarly research journals globally, regardless of discipline, geography or language."

Information about the journal is available with each search result. Abstracts are also available in a collapsible format directly from the search screen. The scholarly article website is somewhat simple, but it is easy to navigate. There are 16 principles of transparency and best practices in scholarly publishing that clearly outline DOAJ policies and standards.

Collection : 6,817,242

Advanced Search Options :   Subject, journal, year

4. Education Resources Information Center

The Education Resources Information Center (ERIC) of the Institution of Education Sciences allows you to search by topic for material related to the field of education. Links lead to other sites, where you may have to purchase the information, but you can search for full-text articles only. You can also search only peer-reviewed sources.

The service primarily indexes journals, gray literature (such as technical reports, white papers, and government documents), and books. All sources of material on ERIC go through a formal review process prior to being indexed. ERIC's selection policy is available as a PDF on their website.

The ERIC website has an extensive FAQ section to address user questions. This includes categories like general questions, peer review, and ERIC content. There are also tips for advanced searches, as well as general guidance on the best way to search the database. ERIC is an excellent database for content specific to education.

Collection : 1,292,897

Advanced Search Options : Boolean

5. arXiv e-Print Archive

The arXiv e-Print Archive is run by Cornell University Library and curated by volunteer moderators, and it now offers OA to more than one million e-prints.

There are advisory committees for all eight subjects available on the database. With a stated commitment to an "emphasis on openness, collaboration, and scholarship," the arXiv e-Print Archive is an excellent STEM resource.

The interface is not as user-friendly as some of the other databases available, and the website hosts a blog to provide news and updates, but it is otherwise a straightforward math and science resource. There are simple and advanced search options, and, in addition to conducting searches for specific topics and articles, users can browse content by subject. The arXiv e-Print Archive clearly states that they do not peer review the e-prints in the database.

Collection : 1,983,891

Good Source of Peer-Reviewed Articles : No

Advanced Search Options :   Subject, date, title, author, abstract, DOI

6. Social Science Research Network

The Social Science Research Network (SSRN) is a collection of papers from the social sciences community. It is a highly interdisciplinary platform used to search for scholarly articles related to 67 social science topics. SSRN has a variety of research networks for the various topics available through the free scholarly database.

The site offers more than 700,000 abstracts and more than 600,000 full-text papers. There is not yet a specific option to search for only full-text articles, but, because most of the papers on the site are free access, it's not often that you encounter a paywall. There is currently no option to search for only peer-reviewed articles.

You must become a member to use the services, but registration is free and enables you to interact with other scholars around the world. SSRN is "passionately committed to increasing inclusion, diversity and equity in scholarly research," and they encourage and discuss the use of inclusive language in scholarship whenever possible.

Collection : 1,058,739 abstracts; 915,452 articles

Advanced Search Options : Term, author, date, network

7. Public Library of Science

Public Library of Science (PLOS) is a big player in the world of OA science. Publishing 12 OA journals, the nonprofit organization is committed to facilitating openness in academic research. According to the site, "all PLOS content is at the highest possible level of OA, meaning that scientific articles are immediately and freely available to anyone, anywhere."

PLOS outlines four fundamental goals that guide the organization: break boundaries, empower researchers, redefine quality, and open science. All PLOS journals are peer-reviewed, and all 12 journals uphold rigorous ethical standards for research, publication, and scientific reporting.

PLOS does not offer advanced search options. Content is organized by topic into research communities that users can browse through, in addition to options to search for both articles and journals. The PLOS website also has resources for peer reviewers, including guidance on becoming a reviewer and on how to best participate in the peer review process.

Collection : 12 journals

Advanced Search Options : None

8. OpenDOAR

OpenDOAR, or the Directory of Open Access Repositories, is a comprehensive resource for finding free OA journals and articles. Using Google Custom Search, OpenDOAR combs through OA repositories around the world and returns relevant research in all disciplines.

The repositories it searches through are assessed and categorized by OpenDOAR staff to ensure they meet quality standards. Inclusion criteria for the database include requirements for OA content, global access, and categorically appropriate content, in addition to various other quality assurance measures. OpenDOAR has metadata, data, content, preservation, and submission policies for repositories, in addition to two OA policy statements regarding minimum and optimum recommendations.

This database allows users to browse and search repositories, which can then be selected, and articles and data can be accessed from the repository directly. As a repository database, much of the content on the site is geared toward the support of repositories and OA standards.

Collection : 5,768 repositories

Other Services : OpenDOAR offers a variety of additional services. Given the nature of the platform, services are primarily aimed at repositories and institutions, and there is a marked focus on OA in general. Sherpa services are OA archiving tools for authors and institutions.

They also offer various resources for OA support and compliance regarding standards and policies. The publication router matches publications and publishers with appropriate repositories.

There are also services and resources from JISC for repositories for cost management, discoverability, research impact, and interoperability, including ORCID consortium membership information. Additionally, a repository self-assessment tool is available for members.

Advanced Search Options :   Name, organization name, repository type, software name, content type, subject, country, region

9. Bielefeld Academic Search Engine

The Bielefeld Academic Search Engine (BASE) is operated by the Bielefeld University Library in Germany, and it offers more than 240 million documents from more than 8,000 sources. Sixty percent of its content is OA, and you can filter your search accordingly.

BASE has rigorous inclusion requirements for content providers regarding quality and relevance, and they maintain a list of content providers for the sake of transparency, which can be easily found on their website. BASE has a fairly elegant interface. Search results can be organized by author, title, or date.

From the search results, items can be selected and exported, added to favorites, emailed, and searched in Google Scholar. There are basic and advanced search features, with the advanced search offering numerous options for refining search criteria. There is also a feature on the website that saves recent searches without additional steps from the user.

Collection : 276,019,066 documents; 9,286 content providers

Advanced Search Options :   Author, subject, year, content provider, language, document type, access, terms of reuse

10. Digital Library of the Commons Repository

Run by Indiana University, the Digital Library of the Commons (DLC) Repository is a multidisciplinary journal repository that allows users to access thousands of free and OA articles from around the world. You can browse by document type, date, author, title, and more or search for keywords relevant to your topic.

DCL also offers the Comprehensive Bibliography of the Commons, an image database, and a keyword thesaurus for enhanced search parameters. The repository includes books, book chapters, conference papers, journal articles, surveys, theses and dissertations, and working papers. DCL advanced search features drop-down menus of search types with built-in Boolean search options.

Searches can be sorted by relevance, title, date, or submission date in ascending or descending order. Abstracts are included in selected search results, with access to full texts available, and citations can be exported from the same page. Additionally, the image database search includes tips for better search results.

Collection : 10,784

Advanced Search Options :   Author, date, title, subject, sector, region, conference

11. CIA World Factbook

The CIA World Factbook is a little different from the other resources on this list in that it is not an online journal directory or repository. It is, however, a useful free online research database for academics in a variety of disciplines.

All the information is free to access, and it provides facts about every country in the world, which are organized by category and include information about history, geography, transportation, and much more. The World Factbook can be searched by country or region, and there is also information about the world's oceans.

This site contains resources related to the CIA as an organization rather than being a scientific journal database specifically. The site has a user interface that is easy to navigate. The site also provides a section for updates regarding changes to what information is available and how it is organized, making it easier to interact with the information you are searching for.

Collection : 266 countries

12. Paperity

Paperity boasts its status as the "first multidisciplinary aggregator of OA journals and papers." Their focus is on helping you avoid paywalls while connecting you to authoritative research. In addition to providing readers with easy access to thousands of journals, Paperity seeks to help authors reach their audiences and help journals increase their exposure to boost readership.

Paperity has journal articles for every discipline, and the database offers more than a dozen advanced search options, including the length of the paper and the number of authors. There is even an option to include, exclude, or exclusively search gray papers.

Paperity is available for mobile, with both a mobile site and the Paperity Reader, an app that is available for both Android and Apple users. The database is also available on social media. You can interact with Paperity via Twitter and Facebook, and links to their social media are available on their homepage, including their Twitter feed.

Collection : 8,837,396

Advanced Search Options : Title, abstract, journal title, journal ISSN, publisher, year of publication, number of characters, number of authors, DOI, author, affiliation, language, country, region, continent, gray papers

13. dblp Computer Science Bibliography

The dblp Computer Science Bibliography is an online index of major computer science publications. dblp was founded in 1993, though until 2010 it was a university-specific database at the University of Trier in Germany. It is currently maintained by the Schloss Dagstuhl – Leibniz Center for Informatics.

Although it provides access to both OA articles and those behind a paywall, you can limit your search to only OA articles. The site indexes more than three million publications, making it an invaluable resource in the world of computer science. dblp entries are color-coded based on the type of item.

dblp has an extensive FAQ section, so questions that might arise about topics like the database itself, navigating the website, or the data on dblp, in addition to several other topics, are likely to be answered. The website also hosts a blog and has a section devoted to website statistics.

Collection : 5,884,702

14. EconBiz

EconBiz is a great resource for economic and business studies. A service of the Leibniz Information Centre for Economics, it offers access to full texts online, with the option of searching for OA material only. Their literature search is performed across multiple international databases.

EconBiz has an incredibly useful research skills section, with resources such as Guided Walk, a service to help students and researchers navigate searches, evaluate sources, and correctly cite references; the Research Guide EconDesk, a help desk to answer specific questions and provide advice to aid in literature searches; and the Academic Career Kit for what they refer to as Early Career Researchers.

Other helpful resources include personal literature lists, a calendar of events for relevant calls for papers, conferences, and workshops, and an economics terminology thesaurus to help in finding keywords for searches. To stay up-to-date with EconBiz, you can sign up for their newsletter.

Collection : 1,075,219

Advanced Search Options :   Title, subject, author, institution, ISBN/ISSN, journal, publisher, language, OA only

15. BioMed Central

BioMed Central provides OA research from more than 300 peer-reviewed journals. While originally focused on resources related to the physical sciences, math, and engineering, BioMed Central has branched out to include journals that cover a broader range of disciplines, with the aim of providing a single platform that provides OA articles for a variety of research needs. You can browse these journals by subject or title, or you can search all articles for your required keyword.

BioMed Central has a commitment to peer-reviewed sources and to the peer review process itself, continually seeking to help and improve the peer review process. They're "committed to maintaining high standards through full and stringent peer review."

Additionally, the website includes resources to assist and support editors as part of their commitment to providing high-quality, peer-reviewed OA articles.

Collection : 507,212

Other Services : BMC administers the International Standard Randomised Controlled Trial Number (ISRCTN) registry. While initially designed for registering clinical trials, since its creation in 2000, the registry has broadened its scope to include other health studies as well.

The registry is recognized by the International Committee of Medical Journal Editors, as well as the World Health Organization (WHO), and it meets the requirements established by the WHO International Clinical Trials Registry Platform.

The study records included in the registry are all searchable and free to access. The ISRCTN registry "supports transparency in clinical research, helps reduce selective reporting of results and ensures an unbiased and complete evidence base."

Advanced Search Options :   Author, title, journal, list

A multidisciplinary search engine, JURN provides links to various scholarly websites, articles, and journals that are free to access or OA. Covering the fields of the arts, humanities, business, law, nature, science, and medicine, JURN has indexed almost 5,000 repositories to help you find exactly what you're looking for.

Search features are enhanced by Google, but searches are filtered through their index of repositories. JURN seeks to reach a wide audience, with their search engine tailored to researchers from "university lecturers and students seeking a strong search tool for OA content" and "advanced and ambitious students, age 14-18" to "amateur historians and biographers" and "unemployed and retired lecturers."

That being said, JURN is very upfront about its limitations. They admit to not being a good resource for educational studies, social studies, or psychology, and conference archives are generally not included due to frequently unstable URLs.

Collection : 5,064 indexed journals

Other Services : JURN has a browser add-on called UserScript. This add-on allows users to integrate the JURN database directly into Google Search. When performing a search through Google, the add-on creates a link that sends the search directly to JURN CSE. JURN CSE is a search service that is hosted by Google.

Clicking the link from the Google Search bar will run your search through the JURN database from the Google homepage. There is also an interface for a DuckDuckGo search box; while this search engine has an emphasis on user privacy, for smaller sites that may be indexed by JURN, DuckDuckGo may not provide the same depth of results.

Advanced Search Options :   Google search modifiers

Dryad is a digital repository of curated, OA scientific research data. Launched in 2009, it is run by a not-for-profit membership organization, with a community of institutional and publisher members for whom their services have been designed. Members include institutions such as Stanford, UCLA, and Yale, as well as publishers like Oxford University Press and Wiley.

Dryad aims to "promote a world where research data is openly available, integrated with the scholarly literature, and routinely reused to create knowledge." It is free to access for the search and discovery of data. Their user experience is geared toward easy self-depositing, supports Creative Commons licensing, and provides DOIs for all their content.

Note that there is a publishing charge associated if you wish to publish your data in Dryad.  When searching datasets, they are accompanied by author information and abstracts for the associated studies, and citation information is provided for easy attribution.

Collection : 44,458

Advanced Search Options : No

Run by the British Library, the E-Theses Online Service (EThOS) allows you to search over 500,000 doctoral theses in a variety of disciplines. All of the doctoral theses available on EThOS have been awarded by higher education institutions in the United Kingdom.

Although some full texts are behind paywalls, you can limit your search to items available for immediate download, either directly through EThOS or through an institution's website. More than half of the records in the database provide access to full-text theses.

EThOS notes that they do not hold all records for all institutions, but they strive to index as many doctoral theses as possible, and the database is constantly expanding, with approximately 3,000 new records added and 2,000 new full-text theses available every month. The availability of full-text theses is dependent on multiple factors, including their availability in the institutional repository and the level of repository development.

Collection : 500,000+

Advanced Search Options : Abstract, author's first name, author's last name, awarding body, current institution, EThOS ID, year, language, qualifications, research supervisor, sponsor/funder, keyword, title

PubMed is a research platform well-known in the fields of science and medicine. It was created and developed by the National Center for Biotechnology Information (NCBI) at the National Library of Medicine (NLM). It has been available since 1996 and offers access to "more than 33 million citations for biomedical literature from MEDLINE, life science journals, and online books."

While PubMed does not provide full-text articles directly, and many full-text articles may be behind paywalls or require subscriptions to access them, when articles are available from free sources, such as through PubMed Central (PMC), those links are provided with the citations and abstracts that PubMed does provide.

PMC, which was established in 2000 by the NLM, is a free full-text archive that includes more than 6,000,000 records. PubMed records link directly to corresponding PMC results. PMC content is provided by publishers and other content owners, digitization projects, and authors directly.

Collection : 33,000,000+

Advanced Search Options : Author's first name, author's last name, identifier, corporation, date completed, date created, date entered, date modified, date published, MeSH, book, conflict of interest statement, EC/RN number, editor, filter, grant number, page number, pharmacological action, volume, publication type, publisher, secondary source ID, text, title, abstract, transliterated title

20. Semantic Scholar

A unique and easy-to-use resource, Semantic Scholar defines itself not just as a research database but also as a "search and discovery tool." Semantic Scholar harnesses the power of artificial intelligence to efficiently sort through millions of science-related papers based on your search terms.

Through this singular application of machine learning, Semantic Scholar expands search results to include topic overviews based on your search terms, with the option to create an alert for or further explore the topic. It also provides links to related topics.

In addition, search results produce "TLDR" summaries in order to provide concise overviews of articles and enhance your research by helping you to navigate quickly and easily through the available literature to find the most relevant information. According to the site, although some articles are behind paywalls, "the data [they] have for those articles is limited," so you can expect to receive mostly full-text results.

Collection : 203,379,033

Other Services : Semantic Scholar supports multiple popular browsers. Content can be accessed through both mobile and desktop versions of Firefox, Microsoft Edge, Google Chrome, Apple Safari, and Opera.

Additionally, Semantic Scholar provides browser extensions for both Chrome and Firefox, so AI-powered scholarly search results are never more than a click away. The mobile interface includes an option for Semantic Swipe, a new way of interacting with your research results.

There are also beta features that can be accessed as part of the Beta Program, which will provide you with features that are being actively developed and require user feedback for further improvement.

Advanced Search Options : Field of study, date range, publication type, author, journal, conference, PDF

Zenodo, powered by the European Organization for Nuclear Research (CERN), was launched in 2013. Taking its name from Zenodotus, the first librarian of the ancient library of Alexandria, Zenodo is a tool "built and developed by researchers, to ensure that everyone can join in open science." Zenodo accepts all research from every discipline in any file format.

However, Zenodo also curates uploads and promotes peer-reviewed material that is available through OA. A DOI is assigned to everything that is uploaded to Zenodo, making research easily findable and citable. You can sort by keyword, title, journal, and more and download OA documents directly from the site.

While there are closed access and restricted access items in the database, the vast majority of research is OA material. Search results can be filtered by access type, making it easy to view the free articles available in the database.

Collection : 2,220,000+

Advanced Search Options : Access, file type, keywords

Check out our roundup of free research databases as a handy one-page PDF.

How to find peer-reviewed articles.

There are a lot of free scholarly articles available from various sources. The internet is a big place. So how do you go about finding peer-reviewed articles when conducting your research? It's important to make sure you are using reputable sources.

The first source of the article is the person or people who wrote it. Checking out the author can give you some initial insight into how much you can trust what you’re reading. Looking into the publication information of your sources can also indicate whether the article is reliable.

Aspects of the article, such as subject and audience, tone, and format, are other things you can look at when evaluating whether the article you're using is valid, reputable, peer-reviewed material. So, let's break that down into various components so you can assess your research to ensure that you're using quality articles and conducting solid research.

Check the Author

Peer-reviewed articles are written by experts or scholars with experience in the field or discipline they're writing about. The research in a peer-reviewed article has to pass a rigorous evaluation process, so it's a foregone conclusion that the author(s) of a peer-reviewed article should have experience or training related to that research.

When evaluating an article, take a look at the author's information. What credentials does the author have to indicate that their research has scholarly weight behind it? Finding out what type of degree the author has—and what that degree is in—can provide insight into what kind of authority the author is on the subject.

Something else that might lend credence to the author's scholarly role is their professional affiliation. A look at what organization or institution they are affiliated with can tell you a lot about their experience or expertise. Where were they trained, and who is verifying their research?

Identify Subject and Audience

The ultimate goal of a study is to answer a question. Scholarly articles are also written for scholarly audiences, especially articles that have gone through the peer review process. This means that the author is trying to reach experts, researchers, academics, and students in the field or topic the research is based on.

Think about the question the author is trying to answer by conducting this research, why, and for whom. What is the subject of the article? What question has it set out to answer? What is the purpose of finding the information? Is the purpose of the article of importance to other scholars? Is it original content?

Research should also be approached analytically. Is the methodology sound? Is the author using an analytical approach to evaluate the data that they have obtained? Are the conclusions they've reached substantiated by their data and analysis? Answering these questions can reveal a lot about the article's validity.

Format Matters

Reliable articles from peer-reviewed sources have certain format elements to be aware of. The first is an abstract. An abstract is a short summary or overview of the article. Does the article have an abstract? It's unlikely that you're reading a peer-reviewed article if it doesn't. Peer-reviewed journals will also have a word count range. If an article seems far too short or incredibly long, that may be reason to doubt it.

Another feature of reliable articles is the sections the information is divided into. Peer-reviewed research articles will have clear, concise sections that appropriately organize the information. This might include a literature review, methodology, results (in the case of research articles), and a conclusion.

One of the most important sections is the references or bibliography. This is where the researcher lists all the sources of their information. A peer-reviewed source will have a comprehensive reference section.

An article that has been written to reach an academic community will have an academic tone. The language that is used, and the way this language is used, is important to consider. If the article is riddled with grammatical errors, confusing syntax, and casual language, it almost definitely didn't make it through the peer review process.

Also consider the use of terminology. Every discipline is going to have standard terminology or jargon that can be used and understood by other academics in the discipline. The language in a peer-reviewed article is going to reflect that.

If the author is going out of their way to explain simple terms, or terms that are standard to the field or discipline, it's unlikely that the article has been peer reviewed, as this is something that the author would be asked to address during the review process.

Publication

The source of the article will be a very good indicator of the likelihood that it was peer reviewed. Where was the article published? Was it published alongside other academic articles in the same discipline? Is it a legitimate and reputable scholarly publication?

A trade publication or newspaper might be legitimate or reputable, but it is not a scholarly source, and it will not have been subject to the peer review process. Scholarly journals are the best resource for peer-reviewed articles, but it's important to remember that not all scholarly journals are peer reviewed.

It's helpful to look at a scholarly source's website, as peer-reviewed journals will have a clear indication of the peer review process. University libraries, institutional repositories, and reliable databases (and now you have a list of legit ones) can also help provide insight into whether an article comes from a peer-reviewed journal.

Common Research Mistakes to Avoid

Research is a lot of work. Even with high standards and good intentions, it's easy to make mistakes. Perhaps you searched for access to scientific journals for free and found the perfect peer-reviewed sources, but you forgot to document everything, and your references are a mess. Or, you only searched for free online articles and missed out on a ground-breaking study that was behind a paywall.

Whether your research is for a degree or to get published or to satisfy your own inquisitive nature, or all of the above, you want all that work to produce quality results. You want your research to be thorough and accurate.

To have any hope of contributing to the literature on your research topic, your results need to be high quality. You might not be able to avoid every potential mistake, but here are some that are both common and easy to avoid.

Sticking to One Source

One of the hallmarks of good research is a healthy reference section. Using a variety of sources gives you a better answer to your question. Even if all of the literature is in agreement, looking at various aspects of the topic may provide you with an entirely different picture than you would have if you looked at your research question from only one angle.

Not Documenting Every Fact

As you conduct your research, do yourself a favor and write everything down. Everything you include in your paper or article that you got from another source is going to need to be added to your references and cited.

It's important, especially if your aim is to conduct ethical, high-quality research, that all of your research has proper attribution. If you don't document as you go, you could end up making a lot of work for yourself if the information you don't write down is something that later, as you write your paper, you really need.

Using Outdated Materials

Academia is an ever-changing landscape. What was true in your academic discipline or area of research ten years ago may have since been disproven. If fifteen studies have come out since the article that you're using was published, it's more than a little likely that you're going to be basing your research on flawed or dated information.

If the information you're basing your research on isn't as up-to-date as possible, your research won't be of quality or able to stand up to any amount of scrutiny. You don't want all of your hard work to be for naught.

Relying Solely on Open Access Journals

OA is a great resource for conducting academic research. There are high-quality journal articles available through OA, and that can be very helpful for your research. But, just because you have access to free articles, that doesn't mean that there's nothing to be found behind a paywall.

Just as dismissing high-quality peer-reviewed articles because they are OA would be limiting, not exploring any paid content at all is equally short-sighted. If you're seeking to conduct thorough and comprehensive research, exploring all of your options for quality sources is going to be to your benefit.

Digging Too Deep or Not Deep Enough

Research is an art form, and it involves a delicate balance of information. If you conduct your research using only broad search terms, you won't be able to answer your research question well, or you'll find that your research provides information that is closely related to your topic but, ultimately, your findings are vague and unsubstantiated.

On the other hand, if you delve deeply into your research topic with specific searches and turn up too many sources, you might have a lot of information that is adjacent to your topic but without focus and perhaps not entirely relevant. It's important to answer your research question concisely but thoroughly.

Different Types of Scholarly Articles

Different types of scholarly articles have different purposes. An original research article, also called an empirical article, is the product of a study or an experiment. This type of article seeks to answer a question or fill a gap in the existing literature.

Research articles will have a methodology, results, and a discussion of the findings of the experiment or research and typically a conclusion.

Review articles overview the current literature and research and provide a summary of what the existing research indicates or has concluded. This type of study will have a section for the literature review, as well as a discussion of the findings of that review. Review articles will have a particularly extensive reference or bibliography section.

Theoretical articles draw on existing literature to create new theories or conclusions, or look at current theories from a different perspective, to contribute to the foundational knowledge of the field of study.

10 Tips for Navigating Journal Databases

Use the right academic journal database for your search, be that interdisciplinary or specific to your field. Or both!

If it's an option, set the search results to return only peer-reviewed sources.

Start by using search terms that are relevant to your topic without being overly specific.

Try synonyms, especially if your keywords aren't returning the desired results.

Even if you've found some good articles, try searching using different terms.

Explore the advanced search features of the database(s).

Learn to use Booleans (AND, OR, NOT) to expand or narrow your results.

Once you've gotten some good results from a more general search, try narrowing your search.

Read through abstracts when trying to find articles relevant to your research.

Keep track of your research and use citation tools. It'll make life easier when it comes time to compile your references.

7 Frequently Asked Questions

1. how do i get articles for free.

Free articles can be found through free online academic journals, OA databases, or other databases that include OA journals and articles. These resources allow you to access free papers online so you can conduct your research without getting stuck behind a paywall.

Academics don't receive payment for the articles they contribute to journals. There are often, in fact, publication fees that scholars pay in order to publish. This is one of the funding structures that allows OA journals to provide free content so that you don't have to pay fees or subscription costs to access journal articles.

2. How Do I Find Journal Articles?

Journal articles can be found in databases and institutional repositories that can be accessed at university libraries. However, online research databases that contain OA articles are the best resource for getting free access to journal articles that are available online.

Peer-reviewed journal articles are the best to use for academic research, and there are a number of databases where you can find peer-reviewed OA journal articles. Once you've found a useful article, you can look through the references for the articles the author used to conduct their research, and you can then search online databases for those articles, too.

3. How Do I Find Peer-Reviewed Articles?

Peer-reviewed articles can be found in reputable scholarly peer-reviewed journals. High-quality journals and journal articles can be found online using academic search engines and free research databases. These resources are excellent for finding OA articles, including peer-reviewed articles.

OA articles are articles that can be accessed for free. While some scholarly search engines and databases include articles that aren't peer reviewed, there are also some that provide only peer-reviewed articles, and databases that include non-peer-reviewed articles often have advanced search features that enable you to select "peer review only." The database will return results that are exclusively peer-reviewed content.

4. What Are Research Databases?

A research database is a list of journals, articles, datasets, and/or abstracts that allows you to easily search for scholarly and academic resources and conduct research online. There are databases that are interdisciplinary and cover a variety of topics.

For example, Paperity might be a great resource for a chemist as well as a linguist, and there are databases that are more specific to a certain field. So, while ERIC might be one of the best educational databases available for OA content, it's not going to be one of the best databases for finding research in the field of microbiology.

5. How Do I Find Scholarly Articles for Specific Fields?

There are interdisciplinary research databases that provide articles in a variety of fields, as well as research databases that provide articles that cater to specific disciplines. Additionally, a journal repository or index can be a helpful resource for finding articles in a specific field.

When searching an interdisciplinary database, there are frequently advanced search features that allow you to narrow the search results down so that they are specific to your field. Selecting "psychology" in the advanced search features will return psychology journal articles in your search results. You can also try databases that are specific to your field.

If you're searching for law journal articles, many law reviews are OA. If you don't know of any databases specific to history, visiting a journal repository or index and searching "history academic journals" can return a list of journals specific to history and provide you with a place to begin your research.

6. Are Peer-Reviewed Articles Really More Legitimate?

The short answer is yes, peer-reviewed articles are more legitimate resources for academic research. The peer review process provides legitimacy, as it is a rigorous review of the content of an article that is performed by scholars and academics who are experts in their field of study. The review provides an evaluation of the quality and credibility of the article.

Non-peer-reviewed articles are not subject to a review process and do not undergo the same level of scrutiny. This means that non-peer-reviewed articles are unlikely, or at least not as likely, to meet the same standards that peer-reviewed articles do.

7. Are Free Article Directories Legitimate?

Yes! As with anything, some databases are going to be better for certain requirements than others. But, a scholarly article database being free is not a reason in itself to question its legitimacy.

Free scholarly article databases can provide access to abstracts, scholarly article websites, journal repositories, and high-quality peer-reviewed journal articles. The internet has a lot of information, and it's often challenging to figure out what information is reliable. 

Research databases and article directories are great resources to help you conduct your research. Our list of the best research paper websites is sure to provide you with sources that are totally legit.

Get Professional Academic Editing

Hire an expert academic editor , or get a free sample, about the author.

Scribendi's in-house editors work with writers from all over the globe to perfect their writing. They know that no piece of writing is complete without a professional edit, and they love to see a good piece of writing transformed into a great one. Scribendi's in-house editors are unrivaled in both experience and education, having collectively edited millions of words and obtained numerous degrees. They love consuming caffeinated beverages, reading books of various genres, and relaxing in quiet, dimly lit spaces.

Have You Read?

"The Complete Beginner's Guide to Academic Writing"

Related Posts

How to Write a Research Proposal

How to Write a Scientific Paper

How to Write a Thesis or Dissertation

Upload your file(s) so we can calculate your word count, or enter your word count manually.

We will also recommend a service based on the file(s) you upload.

English is not my first language. I need English editing and proofreading so that I sound like a native speaker.

I need to have my journal article, dissertation, or term paper edited and proofread, or I need help with an admissions essay or proposal.

I have a novel, manuscript, play, or ebook. I need editing, copy editing, proofreading, a critique of my work, or a query package.

I need editing and proofreading for my white papers, reports, manuals, press releases, marketing materials, and other business documents.

I need to have my essay, project, assignment, or term paper edited and proofread.

I want to sound professional and to get hired. I have a resume, letter, email, or personal document that I need to have edited and proofread.

 Prices include your personal % discount.

 Prices include % sales tax ( ).

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

PubMed Central (PMC) Home Page

PubMed Central ® (PMC) is a free full-text archive of biomedical and life sciences journal literature at the U.S. National Institutes of Health's National Library of Medicine (NIH/NLM)

Discover a digital archive of scholarly articles, spanning centuries of scientific research.

Learn how to find and read articles of interest to you.

Collections

Browse the PMC Journal List or learn about some of PMC's unique collections.

For Authors

Navigate the PMC submission methods to comply with a funder mandate, expand access, and ensure preservation.

For Publishers

Learn about deposit options for journals and publishers and the PMC selection process.

For Developers

Find tools for bulk download, text mining, and other machine analysis.

9.9 MILLION articles are archived in PMC.

Content provided in part by:, full participation journals.

Journals deposit the complete contents of each issue or volume.

NIH Portfolio Journals

Journals deposit all NIH-funded articles as defined by the NIH Public Access Policy.

Selective Deposit Programs

Publisher deposits a subset of articles from a collection of journals.

March 21, 2024

Preview upcoming improvements to pmc.

We are pleased to announce the availability of a preview of improvements planned for the PMC website. These…

Dec. 15, 2023

Update on pubreader format.

The PubReader format was added to PMC in 2012 to make it easier to read full text articles on tablet, mobile, and oth…

We are pleased to announce the availability of a preview of improvements planned for the PMC website. These improvements will become the default in October 2024.

Reference management. Clean and simple.

The top list of academic search engines

1. Google Scholar

4. science.gov, 5. semantic scholar, 6. baidu scholar, get the most out of academic search engines, frequently asked questions about academic search engines, related articles.

Academic search engines have become the number one resource to turn to in order to find research papers and other scholarly sources. While classic academic databases like Web of Science and Scopus are locked behind paywalls, Google Scholar and others can be accessed free of charge. In order to help you get your research done fast, we have compiled the top list of free academic search engines.

Google Scholar is the clear number one when it comes to academic search engines. It's the power of Google searches applied to research papers and patents. It not only lets you find research papers for all academic disciplines for free but also often provides links to full-text PDF files.

• Coverage: approx. 200 million articles
• Abstracts: only a snippet of the abstract is available
• Related articles: ✔
• References: ✔
• Cited by: ✔
• Links to full text: ✔
• Export formats: APA, MLA, Chicago, Harvard, Vancouver, RIS, BibTeX

BASE is hosted at Bielefeld University in Germany. That is also where its name stems from (Bielefeld Academic Search Engine).

• Coverage: approx. 136 million articles (contains duplicates)
• Abstracts: ✔
• Related articles: ✘
• References: ✘
• Cited by: ✘
• Export formats: RIS, BibTeX

CORE is an academic search engine dedicated to open-access research papers. For each search result, a link to the full-text PDF or full-text web page is provided.

• Coverage: approx. 136 million articles
• Links to full text: ✔ (all articles in CORE are open access)
• Export formats: BibTeX

Science.gov is a fantastic resource as it bundles and offers free access to search results from more than 15 U.S. federal agencies. There is no need anymore to query all those resources separately!

• Coverage: approx. 200 million articles and reports
• Links to full text: ✔ (available for some databases)
• Export formats: APA, MLA, RIS, BibTeX (available for some databases)

Semantic Scholar is the new kid on the block. Its mission is to provide more relevant and impactful search results using AI-powered algorithms that find hidden connections and links between research topics.

• Coverage: approx. 40 million articles
• Export formats: APA, MLA, Chicago, BibTeX

Although Baidu Scholar's interface is in Chinese, its index contains research papers in English as well as Chinese.

• Coverage: no detailed statistics available, approx. 100 million articles
• Abstracts: only snippets of the abstract are available
• Export formats: APA, MLA, RIS, BibTeX

RefSeek searches more than one billion documents from academic and organizational websites. Its clean interface makes it especially easy to use for students and new researchers.

• Coverage: no detailed statistics available, approx. 1 billion documents
• Abstracts: only snippets of the article are available
• Export formats: not available

Consider using a reference manager like Paperpile to save, organize, and cite your references. Paperpile integrates with Google Scholar and many popular databases, so you can save references and PDFs directly to your library using the Paperpile buttons:

Google Scholar is an academic search engine, and it is the clear number one when it comes to academic search engines. It's the power of Google searches applied to research papers and patents. It not only let's you find research papers for all academic disciplines for free, but also often provides links to full text PDF file.

Semantic Scholar is a free, AI-powered research tool for scientific literature developed at the Allen Institute for AI. Sematic Scholar was publicly released in 2015 and uses advances in natural language processing to provide summaries for scholarly papers.

BASE , as its name suggest is an academic search engine. It is hosted at Bielefeld University in Germany and that's where it name stems from (Bielefeld Academic Search Engine).

CORE is an academic search engine dedicated to open access research papers. For each search result a link to the full text PDF or full text web page is provided.

Science.gov is a fantastic resource as it bundles and offers free access to search results from more than 15 U.S. federal agencies. There is no need any more to query all those resources separately!

SpringerOpen

The SpringerOpen portfolio has grown tremendously since its launch in 2010, so that we now offer researchers from all areas of science, technology, medicine, the humanities and social sciences a place to publish open access in journals. Publishing with SpringerOpen makes your work freely available online for everyone, immediately upon publication, and our high-level peer-review and production processes guarantee the quality and reliability of the work. Open access books are published by our Springer imprint.

Find the right journal for you

Explore our subject areas, learn all about open access.

• Browse our alphabetical journal list
• Explore our journals by subject
• Tips for finding the right journal
• Find the right journal with our Journal Suggester
• Find out if open access book publishing is right for you

• Visit our subject pages covering all subject areas in science, technology, medicine, the humanities and social sciences

Visit the Springer Author & Reviewer tutorials and learn all about open access, your benefits, mandates, funding, copyright and more in the different interactive tutorials. And take our quiz to test your knowledge!

• C heck out the free tutorials
• Take the quiz

Video library

Your browser needs to have JavaScript enabled to view this video

Our video library contains how-to videos, videos on the research we publish and journal videos. 

• Link to facebook
• Link to linkedin
• Link to twitter
• Link to youtube
• Writing Tips

10 Free Research and Journal Databases

3-minute read

• 6th April 2019

Finding good research can be tough, especially when so much of it is locked behind paywalls . But there are free resources out there if you know where to look. So to help out, we’ve compiled a list of ten free academic search engines and databases that you should check out.

1. Google Scholar

Even if you’ve not used Google Scholar before, you’ll know Google. And, thus, you can probably guess that Google Scholar is a search engine dedicated to academic work. Not everything listed on Google Scholar will be freely available in full. But it is a good place to start if you’re looking for a specific paper, and many papers can be downloaded for free.

CORE is an open research aggregator. This means it works as a search engine for open access research published by organizations from around the world, all of which is available for free. It is also the world’s largest open access aggregator , so it is a very useful resource for researchers!

3. Bielefeld Academic Search Engine (BASE)

Another dedicated academic search engine, BASE offers access to more than 140 million documents from more than 6,000 sources. Around 60% of these documents are open access, and you can filter results to see only research that is available for free online.

4. Directory of Open Access Journals (DOAJ)

The Directory of Open Access Journals (DOAJ) is a database that lists around 12,000 open access journals covering all areas of science, technology, medicine, social science, and the humanities.

PubMed is a search engine maintained by the NCBI, part of the United States National Library of Medicine. It provides access to more than 29 million citations of biomedical research from MEDLINE, life science journals, and online books. The NCBI runs a similar search engine for research in the chemical sciences called PubChem , too, which is also free to use.

Find this useful?

Subscribe to our newsletter and get writing tips from our editors straight to your inbox.

6. E-Theses Online Service (EThOS)

Run by the British Library, EThOS is a database of over 500,000 doctoral theses. More than half of these are available for free, either directly via EThOS or via a link to a university website.

7. Social Science Research Network (SSRN)

SSRN is a database for research from the social sciences and humanities, including 846,589 research papers from 426,107 researchers across 30 disciplines. Most of these are available for free, although you may need to sign up as a member (also free) to access some services.

8. WorldWideScience

WorldWideScience is a global academic search engine, providing access to national and international scientific databases from across the globe. One interesting feature is that it offers automatic translation, so users can have search results translated into their preferred language.

9. Semantic Scholar

Semantic Scholar is an “intelligent” academic search engine. It uses machine learning to prioritize the most important research, which can make it easier to find relevant literature. Or, in Semantic Scholar’s own words, it uses influential citations, images, and key phrases to “cut through the clutter.”

10. Public Library of Science (PLOS)

PLOS is an open-access research organization that publishes several journals. But as well as publishing its own research, PLOS is a dedicated advocate for open-access learning. So if you appreciate the search engines and databases we’ve listed here, check out the rest of the PLOS site to find out more about their campaign to enable access to knowledge.

Share this article:

Post A New Comment

Got content that needs a quick turnaround? Let us polish your work. Explore our editorial business services.

9-minute read

How to Use Infographics to Boost Your Presentation

Is your content getting noticed? Capturing and maintaining an audience’s attention is a challenge when...

8-minute read

Why Interactive PDFs Are Better for Engagement

Are you looking to enhance engagement and captivate your audience through your professional documents? Interactive...

7-minute read

Seven Key Strategies for Voice Search Optimization

Voice search optimization is rapidly shaping the digital landscape, requiring content professionals to adapt their...

4-minute read

Five Creative Ways to Showcase Your Digital Portfolio

Are you a creative freelancer looking to make a lasting impression on potential clients or...

How to Ace Slack Messaging for Contractors and Freelancers

Effective professional communication is an important skill for contractors and freelancers navigating remote work environments....

How to Insert a Text Box in a Google Doc

Google Docs is a powerful collaborative tool, and mastering its features can significantly enhance your...

Make sure your writing is the best it can be with our expert English proofreading and editing.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Download 55 million PDFs for free

Explore our top research interests.

Engineering

Anthropology

• Earth Sciences

• Computer Science

• Mathematics

• Health Sciences

Join 262 million academics and researchers

Track your impact.

Share your work with other academics, grow your audience and track your impact on your field with our robust analytics

Discover new research

Get access to millions of research papers and stay informed with the important topics around the world

Publish your work

Publish your research with fast and rigorous service through Academia.edu Publishing. Get instant worldwide dissemination of your work

Unlock the most powerful tools with Academia Premium

Work faster and smarter with advanced research discovery tools

Search the full text and citations of our millions of papers. Download groups of related papers to jumpstart your research. Save time with detailed summaries and search alerts.

• Advanced Search
• PDF Packages of 37 papers
• Summaries and Search Alerts

Share your work, track your impact, and grow your audience

Get notified when other academics mention you or cite your papers. Track your impact with in-depth analytics and network with members of your field.

• Mentions and Citations Tracking
• Advanced Analytics
• Publishing Tools

Real stories from real people

Used by academics at over 15,000 universities

Get started and find the best quality research

• Academia.edu Publishing
•   We're Hiring!
•   Help Center
• Find new research papers in:
• Cognitive Science
• Academia ©2024

“The only truly modern academic research engine”

Oa.mg is a search engine for academic papers, specialising in open access. we have over 250 million papers in our index..

A free, AI-powered research tool for scientific literature

• Alice Kessler-Harris
• Dialog System
• Bioremediation

New & Improved API for Developers

Introducing semantic reader in beta.

Stay Connected With Semantic Scholar Sign Up What Is Semantic Scholar? Semantic Scholar is a free, AI-powered research tool for scientific literature, based at the Allen Institute for AI.

Open Access. Powered by Scholars. Published by Universities. ®

Digital Commons Network ™

The Digital Commons Network brings together free, full-text scholarly articles from hundreds of universities and colleges worldwide. Curated by university librarians and their supporting institutions, the Network includes a growing collection of peer-reviewed journal articles, book chapters, dissertations, working papers, conference proceedings, and other original scholarly work.

• Advanced search
• Peer review

Discover relevant research today

Advance your research field in the open

Reach new audiences and maximize your readership

ScienceOpen puts your research in the context of

Publications

For Publishers

ScienceOpen offers content hosting, context building and marketing services for publishers. See our tailored offerings

• For academic publishers  to promote journals and interdisciplinary collections
• For open access journals  to host journal content in an interactive environment
• For university library publishing  to develop new open access paradigms for their scholars
• For scholarly societies  to promote content with interactive features

For Institutions

ScienceOpen offers state-of-the-art technology and a range of solutions and services

• For faculties and research groups  to promote and share your work
• For research institutes  to build up your own branding for OA publications
• For funders  to develop new open access publishing paradigms
• For university libraries to create an independent OA publishing environment

For Researchers

Make an impact and build your research profile in the open with ScienceOpen

• Search and discover relevant research in over 94 million Open Access articles and article records
• Share your expertise and get credit by publicly reviewing any article
• Publish your poster or preprint and track usage and impact with article- and author-level metrics
• Create a topical Collection  to advance your research field

Create a Journal powered by ScienceOpen

Launching a new open access journal or an open access press? ScienceOpen now provides full end-to-end open access publishing solutions – embedded within our smart interactive discovery environment. A modular approach allows open access publishers to pick and choose among a range of services and design the platform that fits their goals and budget.

Continue reading “Create a Journal powered by ScienceOpen”   

What can a Researcher do on ScienceOpen?

ScienceOpen provides researchers with a wide range of tools to support their research – all for free. Here is a short checklist to make sure you are getting the most of the technological infrastructure and content that we have to offer. What can a researcher do on ScienceOpen? Continue reading “What can a Researcher do on ScienceOpen?”   

ScienceOpen on the Road

Upcoming events.

• 20 – 22 February – ResearcherToReader Conferece

Past Events

• 09 November – Webinar for the Discoverability of African Research
• 26 – 27 October – Attending the Workshop on Open Citations and Open Scholarly Metadata
• 18 – 22 October – ScienceOpen at Frankfurt Book Fair.
• 27 – 29 September – Attending OA Tage, Berlin .
• 25 – 27 September – ScienceOpen at Open Science Fair
• 19 – 21 September – OASPA 2023 Annual Conference .
• 22 – 24 May – ScienceOpen sponsoring Pint of Science, Berlin.
• 16-17 May – ScienceOpen at 3rd AEUP Conference.
• 20 – 21 April – ScienceOpen attending Scaling Small: Community-Owned Futures for Open Access Books .
• 18 – 20 April – ScienceOpen at the London Book Fair .

What is ScienceOpen?

• Smart search and discovery within an interactive interface
• Researcher promotion and ORCID integration
• Open evaluation with article reviews and Collections
• Business model based on providing services to publishers

Live Twitter stream

Some of our partners:.

Librarians/Admins

• EBSCOhost Collection Manager
• EBSCO Experience Manager
• EBSCO Connect
• Start your research
• EBSCO Mobile App

Clinical Decisions Users

• DynaMed Decisions
• Dynamic Health
• Waiting Rooms
• NoveList Blog

Free Databases

EBSCO provides free research databases covering a variety of subjects for students, researchers and librarians.

Exploring Race in Society

This free research database offers essential content covering important issues related to race in society today. Essays, articles, reports and other reliable sources provide an in-depth look at the history of race and provide critical context for learning more about topics associated with race, ethnicity, diversity and inclusiveness.

EBSCO Open Dissertations

EBSCO Open Dissertations is a collaboration between EBSCO and BiblioLabs to increase traffic and discoverability of ETD research. You can join the movement and add your theses and dissertations to the database, making them freely available to researchers everywhere.

GreenFILE is a free research database covering all aspects of human impact to the environment. Its collection of scholarly, government and general-interest titles includes content on global warming, green building, pollution, sustainable agriculture, renewable energy, recycling, and more. 

Library, Information Science and Technology Abstracts

Library, Information Science & Technology Abstracts (LISTA) is a free research database for library and information science studies. LISTA provides indexing and abstracting for hundreds of key journals, books, research reports. It is EBSCO's intention to provide access to this resource on a continual basis.

Teacher Reference Center

A complimentary research database for teachers,  Teacher Reference Center  (TRC) provides indexing and abstracts for more than 230 peer-reviewed journals.

European Views of the Americas: 1493 to 1750

European Views of the Americas: 1493 to 1750 is a free archive of indexed publications related to the Americas and written in Europe before 1750. It includes thousands of valuable primary source records covering the history of European exploration as well as portrayals of Native American peoples.

Recommended Reading

• Harvard Library
• Research Guides
• HKS Library & Research Services

Beyond Harvard: Library Resources for HKS Alumni

• Free Resources
• Using the HKS Library as an Alum

Introduction

Hks library research guides, articles & books, policy research, open source software.

• Maximize Your Searching
• Other Libraries

License agreements and copyright laws prevent Harvard libraries from offering alumni complete access to online resources. There is, however, a large and growing body of resources available to everyone, thanks to library digitization projects, the open access movement, and other efforts.

Our 40+ expertly curated Research Guides are freely available online. These guides offer subject-specific resources on topics like:

• COVID-19 resources for social scientists
• Campaigns & Elections
• Human Rights
• National & International Security
• Diversity, Equity & Inclusion

If a resource listed in our guides doesn't specify "Harvard Login," it is likely freely available online.

Zotero is a free and open-source citation management software that allows you to manage your research materials, create bibliographies, format citations, take notes, and collaborate with others.

You can download Zotero even after graduation, and refer to Harvard Library's online guide for help.

All the content you've added to Zotero before graduation will stay in your account. Any new content you add after your HKS email becomes inactive will be subject to Zotero's 300MB free limit. You can purchase more storage online.

Make sure to change your email!

Your student HKS email account and address will remain active for one year after your status is officially changed to Graduate in Harvard's systems. Before this happens, add a personal email address to your Zotero account so you don't get locked out of your content.

Login to Zotero online .

Click on your username in the upper right corner, then select "Settings" from the drop-down.

Click on the "Account" tab.

Under "Add email address," add your personal email address.

Once your HKS email becomes inactive, delete that address from this page. Your personal email will become the primary one associated with your Zotero account.

Many online platforms provide free access to articles, citations, and even full-text books.

• Developing Nations Initiative List of free or low cost scholarly literature access programs for residents of developing countries. Eligibility criteria vary by program.
• Digital Access to Scholarship at Harvard (DASH) DASH is Harvard's central open-access repository. It hosts scholarly work by HKS affiliates including faculty, Centers, and students, and other Harvard community members.
• Digital Commons Network Access to 600,000 publications from almost 300 institutions. Search by keyword or topic.
• Digital Public Library of America Online books, maps, photos, and much more from libraries, archives, and museums, freely available to all.
• Directory of Open Access Books Voted one of the best free reference sites by an ALA reference librarian group, this site enables the user to find free, open access books online.
• Directory of Open Access Journals Directory of free, full text, scientific and scholarly journals. Currently contains over 7,500 titles.
• EconBiz Provides citations and some full-text access to articles on business, management, and economics topics in (mainly) scholarly journals; since EconBiz is edited in Germany, it offers excellent coverage of European publications.
• ERIC A wide variety of journal and non-journal literature in the field of education. Not all material is available in full text.
• Google Scholar Search scholarly literature across many disciplines and sources, including legal cases. Not all content in this search is freely available.
• HathiTrust A shared digital repository created by major research libraries. Features millions of digitized texts, some freely available in full-text, some limited to search-only.
• Internet Archive Digitized books, journals, audio, video, and more, all free. In addition to contemporary texts, includes a wealth of digitized historical material.
• Journalist's Resource A project of the Shorenstein Center, this site surfaces scholarly materials that may be relevant to media practitioners, bloggers, educators, students and general readers. Some of their articles outline how journalists can access academic research for free.
• National Academies Press Collection of over 5,000 academic titles available to read online at no cost.
• NBER: National Bureau of Economic Research Free access to working papers 18 months and older.
• Open Access Theses and Dissertations Resource for locating open access graduate theses and dissertations published around the world. more... less... This is an index of over 1.6 million electronic theses and dissertations (ETDs). To the extent possible, the index is limited to records of graduate-level theses that are freely available online.
• OpenDOAR OpenDOAR is an authoritative directory of academic open access repositories. Repositories can be located by subject and location. The site also links to a customized Google search of the open access repositories included in this directory.
• PolicyArchive An open access repository of public policy research by think tanks, NGOs, CRS, and university centers.
• Project Gutenberg Over 49,000 free books available in a variety of downloadable formats.
• PubMed Central (PMC) Repository of articles produced with the support of NIH research grants and made publically available within 12 months of publication. Not all material is available in full text.
• Rand Corporation Publications dating from 1946. Topics include criminal justice, national security, terrorism, international affairs, healthcare and energy.
• RePEc: Research Papers in Economics Provides links to working papers and journal articles in economics. Try the IDEAS service.
• SSRN: Social Science Research Network Contains working papers in social science, law and business/economics fields. Registration is required.
• Open Access Button

Think Tank Search

Think Tank Search searches the websites of institutions that generate public policy research, analysis, and activity. These sites are affiliated with universities, governments, advocacy groups, foundations, and non-governmental organizations. Inclusion is based upon the relevancy of subject area to HKS coursework and scholarship, the availability of the think tank’s research in full-text on the website, and the think tank’s reputation and influence upon policy making. The list represents a mixture of partisan and non-partisan think tanks.

Other Policy Resources

• Policy Commons: Global Think Tanks Collection of research from the world’s leading policy experts, think tanks, IGOs and NGOs. At last count it contains over 3 million publications from more than 24,000 organizations.
• Open Research Reports from JSTOR More than 39,000 research reports from over 140 policy institutes around the world are freely accessible to everyone on JSTOR. The open research reports are discoverable alongside journals, books, and primary sources, and are clearly labeled as their own content type.
• Find Policy A side project of Transparify, search think tank sites grouped by topic and location.
• Open Think Tank Directory A global collection of 2700 think tanks and related organisations.

Repository of think tank publications on EU affairs from the Library of the General Secretariat of the Council of the EU.

Open source software is distributed with its source code, making it freely available for use. There are open-source alternatives to many of the commercial software tools Harvard licenses.

Adobe Creative Cloud Alternatives

Pdf editing (adobe acrobat dc).

• LibreOffice Draw

Visual Effects & Motion Graphics (Adobe After Effects)

• Synfig Studio

Digital Audio (Adobe Audition)

Vector graphics (adobe illustrator), desktop publishing (adobe indesign).

• Microsoft Publisher Not open source, but included in Office 365 subscriptions.

Transcoding (Adobe Media Encoder)

• MLT Framework
• FFmpeg Requires knowledge of command line interface.

Raster Graphics (Adobe Photoshop)

Video editing (adobe premiere pro), arcgis alternative.

• QGIS All the same functionality as ArcGIS in an open-source tool.

NVivo Alternatives

NVivo offers a free 14-day trial and 24-month subscriptions, with discounts available for students. Other free and less expensive options include:

• Dedoose Commercial alternative to NVivo. Offers 30-day trial then payments only per active month. Discounts available.
• qdap R package for qualitative data analysis.
• RQDA R package for qualitative data analysis.
• ELAN Originally developed for linguistics rather than qualitative social science research.
• Taguette Text tagging.

Tableau Alternatives

Tableau Desktop offers a free one-year license for students . Other alternatives include:

• Tableau Public
• Datawrapper Decent amount of features but visualizations are watermarked.
• Google Data Studio Primarily for building dashboards.

Screencasting

Text mining & natural language processing.

• Quanteda R package for textual analysis and natural language processing.
• Voyant Less powerful than Quanteda but easier to learn.
• << Previous: Using the HKS Library as an Alum
• Next: Maximize Your Searching >>
• Last Updated: May 3, 2024 5:27 PM
• URL: https://guides.library.harvard.edu/hks/alumni

Harvard University Digital Accessibility Policy

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• 30 May 2024

Japan’s push to make all research open access is taking shape

• Dalmeet Singh Chawla 0

Dalmeet Singh Chawla is a freelance science journalist based in London.

You can also search for this author in PubMed   Google Scholar

Japan plans to make all publicly funded research available to read in institutional repositories. Credit: Toru Yamanaka/AFP via Getty

The Japanese government is pushing ahead with a plan to make Japan’s publicly funded research output free to read. In June, the science ministry will assign funding to universities to build the infrastructure needed to make research papers free to read on a national scale. The move follows the ministry’s announcement in February that researchers who receive government funding will be required to make their papers freely available to read on the institutional repositories from January 2025.

The Japanese plan “is expected to enhance the long-term traceability of research information, facilitate secondary research and promote collaboration”, says Kazuki Ide, a health-sciences and public-policy scholar at Osaka University in Suita, Japan, who has written about open access in Japan .

The nation is one of the first Asian countries to make notable advances towards making more research open access (OA) and among the first countries in the world to forge a nationwide plan for OA.

The plan follows in the footsteps of the influential Plan S, introduced six years ago by a group of research funders in the United States and Europe known as cOAlition S , to accelerate the move to OA publishing . The United States also implemented an OA mandate in 2022 that requires all research funded by US taxpayers to be freely available from 2026.

Institutional repositories

When the Ministry of Education, Culture, Sports, Science and Technology (MEXT) announced Japan’s pivot to OA in February, it also said that it would invest ¥10 billion (around US$63 million) to standardize institutional repositories — websites dedicated to hosting scientific papers, their underlying data and other materials — ensuring that there will be a mechanism for making research in Japan open.

Among the roughly 800 universities in Japan, more than 750 already have an institutional repository, says Shimasaki Seiichi, director of the Office for Nuclear Fuel Cycles and Decommissioning at MEXT in Tokyo, who was involved with drawing up the plan. Each university will host the research produced by its academics, but the underlying software will be the same.

In 2022, Japan also launched its own national preprint server, Jxiv , but its use remains limited, with only a few hundred preprint articles posted on the platform so far. Ide says that publishing as preprints is not yet habitual for many researchers in Japan, noting that only around one in five respondents to his 2023 survey 1 on Jxiv were even aware that it existed.

Japan’s move to greater access to its research is focusing on ‘green OA’ — in which authors make the author-accepted, but unfinalized, versions of papers available in the digital repositories, says Seiichi.

Seiichi says that gold OA — in which the final copyedited and polished version of a paper is made freely available on the journal site — is not feasible on a wide scale. That’s because the cost to make every paper free to read would be too high for universities. Publishers levy an article-processing charge (APC) if the paper is made free to read, rather than being paywalled, a fee that covers a publisher’s costs.

APCs are increasing at an average rate of 4.3% per year, notes Johan Rooryck, a scholar of French linguistics at Leiden University in the Netherlands, and executive director of cOAlition S.

Rooryck says that Japan’s green OA strategy should be applauded. “It’s definitely something that one should do,” he says. “Especially for all the content that is still behind the paywall.”

Kathleen Shearer, executive director of the Confederation of Open Access Repositories in Montreal, Canada, says that the Japanese plan is “equitable”.

“It doesn’t matter where you publish, whether you have APCs or not, you are still able to comply with an open-access policy,” she says.

She adds that the policy will mean that Japan has a unified record of all research produced by its academics, because all institutional repositories are hosted on the same national server. “Japan is way ahead of the rest of us,” Shearer says. “More countries are moving in this direction but Japan really was one of the first.”

Focusing on institutional repositories will have another benefit: it will not discriminate against research published in Japanese, Shearer says. “A big part of their scholarly ecosystem is represented in Japanese.”

Japanese research is no longer world class — here’s why

The plan to move to OA and support Japanese universities’ repositories comes as Japan grapples with its declining standing in international research.

In a report released last October, MEXT found that Japan’s world-class research status is declining . For instance, Japan’s share in the top 10% of most-cited papers has dipped from 6% to 2%, placing it 13th on the list of nations, despite Japan having the 5th-highest research output.

In March, Japan vowed to triple its number of doctorate holders by 2040, after another report found that the country’s number of PhD graduates is also declining, making it an outlier among the major economies.

doi: https://doi.org/10.1038/d41586-024-01493-8

Ide, K. & Nakayama, J.-I. Genes Cells 28 , 333–337 (2023).

Article   PubMed   Google Scholar  

Download references

Reprints and permissions

Related Articles

• Scientific community
• Research data
• Intellectual-property rights
• Institutions

How I run a virtual lab group that’s collaborative, inclusive and productive

Career Column 31 MAY 24

Biomedical paper retractions have quadrupled in 20 years — why?

News 31 MAY 24

What is science? Tech heavyweights brawl over definition

The AI revolution is coming to robots: how will it change them?

News Feature 28 MAY 24

Ozempic keeps wowing: trial data show benefits for kidney disease

News 24 MAY 24

Guidelines for academics aim to lessen ethical pitfalls in generative-AI use

Nature Index 22 MAY 24

Mexico’s next president is likely to be this scientist — but researchers are split in their support

News 30 MAY 24

Adopt stricter regulation to stop ‘critical mineral’ greenwashing

Correspondence 28 MAY 24

Risks of bridge collapses are real and set to rise — here’s why

Comment 28 MAY 24

Chief Editor

Job Title:  Chief Editor Organisation: Nature Ecology & Evolution Location: New York, Berlin or Heidelberg - Hybrid working Closing date: June 23rd...

New York City, New York (US)

Springer Nature Ltd

Global Talent Recruitment (Scientist Positions)

Global Talent Gathering for Innovation, Changping Laboratory Recruiting Overseas High-Level Talents.

Beijing, China

Changping Laboratory

Postdoctoral Associate - Amyloid Strain Differences in Alzheimer's Disease

Houston, Texas (US)

Baylor College of Medicine (BCM)

Postdoctoral Associate- Bioinformatics of Alzheimer's disease

Postdoctoral associate- alzheimer's gene therapy.

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

The problem with making all academic research free

T here has been an earthquake in my corner of academia that will affect who teaches in prestigious universities and what ideas circulate among educated people around the world.

And it all happened because a concept rooted in good intentions — that academic research should be “open access,” free for everyone to read — has started to go too far.

The premise of open-access publishing is simple and attractive. It can cost libraries thousands of dollars a year to subscribe to academic journals, which sometimes means only academics affiliated with wealthy colleges and universities may access that research. But under open-access publishing, nearly anyone with an internet connection can find and read those articles for free. Authors win, because they find more readers. Academics around the world benefit, because they can access the latest scholarship. And the world wins, because scientific and intellectual progress is facilitated by the free exchange of ideas.

By now this model has taken hold in the natural sciences, especially in biology and biomedicine; during the pandemic many publishers removed paywalls from articles about vaccines and treatments. The Biden administration requires federally funded scholarly publications to be made freely available without any delay.

However, there is no such thing as a free academic article. Even with digital distribution, the expenses of running a journal are considerable. These costs include hosting the websites where people submit, peer-review, and edit articles; copyediting; advertising; preserving journal archives; and maintaining continuity as editors come and go.

As a result, unless journals have a source of revenue other than subscription fees, any move toward open access raises the question of who will cover the costs of publication.

One answer is that the money will come from authors themselves or their academic institutions or other backers. This works well enough in the natural sciences, because those researchers are often funded by grants, and some of that money can be set aside to cover a journal’s fees for publishing scientific articles. The Bill and Melinda Gates Foundation demands that all research funded by the foundation, including the underlying data, be published open access.

According to an MIT study , however, only a small fraction of scholars in the humanities publish their articles on an open-access basis. Unlike biologists and biomedical engineers, humanities scholars such as philosophers and historians do not get grants that can cover the publishing costs.

This means that if open access is to take hold in those fields as well — as many publishers and academics are advocating — the costs will have to be covered by some foundation or other sponsor, by the scholars’ institutions, or even by the scholars themselves. And all these models have serious downsides.

I’m a political philosopher. The earthquake in my field that I mentioned earlier shook one of our most prominent journals: the Journal of Political Philosophy.

Publishing an article in this journal has long made the difference between whether a candidate gets hired, tenured, or promoted at an elite institution of higher education. The high quality has stemmed in large part from the rigorous approach of the founding editor, Robert Goodin.

At the end of 2023, the publisher, Wiley, terminated its contract with Goodin. The reasons were not immediately clear, and over 1,000 academics, including me, signed a petition stating that we would not serve on the editorial board or write or review for the journal until Wiley reinstates Goodin. I recently attended a panel at an American Philosophical Association conference where philosophers voiced their anger and puzzlement about the situation.

One source of the problem appears to be that Wiley now charges the authors of an article or their institutions $3,840 to get published open access in the journal.

The Journal of Political Philosophy is actually hybrid open access, which means it waives the article processing charges for authors who permit their work to appear behind a subscription-only paywall. Nonetheless, Goodin and Anna Stilz , a Princeton professor and Journal of Political Philosophy editorial board member, point out that publishers like Wiley now have a strong incentive to favor open-access articles.

In the old model, in which university libraries subscribed to journals, editors were mainly incentivized to publish first-rate material that would increase subscriptions. In the open-access model, however, now that authors or their universities must cover the costs of processing articles, publishers of humanities journals seem to be incentivized to boost revenue by accepting as many articles as possible. According to Goodin , open access has “been the death knell of quality academic publishing.” The reason that Goodin lost his job, Goodin and Stilz imply, is that Wiley pressured Goodin to accept more articles to increase Wiley’s profits, and he said no. (Wiley representatives say that lines of communication had collapsed with Goodin.)

Early this year, Goodin cofounded a new journal titled simply Political Philosophy . The journal will be published by the Open Library of Humanities, which is subsidized by libraries and institutions around the world. But this version of open-access publishing does not have the financial stability of the old subscription model. Scholars affiliated with the Open Library of Humanities have pointed out that the project has substantial overhead costs, and it relied on a grant from the Andrew W. Mellon Foundation that has already ended. The Open Library of Humanities is an experiment, and I hope that it works, but as of now it publishes only 30 journals , compared with the 1,600 journals that Wiley publishes.

The fact remains that no one has satisfactorily explained how open access could work in the humanities and social sciences.

In his 2023 book “ Athena Unbound : Why and How Scholarly Knowledge Should Be Free for All,” UCLA history professor Peter Baldwin attempts an answer. He points to Latin America, where some national governments cover all expenses of academic publishing. But this proposal ignores the fact that the governments of the United States and other nations probably do not want to pay for humanities and social sciences journals.

Baldwin also floats the idea of preprint depositories where academics could share documents on the cloud before they have undergone the (somewhat expensive) process of peer review. But this means that academics would lose the benefits that come from getting double-blind feedback from one’s peers. This idea would reduce the costs of publishing a journal article, but it would turn much academic writing into fancy blogging.

Ultimately, Baldwin’s solution is that authors might “have to participate directly, giving them skin in the game and helping contain costs.” This means academics might ask their employers to pay the article processing charges, ask a journal for the processing fees to be waived, or dig into their own pockets to pay to publish.

And it might mean less gets published overall. The journal Government and Opposition, published by Cambridge University Press, is entirely open access and charges $3,450 for an article to be published. I’d have to apply for a discount or a waiver to publish there. Or I could do what political philosophers in Japan and Bosnia and Herzegovina have told me they do: avoid submitting to open-access journals. Their universities will not cover their article processing charges except maybe in the top journals, and even the reduced fees can run into hundreds of dollars that these professors do not have.

In “Athena Unbound,” Baldwin notes that Harvard subscribes to 10 times as many periodicals as India’s Institute of Science. One can bemoan this fact, but one may also appreciate that Harvard’s largesse spread enough subscription revenue around to reputable journals to enable academics to avoid paying to publish in them, no matter whether they teach at regional state schools, non-elite private schools, or institutions of higher education in poor countries. For all its flaws, the old model meant that when rich alumni donated to their alma maters, it increased library budgets and thereby made it possible for scholars of poetry and state politics to run and publish in academic journals.

Until we have more evidence that open-access journals in the humanities and social sciences can thrive in the long run, academics need to appreciate the advantages of the subscription model.

Nicholas Tampio is a professor of political science at Fordham University in New York City.

Fats and Cholesterol

When it comes to dietary fat, what matters most is the type of fat you eat. Contrary to past dietary advice promoting low-fat diets , newer research shows that healthy fats are necessary and beneficial for health.

• When food manufacturers reduce fat, they often replace it with carbohydrates from sugar, refined grains, or other starches. Our bodies digest these refined carbohydrates and starches very quickly, affecting blood sugar and insulin levels and possibly resulting in weight gain and disease. ( 1-3 )
• Findings from the Nurses’ Health Study ( 4 ) and the Health Professionals Follow-up Study ( 5 ) show that no link between the overall percentage of calories from fat and any important health outcome, including cancer, heart disease, and weight gain.

Rather than adopting a low-fat diet, it’s more important to focus on eating beneficial “good” fats and avoiding harmful “bad” fats. Fat is an important part of a healthy diet. Choose foods with “good” unsaturated fats, limit foods high in saturated fat, and avoid “bad” trans fat.

• “Good” unsaturated fats — Monounsaturated and polyunsaturated fats — lower disease risk. Foods high in good fats include vegetable oils (such as olive, canola, sunflower, soy, and corn), nuts, seeds, and fish.
• “Bad” fats — trans fats — increase disease risk, even when eaten in small quantities. Foods containing trans fats are primarily in processed foods made with trans fat from partially hydrogenated oil. Fortunately, trans fats have been eliminated from many of these foods.
• Saturated fats , while not as harmful as trans fats, by comparison with unsaturated fats negatively impact health and are best consumed in moderation. Foods containing large amounts of saturated fat include red meat, butter, cheese, and ice cream. Some plant-based fats like coconut oil and palm oil are also rich in saturated fat.
• When you cut back on foods like red meat and butter, replace them with fish, beans, nuts, and healthy oils instead of refined carbohydrates.

Read more about healthy fats in this “Ask the Expert” with HSPH’s Dr. Walter Willett and Amy Myrdal Miller, M.S., R.D., formerly of The Culinary Institute of America

1. Siri-Tarino, P.W., et al., Saturated fatty acids and risk of coronary heart disease: modulation by replacement nutrients. Curr Atheroscler Rep, 2010. 12(6): p. 384-90.

2. Hu, F.B., Are refined carbohydrates worse than saturated fat? Am J Clin Nutr, 2010. 91(6): p. 1541-2.

3. Jakobsen, M.U., et al., Intake of carbohydrates compared with intake of saturated fatty acids and risk of myocardial infarction: importance of the glycemic index. Am J Clin Nutr, 2010. 91(6): p. 1764-8.

4. Hu, F.B., et al., Dietary fat intake and the risk of coronary heart disease in women. N Engl J Med, 1997. 337(21): p. 1491-9.

5. Ascherio, A., et al., Dietary fat and risk of coronary heart disease in men: cohort follow up study in the United States. BMJ, 1996. 313(7049): p. 84-90.

6. Hu, F.B., J.E. Manson, and W.C. Willett, Types of dietary fat and risk of coronary heart disease: a critical review. J Am Coll Nutr, 2001. 20(1): p. 5-19.

Terms of Use

The contents of this website are for educational purposes and are not intended to offer personal medical advice. You should seek the advice of your physician or other qualified health provider with any questions you may have regarding a medical condition. Never disregard professional medical advice or delay in seeking it because of something you have read on this website. The Nutrition Source does not recommend or endorse any products.

This paper is in the following e-collection/theme issue:

Published on 31.5.2024 in Vol 8 (2024)

Controlling Inputter Variability in Vignette Studies Assessing Web-Based Symptom Checkers: Evaluation of Current Practice and Recommendations for Isolated Accuracy Metrics

Authors of this article:

Original Paper

• András Meczner 1, 2 , MSc, MD   ; 
• Nathan Cohen 1 , BSc, MBBS, MSc   ; 
• Aleem Qureshi 1 , BSc, MBBS   ; 
• Maria Reza 1 , BSc, MBBS   ; 
• Shailen Sutaria, MBBS, BMedSci, MSc   ; 
• Emily Blount 1 , MBBS   ; 
• Zsolt Bagyura 2 , MD, PhD   ; 
• Tamer Malak 1 , BM, MSc, DPhil  

1 Healthily, London, United Kingdom

2 Institute for Clinical Data Management, Semmelweis University, Budapest, Hungary

Corresponding Author:

András Meczner, MSc, MD

167-169 Great Portland Street

London, W1W 5PF

United Kingdom

Email: [email protected]

Background: The rapid growth of web-based symptom checkers (SCs) is not matched by advances in quality assurance. Currently, there are no widely accepted criteria assessing SCs’ performance. Vignette studies are widely used to evaluate SCs, measuring the accuracy of outcome. Accuracy behaves as a composite metric as it is affected by a number of individual SC- and tester-dependent factors. In contrast to clinical studies, vignette studies have a small number of testers. Hence, measuring accuracy alone in vignette studies may not provide a reliable assessment of performance due to tester variability.

Objective: This study aims to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. It further aims to investigate the feasibility of measuring isolated aspects of performance.

Methods: Healthily’s SC was assessed using 114 vignettes by 3 groups of 3 testers who processed vignettes with different instructions: free interpretation of vignettes (free testers), specified chief complaints (partially free testers), and specified chief complaints with strict instruction for answering additional symptoms (restricted testers). κ statistics were calculated to assess agreement of top outcome condition and recommended triage. Crude and adjusted accuracy was measured against a gold standard. Adjusted accuracy was calculated using only results of consultations identical to the vignette, following a review and selection process. A feasibility study for assessing symptom comprehension of SCs was performed using different variations of 51 chief complaints across 3 SCs.

Results: Intertester agreement of most likely condition and triage was, respectively, 0.49 and 0.51 for the free tester group, 0.66 and 0.66 for the partially free group, and 0.72 and 0.71 for the restricted group. For the restricted group, accuracy ranged from 43.9% to 57% for individual testers, averaging 50.6% (SD 5.35%). Adjusted accuracy was 56.1%. Assessing symptom comprehension was feasible for all 3 SCs. Comprehension scores ranged from 52.9% and 68%.

Conclusions: We demonstrated that by improving standardization of the vignette testing process, there is a significant improvement in the agreement of outcome between testers. However, significant variability remained due to uncontrollable tester-dependent factors, reflected by varying outcome accuracy. Tester-dependent factors, combined with a small number of testers, limit the reliability and generalizability of outcome accuracy when used as a composite measure in vignette studies. Measuring and reporting different aspects of SC performance in isolation provides a more reliable assessment of SC performance. We developed an adjusted accuracy measure using a review and selection process to assess data algorithm quality. In addition, we demonstrated that symptom comprehension with different input methods can be feasibly compared. Future studies reporting accuracy need to apply vignette testing standardization and isolated metrics.

Introduction

Web-based symptom checkers (SCs) are tools for lay people to assess their symptoms using smartphones, tablets, and computers, providing possible conditions or triage or both. SC triage determines if there is a need to seek medical advice and its degree of urgency. The number of SCs is growing rapidly with generative artificial intelligence (AI) models gaining widespread popularity and being evaluated for their potential use as SCs [ 1 ]. Regulatory bodies often categorize SC as low-risk medical devices requiring only self-certification before introduction in the open market [ 2 ]. Evaluation of SCs is crucial; incorrect advice on triage or condition outcomes may result in patient harm [ 3 ]. Currently, there are no widely accepted criteria to assess their performance.

The 2 most frequently used methods of assessing SC performance are clinical studies and simulated patient studies via vignettes [ 2 ].

The Food and Drugs Administration and European Union Medical Device Regulation strongly recommend studies involving real patients to evaluate the performance of any medical device before its introduction in the market [ 4 , 5 ]. However, clinical trials have limitations. The cost can be prohibitive [ 2 ] and often only small population samples are feasible, limiting generalizability. In addition, these studies are conducted in a health care setting [ 6 , 7 ] and may not represent the population of individuals using an SC, who often use them before deciding whether to seek medical advice. Finally, the need for repeated assessments with every iteration of development might limit the feasibility of using real-patient studies.

Therefore, vignette studies have gained popularity. Vignettes are clinical scenarios described in a few sentences in accordance with a specific disease or differential list of diseases. Vignette studies offer the opportunity to assess multiple SCs simultaneously with a range of diseases and patient presentations within a single study at relatively lower cost [ 8 ]. However, they have several ontological, methodological, and epistemological limitations [ 9 ]. These include issues concerning the study of a rapidly evolving field; the creation of appropriate vignettes that are representative of real cases, populations, and health care settings; definition of gold standards; and trade-off between simple and complex or ambiguous cases [ 9 - 12 ].

Accuracy as an Outcome

Most studies evaluate SC performance by measuring the accuracy of most likely conditions (single or multiple) or triage outcome determined by health care professionals [ 13 - 15 ]. Accuracy of the outcome behaves as a composite measure as it is affected by a number of individual SC- and tester-dependent factors. Some of these tester-dependent factors are only specific to the vignette methodology.

SC-Dependent Factors

SCs vary in their database, algorithm, symptom expression, and comprehension of chief complaints (ie, initial symptoms of a consultation mimicking a real patient’s presenting complaints) [ 16 - 18 ]. Any of these may impact the levels of outcome accuracy.

General Tester-Dependent Factors

Both real-life users and testers are likely to vary in their ability to express their symptoms and to comprehend the ones offered by the SC [ 19 ]. Misinterpretation of symptoms offered by the SC during the consultation may result in the addition of symptoms the user does not have and neglect the ones that they do have in reality. Moreover, how users or testers express the chief complaints can influence the outcomes. Even an SC with the best comprehension abilities will fail if the user expresses their symptoms incorrectly.

Tester-Dependent Factors Specific to Vignette Studies

Some of the general limitations of vignette studies are reported more often in the literature and have been mentioned in previous sections. However, some other tester-dependent limitations specific to vignette methodology measuring the accuracy of outcome are less explored.

Several vignette studies use vignettes that do not define chief complaints [ 13 , 20 ], leaving it to the discretion of the individual testers to select and interpret them. Appropriate determination of the chief complaints is an important aspect. A number of SCs provide different weighting to them compared to symptoms selected during the consultation. Thus, testers by selection of chief complaints can substantially influence the final outcome, including limiting the potential differentials and subsequent symptoms asked during the consultation [ 21 ]. Other studies have opted for an approach, where the chief complaints are prescribed [ 14 , 22 ]. A limitation of most of these studies is that the number of chief complaints was reduced to a single symptom. Therefore, these studies may not represent the complexity of patients presenting with multiple or ill-defined symptoms.

The other limitation of testers in vignette studies is caused by the extra step whereby testers have to understand and translate the vignettes for the SC. Their abilities to convey the intention of the vignette can influence the outcome.

Hypothesis and Aims

We hypothesize that vignette studies measuring SC’s accuracy of outcome as a composite metric is not a reliable method to determine the performance and their result might not be generalizable to a wider population. In clinical studies with a large, diverse population, a wide range of symptom expression and comprehension ability is represented, which is similar to real life. However, in vignette studies, with few testers, the testers’ ability to input the chief complaints as intended or to select the right symptoms throughout the consultation can unduly influence results despite using the same SCs and clinical vignettes. This is why some studies have reported significant tester variability [ 20 ].

The primary aim of this study was to investigate the impact of tester variability on the accuracy of outcome of SCs, using clinical vignettes. The secondary aim was to identify the methodology to measure isolated aspects of SC performance using clinical vignettes. We investigated the following two isolated aspects: (1) SC data and algorithm quality using adjusted accuracy metrics and (2) SC symptom comprehension.

Clinical Vignettes

The Royal College of General Practitioners in the United Kingdom produced 139 vignettes to evaluate an SC in a benchmarking study conducted by the Self-Care Academic Research Unit of Imperial College London [ 20 ]. A total of 25 vignettes were excluded because they described asymptomatic individuals or individuals with long-term conditions that had already been diagnosed or because they lacked agreed-upon outcomes. The remaining 114 vignettes were used (minor alterations were performed to enhance the clarity of symptoms in 5 vignettes; Table S1 in Multimedia Appendix 1 ).

We included 7 testers who were not employed by Healthily and were remunerated on an hourly basis. They were blinded to the real aims of the study. Nonmedical testers, who were either in university or had completed their education, were used for the inter- and intra-agreement analyses. The tester analyzing the comprehension of the SC was a medical student.

Inter- and Intratester Agreement

This study consisted of 3 experimental phases to analyze the difference between inter- and intra-agreement of testers with differing testing instructions. Consultations were conducted by 3 testers using clinical vignettes with different instructions for each phase. The different instructions given for the different phases can be found in Multimedia Appendix 2 .

Each tester was given the set of vignettes to input the data points (ie, gender, age, duration of complaints, symptoms present, and comorbidities) into Healthily AI Smart Symptom Checker (Healthily SC) via the website application [ 23 ]. The testers recorded every data point of the consultation. The 3 test phases were conducted using the OSC versions from June 2022, February 2023, and April 2023, respectively. The interagreement of testers was investigated in each phase. For each vignette, the most likely outcome condition provided by the SC was translated to a numerical value and compared to assess agreement. A score of 0 was assigned to cases where no condition was found as an outcome. Triage was evaluated using a rating system comprising 9 categories (ie, no triage, see a physician, self-limiting, self-care, routine, urgent—within 48 hours, urgent—within 12 hours, emergency and accident department and emergency ambulance).

Phase 1: Baseline

In phase 1, testers were free to select the chief complaints from the vignettes using their own interpretation without any imposed restrictions ( free tester group ). Intertester agreement was calculated for inputted chief complaints and consultation outcome. In addition, intratester agreement was also calculated following a second round where the same testers repeated vignette inputs with the same instructions approximately 6 months later.

For the agreement of input three investigators compared the chief complaints of the testers for similarity in 2 aspects: the exact wording and clinical concept. Exact wording was defined when 2 input symptom matched word for word. Clinical concept was defined when clinical symptoms matched using synonyms.

Phase 2: Restriction of Only Chief Complaints

Phase 2 investigated the effect of the chief complaints on the outcome by predefining the chief complaints for each vignette (partially free tester group). These chief complaints were prepared by 3 medical qualified doctors selecting single or multiple symptoms from each vignette and transforming them into unambiguous plain English terms. Agreement for most likely condition and triage was calculated.

Phase 3: Restriction of All Symptoms

Phase 3 involved further restrictions to investigate the effect of the other vignette study–specific tester-dependent factor: symptom translation from the vignette for the SC. Along with the prescribed chief complaints, strict instructions were given to each tester regarding the additional symptoms offered by the SCs (restricted tester group). Each tester was instructed to decline every symptom that was not specifically written in the vignette except for those that were synonyms or part of a wider logical category. For example, if the vignette described “pain in knee,” the tester would select “pain in leg” but decline a symptom called “pain on walking,” unless it is specified in the vignette explicitly. Agreement for the most likely condition and triage was calculated.

The effect of variability between testers was evaluated by investigating the difference in the accuracy between the testers and groups. Accuracy was measured as the most likely condition outcome of the consultation matching the imperial gold standard [ 20 ]. The accuracy for each tester was calculated and then averaged for an overall measure in each group.

Review and Selection Process for the Development of an Isolated Metric

In the restricted group, the presence of residual variability between tester outcomes and a difference between the highest and lowest performing testers even with the restrictions triggered the implementation of a review and selection process. The aim was to understand the source of variability and to develop an adjusted accuracy measure to assess data and algorithm quality in isolation.

Cases of the restricted group were reviewed by 2 researchers. First, for each case, all the data points from the consultation report for each tester, including symptom duration, comorbidities, age, gender, and symptoms that were selected or declined, were compared to the original vignette. Discrepancies from the initial vignettes were analyzed (eg, missing symptoms, addition of symptoms not described in the vignette, misinterpretation of symptoms, and selection of incorrect duration). Second, we selected tester consultations where no discrepancy was present. Descriptive analysis was used to describe the proportion of cases where consultations were fully identical to the vignettes stratified by matching outcomes between testers. Finally, an adjusted accuracy was calculated by (1) excluding cases where none of the 3 tester’s consultations were identical to the vignette’s data points (Accuracy Excluded ) and (2) including results of a retested consultation for those cases correcting the discrepancies between the vignette and the initially performed consultations (Accuracy Retested ).

SC Comprehension

A feasibility study was conducted in March 2023 to evaluate the symptom comprehension of SCs in isolation. We defined symptom comprehension as the ability of the SC to understand the intended chief complaints inputted by the tester.

A total of 29 (25.4%) vignettes were randomly selected from the 114 vignettes previously used. The chief complaints extracted during phase 2 were used. Furthermore, 3 synonyms of these chief complaints were created for a natural language processing (NLP) input method and 3 for a drop-down menu method (refer to the list of all symptoms in Table S2 in Multimedia Appendix 1 ). NLP is a combination of methods based on linguistics, computer science, and AI, allowing computers to interpret and comprehend human language from written text [ 24 , 25 ].

A single tester inputted all the symptoms across 3 SCs: 1 using free text NLP (Healthily) and 2 using drop-down or search menus (Ada Health App [by Ada Health GmbH] and Infermedica triage [by Infermedica]) [ 23 , 26 , 27 ]. In one of the latter SCs, only 1 symptom can be inputted at a time, while in the other one, multiple symptoms can be inputted at once. The NLP SC has a drop-down option in the subsequent step after the NLP input, which the tester was allowed to use.

The tester had strict instructions to input the exact wording for the chief complaints and documented the response of the SCs.

Both NLP and drop-down menu inputs were scored in a similar fashion. A score of 2 was given when the exact symptom was detected or the same meaning was conveyed. For example, if the input was “lower tummy pain” and this was translated to “lower abdominal pain,” it scored 2. For extraction or offering of a wider logical category symptom, which would later allow other symptoms to be asked about later, a score of 1 was assigned. For example, if “lower tummy pain” was inputted and “abdominal pain” was extracted, it would score 1. An incorrect symptom, for example, if “passing too much urine” was the chief complaints and only “burning on passing urine” was shown, would score 0. The final scores were turned into probabilities.

Statistical Analysis

Fleiss and Cohen κ using Stata 13 SE (StataCorp) and Package Kappaetc and proportion of agreement were measured [ 28 , 29 ]. The κ value was classified as per Landis and Koch [ 30 ] into the following groups: 0.00 and 0.20 as “slight,” between 0.21 and 0.40 as “fair,” between 0.41 and 0.60 as “moderate,” between 0.61 and 0.80 as “substantial,” and between 0.81 and 1.00 as “almost perfect.”

Descriptive statistics were used to analyze the accuracy of the most likely condition and the performance of symptom comprehension. Inferential statistics was used to compare accuracy between tester groups (ANOVA for all 3 groups and student 2-tailed t test to compare partially free and restrictive groups independently to the free group).

Ethical Considerations

The study was reviewed by the Semmelweis University’s Institutional Review Board and determined to be Institutional Review Board exempt. The study did not involve the recruitment of patients or participants or use patient-identifiable data. Healthily holds the rights to the data. The testers were contracted to perform the task and were renumerated on an hourly basis (£15/hour; US $19).

Characteristics of vignettes including demographics, duration of symptoms, expected triage categories, and the medical domains can be found in the Multimedia Appendix 3 .

Testers were aged between 21 and 39 years, and 33% (2/6) were female. More details on their demographics and which phase they participated in can be found in Multimedia Appendix 4 .

Intertester agreement (κ) for the most likely condition (κ=0.49) and triage (κ=0.51) were “moderate” in the free tester group and “substantial” in both the partially free group (κ=0.66 for both the most likely condition and triage) and the restricted group (κ=0.72 for the most likely condition and κ=0.71 for triage). The difference in κ between the restricted and free group is likely significant as the CIs do not overlap. The number of vignettes that had full agreement for the most likely condition increased from 36 to 73 and increased from 48 to 73 for triage, comparing the free tester group to the restricted group. The highest agreement was in the restricted group, with 63.2% full agreement, followed by 27.2% for partial agreement and 9.6% for no agreement for the most likely condition, and 64%, 34.2% and 1.8%, respectively, for triage ( Table 1 ).

Intratester agreement showed κ ranging from 0.41 to 0.50 for the most likely condition and 0.44 to 0.53 for triage. The proportion of agreement ranged from 46.5% to 50.9% for the most likely condition and 50.0% to 59.6% for triage ( Table 2 ).

Negative agreement of the chief complaints selected for the free tester group was demonstrated with κ=−0.4 (95% CI −0.44 to −0.35) for comparing the exact wording and −0.19 (95% CI −0.25 to −0.13) for comparing medical concepts ( Table 3 ). There was no variability (κ=1) for input in the partially free or the restricted tester groups as these were prescribed.

κ intratester agreement for the chief complaints in the free tester group ranged from −0.55 to −0.78 when comparing exact wording and −0.32 to −0.51 when comparing medical concepts ( Table 4 ).

Accuracy of the 3 individual testers for the most likely condition was 43.9%, 50.9% and 57%, respectively, with an average of 50.6% (SD 5.35%) in the restricted group. Detailed individual results of all testers are provided in Multimedia Appendix 5 .

When comparing the accuracy of the 3 different groups, there was no significant difference ( P =.13). There was a 5% ( P =.48) difference in average accuracy between the free and restrictive groups. There was a 10.2% ( P =.05) difference in average accuracy between the free and partially free groups.

Of the 114 vignettes in the restricted group, 51 (44.7%) had a discrepancy between testers for either triage or for the top diagnosis or both. In 55.3% (63/114) of the vignettes, they all received the same most likely condition and triage. For the cases where the outcome among all testers matched, only 3.5% (4/114) bore no perfect resemblance to the vignette’s data points from any of the testers ( Table 5 ). In 3 of these cases, the vignettes were ambiguous. Reviewing the consultations where the testers did not agree on the most likely condition or triage revealed that in 9.6% (11/114) of the cases, none of the tester’s consultation was perfectly identical to the vignette. Overall, there were only 30.7% (35/114) of cases where the consultation of all 3 testers were identical to the data points of the vignette. In 56.2% (64/114) of the cases, 1 or 2 tester’s consultation was identical to the initial vignettes. In 12.3% (15/114) of the cases, none of the testers’ consultations were identical to the vignettes ( Table 5 ).

Analysis of the causes for discrepancy between the vignettes and the completed consultations of the restricted group can be found in Multimedia Appendix 6 .

The average accuracy for all testers was 50.6% (SD 5.35%) in the restricted group. If the accuracy was expanded to count the result for each case from the tester or testers who met the gold standard increased to 63.2%. Adjusted accuracy excluding cases where no consultation of any of the 3 testers were identical to the vignette’s data points (Accuracy Excluded ), the most likely condition accuracy was 55.6%. When an adjusted accuracy was calculated with these cases retested (Accuracy Retested ), the accuracy was similar at 56.1% ( Table 6 ).

The accuracy of the comprehension of chief complaints was easily measurable on both NLP SCs and those that use drop-down symptom input methods. In total, 51 symptoms were assessed (maximum achievable score of 306), covering 29 vignettes (maximum achievable score of 174). The percentage of symptoms understood ranged from 52.9% (162/306) to 68% (208/306) when assessed as stand-alone symptoms and ranged from 55.7% (97/174) to 64.4% (112/174) when symptoms were grouped by their respective vignettes ( Table 7 ).

Principal Findings

We demonstrated significant variations in the agreement of outcome and chief complaints input between different testers of SCs using the same vignettes and when repeated by the same tester over time. Variability between the testers was significantly reduced by restricting testers on the selection of chief complaints and additional symptoms. This demonstrated that tester-dependent factors specific to vignette methodology are partly responsible for the variation and they can be reduced using our restrictive methods. However, even when controlled, significant variability in agreement between testers remained. The lack of agreement resulted in substantial differences in accuracy of condition outcome within the restricted group. The residual variability between restricted testers suggested that general tester-dependent factors and potentially some of the remaining vignette methodology–dependent factors play an important role in determining the outcome and consistency between testers. Studies with a large number of participants, such as clinical trials, can use outcome accuracy as a composite measure accounting for the tester-dependent factors. However, in studies with a small number of testers, such as vignette studies, tester-dependent factors can disproportionately influence the results. Therefore, outcome accuracy as a composite metric in such studies is at risk of reduced reliability and generalizability to a wider population.

Therefore, we established isolated measures of performance. We developed a review and selection process to ensure that only the results of consultations identical to the vignettes are used to calculate an adjusted accuracy (Accuracy Excluded or Accuracy Retested ). Using our adjusted accuracy, we were able to measure an isolated reflection of data and algorithm quality in vignette studies.

Another aspect of SC performance is symptom comprehension, which, as we demonstrated, is feasible to measure in isolation across a range of input methods.

Comparison to Previous Works

No study to date has reported (1) the intratester agreement assessing the variability of chief complaints interpretation and selection, (2) the relationship between symptom input and outcome, or (3) the impact of tester variability on outcome.

Furthermore, most vignette studies do not report the intertester variability of outcome. One possible reason for the lack of data on intertester variability is that many studies have only used 1 sole tester for each test case [ 14 , 22 , 31 , 32 ]. However, as our study demonstrated, individual testers inconsistently input vignettes.

The degree of agreement in outcome for our free tester group was comparable to the study by El-Osta et al [ 20 ] where the same set of vignettes were used with free testers. However, it was lower compared to the study by Semigran et al [ 13 ], which had an agreement of 0.9. This may be explained by a lower number of vignettes in the study by Semigran et al [ 13 ], with only a sample of vignettes undergoing assessment for variability. Shen et al [ 15 ] also reported an agreement (0.74) higher than our free tester group but similar to our restricted group. However, κ value was calculated against the gold standard condition being in the top 3 outcome conditions of the SC [ 15 ], whereas we evaluated whether the testers had an exact match with each other for the top outcome condition. Most studies do not report on the instructions given to testers in answering the questions of the SC during consultation. The high agreement in some studies could be explained by the vignettes being potentially inputted with similar instruction to our restricted group.

Patients with the exact same symptoms but with different chief complaints drive the thought process of doctors [ 33 , 34 ]. Therefore, treating them as the exact same cases for the assessment of SCs is unfair and can result in different outcomes, as demonstrated in our study. Investigators should always assign chief complaints to vignettes with a mix of single and multiple symptoms to mimic real-patient presentation. We have found only 1 peer reviewed paper that provided cases with multiple chief complaints [ 16 ]. In contrast to vignettes that have the exact same symptoms but different chief complaints, vignettes that only differ in the way the chief complaints is expressed should be considered the same case. However, different expressions of the same symptom can and need to be assessed with a large number of cases because there are multiple synonyms or ways of expressing the same symptom, as demonstrated in this study. For example, abdominal pain may be referred to as “pain in the abdomen” or “tummy ache.” Some SCs may understand one of these terms but not others. Interestingly, the importance of assessing comprehension was also mentioned in the preprint paper by Kopka et al [ 12 ] published just before the submission of this paper.

There are multiple methods of assessing NLP [ 35 ], including good examples in the medical field [ 36 ]. However, only a few methods exist assessing drop-down menus and even less to compare NLP versus drop-down menu. Furthermore, NLP evaluations use metrics such as precision, recall, and F 1 values that require false positives for the calculations. An SC that allows a user to correct errors in NLP symptom comprehension complicates the evaluation by allowing the user to remove false positives, while for false negatives, this process brings a drop-down menu into an otherwise NLP-driven symptom comprehension process. Hence, we used an intrinsic evaluation method with a very simple scoring system. Although 1 SC had a higher comprehension accuracy than the other 2, we do not consider it to be truly superior as this is a feasibility study.

Relationship Between Agreement and Accuracy

Despite prescribing the chief complaint(s) in a vignette and restricting additional symptoms during the OSC-user interaction, which are important steps, they are still not sufficient to eliminate all variability among testers in outcomes. The consequences of this residual variability can influence accuracy. When comparing results between the tester groups, the difference for intertester agreement was significant between the free group and the other 2 groups (CIs do not overlap). A statistically significant difference in outcome condition accuracy was observed only between the free and partially free groups. However, significance has to be evaluated with caution in view of the small sample size, and the individual results might hold more lessons. They demonstrate within the restricted group that even with only 3 testers and a relatively good agreement, the accuracy difference between the “highest” and “lowest” performing testers can remain relatively high with 13%. This may also be one of the reasons why systematic reviews on accuracy of SCs show a marked spread of results when comparing individual vignette studies or both clinical and vignette studies [ 2 , 37 - 39 ]. From a detailed review of the tester consultations, accuracy does not clearly correlate with adherence to vignettes. Discrepancies from the vignettes occurred for different cases and for different symptoms; hence, the consequences and the impact on accuracy were varied. Furthermore, even if their outcome was in line with the gold standard, this did not always translate as a spotless representation of the vignette during the consultation.

Therefore, we believe that using adjusted accuracy through a review and selection process is an essential step in studies aiming to measure accuracy reliably. In the restricted phase of our study, there were 15 cases where all testers’ consultation had discrepancies to the vignettes. For this isolated metric, those cases with discrepancies need to either be excluded from analysis or repeated, as was conducted in our study. As shown in our study, only 3.5% (4/114) of cases were in groups where all testers agreed on both the most likely condition and triage. Therefore, we believe that in a real benchmarking exercise, it would be acceptable to only review the cases where the testers have a discrepancy in the most likely condition or triage.

Analyzing the consultation reports revealed that the causes for the discrepancies from the original vignette resulting in the varying outcome between testers are more multifactorial than what we assumed before the study. Some of these causes have implications for future vignette studies. Human error seems to be an important tester-dependent factor. Testers of vignettes may be inherently more prone to errors compared to real patients in studies as they do not report on their own symptoms. The fact that even in the restricted group this was a significant factor influencing outcome suggests that simply increasing the number of testers and applying restrictions would most likely not fully resolve those vignette methodology related limitations like our review and selection process.

Some of the discrepancies were due to the incompatibility between how the Healthily SC works and the instruction given to the testers. For example, in a situation where a user has already declared that they have a cough, they were required to select either a dry or productive cough as there were no “none of them” option. In such a scenario, testers were unable to adhere to restrictive instructions because the vignettes did not specify whether the cough was dry or productive. This raises the point that researchers conducting benchmarking studies should familiarize themselves with the SCs they are evaluating. Our recommended instructions following the takeaway points from this study and the detailed causes for discrepancies can be found in Multimedia Appendices 6 and 7 .

Some other discrepancies were proven to be the consequence of a few ambiguously phrased or incomplete vignettes that can lead to the misinterpretation of symptoms. One of the vignettes specified “multiple sexual partners” but did not specify whether protection was used. If testers were allowed to decide the answer, the test results would become tester dependent and unreliable. Hence, strict instructions are important, even if that might cause some cases to not complete the consultation process. With the review and selection process, these vignettes can either be excluded from the analysis or retested after expanding their information content according to the anticipated answer from a real patient. In the case described above in this paragraph, all our testers confirmed unprotected sexual encounters, and they were all scored as not following the vignette. However, the vignette was then corrected and repeated.

It is impossible to write vignettes that anticipate every question an SC might ask that a real patient would be able to answer. Nevertheless, the vignette creation process has room for improvement. For example, there is a difference between the “main” symptoms and the characteristics of an already established symptom. Most patients would be expected to know whether they had unprotected sexual intercourse; however, some patients may struggle to describe whether the abdominal pain is at the top or middle of their abdomen. Another lesson is that the more detailed a symptom is, the less room it leaves for misinterpretation. Creators could always specify a symptom in as much detail as a real user could; therefore, rather than using the words “rash on leg,” the vignette could state “rash on leg spreading from thigh to knee.” Further research and guidelines are needed to establish how to write suitable vignettes specifically for SCs.

Limitations

This study has several limitations. We have used only 3 testers for each variation of the study, which could have influenced the degree of agreement.

The testers who assessed the agreement between the outcomes were lay people. Previous studies have suggested that health care professionals are more reliable testers, but lay testers are closer to the real user base of SCs; hence, following the recommendation of Painter et al [ 10 ], we opted for the latter approach to assess outcome agreement [ 40 , 41 ]. In contrast, we felt that there is no significant difference between lay or medical testers for assessing comprehension accuracy, as these symptoms were entirely prescribed; hence, a medical student was assigned the task.

Another possible limitation of our study was that only 1 SC was used with only 1 set of vignettes. However, previously, we explored using an independent and smaller vignette set (51 vignettes) collated from 5 different sources and applied these on 3 different SCs using 2 testers. For each SC, the agreement between the 2 testers was <50%, suggesting that the results are independent of SCs and vignettes (Sutaria, S, unpublished data, June 2023).

Furthermore, we used interpretation of κ values by Landis and Koch [ 30 ] who originally developed it for use with only 2 annotators.

Only the most likely condition was considered in our study for the sake of simplicity when calculating agreement and accuracy. Assessing all the outcome conditions from a consultation may have potentially improved the accuracy results. However, the aim was to demonstrate the effect of variability on the accuracy and the importance of using a reliable methodology during the study and not to establish an accuracy score for the Healthily SC. Any full study assessing comprehension conducted by Healthily researchers could bias the results. Therefore, a simple methodology was used to score the results to test the feasibility rather than aiming to determine a true accuracy.

We used a small number of chief complaints and SCs for assessing comprehension accuracy prepared by the researchers. Using focus groups with lay people and larger numbers may be a more appropriate approach in future studies. However, the aim was to assess the feasibility of conducting such an analysis. As the study was financed by an SC company and the investigators are employees of the company, the authors felt that it would be biased to run a real assessment publishing a verdict on the accuracy of the comprehension of any SC.

Conclusions and Recommendations

The authors of vignette studies should understand that they are unable to account for important variables when measuring accuracy using vignettes in the same manner as clinical studies. They should instead aim to measure isolated aspects of accuracy to judge the performance of an SC.

We recommend the following:

• Cautious comparison of triage and condition outcome accuracy in clinical trials against vignette studies
• Using vignettes with selected chief complaints that cover a variation of single and multiple symptoms that are clearly and plainly described
• Giving clear instructions to testers on not selecting any symptom that is not present in the vignette unless it is a synonym or a bigger category of a symptom described in the vignette (refer to the recommendation on instructions in Multimedia Appendix 7 )
• Using multiple testers to input each vignette
• Comparing consultations to the vignette in cases where there is a discrepancy in the outcome between testers. This should be performed blindly without prior knowledge of the expected gold standard. Only consultations that accurately represent the vignette’s data points should be included for the evaluation. Exclusion or repetition of vignettes might be required. Vignettes might need adaptation or correction before repetition. Testers might need to draw attention to the misinterpretation of symptoms and human errors before retesting.
• Measuring and reporting on comprehension of the SCs

Further research is needed for the following:

• To develop methodologies to measure other aspects of performance such as the SC’s ability to express symptoms in an understandable manner during the consultation
• To develop methodologies and guidelines for creating vignettes specifically with the purpose of assessing SCs
• To explore the evaluation and comparison of other, different input methods

Acknowledgments

The authors are grateful to Dr Maureen Baker, CBE, and Johnathon Carr-Brown for reviewing the manuscript and to Martin Cansdale for reviewing the natural language processing–related paragraphs. Healthily paid for the reimbursement of the testers and paid the publications fee.

Data Availability

The data sets generated during and analyzed during this study are published either in the main text or in the multimedia appendices. The exceptions are the clinical vignettes that are not publicly available, as this would compromise their use for continuous evaluation of Healthily’s web-based symptom checkers (SC). However, they are available from Healthily on reasonable request following approval for genuine scientific research. Requests that might compromise the use of the vignettes in further studies run by Healthily or whereby the vignettes might become publicly available will be denied. Data will be not shared with other companies working on SCs or companies that are competitors of Healthily. Every request will be evaluated on a case-by-case basis by Healthily.

Authors' Contributions

AM conceptualized the study and wrote the first draft of the manuscript. TM reviewed and edited the first draft, providing equal contribution to the manuscript. TM solely conducted all statistical analyses. AM designed the methodology of the study, with TM, AQ, and NC contributing to it. TM, NC, and AQ adapted the premade vignettes for the study. NC and AQ analyzed the results of the testers, except the descriptive analysis of the consultations that was undertaken by AM and AQ. ZB, SS, NC, AQ, MR, and EB reviewed and made significant suggestions to the manuscript. All authors contributed to the review and approved the final version of the paper.

Conflicts of Interest

AM, NC, AQ, MR, SS, EB, and TM are all current or ex-employees and shareholders of Healthily. Healthily funded the research.

Test cases.

Instructions given to the testers for assessing intertester agreement.

Characteristics of the vignettes.

Characteristics of the testers.

Individual accuracy results of the testers in the different phases and the averages of each group.

Causes for discrepancy between the vignette and consultations.

Recommended instructions for future testers.

• Levine DM, Tuwani R, Kompa B, Varma A, Finlayson SG, Mehrotra A, et al. The diagnostic and triage accuracy of the GPT-3 artificial intelligence model. medRxiv. Feb 01, 2023:5067. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wallace W, Chan C, Chidambaram S, Hanna L, Iqbal FM, Acharya A, et al. The diagnostic and triage accuracy of digital and online symptom checker tools: a systematic review. NPJ Digit Med. Aug 17, 2022;5(1):118. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hildebrandt DE, Westfall JM, Fernald DH, Pace WD. Harm resulting from inappropriate telephone triage in primary care. J Am Board Fam Med. Sep 01, 2006;19(5):437-442. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Regulation (EU) 2017/745 of the European Parliament and of the Council of 5 April 2017 on medical devices, amending Directive 2001/83/EC, Regulation (EC) No 178/2002 and Regulation (EC) No 1223/2009 and repealing Council Directives 90/385/EEC and 93/42/EEC (Text with EEA relevance. ). European Union. URL: https://eur-lex.europa.eu/eli/reg/2017/745/oj [accessed 2024-05-08]
• Framework for FDA’s real-world evidence program. U.S. Food & Drug Administration. Dec 2018. URL: https://www.fda.gov/media/120060/download [accessed 2023-05-15]
• Powley L, McIlroy G, Simons G, Raza K. Are online symptoms checkers useful for patients with inflammatory arthritis? BMC Musculoskelet Disord. Aug 24, 2016;17(1):362. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Knitza J, Mohn J, Bergmann C, Kampylafka E, Hagen M, Bohr D, et al. Accuracy, patient-perceived usability, and acceptance of two symptom checkers (Ada and Rheport) in rheumatology: interim results from a randomized controlled crossover trial. Arthritis Res Ther. Apr 13, 2021;23(1):112. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Evans SC, Roberts MC, Keeley JW, Blossom JB, Amaro CM, Garcia AM, et al. Vignette methodologies for studying clinicians' decision-making: validity, utility, and application in ICD-11 field studies. Int J Clin Health Psychol. May 2015;15(2):160-170. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ilicki J. Challenges in evaluating the accuracy of AI-containing digital triage systems: a systematic review. PLoS One. Dec 27, 2022;17(12):e0279636. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Painter A, Hayhoe B, Riboli-Sasco E, El-Osta A. Online symptom checkers: recommendations for a vignette-based clinical evaluation standard. J Med Internet Res. Oct 26, 2022;24(10):e37408. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Gottliebsen K, Petersson G. Limited evidence of benefits of patient operated intelligent primary care triage tools: findings of a literature review. BMJ Health Care Inform. May 07, 2020;27(1):e100114. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kopka M, Feufel MA, Berner ES, Schmieding ML. How suitable are clinical vignettes for the evaluation of symptom checker apps? A test theoretical perspective. Digit Health. Aug 21, 2023;9:20552076231194929. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Semigran HL, Linder JA, Gidengil C, Mehrotra A. Evaluation of symptom checkers for self diagnosis and triage: audit study. BMJ. Jul 08, 2015;351:h3480. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hill MG, Sim M, Mills B. The quality of diagnosis and triage advice provided by free online symptom checkers and apps in Australia. Med J Aust. Jun 11, 2020;212(11):514-519. [ CrossRef ] [ Medline ]
• Shen C, Nguyen M, Gregor A, Isaza G, Beattie A. Accuracy of a popular online symptom checker for ophthalmic diagnoses. JAMA Ophthalmol. Jun 01, 2019;137(6):690-692. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Baker A, Perov Y, Middleton K, Baxter J, Mullarkey D, Sangar D, et al. A comparison of artificial intelligence and human doctors for the purpose of triage and diagnosis. Front Artif Intell. Nov 30, 2020;3:543405. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Explainability statement. Healthily, Best Practice AI, Simmons & Simmons and Jacob Turner of Fountain Court Chambers. Sep 17, 2021. URL: https://tinyurl.com/ru5d8h2m [accessed 2023-09-03]
• Hammoud M, Douglas S, Darmach M, Alawneh S, Sanyal S, Kanbour Y. Avey: an accurate ai algorithm for self-diagnosis. medRxiv. Preprint posted online March 11, 2022. [ CrossRef ]
• Tse T, Soergel D. Exploring medical expressions used by consumers and the media: an emerging view of consumer health vocabularies. AMIA Annu Symp Proc. 2003;2003:674-678. [ FREE Full text ] [ Medline ]
• El-Osta A, Webber I, Alaa A, Bagkeris E, Mian S, Taghavi Azar Sharabiani M, et al. What is the suitability of clinical vignettes in benchmarking the performance of online symptom checkers? An audit study. BMJ Open. Apr 27, 2022;12(4):e053566. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Garcia-Esquirol O. Lack of scientific rigour- response to "how accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs". BMJ Open. Jan 8, 2021. [ FREE Full text ]
• Gilbert S, Mehl A, Baluch A, Cawley C, Challiner J, Fraser H, et al. How accurate are digital symptom assessment apps for suggesting conditions and urgency advice? A clinical vignettes comparison to GPs. BMJ Open. Dec 16, 2020;10(12):e040269. [ CrossRef ]
• Healthily homepage. Healthily. URL: https://www.livehealthily.com/symptom-checker [accessed 2023-05-10]
• Harrison CJ, Sidey-Gibbons CJ. Machine learning in medicine: a practical introduction to natural language processing. BMC Med Res Methodol. Jul 31, 2021;21(1):158. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chen X, Xie H, Tao X. Vision, status, and research topics of Natural Language Processing. Natural Lang Process J. 2022;1:100001. [ CrossRef ]
• Ada Health homepage. Ada Health. URL: https://ada.com/ [accessed 2023-06-02]
• Symptomate homepage. Symptomate. URL: https://symptomate.com/ [accessed 2023-06-02]
• StataCorp. Stata statistical software: release 13. StataCorp LP. College Station, TX. StataCorp LP; 2013. URL: https://www.stata.com/manuals13/u.pdf [accessed 2023-05-01]
• Klein D. KAPPAETC: stata module to evaluate interrater agreement. Boston College Department of Economics. 2016. URL: https://ideas.repec.org//c/boc/bocode/s458283.html [accessed 2023-05-01]
• Landis JR, Koch GG. The measurement of observer agreement for categorical data. Biometrics. Mar 1977;33(1):159-174. [ Medline ]
• Schmieding ML, Kopka M, Schmidt K, Schulz-Niethammer S, Balzer F, Feufel MA. Triage accuracy of symptom checker apps: 5-year follow-up evaluation. J Med Internet Res. May 10, 2022;24(5):e31810. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Ćirković A. Evaluation of four artificial intelligence-assisted self-diagnosis apps on three diagnoses: two-year follow-up study. J Med Internet Res. Dec 04, 2020;22(12):e18097. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nierenberg R. The chief complaint driven medical history: implications for medical education. Int J Med Educ. May 30, 2017;8:205-206. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kilian A, Upton LA, Sheagren JN. Reorganizing the history of present illness to improve verbal case presenting and clinical diagnostic reasoning skills of medical students: the all-inclusive history of present illness. J Med Educ Curric Dev. Jun 10, 2020;7:2382120520928996. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Resnik P, Lin J. Evaluation of NLP systems. In: Clark A, Fox C, Lappin S, editors. The Handbook of Computational Linguistics and Natural Language Processing. Hoboken, NJ. John Wiley & Sons; 2010.
• Canales L, Menke S, Marchesseau S, D'Agostino A, Del Rio-Bermudez C, Taberna M, et al. Assessing the performance of clinical natural language processing systems: development of an evaluation methodology. JMIR Med Inform. Jul 23, 2021;9(7):e20492. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Chambers D, Cantrell AJ, Johnson M, Preston L, Baxter SK, Booth A, et al. Digital and online symptom checkers and health assessment/triage services for urgent health problems: systematic review. BMJ Open. Aug 01, 2019;9(8):e027743. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Pairon A, Philips H, Verhoeven V. A scoping review on the use and usefulness of online symptom checkers and triage systems: how to proceed? Front Med (Lausanne). Jan 6, 2022;9:1040926. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Riboli-Sasco E, El-Osta A, Alaa A, Webber I, Karki M, El Asmar ML, et al. Triage and diagnostic accuracy of online symptom checkers: systematic review. J Med Internet Res. Jun 02, 2023;25:e43803. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Fraser H, Coiera E, Wong D. Safety of patient-facing digital symptom checkers. Lancet. Nov 24, 2018;392(10161):2263-2264. [ CrossRef ] [ Medline ]
• Jungmann SM, Klan T, Kuhn S, Jungmann F. Accuracy of a chatbot (Ada) in the diagnosis of mental disorders: comparative case study with lay and expert users. JMIR Form Res. Oct 29, 2019;3(4):e13863. [ FREE Full text ] [ CrossRef ] [ Medline ]

Abbreviations

Edited by A Mavragani; submitted 22.06.23; peer-reviewed by M Kopka; comments to author 21.07.23; revised version received 10.08.23; accepted 24.04.24; published 31.05.24.

©András Meczner, Nathan Cohen, Aleem Qureshi, Maria Reza, Shailen Sutaria, Emily Blount, Zsolt Bagyura, Tamer Malak. Originally published in JMIR Formative Research (https://formative.jmir.org), 31.05.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Formative Research, is properly cited. The complete bibliographic information, a link to the original publication on https://formative.jmir.org, as well as this copyright and license information must be included.

13 Tips for Choosing the Perfect Domain Name

Published: May 27, 2024

Choosing the right domain name for your company is so important — ultimately, it’s how your audience will find and remember you.

The best domain is a custom domain that works for you. Sometimes, the best domain name for you might break some best practice rules. I’ve got websites built on domains that follow best practices, including those with keyword-rich domains, and other domains break the best practice rules big time; they’re even hard to spell.

With over 700 million domain names registered , it can be hard to find one that’s unique and functional. But, in this article, I’ll help you find a custom domain that will help your website stand out from the crowd and possibly even boost your SEO. I’ll help you choose and buy a domain while sharing the insights and tips I’ve learned along the way.

Table of Contents

What Makes A Good Domain Name?

How to choose a domain name for your business, how to choose a domain name for your blog, domain name generator, establish brand identity by picking the perfect domain name.

The best domain names balance memorability, longevity, and sometimes SEO. There’s a lot to think about when choosing the right domain name, but before I get into all those steps, let’s first identify what makes a good domain.

Ideally, you’ll pick one domain name you’ll stick with forever. This might seem scary, and it’s okay to change your domain later, but it’s better if you don’t — so don’t take this decision lightly.

The reason why longevity matters is because you want a domain name that’s stable to avoid confusing your audience.

There’s another lesser-known reason, and it’s related to SEO: a domain name builds authority. Basically, an older domain with high-quality content becomes trusted by Google and is theoretically easier to rank. The longer you spend with a domain and the more you build it up, the bigger the potential loss if you change it.

Again, you can change your domain, and there are lots of things you can do to protect that built authority, but the absolute best practice is to choose a domain that you can stick with.

SEO Considerations

As well as authority, your domain name can help bolster your SEO efforts. Domains including a keyword can help websites rank.

You want to balance the benefits of using the keyword in the URL against using a brand keyword. I’ll discuss the nuances of choosing keyword-rich domains later including when I think it’s appropriate and when to stay with the brand name.

Memorability

Your domain name is how your audience will find and remember you. The best domain names are memorable to your audience so they can easily find you again.

If you are ready to establish a presence online for your business, you’ll want to choose a domain name that matches your brand. This way, customers can easily find and trust your site. There are a few steps you can keep in mind when choosing your domain name.

Free Website Design Inspiration Guide

77 Brilliant Examples of Homepages, Blogs & Landing Pages to Inspire You

• Agency Pages
• Ecommerce Pages
• Tech Company Pages

You're all set!

Click this link to access this resource at any time.

4. Stay unique, specific, and on-brand.

With all of these things to keep in mind, it can be hard to brainstorm a domain name that is clear and concise while also remaining unique and true to your business.

But it’s certainly doable. Keep your business name and what you sell at the forefront of your mind, and dig into your niche to ensure that your domain name attracts the online audience you want.

I always find it hard to come up with domains. I usually have to go through a few versions of ideas and get knocked back a lot by the time I’ve searched its availability to find it’s taken.

Nowadays, I use ChatGPT to help me brainstorm domain name ideas. Using a fake scenario, here’s an example of what I’d ask ChatGPT.

I asked ChatGPT, “Can you help me come up with the best domain name for a new blog? I want to share my budget travel adventures.”

ChatGPT came up with some great starting points.

Domain names are hot commodities, but they are also pretty affordable. As long as your idea isn’t trademarked or already in use, grab it before it’s gone! You can always change it later if you decide it just isn’t working.

I understand that this is a lot to digest. But just as there are generators to help you find excellent hashtags and keywords, there are also tools to help you find domain names (in addition to using ChatGPT).

You can take some of the keywords and brand-related words you’ve brainstormed through the above tips and add them to a domain name generator, which will use those words and related words to create lists of available domain names.

Even if you don’t pick a domain name exactly from the generator, it can help inspire your team to find a domain name that is perfect for your business or blog. Here are some of the top domain name generators to try.

Don't forget to share this post!

Related articles.

The Best Web Hosting for Small Business Sites in 2024: 12 Favorites

Best Video Hosting Sites: The Top 11 Sites in 2023

Domains and Domain Names: What They Are and 10+ Examples

Everything You Need to Know about Custom Domains

The 13 Best Blog Hosting Sites to Launch a Winning Blog in 2024

14 Best Domain Registrars to Consider This Year

Serverless Functions: Your Website's New Best Friend

The Ultimate Guide to Web Hosting

What Is a CDN? Understanding Content Delivery Networks

10 Best Windows Web Hosting Platforms for 2024

Launch your WordPress website with the help of this free guide and checklist.

CMS Hub is flexible for marketers, powerful for developers, and gives customers a personalized, secure experience

Hoxworth Blood Center urges summer blood donations

Donate blood or platelets in june and receive a special thank you gift.

Donors in June will receive this floppy summer hat.

Hoxworth Blood Center, University of Cincinnati, is celebrating and thanking blood donors this June with a special thank you gift.

All blood and platelet donors who donate at any of Hoxworth’s seven Neighborhood Donor Centers or select mobile blood drives will receive an exclusive Hoxworth floppy sun hat!

“It’s that time of the year where summer vacations and activities are in full swing, and unfortunately we tend to see a drop in donor turnout because of that,” says Jackie Marschall, Hoxworth spokesperson. “The need for blood doesn’t take a vacation, which makes it the perfect time to carve out one hour of your week to make an impact that can change someone’s life for the better.”

Hoxworth needs at least 500 donors each day, including 80 platelet donors, to meet the demand of local hospitals, and that’s without any unexpected procedures.

Hoxworth operates seven Neighborhood Donor Centers across the Tri-State with appointments available each day of the week. Appointments are highly encouraged and can be made at Hoxworth.org , by calling 513-451-0910, or by downloading the new Hoxworth Blood Donor app.

Featured photo at top of donated blood. Photo/University of Cincinnati.

About Hoxworth: Hoxworth Blood Center, University of Cincinnati, was founded in 1938 and serves more than 30 hospitals in 18 counties in Southwestern Ohio, Northern Kentucky, and Southeastern Indiana. Annually, Hoxworth collects more than 100,000 units of blood from local donors to help save the lives of patients in area hospitals. Hoxworth Blood Center: Saving Lives Close to Home.

• College of Medicine

Related Stories

Uc research shows renewed promise for covid-19 convalescent plasma.

March 31, 2022

Research from UC shows renewed hope in reducing hospitalizations from COVID-19 through the use of convalescent plasma when given early in the disease course. The study was published in the New England Journal of Medicine.

Hoxworth celebrates March with exclusive gift for donors

March 4, 2022

Hoxworth Blood Center is celebrating the month of March with exclusive 3-in-1 tumbler/koozies for all donors

Celebrate Cicadapocalypse 2021 at Hoxworth Blood Center

May 24, 2021

Celebrate this year’s cicada invasion (or simply escape the hordes of buzzing, red-eyed locusts) by donating blood at Hoxworth Blood Center.