risk management clinical research

Clinical Researcher

Risk-Based Clinical Trial Management: Harnessing the Transformation of RBM to RBQM

Clinical Researcher August 10, 2020

Clinical Researcher—August 2020 (Volume 34, Issue 7)

QUALITY CONCERNS

Patrick Hughes

Risk-based quality management (RBQM) is a system for managing quality throughout a clinical trial. The data-driven elements of this type of strategy have evolved substantially over the past few years, as an extension to the original principles underpinning risk-based monitoring (RBM). This article will outline the difference between RBM and RBQM, highlighting some of the advantages and benefits of managing all areas of quality in a clinical trial. It will also provide a discussion of the implementation of the method alongside some of the challenges related to embracing the change. It will outline how sponsors and contract research organizations (CROs) can harness the power of risk-based trial management, making clinical trials better, faster, and cheaper for the industry and safer for patients.

A Need for Change

From the year 2000, a continual increase in the complexity of clinical trial designs, highly publicized safety issues with marketed drugs, and a slowing of innovation coupled with patent expirations saw the cost and duration of clinical development steadily increase, while profit margins dwindled. While the previous decade had been a time of relative economic health for the biopharmaceutical industry, at the turn of the century drug makers found themselves faced with growing pressure from multiple directions.

Between 2000 and 2012, a review of marketing submissions to the U.S. Food and Drug Administration revealed that about one-third (32%) of all first-cycle review failures, or 16% of submissions overall, were driven by quality issues.{1} The increasing complexity of trials means they take longer and cost more. This dynamic also adds significant risk to the operational success of research, both in terms of recruiting and retaining patients, and in generating the reliable results needed to support ultimate marketing approvals. It is apparent that the traditional way of conducting trials is not fit for the 21st century.

Understanding RBM

RBM, which is most efficiently achieved by sponsors harnessing technology and real-time information to proactively monitor risk, was written into U.S. and European regulatory guidance in 2013. In its simplest form, RBM strategies use software, data inputs, and analytics to monitor risk and support critical thinking and decision making. By giving sponsors the ability to identify and correct issues as and when they arise, RBM can improve data quality and patient safety as well as reduce costs.

At its core, RBM is the operational analogue to the tenets of “quality by design” (QbD). Both models have the same fundamental goal of improving the operational success rate of clinical research through higher quality, shorter timelines, and greater efficiency. QbD and RBM are also linked by methodology, as they both call for ongoing assessment and mitigation of operational risk.

Embracing RBQM

RBQM methodology is a very timely development that sponsors and CROs are now embracing to address the growing crisis in research complexity, duration, and cost. The latest version of the Good Clinical Practice (GCP) quality standard extends the RBM approach to every aspect of study execution, applying the principles to all areas of quality management. The ICH E6(R2) guideline for GCP from the International Council for Harmonization outlines the driving factors of this approach, including the transition away from largely paper-based research to the modern approach of electronic and digital technologies including electronic data capture, electronic clinical outcome assessment, and interactive response technology. This has opened a tremendous opportunity to plan and manage clinical research more effectively and efficiently.

RBQM implementation can be overwhelming for an organization, given the wealth of information that is currently available. Starting simple is the way to maintain focus and concentrate on the elements of RBQM that are most important to gain immediate quick wins and success in the long term. The key to success is to apply thoughtful but simple processes, smart technology, and a focus on evolutionary change management.

Making the Transition

RBQM encompasses all elements of the study, from planning right through to execution. Risk management underpins the overall quality of the trial by identifying, controlling, and communicating. ICH E6(R2) sets out what a gold standard RBQM system should cover:

• Critical process and data identification
• Risk identifications
• Risk evaluation
• Risk control
• Risk communication
• Risk review
• Risk reporting

Further, centralized statistical monitoring (CSM) is a critical component of the operational success of RBQM, as it is a key and under-used weapon for quality oversight. CSM lies at the heart of RBQM (see Figure 1). It interrogates all clinical and key operational data to find anomalies and discrepancies that would remain undetected by traditional techniques. It is more than just computing statistics on a subset of key variables—it is about processing all data and guiding users to where the potential issues might lie, or a “boil the ocean” approach to risk identification and mitigation.

Figure 1: Centralized Statistical Monitoring Model

An effective centralized monitoring approach should include the following three components:

• Data surveillance
• Key risk indicators (KRIs)
• Quality tolerance limits (QTLs)

When it comes to KRIs and QTLs, quality is much more important than quantity. Sponsors and CROs should identify a core set (10 to 15) of appropriate KRIs and focus on ensuring that these are optimized to detect risk as early as possible and minimize likelihood of false alerting.

The same principle should apply to QTLs (four or five), which should focus on the most important study-level risks, or “failure points.” Data surveillance, which is sometimes referred to as CSM, has been under-appreciated and under-utilized by many organizations, but provides an effective independent and objective quality oversight process.

While KRIs and QTLs are designed to monitor for pre-identified areas of risk, data surveillance or CSM can expose forms of study abnormality and misconduct that may be difficult to identify and/or characterize during pre-study risk planning. By running a comprehensive set of well-designed statistical tests across a broad swath of study data, the method can spot atypical patterns that represent potential intentional or non-intentional misconduct. It can flag issues such as fraud, sloppiness, or training needs, as well as malfunctioning or poorly calibrated study equipment.

Elements to Success

RBQM relies on a combination of different tools. A central monitoring platform can act as the enabling technology, encompassing central data review, risk assessment, KRIs, data quality oversight, and issue and action tracking management modules. None of the key components of RBQM implementation, including pre-study risk planning, adaptive/dynamic site monitoring with a significant reduction in source data verification, and centralized monitoring, need to be complex to be effective. Risk findings should be documented thoroughly and accurately for regulatory inspection purposes. A plan should ideally cover the overall objectives, proactive data monitoring, and communication.

The first step in proactive data monitoring is to identify what is possible to mitigate, eliminate, and accept. This all forms part of various plans, including those for data, training, monitoring, statistical analysis, safety, medical monitoring, quality, and other functional plans. KRIs, QTLs, CSM, and risk communication are all crucial to the process to identify risk signals and comply with the regulatory obligations. The entire study team should be aware of the risks and how they are being managed.

Although the many layers of the model may seem daunting at first, sustainable success in adopting RBQM begins with establishing and confirming the primary objectives for adopting the strategy (i.e., what is the organization trying to achieve with RBQM?).

Each of the following three dimensions of value should be considered:

• Improved quality
• Reduced operational costs
• Shorter timelines

Moving Forward

Improving data quality and patient safety, while controlling the spiralling costs of drug development research, were the primary objectives behind the shift toward RBM over the last eight years. The model’s success, combined with advances in clinical trial technology, has seen the approach extended to cover the whole of trial execution in a methodology widely referred to as RBQM. Elements of RBQM can be implemented individually and independently to great success, making clinical trials better, faster, and cheaper for sponsors and CROs and safer for patients.

1. https://jama.jamanetwork.com/article.aspx?articleid=1817795

Patrick Hughes is Co-founder and Chief Commercial Officer of CluePoints.

Sorry, we couldn't find any jobs that match your criteria.

Barriers to Clinical Trial Enrollment: Focus on Underrepresented Populations

Using Simulation to Teach Research

An Approach to a Benefit-Risk Framework

Risk Based Monitoring Toolbox

Introduction.

The Risk-Based Monitoring Toolbox provides information on tools available for risk assessment, monitoring and study conduct, the institutions where they are used, and other relevant details such as links and user feedback. The goal is to enable researchers to create risk-based strategies that are appropriate for their study needs.

Launched end 2015, the Toolbox was created following a systematic literature review on current practices and recommendations as well as a survey of clinical trial units (CTUs). The survey identified existing risk-adapted monitoring tools, risk evaluation methods, and monitoring strategies. 

Click on the chapters below to find relevant information

Why Risk-Based Monitoring?

Risk-based monitoring in clinical trials is the practice of assessing the risks of a clinical study and using this information to decide which monitoring effort is most appropriate. Traditionally, Source Document Verification is carried out on 100% of data and frequent onsite visits are required. In a risk-based approach, depending on the strategy, Source Document Verification may be carried out only for certain targeted data or some data points may be verified only after a predefined trigger, such as an event occurring at a site or a specific metric being reached. Risk-based monitoring can lead to more efficient use of resources without reduction in data quality.

Why a Risk-Based Monitoring Toolbox?

Scope of monitoring.

Monitoring is the act of overseeing the progress of a clinical trial and ensuring that it is conducted, recorded and reported in accordance with the protocol, Standard Operating Procedures (SOPs), Good Clinical Practice (GCP) and the applicable regulatory requirement(s) (European Medicines Agency 2002). The purpose of trial monitoring is to verify that the rights and well-being of human subjects are protected and the reported trial data are accurate, complete, and verifiable from source documents.

However, a broader definition of trial monitoring includes strategies that enhance trial oversight during the design, execution and analysis stages (Baigent 2008). Also, monitoring activities are not limited to onsite visits but must be understood as all onsite and central activities dealing with checks of data and procedures as well as with the overall surveillance and stimulation of the trial progress.

The Risk-Based Approach: New Paradigm, Big Challenge

The European regulation on clinical trials, as well as guidelines from competent authorities, now require the use of the risk-based approach for many aspects of trial conduct, among which monitoring.

However, the approach is not easily grasped, and no instruction manual has been provided. Furthermore, the very concept of risk-based approach implies that the chosen strategy is adapted to local and trial-specific context. So there is no "one-fits-all" model anymore, and most stakeholders are at a loss to define a relevant trial-specific strategy.

The Toolbox: Objective and Content

We set up this toolbox gathering risk-based tools and strategies already proposed and/or validated together. The visitor may compare different tools, appraise their scope, specificities, strengths and weaknesses, and choose those fit to build a specific monitoring strategy.

We considered any type of tool: standard document, checklist, procedure, software, device, etc. However, commercial tools, information technology, and data management tools were not considered because they come under data management more than monitoring activities, and information on these tools may be found easily on the Internet.

Tools Identification and Description

Tools were detected through a systematic review of the published literature (January 2006 to June 2015), a search of the grey literature, and a survey among clinical trial units and academic sponsors within the ECRIN network (2013). The objective was to identify current state of practice and recommendations in clinical trials, key elements of monitoring, risk assessment tools and risk-adapted monitoring tools.

Tools are not directly available from the toolbox. Each tool is described through its scope, format and purpose. Contact information of the team having developed the tool, or reference of published article, is also provided.

Key Elements of Monitoring

Among the different steps of a clinical trial, some may be more crucial for patients’ safety and the validity of study results. 

We assumed that the importance of an element is reflected through the proportion of publications considering this element in monitoring strategies or quality control methods.

We identified 27 publications (Bakobaki 2011; Baigent 2008; Brosteanu 2009; Journot 2011; Kirwan 2008; Méthot 2012; Sandman 2006; Williams 2006; Bertoye 2006; Cooley 2010; De 2011; Grieve 2012; Kirkwood 2013; Matthews 2011; Ansmann 2013; Venet 2012; Macefield 2012; Tudur-Smith 2014; Graham 2012; Heels-Ansdell 2010; McBee 2012; Novik 2014; Shugarts 2012; CTTI (Morrisson 2010; Morrisson 2011); FDA 2013; EMA; MRC/DH; MRC/DH/MHRA) reporting analyses of monitoring strategies and/or methods for quality control of clinical trial data and procedure. These publications differed significantly in the purpose addressed and the topic discussed. However the identified key elements are reported in the table and figure below.

Distribution of Key Elements of Monitoring :

Risk Assessment Tools

The risk-based approach relies on the identification and assessment of risk(s).

Risk is defined by ISO 31000 standard as "the effect of incertitude on objectives". In clinical research, the considered risks are always negative, so risk is best characterised by the occurrence of a negative event.

For years, risk has been interpreted as risk for patient's safety or rights only. However, other types of risk should be considered: for the participant in the study, the institutions and teams in charge of the study conduct, the governance structures, the target population, and the public health stakeholders, etc.

Risk-Adapted Monitoring Tools

On-site monitoring, source data activities .

On-site monitoring (OSM) is still associated with source data verification (SDV) as a key activity. Indeed, for long years SDV was deemed the most important procedure for achieving high data quality. However, the value of extensive SDV has been questioned in the last years, and there is an increasing body of evidence in the literature showing that 100% SDV is a very costly but not very effective measure in terms of high data quality.

Several authors have investigated the impact of data corrections following SDV; their results are summarized e.g. in the survey of (Tantsyura et al. 2015a).

Sheetz et al (Sheetz et al. 2014) present a retrospective analysis of 1168 clinical trials performed by a total of 53 pharmaceutical sponsors. In all trials, electronic data capture was used. The trials cover phases I to IV and a broad range of indications. The authors show that only 3.7% of all data captured where corrected after initial entering. From these, only 1/3 was corrected due to SDV findings, while the others corrections where performed following automatically generated queries or queries raised by data management. That means that only 1.1% of all data had to be corrected due to SDV findings. This in line with Mitchel et al (Mitchel et al. 2011), who reported a rate of data corrections of 6.2%, from which 71% were due to data entry errors. These corrections had only minimal impact on the means of the primary variables. However, a slight increase in variance was seen with uncorrected data, such that an increase of 1% in the sample size would have been enough to counteract the loss of power due to uncorrected data.  Tudur Smith et al (Tudur Smith et al. 2012) present an analysis based on a non-commercial cancer trial comparing two treatments in over 500 patients in 75 UK trial sites. The trial was not blinded, and paper CRF were used for documentation. The authors report discrepancy rates of about 7.8% for the primary endpoint survival and as high as 24.8% for the secondary endpoint progression free survival. However, these discrepancies were at random and had no impact on the study results, with superimposable Kaplan-Meier curves and only slightly different hazard ratios. It is of interest to mention that in this trial, the authors describe a discrepancy rate of 85.6% in the subjective outcome “objective response”. With respect to this finding, the authors conclude that “an independent blinded review committee and tracking system to monitor missing CT scan data could be more efficient than SDV”.

It should be stressed here that the authors cited above do not question the value of on-site monitoring visits in general. It is only the relevance and efficiency of searching for transcription errors that is challenged. For example, Sheetz et al (Sheetz et al. 2014) report that 11% of the adverse events and 3.6% of the serious adverse events were detected by source data review.

Training and Other On-Site Activities

As stated by Baigent et al (Baigent et al. 2008), “on-site monitoring should be … regarded as "mentoring", providing opportunities for training and supporting study staff”. In this section, two papers describing on-site activities beyond SDV are reviewed.

Central Monitoring

Central monitoring or central statistical monitoring comprise all activities for the surveillance of a clinical trial which may be performed centrally, including data management checks, surveillance by statistical description or analysis, medical review, core labs and review committees for the central assessments of  e.g. scans or clinical endpoints. 

Remote Monitoring

With rapidly developing technologies, remote access to source data is an option sometimes discussed as an alternative to on site monitoring. However, data protection issues may pose serious problems with this approach. 

Central Statistical Monitoring

The potential of central statistical monitoring to detect erroneous data and deficient trial sites has been demonstrated e.g. by Bakobaki et al (Bakobaki et al. 2012) and Lindblad et al ( (Lindblad et al. 2014). Bakobaki and colleagues examine a sample of on site monitoring reports, and find that about 95% of all findings could have been identified by means of central monitoring. Lindblad et al analyse data submitted to the Food and Drug Administration (FDA) by clinical trial sponsors. They show that “systematic central monitoring of clinical trial data can identify problems at the same trials and sites identified during FDA site inspections”. 

Recently, a differentiation between central monitoring using key risk indicators pre-identified by prior risk assessment and “centralized statistical monitoring” has been proposed (Buyse M 2014). In this proposal, centralized statistical monitoring is characterised as the application of advanced statistical and bioinformatics methods on all available clinical trial data without any prioritisation (Venet et al. 2012). However, we do not follow this distinction, because in both cases, statistical techniques are required to make best use of the available information. The two approaches should be considered as complementary. Thorough risk assessment lead to identification of anticipated study specific hazards, which have to be adequately surveyed. The central statistical monitoring techniques proposed by Venet et al (Venet et al. 2012) may help to detect unexpected problems, but have a low specificity and may detect issues of low relevance for the integrity of the trial.

Case Examples

In this section we present some case examples for the implementation of risk-based approaches.

Trial Oversight

Different types of oversight committees may be installed in clinical trials, depending on the size and complexity of the trial and the study population involved (Baigent et al. 2008).

We focus here on independent Data Monitoring Committees (DMC, synonyms e.g. Data Monitoring and Safety Board, Data Safety Monitoring Board), and list publications dealing with considerations and procedures which may be of use when installing a DMC. 

Al-Marzouki, Sanaa; Evans, Stephen; Marshall, Tom; Roberts, Ian (2005): Are these data real? Statistical methods for the detection of data fabrication in clinical trials. In:  BMJ (Clinical research ed.)  331 (7511), S. 267–270. DOI: 10.1136/bmj.331.7511.267.

Ansmann E.B, Hecht A, Henn D.K, Leptien S, Stelzer HG. The future of monitoring in clinical research – a holistic approach: Linking risk-based monitoring with quality management principles. GMS German Medical Science 2013; 11:1-8.Baggs, G.; Seth, A.; Oliver, J. S.; Jones, W. M.; liu, L.; Toth, S. M. (2008): Monitoring Clinical Trial Data Using an Unblinded Industry Statistician. In:  Drug Information Journal  42, S. 193–199. Online verfügbar unter doi:10.1177/009286150804200211.

Baigent, Colin; Harrell, Frank E.; Buyse, Marc; Emberson, Jonathan R.; Altman, Douglas G. (2008): Ensuring trial validity by data quality assurance and diversification of monitoring methods. In:  Clinical trials (London, England)  5 (1), S. 49–55. DOI: 10.1177/1740774507087554.

Bakobaki, Julie M.; Rauchenberger, Mary; Joffe, Nicola; McCormack, Sheena; Stenning, Sally; Meredith, Sarah (2012): The potential for central monitoring techniques to replace on-site monitoring: findings from an international multi-centre clinical trial. In:  Clinical trials (London, England)  9 (2), S. 257–264. DOI: 10.1177/1740774511427325.

Bertoye, P. H., S. Courcier-Duplantier, et al. Adaptation of the application of good clinical practice depending on the features of specific research projects. Therapie 2006; 61(4): 279-85, 271-7.

Brosteanu, Oana; Houben, Peggy; Ihrig, Kristina; Ohmann, Christian; Paulus, Ursula; Pfistner, Beate et al. (2009): Risk analysis and risk adapted on-site monitoring in noncommercial clinical trials. In: Clinical trials (London, England) 6 (6), S. 585–596. DOI: 10.1177/1740774509347398.

Buyse, M.; George, S. L.; Evans, S.; Geller, N. L.; Ranstam, J.; Scherrer, B. et al. (1999): The role of biostatistics in the prevention, detection and treatment of fraud in clinical trials. In:  Statistics in medicine  18 (24), S. 3435–3451.

Buyse M (2014): Centralized Statistical Monitoring As a Way to Improve the Quality of Clinical Data. In:  Applied Clinical Trials . Online verfügbar unter  http://www.appliedclinicaltrialsonline.com/centralized-statistical-monitoring-way-improve-quality-clinical-data?pageID=1.

Clinical Trials Transformation Initiative.  https://www.ctti-clinicaltrials.org/

Cooley S, S. B. Triggered monitoring. Applied clinical trials online.2010

DAMOCLES (2005): A proposed charter for clinical trial data monitoring committees: helping them to do their job well. Helping them to do their job well. In:  Lancet (London, England)  365 (9460), S. 711–722. DOI: 10.1016/S0140-6736(05)17965-3.

De, S. Hybrid approaches to clinical trial monitoring: Practical alternatives to 100% source data verification. Sheetz, N.; Wilson, B.; Benedict, J.; Huffman, E.; Lawton, A.; Travers, M. et al. (2014): Evaluating Source Data Verification as a Quality Control Measure in Clinical Trials. In:  Therapeutic Innovation & Regulatory Science  48 (6), S. 671–680. DOI: 10.1177/2168479014554400.

Desmet, L.; Venet, D.; Doffagne, E.; Timmermans, C.; Burzykowski, T.; Legrand, C.; Buyse, M. (2014): Linear mixed-effects models for central statistical monitoring of multicenter clinical trials. In:  Statistics in medicine  33 (30), S. 5265–5279. DOI: 10.1002/sim.6294.

Djali, S.; Janssens, S.; van Yper, S.; van Parijs, J. (2010): How a Data-Driven Quality Management System Can Manage Compliance Risk in Clinical Trials. In:  Drug Information Journal  44, S. 359–373.

Edwards, Phil; Shakur, Haleema; Barnetson, Lin; Prieto, David; Evans, Stephen; Roberts, Ian (2013): Central and statistical data monitoring in the Clinical Randomisation of an Antifibrinolytic in Significant Haemorrhage (CRASH-2) trial. In:  Clinical trials (London, England)  11 (3), S. 336–343. DOI: 10.1177/1740774513514145.

European Commission. Enterprise and Industry Directorate-General. Draft guidance on ‘specific modalities’ for non-commercial clinical trials referred to in Commission Directive 2005/28/EC laying down the principles and detailed guidelines for good clinical practice.http://ec.europa.eu/health/files/pharmacos/docs/doc2006/07_2006/guide_noncommercial_2006_07_27_en.pdf

European Medicines Agency (2002). Note for Guidance on Good Clinical Practice.http://www.emea.europa.eu/docs/en_GB/document_library/Scientific_guideline/2009/09/WC500002874.pdf.

European Medicines Agency (2013): Reflection paper on risk-based quality management in clinical trials. Online verfügbar unter https://www.ema.europa.eu/en/documents/scientific-guideline/reflection-paper-risk-based-quality-management-clinical-trials_en.pdf .

Food and Drug Administration. Guidance for Industry Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring.

Graham, L., S. Anwar, et al. Risk management of non-CTIMP trials: Focus on complex intervention trials. Clinical Trials 2012; 9(4): 517.

Grieve, A. P. (2012): Source Data Verification by Statistical Sampling. Issues in Implementation. In:  Drug Information Journal  46 (3), S. 368–377. DOI: 10.1177/0092861512442057.

Guthrie, Lauren B.; Oken, Emily; Sterne, Jonathan A. C.; Gillman, Matthew W.; Patel, Rita; Vilchuck, Konstantin et al. (2012): Ongoing monitoring of data clustering in multicenter studies. In:  BMC medical research methodology  12, S. 29. DOI: 10.1186/1471-2288-12-29.

Heels-Ansdell, D., S. Walter, et al. Central statistical monitoring of an international thromboprophylaxis trial. American Journal of Respiratory and Critical Care Medicine 2010; 181(1).

Henderson, L. (2013): Distinct RBM Divisions: source document verification and source data review. In:  Applied Clinical Trials  July.

Hughes, Sara; Harris, Julia; Flack, Nancy; Cuffe, Robert L. (2012): The statistician's role in the prevention of missing data. In:  Pharmaceutical statistics  11 (5), S. 410–416. DOI: 10.1002/pst.1528.

Journot V, Pignon JP, Gaultier C, et al. On behalf of the Optimon Collaborative Group. Validation of a risk assessment scale and a risk-adapted monitoring plan for academic clinical research studies – The Pre-Opti¬mon

Journot, Valérie; Pérusat-Villetorte, Sophie; Bouyssou, Caroline; Couffin-Cadiergues, Sandrine; Tall, Aminata; Chêne, Geneviève (2013): Remote pre enrollment checking of consent forms to reduce nonconformity. In:  Clinical trials (London, England)  10 (3), S. 449–459. DOI: 10.1177/1740774513480003.

Kendall, Brigitte; Städeli, Reto; Schegg, Belinda; Olbrich, Martin; Chen, Edmond; Harmelin-Kadouri, Rona et al. (2012): Clinical Trial Educator program - a novel approach to accelerate enrollment in a phase III International Acute Coronary Syndrome Trial. In:  Clinical trials (London, England)  9 (3), S. 358–366. DOI: 10.1177/1740774512440760.

Kirkwood, Amy A.; Cox, Trevor; Hackshaw, Allan (2013): Application of methods for central statistical monitoring in clinical trials. In:  Clinical trials (London, England)  10 (5), S. 783–806. DOI: 10.1177/1740774513494504.

Kirwan, Bridget-Anne; Lubsen, Jacobus; Brouwer, Sophie de; van Dalen, Frederik J.; Pocock, Stuart J.; Clayton, Tim et al. (2008): Quality management of a large randomized double-blind multi-centre trial: the ACTION experience. In: Contemporary clinical trials  29 (2), S. 259–269. DOI: 10.1016/j.cct.2007.10.001.

Lane, J. Athene; Wade, Julia; Down, Liz; Bonnington, Susan; Holding, Peter N.; Lennon, Teresa et al. (2011): A Peer Review Intervention for Monitoring and Evaluating sites (PRIME) that improved randomized controlled trial conduct and performance. In:  Journal of clinical epidemiology  64 (6), S. 628–636. DOI: 10.1016/j.jclinepi.2010.10.003.

Lindblad, Anne S.; Manukyan, Zorayr; Purohit-Sheth, Tejashri; Gensler, Gary; Okwesili, Paul; Meeker-O'Connell, Ann et al. (2014): Central site monitoring: results from a test of accuracy in identifying trials and sites failing Food and Drug Administration inspection. In:  Clinical trials (London, England)  11 (2), S. 205–217. DOI: 10.1177/1740774513508028.

Macefield, R. C., A. D. Beswick, et al. A systematic review of on-site monitoring methods for health-care randomised controlled trials. Clin Trials 2013; 10(1): 104-24.Mealer, Meredith; Kittelson, John; Thompson, B. Taylor; Wheeler, Arthur P.; Magee, John C.; Sokol, Ronald J. et al. (2013): Remote source document verification in two national clinical trials networks: a pilot study. In:  PloS one  8 (12), S. e81890. DOI: 10.1371/journal.pone.0081890.

McBee, W. L., S. Schenning, et al. Effective monitoring strategies in a long-term clinical trial with varying levels of clinic staff knowledge: The AREDS2 experience. Clin Trials 2012; 9(4): 462.

Méthot J, Brisson D, Gaudet D. On-site management of investigational products and drug delivery systems in conformity with Good Clinical Practices (GCPs). Clin Trials 2012; DOI: 10.1177/1740774511431280

Mitchel, J. T.; Kim, Y. J.; Choi J; Park, G.; Cappi, S.; Horn, D. et al. (2011): Evaluation of Data Entry Errors and Data Changes to an Electronic Data Capture Clinical Trial Database. In:  Drug Information Journal  45, S. 421–430.

Morrison, B., J. Neaton, et al. A CTTI survey of current monitoring practices. Clinical Trials 2010; 7(4): 464.

Morrison BW, Cochran CJ, White JG. Monitoring the quality of conduct of clinical trials: a survey of current practices. Clin Trials 2011; 8: 342-9

MRC/DH Joint project to codify good practice in publicly-funded UK clinical trials with medicines  http://www.ct-toolkit.ac.uk

MRC/DH/MHRA Joint Project. Risk-adapted Approaches to the Management of Clinical Trials of Investigational Medicinal Products (Version: 10th October 2011) http://www.mhra.gov.uk/home/groups/l-ctu/documents/websiteresources/con111784.pdf

Nielsen, E.; Hyder, D.; Deng, C. (2014): A Data-Driven Approach to Risk-Based Source Data Verification. In:  Therapeutic Innovation & Regulatory Science  48 (2), S. 173–180. DOI: 10.1177/2168479013496245.

Novik, Y., L. Fleming, et al. Monitoring data and compliance in institutional cancer trials: Consistent quality improvement by internal peer audit process. Clinical Trials 2010; 7(4): 430.

Pocock, Stuart J. (2006): Current controversies in data monitoring for clinical trials. In:  Clinical trials (London, England)  3 (6), S. 513–521. DOI: 10.1177/1740774506073467.

Pogue, Janice M.; Devereaux, P. J.; Thorlund, Kristian; Yusuf, Salim (2013): Central statistical monitoring: detecting fraud in clinical trials. In:  Clinical trials (London, England)  10 (2), S. 225–235. DOI: 10.1177/1740774512469312.

Sandman L, Mosher A, Khan A. Quality assurance in a large clinical trials consortium: The experience of the Tuberculosis Trials Consortium. Contemp Clin Trials 2006; (27): 554–560

Shugarts, P., R. Kozloff, et al. Risk-based approach to monitoring: The way of the future. Clinical Trials 2012; 9(4): 462.Grant, A.; Altman, D.; Babiker, A.; Campbell, M.; Clemens, F.; Darbyshire, J. et al. (2005): Issues in data monitoring and interim analysis of trials. In:  Health Technol Assess  9 (7). DOI: 10.3310/hta9070.

Tantsyura, V.; Dunn, I. M.; Fendt, K.; Kim, Y. J.; Waters, J.; Mitchel, J. (2015a): Risk-Based Monitoring. A Closer Statistical Look at Source Document Verification, Queries, Study Size Effects, and Data Quality. In:  Therapeutic Innovation & Regulatory Science. DOI:  10.1177/2168479015586001.

Tantsyura, V.; Dunn, I. M.; Waters, J.; Fendt, K.; Kim, Y. J.; Viola, D.; Mitchel, J. (2015b): Extended Risk-Based Monitoring Model, On-Demand Query-Driven Source Data Verification, and Their Economic Impact on Clinical Trial Operations. In: Therapeutic Innovation & Regulatory Science. DOI:  10.1177/2168479015596020.

Tantsyura, Vadim; Grimes, Imogene; Mitchel, Jules; Fendt, Kaye; Sirichenko, Sergiy; Waters, Joel et al. (2010): Risk-Based Source Data Verification Approaches: Pros and Cons. In:  Drug Information Journal  44, S. 745–756. Online verfügbar unter doi:10.1177/009286151004400611.

Taylor, R.; McEntegart, D.; Stillman, E. (2002): Statistical techniques to detect fraud and other data irregularities in clinical questionnaire data. In:  Drug Information Journal  36, S. 115–125.

Timmermans, Catherine; Doffagne, Erik; Venet, David; Desmet, Lieven; Legrand, Catherine; Burzykowski, Tomasz; Buyse, Marc (2015a): Statistical monitoring of data quality and consistency in the Stomach Cancer Adjuvant Multi-institutional Trial Group Trial. In:  Gastric cancer : official journal of the International Gastric Cancer Association and the Japanese Gastric Cancer Association. DOI:  10.1007/s10120-015-0533-9.

Timmermans, Catherine; Venet, David; Burzykowski, Tomasz (2015b): Data-driven risk identification in phase III clinical trials using central statistical monitoring. In:  International journal of clinical oncology. DOI:  10.1007/s10147-015-0877-5.

TransCelerate BioPharma Inc (2013): TransCelerate-RBM-Position-Paper-FINAL-30MAY2013. Online verfügbar unter  http://www.transceleratebiopharmainc.com/ , zuletzt geprüft am 08.10.2015.

Tudur Smith, Catrin; Stocken, Deborah D.; Dunn, Janet; Cox, Trevor; Ghaneh, Paula; Cunningham, David; Neoptolemos, John P. (2012): The value of source data verification in a cancer clinical trial. In:  PloS one  7 (12), S. e51623. DOI: 10.1371/journal.pone.0051623.

Tudur Smith C, Williamson P, Jones A, Smyth A, Langton Hewer S, Gamble C (2014): Risk-proportionate clinical trial monitoring: an example approach from a non-commercial trials unit. In:  Trials  15, S. 127. DOI: 10.1186/1745-6215-15-127.

US Food and Drug Administration (2013): Guidance for Industry Oversight of Clinical Investigations — A Risk- Based Approach to Monitoring. Online verfügbar, zuletzt geprüft am 06.10.2015.

Venet D, Doffagne E, Burzykowski T, Beckers F, Tellier Y, Genevois-Marlin E et al. (2012): A statistical approach to central monitoring of data quality in clinical trials. In:  Clinical trials (London, England)  9 (6), S. 705–713. DOI: 10.1177/1740774512447898.

Williams GW. The other side of clinical trial monitoring; assuring data quality and procedural adherence. Clin Trials 2006 3: 530-7

Wilson, B.; Provencher, T.; Gough, J.; Clark, S.; Abdrachitov, R.; Roeck, K. de et al. (2014): Defining a Central Monitoring Capability. Sharing the Experience of TransCelerate BioPharma's Approach, Part 1. In:  Therapeutic Innovation & Regulatory Science  48 (5), S. 529–535. DOI: 10.1177/2168479014546335.

Wu, Xiaoru; Carlsson, Martin (2011): Detecting data fabrication in clinical trials from cluster analysis perspective. In:  Pharmaceutical statistics  10 (3), S. 257–264. DOI: 10.1002/pst.462.

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 22 April 2024

A method for managing scientific research project resource conflicts and predicting risks using BP neural networks

• Xuying Dong 1 &
• Wanlin Qiu 1  

Scientific Reports volume  14 , Article number:  9238 ( 2024 ) Cite this article

Metrics details

• Computer science
• Engineering

This study begins by considering the resource-sharing characteristics of scientific research projects to address the issues of resource misalignment and conflict in scientific research project management. It comprehensively evaluates the tangible and intangible resources required during project execution and establishes a resource conflict risk index system. Subsequently, a resource conflict risk management model for scientific research projects is developed using Back Propagation (BP) neural networks. This model incorporates the Dropout regularization technique to enhance the generalization capacity of the BP neural network. Leveraging the BP neural network’s non-linear fitting capabilities, it captures the intricate relationship between project resource demand and supply. Additionally, the model employs self-learning to continuously adapt to new scenarios based on historical data, enabling more precise resource conflict risk assessments. Finally, the model’s performance is analyzed. The results reveal that risks in scientific research project management primarily fall into six categories: material, equipment, personnel, financial, time, and organizational factors. This study’s model algorithm exhibits the highest accuracy in predicting time-related risks, achieving 97.21%, surpassing convolutional neural network algorithms. Furthermore, the Root Mean Squared Error of the model algorithm remains stable at approximately 0.03, regardless of the number of hidden layer neurons, demonstrating excellent fitting capabilities. The developed BP neural network risk prediction framework in this study, while not directly influencing resource utilization efficiency or mitigating resource conflicts, aims to offer robust data support for research project managers when making decisions on resource allocation. The framework provides valuable insights through sensitivity analysis of organizational risks and other factors, with their relative importance reaching up to 20%. Further research should focus on defining specific strategies for various risk factors to effectively enhance resource utilization efficiency and manage resource conflicts.

Similar content being viewed by others

Optimized backpropagation neural network for risk prediction in corporate financial management

A case study on the relationship between risk assessment of scientific research projects and related factors under the Naive Bayesian algorithm

Xuying Dong & Wanlin Qiu

Prediction of SMEs’ R&D performances by machine learning for project selection

Hyoung Sun Yoo, Ye Lim Jung & Seung-Pyo Jun

Introduction

In the twenty-first century, driven by rapid technological innovation and a substantial increase in research funding, the number of scientific research projects has experienced exponential growth. These projects, serving as pivotal drivers of scientific and technological advancement, encompass a wide array of domains, including natural sciences, engineering, medicine, and social sciences, among others. This extensive spectrum attracts participation from diverse researchers and institutions 1 , 2 . However, this burgeoning landscape of scientific research projects brings forth a set of accompanying challenges and predicaments. Foremost among these challenges is the persistent issue of resource scarcity and the diversity of project requirements. This quandary poses a formidable obstacle to the management and execution of scientific research initiatives. It not only impacts the project’s quality and efficiency but can also cast a shadow on an organization’s reputation and the output of its research endeavors 3 , 4 , 5 . For instance, when two university research projects concurrently require the use of a specific instrument with limited availability or in need of maintenance, it may result in both projects being unable to proceed as planned, leading to resource conflicts. Similarly, competition for research funding from the same source can introduce conflicts in resource allocation decisions by the approval authority. These issues are widespread in research projects, and surveys indicate that project delays or budget overruns due to improper resource allocation are common in scientific research. For example, a study on research projects funded by the National Institutes of Health in the United States revealed that approximately 30% of projects faced delays due to improper resource allocation. In Europe, statistics from the European Union’s Framework Programme for Science and Innovation indicate that resource conflicts have impeded about 20% of transnational collaborative research projects from achieving their established research objectives on time. Furthermore, scientific research projects encompass a spectrum of resource requirements essential for their seamless progression, including but not limited to materials, equipment, skilled personnel, adequate funding, and time 6 . The predicament arises when multiple research initiatives necessitate identical or analogous resources simultaneously, creating a challenge for organizations to provide equitable support during peak demand periods. To mitigate the risks associated with resource conflicts, organizations must continually administer their resource allocation and strike a harmonious equilibrium between resource requisites and their availability 7 .

The Back Propagation (BP) neural network, as a prominent deep learning algorithm, boasts exceptional data processing capabilities. Notably, neural networks possess the capacity to swiftly process extensive datasets and extract intricate mapping relationships within data, rendering them versatile tools employed across various domains, including project evaluation, risk assessment, and cost prediction 8 , 9 . Scientific research project management constitutes a dynamic process. As projects advance and environmental factors evolve, the risk landscape may undergo continuous transformation 10 . The BP neural network’s inherent self-learning ability empowers it to iteratively update its model based on fresh data, enabling seamless adaptation to new circumstances and changes, thereby preserving the model’s real-time relevance 11 . In conclusion, this approach is poised to enhance project management efficiency and quality, mitigate risks, and foster the potential for the successful realization of scientific research projects.

The primary objective of this study is to formulate an evaluation and risk prediction framework for scientific research project management utilizing the BP neural network. This framework aims to address the issues associated with resource discrepancies and conflicts within the realm of scientific research project management. This study addresses the primary inquiry: What types of resource conflict risks exist in scientific research project management? An extensive literature review and empirical data analysis are conducted to answer this question, identifying six main risk categories: materials, equipment, personnel, finance, time, and organizational factors. A comprehensive resource conflict risk index system is constructed based on these categories. To quantitatively assess the importance of different resource conflict risk factors, the Analytic Hierarchy Process (AHP) is employed. This method allowed for the quantification of the influence of each risk factor objectively and accurately by constructing judgment matrices and calculating the weights of each factor. Subsequently, exploration is conducted into the utilization of BP neural networks to construct a resource conflict risk management model for scientific research projects. A BP neural network model is developed incorporating Dropout technology to capture complex correlations between project resource demand and supply. This model self-learns to adapt to new scenarios in historical data, thereby improving prediction accuracy. Research project data is collected from several universities in Xi’an from September 2021 to March 2023 to validate the effectiveness and accuracy of the proposed model. This data is utilized to train and test the model, and its performance is compared with other advanced algorithms such as CNN and BiLSTM. The evaluation is based on two key metrics: accuracy and root mean square error (RMSE), demonstrating excellent fitting ability and prediction accuracy.

The innovation introduced in this study is rooted in the recognition that the proliferation of scientific research initiatives can precipitate resource conflicts and competition, potentially leading to adverse outcomes such as project failure or resource inefficiency. This study harnesses a multi-layer BP neural network as its central computational tool, concomitantly incorporating the establishment of a resource conflict risk index system. This comprehensive model for evaluating and predicting the risks in scientific research project management takes into account both the resource conflict risk index system and the intrinsic characteristics of the BP neural network. This combined approach serves to enhance the efficiency of managing scientific research projects, curtail resource wastage, mitigate the risk of resource conflicts, and ultimately furnish robust support for the enduring success of scientific research endeavors.

Related work

Current research landscape in scientific research project management.

Scientific research projects hold a pivotal role in advancing scientific and technological frontiers, fostering knowledge generation, and driving innovation. Effective project management in this context ensures the timely delivery, adherence to budgetary constraints, and attainment of predefined quality standards. Numerous scholars have contributed to the body of knowledge concerning scientific research project management. Significant risks in scientific research project management include improper resource allocation, time delays, budget overruns, and collaboration challenges. For instance, concerning time management, Khiat 12 illustrated that insufficient project planning or external factors often hinder project deadlines. Regarding financial management, Gao 13 highlighted the lack of transparency in fund allocation and unreasonable budgeting, leading to unnecessary research cost overruns. Previous studies have predominantly concentrated on developing diverse methodologies and tools to identify and assess potential risks in scientific research projects. For instance, quantitative models have been employed by researchers like Jeong et al. 14 to evaluate project failure probabilities and devise corresponding risk mitigation strategies. Concurrently, Matel et al. 15 utilized artificial intelligence (AI), including neural networks and machine learning, to conduct comprehensive analyses of project data and predict potential issues throughout project progression.

The preceding studies offer essential groundwork and insights for the scientific research project management discussed in this study. They illuminate key risks encountered in scientific research project management, including inadequate resource allocation, time constraints, budgetary overruns, and collaboration hurdles. These risks are pervasive in scientific research project management, directly impacting project execution efficiency and outcomes. Moreover, these studies furnish empirical data and case studies, elucidating the underlying causes and mechanisms of these risks. For example, the research conducted by Khiat and Gao offers a nuanced understanding of risk factors, enriching the comprehension of the challenges in scientific research project management. Additionally, these studies introduce diverse methods and tools for identifying and evaluating potential risks in scientific research projects. For instance, the works of Jeong et al. and Matel et al. utilize quantitative models and artificial intelligence techniques to comprehensively analyze project data and forecast potential issues in project advancement. These methodologies and tools serve as valuable resources for constructing the research framework and methodologies in this study. Despite the commendable strides made in employing multidisciplinary approaches to address the challenges posed by scientific research project management, the issues related to resource allocation conflicts and quality assurance during project implementation remain fertile ground for future exploration and active investigation.

Application of BP neural network in project risk and resource management

BP neural networks are renowned for their non-linear fitting and self-learning capabilities, rendering them invaluable for discerning intricate relationships and patterns in project management. Their applications span diverse areas, including resource allocation, risk assessment, schedule forecasting, cost estimation, and more, culminating in heightened efficiency and precision within project management practices. Numerous scholars have ventured into the realm of BP neural network applications within project management. Zhang et al. 16 introduced a real-time network attack detection method underpinned by deep belief networks and support vector machines. Their findings underscore the method’s potential for bolstering network security risk management, extending novel data security safeguards to scientific research project management. Gong et al. 17 devised an AI-driven human resources management system. This system autonomously evaluates employee performance and needs, proffering intelligent managerial recommendations. Bai et al. 18 harnessed BP neural networks to tackle the intricate challenge of selecting service providers for project management portfolios. Leveraging neural networks, they prognosticate the performance of diverse service providers, lending support to project management decision-making. Sivakumar et al. 19 harnessed BP neural networks to prognosticate the prioritization of production facilities in the bus body manufacturing sector. Their work serves as an illustrative testament to the potential of neural networks in the production and resource allocation facets of scientific project management. Liu et al. 20 undertook an analysis of the influential factors and early warning signs pertaining to construction workers’ safety conditions. This investigation underscores the profound implications of neural networks in safety management within the context of engineering and construction project management. Li et al. 21 harnessed optimized BP neural networks to anticipate risks in the financial management arena of listed companies. Their outcomes underscore the utility of neural networks in financial management, providing an exemplar of a risk assessment tool for scientific research project management.

The comprehensive analysis of the aforementioned studies reveals that BP neural networks exhibit substantial capabilities in scrutinizing historical project data, discerning intricate resource demand–supply dynamics, and offering valuable insights for project management decisions and optimizations. These applications underscore the potential of BP neural networks as indispensable tools within the project management domain. Nonetheless, several challenges persist, particularly concerning the real-time adaptability of BP neural networks and their capacity to cater to dynamic project management requisites.

Research in the field of scientific research project resource management and risk prediction

Within the realm of scientific research project resource management and risk prediction, various studies by notable scholars warrant attention. Jehi et al. 22 employed statistical models for risk prediction but overlooked the intricate resource conflict relationships within scientific research projects. Efficient project resource management and accurate risk prediction are pivotal for ensuring smooth project execution and attaining desired outcomes. Asamoah et al. elucidated that scientific research projects necessitate both tangible and intangible resources 23 , encompassing materials, equipment, personnel, funding, and time. The judicious allocation and optimal utilization of these resources significantly influence project progress and outcomes. Misallocation of resources can lead to setbacks such as project delays and budget overruns. Meanwhile, Zwikael et al. identified organizational culture, awareness, support, rewards, and incentive programs as key drivers impacting the effective management of scientific research project benefits 24 . These risks can profoundly affect project advancement and outcomes, underscoring the importance of accurate prediction and adept management. Farooq et al. advocated for scientific project management, emphasizing the need for enhanced risk management strategies and management efficacy to foster sustainable enterprise development 25 .

In conclusion, studies on project resource management and risk prediction encompass diverse facets, including resource allocation, risk assessment, and model development. These efforts offer essential theoretical and methodological underpinnings for the effective execution of scientific research endeavors. Given the ongoing expansion and growing complexity of scientific projects, further research on resource management and risk prediction is imperative to navigate increasingly intricate circumstances.

A comprehensive review of methods employed in scientific research project management and risk assessment reveals a predominant focus on quantitative analysis, qualitative research, and the integration of AI techniques. In particular, the utilization of BP neural networks, as demonstrated in studies such as Sivakumar et al., Liu et al., and Li et al., underscores their capacity to furnish real-time data analysis and decision-making support for project managers. However, it remains evident that challenges persist in harnessing the full potential of BP neural networks in terms of real-time adaptability and resource allocation within the multifaceted landscape of dynamic project management. Hence, this study accentuates the existing methodological challenges associated with resource conflict resolution, risk management, and overall scientific research project management. Through the optimization and refinement of BP neural network applications in risk assessment, this study strives to furnish organizations with effective decision-making tools. Ultimately, the insights gleaned from this study aim to serve as a valuable reference for scientific research project managers as they navigate the complexities of project risk management.

Prediction method for scientific research project management risks based on the BP neural network

Analysis of the construction of a scientific research project management risk system.

Scientific research project management constitutes a specialized discipline encompassing the planning, organization, execution, and oversight of scientific research endeavors. Its primary objective is to facilitate the effective attainment of research objectives and anticipated outcomes. The overarching aim of scientific research project management is to optimize resource allocation, schedule planning, and risk mitigation, thereby ensuring the successful culmination of research projects 26 , 27 . A visual representation of the fundamental task processes integral to scientific research project management is depicted in Fig.  1 .

Schematic representation of key scientific research project management tasks.

Scientific research project management, as illustrated in Fig.  1 , constitutes an essential framework to ensure the efficient and organized execution of scientific research endeavors. It encompasses four core phases: project planning and initiation, project execution and monitoring, project closure and summarization, and project communication and feedback 28 . The meticulous determination of project requisites is of particular significance, encompassing financial resources, personnel, equipment, materials, and more. Failure to ensure the effective utilization and judicious allocation of these resources during project management may introduce the risk of hindrances in the smooth progress and achievement of the research project’s envisioned objectives.

Ongoing scientific research projects necessitate an array of resources, encompassing both tangible assets such as materials, equipment, and funds, and intangible elements like time, personnel expertise, and organizational support 29 , 30 . These resources are intricately interwoven within scientific research projects and collectively influence project success. However, when confronted with limited total resources, resource conflicts can arise when multiple projects vie for the utilization of the same resources. Consequently, this study has devised a resource conflict risk index system tailored for the management of scientific research projects. This system stratifies risks according to the categories of resources implicated in the project implementation process, as depicted in Fig.  2 . In this study, ensuring the representativeness and comprehensiveness of risk assessment for resource conflicts in scientific research project management is pivotal. A multifaceted and systematic approach is adopted to define risk categories. A comprehensive literature review initially identifies common resource conflicts in scientific research project management. Subsequently, through interviews and surveys with industry research project managers, firsthand information on specific challenges and risk factors encountered during project execution is collected. Additionally, referencing international standards and best practices ensures the authority and applicability of risk classification. The outcome of these efforts is illustrated in Fig.  2 , showcasing a meticulously designed resource conflict risk index system. It encompasses six major categories: equipment risk, material risk, personnel risk, financial risk, time risk, and organizational risk, further subdivided into 17 specific sub-items. Acknowledging the complexity and diversity of research projects, it is recognized that, despite efforts made, other potential risks may not be included in the current model. A dynamic iterative approach is proposed to address this challenge, integrate additional risk factors, and continuously optimize the model. Specific steps are outlined to enhance the model’s capabilities. Firstly, establishing a monitoring system to regularly collect user feedback and industry updates allows the prompt discovery and incorporation of new risk factors. Simultaneously, closely monitoring the latest research findings in the domestic and international scientific research project management field ensures the continuous integration of new discoveries from academia. Additionally, a dedicated team conducts regular in-depth reviews of the existing risk index system, adding, deleting, or adjusting the weights of risk factors as needed based on actual requirements. This process enables the model to better adapt to the current project management environment and future trends. Secondly, utilizing the newly integrated dataset to cross-validate the model ensures that the newly added risk factors are appropriately assessed and predicted. By comparing the performance of different versions of the model, a more accurate measurement of the effects of optimization is achieved. Finally, research project managers are encouraged to provide real-time feedback, including the model’s performance in actual applications, overlooked risk points, and improvement suggestions, enhancing the model’s usability and reliability. These methods aim to construct a more refined, flexible, and adaptable scientific research project risk assessment model that continuously evolves to meet changing needs. Through continuous optimization and improvement, this model is believed to more effectively assist project managers in making risk-based decisions and promote the success rate of scientific research projects.

Resource conflict risk indicator system for scientific research project management.

As depicted in Fig.  2 , this risk system underscores the significance of material quality and timely supply in project execution. The establishment of this resource conflict risk indicator system forms a fundamental basis for subsequent model development and risk forecasting, empowering project managers to gain comprehensive insights into and effectively manage resource conflict risks.

Weight analysis process using APH for the risk indicator system

The AHP is primarily employed for the comprehensive analysis of multifaceted problem systems, involving the segmentation of interrelated factors into hierarchical levels. It subsequently facilitates objective assessments at each tier. This method typically deconstructs problems into a tripartite structure comprising the following levels: the objective layer (highest), the criteria layer (intermediate), and the indicator layer (fundamental) 31 , 32 . In this context, the objective layer pertains to the project’s resource conflict risk, which represents the core challenge addressed by this structural model. The criteria layer provides an initial decomposition of the objective layer and establishes the foundational logical framework for third-level indicators. The indicator layer encompasses risk factors, specifically, the potential triggers for resource conflict risks. The weight analysis process, employing the AHP for the risk indicator system, is delineated in Fig.  3 .

Weight analysis process of applying the hierarchical analysis method to the risk indicator system.

In Fig.  3 , the application of the AHP to the weight analysis of the scientific research project management risk indicator system follows a general procedure: sequentially defining individual problems, creating a hierarchical structural model, constructing pairwise comparison matrices, performing hierarchical ranking calculations and consistency tests, and finally, selecting evaluation criteria systematically for assessment.

The initial step involves breaking down the intricate problem into distinct components, creating a hierarchical structure model comprising the target layer, criterion layer, and indicator layer.

In this phase, the assessment of relative importance between elements leads to the formation of a pairwise comparison judgment matrix, denoted as matrix A , as depicted in Eq. ( 1 ).

In Eq. ( 1 ), \(a_{ij} > 0\) , \(a_{ji} = 1/a_{ij}\) , and \(a_{ii} = 1\) .

The AHP calculations are performed following the classic methodology proposed by Rehman 33 . The process begins by computing the product M i of the elements within each row, as illustrated in Eq. ( 2 ).

The next step involves calculating the n -th root of M i , as described in Eq. ( 3 ).

Next, the process involves normalizing \(W = \left[ {W_{1} ,W_{2} , \cdots ,W_{n} } \right]^{T}\) , as shown in Eq. ( 4 ).

Finally, the maximum eigenvalue \(\lambda_{\max }\) is calculated via Eq. ( 5 ).

The calculation of weights and the consistency test of the judgment matrix involve the use of the eigenvalue method to calculate the weight vector of the judgment matrix. This is demonstrated in Eq. ( 6 ).

In Eq. ( 6 ), \(\lambda_{\max }\) denotes the maximum characteristic root of A , Q signifies the eigenvector, and the weight vector is obtained by normalizing Q.

Continuing with the consistency testing, the weight vector must undergo evaluation for consistency. To initiate this evaluation, calculate the Consistency Index ( C.I. ) using Eq. ( 7 ).

Next, it is imperative to determine the corresponding average Random Consistency Index ( R.I. ). Subsequently, the Consistency Ratio ( C.R. ) is computed using the formula presented in Eq. ( 8 ).

If the calculated C.R. is less than 0.1, it indicates that the judgment matrix meets the prescribed consistency criteria, and the assigned weight values for each indicator are considered valid. However, if the calculated C.R. equals or exceeds 0.1, this signals the need for adjustments to the judgment matrix. To address this, the matrix is re-evaluated, and consistency checks are repeatedly performed until the matrix achieves the required level of consistency.

Analyzing the resource conflict risk management model for scientific research projects based on the BP neural network

This section focuses on predicting and evaluating the potential occurrence of various risk factors within scientific research projects. The objective is to facilitate the selection of appropriate response strategies aimed at minimizing losses stemming from risks associated with scientific research endeavors. Resource management within scientific research projects is a complex undertaking, with resource conflict risks influenced by a multitude of factors. Furthermore, as projects evolve, the risk landscape undergoes dynamic changes. In contrast to conventional statistical models, BP neural networks offer distinctive advantages. They employ a combination of forward signal propagation and reverse error-adjustment learning techniques, showcasing exceptional self-learning capabilities, distributed knowledge storage, and associative memory functions 34 . The BP neural network model, rooted in the backpropagation algorithm, evolved from the necessity to simulate biological neural systems and meet the demands of machine learning. Originating in the 1980s, it became a prominent deep learning model, continually iterating and adjusting connection weights to minimize the error between output and target. This learning mechanism allows the BP neural network to adapt to complex non-linear relationships, showcasing robust approximation and generalization capabilities. Over time, enhanced computer hardware and algorithm optimization led to widespread application and development of the BP neural network model. Algorithmically, various improvements, including the momentum method, adaptive learning rate, and regularization, were introduced to boost training speed and generalization ability, addressing challenges such as susceptibility to local minima in traditional BP algorithms. The advent of deep learning saw the integration of the BP neural network into deeper structures like ResNet and CNN, enabling it to handle more intricate tasks and data. The model’s applicability expanded across diverse domains, including image and speech recognition, natural language processing, financial forecasting, and medical diagnosis, yielding breakthrough results. Moreover, technological advancements like big data and cloud computing have enhanced the training and application efficiency of the BP neural network model, presenting new avenues for development. In conclusion, the evolution of the BP neural network model stems from algorithmic refinements, structural enhancements, and broadened applications, providing potent tools for addressing diverse practical challenges. The data transmission process of the BP neural network is illustrated in Fig.  4 .

Data transmission flow chart of the BP neural network.

Figure  4 illustrates the data transmission process in the BP neural network, highlighting forward propagation, which entails processing and transmitting received data information. This unidirectional propagation begins at the input layer, traverses through the hidden layers, and culminates in the output layer to yield the network’s overall output. Let the received input data be denoted as X  = ( x 1 , x 2 …, x n ), with ‘ n ’ signifying the number of neurons in the input layer. The connections between the input layer and the hidden layer initially possess randomized weight values. This citation is derived from Liu et al.’s recommendation 35 to prevent premature convergence to local minima during the training process. Representing the weight of the connection between the i -th neuron in the input layer and the j -th neuron in the hidden layer as W ij . The notation follows Narkhede et al.’s study 36 , which offers a comprehensive explanation of neural network fundamentals and operational principles. The information received by the hidden layer is expressed in Eq. ( 9 ).

In Eq. ( 9 ), i represents the number assigned to neurons in the input layer, ‘ j ’ pertains to the number of neurons in the hidden layer, and A  = ( a 1 , a 2 …, a m ) symbolizes the input variables received by the hidden layer. Upon receiving these variables, the hidden layer neuron transforms them into the output value of the hidden layer using the activation function. The methodology in this section draws from the research by Narengbam et al. 37 on activation functions in deep learning models. Specifically, the treatment of the output layer mirrors that of the hidden layers, and the computation of output layer neurons adheres to the methodology outlined in the cited literature.

In Eq. ( 10 ), Y  = ( y 1 , y 2 …, y m ) represents the output variables of the hidden layer. The computation method for the input and output values of the output layer parallels that of the hidden layer. The weight denoted as v jk signifies the connection between the j -th neuron in the hidden layer and the k -th neuron in the output layer. The information received by the output layer is described in Eq. ( 11 ).

The output value of the output layer neurons, once activated by the activation function, is expressed in Eq. ( 12 ).

At this juncture, the output value O denoted as \(O = \left( {o_{1} ,o_{2} , \cdots ,o_{z} } \right)\) is obtained, signifying the conclusion of the forward propagation process.

In the backpropagation process, the loss function J quantifies the error between the neural network’s output value and the true value (referring to the definition and application of the loss function in neural network optimization as articulated by Özden et al. 38 ), as illustrated in Eq. ( 13 ).

During the neural network’s training process, the weight, denoted as W , and the bias vector, denoted as b , play essential roles. The gradient descent method is employed to optimize the neural network (derived from Kumar et al.’s 39 analysis of the effectiveness of optimization algorithms in deep learning training). Each iteration within the gradient descent method updates the parameters W and b as per Eqs. ( 14 ) and ( 15 ).

where α represents the learning rate. The crucial step involves computing derivatives using backpropagation, employing the BP algorithm to calculate \(\frac{\partial }{{\partial W_{ij}^{\left( l \right)} }}J\left( {W,b;x,y} \right)\) and \(\frac{\partial }{{\partial b_{i}^{\left( l \right)} }}J\left( {W,b;x,y} \right)\) . These two components represent the derivatives of the cost function J ( W , b ; x , y ) for a single sample ( x , y ). Once this derivative is computed, deriving the derivatives of the overall cost function J ( W , b ; x , y ) becomes relatively straightforward. The calculated results are presented in Eqs. ( 16 ) and ( 17 ).

This study aims to develop a resource conflict risk management model tailored to predict and assess the resource conflict risks inherent in scientific research projects during execution. Resource conflicts arise from competition for limited resources like equipment, funding, and personnel among multiple projects. If unaddressed, these conflicts can significantly impede project progress and outcomes. The model’s specific objectives are to analyze project-related information (e.g., project scale, duration, funding, personnel allocation) to predict potential conflict points in resource allocation, enabling project managers to proactively mitigate or avoid conflicts and optimize resource utilization effectively. To achieve these objectives, we employ a BP neural network approach for model construction, chosen for its superior non-linear mapping capability and self-learning characteristics, enabling it to learn from extensive historical project data and identify complex resource conflict risk patterns. The model construction entails key steps: Data preprocessing involves cleaning and normalizing collected project data to meet model input requirements. Feature selection entails choosing highly correlated feature variables associated with resource conflict risks as model inputs based on expert knowledge and data analysis results. Model training and validation involve training the BP neural network with labeled historical project data and evaluating and optimizing model performance through techniques like cross-validation. Through these methods, the developed model accurately predicts resource conflict risks in scientific research project management, providing decision support for project managers to enhance resource utilization efficiency and foster successful project completion.

While the BP neural network possesses robust learning and non-linear fitting capabilities, inadequate training data can lead to suboptimal fitting. In some cases, the network may only excel at learning from a limited dataset, generating a mapping function (typically represented as a weight vector) that closely matches the training dataset. Consequently, it may struggle to generalize well to new data, exhibiting insufficient generalization abilities. This scenario is known as overfitting. To mitigate overfitting, this study introduces the Dropout regularization method 40 when applying the BP neural network to scientific research project risk management. The Dropout method involves freezing nodes within the input and hidden layers. It is particularly useful when specific neuron correlations in the input layer hinder continuous error convergence during training. The node freezing rate should strike a balance—not too low, as it would have an insignificant impact on the neural network, and not too high, which could lead to underfitting. Therefore, this study sets the node freezing rate for the Dropout regularization method at 50%. By incorporating the Dropout method into the BP neural network, the network topology used for managing resource conflict risks in scientific research projects, based on the BP neural network, is depicted in Fig.  5 .

Network topology based on the BP neural network applied to the resource conflict risk management model for scientific research projects.

As depicted in Fig.  5 , this model incorporates a novel approach. During each training iteration, a randomly selected set of neurons, encompassing those associated with equipment, materials, and organizational risk factors, is temporarily frozen. These frozen neurons do not participate in either the forward propagation calculations or the subsequent backpropagation error adjustments within the current training cycle. The weights connecting these neurons to others retain their previous states or revert to their initial values from the last training update. As the next training iteration commences, the neurons previously frozen are unfrozen, and a new batch of neurons is randomly chosen for freezing. This iterative process effectively bolsters the BP neural network’s ability to generalize from limited data, particularly when addressing resource conflict risk management in research projects.

The integration of the Dropout method into the BP neural network introduces further opportunities for optimization. Adjustments to the network’s depth, the number of neurons, and the choice of activation functions within the risk prediction model can be made. The specific optimization procedure for the BP neural network is outlined in Fig.  6 .

Flowchart presenting the pseudocode algorithm for optimizing the BP neural network.

Experimental evaluation

To assess the performance of the resource conflict risk management model developed in this study, a BP neural network was constructed utilizing the ‘newff’ function within MATLAB. Python was employed for data preprocessing and algorithm implementation. The training of the BP neural network involved configuring parameters for net.trainFcn and net. trainParam following network initialization. Training iterations continued until the error met the predefined performance criterion. The dataset utilized in this study consisted of research project information spanning all universities in Xi’an, China, from September 2021 to March 2023. In comprehensively evaluating the performance of the resource conflict risk management model developed in this study, the scope and objectives of data collection are first determined, focusing primarily on scientific research projects at major universities in the Xi’an area. Data sources included publicly available project records, official website information, and pertinent research project databases. The utilization of web scraping techniques facilitates automated data collection, encompassing details such as project names, principal investigators, start and completion dates, funding particulars, research areas, and participating personnel. Rigorous anonymization and encryption measures are implemented to uphold information security. Subsequently, to enhance understanding of the data characteristics, exploratory data analysis is conducted on the cleaned dataset. This involves calculating descriptive statistics, conducting distribution tests, and performing correlation analysis. Such steps aid in identifying the most influential feature variables for the predictive model. Given that raw data often contain missing values, outliers, or inconsistencies, comprehensive data cleaning is executed, which includes imputation of missing values, removal of outlier data, and standardization of data formats. To safeguard individual privacy, sensitive information such as project leader names undergoes anonymization and encryption. Concerning the application of the AHP in this study, this method is employed to ascertain the relative weights of various risk factors (including materials, equipment, funding, time, personnel skills, and organizational support). The operational process involves establishing a pairwise comparison judgment matrix based on expert assessments and historical data analysis. Each element in the matrix reflects the importance of one risk factor relative to another. The weights of each risk factor are determined by calculating the maximum eigenvalue of the judgment matrix and its corresponding eigenvector. Consistency indices and random consistency ratios are used to verify the consistency of the judgment matrix, deeming the derived weights acceptable only when the random consistency ratio is below 0.1. Using these meticulously assigned weighted risk factors throughout the model evaluation process, resource conflict risk prediction is conducted via the BP neural network using data collected from actual scientific research projects.

Subsequently, rigorous data anonymization procedures were applied, including de-identification, data anonymization, and encryption of sensitive information. The data preprocessing workflow encompassed comprehensive data cleaning to rectify missing or outlier data points. Ultimately, data from 8,175 research projects were amassed and segregated into training and testing subsets, with an 80% to 20% partition ratio.

To assess the performance of the model developed in this study, an initial step involved employing the AHP to evaluate the weights assigned to each factor, including materials, equipment, funds, time, personnel skills, and organization. Subsequently, the algorithm presented in this study was combined with the Convolutional Neural Network (CNN) 41 , Bidirectional Long Short-Term Memory (BiLSTM) 42 , and comparative experiments were conducted in alignment with recent studies conducted by Liu et al. and Li et al. The evaluation primarily relied on accuracy and RMSE as key metrics, precisely measuring model prediction accuracy. Additionally, the Garson sensitivity analysis method was employed to assess the sensitivity of risk factors across various algorithms.

Results and discussions

Analysis of weights and sensitivity results of different factors.

The analysis of weights and sensitivities for various factors is depicted in Figs.  7 and 8 .

Weight results of different factors.

Sensitivity results of different factors.

Figure  7 highlights the various risk factors present in scientific research project management, including materials, equipment, funds, time, personnel skills, and organization. A more in-depth examination of the weight of sub-indicators within each factor reveals that A 21 holds the highest weight value, at 0.705, while A 63 carries the smallest weight value. Consequently, the application of the AHP in this study enables a clear representation of the significance of each influencing factor. This, in turn, facilitates a more targeted and informed decision-making process, allowing for decisions that align better with the actual circumstances and desired outcomes.

Figure  8 reveals notable variations in the sensitivity of each risk factor to the model’s output variables. Organizational risk emerges as the most influential factor on the comprehensive risk value, accounting for a relative importance of 20.31%. Following closely are financial risk at 18.84%, personnel risk at 18.30%, material risk at 17.04%, equipment risk at 16.29%, and time risk at 9.24%. A more detailed scrutiny of the sensitivity of individual sub-indicators within each factor uncovers that A52 exhibits the lowest sensitivity, standing at 4.28%, while A63 records the highest sensitivity, reaching 7.84%.

Model performance comparison results under different algorithms

In-depth analysis encompassed evaluating the accuracy and RMSE outcomes of distinct algorithms across diverse indicators, as depicted in Figs.  9 and 10 .

Visual representation of accuracy results achieved by different algorithms across various factors.

RMSE comparison results of each algorithm under different numbers of neurons.

Figure  9 illustrates that the accuracy of various algorithms remains relatively stable across different index factors. Notably, the risk prediction accuracy achieved by the algorithm proposed in this study outperforms other model algorithms across various factors. The highest risk prediction accuracy is observed in the time factor, reaching an impressive 97.21%, while the equipment factor yields the lowest prediction accuracy, hovering around 80%. Upon further comparison of risk prediction accuracy across algorithms, it becomes evident that the model algorithms proposed in this study outperform Li et al.’s model algorithm and Liu et al.’s model algorithm. Additionally, the proposed model algorithm surpasses BiLSTM and CNN. Consequently, this study’s model algorithm effectively identifies risk factors in the management of scientific research projects.

Figure  10 presents the RMSE results of each algorithm, and it is evident that increasing the number of hidden layer neurons does not significantly alter the RMSE values. Specifically, the RMSE of the model algorithm introduced in this study consistently remains around 0.03. In contrast, other model algorithms yield RMSE values exceeding 0.031, indicating higher errors compared to the model proposed in this study. When arranging the RMSE results in ascending order, it becomes apparent that the order is as follows: the model algorithm introduced in this study has the lowest RMSE, followed by Li et al.’s proposed model algorithm, Liu et al.’s proposed model algorithm, BiLSTM, and CNN. Therefore, the research model demonstrates effective risk prediction in scientific research project management, characterized by lower identification errors and superior fitting capabilities.

This study established a resource conflict risk index system for scientific research project management and introduced a BP neural network as a risk prediction model. Leveraging its non-linear fitting and self-learning capabilities, the model effectively captured intricate resource demand and supply dynamics, enabling a more precise assessment of resource conflict risks. The performance evaluation revealed the model’s strength in predicting time-related risks, achieving an accuracy rate of 97.21% with an RMSE consistently around 0.03, indicating strong fitting capabilities. The developed BP neural network model in this study effectively predicts resource conflict risks in scientific research project management, serving as a valuable decision support tool for risk assessment. However, certain limitations are acknowledged in this research. Firstly, the dataset is derived from universities in a specific region (Xi’an), and although sizable, it may not comprehensively represent all types of scientific research projects. Future endeavors could involve incorporating more diverse and extensive data sources to enhance the model’s universality and robustness. Secondly, despite the notable advantages of BP neural networks in addressing non-linear problems, the selection of appropriate network structures and parameter settings remains a challenge. Subsequent work could focus on further enhancing the network’s performance through the exploration of additional optimization algorithms. In terms of future research directions, the following points are proposed: Firstly, considering the integration of various machine learning and deep learning technologies to obtain more comprehensive risk prediction results. Secondly, exploring the application of the model in scientific research projects of different scales and types to validate and broaden its applicability. Lastly, investigating the integration of the model into a real-time project management system can provide project managers with dynamic risk monitoring and warning services.

Data availability

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Ren, S. et al. The emerging driving force of inclusive green growth: Does digital economy agglomeration work?. Bus. Strateg. Environ. 31 (4), 1656–1678 (2022).

Article   Google Scholar  

Wang, W., Hu, Y. & Lu, Y. Driving forces of China’s provincial bilateral carbon emissions and the redefinition of corresponding responsibilities. Sci. Total Environ. 857 , 159404 (2023).

Article   ADS   CAS   PubMed   Google Scholar  

Do, S. T., Nguyen, V. T. & Likhitruangsilp, V. RSIAM risk profile for managing risk factors of international construction joint ventures. Int. J. Constr. Manag. 23 (7), 1148–1162 (2023).

Google Scholar  

Nguyen, H. D., Do, Q. N. H. & Macchion, L. Influence of practitioners’ characteristics on risk assessment in Green Building projects in emerging economies: A case of Vietnam. Eng. Constr. Archit. Manag. 30 (2), 833–852 (2023).

Shayan, S., Pyung Kim, K. & Tam, V. W. Y. Critical success factor analysis for effective risk management at the execution stage of a construction project. Int. J. Constr. Manag. 22 (3), 379–386 (2022).

Alam, I., Sarwar, N. & Noreen, I. Statistical analysis of software development models by six-pointed star framework. PLoS ONE 17 (4), e0264420 (2022).

Article   CAS   PubMed   PubMed Central   Google Scholar  

Pham, H. T. et al. Supply chain risk management research in construction: A systematic review. Int. J. Constr. Manag. 23 (11), 1945–1955 (2023).

Zhao, Y. et al. Predicting delays in prefabricated projects: SD-BP neural network to define effects of risk disruption. Eng. Constr. Archit. Manag. 29 (4), 1753–1776 (2022).

Zhang, X. et al. Application of grey feed forward back propagation-neural network model based on wavelet denoising to predict the residual settlement of goafs. PLoS ONE 18 (5), e0281471 (2023).

Article   PubMed   PubMed Central   Google Scholar  

El Khatib, M., Al Mulla, A. & Al, K. W. The role of blockchain in E-governance and decision-making in project and program management. Adv. Internet Things 12 (3), 88–109 (2022).

Ujong, J. A., Mbadike, E. M. & Alaneme, G. U. Prediction of cost and duration of building construction using artificial neural network. Asian J. Civil Eng. 23 (7), 1117–1139 (2022).

Khiat, H. Using automated time management enablers to improve self-regulated learning. Act. Learn. High. Educ. 23 (1), 3–15 (2022).

Gao, J. Analysis of enterprise financial accounting information management from the perspective of big data. Int. J. Sci. Res. 11 (5), 1272–1276 (2022).

Jeong, J. & Jeong, J. Quantitative risk evaluation of fatal incidents in construction based on frequency and probability analysis. J. Manag. Eng. 38 (2), 04021089 (2022).

Matel, E. et al. An artificial neural network approach for cost estimation of engineering services. Int. J. Constr. Manag. 22 (7), 1274–1287 (2022).

Zhang, H. et al. A real-time and ubiquitous network attack detection based on deep belief network and support vector machine. IEEE/CAA J. Autom. Sin. 7 (3), 790–799 (2020).

Gong, Y. et al. Design and interactive performance of human resource management system based on artificial intelligence. PLoS ONE 17 (1), e0262398 (2022).

Bai, L. et al. Service provider portfolio selection for project management using a BP neural network. Ann. Oper. Res. 308 , 41–62 (2022).

Article   MathSciNet   Google Scholar  

Sivakumar, A. et al. Prediction of production facility priorities using Back Propagation Neural Network for bus body building industries: A post pandemic research article. Qual. Quant. 57 (1), 561–585 (2023).

Article   CAS   PubMed   Google Scholar  

Liu, N. et al. Influencing factors and prewarning of unsafe status of construction workers based on BP neural network. Appl. Sci. 13 (6), 4026 (2023).

Article   CAS   Google Scholar  

Li, X., Wang, J. & Yang, C. Risk prediction in financial management of listed companies based on optimized BP neural network under digital economy. Neural Comput. Appl. 35 (3), 2045–2058 (2023).

Jehi, L. et al. Individualizing risk prediction for positive coronavirus disease 2019 testing: Results from 11,672 patients. Chest 158 (4), 1364–1375 (2020).

Asamoah, R. O. et al. Identifying intangible resources to enhance profitability strategies of Small-Medium Scale Construction Firms (SMSCFs) in developing countries. Int. J. Construct. Manag. 22 (11), 2207–2214 (2022).

Zwikael, O. & Huemann, M. Project benefits management: Making an impact on organizations and society through projects and programs. Int. J. Project Manag. 41 (8), 102538 (2023).

Farooq, R. A review of knowledge management research in the past three decades: A bibliometric analysis. VINE J. Inf. Knowl. Manag. Syst. 54 (2), 339–378 (2024).

Bergevin, M. D. et al. Cache a Killer: Cache Valley virus seropositivity and associated farm management risk factors in sheep in Ontario, Canada. PLoS ONE 18 (8), e0290443 (2023).

Huang, G., Lee, S. M. & Clinciu, D. L. Competitive advantages of organizational project management maturity: A quantitative descriptive study in Australia. PLoS ONE 18 (6), e0287225 (2023).

Yesica, R. et al. Project management office manager’s competences: Systematic literature review. Int. J. Project Organ. Manag. 15 (2), 253–278 (2023).

Yu, C. & Hsiao, Y. C. IT project management resource: Identifying your project’s common goals. Int. J. Inf. Technol. Project Manag. 13 (1), 1–15 (2022).

Qu, S. et al. The performance evaluation of management mode of small water resources projects. PLoS ONE 18 (4), e0282357 (2023).

Wu, Z. et al. Urban flood risk assessment in Zhengzhou, China, based on a D-number-improved analytic hierarchy process and a self-organizing map algorithm. Remote Sens. 14 (19), 4777 (2022).

Article   ADS   Google Scholar  

Lin, C. L., Fan, C. L. & Chen, B. K. Hybrid analytic hierarchy process-artificial neural network model for predicting the major risks and quality of Taiwanese construction projects. Appl. Sci. 12 (15), 7790 (2022).

Rehman, A. et al. Multi-hazard susceptibility assessment using the analytical hierarchy process and frequency ratio techniques in the Northwest Himalayas, Pakistan. Remote Sens. 14 (3), 554 (2022).

Liu, J. et al. Developing a hybrid algorithm based on an equilibrium optimizer and an improved backpropagation neural network for fault warning. Processes 11 (6), 1813 (2023).

Narkhede, M. V., Bartakke, P. P. & Sutaone, M. S. A review on weight initialization strategies for neural networks. Artif. Intell. Rev. 55 (1), 291–322 (2022).

Narengbam, L. & Dey, S. Harris hawk optimization trained artificial neural network for anomaly based intrusion detection system. Concurr. Comput. Pract. Exp. 35 (23), e7771 (2023).

Özden, A. & İşeri, İ. COOT optimization algorithm on training artificial neural networks. Knowl. Inf. Syst. 65 (8), 3353–3383 (2023).

Kumar, G., Singh, U. P. & Jain, S. Swarm intelligence based hybrid neural network approach for stock price forecasting. Comput. Econ. 60 (3), 991–1039 (2022).

Zhao, Y. Application of BP neural network algorithm in visualization system of sports training management. Soft Comput. 27 (10), 6845–6854 (2023).

Nketiah, E. A. et al. Recurrent neural network modeling of multivariate time series and its application in temperature forecasting. PLoS ONE 18 (5), e0285713 (2023).

Kumar, T. A. et al. A novel CNN gap layer for growth prediction of palm tree plantlings. PLoS ONE 18 (8), e0289963 (2023).

Liu, J. et al. Research on reservoir porosity prediction method based on bidirectional longshort-term memory neural network. Prog. Geophys. 37 (5), 1993–2000 (2022).

Download references

Author information

Authors and affiliations.

Institute of Policy Studies, Lingnan University, Tuen Mun, 999077, Hong Kong

Xuying Dong & Wanlin Qiu

You can also search for this author in PubMed   Google Scholar

Contributions

Dong Xuying and Qiu Wanlin studied the specific situation of BP neural network, and combined with the experience of scientific research project management, Dong Xuying designed a scientific research project management evaluation and risk prediction method based on BP neural network. At the same time, Qiu Wanlin collected and analyzed the experimental data in this paper according to the actual situation. Dong Xuying and Qiu Wanlin wrote the first draft together.

Corresponding author

Correspondence to Wanlin Qiu .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Supplementary information., rights and permissions.

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Dong, X., Qiu, W. A method for managing scientific research project resource conflicts and predicting risks using BP neural networks. Sci Rep 14 , 9238 (2024). https://doi.org/10.1038/s41598-024-59911-w

Download citation

Received : 26 September 2023

Accepted : 16 April 2024

Published : 22 April 2024

DOI : https://doi.org/10.1038/s41598-024-59911-w

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• BP neural network
• Scientific research project management
• Regularization
• Resource conflict risk
• Deep learning

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing: AI and Robotics newsletter — what matters in AI and robotics research, free to your inbox weekly.

Risk-Based Monitoring in Clinical Trials: Past, Present, and Future

• Analytical Report
• Open access
• Published: 29 April 2021
• Volume 55 , pages 899–906, ( 2021 )

Cite this article

You have full access to this open access article

• Brian Barnes 1 ,
• Nicole Stansbury 1 ,
• Debby Brown 1 ,
• Lauren Garson 1 ,
• Geoff Gerard 1 ,
• Nickolas Piccoli 1 ,
• Debra Jendrasek 1 ,
• Nick May 1 ,
• Vanesa Castillo 1 ,
• Anina Adelfio   ORCID: orcid.org/0000-0003-0618-1028 1 ,
• Nycole Ramirez 1 ,
• Andrea McSweeney 1 ,
• Ruth Berlien 1 &
• Paula Jo Butler 1  

8251 Accesses

12 Citations

12 Altmetric

Explore all metrics

Risk-based monitoring (RBM) is a powerful tool for efficiently ensuring patient safety and data integrity in a clinical trial, enhancing overall trial quality. To better understand the state of RBM implementation across the clinical trial industry, the Association of Clinical Research Organizations (ACRO) conducted a landscape survey among its member companies across 6,513 clinical trials ongoing at the end of 2019. Of these trials, 22% included at least 1 of the 5 RBM components: key risk indicators (KRIs), centralized monitoring, off-site/remote-site monitoring, reduced source data verification (SDV), and reduced source document review (SDR). The implementation rates for the individual RBM components ranged 8%–19%, with the most frequently implemented component being centralized monitoring and the least frequently implemented being reduced SDR. When the COVID-19 pandemic emerged in early 2020, additional data were collected to assess its impact on trial monitoring, focusing specifically on trials switching from on-site monitoring to off-site/remote-site monitoring. These mid-pandemic data show that the vast majority of monitoring visits were on-site in February 2020, but an even higher percentage were off-site in April, corresponding with the first peak of the pandemic. Despite this shift, similar numbers of non-COVID-related protocol deviations were detected from February through June, suggesting little or no reduction in monitoring effectiveness. The pre- and mid-pandemic data provide two very different snapshots of RBM implementation, but both support the need to promote adoption of this approach while also highlighting an opportunity to capitalize on the recent shift toward greater RBM uptake in a post-pandemic environment.

Avoid common mistakes on your manuscript.

Introduction

Clinical trial management is a complex endeavor requiring careful planning, compliance with regulations, and coordination between multiple stakeholders such as sponsors, investigators, and contract research organizations (CROs). Risk-based monitoring (RBM) of clinical trials has emerged as a more targeted, strategic approach that takes advantage of increased connectivity and advances in data analytics. RBM streamlines and optimizes error detection, which may facilitate replacement of some or all on-site monitoring visits. The aim of RBM is to focus monitoring on those trial processes most likely to affect patient safety and data quality, often using real-time analytics, so that investigators can more quickly and effectively mitigate risks or address errors before they compromise trial quality.

RBM is an important component of a larger framework known as risk-based quality management (RBQM), defined in a 2013 European Medicines Agency (EMA) reflection paper as “a systematic process put in place to identify, assess, control, communicate and review the risks associated with the clinical trial during its lifecycle” [ 1 , 2 , 3 ]. Compared with source data verification (SDV), source document review (SDR), and other forms of monitoring focused on past events, RBM has a stronger focus on the present and future, particularly when it includes real-time monitoring and predictive modeling [ 4 ]. These forward-looking activities impact not only monitoring functions but also overall trial management. In other words, RBQM is a holistic, quality management, systems-based approach to trial implementation, and RBM as a monitoring strategy is an integral part of that approach [ 2 ]. Importantly, RBQM also directly addresses the directives on RBM contained in the ICH E6(R2) guidance [ 5 ].

The key components of RBQM include the following:

Initial Cross-functional Risk Assessment—Involves multiple stakeholders and identifies critical-to-quality (critical data and critical process) risks across the entire trial lifecycle as well as mitigation strategies, which will inform project plans.

Ongoing Cross-functional Risk Assessment—A continuous process of revisiting and adjusting the initial risk assessment and planned mitigations as the trial proceeds based on incoming data and any new developments within or outside of the trial that could affect quality.

Quality Tolerance Limits (QTLs)—Pre-determined limits for specific trial parameters that, when reached, signal that further evaluation is needed to determine if action is warranted.

Key Risk Indicators (KRIs)—Metrics used to assess site performance, either compared to other sites or to established values.

Centralized Monitoring—The remote review of aggregated electronic data, including data analysis.

Off-Site/Remote-site Monitoring—Replacement of some or all on-site monitoring visits with remote-site monitoring visits, where and when allowed by regulatory authorities. When monitoring remotely, a targeted and/or triggered review of documents and data is used.

Reduced SDV—Shift from 100% SDV to more targeted monitoring.

Reduced SDR—Shift from 100% SDR to more targeted monitoring.

The definition of each component was agreed upon by the authors; however, different terminology may be used across the industry. Components 1–3 affect multiple trial activities beyond monitoring and thus help form the “backbone” of the holistic RBQM framework, while Components 4–8 comprise the monitoring activities and tools specific to RBM. Although the distinction between trial-level activities and monitoring is important, the critical risk assessment and QTL-setting functions of RBQM must also be implemented for RBM to be fully successful.

As a relatively mature concept, RBM offers established benefits to trial execution, including enhanced effectiveness of monitoring, increased overall trial quality, greater efficiency, improved patient safety, and better overall value [ 3 , 6 , 7 ]. One major advantage of RBM is its universal application to any phase trial and essentially any type of clinical study.

There are, however, barriers to RBM adoption, including challenges in executing RBM within a complex trial workflow (especially when using new technologies or coordinating with multiple stakeholders), concern regarding regulator acceptance of data, a number of country-specific regulatory limitations, sponsor reluctance on certain types of trials, and sponsor sensitivity to inspector findings at the site level. Despite these challenges, RBM is supported and encouraged by multiple regulatory agencies [ 5 , 7 , 8 ]. In fact, these authorities encouraged increased use of RBM as the COVID-19 pandemic unfolded, with travel restrictions, risk of infection for vulnerable patients, and site closures disrupting all aspects of clinical trials, including regular on-site monitoring activities [ 8 , 9 ].

To shed light on the state of RBM adoption and implementation, the Association of Clinical Research Organizations (ACRO)—a trade association of CROs and technology companies—conducted a landscape survey of RBM use in clinical trials ongoing at the end of 2019 that were managed by several of its member companies. After COVID-19 emerged as a worldwide threat in early 2020, ACRO then gathered additional data from January–June 2020 to determine the impact of the pandemic on trial management, with a specific focus on monitoring. Here, we present both datasets, discussing the insights gained from them into the past and present use of RBM and how changes in trial practices during the pandemic could help shape the future of clinical trial monitoring.

RBM Landscape Survey

Seven ACRO member companies responded to a survey of RBM practices in clinical trials where project management and/or clinical monitoring were within scope of the companies’ services​. A neutral outside vendor collected, blinded, aggregated, and analyzed the data. The dataset included trials that were ongoing as of December 31, 2019, including studies initiated in 2019 and multi-year studies from years prior.

To better understand the RBM landscape, companies participating in the survey were asked to provide data showing how many of their trials implemented the eight RBM/RBQM components: initial cross-functional risk assessment, ongoing cross-functional risk assessment, QTLs, KRIs, Centralized monitoring, off-site/remote-site monitoring, reduced SDV, and reduced SDR. The component definitions presented above were formulated by the authors to provide a good benchmark to support data collection, ensuring that data submissions were consistent in the survey.

Assessment of Trial Disruptions During the COVID-19 Pandemic

Data on RBM detection of on-site/remote visits and protocol deviations from January–June 2019 were provided by three member companies. Further data on trial disruptions during the same period were gathered from additional member companies, as noted.

RBM Landscape Data

The landscape survey of RBM implementation during 2019 included 6,513 clinical trials managed by 7 of ACRO’s CRO member companies. Of the included trials, 47% had at least 1 of the 8 RBQM components (listed above), while 53% had more traditional trial management.

Implementation rates for the 5 RBM components (KRIs, centralized monitoring, off-site/remote-site monitoring, reduced SDR, reduced SDV) ranged from 8 to 19% of trials and were markedly lower than the implementation rates for initial or ongoing risk assessments (33% for both), which are specific to RBQM but critical to the execution of RBM (Fig.  1 ).

2019 Landscape of RBM/RBQM Components in Clinical Trials. Data represent the percentage of all 6,513 trials included in the survey, not just the subset of studies that have at least one RBM component. *The KRI percentage does not include KRIs related to operations or performance.

Looking at the percentage of trials with specific combinations of RBM/RBQM components (Combinations A–H in Fig.  2 ), it is clear most trials employed neither a “holistic” RBQM approach (defined as having seven or eight RBM/RBQM components) nor a full-RBM approach (defined as having all five RBM components). Taking as a point of reference trials having both initial and ongoing risk assessments (Combination B), when centralized monitoring is added (Combination C) there is a 6-percentage-point drop in the percentage of total trials and a 9-point drop when off-site/remote-site monitoring is added (Combination D). Adding reduced SDR (Combination E) results in a smaller 1-point drop in the percentage of total trials.

2019 Implementation of RBM/RBQM Components in Combination. Graph shows only some of the common combinations of components and not all combinations reported in the dataset.

Taken together, the RBM landscape data show that industry adoption is less widespread than expected and implementation is rather piecemeal, with few studies incorporating all five RBM components. These findings also provide a benchmark to better assess future changes in RBM uptake, particularly in situations where trial protocols and regular monitoring practices have been disrupted.

Impact of the COVID-19 Pandemic on Clinical Trial Monitoring

On March 11, 2020, the World Health Organization (WHO) declared the COVID-19 outbreaks spreading across the globe to be a pandemic. This unprecedented worldwide disruption presented major challenges in clinical trial management by forcing companies to rely mainly on remote and centralized monitoring due to site closures and stay-at-home orders. At the same time, the pandemic created something of a “natural experiment,” allowing ACRO to collect early data on the impact of this shift in trial monitoring to complement the larger-scale RBM landscape dataset.

Data from 3 member companies covering trials from January–June 2020 showed remote-site monitoring increased and on-site monitoring decreased at the peak of the pandemic in April compared with the pre-pandemic baseline in February (representative data covering ~ 1,200 trials from 1 company shown in Fig.  3 ). These trends began to reverse themselves post peak, but the percentage of remote-site monitoring visits was still markedly higher in June compared to the baseline percentage.

Increased Remote Monitoring Visits and RBM in Response to the COVID-19 Pandemic. Graph shows data from 1 of 3 companies providing monitoring visit data (n =  ~ 1,200 trials), but trends were similar across all 3 companies.

Remote-site monitoring effectively captured protocol deviations as the pandemic evolved, even during the peak in April 2020 when there was little or no physical access to most trial sites (representative data covering ~ 1,200 trials from 1 company shown in Fig.  4 ). A corresponding peak in COVID-related protocol deviations was also seen that month, declining over time through June, but remaining above the pre-pandemic levels. Notably, the total non-COVID protocol deviations detected each month from March to May were similar to the February baseline, even as the percentage of remote-site monitoring visits increased from 18% in February to a high of 93% in April. This suggests that the rapid shift in monitoring methods allowed for sufficient oversight and monitoring continuity, lending confidence in data quality and patient safety.

Clinical Trial Protocol Deviations Detected by RBM During the COVID-19 Pandemic. Graph shows data from 1 of 3 companies providing protocol deviation data (n =  ~ 1,200 trials), but trends were similar across all 3 companies.

Trial Disruptions During the COVID-19 Pandemic

Additional data shed more light on the scale of the trial disruptions as the pandemic approached its first peak, complementing the monitoring data. In less than a month (March 14–April 6, 2020), 1 company reported that the percentage of institutions where patient or site monitoring visits for the company’s trials were disrupted jumped from 10 to 49%. A second company reported that 33% of planned trial visits were disrupted in March, and by the end of March, approximately 70% of sites were inaccessible. New subject enrollment in trials managed by a third company was reduced by 65% in March 2020 compared with March 2019.

Though less comprehensive than the pre-COVID RBM implementation data, the mid-pandemic trial monitoring and disruptions data help illustrate the mitigating effect of one RBM component on pervasive and potentially crippling disruptions to clinical trial management.

The RBM Landscape

The RBM landscape data—generated from a survey planned before the COVID-19 pandemic covering more than 6,000 clinical trials—provide a pre-pandemic baseline for RBM adoption and implementation. The data collected for this analysis are representative of studies where CROs have contracted services. We acknowledge these data may not be reflective of the entire clinical trial development landscape, as CROs may only perform certain activities outsourced from trial sponsors and not others; however, it is our opinion that this dataset still provides valuable visibility into clinical trial implementation of RBM. Overall, execution of RBM is rather piecemeal, with the individual RBM components being used in 8%–19% of trials and very few trials executing a full-RBM approach. This inconsistent implementation is also seen for the RBQM components critical to the success of RBM, with initial and ongoing risk assessments each implemented in less than half of trials and not always together in the same trial. Not surprisingly, given the poor uptake of RBM, use of holistic RBQM is quite rare, meaning that the full potential of RBM to enhance trial quality—by more efficiently detecting errors compromising patient safety and data validity so that their impact can be mitigated—is not yet being realized.

The key takeaway from the landscape data is that industry adoption of RBM is less extensive than expected, likely because companies are reluctant to fully commit to changing their existing practices and protocols. For example, centralized monitoring, the most frequently used RBM component, was implemented in less than 20% of the trials in our dataset; however, off-site/remote-site monitoring was used in only 10% of the trials, suggesting greater acceptance of remote data evaluation than replacement of on-site visits with remote visits.

One reason often cited for the incomplete adoption and partial implementation of RBM is a hesitance on the part of trial sponsors and CROs to reduce the amount of SDR/SDV in favor of a more targeted approach. For example, site inspections or audits that find discrepancies not critical-to-quality may cause study personnel to rely more on SDV/SDR, even though patient safety and data integrity have not been compromised. In our experience, most sponsors who agree to reduced SDV also accept reduced SDR, and our landscape data are generally consistent with this assessment. There is, however, resistance to reducing SDR (i.e., if you do not look at 100% of the source data, how do you ensure that you do not miss any adverse events?), as shown by the lower implementation rate for reduced SDR (8%) compared with reduced SDV (15%). Despite this, we believe that a risk-based approach to SDR/SDV best serves the interest of sites, patients, and the whole of the clinical research community.

Other explanations for slow RBM adoption are lack of familiarity with different RBM practices, misconceptions that it might not fit into all studies, the complexity of implementing these practices, logistical barriers, the need for new and unfamiliar technology, and an incorrect assumption that RBM methodology data are less likely to satisfy regulators. Many of these challenges can be addressed by educating study sponsors and personnel on RBM implementation and the regulatory landscape, and also managing expectations regarding what efficient monitoring that meets regulatory guidance looks like. At the same time, stronger, more specific regulator guidance and alignment within regulatory agencies is needed, particularly when executing RBM as part of a more holistic end-to-end RBQM framework.

Real-World RBM Implementation

As disruptive as the COVID-19 pandemic has been, it also created a natural experiment by motivating many companies to transition to remote-site monitoring over the same time period to avoid trial interruptions. The real-world utility of a single RBM component—remote-site monitoring—is thus shown in the mid-pandemic data, which offer an early snapshot of adaptations in trial practices, showing that COVID-related protocol deviations were successfully differentiated from non-COVID-related deviations and that little change in the monthly totals of non-COVID deviations was seen even in the absence of physical access to sites. Early data suggest that the effectiveness of remote-site monitoring mid-pandemic was similar to that of on-site monitoring pre-COVID; however, without established RBM principals and recent technological advancements, there likely would have been a larger disruption in monitoring that would have impeded the continuation of many trials.

Based on the present data and a wealth of experience in RBM, as experts in the field, we recommend the following considerations for changes in monitoring methodology due to trial disruptions:

Centralized monitoring and risk-based approaches help provide confidence in the quality of data monitored during external disruptions.

While alternative ways of accessing source documents may be warranted for certain purposes, such as safety oversight or critical endpoint collection in pivotal trials, achieving 100% SDR/SDV for interim analysis or database lock is not an exigent circumstance that warrants unplanned-for remote access to source documents.

Regulatory consistency and direction are needed for remote-site monitoring and resumption of monitoring activities as restrictions are relaxed, with a focus on the benefits of monitoring critical data only.

On April 16, 2020 the FDA clarified its thinking in regard to remote access, emphasizing that sponsors should carefully evaluate technologies and take a risk-based approach to the unplanned-for collection of source data from sites and document changes to the trial monitoring plan.

While we appreciate FDA mentioning risk-based approaches, we think there is potential for stakeholders to interpret the FDA’s flexibility on remote SDV practices as implicit endorsement of 100% SDV now and going forward.

The RBM landscape survey showed that RBM adoption before the COVID-19 pandemic was not as widespread as expected, despite the proven benefits and clear potential of this approach. In addition, few trials implement more than a few of the eight RBM/RBQM components, meaning the full potential of RBM as a vital part of a broader trial management framework is far from being realized. What is clear from the rapid shift from on-site to remote-site monitoring for most clinical trials during the pandemic is that transitioning to an RBM approach without diminishing monitoring effectiveness is possible, even in difficult circumstances.

The current findings and a wealth of practical experience support the uptake of RBM and, potentially, a shift to RBQM. We believe the industry will continue to lean into greater adoption of off-site/remote-site monitoring and other RBM practices in a post-pandemic environment. To facilitate this, companies involved in clinical trial research are encouraged to share their real-world experiences with RBM implementation pre- and mid-pandemic, both the successes and the lessons learned. ACRO will continue gathering data on trial monitoring practices through the pandemic and after it has ended, with the aim of sharing our findings with the larger clinical research community. We further encourage the industry at large to continue to advance best practices and promote adoption of RBM.

ACRO. Establishing Risk-Based Monitoring within a Quality-Based System as “Best Practice” for Clinical Studies – ACRO [Internet]. 2019. https://www.acrohealth.org/rbqm-report/ . Accessed 8 October 2020

ACRO. Risk Based Quality Management (RBQM) – A Collaborative Approach to Holistic Clinical Trial Oversight [Internet]. 2019. https://www.acrohealth.org/oversight-rbqm-paper/ . Accessed 8 October 2020

EMA. Reflection paper on risk based quality management in clinical trials [Internet]. 2013. https://www.ema.europa.eu/en/documents/scientific-guideline/reflection-paper-risk-based-quality-management-clinical-trials_en.pdf

Rosenberg MJ. Key considerations in the transition to risk-based monitoring. Ther Innov Regul Sci. 2014;48(4):428–35.

Article   Google Scholar  

ICH. Guideline for Good Clinical Practice E6(R2) [Internet]. 2016. https://database.ich.org/sites/default/files/E6_R2_Addendum.pdf . Accessed 7 August 2020

Brosteanu O, Houben P, Ihrig K, Ohmann C, Paulus U, Pfistner B, et al. Risk analysis and risk adapted on-site monitoring in noncommercial clinical trials. Clin Trials Lond Engl. 2009;6(6):585–96.

FDA. Oversight of Clinical Investigations — A Risk-Based Approach to Monitoring [Internet]. FDA; 2013 [cited 2020 Oct 8]. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/oversight-clinical-investigations-risk-based-approach-monitoring . Accessed 8 October 2020

EMA. Guidance on the Management of Clinical Trials During the COVID-19 (Coronavirus) Pandemic [Internet]. 2020. https://ec.europa.eu/health/sites/health/files/files/eudralex/vol-10/guidanceclinicaltrials_covid19_en.pdf

FDA. FDA Guidance on Conduct of Clinical Trials of Medical Products during COVID-19 Public Health Emergency [Internet]. FDA; 2020. https://www.fda.gov/regulatory-information/search-fda-guidance-documents/fda-guidance-conduct-clinical-trials-medical-products-during-covid-19-public-health-emergency . Accessed 7 August 2020

Download references

Special thanks to Amy Volpert of BioScience Communications, New York, NY who provided medical writing support for this manuscript. The landscape survey data were collected and analyzed by Kevin Olson of Industry Standard Research, Raleigh, NC. The Association of Clinical Research Organizations (ACRO), a trade association for clinical research and technology companies, based in Washington, DC provided funding and support for the landscape survey project and development of this manuscript.

Author information

Authors and affiliations.

Association of Clinical Research Organizations (ACRO), Washington, DC, USA

Brian Barnes, Nicole Stansbury, Debby Brown, Lauren Garson, Geoff Gerard, Nickolas Piccoli, Debra Jendrasek, Nick May, Vanesa Castillo, Anina Adelfio, Nycole Ramirez, Andrea McSweeney, Ruth Berlien & Paula Jo Butler

You can also search for this author in PubMed   Google Scholar

Contributions

The Association of Clinical Research Organizations (ACRO) hosts a committee that contributed to the development of this paper. The authors listed below, are submitting this manuscript on behalf of the Association of Clinical Research Organizations (ACRO). Brian Barnes 1 , Nicole Stansbury 1 , Debby Brown 1 , Lauren Garson, Geoff Gerard, Nickolas Piccoli, Debra Jendrasek, Nick May, Vanesa Castillo, Anina Adelfio, Nycole Ramirez, Andrea McSweeney, Paula Jo Butler, and Ruth Berlien.

Corresponding author

Correspondence to Anina Adelfio .

Ethics declarations

Conflict of interest.

The authors declare no conflicts of interest.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Barnes, B., Stansbury, N., Brown, D. et al. Risk-Based Monitoring in Clinical Trials: Past, Present, and Future. Ther Innov Regul Sci 55 , 899–906 (2021). https://doi.org/10.1007/s43441-021-00295-8

Download citation

Received : 13 January 2021

Accepted : 09 April 2021

Published : 29 April 2021

Issue Date : July 2021

DOI : https://doi.org/10.1007/s43441-021-00295-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Find a journal
• Publish with us
• Track your research

Publication types

• Case Reports

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List

Prevalence and risk factors of post-COVID-19 condition in adults and children at 6 and 12 months after hospital discharge: a prospective, cohort study in Moscow (StopCOVID)

Ekaterina pazukhina.

1 Laboratory of Health Economics, Institute of Applied Economic Studies, The Russian Presidential Academy of National Economy and Public Administration, Moscow, Russia

2 Center for Advanced Financial Planning, Macroeconomic Analysis and Financial Statistics, Financial Research Institute of the Ministry of Finance of the Russian Federation, Moscow, Russia

Margarita Andreeva

3 Department of Paediatrics and Paediatric Infectious Diseases, Institute of Child’s Health, Sechenov First Moscow State Medical University (Sechenov University), Moscow, Russia

Ekaterina Spiridonova

Polina bobkova, anastasia shikhaleva, yasmin el-taravi, mikhail rumyantsev, aysylu gamirova, anastasiia bairashevskaia, polina petrova, dina baimukhambetova, maria pikuza, elina abdeeva, yulia filippova, salima deunezhewa, nikita nekliudov, polina bugaeva, nikolay bulanov.

4 Tareev Clinic of Internal Diseases, Sechenov First Moscow State Medical University (Sechenov University), Moscow, Russia

Sergey Avdeev

5 Clinic of Pulmonology, Sechenov First Moscow State Medical University (Sechenov University), Moscow, Russia

Valentina Kapustina

6 Department of Internal Medicine №1, Institute of Clinical Medicine, Sechenov First Moscow State Medical University (Sechenov University), Moscow, Russia

Alla Guekht

7 Research and Clinical Center for Neuropsychiatry, Moscow, Russia

8 Pirogov Russian National Research Medical University, Moscow, Russia

Audrey DunnGalvin

9 School of Applied Psychology, University College Cork, Cork City, Ireland

Pasquale Comberiati

10 Department of Clinical and Experimental Medicine, Section of Pediatrics, University of Pisa, Pisa, Italy

Diego G. Peroni

Christian apfelbacher.

11 Institute of Social Medicine and Health Systems Research, Faculty of Medicine, Otto von Guericke University Magdeburg, Magdeburg, Germany

Jon Genuneit

12 Pediatric Epidemiology, Department of Pediatrics, Medical Faculty, Leipzig University, Leipzig, Germany

Luis Felipe Reyes

13 Universidad de La Sabana, Chía, Colombia

14 Clínica Universidad de La Sabana, Chía, Colombia

Caroline L. H. Brackel

15 Department of Pediatric Pulmonology, Emma Children’s Hospital, Amsterdam University Medical Centers, Amsterdam, the Netherlands

16 Department of Pediatrics, Tergooi MC, Hilversum, the Netherlands

Victor Fomin

17 Sechenov First Moscow State Medical University (Sechenov University), Moscow, Russia

Andrey A. Svistunov

Peter timashev.

18 Institute for Regenerative Medicine, Sechenov First Moscow State Medical University (Sechenov University), Moscow, Russia

Lyudmila Mazankova

19 Russian Medical Academy of Continuous Professional Education of the Ministry of Healthcare of the Russian Federation, Moscow, Russia

Alexandra Miroshina

20 ZA Bashlyaeva Children’s Municipal Clinical Hospital, Moscow, Russia

Elmira Samitova

Svetlana borzakova.

21 Research Institute for Healthcare Organization and Medical Management of Moscow Healthcare Department, Moscow, Russia

Elena Bondarenko

Anatoliy a. korsunskiy, gail carson.

22 Nuffield Department of Medicine, ISARIC Global Support Centre, University of Oxford, Oxford, UK

Louise Sigfrid

Janet t. scott.

23 MRC-University of Glasgow Centre for Virus Research, Glasgow, UK

Matthew Greenhawt

24 Department of Pediatrics, Section of Allergy/Immunology, Children’s Hospital Colorado, University of Colorado School of Medicine, Aurora, USA

Danilo Buonsenso

25 Department of Woman and Child Health and Public Health, Fondazione Policlinico Universitario A. Gemelli IRCCS, Rome, Italy

26 Dipartimento di Scienze Biotecnologiche di Base, Cliniche Intensivologiche e Perioperatorie, Università Cattolica del Sacro Cuore, Rome, Italy

27 Center for Global Health Research and Studies, Università Cattolica del Sacro Cuore, Roma, Italia

Malcolm G. Semple

28 Health Protection Research Unit in Emerging and Zoonotic Infections, Institute of Infection, Veterinary and Ecological Sciences, Faculty of Health and Life Sciences, University of Liverpool, Liverpool, UK

29 Department of Respiratory Medicine, Alder Hey Children’s Hospital, Liverpool, UK

John O. Warner

30 Inflammation, Repair and Development Section, National Heart and Lung Institute, Faculty of Medicine, Imperial College London, London, UK

Piero Olliaro

Dale m. needham.

31 Outcomes After Critical Illness and Surgery (OACIS) Research Group, Johns Hopkins University, Baltimore, MD USA

32 Pulmonary and Critical Care Medicine, Department of Medicine, Johns Hopkins University School of Medicine, Baltimore, MD USA

33 Physical Medicine and Rehabilitation, Johns Hopkins University School of Medicine, Baltimore, MD USA

Petr Glybochko

Denis butnaru, ismail m. osmanov, daniel munblit, associated data.

The data that support the findings of this study are available from the corresponding author, DM, upon reasonable request.

Previous studies assessing the prevalence of COVID-19 sequelae in adults and children were performed in the absence of an agreed definition. We investigated prevalence of post-COVID-19 condition (PCC) (WHO definition), at 6- and 12-months follow-up, amongst previously hospitalised adults and children and assessed risk factors.

Prospective cohort study of children and adults with confirmed COVID-19 in Moscow, hospitalised between April and August, 2020. Two follow-up telephone interviews, using the International Severe Acute Respiratory and Emerging Infection Consortium survey, were performed at 6 and 12 months after discharge.

One thousand thirteen of 2509 (40%) of adults and 360 of 849 (42%) of children discharged participated in both the 6- and 12-month follow-ups. PCC prevalence was 50% (95% CI 47–53) in adults and 20% (95% CI 16–24) in children at 6 months, with decline to 34% (95% CI 31–37) and 11% (95% CI 8–14), respectively, at 12 months. In adults, female sex was associated with PCC at 6- and 12-month follow-up (OR 2.04, 95% CI 1.57 to 2.65) and (OR 2.04, 1.54 to 2.69), respectively. Pre-existing hypertension (OR 1.42, 1.04 to 1.94) was associated with post-COVID-19 condition at 12 months. In children, neurological comorbidities were associated with PCC both at 6 months (OR 4.38, 1.36 to 15.67) and 12 months (OR 8.96, 2.55 to 34.82) while allergic respiratory diseases were associated at 12 months (OR 2.66, 1.04 to 6.47).

Conclusions

Although prevalence of PCC declined one year after discharge, one in three adults and one in ten children experienced ongoing sequelae. In adults, females and persons with pre-existing hypertension, and in children, persons with neurological comorbidities or allergic respiratory diseases are at higher risk of PCC.

Supplementary Information

The online version contains supplementary material available at 10.1186/s12916-022-02448-4.

Although most people fully recover from acute infection with severe acute respiratory syndrome coronavirus 2 (SARS CoV-2) and coronavirus disease 2019 (COVID-19) disease, some experience ongoing sequelae [ 1 ]. This wide range of symptoms occurring in the weeks to months after SARS-CoV-2 infection has been referred to as either long COVID, post-COVID-19 condition, or post-acute sequelae of SARS-CoV-2 infection (PASC), amongst other names [ 2 ]. High profile editorials [ 3 , 4 ] drew attention to an increasing number of people experiencing these ongoing sequelae and called for comprehensive research, including risk factors and clinical features.

Most post-COVID research has focused on adults [ 5 ], given the predominance of adult COVID-19 in the first pandemic waves, which appeared to spare children, somewhat. Therefore, there is a more limited number of paediatric studies [ 6 ], although the need for research on COVID-19 consequences in children and young people has been previously acknowledged and has grown in importance with emergence of variants that are affecting children [ 7 ]. Head-to-head comparison of COVID-19 sequelae in children and adults is still lacking.

Many studies have investigated the prevalence and risk factors of long COVID [ 5 ], but heterogeneity in patient assessment and definitions [ 8 ] and lack of data regarding symptom duration are challenges to meta-analyses. Notably, in September 2020, the World Health Organization (WHO) Classification and Terminologies unit created International Classification of Diseases 10 (ICD-10) and ICD-11 codes for post-COVID-19 condition, and in October 2021, a clinical case definition of post-COVID-19 condition was announced, following a Delphi consensus process [ 9 ]. It was defined as a condition occurring “in individuals with a history of probable or confirmed SARS-CoV-2 infection, usually 3 months from the onset of COVID-19 with symptoms that last for at least 2 months and cannot be explained by an alternative diagnosis”. However, this definition was intended for adults, and WHO suggests that a separate definition might be applicable for children.

This prospective study aimed to investigate the prevalence and characteristics of post-COVID-19 condition in previously hospitalised children and adults using standardised follow-up data collection protocols developed by the International Severe Acute Respiratory and Emerging Infection Consortium (ISARIC) Global Adult and Paediatric COVID-19 follow-up working groups.

The study is reported based on the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) checklist for cohort studies ( https://www.strobe-statement.org/ ), which can be found in the supplementary material.

Study design, setting, and participants

This study combines data from two longitudinal prospective cohorts of patients with COVID-19: (a) adults admitted to Sechenov University Hospital Network (four large tertiary adult hospitals) in Moscow, Russia, and (b) children admitted to Z.A. Bashlyaeva Children’s Municipal Clinical Hospital in Moscow, Russia (the primary paediatric COVID-19 hospital in Moscow throughout the time of pandemic). Only patients with positive polymerase chain reaction (PCR) confirmed SARS-CoV-2 infection were included in this study. Details regarding the demographic profile, hospitalisation requirements and origination of these cohorts are comprehensively described elsewhere [ 10 , 11 ]. In brief, to form and define the cohorts, the acute phase data of adult and paediatric patients were extracted from electronic medical records (EMR) and the Local Health Information System (HIS) at the host institutions using ISARIC Core case report form (CRF) for acute phase data collection. The acute-phase datasets included demographics, comorbidities, symptoms on admission, computed tomography results, and disease severity, including use of supportive therapies.

Data collection and entry were performed by a team of trained medical students and physician residents, with extensive relevant research experience, via telephone interviews and the Research Electronic Data Capture (REDCap) database [ 10 – 12 ], with supervision by senior academic researchers.

Given the well-recognised emergence of COVID-19 infection sequelae, this follow-up study was planned to track prevalence and risk factors for the development of such sequelae occurring after hospital discharge. Data were obtained at two follow-up points, at 6 (± 2) and 12 (± 2) months after hospital discharge. These follow-up assessments, collected via telephone interviews, used the Tier 1 ISARIC Long-term Follow-up Study CRF for adult patients and version 1 of the ISARIC COVID-19 Health and Wellbeing Follow Up Survey for Children for paediatric patients, both developed by the ISARIC Global COVID-19 follow-up working group and independently forward and backward translated into Russian. These follow-up assessments evaluated patients’ physical and mental health status and assessed for any newly developed symptoms between hospital discharge and the follow-up assessment, including symptom onset and duration as previously described [ 11 ]. Given the well-recognised emergence of post-COVID infection sequelae, this follow-up study was planned to track prevalence and risk factors for the development of such sequelae occurring after hospital discharge.

The acute-hospitalisation dataset included demographics, symptoms, comorbidities (at the time of hospital admission for COVID-19), chest computed tomography (CT) results, supportive care required, and clinical outcomes at the time of discharge.

Data management

REDCap electronic data capture tools (Vanderbilt University, Nashville, TN, USA) hosted at Sechenov University and Microsoft Excel (Microsoft Corp, Redmond, WA, USA) were used for data collection, storage, and management [ 13 , 14 ].

Definitions

Post-COVID-19 condition was defined as the presence of any symptom which started no later than three months after hospital discharge and lasted for at least 2 months as per the WHO case definition [ 9 ]. Symptom duration was calculated from the time of the hospital discharge in the absence of reliable objective medical record data regarding date of first symptoms appearance.

Patients requiring non-invasive ventilation, invasive ventilation, or intensive care unit (ICU) care during acute phase of COVID-19 were defined as severe.

Symptoms were categorised into nine manifestations: cardiovascular, dermatological, fatigue, gastrointestinal, musculoskeletal, neurocognitive, respiratory, sensory, and sleep (Table S 1 ). Symptom categorisation was based on previously published literature and ISARIC working groups’ discussion [ 10 , 11 ].

Statistical analysis

Descriptive statistics were calculated for baseline characteristics. Continuous variables were summarised as median (interquartile range, IQR) and categorical variables as frequency (percentage). 95% confidence intervals (CIs) were obtained for the estimates of post-COVID-19 condition prevalence using bootstrap methodology (10,000 iterations).

Forest plots were used to present the prevalence of post-COVID-19 condition and different manifestations. Circular dendrograms were used to illustrate coexistence of post-COVID-19 condition manifestations. The phenotypes of post-COVID-19 condition were presented using radial plots. The cut-off for defining a phenotype presentation was set at 2% of respondents reporting multiple manifestations. For children, due to a low number of respondents reporting multiple manifestations, all individuals were presented on the plots.

We included all participants with post-COVID-19 condition in the final analysis, without missing data imputation. In order to control for recall bias at 12-month follow-up, we considered post-COVID-19 condition manifestations only amongst those manifestations which were reported at 6-months and satisfied the WHO definition of post-COVID-19 condition. Only patients completing both 6-month and 12-month follow-up were included in this study analysis similarly to previously published large cohort studies [ 15 ].

Multivariable logistic regression analysis was performed separately for adults and children to investigate associations of demographic characteristics and comorbidities at hospital admission with COVID-19 (limited to those variables reported in > 3% of study participants) and severity of COVID-19 with post-COVID-19 condition prevalence at the time of the follow-up interviews. Selection of the variables was the following: “COVID-19 severity” variable as exposure, “post-COVID-19 condition” as an outcome, comorbidities as covariates, gender, and age as effect modifiers. Twelve variables in adults and seven in children were tested as potential risk factors based on data availability and previous research [ 10 , 11 , 15 – 18 ]. We included all participants for whom the variables of interest were available in the final analysis, without imputing missing data. Odds ratios were calculated together with 95% CIs.

Two-sided p -values were reported for all statistical tests, a p -value below 0.05 was considered to be statistically significant. Statistical analysis was performed in R version 4.0.2 using libraries dplyr, foreign, forestplot, ograph, and ggraph [ 19 ].

Out of 2509 eligible adults and 849 eligible children with laboratory confirmed COVID-19 discharged between April and August 2020, 1994 (79%) and 832 (98%) had contact information, and of these, 1013 (40% of discharged, 51% of those with contact information) adults and 360 children (42% of discharged, 43% of those with contact information) participated in both follow-up interviews and were included in the final analysis (Fig.  1 ). The 6-month follow-up interviews were conducted between November 2020 and March 2021 and the 12-month follow-up between April 2021 and August 2021.

Flow diagram of patients admitted with PCR-confirmed COVID-19 to Sechenov University Hospital Network (adults) and Z.A. Bashlyaeva Children’s Municipal Clinical Hospital (children)

Table  1 shows the demographic and clinical characteristics of the study participants. For adults the median time, after hospital discharge, to the 6- and 12-month assessments was 215 days (IQR 196–235) and 383 days (IQR 376–390). The median age of adult patients was 56.8 years (IQR 47.0–65.8), and 49% (500/1013) were male. The most common pre-existing comorbidity in adults at admission was hypertension (45%, 458/1013), followed by chronic cardiac disease and excessive weight and obesity (20% each, 198/1013) and type II diabetes (15%, 148/1013). Three percent of patients (27/1013) required non-invasive ventilation, invasive ventilation, or ICU care during hospitalisation.

Demographic characteristics of adults admitted to the Sechenov University Hospital Network and children admitted to the Z.A. Bashlyaeva Children’s Municipal Clinical Hospital. Data are n (%) or median (IQR) excluding missing values. ICU, intensive care unit

For children, the median time to the 6-month follow-up was 255 days (IQR 223–270) and to the 12-month follow-up 367 days (IQR 351–379). The median paediatric patient age was 9.5 years (IQR 2.4–14.8), and 48% (174/360) were male. Three percent of children (12/360) required non-invasive ventilation, invasive ventilation, or treatment in the ICU during hospitalisation. The most common comorbidities in children were allergic rhinitis (7%, 26/360) and intestinal problems (7%, 25/360).

Figure  2 shows the temporal trend in post-COVID-19 condition manifestations prevalence. Prevalence was significantly higher in adults compared with children at both 6-month and 12-month follow-up ( p  < 0·001): relative risk of any manifestation 2.51 (2.02 to 3.11) at 6 months and 3.07 (2.26 to 4.16) at 12 months. The difference in prevalence of each specific manifestation between adults and children is shown in tables S 2 and S 3 . The proportion of individuals with at least one post-COVID-19 condition manifestation decreased from 50% (95% CI 47–53) at 6 months to 34% (95% CI 31–37) at 12 months in adults and from 20% (95% CI 16–24) to 11% (95% CI 8–14), respectively, in children. A decline in prevalence was observed across all manifestations.

Forest plots demonstrating the prevalence of post-COVID-19 condition manifestations in adults and children 6 and 12 months after hospital discharge. Sixth-month prevalence is coloured in red, and 12-month prevalence is coloured in blue. Estimates of the prevalence 95% confidence intervals were calculated using the bootstrapping method

In adults, the most common post-COVID-19 condition features at 6-month follow-up included fatigue 25% (95% CI 22–28), respiratory 22% (95%CI 20–25), neuro-cognitive 19% (95% CI 17–21), and dermatological 13% (95% CI 11–15) manifestations. At 12 months after the hospital discharge, the prevalence decreased to 12% (95% CI 10–14), 10% (95% CI 8–11), 9% (95% CI 7–11), and 4% (95% CI 3–5) respectively.

In children, the most common post-COVID-19 condition features at 6-month follow-up were fatigue 9% (95% CI 6–13), dermatological 5% (95% CI 3–7), neuro-cognitive 4% (95% CI 2–6), and sleep-related 4% (95% CI 2–6) manifestations. At the 12-month follow-up, these decreased to 4% (95% CI 2–6), 2% (95% CI 1–4), 2% (95% CI 1–3), and 1% (95% CI 0–1) respectively.

We investigated the phenotypes of post-COVID-19 condition in adults and children, defined as a report of two or more different manifestations at 6-month assessment (Figs.  3 and ​ and4). 4 ). Amongst adults, 28% (287/1013) reported at least two manifestations. We differentiated three prevalent phenotypes at 6 months, namely (a) fatigue/respiratory without neurological manifestations (10%, 29/287); (b) fatigue/respiratory with neurological manifestations (7%, 19/287); and (c) fatigue/neurological without respiratory manifestations (6%, 17/287), regardless of other manifestations reported. By 12 months, 41% (12/29) of people with fatigue/respiratory without neurological manifestations fully recovered, while only 21% (4/19) of fatigue/respiratory with neurological manifestations and 24% (4/17) of fatigue/neurological without respiratory manifestations were symptom-free (Additional file  1 : Figure S1).

Interrelations between the post-COVID-19 condition manifestations in adults and children 6 and 12 months since hospital discharge. Bubble diameter is proportional to the proportion of individuals with the symptom category reported. Line thickness is proportional to the number of individuals with the coexisting manifestations. Cardiovascular, CRD; dermatological, DRM; fatigue, FTG; gastrointestinal, GST; musculoskeletal, MSC; neurocognitive, NRL; respiratory, RSP; sensory, SNS; sleep, SLP

A Radial plots representing post-COVID-19 condition phenotypes in adults at 6 months after discharge and 12 months after discharge. Manifestations are shown for each patient; each segment represents a single patient. Thick black lines are used to distinct phenotypes. Cardiovascular, CRD; dermatological, DRM; fatigue, FTG; gastrointestinal, GST; musculoskeletal, MSC; neurocognitive, NRL; respiratory, RSP; sensory, SNS; sleep, SLP. B Radial plots representing post-COVID-19 condition phenotypes in children at 6 months after discharge and 12 months after discharge. Manifestations are shown for each patient; each segment represents a single patient. Thick black lines are used to distinct phenotypes. Cardiovascular, CRD; dermatological, DRM; fatigue, FTG; gastrointestinal, GST; musculoskeletal, MSC; neurocognitive, NRL; respiratory, RSP; sensory, SNS; sleep, SLP

In children, phenotypes were less feasible to assess due to a smaller case count of post-COVID-19 condition. Seven percent (25/360) of children had a combination of manifestations at 6-month follow-up. The only characteristic phenotype amongst individuals with coexisting manifestations was fatigue/neurological (24%, 6/25), with 50% (3/6) of these having fully resolved by 12 months.

Risk factors association with post-COVID-19 condition 6- and 12-months after hospital discharge were assessed in multivariable regression analysis separately for adults and children (Fig.  5 ).

A Multivariable logistic regression model demonstrating risk factors associated with post-COVID-19 condition in adults at 6-month follow-up. Odds ratios and 95% CIs are presented. B Multivariable logistic regression model demonstrating risk factors associated with post-COVID-19 condition in adults at 12-month follow-up. Odds ratios and 95% CIs are presented. C Multivariable logistic regression model demonstrating risk factors associated with post-COVID-19 condition in children at 6-month follow-up. Odds ratios and 95% CIs are presented. D Multivariable logistic regression model demonstrating risk factors associated with post-COVID-19 condition in children at 12-month follow-up. Odds ratios and 95% CIs are presented

In adults, female sex was the only statistically significant risk factor of post-COVID-19 condition at both 6-month (odds ratio of 2.04 (95% CI 1.57 to 2.65) and 12-month (2.04, 1.54 to 2.69) follow-up. Pre-existing hypertension was also (1.42, 1.04 to 1.94) independently associated with post-COVID-19 condition at 12 months only.

In children, pre-existing neurological comorbidities were associated with post-COVID-19 condition at both 6-months (4.38, 1.36 to 15.67) and 12 months (8.96, 2.55 to 34.82). History of allergic respiratory diseases was a risk factor (2.66, 1.04 to 6.47) for post-COVID-19 condition at 12 months only.

This prospective cohort study with 1013 adults and 360 children, who were previously hospitalised with laboratory confirmed SARS-CoV-2 infection, assessed the 6- and 12-month prevalence of post-COVID-19 condition, according to the WHO case definition, along with phenotypes and risk factors. We found that half of adults and one of five children had post-COVID-19 condition at 6 months follow-up, with fatigue being the most common manifestation. Although prevalence of post-COVID-19 condition declined between 6- and 12-month assessment, one in three adults and one in ten children still had sequelae. Post-COVID-19 condition was experienced by both sexes, with a higher risk amongst adult women. Pre-existing hypertension (adults) and pre-existing neurological comorbidities and allergic respiratory diseases (children) were associated with post-COVID-19 condition.

The prevalence of post-COVID-19 condition was significantly higher in adults and the risk of post-COVID-19 condition was 2.5 and 3 times higher in adults relative to children 6 and 12 months post-hospital discharge, respectively. This was true also for individual symptom groups, except gastrointestinal at 6 months, and cardiovascular, dermatological, and gastrointestinal at 12 months. Although persistent symptoms of COVID-19 have been assessed in many studies [ 5 ], most published research was performed prior to the WHO post-COVID-19 case definition [ 9 ] in absence of agreed terminology and associated data heterogeneity. With differences in methodology, outcome definitions, and absence of symptom duration measurement across cohorts, it is difficult to evaluate the prevalence of post-COVID-19 condition. Another limitation in the existing literature is the inadequate knowledge of COVID-19 sequelae in children [ 6 ] and the lack of direct head-to-head comparison of its features and prevalence in adults and children, which do not allow for complete understanding if manifestations behave differently based on age.

Persistence of symptoms is a worrisome issue, with half of the adults in our study reported post-COVID-19 condition at 6 months, and 34% still experiencing one or more manifestations 12 months after discharge. This finding is consistent with data from China, which reported a high rate of single sequelae symptom prevalence and decrease from 68% at 6 months to 49% at 12 months [ 15 ]. Difference in prevalence may be related differences in post-COVID condition definitions, as the Chinese study was published before the WHO case definition announcement. We found a twofold decrease in the prevalence of post-COVID-19 condition from 20% between 6 and 12 months in children. To our knowledge, this is the first study reporting consequences of COVID-19 in children 1 year after acute episode, though we did detect persistent symptoms 6 months after hospital discharge in children in a previous study especially in older children and those with allergic disease [ 11 ]. Fatigue was the most common manifestation in both children and adults, regardless of the follow-up time point, though proportionally more adults than children reported fatigue. This finding is consistent with prior data [ 5 , 6 ]. In adults and children, respiratory manifestations were reported in 20% and 2%, respectively. This difference may be related to greater severity of viral pneumonia in adults, as well as greater baseline respiratory comorbidity [ 20 ]. More frequent incidental infection rates may also confer some degree of non-specific immunological protection but the association with pre-existing respiratory allergy suggests that allergic hypersensitivity and/or auto-immune responses are involved. More research into pathophysiology and immune mechanisms is required to establish the cause of described association.

One third of individuals with post-COVID-19 condition can be classified by a phenotypes, of combined manifestations. One in five can be characterised by a combination of fatigue and respiratory with or without neurological manifestations. These results are similar to those reported by Taquet and colleagues [ 21 ]. We found that people without neurological manifestations become asymptomatic by 12 months more frequently than those reporting neurological manifestations at 6 months. However, due to a limited number of individuals available for phenotyping, it is premature to make any definitive conclusions.

The risk of post-COVID-19 condition was twice as high in female as in male adult patient at both time points, in line with previous studies assessing persistent symptoms [ 15 , 17 , 22 ]. Pre-existing hypertension was associated with post-COVID-19 condition at 12 months in adults. The association between pre-existing hypertension and higher risk of post-COVID-19 condition 12 months after hospital discharge in adults has not been previously reported [ 23 ], which may be explained by the difference in outcome definition. Pre-existing neurological comorbidities and allergic respiratory diseases were associated with post-COVID-19 in children. While allergic diseases are felt to be protective of developing COVID-19, this may become a risk factor for the sequelae development and merits further consideration. It was previously hypothesised that allergic conditions may increase the risk of long-term consequences following COVID-19 and that eosinophils, mast cells, or Th-2 responses may be potentially involved in the immunopathology of post-COVID-19 condition [ 24 ], but large prospective studies with biological material collection are required to confirm this.

This study has both strengths and limitations. Strengths include the following: (1) using of standardised ISARIC Long-term Follow-up Study CRFs for adults and children; (2) using the WHO post-COVID-19 condition definition; (3) enrolling both adults and children and comparing the two cohorts; and (4) a relatively large sample size of people attending both the 6- and 12-month follow-up visits, one of the longest follow-up assessments of hospitalised patients to-date. Limitations include the following: (1) questions about spectrum composition, as we enrolled only patients from Moscow (which may limit generalisability), a low proportion of whom had severe COVID-19—issues shared with most major COVID-19 cohort studies. This limitation is balanced by their otherwise being a paucity of data from eastern Europe regarding any COVID-19 outcomes, which becomes a novelty; (2) acute data were collected from the electronic medical records with no access to additional information that could be potentially retrieved from the medical notes—we mitigated potential inaccuracies of demographic information reported by the patients/parents/carers at the time of the hospital admission with subsequent verification during follow-up telephone interviews. This is an accepted and common limitation of cohorts assembled using this methodology; (3) a low proportion of patients with severe COVID-19 patients amongst both adults and children in our cohort limits the generalisability of the study findings to hospitalised patients with more mild to moderate COVID-19; (4) parents/caregivers were interviewed in this study and not children themselves, which is an accepted limitation of paediatric research conducted in children of a particular age; (5) a risk of potential selection bias, for instance with those with symptoms more likely to agree to survey and thus overestimating the prevalence of post-COVID-19 condition [ 25 ], as only 68% of adults and 62% of children for whom we had contact information agreed to participate in our study, and 51% and 42% respectively completing both visits—although retention of over 40% is generally considered good and rates here are comparable or higher than in the recent similar cohort studies [ 15 , 26 ]. Any attrition from a cohort may result in a substantial overestimation of the prevalence of post-COVID-19 condition as those who do remain in the cohort may represent a biased sample [ 25 ]. However, we did not find significant differences between respondents and non-respondents (Table S 4 ); (6) telephone interviews were used in this study, and we acknowledge that face-to-face interviews and/or objective measurements would deliver more robust results. However, financial and pandemic restrictions did not allow for this; (7) the study used hospitalised patients’ data. Interpretation of the data gathered from such sample may be prone to collider bias, as the sample is non-random as is conditions on hospital admission.

We used the ISARIC/WHO Clinical Characterisation Protocol, a prospective pandemic preparedness protocol which is agnostic to disease and has a pragmatic design to allow recruitment during pandemic conditions. As we already underlined in previous publications, the reality of conducting research in outbreak conditions is such that appropriate co-enrolment of a control group is practically challenging, primarily because COVID-19 has overshadowed other infections which could be used as comparators, and because of the lack of agreement on a commonly accepted control group [ 10 ].

This study has shown that half of adults and one of five children have post-COVID-19 condition, as per WHO case definition, 6 months after hospital discharge, with fatigue being the most common manifestation. Respiratory manifestations also were a major problem in adults. Although the prevalence of post-COVID-19 condition declined, one in three adults and one in ten children still had manifestations at 12 months follow-up. Post-COVID-19 condition was more common adult women and amongst adults with pre-existing hypertension. In children, pre-existing neurological comorbidities and allergic respiratory diseases were associated with post-COVID-19 condition. Future studies should define COVID-19 as per the new WHO case definition to allow for a better comparability. Further investigation of risk factors and underlying physiological and immunological mechanisms merit further consideration.

Acknowledgements

We are very grateful to the Sechenov University Hospital Network and Z.A. Bashlyaeva Children’s Municipal Clinical Hospital clinical staff and to the patients, parents, carers and families for their kindness and understanding during these difficult times of COVID-19 pandemic. We would also like to thank UK Embassy in Moscow for providing a grant. We would like to express our very great appreciation to ISARIC Global COVID-19 follow-up working group for the survey development. We would like to thank Mr Maksim Kholopov for providing technical support in data collection and database administration. We are grateful to Daria Bessonova, Olga Burencheva, Natalia Chepelova, Natalia Gorbova, Rina Grigoryan, Sapiat Isaeva, Alena Khrapkova, Ildar Khusainov, Tatiana Kokorina, Margaret Kvaratskheliya, Daria Levina, Anna Lunicheva, Margarita Mikheeva, Elizaveta Mikhsin, Roman Movsisyan, Veronika Palchikova, Maxim Privalov, Tatiana Reznikova, Olga Sokova, Ivan Timchenko, Anna Zezyulina, and Mikhail Zhilinsky for their help at different stages of the project. We are very thankful to FLIP, Eat & Talk, Luch, Black Market, and Academia for providing us the workspace in time of need and their support of COVID-19 research. Finally, we would like to extend our gratitude to the Global ISARIC team, the ISARIC global adult and paediatric COVID-19 follow-up working group, and ISARIC Global support centre for their continuous support and expertise and for the development of the outbreak ready standardised protocols for the data collection.

Sechenov StopCOVID Research Team:

Nikol Alekseeva, Elena Artigas, Asmik Avagyan, Lusine Baziyants, Anna Belkina, Anna Berbenyuk, Tatiana Bezbabicheva, Vadim Bezrukov, Semyon Bordyugov, Aleksandra Borisenko, Maria Bratukhina, Ekaterina Bugaiskaya, Julia Chayka, Yulia Cherdantseva, Natalia Degtyareva, Olesya Druzhkova, Alexander Dubinin, Khalisa Elifkhanova, Dmitry Eliseev, Anastasia Ezhova, Aleksandra Frolova, Julia Ganieva, Anastasia Gorina, Cyrill Gorlenko, Elizaveta Gribaleva, Eliza Gudratova, Shabnam Ibragimova, Khadizhat Kabieva, Yulia Kalan, Margarita Kalinina, Nadezhda Khitrina, Bogdan Kirillov, Herman Kiseljow, Maria Kislova, Natalya Kogut, Irina Konova, Mariia Korgunova, Anastasia Kotelnikova, Karina Kovygina, Alexandra Krupina, Anastasia Kuznetsova, Anna Kuznetsova, Baina Lavginova, Elza Lidjieva, Ekaterina Listovskaya, Maria Lobova, Maria Loshkareva, Ekaterina Lyubimova, Daria Mamchich, Nadezhda Markina, Anastasia Maystrenko, Aigun Mursalova, Evgeniy Nagornov, Anna Nartova, Daria Nikolaeva, Georgiy Novoselov, Marina Ogandzhanova, Anna Pavlenko, Olga Perekosova, Erika Porubayeva, Kristina Presnyakova, Anna Pushkareva, Olga Romanova, Philipp Roshchin, Diana Salakhova, Ilona Sarukhanyan, Victoria Savina, Jamilya Shatrova, Nataliya Shishkina, Anastasia Shvedova, Denis Smirnov, Veronika Solovieva, Olga Spasskaya, Olga Sukhodolskaya, Shakir Suleimanov, Nailya Urmantaeva, Olga Usalka, Valeria Ustyan, Yana Valieva, Katerina Varaksina, Maria Varaksina Ekaterina Varlamova, Maria Vodianova, Margarita Yegiyan, Margarita Zaikina, Anastasia Zorina, Elena Zuykova.

Abbreviations

Authors’ contributions.

DM, DBu, and IMO conceptualised the project and formulated research goals and aims. EP, MA, ESp, PBo, PBu, NN, AS, YET, MR, AGa, AGu, NBu, SA, VK, and DM were responsible for the study design and methodology and participated in the overall project design discussions. ADG, PC, DGP, CA, JG, LFR, CLHB, GC, LS, JTS, MG, DaBu, MGS, JOW, PO, and DMN participated in the CRF development and/or provided expert input at different stages of the project. EP implemented the computer code and supporting algorithms and tested of existing code components. DM and EP tested hypotheses and discussed statistical analyses. EP performed statistical analysis. The StopCOVID Research Team, NN, PBu, MA, AGa, AS, AB, PP, DBa, MP, EA, YF, and SD conducted a research and investigation process, specifically performed data extraction, telephone interviews, and data collection. VF, AAS, PT, LM, AM, ESa, SB, EB, AAK, DM, DB, and PG provided study materials, access to patient data, laboratory data, and computing resources. MA, ESp, PBo, PBu, YET, and MR managed activities to annotate metadata and maintain research data for initial use and later reuse. EP prepared visualisation and worked under DM supervision on the data presentation. DM was responsible for the oversight and leadership for the research activity planning and execution. DM, DB, PBo, ES, AS, AG, and EP provided management and coordination for the research activity planning and execution. VF, AAS, PT, IMO, DB, PG, and DM were responsible for the acquisition of the financial support for the project leading to this publication. EP, NN, and DM wrote original draft. All the authors critically reviewed and commented on the manuscript draft at both, pre-and post-submission stages. All authors read and approved the final manuscript.

No external funding.

Availability of data and materials

Declarations.

This study was approved by the Sechenov University Local Ethics Committee on April 22, 2020 (protocol number 08–20, protocol amendment enabling serial follow-up of the cohort was approved on November 13, 2020), and Moscow City Independent Ethics Committee (abbreviate 1, protocol number 74). Parental consent was sought during hospital admission, and consent for the follow-up interview was sought via verbal confirmation during telephone interview. The consent process was approved by the ethics.

Not applicable.

JTS is supported by Welcome. CLHB declares a grant from The Netherlands Organisation for Health Research and Development. CA declares grant by Federal Ministry of Education and Research (BMBF) for EgePan Unimed project involving research activities on post-COVID-19 condition received by his institution. JG declares grants from the German Federal Ministry of Education and Research, German Federal Ministry of Health, and Danone Nutricia Research reception by his institution. He also reports Danone Nutricia Research patents in the area of breast milk composition pending. He acknowledges receiving an honorarium for serving as an Associate Editor for the Journal Pediatric Allergy and Immunology. MGS reports grants from the National Institute of Health Research UK, Medical Research Council UK, and Health Protection Research Unit in Emerging & Zoonotic Infections, University of Liverpool. He is also an independent external and non-remunerated member of Pfizer’s External Data Monitoring Committee for their mRNA vaccine program(s), a Chair of Infectious Disease Scientific Advisory Board of Integrum Scientific LLC, Greensboro, NC, USA, director of MedEx Solutions Ltd. He also declares being a minority owner at Integrum Scientific LLC, Greensboro, NC, USA, and majority owner at MedEx Solutions Ltd. He reports that a gift from Chiesi Farmaceutici S.p.A. was received by his Institution of Clinical Trial Investigational Medicinal Product without encumbrance and distribution of same to trial sites. He also serves as a non-remunerated independent member of HMG UK Scientific Advisory Group for Emergencies (SAGE), COVID-19 Response, and HMG UK New Emerging Respiratory Virus Threats Advisory Group (NERVTAG). LS declares project funding from the Welcome Trust. DB has participated to a peer-to-peer (PAACE) educational program on long COVID, sponsored by Pfizer. JOW reports funding from Danone/Nutricia, Friesland-Campina, and Airsonett. He also serves an Anaphylaxis Campaign clinical and scientific panel chairman and acknowledges travel expenses as a speaker covered by the World Allergy Organisation. DM reports receipt of grants from the British Embassy in Moscow, UK National Institute for Health Research (NIHR) and Russian Foundation for Basic Research. He also Co-Chair of International Severe Acute Respiratory and Emerging Infection Consortium (ISARIC) Global Paediatric Long COVID Working Group, member of ISARIC working group on long-term follow-up in adults, and co-lead of the PC-COS project. Other authors declare that they have no competing interests.

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Ekaterina Pazukhina, Margarita Andreeva, Ekaterina Spiridonova, Polina Bobkova, Anastasia Shikhaleva, Yasmin El-Taravi, Mikhail Rumyantsev, Aysylu Gamirova, Ismail M Osmanov, and Daniel Munblit contributed equally to the paper.

Contributor Information

Sechenov StopCOVID Research Team: Nikol Alekseeva , Elena Artigas , Asmik Avagyan , Lusine Baziyants , Anna Belkina , Anna Berbenyuk , Tatiana Bezbabicheva , Vadim Bezrukov , Semyon Bordyugov , Aleksandra Borisenko , Maria Bratukhina , Ekaterina Bugaiskaya , Julia Chayka , Yulia Cherdantseva , Natalia Degtyareva , Olesya Druzhkova , Alexander Dubinin , Khalisa Elifkhanova , Dmitry Eliseev , Anastasia Ezhova , Aleksandra Frolova , Julia Ganieva , Anastasia Gorina , Cyrill Gorlenko , Elizaveta Gribaleva , Eliza Gudratova , Shabnam Ibragimova , Khadizhat Kabieva , Yulia Kalan , Margarita Kalinina , Nadezhda Khitrina , Bogdan Kirillov , Herman Kiseljow , Maria Kislova , Natalya Kogut , Irina Konova , Mariia Korgunova , Anastasia Kotelnikova , Karina Kovygina , Alexandra Krupina , Anastasia Kuznetsova , Anna Kuznetsova , Baina Lavginova , Elza Lidjieva , Ekaterina Listovskaya , Maria Lobova , Maria Loshkareva , Ekaterina Lyubimova , Daria Mamchich , Nadezhda Markina , Anastasia Maystrenko , Aigun Mursalova , Evgeniy Nagornov , Anna Nartova , Daria Nikolaeva , Georgiy Novoselov , Marina Ogandzhanova , Anna Pavlenko , Olga Perekosova , Erika Porubayeva , Kristina Presnyakova , Anna Pushkareva , Olga Romanova , Philipp Roshchin , Diana Salakhova , Ilona Sarukhanyan , Victoria Savina , Jamilya Shatrova , Nataliya Shishkina , Anastasia Shvedova , Denis Smirnov , Veronika Solovieva , Olga Spasskaya , Olga Sukhodolskaya , Shakir Suleimanov , Nailya Urmantaeva , Olga Usalka , Valeria Ustyan , Yana Valieva , Katerina Varaksina , Maria Varaksina , Ekaterina Varlamova , Maria Vodianova , Margarita Yegiyan , Margarita Zaikina , Anastasia Zorina , and Elena Zuykova