research paper of e learning

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• My Account Login
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 10 January 2022

Predicting students’ performance in e-learning using learning process and behaviour data

• Feiyue Qiu 1 ,
• Guodao Zhang 2 ,
• Xin Sheng 1 ,
• Lei Jiang 1 ,
• Lijia Zhu 1 ,
• Qifeng Xiang 1 ,
• Bo Jiang 3 &
• Ping-kuo Chen 4  

Scientific Reports volume  12 , Article number:  453 ( 2022 ) Cite this article

21k Accesses

43 Citations

6 Altmetric

Metrics details

• Computational science
• Computer science
• Scientific data

E-learning is achieved by the deep integration of modern education and information technology, and plays an important role in promoting educational equity. With the continuous expansion of user groups and application areas, it has become increasingly important to effectively ensure the quality of e-learning. Currently, one of the methods to ensure the quality of e-learning is to use mutually independent e-learning behaviour data to build a learning performance predictor to achieve real-time supervision and feedback during the learning process. However, this method ignores the inherent correlation between e-learning behaviours. Therefore, we propose the behaviour classification-based e-learning performance (BCEP) prediction framework, which selects the features of e-learning behaviours, uses feature fusion with behaviour data according to the behaviour classification model to obtain the category feature values of each type of behaviour, and finally builds a learning performance predictor based on machine learning. In addition, because existing e-learning behaviour classification methods do not fully consider the process of learning, we also propose an online behaviour classification model based on the e-learning process called the process-behaviour classification (PBC) model. Experimental results with the Open University Learning Analytics Dataset (OULAD) show that the learning performance predictor based on the BCEP prediction framework has a good prediction effect, and the performance of the PBC model in learning performance prediction is better than traditional classification methods. We construct an e-learning performance predictor from a new perspective and provide a new solution for the quantitative evaluation of e-learning classification methods.

Similar content being viewed by others

Data-driven unsupervised clustering of online learner behaviour

Robert L. Peach, Sophia N. Yaliraki, … Mauricio Barahona

EEG-based measurement system for monitoring student engagement in learning 4.0

Andrea Apicella, Pasquale Arpaia, … Andrea Pollastro

Self-paced graph memory for learner GPA prediction and it’s application in learner multiple evaluation

Yue Yun, Ruoqi Cao, … Xuequn Shang

Introduction

E-learning has become a typical form of education 1 and an important part of the development of Internet-based education. Due to the impact of the COVID-19 pandemic, e-learning has been widely used worldwide due to its high temporal and spatial flexibility, low knowledge acquisition threshold, and rich learning resources. However, in this mode, teachers cannot easily perceive the learning status of their learners 2 , and questions about the quality of e-learning have been raised. The study of learning performance prediction provides a basis for teachers to adjust their teaching methods for students who may have problems by predicting students’ performance on future exams, reducing the risk of students failing to pass the course, and ensuring the quality of e-learning. Through a large number of empirical studies that investigate the relationship between e-learning behaviour and learning performance, learners’ e-learning behaviour has an important impact on learning performance. Therefore, in recent years, learning performance prediction based on learning process data has received widespread attention. The use of measurement, collection and analysis of learning process data to achieve learning performance prediction 3 can help teachers modify teaching strategies in time and start during students’ learning processes using the role of supervision and early warning 4 .

Research notes that e-learning behaviour data are important to understanding e-learning processes. E-learning behaviour data refers to the data generated by learners in various behavioural activities performed on e-learning platforms or online teaching organizations, which can describe the activity records of learners in the learning process, specifically including the number of login platforms, the number of access to resources, the number of participants in forum discussions, the number of access to resources and other behavioural data 5 . Concurrently, e-learning behaviour involves less private information, and data collection and use are more convenient, which has an important impact on e-learning performance 6 . Therefore, researchers have conducted in-depth research on e-learning behaviour 7 and constructed different learning performance predictors based on e-learning behaviour 8 . Learning performance prediction is usually a binary classification task that divides students into two groups of ”passed” or ”failed” to predict the possibility of passing the test in the future 9 . Because predictive learning is the primary advantage of machine learning technology, it is often used to train the learning performance prediction model using a simple method 10 , 11 . Although this type of predictor can achieve good prediction results, it has certain limitations. First, regarding low generalizability and high computational complexity, a large number of e-learning behaviour data of different dimensions are captured and recorded during the e-learning process. Constructing an e-learning performance predictor is prone to overfitting and high computational complexity. Feature selection can be used to retain key learning behaviours to reduce model operational costs, which is of practical significance for online platforms to provide high-accuracy and low-time-consuming learning performance prediction services. Second, the single input method of e-learning behaviour data requires that e-learning predictors generally use e-learning behaviour data directly as input variables. Few predictors will consider the combined effect of learning behaviour data (i.e., perform feature fusion processing) on the same type of learning behaviour data and then use it for training. Last, key learning behaviour indicators are not standardized, and those found by different researchers ares different. This field of study has failed to identify key behaviour indicators that can be used to effectively predict learning performance 12 , 13 ; thus, the results of the prediction models are frequently affected by platform-specific learning behaviours, which affects the model’s mobility.

To solve these problems, we propose the behaviour classification-based E-learning performance prediction framework (BCEP prediction framework), summarize the classic e-learning classification methods, analyse the e-learning process in detail, propose the process-behaviour classification model (PBC model), and construct an e-learning performance predictor based on the PBC model. The primary contributions of this article are as follows. First, the BCEP prediction framework proposed in this paper includes four steps: data cleaning, behaviour classification, feature fusion, and model training. Compared with general learning performance predictors, this framework yields more accurate predictions of student achievement. During training, computational complexity is reduced, and the model’s mobility and versatility are increased in the application process. Second, the PBC model proposed in this paper divides e-learning behaviours into four categories. The learning performance predictor based on the PBC model is shown to perform markedly better than existing typical classification methods .

This article is composed of 7 summaries, and the remainder of its content is organized as follows. Section 2 summarizes the development status of e-learning performance prediction, focusing on the prediction indicators and methods of e-learning performance prediction. Section 3 describes the BCEP prediction framework in detail. Section 4 reviews the existing learning classification models and designs a new e-learning behaviour classification method-PBC model. Section 5 describes the experiments used to verify the effectiveness of the BCEP prediction framework and PBC model. In Section 6, experimental results are systematically analysed and discussed. Last, Section 7 provides conclusions and prospects for future research.

Related work

Prediction indicators of e-learning performance.

E-learning performance predictors are generally summarized as tendency indicators and behavioural performance indicators 14 . Tendency indicators are inherent attributes of themselves; primarily static data, which are generally collected before the start of the course or semester, such as socioeconomic status 15 , historical academic records 16 and gender 17 , are common indicators of propensity. Many researchers have used propensity indicators to develop learning early-warning models to predict students’ learning in a course, a semester, and other stages. Although the predictors established by these studies achieve good performance, they ignored the role of learning behaviour records. For example, many studies used students’ historical performance or demographic data that were not related to learning. Although these studies can predict learning performance through learner characteristics, this method ignored that most of the tendency indicators were not in the student’s and the teacher’s control, and the students’ changes in curriculum were ignored 18 . In addition, there is a privacy problem with preference indicators, and personal data collected by educational institutions cannot be shared publicly. The behavioural performance index (i.e., the dynamic index reflected by the learner in the learning process 19 , 20 , 21 ) generally did not have such a problem. E-learning behaviour data can accurately describe the time and energy that students spend on a specific course, such as the frequency of access to course materials 22 and the frequency of online discussions 23 . Some studies also tried to use the combination of two indicators to complete learning prediction 24 but encountered problems related to increasing computational costs.

The accumulation of educational big data and the emergence of new methods of connecting and exchanging information have laid the foundation for e-learning behaviour research. Learners’ learning behaviour data are important when analysing changes in learners’ behaviour, preferences, and ability levels 25 , which promotes related research on learning performance prediction based on learning behaviour. Learning input theory explains the relationship between learning behaviour and learning performance 26 and states that learning behaviour is a key factor affecting learning performance and an important indicator for predicting learning performance 27 . Concurrently, many studies have confirmed that there is a significant correlation between student online activities and academic performance 28 , 29 , and observing learning activities at a finer-grained level can strengthen the grasp of learning conditions and promote constructive learning 30 . Therefore, many researchers have explored the correlation between e-learning behaviour and learning performance, and used e-learning behaviour to predict learning performance. For example, Qi 31 reported that there is a significant positive correlation between learners’ e-learning behaviour and the learning effect. Liang et al. 32 recorded student data through a curriculum management system and used regression analysis to find a correlation between learning experience, learning behaviour and learning performance. Comer et al. 33 found that in the e-learning environment, collaborative communication behaviour will deepen students’ understanding of knowledge and encourage students to achieve certain learning achievements. Kokoç and Altun 34 used learning interaction data to predict the learning performance of online learners and found that the access behaviour of learning content, books, forums, and course activities can significantly affect learning outcomes. Some studies looked for a relationship between a certain behavioural activity or several behavioural activities and learning performance. Zheng et al. 35 found that there is a positive correlation between the number of logins and the final grades of students. Qureshi et al. 36 used a questionnaire survey method to find that cooperative learning and learning participation play an intermediary role between social factors and learning performance, and verified that collaborative learning behaviours promote learning performance in e-learning. Shen 37 noted that the proportion of learners’ homework completion and video completion rate in e-learning affect learning.

Based on the literature about learning performance prediction using learning behaviours, analyses of e-learning behaviour is frequently limited to independent e-learning behaviours. Few studies have explored the internal associations and differences between e-learning, specifically categorizing and analysing e-learning behaviours. In previous studies that used learning behaviour classification as primary predictor indicators, researchers only used independent e-learning behaviour data as the input of predictor training instead of the fusion data of learning behaviour classification, which reduced the importance of learning behaviour classification.

Prediction algorithm of e-learning performance

In e-learning performance prediction research, the selection of predictive indicators occupies an important position, and prediction methods also play a key role, particularly feature selection and algorithm selection, which can markedly affect the prediction effect. Therefore, it is necessary to identify relevant research and applications of machine learning and feature selection.

An increasing number of studies have confirmed that when constructing predictive models, multiple data points cannot always guarantee a higher predictive ability. Unnecessary features will affect the generalizability of the model and increase the computational cost of the model. For example, Akram et al. 38 used ten prediction algorithms to predict students with learning difficulties through assignment submission behaviour and found that the prediction performance of all algorithms decreased as the number of input data increased. It is thus necessary to select behavioural features that are meaningful for learning performance from the sample data and then input them into the model for training and learning; thus, feature selection is necessary 39 . Three methods can be used for feature selection: the filter method, the wrapper method, and the embedded method. Madichetty et al. 40 verified that the selection of key features is helpful for classification prediction. The filtering feature selection method has the advantages of strong independence, fast running speed and low computational complexity in machine learning algorithms, but makes it difficult to completely delete redundant features when there are many redundant features and high target relevance 41 . Wrapping methods can be independent of machine learning models but typically have high computational costs 42 . The embedded method embeds feature selection into other algorithms and selects new features during the training process, which can effectively improve the efficiency of model learning 43 .

In addition, machine learning algorithms have unique advantages in solving classification problems. For example, Huang and Lin et al. 44 proposed a multimodal information perception method for flexible manipulators based on machine learning methods to complete gestures, object shapes, sizes and weights to recognize tasks, compared the recognition accuracy of optical sensor information (OSI), pressure sensor information (PSI) and dual sensor information (DSI). They found that the KNN algorithm with DSI performed better than other with regard to recognition accuracy. Cao and Zhang et al. 45 used and improved the deep learning method; used the multitask cascaded convolution network (MTCNN) method to locate the face of cartoon characters; performed face detection and face feature point detection; and recognized the image emotion of cartoon style. Muhammad and Liu et al. 46 extended the application of machine learning to the field of language recognition and translation by sharing dictionary embeddings between the parent language and the child language without using reverse translation or manual noise injection and proposed a language-independent hybrid transfer learning (HTL) method to solve the problem of data sparseness in low-resource languages (LRLs). Machine learning technology has gradually emerged in the development of the learning analysis process, which facilitates the collection and analysis of student and environmental data 47 . In recent years, many machine learning classification algorithms have been applied to the field of learning performance prediction. For example, Jiang et al. 48 built a predictor based on logistic regression, which combined students’ first week of homework performance and social interaction behaviour to predict learners’ performance in the course. Aziz et al. 49 selected five parameters, race, gender, family income, college enrolment mode and average grade point, and used the naïve Bayes classifier to predict the average grade point. Ahuja and Kankane 50 used the K-nearest neighbour algorithm to predict the results of students’ academic acquisition based on the previous academic performance and non-academic factors of college students. Asif et al. 51 used the decision tree algorithm to predict students’ performance at the end of the four-year study plan. Jie-ping et al. 52 proposed a performance prediction method that combined fuzzy clustering and support vector machine regression based on students’ historical performance and behavioural habits.

Behaviour-based classification of the e-learning performance prediction framework

Researchers typically use the behaviour of each e-learning category as an independent predictor of the performance of e-learning to build predictive models. However, different e-learning behaviours have potential correlations and can be classified into different behaviour categories according to different rules. This research innovatively constructs a learning performance predictor from the perspective of behaviour categories and proposes the behaviour classification-based E-learning performance prediction framework (BCEP prediction framework).

The BCEP prediction framework describes the complete process of implementing learning performance predictors through e-learning behaviour categories, as shown in Fig.  1 . The prediction framework includes four core links: (1) data pre-processing, which includes data cleaning and conversion from the original e-learning behaviour data obtained by the e-learning platform obtain standardized e-learning behaviour data; (2) feature selection, which is performed on pre-processed e-learning behaviour data to obtain key e-learning behaviours; (3) feature fusion, which classifies core learning behaviours according to specific rules, constructs a collection of behaviour categories, and then performs feature fusion to obtain the category feature value of each type of e-learning behaviour; and (4) model training, which builds an e-learning performance predictor based on a variety of machine learning algorithms.

Behavior-based classification of e-learning performance prediction framework.

(1) Data pre-processing

The quality of e-learning behaviour data directly affects the accuracy of predictive models. Therefore, the first step is to clean the e-learning behaviour data obtained from the e-learning platform. There is no unified process for data cleaning, but the method should be selected according to the real situation of the data to manage missing values, duplicate values, and abnormal values. Concurrently, e-learning behaviours recorded by e-learning platforms are often not of a single dimension, e-learning behaviour data of different dimensions are often not numerically comparable, and feature selection cannot be performed. The proposed framework solves this problem by standardizing e-learning behaviour data in different dimensions with Z scores.

We define the original e-learning behaviour set \(B \left\{ b_{1}, b_{2}, \ldots . ., b_{n}\right\}\) and the standard e-learning behaviour set \(B\left\{ b^{\prime }, b_{2}^{\prime }, \ldots , b_{n}^{\prime }\right\}\) . Where \(b_{n}\) is the n-th e-learning behaviour recorded by the e-learning platform, and \(b_{n}^{\prime }\) is the n-th e-learning behaviour after standardization. Concurrently, the original e-learning behaviour data and the standard e-learning behaviour data are defined, where n is the n-th e-learning behaviour, and m is the m-th data of the current e-learning behaviour. For example, \(d_{nm}\) is the second behaviour data of the first type of e-learning behaviour recorded by the e-learning platform, and the formula for \(d_{nm}^{\prime }\) is as follows:

Where \(\mu _{\mathrm {b}_{\mathrm {m}}}\) is the average value of the n-th type of e-learning behaviour data, and \(\sigma _{\mathrm {b}_{\mathrm {m}}}\) is the variance of the n-th type of e-learning behaviour data.

(2) Feature selection

Feature selection can select relevant features that are beneficial to the training model from all features, thereby reducing the feature dimension and improving the generalizability, operating efficiency and interpretability of the model. This framework uses the variance filtering method to perform feature selection on standardized e-learning behaviour data. The variance filtering method uses the variance of each feature itself to filter the features. The smaller the variance of the feature, the lower the difference of the sample on this feature, and the smaller the distinguishing effect of the feature on the sample. The threshold is an important parameter of the variance filtering method, which represents the threshold of variance; thus, features with variance less than the threshold will be discarded.

We define the characteristic value set of e-learning behaviour \(V \left\{ v_{1}, v_{2}, \ldots , v_{n}\right\}\) , where \(v_{n}\) is the characteristic value of the n-th e-learning behaviour, and its formula is as follows:

where \(\mu _{b^{\prime }_{m}}\) represents the average value of the n-th standard e-learning behaviour data. The elements in traversal V are compared with the variance threshold. If the current e-learning behaviour feature value is greater than the threshold, the corresponding e-learning behaviour is added to the key e-learning behaviour set; otherwise, it is not added.

(3) Feature fusion

First, according to the e-learning behaviour classification model, the key e-learning behaviour is divided into different e-learning behaviour clusters. We assume that the classification Model M is composed of n types of e-learning behaviour categories ( ie., \(M\left\{ C_{1}, C_{2}, \ldots , C_{n}\right\}\) ). After dividing the e-learning behaviour categories, n e-learning behaviour clusters are generated, and each type of e-learning behaviour cluster includes a varying number of e-learning behaviours, such as \(C_{1}\left\{ b_{1}, b_{2}, \ldots , b_{n}\right\}\) , where, \(b_{n}\) is the n-th e-learning behaviour that meets the standard of \(C_{1}\) .

Then, feature fusion is performed on each e-learning behaviour cluster to obtain the corresponding feature value of the e-learning behaviour category \(C_{1}\left\{ b_{1}, b_{2}, \ldots , b_{n}\right\}\) as an example. The calculation formula of its category feature value is as follows:

In Eq. 3 , \(V_{b_{i}}\) is the \(b_{i}\) characteristic value of the behaviour, \(\lambda =0\) means the student has passed the curriculum, \(\lambda =1\) means failed. Similarly,we construct the feature value set of e-learning behaviour category \(V_{c}\left\{ V_{c1}, V_{c2}, \ldots , V_{ci}\right\}\) where \(V_{C_{i}}\) is the category feature value of the \(C_{1}\) e-learning behavior.

(4) Model training

In the model training session, classic machine learning methods such as SVC, NaÃve Bayes, KNN and Softmax are selected, and the e-learning behaviour category feature value set \(V_{C}\) is used as the feature data to train the e-learning performance prediction model. After many iterations, the best e-learning performance prediction model is selected to predict the e-learning performance of e-learners.

E-learning process—behaviour classification model

The e-learning behaviour classification model is an important component of the BCEP prediction framework that directly affects the prediction effect of the e-learning performance prediction model. This paper summarizes the current mainstream e-learning behaviour classification methods, as shown in Table  1 .

Table  1 shows that most researchers use interactive objects as the basis for the classification of learning behaviours. The primary interactive objects include learning systems, resource content, learning communities, and learners themselves. However, when learners are in different stages of learning, they often engage in different learning behaviours, but the interactive objects may be the same. The classification method based on interactive objects does not fully consider the process of learning. Based on these ideas, this study constructed an e-learning behaviour classification model based on the e-learning process-process-behaviour classification model (PBC model), as shown in Fig.  2 .

The process-behaviour classification model (PBCM.

The e-learning process primarily includes the learning stage, the knowledge acquisition stage, the interactive reflection stage and the learning consolidation stage 5 . The learning stage is the preparation process for learners to officially start e-learning; the knowledge acquisition stage is the most important e-learning process and is also the process by which learners initially acquire knowledge; the interactive reflection stage is a process in which learners interact with teachers and peers and reflect on themselves during the interaction; and the stage of learning consolidation is the process by which learners consolidate internalized knowledge. The model is centred on online learners, and according to the e-learning process, learning behaviour is divided into learning preparation behaviour (LPB), knowledge acquisition behaviour (KAB), interactive learning behaviour (ILB), and learning consolidation behaviour (LCB).

Learning preparation behaviour (LPB ) occurs during the learning stage and is the most basic behaviour of learners in e-learning. Specifically, LPB includes behaviours such as logging in to the learning platform, accessing the primary page of the course, and accessing the course activity interface.

Knowledge acquisition behaviour (KAB) occurs during the knowledge acquisition stage and is the behaviour of online learners directly acquiring knowledge. KAB primarily includes activities such as browsing course content resources, participating in course activities, watching course videos, and accessing resource links.

Interactive learning behaviour (ILB) occurs in the interactive reflection stage and is one of the key learning behaviours in e-learning. ILB has been proven to have a positive effect on the continuity and learning effect of e-learning 61 . Its specific manifestations include participating in seminars, publishing forums, replying to forums, asking questions to teachers, etc..

Learning consolidation behaviour (LCB) occurs in the stage of learning consolidation and refers to the behaviour of learners to strengthen the degree of knowledge mastery, primarily including proposing postclass reflections and completing postclass tests.

Experimental design

Experiments are used to compare the prediction performance of the predictor in the traditional framework and the BCEP prediction framework based on the PBC model proposed in this study to verify the effectiveness of the proposed framework. Its predictors include six machine learning methods: SVC (R), SVC (L), Naïve Bayes, KNN (U), KNN (D) and softmax 10 . We selected the accuracy rate, F1-score and Kappa coefficient as the quantitative indicators to evaluate the prediction performance. To fully verify the BCEP prediction framework, the evaluation indicators also include the time required for the experiment to complete the prediction.

We used a XiaoXin AirPlus14 Laptop to build the experimental environment, which consists of an AMD Ryzen 5600u processor, NVIDIA GeForce MX450 graphics card and a 500-GB hard disk. In terms of software, programming was performed in the Jupyter lab programming platform of the Windows 10 operating system and in the Python programming language for experiments.

Data sources

The Open University Learning Analytics Dataset (OULAD) 62 is considered to be one of the most comprehensive international open datasets in terms of e-learning data diversity, including student demographic data and interaction data between students and VLE. The role of Open University in developing this dataset is to support research in the field of learning analysis by collecting and analysing learner data to provide personalized guidance and optimize learning resources. The dataset contains 7 course modules (AAA   GGG), 22 courses, e-learning behaviour data and learning performance data of 32,593 students. The collection phase of the entire dataset includes collection, selection, and anonymization. We use SAS technology to create a data warehouse to collect data, select data containing student information from 2013 to 2014, and finally anonymize the data. Its design types are time series design, data integration objective and observation design; its measurement type is learning behaviour; its technology type is digital curation, and its factor type is temporal_interval. In this experiment, the DDD course module with the most sample data was selected, and the e-learning data of 6,272 learners who participated in the DDD course were used as the data source for training and verifying the e-learning performance predictor. When learning DDD courses, learners performed 12 e-learning behaviours, as shown in Table   2 .

Experimental design for validation of the BCPF prediction framework

(1) Experimental program

This experiment sets up three experimental groups to compare and verify the effectiveness of the BCEP prediction framework. The difference between the three experimental groups lies in the feature data used to train the learning performance predictor. Group 1 uses online behaviour data that have only undergone data pre-processing as characteristic data. Group 2 uses data that have undergone feature selection but not feature fusion as characteristic data. Group 3 follows the BCEP prediction framework and uses existing features. The data that have undergone feature fusion are selected as feature data. In this experiment, 6 machine learning methods were selected, 18 learning performance predictors were constructed based on the 3-feature data above , and the effectiveness of the BCEP prediction framework was verified by comprehensively comparing the prediction results of the 18 learning performance predictors.

The experimental group using feature selection uses the variance filtering method to feature a selection of 12 online learning behaviours in the dataset, and 8 of them are selected as the feature data for constructing the learning performance predictor according to the variance threshold. The feature data of the three experimental groups after feature selection are shown in Table 3 :

Before using feature fusion, the experimental group classifies e-learning behaviours through the PBC model and then performs feature fusion on behaviour clusters to obtain behaviour category feature values. Finally, the behaviour category feature value is used as the feature data for building the learning performance predictor. A schematic diagram of the learning behaviour classification of Group 3 is shown in Fig.  3 :

Schematic diagram of e-learning behaviour classification in Group 3.

In this experiment, the number and dimensions of data after feature fusion in the 3 experimental groups are shown in Table  4 .

Experimental design for the validation of the PBC model

The classification of e-learning behaviours are generally limited to theoretical research, and its role and scientific validity in learning performance prediction are difficult to verify. This experiment follows the learning performance prediction framework based on behaviour classification and selects three representative behaviour classification models for comparison with the PBC model, including Moore 53 e-learning behaviour classification (a total of three types of behaviour), Wu 60 e-learning behaviour classification (a total of four types of behaviours) and Peng 56 e-learning behaviour classification (a total of five types of behaviours). We use 6 classic machine learning methods to construct 24 learning performance predictors and verify the effectiveness of the PBC model by comprehensively comparing the prediction results of 24 learning performance predictors.

(2) Behaviour classification

First, feature selection is performed on 12 e-learning behaviours in the original dataset, and 8 e-learning behaviours are selected according to the variance threshold. Then, the 8 e-learning behaviours are classified according to the classification methods of the 4 experimental groups, and the classification results are shown in Fig.  4 :

Online student behaviour classification: ( a ) PBCM, ( b ) Moore, ( c ) Wu, and ( d ) Peng.

Predictor implementation and evaluation

This experiment selects six machine learning algorithms that are currently widely used in the field of learning and prediction, such as SVC (R), SVC (L), Naïve Bayes, KNN (U), KNN (D) and softmax. We divided the original data \(70:30\%\) for training and testing 63 , and the expected output value of the predictor was ”qualified” or ”unqualified”.

Common indicators that are used to evaluate predictors include accuracy (ACC), F1-score (F1), and Kappa coefficient (K). ACC is considered to be the most commonly used measurement index, which refers to the proportion of the number of correctly classified samples to the total number of samples, and its formula is as follows:

where \(T_{p}\) is the number of positive samples correctly predicted, \(T_{n}\) is the number of negative samples correctly predicted, and totaldata is the total number of samples.

However, because the accuracy rate cannot fully evaluate the prediction model, further analysis of the F1-score (F1) and Kappa coefficients is required. The formula of F 1 is as follows:

The formula of Kappa is as follows:

where \(P_{o}\) is the observed coincidence ratio, and \(P_{e}\) is the coincidence ratio due to randomness.

Result analysis and discussion

This section presents the experimental results, including ACC, F1, Kappa, and the prediction time of each experimental group as calculated by the six machine learning methods. The BCEP prediction framework and the PBC model are verified by analysing these data.

BCEP prediction framework validation

We designed 3 control groups (different feature data) and used 6 common machine learning algorithms to build 3 types (18) of learning performance predictors to discuss the effectiveness of the BCEP prediction framework based on the prediction effects of the predictors. The ACC, F1, and Kappa of the three types of learning performance predictors are shown in Figs.  5 ,  6 , and 7 , respectively. To verify the role of feature selection, we also compared the experimental time of different experimental groups to complete the prediction task, as shown in Fig.  8 .

Accuracy of the three types of prediction models.

F1-score of the three types of prediction models.

Kappa of the three types of prediction models.

Figure  5 describes the accuracy of the 6 algorithms (SVC(R), SVC(L), naïve Bayes, KNN(U), KNN(D), and softmax) with three different data processing methods. The accuracy rate of Group 1 is distributed between 89.7% and 91.65%, that of Group 2 is between 89.15% and 91.00%, and that of Group 3 is between 95.44% and 97.40%. The accuracy of Group 3 is also shown to be higher than that of the other two experimental groups with all six algorithms, which means that the prediction accuracy based on the proposed BCEP prediction framework is the highest. Experimental results show that the F1-score of Group 1 is between 0.9280 \(\sim\) 0.9423, that of Group 2 is between 0.9246 \(\sim\) 0.9374, and that of Group 3 is between 0.9685 \(\sim\) 0.9818; thus, Group 3 has the highest F1-score. The Kappa of Group 1 is between 0.7473 \(\sim\) 0.7916, that of Group 2 is between 0.7310 \(\sim\) 0.7820, and that of Group 3 is between 0.8865 \(\sim\) 0.9364; thus, Group 3 achieve markedly higher Kappa values. Lastly, the computational time required for Group 1 under each algorithm is 0.0139 s \(\sim\) 0.1489 s, that for Group 2 is 0.0070 s \(\sim\) 0.1199 s, and that for Group 3 is 0.0050 s \(\sim\) 0.1080 s; thus, the computational time required for Group 2 is less than that of Group 1, and Group 3 is the fastest in obtaining prediction results in each algorithm.

In addition, we can compare the indicators of Groups 1, 2, and 3 using Figs.  5 ,  6 ,  7 , and 8 , although the prediction performance of Groups 1 and 2 on different algorithms has both advantages and disadvantages. In general, after applying the feature selection strategy, Group 2 reduces the feature dimension from 12 to 8, the prediction effect is still near that of Group 1, and the speed is increases by 23.56%. These results show that the feature selection strategy can reduce the predictor’s training parameters while maintaining the predictor’s predictive performance and can reduce the time complexity of the operation. Group 3 is based on Group 2, according to the idea of behaviour classification and adopts a feature fusion strategy to further reduce the feature dimension from 8 to 4, and all indicators for each of the 6 machine learning algorithms are better than those of Group 1 and Group 2. Further analysis of Figs.  5 ,  6 , and 7 shows that the accuracy increased by 5.8% and 6.1% on average compared to Groups 1 and 2; the F1-score increased by 4.06% and 4.24%, respectively; Kappa markedly increased by 14.24% and 15.03%, respectively; and the computation time decreased by 41.57% and 23.56%, respectively. The learning performance prediction framework based on behaviour classification proposed in this paper is thus effective in real scenarios. Building a learning performance predictor with this framework can reduce the dimensionality of feature data and also markedly improve prediction performance compared to traditional methods that only use data pre-processing or feature selection strategies to build learning performance predictors based on pre-processing.

PBC model validity verification

The feature fusion link is a critical step of the proposed framework, and the learning behaviour classification model directly determines the effect of feature fusion. We thus designed 3 comparative experiments (different learning behaviour classification models), built 4 types (24) of learning performance predictors based on 6 common machine learning algorithms, and analysed the proposed PBC model based on the prediction effects of these predictors. The effectiveness of the four types of learning performance predictor accuracy, F1-Score, and Kappa is shown in Figs.  9 , 10 , and 11 .

Computation time required for each of the three types of prediction models.

Accuracy of the four types of prediction models.

F1-score of the four types of prediction models.

Kappa of four types of prediction models.

Figure  9 describes the prediction accuracy of the four groups of experiments (PBC model group, Moore group, Wu group, Peng group). The accuracy of the PBC model group is between 95.44% and 97.40%; that of the Moore group is between 94.25% and 96.42%; that of the Wu group is between 95.01% and 96.10%; and that of the Peng group is between 90.89% and 95.34%. The PBC model group thus achieved a higher accuracy with the Naïve Bayes, KNN (U), and KNN (D) algorithms. Figure  10 shows the F1-score results of four sets of experiments. The F1-score of the six predictors based on the PBC model is between 0.9685 and 0.9818, that based on the Moore group is between 0.9606 and 0.9749, that based on the Wu group is between 0.9654 and 0.9727, and that based on the Peng group is between 0.9388 and 0.9677. The six algorithm models based on the PBC model all have higher F1-score results; the Moore-based and Wu-based F1-score performances are equivalent; and the Peng-based F1-score performance is the worst. Figure  11 shows the Kappa values of the comparative experiment. The Kappa interval based on the PBC model group is 0.8865 to 0.9364, that based on the Moore group is 0.8550 to 0.9126, that based on the Wu group is 0.8764 to 0.9043, and that based on the Peng group is 0.7881 0.8844. Thus, except for the SVC(R) algorithm, the Kappa of the other algorithms in the PBC model group are higher, and the indicators of the PBC model group are the most stable.

From Figs.  9 ,  10 , and 11 , further analysis shows that the average accuracy rate of the PBC model group is 0.65%, 0.60%, and 2.02% higher than that of the Moore group, Wu group, and Peng group, respectively, and the upper (lower) limits of the accuracy increase by 0.98% (1.19%), 1.34% (0.33%), and 2.06% (4.56%); the average value of F1- score is higher by 0.45%, 0.41%, and 1.44%, and the upper (lower) limits of F1-score increase by 0.69% (0.79%), 0.90% (0.32%), and 1.41% (3.48%); the average values of Kappa are higher by 1.61%, 1.48%, and 4.86%, respectively, and the upper (lower) limits of Kappa increase by 2.38% (3.15%), 3.20% (1.00%), and 5.20% (9.84%). Thus, the predictor constructed based on the PBC model achieves the best accuracy, F1-score and Kappa value, and its prediction performance is better than that of the Moore classification method and Wu classification method, and markedly better than that of the Peng classification method. Therefore, when performing learning performance prediction tasks, it is effective and better to use the PBC model to divide learning behaviours.

The learning performance predictor is an effective tool to ensure the quality of e-learning. How to build a learning performance predictor with high versatility and high accuracy has become a research hotspot in e-learning. This paper innovatively begins from the starting point of behaviour classification, introduces the learning behaviour feature fusion strategy to the traditional method, proposes the BCEP prediction framework, and proposes the PBC model based on a summary of existing e-learning behaviour classification methods. Experimental results with the OULAD dataset show that the BCEP prediction framework performs markedly better than the traditional learning performance predictor construction method, and the learning performance predictor constructed by this framework is accurate and stable. Subsequent experiments showed that the PBC model proposed in this paper as a feature fusion strategy of the prediction framework is effective and superior to other e-learning behaviour classification methods, and to a certain extent, also provides a new feasible scheme for quantitatively evaluating the pros and cons of e-learning behaviour classification methods.

In future work, we plan to build learning performance predictors for different e-learning platforms using the framework proposed in this article. By recording and analysing the performance of these predictors in real-world applications, we plan to optimize the proposed BCEP prediction framework. Concurrently, considering that the prediction targets for e-learning should be more diversified, in addition to the e-learning performance mentioned in this article, we plan to use similar methods to predict e-learning emotions to achieve better online supervision and early warning through multiangle prediction results and to ensure the quality of online learners’ learning.

Giannakos, N. & Vlamos, P. Empirical examination and the role of experience. Educational webcasts’ acceptance. Br. J. Educ. Technol. 44 , 125–143. https://doi.org/10.1111/j.1467-8535.2011.01279.x (2013).

Article   Google Scholar  

Qu, S., Li, K., Wu, B., Zhang, X. & Zhu, K. Predicting student performance and deficiency in mastering knowledge points in moocs using multi-task learning. Entropy 21 , 1216. https://doi.org/10.3390/e21121216 (2019).

Article   ADS   PubMed Central   Google Scholar  

Gasevic, D., Siemens, G. & Rose, C. P. Guest editorial: Special section on learning analytics. IEEE Trans. Learn. Technol. 10 , 3–5. https://doi.org/10.1109/tlt.2017.2670999 (2017).

Shu, Y., Jiang, Q. & Zhao, W. Accurate alerting and prevention of online learning crisis: An empirical study of a model. Dist. Educ. China https://doi.org/10.13541/j.cnki.chinade.2019.08.004 (2019).

Sun, Y. Characteristics analysis of online learning behavior of distance learners in open university. China Educ. Technol. 2 , 64–71 (2015).

Google Scholar  

Cohen, A. Analysis of student activity in web-supported courses as a tool for predicting dropout. Etr&D-Educ. Technol. Res. Dev. 65 , 1285–1304. https://doi.org/10.1007/s11423-017-9524-3 (2017).

Lin, J. Moocs learner characteristics and study effect analysis research. China Audio-vis. Educ. 2 , 2 (2013).

Balakrishnan Eecs, G.,. Predicting student retention in massive open online courses using hidden markov models. Digit. Collect. 2 , 2 (2013).

Joksimovi, S. et al. How do we model learning at scale a systematic review of research on moocs. Rev. Educ. Res. 88 (1), 43–86. https://doi.org/10.3102/0034654317740335 (2017).

Coussement, K., Phan, M., Caigny, A. D., Benoit, F. & D. & Raes, A.,. Predicting student dropout in subscription-based online learning environments: The beneficial impact of the logit leaf model. Decis. Support Syst. 135 , 113325. https://doi.org/10.1016/j.dss.2020.113325 (2020).

Kotsiantis, S., Pierrakeas, C. & Pintelas, P. Preventing student dropout in distance learning using machine learning techniques. Springer Berlin Heidelberg 18 , 411–426. https://doi.org/10.1080/08839510490442058 (2003).

Lei, Z. & Tong, D. The prediction of academic achievement and analysis of group characteristics for mooc learners based on data mining. Chongqing Higher Educ. Res. 2 , 1–13 (2021).

Yang Zong, H. Z. & Hongtao, S. A logistic regression analysis of learning behaviors and learning outcomes in moocs. Dist. Educ. China https://doi.org/10.13541/j.cnki.chinade.20160527.002 (2016).

Fan, Y. & Wang, Q. Prediction of academic performance and risk: A review of literature on predicative indicators in learning analytics. Dist. Educ. China https://doi.org/10.13541/j.cnki.chinade.2018.01.001 (2018).

Romero, C., Cerezo, R., Bogarín, A. & Sànchez-Santillán, M. Educational process mining: A tutorial and case study using moodle data sets. Data Min. Learn. Anal. Appl. Educ. Res. 2 , 1–28 (2016).

Nawang, H., Makhtar, M. & Shamsudin, S. Classification model and analysis on students’ performance. J. Fundam. Appl. Sci. 9 , 869–885. https://doi.org/10.4314/jfas.v9i6s.65 (2017).

Keogh, E. J. & Mueen, A. Curse of dimensionality. Encycl. Mach. Learn. Data Mining 314–315 , 2017. https://doi.org/10.1007/978-1-4899-7687-1_192 (2017).

Hooshyar, D., Pedaste, M. & Yang, Y. Mining educational data to predict students’ performance through procrastination behavior. Entropy 22 , 12. https://doi.org/10.3390/e22010012 (2020).

Article   ADS   Google Scholar  

Du, X., Yang, J., Shelton, B. E., Hung, J. & Zhang, M. A systematic meta-review and analysis of learning analytics research. Behav. Inf. Technol. 40 , 49–62. https://doi.org/10.1080/0144929X.2019.1669712 (2021).

E.Shelton, B., Yang, J., Hung, J.-L. & Du, X. Two-stage predictive modeling for identifying at-risk students. In Innovative Technologies and Learning, Icitl 2018 , vol. 11003 of Lecture Notes in Computer Science , 578–583, https://doi.org/10.1007/978-3-319-99737-7_61 (Springer, 2018).

Lagus, J., Longi, K., Klami, A. & Hellas, A. Transfer-learning methods in programming course outcome prediction. Acm Trans. Comput. Educ. https://doi.org/10.1145/3152714 (2018).

Marquez-Vera, C. et al. Early dropout prediction using data mining: A case study with high school students. Expert. Syst. 33 , 107–124. https://doi.org/10.1111/exsy.12135 (2016).

Marbouti, F., Diefes-Dux, H. & Madhavan, K. Models for early prediction of at-risk students in a course using standards-based grading. Comput. Educ. 103 , 1–15. https://doi.org/10.1016/j.compedu.2016.09.005 (2016).

Zhao, L. et al. Academic performance prediction based on multisource, multifeature behavioral data. IEEE Access 9 , 5453–5465. https://doi.org/10.1109/access.2020.3002791 (2021).

Kumar, K. & Vivekanandan, V. Advancing learning through smart learning analytics: A review of case studies. Asian Assoc. Open Universities J. (2018).

Yao, Z. A review of the student engagement theory. J. Shunde Polytechnic 16 , 44–52 (2018).

Ma, Z., Su, S. & Zhang, T. Research on the e-learning behavior model based on the theory of learning engagement–taking the course of ”the design and implementation of network teaching platform” as an example. Modern Educational Technology 27 , 74–80 (2017).

F.Agudo-Peregrina, A., Iglesias–Pradas, S., Conde-González, M. A. & Hernández-Garcáa, A. Can we predict success from log data in vles? classification of interactions for learning analytics and their relation with performance in vle-supported f2f and online learning. Computers in human behavior 31 , 542–550, https://doi.org/10.1016/j.chb.2013.05.031 (2014).

Gomez-Aguilar, D. A., Hernandez-Garcia, A., Garcia-Penalvo, J. & Heron, R. Tap into visual analysis of customization of grouping of activities in elearning. Comput. Hum. Behav. 47 , 60–67. https://doi.org/10.1016/j.chb.2014.11.001 (2015).

Kumar, V. S., Pinnell, C. & Paulmani, G. Analytics in Authentic Learning 75–89 (Springer, Berlin, 2018).

Guo, F. & Liu, Q. A study on the correlation between online learning behavior and learning effect–based on the teaching practice of the flipped classroom of blackboard. Higher Educ. Sci. https://doi.org/10.1007/978-981-10-5930-8_6 (2018).

Liang, D., Jia, J., Wu, X., Miao, J. & Wang, A. Analysis of learners’ behaviors and learning outcomes in a massive open online course. Knowl. Manag. E-Learn. Int. J. 6 , 281–298 (2014).

Comer, K. & Clark, C. Peer-to-peer writing in introductory-level moocs. Writing to learn and learning to write across the disciplines. Int. Rev. Res. Open Dist. Learn. 15 , 26–82 (2014).

Kokoç, M. & Altun, A. Effects of learner interaction with learning dashboards on academic performance in an e-learning environment. Behav. Inf. Technol. 40 , 161–175. https://doi.org/10.1080/0144929X.2019.1680731 (2021).

Binbin, Z., Lin, C. H. & Kwon, J. B. The impact of learner-, instructor-, and course-level factors on online learning. Comput. Educ. https://doi.org/10.1016/j.compedu.2020.103851 (2020).

Qureshi, M. A., Khaskheli, A., Qureshi, J. A., Raza, S. A. & Yousufi, S. Q. Factors affecting students’ learning performance through collaborative learning and engagement. Interact. Learn. Environ. https://doi.org/10.1080/10494820.2021.1884886 (2021).

Shen, X., Liu, M., Wu, J. & Dong, X. Towards a model for evaluating students’ online learning behaviors and learning performance. Dist. Educ. China. https://doi.org/10.13541/j.cnki.chinade.2020.10.001 (2020).

Akram, A. et al. Predicting students’ academic procrastination in blended learning course using homework submission data. IEEE Access 7 , 102487–102498. https://doi.org/10.1109/access.2019.2930867 (2019).

Chaity, et al. Feature representations using the reflected rectified linear unit(rrelu) activation. Big Data Mining Anal. 3 , 20–38 (2020).

Madichetty, Sreenivasulu & Sridevi, M. Comparative study of statistical features to detect the target event during disaster. Big Data Mining Anal. 3 , 39–48. https://doi.org/10.26599/BDMA.2019.9020021 (2020).

Saha, S., Ghosh, M., Ghosh, S., Sen, S. & Sarkar, R. Feature selection for facial emotion recognition using cosine similarity-based harmony search algorithm. Appl. Sci. 10 , 2816. https://doi.org/10.3390/app10082816 (2020).

Article   CAS   Google Scholar  

Zigeng, W., Xiao, S. & Rajasekaran R. Novel and efficient randomized algorithms for feature selection. Big Data Mining Anal. 3 , 56–72. https://doi.org/10.26599/BDMA.2020.9020005 (2020).

Chen, L. & Xia, M. A context-aware recommendation approach based on feature selection. Appl. Intell. https://doi.org/10.1007/s10489-020-01835-9 (2020).

Huang, H., Lin, J., Wu, L., Fang, B. & Sun, F. Machine learning-based multi-modal information perception for soft robotic hands. Tsinghua Science and Technology 25 , 255–269, (2019).

Qinchen, Cao & W., Zhang, Y. & Zhu J.,. Deep learning-based classification of the polar emotions of moe-style cartoon pictures. Tsinghua Sci. Technol. 26 , 275–286 (2021).

Muhammad, M., Liu, Y., Sun, M. & Luan, H. Enriching the transfer learning with pre-trained lexicon embedding for low-resource neural machine translation. Tsinghua Sci. Technol. 26 , 2 (2020).

Vieira, C., Parsons, P. & Byrd, V. Visual learning analytics of educational data: A systematic literature review and research agenda. Comput. Educ. 122 , 119–135. https://doi.org/10.1016/j.compedu.2018.03.018 (2018).

Jiang, S., E.Williams, A., Schenke, K., Warschauer, M. & K.O’Dowd, D. Predicting mooc performance with week 1 behavior. In Proceedings of the 7th International Conference on Educational Data Mining, EDM 2014, London, UK, July 4-7, 2014 , 273–275 (International Educational Data Mining Society (IEDMS), 2014).

Aziz, A. A., Ahmad, F. I. & Hassan, H. A framework for studentsa academic performance analysis using naa ve bayes classifier. Jurnal Teknologi 75 , 2 (2015).

Ahuja, R. & Kankane, Y. Predicting the probability of student’s degree completion by using different data mining techniques. 2017 Fourth International Conference on Image Information Processing 474–477, https://doi.org/10.1109/ICIIP.2017.8313763 (2017).

Asif, R., Merceron, A., Ali, S. A. & Haider, N. G. Analyzing undergraduate students’ performance using educational data mining. Comput. Educ. 113 , 177–194. https://doi.org/10.1016/j.compedu.2017.05.007 (2017).

Shen, H., Ju, S. & Sun, J. Performance prediction based on fuzzy clustering and support vector regression. J. East China Normal Univ. 2 , 66–73 (2019).

Moore, M. G. Three types of interaction. Am. J. Dist. Educ. 3 , 1–6. https://doi.org/10.1080/08923648909526659 (1989).

Hillman, D. C., Willis, D. J. & Gunawardena, C. N. Learner-interface interaction in distance education: An extension of contemporary models and strategies for practitioners. Am. J. Dist. Educ. 8 , 30–42. https://doi.org/10.1080/08923649409526853 (1994).

Hirumi, A. A framework for analyzing, designing, and sequencing planned elearning interactions. Quart. Rev. Dist. Educ. 3 , 141–60 (2002).

Peng, W., Yang, Z. & Huang, K. Analysis of online learning behavior and research on its model. China Educ. Technol. 2 , 31–35 (2006).

Malikowski, S. R., Thompson, M. E. & Theis, J. G. A model for research into course management systems: Bridging technology and learning theory. J. Educ. Comput. Res. 36 , 149–73. https://doi.org/10.2190/1002-1t50-27g2-h3v7 (2007).

Veletsianos, G., Collier, A. & Schneider, E. Digging deeper into learners’ experiences in moocs: Participation in social networks outside of moocs, notetaking and contexts surrounding content consumption. Br. J. Educ. Technol. 46 , 570–587. https://doi.org/10.1111/bjet.12297 (2015).

Wu, L., Lao, C., Liu, Q. & Cheng, Y. Online learning behavior analysis model and its application in network learning space. Mod. Educ. Technol. 28 , 46–53. https://doi.org/10.3969/j.issn.1009-8097.2018.06.007 (2018).

Wu, F. & Tian, H. Mining meaningful features of learning behavior: Research on prediction framework of learning outcomes. Open Educ. Res. 25 , 75–82. https://doi.org/10.13966/j.cnki.kfjyyj.2019.06.008 (2019).

Gayman, C. M., Hammonds, F. & Rost, K. A. Interteaching in an asynchronous online class. Scholarsh. Teach. Learn. Psychol. 4 , 231. https://doi.org/10.1037/stl0000126 (2018).

Kuzilek, J., Hlosta, M. & Zdrahal, Z. Open university learning analytics dataset. Sci. Data 4 , 2. https://doi.org/10.1038/sdata.2017.171 (2017).

Wong, T. & Yeh, P. Reliable accuracy estimates from k-fold cross validation. IEEE Trans. Knowl. Data Eng. 32 , 1586–1594. https://doi.org/10.1109/TKDE.2019.2912815 (2019).

Download references

This work was funded by the National Natural Science Foundation of China grant numbers 71872131 and 61977058, in part by of Science and Technology Program of Zhejiang Province (2018C01080)and the STU Scientific Research Initiation Grant (SRIG) under Grant 20007.

Author information

Authors and affiliations.

College of Education, Zhejiang University of Technology, Hangzhou, 310023, China

Feiyue Qiu, Xin Sheng, Lei Jiang, Lijia Zhu & Qifeng Xiang

College of Computer Science and Technology, Zhejiang University of Technology, Hangzhou, 310023, China

Guodao Zhang

Department of Educational Information Technology, East China Normal University, Shanghai, 200062, China

Business School and Research Institute for Guangdong-Taiwan Business Cooperation, Shantou University, Shantou, 515000, China

Ping-kuo Chen

You can also search for this author in PubMed   Google Scholar

Contributions

Conceptualization,G.Z. and P.C.; methodology,X.S.;software,X.S.; validation,L.Z. and Q.X.; formal analysis,X.S. and G.Z.; investigation,G.Z.; resources,G.Z. and X.S.; data curation,L.J.; writing original draft preparation, X.S. and G.Z.; writing review and editing, X.S. and G.Z.; visualization,L.Z. and Q.X.; supervision,G.Z.; project administration,G.Z.;funding acquisition,F.Q. and B.J. All authors have read and agreed to the published version of the manuscript.

Corresponding authors

Correspondence to Xin Sheng or Ping-kuo Chen .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Qiu, F., Zhang, G., Sheng, X. et al. Predicting students’ performance in e-learning using learning process and behaviour data. Sci Rep 12 , 453 (2022). https://doi.org/10.1038/s41598-021-03867-8

Download citation

Received : 05 August 2021

Accepted : 09 December 2021

Published : 10 January 2022

DOI : https://doi.org/10.1038/s41598-021-03867-8

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

A novel methodology using rnn + lstm + ml for predicting student’s academic performance.

• Ashima Kukkar
• Rajni Mohana
• Anand Nayyar

Education and Information Technologies (2024)

Recent advances in Predictive Learning Analytics: A decade systematic review (2012–2022)

• Nabila Sghir
• Amina Adadi
• Mohammed Lahmer

Education and Information Technologies (2023)

Hybrid analysis of the learner’s online behavior based on learning style

• Rihab Balti
• Aroua Hedhili
• Mourad Abed

The role of demographic and academic features in a student performance prediction

• Muhammad Bilal
• Muhammad Omar
• Gyu Sang Choi

Scientific Reports (2022)

By submitting a comment you agree to abide by our Terms and Community Guidelines . If you find something abusive or that does not comply with our terms or guidelines please flag it as inappropriate.

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Systematic research of e-learning platforms for solving challenges faced by Indian engineering students

Asian Association of Open Universities Journal

ISSN : 2414-6994

Article publication date: 2 December 2020

Issue publication date: 21 May 2021

• Supplementary Material

As educational institutes began to address the challenges posed by COVID-19, e-learning came to the foreground as the best bet left. This study is in quest of revealing engineering student's perceptions of the available e-learning platforms, thus surfacing the underlying bottlenecks. Further, it aims at providing solutions that would help enhance the e-learning experience not only in pandemic times but also in the long run.

Design/methodology/approach

This holistic research begins with a comprehensive comparative study about the available e-learning platforms, followed by a primary data analysis through an online survey of 364 engineering students from various colleges and branches. The collected data was analyzed to detect bottlenecks in online learning and suggestions are given for solving some challenges.

On a five-point Likert scale, the available e-learning platforms garnered ratings ranging from 2.81 to 3.46. Google meet was the most preferred platform. However, with a net promoter score (NPS) of 30.36, Microsoft Teams emerged as the most satisfying platform. Technical shortcomings clubbed with psychological and biological factors were found to be taking a toll on e-learning.

Research limitations/implications

This innovative research is based on the perceptions of engineering students hailing majorly from Indian cities, and hence, it may be having educational stream bias and geographical bias. The research could be further extended to cover rural areas and global trends in e-learning.

Originality/value

The research offers a thorough analysis of e-learning platforms, as seen through the lens of engineering students. Furthermore, the analysis does not constrain itself to the technicalities and thus proves to be an all-encompassing one, potent enough to surface critical issues marring the e-learning experience.

• Online teaching
• E-learning platform
• Engineering education
• Open and Distance education

Thakker, S.V. , Parab, J. and Kaisare, S. (2021), "Systematic research of e-learning platforms for solving challenges faced by Indian engineering students", Asian Association of Open Universities Journal , Vol. 16 No. 1, pp. 1-19. https://doi.org/10.1108/AAOUJ-09-2020-0078

Emerald Publishing Limited

Copyright © 2020, Shivangi Viral Thakker, Jayesh Parab and Shubhankar Kaisare

Published in the Asian Association of Open Universities Journal . Published by Emerald Publishing Limited. This article is published under the Creative Commons Attribution (CC BY 4.0) licence. Anyone may reproduce, distribute, translate and create derivative works of this article (for both commercial and non-commercial purposes), subject to full attribution to the original publication and authors. The full terms of this licence may be seen at http://creativecommons.org/licences/by/4.0/legalcode

1. Introduction

Acting as an interface between science and society, engineering surpasses the boundaries of knowledge, analysis and practices the sublime art of organizing forces of technological change. An effective transfer of engineering education stands on the pillars of remembering, understanding, applying, analyzing, evaluating and creating ( Barak, 2013 ).

With the advent of neoliberal market policies, the private sector in engineering and technical education has shifted the focus from philanthropy to profit thereby resulting in a poor quality of education and a mere 25% employability rate ( Choudhury, 2019 ; Gambhir et al. , 2016 ). To add to the existing troubles, COVID-19 has caused education to blow up in the air like an unprecedented display of fireworks. Education systems have no choice but to accept the digital checkmate imposed, ranging from major shutdowns of classroom teaching to spontaneous virtualization ( Xiao, 2018 ). Lack of access to remote learning tools and materials has pushed an alarming number of students not just out of colleges but also out of the system ( Azorín, 2020 ).

COVID-19 has left the post-pandemic education system with two possibilities: a return to traditional education or a transformation towards enhanced education. The key to transformational change will be for systems to focus on their professional capital and find ways to develop teachers' knowledge and skills, support effective collaborative networks that include parents ( McPhee and Söderström, 2012 ). Including educators in the decision- making and communication process ensures easy transformation ( Hollweck and Doucet, 2020 ).

This paper identifies the current perception of available e-learning platforms among engineering students. A comprehensive internal study followed by a thorough analysis helps detect the underlying problems. Further, the paper proposes solutions for these problems to ensure that the transition to e-learning is a smooth endeavor.

2. Literature review

There are recent studies on the demands and needs of engineering education and the exact process of distance learning in the Internet environment. The current effect of the COVID-19 pandemic on the education sector globally and countrywide is studied by few researchers recently. There is a dire need to search for online learning tools available currently and their impact on future e-learning and classroom learning aspects ( Hillier, 2018 ). The following sub-sections give a detailed literature review done on Engineering education requirements, the effect of the pandemic on the education system, various e-learning platforms, and a comparison of online survey methods.

2.1 Engineering education

Fuentes-Del-Burgo and Navarro-Astor (2016) explained in brief how the concepts of “episteme,” “techne” and “phronesis” given by Aristotle are associated with engineering education. Though mainly concerned with Spanish building engineers, it almost gives a worldwide perspective on how various educational factors play an important role in building good engineers and various suggestions to improve them. Barak (2013) discusses how the United States has implemented the three principles of K-12 education and how it can be utilized in other countries to have an overall development of engineering students. The difference between technology and engineering; integration of Bloom's Taxonomy and K-12 education; and the importance of cognitive education on the improvement of engineering students have also been explained.

Choudhury (2019) , surveyed 1178 undergraduate engineering students in Delhi to understand student's perceptions of various criteria of teaching methods used, skills acquired by the students, and involvement of students. This does provide a brief overview of the current situation of engineering education in India and how the current teaching methods can be improved. Upadhayay and Vrat (2017) , have analyzed the growth of India's technical education from the system's dynamic perspective followed by a comparison with the Gartner Hype cycle. The concept of the Boomerang effect has been introduced in this paper to compare it with the current movement of Indian technical education along with concerns over the quality of technical education currently in India. Gambhir et al. (2016) echo the same concerns and have developed a methodology to build a quality model for the integration of various factors in a technical institute.

2.2 COVID-19 pandemic's effect on education

UNESCO report (2020) gives some interesting numbers on how much the pandemic has affected the education system globally. The study indicates that 60.9% of the enrolled student population has been affected with over 1 billion student learners affected and 107 nationwide closures (as of July 2020). The study shows that this pandemic has crippled the education system all over the world further emphasizing the need for new measures to be taken to handle this inopportune time.

Carter et al. (2020) have discussed the effect of COVID-19 on classroom education and how e-learning would need to come to the forefront. The concept of self-regulated learning has been introduced along with its components and ways to integrate it with online learning. Hollweck and Doucet (2020) have also discussed the effects of COVID-19 on education, but they have created an interesting analogy with supernova. They have compared COVID-19 with a supernova in terms that after a supernova event everything changes for the better and the status quo are broken which was unraveling before. Similarly, the paper takes this pandemic as an opportunity to disrupt the status quo and build the education system in a much better way.

Further, Fullan (2020) reiterates that evolution could have wonderful things in store for us - but only if we do our part to shape it and thus hints to take this pandemic positively. Izumi et al. (2020) have similarly gone about discussing the issue of COVID-19 on the transition from classroom learning to online learning and the financial strains of the same. They have surveyed to understand the preparedness for such a transition and the available infrastructure. This has given great insight into the current capacity of the world to transform itself from classroom teaching to online learning.

Lall and Singh (2020) have discussed the impact of COVID-19 on India, emphasizing the importance of a smooth transition towards online learning. A survey to gauge the current perception towards online learning, drawbacks of it, and also the preferred mode of learning was done, which gives a great insight into what factors affect the success of online learning. Similar survey-based research has been done by Basilaia and Kvavadze (2020) with great emphasis on the transition to online learning in Georgia. A brief discussion on the social impact of this immediate transition from classroom learning to online learning has been done by Kufi et al. (2020) along with the importance of free online courses and how resource use should be done in schools to tackle this situation.

2.3 E-learning

Harper et al. (2004) explain distance learning, the advancement of the same along with the impact of government involvement on distance learning. The authors embellish the current details with information on the role of participants in the success of distance learning, change in the organizational structure required for the success of distance learning, and the pros and cons of it in long term perspectives. Along similar lines, Au et al. (2018) discuss the success factors for students learning online such as time management, online examination conduction and flexibility. Jones et al. (2014) discuss how the introduction of technology affects the temporal experience of the learner and states the importance of time flexibility which must be introduced in online learning. This, in a direct sense, gives an understanding of how synchronous and asynchronous ways of teaching can affect the learning capacity of a student. Though Fang et al. (2019) discuss the evolution of MOOCs from 2009 to 2018 in language learning through literature study, they also reiterate similar points and their results show that online learning courses and platforms have raised the time and space for learning, which has made it flexible.

Veletsianos and Houlden (2019) have discussed various themes associated with distance learning. These themes have been closely associated with flexibility and further discuss various approaches towards it from a pedagogical, liberal, temporal and cultural point of view for the past 40 years of distance learning. A similar analysis of the flexibility of transactional education has been done by Paul et al. (2015) . Even Naidu (2019) has given a brief overview of how open learning, flexible learning, and e-learning are very dynamic with narrative changing at any given point of time. Its psychological impact on students also has been reviewed along with various advantages and disadvantages of it. A similar yet a very unique study also has been done by Estacio and Raga (2017) , where they have used machine learning models and correlated the quantitative data available from Moodle to the online learning behavior of students, where the grades obtained are used as a determining medium.

Major et al. (2014) explained the various pedagogical approaches which can be used in distance learning like constructivist, problem-based learning, holistic approach, teamwork. They further provided an overview of how the transition to the online setting must be done along with the technological challenges associated with it. Joanna Rabiega-Wiśniewska (2020) conducted a case study on the current perception of e-learning at Maria Grzegorzewska University, Poland. The study does indicate a neutral stance over liking of the immediate change in learning method, but with 91% of students having a stable Internet connection; it's a good sign nonetheless. A brief understanding of the type of scaling system to be used in such surveys has been explained, which shall be imbibed in this paper to enhance and avoid response bias and to evaluate NPS. Marengo and Marengo (2005) have discussed in brief the actual organizational structure and proper education requirements through Kirkpatrick's taxonomy which needs to be imbibed in e-learning. The concept of blended learning also has been introduced in this paper with its pros and cons in economic terms. The study has effectively discussed various costs involved in e-learning along with the benefits gained. These costs have played an important role in deciding parametric questions to be asked to the students for correct evaluation of the current perception of e-learning tools among engineering students.

2.3.1 Comparison of online platforms for e-learning

A comprehensive comparative study becomes crucial to determine the publicly known best available tools as floating a survey on the unpopular tools may hamper the survey outcome significantly. To aid this comparison, identification of parameters to be compared must be identified. Agrawal et al. (2016) discuss in brief how a parametric survey needs to be conducted and the importance of information quality, service quality, system quality in the success of e-learning. Further, James-Gordon et al. (2003) explain the importance of security features required for e-learning to not be a hindrance for people and an understanding of how market demand or the popularity of a learning platform affects its overall success. Wong (2015) mentions the importance of flexibility of the platforms provided for MOOCs and this flexibility will play a big role in these e-learning platforms as well. Keeping these factors in mind, various parameters such as features provided, platforms which the tool supports, typical customers the tool attracts, customer support provision, price of the tool, overall customer perception about the tool, third party integration, the scope of the tool have been devised for comparison.

2.3.2 Comparison of survey methods

Surveys can be conducted in two ways: online and offline. Offline surveys are generally avoided as they have a localized outreach and getting timely responses is a big task. Online surveys break the barriers of distance and have a hassle-free response collection process. Online survey forms have an easy build coupled with cost-effectiveness. There are various online survey platforms available and a proper comparison must be done among them to find out an apt option for the survey.

To accurately garner student perceptions, the online survey tool should be selected with keen consideration. Along with cost-effectiveness, this tool should bring the magical combination of accuracy and customization. To narrow down on the best survey tool, it was necessary to adopt a comprehensive approach that compared these tools based on the parameters like permissible number of questions, permissible number of responses, data export availability and options, number of free surveys allowed, customization and its scale. Table 1 displays the permissible values with the free version of the tool along with the cost to upgrade to the premium version.

3. Methodology

3.1 data collection tool.

A comprehensive study of seven e-learning platforms (Zoom, Google Meet, Microsoft Teams, GoToWebinar, Zoho Meeting, Adobe Connect and GoToMeeting) was performed to gauge the consistency and performance of platforms based on features, security, customer support and third-party integrations. This study acted as predictive analysis to understand what could be the student's standpoint and then understand how much it varies. Further, a comparative analysis was adopted to find the most suitable online survey platform. A survey-based approach was adopted to gauge the perception of engineering students on the available e-learning tools. Through the review done above, Google Forms was finalized as the survey platform.

3.2 Questionnaire for survey

For drawing valuable insights, it is vital to analyze the most critical parameters. A respondent friendly survey was constructed on Google Forms wherein the respondents had to rate the e-learning platforms based on the parameters like video quality, audio quality, privacy/security, multi-device support, user-friendliness of the interface, screen sharing, chat features, host's control and quality of meeting recording. Figure 1 shows the flow of the questionnaire.

The questionnaire ratings were taken on a five-point Likert scale developed by Rensis Likert ( Reichheld, 2003 ) as this type of scale is used in attitude research projects ( Joanna Rabiega-Wiśniewska, 2020 ). An odd-numbered Likert scale was used to avoid emotion bias and to provide an option for indecision, negativity, and positivity ( Croasmunand Ostrom, 2011 ).

3.3 Distribution channel

A robust distribution channel ensures a greater number of responses from students, spread across various engineering colleges and branches. To achieve the same, the survey form was circulated through platforms like WhatsApp, Gmail, Instagram, LinkedIn and personal calling.

These tools and methods helped in collecting responses from students spread across 12 branches and 49 colleges. The responses generated from surveys generally depict a bell curve. In such cases, if the sample size or the number of respondents is very large, the confidence interval narrows down and errors decrease. Error reduction is good, but the confidence interval should not decrease to a point where it starts showing that negligible people have positive responses. Now, with a decrease in sample size, the confidence interval increases but the error also increases. Thus, selecting the number of respondents is a double-edged sword as a perfect balance has to be struck among confidence interval and error. Hence, an optimal range of 350–400 responses was chosen and the survey form was closed on receiving 364 responses.

4. Data analysis

4.1 respondents profiles.

A total of 364 responses were collected from 49 colleges across India. It was ensured that all the respondents have extensively used the platforms voted by them for at least a month. This data needs to be sorted into various categories to identify trends and gain insights from them. These responses were analyzed branch–wise and year-wise to check whether there is slight response bias, to identify trends, and to draw insights based on the same (see Figures 2 and 3 ).

The Mechanical branch accounted for 44.23% of responses and had the maximum number of responses. Computer Science and Engineering (CSE) branch was second to the Mechanical branch and held 23.07% of responses. Information Technology (IT) branch and Electronics and Telecommunication branch (EXTC) had an almost similar number of respondents and contributed 9.89 and 9.07% of responses respectively. This does indicate that the perception generated was slightly biased towards the requirements of Mechanical Engineering students, but on a closer look at the data, the platforms selected and the ratings given by other branches were on similar lines as the Mechanical branch. Last year students of engineering submitted the maximum responses indicating that maximum awareness, for now, has been limited to certain students only with the further scope for improvement. As first-year students had just been admitted to their respective colleges when the survey was conducted, they were not exposed to the e-learning environment thus resulting in a fewer number of responses from the first year.

4.2 Net promoter score of platforms

The data, collected from the survey responses of 364 students, was analyzed firstly by segregating and making a college wise distribution of responses to check the demographic reach of the survey. A wider demographic reach ensures a varied perspective thereby eliminating regional bias. Branch wise distribution of responses was also plotted to check for singular branch bias for a particular online learning tool. A similar approach was used to check singular year bias by plotting the year-wise distribution of responses. This data was crucial in understanding how the perception is influenced by branch and year of study.

Awareness of platforms was analyzed to check the popularity or reach of each platform irrespective of its liking or disliking. The average ratings of each platform based on the nine parameterized survey outcomes provided insights as to which platform has been consistent in providing all the features satisfactorily to its target audience.

Further, an NPS for each online learning platform was evaluated. NPS is a loyalty index introduced by Frederick F. Reichheld in 2003, primarily used to evaluate how much a product has been liked by the customers and can be used for further product referrals. Promoters are individuals who strongly recommend the product and are convinced of the parameter, thus rating it 4 or 5. Detractors are individuals who are unsatisfied with the product or some parameter of it, thus rating it 1 or 2. Individuals, who give a rating of 3, lie between these two categories and are called passives. NPS for a particular platform, on a 5-point Likert scale, is evaluated as: NPS = ( Number   of    promoters − Number   of   detractors )  *  100 Number   of    respondents   who    have   used   that   platform

(−100 to 0): Needs improvement

(0–30): Good

(30–70): Very good

(70–100): Excellent

The above ranges helped to boil down the overall user sentiment into a single quantifiable value and classify the platform on the same. Finally, the preference percentage was plotted for each platform to understand the current perception and to recognize which platform currently is ruling the roost in the online learning world among engineering students.

5. Results and interpretations

5.1 internal study outcomes.

The internal study focused on performing a comparative analysis of the available e-learning platforms. By comparing these platforms based on the offered features, integrations, reviews, and pricing, the study aimed at finding a platform that provided a complete package to its users at a reasonable subscription cost. Table 2 provides an overview of the internal study outcome.

From Table 2 , it is evident that Zoom and Microsoft Teams are the best platforms with 44 and 67 features available respectively. A closer introspection does reveal a shortcoming of Microsoft Teams over Zoom that is the absence of an attendance management system. In terms of security aspects, Google Meet, GoToMeeting, and GoToWebinar do not have access control and an activity dashboard thereby making these platforms weak. The only salvation for Google Meet is that it has a better API. A bird's eye view indicates that Google Meets supports all the platforms available to people, whereas Zoom and Microsoft Teams do not support the Windows phone app. Microsoft Teams does not attract freelancers and does not provide customer support over the phone. Other platforms satisfactorily provide this, thereby leaving Microsoft Teams with a massive scope of improvement in this aspect. Google Meet is the best in this aspect followed closely by Zoom.

As visible from both Table 3 , Microsoft Teams is the most feasible platform whereas GoToWebinar is the highest priced platform. Zoom and Google Meet are also priced affordably but Microsoft Teams wins the battle in pricing. Table 4 shows that the rankings of all the platforms are not too bad, all crossing 4 stars, but the number of reviews given for Zoom and Google Meet shows that they are the most popular platforms among the others. Zoho Meeting though not as popular, has been highly ranked by those who have used it. Zoom and Google Meet are closely followed by Microsoft Teams which ranks third in popularity. The pricing of GoToWebinar and Adobe Connect does surely reflect their lack of popularity amongst general people. Table 4 does show that Zoom is the platform with the highest number of Third-Party Integrations amounting to whopping 170 integrations. It is closely followed by Microsoft Teams with 154 Third-Party integrations. Other platforms need improvements in this aspect with 86 integrations from GoToWebinar and then an equally shocking drop to 15 integrations from Google Meet. This does show that Third-Party Integrations are surely a challenge for these platforms, Zoom and Microsoft Teams being the only exceptions.

These results show that Zoom has the best balance among features, overview, pricing, popularity, third-party integrations as compared to other platforms. Though just by score value, Microsoft Teams should have followed as the next best; the graph shows high inconsistencies in these parameters. This indicates that due notice over certain parameters has not been given in Microsoft Teams. This makes Google Meet slightly more favorable over Microsoft Teams. Adobe Connect and Zoho Meeting do not make a case to prove their chance in the education sector with even GoToWebinar becoming a rare case of use due to its high price (see Figures 4–6 ).

5.2 Awareness of platforms

Zoom and Google Meets are the most publicly known platforms with an astounding awareness percentage of 86 and 81.6% respectively. Adobe connect and Zoho Meeting is the least known ones and the perception matches the internal study where the higher pricing and fewer features value seen in the graph of these tools had made them possibly least known ones. Thus, there are increased chances of Google Meets and Zoom ruling the roost in the online education industry as these are the platforms mostly used (see Table 5 ).

5.3 Comparison of platforms based on survey results

These ratings indicate that Microsoft Teams is the best platform followed by Google Meets, Zoom, GoToWebinar, GoToMeeting, Zoho Meeting and Adobe Connect. Microsoft Teams had a maximum rating of 3.46 closely followed by Google Meets with a rating of 3.45. No platform had an average rating beyond 4. These passive ratings indicate an even greater perspective over the audience being a low tolerant one with a keen eye towards perfection. Considering this it can be predicted that the NPS would not be very high and in a rare case, it would breach the barrier of 30 (see Table 6 ).

5.4 Net promoter score

Microsoft Teams has the maximum NPS of 30.36, overcoming the Good band and entering the Very Good band. Adobe Connect has the worst NPS of −50 indicating that it needs improvement. Also, the NPS of almost all platforms is closely ranged showing that the competition is very stiff (see Figures 7 and 8 ).

5.5 Preference of platforms

A majority of the audience has given preference to Google Meets followed by Zoom and Microsoft Teams respectively. Here, though the quality of Microsoft Teams is much higher than that of both Zoom and Google Meets, lack of awareness of Microsoft Teams has resulted in it being ruled out of favor. There is also a curious case of Zoom, but a closer introspection shows that concerns over privacy and security of the platform have caused it to be not as favorable as Google Meet, though being superior in other features. Zoho Meetings and Adobe Connect have not made it through as far as students' preference is concerned.

6. Challenges and solutions

The survey respondents highlighted several shortcomings which were barring them from having an effective e-learning experience. Along with these shortcomings, the respondents expressed their desire for certain additional features which would greatly boost the e-learning experience.

6.1 Security concerns

A high number of students are attending digital classrooms and it has become easier for cybercriminals to hijack meetings. Events of video hijacking by uninvited parties to disrupt the usual proceedings have been on the rise since the global quarantine began. Spreading hateful comments, racist and obscene content on these platforms has given rise to a new kind of Internet trolling. Further, unwarranted logins to the enterprise cloud architecture have resulted in immense data breaches.

To greatly reduce such malpractices, the responsibility lies on the shoulders of the platform, the host, and the attendees. Platforms have been striving to enhance their security measures and have also created robust privacy policies. Hosts should secure meetings with a passcode and use private distribution channels to invite participants. Also, disabling features like join before host and participant screen sharing would provide a greater immunity against hijacking. Attendees should refrain from sharing the meeting details on public platforms and avoid clicking on any malicious links.

6.2 Online engagement concerns

6.2.1 proctor mode.

After spending huge amounts on these e-learning platforms, educational institutions do not prefer using separate applications designed specifically for proctoring. This leaves them with two broad options which are to either conduct examinations without proctoring or to use the same e-learning platform for proctoring. The former invites a large number of unfair practices and thus is unjust for diligent students ( Nguyen, 2015 ). The latter requires all the participants to switch on their video which consumes a great amount of bandwidth resulting in lags. Even if incoming videos are disabled, the bandwidth problem persists with the host which leads to difficulties in proctoring ( Gillett-Swan, 2017 ; Dhawan, 2020 ). Thus, the introduction of a specialized proctor mode on these platforms is a desire of many students.

6.2.2 Lecture mode

Survey respondents reported that mischiefs by certain students (e.g.: disturbing annotations on the screen, muting the instructor, etc.) disrupt the flow of the lectures. Though the platforms have provided certain host-specific features, the spontaneous virtualization of education resulted in the instructors getting insufficient time to adapt themselves to these features. This issue has also been highlighted by Moradimokhles and Hwang (2020) as a limitation of online learning. As a result, a majority of them are not aware of or are unable to use all the features they have at their disposal. Even before the pandemic hit, this adaptability was an issue that was highlighted by Parkes et al. , (2014) . As a result, a majority of them are not aware of or are unable to use all the features they have at their disposal. Bringing all these features under a single button of lecture mode would thus help in conducting lectures smoothly without mischiefs.

6.3 Introduction of new features

In addition to the existing features, the respondents expressed the need for certain features. A large number of instructors annotate the content to provide a lucid explanation. However, the students can download the file without any annotations. An option to download it with annotations would ensure a quicker grasping of the concept when students revisit that concept. An inbuilt notepad that can be opened along with the lecture content, in a split-screen mode, would make the notes taking process hassle-free. Live polling would facilitate the instructor in a variety of ways. Similarly, live quizzes with leader-boards would not only add an element of fun to the learning but would also be an indicator of how much the students have learned ( Huang et al. (2019a , b) , Seaborn and Fels (2015) ). The platforms should further be compatible with augmented reality and virtual reality as these would greatly increase the level of understanding ( Bower, 2017 ; El Kabtane et al. , 2019 ; Uhomoibhi et al. , 2019 ). The presence of a virtual user guide along with a chat box would help in resolving the basic issues faced by a great number of users. The ability to rewind live lectures, like YouTube Live, would help students who have missed out on certain important parts of a lecture.

7. Conclusion

Almost all the platforms are sufficient for learning for the time being but have shortcomings that need to be improved to adapt to this fast-changing education sector. There is a large amount of concern over the video and audio quality of all the platforms and the students feel that the platforms are not updated as per current requirements. As per this research study, Google Meet is the best platform among students followed by Zoom and Microsoft Teams respectively, even though NPS indicates Microsoft Teams is the best. If Microsoft teams can improve its social presence, it can prove to be a strong competitor for both Zoom and Google Meet.

The available online learning tools are not the best means to study a holistic curriculum of theory and practical combined. These tools will have to be more adaptable, more technically friendly for the audience to achieve high effectiveness. Along with online learning, methods to improve motivation to study through these means need to be developed ( Roberts et al. , 2018 ). Available e-learning tools serve the basic purpose but integrations of these platforms with other platforms must be improved to give a wider, more enriching experience. Keeping these points in mind, it would not be wrong to conclude that currently, online learning is the best bet left to counter this unprecedented situation in India, but infrastructure development for such platforms needs to be enhanced to consider this method of learning completely fruitful.

8. Limitations and future scope

This research is based on the perceptions of engineering students hailing majorly from Indian cities and is thus subject to educational stream bias and geographical bias. Curbing the educational stream bias by incorporating respondents from other streams could help in understanding the shortcomings on a broader level. Expanding the respondent base by breaking the geographical barriers would help in further understanding the overall access to technology and its implications on the e-learning experience. Furthermore, the overall experience, perceptions, and awareness of students about these platforms are subject to the instructor's ICT proficiency along with the availability and compatibility with the existing infrastructure in the institutions. Considering the unequal penetration of technology across the various socioeconomic classes, an equal amount of focus should be laid on bridging these gaps ( Zhao, 2016 ). As the COVID-19 pandemic is heralding the end of a largely obsolete educational system, developing solutions on a global level while keeping in mind the issues on local levels would bolster the possibility of redesigning a better education system on the bedrocks of equity, excellence and student well-being.

Questionnaire flow chart

Respondents' profiles

Overall aspect comparison of platforms (Individual platform level comparison is given in Appendix )

Platform awareness among respondents

Average survey ratings (For individual platform level comparison, see Appendix )

Platform-based average survey ratings

Comparison of platforms based on NPS

Survey respondents' preference for platforms

Comparison of survey platforms based on features

Overall comparison of platforms

Feature-based comparison of platforms

Overall survey ratings

The appendix file are available online for this article.

Agrawal , V. , Agrawal , A. and Agarwal , S. ( 2016 ), “ Assessment of factors for e-learning: an empirical investigation ”, Industrial and Commercial Training , Vol. 48 No. 8 , pp. 409 - 415 .

Au , O.T.S. , Li , K. and Wong , T.M. ( 2018 ), “ Student persistence in open and distance learning: success factors and challenges ”, Asian Association of Open Universities Journal , Vol. 13 No. 2 , pp. 191 - 202 .

Azorín , C. ( 2020 ), “ Beyond COVID-19 supernova. Is another education coming? ”, Journal of Professional Capital and Community . doi: 10.1108/JPCC-05-2020-0019 .

Barak , M. ( 2013 ), “ Teaching engineering and technology: cognitive, knowledge and problem-solving taxonomies ”, Journal of Engineering, Design and Technology , Vol. 11 No. 3 , pp. 316 - 333 .

Basilaia , G. and Kvavadze , D. ( 2020 ), “ Transition to online education in schools during a SARS-CoV-2 coronavirus (COVID-19) pandemic in Georgia ”, Pedagogical Research , Vol. 5 No. 4 , pp. 1 - 9 .

Bower , M. ( 2017 ), “ Designing for learning using virtual worlds ”, Design of Technology-Enhanced Learning , Emerald Publishing Limited , pp. 305 - 364 .

Carter , R.A. Jr , Rice , M. , Yang , S. and Jackson , H.A. ( 2020 ), “ Self-regulated learning in online learning environments: strategies for remote learning ”, Information and Learning Sciences , Vol. 121 No. 5 , pp. 321 - 329 .

Choudhury , P.K. ( 2019 ), “ Student assessment of the quality of engineering education in India: evidence from a field survey ”, Quality Assurance in Education , Vol. 27 No. 1 , pp. 103 - 126 .

Croasmun , J.T. and Ostrom , L. ( 2011 ), “ Using likert-type scales in the social sciences ”, Journal of Adult Education , Vol. 40 No. 1 , pp. 19 - 22 .

Dhawan , S. ( 2020 ), “ Online learning: a panacea in the time of COVID-19 crisis ”, Journal of Education Technology Systems , Vol. 49 No. 1 , pp. 5 - 22 .

El Kabtane , H. , El Adnani , M. , Sadgal , M. and Mourdi , Y. ( 2019 ), “ Augmented reality-based approach for interactivity in MOOCs ”, International Journal of Web Information Systems , Vol. 15 No. 2 , pp. 134 - 154 .

Estacio , R.R. and Raga , R.C. Jr ( 2017 ), “ Analyzing students online learning behavior in blended courses using Moodle ”, Asian Association of Open Universities Journal , Vol. 12 No. 1 , pp. 52 - 68 .

Fang , J.W. , Hwang , G.J. and Chang , C.Y. ( 2019 ), “ Advancement and the foci of investigation of MOOCs and open online courses for language learning: a review of journal publications from 2009 to 2018 ”, Interactive Learning Environments , pp. 1 - 19 , doi: 10.1080/10494820.2019.1703011 .

Fuentes-Del-Burgo , J. and Navarro-Astor , E. ( 2016 ), “ What is engineering education for? Listening to the voices of some Spanish building engineers ”, Journal of Engineering, Design and Technology , Vol. 14 No. 4 , pp. 897 - 919 .

Fullan , M. ( 2020 ), “ The battle of the century: catastrophe versus evolutionary nirvana ”, Australian Education Leader , Vol. 42 No. 1 , pp. 8 - 10 .

Gambhir , V. , Wadhwa , N.C. and Grover , S. ( 2016 ), “ Quality concerns in technical education in India ”, Quality Assurance in Education , Vol. 24 No. 1 , pp. 2 - 25 .

Gillett-Swan , J. ( 2017 ), “ The challenges of online learning: supporting and engaging the isolated learner ”, Journal of Learning Design , Vol. 10 No. 1 , pp. 20 - 30 .

Harper , K.C. , Chen , K. and Yen , D.C. ( 2004 ), “ Distance learning, virtual classrooms, and teaching pedagogy in the Internet environment ”, Technology in Society , Vol. 26 No. 4 , pp. 585 - 598 .

Hillier , M. ( 2018 ), “ Bridging the digital divide with off-line e-learning ”, Distance Education , Vol. 39 No. 1 , pp. 110 - 12 .

Hollweck , T. and Doucet , A. ( 2020 ), “ Pracademics in the pandemic: pedagogies and professionalism ”, Journal of Professional Capital and Community , pp. 1 - 11 .

Huang , B. , Hew , K.F. and Lo , C.K. ( 2019a ), “ Investigating the effects of gamification-enhanced flipped learning on undergraduate students' behavioral and cognitive engagement ”, Interactive Learning Environments , Vol. 27 No. 8 , pp. 1106 - 1126 .

Huang , B. , Hwang , G.J. , Hew , K.F. and Warning , P. ( 2019b ), “ Effects of gamification on students' online interactive patterns and peer-feedback ”, Distance Education , Vol. 40 No. 3 , pp. 350 - 379 .

Izumi , T. , Sukhwani , V. , Surjan , A. and Shaw , R. ( 2020 ), “ Managing and responding to pandemics in higher educational institutions: initial learning from COVID-19 ”, International Journal of Disaster Resilience in the Built Environment , pp. 1 - 16 .

James-Gordon , Y. , Young , A. and Bal , J. ( 2003 ), “ External environmental forces affecting e-learning providers ”, Marketing Intelligence and Planning , Vol. 21 No. 3 , pp. 168 - 172 .

Joanna Rabiega-Wiśniewska ( 2020 ), “ When students faced online learning in coronavirus times: a case study ”, The International Virtual Conference on Education , Teaching and Learning , pp. 1 - 11 .

Jones , P. , Skinner , H. and Leeds , B. ( 2014 ), “ Temporal experiences of e-learning by distance learners ”, Education + Training , Vol. 56 No. 3 , pp. 179 - 189 .

Kufi , E.F. , Negassa , T. , Melaku , R. and Mergo , R. ( 2020 ), “ Impact of corona pandemic on educational undertakings and possible breakthrough mechanisms ”, BizEcons Quarterly , Vol. 11 No. 1 , pp. 3 - 14 .

Lall , S. and Singh , N. ( 2020 ), “ COVID-19: unmasking the new face of education ”, International Journal of Research in Pharmaceutical Sciences (SPL) , Vol. 11 SPL1 , pp. 48 - 53 .

Major , A.E. , Chaudhury , S.R. , Gilbertson , B.M. and King , D.T. Jr ( 2014 ), “ An integrated science course moves online: four concurrent perspectives ”, Journal of Applied Research in Higher Education , Vol. 6 No. 2 , pp. 248 - 268 .

Marengo , A. and Marengo , V. ( 2005 ), “ Measuring the economic benefits of e-learning: a proposal for a new index for academic environments ”, Journal of Information Technology Education , Vol. 4 No. 1 , pp. 329 - 346 .

McPhee , I. and Söderström , T. ( 2012 ), “ Distance, online and campus higher education: reflections on learning outcomes ”, Campus-Wide Information Systems , Vol. 29 No. 3 , pp. 144 - 155 .

Moradimokhles , H. and Hwang , G.J. ( 2020 ), “ The effect of online vs. blended learning in developing English language skills by nursing student: an experimental study ”, Interactive Learning Environments , Vol. 28 , pp. 1 - 10 .

Naidu , S. ( 2019 ), “ The changing narratives of open, flexible and online learning ”, Distance Education , Vol. 40 No. 2 , pp. 149 - 152 .

Nguyen , T. ( 2015 ), “ The effectiveness of online learning: beyond No significant difference and future horizons ”, MERLOT Journal of Online Learning and Teaching , Vol. 11 No. 2 , pp. 309 - 319 .

Parkes , M. , Stein , S. and Reading , C. ( 2014 ), “ Student preparedness for university e-learning environments ”, The Internet and Higher Education , Vol. 25 , pp. 1 - 10 .

Paul , R.C. , Swart , W. , Zhang , A.M. and MacLeod , K.R. ( 2015 ), “ Revisiting Zhang's scale of transactional distance: refinement and validation using structural equation modeling ”, Distance Education , Vol. 36 No. 3 , pp. 364 - 382 .

Reichheld , F.F. ( 2003 ), “ The one number you need to grow ”, Harvard Business Review , Vol. 81 No. 12 , pp. 46 - 55 .

Roberts , J. , Kigotho , M. and Stagg , A. ( 2018 ), “ Expanding horizons in open and distance learning ”, Distance Education , Vol. 39 No. 1 , pp. 1 - 3 .

Seaborn , K. and Fels , D.I. ( 2015 ), “ Gamification in theory and action: a survey ”, International Journal of Human-Computer Studies , Vol. 74 , pp. 14 - 31 .

Uhomoibhi , J. , Onime , C. and Wang , H. ( 2019 ), “ A study of developments and applications of mixed reality cubicles and their impact on learning ”, International Journal of Information and Learning Technology , Vol. 37 , pp. 15 - 31 .

UNESCO ( 2020 ), COVID-19 Impact on Education , UNESCO , available at: https://en.unesco.org/covid19/educationresponse .

Upadhayay , L. and Vrat , P. ( 2017 ), “ Policy boomerang in technical education: a system dynamics perspective ”, Journal of Advances in Management Research , Vol. 14 No. 2 , pp. 143 - 161 .

Veletsianos , G. and Houlden , S. ( 2019 ), “ An analysis of flexible learning and flexibility over the last 40 years of distance education ”, Distance Education , Vol. 40 No. 4 , pp. 454 - 468 .

Wong , B.T.M. ( 2015 ), “ Pedagogic orientations of MOOC platforms: influence on course delivery ”, Asian Association of Open Universities Journal , Vol. 10 No. 2 , pp. 49 - 66 .

Xiao , J. ( 2018 ), “ On the margins or at the center? Distance education in higher education ”, Distance Education , Vol. 39 No. 2 , pp. 259 - 274 .

Zhao , Y. ( 2016 ), “ From deficiency to strength: shifting the mindset about education inequality ”, Journal of Social Issues , Vol. 72 No. 4 , pp. 716 - 735 .

Acknowledgements

The authors are grateful to all the students of various engineering colleges for responding to the survey and using the online platforms for a month as a prerequisite for the survey.

Corresponding author

Supplementary materials.

AAOUJ-09-2020-0078_suppl2.docx (44 KB) AAOUJ-09-2020-0078_suppl1.docx (42 KB) AAOUJ-09-2020-0078_suppl3.docx (27 KB)

Related articles

We’re listening — tell us what you think, something didn’t work….

Report bugs here

All feedback is valuable

Please share your general feedback

Join us on our journey

Platform update page.

Visit emeraldpublishing.com/platformupdate to discover the latest news and updates

Questions & More Information

Answers to the most commonly asked questions here

e learning Recently Published Documents

Total documents.

• Latest Documents
• Most Cited Documents
• Contributed Authors
• Related Sources
• Related Keywords

Possibilities and Challenges of Online Education in India During the COVID-19 Pandemic

The undergraduate and postgraduate studies of colleges and universities in India have been affected badly amidst the lockdowns for COVID-19 pandemic. The Government has insisted to start the academic activity through online platforms. The biggest concern for the academic institutions now is to select an appropriate e-learning platform. This paper compares different features and facilities available in some widely used online platforms and analyze their suitability from the perspective of socio-economic constraints of students in India. A generic framework for conducting online classes is described that meets the special requirements of the unprivileged students. Some strategic plans to overcome the challenges are identified and suggested. A technical solution for implementation of time-bound assessment module is also proposed.

Determinants of e-learning acceptance amongst Iranian postgraduate students

E-learning can address some of the unmet needs of learners and educational communities; however, not all learners and educators accept e-learning as a delivery modality. This research endeavored to study the factors which affect e-learning acceptance among Iranian post-graduate students using the Davis Technology Acceptance Model (TAM) and to identify the changes which would facilitate their improved acceptance and subsequent wider use of e-learning. This descriptive-correlation study was conducted by surveying 320 Iranian postgraduate students using a self-reporting questionnaire. Structural equation modeling was used for data analysis through LISREL software. Results revealed sufficient validity and reliability of the TAM among Iranian postgraduate medical students. Perceived ease of use, perceived usefulness of e-learning, students’ attitudes toward e-learning, and the intention to use e-learning positively affected e-learning’s acceptance (p < 0.05) among Iranian postgraduate students. According to the results, attitudes toward e-learning have more predictive power than other TAM constructs. Therefore, emphasis on students’ favorable attitudes toward e-learning can be effective in accelerating its acceptance and will progress students’ learning outcomes.

Gamification and e-learning for young learners: A systematic literature review, bibliometric analysis, and future research agenda

An evaluation study of e-learning course at the duhok polytechnic university a case study.

In last decade, there has been a growing awareness in E-Learning which is the most recent type of distance education, and becomes a highly significant model of educational technology. Today e-Learning studies indicate that there is an increasing use of this educational scheme for students, trainers and instructors which improve the learning capability of the students and trainee by depending on student-centered learning (SCL) scheme rather than using the traditional teacher or trainer centered teaching mechanism.However, students’ awareness for the advantage of using e-Learning has not been significantly investigated, especially in Iraq. The aim of this article is to inspect students’ awareness of the using of e-Learning at the Dohuk Polytechnic University. Participants involved in this study were 100 students studying the C++ programming language course during 2015-2016 and 2017-2018 academic years and 210 students studying computer application course during 2018-2019.

An Evaluation Study of an E-Learning Course at the Duhok Polytechnic University

Paradigm shift in education sector amidst covid-19 to improve online engagement.

The study aims to identify social, intellectual and conceptual structures along with key areas, contributors, current dynamics and suggest future research directions in the field of engagement with e-learning systems. An objective analysis of a sample of 358 articles taken from the Web of Science database, supported by subjective assessments based on the research focused on the integration of management into e-learning domain. Citations and PageRank metrics were used to identify the most influential papers along with most influential authors. To understand the intellectual structure of the research area, a co-citation network was developed. The study may help to explore effective ways of delivering education during a crisis, while also taking a sustainable approach to the promotion of education through online methods. By understanding the behavior of learners towards various forms of content delivery, policy makers at national level can develop a framework to implement it nationwide.

Moderating the Neuropsychological Impact of Online Learning on Psychology Students

<p style="text-align: justify;">The purpose of the study was to identify what neuropsychological effect online learning had on psychology students and how it could be moderated. The study was descriptive and combined qualitative and quantitative methods to address the research questions. The study relied on three phases such as baseline study, experiment, and reporting. The experiment utilised neuropsychology tests adopted from the NeurOn platform. It was found that the Psychology students’ perceptions of e-learning and their emotional reaction to them were found not to be appreciative. The practices in breathing exercises, meditation, or yoga were proved to be able to moderate the impact of online learning on the experimental group students’ attentional capacities, memory processes, and cognition abilities. The above findings were supported by the results obtained for the neuropsychology tests and the experimental group students’ self-reflections yielded from the use of the MovisensXS App. The students confirmed that breathing exercises, meditation, or yoga reduced study stress and burnout caused by e-learning and improved their academic performance. The focus group online discussion also showed that integration of breathing exercises, meditation, and yoga helped the experimental group students keep emotional balance, concentrate on their studies easier, remember more information, and meet deadlines in completing assignments. The education scientists are suggested to study how the e-learning curriculum could be reshaped so that it used relaxation practices on regular basis.</p>

Academic Experiences of “Zoom-Fatigue” as a Virtual Streaming Phenomenon During the COVID-19 Pandemic

This phenomenological exploratory multiple-case study design was conducted at an open distance e-learning university and a traditional contact residential university and it was found that the participants viewed video conferencing under the COVID-19 lockdown period as an exhausting experience. A second major finding revealed that the participants were empowered with digital literacy skills to use video conferencing effectively. The current findings add to a growing body of literature on video conferencing with a focus on Zoom fatigue. Further research might explore the lived zoom experiences of administrators, students and a larger group of faculties over a longer period. The study findings must be considered when planning and implementing video conferencing for academics and students in open distance e-learning contexts. This study showed that video conferencing is one tool in the emergence of a digital zoom revolution that has radically changed the workspace. The evidence from this study suggests that zoom fatigue is a reality check for work-related health management.

Academia : Web Platform for E-learning.

Abstract: This paper illustrates how we can improve the existing manual system with the help of E-learning management system. The method aims to build an E-learning web application having better and safer user experience and provides an interactive teaching-learning platform for students and teachers. E-learning Management System is way of solving the educational problems using the modern technologies. It gives an error free, secure, reliable and fast management system. It can assist the user to concentrate on learning rather to concentrate on the record keeping and other stuff. It will help organization in better utilization of resources. Keywords: Web application, Database, backend, frontend, platform, E-learning, Frameworks

Attitudes of primary school teachers towards e-learning in teaching Arabic and the obstacles they face

The study aimed to identify the trends of primary stage laborers towards e-learning in teaching the Arabic language, and the obstacles they face from their point of view. The study relied on the descriptive and analytical approach, and the study population consisted of (170) male and female teachers working in public schools in Jerash governorate. The study sample (75) male and female teachers, who were chosen randomly. The results of the study showed that the attitudes of primary school teachers towards e-learning came at a positive level, with an arithmetic average (3.61), and the obstacles facing primary school teachers in e-learning came with a high degree, with an arithmetic average (3.84). The study recommended giving attention to providing electronic learning tools to include all schools, connecting all schools to the Internet and providing computers in proportion to the number of students in schools, and the study also recommended the necessity of dissolving all obstacles in e-learning, in terms of providing Internet labs in schools in proportion to the number of students and courses Scholastic.

Export Citation Format

Share document.

• Open access
• Published: 07 October 2022

Advances in e-learning in undergraduate clinical medicine: a systematic review

• T. Delungahawatta 1 ,
• S. S. Dunne 1 ,
• S. Hyde 1 ,
• L. Halpenny 1 ,
• D. McGrath 1 , 2 ,
• A. O’Regan 1 &
• C. P. Dunne 1 , 2  

BMC Medical Education volume  22 , Article number:  711 ( 2022 ) Cite this article

4745 Accesses

10 Citations

3 Altmetric

Metrics details

E-learning is recognised as a useful educational tool and is becoming more common in undergraduate medical education. This review aims to examine the scope and impact of e-learning interventions on medical student learning in clinical medicine, in order to aid medical educators when implementing e-learning strategies in programme curricula.

A systematic review compliant with PRISMA guidelines that appraises study design, setting and population, context and type of evaluations. Specific search terms were used to locate articles across nine databases: MEDLINE/PubMed, ScienceDirect, EMBASE, Cochrane Library, ERIC, Academic Search Complete, CINAHL, Scopus and Google Scholar. Only studies evaluating e-learning interventions in undergraduate clinical medical education between January 1990 and August 2021 were selected. Of the 4,829 papers identified by the search, 42 studies met the inclusion criteria.

The 42 studies included varied in scope, cognitive domain, subject matter, design, quality and evaluation. The most popular approaches involved multimedia platforms (33%) and case-based approaches (26%), were interactive (83%), asynchronous (71%) and accessible from home (83%). Twelve studies (29%) evaluated usability, all of which reported positive feedback. Competence in use of technology, high motivation and an open attitude were key characteristics of successful students and preceptors.

Conclusions

Medical education is evolving consistently to accommodate rapid changes in therapies and procedures. In today’s technologically adept world, e-learning is an effective and convenient pedagogical approach for the teaching of undergraduate clinical medicine.

Peer Review reports

E-learning, a pedagogical approach supported by the principles of connectivism learning theory, involves the use of technology and electronic media in knowledge transfer [ 1 , 2 ]. Connectivism views knowledge as a fluid entity circulated through technology enabled networks that foster interactions between individuals, organizations, and societies at large [ 2 ]. Based on this conceptual framework, medical curricula can potentially benefit from enhanced communication and knowledge exchange using technology.

Common e-learning instructional designs in clinical medicine include “online and offline computer-based programmes, massive open online courses, virtual reality environments, virtual patients, mobile learning, digital game-based learning and psychomotor skills trainers”[ 1 ]. To maximize the potential for e-learning, it seems rational that the roles and needs of the e-learner, e-teacher and host institution should be defined and appreciated. According to the Association for Medical Education in Europe (AMEE), an e-learner is any individual taught in an online learning environment [ 1 ]. As the role of the e-learner is central to the learning process, effective e-learning strategies should consider potential learning challenges encountered by the e-learner. Employing skilled e-teachers and providing them with sufficient supports are also important considerations. Furthermore, institutional management of the content versus process elements of educational technology use should best align with the objectives of the program [ 1 ]. For example, if the intent is to provide student access to digital content, then managing sound or video files, podcasts, and online access to research papers, clinical protocols, or reference materials, should be prioritized. On the other hand, if the focus is on student participation in digital activities, then managing processes such as discussion boards and test-taking should take precedence. Accounting for the role of the e-learner, e-teacher, and host institution in this manner, can result in successful implementation of an e-learning system. In fact, e-learning has been shown to be at least as effective as, and can serve as an adjunct to, face-to-face teaching and learning methods [ 3 , 4 , 5 ].

An institution may choose to employ educational technologies for the entirety of the course or provide a combination of online and in-class interactions, with the latter approach referred to as ‘blended learning’ [ 1 ]. Incorporation of e-learning into the curriculum allows for new avenues of interactive knowledge and skill transfer between teachers and students and amongst students. Interactions are not limited to face-to-face conversations but can involve text, audio, images, or video, thereby enriching the learning experience. Giving access to a greater breadth of learning resources further develops lifelong learning skills in students as they are required to independently evaluate and extract the pertinent information [ 1 ]. E-learning interventions can also be accessed at any time from almost any location, which facilitates a student-centred approach through self-directed and flexible learning [ 6 ]. As such, e-learning is an attractive instructional undergraduate health education approach [ 7 ].

To date, e-learning interventions in the sciences, particularly anatomy [ 8 ] and physiology [ 9 ], and postgraduate medical training [ 3 , 4 ] have been described. However, their use has not been reviewed systematically in the specific context of augmenting, enhancing or supporting student learning in undergraduate clinical medicine [ 10 ], or replacing face-to-face learning with online learning in the case of COVID-19 emergency remote teaching. In 2014, survey responses from senior medical students in Illinois, reported use of online collaborative authoring, multimedia, social-networking, and communication tools as point of opportunity study resources during clinical rotations [ 11 ]. Additionally, the COVID-19 pandemic has necessitated stepping away from traditional classroom and bedside teaching, and development of more flexible course delivery. A recent survey by Barton et al. collected 1,626 responses from medical students across 41 medical schools in the United Kingdom during the COVID lockdown. Results of study resources accessed daily showed that 41.6% of students used information provided by university (PowerPoint lecture slides, personal notes), 29.6% accessed free websites and question banks, and 18.4% accessed paid websites and question banks [ 12 ]. The work therefore suggests a strong tendency for students to supplement university materials with online resources [ 12 , 13 ]. The popularity of online learning platforms seems to stem from an association with achieving higher exam scores [ 14 , 15 ], ability to self-monitor knowledge gaps [ 16 ], improved knowledge retention from repeat exposure [ 17 , 18 ], and to practice exam technique [ 16 ].

Medical school educators are, therefore, called to evaluate e-learning approaches and to consider incorporation of suitable strategies into current curricula to ensure equitable access and student success. Thus, we aimed to systematically review the scope and impact of e-learning interventions published regarding undergraduate clinical medicine, and to inform medical educators of the effectiveness and character of various online learning environments.

The Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines are used for the reporting of this systematic review [ 19 ]. The PRISMA checklist is included as Additional File 1 .

Search methods

The early 1990s marked the commercial availability of computer-based learning multimedia [ 20 ] as well as the emergence of online education programs [ 21 ]. Thus, medical subject headings (MeSH), key words and specific database headings were used to locate articles published between January 1990 and August 2021: ‘e-learning’ or ‘digital resources’ or ‘internet learning resources’ AND ‘medical education’ AND ‘undergraduate’ AND ‘techniques’ or ‘programmes’ or ‘interventions’. The search was piloted on PubMed and adapted subsequently for the databases. A total of nine databases were searched: MEDLINE/PubMed, ScienceDirect, EMBASE, Cochrane Library, ERIC, Academic Search Complete, CINAHL, Scopus, Google Scholar and grey literature. The bibliographies of each selected paper were searched manually for further studies. Websites of medical education organisations were searched for position statements and guidelines, including the Association for the Study of Medical Education, AMEE and the British Medical Journal.

Inclusion and exclusion criteria

Only studies in the English language that evaluated an e-learning intervention in subjects related to clinical medicine were selected. These included: family medicine, surgery, internal medicine, radiology, psychiatry, dermatology, paediatrics and obstetrics. Studies that did not involve undergraduate medical students, were based on pre-clinical sciences or were not focussed on an e-learning intervention were excluded. Studies that focussed on the use of internet for assessment and course administration only were not included. Additionally, studies that described interventions but not their evaluation were excluded. Of the 4,829 papers identified by the search, 42 studies were deemed eligible for inclusion in this review.

Data extraction and analysis

AMEE guidelines on e-learning interventions [ 1 ] were used to modify a previous data extraction tool that had been used in a systematic evaluation of effectiveness of medical education interventions [ 22 ]. This was subsequently piloted and refined by three of the authors until consensus was achieved to form the data extraction tool (see Additional File 2 ). With application of connectivism, individual elements of e-learning were identified to infer and appreciate their collective effects on the learning process. More specifically, data was extracted by examining two central questions: how and when to use e-learning in undergraduate clinical medical education. The primary outcomes relating to how to use e-learning were: instructional features that made the e-learning intervention effective; usability features; assessment of effectiveness and quality of the intervention. Primary outcomes relating to when were: the context, and the learner and preceptor characteristics. In addition to the outcomes measured, descriptive data was also extracted to summarise the studies including: the study design, setting and population; context and discipline; type of evaluations. All selected papers were filed in an Endnote library and the data extraction tool for each was stored in an Excel file, a summary of which is provided as Additional File 2 and Additional File 3 .

Guidelines for evaluating papers on medical education interventions from the Education Group for Guidelines on Evaluation were used as a framework to assign a global score for the strength of each paper [ 23 ]. Among these guidelines, significant value is placed on development of strong intervention rationale and intervention evaluation methods [ 23 ]. The impact of the evaluation was also measured using Kirkpatrick’s levels, a recognised system of understanding the effect of interventions [ 24 ]. The first level, reaction, is a measure of learner satisfaction with the intervention [ 24 ]. The second Kirkpatrick level, learning, is a measure of change in knowledge, skills, or experience. The third Kirkpatrick level of behaviour is a measure of behavioural change. The final level, results, is a measure of overall impact on the organization (i.e., improved quality of work, reduction in time wasted, better patient care).

Search results

A total of 4,829 papers were retrieved from database and manual searches, and this number was reduced to 42 after removal of duplicates and application of inclusion/exclusion criteria at set stages (see Fig.  1 for the PRISMA flow diagram). Two papers were retrieved from manual searches of bibliographies [ 25 , 26 ]. The main reasons for excluding studies were a lack of focus on undergraduate medical students (112 studies) or absence of an e-learning intervention (34 studies).

PRISMA flow diagram

Design of included studies

The year of publication ranged from 2003 to 2021, with most conducted within the past ten years (31 studies). Interventions were conducted in nine different countries, mainly the United States (13 studies) and Germany (9 studies). More than half of the studies were conducted in the European Union (21 studies). Several research designs were described, including 17 observational studies [ 25 , 27 , 28 , 29 , 30 , 31 , 32 , 33 , 34 , 35 , 36 , 37 , 38 , 39 , 40 , 41 , 42 ], 13 randomised control trials [ 26 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ], three non-randomised control trials [ 55 , 56 , 57 ], eight qualitative studies [ 58 , 59 , 60 , 61 , 62 , 63 , 64 , 65 ], and one mixed methods study [ 66 ]. Thirteen of the total studies included data collection both pre- and post- intervention [ 25 , 27 , 31 , 34 , 36 , 38 , 39 , 45 , 48 , 52 , 53 , 54 , 61 ]. Six studies had follow-up data (collected weeks to months after intervention) [ 34 , 45 , 49 , 52 , 54 , 56 ] and twelve papers reported ethical approval [ 28 , 29 , 30 , 31 , 33 , 34 , 39 , 40 , 42 , 46 , 49 , 54 ]. Furthermore, eight studies described learning theories in the development or evaluation of medical curricula [ 29 , 30 , 33 , 49 , 51 , 52 , 56 , 58 ]. Of these studies, five referenced constructivism [ 29 , 49 , 51 , 52 , 58 ] three studies highlighted cognitivism [ 30 , 56 , 59 ], and one study evaluated behaviourist learning theory [ 33 ].

Study population

Students in the third year of medical school experiencing clinical exposure were the most commonly studied (sixteen studies), with fourteen studies involving multiple cohorts of students (see Additional File 3 ). Sample sizes ranged from 10 to 42,190 individuals. The most common disciplines investigated were interdisciplinary (13 studies), surgery (8 studies), radiology (7 studies), and dermatology (4 studies) (see Fig.  2 Intervention Discipline).

Intervention discipline

Intervention characteristics

Twelve types of intervention were described and the most commonly used were multimedia platforms (fourteen studies) and case-based learning (eleven studies), as per Additional File 2 and Fig.  3 . In terms of cognitive domain, 27 interventions were in the domain of knowledge [ 25 , 26 , 27 , 29 , 30 , 32 , 33 , 34 , 35 , 39 , 40 , 42 , 43 , 47 , 48 , 50 , 52 , 53 , 54 , 57 , 60 , 61 , 62 , 63 , 64 , 66 , 67 ]; eight were in the domain of skills [ 9 , 30 , 31 , 36 , 37 , 46 , 49 , 51 ] and seven in combined knowledge and skills [ 38 , 41 , 44 , 45 , 56 , 59 , 65 ]. The interventions ranged in duration from a single session to a complete academic year. Thirteen of the interventions were synchronous, where users log on at a given time [ 8 , 26 , 27 , 31 , 33 , 34 , 37 , 43 , 47 , 51 , 52 , 58 , 66 ], and the remaining 29 used an asynchronous platform (users logging on independently in their own time). Seven were accessible in a classroom setting only [ 26 , 27 , 36 , 47 , 52 , 58 , 66 ] while the others could be accessed from home (Fig. 4 ).

Intervention type

Reported roles for e-learning within the curriculum included a revision aid for examinations [ 58 ]; the flipped classroom concept [ 44 , 57 ], whereby lectures held after an e-lecture become an interactive session; to facilitate an online community where knowledge could be discussed/ shared [ 25 ]; and, enabling just-in-time learning through timely access to facts [ 30 , 31 , 37 ]. Seven (17%) of the 42 interventions were didactic in approach [ 27 , 30 , 37 , 55 , 57 , 63 , 65 ], while the others were interactive. Twelve studies described a collaborative approach, whereby students discussed cases and problems with one another and engaged in role-plays [ 25 , 26 , 36 , 38 , 40 , 41 , 42 , 46 , 52 , 59 , 61 , 66 ]. The context of e-learning in relation to the curriculum was not stated in ten of the studies but another thirteen studies used the terms “adjunct”, “complement”, “supplement”,”hybrid” and “blended” to illustrate the common theme of integrating e-learning with traditional learning [ 25 , 29 , 30 , 32 , 44 , 45 , 46 , 47 , 50 , 56 , 57 , 58 , 62 , 63 ]. Seven studies describe temporary replacement of traditional curricula with e-learning platforms in response to COVID-19 [ 33 , 40 , 41 , 42 , 61 , 62 , 64 ]. Eight studies described a pilot phase or the inclusion of students in the development of the intervention [ 33 , 37 , 44 , 45 , 48 , 49 , 53 , 66 ]. Nineteen of the interventions had a built-in assessment, with multiple choice questions being used in most cases, to evaluate whether an improvement in learning had taken place [ 25 , 27 , 31 , 34 , 37 , 39 , 43 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 54 , 55 , 59 , 66 ]. Justification for the chosen assessment strategy or a statement on its suitability was included in two studies [ 50 , 66 ]. Kourdioukova et al. reported an improvement in knowledge and skills with computer supported collaborative case-based approach as judged by in-built multiple-choice questions (MCQ), suggesting the importance of content-specific scripting [ 66 ]. Schneider et al. used a combination of MCQ and survey, and justified their use by demonstrating that learning improved with the intervention compared to the control [ 50 ]. Five of the interventions used end of module assessments as the marker of quality [ 26 , 29 , 53 , 56 , 57 ], with one stating that this was not a suitable mechanism due to its inability to assess the students’ ability to take a patient history or perform a clinical examination [ 53 ].

Intervention evaluation

Each study was given a global rating from 1–5 based on guideline criteria from the Education Group for Guidelines on Evaluation, including whether learning outcomes and curricular context were outlined and the power and rigor of the studies [ 23 ] (Additional File 2 ). Accordingly, eleven studies scored 4/5; two scored 3.5/5; twelve studies scored 3/5; twelve studies scored 2.5/5; and five scored 2/5 (σ = 0.138).

Intervention effectiveness and acceptability

Nine studies described an impact matching a Kirkpatrick level 1, where the student reaction to e-learning intervention was evaluated using student surveys or questionnaires [ 32 , 35 , 44 , 58 , 60 , 61 , 62 , 64 , 65 ]. All these studies report that most students were satisfied with the addition of an e-learning intervention. For instance, Orton et al. note that over 91% of survey responses either ‘strongly agreed’ or ‘agreed’ that use of computer-based virtual patients enabled learning [ 35 ].

Twenty-one (50%) of the 42 studies evaluated acceptability [ 26 , 30 , 32 , 33 , 36 , 37 , 40 , 41 , 42 , 44 , 48 , 53 , 54 , 55 , 56 , 57 , 58 , 63 , 65 , 66 , 68 ]. Of these, 17 reported that the intervention was acceptable. A neutral attitude was reported to a radiology e-learning intervention that involved peer collaboration and was found to be time consuming[ 66 ]. Attitude in another study was much more favourable in junior years than in senior years, with the authors commenting on the conflict between completing assignments and preparing for high stakes examinations [ 55 ]. Another study that focussed on acceptability, with positive outcomes, found that perceived utility and ease of use were the key factors [ 30 ]. Twelve (57%) of the 21 studies further evaluated usability [ 30 , 36 , 37 , 40 , 41 , 42 , 44 , 53 , 56 , 57 , 58 , 65 ], all with positive outcomes, but only one used a formal usability assessment tool [ 58 ]. In that study, Farrimond et al. found that a usable intervention should be: simple and intuitive to use and, from a learner perspective, interactive and enjoyable [ 58 ]. In the development of virtual lectures, ease of navigation, audio-visual quality and accessibility were the key usability features [ 57 ]. Wahlgren et al. concluded that as well as navigation, interactivity is a priority for e-learning development [ 53 ]. Regarding mobile learning, the display should be adaptable to varying screen sizes, termed ‘chunking’, and it should be suitable for a number of platforms [ 30 ].

Twenty-nine (69%) of the 42 studies described an impact matching a Kirkpatrick level 2, where evaluation of whether learning took place was assessed through post intervention scores [ 25 , 27 , 31 , 36 , 38 , 39 , 47 , 48 , 50 , 52 , 53 , 54 , 56 , 57 , 61 ], final exam results [ 26 , 29 , 45 , 66 ], direct observation [ 28 , 31 , 33 , 43 , 46 , 51 , 55 ] and student survey [ 25 , 26 , 30 , 37 , 38 , 39 , 40 , 41 , 42 , 45 , 48 , 49 , 53 , 54 , 56 , 65 , 66 ]. Among these studies, two studies had included both pre- and post- intervention evaluations but neither had a control group nor longer term follow-up [ 25 , 27 ]. One randomised control trial showed a statistically significant improvement in factual knowledge acquisition after participation in an online module as judged based on performance in end of year assessments, compared to a traditional teaching control group (84.8% ± 1.3 vs. 79.5% ± 1.4, p  = 0.006, effect size 0.67) [ 26 ]. Likewise, Davis et al., found that the use of a procedural animation video on mobile device resulted in higher medical student scores on skills checklist (9.33 ± 2.65 vs. 4.52 ± 3.64, p  < 0.001, effect size 1.5) [ 30 ]. Similarly, in Sijstermans et al., mean students’ self-evaluation of their skills using five-point Likert scale questionnaire, before and after two patient stimulations showed improvement (3.91 ± 0.28 vs 3.56 ± 0.34, P  < 0.0001, effect size 1.12). Furthermore, in one study employing a problem-based e-learning approach, the number of first-class honours awarded were found to be significantly improved when compared to control group [ 29 ]. However, in another study using a problem-based e-learning intervention, no significant difference was found between control and intervention groups in subsequent examinations ( p  = 0.11) [ 53 ]. In contrast, Al Zahrani et al. found that delivery of new e-learning platforms (Blackboard Collaborate, ZOOM) in response to COVID-19 was poorly accepted by students, whereby 59.2% did not feel adequately educated on learning outcomes, 30% felt no educational difference between e-learning and traditional curriculums, and 56.1% felt e-learning is insufficient as an educational tool for the health sciences [ 40 ].

Four studies demonstrated a change in student behaviour in line with Kirkpatrick level 3 [ 50 , 52 , 59 , 63 ]. In de Villiers et al., it was found that students were using podcasts to learn course content and the classroom teaching setting to strengthen their understanding, inadvertently accepting the flipped classroom approach [ 63 ]. In Sward et al., students who were assigned to a gaming intervention were more willing to engage in answer creating and answer generating as well as independent study of subject materials prior to session time [ 52 ]. Similarly, in Schneider et al., students in the computer case-based intervention group were found to invest more time into studying course subjects (38.5 min vs 15.9 min) which resulted in significantly higher test scores [ 50 ]. Finally, in Moriates et al., following the integration of value-based modules, students have reported increased awareness of patient needs and discussions with peers regarding value-based decision-making during clerkship [ 59 ].

Learner and preceptor characteristics

Learner characteristics identified to enable successful e-learning include: good digital skills, less resistance to change [ 32 ] and a willingness to collaborate with peers [ 66 ]. Preceptor characteristics were not described in most of the studies, but the role involved guiding students through their learning [ 33 , 46 , 61 , 66 ], selection of topics of broad interest to students [ 60 ], technical support [ 54 ], student evaluation[ 28 , 31 , 37 , 40 , 42 , 45 , 46 , 49 , 51 ], content development and management [ 32 , 41 , 42 , 46 , 54 , 62 ] and providing feedback and clear instruction on what is expected of the learners [ 28 , 37 , 40 , 42 , 51 , 54 , 60 ].

The COVID-19 pandemic resulted in global university closures during periods of lockdown, necessitating educators to quickly adopt alternate pedagogical approaches. As a result, there has been a substantial increase in the use of e-learning, by which teaching and learning activities occur at a distance on online platforms [ 69 ].

In enabling a shift in the control of knowledge acquisition and distribution from the teacher to the student, e-learning facilitates the learning process. Learners filter the available information, develop new perspectives, log into networks to share their understanding, and repeat the cycle [ 2 ]. This view of learning as a fluid and dynamic process is the basis of the learning theory of connectivism and highlights the benefit of this instructional design in medical education – a field amenable to rapid changes in therapies and procedures. In fact, educational theorists have significantly influenced the development of medical curricula throughout history. Amongst the 25 higher impact studies (achieving a global score greater or equal to 3), only 7 studies (28%) were found to have described theoretical underpinnings [ 30 , 33 , 49 , 51 , 52 , 58 , 59 ]. Initially, the behaviourist perspective supported pedological practices [ 70 ]. Behaviourism described learning as largely deriving from responses to external stimuli and led to curricula aimed to influence behaviour through reward and positive and negative reinforcement. In one study reviewed, the lack of direct observation of non-verbal communication by instructors was seen as a significant learning challenge in the virtual environment [ 33 ]. A shift from behaviourism to cognitivism later ensued with the belief that the brain is much more than a ‘black box’ and learning rather involved mental processing and organization of knowledge, and memory functions [ 70 ]. With the recognition of individual differences in the learning process, online systems attempted to introduce interventions that suited multiple learning strategies. For example, learning from auditory narration with animation was found to be more effective than use of text with animation [ 71 ]. This review further highlighted the impact of repetition [ 30 ] and clinical reasoning [ 56 , 59 ] on the learning process. More recently, constructivist learning theory and the perception that learners incorporate new information into pre-existing knowledge schemas has greatly contributed to reformation of medical education [ 70 ]. Incorporating real world connections [ 29 , 49 , 58 ], building on motivations [ 52 ], application of feedback [ 51 ] and continuous reflection [ 49 ] has been noted in this review as important factors in knowledge handling and retention. Presently, e-learning interventions often utilize aspects of more than one theoretical perspective. For instance, problem-based learning interventions have emphasised the critical thinking processes of cognitivism and the self-direction of constructivism [ 29 ]. While primary studies have increased the reporting of underlying theory over time, there is still a significant lack of discussion – future work should reference theoretical principles to objectively frame and assess online education.

In addition to recognizing the needs of the e-learner, identifying required skills of e-teachers and developing content that appropriately supplement the curriculum are vital to ensuring successful implementation of an e-learning system [ 1 ]. Therefore, this study involved review of studies published between 1990 and 2021, assessing the effectiveness and character of various online learning environments in undergraduate clinical medical education. Specifically, these studies involved medical students pursuing medicine as a primary degree and those enrolled with prior degrees.

Intervention design

Critical appraisal of the collected studies using EGGE criteria, identified seventeen studies (40%) meeting a global rating of less than 3. The EGGE criteria encompass a standardized framework by which quality indicators can be recognized. Lower ratings of included studies suggests that conducting and reporting of e-learning interventions is largely lacking in methodological rigour and therefore limits transferability of study results. This finding is consistent with conclusions from a review by Kim et al., describing how most of the existing literature on e-learning interventions have little quantitative data, evaluate a limited range of outcomes and have significant gaps in study designs [ 72 ]. Additionally, only 13 (31%) randomized control trials (RCTs) were included in the review [ 26 , 43 , 44 , 45 , 46 , 47 , 48 , 49 , 50 , 51 , 52 , 53 , 54 ]. Amongst these studies, five reported pre and post test scores [ 45 , 48 , 52 , 53 , 54 ], three of which report long term follow up [ 45 , 52 , 54 ]. Interestingly, all the RCTs report no significant differences in knowledge mastery between control and intervention groups. However, in the immediate short term, e-learning interventions were associated with greater learner satisfaction. For example, in Lee et al., mobile learning with interactive multimedia had higher satisfaction scores compared with conventional Microsoft PowerPoint Show content, despite non-significant differences in knowledge gain [ 48 ]. Similarly, in the study by Wahlgren et al., the majority of students in the intervention group reported that the interactive computerised cases enabled better understanding of disease diagnosis and management, particularly referencing the user-friendliness and feedback [ 53 ]. Yet, knowledge gain as assessed by post-intervention examination scores did not show statistically significant differences between the two groups. Systematic reviews examining the effect of e-learning on nursing education have also demonstrated no differences between e-learning and traditional teaching modalities but report high satisfaction rates with the former [ 73 , 74 ]. While these studies suggest that e-learning is as effective as traditional educational methods, higher student satisfaction levels are indicative of more effective learning programs [ 75 ]. Therefore, the lack of longitudinal data may limit our ability to accurately evaluate the impact of e-learning technologies.

Many of the studies in this review used virtual patient and case-based pedagogical methods reflecting an educational trend towards more critical thinking [ 76 ]. Thirty-five of the interventions under review used an interactive approach, encouraging a style in which students collaborated and discussed ideas with their peers and tutors, the importance of which has been recognised [ 77 ]. Two studies of mobile learning identified wasted time for students as a concern that could be addressed by allowing immediate access to information that would soon be required [ 30 , 55 ]. This ‘just in time learning’, defined as a “brief educational experience targeting a specific need or clinical question” [ 78 ], can be facilitated through e-learning. Ten of the included studies concluded that an integrated approach works best, whereby educators do not seek to replace traditional methods but rather supplement them. This has previously been described as a ‘blended-learning’ style [ 77 ]. A recent study suggests that students thrive in blended- versus self-directed virtual reality environments due to face-to-face teacher support [ 79 ].

Despite variability in methodological design, several studies of e-learning across domains of education, politics, business, and military training have shown knowledge gains assessed by pre- versus post-intervention tests [ 80 ]. Similarly, subjects within the studies we have reviewed have reported e-learning interventions to be conducive to learning [ 32 , 35 , 36 , 44 , 58 , 60 , 61 , 62 , 64 , 65 ], have demonstrated improvements in learning [ 25 , 26 , 27 , 29 , 30 , 31 , 34 , 36 , 37 , 38 , 39 , 43 , 46 , 48 , 49 , 54 , 55 , 56 , 57 , 66 ] and modified learning strategies [ 50 , 52 , 63 ]. The specific features of e-learning strategies most likely to enhance the learning experience may include: peer-to-peer learning [ 52 ], making use of wasted time [ 30 , 40 , 41 , 42 , 81 ], feedback from clinicians and ongoing technical support [ 32 , 82 ], consolidation of information and skill through repetition [ 52 , 82 , 83 ], and convenience of online content access [ 25 , 30 , 40 , 41 , 42 ]. Usability of the intervention has specifically featured strongly in this review. Vital features of e-learning interventions facilitating its use may include: interactive software, active learning promotion (built-in quizzes following cases), asynchronous use, multimedia platforms (i.e., slideshows, videos, images), ease of use and adaptability [ 76 , 81 , 84 ]. Unsurprisingly, students are more engaged with educational material after the typical 9-to-5 work hours [ 25 , 35 ]. Whereas traditional learning opportunities may be restricted to these hours, the flexibility of being able to access online resources outside of this timeframe, may better facilitate achievement of learning objectives [ 25 , 35 ]. Additionally, the use of discussion boards [ 78 ] and games [ 77 ] may facilitate active learning and feedback to be sought and received in a timely manner. Furthermore, quality assurance is recognized as a critical factor, and if considered at the planning stage of an intervention and built into e-learning interventions, may lead to more favourable outcomes [ 23 ]. Engagement with students in this manner is in keeping with the AMEE recommended goals of e-learning [ 1 ]. Several studies also highlight how online learning might provide an encouraging environment for the development of knowledge and skills, relatively easily tailored to individual learning preferences and prior knowledge, and with the possibility of compensating for a lack of accessibility of patients or teachers [ 35 , 36 , 38 , 63 , 85 ]. Furthermore, the ability to access an extensive network of additional resources may allow students to take control of their learning and regulate the volume of information studied [ 36 ].

While our review found improved learning outcomes, other systematic reviews assessing the effectiveness of technology and electronic media in health education, report equivocal findings [ 77 , 86 ]. Proposed factors that may limit learning capacity include: hesitancy to adopt changes by students and teachers, poor technical or financial support, limited technological skills, and the lack of direct and personalized teacher communication [ 25 , 32 , 82 , 87 ]. For example, Davies et al. suggests that an open outlook on mobile device usage was required by students and clinicians, to limit non-use and acquire potential benefits [ 30 ]. In another study conducted by Alsoufi et al., online medical education programs implemented in Libya in response to COVID-19 were found to be negatively received by respondents [ 87 ]. Financial and technical barriers and the lack of hands-on bedside teaching were stated by respondents as limitations to acceptance of e-learning. The shift to online medical learning in the Philippines during the COVID-19 pandemic also identified lack of access to computers and the internet as a significant barrier [ 82 ]. Of course, with these later interventions, the rapid onset of the pandemic required development of e-learning platforms with relatively little training and preparation. As such, the logistics of e-learning curricula as it pertains to specific communities may not have been foreseen. Another reason for such discrepancies may be the underlying discipline in which the intervention is being evaluated [ 47 ]. For instance, the use of only e-learning materials when teaching new skills may not be sufficient, as the direct observation and guidance of an expert is valuable [ 88 ]. A blended-learning environment may be more appropriate in these circumstances [ 47 ]. Indeed, viewing e-learning as a complement rather than replacement of traditional approaches is already well accepted amongst students [ 80 ].

Learner, preceptor and institution characteristics

The twenty-first century learners are known to be avid consumers of various digital platforms. However, studies have shown an incongruence between their ability to use technology for entertainment and ability to use it for educational purposes [ 89 ]. Most students require guidance to synthesize information and create new understanding. In fact, students in middle school through undergraduate level studies have consistently demonstrated poor digital research skills [ 90 , 91 ]. Furthermore, students may require adjustment of learning practices to best engage with the presented e-learning platform. For example, use of PowerPoint presentations or handouts in replacement of in-class teaching can cause visual and auditory learners to require more time to comprehend the information [ 82 ]. Therefore, in addition to carrying an acceptant attitude and a willingness to collaborate with peers, the ability to engage with and extract relevant content from online resources, is a characteristic linked to success in e-learning [ 32 , 66 ].

Nevertheless, recognition of the need for continued mentoring and support in the online learning environment, requires appreciation of the role of the e-teacher. Preceptors’ roles involve development and delivery of the intervention and acting as a resource person for the duration of the module [ 68 ]. In our previous discussion of e-learning strategy effectiveness, two further roles of the e-teacher can be recognized. Firstly, the e-teacher is instrumental in providing timely feedback, one of the main features associated with improved e-learning outcomes [ 32 ]. E-teachers should actively monitor student activity and provide feedback or support where needed [ 92 ]. Secondly, success of e-learning is also strongly related to the motivation of the students and indirectly the motivation demonstrated by the e-teacher [ 30 , 92 ]. The ARCS motivational model highlights four components needed to create a highly motivational e-learning system: maintain student attention, content relevance, student confidence, student satisfaction [ 93 ]. If e-teachers can convey subject material through strategies which encompass use of interactive multimedia, humour, and inquiry for instance, they can satisfy the first component of attention [ 92 ]. Generating activities that best illustrate main ideas, tailoring to the learner knowledge level and providing positive feedback are examples of methods to instil content relevance, student confidence and student satisfaction, accordingly. In Gradl-Dietsch et al., combination of video-based learning, team-based learning and peer-teaching, along with practical skills teaching in point of care ultrasound, feedback from peer teachers, and positive instructor-learner interactions, collectively fulfil the components of the ARCS model [ 54 ]. In Sox et al., the use of a web-based module to teach oral case presentation skills satisfied student attention and content relevance [ 51 ]. However, poor adherence to module largely due to time constraints, can be suggestive of poor student satisfaction. As a result, student confidence and the quality of oral case presentations did not differ from controls (faculty-led feedback sessions). As suggested by the authors, a combination of web module with direct faculty feedback may better instil student confidence and satisfaction with module content, and thereby improve student performance [ 51 ]. Recent studies have shown that the digital literacy skills of most instructors are inadequate [ 90 , 91 ]. Therefore, institutions need to invest into the provision of training programs and supports to allow e-teachers to develop and strengthen competencies needed to sufficiently handle educational technologies [ 92 , 94 , 95 ]. For example, the use of offline tablet-based materials was shown to improve medical education in Zambia, but reported usage amongst healthcare workers was low [ 95 ]. Authors suggest that a lack of training in tablet use was the underlying reason. Taken together, while the role of the teacher has changed compared to traditional pedological approaches, their actions can still heavily influence student learning outcomes.

Limitations and future directions

In a field where technology is changing faster than studies can be completed and interventions are evolving rapidly, medical education research has become a challenging topic of debate. Research can “provide the evidence to prove—and improve—the quality and effectiveness of teaching” and therefore advise the restructuring of curricula to respond to advances in science and technology [ 96 ]. In this review, 29 studies received a global score of 3 or less out of 5, highlighting a lack of transparency and rigour in most of the studies. This justifies a need for a standardised approach for reporting medical education interventions. Pre- and post-intervention testing is informative, but follow-up months later would be an important measure of knowledge retention and therefore intervention effectiveness. Moreover, most of the studies in this review examined knowledge or skill development but few examined higher Kirkpatrick levels. The inclination towards focus on the lower levels of the Kirkpatrick model may stem from difficulty following students in the field to evaluate long-term results of the educational intervention on student behaviours (level three) and the organization at large (level four) [ 97 ]. Future work on the evaluation of associated changes in behaviour, professional practice or patient outcomes would be valuable. Other e-learning characteristics that can be evaluated in future work (Fig. 4 ) may include the capacity for adaptivity (to accommodate changing student needs and performance) and collaboration [ 98 ]. Including descriptions of curricula context can also facilitate the exploration of which e-learning strategies are best suited for specific medicine disciplines and socioeconomic settings. The use of internet resources by both students and patients alike, and the exponential growth in social media influence may also provide a platform for future e-learning interventions [ 99 ].

Future intervention design recommendations

Over the past twenty years and with the recent advent of the COVID-19 pandemic, there has been a substantial increase in the use of e-learning. This review found that e-learning interventions are positively perceived by students and associated with improvements in learning. Improved learning outcomes are closely correlated with interactive, asynchronous, easily accessible and usable interventions, and those involving students and preceptors with digital skills, high motivation and receptive attitudes. While further exploration of the strengths and weaknesses of e-learning technologies is warranted, use of online platforms is a creditable educational tool for undergraduate clinical medicine.

Abbreviations

Association for Medical Education in Europe

Preferred Reporting Items for Systematic Reviews and Meta-Analyses

Ellaway R, Masters K. AMEE Guide 32: e-Learning In Medical Education - Part 1: Learning. Teaching And Assessment Med Teach. 2008;30(5):455–73.

Google Scholar  

Goldie JG. Connectivism: A Knowledge Learning Theory For The Digital Age? Med Teach. 2016;38(10):1064–9.

Article   Google Scholar  

Maertens H, Madani A, Landry T, Vermassen F, Van Herzeele I, Aggarwal R. Systematic Review Of E-Learning For Surgical Training. Br J Surg. 2016;103(11):1428–37.

Tarpada SP, Morris MT, Burton DA. E-Learning In Orthopedic Surgery Training: A Systematic Review. J Orthop. 2016;13(4):425–30.

Feng J-Y, Chang Y-T, Chang H-Y, Erdley WS, Lin C-H, Chang Y-J. Systematic Review of Effectiveness of Situated E-Learning on Medical and Nursing Education. Worldviews Evid Based Nurs. 2013;10(3):174–83.

Price Kerfoot B, Masser BA, Hafler JP. Influence Of New Educational Technology On Problem-Based Learning At Harvard Medical School. Med Educ. 2005;39(4):380–7.

Guarino S, Leopardi E, Sorrenti S, De Antoni E, Catania A, Alagaratnam S. Internet-Based Versus Traditional Teaching And Learning Methods. Clin Teach. 2014;11(6):449–53.

Trelease RB. From Chalkboard, Slides, And Paper To E-Learning: How Computing Technologies Have Transformed Anatomical Sciences Education. Anat Sci Educ. 2016;9(6):583–602.

Felder E, Fauler M, Geiler S. Introducing E-Learning/Teaching In A Physiology Course For Medical Students: Acceptance By Students And Subjective Effect On Learning. Adv Physiol Educ. 2013;37(4):337–42.

Martin EA. Concise medical dictionary. 9th ed. Oxford: Oxford University Press; 2015.

Book   Google Scholar  

Han H, Nelson E, Wetter N. Medical students’ online learning technology needs. Clin Teach. 2014;11(1):15–9.

Barton J, Rallis KS, Corrigan AE, Hubbard E, Round A, Portone G, et al. Medical students’ pattern of self-directed learning prior to and during the coronavirus disease 2019 pandemic period and its implications for Free Open Access Meducation within the United Kingdom. J Educ Eval Health Prof. 2021;18:5.

O’Hanlon R, Laynor G. Responding to a new generation of proprietary study resources in medical education. J Med Libr Assoc. 2019;107(2):251–7.

Zhang J, Peterson RF, Ozolins IZ. Student approaches for learning in medicine: What does it tell us about the informal curriculum? BMC Med Educ. 2011;11(1):87.

Cooper AL, Elnicki DM. Resource utilisation patterns of third-year medical students. Clin Teach. 2011;8(1):43–7.

Wynter L, Burgess A, Kalman E, Heron JE, Bleasel J. Medical students: what educational resources are they using? BMC Med Educ. 2019;19(1):36.

Larsen DP, Butler AC, Roediger HL 3rd. Test-enhanced learning in medical education. Med Educ. 2008;42(10):959–66.

Augustin M. How to learn effectively in medical school: test yourself, learn actively, and repeat in intervals. Yale J Biol Med. 2014;87(2):207–12.

Moher D, Liberati A, Tetzlaff J, Altman DG. Reprint—Preferred Reporting Items for Systematic Reviews and Meta-Analyses: The PRISMA Statement. Phys Ther. 2009;89(9):873–80.

Hammond M, editor Learning From Experience: Approaching The Research Of CD-ROM In Schools. WCCE; 1995.

Kentnor HE. Distance Education and the Evolution of Online Learning in the United States. Curric Teach. 2015;17:21–34.

Barry Issenberg S, McGaghie WC, Petrusa ER, Lee Gordon D, Scalese RJ. Features And Uses Of High-Fidelity Medical Simulations That Lead To Effective Learning: A BEME Systematic Review. Med Teach. 2005;27(1):10–28.

Evaluation. EGfGo. Guidelines for Evaluating Papers on Educational Interventions. Education Group for Guidelines on Evaluation ed: BMJ; 1999. p. 1265–7.

Kirkpatrick D, Kirkpatrick J. Transferring Learning to Behavior : Using the Four Levels to Improve Performance. Oakland, United States: Berrett-Koehler Publishers, Incorporated; 2005.

Bernardo V, Ramos MP, Plapler H, De Figueiredo LF, Nader HB, Ancao MS, et al. Web-Based Learning In Undergraduate Medical Education: Development And Assessment Of An Online Course On Experimental Surgery. Int J Med Inform. 2004;73(9–10):731–42.

Raupach T, Munscher C, Pukrop T, Anders S, Harendza S. Significant Increase In Factual Knowledge With Web-Assisted Problem-Based Learning As Part Of An Undergraduate Cardio-Respiratory Curriculum. Adv Health Sci Educ Theory Pract. 2010;15(3):349–56.

Casillas JM, Gremeaux V. Evaluation Of Medical Students’ Expectations For Multimedia Teaching Materials: Illustration By An Original Method Using The Evaluation Of A Web Site On Cardiovascular Rehabilitation. Ann Phys Rehabil Med. 2012;55(1):25–37.

Cevik AA, Shaban S, El Zubeir M, Abu-Zidan FM. The Role Of Emergency Medicine Clerkship E-Portfolio To Monitor The Learning Experience Of Students In Different Settings: A Prospective Cohort Study. Int J Emerg Med. 2018;11(1):24.

Corrigan M, Reardon M, Shields C, Redmond H. “SURGENT” – Student E-Learning For Reality: The Application Of Interactive Visual Images To Problem-Based Learning In Undergraduate Surgery. J Surg Educ. 2008;65(2):120–5.

Davies BS, Rafique J, Vincent TR, Fairclough J, Packer MH, Vincent R, et al. Mobile Medical Education (Momed) - How Mobile Information Resources Contribute To Learning For Undergraduate Clinical Students - A Mixed Methods Study. BMC Med Educ. 2012;12:1.

de Sena DP, Fabricio DD, Lopes MH, da Silva VD. Computer-Assisted Teaching Of Skin Flap Surgery: Validation Of A Mobile Platform Software For Medical Students. PLoS ONE. 2013;8(7):e65833.

Howlett D, Vincent T, Gainsborough N, Fairclough J, Taylor N, Cohen J, et al. Integration Of A Case-Based Online Module Into An Undergraduate Curriculum: What Is Involved And Is It Effective? E-Learn. 2009;6(4):372–84.

Khalil R, Mansour AE, Fadda WA, Almisnid K, Aldamegh M, Al-Nafeesah A, et al. The Sudden Transition To Synchronized Online Learning During The COVID-19 Pandemic In Saudi Arabia: A Qualitative Study Exploring Medical Students’ Perspectives. BMC Med Educ. 2020;20(1):285.

Ogura A, Hayashi N, Negishi T, Watanabe H. Effectiveness of an e-Learning Platform for Image Interpretation Education of Medical Staff and Students. J Digit Imaging. 2018;31(5):622–7.

Orton E, Mulhausen P. E-Learning Virtual Patients For Geriatric Education. Gerontol Geriatr Educ. 2008;28(3):73–88.

Sijstermans R, Jaspers MW, Bloemendaal PM, Schoonderwaldt EM. Training Inter-Physician Communication Using The Dynamic Patient Simulator. Int J Med Inform. 2007;76(5–6):336–43.

Tews M, Brennan K, Begaz T, Treat R. Medical Student Case Presentation Performance And Perception When Using Mobile Learning Technology In The Emergency Department. Med Educ Online. 2011;16. https://doi.org/10.3402/meo.v16i0.7327 .

Wunschel M, Leichtle U, Wulker N, Kluba T. Using A Web-Based Orthopaedic Clinic In The Curricular Teaching Of A German University Hospital: Analysis Of Learning Effect, Student Usage And Reception. Int J Med Inform. 2010;79(10):716–21.

Zayed MA, Lilo EA, Lee JT. Impact of an Interactive Vascular Surgery Web-Based Educational Curriculum on Surgical Trainee Knowledge and Interest. J Surg Educ. 2017;74(2):251–7.

Al Zahrani EM, Al Naam YA, AlRabeeah SM, Aldossary DN, Al-Jamea LH, Woodman A, et al. E- Learning experience of the medical profession’s college students during COVID-19 pandemic in Saudi Arabia. BMC Med Educ. 2021;21(1):443.

Dost S, Hossain A, Shehab M, Abdelwahed A, Al-Nusair L. Perceptions of medical students towards online teaching during the COVID-19 pandemic: a national cross-sectional survey of 2721 UK medical students. BMJ Open. 2020;10(11): e042378.

Coffey CS, MacDonald BV, Shahrvini B, Baxter SL, Lander L. Student Perspectives on Remote Medical Education in Clinical Core Clerkships During the COVID-19 Pandemic. Medical Science Educator. 2020;30(4):1577–84.

Diekhoff T, Kainberger F, Oleaga L, Dewey M, Zimmermann E. Effectiveness Of The Clinical Decision Support Tool ESR Eguide For Teaching Medical Students The Appropriate Selection Of Imaging Tests: Randomized Cross-Over Evaluation. Eur Radiol. 2020;30(10):5684–9.

Dombrowski T, Wrobel C, Dazert S, Volkenstein S. Flipped Classroom Frameworks Improve Efficacy In Undergraduate Practical Courses – A Quasi-Randomized Pilot Study In Otorhinolaryngology. BMC Med Educ. 2018;18(1):294.

Hari R, Kälin K, Harris M, Walter R, Serra A. Comparing Blended Learning With Faculty-Led Ultrasound Training: Protocol For A Randomised Controlled Trial (The SIGNATURE Trial). Praxis (Bern 1994). 2020;109(8):636–40.

Herrmann-Werner A, Weber H, Loda T, Keifenheim KE, Erschens R, Mölbert SC, et al. “But Dr Google Said…” - Training Medical Students How To Communicate With E-Patients. Med Teach. 2019;41(12):1434–40.

Jenkins S, Goel R, Morrell DS. Computer-Assisted Instruction Versus Traditional Lecture For Medical Student Teaching Of Dermatology Morphology: A Randomized Control Trial. J Am Acad Dermatol. 2008;59(2):255–9.

Lee LA, Chao YP, Huang CG, Fang JT, Wang SL, Chuang CK, et al. Cognitive Style and Mobile E-Learning in Emergent Otorhinolaryngology-Head and Neck Surgery Disorders for Millennial Undergraduate Medical Students: Randomized Controlled Trial. J Med Internet Res. 2018;20(2): e56.

Plackett R, Kassianos AP, Kambouri M, Kay N, Mylan S, Hopwood J, et al. Online Patient Simulation Training To Improve Clinical Reasoning: A Feasibility Randomised Controlled Trial. BMC Med Educ. 2020;20(1):245.

Schneider AT, Albers P, Muller-Mattheis V. E-Learning in Urology: Implementation of the Learning and Teaching Platform CASUS(R) - Do Virtual Patients Lead to Improved Learning Outcomes? A Randomized Study among Students. Urol Int. 2015;94(4):412–8.

Sox CM, Tenney-Soeiro R, Lewin LO, Ronan J, Brown M, King M, et al. Efficacy of a Web-Based Oral Case Presentation Instruction Module: Multicenter Randomized Controlled Trial. Acad Pediatr. 2018;18(5):535–41.

Sward KARNP, Richardson SRNP, Kendrick JMD, Maloney CMDP. Use of a Web-Based Game to Teach Pediatric Content to Medical Students. Acad Pediatr. 2008;8(6):354–9.

Wahlgren CF, Edelbring S, Fors U, Hindbeck H, Stahle M. Evaluation Of An Interactive Case Simulation System In Dermatology And Venereology For Medical Students. BMC Med Educ. 2006;6:40.

Gradl-Dietsch G, Menon AK, Gürsel A, Götzenich A, Hatam N, Aljalloud A, et al. Basic Echocardiography For Undergraduate Students: A Comparison Of Different Peer-Teaching Approaches. Eur J Trauma Emerg Surg. 2018;44(1):143–52.

Davis JS, Garcia GD, Wyckoff MM, Alsafran S, Graygo JM, Withum KF, et al. Use Of Mobile Learning Module Improves Skills In Chest Tube Insertion. J Surg Res. 2012;177(1):21–6.

Roesch A, Gruber H, Hawelka B, Hamm H, Arnold N, Popal H, et al. Computer Assisted Learning In Medicine: A Long-Term Evaluation Of The “Practical Training Programme Dermatology 2000.” Med Inform Internet Med. 2003;28(3):147–59.

Sendra-Portero F, Torales-Chaparro OE, Ruiz-Gomez MJ, Martinez-Morillo M. A Pilot Study To Evaluate The Use Of Virtual Lectures For Undergraduate Radiology Teaching. Eur J Radiol. 2013;82(5):888–93.

Farrimond H, Dornan TL, Cockcroft A, Rhodes LE. Development And Evaluation Of An E-Learning Package For Teaching Skin Examination. Action Research Br J Dermatol. 2006;155(3):592–9.

Moriates C, Valencia V, Stamets S, Joo J, MacClements J, Wilkerson L, et al. Using Interactive Learning Modules to Teach Value-Based Health Care to Health Professions Trainees Across the United States. Acad Med. 2019;94(9):1332–6.

Naeger DM, Straus CM, Phelps A, Courtier J, Webb EM. Student-Created Independent Learning Modules: An Easy High-Value Addition To Radiology Clerkships. Acad Radiol. 2014;21(7):879–87.

Smith E, Boscak A. A Virtual Emergency: Learning Lessons From Remote Medical Student Education During The COVID-19 Pandemic. Emerg Radiol. 2021;28(3):445–52.

Taurines R, Radtke F, Romanos M, König S. Using Real Patients In E-Learning: Case-Based Online Training In Child And Adolescent Psychiatry. GMS J Med Educ. 2020;37(7):Doc96-Doc.

De Villiers M, Walsh S. How Podcasts Influence Medical Students’ Learning – A Descriptive Qualitative Study. Afr J Health Prof Educ. 2015;7(1):130–3.

Wagner-Menghin M, Szenes V, Scharitzer M, Pokieser P. Designing Virtual Patient Based Self-Study Quizzes Covering Learning Goals In Clinical Diagnostic Sciences For Undergraduate Medical Students - The Radiology Example. GMS J Med. 2020;37(7):Doc91.

Nelson TM. Preparing for Practice: Strengthening Third-Year Medical Students’ Awareness of Point-of-Care Resources. Med Ref Serv Q. 2018;37(3):312–8.

Kourdioukova EV, Verstraete KL, Valcke M. The Quality And Impact Of Computer Supported Collaborative Learning (CSCL) In Radiology Case-Based Learning. Eur J Radiol. 2011;78(3):353–62.

Clark RC, Mayer RE. E-Learning and the Science of Instruction: Proven Guidelines for Consumers and Designers of Multimedia Learning. Hoboken: Center for Creative Leadership; 2016.

Gruner D, Pottie K, Archibald D, Allison J, Sabourin V, Belcaid I, et al. Introducing global health into the undergraduate medical school curriculum using an e-learning program: a mixed method pilot study. BMC Med Educ. 2015;15(1):142.

UNESCO. National education responses to COVID-19: summary report of UNESCO's online survey. United Nations Educational, Scientific and Cultural Organization.

Mann KV. Theoretical Perspectives In Medical Education: Past Experience And Future Possibilities. Med Ed. 2011;45(1):60–8.

Moreno R, Mayer RE. Cognitive Principles Of Multimedia Learning: The Role Of Modality And Contiguity. J Educ Psychol. 1999;91(2):358–68.

Kim S. The Future of e-Learning in Medical Education: Current Trend and Future Opportunity. J Educ Eval Health Prof. 2006;3:3.

Du S, Liu Z, Liu S, Yin H, Xu G, Zhang H, et al. Web-based distance learning for nurse education: a systematic review. Int Nurs Rev. 2013;60(2):167–77.

Lahti M, Hätönen H, Välimäki M. Impact of e-learning on nurses’ and student nurses knowledge, skills, and satisfaction: A systematic review and meta-analysis. Int J Nurs Stud. 2014;51(1):136–49.

Kirkpatrick DL. The Four Levels of Evaluation. In: Brown SM, Seidner CJ, editors. Evaluating Corporate Training: Models and Issues. Dordrecht: Springer, Netherlands; 1998. p. 95–112.

Chapter   Google Scholar  

Fuks A, Boudreau JD, Cassell EJ. Teaching Clinical Thinking To First-Year Medical Students. Med Teach. 2009;31(2):105–11.

Chumley-Jones HS, Dobbie A, Alford CL. Web-Based Learning: Sound Educational Method Or Hype? A Review Of The Evaluation Literature. Acad Med. 2002;77(10 Suppl):S86-93.

Kahn CE Jr, Ehlers KC, Wood BP. Radiologists’ Preferences for Just-in-Time Learning. J Digit Imaging. 2006;19(3):202–6.

Fairén M, Moyés J, Insa E. VR4Health: Personalized teaching and learning anatomy using VR. J Med Syst. 2020;44(5):94.

Ruiz JG, Mintzer MJ, Leipzig RM. The impact of E-learning in medical education. Acad Med. 2006;81(3):207–12.

Klímová B. Mobile Learning in Medical Education. J Med Syst. 2018;42(10):194.

Baticulon RE, Sy JJ, Alberto NRI, Baron MBC, Mabulay REC, Rizada LGT, et al. Barriers to Online Learning in the Time of COVID-19: A National Survey of Medical Students in the Philippines. Medical Science Educator. 2021;31(2):615–26.

Mehrpour SR, Aghamirsalim M, Motamedi SM, Ardeshir Larijani F, Sorbi R. A supplemental video teaching tool enhances splinting skills. Clin Orthop Relat Res. 2013;471(2):649–54.

Larvin M. E-Learning In Surgical Education And Training. ANZ J Surg. 2009;79(3):133–7.

Lane J, Slavin S. Simulation in Medical Education: A Review. Simulation & Gaming - Simulat Gaming. 2001;32.

Bell DS, Fonarow GC, Hays RD, Mangione CM. elf-study from web-based and printed guideline materials. A randomized, controlled trial among resident physicians. Annals of internal medicine. 2000;132(12):938–46.

Alsoufi A, Alsuyihili A, Msherghi A, Elhadi A, Atiyah H, Ashini A, et al. Impact of the COVID-19 pandemic on medical education: Medical students’ knowledge, attitudes, and practices regarding electronic learning. PLoS ONE. 2020;15(11):e0242905.

Rogers DA, Regehr G, Yeh KA, Howdieshell TR. Computer-assisted Learning versus a Lecture and Feedback Seminar for Teaching a Basic Surgical Technical Skill. The American journal of surgery. 1998;175(6):508–10.

Guri RS. E-Teaching In Higher Education: An Essential Prerequisite For E-Learning. J New Approaches Educ Res. 2018;7(2):93–7.

Alexander B, Adams-Becker S, Cummins M, Hall-Giesinger C. Digital Literacy in Higher Education, Part II: An NMC Horizon Project Strategic Brief. . Austin, Texas; 2017.

Wineburg S, McGrew S, Breakstone J, Ortega T. Evaluating Information: The Cornerstone of Civic Online Reasoning. Stanford Digital Repository; 2016.

Yengin İ, Karahoca D, Karahoca A, Yücel A. Roles Of Teachers In E-Learning: How To Engage Students & How To Get Free E-Learning And The Future. Procedia Soc. 2010;2(2):5775–87.

Keller J, Suzuki K. Learner Motivation and E-learning Design: A Multinationally Validated Process. Learn Media Technol. 2004;29:229–39.

Howe DL, Heitner KL, Dozier A, Silas S. Health Professions Faculty Experiences Teaching Online During the COVID-19 Pandemic. ABNF J. 2021;32(1):6–11.

Barteit S, Neuhann F, Bärnighausen T, Bowa A, Lüders S, Malunga G, et al. Perspectives of Nonphysician Clinical Students and Medical Lecturers on Tablet-Based Health Care Practice Support for Medical Education in Zambia, Africa: Qualitative Study. JMIR Mhealth Uhealth. 2019;7(1):e12637.

Easton G. Primary care education research-time to raise our game? Educ Prim Care. 2014;25(6):304–7.

Cahapay M. Kirkpatrick Model: Its Limitations As Used in Higher Education Evaluation. IJATE. 2021;8:135–44.

Byrnes KG, Kiely PA, Dunne CP, McDermott KW, Coffey JC. Communication, collaboration and contagion: “Virtualisation” of anatomy during COVID-19. Clin Anat. 2021;34(1):82–9.

Dunne S, Cummins NM, Hannigan A, Shannon, Dunne C, Cullen W. A method for the design and development of medical or health care information websites to optimize search engine results page rankings on Google. Journal of Medical Internet Research. 2013;15(8):e2632.

Download references

Acknowledgements

Not applicable

No funding received.

Author information

Authors and affiliations.

School of Medicine, University of Limerick, Limerick, Ireland

T. Delungahawatta, S. S. Dunne, S. Hyde, L. Halpenny, D. McGrath, A. O’Regan & C. P. Dunne

Centre for Interventions in Infection, Inflammation & Immunity (4I), University of Limerick, Limerick, Ireland

D. McGrath & C. P. Dunne

You can also search for this author in PubMed   Google Scholar

Contributions

CPD, AOR, SH and DM conceived the study and its design. LH and AOR performed initial analysis. TD and SSD drafted and revised the manuscript. All authors co-wrote and approved the final manuscript.

A uthors’ information

Not applicable.

Corresponding author

Correspondence to C. P. Dunne .

Ethics declarations

Ethics approval and consent to participate, availability of data and material.

All data generated or analysed during this study are included in this published article [and its supplementary information files].

Consent for publication

Competing interests.

The authors declare that they have no competing interests.

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Supplementary Information

Additional file 1:.

PRISMA Checklist.

Additional file 2:  

Summary of Review Results.

Additional file 3:  

Intervention Characteristics.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ . The Creative Commons Public Domain Dedication waiver ( http://creativecommons.org/publicdomain/zero/1.0/ ) applies to the data made available in this article, unless otherwise stated in a credit line to the data.

Reprints and permissions

About this article

Cite this article.

Delungahawatta, T., Dunne, S.S., Hyde, S. et al. Advances in e-learning in undergraduate clinical medicine: a systematic review. BMC Med Educ 22 , 711 (2022). https://doi.org/10.1186/s12909-022-03773-1

Download citation

Received : 26 February 2022

Accepted : 26 September 2022

Published : 07 October 2022

DOI : https://doi.org/10.1186/s12909-022-03773-1

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Online learning
• Distance learning
• Medical education
• Medical students
• Clinical medicine
• Systematic review

BMC Medical Education

ISSN: 1472-6920

• Submission enquiries: [email protected]
• General enquiries: [email protected]

ORIGINAL RESEARCH article

This article is part of the research topic.

Integration-Focused Approaches of Educational Systems Across the EU

Navigating the Peer-to-Peer Workflow in Non-Formal Education Through an Innovative E-learning Platform: A Case Study of the KIDS4ALLL Educational Project in Hungary and Italy Provisionally Accepted

• 1 University of Turin, Italy
• 2 TÁRKI Social Research Institute, Hungary

The final, formatted version of the article will be published soon.

The digital revolution is affecting all aspects of life, radically transforming everyday tasks and routines. The ability to cope with new challenges in life, including new forms of learning are key skills in the 21st-century, however, education systems often struggle with tackling digital inequalities. A digital learning platform developed by the KIDS4ALLL educational project, implemented in face-to-face student interactions, aims to mitigate the divide and the resulting social disadvantages among children with and without migration/ethnic minority background. Analysing data collected during the pilot phase of the project in two of the participating countries, Italy and Hungary, this paper examines how students and teaching staff adapt to a newly introduced digital learning tool based on peer-to-peer workflows. Firstly, it examines the role of educators' interpersonal competences in navigating the innovative learning activities and delves into how they use them and how they manage resources. Secondly, the study explores what attitudes and behaviours are observed among students engaged in the proposed peer-led activities, in particular in terms of their ability to cope with uncertainty and complexity. The analytical framework of the paper is based on two cultural dimensions offered by Hofstede (2001), the index of uncertainty avoidance (UAI) and power distance (PDI), and it utilizes the personal, social and learning-to-learn competence of the 8 LLL Key Competences as defined by the European Commission to conceptualize the skills of educators and students. Interpreting data from Italy and Hungary in their respective social and educational contexts, the study finds that the most important features that proved to be effective and useful during the pilot phase were the democratic power-relations between students and educators, the peer-to-peer scheme and its further development to the peer-for-peer approach. The child-friendly and real-life-related new curriculum and its appealing digital learning platform, embedded into a flexible, playful and child-centred pedagogical approach, were also successful. These are all complementing the traditional, formal school environment and pedagogy which, despite all developments in formal education in the past decades, can be characterized as teacher-centred and frontal.

Keywords: peer-to-peer learning, Educational inclusion, LLL Key competences, uncertainty avoidance, Power distance

Received: 10 Jan 2024; Accepted: 08 Apr 2024.

Copyright: © 2024 Schroot, Lőrincz and Bernát. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY) . The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.

* Correspondence: Dr. Tanja Schroot, University of Turin, Turin, Italy

People also looked at

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• Springer Nature - PMC COVID-19 Collection

Gamification of e-learning in higher education: a systematic literature review

Amina khaldi.

Ecole Nationale Supérieure d’Informatique ESI, Ex INI, Algiers, Algeria

Rokia Bouzidi

Fahima nader, associated data.

All data generated or analyzed during this study are included in this published article.

In recent years, university teaching methods have evolved and almost all higher education institutions use e-learning platforms to deliver courses and learning activities. However, these digital learning environments present significant dropout and low completion rates. This is primarily due to the lack of student motivation and engagement. Gamification which can be defined as the application of game design elements in non-game activities has been used to address the issue of learner distraction and stimulate students’ involvement in the course. However, choosing the right combination of game elements remains a challenge for gamification designers and practitioners due to the lack of proven design approaches, and there is no one-size-fits-all approach that works regardless of the gamification context. Therefore, our study focused on providing a comprehensive overview of the current state of gamification in online learning in higher education that can serve as a resource for gamification practitioners when designing gamified systems. In this paper, we aimed to systematically explore the different game elements and gamification theory that have been used in empirical studies; establish different ways in which these game elements have been combined and provide a review of the state-of-the-art of approaches proposed in the literature for gamifying e-learning systems in higher education. A systematic search of databases was conducted to select articles related to gamification in digital higher education for this review, namely, Scopus and Google Scholar databases. We included studies that consider the definition of gamification as the application of game design elements in non-game activities, designed for online higher education. We excluded papers that use the term of gamification to refer to game-based learning, serious games, games, video games, and those that consider face-to-face learning environments. We found that PBL elements (points, badges, and leaderboards), levels, and feedback and are the most commonly used elements for gamifying e-learning systems in higher education. We also observed the increasing use of deeper elements like challenges and storytelling. Furthermore, we noticed that of 39 primary studies, only nine studies were underpinned by motivational theories, and only two other studies used theoretical gamification frameworks proposed in the literature to build their e-learning systems. Finally, our classification of gamification approaches reveals the trend towards customization and personalization in gamification and highlights the lack of studies on content gamification compared to structural gamification.

Introduction

In recent years, most universities use e-learning platforms to deliver courses. Teaching in the form of e-learning is a modern supplement, and sometimes even an alternative to traditional education (Górska, 2016 ). Especially since the last few years, with the spread of the Covid-19 crisis, higher education institutions had to shift from traditional teaching to online teaching as an alternative to resume learners' learning (Sofiadin & Azuddin, 2021 ). However, over time, these digital environments brought several challenges. On one hand, student motivation decreases, resulting in a lack of engagement and participation in courses. On the other hand, instructors struggle to maintain learners’ attention, leading to the eventual abandonment of online education systems. To solve this problem and create engaging e-learning platforms, the gamification technique was proposed.

Game technologies create opportunities for higher education institutions to redesign and innovate their e-learning models to support learning experiences among learners (Alhammad & Moreno, 2018 ). The introduction and growing expansion of gamification in education and learning contexts promotes critical reflection on the development of projects that transform students’ learning experiences (Garone & Nesteriuk, 2019 ). However, is it that simple to create effective gamified e-learning systems especially in the context of higher education?

Early applied work on gamification of educational settings suggested positive-learning, but mixed results have been obtained (Seaborn & Fels, 2015 ). While gamification in general learning systems is known to have a positive impact on student motivation, evidence on its effectiveness in higher education settings is mixed and still uncertain due to the complicated environment in the higher education context. First, the level of difficulty of study is higher at the university than at lower levels of education, and students are more aware of the importance of education they have chosen (Urh et al., 2015 ). Moreover, tertiary education is characterized by the variety of students’ profiles, needs and learning methods; thereby, each game element and even each combination of game elements affects each student differently. Given this diversity of features in the higher education context and the increasing number of inter- and multidisciplinary programs, the process of applying gamification is becoming more complex.

The purpose of this systematic review was to provide a comprehensive overview of the current state of gamification in e-learning in higher education. We focused on identifying how designers currently deal with gamification in the digital higher education context, what game elements they use, how these elements are combined, and what gamification theories are used. In addition, this study sought to find data on existing gamification approaches in the literature, especially those suggested to be applied in digital higher education. Our study differs from previous studies in several ways. In our study, we first wanted to compare our results with previous research’s results that addressed the same research questions in terms of trends in the use of game elements, i.e. whether designers who develop gamified e-learning systems still use classic game elements such as points, badges, and leaderboards, or whether they expand the list of game elements used to include deeper game elements like challenges, storytelling, and so on. We then focused on the underpinning gamification theories used in empirical work, and specifically we sought to understand whether empirical research is beginning to use the various gamification frameworks available in the literature, or whether it is still relying on theories and methods that are highly theoretical and do not provide clear guidance to designers when choosing the right set of game elements (Toda et al., 2020 ). Also, in our study, we sought to find out how game elements are combined in gamified learning systems in higher education. Previous studies have not fully explored this point, with the exception of the study (Dichev & Dicheva, 2017 ). Finally, we proposed a classification of gamification approaches proposed in the context of e-learning in higher education based on several relevant criteria.

The remainder of this manuscript has the following structure. " Related works " section, briefly reviews some of the most relevant review papers. " Systematic literature review methodology " section, systematic literature review methodology, presents the approach we followed in conducting our paper retrieval. " Results of the search " section, results of the research, presents responses to our defined research questions. " Discussion and limitations " section is dedicated for discussion of the results; and finally, we conclude.

Related works

Prior reviews.

This section briefly reviews some of the relevant literature reviews on gamification in higher education related to the topic of our systematic review. The objective is to be able to compare our findings later in the results section to prior reviews’ findings and to shed a more realistic light on any advances in gamification in e-learning in the context of higher education.

Dichev and Dicheva ( 2017 ) critically reviewed the advancement of educational gamification. This review paper was the only one to address the issue of combining game elements in gamified learning systems. The authors found that in all reviewed works, no justification is given for the selection of particular game elements. The study concluded that there is a need for further studies to improve our understanding of how individual game elements are associated with behavioral and motivational outcomes and how they function in an educational context.

Ozdamli ( 2018 ) examined 313 studies on gamification in education. It used content analysis to determine trends in gamification research. The study sought to determine the distribution of empirical research based on a variety of criteria, namely: distribution of studies based on years, number of authors, type of publication, paradigms, research sample, environments, theory/model/strategy, learning area and distribution of game components, mechanics and dynamics. The author found that motivational theories are the most frequently used approach in gamification studies and that the most frequently used game components are goals, rewards and progression sticks.

Khalil et al. ( 2018 ) reviewed the state of the art on gamification in MOOCs (Massive Open Online Course) by answering eight research questions. One of these questions sought to identify elements of gamification that have been implemented or proposed for implementation in MOOCs. The study found that the most commonly used elements in the application of gamification in MOOCs are badges, leaderboards, progress, and challenges. According to the study, progress and challenges are used more frequently in MOOCs than points.

The paper (Alhammad & Moreno, 2018 ) studied gamification in the context of software engineering (SE) education. The study sought to understand how gamification was applied in the SE curriculum and what game elements were used. The study identified four gamification approaches from the primary studies analyzed: papers that implemented gamification by following an existing gamification approach in the literature, papers that adapted psychological and educational theories as gamification approaches, papers that designed and followed their own gamification approach, and finally, papers that did not follow any specific gamification approach. In addition, leaderboards, points and levels were found to be the most frequently used gaming components. Similarly, challenges, feedback, and rewards were the most commonly used mechanics, and progression was the most commonly used dynamic.

Majuri et al. ( 2018 ) reviewed 128 empirical research papers in the literature on gamification in education and learning. It was found that points, challenges, badges and leaderboards are the most commonly used gamification affordances in education which are affordances that refer to achievement and progression while social and immersion-oriented affordances are much less common.

In the paper (Zainuddin et al., 2020 ), the authors addressed a research question related to our research area, namely the underlying theoretical models used in gamification research. It was found that in the studies that implicitly mention their theoretical underpinnings, self-determination theory is the most commonly used, followed by flow theory and goal-setting, while the other studies do not provide any theoretical content.

More recently, van Gaalen et al. ( 2021 ) reviewed 44 research studies in the health professions education literature. The study addressed the question of what game attributes are used in gamified environments, and sought to understand the use of theory throughout the gamification process. The study used Landers ( 2014 )’s framework to categorize the identified game elements into game attributes and revealed that in most reviewed studies the game attributes ‘assessment’ and/or ‘conflict/challenge’ were embedded in the learning environment. Regarding the use of theory in gamification processes, most of the identified studies on gamification in health professions education were not theory-based, or theoretical considerations were not included or not yet developed.

Finally, the authors of the paper (Kalogiannakis et al., 2021 ) performed a systematic literature review on gamification in science education by reviewing 24 empirical research papers. A research question related to our field of study was addressed in this review, namely, what learning theory is used, and what game elements are incorporated into gaming apps. The findings of the studyshowed that most articles did not provide details about the theoretical content or the theory on which they were based. The few articles that used theoretical frameworks were based on self-determination theory SDT, flow theory, goal-setting theory, cognitive theory of multimedia learning and motivation theory. In addition, the study found that the most common game elements and mechanics used in gamified science education environments were competitive setup, leaderboards, points and levels.

Systematic literature review methodology

In this paper of systematic review, we followed a methodology to identify how gamification technique has been used in digital learning environments, specifically in higher education. We sought to identify the game elements that have been used the most, the way they have been combined, and the different frameworks proposed in the literature for gamification of e-learning systems in higher education. A systematic literature review is a means of identifying, evaluating and interpreting all available research relevant to a particular research question, or topic area, or phenomenon of interest (Kitchenham, 2004 ). Kitchenham ( 2004 ) summarizes the stages of a systematic review in three main phases: Planning the Review, Conducting the Review, and Reporting the Review. The first phase ‘Planning the Review’ includes the formulation of research questions, identification of key concepts and constructing the search queries. The second phase ‘Conducting the Review’ consists on study selection based on inclusion and exclusion criteria. Finally, the third phase ‘Reporting the Review’ relates to data extraction and responding to research questions. In the following, we detail the main steps of each phase.

Search strategy

We started by identifying the main goal of this systematic literature review by clearly formulating the following research questions:

Which game elements and gamification theories are used in gamified learning systems?

How these game elements are combined?

Which gamification design approaches are available in the literature?

Then, we constructed a list of key concepts that are: gamification, e-learning and higher education. After that, we identified the alternative terms for each of the key concepts as some authors may refer to the same concept using a different term. For the concept of gamification, we identified this list of free text terms: gamify, game elements, game dynamics, game mechanics, game components, game aesthetics and gameful. For the two other concepts of e-learning and higher education, we identified these terms: education, educational, learning, teaching, course, syllabus, syllabi, curriculum, and curricula.

We formulated two search queries based on the terms identified previously:

• For research questions 1and 2:

(gamif* OR gameful OR “game elements” OR “game mechanics” OR “game dynamics” OR “game components” OR “game aesthetics”) AND (education OR educational OR learning OR teaching OR course OR syllabus OR syllabi OR curriculum OR curricula).

• (2) For research question 3:

(gamif* OR gameful OR “game elements” OR “game mechanics” OR “game dynamics” OR “game components” OR “game aesthetics”) AND (education OR educational OR learning OR teaching OR course OR syllabus OR syllabi OR curriculum OR curricula) AND (framework OR method OR design OR model OR approach OR theory OR strategy).

We conducted our research by searching the databases using the search query formulated previously. We performed our search in the Scopus and Google Scholar databases as the first is one of the most professional indexing databases and the second is the most popular, so it helps to identify further eligible studies. The search was performed in December 2021. Although the Scopus database indexed the publication abstracts, most of the articles were not available through Scopus, and the articles were retrieved from the following publishers:

• SEMANTIC SCHOLAR,
• (Hallifax et al. ) SAGE,
• Science Direct.

The exception was some articles that could not be accessed. We also performed a backward snowballing search to identify further relevant studies by scanning and searching the references of papers marked as potentially relevant (Dichev & Dicheva, 2017 ; Mora et al., 2017 ; Gari & Radermacher, 2018 ; Khalil et al., 2018 ; Ozdamli, 2018 ; Subhash & Cudney, 2018 ; da Silva et al., 2019 ; Hallifax et al., 2019a , 2019b ; Legaki & Hamari, 2020 ; Zainuddin et al., 2020 ; Saleem et al., 2021 ; Swacha, 2021 ; van Gaalen et al., 2021 ) in search of other relevant studies.

Inclusion and exclusion criteria

In the following table, we summarized the inclusion and exclusion criteria that we considered when we screened full text articles (Table ​ (Table1 1 ).

Study selection

To select the relevant studies for this systematic review, a manual screening was performed. First, we reviewed the titles and abstracts of different records that were retrieved. Then, citations were imported to Endnote and duplicate records were removed. After that, we read the full text of all retained articles for inclusion and exclusion based on the eligibility criteria. In case of uncertainty, discussion was organized with the research team to reach consensus about the articles in question.

Data extraction

We developed a data extraction form that was refined and discussed until consensus was obtained. The extraction form was then used by the review author to extract data from all included studies. In this part of this paper, we have considered two types of papers: papers representing case studies to extract the game elements used in the developed e-learning systems, the underpinning theories behind the gamification process and the way game elements were combined with each other. The second type of retrieved papers is about framework proposals, from which we could identify models, approaches, and design processes proposed in the literature for gamifying digital learning environments in tertiary education level.

Results of the search

General results.

In this literature review, we reported the most extensive overview of the empirical research literature on gamification of e-learning in higher education to date. The selection process of relevant studies is shown in Fig.  1 . We analyzed a total of 90 papers to respond to the three research questions formulated previously. First, we retrieved 39 papers in the form of empirical studies carried out at university level and analyzed them to identify what game elements are used, what gamification theories are used to guide the gamification process, and how these game elements are combined. We then identified a variety of 51 papers of type theoretical proposals intended to guide the gamification process. Since higher education is part of general learning systems, we included in this review papers that propose gamification approaches for general contexts and general learning systems. Indeed, we identified 16 papers for general application of gamification, 18 papers for gamifying general learning systems and 17 approaches intended to be applied to e-learning systems in higher education.

Flow diagram of the articles selection process

Answering research questions

In following, we will answer the three research questions formulated at the beginning of this article:

Education applications of gamification refer to using game elements for scholastic development in formal and informal settings (Seaborn & Fels, 2015 ). In our case, we were interested in collecting relevant experimental studies on gamification of e-learning systems in higher education. In the following table (Table 2 ), we list and examine 39 experimental studies that have implemented a digital learning system at the higher education level to answer RQ1 . For each study, we analyzed the game elements that were incorporated and the gamification approaches that were followed during the gamification process. For ease of reference, the game elements that were used in e-learning systems to improve student engagement and the underpinning theories are summarized in Table ​ Table2. 2 . More detailed descriptions of the 39 empirical studies are presented in “Appendix”.

Experimental studies on gamification of e-learning in higher education

By analyzing the game elements listed in Table ​ Table2, 2 , we noticed that PBL elements (points, badges, and leaderboards), levels, and feedback are the most commonly used elements for gamifying e-learning systems in higher education. This is in line with other reviews’ findings, e.g. (Dichev & Dicheva, 2017 ).

Furthermore, in response to what (Dichev & Dicheva, 2017 ) stated about the fact that gamification with “deeper game elements” (Enders, 2013 ) by incorporating game design principles involving game mechanics and dynamics such as challenges, choice, low-risk failure, role-play or narrative is still scarce, we noted in our systematic literature review that recent studies explore new game elements. Indeed, among the 39 studies analyzed in Table ​ Table2, 2 , there are 20 primary studies that used “deeper game elements” (Enders, 2013 ) like challenges and storytelling (narrative). Among these, challenges are the most popular ones.

In Seaborn and Fels ( 2015 ), the authors noted that till 2015, the majority of applied research on gamification was not grounded in theory and did not use gamification frameworks in the design of the system under study. Likewise, in this systematic review, by analyzing the 39 empirical studies listed in Table ​ Table2, 2 , we noticed that most studies were not underpinned by gamification theories. This is in line with the findings of other recent studies, such as van Gaalen et al. ( 2021 ) and Kalogiannakis et al. ( 2021 ). Indeed, of the 39 primary studies analyzed in our systematic review, only nine papers (Smith, 2017 ; Kyewski & Krämer, 2018 ; Pilkington, 2018 ; Tsay et al., 2018 ; van Roy & Zaman, 2019 ; De-Marcos et al., 2020 ; Facey-Shaw et al., 2020 ; Sanchez et al., 2020 ; Dikcius et al., 2021 ) adapted theoretical approaches and used them as gamification approaches. These are a set of social and motivational theories resumed in a variety of six different theories, namely: self-determination theory-SDT, Social comparison theory, social exchange theory-SET, cognitive evaluation theory-CET, situated motivational affordance theory, theory of gamified learning (Landers, 2014 ) and user-centered design (Nicholson, 2012 ). Self-determination theory is considerably the most popular one. These findings are correlated with other reviews’ findings such as Zainuddin et al. ( 2020 ) and Kalogiannakis et al. ( 2021 ). Only two other primary studies Marín et al. ( 2019 ) and Dias ( 2017 ) used existing theoretical gamification frameworks to build their gamified e-learning systems. For the remaining papers, some built their owngamification design based on guidelines from the literature whereas others did not cite any theory. Hence, we notice that this distribution is in line with (Alhammad & Moreno, 2018 )’s review findings regarding the use of four different categories of gamification approaches in primary studies, namely, papers that followed existing gamification frameworks, papers that adapted motivational theories to their needs, papers that built their own approach, and finally, those that didn’t follow any specific approach. We also noticed that motivational theories are the most frequently used approach, as noted in Ozdamli ( 2018 ).

Gamification approaches

For this research question, we sought to identify how game elements are combined in gamified learning systems in higher education. Previous studies have not fully explored this point except the paper (Dichev & Dicheva, 2017 ). By analyzing the different empirical studies involved in this systematic literature review (listed in Table ​ Table2), 2 ), we noticed the lack of detailed information about how instructors and designers combined different game elements. Indeed, in all reviewed papers, the authors listed only the game elements employed to gamify their learning systems. In addition, no study provided any justification of the choice made about the sets of game elements to use, nor the way they combined them in the gamified learning systems.

In the reviewed collection, five studies employed one single game element (Coleman, 2018 ; Garnett & Button, 2018 ; Kyewski & Krämer, 2018 ; Facey-Shaw et al., 2020 ; Dikcius et al., 2021 ), three other studies gamified systems using two game elements (Fajiculay et al., 2017 ; Smith, 2017 ; Donnermann et al., 2021 ), five other studies used three game elements (Hisham & Sulaiman, 2017 ; Kasinathan et al., 2018 ; Romero-Rodriguez et al., 2019 ; Khaleel et al., 2020 ; Sanchez et al., 2020 ) while the remaining ones used more than three elements.

This happens due to the lack of studies that provide clear guidelines and justifications for the combination of game elements (Toda et al., 2020 ).

In this section, we will approach RQ3 . We first synthesize the current literature on gamification approaches in a general context. Then, we present a set of gamification approaches for general learning systems. Finally, we list a set of approaches proposed specifically for higher education within e-learning environments. We briefly described each approach in the table below (Table 3 ).

In the table above, we investigated a total of 51 gamification approaches in three different contexts. The first set of approaches (the first 16 rows of Table ​ Table3) 3 ) was designed for general use, i.e., for all contexts such as learning, health, marketing and entrepreneurship. While the second set of approaches (the next 18 rows of Table ​ Table3) 3 ) targeted general learning contexts, i.e., without any restriction on educational level. Finally, the third set of approaches (the last 17 rows of Table ​ Table3) 3 ) was intended to be applied in a specific context, namely digital higher education.

Given our review’s main interest in e-learning in higher education, we will classify the last 17 approaches of Table ​ Table3, 3 , which correspond to those designed for e-learning systems in higher education, into several classes based on different relevant criteria that we will detail below. The paper (Saggah et al., 2020 ) proposes categorizing gamification design frameworks into three categories: scenario-based, high-level approach, and Gamification elements guidance. Inspired by this categorization, we propose our categorization, which will be used to classify the different gamification approaches in e-learning in higher education. A description of each category is given in what follows, and our classification results are shown in Table ​ Table4 4 .

• High-level approach This group categorizes papers that provide an overview of the design process that serves as a general high-level guideline containing the global phases without detailing which game elements to use and how to implement them.
• Gamification elements guidance This group categorizes papers that provide a conceptualization of the gamification elements that can be used in educational environments. These studies can include implementation guidance.
• Scenario based This group categorizes papers that provide a descriptive outline of the design process. In other words, these papers propose gamification approaches by describing their application through real empirical studies experimented in real learning environments.
• Type from student perspective (adaptive gamification/one size fits all gamification) Adaptive gamification considers that users have different motivations, so it consists of personalizing learning experiences according to each learner profile. Whereas ‘one size fits all’ gamification uses the same gamified system (gamification elements, rules, etc.) for all learners. For ease of use, we will use ‘A’ character for adaptive approaches and x for ‘one size fits all’ ones.
• Profundity from pedagogical perspective (structural gamification versus content gamification) structural gamification refers to the application of game design elements to motivate the learner through an instructional content without changing it (Garone & Nesteriuk, 2019 ). It can be made by using clear goals, rewards for achievements, progression system and status, challenge and feedback (Garone & Nesteriuk, 2019 ). Content gamification is the application of elements, mechanics and game thinking to make the content more game-like (Garone & Nesteriuk, 2019 ). It is a one-time structure created only for a specific content or learning objectives and hence cannot be reused for any content (Sanal, 2019 ). Garone and Nesteriuk ( 2019 ) states that elements that can be used in content gamification are story and narrative; challenge, curiosity and exploration; characters and avatars; interactivity, feedback and freedom to fail (Kapp, 2014 ). According to Kapp ( 2014 ), the combination of both structural and content gamification, is the most effective way to build high engaging and motivating environments. For ease of use, we will use ‘C’ character for content approaches and x for structural ones.
• Validation This group categorizes papers that provided a validation of the proposed approach through empirical evidence showing its application to e-learning systems in higher education.

Classification of gamification approaches (context of e-learning in higher education)

Table ​ Table4 4 represents the results of our classification of gamification approaches in the context of e-learning in higher education. Regarding the level of detail, we noticed that most of the analyzed approaches (with a number of 9 out of a total of 17) are of the type of gamification elements guidance (Urh et al., 2015 ; Huang & Hew, 2018 ; Alsubhi & Sahari, 2020 ; Kamunya et al., 2020 ; Winanti et al., 2020 ; Alsubhi et al., 2021 ; Júnior & Farias, 2021 ; Sofiadin & Azuddin, 2021 ; Yamani, 2021 ). This number is followed by a number of 5 approaches of type scenario based (Mi et al., 2018 ; Legaki et al., 2020 ; Al Ghawail et al., 2021 ; Bencsik et al., 2021 ; Fajri et al., 2021 ), and finally, only 2 approaches are categorized as high-level approaches (Carreño, 2018 ; de la Peña et al., 2021 ). It is worth saying that scenario-based approaches are, in most cases, the most difficult to reproduce in other educational environments, as they are very specific, and each environment has its own characteristics. In contrast, high-level approaches are more general and need to be tailored according to the context. Finally, gamification elements guidance approaches can strongly help implement gamified learning environments as they provide a handy catalog of elements that can be injected easily into learning environments.

Furthermore, Table ​ Table4 4 shows that most of the suggested design approaches in the literature are not empirically explored (for example, by using a control and comparing gamified and non-gamified systems). Indeed, of the 17 gamification approaches in the context of e-learning in higher education analyzed, only four approaches have been applied and evaluated by empirical evidence (Huang & Hew, 2018 ; Alsubhi et al., 2021 ; de la Peña et al., 2021 ; Júnior & Farias, 2021 ). Among those four studies, one work was validated with experts (Alsubhi et al., 2021 ).

Moreover, Table ​ Table4 4 shows that of the 17 gamification approaches proposed for application to online learning systems in the context of higher education, two approaches (Carreño, 2018 ; Kamunya et al., 2020 ) fall into the category of adaptive gamification. This shows the trendy nature of personalization in higher education. Finally, Table ​ Table4 4 shows that the 17 approaches that have been proposed to gamify online learning systems in higher education focus solely on structured gamification, neglecting the content side of online learning systems.

Discussion and limitations

Through this systematic review, we identified several papers on the gamification of e-learning in the higher education context. In recent years, the research on gamification in e-learning has been getting traction, and the number of research articles and systematic reviews of research articles is increasing. As a summary of the existing approaches of gamification in e-learning in higher education, we notice the following points:

Gamification of e-learning in higher education: a trending area of research

The systematic review showed that gamification of learning systems is nowadays a hot topic, and research in this field is growing rapidly as well as for e-learning in higher education context, as it is shown by Fig. ​ Fig.2 2 .

Number of publications per year

Gamification design gaps and tendencies

In general, gamification theory helps in training and shaping participant behavior, however, in our systematic literature review, we observed from RQ1 that the majority of applied research on gamification is not grounded in theory and did not use gamification frameworks in the design of the learning system under study. This highlights the fact that there is a real gap between theoretical and applied work on gamification. One reason may be that existing approaches are very theoretical and cannot strongly assist designers and practitioners when gamifying learning systems, as pointed out by Toda et al. ( 2020 ). This also explains our results to the second research question RQ2 regarding the lack of detail on the combination of game elements used in the experimental studies and the motivation behind choosing specific game elements over others.

To better understand this phenomenon and to find a rationale for this lack of using theory and, thus, the lack of logic behind the use of certain game elements over others and their random linking and combination in gamified learning systems, we addressed the research question RQ3. In the latter, we analyzed the gamification approaches available in the literature and classified them into different categories based on a variety of criteria. Our results revealed that the gamification elements guidance approaches that provide taxonomies of game elements that can be incorporated into learning systems constitute the majority of the approaches that have been proposed for application in online learning in higher education. Those did not provide the psychological and behavioral changes that correspond to each game element. Instead, the older gamification theory was based simply on the behavioral outcomes that come from using gamification and the motivational needs behind it and did not provide details on how to implement them or details on what elements to use.

Using appropriate game elements can lead to higher levels of user motivation, whereas inappropriate game elements can demotivate users (Hallifax et al., 2019a , 2019b ). Thus, it is essential to choose the right combination of game elements that perfectly matches the desired behavior change. To do this, we must first explore the effect of each game element separately (Dichev & Dicheva, 2017 ). Thus, further studies are needed to improve our understanding of how individual game elements relate to behavioral and motivational outcomes so that we can identify their contribution in studies that mix multiple game elements (Dichev & Dicheva, 2017 ). An example of such study was provided in the health domain in the paper (Hervas et al., 2017 ). The latter proposed a taxonomy of gamification elements used in the domain of health by relating them to psychological fundamentals on behavioral changes, like Self-efficacy, Social influence, and Behavioral momentum. This work can facilitate researchers' empirical validation of gamification theory by building contexts and scenarios from ready-made taxonomies of gamification elements that target a specific behavioral outcome.

On the other hand, through our systematic literature review, we can see from RQ3 the recent emergence of data-driven approaches through machine learning techniques (Knutas et al., 2019 ; Duggal et al., 2021 ). These techniques help to create gamification designs suitable for the gamified context, especially when it comes to customizing the game elements to be incorporated into the final gamified system to the students' profiles.

In many learning environments, pedagogy assumes that all learners have homogeneous characteristics (Kamunya et al., 2020 ). However, Schöbel and Söllner ( 2016 ) argue that most gamification projects are not working because they are designed for a group of system users without considering the personal needs of each user. Hence the advantage of personalized training to the learner where all learners differ in preference, style and abilities with regard to the learning processes with or without technology mediation (Naik & Kamat, 2015 ). In this context, we noted the existence of two gamification approaches designed for online learning in higher education (Carreño, 2018 ; Kamunya et al., 2020 ). This is put into practice by tailoring the gamification elements to users' individual preferences. A recent related problem is the lack of adaptation of gamification to the content being gamified.

Another recent and relevant issue is the extreme lack of content gamification. Indeed, the motivational impact of certain game elements varies with the user activity or the domain of gamified systems (Hallifax et al., 2019a , 2019b ). Therefore, there is a great need for further exploration and experimentation in this immature area to provide a gamified design to satisfy users’ preferences as well as the task at hand. In other words, personalization in gamification should extend to content, as it does with user profiles, for example, by applying machine learning techniques to tailor the choice of game elements to gamified content.

Another common study design issue illuminated by our review is the lack of validation of the proposed gamification approaches through statistical analyses. In addition, most applied research on the gamification of online learning systems in higher education has not explored the gamification frameworks suggested in the literature.

Conclusion and future work

In this work, we conducted a review of the literature on gamification elements used in digital higher education, the way they are combined, and the different gamification approaches proposed in the literature to gamify learning systems. We analyzed a total of 90 papers to answer the three research questions formulated for this study.

This review identified points, badges, leaderboards, levels, feedback, and challenges as the most commonly used game elements in digital higher education. However, in terms of using gamification theory, our review found that the majority of applied gamification research is not theory-based and has not used gamification frameworks in the design of gamified learning systems. Although some experimental studies attempt to adapt psychological and educational theories available in the literature as gamification approaches, the resulting systems are not very clear, and there is no rationale for choosing certain game elements over others. Consequently, it can be concluded that these gamification approaches cannot strongly assist designers and practitioners in gamifying their learning systems. In addition, theoretical gamification approaches in e-learning in higher education should focus on understanding the effect of each single game design element and the behavioral changes that outcome from its use.

Moreover, based on the results of this review, we can observe the trend towards data-driven approaches through the use of machine learning techniques, especially in adaptive gamification approaches. This involves the adaptation of gamification elements to user profiles. On the other hand, although we have noticed the increasing use of gamification elements that are suitable for content gamification and make the content more game-like, such as storytelling and challenges, there is still a lack of gamification approaches that address content gamification. In fact, this is still an immature research area in gamification design in e-learning in higher education.  Future works should pay more attention to the pedagogical side of learning systems and the task under gamification. Apart from that, further research is required to compare theory-driven to data-driven gamification approaches, in terms of which one is the better or perhaps evaluate the effectiveness of a combination of the two, and go so far as to propose a hybrid gamification approach, which does not exist yet and might solve several gamification design issues.

Regarding future work, efforts should focus on building a holistic approach by considering all the aspects that constitute the environment. Among those,  personalization according to students’ profiles, gamified subject, educational context, learner’s culture, learner’s preferences, level, playing motivations and experience with games.

Finally, we have seen that most of the design approaches suggested in the literature are not empirically explored. Therefore, statistical analyses and comparative studies should be conducted to draw more robust and generalizable conclusions to validate the existing gamification approaches in the literature.

Acknowledgements

Not applicable.

Author contributions

The authors worked together on the manuscript. All authors have read and approved the final manuscript.

Availability of data and materials

Declarations.

The authors declare that they have no competing interests.

Publisher's Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Contributor Information

Amina Khaldi, Email: zd.ise@idlahk_af .

Rokia Bouzidi, Email: zd.ise@idizuob_ra .

Fahima Nader, Email: zd.ise@redan_f .

• Adams SP, Du Preez R. Supporting student engagement through the gamification of learning activities: A design-based research approach. Technology, Knowledge and Learning. 2021; 27 :119–138. doi: 10.1007/s10758-021-09500-x. [ CrossRef ] [ Google Scholar ]
• Ahmed HD, Asiksoy G. The effects of gamified flipped learning method on student’s innovation skills, self-efficacy towards virtual physics lab course and perceptions. Sustainability (switzerland) 2021; 13 (18):10163. doi: 10.3390/su131810163. [ CrossRef ] [ Google Scholar ]
• Al Ghawail EA, Yahia SB, et al. Gamification model for developing E-learning in Libyan Higher Education. Smart education and e-learning 2021. Springer; 2021. pp. 97–110. [ Google Scholar ]
• Alcivar I, Abad A. Design and evaluation of a gamified system for ERP training. Computers in Human Behavior. 2016; 58 :109–118. doi: 10.1016/j.chb.2015.12.018. [ CrossRef ] [ Google Scholar ]
• Alhammad MM, Moreno AM. Gamification in software engineering education: A systematic mapping. Journal of Systems and Software. 2018; 141 :131–150. doi: 10.1016/j.jss.2018.03.065. [ CrossRef ] [ Google Scholar ]
• Allen MWMW. Designing successful e-learning: Forget what you know about instructional design and do something interesting. Wiley; 2007. [ Google Scholar ]
• Alsubhi, M., Ashaari, N., et al. (2021). Design and evaluation of an engagement framework for e-learning gamification. International Journal of Advanced Computer Science and Applications , 12 .
• Alsubhi MA, Sahari N. A conceptual engagement framework for gamified E-learning platform activities. International Journal of Emerging Technologies in Learning. 2020; 15 (22):4–23. doi: 10.3991/ijet.v15i22.15443. [ CrossRef ] [ Google Scholar ]
• Andrade FRH, Mizoguchi R, et al. The bright and dark sides of gamification. Springer; 2016. [ Google Scholar ]
• Aparicio, A. F., Vela, F. L. G., et al. (2012). Analysis and application of gamification. In Proceedings of the 13th international conference on Interacción Persona-Ordenador . Elche, Spain: Association for Computing Machinery. Article 17.
• Aşıksoy G. The effects of the gamified flipped classroom environment (GFCE) on students’ motivation, learning achievements and perception in a physics course. Quality and Quantity. 2018; 52 :129–145. doi: 10.1007/s11135-017-0597-1. [ CrossRef ] [ Google Scholar ]
• Asiksoy G, Canbolat S. The effects of the gamified flipped classroom method on petroleum engineering students' pre-class online behavioural engagement and achievement. International Journal of Engineering Pedagogy. 2021; 11 (5):19–36. doi: 10.3991/ijep.v11i5.21957. [ CrossRef ] [ Google Scholar ]
• Bencsik A, Mezeiova A, et al. Gamification in higher education (case study on a management subject) International Journal of Learning, Teaching and Educational Research. 2021; 20 (5):211–231. doi: 10.26803/ijlter.20.5.12. [ CrossRef ] [ Google Scholar ]
• Bennani, S., Maalel, A., et al. (2021). Towards an adaptive gamification model based on ontologies. In 2021 IEEE/ACS 18th international conference on computer systems and applications (AICCSA) .
• Bernik A. Gamification framework for E-learning systems in higher education. Tehnički Glasnik. 2021; 15 (2):184–190. doi: 10.31803/tg-20201008090615. [ CrossRef ] [ Google Scholar ]
• Bernik A, Radošević D, et al. Research on efficiency of applying gamified design into University's e-courses: 3D modeling and programming. Journal of Computer Science. 2017; 13 (12):718–727. doi: 10.3844/jcssp.2017.718.727. [ CrossRef ] [ Google Scholar ]
• Bernik A, Radošević D, et al. Achievements and usage of learning materials in computer science hybrid courses. Journal of Computer Science. 2019; 15 (4):489–498. doi: 10.3844/jcssp.2019.489.498. [ CrossRef ] [ Google Scholar ]
• Böckle, M., Micheel, I., et al. (2018). A design framework for adaptive gamification applications.
• Buckley P, Doyle E. Individualising gamification: An investigation of the impact of learning styles and personality traits on the efficacy of gamification using a prediction market. Computers and Education. 2017; 106 :43–55. doi: 10.1016/j.compedu.2016.11.009. [ CrossRef ] [ Google Scholar ]
• Carreño, A. M. (2018). A framework for agile design of personalized gamification services.
• Castro TC, Gonçalves LS. The use of gamification to teach in the nursing field. Revista Brasileira De Enfermagem. 2018; 71 (3):1038–1045. doi: 10.1590/0034-7167-2017-0023. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Cechetti NP, Bellei EA, et al. Developing and implementing a gamification method to improve user engagement: A case study with an m-Health application for hypertension monitoring. Telematics Informatics. 2019; 41 :126–138. doi: 10.1016/j.tele.2019.04.007. [ CrossRef ] [ Google Scholar ]
• Chou YK. Actionable gamification: Beyond points, badges, and leaderboards. Createspace Independent Publishing Platform; 2015. [ Google Scholar ]
• Coleman JD. Engaging undergraduate students in a co-curricular digital badging platform. Education and Information Technologies. 2018; 23 (1):211–224. doi: 10.1007/s10639-017-9595-0. [ CrossRef ] [ Google Scholar ]
• de la Peña D, Lizcano D, et al. Learning through play: Gamification model in university-level distance learning. Entertainment Computing. 2021; 39 :100430. doi: 10.1016/j.entcom.2021.100430. [ CrossRef ] [ Google Scholar ]
• De-Marcos L, Garcia-Cabot A, et al. Gamifying massive online courses: Effects on the social networks and course completion rates. Applied Sciences (switzerland) 2020; 10 (20):1–17. [ Google Scholar ]
• Deterding, S., Dixon, D., et al. (2011b). From game design elements to gamefulness: Defining gamification.
• Deterding, S., Dixon, D., et al. (2011a). From game design elements to gamefulness: Defining "gamification". In Proceedings of the 15th international academic mindtrek conference: envisioning future media environments (pp. 9–15). Tampere, Finland: Association for Computing Machinery.
• Deterding S. Gamification: Designing for motivation. Interactions. 2012; 19 (4):14–17. doi: 10.1145/2212877.2212883. [ CrossRef ] [ Google Scholar ]
• Dias J. Teaching operations research to undergraduate management students: The role of gamification. The International Journal of Management Education. 2017; 15 (1):98–111. doi: 10.1016/j.ijme.2017.01.002. [ CrossRef ] [ Google Scholar ]
• Dichev C, Dicheva D. Gamifying education: What is known, what is believed and what remains uncertain: A critical review. International Journal of Educational Technology in Higher Education. 2017; 14 (1):9. doi: 10.1186/s41239-017-0042-5. [ CrossRef ] [ Google Scholar ]
• Dicheva D, Dichev C, et al. Gamification in education: A systematic mapping study. Educational Technology & Society. 2015; 18 :75–88. [ Google Scholar ]
• Dikcius V, Urbonavicius S, et al. Learning marketing online: The role of social interactions and gamification rewards. Journal of Marketing Education. 2021; 43 (2):159–173. doi: 10.1177/0273475320968252. [ CrossRef ] [ Google Scholar ]
• Donath L, Mircea G, et al. E-learning platforms as leverage for education for sustainable development. European Journal of Sustainable Development. 2020; 9 (2):1–19. doi: 10.14207/ejsd.2020.v9n2p1. [ CrossRef ] [ Google Scholar ]
• Donnermann M, Lein M, et al. Social robots and gamification for technology supported learning: An empirical study on engagement and motivation. Computers in Human Behavior. 2021; 121 :106792. doi: 10.1016/j.chb.2021.106792. [ CrossRef ] [ Google Scholar ]
• Duggal K, Gupta LR, et al. Gamification and machine learning inspired approach for classroom engagement and learning. Mathematical Problems in Engineering. 2021; 2021 :9922775. doi: 10.1155/2021/9922775. [ CrossRef ] [ Google Scholar ]
• Enders B. GAMIFICATION, GAMES, AND LEARNING: What managers and practitioners need to know. The E-learning Guild; 2013. [ Google Scholar ]
• Facey-Shaw L, Specht M, et al. Do badges affect intrinsic motivation in introductory programming students? Simulation and Gaming. 2020; 51 (1):33–54. doi: 10.1177/1046878119884996. [ CrossRef ] [ Google Scholar ]
• Fajiculay JR, Parikh BT, et al. Student perceptions of digital badges in a drug information and literature evaluation course. Currents in Pharmacy Teaching and Learning. 2017; 9 (5):881–886. doi: 10.1016/j.cptl.2017.05.013. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Fajri FA, R. K. Haribowo P,, et al. Gamification in e-learning: The mitigation role in technostress. International Journal of Evaluation and Research in Education. 2021; 10 (2):606–614. [ Google Scholar ]
• García F, Pedreira O, et al. A framework for gamification in software engineering. Journal of Systems and Software. 2017; 132 :21–40. doi: 10.1016/j.jss.2017.06.021. [ CrossRef ] [ Google Scholar ]
• Gari, M. R. N., & Radermacher, A. D. (2018). Gamification in computer science education: A systematic literature review. In ASEE annual conference and exposition, conference proceedings .
• Garnett T, Button D. The use of digital badges by undergraduate nursing students: A three-year study. Nurse Education in Practice. 2018; 32 :1–8. doi: 10.1016/j.nepr.2018.06.013. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Garone P, Nesteriuk S. Gamification and learning: A comparative study of design frameworks. Springer; 2019. [ Google Scholar ]
• Górska D. E-learning in Higher Education. The Person and the Challenges. the Journal of Theology, Education, Canon Law and Social Studies Inspired by Pope John Paul II. 2016; 6 (2):35. doi: 10.15633/pch.1868. [ CrossRef ] [ Google Scholar ]
• Guérard-Poirier N, Beniey M, et al. An educational network for surgical education supported by gamification elements: Protocol for a randomized controlled trial. JMIR Research Protocols. 2020; 9 (12):e21273. doi: 10.2196/21273. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Gunawan FE, Jupiter, Gamification analysis and implementation in online learning. ICIC Express Letters. 2018; 12 (12):1195–1204. [ Google Scholar ]
• Gündüz, A. Y., & Akkoyunlu, B. (2020). Effectiveness of gamification in flipped learning. SAGE Open , 10 (4).
• Hallifax, S., Serna, A., et al. (2019a). Factors to consider for tailored gamification. In Proceedings of the Annual Symposium on Computer-Human Interaction in Play (pp. 559–572). Barcelona, Spain: Association for Computing Machinery.
• Hallifax, S., Serna, A., et al. (2019b). Adaptive gamification in education: A literature review of current trends and developments. In Lecture notes in computer science (including subseries lecture notes in artificial intelligence and lecture notes in bioinformatics) (Vol. 11722 LNCS, pp. 294–307).
• Hervas, R., Ruiz-Carrasco, D., et al. (2017). Gamification mechanics for behavioral change: A systematic review and proposed taxonomy. In ACM international conference proceeding series .
• Hisham FBMN, Sulaiman S. Adapting gamification approach in massive open online courses to improve user engagement. UTM Computing Proceedings Innovation in Computing Technology and Applications. 2017; 2 :1–6. [ Google Scholar ]
• Huang B, Hew KF. Implementing a theory-driven gamification model in higher education flipped courses: Effects on out-of-class activity completion and quality of artifacts. Computers and Education. 2018; 125 :254–272. doi: 10.1016/j.compedu.2018.06.018. [ CrossRef ] [ Google Scholar ]
• Hunicke, R., Leblanc, M. G., et al. (2004). MDA: A formal approach to game design and game research.
• Jianu, E. M., & Vasilateanu, A. (2017). Designing of an e-learning system using adaptivity and gamification. In 2017 IEEE international systems engineering symposium (ISSE) .
• Júnior, E., & Farias, K. (2021). ModelGame: A quality model for gamified software modeling learning. In 15th Brazilian symposium on software components, architectures, and reuse (pp. 100–109). Joinville, Brazil: Association for Computing Machinery.
• Kalogiannakis M, Papadakis S, et al. Gamification in science education. A systematic review of the literature. Education Sciences. 2021; 11 (1):22. doi: 10.3390/educsci11010022. [ CrossRef ] [ Google Scholar ]
• Kamunya, S., Mirirti, E., et al. (2020). An adaptive gamification model for e-learning. In 2020 IST-Africa conference (IST-Africa) .
• Kapp, K. M. (2012). The gamification of learning and instruction: game-based methods and strategies for training and education.
• Kapp KMBLMR. The gamification of learning and instruction fieldbook: Ideas into practice. Wiley; 2014. [ Google Scholar ]
• Kasinathan V, Mustapha A, et al. Questionify: Gamification in education. International Journal of Integrated Engineering. 2018; 10 (6):139–143. doi: 10.30880/ijie.2018.10.06.019. [ CrossRef ] [ Google Scholar ]
• Khaleel FL, Ashaari NS, et al. An empirical study on gamification for learning programming language website. Jurnal Teknologi. 2019; 81 (2):151–162. [ Google Scholar ]
• Khaleel FL, Ashaari NS, et al. The impact of gamification on students learning engagement. International Journal of Electrical and Computer Engineering. 2020; 10 (5):4965–4972. [ Google Scholar ]
• Khalil, M., Wong, J., et al. (2018). Gamification in MOOCs: A review of the state of the art. In IEEE global engineering education conference, EDUCON .
• Kim JT, Lee W-H. Dynamical model for gamification of learning (DMGL) Multimedia Tools and Applications. 2015; 74 (19):8483–8493. doi: 10.1007/s11042-013-1612-8. [ CrossRef ] [ Google Scholar ]
• Kitchenham B. Procedures for performing systematic reviews. Software Engineering Group Department of Computer Science, Keele University; 2004. [ Google Scholar ]
• Knutas A, van Roy R, et al. A process for designing algorithm-based personalized gamification. Multimedia Tools and Applications. 2019; 78 (10):13593–13612. doi: 10.1007/s11042-018-6913-5. [ CrossRef ] [ Google Scholar ]
• Kyewski E, Krämer NC. To gamify or not to gamify? An experimental field study of the influence of badges on motivation, activity, and performance in an online learning course. Computers and Education. 2018; 118 :25–37. doi: 10.1016/j.compedu.2017.11.006. [ CrossRef ] [ Google Scholar ]
• Landers RN. Developing a theory of gamified learning: Linking serious games and gamification of learning. Simulation & Gaming. 2014; 45 (6):752–768. doi: 10.1177/1046878114563660. [ CrossRef ] [ Google Scholar ]
• Lavoué E, Monterrat B, et al. Adaptive gamification for learning environments. IEEE Transactions on Learning Technologies. 2019; 12 (1):16–28. doi: 10.1109/TLT.2018.2823710. [ CrossRef ] [ Google Scholar ]
• Legaki, N. Z., & Hamari, J. (2020). Gamification in statistics education: A literature review. In CEUR workshop proceedings .
• Legaki NZ, Xi N, et al. The effect of challenge-based gamification on learning: An experiment in the context of statistics education. International Journal of Human Computer Studies. 2020; 144 :102496. doi: 10.1016/j.ijhcs.2020.102496. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Llorens-Largo F, Gallego-Durán FJ, et al. Gamification of the learning process: Lessons learned. IEEE Revista Iberoamericana De Tecnologias Del Aprendizaje. 2016; 11 (4):227–234. doi: 10.1109/RITA.2016.2619138. [ CrossRef ] [ Google Scholar ]
• Majuri, J., Koivisto, J., et al. (2018). Gamification of education and learning: A review of empirical literature. In CEUR workshop proceedings .
• Marín B, Frez J, et al. An empirical investigation on the benefits of gamification in programming courses. ACM Transactions on Computing Education. 2019; 19 (1):1–22. doi: 10.1145/3231709. [ CrossRef ] [ Google Scholar ]
• Mi, Q., Keung, J., et al. (2018). A gamification technique for motivating students to learn code readability in software engineering. In Proceedings—2018 international symposium on educational technology, ISET 2018 .
• Milenković I, Šošević U, et al. Improving student engagement in a biometric classroom: The contribution of gamification. Universal Access in the Information Society. 2019; 18 (3):523–532. doi: 10.1007/s10209-019-00676-9. [ CrossRef ] [ Google Scholar ]
• Mora A, Riera D, et al. Gamification: A systematic review of design frameworks. Journal of Computing in Higher Education. 2017; 29 (3):516–548. doi: 10.1007/s12528-017-9150-4. [ CrossRef ] [ Google Scholar ]
• Morschheuser, B., Werder, K., et al. (2017). How to gamify? A method for designing gamification.
• Morschheuser B, Hassan L, et al. How to design gamification? A method for engineering gamified software. Information and Software Technology. 2018; 95 :219–237. doi: 10.1016/j.infsof.2017.10.015. [ CrossRef ] [ Google Scholar ]
• Naik, V., & Kamat, V. V. (2015). Adaptive and gamified learning environment (AGLE). In 2015 IEEE seventh international conference on technology for education (T4E) (pp. 7–14).
• Nicholson, S. (2012). A user-centered theoretical framework for meaningful gamification.
• Nielson, B. (2017). Gamification mechanics vs. gamification dynamics . Retrieved from https://www.yourtrainingedge.com/gamification-mechanics-vs-gamification-dynamics/ .
• Ozdamli, S. K. A. F. (2018). A review of research on gamification approach in education.
• Pakinee A, Puritat K. Designing a gamified e-learning environment for teaching undergraduate ERP course based on big five personality traits. Education and Information Technologies. 2021; 26 (4):4049–4067. doi: 10.1007/s10639-021-10456-9. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Park J, De L, et al. GAMESIT: A gamified system for information technology training. Computers and Education. 2019; 142 :103643. doi: 10.1016/j.compedu.2019.103643. [ CrossRef ] [ Google Scholar ]
• Pérez-López IJ, Rivera García E, et al. “The prophecy of the chosen ones”: An example of gamification applied to university teaching. Revista Internacional De Medicina y Ciencias De La Actividad Fisica y Del Deporte. 2017; 17 (66):243–260. [ Google Scholar ]
• Pilkington C. A playful approach to fostering motivation in a distance education computer programming course: Behaviour change and student perceptions. International Review of Research in Open and Distance Learning. 2018; 19 (3):282–298. doi: 10.19173/irrodl.v19i3.3664. [ CrossRef ] [ Google Scholar ]
• Rivera ES, Garden CLP. Gamification for student engagement: A framework. Journal of Further and Higher Education. 2021; 45 (7):999–1012. doi: 10.1080/0309877X.2021.1875201. [ CrossRef ] [ Google Scholar ]
• Rodríguez I, Puig A, et al. Towards adaptive gamification: A method using dynamic player profile and a case study. Applied Sciences. 2022; 12 (1):486. doi: 10.3390/app12010486. [ CrossRef ] [ Google Scholar ]
• Romero-Rodriguez LM, Ramirez-Montoya MS, et al. Gamification in MOOCs: Engagement application test in energy sustainability courses. IEEE Access. 2019; 7 :32093–32101. doi: 10.1109/ACCESS.2019.2903230. [ CrossRef ] [ Google Scholar ]
• Ropero-Padilla C, Rodriguez-Arrastia M, et al. A gameful blended-learning experience in nursing: A qualitative focus group study. Nurse Education Today. 2021; 106 :105109. doi: 10.1016/j.nedt.2021.105109. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• van Roy, R., & Zaman, B. (2017). Why gamification fails in education and how to make it successful: Introducing nine gamification heuristics based on self-determination theory (pp. 485–509).
• Ryan R, Deci E. Self-determination theory and the facilitation of intrinsic motivation, social development, and well-being. The American Psychologist. 2000; 55 :68–78. doi: 10.1037/0003-066X.55.1.68. [ PubMed ] [ CrossRef ] [ Google Scholar ]
• Saggah, A., Atkins, A. S., et al. (2020). A review of gamification design frameworks in education. In 2020 Fourth international conference on intelligent computing in data sciences (ICDS) .
• Saleem AN, Noori NM, et al. Gamification applications in E-learning: A literature review. Technology, Knowledge and Learning. 2021; 27 :139–159. doi: 10.1007/s10758-020-09487-x. [ CrossRef ] [ Google Scholar ]
• Sanal, A. (2019). Content gamification vs structured gamification in E-learning . Retrieved from https://playxlpro.com/content-gamification-vs-structured-gamification-in-e-learning/ .
• Sanchez DR, Langer M, et al. Gamification in the classroom: Examining the impact of gamified quizzes on student learning. Computers and Education. 2020; 144 :103666. doi: 10.1016/j.compedu.2019.103666. [ CrossRef ] [ Google Scholar ]
• Schöbel, S., & Söllner, M. (2016). How to gamify information systems—Adapting gamification to individual user preferences.
• Schonfeld, E. (2010). SCVNGR's secret game mechanics playdeck.
• Seaborn K, Fels DI. Gamification in theory and action: A survey. International Journal of Human-Computer Studies. 2015; 74 :14–31. doi: 10.1016/j.ijhcs.2014.09.006. [ CrossRef ] [ Google Scholar ]
• da Silva, R. J. R., Rodrigues, R. G., et al. (2019)."Gamification in management education: A systematic literature review. BAR - Brazilian Administration Review , 16 (2).
• Simões J, Redondo RPD, et al. A social gamification framework for a K-6 learning platform. Computers in Human Behavior. 2013; 29 :345–353. doi: 10.1016/j.chb.2012.06.007. [ CrossRef ] [ Google Scholar ]
• Smith T. Gamified modules for an introductory statistics course and their impact on attitudes and learning. Simulation and Gaming. 2017; 48 (6):832–854. doi: 10.1177/1046878117731888. [ CrossRef ] [ Google Scholar ]
• Sofiadin, A., & Azuddin, M. (2021). An initial sustainable e-learning and gamification framework for higher education. In International conferences on mobile learning 2021 and educational technologies 2021 .
• Subhash S, Cudney EA. Gamified learning in higher education: A systematic review of the literature. Computers in Human Behavior. 2018; 87 :192–206. doi: 10.1016/j.chb.2018.05.028. [ CrossRef ] [ Google Scholar ]
• Swacha J. State of research on gamification in education: A bibliometric survey. Education Sciences. 2021; 11 :69. doi: 10.3390/educsci11020069. [ CrossRef ] [ Google Scholar ]
• Toda, A. M., Oliveira, W., et al. (2019). A taxonomy of game elements for gamification in educational contexts: Proposal and evaluation. In 2019 IEEE 19th international conference on advanced learning technologies (ICALT) .
• Toda A, Toledo Palomino P, et al. How to gamify learning systems? An experience report using the design sprint method and a taxonomy for gamification elements in education. Educational Technology & Society. 2020; 22 :47–60. [ Google Scholar ]
• Towongpaichayont W. A guideline of designing gamification in the classroom and its case study. ICIC Express Letters. 2021; 15 (6):639–647. [ Google Scholar ]
• Tsay CHH, Kofinas A, et al. Enhancing student learning experience with technology-mediated gamification: An empirical study. Computers and Education. 2018; 121 :1–17. doi: 10.1016/j.compedu.2018.01.009. [ CrossRef ] [ Google Scholar ]
• Urh M, Vukovic G, et al. The model for introduction of gamification into E-learning in higher education. Procedia - Social and Behavioral Sciences. 2015; 197 :388–397. doi: 10.1016/j.sbspro.2015.07.154. [ CrossRef ] [ Google Scholar ]
• Uz Bilgin C, Gul A. Investigating the effectiveness of gamification on group cohesion, attitude, and academic achievement in collaborative learning environments. TechTrends. 2020; 64 (1):124–136. doi: 10.1007/s11528-019-00442-x. [ CrossRef ] [ Google Scholar ]
• van Gaalen AEJ, Brouwer J, et al. Gamification of health professions education: A systematic review. Advances in Health Sciences Education. 2021; 26 (2):683–711. doi: 10.1007/s10459-020-10000-3. [ PMC free article ] [ PubMed ] [ CrossRef ] [ Google Scholar ]
• van Roy R, Zaman B. Unravelling the ambivalent motivational power of gamification: A basic psychological needs perspective. International Journal of Human Computer Studies. 2019; 127 :38–50. doi: 10.1016/j.ijhcs.2018.04.009. [ CrossRef ] [ Google Scholar ]
• Wendy Hsin-Yuan Huang DS. Gamification of education. Rotman School of Management, University of Toronto; 2013. [ Google Scholar ]
• Werbach, K., & Hunter, D. (2012). For the win: How game thinking can revolutionize your business.
• Winanti W, Abbas BS, et al. Gamification framework for programming course in higher education. Journal of Games, Game Art, and Gamification. 2020; 5 (2):54–57. doi: 10.21512/jggag.v5i2.7479. [ CrossRef ] [ Google Scholar ]
• Wongso, O., Rosmansyah, Y., et al. (2014). Gamification framework model, based on social engagement in e-learning 2.0. In 2014 2nd international conference on technology, informatics, management, engineering & environment (pp. 10–14).
• Yamani H. A conceptual framework for integrating gamification in elearning systems based on instructional design model. International Journal of Emerging Technologies in Learning. 2021; 16 :14–33. doi: 10.3991/ijet.v16i04.15693. [ CrossRef ] [ Google Scholar ]
• Yildirim I. The effects of gamification-based teaching practices on student achievement and students' attitudes toward lessons. Internet and Higher Education. 2017; 33 :86–92. doi: 10.1016/j.iheduc.2017.02.002. [ CrossRef ] [ Google Scholar ]
• Zainuddin Z, Chu SKW, et al. The impact of gamification on learning and instruction: A systematic review of empirical evidence. Educational Research Review. 2020; 30 :100326. doi: 10.1016/j.edurev.2020.100326. [ CrossRef ] [ Google Scholar ]
• Zaric N, Lukarov V, et al. A fundamental study for gamification design: Exploring learning tendencies’ effects. International Journal of Serious Games. 2020; 7 (4):3–25. doi: 10.17083/ijsg.v7i4.356. [ CrossRef ] [ Google Scholar ]
• Zhao D, Playfoot J, et al. An innovative multi-layer gamification framework for improved STEM learning experience. IEEE Access. 2022; 10 :3879–3889. doi: 10.1109/ACCESS.2021.3139729. [ CrossRef ] [ Google Scholar ]
• Zichermann G, Cunningham C. Gamification by design: Implementing game mechanics in web and mobile apps. O'Reilly Media, Inc; 2011. [ Google Scholar ]

This paper is in the following e-collection/theme issue:

Published on 9.4.2024 in Vol 10 (2024)

Discovering Subgroups of Children With High Mortality in Urban Guinea-Bissau: Exploratory and Validation Cohort Study

Authors of this article:

Original Paper

• Andreas Rieckmann 1 , PhD   ; 
• Sebastian Nielsen 2, 3 , PhD   ; 
• Piotr Dworzynski 4 , PhD   ; 
• Heresh Amini 5, 6 , PhD   ; 
• Søren Wengel Mogensen 7 , PhD   ; 
• Isaquel Bartolomeu Silva 2, 3 , MSc   ; 
• Angela Y Chang 8, 9 , ScD   ; 
• Onyebuchi A Arah 10, 11, 12 , MD, PhD   ; 
• Wojciech Samek 13, 14, 15 , Dr rer nat   ; 
• Naja Hulvej Rod 1 , PhD, DMSc   ; 
• Claus Thorn Ekstrøm 16 , PhD   ; 
• Christine Stabell Benn 2, 3, 8 , MD, PhD, DMSc   ; 
• Peter Aaby 2, 3 , DMSc   ; 
• Ane Bærent Fisker 2, 3 , MD, PhD  

1 Section of Epidemiology, Department of Public Health, University of Copenhagen, Copenhagen, Denmark

2 Bandim Health Project, INDEPTH Network, Bissau, Guinea-Bissau

3 Bandim Health Project, Research unit Odense Patient Data Explorative Network (OPEN), Department of Clinical Research, Odense University Hospital and University of Southern Denmark, Odense, Denmark

4 Novo Nordisk Foundation Center for Basic Metabolic Research, University of Copenhagen, Copenhagen, Denmark

5 Department of Environmental Medicine and Climate Science, Icahn School of Medicine at Mount Sinai, New York, NY, United States

6 Institute for Climate Change, Environmental Health, and Exposomics, Icahn School of Medicine at Mount Sinai, New York, NY, United States

7 Department of Automatic Control, Lund University, Lund, Sweden

8 Danish Institute for Advanced Study, University of Southern Denmark, Odense, Denmark

9 The Interdisciplinary Centre on Population Dynamics, University of Southern Denmark, Odense, Denmark

10 Department of Epidemiology, Fielding School of Public Health, University of California, Los Angeles, Los Angeles, CA, United States

11 Department of Statistics and Data Science, College of Letters and Science, University of California, Los Angeles, Los Angeles, CA, United States

12 Research Unit for Epidemiology, Department of Public Health, University of Aarhus, Aarhus, Denmark

13 Department of Artificial Intelligence, Fraunhofer Heinrich Hertz Institute, Berlin, Germany

14 Department of Electrical Engineering and Computer Science, Technical University of Berlin, Berlin, Germany

15 Berlin Institute for the Foundations of Learning and Data, Berlin, Germany

16 Section of Biostatistics, Department of Public Health, University of Copenhagen, Copenhagen, Denmark

Corresponding Author:

Andreas Rieckmann, PhD

Section of Epidemiology

Department of Public Health

University of Copenhagen

Øster Farimagsgade 5

Copenhagen, 1353

Phone: 45 35326765

Email: [email protected]

Background: The decline in global child mortality is an important public health achievement, yet child mortality remains disproportionally high in many low-income countries like Guinea-Bissau. The persisting high mortality rates necessitate targeted research to identify vulnerable subgroups of children and formulate effective interventions.

Objective: This study aimed to discover subgroups of children at an elevated risk of mortality in the urban setting of Bissau, Guinea-Bissau, West Africa. By identifying these groups, we intend to provide a foundation for developing targeted health interventions and inform public health policy.

Methods: We used data from the health and demographic surveillance site, Bandim Health Project, covering 2003 to 2019. We identified baseline variables recorded before children reached the age of 6 weeks. The focus was on determining factors consistently linked with increased mortality up to the age of 3 years. Our multifaceted methodological approach incorporated spatial analysis for visualizing geographical variations in mortality risk, causally adjusted regression analysis to single out specific risk factors, and machine learning techniques for identifying clusters of multifactorial risk factors. To ensure robustness and validity, we divided the data set temporally, assessing the persistence of identified subgroups over different periods. The reassessment of mortality risk used the targeted maximum likelihood estimation (TMLE) method to achieve more robust causal modeling.

Results: We analyzed data from 21,005 children. The mortality risk (6 weeks to 3 years of age) was 5.2% (95% CI 4.8%-5.6%) for children born between 2003 and 2011, and 2.9% (95% CI 2.5%-3.3%) for children born between 2012 and 2016. Our findings revealed 3 distinct high-risk subgroups with notably higher mortality rates, children residing in a specific urban area (adjusted mortality risk difference of 3.4%, 95% CI 0.3%-6.5%), children born to mothers with no prenatal consultations (adjusted mortality risk difference of 5.8%, 95% CI 2.6%-8.9%), and children from polygamous families born during the dry season (adjusted mortality risk difference of 1.7%, 95% CI 0.4%-2.9%). These subgroups, though small, showed a consistent pattern of higher mortality risk over time. Common social and economic factors were linked to a larger share of the total child deaths.

Conclusions: The study’s results underscore the need for targeted interventions to address the specific risks faced by these identified high-risk subgroups. These interventions should be designed to work to complement broader public health strategies, creating a comprehensive approach to reducing child mortality. We suggest future research that focuses on developing, testing, and comparing targeted intervention strategies unraveling the proposed hypotheses found in this study. The ultimate aim is to optimize health outcomes for all children in high-mortality settings, leveraging a strategic mix of targeted and general health interventions to address the varied needs of different child subgroups.

Introduction

Child mortality in Guinea-Bissau has decreased significantly over the past 40 years but is still unacceptably high (1 in 13 children dying before the age of 5 years in 2021 [ 1 ]). Thus, there is a constant need to design relevant interventions to reduce mortality [ 2 , 3 ]. In particular, identifying subgroups of children at high risk of dying may inform targeted preventive or risk-mitigating interventions to supplement population-wide approaches [ 4 , 5 ].

To identify actionable points for interventions to prevent or mitigate risk, we want to document the fuller causal structure. This spans distal causes such as social and economic conditions, legal rights, and welfare policies to immediate causes such as congenital malformations or infectious agents [ 6 ]. However, obtaining high-quality data on these factors can be challenging, particularly in low-income countries. One potential data source is Health and Demographic Surveillance Systems (HDSS), which collect individual-level data on demographics and health for a portion of the population [ 7 ].

In this exploratory study, we used HDSS data from urban Bissau, the capital of Guinea-Bissau, to identify subgroups of children at high risk of dying before 3 years of age. We analyzed data from 2003 to 2019, where the birth years from 2003 to 2011 were used to identify risk factors and high-risk groups, which we then tested in the birth years from 2012 to 2016. This allowed us to focus on factors consistently associated with high mortality over time. To do this, we used 3 different types of analyses, that are, spatial analysis to map child mortality in specific areas, regression analysis to identify single risk factors associated with high mortality, and a machine learning model to identify multifactorial risk groups. By integrating these approaches, we aimed to discover subgroups of children with high mortality without being limited to prior hypotheses [ 8 ]. Such discoveries are necessary for developing new hypotheses and identifying interventions to reduce child mortality.

Study Population and Follow-Up

The study population included children living in Bissau, the capital city of Guinea-Bissau. All the children were part of the HDSS Bandim Health Project [ 9 ] and were seen by data collectors within the first 6 weeks of life. Children under 3 years of age are routinely visited every 3-4 months to collect vital and health information. Many recorded child deaths in this population are due to infectious diseases such as respiratory infections, malaria, and diarrhea [ 10 , 11 ].

Follow-up for this study began at 6 weeks of age to ensure that a sufficient proportion of children had their baseline information recorded. Children, who died before 6 weeks of age or did not have complete baseline information, were excluded from the study (30,441/51,446, see flowchart in Figure S1 in Multimedia Appendix 1 ). To account for the potential selection bias caused by migrating children, inverse probability of censoring weights (IPCW) was used in all analyses and presented results ( Multimedia Appendix 2 ) [ 12 ].

Baseline Information

To identify relevant factors for child mortality, available baseline information was divided into environmental, household, and individual and birth domains. Figure 1 depicts the assumed causal structure [ 12 ] or the data-generating process linking these domains. Operational definitions of the variables and a visualization of their pairwise associations can be found in Table S1 and Figure S2 in Multimedia Appendix 1 , respectively.

A temporal split of the data, rather than a random split, was used in this study. This allowed us to determine whether identified subgroups consistently had a higher mortality risk across different time periods and thus, may be relevant subgroups for future intervention. The sample size allowed us to divide the data into 2 cohorts. The temporal validation cohort also allowed us to test the robustness of our findings, as there have been various changes over time that may have affected child mortality, such as a significant decline in respiratory infections [ 11 , 13 ].

To describe the temporal trends in child mortality, the Kaplan-Meier estimator was used to calculate overall risks and risks split by age, accounting for censoring during follow-up. All subsequent analyses excluded censored children and used IPCW to adjust for selection bias.

We conducted the following 3 analyses to investigate the association between the baseline information collected before 6 weeks of age and mortality during the entire follow-up period (from 6 weeks to 3 years). All association measures were reported as mortality risk difference (MRD), with 0 indicating no difference between the compared subgroups. The MRD was expressed as a percentage (ie, the difference in deaths per 100 children). All estimates in the results section are adjusted mortality risk differences (aMRD).

Spatial Analysis

We examined whether certain residential areas had a higher mortality risk than others by mapping the children’s households at baseline and moving a sliding window (250 m × 250 m) 10 m at a time to visualize the mortality risk across the study area. Estimates were only presented when at least 100 children were included within the sliding window to avoid small cell sizes. The estimates were adjusted for the linear effect of birth year, as child mortality has approximately decreased linearly by birth year (Figure S3 in Multimedia Appendix 1 ).

Single Risk Factors

We used generalized linear regressions to investigate the associations between single factors and a higher risk of child mortality. Adjustments were made according to the assumed causal structure (depicted in Figure 1 ) by blocking the common causes in higher-order domains using the backdoor criterion [ 12 ].

Multifactorial Risk Groups

We applied the Causes of Outcome Learning approach [ 14 , 15 ] to identify vulnerable subgroups with a combination of baseline information that was associated with a higher risk of child mortality. This causal inference-inspired machine learning approach has been optimized to prevent causal biases such as confounding by calendar time and collider bias, which could occur in other supervised clustering approaches. Details of the implementation of the Causes of Outcome Learning approach can be found in Multimedia Appendix 3 . Since this approach is optimized for interactions in subpopulations, it is expected to find other patterns than the first-order linear regression which averages across the entire population.

Summarization and Causal Modeling

To summarize the findings from the 3 analyses, key statistics such as prevalence, crude risks, and identification of synergistic associations [ 16 ] (where the risk from simultaneous exposure to multiple factors is greater than the sum of the individual risks) were calculated. Adjusted risk differences were determined using causal modeling (targeted maximum likelihood estimation [TMLE] [ 17 ]) for the defined subgroups compared to all other children. The probability of the estimates from the hypothesis-generating and temporal validation cohorts being similar was also calculated. In addition, a combined estimate for both cohorts was obtained to estimate the population attributable fraction (PAF) [ 18 ], which represents the fraction of all mortality that would be prevented if the causal exposure of interest was removed. The analyses were conducted using R (version 4.2; R Core Team), and some sentences were revised using ChatGPT (OpenAI) to improve clarity.

Ethical Considerations

The study does not include biologically, physically invasive, or potentially dangerous procedures. The HDSS collection of data is at the request of the Ministry of Health, Guinea-Bissau.

A total of 51,446 children were registered between 2003 and 2019, with 30,441 being excluded from our analysis due to registration after 6 weeks of age, lack of follow-up information, death by 6 weeks of age, missing baseline information, or emigration during follow-up (see flowchart in Figure S1 in Multimedia Appendix 1 ). The study sample included 21,005 children, which was weighted to an analytical sample of 27,998 children using IPCW to account for nonrandom emigration. The hypothesis-generating and temporal validation cohorts were based on weighted samples of 19,311 and 8687 children, respectively. The weights were not extreme ( Multimedia Appendix 2 ). The mortality risk during the follow-up period (from 6 weeks to 3 years of age) was 5.2% (95% CI 4.8%-5.6%) in the hypothesis-generating cohort and 2.9% (95% CI 2.5%-3.3%) in the temporal validation cohort.

We explored the results from the spatial analysis of the hypothesis-generating cohort, which gave rise to defining 4 areas; A, B, C, and D where the child mortality rate was considerably high. We marked these areas with circles on top of the spatial results in Figure 2 . By comparing children living in the residential areas marked by circles A, B, C, and D (constituting between 1% [n=253] and 3% [n=533] of children) to those living outside these areas ( Figure 2 ), the aMRD was 4.5% (95% CI –0.6% to 9.6%), 1.9% (95% CI –1.2% to 5.0%), 3.3% (95% CI –0.3% to 6.9%), and 4.0% (95% CI 0.1%-8.0%), respectively. When the 4 suggested high-risk residential areas were assessed in the temporal validation cohort, only area D still tended to exhibit higher mortality though the estimate was associated with more uncertainty (aMRD of 2.0%, 95% CI –2.8% to 6.7%) ( Table 1 and Figure S4 in Multimedia Appendix 1 ). The combined estimate for both cohorts for area D was an aMRD of 3.4% (95% CI 0.3%-6.5%). If causal, the excess risk translates to a PAF of 1.1% of all deaths.

a aMRD: adjusted mortality risk difference.

b TMLE: targeted maximum likelihood estimation.

c The additional risk above the linear effect of the single factors is presented (ie, the parameter for the subgroup parameter while adjusting for each of the variables used to create the subgroup definition). Also adjusted for calendar time but no potential confounders.

d The additional risk-adjusted using targeted maximum likelihood estimation with using linear models.

e N/A: not applicable.

f Adjustment for calendar time.

g These findings were considered consistent across the cohorts by the authors.

h Adjustment for calendar time and environmental factors.

i HDSS: Health and Demographic Surveillance Systems.

j Adjustment for calendar time and environmental and household factors.

In the hypothesis-generating cohort, children of mothers with less education than 7 years compared to those with 7 years or more of education were common (n=11,739, 60.8%) and had an aMRD of 1.6% (95% CI 0.9%-2.4%; Table 2 ). Crowding (ie, having multiple children in the household under 3 years of age) was common (n=16,054, 83%) and associated with an aMRD of 1.6% (95% CI 0.6%-2.6%) compared to being the sole child ( Table 2 ). Having functioning electricity, a television, and a toilet inside the house indicates higher wealth, which was associated with lower child mortality. Across both cohorts with very similar estimates, both less maternal schooling and more crowding were associated with an aMRD of approximately 1.5% ( Table 1 , the column “aMRD visualizations” shows virtually the same estimates). If causal 19.4% of all deaths could be attributed to low-maternal education and 30.0% to crowding.

The most pronounced environmental factor was living within 50 m of a major road, associated with an aMRD of 2.1% (95% CI 0.6%-3.6%) compared with children living further away.

Children of mothers lost to follow-up were at a marked increased mortality risk. Still, they constituted a very small number of children in the temporal validation cohort (additional explanation in Table S1 in Multimedia Appendix 1 ).

Being a twin was consistently associated with higher mortality, with an aMRD of 4.3% (95% CI 1.0%-7.7%) and 3.0% (95% CI –0.5% to 6.5%) in the hypothesis-generating and temporal validation cohorts, respectively ( Table 1 ).

No prenatal consultation was recorded for 4% (n=834) of the children in the hypothesis-generating cohort and was associated with an aMRD of 6.4% (95% CI 2.5%-10.2%; Table 1 ). In the temporal validation data set, this was associated with an aMRD of 2.8% (95% CI –0.9% to 6.6%).

a For the environmental category, adjusted risk differences are adjusted for a calendar effect, risk differences in the household category are additionally adjusted for the environmental variables, and risk differences in the information related to the delivery category are further adjusted for the household variables.

b The prevalence differs from 10% to 90% because the cutoff is based on data from both cohorts (2003-2016).

c Reference group.

d N/A: not applicable, since the effect of sex and birth season is not expected to be confounded and thus does not need adjustment.

In the hypothesis-generating cohort, twins born in the rainy season had higher mortality risk compared with those born in the dry season (aMRD of 7.0%, 95% CI 2.0%-11.9%; Figure 3 , group 4, and Table 1 ), but this association was not found in the temporal validation data (aMRD 1.3%, 95% CI –3.0% to 5.6%; Table 1 ). Children of polygamous families born in the dry season had an aMRD of 1.8% (95% CI 0.2%-3.3%) compared to all other children ( Figure 3 , group 5, and Table 1 ). This subgroup constituted 9% (n=1770) of the children in the hypothesis-generating cohort, and the finding was consistent in the temporal validation cohort (an aMRD of 1.7%, 95% CI –0.4% to 3.9%, covering 8% [n=720] of all children; Table 1 , see the column with aMRD visualization for consistency). If these associations are causal, 3.3% of all deaths could be attributed to this combination. A supplementary analysis of both cohorts (birth years 2003-2016) was conducted to understand the phenomenon better. The results suggested that (1) the finding was not artificially introduced by the IPCW approach; (2) the increased risk was highest in the first half-year of follow-up (6 weeks to 7 months of age) but continued throughout the entire follow-up period (up to 3 years of age); (3) the association varied across birth years without any trend; (4) the association was strongest among the Manjaco and Mancanha ethnic groups; (5) the association was most pronounced in the eastern part of the HDSS area; and (6) the association was not confounded by crowding, but was driven by the strata of children living in a household with other children under 3 years of age ( Multimedia Appendix 4 ).

Principal Findings

In this study, we aimed to discover subgroups of children with high mortality in urban Guinea-Bissau. We used complementary analyses and split the data into a hypothesis-generating cohort and a temporal validation cohort. Of children with high mortality who may be targeted for interventions, we identified (1) a residential area (area D), (2) children of mothers who did not attend prenatal consultations, and (3) children born in polygamous families during the dry season had excess mortality risk throughout the study period. Of population-wide findings, maternal education and household crowding were important factors.

Limitations

Excluding children without complete information (Figure S1 in Multimedia Appendix 1 ), and conditioning on children being alive at 6 weeks of age [ 19 ] may have limited the generalizability of our findings. The HDSS is a valuable source of information but is focused on specific key health indicators as data collection in Guinea-Bissau is resource-demanding. Thus, we lacked information about other relevant baseline characteristics such as vector-borne diseases, health care and systems, weather, pollution, water and sanitation, community relations (social capital), and household and macroeconomic conditions. We did not include factors that varied during follow-up, such as season, vaccinations, vitamins, and other health campaigns. This may be important as child mortality is considerably higher in the rainy season [ 19 , 20 ]. Live and nonlive vaccines have been shown to affect child mortality more generally than their effects on the targeted diseases explained [ 21 ].

We acknowledge that data gathered from the real world may have some natural limitations in terms of completeness and accuracy, which could potentially affect the reliability of the identified risk factors. Efforts to account for missing or incomplete data were made where feasible. Furthermore, as this study concentrates on urban Guinea-Bissau, its findings may not readily apply to different socioeconomic and cultural contexts. Further research in varied settings is necessary to validate and understand the transferability of these factors.

To acknowledge the challenges in establishing causality, we integrated multiple methods and used an inductive-deductive research methodology (ie, take the learnings from the hypothesis-generating cohort to be tested in the temporal validation cohort) [ 22 ]. This approach guided us to propose future research directions to validate and understand the mechanisms driving the observed phenomena. While total effects can be diluted in the Causes of Outcome Learning approach, particularly when including individual and birth-related factors in the model ( Figure 3 ), our methodology is strengthened by using a causal structure ( Table 1 ) for adjustments and the TMLE approach. This enhances the validity of our findings and contributes toward a more robust inference of causality by better adjustment and more robust model specification [ 17 ]. It could be explored if other novel machine learning methods could supplement the findings [ 23 ].

Interpretation

The temporal split allowed us to investigate consistency across 2 time periods. While a lack of consistency can be due to chance, it may also reflect changes in the causal structure over time. We found that children of mothers not under HDSS surveillance were strongly associated with child mortality in the hypothesis-generating cohort. However, in the temporal validation cohort, close to no children had mothers who were not under HDSS surveillance. Changes in data collection methods might explain this discrepancy; after 2013, mothers of children in new families were registered by the same data collector as the child, whereas before 2013, the mother’s registration was handled by a separate team. The increased risk among twins born in the rainy season in the hypothesis-generating cohort may have occurred by chance. Still, it could also indicate better health care for high-risk children in the validation cohort.

Local Environmental Factors

Previous spatial studies have shown large differences in disparities within and between countries [ 24 , 25 ], and temporal persistence at local levels [ 26 ]. The population movement in Bissau may have made it more difficult to identify high-risk residential areas. Area D contains a busy market called Caracol which is known for traditional medicine and care. High population density and possibly high infectious load, may offer 1 explanation for the high risk in residential area D. The proportion of mothers with less than 7 years of schooling was similar in area D as outside of it (Figure S5 in Multimedia Appendix 1 ).

To further understand and address the high mortality rate in this residential area, several future studies could be conducted, such as (1) qualitative study following families in this area may add insight and create new hypotheses; (2) network analysis to reveal contact patterns and exposed jobs most relevant in this area; and (3) spatial analysis of distance to specific proximate places (eg, places for traditional medicine and care), infrastructures (eg, wells), or potentially hazardous areas (eg, waste collection areas).

Lack of Prenatal Consultations

Prenatal consultations are designed to prevent early child mortality and may directly affect maternal behavior. The association between lack of a prenatal consultation and mortality is reflected in other studies [ 27 ], but we cannot exclude that some of the association was confounded by social and economic factors, as well as health care-seeking behavior. This may be especially important as we are considering postneonatal mortality. Various mechanisms may contribute to postneonatal mortality, such as out-of-pocket fees associated with increased child mortality in sub-Saharan Africa [ 28 ]. In Guinea-Bissau, the expansion of free antenatal care was, however, not associated with reduced perinatal mortality [ 29 ], and thus some of the observed associations of prenatal consultations may reflect confounding.

To further understand and address the lack of prenatal consultations and its impact on child mortality, a number of future studies could be conducted, such as (1) studies examining various characteristics of mothers not participating in prenatal consultation and their outcomes to understand further how this subgroup is associated with mortality and morbidity, (2) assess the effectiveness of interventions such as active home visits with prenatal consultations in reducing child mortality, and (3) explore if health care decisions during prenatal consultations can be assisted by artificial intelligence–based assessment systems [ 30 ].

Family Type and Birth Season

Connecting children of polygamous families born in the dry season, an HDSS-based study from the Gambia identified that children born in the harvest season (January-June, approximately equivalent to the dry season in Guinea-Bissau which is December-May) were at increased postneonatal mortality risk [ 31 ] and a study from Ghana found that children from polygamous families had higher child mortality than those of monogamous families [ 32 ]. We could not identify other studies assessing the combination of family type and birth season. Within our study, we further observed that the finding was not indicated to be confounded by crowding, though residual confounding may persist. However, we found that the pattern was only present for children living in households with other children under 3 years of age. Some mothers travel to rural villages to harvest cashew nuts in the late dry season and return in the rainy season. One explanation may lie in the divided attention between labor in the cashew plantations and care for other children (potentially in a different environment). With reduced or limited breastfeeding during the cashew harvest, children may lose maternal antibodies and thus become more susceptible to infections. How these mechanisms interact with family structure is still to be understood.

To further understand and address the association between birth season and family type on child mortality, several future studies could be conducted, such as (1) investigation into accidents as causes of death may reveal if the combination of shared child attendance and busy months in relation to the harvest increases the risk of domestic accidents, (2) interviews with these families may give insight into the observed phenomenon, and (3) triangulating the findings with other health-related behaviors such as vaccination uptake may help uncover mechanisms.

Resource Prioritization

As repeatedly described in the literature, social and economic factors affecting a wide part of the population strongly predict mortality [ 33 ]. In our data, social and economic factors may account for 20%-30% of all deaths in children aged 6 weeks to 3 years. In contrast, the 3 subgroups of children with high mortality identified in our study may represent a smaller fraction of the overall mortality burden (less than 5%), but they are characterized by significantly higher absolute mortality risks. This distinction raises important questions about the feasibility and potential impact of targeted interventions for these subgroups as compared to more widespread, universal public health strategies. While recognizing the challenges in reaching these smaller subgroups, targeted interventions could be crucial in addressing their disproportionately high mortality risks. Therefore, it is imperative to consider both cost-effectivity and equity in designing these interventions, ensuring they complement broader public health measures to provide comprehensive and effective child health care.

Demonstration of a Novel Approach for Targeted Public Health Research

This study not only provides insights into child mortality in urban Guinea-Bissau but also demonstrates the practical application of the Causes of Outcome Learning approach [ 14 ] on real-world data. Our findings illustrate how this approach effectively deciphers complex patterns and suggests potential synergistic causes in public health data, revealing phenomena that would be overlooked by traditional analytical methods. Future research should focus on identifying when the Causes of Outcome Learning approach is most effective and on refining the methodology to improve its accuracy and adaptability for a variety of public health research questions and study designs.

Conclusions

Reaching the Sustainable Development Goal of reducing under-5 child mortality to below 1 in 25 children by 2030 will require a range of interventions. By using several different and complementary approaches, we were able to identify subgroups of children at a high mortality risk that would not be evident otherwise. These high-risk children live in a specific area near a marked area known for traditional medicine and care, have mothers who did not attend prenatal consultations, and were born in the dry season and in polygamous families. We have suggested several future studies that may help explore these hypotheses. Potential targeted interventions should be evaluated in comparison with the impact of population-wide structural interventions both from cost-effectivity aspects and equity aspects and tested under proper evaluation schemes [ 34 ] to reduce child mortality.

Acknowledgments

The study would not have been possible without the dedicated work of the many data collectors, supervisors, and mothers of children under surveillance who were willing to provide answers to the questions. AR was supported by an International Postdoctoral Grant (9034-00006B) from the Independent Research Fund Denmark. HA was supported by Novo Nordisk Foundation Challenge Programme (NNF17OC0027812) and by National Institutes of Health (UL1TR004419 and P30ES023515). SWM was supported by a DFF-International Postdoctoral Grant (0164-00023B) from the Independent Research Fund Denmark. ABF was supported by an Ascending investigator grant from Lundbeck Foundation (R313-2019-635) and a Sapere Aude grant from Independent Research Fund Denmark (9060-00018B). Many funders supported the data collection at Bandim Health Project over the years (full list available [ 35 ]).

Authors' Contributions

The study was based on secondary analyses of data collected at the Bandim Health Project under the supervision of SWM, IBS, PA, and ABF. All authors contributed to the study concept and design. AR, SN, and ABF extracted and verified the data. AR is responsible for all the data analyses. AR, HA, and SWM contributed to the spatial analyses. AR, PD, WS, NHR, and CTE contributed to the implementation of the Causes of Outcome Learning approach. All authors contributed to the interpretation of the results. AR, SN, and ABF contributed to drafting the study. All authors critically revised the study for important intellectual content.

Conflicts of Interest

None declared.

The variable operationalisation, flowchart, pairwise associations between basic HDSS information, temporal overview, geo-spatial patterns of the temporal validation data adjusting for a linear effect of calendar time, and geo-spatial patterns of the prevalence of mothers with fewer than 7 years of schooling.

The inverse probability of censoring weights.

The Causes of Outcome Learning method.

The analysis regarding the birth season and polygamous families.

• Child mortality, stillbirth, and causes of death estimates. IGME. URL: https://childmortality.org/data/Guinea-Bissau [accessed 2022-06-22]
• Dowell SF, Zaidi A, Heaton P. Why child health and mortality prevention surveillance? Clin Infect Dis. 2019;69(Suppl 4):S260-S261. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hay SI. Maintaining progress for the most beautiful chart in the world. Int Health. 2019;11(5):344-348. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Horton R. Offline: in defence of precision public health. Lancet. 2018;392(10157):1504. [ CrossRef ] [ Medline ]
• Olstad DL, McIntyre L. Reconceptualising precision public health. BMJ Open. 2019;9(9):e030279. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rose GA. Strategy of Preventive Medicine. Oxford. Oxford University Press; 1994.
• Sankoh O, Byass P. The INDEPTH network: filling vital gaps in global epidemiology. Int J Epidemiol. 2012;41(3):579-588. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Yanai I, Lercher M. A hypothesis is a liability. Genome Biol. 2020;21(1):231. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Bandim Health Project: a health and demographic surveillance system site situated in Guinea-Bissau, West Africa. Bandim Health Project. URL: https://www.bandim.org/ [accessed 2023-12-14]
• Fisker AB, Bale C, Rodrigues A, Balde I, Fernandes M, Jørgensen MJ, et al. High-dose vitamin A with vaccination after 6 months of age: a randomized trial. Pediatrics. 2014;134(3):e739-e748. [ CrossRef ] [ Medline ]
• Martins CL, Benn CS, Andersen A, Balé C, Schaltz-Buchholzer F, Do VA, et al. A randomized trial of a standard dose of Edmonston-Zagreb measles vaccine given at 4.5 months of age: effect on total hospital admissions. J Infect Dis. 2014;209(11):1731-1738. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Hernán MA, Robins JM. Causal Inference: What If. Boca Raton. Chapman & Hall/CRC; 2020.
• Nielsen S, Fisker AB, da Silva I, Byberg S, Biering-Sørensen S, Balé C, et al. Effect of early two-dose measles vaccination on childhood mortality and modification by maternal measles antibody in Guinea-Bissau, West Africa: a single-centre open-label randomised controlled trial. EClinicalMedicine. 2022;49:101467. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rieckmann A, Dworzynski P, Arras L, Lapuschkin S, Samek W, Arah OA, et al. Causes of outcome learning: a causal inference-inspired machine learning approach to disentangling common combinations of potential causes of a health outcome. Int J Epidemiol. 2022;51(5):1622-1636. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rieckmann A, Dworzynski P, Arras L, Ekstrøm CT. CoOL: Causes of Outcome Learning. CRAN. URL: https://cran.r-project.org/package=CoOL [accessed 2022-05-27]
• VanderWeele TJ, Knol MJ. A tutorial on interaction. Epidemiol Methods. 2014;3(1):33-72. [ FREE Full text ] [ CrossRef ]
• van der Laan MJ, Rubin D. Targeted maximum likelihood learning. Int J Biostat. 2006;2(1):11. [ FREE Full text ] [ CrossRef ]
• Mansournia MA, Altman DG. Population attributable fraction. BMJ. 2018;360:k757. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Nielsen BU, Byberg S, Aaby P, Rodrigues A, Benn CS, Fisker AB. Seasonal variation in child mortality in rural Guinea-Bissau. Trop Med Int Health. 2017;22(7):846-856. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Veirum JE, Biai S, Jakobsen M, Sandström A, Hedegaard K, Kofoed PE, et al. Persisting high hospital and community childhood mortality in an urban setting in Guinea-Bissau. Acta Paediatr. 2007;96(10):1526-1530. [ CrossRef ] [ Medline ]
• Benn CS, Fisker AB, Rieckmann A, Sørup S, Aaby P. Vaccinology: time to change the paradigm? Lancet Infect Dis. 2020;20(10):e274-e283. [ CrossRef ] [ Medline ]
• Shu X, Ye Y. Knowledge discovery: methods from data mining and machine learning. Soc Sci Res. 2023;110:102817. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Qiu W, Chen H, Dincer AB, Lundberg S, Kaeberlein M, Lee SI. Interpretable machine learning prediction of all-cause mortality. Commun Med (Lond). 2022;2:125. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Golding N, Burstein R, Longbottom J, Browne AJ, Fullman N, Osgood-Zimmerman A, et al. Mapping under-5 and neonatal mortality in Africa, 2000-15: a baseline analysis for the Sustainable Development Goals. Lancet. 2017;390(10108):2171-2182. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Wagner Z, Heft-Neal S, Bhutta ZA, Black RE, Burke M, Bendavid E. Armed conflict and child mortality in Africa: a geospatial analysis. Lancet. 2018;392(10150):857-865. [ CrossRef ]
• Sartorius B, Kahn K, Collinson MA, Vounatsou P, Tollman SM. Survived infancy but still vulnerable: spatial-temporal trends and risk factors for child mortality in the Agincourt rural sub-district, South Africa, 1992-2007. Geospat Health. 2011;5(2):285-295. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rocha TAH, de Thomaz EBAF, de Almeida DG, da Silva NC, Queiroz RCDS, Andrade L, et al. Data-driven risk stratification for preterm birth in Brazil: a population-based study to develop of a machine learning risk assessment approach. Lancet Reg Health Am. 2021;3:100053. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Simmons RA, Anthopolos R, O'Meara WP. Effect of health systems context on infant and child mortality in sub-Saharan Africa from 1995 to 2015, a longitudinal cohort analysis. Sci Rep. Aug 11, 2021;11(1):16263. [ CrossRef ] [ Medline ]
• Damerow SM, Lundgren VM, Dunga MJ, Martins JSD, Adrian HV, Jensen AM, et al. 155:oral monitoring the impact of health system strengthening for maternal and child health in Guinea-Bissau: focus on universal health coverage removes focus from stagnating perinatal mortality. BMJ Glob Health. 2022;7(Suppl 2):A34-A35. [ FREE Full text ] [ CrossRef ]
• Schmude M, Salim N, Azadzoy H, Bane M, Millen E, O'Donnell L, et al. Investigating the potential for clinical decision support in sub-Saharan Africa with AFYA (artificial intelligence-based assessment of health symptoms in Tanzania): protocol for a prospective, observational pilot study. JMIR Res Protoc. 2022;11(6):e34298. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Jarde A, Mohammed NI, Gomez P, Saine PC, D'Alessandro U, Roca A. Risk factors of infant mortality in rural the Gambia: a retrospective cohort study. BMJ Paediatr Open. 2021;5(1):e001190. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Kanmiki EW, Bawah AA, Agorinya I, Achana FS, Awoonor-Williams JK, Oduro AR, et al. Socio-economic and demographic determinants of under-five mortality in rural northern Ghana. BMC Int Health Hum Rights. 2014;14:24. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Marmot M. Social determinants of health inequalities. Lancet. 2005;365(9464):1099-1104. [ FREE Full text ] [ CrossRef ] [ Medline ]
• Rieckmann A, Benn CS. The importance of randomised vs non-randomised trials. Lancet. 2019;394(10199):634. [ CrossRef ] [ Medline ]
• Bandim Health Project. Bandim Health Project. URL: https://www.bandim.org [accessed 2024-03-16]

Abbreviations

Edited by C Argyropoulos; submitted 18.04.23; peer-reviewed by H Nguyen, E Vashishtha; comments to author 10.12.23; revised version received 22.12.23; accepted 23.01.24; published 09.04.24.

©Andreas Rieckmann, Sebastian Nielsen, Piotr Dworzynski, Heresh Amini, Søren Wengel Mogensen, Isaquel Bartolomeu Silva, Angela Y Chang, Onyebuchi A Arah, Wojciech Samek, Naja Hulvej Rod, Claus Thorn Ekstrøm, Christine Stabell Benn, Peter Aaby, Ane Bærent Fisker. Originally published in JMIR Public Health and Surveillance (https://publichealth.jmir.org), 09.04.2024.

This is an open-access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided the original work, first published in JMIR Public Health and Surveillance, is properly cited. The complete bibliographic information, a link to the original publication on https://publichealth.jmir.org, as well as this copyright and license information must be included.