research papers on recommender systems

• Survey paper
• Open access
• Published: 03 May 2022

A systematic review and research perspective on recommender systems

• Deepjyoti Roy   ORCID: orcid.org/0000-0002-8020-7145 1 &
• Mala Dutta 1  

Journal of Big Data volume  9 , Article number:  59 ( 2022 ) Cite this article

78k Accesses

139 Citations

9 Altmetric

Metrics details

Recommender systems are efficient tools for filtering online information, which is widespread owing to the changing habits of computer users, personalization trends, and emerging access to the internet. Even though the recent recommender systems are eminent in giving precise recommendations, they suffer from various limitations and challenges like scalability, cold-start, sparsity, etc. Due to the existence of various techniques, the selection of techniques becomes a complex work while building application-focused recommender systems. In addition, each technique comes with its own set of features, advantages and disadvantages which raises even more questions, which should be addressed. This paper aims to undergo a systematic review on various recent contributions in the domain of recommender systems, focusing on diverse applications like books, movies, products, etc. Initially, the various applications of each recommender system are analysed. Then, the algorithmic analysis on various recommender systems is performed and a taxonomy is framed that accounts for various components required for developing an effective recommender system. In addition, the datasets gathered, simulation platform, and performance metrics focused on each contribution are evaluated and noted. Finally, this review provides a much-needed overview of the current state of research in this field and points out the existing gaps and challenges to help posterity in developing an efficient recommender system.

Introduction

The recent advancements in technology along with the prevalence of online services has offered more abilities for accessing a huge amount of online information in a faster manner. Users can post reviews, comments, and ratings for various types of services and products available online. However, the recent advancements in pervasive computing have resulted in an online data overload problem. This data overload complicates the process of finding relevant and useful content over the internet. The recent establishment of several procedures having lower computational requirements can however guide users to the relevant content in a much easy and fast manner. Because of this, the development of recommender systems has recently gained significant attention. In general, recommender systems act as information filtering tools, offering users suitable and personalized content or information. Recommender systems primarily aim to reduce the user’s effort and time required for searching relevant information over the internet.

Nowadays, recommender systems are being increasingly used for a large number of applications such as web [ 1 , 67 , 70 ], books [ 2 ], e-learning [ 4 , 16 , 61 ], tourism [ 5 , 8 , 78 ], movies [ 66 ], music [ 79 ], e-commerce, news, specialized research resources [ 65 ], television programs [ 72 , 81 ], etc. It is therefore important to build high-quality and exclusive recommender systems for providing personalized recommendations to the users in various applications. Despite the various advances in recommender systems, the present generation of recommender systems requires further improvements to provide more efficient recommendations applicable to a broader range of applications. More investigation of the existing latest works on recommender systems is required which focus on diverse applications.

There is hardly any review paper that has categorically synthesized and reviewed the literature of all the classification fields and application domains of recommender systems. The few existing literature reviews in the field cover just a fraction of the articles or focus only on selected aspects such as system evaluation. Thus, they do not provide an overview of the application field, algorithmic categorization, or identify the most promising approaches. Also, review papers often neglect to analyze the dataset description and the simulation platforms used. This paper aims to fulfil this significant gap by reviewing and comparing existing articles on recommender systems based on a defined classification framework, their algorithmic categorization, simulation platforms used, applications focused, their features and challenges, dataset description and system performance. Finally, we provide researchers and practitioners with insight into the most promising directions for further investigation in the field of recommender systems under various applications.

In essence, recommender systems deal with two entities—users and items, where each user gives a rating (or preference value) to an item (or product). User ratings are generally collected by using implicit or explicit methods. Implicit ratings are collected indirectly from the user through the user’s interaction with the items. Explicit ratings, on the other hand, are given directly by the user by picking a value on some finite scale of points or labelled interval values. For example, a website may obtain implicit ratings for different items based on clickstream data or from the amount of time a user spends on a webpage and so on. Most recommender systems gather user ratings through both explicit and implicit methods. These feedbacks or ratings provided by the user are arranged in a user-item matrix called the utility matrix as presented in Table 1 .

The utility matrix often contains many missing values. The problem of recommender systems is mainly focused on finding the values which are missing in the utility matrix. This task is often difficult as the initial matrix is usually very sparse because users generally tend to rate only a small number of items. It may also be noted that we are interested in only the high user ratings because only such items would be suggested back to the users. The efficiency of a recommender system greatly depends on the type of algorithm used and the nature of the data source—which may be contextual, textual, visual etc.

Types of recommender systems

Recommender systems are broadly categorized into three different types viz. content-based recommender systems, collaborative recommender systems and hybrid recommender systems. A diagrammatic representation of the different types of recommender systems is given in Fig.  1 .

Content-based recommender system

In content-based recommender systems, all the data items are collected into different item profiles based on their description or features. For example, in the case of a book, the features will be author, publisher, etc. In the case of a movie, the features will be the movie director, actor, etc. When a user gives a positive rating to an item, then the other items present in that item profile are aggregated together to build a user profile. This user profile combines all the item profiles, whose items are rated positively by the user. Items present in this user profile are then recommended to the user, as shown in Fig.  2 .

One drawback of this approach is that it demands in-depth knowledge of the item features for an accurate recommendation. This knowledge or information may not be always available for all items. Also, this approach has limited capacity to expand on the users' existing choices or interests. However, this approach has many advantages. As user preferences tend to change with time, this approach has the quick capability of dynamically adapting itself to the changing user preferences. Since one user profile is specific only to that user, this algorithm does not require the profile details of any other users because they provide no influence in the recommendation process. This ensures the security and privacy of user data. If new items have sufficient description, content-based techniques can overcome the cold-start problem i.e., this technique can recommend an item even when that item has not been previously rated by any user. Content-based filtering approaches are more common in systems like personalized news recommender systems, publications, web pages recommender systems, etc.

Collaborative filtering-based recommender system

Collaborative approaches make use of the measure of similarity between users. This technique starts with finding a group or collection of user X whose preferences, likes, and dislikes are similar to that of user A. X is called the neighbourhood of A. The new items which are liked by most of the users in X are then recommended to user A. The efficiency of a collaborative algorithm depends on how accurately the algorithm can find the neighbourhood of the target user. Traditionally collaborative filtering-based systems suffer from the cold-start problem and privacy concerns as there is a need to share user data. However, collaborative filtering approaches do not require any knowledge of item features for generating a recommendation. Also, this approach can help to expand on the user’s existing interests by discovering new items. Collaborative approaches are again divided into two types: memory-based approaches and model-based approaches.

Memory-based collaborative approaches recommend new items by taking into consideration the preferences of its neighbourhood. They make use of the utility matrix directly for prediction. In this approach, the first step is to build a model. The model is equal to a function that takes the utility matrix as input.

Model = f (utility matrix)

Then recommendations are made based on a function that takes the model and user profile as input. Here we can make recommendations only to users whose user profile belongs to the utility matrix. Therefore, to make recommendations for a new user, the user profile must be added to the utility matrix, and the similarity matrix should be recomputed, which makes this technique computation heavy.

Recommendation = f (defined model, user profile) where user profile  ∈  utility matrix

Memory-based collaborative approaches are again sub-divided into two types: user-based collaborative filtering and item-based collaborative filtering. In the user-based approach, the user rating of a new item is calculated by finding other users from the user neighbourhood who has previously rated that same item. If a new item receives positive ratings from the user neighbourhood, the new item is recommended to the user. Figure  3 depicts the user-based filtering approach.

User-based collaborative filtering

In the item-based approach, an item-neighbourhood is built consisting of all similar items which the user has rated previously. Then that user’s rating for a different new item is predicted by calculating the weighted average of all ratings present in a similar item-neighbourhood as shown in Fig.  4 .

Item-based collaborative filtering

Model-based systems use various data mining and machine learning algorithms to develop a model for predicting the user’s rating for an unrated item. They do not rely on the complete dataset when recommendations are computed but extract features from the dataset to compute a model. Hence the name, model-based technique. These techniques also need two steps for prediction—the first step is to build the model, and the second step is to predict ratings using a function (f) which takes the model defined in the first step and the user profile as input.

Recommendation = f (defined model, user profile) where user profile  ∉  utility matrix

Model-based techniques do not require adding the user profile of a new user into the utility matrix before making predictions. We can make recommendations even to users that are not present in the model. Model-based systems are more efficient for group recommendations. They can quickly recommend a group of items by using the pre-trained model. The accuracy of this technique largely relies on the efficiency of the underlying learning algorithm used to create the model. Model-based techniques are capable of solving some traditional problems of recommender systems such as sparsity and scalability by employing dimensionality reduction techniques [ 86 ] and model learning techniques.

Hybrid filtering

A hybrid technique is an aggregation of two or more techniques employed together for addressing the limitations of individual recommender techniques. The incorporation of different techniques can be performed in various ways. A hybrid algorithm may incorporate the results achieved from separate techniques, or it can use content-based filtering in a collaborative method or use a collaborative filtering technique in a content-based method. This hybrid incorporation of different techniques generally results in increased performance and increased accuracy in many recommender applications. Some of the hybridization approaches are meta-level, feature-augmentation, feature-combination, mixed hybridization, cascade hybridization, switching hybridization and weighted hybridization [ 86 ]. Table 2 describes these approaches.

Recommender system challenges

This section briefly describes the various challenges present in current recommender systems and offers different solutions to overcome these challenges.

Cold start problem

The cold start problem appears when the recommender system cannot draw any inference from the existing data, which is insufficient. Cold start refers to a condition when the system cannot produce efficient recommendations for the cold (or new) users who have not rated any item or have rated a very few items. It generally arises when a new user enters the system or new items (or products) are inserted into the database. Some solutions to this problem are as follows: (a) Ask new users to explicitly mention their item preference. (b) Ask a new user to rate some items at the beginning. (c) Collect demographic information (or meta-data) from the user and recommend items accordingly.

Shilling attack problem

This problem arises when a malicious user fakes his identity and enters the system to give false item ratings [ 87 ]. Such a situation occurs when the malicious user wants to either increase or decrease some item’s popularity by causing a bias on selected target items. Shilling attacks greatly reduce the reliability of the system. One solution to this problem is to detect the attackers quickly and remove the fake ratings and fake user profiles from the system.

Synonymy problem

This problem arises when similar or related items have different entries or names, or when the same item is represented by two or more names in the system [ 78 ]. For example, babywear and baby cloth. Many recommender systems fail to distinguish these differences, hence reducing their recommendation accuracy. To alleviate this problem many methods are used such as demographic filtering, automatic term expansion and Singular Value Decomposition [ 76 ].

Latency problem

The latency problem is specific to collaborative filtering approaches and occurs when new items are frequently inserted into the database. This problem is characterized by the system’s failure to recommend new items. This happens because new items must be reviewed before they can be recommended in a collaborative filtering environment. Using content-based filtering may resolve this issue, but it may introduce overspecialization and decrease the computing time and system performance. To increase performance, the calculations can be done in an offline environment and clustering-based techniques can be used [ 76 ].

Sparsity problem

Data sparsity is a common problem in large scale data analysis, which arises when certain expected values are missing in the dataset. In the case of recommender systems, this situation occurs when the active users rate very few items. This reduces the recommendation accuracy. To alleviate this problem several techniques can be used such as demographic filtering, singular value decomposition and using model-based collaborative techniques.

Grey sheep problem

The grey sheep problem is specific to pure collaborative filtering approaches where the feedback given by one user do not match any user neighbourhood. In this situation, the system fails to accurately predict relevant items for that user. This problem can be resolved by using pure content-based approaches where predictions are made based on the user’s profile and item properties.

Scalability problem

Recommender systems, especially those employing collaborative filtering techniques, require large amounts of training data, which cause scalability problems. The scalability problem arises when the amount of data used as input to a recommender system increases quickly. In this era of big data, more and more items and users are rapidly getting added to the system and this problem is becoming common in recommender systems. Two common approaches used to solve the scalability problem is dimensionality reduction and using clustering-based techniques to find users in tiny clusters instead of the complete database.

Methodology

The purpose of this study is to understand the research trends in the field of recommender systems. The nature of research in recommender systems is such that it is difficult to confine each paper to a specific discipline. This can be further understood by the fact that research papers on recommender systems are scattered across various journals such as computer science, management, marketing, information technology and information science. Hence, this literature review is conducted over a wide range of electronic journals and research databases such as ACM Portal, IEEE/IEE Library, Google Scholars and Science Direct [ 88 ].

The search process of online research articles was performed based on 6 descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*”, “Hybrid Recommend*”. The following research papers described below were excluded from our research:

News articles.

Master’s dissertations.

Non-English papers.

Unpublished papers.

Research papers published before 2011.

We have screened a total of 350 articles based on their abstracts and content. However, only research papers that described how recommender systems can be applied were chosen. Finally, 60 papers were selected from top international journals indexed in Scopus or E-SCI in 2021. We now present the PRISMA flowchart of the inclusion and exclusion process in Fig.  5 .

PRISMA flowchart of the inclusion and exclusion process. Abstract and content not suitable to the study: * The use or application of the recommender system is not specified: **

Each paper was carefully reviewed and classified into 6 categories in the application fields and 3 categories in the techniques used to develop the system. The classification framework is presented in Fig.  6 .

Classification framework

The number of relevant articles come from Expert Systems with Applications (23%), followed by IEEE (17%), Knowledge-Based System (17%) and Others (43%). Table 3 depicts the article distribution by journal title and Table 4 depicts the sector-wise article distribution.

Both forward and backward searching techniques were implemented to establish that the review of 60 chosen articles can represent the domain literature. Hence, this paper can demonstrate its validity and reliability as a literature review.

Review on state-of-the-art recommender systems

This section presents a state-of-art literature review followed by a chronological review of the various existing recommender systems.

Literature review

In 2011, Castellano et al. [ 1 ] developed a “NEuro-fuzzy WEb Recommendation (NEWER)” system for exploiting the possibility of combining computational intelligence and user preference for suggesting interesting web pages to the user in a dynamic environment. It considered a set of fuzzy rules to express the correlations between user relevance and categories of pages. Crespo et al. [ 2 ] presented a recommender system for distance education over internet. It aims to recommend e-books to students using data from user interaction. The system was developed using a collaborative approach and focused on solving the data overload problem in big digital content. Lin et al. [ 3 ] have put forward a recommender system for automatic vending machines using Genetic algorithm (GA), k-means, Decision Tree (DT) and Bayesian Network (BN). It aimed at recommending localized products by developing a hybrid model combining statistical methods, classification methods, clustering methods, and meta-heuristic methods. Wang and Wu [ 4 ] have implemented a ubiquitous learning system for providing personalized learning assistance to the learners by combining the recommendation algorithm with a context-aware technique. It employed the Association Rule Mining (ARM) technique and aimed to increase the effectiveness of the learner’s learning. García-Crespo et al. [ 5 ] presented a “semantic hotel” recommender system by considering the experiences of consumers using a fuzzy logic approach. The system considered both hotel and customer characteristics. Dong et al. [ 6 ] proposed a structure for a service-concept recommender system using a semantic similarity model by integrating the techniques from the view of an ontology structure-oriented metric and a concept content-oriented metric. The system was able to deliver optimal performance when compared with similar recommender systems. Li et al. [ 7 ] developed a Fuzzy linguistic modelling-based recommender system for assisting users to find experts in knowledge management systems. The developed system was applied to the aircraft industry where it demonstrated efficient and feasible performance. Lorenzi et al. [ 8 ] presented an “assumption-based multiagent” system to make travel package recommendations using user preferences in the tourism industry. It performed different tasks like discovering, filtering, and integrating specific information for building a travel package following the user requirement. Huang et al. [ 9 ] proposed a context-aware recommender system through the extraction, evaluation and incorporation of contextual information gathered using the collaborative filtering and rough set model.

In 2012, Chen et al. [ 10 ] presented a diabetes medication recommender model by using “Semantic Web Rule Language (SWRL) and Java Expert System Shell (JESS)” for aggregating suitable prescriptions for the patients. It aimed at selecting the most suitable drugs from the list of specific drugs. Mohanraj et al. [ 11 ] developed the “Ontology-driven bee’s foraging approach (ODBFA)” to accurately predict the online navigations most likely to be visited by a user. The self-adaptive system is intended to capture the various requirements of the online user by using a scoring technique and by performing a similarity comparison. Hsu et al. [ 12 ] proposed a “personalized auxiliary material” recommender system by considering the specific course topics, individual learning styles, complexity of the auxiliary materials using an artificial bee colony algorithm. Gemmell et al. [ 13 ] demonstrated a solution for the problem of resource recommendation in social annotation systems. The model was developed using a linear-weighted hybrid method which was capable of providing recommendations under different constraints. Choi et al. [ 14 ] proposed one “Hybrid Online-Product rEcommendation (HOPE) system” by the integration of collaborative filtering through sequential pattern analysis-based recommendations and implicit ratings. Garibaldi et al. [ 15 ] put forward a technique for incorporating the variability in a fuzzy inference model by using non-stationary fuzzy sets for replicating the variabilities of a human. This model was applied to a decision problem for treatment recommendations of post-operative breast cancer.

In 2013, Salehi and Kmalabadi [ 16 ] proposed an e-learning material recommender system by “modelling of materials in a multidimensional space of material’s attribute”. It employed both content and collaborative filtering. Aher and Lobo [ 17 ] introduced a course recommender system using data mining techniques such as simple K-means clustering and Association Rule Mining (ARM) algorithm. The proposed e-learning system was successfully demonstrated for “MOOC (Massively Open Online Courses)”. Kardan and Ebrahimi [ 18 ] developed a hybrid recommender system for recommending posts in asynchronous discussion groups. The system was built combining both collaborative filtering and content-based filtering. It considered implicit user data to compute the user similarity with various groups, for recommending suitable posts and contents to its users. Chang et al. [ 19 ] adopted a cloud computing technology for building a TV program recommender system. The system designed for digital TV programs was implemented using Hadoop Fair Scheduler (HFC), K-means clustering and k-nearest neighbour (KNN) algorithms. It was successful in processing huge amounts of real-time user data. Lucas et al. [ 20 ] implemented a recommender model for assisting a tourism application by using associative classification and fuzzy logic to predict the context. Niu et al. [ 21 ] introduced “Affivir: An Affect-based Internet Video Recommendation System” which was developed by calculating user preferences and by using spectral clustering. This model recommended videos with similar effects, which was processed to get optimal results with dynamic adjustments of recommendation constraints.

In 2014, Liu et al. [ 22 ] implemented a new route recommendation model for offering personalized and real-time route recommendations for self-driven tourists to minimize the queuing time and traffic jams infamous tourist places. Recommendations were carried out by considering the preferences of users. Bakshi et al. [ 23 ] proposed an unsupervised learning-based recommender model for solving the scalability problem of recommender systems. The algorithm used transitive similarities along with Particle Swarm Optimization (PSO) technique for discovering the global neighbours. Kim and Shim [ 24 ] proposed a recommender system based on “latent Dirichlet allocation using probabilistic modelling for Twitter” that could recommend the top-K tweets for a user to read, and the top-K users to follow. The model parameters were learned from an inference technique by using the differential Expectation–Maximization (EM) algorithm. Wang et al. [ 25 ] developed a hybrid-movie recommender model by aggregating a genetic algorithm (GA) with improved K-means and Principal Component Analysis (PCA) technique. It was able to offer intelligent movie recommendations with personalized suggestions. Kolomvatsos et al. [ 26 ] proposed a recommender system by considering an optimal stopping theory for delivering books or music recommendations to the users. Gottschlich et al. [ 27 ] proposed a decision support system for stock investment recommendations. It computed the output by considering the overall crowd’s recommendations. Torshizi et al. [ 28 ] have introduced a hybrid recommender system to determine the severity level of a medical condition. It could recommend suitable therapies for patients suffering from Benign Prostatic Hyperplasia.

In 2015, Zahálka et al. [ 29 ] proposed a venue recommender: “City Melange”. It was an interactive content-based model which used the convolutional deep-net features of the visual domain and the linear Support Vector Machine (SVM) model to capture the semantic information and extract latent topics. Sankar et al. [ 30 ] have proposed a stock recommender system based on the stock holding portfolio of trusted mutual funds. The system employed the collaborative filtering approach along with social network analysis for offering a decision support system to build a trust-based recommendation model. Chen et al. [ 31 ] have put forward a novel movie recommender system by applying the “artificial immune network to collaborative filtering” technique. It computed the affinity of an antigen and the affinity between an antibody and antigen. Based on this computation a similarity estimation formula was introduced which was used for the movie recommendation process. Wu et al. [ 32 ] have examined the technique of data fusion for increasing the efficiency of item recommender systems. It employed a hybrid linear combination model and used a collaborative tagging system. Yeh and Cheng [ 33 ] have proposed a recommender system for tourist attractions by constructing the “elicitation mechanism using the Delphi panel method and matrix construction mechanism using the repertory grids”, which was developed by considering the user preference and expert knowledge.

In 2016, Liao et al. [ 34 ] proposed a recommender model for online customers using a rough set association rule. The model computed the probable behavioural variations of online consumers and provided product category recommendations for e-commerce platforms. Li et al. [ 35 ] have suggested a movie recommender system based on user feedback collected from microblogs and social networks. It employed the sentiment-aware association rule mining algorithm for recommendations using the prior information of frequent program patterns, program metadata similarity and program view logs. Wu et al. [ 36 ] have developed a recommender system for social media platforms by aggregating the technique of Social Matrix Factorization (SMF) and Collaborative Topic Regression (CTR). The model was able to compute the ratings of users to items for making recommendations. For improving the recommendation quality, it gathered information from multiple sources such as item properties, social networks, feedback, etc. Adeniyi et al. [ 37 ] put forward a study of automated web-usage data mining and developed a recommender system that was tested in both real-time and online for identifying the visitor’s or client’s clickstream data.

In 2017, Rawat and Kankanhalli [ 38 ] have proposed a viewpoint recommender system called “ClickSmart” for assisting mobile users to capture high-quality photographs at famous tourist places. Yang et al. [ 39 ] proposed a gradient boosting-based job recommendation system for satisfying the cost-sensitive requirements of the users. The hybrid algorithm aimed to reduce the rate of unnecessary job recommendations. Lee et al. [ 40 ] proposed a music streaming recommender system based on smartphone activity usage. The proposed system benefitted by using feature selection approaches with machine learning techniques such as Naive Bayes (NB), Support Vector Machine (SVM), Multi-layer Perception (MLP), Instance-based k -Nearest Neighbour (IBK), and Random Forest (RF) for performing the activity detection from the mobile signals. Wei et al. [ 41 ] have proposed a new stacked denoising autoencoder (SDAE) based recommender system for cold items. The algorithm employed deep learning and collaborative filtering method to predict the unknown ratings.

In 2018, Li et al. [ 42 ] have developed a recommendation algorithm using Weighted Linear Regression Models (WLRRS). The proposed system was put to experiment using the MovieLens dataset and it presented better classification and predictive accuracy. Mezei and Nikou [ 43 ] presented a mobile health and wellness recommender system based on fuzzy optimization. It could recommend a collection of actions to be taken by the user to improve the user’s health condition. Recommendations were made considering the user’s physical activities and preferences. Ayata et al. [ 44 ] proposed a music recommendation model based on the user emotions captured through wearable physiological sensors. The emotion detection algorithm employed different machine learning algorithms like SVM, RF, KNN and decision tree (DT) algorithms to predict the emotions from the changing electrical signals gathered from the wearable sensors. Zhao et al. [ 45 ] developed a multimodal learning-based, social-aware movie recommender system. The model was able to successfully resolve the sparsity problem of recommender systems. The algorithm developed a heterogeneous network by exploiting the movie-poster image and textual description of each movie based on the social relationships and user ratings.

In 2019, Hammou et al. [ 46 ] proposed a Big Data recommendation algorithm capable of handling large scale data. The system employed random forest and matrix factorization through a data partitioning scheme. It was then used for generating recommendations based on user rating and preference for each item. The proposed system outperformed existing systems in terms of accuracy and speed. Zhao et al. [ 47 ] have put forward a hybrid initialization method for social network recommender systems. The algorithm employed denoising autoencoder (DAE) neural network-based initialization method (ANNInit) and attribute mapping. Bhaskaran and Santhi [ 48 ] have developed a hybrid, trust-based e-learning recommender system using cloud computing. The proposed algorithm was capable of learning online user activities by using the Firefly Algorithm (FA) and K-means clustering. Afolabi and Toivanen [ 59 ] have suggested an integrated recommender model based on collaborative filtering. The proposed model “Connected Health for Effective Management of Chronic Diseases”, aimed for integrating recommender systems for better decision-making in the process of disease management. He et al. [ 60 ] proposed a movie recommender system called “HI2Rec” which explored the usage of collaborative filtering and heterogeneous information for making movie recommendations. The model used the knowledge representation learning approach to embed movie-related information gathered from different sources.

In 2020, Han et al. [ 49 ] have proposed one Internet of Things (IoT)-based cancer rehabilitation recommendation system using the Beetle Antennae Search (BAS) algorithm. It presented the patients with a solution for the problem of optimal nutrition program by considering the objective function as the recurrence time. Kang et al. [ 50 ] have presented a recommender system for personalized advertisements in Online Broadcasting based on a tree model. Recommendations were generated in real-time by considering the user preferences to minimize the overhead of preference prediction and using a HashMap along with the tree characteristics. Ullah et al. [ 51 ] have implemented an image-based service recommendation model for online shopping based random forest and Convolutional Neural Networks (CNN). The model used JPEG coefficients to achieve an accurate prediction rate. Cai et al. [ 52 ] proposed a new hybrid recommender model using a many-objective evolutionary algorithm (MaOEA). The proposed algorithm was successful in optimizing the novelty, diversity, and accuracy of recommendations. Esteban et al. [ 53 ] have implemented a hybrid multi-criteria recommendation system concerned with students’ academic performance, personal interests, and course selection. The system was developed using a Genetic Algorithm (GA) and aimed at helping university students. It combined both course information and student information for increasing system performance and the reliability of the recommendations. Mondal et al. [ 54 ] have built a multilayer, graph data model-based doctor recommendation system by exploiting the trust concept between a patient-doctor relationship. The proposed system showed good results in practical applications.

In 2021, Dhelim et al. [ 55 ] have developed a personality-based product recommending model using the techniques of meta path discovery and user interest mining. This model showed better results when compared to session-based and deep learning models. Bhalse et al. [ 56 ] proposed a web-based movie recommendation system based on collaborative filtering using Singular Value Decomposition (SVD), collaborative filtering and cosine similarity (CS) for addressing the sparsity problem of recommender systems. It suggested a recommendation list by considering the content information of movies. Similarly, to solve both sparsity and cold-start problems Ke et al. [ 57 ] proposed a dynamic goods recommendation system based on reinforcement learning. The proposed system was capable of learning from the reduced entropy loss error on real-time applications. Chen et al. [ 58 ] have presented a movie recommender model combining various techniques like user interest with category-level representation, neighbour-assisted representation, user interest with latent representation and item-level representation using Feed-forward Neural Network (FNN).

Comparative chronological review

A comparative chronological review to compare the total contributions on various recommender systems in the past 10 years is given in Fig.  7 .

Comparative chronological review of recommender systems under diverse applications

This review puts forward a comparison of the number of research works proposed in the domain of recommender systems from the year 2011 to 2021 using various deep learning and machine learning-based approaches. Research articles are categorized based on the recommender system classification framework as shown in Table 5 . The articles are ordered according to their year of publication. There are two key concepts: Application fields and techniques used. The application fields of recommender systems are divided into six different fields, viz. entertainment, health, tourism, web/e-commerce, education and social media/others.

Algorithmic categorization, simulation platforms and applications considered for various recommender systems

This section analyses different methods like deep learning, machine learning, clustering and meta-heuristic-based-approaches used in the development of recommender systems. The algorithmic categorization of different recommender systems is given in Fig.  8 .

Algorithmic categorization of different recommender systems

Categorization is done based on content-based, collaborative filtering-based, and optimization-based approaches. In [ 8 ], a content-based filtering technique was employed for increasing the ability to trust other agents and for improving the exchange of information by trust degree. In [ 16 ], it was applied to enhance the quality of recommendations using the account attributes of the material. It achieved better performance concerning with F1-score, recall and precision. In [ 18 ], this technique was able to capture the implicit user feedback, increasing the overall accuracy of the proposed model. The content-based filtering in [ 30 ] was able to increase the accuracy and performance of a stock recommender system by using the “trust factor” for making decisions.

Different collaborative filtering approaches are utilized in recent studies, which are categorized as follows:

Model-based techniques

Neuro-Fuzzy [ 1 ] based technique helps in discovering the association between user categories and item relevance. It is also simple to understand. K-Means Clustering [ 2 , 19 , 25 , 48 ] is efficient for large scale datasets. It is simple to implement and gives a fast convergence rate. It also offers automatic recovery from failures. The decision tree [ 2 , 44 ] technique is easy to interpret. It can be used for solving the classic regression and classification problems in recommender systems. Bayesian Network [ 3 ] is a probabilistic technique used to solve classification challenges. It is based on the theory of Bayes theorem and conditional probability. Association Rule Mining (ARM) techniques [ 4 , 17 , 35 ] extract rules for projecting the occurrence of an item by considering the existence of other items in a transaction. This method uses the association rules to create a more suitable representation of data and helps in increasing the model performance and storage efficiency. Fuzzy Logic [ 5 , 7 , 15 , 20 , 28 , 43 ] techniques use a set of flexible rules. It focuses on solving complex real-time problems having an inaccurate spectrum of data. This technique provides scalability and helps in increasing the overall model performance for recommender systems. The semantic similarity [ 6 ] technique is used for describing a topological similarity to define the distance among the concepts and terms through ontologies. It measures the similarity information for increasing the efficiency of recommender systems. Rough set [ 9 , 34 ] techniques use probability distributions for solving the challenges of existing recommender models. Semantic web rule language [ 10 ] can efficiently extract the dataset features and increase the model efficiency. Linear programming-based approaches [ 13 , 42 ] are employed for achieving quality decision making in recommender models. Sequential pattern analysis [ 14 ] is applied to find suitable patterns among data items. This helps in increasing model efficiency. The probabilistic model [ 24 ] is a famous tool to handle uncertainty in risk computations and performance assessment. It offers better decision-making capabilities. K-nearest neighbours (KNN) [ 19 , 37 , 44 ] technique provides faster computation time, simplicity and ease of interpretation. They are good for classification and regression-based problems and offers more accuracy. Spectral clustering [ 21 ] is also called graph clustering or similarity-based clustering, which mainly focuses on reducing the space dimensionality in identifying the dataset items. Stochastic learning algorithm [ 26 ] solves the real-time challenges of recommender systems. Linear SVM [ 29 , 44 ] efficiently solves the high dimensional problems related to recommender systems. It is a memory-efficient method and works well with a large number of samples having relative separation among the classes. This method has been shown to perform well even when new or unfamiliar data is added. Relational Functional Gradient Boosting [ 39 ] technique efficiently works on the relational dependency of data, which is useful for statical relational learning for collaborative-based recommender systems. Ensemble learning [ 40 ] combines the forecast of two or more models and aims to achieve better performance than any of the single contributing models. It also helps in reducing overfitting problems, which are common in recommender systems.

SDAE [ 41 ] is used for learning the non-linear transformations with different filters for finding suitable data. This aids in increasing the performance of recommender models. Multimodal network learning [ 45 ] is efficient for multi-modal data, representing a combined representation of diverse modalities. Random forest [ 46 , 51 ] is a commonly used approach in comparison with other classifiers. It has been shown to increase accuracy when handling big data. This technique is a collection of decision trees to minimize variance through training on diverse data samples. ANNInit [ 47 ] is a type of artificial neural network-based technique that has the capability of self-learning and generating efficient results. It is independent of the data type and can learn data patterns automatically. HashMap [ 50 ] gives faster access to elements owing to the hashing methodology, which decreases the data processing time and increases the performance of the system. CNN [ 51 ] technique can automatically fetch the significant features of a dataset without any supervision. It is a computationally efficient method and provides accurate recommendations. This technique is also simple and fast for implementation. Multilayer graph data model [ 54 ] is efficient for real-time applications and minimizes the access time through mapping the correlation as edges among nodes and provides superior performance. Singular Value Decomposition [ 56 ] can simplify the input data and increase the efficiency of recommendations by eliminating the noise present in data. Reinforcement learning [ 57 ] is efficient for practical scenarios of recommender systems having large data sizes. It is capable of boosting the model performance by increasing the model accuracy even for large scale datasets. FNN [ 58 ] is one of the artificial neural network techniques which can learn non-linear and complex relationships between items. It has demonstrated a good performance increase when employed in different recommender systems. Knowledge representation learning [ 60 ] systems aim to simplify the model development process by increasing the acquisition efficiency, inferential efficiency, inferential adequacy and representation adequacy. User-based approaches [ 2 , 55 , 59 ] specialize in detecting user-related meta-data which is employed to increase the overall model performance. This technique is more suitable for real-time applications where it can capture user feedback and use it to increase the user experience.

Optimization-based techniques

The Foraging Bees [ 11 ] technique enables both functional and combinational optimization for random searching in recommender models. Artificial bee colony [ 12 ] is a swarm-based meta-heuristic technique that provides features like faster convergence rate, the ability to handle the objective with stochastic nature, ease for incorporating with other algorithms, usage of fewer control parameters, strong robustness, high flexibility and simplicity. Particle Swarm Optimization [ 23 ] is a computation optimization technique that offers better computational efficiency, robustness in control parameters, and is easy and simple to implement in recommender systems. Portfolio optimization algorithm [ 27 ] is a subclass of optimization algorithms that find its application in stock investment recommender systems. It works well in real-time and helps in the diversification of the portfolio for maximum profit. The artificial immune system [ 31 ]a is computationally intelligent machine learning technique. This technique can learn new patterns in the data and optimize the overall system parameters. Expectation maximization (EM) [ 32 , 36 , 38 ] is an iterative algorithm that guarantees the likelihood of finding the maximum parameters when the input variables are unknown. Delphi panel and repertory grid [ 33 ] offers efficient decision making by solving the dimensionality problem and data sparsity issues of recommender systems. The Firefly algorithm (FA) [ 48 ] provides fast results and increases recommendation efficiency. It is capable of reducing the number of iterations required to solve specific recommender problems. It also provides both local and global sets of solutions. Beetle Antennae Search (BAS) [ 49 ] offers superior search accuracy and maintains less time complexity that promotes the performance of recommendations. Many-objective evolutionary algorithm (MaOEA) [ 52 ] is applicable for real-time, multi-objective, search-related recommender systems. The introduction of a local search operator increases the convergence rate and gets suitable results. Genetic Algorithm (GA) [ 2 , 22 , 25 , 53 ] based techniques are used to solve the multi-objective optimization problems of recommender systems. They employ probabilistic transition rules and have a simpler operation that provides better recommender performance.

Features and challenges

The features and challenges of the existing recommender models are given in Table 6 .

Simulation platforms

The various simulation platforms used for developing different recommender systems with different applications are given in Fig.  9 .

Simulation platforms used for developing different recommender systems

Here, the Java platform is used in 20% of the contributions, MATLAB is implemented in 7% of the contributions, different fold cross-validation are used in 8% of the contributions, 7% of the contributions are utilized by the python platform, 3% of the contributions employ R-programming and 1% of the contributions are developed by Tensorflow, Weka and Android environments respectively. Other simulation platforms like Facebook, web UI (User Interface), real-time environments, etc. are used in 50% of the contributions. Table 7 describes some simulation platforms commonly used for developing recommender systems.

Application focused and dataset description

This section provides an analysis of the different applications focused on a set of recent recommender systems and their dataset details.

Recent recommender systems were analysed and found that 11% of the contributions are focused on the domain of healthcare, 10% of the contributions are on movie recommender systems, 5% of the contributions come from music recommender systems, 6% of the contributions are focused on e-learning recommender systems, 8% of the contributions are used for online product recommender systems, 3% of the contributions are focused on book recommendations and 1% of the contributions are focused on Job and knowledge management recommender systems. 5% of the contributions concentrated on social network recommender systems, 10% of the contributions are focused on tourist and hotels recommender systems, 6% of the contributions are employed for stock recommender systems, and 3% of the contributions contributed for video recommender systems. The remaining 12% of contributions are miscellaneous recommender systems like Twitter, venue-based recommender systems, etc. Similarly, different datasets are gathered for recommender systems based on their application types. A detailed description is provided in Table 8 .

Performance analysis of state-of-art recommender systems

The performance evaluation metrics used for the analysis of different recommender systems is depicted in Table 9 . From the set of research works, 35% of the works use recall measure, 16% of the works employ Mean Absolute Error (MAE), 11% of the works take Root Mean Square Error (RMSE), 41% of the papers consider precision, 30% of the contributions analyse F1-measure, 31% of the works apply accuracy and 6% of the works employ coverage measure to validate the performance of the recommender systems. Moreover, some additional measures are also considered for validating the performance in a few applications.

Research gaps and challenges

In the recent decade, recommender systems have performed well in solving the problem of information overload and has become the more appropriate tool for multiple areas such as psychology, mathematics, computer science, etc. [ 80 ]. However, current recommender systems face a variety of challenges which are stated as follows, and discussed below:

Deployment challenges such as cold start, scalability, sparsity, etc. are already discussed in Sect. 3.

Challenges faced when employing different recommender algorithms for different applications.

Challenges in collecting implicit user data

Challenges in handling real-time user feedback.

Challenges faced in choosing the correct implementation techniques.

Challenges faced in measuring system performance.

Challenges in implementing recommender system for diverse applications.

Numerous recommender algorithms have been proposed on novel emerging dimensions which focus on addressing the existing limitations of recommender systems. A good recommender system must increase the recommendation quality based on user preferences. However, a specific recommender algorithm is not always guaranteed to perform equally for different applications. This encourages the possibility of employing different recommender algorithms for different applications, which brings along a lot of challenges. There is a need for more research to alleviate these challenges. Also, there is a large scope of research in recommender applications that incorporate information from different interactive online sites like Facebook, Twitter, shopping sites, etc. Some other areas for emerging research may be in the fields of knowledge-based recommender systems, methods for seamlessly processing implicit user data and handling real-time user feedback to recommend items in a dynamic environment.

Some of the other research areas like deep learning-based recommender systems, demographic filtering, group recommenders, cross-domain techniques for recommender systems, and dimensionality reduction techniques are also further required to be studied [ 83 ]. Deep learning-based recommender systems have recently gained much popularity. Future research areas in this field can integrate the well-performing deep learning models with new variants of hybrid meta-heuristic approaches.

During this review, it was observed that even though recent recommender systems have demonstrated good performance, there is no single standardized criteria or method which could be used to evaluate the performance of all recommender systems. System performance is generally measured by different evaluation matrices which makes it difficult to compare. The application of recommender systems in real-time applications is growing. User satisfaction and personalization play a very important role in the success of such recommender systems. There is a need for some new evaluation criteria which can evaluate the level of user satisfaction in real-time. New research should focus on capturing real-time user feedback and use the information to change the recommendation process accordingly. This will aid in increasing the quality of recommendations.

Conclusion and future scope

Recommender systems have attracted the attention of researchers and academicians. In this paper, we have identified and prudently reviewed research papers on recommender systems focusing on diverse applications, which were published between 2011 and 2021. This review has gathered diverse details like different application fields, techniques used, simulation tools used, diverse applications focused, performance metrics, datasets used, system features, and challenges of different recommender systems. Further, the research gaps and challenges were put forward to explore the future research perspective on recommender systems. Overall, this paper provides a comprehensive understanding of the trend of recommender systems-related research and to provides researchers with insight and future direction on recommender systems. The results of this study have several practical and significant implications:

Based on the recent-past publication rates, we feel that the research of recommender systems will significantly grow in the future.

A large number of research papers were identified in movie recommendations, whereas health, tourism and education-related recommender systems were identified in very few numbers. This is due to the availability of movie datasets in the public domain. Therefore, it is necessary to develop datasets in other fields also.

There is no standard measure to compute the performance of recommender systems. Among 60 papers, 21 used recall, 10 used MAE, 25 used precision, 18 used F1-measure, 19 used accuracy and only 7 used RMSE to calculate system performance. Very few systems were found to excel in two or more matrices.

Java and Python (with a combined contribution of 27%) are the most common programming languages used to develop recommender systems. This is due to the availability of a large number of standard java and python libraries which aid in the development process.

Recently a large number of hybrid and optimizations techniques are being proposed for recommender systems. The performance of a recommender system can be greatly improved by applying optimization techniques.

There is a large scope of research in using neural networks and deep learning-based methods for developing recommender systems. Systems developed using these methods are found to achieve high-performance accuracy.

This research will provide a guideline for future research in the domain of recommender systems. However, this research has some limitations. Firstly, due to the limited amount of manpower and time, we have only reviewed papers published in journals focusing on computer science, management and medicine. Secondly, we have reviewed only English papers. New research may extend this study to cover other journals and non-English papers. Finally, this review was conducted based on a search on only six descriptors: “Recommender systems”, “Recommendation systems”, “Movie Recommend*”, “Music Recommend*”, “Personalized Recommend*” and “Hybrid Recommend*”. Research papers that did not include these keywords were not considered. Future research can include adding some additional descriptors and keywords for searching. This will allow extending the research to cover more diverse articles on recommender systems.

Availability of data and materials

Not applicable.

Castellano G, Fanelli AM, Torsello MA. NEWER: A system for neuro-fuzzy web recommendation. Appl Soft Comput. 2011;11:793–806.

Article   Google Scholar  

Crespo RG, Martínez OS, Lovelle JMC, García-Bustelo BCP, Gayo JEL, Pablos PO. Recommendation system based on user interaction data applied to intelligent electronic books. Computers Hum Behavior. 2011;27:1445–9.

Lin FC, Yu HW, Hsu CH, Weng TC. Recommendation system for localized products in vending machines. Expert Syst Appl. 2011;38:9129–38.

Wang SL, Wu CY. Application of context-aware and personalized recommendation to implement an adaptive ubiquitous learning system. Expert Syst Appl. 2011;38:10831–8.

García-Crespo Á, López-Cuadrado JL, Colomo-Palacios R, González-Carrasco I, Ruiz-Mezcua B. Sem-Fit: A semantic based expert system to provide recommendations in the tourism domain. Expert Syst Appl. 2011;38:13310–9.

Dong H, Hussain FK, Chang E. A service concept recommendation system for enhancing the dependability of semantic service matchmakers in the service ecosystem environment. J Netw Comput Appl. 2011;34:619–31.

Li M, Liu L, Li CB. An approach to expert recommendation based on fuzzy linguistic method and fuzzy text classification in knowledge management systems. Expert Syst Appl. 2011;38:8586–96.

Lorenzi F, Bazzan ALC, Abel M, Ricci F. Improving recommendations through an assumption-based multiagent approach: An application in the tourism domain. Expert Syst Appl. 2011;38:14703–14.

Huang Z, Lu X, Duan H. Context-aware recommendation using rough set model and collaborative filtering. Artif Intell Rev. 2011;35:85–99.

Chen RC, Huang YH, Bau CT, Chen SM. A recommendation system based on domain ontology and SWRL for anti-diabetic drugs selection. Expert Syst Appl. 2012;39:3995–4006.

Mohanraj V, Chandrasekaran M, Senthilkumar J, Arumugam S, Suresh Y. Ontology driven bee’s foraging approach based self-adaptive online recommendation system. J Syst Softw. 2012;85:2439–50.

Hsu CC, Chen HC, Huang KK, Huang YM. A personalized auxiliary material recommendation system based on learning style on facebook applying an artificial bee colony algorithm. Comput Math Appl. 2012;64:1506–13.

Gemmell J, Schimoler T, Mobasher B, Burke R. Resource recommendation in social annotation systems: A linear-weighted hybrid approach. J Comput Syst Sci. 2012;78:1160–74.

Article   MathSciNet   Google Scholar  

Choi K, Yoo D, Kim G, Suh Y. A hybrid online-product recommendation system: Combining implicit rating-based collaborative filtering and sequential pattern analysis. Electron Commer Res Appl. 2012;11:309–17.

Garibaldi JM, Zhou SM, Wang XY, John RI, Ellis IO. Incorporation of expert variability into breast cancer treatment recommendation in designing clinical protocol guided fuzzy rule system models. J Biomed Inform. 2012;45:447–59.

Salehi M, Kmalabadi IN. A hybrid attribute–based recommender system for e–learning material recommendation. IERI Procedia. 2012;2:565–70.

Aher SB, Lobo LMRJ. Combination of machine learning algorithms for recommendation of courses in e-learning System based on historical data. Knowl-Based Syst. 2013;51:1–14.

Kardan AA, Ebrahimi M. A novel approach to hybrid recommendation systems based on association rules mining for content recommendation in asynchronous discussion groups. Inf Sci. 2013;219:93–110.

Chang JH, Lai CF, Wang MS, Wu TY. A cloud-based intelligent TV program recommendation system. Comput Electr Eng. 2013;39:2379–99.

Lucas JP, Luz N, Moreno MN, Anacleto R, Figueiredo AA, Martins C. A hybrid recommendation approach for a tourism system. Expert Syst Appl. 2013;40:3532–50.

Niu J, Zhu L, Zhao X, Li H. Affivir: An affect-based Internet video recommendation system. Neurocomputing. 2013;120:422–33.

Liu L, Xu J, Liao SS, Chen H. A real-time personalized route recommendation system for self-drive tourists based on vehicle to vehicle communication. Expert Syst Appl. 2014;41:3409–17.

Bakshi S, Jagadev AK, Dehuri S, Wang GN. Enhancing scalability and accuracy of recommendation systems using unsupervised learning and particle swarm optimization. Appl Soft Comput. 2014;15:21–9.

Kim Y, Shim K. TWILITE: A recommendation system for twitter using a probabilistic model based on latent Dirichlet allocation. Inf Syst. 2014;42:59–77.

Wang Z, Yu X, Feng N, Wang Z. An improved collaborative movie recommendation system using computational intelligence. J Vis Lang Comput. 2014;25:667–75.

Kolomvatsos K, Anagnostopoulos C, Hadjiefthymiades S. An efficient recommendation system based on the optimal stopping theory. Expert Syst Appl. 2014;41:6796–806.

Gottschlich J, Hinz O. A decision support system for stock investment recommendations using collective wisdom. Decis Support Syst. 2014;59:52–62.

Torshizi AD, Zarandi MHF, Torshizi GD, Eghbali K. A hybrid fuzzy-ontology based intelligent system to determine level of severity and treatment recommendation for benign prostatic hyperplasia. Comput Methods Programs Biomed. 2014;113:301–13.

Zahálka J, Rudinac S, Worring M. Interactive multimodal learning for venue recommendation. IEEE Trans Multimedia. 2015;17:2235–44.

Sankar CP, Vidyaraj R, Kumar KS. Trust based stock recommendation system – a social network analysis approach. Procedia Computer Sci. 2015;46:299–305.

Chen MH, Teng CH, Chang PC. Applying artificial immune systems to collaborative filtering for movie recommendation. Adv Eng Inform. 2015;29:830–9.

Wu H, Pei Y, Li B, Kang Z, Liu X, Li H. Item recommendation in collaborative tagging systems via heuristic data fusion. Knowl-Based Syst. 2015;75:124–40.

Yeh DY, Cheng CH. Recommendation system for popular tourist attractions in Taiwan using delphi panel and repertory grid techniques. Tour Manage. 2015;46:164–76.

Liao SH, Chang HK. A rough set-based association rule approach for a recommendation system for online consumers. Inf Process Manage. 2016;52:1142–60.

Li H, Cui J, Shen B, Ma J. An intelligent movie recommendation system through group-level sentiment analysis in microblogs. Neurocomputing. 2016;210:164–73.

Wu H, Yue K, Pei Y, Li B, Zhao Y, Dong F. Collaborative topic regression with social trust ensemble for recommendation in social media systems. Knowl-Based Syst. 2016;97:111–22.

Adeniyi DA, Wei Z, Yongquan Y. Automated web usage data mining and recommendation system using K-Nearest Neighbor (KNN) classification method. Appl Computing Inform. 2016;12:90–108.

Rawat YS, Kankanhalli MS. ClickSmart: A context-aware viewpoint recommendation system for mobile photography. IEEE Trans Circuits Syst Video Technol. 2017;27:149–58.

Yang S, Korayem M, Aljadda K, Grainger T, Natarajan S. Combining content-based and collaborative filtering for job recommendation system: A cost-sensitive Statistical Relational Learning approach. Knowl-Based Syst. 2017;136:37–45.

Lee WP, Chen CT, Huang JY, Liang JY. A smartphone-based activity-aware system for music streaming recommendation. Knowl-Based Syst. 2017;131:70–82.

Wei J, He J, Chen K, Zhou Y, Tang Z. Collaborative filtering and deep learning based recommendation system for cold start items. Expert Syst Appl. 2017;69:29–39.

Li C, Wang Z, Cao S, He L. WLRRS: A new recommendation system based on weighted linear regression models. Comput Electr Eng. 2018;66:40–7.

Mezei J, Nikou S. Fuzzy optimization to improve mobile health and wellness recommendation systems. Knowl-Based Syst. 2018;142:108–16.

Ayata D, Yaslan Y, Kamasak ME. Emotion based music recommendation system using wearable physiological sensors. IEEE Trans Consum Electron. 2018;64:196–203.

Zhao Z, Yang Q, Lu H, Weninger T. Social-aware movie recommendation via multimodal network learning. IEEE Trans Multimedia. 2018;20:430–40.

Hammou BA, Lahcen AA, Mouline S. An effective distributed predictive model with matrix factorization and random forest for big data recommendation systems. Expert Syst Appl. 2019;137:253–65.

Zhao J, Geng X, Zhou J, Sun Q, Xiao Y, Zhang Z, Fu Z. Attribute mapping and autoencoder neural network based matrix factorization initialization for recommendation systems. Knowl-Based Syst. 2019;166:132–9.

Bhaskaran S, Santhi B. An efficient personalized trust based hybrid recommendation (TBHR) strategy for e-learning system in cloud computing. Clust Comput. 2019;22:1137–49.

Han Y, Han Z, Wu J, Yu Y, Gao S, Hua D, Yang A. Artificial intelligence recommendation system of cancer rehabilitation scheme based on IoT technology. IEEE Access. 2020;8:44924–35.

Kang S, Jeong C, Chung K. Tree-based real-time advertisement recommendation system in online broadcasting. IEEE Access. 2020;8:192693–702.

Ullah F, Zhang B, Khan RU. Image-based service recommendation system: A JPEG-coefficient RFs approach. IEEE Access. 2020;8:3308–18.

Cai X, Hu Z, Zhao P, Zhang W, Chen J. A hybrid recommendation system with many-objective evolutionary algorithm. Expert Syst Appl. 2020. https://doi.org/10.1016/j.eswa.2020.113648 .

Esteban A, Zafra A, Romero C. Helping university students to choose elective courses by using a hybrid multi-criteria recommendation system with genetic optimization. Knowledge-Based Syst. 2020;194:105385.

Mondal S, Basu A, Mukherjee N. Building a trust-based doctor recommendation system on top of multilayer graph database. J Biomed Inform. 2020;110:103549.

Dhelim S, Ning H, Aung N, Huang R, Ma J. Personality-aware product recommendation system based on user interests mining and metapath discovery. IEEE Trans Comput Soc Syst. 2021;8:86–98.

Bhalse N, Thakur R. Algorithm for movie recommendation system using collaborative filtering. Materials Today: Proceedings. 2021. https://doi.org/10.1016/j.matpr.2021.01.235 .

Ke G, Du HL, Chen YC. Cross-platform dynamic goods recommendation system based on reinforcement learning and social networks. Appl Soft Computing. 2021;104:107213.

Chen X, Liu D, Xiong Z, Zha ZJ. Learning and fusing multiple user interest representations for micro-video and movie recommendations. IEEE Trans Multimedia. 2021;23:484–96.

Afolabi AO, Toivanen P. Integration of recommendation systems into connected health for effective management of chronic diseases. IEEE Access. 2019;7:49201–11.

He M, Wang B, Du X. HI2Rec: Exploring knowledge in heterogeneous information for movie recommendation. IEEE Access. 2019;7:30276–84.

Bobadilla J, Serradilla F, Hernando A. Collaborative filtering adapted to recommender systems of e-learning. Knowl-Based Syst. 2009;22:261–5.

Russell S, Yoon V. Applications of wavelet data reduction in a recommender system. Expert Syst Appl. 2008;34:2316–25.

Campos LM, Fernández-Luna JM, Huete JF. A collaborative recommender system based on probabilistic inference from fuzzy observations. Fuzzy Sets Syst. 2008;159:1554–76.

Funk M, Rozinat A, Karapanos E, Medeiros AKA, Koca A. In situ evaluation of recommender systems: Framework and instrumentation. Int J Hum Comput Stud. 2010;68:525–47.

Porcel C, Moreno JM, Herrera-Viedma E. A multi-disciplinar recommender system to advice research resources in University Digital Libraries. Expert Syst Appl. 2009;36:12520–8.

Bobadilla J, Serradilla F, Bernal J. A new collaborative filtering metric that improves the behavior of recommender systems. Knowl-Based Syst. 2010;23:520–8.

Ochi P, Rao S, Takayama L, Nass C. Predictors of user perceptions of web recommender systems: How the basis for generating experience and search product recommendations affects user responses. Int J Hum Comput Stud. 2010;68:472–82.

Olmo FH, Gaudioso E. Evaluation of recommender systems: A new approach. Expert Syst Appl. 2008;35:790–804.

Zhen L, Huang GQ, Jiang Z. An inner-enterprise knowledge recommender system. Expert Syst Appl. 2010;37:1703–12.

Göksedef M, Gündüz-Öğüdücü S. Combination of web page recommender systems. Expert Syst Appl. 2010;37(4):2911–22.

Shao B, Wang D, Li T, Ogihara M. Music recommendation based on acoustic features and user access patterns. IEEE Trans Audio Speech Lang Process. 2009;17:1602–11.

Shin C, Woo W. Socially aware tv program recommender for multiple viewers. IEEE Trans Consum Electron. 2009;55:927–32.

Lopez-Carmona MA, Marsa-Maestre I, Perez JRV, Alcazar BA. Anegsys: An automated negotiation based recommender system for local e-marketplaces. IEEE Lat Am Trans. 2007;5:409–16.

Yap G, Tan A, Pang H. Discovering and exploiting causal dependencies for robust mobile context-aware recommenders. IEEE Trans Knowl Data Eng. 2007;19:977–92.

Meo PD, Quattrone G, Terracina G, Ursino D. An XML-based multiagent system for supporting online recruitment services. IEEE Trans Syst Man Cybern. 2007;37:464–80.

Khusro S, Ali Z, Ullah I. Recommender systems: Issues, challenges, and research opportunities. Inform Sci Appl. 2016. https://doi.org/10.1007/978-981-10-0557-2_112 .

Blanco-Fernandez Y, Pazos-Arias JJ, Gil-Solla A, Ramos-Cabrer M, Lopez-Nores M. Providing entertainment by content-based filtering and semantic reasoning in intelligent recommender systems. IEEE Trans Consum Electron. 2008;54:727–35.

Isinkaye FO, Folajimi YO, Ojokoh BA. Recommendation systems: Principles, methods and evaluation. Egyptian Inform J. 2015;16:261–73.

Yoshii K, Goto M, Komatani K, Ogata T, Okuno HG. An efficient hybrid music recommender system using an incrementally trainable probabilistic generative model. IEEE Trans Audio Speech Lang Process. 2008;16:435–47.

Wei YZ, Moreau L, Jennings NR. Learning users’ interests by quality classification in market-based recommender systems. IEEE Trans Knowl Data Eng. 2005;17:1678–88.

Bjelica M. Towards TV recommender system: experiments with user modeling. IEEE Trans Consum Electron. 2010;56:1763–9.

Setten MV, Veenstra M, Nijholt A, Dijk BV. Goal-based structuring in recommender systems. Interact Comput. 2006;18:432–56.

Adomavicius G, Tuzhilin A. Toward the next generation of recommender systems: a survey of the state-of-the-art and possible extensions. IEEE Trans Knowl Data Eng. 2005;17:734–49.

Symeonidis P, Nanopoulos A, Manolopoulos Y. Providing justifications in recommender systems. IEEE Transactions on Systems, Man, and Cybernetics - Part A: Systems and Humans. 2009;38:1262–72.

Zhan J, Hsieh C, Wang I, Hsu T, Liau C, Wang D. Privacy preserving collaborative recommender systems. IEEE Trans Syst Man Cybernet. 2010;40:472–6.

Burke R. Hybrid recommender systems: survey and experiments. User Model User-Adap Inter. 2002;12:331–70.

Article   MATH   Google Scholar  

Gunes I, Kaleli C, Bilge A, Polat H. Shilling attacks against recommender systems: a comprehensive survey. Artif Intell Rev. 2012;42:767–99.

Park DH, Kim HK, Choi IY, Kim JK. A literature review and classification of recommender systems research. Expert Syst Appl. 2012;39:10059–72.

Download references

Acknowledgements

We thank our colleagues from Assam Down Town University who provided insight and expertise that greatly assisted this research, although they may not agree with all the interpretations and conclusions of this paper.

No funding was received to assist with the preparation of this manuscript.

Author information

Authors and affiliations.

Department of Computer Science & Engineering, Assam Down Town University, Panikhaiti, Guwahati, 781026, Assam, India

Deepjyoti Roy & Mala Dutta

You can also search for this author in PubMed   Google Scholar

Contributions

DR carried out the review study and analysis of the existing algorithms in the literature. MD has been involved in drafting the manuscript or revising it critically for important intellectual content. Both authors read and approved the final manuscript.

Corresponding author

Correspondence to Deepjyoti Roy .

Ethics declarations

Ethics approval and consent to participate, consent for publication, competing interests.

On behalf of all authors, the corresponding author states that there is no conflict of interest.

Additional information

Publisher's note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Roy, D., Dutta, M. A systematic review and research perspective on recommender systems. J Big Data 9 , 59 (2022). https://doi.org/10.1186/s40537-022-00592-5

Download citation

Received : 04 October 2021

Accepted : 28 March 2022

Published : 03 May 2022

DOI : https://doi.org/10.1186/s40537-022-00592-5

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Recommender system
• Machine learning
• Content-based filtering
• Collaborative filtering
• Deep learning

Advertisement

Recommender systems in the healthcare domain: state-of-the-art and research issues

• Open access
• Published: 17 December 2020
• Volume 57 , pages 171–201, ( 2021 )

Cite this article

You have full access to this open access article

• Thi Ngoc Trang Tran   ORCID: orcid.org/0000-0002-3550-8352 1 ,
• Alexander Felfernig 1 ,
• Christoph Trattner 2 &
• Andreas Holzinger 3  

40k Accesses

127 Citations

3 Altmetric

Explore all metrics

Nowadays, a vast amount of clinical data scattered across different sites on the Internet hinders users from finding helpful information for their well-being improvement. Besides, the overload of medical information (e.g., on drugs, medical tests, and treatment suggestions) have brought many difficulties to medical professionals in making patient-oriented decisions. These issues raise the need to apply recommender systems in the healthcare domain to help both, end-users and medical professionals, make more efficient and accurate health-related decisions. In this article, we provide a systematic overview of existing research on healthcare recommender systems. Different from existing related overview papers, our article provides insights into recommendation scenarios and recommendation approaches. Examples thereof are food recommendation, drug recommendation, health status prediction, healthcare service recommendation, and healthcare professional recommendation. Additionally, we develop working examples to give a deep understanding of recommendation algorithms. Finally, we discuss challenges concerning the development of healthcare recommender systems in the future.

Similar content being viewed by others

Health Recommender Systems: A Survey

Health Recommender Systems

Introduction on Health Recommender Systems

Explore related subjects.

• Artificial Intelligence

Avoid common mistakes on your manuscript.

1 Introduction

In the past decades, a considerable amount of clinical data representing patients’ health status (e.g., medical reports, laboratory results, and disease treatment plans) have been collected. This has remarkably increased digital information available for patient-oriented decision making . Such digital information is often scattered across different sites, which hinders users from finding useful information for their well-being improvement. Besides, more drugs, tests, and treatment recommendations are available for medical staff daily, which triggers difficulties in deciding appropriate remedies for patients (Stark et al. 2019 ; Wiesner and Pfeifer 2014 ). In this context, recommender systems for medical use should be implemented to bridge these gaps and support both, patients and medical professionals, to make better healthcare-related decisions. Recommender systems have been integrated into online retailers, streaming services, and social networks to facilitate users’ item selection process (Felfernig and Gula 2006 ; Tran et al. 2018 ). Recently, these systems have been widely applied to the healthcare domain (so-called Health Recommender Systems - HRS ) to better support medical suggestions. Different from the precursors in the same domain (e.g., medical expert systems), HRS offer a better personalization that increases the details of provided recommendations and improves users’ understanding of their medical condition. These systems also provide patients with a better experience, improve their health condition, and motivate them to follow a healthier lifestyle. Moreover, they also assist healthcare professionals with disease predictions/treatments (Holzinger et al. 2016 ; Pincay et al. 2019 ; Sahoo et al. 2019 ; Schäfer et al. 2017 ; Wiesner and Pfeifer 2014 ). HRS should analyze patients’ health status and recommend personalized diets, exercise routines, medications, disease diagnoses, or other healthcare services. HRS’s great concern is to send the necessary information to patients at the right time while ensuring the accuracy, trustworthiness, and privacy of patient information (Sahoo et al. 2019 ). Moreover, these systems are expected to minimize the cost of the healthcare-related decision making process (in terms of time and effort ) (Valdez et al. 2016 ).

Although several studies have been carried out on HRS, they target a specific disease or recommendation context. This raises a need for a comprehensive overview that provides a “ full landscape ” of recommendation scenarios supported by HRS (Pincay et al. 2019 ). In the current literature, only a few endeavors summarize current approaches to designing and implementing HRS. For instance, Sezgin and Özkan (Sezgin and Özkan 2013 ) and Wiesner and Pfeifer (Wiesner and Pfeifer 2014 ) discussed some recommendation scenarios (e.g., drug recommendations, medical information suggestions , and disease predictions ) and various methods to evaluate the effectiveness of HRS. Calero Valdez et al. (Valdez et al. 2016 ) provided a literature review, in which a framework to develop HRS was proposed. Stark et al. (Stark et al. 2019 ) analyzed 13 existing studies and categorized them according to criteria, such as disease, data storage, interface, data collection, data preparation, and recommendation techniques . Finally, Pincay et al. (Pincay et al. 2019 ) presented an overview of the methods and techniques used to design and implement HRS. Compared to the mentioned related work, our article presents a broader picture of recommendation scenarios supported by HRS with a different set of considered studies. The discussed scenarios focus on two types of users: end-users (healthy users and patients) and healthcare professionals (e.g., doctors, nurses, clinicians, or physicians). For end-users, HRS provide nutritional information, medications, treatment plans, diagnoses/disease predictions, physical activities or other healthcare services (e.g., finding good doctors or proper medical services for patients) (Wiesner and Pfeifer 2014 ). For healthcare professionals, HRS use medical resources to assist them in creating more precise suggestions for patients. For each recommendation scenario, we summarize recommendation algorithms and develop corresponding working examples. Besides analyzing existing studies, we discuss research challenges as well as potential directions for future HRS. The remainder of the article is structured as follows. In Section  2 , we present the used methodology for our literature analysis. In Section  3 , we discuss basic approaches that are frequently used in recommender systems and necessary adaptations to apply these in the healthcare domain. In Section  4 , we present recommendation scenarios supported by HRS and used recommendation techniques. In Section  5 , we summarize evaluation methods employed in the mentioned approaches. Finally, we discuss open issues for future work in Section  6 and conclude the article in Section  7 .

2 Research methodology

This study was performed based on a bibliographic review method, which provides a systematical analysis of domain-specific knowledge. We first collected a set of studies concerning HRS using the following keywords: “health recommender systems”, “medicine recommender systems”, “recommender systems in the wellness domain” , and “e-Health systems” . To have a deeper look at recommendation scenarios in the healthcare domain, we searched for references using additional keywords: “food recommendation”, “nutrition recommendation”, “drug recommendation”, “heath status prediction”, “healthcare service recommendation”, “physical activity recommendation”, and “doctor recommendation” . Besides, to ensure the quality of the references, we selected the papers using some criteria proposed by Stark et al. (Stark et al. 2019 ): (i) published from 2000 onward; (ii) well referenced with more than 15 sources, (iii) providing logical and reasonable findings of the domain, and (iv) presenting a detailed discussion on recommendation techniques. We searched for references in well-known digital libraries, such as Google Scholars Footnote 1 , Springer Footnote 2 , ACM Footnote 3 , ResearchGate Footnote 4 , Science Direct Footnote 5 , and PubMed Footnote 6 . In this context, we checked the title, keywords, abstract, conclusion, tables, and figures of the collected papers. Finally, we filtered out 98 papers that meet the mentioned criteria and have a strong relationship with our work. From these, we selected and analyzed 37 studies, which provide detailed discussions on recommendation approaches in the healthcare domain. These studies are summarized in Section  4 : eight papers related to food recommendation, 18 papers on drug recommendation, three papers related to health status prediction, four papers on physical activity recommendation, and four papers on healthcare professional recommendation. Additionally, we analyzed 32 papers to find out open issues for HRS and potential directions for future work. The remaining papers are cited in other sections of this article. Most papers cited in our work were published in prestigious conferences, such as ACM Conference on Recommender Systems, ACM Conference on User Modelling, Adaptation and Personalization, IEEE International Conference on e-Health Networking, and International Conference on Software Engineering and Knowledge Engineering. For journal articles, we selected the ones published in journals on computer science (e.g., Journal of Computer Applications, Journal of Expert System Applications, Journal of Computing Sciences in Colleges, Journal of Data Mining and Knowledge Discovery, and Journal of Engineering and Technology) and on medicine (e.g., Journal of Science Translational Medicine, International Journal of Basic Science in Medicine, and Journal of Biomedical Semantics, and Journal of Biomedical Informatics).

3 Basic techniques in recommender systems

Collaborative filtering (Aberg 2006 ; Bankhele et al. 2017 ; Berkovsky and Freyne 2010 ; Davis et al. 2009 ; Dharia et al. 2016 ; Han et al. 2018 ; Narducci et al. 2015 ; Nasiri et al. 2016 ; Stark et al. 2017 ; Zhang et al. 2016 ), content-based (Aberg 2006 ; Dharia et al. 2016 ; Han et al. 2018 ), knowledge-based (Ali et al. 2018 ; Doulaverakis et al. 2012 ; Mahmoud and Elbeh 2016 ), and hybrid approaches (Aberg 2006 ; Dharia et al. 2016 ; Han et al. 2018 ) are the basic recommendation techniques that can be used in HRS. Besides, other algorithms are also applied to generate recommendations in the healthcare domain, such as ant colony algorithm (Rehman et al. 2017 ), classification (Hussein et al. 2012 ; Shimada et al. 2005 ), clustering (Rokicki et al. 2015 ), decision tree (Bresso et al. 2013 ), logistic regression (Huang et al. 2011 ), natural language processing (Gujar et al. 2018 ), inductive logic programming (Bresso et al. 2013 ), ontologies (Chen et al. 2011 ; Chen et al. 2012 ; Donciu et al. 2011 ; Doulaverakis et al. 2012 ; Faiz et al. 2014 ; Mahmoud and Elbeh 2016 ), sparse canonical correlation (Yamanishi et al. 2012 ), support vector machines (Huang et al. 2011 ), semantic technologies (Donciu et al. 2011 ; Faiz et al. 2014 ; Medvedeva et al. 2007 ), multi-criteria decision making (Chen et al. 2011 ; Chen et al. 2012 ), graph-based recommendations (Stark et al. 2017 ), context-aware recommendation (Ali et al. 2018 ), and matrix factorization (Zhang et al. 2015 ). In this section, we present basic recommendation techniques applied in the healthcare domain. Other techniques will be discussed in Section  4 .

3.1 Aspects of recommender systems

There are three main aspects that need to be considered in recommender systems: usage context, users , and items (Sánchez-Bocanegra et al. 2015 ). Usage context describes the environment where all elements (e.g., items, users, and their relationship) interact with each other. Users are the end-users of recommender systems, and items are the elements that users are looking for. In the healthcare domain, additional aspects concerning the mentioned elements should be considered to generate more precise recommendations.

Usage context

The usage context in HRS consists of contextual factors and multi-factorial goal settings that can influence on how items are recommended or presented. Contextual factors indicate dynamic attributes which might affect a specific activity (e.g., time to take medicine - the optimal time to take fat-soluble vitamins is with dinner) and dynamic factors from users (such as emotional states). The inclusion of such contextual information in the sequence of recent contexts of a user can help to better understand contexts that led to the user’s current behavior and preferences. For multi-factorial goal settings , different domain-specific criteria should be considered when evaluating an item. In e-commerce domains, people might naively think that the “ most preferred items ” are more likely to be recommended to users. However, this idea needs to be reconsidered in the healthcare domain since items that are the best for this user might not be good for others (Valdez et al. 2016 ). For instance, although diuretics and blood pressure-lowering medicines are good for patients suffering from hypertension, these drugs can be dangerous for diabetes or gout patients. Besides, even patients have the same diseases, remedies for this patient could not always be suggested to others since they might have different health conditions.

HRS are able to support two types of users: end-users and healthcare professionals . End-users could be healthy users or patients. For each end-user, the system has to save a user profile describing his/her health condition. For instance, the profile of a cardiovascular patient includes the following information: name, birthday, weight, height, cardiovascular type , and blood pressure measurement . This information helps HRS identify appropriate medications for the user. Healthcare professionals can be doctors, nurses, physicians, clinicians, or pharmacists . Besides, medical researchers and policy makers can also benefit from HRS (Valdez et al. 2016 ).

HRS can offer recommendations concerning different categories, such as diets to optimize nutrition, physical activities/sports that match the user’s requirements and needs, recommended diagnoses of patients to doctors or nurses, treatments/medications for a specific disease, and medical information/sources that motivate(s) users to follow a healthy lifestyle and improve their well-being (Valdez et al. 2016 ).

3.2 Basic recommendation techniques

The information of the mentioned elements can be the input of algorithms that generate personalized recommendations to patients.

Collaborative Filtering (CF)

CF recommends items to a user based on the following idea: “If users shared the same interests in the past, then they would have similar tastes” (Jannach 2011 ). In the context of HRS, this approach can be interpreted as follows: “If patients share similar disease profiles/health conditions, then they would have similar treatments/health-care services”.

Content-based Filtering (CB)

This approach looks for items similar to those that the user liked in the past and match the user profile (Lops et al. 2011 ; Ricci et al. 2010 ; Sánchez-Bocanegra et al. 2015 ). In HRS, this approach suggests healthcare services that fit the patient’s health condition/disease situation and are similar to those assigned to him/her in the past.

Knowledge-based Recommendation (KB)

This approach is applied to domains where the quantity of available item ratings is quite limited (e.g., cars , apartments , and financial services ) or when the user wants to define his/her requirements for items explicitly (e.g., “the food should not contain cheese since I am allergic to milk products”). This approach creates recommendations based on knowledge about the items , explicit user preferences , and a set of constraints describing the dependencies between users’ preferences and items’ properties (Felfernig and Burke 2008 ).

Hybrid Recommendation (HyR)

The idea of this approach is to combine the aforementioned recommendation techniques to make use of the advantages of one approach and fix the disadvantages of another approach (Ricci et al. 2010 ). For instance, CF usually faces a cold-start problem triggered when a new item is added to the system and has no user ratings, whereas CB can tackle this issue since the prediction for new items is generally based on available descriptions of these items.

4 Recommendation scenarios in the healthcare domain

HRS offer users various types of recommendations that help to improve their well-being. These systems also assist healthcare professionals in making more precise patient-oriented decisions. In the following subsections, we provide a detailed discussion on recommendation scenarios and corresponding recommendation approaches (see also Table  1 ).

4.1 Food recommendation

Due to the extensive growth of food variety and busy lifestyles, people have been facing the issue of making healthy food decisions to reduce the risk of chronic diseases (Ge et al. 2015 ; Robertson et al. 2004 ). In this context, food recommender systems can motivate users to change their eating behaviors or suggest healthier food choices (Tran et al. 2018 ; Trattner and Elsweiler 2017 ; Yang et al. 2017 ). In the following, we summarize scenarios where food recommender systems support users in optimizing their nutrition intake. Studies on food recommendation were presented in our earlier survey (Tran et al. 2018 ). However, different from (Tran et al. 2018 ) where these studies were grouped based on recommendation techniques, in this article, they are grouped according to dietary needs. Besides, we include additional studies on “food-substitutes suggestions” to increase the coverage of the article. For the studies already presented in our earlier survey, we shortly mention the general idea of recommendation algorithms, and for further details, we refer to (Tran et al. 2018 ).

Recommend proper diets

Many people are suffering from health problems concerning inappropriate eating habits. Thus, one of the main functions of food recommender systems is to understand the eating behavior and recommend proper diets to users. There exist in the current literature some systems that fulfill this function. For instance, Aberg et al. (Aberg 2006 ) developed a menu-planning tool to deal with the malnutrition of the elderly. Rehman et al. (Rehman et al. 2017 ) highlighted the appropriateness of selected diets by proposing a cloud-based food recommender system so-called Diet-Right . This system uses an ant colony algorithm to generate an optimal food list and to suggest proper food for users according to their pathological reports.

Prevent/Cure Food-based Illness

Lacking the nutritional understanding of users leads to poor selections of ingredients and causes food-related diseases . To prevent these issues, food recommender systems have been developed to provide nutrition recommendations that consider both the preferences and health conditions of patients. For instance, Rokicki et al. (Rokicki et al. 2015 ) suggested menus that best match the patients’ tastes and dietary restrictions. Ueta et al. (Ueta et al. 2011 ) proposed a goal-oriented recipe recommendation to provide a user with a nutrient list for treating his/her health problem.

Suggest Food Substitutes

Another approach of food recommender systems is to identify a substitute relationship between food pairs as the first step towards “ similar but healthier ” food recommendations (Achananuparp and Weber 2016 ). In this approach, foods are assumed to be similar dietarily if they are consumed in similar contexts. For instance, “a chicken sandwich can be a substitute for a turkey sandwich if they are both consumed with french fries and salad” (Achananuparp and Weber 2016 ). This approach analyzes the real-world self-reported food consumption of users created by the MyFitnessPal Footnote 7 . The consumed food items and corresponding contexts are represented in a food-context matrix , in which the rows represent food items, and the columns represent a context. The similarity between two food items is measured using the Cosine similarity between the corresponding row vectors in the matrix. The higher the similarity score, the higher the probability of food pairs to be suitable substitutes for each other. Elsweiler et al. (Elsweiler et al. 2017 ) investigated the feasibility of replacing recipes consumed by a user with similar and healthier alternatives. To find appropriate recipe substitutions, the authors applied an ingredient-network approach (Teng et al. 2012 ) to establish recipe pairs based on their pairwise similarities. Thereafter, they looked at the distribution of health features across pairs to identify healthier replacements. Finally, the rating distributions within pairs were considered to find out replacements with higher ratings than the original recipes’ rating.

Food Recommendation for Groups

In many real-world scenarios, food consumption is a good example of a group activity, for instance, deciding on the menu for a birthday party with friends or daily meals with family members (Elahi et al. 2014 ; Felfernig et al. 2018 ). In these scenarios, recommendations should be tailored in such a way that assures the maximum satisfaction of each member and the group as a whole (O’Connor et al. 2001 ). Berkovsky and Freyne (Berkovsky and Freyne 2010 ) examined the applicability of CF strategies to discover the best strategy for group recommendations. The authors discussed two group-based recommendation strategies: aggregated models strategy and aggregated prediction strategy Footnote 8 . These strategies recommend a list of recipes to the whole family by considering the task of recommending top-k recipes. Similar work was conducted by Elahi et al. (Elahi et al. 2014 ), where a novel interactive environment for groups was developed in planning meals through a conversational process based on critiquing (Chen and Pu 2012 ). For an example of food recommendation for groups, we refer to our earlier survey (Tran et al. 2018 ).

4.2 Drug recommendation

4.2.1 drug recommendation for curing diseases.

Medication errors are one of the most serious medical errors that could threaten patients’ life (Charkhat Gorgich et al. 2015 ). More than 42% of these errors are caused by doctors who have limited experiences/knowledge about drugs and diseases (Bao and Jiang 2016 ). Another reason lies in the increasing number of available drug information, which has brought obstacles concerning the discovery of relevant drugs and drug-disease interactions (Doulaverakis et al. 2012 ). In this context, drug recommender systems have been developed to assist end-users and healthcare professionals in identifying accurate medications for a specific disease.

Diabetes disease

Diabetes is one of the most popular diseases caused by busy lifestyles with a lack of physical activities and unhealthy eating habits (Bankhele et al. 2017 ; Mahmoud and Elbeh 2016 ). Plenty of drug recommender systems have been developed to help end-users effectively control diabetes and avoid future complications. These systems also assist medical professionals in giving precise medicine recommendations to patients. Chen et al. (Chen et al. 2011 ) created anti-diabetic drug recommendations based on patient ontology knowledge and multi-criteria decision making . Mahmoud et al. (Mahmoud and Elbeh 2016 ) utilized ontologies to represent knowledge about patients’ profiles and anti-diabetes drugs. This system additionally combines ontologies with rule-based decision making to provide restrictions on target treatment goals and medicines with dose prescription. The defined rules select drugs for each patient based on his/her profile. An example rule of selecting drugs can be described as follows: “If a patient is under 60 years old, suffering a liver problem, and used Sulfonzlureas (Glipizide), then starting dose should be 2.5mg daily” (Mahmoud and Elbeh 2016 ). Medvedeva et al. (Medvedeva et al. 2007 ) developed a web-based case-similarity retrieval system to enable doctors to share their knowledge with the community and to optimize disease treatments for their patients. In this system, patient histories are utilized by doctors to select suitable treatment plans for patients. Bankhele et al. (Bankhele et al. 2017 ) proposed a recommendation approach based on the CF technique to suggest proper medications to diabetes patients. A patient has to register in the system and then enters a predefined set of attributes, such as age, insulin, glucose, BMI, BP, and triceps thickness , which are then analyzed to create personalized recommendations. A user-based CF is applied to find patients whose attributes best match the active patient’s attributes. This matching is done using Formula ( 1 ), where P is the attribute set of patients a and b ; r a , p is the value of patient a for attribute p with \(\overline {r_{a}}\) as the mean over set P of attributes p ; r b , p is the value of patient b for the attribute p with \(\overline {r_{b}}\) as the mean over set P of attributes p .

For demonstration purposes, we introduce an example describing the drug recommendation process using the approach presented in (Bankhele et al. 2017 ). Assume, Tom is an active patient who has entered to the system some attributes of his health status (see Table  2 ). The data of patients who share similar attributes with Tom ( p a t i e n t 1 ... p a t i e n t 4 ) is summarized in Table  2 . Based on Formula ( 1 ), the similarity scores of patients with regard to current user attributes are calculated as follows:

s i m ( T o m , p a t i e n t 1 ) = 0.86 \(\checkmark \) ; s i m ( T o m , p a t i e n t 2 ) = 0.51

s i m ( T o m , p a t i e n t 3 ) = 0.49; s i m ( T o m , p a t i e n t 4 ) = 0.04

The calculations show that p a t i e n t 1 is the most similar to Tom . Thus, the drugs prescribed for this patient can be recommended to Tom .

Migraine Disease

Stark et al. (Stark et al. 2017 ) proposed a drug recommender system assisting doctors in writing more appropriate and accurate prescriptions to migraine-disease patients. This system uses a graph database to store patients’ information. The database is organized as nodes and edges . Nodes represent patients’ information, diseases, allergies, and drugs, whereas edges represent the relationships between nodes. Using a CF approach, drug recommendations are created as follows:

Filter out patients who are similar to the active patient in terms of gender (male/female), aura (yes/no), and the type of migraine (acute/chronic).

Calculate the similarity level between each neighbor and the active patient according to the following features: age, allergies, disease history, preexisting conditions, current drug prescription, and blood pressure. Each feature is weighted depending to its importance. For instance, age and disease history are more important than other features. Therefore, these features have a higher weight compared to others: w a g e = w d i s e a s e H i s t o r y = 3 and w a l l e r g i e s = w p r e e x i s t i n g C o n d i t i o n s = w b l o o d P r e s s u r e = 1.

Sum up all features’ scores. Only drugs consumed by the patients who are at least 80% similar to the current patient will be included in the recommendation.

Infectious Diseases

Shimada et al. (Shimada et al. 2005 ) developed a recommender system that helps doctors select proper first-line drugs for patients suffering from infectious diseases. Before giving suggestions, doctors have to know the ability of patients to protect themselves from risk factors. For this, a risk-level classification method that utilizes clinical information of patients is applied. This method assigns each risk factor to a score representing its impact degree on the patient. Besides, a knowledge base consisting of risk factors and their impact degrees are also constructed. The system returns risk levels that are helpful for precisely predicting the patient’s health condition and then recommending to him/her appropriate drugs.

Other Diseases

Besides drug recommendations for specific diseases, plenty of recommender systems have been developed for undefined diseases. For instance, GalenOWL (Doulaverakis et al. 2012 ) allows doctors to search for drug information and recommends suitable drugs to patients based on their disease, allergies, and drug interactions in the past. This system uses ontologies and ICD-codes to store rules about drugs and their interactions. These rules are the system input to generate the most fitting drugs for patients. Based on the GalenOWL system, a semantic framework so-called Panacea (Doulaverakis et al. 2014 ) was developed to assist physicians in prescribing drugs according to drugs’ active substance indications and contraindications. Panacea generates drug recommendations based on standardized medical terminologies and rules describing drug-drug and drug-disease interactions. This system outperforms GalenOWL while guaranteeing the same recommendation quality. Similar to Panacea , SemMed (Rodríguez et al. 2009 ) was developed based on semantic web technologies. This system provides patients with correct drugs and treatment recommendations that are proper to heal a concrete pathology. Besides, it helps healthcare professionals avoid mistakes in the drug interaction process and discard factors causing risks to patients, such as drug allergies or contraindications.

4.2.2 Predict drug side effects

Drug side effects or adverse drug reactions (ADR) are one of the leading causes of morbidity and mortality in health care (Galeano and Paccanaro 2018 ). As reported by the American Institute of Medicine, unexpected drug side effects cause 100,000 deaths annually in the USA (Gurwitz et al. 2003 ). Thus, medical researchers have taken heed of developing systems for drug discovery (Zhang et al. 2016 ). One of the first ideas of drug side-effect predictions is to utilize structure-activity or quantitative structure-property relationships. For instance, Fliri et al. (Fliri et al. 2006 ) translated adverse effect data derived from 1,045 prescription drug labels into effect spectra, and then showed their utility for diagnosing induced effects of drugs. Fukuzaki et al. (Fukuzaki et al. 2009 ) designed a model to list drug side effects by searching for cooperative pathways shared among gene expression profiles. The general idea of this work is: “A drug is produced to affect a specific gene. However, if the drug inadvertently activates other genes, then it might cause side effects”. In this approach, each pathway is represented as a graph with vertexes and edges. Each vertex represents a gene that indicates an item-set showing a set of drugs or conditions activating the gene. Each edge indicates a gene interaction. Based on this graph, sub-pathways showing side effects can be found based on the item-sets (i.e., activation conditions) shared between them.

Recently, some methods based on machine-learning have been employed to predict potential side effects of drugs. “In silico” is the most common method which creates side-effect predictions based on the structure chemistry and biological features of drugs, such as target proteins, protein-protein interactions, or gene ontology annotations (Zhang et al. 2015 ). Bresso et al. (Bresso et al. 2013 ) used this method to characterize side-effect profiles shared by several drugs. Huang et al. (Huang et al. 2011 ) utilized the drug targets, protein-protein interactions, and gene ontology annotations, and after that applied the support vector machine and logistic regression techniques to create predictions. Yamanishi et al. (Yamanishi et al. 2012 ) combined drug structures (from chemical profiles) and target proteins (from biological profiles) and then adopted the Sparse Canonical Correlation to predict potential side-effect profiles of drug candidate molecules.

The prediction methods mentioned above face some limitations concerning the availability of chemistry structures, considerable required computational power, and a high amount of false positives (Deshpande and Butte 2011 ). Besides, they are usually done in clinical trials, where many side effects could not be detected until drugs are approved. This raises a critical need to predict potential or missing side effects for drugs (Zhang et al. 2016 ). A few drug recommender systems were developed to address this need. One example thereof was proposed by Zhang et al. (Zhang et al. 2016 ), in which the potential side-effect prediction is formed as a recommendation task. An integrated neighborhood-based method is applied to make predictions. This method is an extension of the classic neighborhood-based recommendation , which utilizes known side effects of similar drugs. We will present the detail of this recommendation method using the following example.

Given a target drug d , a list of four approved drugs { d 1 , d 2 , d 3 , d 4 }, and corresponding side effects as shown in Table  3 , we predict the probability of s 5 and s 6 to be the side effects of drug d . The prediction is formulated as a recommendation problem, in which drugs, side effects, and drug-side effects associations are combined. The prediction process is conducted in the following steps:

Step 1 : Calculate drug-drug similarity based on side effect profiles. Given two drugs d i and d k whose side effect profiles are S i and S k , the Jaccard similarity is used to calculate their similarity s i m ( i , k ) (see Formula ( 2 )).

\(sim(d,d_{1}) = \frac {2}{4} = \textbf {0.5} \checkmark \) ; \(sim(d,d_{2}) = \frac {1}{4} = 0.25\)

\(sim(d,d_{3}) = \frac {3}{4} = \textbf {0.75} \checkmark \) ; \(sim(d,d_{4}) = \frac {1}{4} = 0.25\)

Step 2 : A set of neighbor drugs of the target drug d are determined by filtering similarity scores with a pre-defined threshold 𝜃 . In this example, we assume 𝜃 = 0.5, which means only drugs d 1 and d 3 are selected to be the neighbors of d .

Step 3 : Calculate the probability of drug d i inducing side effect s j - p r o b ( d i , s j ) by aggregating the known side effect s j of its neighbors (see Formula ( 3 )).

\(prob(d,s_{5}) = \frac {0 \times 0.5 + 1 \times 0.75}{0.5 + 0.75} = 0.6\) ; \(prob(d,s_{6}) = \frac {1 \times 0.5 + 1 \times 0.75}{0.5 + 0.75} = \textbf {1} \checkmark \)

The probability p r o b ( d , s 6 ) = 1 > p r o b ( d , s 5 ), meaning that s 6 is chosen as the potential side effect of the target drug d .

4.3 Health status prediction

In the past decades, predicting risks concerning specific diseases has become an intensive research topic (Davis et al. 2009 ), where the number of related studies for chronic diseases has been increasing significantly. The reason lies in the rapid growth of these diseases worldwide (Hussein et al. 2012 ). Long-term diseases prevent patients from physical activities and trigger burdens concerning time and money of the treatment process (Nasiri et al. 2016 ). To help patients avoid these diseases, HRS have been developed to detect disease symptoms as early as possible. Moreover, they can assist healthcare professionals in making proper treatment plans for patients. Davis et al. (Davis et al. 2009 ) and Nasiri et al. (Nasiri et al. 2016 ) proposed recommender systems to predict risk factors (e.g., potential complications or further diseases ) that a target patient with a chronic disease would face in the future. These systems applied CF, which is based on the intuitive assumption “patients who share similar diseases and health status might face the same risk factors”. Predictions of disease risks can be generated based on a set of similar patients’ information. The traditional CF technique is modified to make it suitable for the healthcare domain. The reason for this modification lies in the rating values of items. The patients’ ratings are non-ordinal values; they are binary ( 1/0 - the patient is facing/not facing a risk factor j ). For this approach, given an active patient a , a set of patients I , and a set of risk factors J , the risk factor prediction is generated in the following steps:

Step 1 : Calculate the similarity between patient a and each patient i ∈ I using Formula ( 4 ), where v a , j is the vote of patient a for risk factor j :

Step 2 : Find the most similar patients to patient a based on the similarity scores. The most similar patient has the greatest similarity score.

Step 3 : Calculate the prediction score of a risk factor j which have not been faced by patient a using Formulae ( 5 ) - ( 7 ), where p ( a , j ) is the prediction score for the patient a on risk factor j , \(\overline {v_{j}}\) is the average vote of all patients who have faced risk factor j , w ( a , i ) is the similarity between patients a and i (see Formula ( 4 )), and | I j | is the number of patients who have faced risk factor j . The normalized constant k ensures the prediction score within the range of possible votes.

For a better understanding, we exemplify the mentioned approach using a specific example. Assume Maria is an active patient suffering from the diabetes disease. She is now facing some risk factors, such as nerve damage , eye damage , slow healing , skin issues . p 1 ... p 4 are the patients who share similar profiles to Maria (see Table  4 ). To predict which risk factors that Maria might face in the future, the prediction scores for the not-been-faced risk factors are calculated: p(Maria, kidneyDamage) = 0.89, p(Maria, hearingImpairment) = 0.90 , p(Maria, heartDisease) = 0.69 . This shows Maria might face “hearing impairment” in the near future.

Besides recommendation techniques, machine learning approaches have been employed to generate disease predictions. For instance, Lafta et al. (Lafta et al. 2015 ) proposed an innovative time series prediction algorithm to support the decision making process of heart-disease patients. Particularly, the algorithm helps to decide whether a medical measurement, such as a heart-rate test, needs to be taken today based on the patient’s measurement readings for the past k days. Hussein et al. (Hussein et al. 2012 ) presented a Chronic Disease Diagnosis (CDD) recommender system using the Random Forest - RF classification model (Özcift 2011 ). Footnote 9 This system requires three types of input information to build up predictions for undiagnosed patients: (1) training data consisting of medical records of previous diabetic patients; (2) demographic data showing the patient’s profile, such as name, age, and education level ; and (3) the medical data of active patient referring to two types of tests: home-tests (e.g., blood sugar level, blood pressure, and weight) and lab-tests from the laboratory. The system’s output information includes a prediction representing the patient’s disease risks and a recommendation showing disease risk status acknowledgment that the patient is looking for.

4.4 Physical activity recommendation

Besides recommendations of disease treatment plans, suggestions on physical activities have become another focus of HRS. Physical-activity recommendations help to decrease the probability of becoming frail of patients and prevent them from further health complications (Valdez et al. 2016 ). Moreover, they also encourage users to follow daily activities that meet their calorie-burn goals. Runner (Donciu et al. 2011 ) and Shade (Faiz et al. 2014 ) provide users with food and exercise recommendations to keep them stay healthily. These studies generate a recommendation based on the fact that “what and when you eat during and after exercise can be just as important” (Donciu et al. 2011 ). These recommendations are tailored based on users’ health status, goals, and preferences, which are usually collected from different sources, such as foods, physical activities, elderly/diabetes/runner domains, user-health state, and user preferences. Therefore, ontologies and semantics technologies (Orgun and Vu 2006 ) are utilized to address the heterogeneous issues of user data. The recommendation process can be done as follows: First, an initial set of exercises for the user is selected based on his/her physical health status and exercise goal. Thereafter, the usage history and prior feedback regarding difficulty and enjoyment levels are used to adapt the selected exercises before sending it to the user.

Dharia et al. (Dharia et al. 2016 ) proposed a system to suggest personalized workout session recommendations based on the contextual data of users, such as past activities, preferences, and physical state. The recommendation process is performed as follows. The user first enters his/her contextual data. Thereafter, the system collects all the contacts and calendars events from the user’s device and employs a hybrid approach to recommend fitness sessions to the user. This approach combines CB and CF recommendations, in which the CB considers the user’s preferences, and the CF considers the preferences of similar users. The system also offers available slots in the user’s calendar so that he/she can re-schedule sessions anytime.

Imran Ali et al. (Ali et al. 2018 ) developed a hybrid framework that provides physical activity and diet recommendations using context-aware recommendation (Verbert et al. 2012 ) and knowledge-based recommendation (Burke 2000 ). The proposed framework consists of a multi-stage recommender system which supports the following modules:

Module 1 (Data acquisition and processing) , which stores the demographic information and preferred activities of users collected from sensory devices.

Module 2 (Context generation) , which saves the current activity, location, weather conditions, and emotional state of the user.

Module 3 (Expert knowledge repository) , which represents rules as IF-THEN form, which are then adopted to create recommendations. For instance, “IF a patient is pregnant and facing the gestational diabetes mellitus, THEN she should do a 20-30 minute moderate-intensity exercise on almost every day of the week” (Colberg et al. 2016 ).

Module 4 (Multi-stage recommender) , which utilizes the user information collected from Modules 1 and 2 to create a comprehensive recommendation to the user. The recommendation process is done in two stages. In Stage 1 , the system calculates the user’s calorie-burn, in-take targets, and a generic set of physical activity recommendations. Additionally, a case-based reasoning mechanism is used to infer the most relevant rules from the knowledge-base. In Stage 2 , the recommendations generated in Stage 1 are refined in a personalized manner. A contextual matrix is created to recommend suitable activities to the user at a given time. This matrix is calculated based on the user’s surveyed results to filter out proper physical activities in different contexts. For instance, “since the user is now staying at home, stretching seems to be more appropriate for him than running”.

Module 5 (Explanations of suggested activities) , which are sent together with recommendations to describe as to why a specific physical activity has been recommended to the user. For instance, “you should run at least one hour daily to improve your current health condition and meet one of your calorie-burn targets”. Additional explanations based on the context can also be provided, e.g., “it is quite cold today, hence consider to bring a sports jacket with you before going out”.

4.5 Healthcare professional recommendations

In recent years, there has been a significant increase in the amount of available medical information, which results in some difficulties for patients when searching for suitable doctors. What concerns patients greatly is how to find medical professionals with the best expertise for resolving their health issues (Hoens et al. 2010 ; Narducci et al. 2015 ). Most existing healthcare providers do not provide patients with full infrastructure or service design implementations that assist them in fulfilling this task. This gap raises an open topic on patient-doctor matchmaking, in which patients can find the right doctors to build a trust relationship (Han et al. 2018 ). Han et al. (Han et al. 2018 ) proposed a hybrid recommender system, in which family-doctor recommendations are made based on the level of available information about users. The authors discussed three use cases of generating recommendations:

Use case 1 (New patient): The patient has recently joined the network, and only basic demographic information is available. The CB recommendation is used to create recommendations based on similar demographic profiles.

Use case 2 (Existing patient with no interactions with primary care doctors): The patient has already visited specialists or hospitals, but has not visited family doctors yet. The activities of other patients in previous visits are utilized to narrow down the doctor list. Besides, a complementary data set describing hospital inpatient procedures and certain types of diseases of patients are used to create the patient profiles and then generate recommendations using the CB recommendation approach.

Use case 3 (Existing patient with prior interactions with primary care doctors): The CF recommendation approach is applied to look for doctors visited by similar patients (i.e., patients who have visited the same doctors earlier).

Zhang et al. (Zhang et al. 2016 ) proposed an iDoctor system to provide users with personalized doctor recommendations. This system explores the emotions and preferences of users about doctors through their ratings and reviews. Three modules are integrated into the system: sentiment analysis, topic modeling, and hybrid matrix factorization . The sentiment analysis module calculates the emotional offset from user reviews. The topic modeling module extracts user preferences and doctor features (e.g., specialty, fee range, and prescribing habits) from user reviews. The extracted information is used in the hybrid matrix factorization module to predict the rating for doctors.

Gujar et al. (Gujar et al. 2018 ) proposed a recommendation approach based on the “word of mouth” recommendation (e.g., asking friends or relatives), which is often used in reality to find doctors. The authors developed a recommender system to identify the location, contact, and other necessary information of medical specialists. They used the CoreNLP technique to generate doctor recommendations based on the review of previous users. The recommended doctors are filtered out based on some criteria, such as fewer fees, more experienced, nearest location , and feedback reviews of doctors . Different from the studies discussed earlier, this system allows patients to give their feedback on recommended doctors, which are then used to improve the quality of future recommendations. Based on a similar recommendation mechanism as mentioned in (Gujar et al. 2018 ), Narducci et al. (Narducci et al. 2015 ) presented a social network so-called HealthNet , where a recommendation component is integrated to suggest doctors and hospitals which best fit a specific patient profile. In HealthNet , a patient enters his/her health data, such as conditions , treatments (drugs, surgeries, or side effects), health indicators (blood pressure, body weight, laboratory analysis, etc.), consulted doctors, and hospitalizations . Based on the input data, the system search for similar patients stored in the database. The similarity between the active patient p and another patient p ′ in the database is estimated using Formula ( 8 ), where:

k and n are the numbers of conditions of patients p and p ′ respectively.

z and r are the numbers of treatments of patients p and p ′ respectively.

p c and p c ′ are the conditions of patients p and p ′ respectively.

p t and p t ′ are the treatments of patients p and p ′ respectively.

\(s_{c}(p_{c_{i}},p'_{c_{j}})\) is the similarity score between the condition c i of patient p and the condition c j of patient p ′ (see Formula ( 9 )). If these two conditions are the same, then this score is the logarithm of the ratio between the number of conditions in the database ( # C ) and the number of patients affected by that condition ( \(P_{c_{i}}\) ). Otherwise, the s c is computed as the number of edges in the shortest path s p , which connects the two conditions in the disease hierarchy Footnote 10 . The idea of this rule is to figure out whether two patients are affected by similar disease conditions. For instance, dilated cardiomyopathy and coronary artery conditions of two patients can be considered the same since they both refer to heart-muscle failures. In this context, the experiences of consulted doctors/hospitals of this patient could be useful for another (Narducci et al. 2015 ).

\(s_{t}(p_{t_{i}}, p^{\prime }_{t_{j}})\) is the similarity score between the treatment t i of patient p and the treatment t j of patient \(p^{\prime }\) (see Formula ( 10 )).

α refers to the contribution of conditions and treatments to patients’ similarity.

β indicates the weight of the community (patients) and the ministry indicator.

Given an active patient p i , the relevant doctors and hospitals for this patient can be estimated using Formulae ( 11 ) and ( 12 ).

For the demonstration purposes, we present a simple example to show how relevant doctors and hospitals can be suggested to the patient using the mentioned approach. Assume an active patient p i is having heart disease and suffering from condition c so-called dilated cardiomyopathy . This patient was applied the treatment t - nitrates . Now, he needs doctor and hospital recommendations that can effectively resolve his health problems. These recommendations can be generated based on the relevant information from other patients in the system. Assume two heart-disease patients ( p 1 and p 2 ) have visited doctors X and Y from hospitals A and B respectively. The information of these patients is summarized in Table  5 .

Patient p 1 has a higher similar disease condition to patient p i compared to patient p 2 ( p i and p 1 have trouble with heart muscle failures). Therefore, we assume the distance in the disease hierarchy tree between p i and p 1 is 2 and between p i and p 2 is 3. Based on Formula ( 9 ), the condition similarities would be s p ( c i , c 1 ) = 1/2 and s p ( c i , c 2 ) = 1/3. We assume disease conditions and treatments have the same impacts on the patient similarity scores (i.e., α = 0.5), and the community and ministry have the same weights (i.e., β = 0.5). The ratings of the ministry for hospitals A and B are q ( A ) = 4.0 and q ( B ) = 4.5 respectively. The necessary calculations are presented below, which show that doctor X and hospital A are recommended to patient p i .

s c ( p i , p 1 ) = 1/2; s c ( p i , p 2 ) = 1/3; s t ( p i , p 1 ) = 0; s t ( p i , p 2 ) = 0

s ( p i , p 1 ) = (0.5 ∗ 0.5)/2 = 0.125; s ( p i , p 2 ) = (0.5 ∗ 0.33)/2 = 0.083

s c o r e D o c t o r ( X , p i ) = 0.125 ∗ 4.1= 0.513 \(\checkmark \)

s c o r e D o c t o r ( Y , p i ) = 0.083 ∗ 4.5 = 0.375

s c o r e H o s p i t a l ( A , p i ) = 0.5 ∗ 0.125 ∗ 4.2 ∗ 0.5 ∗ 4= 0.525 \(\checkmark \)

s c o r e H o s p i t a l ( B , p i ) = 0.5 ∗ 0.083 ∗ 4.8 ∗ 0.5 ∗ 4.5 = 0.45

5 Evaluating health recommender systems

The most common evaluation method applied in the aforementioned recommendation approaches is offline evaluation (Trattner et al. 2018 ), estimating the prediction quality of a recommendation approach using existing data sets. With this method, accuracy metrics are used to compare recommendations determined by a recommender system with a predefined set of real-world user opinions (also known as ground truth (Shani and Gunawardana 2011 )). For instance, Achananuparp et al. (Achananuparp and Weber 2016 ) constructed a real-world food consumption from MyFitnessPal’s public food diary entries Footnote 11 and obtained group truth judgments of food substitutes from a crowdsourcing service. The authors used classification metrics “precision”, “mean average precision”, and “normalized discounted cumulative gain” to measure the method accuracy. Similar evaluation methods were applied in (Galeano and Paccanaro 2018 ; Hussein et al. 2012 ; Ueta et al. 2011 ; Yamanishi et al. 2012 ; Zhang et al. 2016 ), where metrics “precision” and “recall” were used to evaluate and compare the prediction performance of the recommendation algorithms. Besides classification metrics, error metrics (Trattner et al. 2018 ) were also employed to measure the error made by a recommender system when predicting an item rating (Galeano and Paccanaro 2018 ; Hussein et al. 2012 ; Narducci et al. 2015 ; Zhang et al. 2016 ). For instance, Narducci et al. (Narducci et al. 2015 ) carried out a preliminary evaluation, where the “Mean Absolute Error” was computed to compare their semantic approach based on the disease hierarchy to a simple string matching baseline. Another offline evaluation approach was cross validation (Dubitzky 2009 ) that allows to evaluate the performance of recommendation algorithms (Bresso et al. 2013 ; Elsweiler et al. 2017 ; Han et al. 2018 ; Zhang et al. 2016 ). Han et al. (Han et al. 2018 ) determined hyper-parameters for their model by performing a temporal cross-validation , which chronologically splits the data into train and test sets over the years. To consider item relevance and item position in a recommendation list, Achananuparp et al. (Achananuparp and Weber 2016 ) computed the “ discounted cumulative gain (DCG) ” metric based on the idea that items appearing lower in a recommendation list should be personalized by downgrading relevance values logarithmically (Trattner et al. 2018 ).

Compared to offline evaluation, much lesser number of studies employed online evaluation to test recommendation algorithms’ accuracy in HRS (Berkovsky and Freyne 2010 ; Donciu et al. 2011 ; Ueta et al. 2011 ). The idea of online evaluation is to use A/B testing or laboratory studies to evaluate an algorithm, a user interface, or the whole system (Trattner et al. 2018 ). For instance, using a dataset of explicit recipe ratings, Berkovsky et al. (Berkovsky and Freyne 2010 ) conducted a user study to observe families’ interaction with an experimental eHealth portal. This study aimed to uncover a recommendation strategy that could be most suitable for implementing an aggregation strategy in a group recommender system. Another approach was “direct test” , which was employed in some studies on drug recommendations (Doulaverakis et al. 2012 ; Mahmoud and Elbeh 2016 ). These tests were conducted with medical experts (e.g., doctors, clinicians, physicians, or nurses), where they were asked for feedback on the preciseness of recommendation outcomes. Mahmoud et al. (Mahmoud and Elbeh 2016 ) carried out a study in which experts evaluated recommendation results of the developed recommender system using a specific number of data sets. After collecting experts’ feedback, the precision was calculated. This value indicates the exactness or the quality of recommendation results. A true positive rate shows that the expert agreed with the recommendation result. In contrast, a false positive rate indicates that the expert disagreed with the recommendation result (Mahmoud and Elbeh 2016 ).

In summary, most of the existing studies discussed in this article use evaluation methods that have been previously developed in traditional recommendation domains and mainly focus on evaluating the accuracy of recommendation algorithms. However, the quality of HRS should be further evaluated according to other features beyond the accuracy, such as trust, causability, robustness, privacy, ethics, user satisfaction, uncertainty, effectiveness , and in-situ evaluation . Up to now, how to employ evaluation methods considering the mentioned evaluation perspectives has remained an open issue. For further discussion on the mentioned perspectives, see Section  6.4 .

6 Open issues for future work

Although the current literature has shown many benefits of HRS to improve their health conditions, there still exist some gaps regarding developing and evaluating HRS that need to be bridged. In the following, we discuss some research challenges that HRS face and corresponding solutions to tackle them.

6.1 Constructing user profiles

In HRS, besides user preferences that are typically used in recommender systems, further user information should be collected to obtain relevant, diverse, and precise recommendations. Thereof are demographic information, current health condition, diseases/allergies, treatments/surgeries/diagnoses experienced in the past, physical activities, nutrition needs, eating habits, feelings, and experiences. Although many sources exist to accumulate this information, recording such information could not avoid faults (Mika 2011 ). Hence, it is critical to require a standard concerning data formats, the authenticity of data sources, and automated update intervals (Schäfer et al. 2017 ) to ensure the quality of obtained information. Besides, the user profile parameters could be conflict with each other (e.g., user preferences vs. health conditions). To guarantee optimal suggestions in terms of balancing between user satisfaction and healthiness, parameters in the user profile need to be deemed wholly and appropriately. In some cases, the parameters regarding health conditions should be taken precedence over those concerning user preferences. For instance, a user diagnosed with a high risk of having diabetes should not be recommended any food containing trans-fats, even this recommendation goes against his preference.

6.2 Early disease detection

Many reports show that patients suffering from chronic diseases or cancers are often not well perceived about their disease or treatment options until the disease situations fall in late stages (Schäfer et al. 2017 ). The late detection of such diseases causes a low probability of completely curing the disease, and in some cases, this could threaten patients’ lives (e.g., the late stage of cancers). In such a context, besides assisting patients in finding suitable treatment methods (Davis et al. 2009 ; Nasiri et al. 2016 ), HRS should offer users a health education functionality that helps to improve the perception of users concerning the diseases. Stettinger et al. (Stettinger et al. 2020 ) developed an e-learing application so-called KnowledgeCheckr Footnote 12 , which provides intuitive learning contents and suggests learning units in a personalized fashion. With this application, helpful information concerning disease descriptions and related symptoms can be transferred to patients. Moreover, HRS should analyze the underlying health condition of patients and predict early diseases that users might face in the near future. Also, necessary diagnoses and information of corresponding healthcare professionals should be delivered to patients. For chronic and life-threatening diseases, such early disease detection can minimize disease complications and the treatment process’ burdens.

6.3 Persuasive recommendations

One focus of HRS is to track users’ daily activities and motivate them to adjust their routines or habits positively. However, it seems to be a big challenge to change habits that have become so deeply entrenched over the years (Tran et al. 2018 ). Therefore, researchers have recently paid attention to develop persuasive systems, in which various strategies and persuasion principles are discovered to encourage users to adopt and maintain beneficial behaviors and attitudes. For instance, Thomas et al. (Thomas et al. 2017 ) investigated argument-based approaches in which motivating arguments are created to change the eating habit of users healthily. These studies indicate that it is necessary to produce persuasive arguments based on user attributes such as age, gender, or personality. Although these studies show positive effects on the behavior changing of users, it does not guarantee full acceptance of changes. The argumentation-based approaches have been proved to be sufficiently effective for patients in the late stages of the disease, whereas they show a lower effect for the patients in the early stages of the disease (Nguyen and Masthoff 2008 ). This raises an open issue of developing arguments that are strong, relevant, and convincing enough to bring actual changes for those in the early phases of health risks. Besides, while many efforts have been conducted to estimate the arguments’ perceived persuasiveness, measuring the actual persuasiveness of arguments is still an open issue. In fact, what people perceive to be persuasive is not necessarily what will persuade them to act. In the healthcare domain, this can be interpreted that users might be unwilling to change their behavior, even though they are aware of the risks triggered by unhealthy habits (Nguyen and Masthoff 2008 ). For instance, although some people may perceive the harmful effects of smoking, they are not ready to give it up. On the other hand, changing users’ behavior or attitude is a long-term process with plenty of steps. In this context, the question is “how to generate persuasive arguments that motivate users as much as possible” . In the healthy food domain, the answer to this question could be to develop food recommender systems, where theories from health psychology are integrated to stimulate users to comply with healthy eating behaviors (Schäfer and Willemsen 2019 ). One approach is to apply a simple change at a specific time until the user behavior becomes habitual. Another approach is to compare nutrients consumed by the user to the ones acquired from reliable sources (e.g., USDA, DACH Footnote 13 ) (Snooks 2009 ).

6.4 Further aspects for evaluating health recommender systems

Typically, the evaluation of recommender systems emphasizes the accuracy metrics (Powers 2011 ) (see also Section  5 ). However, in the healthcare domain, recommender systems’ quality needs to be measured based on aspects beyond the accuracy objectives (Valdez et al. 2016 ).

Trust is one of the most important criteria that should be considered when evaluating recommender systems (O’Donovan and Smyth 2005 ). This is even more critical for HRS to convince patients to follow health-related recommendations. This aspect can be enhanced by providing explanations for recommendations (Tran et al. 2019 ). Similar to other domains, explanations in the healthcare domain should show how a suggestion has been created for the user (Elahi et al. 2014 ), e.g., “According to the tests you did last week, we have detected that the level of uric acid in your blood is still really high. Therefore, r i seems to be the most appropriate recipe for you since it has no ingredients containing purines”. Besides, effective visualizations should be included in HRS to further explain recommended items (Valdez et al. 2016 ). For instance, in food recommender systems, a table showing the description of the nutrition value of food items should be provided to the users to emphasize the healthiness of a recommended recipe (Tran et al. 2018 ).

Causability helps users understand why specific recommendations have been made. This criterion is useful in many domains and especially crucial in the medical domain to enhance trust in the results and enable domain experts to retrace, understand, and explain why a particular recommendation was given. This does not necessarily mean that everything must be explained automatically, but that a domain expert has a chance to understand it on demand. To measure the understandability of recommendations, the concept of causability can be helpful. In the same way that usability encompasses measurements for the quality of use, causability encompasses measurements for the quality of explanations (Holzinger et al. 2019 ).

Robustness is related to the trustworthiness of a recommender system. In HRS, sometimes, end-users could not be differentiated from potential attackers, which causes a degradation of trust in the objectivity and accuracy of the system (Valdez et al. 2016 ). To ensure secure HRS for users, future studies should model potential attacks and investigate the impacts of such attacks on recommendation algorithms (Mobasher et al. 2007 ).

Privacy is referred to as the ability of HRS to preserve patients’ preferences and medical information. The leak of such information raises the doubts of patients and consequently decreases the willingness to share their sensitive medical data with HRS (Valdez et al. 2016 ). The most common approach to address the privacy concern is data encryption that provides data confidentiality while utilizing the user data to generate precise recommendations (Hoens et al. 2010 ). However, this method requires highly overhead computation- and communication-wise, which significantly decreases the performance of HRS (Verhaert et al. 2018 ). Although there exist some studies to improve the data encryption approach, some of them still face the issue concerning the low efficiency of the system (Hoens et al. 2010 ; Verhaert et al. 2018 ). Up to now, developing HRS ensuring trade-offs between the privacy and the high performance of recommendation algorithms is still an open issue.

Ethics has been raised in recommender systems to help users pick up morally appropriate items during the post-recommendation process (Tang and Winoto 2016 ). In HRS, ethics should be considered more strictly to prevent recommendations from directions that could harm the healthiness of patients (Valdez et al. 2016 ). The principle of “first do no harm” should be kept in mind when developing HRS to minimize potential risks and maximize benefits for users. The healthiness of patients is the most crucial criterion when creating recommendations, even this might be against patients’ preferences (Tang and Winoto 2016 ).

User satisfaction with recommendations can be different depending on user diversity. Some studies have been performed recently to have a deep look at modeling user satisfaction for the purpose of predicting satisfaction models (Chen et al. 2019 ; Nguyen et al. 2017 ). In HRS, it would make sense to investigate the relationship between health-related recommendations and users’ satisfaction from different user groups, e.g., patients, doctors, nurses, physicians, and medical researchers (Valdez et al. 2016 ). The differences in expertise, overview knowledge, and recommendation tasks of these users could influence their satisfaction with recommended items.

Uncertainty in HRS links to potential risks, such as imprecise predictions since user preferences are not always captured well, or the inability to find a perfect pattern because of incomplete data. The risks could result in a reduced quality of the patient’s life. Therefore, when developing HRS, system designers should find ways to visualize uncertainty in a set of recommendations, allowing users to evaluate the option adequately before making a final decision (Valdez et al. 2016 ).

Effectiveness is referred to as the ability of HRS to help patients meet their desired changes in health. To measure this aspect, we need to consider which health parameters to be assessed or which medical tests to be employed to ensure medical effectiveness. For instance, in HRS that support the lose-weight targets of users, the effectiveness should be assessed based on both short-term and long-term recommendations. The reason is that, in some cases, short-term recommendations could burden or conflict with long-term ones. For instance, crash diets could help a patient lose weight quickly since it cuts the calories too low and makes drastic changes regarding food types to be consumed. However, this reduces the metabolism of the patient’s body and consequently burdens the long-term weight loss (Valdez et al. 2016 ).

In-Situ Evaluation indicates real-life non-laboratory settings that have to be evaluated to prove the worthiness of HRS. This evaluation paradigm should be able to precisely evaluate the ability of HRS to improve the quality of care (concerning accuracy, relevance, and early diagnosis ) and reduce the cost of care. Besides, it should be capable of evaluating the robustness to false information and the ability to consider potential health risks based on various dimensions (e.g., age, culture, ethnicity, etc.). Moreover, long-term behavioral effects must also be investigated in-situ evaluation to address the complexity of health and health behaviors (Schäfer et al. 2017 ).

6.5 Bundle recommendation

In the healthy food domain, users might require recommendations of a complete meal with the combination of many recipes or a food schedule for more than one day (e.g., foods for next week). This issue is known as bundle recommendation , which is a new research branch of recommender systems. The idea here is to recommend a sequence of items instead of separated ones. Recommending a complete meal is quite complicated since the system has to consider not only the preferences of users but also other aspects, such as the meal variety, weather and season , the healthiness of recipes, health problems, or nutrition needs . Thus, approaches to generate bundle recommendations in the healthy food domain have remained an open issue.

6.6 Group decision making

For some scenarios (i.e., diet recommendation), recommended items could involve groups of users rather than individual users (e.g., recommend a menu for a Christmas party). The current literature shows a limited number of studies on food recommender systems for groups. Therefore, it is still an open topic that needs to be analyzed in future research. Group recommender systems usually attach the requirements/preferences of different users into group recommendation. This is the crucial idea discussed in many related studies (Berkovsky and Freyne 2010 ; Felfernig et al. 2018 ; O’Connor et al. 2001 ). Recommending a joint meal for a group of users is a complicated task since different goals and dietary constraints of group members should be taken into account. While we have a solution for merging the constraints (Atas et al. 2019 ), a solution for merging goals is still an open issue. Besides, recommendations generated for groups should assure fairness among group members, which means negotiation and argumentation mechanisms have to be developed to support group members in expressing acceptable trade-offs (Felfernig et al. 2014 ). For instance, in a meal plan for a week, users’ preferences ignored in previous meals should have a stronger influence on the upcoming meals. On the other hand, although different aggregation approaches have been applied to generate group recommendations, they do not ensure that recommended items reflect a high agreement level among group members (Castro et al. 2015 ). In this context, a consensus making process is needed to bring individual preferences closer to each other before delivering group recommendations. Further issues need to be considered to accelerate such a process. One promising solution is to enrich user interfaces that allow group members to share their preferences (Nguyen and Ricci 2017 ). Besides, psychological aspects (e.g., personality and emotions) beyond group members’ preferences are also crucial to be taken into account in group decision making. This draws an open topic regarding the influence of group members’ personality and emotions on group recommendation strategies (Quijano-Sanchez et al. 2013 ).

7 Conclusion

Health recommender systems have emerged as tools to support patients and healthcare professionals to make better health-related decisions. In this article, we have given insights into recommendation scenarios offered by these systems, such as food recommendation, drug recommendation, health status prediction, physical activity recommendation, and healthcare professional recommendation. For each recommendation scenario, various algorithms have been employed, which are based on recommendation techniques (e.g., CF, CB, KB, HyR, and context-based recommendations) or machine learning techniques (e.g., classification, clustering, decision tree, natural language processing, logic programming, ontologies, and semantic technologies). Although the proposed HRS bring many benefits in terms of health-related improvements, there still exist a number of challenges that need to be tackled for the better development of these systems in the future.

https://scholar.google.at/

https://link.springer.com/

https://dl.acm.org/

https://www.researchgate.net/

https://www.sciencedirect.com/

https://pubmed.ncbi.nlm.nih.gov/

https://en.wikipedia.org/wiki/MyFitnessPal

For details of these strategies, we refer to (Felfernig et al. 2018 ; Tran et al. 2018 ).

Other decision tree classification models such as J48 (Kaur and Raghava 2003 ), Decision Stump (Noh et al. 2003 ), REP tree (Park et al. 2006 ) have been applied to predict the risk factors of chronic diseases.

http://apps.who.int/classifications/icd10/browse/2010/en

https://www.myfitnesspal.com

https://www.knowledgecheckr.com

http://www.sge-ssn.ch .

Aberg, J. (2006). Dealing with malnutrition: a meal planning system for elderly. In AAAI Spring symposium: argumentation for consumers of healthcare, (pp. 1–7). AAAI .

Achananuparp, P., & Weber, I. (2016). Extracting food substitutes from food diary via distributional similarity. In Proceedings of the. 10th ACM Conference on Recommender Systems, (pp. 1–5) .

Ali, S.I., Amin, M.B., Kim, S., & Lee, S. (2018). A hybrid framework for a comprehensive physical activity and diet recommendation system. In Mokhtari, M., Abdulrazak, B., & Aloulou, H. (Eds.) Smart homes and health telematics, designing a better future: urban assisted living (ICOST 2018), (pp 101–109). Springer International Publishing, Cham .

Atas, M., Tran, T.N.T., Felfernig, A., Polat-Erdeniz, S., Samer, R., & Stettinger, M. (2019). Towards similarity-aware constraint-based recommendation. In Advances and trends in artificial intelligence, lecture notes in computer science, (pp. 287–299). Springer .

Bankhele, S., Mhaske, A., & Bhat, S. (2017). V., s.: a diabetic healthcare recommendation system. International Journal of Computer Applications , 167 , 14–18.

Article   Google Scholar  

Bao, Y., & Jiang, X. (2016). An intelligent medicine recommender system framework. In 2016 IEEE 11Th conference on industrial electronics and applications (ICIEA), (pp. 1383–1388) .

Berkovsky, S., & Freyne, J. (2010). Group-based recipe recommendations: Analysis of data aggregation strategies. In Proceedings of the Fourth ACM Conference on Recommender Systems, RecSys ’10, (pp 111–118). ACM, New York, NY, USA .

Bresso, E., Grisoni, R., Marchetti, G., Karaboga, A., Souchet, M., Devignes, M.D., & Smail, M. (2013). Integrative relational machine-learning approach for understanding drug side-effect profiles. BMC bioinformatics , 14 , 207.

Burke, R. (2000). Knowledge-based recommender systems. In Encyclopedia of library and information systems, vol. 69, (pp. 180–200). Marcel dekker .

Cami, A., Arnold, A., Manzi, S., & Reis, B. (2011). Predicting adverse drug events using pharmacological network models. Science translational medicine , 3 , 114–127.

Castro, J., Quesada, F.J., Palomares, I., & Martínez-López, L. (2015). A consensus-driven group recommender system. International Journal of Intelligent Systems , 30 (8), 887–906.

Charkhat Gorgich, E.A., Barfroshan, S., Ghoreishi, G., & Yaghoobi, M. (2015). Investigating the causes of medication errors and strategies to prevention of them from nurses and nursing student viewpoint. Global Journal of Health Science , 8 , 220.

Chen, L., & Pu, P. (2012). Critiquing-based recommenders: survey and emerging trends. User Modeling and User-Adapted Interaction , 22 (1-2), 125–150.

Chen, L., Yang, Y., Wang, N., Yang, K., & Yuan, Q. (2019). How serendipity improves user satisfaction with recommendations? a large-scale user evaluation. In The world wide web conference, WWW’19, (pp 240–250). Association for Computing Machinery, New York, NY, USA .

Chen, R.C., Chiu, J.Y., & Bau, C.T. (2011). The recommendation of medicines based on multiple criteria decision making and domain ontology — an example of anti-diabetic medicines. In The Proceedings of the International Conference on Machine Learning and Cybernetics, vol. 1, (pp. 27–32) .

Chen, R.C., Huang, Y.H., Bau, C.T., & Chen, S.M. (2012). A recommendation system based on domain ontology and SWRL for anti-diabetic drugs selection. Expert Systems with Applications , 39 (4), 3995–4006.

Colberg, S., Sigal, R., Yardley, J., Riddell, M., Dunstan, D., Dempsey, P., Horton, E., Castorino, K., & Tate, D. (2016). Physical activity/exercise and diabetes: a position statement of the american diabetes association. Diabetes Care , 39 , 2065–2079.

Davis, D.A., Chawla, N.V., Christakis, N.A., & Barabási, A. L. (2009). Time to care: a collaborative engine for practical disease prediction. Data Mining and Knowledge Discovery , 20 , 388–415.

Article   MathSciNet   Google Scholar  

Deshpande, T., & Butte, A. (2011). Exploiting drug-disease relationships for computational drug repositioning. Briefings in bioinformatics , 12 , 303–11.

Dharia, S., Jain, V., Patel, J., Vora, J., Chawla, S., & Eirinaki, M. (2016). Pro-fit: a personalized fitness assistant framework. In SEKE .

Donciu, M., Ionita, M., & Dascalu, M. (2011). Trausan-matu, S.: The runner - recommender system of workout and nutrition for runners. 2011 13th International Symposium on Symbolic and Numeric Algorithms for Scientific Computing (SYNASC) , (pp. 230–238).

Doulaverakis, C., Nikolaidis, G., Kleontas, A., & Kompatsiaris, I. (2012). Galenowl: Ontology-based drug recommendations discovery. Journal of biomedical semantics , 3 , 14.

Doulaverakis, C., Nikolaidis, G., Kleontas, A., & Kompatsiaris, I. (2014). Panacea, a semantic-enabled drug recommendations discovery framework. Journal of biomedical semantics , 5 , 13.

Dubitzky, W. (2009). Fundamentals of data mining in genomics and proteomics . Berlin: Springer.

MATH   Google Scholar  

Elahi, M., Ge, M., Ricci, F., Massimo, D., & Berkovsky, S. (2014). Interactive food recommendation for groups. In RECSYS, vol. 1247 .

Elsweiler, D., Trattner, C., & Harvey, M. (2017). Exploiting food choice biases for healthier recipe recommendation. In Proceedings of the 40th Conference on Research and Development in Information Retrieval, SIGIR ’17, (pp 575–584). Association for Computing Machinery, NY, USA .

Faiz, I., Mukhtar, H., & Khan, S. (2014). An integrated approach of diet and exercise recommendations for diabetes patients. In IEEE 16th International Conference on e-Health Networking, Applications and Services (Healthcom), pp 537–542 .

Felfernig, A., Atas, M., Helic, D., Tran, T.N.T., Stettinger, M., & Samer, R. (2018). Group Recommender systems: An Introduction, chap. Algorithms for Group Recommendation , (pp. 27–58). Berlin: Springer.

Book   Google Scholar  

Felfernig, A., & Burke, R. (2008). Constraint-based recommender systems: Technologies and research issues. In Proceedings of the 10th International Conference on Electronic Commerce, ICEC’08, (pp. 3:1–3:10). ACM, New York, NY, USA .

Felfernig, A., & Gula, B. (2006). An empirical study on consumer behavior in the interaction with knowledge-based recommender applications. In The 8th IEEE international conference on e-commerce technology and the 3rd IEEE international conference on enterprise computing, e-commerce, and e-services (CEC/EEE’06), (pp. 37–37) .

Felfernig, A., Stettinger, M., Ninaus, G., Jeran, M., Reiterer, S., Falkner, A.A., Leitner, G., & Tiihonen, J. (2014). Towards open configuration. In Proceedings of the 16th International Configuration Workshop, Novi Sad, Serbia, September 25-26, 2014., (pp. 89–94) .

Fliri, A., Loging, W., & Thadeio, P. (2006). Analysis of drug-induced effect patterns to link structure and side effects of medicines. Nature chemical biology , 1 , 389–97.

Fukuzaki, M., Seki, M., Kashima, H., & Sese, J. (2009). Side effect prediction using cooperative pathways. In IEEE International conference on bioinformatics and biomedicine, (pp. 142–147) .

Galeano, D., & Paccanaro, A. (2018). A recommender system approach for predicting drug side effects. In IJCNN 2018: International joint conference on neural networks, (pp. 1–7). IEEE xplore .

Ge, M., Ricci, F., & Massimo, D. (2015). Health-aware food recommender system. In Proceedings of the 9th ACM Conference on Recommender Systems, RecSys ’15, (pp. 333–334). Association for Computing Machinery, New York, NY, USA .

Gujar, D., Biyani, R., Bramhane, T., Bhosale, S., & Vaidya, T.P. (2018). Disease prediction and doctor recommendation system. International Research Journal of Engineering and Technology (IRJET) , 5 , 3207–3209.

Google Scholar  

Gurwitz, J., Field, T., Harrold, L., Rothschild, J., DeBellis, K., Seger, A., Cadoret, C., Fish, L., Garber, L., Kelleher, M., & Bates, D. (2003). Incidence and preventability of adverse drug events among older persons in the ambulatory setting. JAMA: Journal of the American Medical Association , 289 , 1107–16.

Han, Q., Ji, M., Martínez de Rituerto de Troya, I., Gaur, M., & Zejnilovic, L. (2018). A hybrid recommender system for patient-doctor matchmaking in primary care. In The 5th IEEE international conference on data science and advanced analytics (DSAA), (pp. 1–10) .

Hoens, T.R., Blanton, M., & Chawla, N.V. (2010). Reliable medical recommendation systems with patient privacy. In Proceedings of the 1st ACM International Health Informatics Symposium, IHI ’10, (pp. 173–182). Association for Computing Machinery, New York, NY, USA .

Holzinger, A., Carrington, A., & Mu̇ller, H. (2019). Measuring the quality of explanations: The system causability scale (scs) - comparing human and machine explanations. KI - Künstliche Intelligenz (German Journal of Artificial intelligence) Special Issue on Interactive Machine Learning 24 (2).

Holzinger, A., Valdez, A.C., & Ziefle, M. (2016). Towards interactive recommender systems with the doctor-in-the-loop. In Weyers, B., & Dittmar, A. (Eds.) Mensch und computer 2016 – workshopband. Gesellschaft für Informatik e.V., Aachen .

Huang, L.C., Wu, X., & Chen, J. (2011). Predicting adverse side effects of drugs. BMC genomics 12 Suppl , 5 , S11.

Hussein, A.S., Omar, W.M., Li, X., & Ati, M. (2012). Accurate and reliable recommender system for chronic disease diagnosis. In The first international conference on global health challenges - global health 2012, (pp. 113–118) .

Jannach, D. (2011). Recommender systems : an introduction Cambridge University Press.

Kaur, H., & Raghava, G.P.S. (2003). A neural-network based method for prediction of gamma-turns in proteins from multiple sequence alignment. Protein Society , 12 , 923–929.

Lafta, R., Zhang, J., Tao, X., Li, Y., & Tseng, V.S. (2015). An intelligent recommender system based on short-term risk prediction for heart disease patients. In 2015 IEEE/WIC/ACM International conference on web intelligence and intelligent agent technology (WI-IAT), vol. 3, (pp. 102–105) .

Lops, P., Gemmis, M.D., & Semeraro, G. (2011). Content-based Recommender systems: State of the Art and Trends , (pp. 73–105). Boston: Springer.

Mahmoud, N., & Elbeh, H. (2016). IRS-t2d : Individualize recommendation system for type2 diabetes medication based on ontology and SWRL . In Proceedings of the 10th International Conference on Informatics and Systems, INFOS ’16, (pp. 203–209). Association for Computing Machinery, NY, USA .

Medvedeva, O., Knox, T., & Paul, J. (2007). Diatrack: web-based application for assisted decision-making in treatment of diabetes. Journal of Computing Sciences in Colleges , 23 , 154–161.

Mika, S. (2011). Challenges for nutrition recommender systems. In Workshop Proceedings on Context Aware Intelligent Assistance, (pp. 25–33). CEUR-WS.org .

Mobasher, B., Burke, R., Bhaumik, R., & Williams, C. (2007). Toward trustworthy recommender systems: An analysis of attack models and algorithm robustness. ACM Transactions on Internet Technology , 7 (4), 23–es.

Narducci, F., Musto, C., Polignano, M., de Gemmis, M., Lops, P., & Semeraro, G. (2015). A recommender system for connecting patients to the right doctors in the healthnet social network. In WWW’2015 Companion, (pp. 81–82) .

Nasiri, M., Minaei, B., & Kiani, A. (2016). Dynamic recommendation: Disease prediction and prevention using recommender system. International Journal of Basic Science in Medicine, (pp. 13–17).

Nguyen, H., & Masthoff, J. (2008). Designing persuasive dialogue systems: Using argumentation with care. In H. Oinas-kukkonen, p.f.v. hasle, M. Harjumaa, K. Segerståhl, P. Øhrstrøm (eds.) Persuasive Technology, Third International Conference, PERSUASIVE 2008, Oulu, Finland, June 4-6, 2008. Proceedings, Lecture Notes in Computer Science, vol. 5033, (pp. 201–212). Springer .

Nguyen, T., Harper, F., Terveen, L., & Konstan, J. (2017). User personality and user satisfaction with recommender systems. Information Systems Frontiers , 20 , 1–17.

Nguyen, T.N., & Ricci, F. (2017). A chat-based group recommender system for tourism. Information Technology & Tourism , 18 (2), 5–28.

Noh, H.J., Kwak, M.J., & Han, I.G. (2003). Handling incomplete data problem in collaborative filtering system. Journal of Intelligent Information Systems , 9 , 51–63.

O’Connor, M., Cosley, D., Konstan, J.A., & Riedl, J. (2001). Polylens: a recommender system for groups of users. In Proceedings of the Seventh Conference on European Conference on Computer Supported Cooperative Work, ECSCW’01, (pp. 199–218). Kluwer Academic Publishers, Norwell, USA .

O’Donovan, J., & Smyth, B. (2005). Trust in recommender systems. In International Conference on Intelligent User Interfaces, Proceedings IUI, (pp. 167–174) .

Orgun, B., & Vu, J. (2006). Hl7 ontology and mobile agents for interoperability in heterogeneous medical information systems. Computers in Biology and Medicine , 36 (7), 817–836. Special Issue on Medical Ontologies.

Özcift, A. (2011). Random forests ensemble classifier trained with data resampling strategy to improve cardiac arrhythmia diagnosis. Computers in biology and medicine , 41 , 265–71.

Park, J., Tyan, H., & Kuo, C.C.J. (2006). Internet traffic classification for scalable qos provision. In 2006 IEEE International conference on multimedia and expo, (pp. 1221–1224) .

Pincay, J., Terán, L., & Portmann, E. (2019). Health recommender systems: a state-of-the-art review. In 2019 Sixth international conference on edemocracy egovernment (ICEDEG), (pp. 47–55) .

Powers, D.M.W. (2011). Evaluation: from precision, recall and f-measure to roc., informedness, markedness & correlation. Journal of Machine Learning Technologies , 2 (1), 37–63.

MathSciNet   Google Scholar  

Quijano-Sanchez, L., Recio-Garcia, J.A., Diaz-Agudo, B., & Jimenez-Diaz, G. (2013). Social factors in group recommender systems. ACM Transactions on Intelligent Systems and Technology , 4 (1), 8:1–8:30.

Rehman, F., Khalid, O., Haq, N., Khan, A.U.R., Bilal, K., & Madani, S. (2017). Diet-right: a smart food recommendation system. KSII on Internet and Information Systems , 11 , 2910–2925.

Ricci, F., Rokach, L., Shapira, B., & Kantor, P.B. (2010). Recommender systems handbook , 1st edn. Berlin: Springer.

Robertson, A., Tirado, M., Lobstein, T., Jermini, M., Knai, C., Jensen, J., Ferro-Luzzi, A., & James, W. (2004). Food and health in europe: a new basis for action. WHO regional publications. European series (pp. i–xvi), 1.

Rodríguez, A., Jiménez, E., Fernández, J., Eccius, M., Miguel Gómez, J., Alor-Hernández, G., Posada-Gómez, R., & Laufer, C. (2009). Semmed: Applying semantic web to medical recommendation systems. 2009 First International Conference on Intensive Applications and Services , (pp. 47–52).

Rokicki, M., Herder, E., & Demidova, E. (2015). Vol What’s on my plate: Towards recommending recipe variations for diabetes patients. In Cristea, A.I., Masthoff, J., Said, A., & Tintarev, N. (Eds.) UMAP Workshops, CEUR Workshop Proceedings, vol 1388. CEUR-WS.org .

Sahoo, A.K., Pradhan, C., Barik, R.K., & Dubey, H. (2019). Deepreco: Deep learning based health recommender system using collaborative filtering. Computation , 7 , 25.

Sánchez-Bocanegra, C.L., Sánchez-Laguna, F., & Sevillano, J.L. (2015). Introduction on health recommender systems. Methods in molecular biology (Clifton N.J.) , 1246 , 131–46.

Schäfer, H., Hors-Fraile, S., Karumur, R.P., Calero Valdez, A., Said, A., Torkamaan, H., Ulmer, T., & Trattner, C. (2017). Towards health (aware) recommender systems. In Proceedings of the 2017 International Conference on Digital Health, DH ’17, (pp. 157–161). ACM, New York, NY, USA .

Schäfer, H., & Willemsen, M. (2019). Rasch-based tailored goals for nutrition assistance systems. In Proceedings of the 24th International Conference on Intelligent User Interfaces, IUI ’19, (pp. 18–29). Association for Computing Machinery, New York, NY, USA .

Sezgin, E., & Özkan, S. (2013). A systematic literature review on health recommender systems. In 2013 E-health and bioengineering conference, EHB 2013, (pp. 1–4) .

Shani, G., & Gunawardana, A. (2011). Evaluating recommendation systems. In Ricci, F., Rokach, L., Shapira, B., & Kantor, P.B. (Eds.) Recommender systems handbook, (pp. 257–297). Springer .

Shimada, K., Takada, H., Mitsuyama, S., Ban, H., Matsuo, H., Otake, H., Kunishima, H., Kanemitsu, K., & Kaku, M. (2005). Drug-recommendation system for patients with infectious diseases. Annual Symposium proceedings. AMIA Symposium , (pp. 11–12).

Snooks, M. (2009). Health psychology: Biological, Psychological, and Sociocultural Perspectives, chapter 5: Applications of Health Psychology to Eating behaviors: Improving health through nutritional changes Jones & Bartlett Learning.

Stark, B., Knahl, C., Aydin, M., & Elish, K. (2019). A literature review on medicine recommender systems. International journal of advanced computer science and applications 10 (8).

Stark, B., Knahl, C., Aydin, M., Samarah, M., & Elish, K.O. (2017). Betterchoice: a migraine drug recommendation system based on neo4j. In 2017 2Nd IEEE international conference on computational intelligence and applications (ICCIA), (pp. 382–386) .

Stettinger, M., Tran, T., Pribik, I., Leitner, G., Felfernig, A., Samer, R., Atas, M., & Wundara, M. (2020). KNOWLEDGECHECKR: Intelligent techniques for counteracting forgetting. In G.d. giacomo, A. Catalȧ, B. Dilkina, M. Milano, S. Barro, A. Bugarín, J. Lang (eds.) ECAI 2020 - 24th European Conference on Artificial Intelligence, 29 August-8 September 2020, Santiago de Compostela, Spain, August 29 - September 8, 2020 - Including 10th Conference on Prestigious Applications of Artificial Intelligence (PAIS 2020), Frontiers in Artificial Intelligence and Applications, vol. 325, (pp. 3034–3039). IOS Press .

Tang, T.Y., & Winoto, P. (2016). I should not recommend it to you even if you will like it: the ethics of recommender systems. New Review of Hypermedia and Multimedia , 22 (1-2), 111–138.

Teng, C.Y., Lin, Y.R., & Adamic, L.A. (2012). Recipe recommendation using ingredient networks. In Proceedings of the 4th Annual ACM Web Science Conference, WebSci ’12, (pp. 298–307). Association for Computing Machinery, New York, NY, USA .

Thomas, R.J., Masthoff, M., Oren, N., de Vries, P.W., Oinas-Kukkonen, H., Siemons, L., Jong, N.B., & van Gemert-Pijnen, L. (2017). Adapting healthy eating messages to personality. In Persuasive Technology: Development and Implementation of Personalized Technologies to Change Attitudes and Behaviors - 12th International Conference, PERSUASIVE 2017, Amsterdam, The Netherlands, April 4-6, 2017, Proceedings, Lecture Notes in Computer Science, vol. 10171, (pp. 119–132). Springer .

Tran, T.N.T., Atas, M., Felfernig, A., Le, V.M., Samer, R., & Stettinger, M. (2019). Towards social choice-based explanations in group recommender systems. In Proceedings of the 27th ACM Conference on User Modeling, Adaptation and Personalization, UMAP ’19, (pp. 13–21). Association for Computing Machinery, New York, NY, USA .

Tran, T.N.T., Atas, M., Felfernig, A., & Stettinger, M. (2018). An overview of recommender systems in the healthy food domain. Journal of Intelligent Information Systems , 50 (3), 501–526.

Trattner, C., & Elsweiler, D. (2017). Food recommender systems: Important contributions, challenges and future research directions. CoRR abs/1711.02760.

Trattner, C., Said, A., Boratto, L., & Felfernig, A. (2018). Evaluating Group Recommender Systems , (pp. 59–71). Manhattan: Springer International Publishing Cham.

Ueta, T., Iwakami, M., & Ito, T. (2011). A recipe recommendation system based on automatic nutrition information extraction. In Proceedings of the 5th international conference on knowledge science, engineering and management, KSEM’11, (pp. 79–90). Springer: Berlin .

Valdez, A.C., Ziefle, M., Verbert, K., Felfernig, A., & Holzinger, A. (2016). Recommender systems for health informatics: State-of-the-art and future perspectives. Lecture Notes in Computer Science , 9605 .

Verbert, K., Manouselis, N., Ochoa, X., Wolpers, M., Drachsler, H., Bosnic, I., & Duval, E. (2012). Context-aware recommender systems for learning: a survey and future challenges. IEEE Transactions on Learning Technologies , 5 (4), 318–335.

Verhaert, D., Nateghizad, M., & Zekeriya, E. (2018). An efficient privacy-preserving recommender system for e-healthcare system. In International conference on security and cryptography, (pp. 1–12) .

Wiesner, M., & Pfeifer, D. (2014). Health recommender systems: Concepts, requirements, technical basics and challenges. International Journal of Environmental Research and Public Health , 11 , 2580–2607.

Yamanishi, Y., Pauwels, E., & Kotera, M. (2012). Drug side-effect prediction based on the integration of chemical and biological spaces. Journal of chemical information and modeling, 52 .

Yang, L., Hsieh, C., Yang, H., Pollak, J., Dell, N., Belongie, S., Cole, C., & Estrin, D. (2017). Yum-me: A personalized nutrient-based meal recommender system. ACM Transactions on Information Systems 36 (1).

Zhang, Q., Zhang, G., Lu, J., & Wu, D. (2015). A framework of hybrid recommender system for personalized clinical prescription. In 2015 10Th international conference on intelligent systems and knowledge engineering (ISKE), (pp. 189–195) .

Zhang, W., Zou, H., Luo, L., Liu, Q., Wu, W., & Xiao, W. (2016). Predicting potential side effects of drugs by recommender methods and ensemble learning. Neurocomputing , 173 (P3), 979–987.

Zhang, Y., Chen, M., Huang, D., Wu, D., & Li, Y. (2016). idoctor: Personalized and professionalized medical recommendations based on hybrid matrix factorization. Future Generation Computer Systems , 66 , 30–35.

Download references

Open Access funding provided by Graz University of Technology.

Author information

Authors and affiliations.

Institute of Software Technology, Graz University of Technology, Graz, Austria

Thi Ngoc Trang Tran & Alexander Felfernig

Information Science and Media Studies Department, University of Bergen, Bergen, Norway

Christoph Trattner

Institute for Medical Informatics, Statistics and Documentation, Medical University of Graz, Graz, Austria

Andreas Holzinger

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Thi Ngoc Trang Tran .

Additional information

Publisher’s note.

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article's Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article's Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Tran, T.N.T., Felfernig, A., Trattner, C. et al. Recommender systems in the healthcare domain: state-of-the-art and research issues. J Intell Inf Syst 57 , 171–201 (2021). https://doi.org/10.1007/s10844-020-00633-6

Download citation

Received : 04 June 2020

Revised : 18 November 2020

Accepted : 19 November 2020

Published : 17 December 2020

Issue Date : August 2021

DOI : https://doi.org/10.1007/s10844-020-00633-6

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• Health recommender systems
• Food recommendation
• Drug recommendation
• Health status prediction
• Healthcare service recommendation
• Healthcare professionals recommendation
• Find a journal
• Publish with us
• Track your research

To read this content please select one of the options below:

Please note you do not have access to teaching notes, recommender systems: a systematic review of the state of the art literature and suggestions for future research.

ISSN : 0368-492X

Article publication date: 15 March 2018

Issue publication date: 2 May 2018

This paper aims to identify, evaluate and integrate the findings of all relevant and high-quality individual studies addressing one or more research questions about recommender systems and performing a comprehensive study of empirical research on recommender systems that have been divided into five main categories. To achieve this aim, the authors use systematic literature review (SLR) as a powerful method to collect and critically analyze the research papers. Also, the authors discuss the selected recommender systems and its main techniques, as well as their benefits and drawbacks in general.

Design/methodology/approach

In this paper, the SLR method is utilized with the aim of identifying, evaluating and integrating the findings of all relevant and high-quality individual studies addressing one or more research questions about recommender systems and performing a comprehensive study of empirical research on recommender systems that have been divided into five main categories. Also, the authors discussed recommender system and its techniques in general without a specific domain.

The major developments in categories of recommender systems are reviewed, and new challenges are outlined. Furthermore, insights on the identification of open issues and guidelines for future research are provided. Also, this paper presents the systematical analysis of the recommender system literature from 2005. The authors identified 536 papers, which were reduced to 51 primary studies through the paper selection process.

Originality/value

This survey will directly support academics and practical professionals in their understanding of developments in recommender systems and its techniques.

• Recommender system
• Collaborative filtering
• Demographic filtering

Alyari, F. and Jafari Navimipour, N. (2018), "Recommender systems: A systematic review of the state of the art literature and suggestions for future research", Kybernetes , Vol. 47 No. 5, pp. 985-1017. https://doi.org/10.1108/K-06-2017-0196

Emerald Publishing Limited

Copyright © 2018, Emerald Publishing Limited

Related articles

All feedback is valuable.

Please share your general feedback

Report an issue or find answers to frequently asked questions

Contact Customer Support

Click through the PLOS taxonomy to find articles in your field.

For more information about PLOS Subject Areas, click here .

Loading metrics

Open Access

Peer-reviewed

Research Article

A collaborative approach for research paper recommender system

Roles Conceptualization, Data curation, Formal analysis, Methodology, Project administration, Writing – original draft, Writing – review & editing

Affiliations Department of Computer Science, Faculty of Computer Science and Information Technology, Bayero University, Kano, Nigeria, Department of Information Systems, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

Roles Supervision, Validation, Visualization, Writing – review & editing

* E-mail: [email protected]

Affiliation Department of Information Systems, Faculty of Computer Science and Information Technology, University of Malaya, Kuala Lumpur, Malaysia

Roles Funding acquisition, Resources

Affiliation Sekolah Tinggi Pariwisata Ambarrukmo, Yogyakarta, Indonesia

Affiliation Faculty of Information Technology and Business, Universitas Teknologi Yogyakarta, Yogyakarta, Indonesia

Roles Formal analysis, Investigation, Project administration, Resources, Validation, Visualization, Writing – review & editing

Affiliations Faculty of Information Technology and Business, Universitas Teknologi Yogyakarta, Yogyakarta, Indonesia, AMCS Research Center, Yogyakarta, Indonesia

• Khalid Haruna, 
• Maizatul Akmar Ismail, 
• Damiasih Damiasih, 
• Joko Sutopo, 
• Tutut Herawan

• Published: October 5, 2017
• https://doi.org/10.1371/journal.pone.0184516
• Reader Comments

Research paper recommenders emerged over the last decade to ease finding publications relating to researchers’ area of interest. The challenge was not just to provide researchers with very rich publications at any time, any place and in any form but to also offer the right publication to the right researcher in the right way. Several approaches exist in handling paper recommender systems. However, these approaches assumed the availability of the whole contents of the recommending papers to be freely accessible, which is not always true due to factors such as copyright restrictions. This paper presents a collaborative approach for research paper recommender system. By leveraging the advantages of collaborative filtering approach, we utilize the publicly available contextual metadata to infer the hidden associations that exist between research papers in order to personalize recommendations. The novelty of our proposed approach is that it provides personalized recommendations regardless of the research field and regardless of the user’s expertise. Using a publicly available dataset, our proposed approach has recorded a significant improvement over other baseline methods in measuring both the overall performance and the ability to return relevant and useful publications at the top of the recommendation list.

Citation: Haruna K, Akmar Ismail M, Damiasih D, Sutopo J, Herawan T (2017) A collaborative approach for research paper recommender system. PLoS ONE 12(10): e0184516. https://doi.org/10.1371/journal.pone.0184516

Editor: Feng Xia, Dalian University of Technology, CHINA

Received: June 10, 2017; Accepted: August 27, 2017; Published: October 5, 2017

Copyright: © 2017 Haruna et al. This is an open access article distributed under the terms of the Creative Commons Attribution License , which permits unrestricted use, distribution, and reproduction in any medium, provided the original author and source are credited.

Data Availability: All study files are available from: https://figshare.com/articles/Supporting_Information_Dataset_docx/5368408 .

Funding: The authors received no specific funding for this work.

Competing interests: The authors have declared that no competing interests exist.

1. Introduction

The overabundance of information that is available over the internet makes information seeking a difficult task. Researchers find it difficult to access and keep track of the most relevant and promising research papers of their interest [ 1 ]. The easiest and the most common approach used in searching for related publications is to send a query message asking the web to provide you with specific information [ 2 ]. However, the results from this approach largely depend on how good the user is in fine-tuning the query message beside its inability to personalize the searching results.

Another classical approach used by most researchers is to follow the list of references from the documents they already possessed [ 3 ]. Even though this approach might be quite effective in some instances, it does not guarantee full coverage of recommending research papers and cannot trace papers published after the possessed paper. In addition, the list of references may not be publicly available and therefore hard for the researchers to access.

An alternative approach that has been proposed in the literature is the use of research paper recommender systems [ 4 , 5 ], to automatically suggest relevant papers to the researchers based on some initial information provided by the users that are more elaborate than a few keywords.

To provide more accurate and relevant recommendations, recommender systems incorporate the users’ contexts and the possible contextual information of the consumed contents [ 6 ]. Different researchers proposed the use of a different user provided information such as the use of a list of citations [ 7 ], the list of papers authored by an author [ 8 ], part of paper text [ 2 ], a single paper [ 9 ], and so on. In these approaches, a user profile is constructed from this initial information to represent the interests of the users and the system search for items or other profiles similar to the one provided to generate recommendations. The challenge was not just to provide a very rich recommendation to researchers at any time, any place and in any form but to also offer the right paper to the right researcher in the right way [ 10 – 12 ].

The major limitation of the existing approaches is their assumption of the availability of the whole content of the recommending papers to be freely accessible, which is not always true due to factors such as copyright restrictions. In an attempt to address this problem, Liu, et al . in [ 3 ] applied the concept of the collaborative approach to mine the hidden associations that exists between a target paper and its references to provide a unique and useful list of research papers as recommendations.

Motivated from [ 3 ], this paper presents a collaborative approach for research paper recommender system. In addition to mining the hidden associations between a target paper and its references, in this paper, we also put into cognizance the hidden associations between the target paper’s citations (see section 3). Similar to [ 3 ], our task is not to apply a direct relation between paper-citation relations because, in one way or the other, a researcher who is in possession of a research paper directly or indirectly has access to its limited references and also to its citations. Our aim is to identify the latent associations that exist between research papers based on the perspective of paper-citation relations. A candidate paper is qualified for consideration in [ 3 ] if it cited any of the target paper’s references. In our proposed approach, a candidate paper is qualified for consideration if and only if it cited any of the target paper’s references and there exist another paper which cited both the candidate and the target papers simultaneously. We then measure and weigh the extent of similarity between the target paper and the qualified candidate papers and recommend the top-N most similar papers based on the assumption that if there exist significant co-occurrence between the target paper and the qualified candidate papers, then there exist some extent of similarities between them. This strictness in qualifying a candidate paper helps in enhancing the overall performance of the approach and the ability to return relevant and useful recommendations at the top of the recommendation list.

The major contributions of our proposed approach are as follows;

• We utilized the advantages of publicly available contextual metadata to propose an independent research paper that does not require a priori user profile.
• Our approach provides personalized recommendations regardless of the research field and regardless of user expertise.

The outline of the rest of the paper is as follows. We first present some related works on recommending research papers. We then detailed our proposed approach. Next, we described our experiments, starting with the dataset and the baseline methods, followed by the evaluation procedures. We then discuss our findings and lastly conclude the paper with a brief concluding remark and future research directions.

2. Related work

Research paper recommenders that provide the best suggestions for all alternatives emerged over the last decade to help researchers on seemingly finding works of their interest over the Cyber Ocean of information. Collaborative filtering (CF) is one of the most successful techniques used in recommender systems [ 13 ]. It is a method which recommends items to target users based on what other similar users have previously preferred [ 14 – 16 ]. It has been used in various applications such as in recommending movies [ 17 ], audio CD [ 18 ], e-commerce [ 19 ], music [ 20 ], Usenet news [ 16 ], research papers [ 7 , 21 – 24 ] among others (see [ 25 ]). Some researchers [ 13 , 21 , 26 ], have criticized the use of this technique to recommend scholarly papers. Precisely the authors in [ 21 , 26 ], claimed that collaborative filtering is only effective in a domain where the number of users seeking recommendation is higher than the number of items to be recommended, such domains include movies [ 27 ], music [ 28 ], news [ 29 ] etc. While the argument in [ 13 ], is that researchers are not willing to spend their valuable time to provide explicit ratings to their consumed research papers, and therefore, leading to insufficient ratings by the researchers to the research papers. Furthermore, for a user to receive useful recommendations, a tangible number of ratings is required.

Nevertheless, despite these aforementioned problems, a significant amount of papers can be traced, which suggest relevant papers to researchers based on collaborative filtering by mining latent associations between scholarly papers. These associations are either directly obtained by taking into consideration paper citations as rating scores [ 7 ], or by monitoring the researchers’ actions implicitly [ 30 , 31 ]. Applying citation analysis such as bibliographical coupling [ 32 ] and co-citation analysis [ 33 ] has also been used to identify similar papers to a target paper [ 34 ].

The relationships among research papers have been categorized into direct and indirect relations in a survey conducted by [ 35 ]. In the paper, three approaches were identified for detecting the relationships between papers based on the perspective of paper sources. Namely, citation context, citation analysis, and content-based. The authors claimed that content-based approach becomes less appropriate in detecting relationships across research papers, due to its inability to accommodate some specific characteristics that exist in the research papers like author and citations. Therefore, it becomes suitable only for identifying similarity relations across regular documents. On the other hand, the use of citation analysis can generate more relations between research papers but cannot generate relations from semantic text. This weakness is addressed by using citation context based approach, which depicts more emphasis on determining some important features in the text classification process to increase classification performance.

A context-based collaborative framework (CCF) that uses only easily obtained citations relations as source data was proposed in [ 3 ]. The framework employs an association-mining technique to obtain a paper representation of the paper citation context. A pairwise comparison was then performed to compute the extent of similarities between papers. The use of collaborative filtering has also been explored in [ 7 ], by using citation-web between scholarly papers to create a rating matrix. The aim was to use the paper-citation relation to recommend some additional references to the input paper. In doing that, the authors investigated the use of six different algorithms for selecting citations. Using offline evaluation, they discovered large disparity in the returned accuracy by each of the six algorithms.

The authors in [ 6 ], hypothesized the author’s previous publications to constitute a clear signal of the latent interests of a researcher. The key part of their model was to enhance the user profile with the information coming directly from the references to the researcher’s previous works as well as the papers that cited them. However, the approach increases the well-known sparsity problem. To alleviate this problem, they extend their work in [ 8 ], to mine potential citations papers using imputed similarities through the use of collaborative filtering. They also refined the use of citing papers in characterizing a target candidate paper using fragments in the citation and potential citation papers. Whilst the approach works well for researchers with a single discipline, it generates poor results for the multidisciplinary researchers. To overcome this problem, an adaptive neighbor selection approach was proposed in [ 2 ], to overcome imputation-based collaborative filtering problem. Whereas authors in [ 2 , 6 , 8 ], recommend papers relevant to the researcher’s interest, they also addressed the serendipitous scholarly paper recommendation in [ 36 ].

On another development, the increasing number of research communities and social networking sites such as LiveJournal and MySpace have brought new opportunities for research paper recommendation systems. Researches show that users in online social networks tend to form knit groups [ 37 ], with strongly large connected components [ 38 ].

Several kinds of research have considered the social group formation and community membership in social networks and their use in recommender systems [ 39 – 46 ]. These researchers utilized the influence of social properties to suggest relevant information to individual or group of users based on social ties, which can either be strong or weak depending on the tie strength that represents the closeness and interaction frequency between the information source and recipient [ 47 , 48 ].

Recommendations from strong ties are believed to be more persuasive than those from weak ties [ 49 – 51 ]. This is because information transferred by strong ties is likely to be perceived as more relevant and reliable. To be specific, the authors of [ 45 , 52 ] proposed a novel algorithm called socially aware recommendation of scholarly papers (SARSP) that utilizes the aspect of social learning and networking for conference participants through the construction of relations in folksonomies and social ties. The algorithm recommends research papers issued by an active participant to other conference participants based on the computation of their social ties. This approach has been extended in [ 53 ], to include personality behavior in addition to social relations among smart conference attendees. A more detail survey on scholarly data is presented in [ 54 ] for more exploration.

The major challenge with the previous researches is that all the contextual information from the recommended, referenced and cited papers must be fully accessible to the recommenders, which are not always freely available due to factors such as copyright restrictions. Another major problem with the existing research paper recommender systems is their dependency on a priori user profile, which makes the system to work well only when it already has a number of registered users, a major hurdle for the construction of new recommender system. Furthermore, the recommendation coverage of most of the current paper recommenders are limited to a certain field of research, this is because recommending papers are stored prior and therefore the system cannot effectively scan the entire databases to find connections between papers. Moreover, most of the existing research paper frameworks are designed to work only on a single discipline, and therefore cannot be used to address the problems of multidisciplinary scholars. While the use of keyword-based query information retrieval technique through search engines is able to scan all document for relevant text, it also provides 100s of irrelevant documents, besides its inability to provide personalize results to the individual researchers.

Different from the existing works, in this paper, we propose a new approach based on collaborative filtering that utilizes only publicly available contextual metadata to personalize recommendations based on the hidden associations that exist between research papers. Our proposed approach does not only provide personalized recommendations regardless of the research field and regardless of user expertise but also handles multi-disciplinary problems.

3. Proposed collaborative research paper recommendation approach

Even though some researchers [ 6 , 13 , 21 , 26 ], claimed content based to be the most suitable approach when dealing with scholarly domain, other researchers [ 35 ] argued on its suitability because only become suitable in identifying similarity relations across regular documents but lacks some important features to effectively detect relationships across research papers.

In this paper, we are motivated to leverage the advantages of collaborative approach as it has proved to be effective in the domains of movies [ 27 ], music [ 28 ], news [ 29 ], e-commerce [ 19 ], etc. The unsuitability of the collaborative approach to research paper recommenders was referred to the lack of ratings to research papers by the researchers [ 13 ]. In bringing a solution to this problem, we mined rating score between researchers and research papers based on paper-citation relations. We use C ij to denote citation score between paper i and a cited-paper j from a paper-citation matrix C . If paper i cited a paper j , C ij = 1 otherwise C ij = 0.

We initiate our approach by first transforming all the recommending papers (in our dataset) into a paper-citation relations matrix in which, the rows and the columns respectively represent the recommending papers and their citations. Our approach aimed to deal with scenarios in which: (a) A researcher who finds an interesting paper after some initial searches, wants to get more other related papers similar to it. (b) A student received a paper by his supervisor to start a research in the topic area covered by it. (c) A reviewer wants to explore more based on a received paper that addresses a subject matter which he is not a specialist in. (d) A researcher who wants to explore more from his previous publication(s). In all these cases, we consider a situation where the references and citations of the possessed paper that indicate the user’s preferences are publicly available (which is usually the case in almost all the major academic databases).

Algorithm 1. Algorithm representing proposed approach.

Algorithm: Collaborative Research Paper Recommendation

Input: Target Paper

Output: Top-N Recommendation

Given a target paper p i as a query,

• For each of the references Rf j , extract all other papers p ci that also cited Rf j other than the target paper p i .
• For each of the citations Cf j , extract all other papers p ri that Cf j referenced other than the target paper p i .
• Qualify all the candidate papers p c from p ci that has been referenced by at least any of the p ri

• Recommend the top-N most similar papers to the user.

We accept the user’s query in order to identify the target-paper. Once the target paper is identified, we apply algorithm 1. The algorithm retrieves all the target paper’s references and citations. For each of the references, it extracts all other papers from the web (google scholar to be precise) that also cited any of those target paper’s references. In addition, for each of the target paper’s citations, it extracts all other papers from the web that referenced any of those target paper’s citations (in other words, all the references to the target paper’s citations) and we refer to these extracted papers as the target papers nearest neighbors. For each of the neighboring papers, we qualify candidate papers that are co-cited with the target paper and which has been referenced by at least any of the target papers references. We then measure the degree of similitude between these qualified candidate papers and the target paper by measuring their collaborative similarity using Jaccard similarity measure given by Eq (1) . We then recommend the top-N most comparable papers to the researcher.

Z 11 Represents the total number of attributes where X and Y both having a value of 1.

Z 01 Represents the total number of attributes where the attribute of X is 0 and the attribute of Y is 1.

Z 10 Represents the total number of attributes where the attribute of X is 1 and the attribute of Y is 0.

To illustrate our approach further, Fig 1 represents a target-paper ( p i ) with references ( Rf 1 to Rf N ) and citations ( Cit . 1 to Cit . N ). Each of the references of the target paper has other citations from any of Rec . 1 to Rec . N and/or Cit . 1 to Cit . N other than the target-paper ( p i ). Also, each of the citations to the target paper has other references from any of Rec . 1 to Rec . N and/or Rf 1 to Rf N other than the target-paper ( p i ). Our approach qualifies recommending papers ( Rec . 1 to Rec . N ) that are co-cited with the target paper and which has been referenced by at least any of the target papers references.

• PPT PowerPoint slide
• PNG larger image
• TIFF original image

https://doi.org/10.1371/journal.pone.0184516.g001

For example, from Fig 1 , Rec . 1 and Rec . 2 are co-cited with the target paper by Cit . 1 . However, Ref . 2 does not have any connection to any of the target paper’s references and therefore disqualified by step 3 of our proposed algorithm. On the other hand, Rec . 1 does not only being co-cited with the target paper by Cit . 1 but also referenced one of the target papers references Ref . 1 . As can be observed from Fig 1 , only Rec . 1 and Rec . 3 are qualified candidate papers.

In the following section, we present the experiments setup.

4. Experiments setup

4.1 dataset.

We utilize the publicly available dataset presented in [ 2 ]. The dataset contained the publication list of 50 researchers whose research interests are from different fields of computer science that range from information retrieval, software engineering, user interface, security, graphics, databases, operating systems, embedded systems and programming languages. We retrieved every one of their references and citations and extracted from google scholar, every other paper that cited any of the references as well as all the references of each of the target paper’s citations. Some statistics of the utilized dataset is presented in Table 1 .

https://doi.org/10.1371/journal.pone.0184516.t001

4.2 Baseline methods

In assessing the effectiveness of our proposed framework, we compare the recommendation results with two baselines presented in [ 7 ] and [ 3 ]. The pattern introduced in [ 7 ] views citation relation matrix as a rating score and generates the recommendation based on common citations between the target paper and its neighboring papers. Given a target paper, the algorithm counts the number of times other citations were co-cited with it. The algorithm then recommends citations with the highest total co-citations summed over all recommending papers. The assumption was that, the more the co-citation in like manner between papers the higher their similarity. While [ 3 ], mined the hidden relationship between a target paper and all of its references. The task was to quantify the degree of closeness between the target paper and the other papers that also cited any of the target paper’s references. The rationale behind the approach was that, if two papers are significantly co-occurring with the same citing paper(s), then they should be similar to some extent.

4.3 Evaluation metrics

In order to evaluate the quality of our approach, for each of the target papers, we performed 5-fold cross validation to its references and citations by selecting 20% as a test set. We then assess the general performance using the three most commonly used evaluation metrics in retrieval systems: precision, recall and F1 measures. Precision given by Eq (2) , measures the capability of the system to reclaim as much relevant research papers as possible in response to the target paper request.

On the other hand, recall given by Eq (3) , measures the capability of the system to reclaim as few irrelevant research papers as possible in response to the target paper request.

Moreover, F1 measure given by Eq (4) is the harmonic mean between the precision and recall.

As users often scan only documents presented at the top ranked of the recommendation list, we feel imperative to also measure the system’s ability to provide useful recommendations at the top of the recommendation list using the two most widely used ranked information retrieval evaluation measures: Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR).

5. Results and discussions

To be specific, the results of each evaluation metric in this section represent the overall averages over all the 50 researchers of our dataset. We start the comparison by assessing the general performance of our proposed approach in returning relevant research papers with the baseline methods based on the three most commonly used information retrieval evaluation metrics. Figs 2 – 4 , demonstrate the comparisons based on precision, recall and F1 evaluation measures respectively. As can be seen from Fig 2 , the precision results of our proposed approach has significantly outperformed the baseline methods (Context-Based Collaborative Filtering (CCF) proposed by [ 3 ] and Co-citation method proposed by [ 7 ]) in returning relevant research papers for all N recommendations values. This is because our approach is able to critically remove recommending papers that are less related to the target paper.

https://doi.org/10.1371/journal.pone.0184516.g002

https://doi.org/10.1371/journal.pone.0184516.g003

https://doi.org/10.1371/journal.pone.0184516.g004

Fig 3 depicts the comparison based on recall. As can be seen from the figure, the performance difference between our proposed approach and CCF is very much insignificant. In fact, the CCF method is even slightly better than our proposed approach when N = 5 and when N = 20. However, our proposed approach began to show the significant difference as the number of N increases, specifically when N is above 20. The low performance based on recall of our proposed approach is as a result of strict rules in qualifying a candidate paper. Thus, our approach is only after the most significant related recommending papers to the target paper and therefore leaving a lot of other less related papers unrecalled. Furthermore, Fig 4 depicts the harmonic mean between the precision and recall ( F 1 measure), and from the figure, the performance difference between our proposed approach and CCF is also insignificant for values of N less than or equals to 20. However, our approach began to show significant improvement over CCF when N is greater than 20. In all the three measures, the Co-citation method performs very low compared to our proposed approach. This is because the Co-citation method does not infer the hidden associations between paper-citation relations rather applies direct relations between a target paper and its neighboring papers.

Conclusively, the general performance of our proposed approach has outstandingly outperformed the baseline methods based on precision for all values of N . On the other hand, our proposed approach performs worse than CCF in a recommendation list of 5 based on recall and F1 performance measures. However, the major reason behind the low performance of our proposed approach based on recall is the strict rules in qualifying a candidate paper.

Our proposed approach is designed to favor precision which has more influence on user satisfaction than recall. This is because precision is the key element in the process of implementing a search solution [ 55 ]. Poor precision damages the reputation of a search system and discourages its use. High precision generally impresses search users [ 55 ]. That is why our proposed approach is only after the most significant related recommending papers to the target paper (the result of this can easily be seen from Fig 2 ), and therefore leaving a lot of other less related papers unrecalled. This is because recall is particularly important in applications where the user cannot afford to miss information such as issues related to security or compliance applications. The recall has less influence on user satisfaction than precision. Many searchers, especially on the Web, are satisfied by precise results, even where recall is low [ 56 ]. Notwithstanding, our proposed approach starts to show large disparities with the baseline methods when the number of N is above 5 for both recall and F 1 measures. Therefore, a very large N value is extremely important in order to recall as much qualitative and useful recommendations as possible.

Due to the fact that users usually scan only the top of the recommendation list, we also make the comparison based on how our approach is able to return relevant research papers at the top of the recommendation list. Figs 5 and 6 depict our comparisons based on Mean Average Precision (MAP) and Mean Reciprocal Rank (MRR) respectively. As can be seen from Fig 5 , that our proposed method has significantly outperformed the baseline methods based on mean average precision (MAP) in all cases in returning the relevant recommendations at the top of the recommendation list. Moreover, the comparison based on mean reciprocal rank (MRR) depicted by Fig 6 has also revealed that our proposed approach has outstandingly outperformed the baseline methods in all scenarios. It can easily be seen from the figure that our approach is able to return a relevant research paper at either rank 1 or rank 2 of the recommendation list for all queries.

https://doi.org/10.1371/journal.pone.0184516.g005

https://doi.org/10.1371/journal.pone.0184516.g006

As we have pointed out earlier, all these improvements are largely due to the strictness in qualifying a candidate paper which removed less relevant papers to the target paper. This, therefore, increases the system’s ability to return relevant and useful recommendations at the top of the recommendation list.

6. Conclusion and future work

In this paper, we utilized the publicly available contextual metadata to leverage the advantages of collaborative filtering approach in recommending a set of related papers to a researcher based on paper-citation relations. The approach mined the hidden associations between a research paper and its references and citations using paper-citation relations. The rationale behind the approach is that, if two papers are significantly co-occurring with the same citing paper(s), then they should be similar to some extent.

As demonstrated using a publicly available dataset, our proposed method outperforms the baseline methods in measuring both the overall performance and the ability to return relevant and useful research papers at the top of the recommendation list. Based on the three most commonly used information retrieval system metrics, our proposed approach have significantly improved the baseline methods based on precision, recall and F1 measures. Our proposed approach has also recorded significant improvements over the baseline methods in providing relevant and useful recommendations at the top of the recommendation list based on mean average precision (MAP) and mean reciprocal rank (MRR).

In addition to considering the collaborative relations among research papers, our next line of research is to also put into cognizance the public contextual contents, such as titles and abstracts of the recommending papers for better performances.

Supporting information

S1 dataset. the detail of the complete dataset can be accessed via https://figshare.com/articles/supporting_information_dataset_docx/5368408 ..

https://doi.org/10.1371/journal.pone.0184516.s001

• View Article
• Google Scholar
• 4. B. Gipp, J. Beel, and C. Hentschel, "Scienstein: A research paper recommender system," in Proceedings of the international conference on emerging trends in computing (icetic’09), 2009, pp. 309–315.
• 5. J. Beel, S. Langer, M. Genzmehr, and A. Nürnberger, "Introducing Docear's research paper recommender system," in Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries, 2013, pp. 459–460.
• 6. K. Sugiyama and M.-Y. Kan, "Scholarly paper recommendation via user's recent research interests," in Proceedings of the 10th annual joint conference on Digital libraries, 2010, pp. 29–38.
• 7. S. M. McNee, I. Albert, D. Cosley, P. Gopalkrishnan, S. K. Lam, A. M. Rashid, et al., "On the recommending of citations for research papers," in Proceedings of the 2002 ACM conference on Computer supported cooperative work, 2002, pp. 116–125.
• 8. K. Sugiyama and M.-Y. Kan, "Exploiting potential citation papers in scholarly paper recommendation," in Proceedings of the 13th ACM/IEEE-CS joint conference on Digital libraries, 2013, pp. 153–162.
• 9. C. Nascimento, A. H. Laender, A. S. da Silva, and M. A. Gonçalves, "A source independent framework for research paper recommendation," in Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, 2011, pp. 297–306.
• 10. C. Prahalad, "Beyond CRM: CK Prahalad predicts customer context is the next big thing," American Management Association MwWorld, 2004.
• 11. K.-L. Skillen, L. Chen, C. D. Nugent, M. P. Donnelly, W. Burns, and I. Solheim, "Ontological user profile modeling for context-aware application personalization," in Ubiquitous Computing and Ambient Intelligence, ed: Springer, 2012, pp. 261–268.
• 12. Z. Yu, Y. Nakamura, S. Jang, S. Kajita, and K. Mase, "Ontology-based semantic recommendation for context-aware e-learning," in Ubiquitous Intelligence and Computing, ed: Springer, 2007, pp. 898–907.
• 13. R. Torres, S. M. McNee, M. Abel, J. A. Konstan, and J. Riedl, "Enhancing digital libraries with TechLens," in Digital Libraries, 2004. Proceedings of the 2004 Joint ACM/IEEE Conference on, 2004, pp. 228–236.
• 16. P. Resnick, N. Iacovou, M. Suchak, P. Bergstrom, and J. Riedl, "GroupLens: an open architecture for collaborative filtering of netnews," in Proceedings of the 1994 ACM conference on Computer supported cooperative work, 1994, pp. 175–186.
• 18. U. Shardanand and P. Maes, "Social information filtering: algorithms for automating “word of mouth”," in Proceedings of the SIGCHI conference on Human factors in computing systems, 1995, pp. 210–217.
• 20. N. Hariri, B. Mobasher, and R. Burke, "Context-aware music recommendation based on latenttopic sequential patterns," in Proceedings of the sixth ACM conference on Recommender systems, 2012, pp. 131–138.
• 21. N. Agarwal, E. Haque, H. Liu, and L. Parsons, "Research paper recommender systems: A subspace clustering approach," in International Conference on Web-Age Information Management, 2005, pp. 475–491.
• 22. K. Chandrasekaran, S. Gauch, P. Lakkaraju, and H. P. Luong, "Concept-based document recommendations for citeseer authors," in International Conference on Adaptive Hypermedia and Adaptive Web-Based Systems, 2008, pp. 83–92.
• 23. M. Gori and A. Pucci, "Research paper recommender systems: A random-walk based approach," in Web Intelligence, 2006. WI 2006. IEEE/WIC/ACM International Conference on, 2006, pp. 778–781.
• 24. A. Kodakateri Pudhiyaveetil, S. Gauch, H. Luong, and J. Eno, "Conceptual recommender system for CiteSeerX," in Proceedings of the third ACM conference on Recommender systems, 2009, pp. 241–244.
• 25. K. Haruna, M. A. Ismail, and S. M. Shuhidan, "Domain of Application in Context-Aware Recommender Systems: A Review," Knowledge Management International Conference (KMICe) 2016, 29–30 August 2016, Chiang Mai, Thailand 2016.
• 27. A. Azaria, A. Hassidim, S. Kraus, A. Eshkol, O. Weintraub, and I. Netanely, "Movie recommender system for profit maximization," in Proceedings of the 7th ACM conference on Recommender systems, 2013, pp. 121–128.
• 28. Schedl M., Knees P., McFee B., Bogdanov D., and Kaminskas M., "Music recommender systems," in Recommender Systems Handbook , ed: Springer, 2015, pp. 453–492.
• 30. D. M. Pennock, E. Horvitz, S. Lawrence, and C. L. Giles, "Collaborative filtering by personality diagnosis: A hybrid memory-and model-based approach," in Proceedings of the Sixteenth conference on Uncertainty in artificial intelligence, 2000, pp. 473–480.
• 32. Fano R., "Information theory and the retrieval of recorded information," Documentation in Action , Shera JH Kent A. Perry JW (Edts), New York: Reinhold Publ. Co, pp. 238–244, 1956.
• 33. I. V. Marshakova, "System of document connections based on references," Nauchno-Tekhnicheskaya Informatsiya Seriya 2-Informatsionnye Protsessy I Sistemy, pp. 3–8, 1973.
• 34. C. L. Giles, K. D. Bollacker, and S. Lawrence, "CiteSeer: An automatic citation indexing system," in Proceedings of the third ACM conference on Digital libraries, 1998, pp. 89–98.
• 35. Y. Sibaroni, D. H. Widyantoro, and M. L. Khodra, "Survey on research paper's relations," in Information Technology Systems and Innovation (ICITSI), 2015 International Conference on, 2015, pp. 1–6.
• 36. K. Sugiyama and M.-Y. Kan, "Serendipitous recommendation for scholarly papers considering relations among researchers," in Proceedings of the 11th annual international ACM/IEEE joint conference on Digital libraries, 2011, pp. 307–310.
• 38. Kumar R., Novak J., and Tomkins A., "Structure and evolution of online social networks," in Link mining : models , algorithms , and applications , ed: Springer, 2010, pp. 337–357.
• 39. F. C. T. Chua, H. W. Lauw, and E.-P. Lim, "Predicting item adoption using social correlation," in Proceedings of the 2011 SIAM International Conference on Data Mining, 2011, pp. 367–378.
• 40. I. Konstas, V. Stathopoulos, and J. M. Jose, "On social networks and collaborative recommendation," in Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 2009, pp. 195–202.
• 41. H. Ma, I. King, and M. R. Lyu, "Learning to recommend with social trust ensemble," in Proceedings of the 32nd international ACM SIGIR conference on Research and development in information retrieval, 2009, pp. 203–210.
• 42. H. Ma, D. Zhou, C. Liu, M. R. Lyu, and I. King, "Recommender systems with social regularization," in Proceedings of the fourth ACM international conference on Web search and data mining, 2011, pp. 287–296.
• 43. L. Backstrom, D. Huttenlocher, J. Kleinberg, and X. Lan, "Group formation in large social networks: membership, growth, and evolution," in Proceedings of the 12th ACM SIGKDD international conference on Knowledge discovery and data mining, 2006, pp. 44–54.
• 52. F. Xia, N. Y. Asabere, H. Liu, N. Deonauth, and F. Li, "Folksonomy based socially-aware recommendation of scholarly papers for conference participants," in Proceedings of the 23rd International Conference on World Wide Web, 2014, pp. 781–786.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

Preview improvements coming to the PMC website in October 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• J Med Internet Res
• v.23(6); 2021 Jun

Health Recommender Systems: Systematic Review

Robin de croon.

1 Department of Computer Science, KU Leuven, Leuven, Belgium

Leen Van Houdt

Nyi nyi htun, gregor Štiglic.

2 Faculty of Health Sciences, University of Maribor, Maribor, Slovenia

Vero Vanden Abeele

Katrien verbert, associated data.

Coded data set of all included papers.

Overview of recommended items by 73 studies.

Overview of evaluation approaches.

Health recommender systems (HRSs) offer the potential to motivate and engage users to change their behavior by sharing better choices and actionable knowledge based on observed user behavior.

We aim to review HRSs targeting nonmedical professionals (laypersons) to better understand the current state of the art and identify both the main trends and the gaps with respect to current implementations.

We conducted a systematic literature review according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines and synthesized the results. A total of 73 published studies that reported both an implementation and evaluation of an HRS targeted to laypersons were included and analyzed in this review.

Recommended items were classified into four major categories: lifestyle, nutrition, general health care information, and specific health conditions. The majority of HRSs use hybrid recommendation algorithms. Evaluations of HRSs vary greatly; half of the studies only evaluated the algorithm with various metrics, whereas others performed full-scale randomized controlled trials or conducted in-the-wild studies to evaluate the impact of HRSs, thereby showing that the field is slowly maturing. On the basis of our review, we derived five reporting guidelines that can serve as a reference frame for future HRS studies. HRS studies should clarify who the target user is and to whom the recommendations apply, what is recommended and how the recommendations are presented to the user, where the data set can be found, what algorithms were used to calculate the recommendations, and what evaluation protocol was used.

Conclusions

There is significant opportunity for an HRS to inform and guide health actions. Through this review, we promote the discussion of ways to augment HRS research by recommending a reference frame with five design guidelines.

Introduction

Research goals.

Current health challenges are often related to our modern way of living. High blood pressure, high glucose levels, and physical inactivity are all linked to a modern lifestyle characterized by sedentary living, chronic stress, or a high intake of energy-dense foods and recreational drugs [ 1 ]. Moreover, people usually make poor decisions related to their health for distinct reasons, for example, busy lifestyles, abundant options, and a lack of knowledge [ 2 ]. Practically, all modern lifestyle health risks are directly affected by people’s health decisions [ 3 ], such as an unhealthy diet or physical inactivity, which can contribute up to three-fourth of all health care costs in the United States [ 4 ]. Most risks can be minimized, prevented, or sometimes even reversed with small lifestyle changes. Eating healthily, increasing daily activities, and knowing where to find validated health information could lead to improved health status [ 5 ].

Health recommender systems (HRSs) offer the potential to motivate and engage users to change their behavior [ 6 ] and provide people with better choices and actionable knowledge based on observed behavior [ 7 - 9 ]. The overall objective of the HRS is to empower people to monitor and improve their health through technology-assisted, personalized recommendations. As one approach of modern health care is to involve patients in the cocreation of their own health, rather than just leaving it in the hands of medical experts [ 10 ], we limit the scope of this paper to HRSs that focus on laypersons, for example, nonhealth care professionals. These HRSs are different from clinical decision support systems that provide recommendations for health care professionals. However, laypersons also need to understand the rationale of recommendations, as echoed by many researchers and practitioners [ 11 ]. This paper also studies the role of a graphical user interface. To guide this study, we define our research questions (RQs) as follows:

RQ1: What are the main applications of the recent HRS, and what do these HRSs recommend?

RQ2: Which recommender techniques are being used across different HRSs?

RQ3: How are the HRSs evaluated, and are end users involved in their evaluation?

RQ4: Is a graphical user interface designed, and how is it used to communicate the recommended items to the user?

Recommender Systems and Techniques

Recommender techniques are traditionally divided into different categories [ 12 , 13 ] and are discussed in several state-of-the-art surveys [ 14 ]. Collaborative filtering is the most used and mature technique that compares the actions of multiple users to generate personalized suggestions. An example of this technique can typically be found on e-commerce sites, such as “Customers who bought this item also bought...” Content-based filtering is another technique that recommends items that are similar to other items preferred by the specific user. They rely on the characteristics of the objects themselves and are likely to be highly relevant to a user’s interests. This makes content-based filtering especially valuable for application domains with large libraries of a single type of content, such as MedlinePlus’ curated consumer health information [ 15 ]. Knowledge-based filtering is another technique that incorporates knowledge by logic inferences. This type of filtering uses explicit knowledge about an item, user preferences, and other recommendation criteria. However, knowledge acquisition can also be dynamic and relies on user feedback. For example, a camera recommender system might inquire users about their preferences, fixed or changeable lenses, and budget and then suggest a relevant camera. Hybrid recommender systems combine multiple filtering techniques to increase the accuracy of recommendation systems. For example, the companies you may want to follow feature in LinkedIn uses both content and collaborative filtering information [ 16 ]: collaborative filtering information is included to determine whether a company is similar to the ones a user already followed, whereas content information ensures whether the industry or location matches the interests of the user. Finally, recommender techniques are often augmented with additional methods to incorporate contextual information in the recommendation process [ 17 ], including recommendations via contextual prefiltering, contextual postfiltering, and contextual modeling [ 18 ].

HRSs for Laypersons

Ricci et al [ 12 ] define recommender systems as:

Recommender Systems (RSs) are software tools and techniques providing suggestions for items to be of use to a user [ 13 , 19 , 20 ]. The suggestions relate to various decision-making processes, such as what items to buy, what music to listen to, or what online news to read.

In this paper, we analyze how recommender systems have been used in health applications, with a focus on laypersons. Wiesner and Pfeifer [ 21 ] broadly define an HRS as:

a specialization of an RS [recommender system] as defined by Ricci et al [ 12 ]. In the context of an HRS, a recommendable item of interest is a piece of nonconfidential, scientifically proven or at least generally accepted medical information.

Researchers have sought to consolidate the vast body of literature on HRSs by publishing several surveys, literature reviews, and state-of-the-art overviews. Table 1 provides an overview of existing summative studies on HRSs that identify existing research and shows the number of studies included, the method used to analyze the studies, the scope of the paper, and their contribution.

An overview of the existing health recommender system overview papers.

Review	Papers, n	Method	Scope	Contribution
Sezgin and Özkan (2013) [ ]	8	Systematic review	Provides an overview of the literature in 2013	Identifying challenges (eg, cyber-attacks, difficult integration, and data mining can cause ethical issues) and opportunities (eg, integration with personal health data, gathering user preferences, and increased consistency)
Calero Valdez et al (2016) [ ]	17	Survey	Stresses the importance of the interface and HCI of an HRS	Providing a framework to incorporate domain understanding, evaluation, and specific methodology into the development process
Kamran and Javed (2015) [ ]	7	Systematic review	Provides an overview of existing recommender systems with more focus on health care systems	Proposing a hybrid HRS
Afolabi et al (2015) [ ]	22	Systematic review	Research empirical results and practical implementations of HRSs	Presenting a novel proposal for the integration of a recommender system into smart home care
Ferretto et al (2017) [ ]	8	Systematic review	Identifies and analyzes HRSs available in mobile apps	Identifying HRSs that do not have many mobile health care apps
Hors-Fraile et al 2018 [ ]	19	Systematic review	Identifies, categorizes, and analyzes existing knowledge on the use of HRSs for patient interventions	Proposing a multidisciplinary taxonomy, including integration with electronic health records and the incorporation of health promotion theoretical factors and behavior change theories
Schäfer et al (2017) [ ]	24	Survey	Discusses HRSs to find personalized, complex medical interventions or support users with preventive health care measures	Identifying challenges subdivided into patient and user challenges, recommender challenges, and evaluation challenges
Sadasivam et al (2016) [ ]	15	Systematic review	Research limitations of current CTHC systems	Identifying challenges of incorporating recommender systems into CTHC. Proposing a future research agenda for CTHC systems
Wiesner and Pfeifer (2014) [ ]	Not reported	Survey	Introduces HRSs and explains their usefulness to personal health record systems	Outlining an evaluation approach and discussing challenges and open issues
Cappella et al (2015) [ ]	Not reported	Survey	Explores approaches to the development of a for archives of public health messages	Reflecting on theory development and applications

a HCI: human-computer interaction.

b HRS: health recommender system.

c CTHC: computer-tailored health communication.

As can be seen in Table 1 , the scope of the existing literature varies greatly. For example, Ferretto et al [ 26 ] focused solely on HRSs in mobile apps. A total of 3 review studies focused specifically on the patient side of the HRS: (1) Calero Valdez et al [ 23 ] analyzed the existing literature from a human-computer interaction perspective and stressed the importance of a good HRS graphical user interface; (2) Schäfer et al [ 28 ] focused on tailoring recommendations to end users based on health context, history, and goals; and (3) Hors-Fraile et al [ 27 ] focused on the individual user by analyzing how HRSs can target behavior change strategies. The most extensive study was conducted by Sadasivam et al [ 29 ]. In their study, most HRSs used knowledge-based recommender techniques, which might limit individual relevance and the ability to adapt in real time. However, they also reported that the HRS has the opportunity to use a near-infinite number of variables, which enables tailoring beyond designer-written rules based on data. The most important challenges reported were the cold start [ 31 ] where limited data are available at the start of the intervention, limited sample size, adherence, and potential unintended consequences [ 29 ]. Finally, we observed that these existing summative studies were often restrictive in their final set of papers.

Our contributions to the community are four-fold. First, we analyze a broader set of research studies to gain insights into the current state of the art. We do not limit the included studies to specific devices or patients in a clinical setting but focus on laypersons in general. Second, through a comprehensive analysis, we aim to identify the applications of recent HRS apps and gain insights into actionable knowledge that HRSs can provide to users (RQ1), to identify which recommender techniques have been used successfully in the domain (RQ2), how HRSs have been evaluated (RQ3), and the role of the user interface in communicating recommendations to users (RQ4). Third, based on our extensive literature review, we derive a reference frame with five reporting guidelines for future layperson HRS research. Finally, we collected and coded a unique data set of 73 papers, which is publicly available in Multimedia Appendix 1 [ 7 - 9 , 15 , 32 - 100 ] for other researchers.

Search Strategy

This study was conducted according to the key steps required for systematic reviews according to PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines [ 101 ]. A literature search was conducted using the ACM Digital Library (n=2023), IEEExplore (n=277), and PubMed (n=93) databases. As mentioned earlier, in this systematic review we focused solely on HRSs with a focus on laypersons. However, many types of systems, algorithms, and devices can be considered as a HRS. For example, push notifications in a mobile health app or health tips prompted by web services can also be considered as health-related recommendations. To outline the scope, we limited the search terms to include a recommender or recommendation, as reported by the authors. The search keywords were as follows, using an inclusive OR: ( recommender OR recommendation systems OR recommendation system ) AND (health OR healthcare OR patient OR patients ).

In addition, a backward search was performed by examining the bibliographies of the survey and review papers discussed in the Introduction section and the reference list of included studies to identify any additional studies. A forward search was performed to search for articles that cited the work summarized in Table 1 .

Study Inclusion and Exclusion Criteria

As existing work did not include many studies ( Table 1 ) and focused on a specific medical domain or device, such as mobile phones, this literature review used nonrestrictive inclusion criteria. Studies that met all the following criteria were included in the review: described an HRS whose primary focus was to improve health (eg, food recommenders solely based on user preferences [ 102 ] were not included); targeted laypersons (eg, activity recommendations targeted on a proxy user such as a coach [ 103 ] were not included); implemented the HRS (eg, papers describing an HRS concept are not included); reported an evaluation, either web-based or offline evaluation; peer-reviewed and published papers; published in English.

Papers were excluded when one of the following was true: the recommendations of HRSs were unclear; the full text was unavailable; or a newer version was already included.

Finally, when multiple papers described the same HRS, only the latest, relevant full paper was included.

Classification

To address our RQs, all included studies were coded for five distinct coding categories.

Study Details

To contextualize new insights, the publication year and publication venue were analyzed.

Recommended Items

HRSs are used across different health domains. To provide details on what is recommended, all papers were coded according to their respective health domains. To not limit the scope of potential items, no predefined coding table was used. Instead, all papers were initially coded by the first author. These resulting recommendations were then clustered together in collaboration with the coauthors into four categories, as shown in Multimedia Appendix 2 .

Recommender Techniques

This category encodes the recommender techniques that were used: collaborative filtering [ 104 ], content-based filtering [ 105 ], knowledge-based filtering [ 106 ], and their hybridizations [ 107 ]. Some studies did not specify any algorithmic details or compared multiple techniques. Finally, when an HRS used contextual information, it was coded whether they used pre- or postfiltering or contextual modeling.

Evaluation Approach

This category encodes which evaluation protocols were used to measure the effect of HRSs. We coded whether the HRSs were evaluated through offline evaluations (no users involved), surveys, heuristic feedback from expert users, controlled user studies, deployments in the wild , and randomized controlled trials (RCTs). We also coded sample size and study duration and whether ethical approval was gathered and needed.

Interface and Transparency

Recommender systems are often perceived as a black box , as the rationale for recommendations is often not explained to end users. Recent research increasingly focuses on providing transparency to the inner logic of the system [ 11 ]. We encoded whether explanations are provided and, in this case, how such transparency is supported in the user interface. Furthermore, we also classified whether the user interface was designed for a specific platform, categorized as mobile , web , or other.

Data Extraction, Intercoder Reliability, and Quality Assessment

The required information for all included technologies and studies was coded by the first author using a data extraction form. Owing to the large variety of study designs, the included studies were assessed for quality (detailed scores given in Multimedia Appendix 1 ) using the tool by Hawker et al [ 108 ]. Using this tool, the abstract and title , introduction and aims , method and data , sample size (if applicable), data analysis , ethics and bias , results , transferability or generalizability , and implications and usefulness were allocated a score between 1 and 4, with higher scoring studies indicating higher quality. A random selection with 14% (10/73) of the papers was listed in a spreadsheet and coded by a second researcher following the defined coding categories and subcategories. The decisions made by the second researcher were compared with the first. With the recommended items ( Multimedia Appendix 2 ), there was only one small disagreement between physical activity and leisure activity [ 32 ], but all other recommended items were rated exactly the same; the recommender techniques had a Cohen κ value of 0.71 ( P <.001) and the evaluation approach scored a Cohen κ value of 0.81 ( P <.001). There was moderate agreement (Cohen κ=0.568; P <.001) between the researchers concerning the quality of the papers. The interfaces used were in perfect agreement. Finally, the coding data are available in Multimedia Appendix 1 .

The literature in three databases yielded 2340 studies, of which only 23 were duplicates and 53 were full proceedings, leaving 2324 studies to be screened for eligibility. A total of 2161 studies were excluded upon title or abstract screening because they were unrelated to health or targeted at medical professionals or because the papers did not report an evaluation. Thus, the remaining 163 full-text studies were assessed for eligibility. After the removal of 90 studies that failed the inclusion criteria or met the exclusion criteria, 73 published studies remained. The search process is illustrated in Figure 1 .

Flow diagram according to the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines. EC: exclusion criteria; IC: inclusion criteria.

All included papers were published in 2009 or later, following an upward trend of increased popularity. The publication venues of HRSs are diverse. Only the PervasiveHealth [ 33 - 35 ], RecSys [ 36 , 37 , 109 ], and WI-IAT [ 38 - 40 ] conferences published 3 papers each that were included in this study. The Journal of Medical Internet Research was the only journal that occurred more frequently in our data set; 5 papers were published by Journal of Medical Internet Research [ 41 - 45 ]. The papers were first rated using Hawker tool [ 108 ]. Owing to a large number of offline evaluations, we did not include the sample score to enable a comparison between all included studies. The papers received an average score of 24.32 (SD 4.55, max 32; data set presented in Multimedia Appendix 1 ). Most studies scored very poor on reporting ethics and potential biases, as illustrated in Figure 2 . However, there is an upward trend over the years in more adequate reporting of ethical issues and potential biases. The authors also limited themselves to their specific case studies and did not make any recommendations for policy (last box plot is presented in Figure 2 ). All 73 studies reported the use of different data sets. Although all recommended items were health related, only Asthana et al [ 46 ] explicitly mentioned using electronic health record data. Only 14% (10/73) [ 7 , 47 - 55 ] explicitly reported that they addressed the cold-start problem.

Distribution of the quality assessment using Hawker tool.

Most HRSs operated in different domains and thus recommended different items. In this study, four nonmutually exclusive categories of recommended items were identified: lifestyle 33% (24/73), nutrition 36% (26/73), general health information 32% (23/73), and specific health condition–related recommendations 12% (9/73). The only significant trend we found is the increasing popularity of nutrition advice . Multimedia Appendix 2 shows the distribution of these recommended items.

Many HRSs, 33% (24/73) of the included studies, suggest lifestyle-related items, but they differ greatly in their exact recommendations. Physical activity is often recommended. Physical activities are often personalized according to personal interests [ 56 ] or the context of the user [ 35 ]. In addition to physical activities, Kumar et al [ 32 ] recommend eating, shopping, and socializing activities. One study analyzes the data and measurements to be tracked for an individual and then recommends the appropriate wearable technologies to stimulate proactive health [ 46 ]. A total of 7 studies [ 7 , 9 , 42 , 53 , 57 - 59 ] more directly try to convince users to alter their behavior by recommending them to change, or alter their behavior: for example, Rabbi et al [ 7 ] learn “a user’s physical activity and dietary behavior and strategically suggests changes to those behaviors for a healthier lifestyle . ” In another example, both Marlin et al [ 59 ] and Sadasivam et al [ 42 ] motivate users to stop smoking by providing them with tailored messages, such as “Keep in mind that cravings are temporary and will pass.” Messages could reflect the theoretical determinants of quitting, such as positive outcome expectations and self-efficacy enhancing small goals [ 42 ].

The influence of food on health is also clear from the large subset of HRSs dealing with nutrition recommendations. A mere 36% (26/73) of the studies recommend nutrition-related information, such as recipes [ 50 ], meal plans [ 36 ], restaurants [ 60 ], or even help with choosing healthy items from a restaurant menu [ 61 ]. Wayman and Madhvanath [ 37 ] provide automated, personalized, and goal-driven dietary guidance to users based on grocery receipt data. Trattner and Elsweiler [ 62 ] use postfiltering to focus on healthy recipes only and extended them with nutrition advice, whereas Ge et al [ 48 ] require users to first enter their preferences for better recommendations. Moreover, Gutiérrez et al [ 63 ] propose healthier alternatives through augmented reality when the users are shopping. A total of 7 studies specifically recommend healthy recipes [ 47 , 48 , 50 , 62 , 64 - 66 ]. Most HRSs consider the health condition of the user, such as the DIETOS system [ 67 ]. Other systems recommend recipes that are synthesized based on existing recipes and recommend new recipes [ 64 ], assist parents in making appropriate food for their toddlers [ 47 ], or help users to choose allergy-safe recipes [ 65 ].

General Health Information

According to 32% (23/73) of the included studies, providing access to trustworthy health care information is another common objective. A total of 5 studies focused on personalized, trustworthy information per se [ 15 , 55 , 68 - 70 ], whereas 5 others focused on guiding users through health care forums [ 52 , 71 - 74 ]. In total, 3 studies [ 55 , 68 , 69 ] provided personalized access to general health information. For example, Sanchez Bocanegra et al [ 15 ] targeted health-related videos and augmented them with trustworthy information from the United States National Library of Medicine (MedlinePlus) [ 110 ]. A total of 3 studies [ 52 , 72 , 74 ] related to health care forums focused on finding relevant threads. Cho et al [ 72 ] built “an autonomous agent that automatically responds to an unresolved user query by posting an automated response containing links to threads discussing similar medical problems.” However, 2 studies [ 71 , 73 ] helped patients to find similar patients. Jiang and Yang [ 71 ] investigated approaches for measuring user similarity in web-based health social websites, and Lima-Medina et al [ 73 ] built a virtual environment that facilitates contact among patients with cardiovascular problems. Both studies aim to help users seek informational and emotional support in a more efficient way. A total of 4 studies [ 41 , 75 - 77 ] helped patients to find appropriate doctors for a specific health problem, and 4 other studies [ 51 , 78 - 80 ] focused on finding nearby hospitals. A total of 2 studies [ 78 , 79 ] simply focused on the clinical preferences of the patients, whereas Krishnan et al [ 111 ] “provide health care recommendations that include Blood Donor recommendations and Hospital Specialization.” Finally, Tabrizi et al [ 80 ] considered patient satisfaction as the primary feature of recommending hospitals to the user.

Specific Health Conditions

The last group of studies (9/73, 12%) focused on specific health conditions. However, the recommended items vary significantly. Torrent-Fontbona and Lopez Ibanez [ 81 ] have built a knowledge-based recommender system to assist diabetes patients in numerous cases, such as the estimated carbohydrate intake and past and future physical activity. Pustozerov et al [ 43 ] try to “reduce the carbohydrate content of the desired meal by reducing the amount of carbohydrate-rich products or by suggesting variants of products for replacement.” Li and Kong [ 82 ] provided diabetes-related information, such as the need for a low-sodium lunch, targeted on American Indians through a mobile app. Other health conditions supported by recommender systems include depression and anxiety [ 83 ], mental disorders [ 45 ], and stress [ 34 , 54 , 84 , 85 ]. Both the mental disorder [ 45 ] and the depression and anxiety [ 83 ] HRSs recommend mobile apps. For example, the app MoveMe suggests exercises tailored to the user’s mood. The HRS to alleviate stress includes recommending books to read [ 54 ] and meditative audios [ 85 ].

The recommender techniques used varied greatly. Table 2 shows the distributions of these recommender techniques.

Overview of the different recommender techniques used in the studies.

Main technique	Study	Total studies, n (%)
Collaborative filtering	[ , , ]	3 (4)
Content-based filtering	[ , , , , , , ]	7 (10)
Knowledge-based filtering	[ , , , , , , , , , , , , - ]	16 (22)
Hybrid	[ , , , , , - , , - , , , , , , , , , , , , , , , - , ]	32 (44)
Context-based techniques	[ , , , ]	4 (5)
Not specified	[ , , ]	3 (4)
Comparison between techniques	[ , , , , , , , ]	8 (11)

a The papers are classified based on how the authors reported their techniques.

Recommender Techniques in Practice

The majority of HRSs (49/73, 67%) rely on knowledge-based techniques, either directly (17/49, 35%) or in a hybrid approach (32/49, 65%). Knowledge-based techniques are often used to incorporate additional information of patients into the recommendation process [ 112 ] and have been shown to improve the quality of recommendations while alleviating other drawbacks such as cold-start and sparsity issues [ 14 ]. Some studies use straightforward approaches, such as if-else reasoning based on domain knowledge [ 9 , 79 , 81 , 82 , 88 , 90 , 100 ]. Other studies use more complex algorithms such as particle swarm optimization [ 57 ], fuzzy logic [ 68 ], or reinforcement algorithms [ 44 , 84 ].

In total, 32 studies reported using a combination of recommender techniques and are classified as hybrid recommender systems . Different knowledge-based techniques are often combined. For example, Ali et al [ 56 ] used a combination of rule-based reasoning, case-based reasoning, and preference-based reasoning to recommend personalized physical activities according to the user’s specific needs and personal interests. Asthana et al [ 46 ] combined the knowledge of a decision tree and demographic information to identify the health conditions. When health conditions are known, the system knows which measurements need to be monitored. A total of 7 studies used a content-based technique to recommend educational content [ 15 , 72 , 87 ], activities [ 32 , 86 ], reading materials [ 54 ], or nutritional advice [ 63 ].

Although collaborative filtering is a popular technique [ 113 ], it is not used frequently in the HRS domain. Marlin et al [ 59 ] used collaborative filtering to personalize future smoking cessation messages based on explicit feedback on past messages. This approach is used more often in combination with other techniques. A total of 2 studies [ 38 , 92 ] combined content-based techniques with collaborative filtering. Esteban et al [ 92 ], for instance, switched between content-based and collaborative approaches. The former approach is used for new physiotherapy exercises and the latter, when a new patient is registered or when previous recommendations to a patient are updated.

Context-Based Recommender Techniques

From an HRS perspective, context is described as an aggregate of various information that describes the setting in which an HRS is deployed, such as the location, the current activity, and the available time of the user. A total of 5 studies use contextual information to improve their recommendations but use a different technique; a prefilter uses contextual information to select or construct the most relevant data for generating recommendations. For example, in Narducci et al [ 75 ], the set of potentially similar patients was restricted to consultation requests in a specific medical area. Rist et al [ 33 ] applied a rule-based contextual prefiltering approach [ 114 ] to filter out inadequate recommendations, for example, “if it is dark outside, all outdoor activities, such as ‘take a walk,’ are filtered out” [ 33 ] before they are fed to the recommendation algorithm. However, a postfilter removes the recommended items after they are generated, such as filtering outdoor activities while it is raining. Casino et al [ 97 ] used a postfiltering technique by running the recommended items through a real-time constraint checker . Finally, contextual modeling, which was used by 2 studies [ 35 , 58 ], uses contextual information directly in the recommendation function as an explicit predictor of a user’s rating for an item [ 114 ].

Location, agenda, and weather are examples of contextual information used by Lin et al [ 35 ] to promote the adoption of a healthy and active lifestyle. Cerón-Rios et al [ 58 ] used a decision tree to analyze user needs, health information, interests, time, location, and lifestyle to promote healthy habits. Casino et al [ 97 ] gathered contextual information through smart city sensor data to recommend healthier routes. Similarly, contextual information was acquired by Rist et al [ 33 ] using sensors embedded in the user’s environment.

Comparisons

A total of 8 papers compared different recommender techniques to find the most optimal algorithm for a specific data set, end users, domain, and goal. Halder et al [ 52 ] used two well-known health forum data sets (PatientsLikeMe [ 115 ] and HealthBoards [ 116 ]) to compare 7 recommender techniques (among collaborative filtering and content-based filtering) and found that a hybrid approach scored best [ 52 ]. Another example is the study by Narducci et al [ 75 ], who compared four recommendation algorithms: cosine similarity as a baseline, collaborative filtering, their own HealthNet algorithm, and a hybrid of HealthNet and cosine similarity. They concluded that a prefiltering technique for similar patients in a specific medical area can drastically improve the recommendation accuracy [ 75 ]. The average and SD of the resulting ratings of the two collaborative techniques are compared with random recommendations by Li et al [ 60 ]. They show that a hybrid approach of a collaborative filter augmented with the calculated health level of the user performs better. In their nutrition-based meal recommender system, Yang et al [ 49 ] used item-wise and pairwise image comparisons in a two-step process. In conclusion, the 8 studies showed that recommendations can be improved when the benefits of multiple recommender techniques are combined in a hybrid solution [ 60 ] or contextual filters are applied [ 75 ].

HRSs can be evaluated in multiple ways. In this study, we found two categories of HRS evaluations: (1) offline evaluations that use computational approaches to evaluate the HRS and (2) evaluations in which an end user is involved. Some studies used both, as shown in Multimedia Appendix 3 .

Offline Evaluations

Of the total studies, 47% (34/73) do not involve users directly in their method of evaluation. The evaluation metrics also vary greatly, as many distinct metrics are reported in the included papers ( Multimedia Appendix 3 ). Precision 53% (18/34), accuracy 38% (13/34), performance 35% (12/34), and recall 32% (11/34) were the most commonly used offline evaluation metrics. Recall has been used significantly more in recent papers, whereas accuracy also follows an upward trend. Moreover, performance was defined differently across studies. Torrent-Fontbona and Lopez Ibanez [ 81 ] compared the “amount of time in the glycaemic target range by reducing the time below the target” as performance. Cho et al [ 72 ] compared the precision and recall to report the performance. Clarke et al [ 84 ] calculated their own reward function to compare different approaches, and Lin et al [ 35 ] measured system performance as the number of messages sent in their in the wild study. Finally, Marlin et al [ 59 ] tested the predictive performance using a triple cross-validation procedure.

Other popular offline evaluation metrics are accuracy-related measurements, such as mean absolute (percentage) error, 18% (6/34); normalized discounted cumulative gain (nDCG), 18% (6/34); F 1 score, 15% (5/34); and root mean square error, 15% (5/34). The other metrics were measured inconsistently. For example, Casino et al [ 97 ] reported that they measure robustness but do not outline what they measure as robustness. However, they measured the mean absolute error. Torrent-Fontbona and Lopez Ibanez [ 81 ] defined robustness as the capability of the system to handle missing values. Effectiveness is also measured with different parameters, such as its ability to take the right classification decisions [ 75 ] or in terms of key opinion leaders’ identification [ 41 ]. Finally, Li and Zaman [ 68 ] measured trust with a proxy: “evaluate the trustworthiness of a particular user in a health care social network based on factors such as role and reputation of the user in the social community” [ 68 ].

User Evaluations

Of the total papers, 53% (39/73) included participants in their HRS evaluation, with an average sample size of 59 (SD 84) participants (excluding the outlier of 8057 participants, as recruited in the study by Cheung et al [ 83 ]). On average, studies ran for more than 2 months (68, SD 56 days) and included all age ranges. There is a trend of increasing sample size and study duration over the years. However, only 17 studies reported the study duration; therefore, these trends were not significant. Surveys (12/39, 31%), user studies (10/39, 26%), and deployments in the wild (10/39, 26%) were the most used user evaluations. Only 6 studies used an RCT to evaluate their HRS. Finally, although all the included studies focused on HRSs and were dealing with sensitive data, only 12% (9/73) [ 9 , 34 , 42 - 45 , 73 , 83 , 95 ] reported ethical approval by a review board.

No universal survey was found, as all the studies deployed a distinct survey. Ge et al [ 48 ] used the system usability scale and the framework of Knijnenburg et al [ 117 ] to explain the user experience of recommender systems. Esteban et al [ 95 ] designed their own survey with 10 questions to inquire about user experience. Cerón-Rios [ 58 ] relied on the ISO/IEC (International Organization of Standardization/International Electrotechnical Commission) 25000 standard to select 7 usability metrics to evaluate usability. Although most studies did not explicitly report the surveys used, user experience was a popular evaluation metric, as in the study by Wang et al [ 69 ]. Other metrics range from measuring user satisfaction [ 69 , 99 ] and perceived prediction accuracy [ 59 ] (with 4 self-composed questions). Nurbakova et al [ 98 ] combined data analytics with surveys to map their participants’ psychological background, including orientations to happiness measured using the Peterson scale [ 118 ], personality traits using the Mini-International Personality Item Pool [ 119 ], and Fear of Missing Out based on the Przybylski scale [ 120 ].

Single-Session Evaluations (User Studies)

A total of 10 studies recruited users and asked them to perform certain tasks in a single session. Yang et al [ 49 ] performed a 60-person user study to assess its feasibility and effectiveness. Each participant was asked to rate meal recommendations relative to those made using a traditional survey-based approach. In a study by Gutiérrez et al [ 63 ], 15 users were asked to use the health augmented reality assistant and measure the qualities of the recommender system, users’ behavioral intentions, perceived usefulness, and perceived ease of use. Jiang and Xu [ 77 ] performed 30 consultations and invited 10 evaluators majoring in medicine and information systems to obtain an average rating score and nDCG. Radha et al [ 8 ] used comparative questions to evaluate the feasibility. Moreover, Cheng et al [ 89 ] used 2 user studies to rank two degrees of compromise (DOC). A low DOC assigns more weight to the algorithm, and a high DOC assigns more weight to the user’s health perspective. Recommendations with a lower DOC are more efficient for the user’s health, but recommendations with a high DOC could convince users to believe that the recommended action is worth doing. Other approaches used are structured interviews [ 58 ], ranking [ 86 , 89 ], asking for unstructured feedback [ 40 , 88 ], and focus group discussions [ 87 ]. Finally, 3 studies [ 15 , 75 , 90 ] evaluated their system through a heuristic evaluation with expert users.

In the Wild

Only 2 studies tested their HRS into the wild recruited patients (people with a diagnosed health condition) in their evaluation. Yom-Tov et al [ 44 ] provided 27 sedentary patients with type 2 diabetes with a smartphone-based pedometer and a personal plan for physical activity. They assessed the effectiveness by calculating the amount of activity that the patient performed after the last message was sent. Lima-Medina et al [ 73 ] interviewed 45 patients with cardiovascular problems after a 6-month study period to measure (1) social management results, (2) health care plan results, and (3) recommendation results. Rist et al [ 33 ] performed an in-situ evaluation in an apartment of an older couple and used the data logs to describe the usage but augmented the data with a structured interview.

Yang et al [ 49 ] conducted a field study of 227 anonymous users that consisted of a training phase and a testing phase to assess the prediction accuracy. Buhl et al [ 99 ] created three user groups according to the recommender technique used and analyzed log data to compare the response rate, open email rate, and consecutive log-in rate. Similarly, Huang et al [ 76 ] compared the ratio of recommended doctors chosen and reserved by patients with the recommended doctors. Lin et al [ 35 ] asked 6 participants to use their HRSs for 5 weeks, measured system performance, studied user feedback to the recommendations, and concluded with an open-user interview. Finally, Ali et al [ 56 ] asked 10 volunteers to use their weight management systems for a couple of weeks. However, they do not focus on user-centric evaluation, as “only a prototype of the [...] platform is implemented.”

Rabbi et al [ 7 ] followed a single case with multiple baseline designs [ 121 ]. Single-case experiments achieve sufficient statistical power with a large number of repeated samples from a single individual. Moreover, Rabbi et al [ 7 ] argued that HRSs suit this requirement “since enough repeated samples can be collected with automated sensing or daily manual logging [ 121 ].” Participants were exposed to 2, 3, or 4 weeks of the control condition. The study ran for 7-9 weeks to compensate for the novelty effects. Food and exercise log data were used to measure changes in food calorie intake and calorie loss during exercise.

Randomized Controlled Trials

Only 6 studies followed an RCT approach. In the RCT by Bidargaddi et al [ 45 ], a large group of patients (n=192) and control group (n=195) were asked to use a web-based recommendation service for 4 weeks that recommended mental health and well-being mobile apps. Changes in well-being were measured using the Mental Health Continuum-Short Form [ 122 ]. The RCT by Sadasivam et al [ 42 ] enrolled 120 current smokers (n=74) and control group (n=46) as a follow-up to a previous RCT [ 123 ] that evaluated their portal to specifically evaluate the HRS algorithm. Message ratings were compared between the intervention and control groups.

Cheung et al [ 83 ] measured app loyalty through the number of weekly app sessions over a period of 16 weeks with 8057 users. In the study by Paredes et al [ 34 ], 120 participants had to use the HRS for at least 26 days. Self-reported stress assessment was performed before and after the intervention. Agapito et al [ 67 ] used an RCT with 40 participants to validate the sensitivity (true positive rate/[true positive rate+false negative rate]) and specificity (true negative rate/[true negative rate+false positive rate]) of the DIETOS HRS. Finally, Luo et al [ 93 ] performed a small clinical trial for more than 3 months (but did not report the number of participants). Their primary outcome measures included two standard clinical blood tests: fasting blood glucose and laboratory-measured glycated hemoglobin, before and after the intervention.

Only 47% (34/73) of the studies reported implementing a graphical user interface to communicate the recommended health items to the user. As illustrated in Table 3 , 53% (18/34) use a mobile interface, usually through a mobile (web) app, whereas 36% (14/34) use a web interface to show the recommended items. Rist et al [ 33 ] built a kiosk into older adults’ homes, as illustrated in Figure 3 . Gutiérrez et al [ 63 ] used Microsoft HoloLens to project healthy food alternatives in augmented reality surrounding a physical object that the user holds, as shown in Figure 4 .

Distribution of the interfaces used among the different health recommender systems (n=34).

Interface	Study	Total studies, n (%)
Mobile	[ , , , , , , , , , , , , - , , , ]	18 (53)
Web	[ , , , , , , , , , , , , , ]	14 (41)
Kiosk	[ ]	1 (3)
HoloLens	[ ]	1 (3)

Rist et al installed a kiosk in the home of older adults as a direct interface to their health recommender system.

An example of the recommended healthy alternatives by Gutiérrez et al.

Visualization

A total of 7 studies [ 33 , 34 , 37 , 63 , 79 , 88 , 97 ] or approximately one-fourth of the studies with an interface included visualizations. However, the approach used was different for all studies, as shown in Table 4 . Showing stars to show the relevance of a recommended item are only used by Casino et al [ 97 ] and Gutiérrez et al [ 63 ]. Wayman and Madhvanath [ 37 ] also used bar charts to visualize the progress toward a health goal. They visualize the healthy proportions, that is, what the user should eat. Somewhat more complex visualizations are used by Ho and Chen [ 88 ] who visualized the user’s ECG zones. Paredes et al [ 34 ] presented an emotion graph as an input screen. Rist et al [ 33 ] visualized an example of how to perform the recommended activity.

Distribution of the visualizations used among the different health recommender systems (n=7).

Visualization technique	Study	Total studies, n (%)
Bar charts	Wayman and Madhvanath [ ] and Gutiérrez et al [ ]	2 (29)
Heatmap	Ho and Chen [ ]	1 (14)
Emotion graph	Paredes et al [ ]	1 (14)
Visual example of action	Rist et al [ ]	1 (14)
Map	Avila-Vazquez et al [ ]	1 (14)
Star rating	Casino et al [ ]	1 (14)

Transparency

In the study by Lage et al [ 87 ], participants expressed that:

they would like to have more control over recommendations received. In that sense, they suggested more information regarding the reasons why the recommendations are generated and more options to assess them.

A total of 7 studies [ 7 , 37 , 41 , 45 , 63 , 66 , 82 ] explained the reasoning behind recommendations to end users at the user interface. Gutiérrez et al [ 63 ] provided recommendations for healthier food products and mentioned that the items ( Figure 4 ) are based on the users’ profile. Ueta et al [ 66 ] explained the relationship between the recommended dishes and a person’s health conditions. For example, a person with acne can see the following text: “15 dishes that contained Pantothenic acid thought to be effective in acne a lot became a hit” [ 66 ]. Li and Kong [ 82 ] showed personalized recommended health actions in a message center. Color codes are used to differentiate between reminders, missed warnings, and recommendations. Rabbi et al [ 7 ] showed tailored motivational messages to explain why activities are recommended. For example, when the activity walk near East Ave is recommended, the app shows the additional message:

1082 walks in 240 days, 20 mins of walk everyday. Each walk nearly 4 min. Let us get 20 mins or more walk here today 7

Wayman and Madhvanath [ 37 ] first visualized the user’s personal nutrition profile and used the lower part of the interface to explain why the item was recommended. They provided an illustrative example of spaghetti squash. The explanation shows that:

This product is high in Dietary_fiber, which you could consume more of. Try to get 3 servings a week 37

Guo et al [ 41 ] recommended doctors and showed a horizontal bar chart to visualize the user’s values compared with the average values. Finally, Bidargaddi et al [ 45 ] visualized how the recommended app overlaps with the goal set by the users, as illustrated in Figure 5 .

A screenshot from the health recommender system of Bidargaddi et al. Note the blue tags illustrating how each recommended app matches the users’ goals.

Principal Findings

HRSs cover a multitude of subdomains, recommended items, implementation techniques, evaluation designs, and means of communicating the recommended items to the target user. In this systematic review, we clustered the recommended items into four groups: lifestyle, nutrition, general health care information, and specific health conditions. There is a clear trend toward HRSs that provide well-being recommendations but do not directly intervene in the user’s medical status. For example, almost 70% (50/73; lifestyle and nutrition) focused on no strict medical recommendations. In the lifestyle group, physical activities (10/24, 42%) and advice on how to potentially change behavior (7/24, 29%) were recommended most often. In the nutrition group, these recommendations focused on nutritional advice (8/26, 31%), diets (7/26, 27%), and recipes (7/26, 27%). A similar trend was observed in the health care information group, where HRSs focused on guiding users to the appropriate environments such as hospitals (5/23, 22%) and medical professionals (4/23, 17%) or on helping users find qualitative information (5/23, 22%) on validated sources or from experiences by similar users and patients on health care forums (3/23, 13%). Thus, they only provide general information and do not intervene by recommending, for example, changing medication. Finally, when HRSs targeted specific health conditions, they recommended nonintervening actions, such as meditation sessions [ 84 ] or books to read [ 54 ].

Although collaborative filtering is commonly the most used technique in other domains [ 124 ], here only 3 included studies reported the use of a collaborative filtering approach. Moreover, 43% (32/73) of the studies applied a hybrid approach, showing that HRS data sets might need special attention, which might also be the reason why all 73 studies used distinct data sets. In addition, the HRS evaluations varied greatly and were divided over evaluations where the end user was involved and evaluations that did not evolve users (offline evaluations). Only 47% (34/73) of the studies reported implementing a user interface to communicate recommendations to the user, despite the need to show the rationale of recommendations, as echoed by many researchers and practitioners [ 11 ]. Moreover, only 15% (7/47) included a (basic) visualization.

Unfortunately, this general lack of agreement on how to report HRSs might introduce researcher bias, as a researcher is currently completely unconstrained in defining what and how to measure the added value of an HRS. Therefore, further debate in the health recommender community is needed on how to define and measure the impact of HRSs. On the basis of our review and contribution to this discussion, we put forward a set of essential information that researchers should report in their studies.

Considerations for Practice

The previously discussed results have direct implications in practice and provide suggestions for future research. Figure 6 shows a reference frame of these requirements that can be used in future studies as a quality assessment tool.

A reference frame to report health recommender system studies. On the basis of the results of this study, we suggest that it should be clear what and how items are recommended (A), who the target user is (B), which data are used (C), and which recommender techniques are applied (D). Finally, the evaluation design should be reported in detail (E).

Define the Target User

As shown in this review, HRSs are used in a plethora of subdomains and each domain has its own experts. For example, in nutrition, the expert is most likely a dietician. However, the user of an HRS is usually a layperson without the knowledge of these domain experts, who often have different viewing preferences [ 125 ]. Furthermore, each user is unique. All individuals have idiosyncratic reasons for why they act, think, behave, and feel in a certain way at a specific stage of their life [ 126 ]. Not everybody is motivated by the same elements. Therefore, it is important to know the target user of the HRS. What is their previous knowledge, what are their goals, and what motivates them to act on a recommended item?

Show What Is Recommended (and How)

Researchers have become aware that accuracy is not sufficient to increase the effectiveness of a recommender system [ 127 ]. In recent years, research on human factors has gained attention. For example, He et al [ 11 ] surveyed 24 existing interactive recommender systems and compared their transparency, justification, controllability, and diversity. However, none of these 24 papers discussed HRSs. This indicates the gap between HRSs and recommender systems in other fields. Human factors have gained interest in the recommender community by “combining interactive visualization techniques with recommendation techniques to support transparency and controllability of the recommendation process” [ 11 ]. However, in this study, only 10% (7/73) explained the rationale of recommendations and only 10% (7/73) included a visualization to communicate the recommendations to the user. We do not argue that all HRSs should include a visualization or an explanation. However, researchers should pay attention to the delivery of these recommendations. Users need to understand, believe, and trust the recommended items before they can act on it.

To compare and assess HRSs, researchers should unambiguously report what the HRS is recommending. After all, typical recommender systems act like a black box , that is, they show suggestions without explaining the provenance of these recommendations [ 11 ]. Although this approach is suitable for typical e-commerce applications that involve little risk, transparency is a core requirement in higher risk application domains such as health [ 128 ]. Users need to understand why a recommendation is made, to assess its value and importance [ 12 ]. Moreover, health information can be cumbersome and not always easy to understand or situate within a specific health condition [ 129 ]. Users need to know whether the recommended item or action is based on a trusted source, tailored to their needs, and actionable [ 130 ].

Report the Data Set Used

All 73 studies used a distinct data set. Furthermore, some studies combine data from multiple databases, making it even more difficult to judge the quality of the data [ 35 ]. Nonetheless, most studies use self-generated data sets. This makes it difficult to compare and externally validate HRSs. Therefore, we argued that researchers should clarify the data used and potentially share whether these data are publicly available. However, in health data are often highly privacy sensitive and cannot be shared among researchers.

Outline the Recommender Techniques

The results show that there is no panacea for which recommender technique to use. The included studies differ from logic filters to traditional recommender techniques, such as collaborative filtering and content-based filtering to hybrid solutions and self-developed algorithms. However, with 44% (32/73), there is a strong trend toward the use of hybrid recommender techniques. The low number of collaborative filter techniques might be related to the fact that the evaluation sample sizes were also relatively low. Unfortunately, some studies have not fully disclosed the techniques used and only reported on the main algorithm used. It is remarkable that studies published in high-impact journals, such as studies by Bidargaddi et al [ 45 ] and Cheung et al [ 83 ], did not provide information on the recommender technique used. Nonetheless, disclosing the recommender technique allows other researchers not only to build on empirically tested technologies but also to verify whether key variables are included [ 29 ]. User data and behavior data can be identified to augment theory-based studies [ 29 ]. Researchers should prove that the algorithm is capable of recommending valid and trustworthy recommendations to the user based on their available data set.

Elaborate on the Evaluation Protocols

HRSs can be evaluated using different evaluation protocols. However, the protocol should be outlined mainly by the research goals of the authors. On the basis of the papers included in this study, we differentiate between the two approaches. In the first approach, the authors aim to influence their users’ health, for example, by providing personalized diabetes guidelines [ 81 ] or prevention exercises for users with low back pain [ 95 ]. Therefore, the end user should always be involved in both the design and evaluation processes. However, only 8% (6/73) performed an RCT and 14% (10/73) deployed their HRS in the wild. This lack of user involvement has been noted previously by researchers and has been identified as a major challenge in the field [ 27 , 28 ]. Nonetheless, in other domains, such as job recommenders [ 131 ] or agriculture [ 132 ], user-centered design has been proposed as an important methodology in the design and development of tools used by end users, with the purpose of gaining trust and promoting technology acceptance, thereby increasing adoption with end users. Therefore, we recommend that researchers evaluate their HRSs with actual users. A potential model for a user-centric approach to recommender system evaluation is the user-centric framework proposed by Knijnenburg et al [ 117 ].

Research protocols need to be elaborated and approved by an ethical review board to prevent any impact on users. Authors should report how they informed their users and how they safeguarded the privacy of the users. This is in line with the modern journal and conference guidelines. For example, editorial policies of the Journal of Medical Internet Research state that “when reporting experiments on human subjects, authors should indicate IRB (Institutional Rese[a]rch Board, also known as REB) approval/exemption and whether the procedures followed were in accordance with the ethical standards of the responsible committee on human experimentation” [ 133 ]. However, only 12% (9/73) reported their approval by an ethical review board. Acquiring review board approval will help the field mature and transition from small incremental studies to larger studies with representative users to make more reliable and valid findings.

In the second approach, the authors aim to design a better algorithm, where better is again defined by the authors. For example, the algorithm might perform faster, be more accurate, and be more efficient in computing power. Although the F 1 score, the mean absolute error, and nDCG are well defined and known within the recommender domain, other parameters are more ambiguous. For example, the performance or effectiveness can be assessed using different measurements. However, a health parameter can be monitored, such as the duration that a user remains within healthy ranges [ 81 ]. Furthermore, it could be a predictive parameter, such as an improved precision and recall as a proxy for performance [ 72 ]. Unfortunately, this difference makes it difficult to compare health recommendation algorithms. Furthermore, this inconsistency in measurement variables makes it infeasible to report in this systematic review which recommender techniques to use. Therefore, we argue that HRS algorithms should always be evaluated for other researchers to validate the results, if needed.

Limitations

This study has some limitations that affect its contribution. Although an extensive scope search was conducted in scientific databases and most relevant health care informatic journals, some relevant literature in other domains might have been excluded. The keywords used in the search string could have impacted the results. First, we did not include domain-specific constructs of health, such as asthma, pregnancy, and iron deficiency. Many studies may implicitly report healthy computer-generated recommendations when they research the impact of a new intervention. In these studies, however, building an HRS is often not their goal and, therefore, was excluded from this study. Second, we searched for papers that reported studying an HRS; nonincluded studies might have built an HRS but did not report it as such. Considering our RQs, we deemed it important that authors explicitly reported their work as a recommender system. To conclude, in this study, we provide a large cross-domain overview of health recommender techniques targeted to laypersons and deliver a set of recommendations that could help the field of HRS mature.

This study presents a comprehensive report on the use of HRS across domains. We have discussed the different subdomains HRS applied in, the different recommender techniques used, the different manners in which they are evaluated, and finally, how they present the recommendations to the user. On the basis of this analysis, we have provided research guidelines toward a consistent reporting of HRSs. We found that although most applications are intended to improve users’ well-being, there is a significant opportunity for HRSs to inform and guide users’ health actions. Although many of the studies present a lack of a user-centered evaluation approach, some studies performed full-scale RCT evaluations or elaborated in the wild studies to validate their HRS, showing the field of HRS is slowly maturing. On the basis of this study, we argue that it should always be clear what the HRS is recommending and to whom these recommendations are for. Graphical assets should be added to show how recommendations are presented to users. Authors should also report which data sets and algorithms were used to calculate the recommendations. Finally, detailed evaluation protocols should be reported.

We conclude that the results motivate the creation of richer applications in future design and development of HRSs. The field is maturing, and interesting opportunities are being created to inform and guide health actions.

Acknowledgments

This work was part of the research project PANACEA Gaming Platform with project HBC.2016.0177, which was financed by Flanders Innovation & Entrepreneurship and research project IMPERIUM with research grant G0A3319N from the Research Foundation-Flanders (FWO) and the Slovenian Research Agency grant ARRS-N2-0101. Project partners were BeWell Innovations and the University Hospital of Antwerp.

Abbreviations

DOC	degrees of compromise
HRS	health recommender system
ISO/IEC	International Organization of Standardization/International Electrotechnical Commission
nDCG	normalized discounted cumulative gain
PRISMA	Preferred Reporting Items for Systematic Reviews and Meta-Analyses
RCT	randomized controlled trial
RQ	research question

Multimedia Appendix 1

Multimedia appendix 2, multimedia appendix 3.

Conflicts of Interest: None declared.

Subscribe to the PwC Newsletter

Join the community, add a new evaluation result row, recommendation systems.

1618 papers with code • 54 benchmarks • 56 datasets

The Recommendation Systems task is to produce a list of recommendations for a user. The most common methods used in recommender systems are factor models (Koren et al., 2009; Weimer et al., 2007; Hidasi & Tikk, 2012) and neighborhood methods (Sarwar et al., 2001; Koren, 2008). Factor models work by decomposing the sparse user-item interactions matrix to a set of d dimensional vectors one for each item and user in the dataset. Factor models are hard to apply in session-based recommendations due to the absence of a user profile. On the other hand, neighborhood methods, which rely on computing similarities between items (or users) are based on co-occurrences of items in sessions (or user profiles). Neighborhood methods have been used extensively in session-based recommendations.

( Image credit: CuMF_SGD )

Benchmarks Add a Result

--> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> --> -->

Trend	Dataset	Best Model	Paper	Code	Compare
		GLocal-K
		HyperML
		GHRS
		Bayesian timeSVD++ flipped
		SSCF
		ConvNCF
		H+Vamp Gated
		NESCL
		GLocal-K
		I-CFN
		EASE
		IGMC
		C2CRS
		CARCA Abs + Con
		ProxyRCA
		MG-GAT
		GRALS
		DANSER
		ProxyRCA
		GRALS
		Fashion GAE
		Ekar*
		CARCA Learnt + Con
		INN
		KTUP (soft)
		DGRec
		TransCF
		HetroFair
		DANSER
		TLSAN
		NeuMF
		TransCF
		TransCF
		TransCF
		TransCF
		RATE-CSE
		RATE-CSE
		RANK-CSE
		RANK-CSE
		RANK-CSE
		DGRec
		KGNN-LS
		HGN
		HGN
		SASRec
		Multi-Gradient Descent
		HAKG
		LT-OCF
		CFM
		SASRec
		HetroFair
		HetroFair
		HetroFair
		HetroFair

Most implemented papers

Neural collaborative filtering.

hexiangnan/neural_collaborative_filtering • WWW 2017

When it comes to model the key factor in collaborative filtering -- the interaction between user and item features, they still resorted to matrix factorization and applied an inner product on the latent features of users and items.

Wide & Deep Learning for Recommender Systems

Memorization of feature interactions through a wide set of cross-product feature transformations are effective and interpretable, while generalization requires more feature engineering effort.

FiBiNET: Combining Feature Importance and Bilinear feature Interaction for Click-Through Rate Prediction

In this paper, a new model named FiBiNET as an abbreviation for Feature Importance and Bilinear feature Interaction NETwork is proposed to dynamically learn the feature importance and fine-grained feature interactions.

Session-based Recommendations with Recurrent Neural Networks

We apply recurrent neural networks (RNN) on a new domain, namely recommender systems.

DeepFM: A Factorization-Machine based Neural Network for CTR Prediction

Learning sophisticated feature interactions behind user behaviors is critical in maximizing CTR for recommender systems.

Neural Graph Collaborative Filtering

Further analysis verifies the importance of embedding propagation for learning better user and item representations, justifying the rationality and effectiveness of NGCF.

xDeepFM: Combining Explicit and Implicit Feature Interactions for Recommender Systems

On one hand, the xDeepFM is able to learn certain bounded-degree feature interactions explicitly; on the other hand, it can learn arbitrary low- and high-order feature interactions implicitly.

Variational Autoencoders for Collaborative Filtering

dawenl/vae_cf • 16 Feb 2018

This non-linear probabilistic model enables us to go beyond the limited modeling capacity of linear factor models which still largely dominate collaborative filtering research. We introduce a generative model with multinomial likelihood and use Bayesian inference for parameter estimation.

Deep Learning Recommendation Model for Personalization and Recommendation Systems

With the advent of deep learning, neural network-based recommendation models have emerged as an important tool for tackling personalization and recommendation tasks.

MaskNet: Introducing Feature-Wise Multiplication to CTR Ranking Models by Instance-Guided Mask

We also turn the feed-forward layer in DNN model into a mixture of addictive and multiplicative feature interactions by proposing MaskBlock in this paper.

Scientific paper recommendation systems: a literature review of recent publications

New Citation Alert added!

This alert has been successfully added and will be sent to:

You will be notified whenever a record that you have chosen has been cited.

To manage your alert preferences, click on the button below.

New Citation Alert!

Please log in to your account

Information & Contributors

Bibliometrics & citations, view options.

• Stergiopoulos V Vassilakopoulos M Tousidou E Corral A (2024) An academic recommender system on large citation data based on clustering, graph modeling and deep learning Knowledge and Information Systems 10.1007/s10115-024-02094-7 66 :8 (4463-4496) Online publication date: 1-Aug-2024 https://dl.acm.org/doi/10.1007/s10115-024-02094-7
• Li W Xie Y Jiang H Sun Y (2023) Differentiable Topics Guided New Paper Recommendation Neural Information Processing 10.1007/978-981-99-8076-5_4 (44-56) Online publication date: 20-Nov-2023 https://dl.acm.org/doi/10.1007/978-981-99-8076-5_4

Index Terms

Information systems

Information retrieval

Retrieval tasks and goals

Document filtering

Information extraction

Information systems applications

Recommendations

Acknowledgments in scientific publications: presence in spanish science and text patterns across disciplines.

The acknowledgments in scientific publications are an important feature in the scholarly communication process. This research analyzes funding acknowledgment presence in scientific publications and introduces a novel approach for discovering text ...

Retracted publications in the biomedical literature with authors from mainland China

The number of retracted articles with Chinese authors has raised much attention, but no systematic study has specifically explored the retraction of academic publications by researchers from mainland China. Here, we determined the characteristics of ...

Scholarly recommendation systems: a literature survey

A scholarly recommendation system is an important tool for identifying prior and related resources such as literature, datasets, grants, and collaborators. A well-designed scholarly recommender significantly saves the time of researchers and can ...

Information

Published in.

Springer-Verlag

Berlin, Heidelberg

Publication History

Author tags.

• Paper recommendation system
• Publication suggestion
• Literature review
• Research-article

Funding Sources

• Technische Hochschule Köln (3331)

Contributors

Other metrics, bibliometrics, article metrics.

• 1 Total Citations View Citations
• 0 Total Downloads
• Downloads (Last 12 months) 0
• Downloads (Last 6 weeks) 0

View options

Login options.

Check if you have access through your login credentials or your institution to get full access on this article.

Full Access

Share this publication link.

Copying failed.

Share on social media

Affiliations, export citations.

• Please download or close your previous search result export first before starting a new bulk export. Preview is not available. By clicking download, a status dialog will open to start the export process. The process may take a few minutes but once it finishes a file will be downloadable from your browser. You may continue to browse the DL while the export process is in progress. Download
• Download citation
• Copy citation

We are preparing your search results for download ...

We will inform you here when the file is ready.

Your file of search results citations is now ready.

Your search export query has expired. Please try again.

Information

• Author Services

Initiatives

You are accessing a machine-readable page. In order to be human-readable, please install an RSS reader.

All articles published by MDPI are made immediately available worldwide under an open access license. No special permission is required to reuse all or part of the article published by MDPI, including figures and tables. For articles published under an open access Creative Common CC BY license, any part of the article may be reused without permission provided that the original article is clearly cited. For more information, please refer to https://www.mdpi.com/openaccess .

Feature papers represent the most advanced research with significant potential for high impact in the field. A Feature Paper should be a substantial original Article that involves several techniques or approaches, provides an outlook for future research directions and describes possible research applications.

Feature papers are submitted upon individual invitation or recommendation by the scientific editors and must receive positive feedback from the reviewers.

Editor’s Choice articles are based on recommendations by the scientific editors of MDPI journals from around the world. Editors select a small number of articles recently published in the journal that they believe will be particularly interesting to readers, or important in the respective research area. The aim is to provide a snapshot of some of the most exciting work published in the various research areas of the journal.

Original Submission Date Received: .

• Active Journals
• Find a Journal
• Proceedings Series
• For Authors
• For Reviewers
• For Editors
• For Librarians
• For Publishers
• For Societies
• For Conference Organizers
• Open Access Policy
• Institutional Open Access Program
• Special Issues Guidelines
• Editorial Process
• Research and Publication Ethics
• Article Processing Charges
• Testimonials
• Preprints.org
• SciProfiles
• Encyclopedia

Article Menu

• Subscribe SciFeed
• Author Biographies
• Google Scholar
• on Google Scholar
• Table of Contents

Find support for a specific problem in the support section of our website.

Please let us know what you think of our products and services.

Visit our dedicated information section to learn more about MDPI.

JSmol Viewer

Recommender systems and over-the-top services: a systematic review study (2010–2022).

1. Introduction

2. background, 3.1. data collection, 3.2. protocol and coding, 4.1. what research objectives have been pursued, 4.1.1. data management as a strategic practice in the media industry, 4.1.2. the impact of recommender systems on viewers’ experience, 4.1.3. the impact of recommender systems on culture and cultural production, 4.1.4. algorithmic bias, inclusion, diversity, and digital divides, 4.1.5. globalization, power, and the political economy of ott services, 4.2. what concepts have been developed and/or applied, 4.3. what methodologies have been privileged, 4.4. which ott platforms have received the most research attention, 5. future directions, 6. final remarks, author contributions, data availability statement, conflicts of interest.

• AI HLEG. 2019. High-level expert group on artificial intelligence. In Ethics Guidelines for Trustworthy AI ; Brussels: European Commission, 6p. [ Google Scholar ]
• Ananny, Mike. 2016. Toward an Ethics of Algorithms: Convening, Observation, Probability, and Timeliness. Science, Technology, & Human Values 41: 93–117. [ Google Scholar ] [ CrossRef ]
• Ang, Ien. 2006. Desperately Seeking the Audience . London: Routledge. First published 1991. [ Google Scholar ]
• Baker, Djoymi. 2017. Terms of Excess: Binge-Viewing, Epic-Viewing, and the Netflix Effect. The Age of Netflix: Critical Essays on Streaming Media, Digital Delivery and Instant Access . Jefferson: McFarland and Company, pp. 31–54. [ Google Scholar ]
• Baumeister, Roy F., and Mark R. Leary. 1997. Writing Narrative Literature Reviews. Review of General Psychology 1: 311–20. [ Google Scholar ] [ CrossRef ]
• Beer, David. 2017. The Social Power of Algorithms. Information, Communication & Society 20: 1–13. [ Google Scholar ] [ CrossRef ]
• Behrens, Ronny, Natasha Zhang Foutz, Michael Franklin, Jannis Funk, Fernanda Gutierrez-Navratil, Julian Hofmann, and Ulrike Leibfried. 2021. Leveraging Analytics to Produce Compelling and Profitable Film Content. Journal of Cultural Economics 45: 171–211. [ Google Scholar ] [ CrossRef ]
• Benavides Almarza, Cristóbal Fernando, and Ligia García-Béjar. 2021. ¿ Por qué ven Netflix quienes ven Netflix?: Experiencias de engagement de jóvenes mexicanos frente a quien revolucionó el consumo audiovisual. Revista de Comunicación 20: 29–47. [ Google Scholar ] [ CrossRef ]
• Bonini, Tiziano, and Eleonora Maria Mazzoli. 2022. A convivial-agonistic framework to theorise public service media platforms and their governing systems. New Media & Society 24: 922–941. [ Google Scholar ]
• Borrajo, E. N., J. Clares-Gavilán, and J. Sánchez-Navarro. 2020. Impacto de los servicios over-the-top en la generación de comunidades de gustos y nichos globales: Netflix como estudio de caso. Comunicació: Revista de Recerca i d’Anàlisi [Societat Catalana de Comunicació] 37: 93–11. [ Google Scholar ]
• Bourdon, Jérôme, and Cécile Méadel. 2014. Television Audiences across the World: Deconstructing the Ratings Machine. London: Palgrave Macmillan. [ Google Scholar ]
• Bucher, Taina. 2018. If… Then: Algorithmic Power and Politics . Oxford: Oxford University Press. [ Google Scholar ]
• Burroughs, Benjamin. 2019. House of Netflix: Streaming media and digital lore. Popular Communication 17: 1–17. [ Google Scholar ] [ CrossRef ]
• Caers, Ralf, Tim De Feyter, Marijke De Couck, Talia Stough, Claudia Vigna, and Cind Du Bois. 2013. Facebook: A Literature Review. New Media & Society 15: 982–1002. [ Google Scholar ] [ CrossRef ]
• Chen, Liang, Shirley S. Ho, and May O. Lwin. 2017. A Meta-Analysis of Factors Predicting Cyberbullying Perpetration and Victimization: From the Social Cognitive and Media Effects Approach. New Media & Society 19: 1194–213. [ Google Scholar ] [ CrossRef ]
• Colbjørnsen, Terje. 2021. The streaming network: Conceptualizing distribution economy, technology, and power in streaming media services. Convergence 27: 1264–87. [ Google Scholar ] [ CrossRef ]
• Couldry, Nick. 2020. Recovering Critique in an Age of Datafication. New Media & Society 22: 1135–51. [ Google Scholar ] [ CrossRef ]
• Cowgill, Bo, and Catherine E. Tucker. 2020. Algorithmic Fairness and Economics. In Columbia Business School Research Paper . Available online: https://ssrn.com/abstract=3361280 (accessed on 12 July 2024). [ CrossRef ]
• Dogruel, Leyla, Philipp Masur, and Sven Joeckel. 2021. Development and Validation of an Algorithm Literacy Scale for Internet Users. Communication Methods and Measures 16: 115–33. [ Google Scholar ] [ CrossRef ]
• Dourish, Paul. 2016. Algorithms and Their Others: Algorithmic Culture in Context. Big Data & Society 3: 2053951716665128. [ Google Scholar ] [ CrossRef ]
• Eklund, Oliver. 2022. Custom Thumbnails: The Changing Face of Personalisation Strategies on Netflix. Convergence 28: 737–60. [ Google Scholar ] [ CrossRef ]
• Elkins, Evan. 2019. Algorithmic cosmopolitanism: On the global claims of digital entertainment platforms. Critical Studies in Media Communication , 36. [ Google Scholar ] [ CrossRef ]
• Fernández-Manzano, Eva-Patricia, and María-Isabel González-Vasco. 2018. Analytic Surveillance: Big Data Business Models in the Time of Privacy Awareness. El Profesional de La Información (EPI) 27: 402–9. [ Google Scholar ] [ CrossRef ]
• Fernández-Manzano, Eva-Patricia, Elena Neira, and Judith Clares-Gavilán. 2016. Data management in audiovisual business: Netflix as a case study. Profesional de la Información 25: 568–77. [ Google Scholar ] [ CrossRef ]
• Fleischer, Rasmus. 2020. Universal Spotification? The Shifting Meanings of ‘Spotify’ as a Model for the Media Industries. Popular Communication 19: 14–25. [ Google Scholar ] [ CrossRef ]
• Frey, Mattias. 2021. Netflix Recommends: Algorithms, Film Choice, and the History of Taste . Berkeley: University California Press. [ Google Scholar ]
• Gaw, Fatima. 2022. Algorithmic Logics and the Construction of Cultural Taste of the Netflix Recommender System. Media, Culture & Society 44: 706–25. [ Google Scholar ]
• Gillespie, Tarleton. 2010. The Politics of ‘Platforms’. New Media & Society 12: 347–64. [ Google Scholar ] [ CrossRef ]
• Gillespie, Tarleton, and Nick Seaver. 2016. Critical Algorithm Studies: A Reading List. Social Media Collective , 15. Available online: https://socialmediacollective.org/reading-lists/critical-algorithm-studies/ (accessed on 1 September 2017).
• Goffey, Andrew. 2019. Automation Anxieties and Infrastructural Technologies. New Formations 98: 29–47. [ Google Scholar ] [ CrossRef ]
• Gomez-Uribe, Carlos A., and Neil Hunt. 2016. The Netflix Recommender System: Algorithms, Business Value, and Innovation. ACM Transactions on Management Information Systems (TMIS) 6: 1–19. [ Google Scholar ] [ CrossRef ]
• Gupta, Manjul, Carlos M. Parra, and Denis Dennehy. 2021. Questioning Racial and Gender Bias in AI-Based Recommendations: Do Espoused National Cultural Values Matter? Information Systems Frontiers 24: 1465–81. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Gusenbauer, Michael, and Neal R. Haddaway. 2020. Which Academic Search Systems Are Suitable for Systematic Reviews or Meta-Analyses? Evaluating Retrieval Qualities of Google Scholar, PubMed, and 26 Other Resources. Research Synthesis Methods 11: 181–217. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Guzman, Andrea L., and Seth C. Lewis. 2020. Artificial Intelligence and Communication: A Human–Machine Communication Research Agenda. New Media & Society 22: 70–86. [ Google Scholar ] [ CrossRef ]
• Hallinan, Blake, and Ted Striphas. 2016. Recommended for you: The Netflix Prize and the production of algorithmic culture. New Media & Society 18: 117–37. First published 2014. [ Google Scholar ] [ CrossRef ]
• Hepp, Andreas. 2020. Artificial Companions, Social Bots and Work Bots: Communicative Robots as Research Objects of Media and Communication Studies. Media, Culture & Society 42: 1410–26. [ Google Scholar ] [ CrossRef ]
• Heredia-Ruiz, Verónica, Ana Catalina Quirós-Ramírez, and Beatriz Eugenia Quiceno-Castañeda. 2021. Netflix: Catálogo de contenido y flujo televisivo en tiempos de big data. Revista de Comunicación 20: 117–36. [ Google Scholar ] [ CrossRef ]
• Higgins, Julian P. T., James Thomas, Jacqueline Chandler, Miranda Cumpston, Tianjing Li, Matthew J. Page, and Vivian A. Welch. 2019. Cochrane Handbook for Systematic Reviews of Interventions . Hoboken: John Wiley & Sons. [ Google Scholar ]
• Hildén, Jockum. 2021. The Public Service Approach to Recommender Systems: Filtering to Cultivate. Television & New Media 23: 777–96. [ Google Scholar ] [ CrossRef ]
• Johnson, Catherine. 2019. Online TV Data and Algorithm . London: Routledge. [ Google Scholar ]
• Kelly, John-Paul. 2019. Television by the Numbers: The Challenges of Audience Measurement in the Age of Big Data. Convergence 25: 113–32. [ Google Scholar ] [ CrossRef ]
• Kennedy, Jenny, and Indigo Holcombe-James. 2022. ‘It’s Almost Impossible to Buy a Dumb TV’ Experiences of Automated Decision-Making and Smart Technologies in Low-Income Homes. Telematics and Informatics 68: 101767. [ Google Scholar ] [ CrossRef ]
• Khoo, Olivia. 2022. Picturing Diversity: Netflix’s Inclusion Strategy and the Netflix Recommender Algorithm (NRA). Television & New Media 24: 281–97. [ Google Scholar ] [ CrossRef ]
• Kim, Taeyoung. 2022. Critical Interpretations of Global-Local Co-Productions in Subscription Video-on-Demand Platforms: A Case Study of Netflix’s YG Future Strategy Office. Television & New Media 23: 405–21. [ Google Scholar ]
• Kitchin, Rob. 2017. Thinking Critically about and Researching Algorithms. Information, Communication & Society 20: 14–29. [ Google Scholar ] [ CrossRef ]
• Klatt, Tyler. 2022. The Streaming Industry and the Great Disruption: How Winning a Golden Globe Helps Amazon Sell More Shoes. Media, Culture & Society 44: 1541–58. [ Google Scholar ]
• Kowalski, Robert. 1979. Algorithm = logic + control. Communications of the ACM 22: 424–36. [ Google Scholar ] [ CrossRef ]
• Kwon, Yeeun, Jaecheol Park, and Jai-Yeol Son. 2020. Accurately or Accidentally? Recommendation Agent and Search Experience in over-the-Top (OTT) Services. Internet Research 31: 562–86. [ Google Scholar ] [ CrossRef ]
• Lee, Francis, and Lotta Björklund Larsen. 2019. How Should We Theorize Algorithms? Five Ideal Types in Analyzing Algorithmic Normativities. Big Data & Society 6: 2053951719867349. [ Google Scholar ] [ CrossRef ]
• Lobato, Ramon. 2019. Netflix Nations: The Geography of Digital Distribution . New York: New York University Press. [ Google Scholar ]
• Lotz, Amanda D. 2007. The Television Will Be Revolutionized . New York and London: New York University Press. [ Google Scholar ]
• Lotz, Amanda D. 2018. We Now Disrupt this Broadcast. How Cable Transformed Television and the Internet Revolutionized It All . Cambridge: The MIT Press. [ Google Scholar ]
• Lotz, Amanda D. 2022. Netflix and Streaming Video: The Business of Subscriber-Funded Video on Demand . Cambridge: Polity Press. [ Google Scholar ]
• Lotz, Amanda D., Ramon Lobato, and Julian Thomas. 2018. Internet-distributed television research: A provocation. Media Industries 5: 35–47. [ Google Scholar ] [ CrossRef ]
• Lowrie, Ian. 2017. Algorithmic Rationality: Epistemology and Efficiency in the Data Sciences. Big Data & Society 4: 2053951717700925. [ Google Scholar ] [ CrossRef ]
• McKelvey, Fenwick, and Robert Hunt. 2019. Discoverability: Toward a Definition of Content Discovery through Platforms. Social Media+ Society 5: 2056305118819188. [ Google Scholar ] [ CrossRef ]
• McKenzie, Jordi, Paul Crosby, and Sunny Y. Shin. 2022. Netflix Chills and Revamps Its Viewing Metrics: Preliminary Analysis and Opportunities for Research. Poetics 96: 101738. [ Google Scholar ] [ CrossRef ]
• Meyerend, Daniel. 2023. The Algorithm Knows I’m Black: From Users to Subjects. Media, Culture & Society 45: 629–45. [ Google Scholar ]
• Michalis, Maria. 2022. Trends and Perspectives on Digital Platforms and Digital Television in Europe|Public Service Broadcasting in the Online Television Environment: The Case for PSB VoD Players and the Role of Policy Focusing on the BBC IPlayer. International Journal of Communication 16: 20. [ Google Scholar ]
• Modgil, Sachin, Rohit Kumar Singh, Shivam Gupta, and Denis Dennehy. 2021. A Confirmation Bias View on Social Media Induced Polarisation During COVID-19. Information Systems Frontiers 26: 417–41. [ Google Scholar ] [ CrossRef ]
• Morley, David. 2006. Unanswered questions in audience research. The Communication Review 9: 101–21. [ Google Scholar ] [ CrossRef ]
• Møller Hartley, Jannie, Mette Bengtsson, Anna Schjøtt Hansen, and Morten Fischer Sivertsen. 2021. Researching Publics in Datafied Societies: Insights from Four Approaches to the Concept of ‘Publics’ and a (Hybrid) Research Agenda. New Media & Society 25: 1668–86. [ Google Scholar ] [ CrossRef ]
• Nah, Seungahn, Jasmine McNealy, Jang Hyun Kim, and Jungseock Joo. 2020. Communicating Artificial Intelligence (AI): Theory, Research, and Practice. Communication Studies 71: 369–72. [ Google Scholar ] [ CrossRef ]
• Napoli, Philip M. 2003. Audience Economics: Media Institutions and the Audience Marketplace . New York: Columbia University Press. [ Google Scholar ]
• Napoli, Philip M. 2010. Revisiting ‘mass communication’ and the ‘work’ of the audience in the new media environment. Media, Culture & Society 32: 505–16. [ Google Scholar ] [ CrossRef ]
• Napoli, Philip M. 2011. Audience Evolution. New Technologies and the Transformation of Media Audiences . New York: Columbia University Press. [ Google Scholar ]
• Navar-Gill, Annemarie. 2020. The Golden Ratio of Algorithms to Artists? Streaming Services and the Platformization of Creativity in American Television Production. Social Media+ Society 6: 2056305120940701. [ Google Scholar ] [ CrossRef ]
• Ortega, Vicente Rodríguez. 2022. ‘We Pay to Buy Ourselves’: Netflix, Spectators & Streaming. Journal of Communication Inquiry 47: 126–44. [ Google Scholar ] [ CrossRef ]
• Pajkovic, Niko. 2022. Algorithms and Taste-Making: Exposing the Netflix Recommender System’s Operational Logics. Convergence 28: 214–35. [ Google Scholar ] [ CrossRef ]
• Pasquale, Frank. 2015. Black Box Society: The Secret Algorithms That Control Money and Information . Cambridge: Harvard University Press. [ Google Scholar ]
• Petticrew, Mark, and Helen Roberts. 2008. Systematic Reviews in the Social Sciences: A Practical Guide . Hoboken: John Wiley & Sons. [ Google Scholar ]
• Pérez-Escolar, Marta, and Fernando Canet. 2022. Research on Vulnerable People and Digital Inclusion: Toward a Consolidated Taxonomical Framework. Universal Access in the Information Society 22: 1059–72. [ Google Scholar ] [ CrossRef ] [ PubMed ]
• Pilipets, Elena. 2019. From Netflix Streaming to Netflix and Chill: The (Dis)Connected Body of Serial Binge-Viewer. Social Media + Society 5: 1–13. [ Google Scholar ] [ CrossRef ]
• Purssell, Edward, and Niall McCrae. 2020. How to Perform a Systematic Literature Review: A Guide for Healthcare Researchers, Practitioners and Students . Berlin/Heidelberg: Springer. [ Google Scholar ]
• Shapiro, Stephen. 2020. Algorithmic Television in the Age of Large-scale Customization. Television & New Media 21: 658–63. [ Google Scholar ] [ CrossRef ]
• Shin, Donghee, Azmat Rasul, and Anestis Fotiadis. 2021. Why Am I Seeing This? Deconstructing Algorithm Literacy through the Lens of Users. Internet Research 32: 1214–34. [ Google Scholar ] [ CrossRef ]
• Siles, Ignacio, Johan Espinoza-Rojas, Adrián Naranjo, and María Fernanda Tristán. 2019. The Mutual Domestication of Users and Algorithmic Recommendations on Netflix. Communication, Culture & Critique 12: 499–518. [ Google Scholar ] [ CrossRef ]
• Skoric, Marko M., Qinfeng Zhu, Debbie Goh, and Natalie Pang. 2016. Social Media and Citizen Engagement: A Meta-Analytic Review. New Media & Society 18: 1817–39. [ Google Scholar ] [ CrossRef ]
• Spicer, Stuart Gordon, Laura Louise Nicklin, Maria Uther, Joanne Lloyd, Helen Lloyd, and James Close. 2021. Loot Boxes, Problem Gambling and Problem Video Gaming: A Systematic Review and Meta-Synthesis. New Media & Society 24: 1001–22. [ Google Scholar ] [ CrossRef ]
• Steiner, Emil. 2017. Binge-watching in practice. The ritual, motives and feelings of streaming video viewers. In The Age of Netflix . Edited by Cory Barker and Myc Wiatrowski. Jefferson: McFarland & Company, Inc., pp. 141–61. [ Google Scholar ]
• Striphas, Ted. 2015. Algorithmic Culture. European Journal of Cultural Studies 18: 395–412. [ Google Scholar ] [ CrossRef ]
• Suri, Harsh. 2017. Meta-Analysis, Systematic Reviews and Research Syntheses. In Research Methods in Education . London: Routledge, pp. 427–39. [ Google Scholar ]
• Van Dijck, José. 2014. Datafication, Dataism and Dataveillance: Big Data between Scientific Paradigm and Ideology. Surveillance & Society 12: 197–208. [ Google Scholar ]
• van Es, Karin. 2022. Netflix & Big Data: The Strategic Ambivalence of an Entertainment Company. Television & New Media 24: 656–72. [ Google Scholar ] [ CrossRef ]
• Wayne, Michael L. 2021. Netflix Audience Data, Streaming Industry Discourse, and the Emerging Realities of ‘Popular’ Television. Media, Culture & Society 44: 193–209. [ Google Scholar ] [ CrossRef ]
• Williams, Joshua R. 2019. The Use of Online Social Networking Sites to Nurture and Cultivate Bonding Social Capital: A Systematic Review of the Literature from 1997 to 2018. New Media & Society 21: 2710–29. [ Google Scholar ] [ CrossRef ]
• Williams, Raymond. 2003. Television: Technology and Cultural Form . London: Routledge. First published 1974. [ Google Scholar ]
• Wolff, Michael. 2015. Television Is the New Television. The Unexpected Triumph of Old Media in the Digital Age . New York: Portfolio/Penguin. [ Google Scholar ]
• Zarouali, Brahim, Sophie C. Boerman, and Claes H. de Vreese. 2021. Is This Recommended by an Algorithm? The Development and Validation of the Algorithmic Media Content Awareness Scale (AMCA-Scale). Telematics and Informatics 62: 101607. [ Google Scholar ] [ CrossRef ]
• Zhang, Yin, and Louis Leung. 2015. A Review of Social Networking Service (SNS) Research in Communication Journals from 2006 to 2011. New Media & Society 17: 1007–24. [ Google Scholar ] [ CrossRef ]
• Zhao, Elaine Jing. 2021. Reconfiguring audience measurement in platform ecologies of video streaming: iQiyi’s pivot toward data-driven fandom and algorithmic metrics. International Journal of Communication 15: 21. [ Google Scholar ]
• Ziewitz, Malte. 2016. Governing Algorithms: Myth, Mess, and Methods. Science, Technology, & Human Values 41: 3–16. [ Google Scholar ] [ CrossRef ]

Click here to enlarge figure

Thematic Category	Study	Year	Key Concept	Method	Participants	OTT Service
Data management as a strategic practice in the media industry	( )	2016	Business intelligence	Cross-sectional (bibliographic reviews, analysis of the data published by the firm itself, information provided by its staff in discussion forums, and analysis of data provided by specialized press)	N/A	Netflix
	( )	2019	Data divide; big data	General literature review	N/A	Netflix
	( )	2019	Streaming lore	Cross-sectional (general literature review and media content analysis)	N/A	Netflix
	( )	2018	Privacy; security risks	General literature review	N/A	Netflix; Movistar+; HBO; Amazon Prime; Sky; Hulu; Disney; ESPN; Apple TV
	( )	2020	Data behavioralism; streaming prestige television; datalogic turn; algorithmic television; Netflixism; algorithmic turn	General literature review	N/A	Netflix
	( )	2020	Spotification	Media content analysis	N/A	N/A
	( )	2021	Data-driven fandom	General literature review	N/A	iQiyi
	( )	2021	Flow TV	Content analysis	N/A	Netflix
	( )	2022	Viewing metrics	Descriptive analysis of the three distinct datasets released by Netflix	N/A	Netflix
	( )	2022	Conglomeration, flywheel economics, disruption	Case study	N/A	Amazon Prime Video
	( )	2022	The myth of big data; data-driven organization; data-driven mindset; data–human divide	Discourse analysis	N/A	Netflix
The impact of recommender systems on viewers’ experience	( )	2019	Mutual domestication	Interviews; inductive analysis of practices and profiles on the platform	25 interviewees	Netflix
	( )	2019	Binge-watching; attachment	General literature review; network analysis	N/A	Netflix
	( )	2020	Perceived diagnosticity; perceived serendipity	Online survey	212 survey respondents	Netflix
	( )	2021	Algorithmic awareness	Scale development and validation	5 experts; 26 respondents	YouTube; Netflix
	( )	2021	Engagement	Survey	574 respondents	Netflix
	( )	2021	Algorithmic literacy; algorithmic divide	Online survey	775 survey respondents	Amazon; Netflix
	( )	2022	Personalization tactics, thumbnails as paratext	Pilot survey	6 participants	Netflix
	( )	2022	Planned differentiation	General literature review	N/A	Netflix
The impact of recommender systems in culture and cultural production	( )	2014	Algorithmic culture	General literature review	N/A	Netflix
	( )	2019	Discoverability	General literature review	N/A	Netflix; YouTube; HBO; Amazon Prime
	( )	2020	Platformization of creativity	Cross-sectional (fieldwork, interviews, and discourse analysis)	13 TV screenwriters	Netflix; Amazon Prime; Hulu
	( )	2020	Taste communities; global niches	Case study	N/A	Netflix
	( )	2022	Taste-making	Reverse engineering (taste personas)	N/A	Netflix
	( )	2022	Algorithmic logics	Cross-sectional (reverse engineering: analysis of 60 documents and 100 media reports; and phenomenological approach: coding of 990 tweets)	N/A	Netflix
	( )	2022	Global SVoD players	Cross-sectional (semi-structured, in-depth interviews)	15 bureaucrats, 10 industry insiders	Netflix
Algorithmic bias, inclusion, diversity, and digital divides	( )	2023	Algorithmic representations of race	General literature review	N/A	Netflix
	( )	2022	Digital divide; digital inclusion; digital exclusion	Cross-sectional (survey, and semi-structured interviews)	46 households (evaluation). 3 households as examples in the paper	Netflix, Disney Plus
	( )	2021	Exposure diversity; personalization; selective exposure; nudging	Thematic semi-structured interviews	10 interviewees	Public service media in Europe
	( )	2022	Inclusion strategy; algorithmic cultures	Case study	N/A	Netflix
Globalization, power, and the political economy of OTT services	( )	2019	Globalization; cosmopolitanism	Discourse analysis	N/A	Netflix
	( )	2020	Streaming network	General literature review; financial and other available information regarding four streaming services	N/A	Spotify; Apple Music; Netflix; Kindle
	( )	2022	Agonistic pluralism; conviviality	General literature review	N/A	N/A

The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Share and Cite

Vicente, P.N.; Burnay, C.D. Recommender Systems and Over-the-Top Services: A Systematic Review Study (2010–2022). Journal. Media 2024 , 5 , 1259-1278. https://doi.org/10.3390/journalmedia5030080

Vicente PN, Burnay CD. Recommender Systems and Over-the-Top Services: A Systematic Review Study (2010–2022). Journalism and Media . 2024; 5(3):1259-1278. https://doi.org/10.3390/journalmedia5030080

Vicente, Paulo Nuno, and Catarina Duff Burnay. 2024. "Recommender Systems and Over-the-Top Services: A Systematic Review Study (2010–2022)" Journalism and Media 5, no. 3: 1259-1278. https://doi.org/10.3390/journalmedia5030080

Article Metrics

Further information, mdpi initiatives, follow mdpi.

Subscribe to receive issue release notifications and newsletters from MDPI journals

IEEE Account

• Change Username/Password
• Update Address

Purchase Details

• Payment Options
• Order History
• View Purchased Documents

Profile Information

• Communications Preferences
• Profession and Education
• Technical Interests
• US & Canada: +1 800 678 4333
• Worldwide: +1 732 981 0060
• Contact & Support
• About IEEE Xplore
• Accessibility
• Terms of Use
• Nondiscrimination Policy
• Privacy & Opting Out of Cookies

A not-for-profit organization, IEEE is the world's largest technical professional organization dedicated to advancing technology for the benefit of humanity. © Copyright 2024 IEEE - All rights reserved. Use of this web site signifies your agreement to the terms and conditions.

Measuring Bias in Job Recommender Systems: Auditing the Algorithms

We audit the job recommender algorithms used by four Chinese job boards by creating fictitious applicant profiles that differ only in their gender. Jobs recommended uniquely to the male and female profiles in a pair differ modestly in their observed characteristics, with female jobs advertising lower wages, requesting less experience, and coming from smaller firms. Much larger differences are observed in these ads’ language, however, with women’s jobs containing 0.58 standard deviations more stereotypically female content than men’s. Using our experimental design, we can conclude that these gender gaps are generated primarily by content-based matching algorithms that use the worker’s declared gender as a direct input. Action-based processes like item-based collaborative filtering and recruiters’ reactions to workers’ resumes contribute little to these gaps.

We thank Sarah Bana, Kelly Bedard, Clément de Chaisemartin, Joanna Lahey, Heather Royer, Benjamin Villena Roldan, Kim Weeden, Catherine Weinberger, seminar participants at the University of Oregon and UC San Diego, and participants at the Chinese Labor Economists Society Conference (Shandong University) and LM2C2 Workshop on Gender and Labor Market Mismatches, Santiago, Chile for many helpful comments. This study was approved under UCSB IRB No.17-20-0451 and is in the AEA RCT Registry under AEARCTR-0006101. The authors declare that they have no relevant financial relationships or other conflicts of interest affecting this manuscript. The views expressed herein are those of the authors and do not necessarily reflect the views of the National Bureau of Economic Research.

MARC RIS BibTeΧ

Download Citation Data

• randomized controlled trials registry entry

Working Groups

More from nber.

In addition to working papers , the NBER disseminates affiliates’ latest findings through a range of free periodicals — the NBER Reporter , the NBER Digest , the Bulletin on Retirement and Disability , the Bulletin on Health , and the Bulletin on Entrepreneurship  — as well as online conference reports , video lectures , and interviews .