word puzzle research paper

Thank you for visiting nature.com. You are using a browser version with limited support for CSS. To obtain the best experience, we recommend you use a more up to date browser (or turn off compatibility mode in Internet Explorer). In the meantime, to ensure continued support, we are displaying the site without styles and JavaScript.

• View all journals
• Explore content
• About the journal
• Publish with us
• Sign up for alerts
• Open access
• Published: 02 May 2022

MODEL, GUESS, CHECK: Wordle as a primer on active learning for materials research

• Keith A. Brown   ORCID: orcid.org/0000-0002-2379-2018 1  

npj Computational Materials volume  8 , Article number:  97 ( 2022 ) Cite this article

4354 Accesses

8 Citations

7 Altmetric

Metrics details

• Techniques and instrumentation
• Theory and computation

Research and games both require the participant to make a series of choices. Active learning is a process borrowed from machine learning for algorithmically making choices that has become increasingly used to accelerate materials research. While this process may seem opaque to researchers outside the field of machine learning, examining active learning in games provides an accessible way to showcase the process and its virtues. Here, we examine active learning through the lens of the game Wordle to both explain the active learning process and describe the types of research questions that arise when using active learning for materials research.

A challenge common across research, whether computational or experimental, is deciding which experiment to perform next. While many of these choices are made ad hoc by a researcher in front of an instrument, there is a community, with roots tracing back centuries, that studies how to optimally select experiments 1 . Widely used strategies include one factor at a time (OFAT) optimization and selecting experiments evenly across the available parameter space (e.g., grid-based searching). In contrast with strategies that select all experiments before performing an experimental campaign, active learning is a branch of machine learning in which experiments are selected sequentially using knowledge gained from all prior experiments 2 , 3 . Encouragingly, recent studies have reported that active learning outperforms both design of experiments that define a set of experiments at the outset and experiments guided by human intuition 4 , 5 . Indeed, examples of active learning in materials research have blossomed in recent years 6 , 7 , 8 , 9 , 10 , 11 . One area that has especially shown a spotlight on this trend is autonomous experimentation in which a robotic system is used to carry out experiments that are chosen by active learning 12 . Such systems have emerged in a broad range of fields including mechanics 13 , 14 , biology 15 , 16 , chemistry 17 , 18 , nanotechnology 19 , and microscopy 20 , 21 . In light of these advances in autonomous experimentation, and the demonstrated advantage of active learning, it is increasingly clear that there are tremendous opportunities for applying concepts of active learning in a wide range of research fields.

Despite the benefits of adopting active learning in research, there are still hurdles associated with learning how to implement active learning and communicating the advantages to researchers far removed from the fields of machine learning or data-driven science. Interestingly, the introduction and rapid popularization of the word game Wordle produced by Josh Wardle provides a fascinating platform for overcoming both of these challenges. In Wordle, the player has six guesses to identify a specific five-letter word. After each guess, feedback is provided about whether each letter is present in the target word and whether the letter is in the correct location. Part of the popularity of this game stems from the fact that it can only be played once a day, which further raises the stakes of each guess. At a glance, this game shares some extremely salient connections to experimental selection during a research campaign. First, a Wordle player can guess any five-letter word, meaning that there is a large, but finite, parameter space that is known ahead of time. This is often true in a research campaign where the knobs that can be turned are known, even if their importance has yet to be determined. Second, the budget of available experiments is limited—to six by the programmer in Wordle—and by the natural constraints of time and other resources in research. Finally, the goal of Wordle, to identify the target word, mirrors the goal of finding a maximum or minimum property in some parameter space, which is a common task in many research studies. Despite these similarities, the goals of Wordle and research often have meaningful differences. For example, a researcher may not know when they have found the best experiment, the goal of a research campaign is often to learn the behavior of some property rather than simply find its extrema, and there can be more than one best experiment in a parameter space. Those caveats aside, the task of optimization is extremely common and an obvious first step in any more complex research goal.

Given the similarities between Wordle and research campaigns, it is our hope that one can gain insight into active learning and motivation for adopting it more generally by studying active learning in Wordle. Taking inspiration from a Bayesian optimization formalization of active learning, we can lay out an iterative cycle for selecting and interpreting guesses (Fig. 1A ). The first task is to define a surrogate model that encompasses our knowledge about the system. As part of this, we must define the parameter space, which for Wordle, amounts to identifying all valid five-letter words, of which we identified 12,478. To illustrate why active learning is needed to play Wordle, given a parameter space this large, selecting guesses uniformly at random from the available words means that the player will win only 0.05% of the time. In a materials research campaign, defining the parameter space is akin to identifying all available materials and the valid ranges of experimental or computational parameters that can be tuned. Once we have defined the parameter space, we need to build a surrogate model of our belief about the space. In Wordle, this can be the belief of whether a given word is possibly correct, which is initially uniform for all available words. This surrogate model is implemented as a look up table in which each word is assigned a probability that it is the correct word. If a given word is ruled out then its probability is set to 0 and the rest of the space is renormalized. More complex and versatile surrogate models, such as Gaussian process regressions, can be constructed that leverage the idea that points close to one another in parameter space should behave similarly, but a simple look up table is sufficient for illustrating the process. In either case, this surrogate model is updated as experiments are performed, which in Wordle amounts to ruling out words that are inconsistent with the responses to prior guesses. To show the value of iteratively incorporating this knowledge, if a player randomly selects words from this ever narrowing field of possibilities, they will win Wordle 85% of the time. This striking shift in outcomes illustrates the value of selecting each experiment using all available knowledge.

A Diagram showing the anatomy of active learning in a game of Wordle. A surrogate model describes the present belief about the system. During the first turn ( N  = 0), all words are assumed to be candidates for the winning word. Next, a decision-making policy is employed to quantify the relative value of each word. Subsequently, the highest value word is guessed and the response is used to update the surrogate model. This process continues until the correct word is found or the player loses. B Ranked order evaluation of three decision-making policies for all words. Decision policies include ‘Eliminate’ in which each word’s value is estimated by the number of words that could be eliminated if it was guessed, ‘Knowledge’ in which the value of each word is determined by the number of unique outcomes that could arise from guessing the word, and ‘Letters’ in which the value of each word is based on the abundance of its letters in the catalog of words. C Probability of different outcomes from playing Wordle guided by different strategies. Color indicates the number of guesses required to find the correct answer with red indicating that seven or more guesses were required. Strategies are either ‘Hard’ in that they require that words are selected from those that have not been ruled out or ‘Normal’ in that all words can be selected. ‘Random’ indicates that words were selected uniformly at random. The code used to produce these results can be found at https://github.com/kabrownlab/wordle .

Randomly guessing from the words that remain in contention takes advantage of the information from the results of previous guesses, but not the information embodied by the parameter space itself. A major goal in active learning is determining the expected value of a given guess. In a materials research campaign, this could be related to a materials property, such as its Seebeck coefficient or fracture toughness. In Wordle, the value of a given word is related to how much it helps identify the correct word. Given this, there are many different ways to value a potential guess (Fig. 1B ). Three examples include (1) assigning a score to each word based on how common the letters in the word are, (2) prioritizing words that allow the player to eliminate the most words, or (3) targeting words that lead to the most possible outcomes and therefore the most unique information. Each choice represents a decision-making policy that assigns a value to any given point in parameter space based on the current state of the surrogate model. Here, we find that selecting words based on how many possible words they eliminate or how much information they provide leads to victory >90% of the time, showing a ~5% improvement over randomly selecting from the available words. However, choosing based on the commonality of letters actually performs worse than randomly selecting from potential winners, highlighting the importance of choosing a decision-making policy wisely. As shown by this example, the benefits and risks associated with choosing different decision-making policies is both highly impactful on the pace of learning and a fascinating lens through which to view materials research.

Part of the value of active learning is that it opens the door for the algorithm to make choices based on relationships inside the data that might be difficult for human users to intuit. Conceptually, this can amount to selecting experiments that themselves are not likely to lead to high performance but improve chances for future success. To apply this concept in Wordle, consider choosing between two words as potential guesses: word 1 has some probability of being a winning word but would provide little actionable information if it is not the target word. In contrast, word 2 has already been ruled out from being a winning word, but the response from this word would on average greatly increase the player’s chance of success by providing actionable information such as ruling out other words. This latter approach, namely selecting the words with the most possible outcomes even among those that have been ruled out produces success a remarkable 99.7% of the time. This interesting result can be conceptualized as coming from the dichotomy between exploration and exploitation in acknowledging that if your budget is known, which of course in Wordle it is, it is beneficial to spend earlier guesses exploring parameter space before focusing on regions in which success is expected. In active learning, there are decision-making policies that naturally balance exploration and exploitation such as the expected improvement policy that selects points in parameter space that are judged to be most likely to increase, in the case of maximization, the current value of the maximum.

While analyzing the algorithmic process of sequentially selecting guesses for Wordle provides insight into the process for materials development, it also raises a key shortcoming of purely algorithmic active learning. The discussed algorithms assume that all words initially have an equal chance of being correct, which discounts the bias against esoteric words in the target word selection by the game’s creator. There are two ways one could envision trying to take advantage of this information. First, one could imagine including information about word popularity as prior knowledge in the active learning loop. This could amount to, for example, weighting the initial probability of each word in proportion to how often it is used in literature. It is also worth emphasizing that prior knowledge has already been introduced by only considering combinations of five-letter words that are English words. Second, this could also be dynamically addressed using human-in-the-loop (HITL) active learning in which the human-machine partnership is leveraged to further accelerate the learning process. Generally, HITL entails finding a productive way to combine the best attributes of each member of the partnership. This has been productively used to outperform algorithms or humans alone in fields including radiology 22 and robotics 23 . In the present example, HITL could entail using the algorithm to identify a short list of high value words and allow the human to select from these words as a way of considering the relative popularity of each word. Advances in this area are particularly interesting in materials research where the insight of researchers may be difficult to quantify but could be productively employed through such a partnership.

Formalizing the selection of choices, whether in games or materials research, forces one to define and consider the goals and important information present in a system. Because much materials research still relies on relatively simple heuristics for experimental selection, it is our hope that this comment, and the broader efforts to introduce active learning into materials research that it reflects, will spark the curiosity of new researchers and engage them in the process of looking more deeply at how experiments are selected. Such interest can be complemented by easy-to-implement active learning packages in a variety of programming languages and high quality tutorial articles 24 , 25 . Along these lines, the code used to generate the results presented in this work are posted at https://github.com/kabrownlab/wordle . Perhaps most importantly, the incorporation of active learning elevates the conversation and thoughts in the research enterprise. For example, rather than thinking about which word to select in Wordle, instead the player thinks about what defines a word’s value. This level of discourse centered around selecting and refining decision-making policies and surrogate models is an exciting prospect for the materials community.

Data availability

The code used to produce the data in this paper can be found at https://github.com/kabrownlab/wordle .

Hicks, C. R. Fundamental concepts in the design of experiments. (Holt, New York, 1964).

Cohn, D. A., Ghahramani, Z. & Jordan, M. I. Active learning with statistical models. J. Artif. Intell. Res. 4 , 129–145 (1996).

Article   Google Scholar  

Pereira-Santos, D., Prudêncio, R. B. C. & de Carvalho, A. C. Empirical investigation of active learning strategies. Neurocomputing 326 , 15–27 (2019).

Liang, Q. et al. Benchmarking the performance of Bayesian optimization across multiple experimental materials science domains. npj Comput. Mater. 7 , 188 (2021).

Chen, D., Skouras, M., Zhu, B. & Matusik, W. Computational discovery of extremal microstructure families. Sci. Adv. 4 , eaao7005 (2018).

Lookman, T., Balachandran, P. V., Xue, D. & Yuan, R. Active learning in materials science with emphasis on adaptive sampling using uncertainties for targeted design. npj Comput. Mater. 5 , 1–17 (2019).

Bassman, L. et al. Active learning for accelerated design of layered materials. npj Comput. Mater. 4 , 74 (2018).

Tian, Y. et al. Role of uncertainty estimation in accelerating materials development via active learning. J. Appl. Phys. 128 , 014103 (2020).

Article   CAS   Google Scholar  

Tran, K. & Ulissi, Z. W. Active learning across intermetallics to guide discovery of electrocatalysts for CO 2 reduction and H 2 evolution. Nat. Catal. 1 , 696 (2018).

Kusne, A. G. et al. On-the-fly closed-loop materials discovery via Bayesian active learning. Nat. Commun. 11 , 1–11 (2020).

Sun, S. et al. Accelerated development of perovskite-inspired materials via high-throughput synthesis and machine-learning diagnosis. Joule 3 , 1437–1451 (2019).

Stach, E. et al. Autonomous experimentation systems for materials development: a community perspective. Matter 4 , 2702–2726 (2021).

Gongora, A. E. et al. Using simulation to accelerate autonomous experimentation: a case study using mechanics. iScience 24 , 102262 (2021).

Gongora, A. E. et al. A Bayesian experimental autonomous researcher for mechanical design. Sci. Adv. 6 , eaaz1708 (2020).

King, R. D. et al. Functional genomic hypothesis generation and experimentation by a robot scientist. Nature 427 , 247 (2004).

King, R. D. et al. The automation of science. Science 324 , 85–89 (2009).

Coley, C. W. et al. A robotic platform for flow synthesis of organic compounds informed by AI planning. Science 365 , eaax1566 (2019).

Häse, F., Roch, L. M. & Aspuru-Guzik, A. Next-generation experimentation with self-driving laboratories. Trends Chem. 1 , 282–291 (2019).

Nikolaev, P. et al. Autonomy in materials research : a case study in carbon nanotube growth. npj Comput. Mater. 2 , 16031 (2016).

Ziatdinov, M.A. et al. Hypothesis learning in automated experiment: application to combinatorial materials libraries. Adv. Mater. Early View, 2201345 (2022).

Roccapriore, K. M., Dyck, O., Oxley, M. P., Ziatdinov, M. & Kalinin, S. V. Automated experiment in 4D-STEM: exploring emergent physics and structural behaviors. Preprint at https://doi.org/10.48550/arXiv.2112.04479 (2021).

Patel, B. N. et al. Human–machine partnership with artificial intelligence for chest radiograph diagnosis. npj Digit. Med. 2 , 1–10 (2019).

CAS   Google Scholar  

Zhang, J. et al. Human-in-the-loop optimization of exoskeleton assistance during walking. Science 356 , 1280–1284 (2017).

Frazier, P. I. A tutorial on Bayesian optimization. Preprint at https://doi.org/10.48550/arXiv.1807.02811 (2018).

Astudillo, R. & Frazier, P. I. Thinking inside the box: a tutorial on grey-box Bayesian optimization. In 2021 Winter Simulation Conference (WSC) 1-15 (2021) Preprint at https://doi.org/10.48550/arXiv.2201.00272 (2022).

Download references

Acknowledgements

We acknowledge fruitful discussions with Sam Ribnick, Alan Gardner, and Kelsey Snapp.

Author information

Authors and affiliations.

Department of Mechanical Engineering, Physics Department, and Division of Materials Science & Engineering, Boston University, Boston, MA, USA

Keith A. Brown

You can also search for this author in PubMed   Google Scholar

Contributions

K.A.B. conceptualized this work, performed the simulations, and wrote the manuscript.

Corresponding author

Correspondence to Keith A. Brown .

Ethics declarations

Competing interests.

The authors declare no competing interests.

Additional information

Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Rights and permissions

Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons license, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons license and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this license, visit http://creativecommons.org/licenses/by/4.0/ .

Reprints and permissions

About this article

Cite this article.

Brown, K.A. MODEL, GUESS, CHECK: Wordle as a primer on active learning for materials research. npj Comput Mater 8 , 97 (2022). https://doi.org/10.1038/s41524-022-00787-7

Download citation

Received : 19 February 2022

Accepted : 13 April 2022

Published : 02 May 2022

DOI : https://doi.org/10.1038/s41524-022-00787-7

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

This article is cited by

How games can make behavioural science better.

• Samuel A. Mehr

Nature (2023)

Quick links

• Explore articles by subject
• Guide to authors
• Editorial policies

Sign up for the Nature Briefing newsletter — what matters in science, free to your inbox daily.

Research Paper Word Search

Description.

credible source Microsoft Word Body Paragraph works cited Concession Conclusion Cover Page Quotation Paragraph notecard citation Website Thesis Author Format MLA

Informative/Expository Writing

Research Word Search

Word Search

Works Cited and MLA

MLA Writing Vocabulary

Persuasive Writing

Frequently Asked Questions

What is a word search.

A word search is a puzzle where there are rows of letters placed in the shape of a square, and there are words written forwards, backwards, horizontal, vertical or diagonal. There will be a list of words for the player to look for and the goal of the player is to find those words hidden in the word search puzzle, and highlight them.

How do I choose the words to use in my word search?

Once you’ve picked a theme, choose words that have a variety of different lengths, difficulty levels and letters. You don’t need to worry about trying to fit the words together with each other because WordMint will do that for you!

How are word searches used in the classroom?

Word search games are an excellent tool for teachers, and an excellent resource for students. They help to encourage wider vocabulary, as well as testing cognitive abilities and pattern-finding skills.

Because the word search templates are completely custom, you can create suitable word searches for children in kindergarten, all the way up to college students.

Who is a word search suitable for?

One of the common word search faq’s is whether there is an age limit or what age kids can start doing word searches. The fantastic thing about word search exercises is, they are completely flexible for whatever age or reading level you need.

Word searches can use any word you like, big or small, so there are literally countless combinations that you can create for templates. It is easy to customise the template to the age or learning level of your students.

How do I create a word search template?

For the easiest word search templates, WordMint is the way to go!

Pre-made templates

For a quick an easy pre-made template, simply search through WordMint’s existing 500,000+ templates . With so many to choose from, you’re bound to find the right one for you!

Create your own from scratch

• Log in to your account (it’s free to join!)
• Head to ‘My Puzzles’
• Click ‘Create New Puzzle’ and select ‘Word Search’
• Select your layout, enter your title and your chosen words
• That’s it! The template builder will create your word search template for you and you can save it to your account, export as a Word document or PDF and print!

How can I print my word search template?

All of our templates can be exported into Microsoft Word to easily print, or you can save your work as a PDF to print for the entire class. Your puzzles get saved into your account for easy access and printing in the future, so you don’t need to worry about saving them at work or at home!

Can I create a word search in other languages?

Word searches are a fantastic resource for students learning a foreign language as it tests their reading comprehension skills in a fun, engaging way.

We have full support for word search templates in Spanish, French and Japanese with diacritics including over 100,000 images.

Academia.edu no longer supports Internet Explorer.

To browse Academia.edu and the wider internet faster and more securely, please take a few seconds to  upgrade your browser .

Enter the email address you signed up with and we'll email you a reset link.

• We're Hiring!
• Help Center

Word Search Game as a Valuable Technique in Enhancing Students’ English Vocabulary Mastery

The research purpose is to find out whether or not word search game is effective to improve the students’ vocabulary mastery, especially for tenth grade students of State Vocational High School (SMK) Negeri 1 Kebumen in the academic year of 2016/2017.The subject of the study was the students of SMK Negeri 1 Kebumen. The population of this research is the tenth grade students of SMK Negeri 1 Kebumen in the academic year of 2016/2017. As sample, the researcher took two classes, class AK1 consisting of 32 students of control group and 32 students of the experimental group from class AK2.The result shows that there is an enhancement of students’ English vocabulary mastery after using word search game at the students of that school. It could be seen from the results of t-value is 10.156. Referring to 0.05 significance level, the t-table value is 2.000 with the degree of freedom is 64, the calculation shows that t-value is higher than t-table (10.156>2.000). In addition, , the enhancem...

Related Papers

Dona Maulana

To have excellent English proficiency, people need to mastery English vocabularies. Vocabulary is like bricks to construct an English skill house. However, learning this component is not easy, especially for students who do not use English as their daily conversation. Thus, the researcher has aim to find the effectiveness of game called word search puzzle towards students' vocabulary mastery. The population of this study are student's grade 8 th at SMPN 1 Karangrejo with students at VIII C and VIII D as the sample. Some data had been collected through quasi experimental method. The researcher gained the conclusion that the word search puzzle is effective to increase students' vocabulary mastery. It can be seen from statistical computation said that Sig˃ α (Sig = 0.15 and α= 0.05). In addition, the score of the experimental class is higher than in control class.

Devi Simatupang

The purpose of the research is to find out whether there is any significant effect of the word search game compared to the board game towards the students’ vocabulary mastery at the fourth grade SD Swasta Karya Tani Kandis. The design used in this research is experimental research. The location for this research was at SD Swasta Karya Tani Kandis. The vocabulary test was used to find out the students’ ability in vocabulary. The result of this research shown that the data obtained from the students’ pre-test to find the homogeneity to see the equal ability from both classes. and post-test scores of experimental classes 1 and 2 to know the different scores of two experimental classes. The test was to answer the questions based on the vocabulary test. Before treatment, the researcher gave the pre-test to IVA (experiment class 1) and IVB (experiment class 2). The researcher taught within 6 (Six) meetings including pre-test and post-test. Based on the data analysis it can be seen that Si...

Journal of English Language Teaching and Learning

Hafidz Yudha R.

intan zulfira pane

The research was intended to know the effectiveness of Word FindPuzzle in the students’ vocabulary mastery which was observed andanalyzed from students of tenth grade at SMA Baitul Aziz BandarKlippa. The method used was quatitative research by using theexperimental research design. The population of this study wastenth grade students of SMA Baitul Aziz Bandar Klippa. In takingthe sample of this research, the researchers took some students ineach classes as the sample used and divided into 2 classes,experimental and control. There were 34 students in theexperimental and 26 students in control. The researchers taught byusing the Word Find Puzzle in the experimental class and taught byusing lecturing method in control class.The researchers used pretest,treatment, and post-test. In control class the score increase59,23 between pre-test and post-test, but in experimental class thescore increase about 68,85. By calculated by used SPSS V24, Theresearcherss concluded that there was a signif...

Kairos English Language Teaching Journal

Eric Pandiangan

Vocabulary is one of components in the language learning because it is the basic for the people to speak the language. Therefore this study was conducted to improve students’ vocabulary mastery by applying Word Search Puzzle. This research design is classroom action research. The subjects of this research were the seventh grade of SMP Trisakti 1 Medan. The result showed in cycle I, there were 5 students (15,63%) whose vocabulary scores had achieved that the target of indicator with mean score is 52,89 (KKM 70) and there were 27 students (78,13%) whose scores were still far from the target. As there were 27 students whose scores below the target, so the indicator of the research had not been fulfilled yet in the first cycle. Nevertheless, in order to fulfill the indicator which is 75% student must be able to get score 70 or more, so the next cycle was conducted. From data in cycle II, there were only 5 students (22%) whose scores were still far from target. In other hand, total s...

Proceedings of the Eighth International Conference on Languages and Arts (ICLA-2019)

Havid Ardi UNP

International Journal of Multidisciplinary Approach and Studies

Sydney Jay Villarin

This paper is a study on the grade 8 students' vocabulary achievement using word games. The study was conducted at Kiwalan National High School and aimed to determine the effectiveness of word games as a learning method in improving students' vocabulary achievement. The data was collected through questionnaires and was analyzed through percentage, mean, t-test, and paired t-test. The results showed that there was no significant difference between the students' pretest scores in vocabulary when grouped to Word Games method and Traditional method with a p-value of 0.835, but there was a significant difference on the students' posttest scores in vocabulary when taught to Word Games method and Traditional method with a p-value of 0.0015. Additionally, paired t-test revealed that there was a significant difference between the students' pretest and posttest scores in vocabulary when exposed to Word Games method with a p-value of 0.00016, while the p-value of 0.257 achieved by the group of students taught to Traditional method was not statistically significant. Based on these findings, it is recommended that word games be implemented in vocabulary classes as an effective method for improving students' vocabulary achievement.

Hanswaty Noho

The aim of this research is to investigate whether the games techniques can improve the students‟ vocabulary in the English learning. This research was conducted the second class of SMK Almamater Telaga by taking 30 students as a sample. The method used is quasi experimental one with research design one group of pre-test and posttest design. The data were collected from a test, notably vocabulary multiple choice test consists of 35 items. Each item has been analyzed by using item analysis in order to take its validity and reliability. The results indicate that the students‟ vocabulary can be expanded by using the Games technique. It is proved by the result of statistical analysis in which the criteria used to verify the hypothesis of this research was received Ho if t count< t test. In this cases, t count = -2.36 and t list = 2.00.By looking the result of the data analysis, it can be concluded that the students‟ vocabulary increase after treatment. It means that the vocabulary te...

vera fitri khoirunnisa

This descriptive study aims to describe the implementation of using match words game in teaching English vocabulary for the students at SMP Penda Tawangmangu. The problem in this study focuses on the games used, the vocabulary teaching process, then the problems faced by teachers and students, and problem solving by the teacher. Games are tools that make it easy for children to learn new things. The students very enjoyed learning while playing. The author takes the students from seventh grade (VII A) and eighth grade (VIII D) students as research subjects. Data is taken from classroom observations and interviews with students. The researcher acts as a teacher in implementing match words games. Based on the research findings, it can be concluded that the problem faced by teacher in teaching vocabulary is that the class is not conducive and the problem faced by students is the lack of vocabulary they know, making students difficult when carrying out the game. Problem solving applied b...

Loading Preview

Sorry, preview is currently unavailable. You can download the paper by clicking the button above.

RELATED PAPERS

yanna queencer

Islamic University of Malang

Piping Rahadianto

English Education:Journal of English Teaching and Research

Jaftiyatur Rohaniyah

Hanane Brahimi

Journal of Psychology and Instructions

Amelia Kartadi

erikson saragih

CAHAYA PENDIDIKAN

Sri Sugiharti

Prosiding Seminas Competitive Advantage

CHOLICHUL HADI

Fitri Handayani Suleman

JOEY: Journal of English Ibrahimy

Dwi Wijaksono

jerni gultom

Fahmi Abbad

Ahmad Fahmi

Edumaspul - Jurnal Pendidikan

Hasan Hasan

tressya susanti

Elsa Wijiyanti

Jurnal Serunai Bahasa Inggris

Aminatuz Zuhriyah

ETERNAL (English, Teaching, Learning, and Research Journal)

novalia tanasy

Harry Dermawan

Journal of World Englishes and Educational Practices (JWEEP) ISSN: 2707-7586

Amr Mahmoud Abdeldaim Mohamed

International Journal of Scientific & Technology Research

Ranmeet Kaur

kennita septriana

Maryati Salmiah

Leroy Holman Siahaan

•   We're Hiring!
•   Help Center
• Find new research papers in:
• Health Sciences
• Earth Sciences
• Cognitive Science
• Mathematics
• Computer Science
• Academia ©2024

what is the point? teaching graduate students how to construct political science research puzzles

• Teaching and Learning
• Open access
• Published: 31 July 2017
• Volume 17 , pages 634–648, ( 2018 )

Cite this article

You have full access to this open access article

• Karl Gustafsson 1 &
• Linus Hagström 2  

97k Accesses

9 Citations

290 Altmetric

Explore all metrics

One of the key challenges graduate students face is how to come up with a good rationale for their theses. Unfortunately, the methods literature in and beyond political science does not provide much advice on this important issue. While focusing on how to conduct research, this literature has largely neglected the question of why a study should be undertaken. The limited discussions that can be found suggest that new research is justified if it (1) fills a ‘gap’; (2) addresses an important real-world problem; and/or (3) is methodologically rigorous. This article discusses the limitations of these rationales. Then, it proposes that research puzzles are more useful for clarifying the nature and importance of a contribution to existing research, and hence a better way of justifying new research. The article also explores and clarifies what research puzzles are, and begins to devise a method for constructing them out of the vague ideas and questions that often trigger a research process.

Similar content being viewed by others

How the Research Design Can Be a Structure, a Process, and a Product for Learning Political Science

The savvy consumer of political science research, building qualitative methods skills through research design.

Avoid common mistakes on your manuscript.

Introduction

As political scientists, we spend much time writing: book manuscripts, conference papers and article drafts. Some texts are published in high-ranking peer-reviewed journals or by prestigious university presses, while others end up as working papers or even unpublished in desk drawers. Our peers clearly view some texts as meaningful, while others provoke their irritation or scorn.

As scholars, we not only write, but also read academic texts written by others: student theses, conference papers, manuscripts for peer review and published texts. Some of these works immediately capture our interest, and we keep reading almost effortlessly. Others can be ‘boring, dogmatic, formalistic, and unsurprising’ (Löwenheim, 2010 : 1043). Why do we find some texts meaningful, while others come across as pointless, even though they may all appear to contain necessary components – aim, previous research, theory, method, data, analysis – and their authors have invested much time and effort in them?

The watershed is arguably whether a study has a clear rationale or justification. How to come up with one is a key challenge facing graduate students. Yet, the methods literature in and beyond political science provides them with scant advice on how to do so. While the social science methods literature is sizeable and covers many narrow issues, it has prioritized the question of how to conduct research, largely neglecting why a study should be undertaken. A rationale, it suggests, is provided by (1) filling a ‘gap’ in previous research; (2) addressing an important real-world problem; and/or (3) exercising methodological rigour.

While all three are necessary research components, we argue that they provide weak justifications for new research. Instead, we suggest that a well-constructed research puzzle is more useful as it clearly situates new research vis-à-vis the state of the art by explaining both how it contributes and why the contribution is necessary. Constructing research puzzles is not the only method for justifying new research, but we contend it is among the best ones. Neither the term research puzzle, nor the practice of formulating them, is new, however. Some works mention research puzzles but fail to explain their construction and function (Van Evera, 1997 : 97–103; George and Bennett, 2004 : 74–79; Flick, 2007 : 22–23; Bloomberg and Volpe, 2008 : 5–7; Della Porta and Keating, 2008 : 266–267; Blaikie, 2010 : 45–50; Flick, 2014 : 12–13). This article contributes by fully explaining the advantage of research puzzles and by demonstrating how to construct them.

Our argument has educational value; it demonstrates how graduate students can conceive a clear rationale for their theses or dissertations. It also helps supervisors provide students with good advice. In fact, even established scholars can benefit from thinking more systematically about research puzzles, as it may help them communicate their contributions more clearly than is often the case in conference papers, manuscripts submitted for review and sometimes even in published work. Finally, we believe a more diffuse practice of formulating research puzzles could help facilitate debate and possibly even understanding across the boundaries that currently divide political science. Scholars might use different theories and methods for understanding or explaining politics, but we suggest they can construct similar research puzzles. Puzzles are a succinct way of delineating the nature of, and the need for, a specific contribution, regardless of whether qualitative or quantitative methods are used; whether ‘why questions’ or ‘how possible questions’ are asked; and, indeed, whether the research is ‘problem’, ‘theory’ or ‘method’ driven, in the language of Ian Shapiro ( 2002 ).

This article’s aim is threefold: to explain why it is important to think hard about research puzzles; to clarify what a research puzzle is; and to start devising a method for constructing puzzles out of the vague ideas and questions that research processes often depart from. In the remainder, we first critically evaluate the three most widespread ideas on how to motivate research, that is, research gaps, real-world problems and methodological rigour. We then outline what a puzzle is, explain why puzzles are useful and indeed necessary, and present a strategy for constructing puzzles premised on problematization and abduction. Finally, we illustrate how our advice can be used to help students formulate research puzzles.

How to justify new research

Critically assessing the three main suggestions identified above for justifying new research, this section also begins to delineate our alternative approach.

In their immensely influential book on research design, Gary King, Robert O. Keohane and Sidney Verba argue that making a ‘[c]ontribution to an identifiable scholarly literature’ is one of two important criteria for choosing a topic ( 1994 : 15; see also Collier et al , 2004 : 37–38). Such contributions, they suggest, can be made in various ways, one of which is to ‘[a]rgue that an important topic has been overlooked in the literature and then proceed to contribute a systematic study to the area’ (King et al , 1994 : 17). We agree about the need to clarify the contribution to existing research, and view close familiarity with the previous literature as crucial. However, we dispute the idea that a gap – understood as a topic that has not previously been analysed – sufficiently motivates new research.

Other well-cited works on research design and methods also refer to gaps. Alexander L. George and Andrew Bennett write: ‘The problem should be embedded in a well-informed assessment that identifies gaps in the current state of knowledge, acknowledges contradictory theories, and notes inadequacies in the evidence for existing theories’ ( 2004 : 74). While gaps are only part of what motivates new research for George and Bennett, they are nonetheless central. When addressing ‘topic selection’, Stephen Van Evera suggests: ‘After each graduate school class, write an audit memo about the subject area of the course asking what was missing . What important questions went unasked ?’ ( 1997 : 98, emphasis added). Furthermore, he advises that the introductory chapter of a dissertation should highlight ‘the holes in the current literature’ and the questions that ‘have not been explored’ ( 1997 : 101, emphasis in the original).

Some may object that few established scholars consider gap-filling sufficient motivation for new research, but the idea remains influential. We often find that students justify their theses by arguing that a topic has been neglected and, indeed, that their supervisors and/or textbooks have taught them that such a rationale is both necessary and sufficient. However, previous neglect does not automatically make the study of a topic necessary. On the contrary, such inattention could indicate that it lacks implications for previous research.

The main problem with the preoccupation with gaps is arguably that it focuses on mere inclusion/exclusion, rather than on why a certain gap is problematic and should be filled. Hence, gap filling under-problematizes the relationship to previous research. By failing to challenge assumptions in the existing literature, it risks reinforcing dominant theories (Alvesson and Sandberg, 2013 ). While we consider a gap to be insufficient reason for undertaking a study (Schmitter, 2008 : 267; Alvesson and Sandberg, 2013 : chapter 4), it is potentially a useful starting point, from which to proceed towards a research puzzle.

A similar argument suggests that new research can be motivated by studying what has yet to be sufficiently explored (Booth et al , 2008 : chapters 3 and 4). Such a stance implies the possibility of achieving complete or sufficient knowledge. However, if we consider all scientific knowledge provisional, which is reasonable, no answer can constitute the ‘last word’ in a debate or on a certain topic. 1 While existing research may be valuable, it will inevitably have missed important aspects or failed to illuminate them fully. To avoid falling into the gap trap once again, however, it is crucial to explain exactly why particular shortcomings need to be remedied and why certain understandings and explanations are worth pursuing beyond individual motivations.

Real-world problems

The argument that new research is justified if it addresses pressing real-world problems is also influential in the methods literature. For example, King, Keohane and Verba’s second criterion for how to choose a topic is that it is ‘‘important’ in the real world’ ( 1994 : 15). Van Evera similarly emphasizes that political scientists should address questions ‘relevant to real problems facing the real world’ ( 1997 : 97). 2 Shapiro’s suggestion that scholars should be problem driven rather than theory or methods driven, is similarly based on an understanding of ‘problems’ as ‘the great questions of the day’ ( 2002 : 597). To Shapiro, examples include questions such as ‘what the conditions are that make transitions to democracy more or less likely, or what influences the fertility rates of poor women’ (Shapiro, 2002 : 593; see also Flick, 2014 : 12; Booth et al , 2008 : chapter 4). In politics, numerous generic problems require attention. A case in point, central to the International Relations (IR) sub-discipline, is why states go to war. This is undoubtedly an important question, but formulated as such it is a political problem rather than a research puzzle.

Norman Blaikie defines a social problem as a state of affairs in society which policymakers, pundits and sociologists deem inadequate, and therefore in need of attention or a solution. A sociological problem, by contrast, is one that sociologists consider in need of a better explanation or enhanced understanding (Blaikie, 2010 : 45). While scholars can pay attention to, and propose solutions for, social and political problems, we believe they need to frame their research differently from for example the media or the government. Hence, while political problems involve phenomena in need of political attention and resolution, research puzzles pinpoint issues in previous research in need of scholarly attention and resolution. This does not mean that research puzzles cannot have real-world significance, or that researchers should shun political problems. On the contrary, compelling research puzzles often have political significance (Mosser, 2010 ). A real-world problem might be the starting point for a research project, but is in itself insufficient as a justification for new research without an explanation of what makes the existing academic knowledge pertaining to it inadequate.

Another reason why scholars should refrain from basing their research only on what is considered a political problem is that they risk being reduced to useful idiots. Doing such research does not automatically lead scholars to accept established definitions of problems, but it sets certain boundaries – the uncritical acceptance of which increases the risk of adopting status quo-oriented approaches. Of course, even when researchers control the formulation of problems and puzzles, the process is inevitably influenced by individual or collective norms and values (Rosenau, 1980 : 31; Mosser, 2010 : 1078). We argue below that such assumptions should be scrutinized and problematized as far as possible.

Methodological rigour

Apart from gaps and real-world problems, the methods literature implies that rigorous research design and sophisticated methods themselves justify new research. However, as mentioned, the methods literature mainly focuses on how to do research, while discussions of why certain research is necessary are less common (e.g. Patton, 2001 ; Gerring, 2001 ; Marsh and Stoker, 2002 ; Klotz and Lynch, 2007 ; Silverman, 2011 ; Bryman, 2012 ). George and Bennett, for example, focus mostly on the purposes of case studies ( 2004 : 75–79), and King, Keohane and Verba argue for the primacy of inference as a methodological principle, and do not clearly distinguish between ‘topics’, ‘research questions’ and ‘puzzles’ ( 1994 : 14–19). Van Evera, similarly, is mostly preoccupied with arguing why case study method can be used in positivist theory-testing ( 1997 : chapters 1 and 2). In comparison, his discussion of ‘topic selection’ seems more like an afterthought.

While rigorous methodology is necessary in all research, the works discussed above arguably espouse an excessively narrow understanding of what qualifies as such. More importantly for this article’s purposes, we believe the existing methods literature can help determine how things are connected, but is of little use for arguing why knowledge about such connections is interesting or important in the first place. Hence, that empirical data and theory ‘fit together’ in the analysis is insufficient. Instead, a strong argument is necessary as to why new research can provide an explanation or understanding that differs from, and preferably supersedes, those found in existing scholarship.

One reason for this neglect is arguably the entrenchment in the social sciences of Karl Popper’s ( 2002 [1934/1959]: 7–8) distinction between contexts of ‘discovery’ and ‘justification’. Most agree with Popper that methods are essential in the context of justification – where hypotheses are tested and the inquiry is carried out. In contrast, the context of discovery – where the inquiry’s aim is conceived – is often characterized as an irrational act involving intuition, coincidence and wild guesswork. Many methodologists therefore seem to believe that it is impossible to prescribe a method for developing new ideas. Against this deep-seated belief, we argue that it can be done by thinking methodically about research puzzles.

Research puzzles: What, why and how?

Both gaps and real-world problems can be used as starting points when developing research puzzles, and methodological rigour is important in all research projects, but none of these propositions sufficiently motivate new research. This section clarifies what a research puzzle is, why it is useful, and how one can be conceived.

What a puzzle is, and how to develop one

In a 1980 book chapter, James Rosenau emphasized the importance of genuine puzzlement. He exemplified by asking: ‘Why are most governments unable to control inflation?’ ( 1980 : 36). While we may agree that this is puzzling, without a clear connection to previous knowledge it resembles a political problem rather than a political science research puzzle. Rosenau’s example takes the form of a ‘why x -question’, but even someone lacking the most rudimentary knowledge of state finances can formulate such a question (Zinnes 1980 : 338). Developing the question into a research puzzle thus requires asking ‘what is puzzling about how earlier research has described or explained this (allegedly puzzling) phenomenon?’

We propose that the following formula succinctly captures what research puzzles look like: ‘Why x despite y ?’, or ‘How did x become possible despite y ?’ 3 A puzzle thus formulated is admittedly a research question, but one requiring much closer familiarity with the state of the art than a ‘why x -question’. The researcher considers the phenomenon x puzzling since it happens despite y – that is, previous knowledge that would seem contradicted by its occurrence. Hence, puzzlement arises when things do not fit together as anticipated, challenging existing knowledge.

It might be objected that some post-positivist approaches frame their research differently and indeed that a formula for devising puzzles serves knowledge-producing and hence political purposes the same way all methods do – by enacting the ‘worlds’ it analyses (Aradau and Huysmans, 2014 ). Put differently, it could be argued that our formula takes the x and the y as objectively existing and true. While this critique has a point, the x and the y do not need to be viewed as truths, but could be regarded as broadly shared beliefs or reasons for believing that something might be true. Post-positivist approaches could address research puzzles constructed in line with our formula, and influential studies do so (e.g. Campbell, 1992 ; Doty, 1993 ; Weldes and Saco, 1996 ). Hence, we argue that puzzles are impartial to theoretical approach and that social science research, regardless of ontology and epistemology, benefits from constructing clear research puzzles, or from explicating tacit puzzles that sometimes exist between the lines. Research puzzles can increase communicability within and between academic paradigms and therefore enhance the likelihood that a study can become influential and have impact beyond the circle of theoretically or methodologically like-minded scholars.

As research puzzles pinpoint what is considered deviant or unexpected rather than normal, typical or expected, some may object that scholarship should describe and explain general patterns rather than exceptions. While we agree that social science should aim for generality, such an aim does not preclude addressing deviance. The discovery of unexpected deviation from a pattern established in earlier research can produce new knowledge that not simply confirms, but questions what we collectively believe we know. What has hitherto been considered a pattern is destabilized by conflicting observations or interpretations. Indeed, one might even argue that such puzzling ‘anomalies’ are important drivers of scientific progress, regardless of whether they lead to the correction or refinement of an existing theory or its abandonment in favour of an alternative theory (Kuhn, 1970 ; Lakatos, 1970 ; Vasquez, 1997 ; Elman and Elman, 2002 ). Discussions of anomalies in political science have primarily focused on their role in assessing the theoretical progress of research programs, rather than on puzzles as motivations for new research (e.g. Vasquez, 1997 ; Elman and Elman, 2002 ). While being informed by discussions in the former literature, this article is concerned with the latter issue.

The discussion above implies that a new explanation or theory can only be justified if it is seen as different from, and possibly opposed to, established knowledge. Without such differentiation, there is no way of determining whether it illuminates things better than previous research. The detection of unexpected difference – of tensions in the empirics or how they are interpreted – indicates the need for new research.

Exciting and innovative research often advances arguments that appear counterintuitive, which is why we agree with management scholars Mats Alvesson and Dan Kärreman that ‘a desire to become challenged, surprised, bewildered, and confused’ should ‘take centre stage in research’ ( 2007 : 1269). We have the same desire when we read detective stories. The mere fact that a murder has taken place is usually insufficient; the impetus to continue reading is instead provided by complicating factors. Zinnes likens the relationship between the researcher and a puzzle to ‘a detective confronted with a murder in a room with doors locked from the inside and no possible weapon within sight’ ( 1980 : 318). There is not just a corpse, but circumstances that make the murder seem truly puzzling. A detective story where the murderer is precisely the person who appeared guilty at the outset, by contrast, is not worth reading. Academic texts are no different; if they merely confirm what influential theories have long argued, they will make only limited contributions.

Problematization

We believe researchers should continue to wrestle with theories, explanations, assumptions and variables that people have begun to treat as ‘common sense’. Since new research is only new in relation to the old, this is also a useful way of deciding what to do new research on. Entering into critical dialogue with existing research can shed new light on theories and empirical phenomena alike. This is a strategy premised on ‘problematization’, that is, the practice of disrupting ‘taken-for-granted ‘truths’’ (Bacchi, 2012 : 1).

Problematization can be used to turn a ‘why x -question’ into a research puzzle. For example, the social/political problem mentioned above, ‘why do states go to war?’, is clearly a ‘why x -question’. It might be argued that the devastation and suffering brought about by wars at least makes the question implicitly puzzling. We are also eager to know why wars break out, hoping such enhanced understanding might help prevent future wars. However, since this argument depends on what is considered important in a particular society at a certain point in time, it is again a potentially status quo-oriented knowledge interest.

Creating a research puzzle, by contrast, necessitates making an inventory of previous research on the outbreak of wars, and problematizing parts of its assumptions or findings. To make the original question more puzzling, one should ask ‘why not x ?’, that is, inquire into whether there is reason to believe that things are not connected as suggested by the original ‘why x -question’. Hence, we could ask: ‘Why not expect states to go to war?’ The answer might be found in complex interdependence, regional integration, democratic peace or other theories that suggest that wars are less likely under certain circumstances (e.g. Keohane and Nye, 1977 ; Russett, 1993 ). We can then develop the original question: ‘Why do some states go to war despite the existence of complex interdependence, regional integration, or democracy? (Examples would include the Kargil War between India and Pakistan in 1999 and the Russo-Georgian War in 2008.) The clear link to existing research makes the original question truly puzzling.

Some students not only motivate their theses with the dearth of research on a topic, but complain that there is no y in the light of which x is puzzling, that is, that there is no ‘previous research’. A way of dealing with such a situation is to think more broadly, either theoretically or empirically. It may be possible, for example, to contextualize an issue by discussing what it is a case of, or how influential theories might tackle it. There may be no previous research on a certain war, but there are plenty of studies on other wars, and theories about the general phenomenon. Moreover, theories not usually applied to wars could also help provide new insights.

We can problematize not only approaches that we disagree with, but also ones that we are largely sympathetic to. Less thorough problematization involves demonstrating that parts of an explanation or theory are problematic, despite being valuable in other respects. More ambitious problematization may challenge the ontological or epistemological assumptions on which previous research is premised. A case in point is research raising the question of how to recognize a phenomenon when we see it. Such studies focus on key variables in an academic literature that are insufficiently substantiated or taken for granted, and can demonstrate that influential explanations rest on shaky ground. For instance, power transition is one explanation for why wars occur (Organski and Kugler, 1981 ), but due to conceptual complexities associated with the concept of power one could infer that power transitions are more difficult to spot than previous research admits (Chan, 2005 ). Despite such shakiness in the independent variable or explanans , scholars often continue to do research that takes the veracity of the explanation for granted. Similarly, a study can challenge influential ideas about what should be explained – the dependent variable or explanandum (Shapiro, 2002 : 613–615). One could ask, for example, how we can recognize a war when we see one and how to differentiate it from ‘skirmishes’ or other clashes. Problematization is thus a way to construct puzzles premised on the question why previous research treats key variables as unproblematic, despite lingering problems related to conceptualization, measurement or interpretation.

Some assumptions are specific to one theory, while others are shared by several – even seemingly opposing – theories or entire paradigms. Research that challenges widespread conventions can completely overturn the basis for what might previously have seemed like a debate between conflicting positions (Alvesson and Sandberg, 2013 ). The more extensive or influential the problematized research, the greater its potential contribution.

Finally, we believe scholars must continue to problematize and reflect critically not only on previous research, but also on their own assumptions, concepts, theories, methods and conclusions. Since analysis, and the choices it is premised on always serves some interests, the distinction between political problems and political science puzzles may appear to get blurred. Indeed, by making knowledge claims, research inevitably projects power (Ackerly and True, 2008 ; Dauphinee, 2010 ). However, acknowledging the inevitably political nature of research is not equivalent to saying that political problems are sufficient motivation. It is rather a reminder that problematization must be premised on continued reflexivity (e.g. Amoureux and Steele, 2016 ).

Our experience is that graduate students and scholars take interest in phenomena and ask questions for very good reasons, but often fail to justify new research in a way that is sufficiently persuasive to others. Therefore, we believe it is useful to articulate an explicit research puzzle as early as possible in the research process (Rosenau, 1980 : 34). Using problematization as a strategy helps justifying why an issue one has already decided to analyse deserves attention, but more fundamentally it helps improving the initial questions and alternative interpretations that one might bring to the table. Once established, the puzzle provides both a rationale and direction for the research process. The choices of aim, research questions, theory and, to some extent, methods and materials all tend to follow. In our experience, the careful construction of a research puzzle amounts to about half the job, especially when writing a thesis or journal article. Sufficient time should thus be allocated to it. Conversely, the absence of a research puzzle risks inviting inconsistencies into the research, some of which may be irreconcilable.

In conceptualizing the research puzzle as preferably preceding the investigation, we anticipate the objection that our advice is biased towards deduction. There is some truth in the allegation. Yet, it is highly unlikely that scholars undertake an investigation completely inductively, without being guided by any prior assumptions or preconceptions. Since data are always theory dependent (Kuhn, 1970 ), any investigation is inevitably characterized by a degree of deduction. Having said that, most scholars arguably also engage in continuous inductive reflection, often even before realizing that a research process has begun. Since some ideas antedate each investigation, they should be articulated as early as possible. At the same time, it is important to be prepared for the possibility that one’s initial research puzzle will require continual improvement and that a final version can only be constructed after completing the inquiry.

Charles Sanders Peirce called this continual movement to and fro between theory and empirics ‘abduction’ ( 1934 ). Although it has become associated with the context of justification, for Peirce and others abduction pertains more closely to the context of discovery, where hypotheses – and research puzzles – are formulated (cf. Hanson, 1958 : 72). Abductive reasoning often departs from puzzling cases, which need to be ‘rendered intelligible’ (Glynos and Howarth, 2007 : 34; see also Beach and Pedersen, 2013 : 19). The researcher seeks to conceptualize a plausible interpretation that can cast light on a specific case, but then moves on to inquire whether the interpretation can be extended to other cases. As the research process proceeds, the interpretation’s scope, quality and distinctiveness develop in parallel. This shows how closely entwined the formulation of the research puzzle is with the actual investigation, and why the puzzle must be continually honed as the inquiry progresses.

From topic to research puzzle: power shift in East Asia

This section puts our advice to use by demonstrating how to develop a vague idea into a research puzzle. Our example takes the form of a dialogue between person A, perhaps a graduate student, and person B, possibly a supervisor. The dialogue is fictitious although the example is closely related to our own empirical research interest – international politics in East Asia. After most of B’s utterances, we formulate more general advice in brackets. This advice can hopefully serve as steps towards a recipe for formulating research puzzles, although the order of the steps is not fixed (see Table  1 ).

A: I want to do research on the East Asian power shift.

B: Since this is just a topic, I will ask questions to help you make necessary distinctions and construct a research puzzle. First, what power shift? [Make distinctions to narrow down your interest from a topic to an approximation of a research puzzle]

A: The current power shift from Japan to China.

B: Why do you find this ostensible power shift interesting? [Explicate the motives and preconceptions underlying your interest in an issue]

A: East Asia is increasingly significant in world politics. If a power shift occurs there it could have repercussions beyond the region.

B: This sounds like a political problem. Scholars also care about it, but must relate their interest to previous knowledge. What does the existing academic literature say about an East Asian power shift? [Approach your topic as a political science problem, rather than just a political problem]

A: Having read key texts I understand that previous research views Japan’s power as diminishing relative to China’s.

B: How can you contribute to this research? [Make sure your research aims to produce new knowledge]

A: Existing research primarily analyses issues related to economic and military capabilities, and regional territorial conflicts. There is a gap concerning who has agenda - setting power in regional organizations.

B: Since the existence of a gap has little intrinsic value, you must clarify why it is necessary to fill it: How could research on agenda - setting power in regional organizations help illuminate East Asian power relations? [A gap in previous research is a necessary, but insufficient, argument for new research]

A: We can assume that Japan has been influential in regional organizations for decades and has exercised power over the agendas of these organizations. If power is indeed shifting from Japan to China, it should be detectable in this area.

B: You are making many assumptions, each of which deserves its own investigation. For example, what exactly is agenda - setting power and how can we recognize its exercise when we see it? Moreover, do we know Japan previously exercised such power? And are regional organizations important in East Asia? [Problematize the often commonsensical assumptions on which that previous knowledge is based]

A: How do I address those questions?

B: Absorb yourself even more in the relevant literature. [Since knowledge is necessary for constructing a research puzzle, read broadly in fields related to your problem area]

A: After reading more, I understand that agenda - setting power is the capacity to decide which questions are included on or excluded from the political agenda. I also realize that while East Asian regional organizations are deemed less important than, for example, European ones, they are nonetheless active in important issue areas. There is no research that explores whether Japan has exercised such agenda - setting power in the past.

B: Okay, do decisions by organizations such as ASEAN  +  3 (Association of Southeast Asian Nations plus China, Japan and South Korea) somehow contradict your assumptions about regional agenda - setting power? [If the empirical record can be interpreted as conforming to existing assumptions, it is not clear why further research is necessary]

A: Yes, I have identified a common assumption in the literature that Japan had agenda - setting power commensurate with its economic capabilities, but empirical analysis demonstrating this is lacking.

B: This gets you close to a research puzzle, but a different one than you envisaged: Why has Japan been attributed so much agenda - setting power despite limited empirical evidence? [As your understanding deepens, be prepared to tweak your research puzzle accordingly]

A: Okay, can I do my research now?

B: Not yet. First you need to consider whether this discrepancy applies only to Japan or is a broader phenomenon. For example, if Japan was attributed agenda - setting power before the ostensible power shift, is it possible that China is currently ascribed such power on similarly shaky grounds? This would suggest the existence of a meta - level agenda - setting power that previous analyses have missed. [The bigger the target, the greater the potential contribution, so determine whether your observation is a case of a wider phenomenon]

A: Can the discrepancy between assumptions about Japan and the empirics also motivate a more thorough empirical analysis of agenda - setting power in East Asia?

B: Absolutely, and the analysis does not have to be limited to East Asia. The target could be an influential theory, according to which agenda - setting power follows from economic capabilities: the hypotheses derived from such a theory would seemingly be falsified by available empirics. In other words, theory C states that agenda - setting power follows from economic capabilities. The available empirics, however, do not support the assumption that Japan has had agenda - setting power commensurate with its economic strength. Why, has Japan not had considerable agenda - setting power, despite its possession of huge economic capabilities for so long? How can this anomaly for theory C be explained? [Clarify to what theory, explanation, or interpretation your own thesis would be counterintuitive]

A: Yes, that certainly is an interesting research question.

B: It is not simply a question. Since it contains both an observation and a theoretical understanding to which the observation is paradoxical, it is a research puzzle. [Construct a clear research puzzle using the formula ‘why x despite y ’]

A: Okay, great, thanks.

B: I know you are eager to get started with your thesis, but constructing research puzzles is like peeling an onion: each layer hides another layer. For example, I suggested that the notion of meta - level agenda - setting power might help us understand the puzzle we just constructed. If you aim to contribute mainly to the study of East Asian international relations, you might be able to draw on theoretical work on meta - power or discursive power to craft an explanation. But you could also ask: does anything about my puzzle remain puzzling to this literature? If so, you could make a theoretical contribution extending beyond what you originally intended. [Pinpoint a research puzzle early in the research process but be prepared to find other potentially more significant puzzles that lead to greater contributions – possibly at higher levels of abstraction – as your knowledge expands]

One of the greatest challenges for graduate students is how to come up with a clear rationale for their dissertations and theses. This article has argued that research puzzles are more useful for that endeavour than the most common propositions in the existing methods literature: that new research should fill gaps, address important real-world problems and/or be methodologically rigorous. In contrast, a research puzzle often provides a sufficient justification. Research puzzles are different from societal or political problems, primarily because they are explicitly contextualized in relation to previous knowledge and research. Good research puzzles have in common that they problematize something in existing research. A well-formulated research puzzle provides direction and coherence to the research process and governs all other key choices, such as the aim, research questions, material and method. Our argument should not be misinterpreted as suggesting that research puzzles are the only way for providing new research with a rationale. Future research should identify and develop ways for constructing other types of justifications.

The present article is no exception; criticism of it is likely to enhance our collective understanding of research puzzles.

Neither King, Keohane and Verba nor Van Evera argues that a real-world problem is sufficient motivation for new research. Instead, the former consider it a necessary criterion along with the need to make a specific contribution to an identifiable scholarly literature ( 1994 : 15–17), and the latter puts it alongside gaps and ‘key disputes of fact or theory’ ( 1997 : 99). Nonetheless, both works strongly emphasize the importance of real-world problems.

Because of space restrictions, the remainder of this article discusses ‘why x -questions’, but the discussion is applicable to ‘how possible questions’ too.

Ackerly, B. and True, J. (2008) ‘Reflexivity in practice: Power and ethics in feminist research on international relations’, International Studies Review 10(4): 693–707.

Article   Google Scholar  

Alvesson, M. and Kärreman, D. (2007) ‘Constructing mystery: Empirical matters in theory development’, Academy of Management Review 32(4): 1265–1281.

Alvesson, M. and Sandberg, J. (2013) Constructing Research Questions: Doing Interesting Research , London: Sage.

Book   Google Scholar  

Amoureux, J.L. and Steele, B.J. (eds.) (2016) Reflexivity and International Relations: Positionality, Critique, and Practice , London: Routledge.

Google Scholar  

Aradau, C. and Huysmans, J. (2014) ‘Critical methods in international relations: The politics of techniques, devices and acts’, European Journal of International Relations 20(3): 596–619.

Bacchi, C. (2012) ‘Why study problematizations? Making politics visible’, Open Journal of Political Science 2(1): 1–8.

Beach, D. and Pedersen, R.B. (2013) Process - Tracing Methods: Foundations and Guidelines , Ann Arbor: University of Michigan Press.

Blaikie, N. (2010) Designing Social Research , 2nd edn, Cambridge: Polity Press.

Bloomberg, L.D. and Volpe, M. (2008) Completing Your Dissertation: A Roadmap from Beginning to End , London: Sage.

Booth, W.C, Colomb, G.G. and Williams, J.M. (2008) The Craft of Research , Chicago: The University of Chicago Press.

Bryman, A. (2012) Social Research Methods , 4th edn, Oxford: Oxford University Press.

Campbell, D. (1992) Writing Security: United States Foreign Policy and the Politics of Identity , Manchester: Manchester University Press.

Chan, S. (2005) ‘Is there a power transition between the US and China? The different faces of national power’, Asian Survey 45(5): 687–701.

Collier, D., Seawright, J. and Munck, G.L. (2004) ‘The Quest for Standards: King, Keohane and Verba’s Designing Social Enquiries’, in H.E. Brady and D. Collier (eds.) Rethinking Social Inquiry: Diverse Tools, Shared Standards , Lanham, MD: Rowman and Littlefield, pp. 21–50.

Dauphinee, E. (2010) ‘The ethics of autoethnography’, Review of International Studies 36(3): 799–818.

Della Porta, D. and Keating, M. (eds.) (2008) Approaches and Methodologies in the Social Sciences: A Pluralist Perspective , Cambridge: Cambridge University Press.

Doty, R.L. (1993) ‘Foreign policy as social construction: A post-positivist analysis of US counterinsurgency policy in the Philippines’, International Studies Quarterly 37(3): 297–320.

Elman, C. and Elman, M.F. (2002) ‘How not to be Lakatos intolerant: Appraising progress in IR research’, International Studies Quarterly 46(2): 231–262.

Flick, U. (2007) Designing Qualitative Research , London: Sage.

Flick, U. (2014) An Introduction to Qualitative Research , 5th edn, London: Sage.

George, A.L. and Bennett, A. (2004) Case Studies and Theory Development in the Social Sciences , Cambridge: MIT Press.

Gerring, J. (2001) Social Science Methodology: A Critical Framework , Cambridge: Cambridge University Press.

Glynos, J. and Howarth, D. (2007) Logics of Critical Explanation in Social and Political Theory , London: Routledge.

Hanson, N.R. (1958) Patterns of Discovery: An Inquiry into the Conceptual Foundations of Science , Cambridge: Cambridge University Press.

Keohane, R.O. and Nye, J.S. Jr (1977) Power and Interdependence: World Politics in Transition , Boston: Little, Brown.

King, G., Keohane. R.O. and Verba, S. (1994) Designing Social Inquiry: Scientific Inference in Qualitative Research , Princeton: Princeton University Press.

Klotz, A. and Lynch, C. (2007) Strategies for Research in Constructivist International Relations , Armonk, NY: ME Sharpe.

Kuhn, T.S. (1970) The Structure of Scientific Revolutions , 2nd ed. Chicago: University of Chicago Press.

Lakatos, I. (1970) ‘Falsification and the Methodology of Scientific Research Programmes’ in I. Lakatos and A. Musgrave (eds.) Criticism and the Growth of Knowledge , Cambridge: Cambridge University Press, pp. 91–196

Chapter   Google Scholar  

Löwenheim, O. (2010) ‘The ‘I’ in IR: An autoethnographic account’, Review of International Studies 36(4): 1023–1045.

Marsh, D. and Stoker, G. (2002) Theory and Method in Political Science , 2nd ed. New York: Palgrave Macmillan.

Mosser, M.W. (2010) ‘Puzzles versus problems: The alleged disconnect between academics and military practitioners’, Perspectives on Politics 8(4): 1077–1086.

Organski, A.F.K. and Kugler, J. (1981) The War Ledger , Chicago: University of Chicago Press.

Patton, M.Q. (2001) Qualitative Evaluation and Research Methods , 3rd ed. London: Sage.

Peirce, C.S. (1934) Collected Papers of Charles Sanders Peirce , vols. 5–6, Cambridge, MA: Harvard University Press.

Popper, K. (2002 [1934/1959]) The Logic of Scientific Discovery , New York: Routledge. First published in German in 1934; first English translation in 1959.

Rosenau, J.N. (1980) ‘Thinking theory thoroughly’, in J.N. Rosenau (ed.) The Scientific Study of Foreign Policy , London: Frances Pinter, pp. 29–37.

Russett, B. (1993) Grasping the Democratic Peace , Princeton, NJ: Princeton University Press.

Schmitter, P. (2008) ‘The design of social and political research’, in D. Della Porta and M. Keating (eds.) Approaches and Methodologies in the Social Sciences: A Pluralist Perspective , Cambridge: Cambridge University Press, pp. 263–295.

Shapiro, I. (2002) ‘Problems, methods, and theories in the study of politics, or what’s wrong with political science and what to do about it’, Political Theory 30(4): 596–619.

Silverman, D. (ed.) (2011) Qualitative Research , 3rd ed. London: Sage.

Van Evera, S. (1997) Guide to Methods for Students of Political Science , Ithaca: Cornell University Press.

Vasquez, J.A. (1997) ‘The realist paradigm and degenerative versus progressive research programs: An appraisal of neotraditional research on Waltz’s balancing proposition’, American Political Science Review 91(4): 899–912.

Weldes, J. and D. Saco (1996) ‘Making state action possible: The United States and the discursive construction of ‘The Cuban Problem’, 1960–1994’, Millennium: Journal of International Studies 25(2): 361–395.

Zinnes, D.A. (1980) ‘Three puzzles in search of a researcher’, International Studies Quarterly 24(3): 315–342.

Download references

Acknowledgements

The authors would like to express their gratitude to participants in seminars at Freie Universität Berlin, Lund University and Warwick University, as well as to Stefan Borg, Niklas Bremberg, Johan Engvall, Johan Eriksson, Ulv Hanssen, Björn Jerdén, Regina Karp, Wrenn Yennie Lindgren, Tom Lundborg, Nicola Nymalm, Gunilla Reischl, Anke Schmidt-Felzmann, Oliver Turner, Mikael Weissmann and Stephanie Winkler for their useful comments on earlier versions of this article.

Author information

Authors and affiliations.

Swedish Institute of International Affairs, P.O. Box 27035, 10251, Stockholm, Sweden

Karl Gustafsson

Swedish Defence University, P.O. Box 27805, 11593, Stockholm, Sweden

Linus Hagström

You can also search for this author in PubMed   Google Scholar

Corresponding author

Correspondence to Karl Gustafsson .

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License ( http://creativecommons.org/licenses/by/4.0/ ), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made.

Reprints and permissions

About this article

Gustafsson, K., Hagström, L. what is the point? teaching graduate students how to construct political science research puzzles. Eur Polit Sci 17 , 634–648 (2018). https://doi.org/10.1057/s41304-017-0130-y

Download citation

Published : 31 July 2017

Issue Date : December 2018

DOI : https://doi.org/10.1057/s41304-017-0130-y

Share this article

Anyone you share the following link with will be able to read this content:

Sorry, a shareable link is not currently available for this article.

Provided by the Springer Nature SharedIt content-sharing initiative

• political science
• problematization
• research puzzles
• research design
• Find a journal
• Publish with us
• Track your research

• Corpus ID: 147600381

USING WORD-SEARCH-PUZZLE GAMES FOR IMPROVING VOCABULARY KNOWLEDGE OF IRANIAN EFL LEARNERS

• Vossoughi Hossein , Zargar Marzieh
• Published 2009
• Education, Linguistics, Computer Science

Tables from this paper

19 Citations

The effectiveness of word-search puzzle games on vocabulary mastery for the students of first grade in mts muhammadiyah 1 malang, the impact of using problem-solving puzzles on iranian intermediate efl learners' lexical knowledge, mastering vocabulary by using crossword puzzle, fostering vocabulary enhancement through language games in the context of blended learning environment, the effect of puzzle-based learning on secondary school students’ attitudes toward english lesson and self-efficacy beliefs in english, the effect of using quizlet on students’ vocabulary achievement, using anagram technique to develop students’ vocabulary mastery to the seventh grade students, increasing students’ vocabulary by using mind mapping technique at smp n 2 bitung, tiktok’s effect on english vocabulary learning: an empirical study of english major curriculum, the effectiveness of crossword puzzle in learning vocabulary (a quasy-experimental study at the seventh grade students of mts. muhammadiyah 1 ciputat), 8 references, the vocabulary book: learning & instruction, how to teach vocabulary, the practice of english language teaching, techniques in teaching vocabulary, related papers.

Showing 1 through 3 of 0 Related Papers

• Brain Development
• Childhood & Adolescence
• Diet & Lifestyle
• Emotions, Stress & Anxiety
• Learning & Memory
• Thinking & Awareness
• Alzheimer's & Dementia
• Childhood Disorders
• Immune System Disorders
• Mental Health
• Neurodegenerative Disorders
• Infectious Disease
• Neurological Disorders A-Z
• Body Systems
• Cells & Circuits
• Genes & Molecules
• The Arts & the Brain
• Law, Economics & Ethics
• Neuroscience in the News
• Supporting Research
• Tech & the Brain
• Animals in Research
• BRAIN Initiative
• Meet the Researcher
• Neuro-technologies
• Tools & Techniques
• Core Concepts
• For Educators
• Ask an Expert
• The Brain Facts Book

Kinds of Research Word Puzzle

• Published 20 May 2019
• Author Emma Lindberg
• Source BrainFacts/SfN

Chapter 16 of the Brain Facts book introduces readers to some of the most important research methods for studying the brain, including types of microscopy, animal models, and cutting-edge molecular techniques.  After reading and reviewing this chapter with your students, use the provided worksheet to test their knowledge of the material. 

Integration into the Curriculum

• Biology, AP Biology
• Anatomy and Physiology
• The Brain Facts book (a PDF version of the book is available at the link)
• One copy per student of the Kinds of Research word puzzle worksheet  (you can access an interactive PDF version here )
• One copy of the Kinds of Research word puzzle Teachers Sheet for grading

Instructions

• Have your students read Chapter 16: Kinds of Research.
• Distribute a copy of the Kinds of Research word puzzle worksheet to each student. 
• Review the “Instructions” listed at the top of the worksheet.
• Ask your students to complete the worksheet using the book or their notes. 

About the Author

Emma Lindberg

Emma is the former Outreach Associate at the Society for Neuroscience. She helped find and create Educator Resources for BrainFacts.org , as well as promote the site at conferences and events across the country.

CONTENT PROVIDED BY

BrainFacts/SfN

Also In For the Classroom

Popular articles on BrainFacts.org

Find a Neuroscientist

Engage local scientists to educate your community about the brain.

BrainFacts Book

Download a copy of the newest edition of the book, Brain Facts: A Primer on the Brain and Nervous System.

Educator Resources

Explain the brain to your students with a variety of teaching tools and resources.

SUPPORTING PARTNERS

• Privacy Notice
• Accessibility Policy
• Terms and Conditions
• Manage Cookies

Some pages on this website provide links that require Adobe Reader to view.

An official website of the United States government

The .gov means it’s official. Federal government websites often end in .gov or .mil. Before sharing sensitive information, make sure you’re on a federal government site.

The site is secure. The https:// ensures that you are connecting to the official website and that any information you provide is encrypted and transmitted securely.

• Publications
• Account settings

The PMC website is updating on October 15, 2024. Learn More or Try it out now .

• Advanced Search
• Journal List
• JMIR Serious Games
• v.10(3); Jul-Sep 2022

The Effectiveness of Serious Games in Improving Memory Among Older Adults With Cognitive Impairment: Systematic Review and Meta-analysis

Alaa abd-alrazaq.

1 AI Center for Precision Health, Weill Cornell Medicine-Qatar, Doha, Qatar

Dari Alhuwail

2 Information Science Department, Kuwait University, Kuwait, Kuwait

3 Health Informatics Unit, Dasman Diabetes Institute, Kuwait, Kuwait

Eiman Al-Jafar

4 Kuwait Health Informatics Association, Kuwait, Kuwait

Arfan Ahmed

Farag shuweihdi.

5 Leeds Institute of Health Sciences, School of Medicine, University of Leeds, Leeds, United Kingdom

Shuja Mohd Reagu

6 Mental Health Services, Hamad Medical Corporation, Doha, Qatar

Mowafa Househ

7 Division of Information and Computing Technology, College of Science and Engineering, Hamad Bin Khalifa University, Qatar Foundation, Doha, Qatar

Associated Data

PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) checklist.

Search strategy.

Data extraction form.

Excluded studies.

Reviewers’ judgments about each risk of bias domain for each included study.

Grading of Recommendations Assessment, Development, and Evaluation profile for comparison of serious games with control, conventional exercises, and conventional cognitive activities for verbal memory.

Moderation analyses for verbal memory.

Grading of Recommendations Assessment, Development, and Evaluation profile for comparison of serious games with control, conventional exercises, conventional cognitive activities, and other serious games for nonverbal memory.

Moderation analyses for nonverbal memory.

Grading of Recommendations Assessment, Development, and Evaluation profile for comparison of serious games with control, conventional exercises, conventional cognitive activities, and other serious games for working memory.

Moderation analyses for working memory.

Memory, one of the main cognitive functions, is known to decline with age. Serious games have been used for improving memory in older adults. The effectiveness of serious games in improving memory has been assessed by many studies. To draw definitive conclusions about the effectiveness of serious games, the findings of these studies need to be pooled and aggregated.

This study aimed to assess the effectiveness of serious games in improving memory in older adults with cognitive impairment.

A systematic review of randomized controlled trials was carried out. The search sources included 8 databases, the reference lists of the included studies and relevant reviews, and the studies that cited the included studies. In total, 2 reviewers (AA and MH) independently carried out the study selection, data extraction, risk of bias assessment, and quality of evidence appraisal. Extracted data were synthesized using a narrative approach and a statistical approach (ie, multilevel meta-analysis), as appropriate.

Of the 618 citations retrieved, 18 (2.9%) met the eligibility criteria for this review. Of these 18 studies, 15 (83%) randomized controlled trials were included in 10 multilevel meta-analyses. We found that serious games were more effective than no or passive interventions in improving nonverbal memory ( P =.02; standardized mean difference [SMD]=0.46, 95% CI 0.09-0.83) and working memory ( P =.04; SMD=0.31, 95% CI 0.01-0.60) but not verbal memory ( P =.13; SMD=0.39, 95% CI −0.11 to 0.89). The review also showed that serious games were more effective than conventional exercises in improving verbal memory ( P =.003; SMD=0.46, 95% CI 0.16-0.77) but not nonverbal memory ( P =.30; SMD=−0.19, 95% CI −0.54 to 0.17) or working memory ( P =.99; SMD=0.00, 95% CI −0.45 to 0.45). Serious games were as effective as conventional cognitive activities in improving verbal memory ( P =.14; SMD=0.66, 95% CI −0.21 to 1.54), nonverbal memory ( P =.94; SMD=−0.01, 95% CI −0.32 to 0.30), and working memory ( P =.08; SMD=0.37, 95% CI −0.05 to 0.78) among older adults with cognitive impairment. Finally, the effect of adaptive serious games on working memory was comparable with that of nonadaptive serious games ( P =.08; SMD=0.18, 95% CI −0.02 to 0.37).

Conclusions

Serious games have the potential to improve verbal, nonverbal, and working memory in older adults with cognitive impairment. However, our findings should be interpreted cautiously given that most meta-analyses were based on a few studies (≤3) and judged to have a low quality of evidence. Therefore, serious games should be offered as a supplement to existing proven and safe interventions rather than as a complete substitute until further, more robust evidence is available. Future studies should investigate the short- and long-term effects of serious games on memory and other cognitive abilities among people of different age groups with or without cognitive impairment.

Introduction

Life expectancy has increased worldwide as people have better access to health care services and an improved standard of living. As a result, people are living longer [ 1 - 3 ]. According to the United Nations World Population Aging 2020 report [ 4 ], the number of people aged ≥65 years has increased up to 727 million worldwide. The older population group is expected to increase to 16% by 2050 compared with 9.3% in 2020 [ 4 ]. The older population group is more likely to develop cognitive impairment [ 5 , 6 ], which is a decline in cognitive abilities and functions such as memory, attention, concentration, learning, and language [ 7 , 8 ]. According to the Alzheimer’s Association, approximately 12% to 18% of people aged ≥60 years have mild cognitive impairment (MCI) [ 9 ].

MCI refers to a decline in the ability to learn new information or recall stored information and occurs along a continuum that ranges from normal to severely impaired cognition [ 10 ]. Although inconsistencies exist in screening for MCIs, it is certain that they occur because of brain changes owing to multiple factors, including older age, injuries to the brain, diabetes, hypertension, stroke, depression, and physical inactivity [ 11 ]. Memory is one of the main cognitive functions that decline with age. Memory is known as the ability of the brain to hold information and recall it as needed. There are different types of memory: verbal, nonverbal, and working memory. Verbal memory refers to a person’s ability to remember what they read or hear of information that was already learned [ 12 ]. On the other hand, nonverbal memory refers to storing, retrieving, and remembering nonverbal information, content, or experiences, such as images, feelings, tastes, sounds, shapes, and smells [ 13 ]. Furthermore, memory is divided into 3 types according to the period for which the memorized information is retained: short-term, long-term, and working memory. Short-term memory temporarily holds a limited amount of information [ 14 ], whereas long-term memory refers to the relatively permanent storage and recall of information [ 15 ]. Working memory refers to the temporary storage of a limited amount of information to be used in the execution of cognitive activities such as learning, reasoning, and comprehension [ 16 ].

Several nonpharmacological interventions can be used to improve memory, such as physical exercise, cognitive behavioral therapy, psychosocial therapy, good nutrition, and serious games [ 17 ]. Serious games are defined as electronic games that are played for purposes beyond leisure to promote the users’ mental, physical, and social well-being [ 18 , 19 ]. Recent evidence suggests that exergames are effective in improving physical and cognitive function in people with MCIs [ 20 ] as well as their compliance and adherence to medical interventions embedded in serious games [ 21 , 22 ]. Previous systematic reviews have shown that serious games have the potential to prevent or alleviate mental disorders such as depression [ 23 ], anxiety [ 24 ], and cognitive impairment [ 25 ]. Several types of serious games have been used to improve cognitive abilities, namely (1) cognitive training games (which deliver cognitive activities to maintain or improve cognitive functions) and (2) exergames (which entail physical exercises as part of the intended gameplay [ 25 ]). Compared with conventional exercise and cognitive training, serious games can positively affect mood, social functioning, mental health well-being, and cognitive flexibility in older adults [ 26 - 29 ].

Research Problem and Objectives

The effectiveness of serious games in improving memory has been assessed by many studies. To draw definitive conclusions about the effectiveness of serious games, the findings of these studies need to be pooled and aggregated. Several systematic reviews have summarized the evidence from these studies; however, they had a different aim and scope from this review. Specifically, these reviews (1) focused on healthy older adults and not necessarily those with cognitive impairment [ 17 , 30 - 33 ] (therefore, future reviews should consider older adults with cognitive impairment), (2) included pilot randomized controlled trials (RCTs) and quasi-experiments [ 17 , 20 , 33 , 34 ] (thus, future reviews should include only RCTs), (3) performed an outdated search (>5 years [ 17 , 32 , 34 ]; therefore, an updated review or a new review are required), (4) did not assess the quality of evidence [ 17 , 20 , 30 , 33 , 34 ] (thus, the quality of the evidence should be assessed in future reviews), (5) only focused on a specific type of serious game such as cognitive training games [ 30 , 34 ] and exergames [ 17 , 20 , 33 ] (hence, future reviews should consider all types of serious games), (6) focused on a certain type of memory (working memory [ 34 ]; therefore, all types of memory should be considered in upcoming reviews), or (7) did not compare the effect of serious games with a specific comparator (eg, no intervention, conventional exercises, or conventional cognitive activities [ 17 , 20 , 30 , 33 , 34 ]; thus, further reviews are needed to compare the effect of serious games with a specific comparator). To address the aforementioned gaps, this study aimed to assess the effectiveness of serious games in improving memory among older adults with cognitive impairment. This review focused only on memory as other cognitive domains—for example, global cognition [ 25 ], executive functions [ 35 ], and processing speed [ 36 ]—were targeted by previous reviews.

The authors followed the expanded version of the PRISMA (Preferred Reporting Items for Systematic Reviews and Meta-Analyses) guidelines to conduct a systematic review and meta-analyses ( Multimedia Appendix 1 ). The protocol for this review was registered with PROSPERO (CRD42021292150).

Eligibility Criteria

This review included only RCTs that looked at the effectiveness of serious games in improving memory in older adults with cognitive impairment. The target intervention in this review was serious games supplied on any digital platform, such as computers (PCs), consoles (Xbox and PlayStation), mobile phones, handheld devices, Nintendo, or any other computerized device. Furthermore, components of gaming had to be used as an important and major technique for reaching the intervention’s goal. Serious games had to be used solely for the purpose of therapy. Studies combining serious games with other interventions were eligible if the control group received the same adjacent intervention. Nondigital games and those used for other purposes, such as monitoring, screening, diagnosis, and research, were excluded.

The study focused on the older adult population (aged ≥60 years) who had any type of cognitive impairment or condition (eg, MCI, Alzheimer disease, or dementia). Their diagnosis had to be confirmed by checking the inclusion criteria or baseline scores against standardized diagnostic criteria (eg, Mini-Mental State Examination and Montreal Cognitive Assessment). This review did not focus on healthy older adults, health care providers, or caregivers. No restrictions were applied regarding sex and ethnicity.

The main outcome of interest in this review was memory regardless of the type (verbal, nonverbal, or working memory) and regardless of the tool used for measuring the outcome. Studies were excluded if they assessed only other cognitive outcomes (eg, language and processing speed), cost-effectiveness, acceptance, feasibility, or satisfaction. This review focused on outcome data that were measured immediately after the intervention rather than on follow-up data.

Only RCTs conducted in English and from 2010 onward were considered. Pilot or feasibility RCTs, quasi-experiments, observational studies, and reviews were omitted. Studies published as journal articles, conference proceedings, or dissertations were included. Reviews, conference abstracts, proposals, editorials, and commentaries were all excluded. Finally, no restrictions related to the country of publication, comparator, or study setting were applied.

Information Sources and Search Strategy

The studies that were relevant to this review were found by searching 7 bibliographic databases: MEDLINE (via Ovid), PsycINFO (via Ovid), EMBASE (via Ovid), CINAHL (via EBSCO), IEEE Xplore, ACM Digital Library, and Scopus. Furthermore, we searched the search engine Google Scholar. Owing to the high number of papers obtained through Google Scholar, only the first 10 pages (ie, 100 records) were taken into account as they were automatically ordered based on their relevance [ 37 ]. The first author (AA) conducted the search on August 6, 2021. An automatic alert was set up to retrieve studies that were added to the databases after that date; this continued for 16 weeks (ending on December 5, 2021). Forward reference list checking (ie, screening studies that cited the included studies) and backward reference list checking (ie, screening the reference lists of the included studies and relevant reviews) were carried out to retrieve further studies.

To develop the search query for this review, the authors consulted 2 experts in digital mental health and checked the search queries used in other systematic reviews within this field. The chosen search terms were related to the target population (eg, cognitive impairment), target intervention (eg, serious games and exergames), and target study design (eg, RCTs). Multimedia Appendix 2 summarizes the search query that was used for searching each of the 8 databases.

Selection Process

Relevant studies were identified taking the following steps. First, the obtained studies were imported into EndNote X8 (Clarivate Analytics) to identify and delete duplicate items. Second, the titles and abstracts of all the retrieved studies were evaluated in the second phase by 2 reviewers (AA and MH) working independently. Finally, the 2 reviewers independently evaluated the entire texts of the studies included in the previous step. Any disagreements in the 2 previous steps were resolved via discussion. The interrater agreement (Cohen κ) in steps 2 and 3 was 0.94 and 0.96, respectively, indicating a near-perfect level of interrater agreement [ 38 ].

Data Collection Process

In total, 2 independent reviewers (AA and MH) used Microsoft Excel to extract data from all the included studies. The data extraction form used to extract data from the included studies was pilot-tested using 2 of the included studies ( Multimedia Appendix 3 ). The reviewers’ disagreements were resolved through discussion. An interrater agreement of 0.85 was observed, indicating a near-perfect degree of agreement. If data such as the mean, SD, and sample size were unavailable from the published studies, contact was made with the first and corresponding authors in an attempt to retrieve them.

Study Risk of Bias Assessment

The Cochrane Collaboration recommends assessing the risk of bias via 2 independent reviewers (AA and MH) using the Risk of Bias 2 (RoB 2) tool [ 39 ]; as such, these guidelines were followed for this review. The RoB 2 tool assesses the risk of bias in 5 domains of RCTs: randomization process, deviations from intended interventions, missing outcome data, measurement of the outcome, and selection of the reported result [ 39 ]. The risk of bias judgments in these domains were used to determine the overall risk of bias of each included study. Any inconsistencies in decisions between the reviewers were resolved by consulting a third reviewer. Interrater agreement between the reviewers was near perfect (Cohen κ=0.93) [ 38 ].

Synthesis Methods

A narrative and statistical approach was used to synthesize the information acquired. In our narrative synthesis, we used texts and tables to describe the characteristics of the included studies (demographic, intervention, comparator, and outcome variables). The results of the experiments were categorized and pooled based on measured outcome (ie, verbal, nonverbal, and working memory) and the comparator (ie, control, conventional exercises, conventional cognitive training, and other serious games). A meta-analysis was conducted when at least two studies with the same measured outcome and comparator submitted enough data (ie, mean, SD, and number of participants in each intervention group). Owing to the type of data for the outcome of interest (memory) being continuous and the methods used to measure the outcome being variable throughout the included studies, the standardized mean difference (SMD; Cohen d ) was used to analyze the overall effect of each study. The random effects model was used for the analysis because of the high clinical heterogeneity among the meta-analyzed studies in terms of serious game characteristics (eg, type, duration, frequency, and period), population characteristics (eg, sample size, mean age, and health condition), and outcome measures (ie, tools and follow-up period). As several studies used more than one outcome measure to assess memory, the dependency on effect sizes within or across studies will be introduced in the meta-analysis. As a result, a multilevel meta-analysis considering the dependency on effect sizes and sampling covariance between the effect sizes was used [ 40 - 42 ]. Namely, the multilevel meta-analysis should be applied when effect sizes within the same study are very likely to be more similar to each other than the effect sizes across studies [ 42 ]. The R (version 4.3.1; R Foundation for Statistical Computing) statistical package was used to perform the analysis. We used the function rma.mv in the library metafor , which is a library in R , to perform the multilevel meta-analysis [ 43 ].

If we observed a statistically significant difference between the groups in a meta-analysis, we further sought to examine if it was clinically significant. The phrase “minimal clinically important difference” (MCID) refers to the smallest change in a measured outcome that a patient would consider worthwhile and significant enough to warrant a change in treatment. The MCID boundaries were calculated as 0.5 times the SMD of the meta-analyzed studies.

We calculated I 2 and a chi-square P value to investigate the degree and statistical significance of the heterogeneity in the meta-analyzed studies, respectively. A chi-square P value of ≤.05 suggests heterogeneous meta-analyzed studies [ 44 ]. When I 2 ranged from 0% to 40%, 30% to 60%, 50% to 90%, and 75% to 100%, the degree of heterogeneity was judged to be insignificant, moderate, substantial, or considerable, respectively [ 44 ].

Certainty of Evidence

To appraise the overall quality of evidence resulting from the meta-analyses, we applied the Grading of Recommendations Assessment, Development, and Evaluation approach [ 45 ], which assesses the quality of evidence based on 5 domains: risk of bias, inconsistency (ie, heterogeneity), indirectness, imprecision, and publication bias [ 45 ]. In total, 2 reviewers independently rated the overall quality of the meta-analyzed evidence, and disagreements were resolved through discussion. The interrater agreement of the reviewers was considered near perfect (Cohen κ=0.87) [ 38 ].

Study Selection

The total number of records retrieved by searching the predefined databases was 618 ( Figure 1 ). Of these 618 records, 161 (26.1%) duplicates were removed using the EndNote software. Checking titles and abstracts of the remaining records led to the exclusion of 52.3% (323/618). After reading the full texts of the remaining 134 publications, 116 (86.6%) were excluded, mainly because of the population (n=67, 57.8%). The list of studies that were excluded after screening the full texts is provided in Multimedia Appendix 4 . No additional studies were found through backward and forward reference list checking. In total, 18 RCTs were included in this review [ 46 - 63 ]. Of these 18 studies, 15 (83%) were included in 10 meta-analyses [ 47 - 49 , 51 , 52 , 54 - 63 ]. A total of 17% (3/18) of the studies were excluded from the meta-analyses because 33% (1/3) [ 46 ] did not report the data required for the meta-analysis (eg, mean and SD) and 67% (2/3) [ 53 , 61 ] compared serious games with other serious games that had different characteristics; therefore, including them in a meta-analysis would not make sense.

Flowchart of the study selection process.

Study Characteristics

The included studies were published between 2012 and 2021 ( Table 1 ). The year in which the largest number of included studies was published was 2015 (4/18, 22%). The included studies were carried out in 13 different countries, and there was a general equal distribution of studies in these countries. All the included studies were peer-reviewed journal articles except for a book chapter included (1/18, 6%). The trial type was parallel RCT in most of the included studies (17/18, 94%).

Characteristics of the studies and populations (N=18).

a RCT: randomized controlled trial.

b MCI: mild cognitive impairment.

c AD: Alzheimer disease.

The sample size of the included studies varied from 20 to 209, with an average of 81. The mean age of the participants in the included studies ranged from 66 to 83.1 years, with an average of 74.5 years. The percentage of men in the included studies ranged from 21.5% to 71%, with an average of 46.5%. The participants in most of the included studies had MCI (14/18, 78%). Participants were recruited from clinical settings in 67% (12/18) of the studies, from the community in 28% (5/18) of the studies, and from both clinical settings and the community in 6% (1/18) of the studies.

Serious games alone were used as interventions in 89% (16/18) of the included studies, whereas the remaining 11% (2/18) of the studies used serious games combined with conventional exercises [ 48 ] or sham exercises [ 49 ] ( Table 2 ). The included studies used 16 different serious games. On the basis of the therapeutic modality that they delivered, the serious games used in the included studies were grouped into 2 types: cognitive training games (16/18, 89%) and exergames (2/18, 11%). Games were designed with a “serious” purpose from the beginning (designed serious games) in all studies except for 6% (1/18) that used a purpose-shifted game (which was not designed as a serious game from the start but rather was used for a serious purpose). The most common platform used for playing the games were computers (14/18, 78%). In 67% (12/18) of the studies, serious games were played under the supervision of health care providers or caregivers. The duration of the games in the included studies ranged from 7 to 90 minutes, and the most common duration was 60 minutes (7/18, 39%). The frequency of playing the games varied between 2 and 7 times per week, but it was 2 times per week in half of the studies (9/18, 50%). The period of intervention ranged from 2 to 25 weeks, but it was ≤12 weeks in 72% (13/18) of the studies.

Characteristics of the interventions (N=18).

a NR: not reported.

b VR: virtual reality.

The comparison groups received only passive interventions in 39% (7/18) of the studies, whereas they received only active interventions in 44% (8/18) of the studies (eg, conventional exercises and conventional cognitive activities; Table 3 ). In total, 17% (3/18) of the studies delivered both active and passive interventions as comparators. The duration of the active comparators ranged from 7 to 100 minutes. The frequency of the active comparators varied between 2 and 7 times per week. The period of the active comparators varied between 2 and 25 weeks. Most of the included studies (16/18, 89%) measured more than one outcome. The measured outcomes were verbal memory in 78% (14/18) of the studies, nonverbal memory in 61% (11/18) of the studies, and working memory in 67% (12/18) of the studies. The studies used 32 different tools to measure these outcomes, but the most common tool used was the Wechsler Memory Scale Third Edition (7/18, 39%). The outcomes were measured immediately after the intervention in all the included studies (18/18, 100%). The follow-up period ranged from 4 to 264 weeks. Participant attrition was reported in 89% (16/18) of the studies, and it ranged from 0 to 23.

Characteristics of the comparators and outcomes (N=18).

a N/A: not applicable.

b VM: verbal memory.

c HVLT: Hopkins Verbal Learning Test.

d RAVLT: Rey Auditory Verbal Learning Test.

e RBMT: Rivermead Behavioral Memory Test.

f NR: not reported.

g ACE-R: Addenbrooke Cognitive Examination-Revised.

h NVM: nonverbal memory.

i BVRT-R: Benton Visual Retention Test-Revised, Fifth Edition.

j WMS-III-LM: Wechsler Memory Scale Third Edition-Logical Memory.

k WMS-R-VR-II: Wechsler Memory Scale-Revised-Visual Reproductions II.

l WMS-R-LM: Wechsler Memory Scale-Revised-Logical Memory.

m BSRT: Buschke Selective Reminding Test.

n WM: working memory.

o CVLT: California Verbal Learning Test.

p SBTT: spatial n-back task test.

q WMS-IV-VPA-II: Wechsler Memory Scale Fourth Edition-Verbal Paired Associates II.

r WMS-IV-SS: Wechsler Memory Scale Fourth Edition-Symbol Span.

s WAIS-DSB: Wechsler Adult Intelligence Scale-Digit Span Backwards.

t WMS-R-DSB: Wechsler Memory Scale-Revised-Digit Span Backwards.

u TSWRT: two-syllable word repetition test.

v WMS-III-FP: Wechsler Memory Scale Third Edition-Family Pictures.

w WMS-III-DST: Wechsler Memory Scale Third Edition-Digit Span Test.

x WMS-III-VSST: Wechsler Memory Scale Third Edition-Visual-Spatial Span Test.

y ROCFT: Rey-Osterrieth complex figure test.

z SVLT: Seoul Verbal Learning Test.

aa WMS-III-DSB: Wechsler Memory Scale Third Edition-Digit Span Backwards Test.

ab WMS-III-FII: Wechsler Memory Scale Third Edition-Faces II.

ac CVLT-II: California Verbal Learning Test-Second Edition.

ad WMS-III-SS: Wechsler Memory Scale Third Edition-Symbol Span.

ae WMS-III-LNS: Wechsler Memory Scale Third Edition-Letter-Number Sequencing.

af ROCFT-R: Rey-Osterrieth complex figure test-Revised.

ag BEM-WLTR: Batterie d’Efficience Mnesique-word list total recall.

ah MMSE-R: Mini-Mental State Examination-Recall.

ai SRT: source recognition task.

aj n-BT: n-back task.

ak RST: reading span task.

al LLT-R: Location Learning Test-Revised.

am WAIS-III-DS: Wechsler Adult Intelligence Scale Second Edition-Digit Span.

Risk of Bias in the Studies

An appropriate random allocation sequence for the randomization process was used in 44% (8/18) of the studies. Researchers in 39% (7/18) of the studies concealed the allocation sequence until participants were assigned to the interventions. The groups were comparable at baseline in all studies (18/18, 100%). Thus, the risk of bias owing to the randomization process was rated as low in only 33% (6/18) of the studies ( Figure 2 ).

Review authors’ judgments about each “risk of bias” domain.

Participants and those who delivered the interventions were aware of the assigned interventions during the trial in 67% (12/18) and 83% (15/18) of the studies, respectively. None of the studies reported a deviation from the intended intervention because of experimental contexts; however, 11% (2/18) of the studies provided insufficient information to verify if protocol deviations had occurred. Appropriate analysis methods (eg, intention-to-treat analysis) were used in 89% (16/18) of the studies to estimate the effect of the intervention. According to these judgments, the risk of bias because of deviations from the intended interventions was low in 78% (14/18) of the studies ( Figure 2 ).

Missing outcome data were <5% in 44% (8/18) of the studies. There was evidence that the findings were not biased by missing outcome data in only 6% (1/18) of the studies. The missing outcome data resulted from reasons that were documented and not related to the outcome in 28% (5/18) of the studies. Therefore, there was a low risk of bias because of missing outcome data in 78% (14/18) of the studies ( Figure 2 ).

In all the included studies (18/18, 100%), the outcomes of interest were evaluated using appropriate measures, and the measurement methods were comparable across the intervention groups. The assessor of the outcome was aware of the assigned interventions in 39% (7/18) of the studies, but it was unlikely that the assessment of the outcome was influenced by knowledge of the intervention received in these studies. Accordingly, all studies (18/18, 100%) had a low risk of bias in the “measuring the outcome” domain ( Figure 2 ).

In total, 28% (5/18) of the studies published their protocols in sufficient detail. In all studies (18/18, 100%), the reported outcome measurements did not differ from those specified in the analysis plan, and there was no evidence that the studies selected their results from many results produced from multiple eligible analyses of the data. On the basis of these judgments, the risk of bias because of the selection of the reported results was considered low in 28% (5/18) of the studies ( Figure 2 ).

In the last domain, “overall bias,” the risk of bias was considered high in 22% (4/18) of the studies as they were judged as having a high risk of bias in at least one domain. A total of 61% (11/18) of the studies raised some concerns in the domain of overall bias as they had some issues in at least one of the domains and were not at high risk for any domain. The remaining 17% (3/18) of the studies were judged to be at low risk of bias for the domain of overall bias given that they were rated to be at low risk of bias for all domains. The reviewers’ judgments about each “risk of bias” domain for each included study are presented in Multimedia Appendix 5 [ 46 - 63 ].

Results of the Studies

As mentioned earlier, the included studies assessed the effect of serious games on 3 outcomes: verbal, nonverbal, and working memory. The results of the included studies were divided into 3 groups based on these outcomes. Furthermore, the results for each outcome were grouped based on the comparator used in the studies (ie, control [no or passive interventions], conventional exercises, conventional cognitive activities, and other serious games).

Verbal Memory

Serious games versus control.

The effect of serious games on verbal memory was compared with that of no or passive interventions in 44% (8/18) of the studies [ 46 , 47 , 49 , 52 , 54 - 57 ]. A total of 13% (1/8) of these studies were not included in the meta-analysis given that they did not report the required data and we could not obtain them when contacting the authors. Of the 7 studies included in the meta-analysis, 2 (29%) assessed verbal memory using 2 different measures [ 55 , 57 ]. Therefore, we included the results of all these measures in the meta-analysis to form 9 comparisons ( Figure 3 [ 47 , 49 , 52 , 54 - 57 ]). The meta-analysis showed no statistically significant difference ( P =.13) in verbal memory between the serious game and control groups (SMD=0.39, 95% CI −0.11 to 0.89). The statistical heterogeneity of the evidence was considerable ( P <.001; I 2 =89.5%). The high heterogeneity may be attributed to differences in sample size, participants’ health condition, period of the intervention, and outcome measures among the studies included in this analysis. The quality of the evidence was very low as it was downgraded by 5 levels owing to a high risk of bias, heterogeneity, and imprecision ( Multimedia Appendix 6 ).

Forest plot of 7 studies (9 comparisons) comparing the effect of serious games with that of control on verbal memory. RE: random effect; SMD: standardized mean difference [ 49 , 51 , 54 , 56 - 59 ].

We conducted subgroup analyses, also known as moderator analyses [ 64 ], to investigate whether different characteristics of the population (ie, sample size, health condition, and recruitment setting) and intervention (ie, delivery method, duration, frequency, and period) moderated the effect of serious games on verbal memory. As shown in Multimedia Appendix 7 , there was no statistically significant difference among all characteristics of the population and intervention except for the health condition of the participants ( P =.003) and the period of the intervention ( P =.05).

Serious Games Versus Conventional Exercises

The effect of serious games was compared with that of conventional exercises in 17% (3/18) of the studies [ 48 , 49 , 51 ] ( Figure 4 [ 48 , 49 , 51 ]). A meta-analysis of the results of these studies showed a statistically significant difference in verbal memory ( P =.003) between the groups, favoring serious games over conventional exercises (SMD=0.46, 95% CI 0.16-0.77). This difference was also clinically important as the overall effect was outside MCID boundaries (−0.23 to 0.23) and its 95% CI did not cross the “no effect” line (zero effect). For this outcome, the MCID boundaries were calculated as –0.5 times to +0.5 times the SMD value (0.46). The statistical heterogeneity of the evidence was not a concern ( P =.34; I 2 =0%). The quality of the evidence was very low as it was downgraded by 3 levels owing to a high risk of bias and imprecision ( Multimedia Appendix 6 ).

Forest plot of 3 studies comparing the effect of serious games with that of conventional exercises on verbal memory. RE: random effect; SMD: standardized mean difference [ 50 , 51 , 53 ].

Serious Games Versus Conventional Cognitive Activities

In total, 11% (2/18) of the studies examined the effect of serious games in comparison with conventional cognitive activities [ 57 , 59 ]. These studies assessed verbal memory using 2 different measures. Thus, we included the results of all these measures in a meta-analysis to form 4 comparisons ( Figure 5 [ 57 , 59 ]). The meta-analysis showed no statistically significant difference ( P =.14) in verbal memory between the groups (SMD=0.66, 95% CI −0.21 to 1.54). The statistical heterogeneity of the evidence was substantial ( P <.001; I 2 =76.3%). The high heterogeneity may be attributed to differences in the platform of the intervention, period of the intervention, and outcome measures among the studies included in this analysis. The quality of the evidence was very low as it was downgraded by 5 levels owing to a high risk of bias, heterogeneity, and imprecision ( Multimedia Appendix 6 ).

Forest plot of 2 studies (4 comparisons) comparing the effect of serious games with that of conventional cognitive activities on verbal memory. RE: random effect; SMD: standardized mean difference [ 59 , 61 ].

Serious Games Versus Other Serious Games

In total, 17% (3/18) of the studies compared the effect of serious games on verbal memory with that of other serious games [ 50 , 53 , 58 ]. Specifically, Gooding et al [ 50 ] compared the effect of a cognitive training game that included empirically validated motivational teaching and rehabilitation techniques (BrainFitnessPlus) with 2 other games: the same previous game without the aforementioned techniques (BrainFitness) and commercially available computer games and puzzles (ie, Brain Age, Sudoku, and crossword puzzles). The study found a statistically significant difference in memory between the groups, favoring BrainFitnessPlus and BrainFitness over commercially available computer games as measured by the Buschke Selective Reminding Test-Delay (BSRT-Delay) and the Wechsler Memory Scale Third Edition-Logical Memory II (WMS-III-LM-II) and favoring BrainFitness over commercially available computer games as measured by the BSRT-Delay only. However, there was no significant difference in memory between the BrainFitnessPlus and BrainFitness groups as measured by the BSRT-Delay and the WMS-III-LM-II [ 50 ].

The second trial compared the effect of a cognitive training game with that of exergames [ 53 ]. The study found no statistically significant difference ( P =.76) in memory between the groups. The last study in this group compared the effect of a cognitive training game that adjusts the level of difficulty of the tasks based on an individual’s mastery on each level (ie, adaptive game) with the same game but without adjustment of the level of difficulty of the tasks (ie, nonadaptive game) [ 58 ]. The study showed no statistically significant difference between the groups as measured by the WMS-III-LM-II ( P =.76) and the California Verbal Learning Test Total Hits ( P =.30), but there was a statistically significant difference between the groups as measured by the California Verbal Learning Test II Long Delay Free Recall ( P =.03), favoring the adaptive game over the nonadaptive game [ 58 ].

Nonverbal Memory

The effect of serious games on nonverbal memory was compared with that of no or passive interventions in 44% (8/18) of the studies [ 49 , 54 - 57 , 60 - 62 ]. Of these 8 studies, 2 (25%) assessed nonverbal memory using 2 different measures [ 55 , 57 ]. Therefore, we included the results of all these measures in the meta-analysis to form 9 comparisons ( Figure 6 ) [ 49 , 54 - 57 , 60 - 62 ]. The meta-analysis showed a statistically significant difference ( P =.02) in nonverbal memory between the groups, favoring serious games over no or passive interventions (SMD=0.46, 95% CI 0.09-0.83). This difference was also clinically important as the overall effect was outside MCID boundaries (−0.23 to 0.23) and its CI did not cross the “no effect” line (zero effect). For this outcome, the MCID boundaries were calculated as –0.5 times to +0.5 times the SMD value (0.46). The statistical heterogeneity of the evidence was substantial ( P <.001; I 2 =80.1%). The high heterogeneity may be attributed to differences in sample sizes, participants’ health conditions, duration of the intervention, period of the intervention, platform of the intervention, and outcome measures among the studies included in this analysis. The quality of the evidence was very low as it was downgraded by 5 levels owing to a high risk of bias, heterogeneity, and imprecision ( Multimedia Appendix 8 ). Subgroup analyses showed no statistically significant difference for all characteristics of the population and intervention ( P >.05; Multimedia Appendix 9 ).

Forest plot of 8 studies (10 comparisons) comparing the effect of serious games with that of control on nonverbal memory. RE: random effect; SMD: standardized mean difference [ 51 , 56 - 59 , 62 - 64 ].

The effect of serious games on nonverbal memory was compared with that of conventional exercises in 11% (2/18) of the studies [ 49 , 62 ]. As shown in Figure 7 [ 49 , 62 ], there was no statistically significant difference ( P =.30) in nonverbal memory between the groups (SMD=−0.19, 95% CI −0.54 to 0.17). The statistical heterogeneity of the evidence was not a concern ( P =.90; I 2 =0%). The quality of the evidence was very low as it was downgraded by 3 levels owing to a high risk of bias and imprecision ( Multimedia Appendix 8 ).

Forest plot of 2 studies comparing the effect of serious games with that of conventional exercises on nonverbal memory. RE: random effect; SMD: standardized mean difference [ 51 , 64 ].

The effect of serious games on nonverbal memory was compared with that of conventional cognitive activities in 11% (2/18) of the studies [ 57 , 59 ]. Of these 2 studies, 1 (50%) assessed nonverbal memory using 2 different measures [ 57 ]. Therefore, we included the results of all these measures in the meta-analysis to form 3 comparisons ( Figure 8 [ 57 , 59 ]). The meta-analysis showed no statistically significant difference ( P =.94) in nonverbal memory between the groups (SMD=−0.01, 95% CI −0.32 to 0.30). The statistical heterogeneity of the evidence was not a concern ( P =.74; I 2 =0%). The quality of the evidence was very low as it was downgraded by 4 levels owing to a high risk of bias and imprecision ( Multimedia Appendix 8 ).

Forest plot of 2 studies (3 comparisons) comparing the effect of serious games with that of conventional cognitive activities on nonverbal memory. RE: random effect; SMD: standardized mean difference [ 59 , 61 ].

In total, 17% (3/18) of the studies compared the effect of serious games on nonverbal memory with that of other serious games [ 50 , 58 , 61 ]. Specifically, Gooding et al [ 50 ] compared the effect of BrainFitnessPlus with that of BrainFitness and commercially available computer games. The study showed no statistically significant difference in memory between any 2 of these groups [ 50 ].

The second study compared the effect of an adaptive serious game with that of a nonadaptive serious game [ 58 ]. The study showed no statistically significant difference in nonverbal memory between the groups as measured by the Rey-Osterrieth complex figure test-delayed recall ( P =.25) and the Wechsler Memory Scale Third Edition-Faces II ( P =.61) [ 58 ].

The last study in this group assessed the effect of 2 cognitive training games [ 61 ]. Both games consisted of a study and a test phase. In each session of the study phase, both games asked participants to read and remember 16 words presented one at a time on a computer screen for 3 seconds followed by a 1-second white screen [ 61 ]. In the test phase, participants were asked to recognize the 16 study words, which were mixed with 16 new words in the first game (recollection training game) and 32 new words in the second game (recognition practice game) [ 61 ]. The study showed no statistically significant difference ( P =.17) in nonverbal memory between the 2 groups [ 61 ].

Working Memory

The effect of serious games on working memory was compared with that of control (no or passive interventions) in 39% (7/18) of the studies [ 52 , 54 - 57 , 61 , 62 ]. Of these 7 studies, 4 (57%) assessed working memory using more than one measure [ 54 , 55 , 61 , 62 ]. Therefore, we included the results of all these measures in the meta-analysis to form 13 comparisons ( Figure 9 ) [ 52 , 54 - 57 , 61 , 62 ]. The meta-analysis showed a statistically significant difference ( P =.04) in working memory between the groups, favoring serious games over no or passive interventions (SMD=0.31, 95% CI 0.01-0.60). This difference was also clinically important as the overall effect was outside MCID boundaries (−0.16 to 0.16) and its CI did not cross the “no effect” line (zero effect). For this outcome, the MCID boundaries were calculated as –0.5 times to +0.5 times the SMD value (0.31). The statistical heterogeneity of the evidence was substantial ( P <.001; I 2 =78.3%). The high heterogeneity may be attributed to differences in sample sizes, percentage of men, participants’ health conditions, duration of the intervention, period of the intervention, and outcome measures among the studies included in this analysis. The quality of the evidence was very low as it was downgraded by 5 levels owing to a high risk of bias, heterogeneity, and imprecision ( Multimedia Appendix 10 ). Subgroup analyses showed no statistically significant difference for all characteristics of the population and intervention ( P >.05; Multimedia Appendix 11 ).

Forest plot of 7 studies (13 comparisons) comparing the effect of serious games with that of control on working memory. RE: random effect; SMD: standardized mean difference [ 54 , 56 - 59 , 63 , 64 ].

The effect of serious games on working memory was compared with that of conventional exercise in 11% (2/18) of the studies [ 51 , 62 ]. Both studies assessed working memory using 2 different measures. Thus, we included the results of all these measures. As shown in Figure 10 [ 51 , 62 ], there was no statistically significant difference ( P =.99) in working memory between the serious game and conventional exercise groups (SMD=0.00, 95% CI −0.45 to 0.45). The statistical heterogeneity of the evidence was moderate ( P =.10; I 2 =50.9%). The quality of the evidence was very low as it was downgraded by 5 levels owing to a high risk of bias and imprecision ( Multimedia Appendix 10 ).

Forest plot of 2 studies (4 comparisons) comparing the effect of serious games with that of conventional exercises on working memory. RE: random effect; SMD: standardized mean difference [ 53 , 64 ].

The effect of serious games on working memory was compared with that of conventional cognitive activities in 11% (2/18) of the studies [ 57 , 59 ] ( Figure 11 [ 57 , 59 ]). A meta-analysis of the results of these studies showed no statistically significant difference ( P =.08) in working memory between the groups (SMD=0.37, 95% CI −0.05 to 0.78). The statistical heterogeneity of the evidence was not a concern ( P =.65; I 2 =0%). The quality of the evidence was very low as it was downgraded by 3 levels owing to a high risk of bias and imprecision ( Multimedia Appendix 10 ).

Forest plot of 2 studies comparing the effect of serious games with that of conventional cognitive activities on working memory. RE: random effect; SMD: standardized mean difference [ 59 , 61 ].

The effect of serious games on working memory was compared with that of other serious games in 22% (4/18) of the studies [ 53 , 58 , 61 , 63 ]. Specifically, the first study compared the effect of a cognitive training game with that of exergames [ 53 ]. The study found a statistically significant difference ( P <.001) in memory between the groups, favoring cognitive training games over exergames [ 53 ]. The second study assessed the effect of 2 cognitive training games on working memory: a recollection training game and a recognition practice game [ 61 ]. The study showed no statistically significant difference in working memory between the 2 groups as measured by the n-back task ( P =.78) and reading span task ( P =.76) [ 61 ].

The remaining 50% (2/4) of the studies compared the effect of adaptive serious games with that of nonadaptive serious games [ 58 , 63 ]. Of the 2 studies, 1 (50%) assessed working memory using 4 different measures [ 58 ], whereas the other study (50%) used 2 different measures to do so [ 63 ]. Hence, we included the results of all these measures in the meta-analysis to form 6 comparisons. As shown in Figure 12 [ 58 , 63 ], there was no statistically significant difference ( P =.08) in working memory between adaptive serious games and nonadaptive serious games (SMD=0.18, 95% CI −0.02 to 0.37). The statistical heterogeneity of the evidence was not a concern ( P =.99; I 2 =0%). The quality of the evidence was low as it was downgraded by 2 levels owing to a high risk of bias and imprecision ( Multimedia Appendix 10 ).

Forest plot of 2 studies (6 comparisons) comparing the effect of adaptive serious games with that of nonadaptive serious games on working memory. RE: random effect; SMD: standardized mean difference [ 60 , 65 ].

Principal Findings

This study summarized the evidence regarding the effectiveness of serious games in improving memory. Our meta-analyses showed that serious games are more effective than no or passive interventions in improving nonverbal and working memory. Surprisingly, we found that serious games are as effective as no or passive interventions in improving verbal memory, which, therefore, deems serious games ineffective. This review demonstrated that serious games are more effective than conventional exercises in improving verbal memory. However, we found that serious games are as effective as conventional exercises in improving nonverbal and working memory, indicating that serious games are comparable with conventional exercises. Evidence suggests that cognitive training and exercise work through distinct neuronal mechanisms and, therefore, if combined, they might have synergistic and more effective results compared with being used as separate interventions [ 65 , 66 ]. Studying this synergistic relationship will become important in future primary research and trials. With the advances in virtual reality technologies, their availability, and rising applications of the metaverse [ 67 ], more evidence is needed to assess the effectiveness of virtual reality–based exergames in improving memory [ 68 ].

The meta-analyses in this review showed that serious games are as effective as conventional cognitive training in improving verbal, nonverbal, and working memory, meaning that serious games and conventional cognitive training are comparable. Furthermore, we found that the effect of adaptive serious games is similar to that of nonadaptive serious games in improving working memory.

The findings of our review and previous reviews were consistent for some outcomes and different for others. Specifically, a systematic review conducted by Lampit et al [ 32 ] compared the effect of cognitive training games with that of passive and active interventions on verbal, nonverbal, and working memory in healthy older adults. Consistent with our findings, the review found no statistically significant difference ( P >.05) in the effect of cognitive training games and of no or passive interventions on verbal memory, and there was a statistically significant difference in working memory between the groups, favoring cognitive training games over no or passive interventions [ 32 ]. In contrast to our findings, Lampit et al [ 32 ] did not find a statistically significant difference in nonverbal memory between the groups. The contrary finding may be attributed to the following reasons: (1) although the number of participants was ≥100 in 17% (3/18) of the studies in our meta-analysis, the number of participants was <100 in all studies included in the meta-analysis by Lampit et al [ 32 ]; (2) the total number of training hours was >15 in only 2 studies included in the meta-analysis by Lampit et al [ 32 ], whereas the total number of training hours was >15 in 28% (5/18) of the studies in our meta-analysis; and (3) all studies meta-analyzed in our review (15/18, 83%) recruited participants with cognitive impairment, whereas all studies meta-analyzed in the review by Lampit et al [ 32 ] recruited participants without cognitive impairment.

In contrast to our findings, Lampit et al [ 32 ] found a statistically significant difference in verbal, nonverbal, and working memory between the groups, favoring cognitive training games over active interventions [ 32 ]. This may be attributed to the following reasons: (1) our findings related to these comparisons are based on meta-analyses of 11% (2/18) of the studies, whereas the findings of Lampit et al [ 32 ] are based on meta-analyses of 6 to 15 studies; (2) Lampit et al [ 32 ] compared cognitive training games with all active interventions, but this review compared cognitive training games with specific active interventions (ie, conventional exercises and conventional cognitive training); (3) all studies meta-analyzed in our review (15/18, 83%) recruited participants with cognitive impairment, whereas all studies meta-analyzed in the review by Lampit et al [ 32 ] recruited participants without cognitive impairment; and (4) the review by Lampit et al [ 32 ] included some pilot RCTs, whereas our review included only RCTs.

Another review examined the effect of cognitive training games on verbal and working memory among healthy older adults regardless of the comparator type (ie, passive and active controls) [ 30 ]. Meta-analyses in that review showed a statistically significant difference in verbal memory ( P =.03) and working memory ( P <.001) between the groups, favoring cognitive training games over all types of comparators [ 30 ].

Hill et al [ 34 ] conducted a systematic review to assess the effect of cognitive training games on verbal, nonverbal, and working memory among people with MCI or dementia regardless of the comparator type. For people with MCI, the review found a statistically significant difference between the groups in verbal and working memory ( P <.001), favoring all types of comparators [ 34 ]. In contrast, there was no statistically significant effect of cognitive training games on nonverbal memory when compared with all types of comparators [ 34 ]. For people with dementia, the review showed no statistically significant effect of cognitive training games on verbal, nonverbal, or working memory when compared with all types of comparators [ 34 ].

Research and Practical Implications

Research implications.

Given that the review focused on memory among older adults with cognitive impairment, future reviews should assess the effectiveness of serious games on other cognitive functions (eg, learning, language, executive function, and processing speed) in young and older adults with or without cognitive impairment. In this review, a few studies (≤3) were included in the meta-analyses that compared serious games with active interventions (ie, conventional exercises, conventional cognitive training, and nonadaptive serious games); therefore, our findings regarding these comparisons remain inconclusive. Thus, there is a pressing need to conduct more studies to compare the effect of serious games on memory with active interventions.

Most studies in this review (12/18, 67%) were carried out in clinical settings, thus offering the researchers more control over the experiments. However, the participants may have been stressed by playing these games outside the environment that they were used to. Therefore, more studies should be conducted in the community and home settings, allowing participants to be at ease and enabling the researchers to examine other factors that could come into play, such as environmental conditions (eg, room temperature and lighting).

In this review, the long-term effect of serious games was not assessed as few studies reported follow-up data, and the follow-up period was not consistent among the studies. Further studies should assess the long-term effect of serious games on memory. Most of the included studies (15/18, 83%) did not report the mean and SD of pre-post intervention change in memory for each group. Researchers should report this information to accurately calculate effect sizes.

Future studies should also examine and compare the effectiveness of playing serious games in multiplayer mode with other members of the family or community as this has not been assessed in previous studies. We urge researchers to conduct and report RCTs following recommended guidelines or tools (eg, RoB 2 [ 39 ]) to avoid the biases identified in this review.

Practical Implications

This review shows that serious games can be effective in improving verbal, nonverbal, and working memory. However, these findings should be interpreted cautiously given that most meta-analyses were based on a few studies (≤3) and judged to have a low quality of evidence for the following reasons: most of the included studies (11/18, 61%) were judged to have some concerns regarding the overall bias, the heterogeneity of the evidence was high in approximately half of the meta-analyses (4/10, 40%), and the total effect sizes were imprecise in all meta-analyses (10/10, 100%). On the basis of our review findings, serious games are still not ready as substitutes for real-world interactions and experiences; they should still be used as a supplement rather than an alternative method for interventions targeting the improvement of verbal, nonverbal, and working memory until more evidence suggests otherwise.

Despite the ubiquity and availability of smart mobile devices (ie, tablets and smartphones), only 6% (1/18) of the included studies used them [ 60 ]. Mobile devices can be more pervasive and accessible than PCs or commercially available gaming consoles. Studies estimate that, in 2021 alone, approximately 15 billion mobile devices exist worldwide and are used by >7.1 billion users [ 69 ]; this is expected to rise. Game and app developers should invest in creating serious games on mobile devices that target improving verbal, nonverbal, and working memory.

Limitations

This review cannot comment on the effectiveness of serious games (1) delivered on nondigital platforms, (2) used for other purposes (eg, screening or diagnosis), (3) used for improving other cognitive abilities (eg, learning, processing speed, and executive functions), (4) among other age groups, or (5) among those without cognitive impairment. This is because such interventions, outcomes, and populations were beyond the scope of this review.

It is likely that we missed some relevant studies as this review did not search some databases (eg, PubMed and the Cochrane Library [CENTRAL]) and excluded studies that were quasi-experiments, pilot RCTs, published before 2010, or written in non-English languages. The quality of the evidence was very low in all meta-analyses except for 10% (1/10); this may decrease the internal validity of our findings. We cannot comment on the long-term effect of serious games on memory as this review focused on the short-term effect of serious games by meta-analyzing only postintervention data rather than follow-up data. This is because the follow-up period was not consistent among the studies.

The effect size for each meta-analyzed study was likely overestimated or underestimated in this review given that the authors used postintervention data for each group to assess the effect size rather than the pre-post intervention change for each group. Postintervention outcome data were used as most studies (15/18, 83%) did not report the mean and SD for pre-post intervention change in memory for each group, and there was no statistically significant difference in memory between the groups at baseline in all studies (18/18, 100%).

Serious games may have a significant role to play in improving verbal, nonverbal, and working memory in older adults with cognitive impairment. However, these findings should be treated with caution given that most meta-analyses (7/10, 70%) were based on a few studies (≤3) and judged to have a low quality of evidence for the following reasons: most of the included studies (11/18, 61%) were judged to have some concerns regarding the overall bias, the heterogeneity of the evidence was high in approximately half of the meta-analyses (4/10, 40%), and the total effect sizes were imprecise in all meta-analyses (10/10, 100%). Therefore, serious games should be offered as a supplement to existing proven and safe interventions rather than as a complete substitute until further, more robust evidence is available. Further reviews are necessary to investigate the short- and long-term effect of serious games on memory and other cognitive abilities (eg, executive function, processing speed, and learning) among people of different age groups with or without cognitive impairment.

Abbreviations

Multimedia Appendix 1

Multimedia appendix 2, multimedia appendix 3, multimedia appendix 4, multimedia appendix 5, multimedia appendix 6, multimedia appendix 7, multimedia appendix 8, multimedia appendix 9, multimedia appendix 10, multimedia appendix 11.

Conflicts of Interest: None declared.