Predicting students' knowledge after playing a serious game based on learning analytics data: A case study

Serious games have proven to be a powerful tool in education to engage, motivate, and help students learn. However, the change in student knowledge after playing games is usually measured with traditional (paper) prequestionnaires–postquestionnaires. We propose a combination of game learning analytics and data mining techniques to predict knowledge change based on in-game student interactions. We have tested this approach in a case study for which we have conducted preexperiments–postexperiments with 227 students playing a previously validated serious game on first aid techniques. We collected student interaction data while students played, using a game learning analytics infrastructure and the standard data format Experience API for Serious Games. After data collection, we developed and tested prediction models to determine whether knowledge, given as posttest results, can be accurately predicted. Additionally, we compared models both with and without pretest information to determine the importance of previous knowledge when predicting postgame knowledge. The high accuracy of the obtained prediction models suggests that serious games can be used not only to teach but also to measure knowledge acquisition after playing. This will simplify serious games application for educational settings and especially in the classroom easing teachers' evaluation tasks.     

Assessing the educational values of digital games

Abstract   In recent years, digital games have assumed an important place in the lives of children and adolescents. Effective content design is crucial to the success of digital game-based learning. Therefore, the tool for assessing the effectiveness of game design is accordingly very important for parents and teachers, so that they may encourage or discourage students to play. The purpose of this study is to develop an assessment tool to examine the educational values of digital games. In the first phase of this research, the research team developed the indices for assessing the educational values of digital games. An expert panel consisting of game scholars and professional game designers was established to construct the indices for evaluating digital games in three focus group discussions. Seventy-four game evaluation indices were sorted into seven categories: mentality change, emotional fulfilment, knowledge enhancement, thinking skill development, interpersonal skill development, spatial ability development and bodily coordination. In the second phase of the research, the game designers were asked to assess certain games by using the 74 indices. Meanwhile, the game scholars were also asked to evaluate the same pool of games by the same indices. The assessments by both the scholars and designers were then compared and the similarities were found. This research provided a preliminary framework for future game designers, parents and teachers in assessing educational values of digital games.     

The effectiveness of adaptive versus non-adaptive learning with digital educational games

For the training of academic skills, digital educational games with integrated adaptivity are promising. Adaptive games are considered superior to non-adaptive games, because they constantly assess children's performance, and accordingly adapt the difficulty of the tasks corresponding to the children's individual level. However, empirical evidence with regard to the effectivity of adaptive compared to non-adaptive games is limited. A study was conducted with 191 children from the third year of Kinder garten who were enrolled in one of three conditions, that is, playing an adaptive version of the reading game (RG), a non-adaptive version of the RG or training with pen-and-paper exercises. In all three conditions, children trained emergent reading (phonological awareness and letter knowledge) once a week for 30 min over a period of 5 weeks. Children's performance on cognitive (phonological awareness, letter knowledge, reading fluency) and non-cognitive (motivation, self-concept) factors was assessed. Results revealed a significant improvement in phonological awareness and letter knowledge in all conditions. However, no differences between the conditions were observed with respect to children's improvement on phonological awareness and letter knowledge or on their post-test scores for reading fluency. With regard to motivation and self-concept, again, no differences in these non-cognitive factors were observed across conditions.     

Machine learning in digital games: a survey

Artificial intelligence for digital games constitutes the implementation of a set of algorithms and techniques from both traditional and modern artificial intelligence in order to provide solutions to a range of game dependent problems. However, the majority of current approaches lead to predefined, static and predictable game agent responses, with no ability to adjust during game-play to the behaviour or playing style of the player. Machine learning techniques provide a way to improve the behavioural dynamics of computer controlled game agents by facilitating the automated generation and selection of behaviours, thus enhancing the capabilities of digital game artificial intelligence and providing the opportunity to create more engaging and entertaining game-play experiences. This paper provides a survey of the current state of academic machine learning research for digital game environments, with respect to the use of techniques from neural networks, evolutionary computation and reinforcement learning for game agent control.     

An activity theory-based model for serious games analysis and conceptual design

There are currently a number of models, frameworks and methodologies for serious games analysis and design that provide useful interpretations of the possibilities and limitations offered by serious games. However, these tools focus mostly on high-level aspects and requirements and do not help understand how such high-level requirements can be concretely satisfied. In this paper, we present a conceptual model, called Activity Theory-based Model of Serious Games (ATMSG), that aims to fill this gap. ATMSG supports a systematic and detailed representation of educational serious games, depicting the ways that game elements are connected to each other throughout the game, and how these elements contribute to the achievement of the desired pedagogical goals. Three evaluation studies indicate that ATMSG helped participants, particularly those with gaming experience, identify and understand the roles of each component in the game and recognize the game's educational objectives.     

Supporting conceptual understanding of the associative and distributive properties through digital gameplay

There has been a steady rise in the support for games as learning environments. This support is largely based on the strong levels of engagement and motivation observed during gameplay. What has proven difficult is the ability to consistently design and develop learning games that are both engaging and educationally viable. Those in the game‐based learning community contend the solution is to tightly integrate instructional objective(s) within the game's mechanics. In order to investigate whether or not intrinsic integration would aid in game design, an experimental study was conducted with 111 fourth and fifth graders in which three versions of a game were designed in order to determine which approach would have a more significant impact on conceptual understanding of the associative and distributive properties. It was found that those who played the intrinsic version of the game (properties mapped to game mechanics) had significant learning gains compared with those who played the extrinsic version. Perceptions of the game are also discussed, along with future directions.     

Exploring students' flow experiences in business simulation games

Business simulation games are a motivational and engaging tool for teaching business management. However, relatively little is known about what factors contribute to their success. This study explores the role of flow experienced while using business simulation games. Specifically, this research investigates the influence of challenge, skills, feedback, and goal clarity on students' flow experience. It also explores the impact of flow on generic skills development, perceived learning, and satisfaction. Based on a survey of 167 undergraduate students who used a classroom‐based business simulation game, the findings show that the challenge provided by the game, students' skills to meet the challenge, and instant feedback positively influenced students' flow. Conversely, goal clarity did not foster the optimal experience. Flow positively impacted generic skills development, perceived learning, and satisfaction.     

Maximizing learning without sacrificing the fun: Stealth assessment,adaptivity and learning supports in educational games

In this study, we investigated the validity of a stealth assessment of physics understanding in an educational game, as well as the effectiveness of different game-level delivery methods and various in-game supports on learning. Using a game called Physics Playground, we randomly assigned 263 ninth- to eleventh-grade students into four groups: adaptive, linear, free choice and no-treatment control. Each condition had access to the same in-game learning supports during gameplay. Results showed that: (a) the stealth assessment estimates of physics understanding were valid—significantly correlating with the external physics test scores; (b) there was no significant effect of game-level delivery method on students' learning; and (c) physics animations were the most effective (among eight supports tested) in predicting both learning outcome and in-game performance (e.g. number of game levels solved). We included student enjoyment, gender and ethnicity in our analyses as moderators to further investigate the research questions.     

Supporting teachers' reflection and learning through structured digital teaching portfolios

Digital teaching portfolios have been proposed as an effective tool for teacher learning and professional development, but there is a lack of empirical evidence supporting their effectiveness. This study proposed the design of a structured digital portfolio equipped with multiple aids (e.g. self-assessment, peer assessment, discussion and journal writing) for the professional development of teachers. This study also empirically evaluated the reflection and professional development as demonstrated in digital teaching portfolios with multiple supporting measures. Forty-four in-service substitute teachers participated in a course of classroom assessment and used a Web-based portfolio system. Based on the framework of teacher reflective thinking developed by Sparks-Langer et al. , we found that most teachers demonstrated moderate levels of reflection in their journals but only one-third of them showed the highest level of reflection. We also found that the professional knowledge of teachers about classroom assessment – as shown by their implementation of it – improved significantly during the construction of portfolios. The above findings also represent good evidence that digital portfolios with multiple aids are beneficial to teacher reflection and professional development.     

Supporting digitally enhanced learning through measurement in higher education: Development and validation of a university students' digital competence scale

Digital competence is critical for university students to adapt to and benefit from digitally enhanced learning. Prior studies on its measurement mostly focus on educators and relied on factor analyses. However, there is a lack of valid and convenient tools to measure university students' digital competence. This study aimed to develop a digital competence scale for university students (DC-US) in digitally enhanced learning with robust psychometric properties. An initial DC-US with 23 items was proposed to measure the single latent trait of digital competence. It was validated and refined continuously through a pilot study, a main study and a predictive validity study in three datasets involving 825 participants altogether, using factor analyses, Rasch analyses and the partial least squares modelling. The final DC-US turned out to comprise two subscales: technical literacy and digital skills, with 10 items retained, and manifested high internal consistency, unidimensionality and measurement invariance. The scale also demonstrated strong predictive validity, with technical literacy greatly predicting digital skills, which negatively predicted technostress. The DC-US enables instructors and school administrators to conveniently obtain preliminary information of university students' digital competence, informing their digital class preparation and development of timely interventions for addressing digital deficiencies.     

Assessing implicit science learning in digital games

Building on the promise shown in game-based learning research, this paper explores methods for Game-Based Learning Assessments (GBLA) using a variety of educational data mining techniques (EDM). GBLA research examines patterns of behaviors evident in game data logs for the measurement of implicit learning—the development of unarticulated knowledge that is not yet expressible on a test or formal assessment. This paper reports on the study of two digital games showing how the combination of human coding with EDM has enabled researchers to measure implicit learning of Physics. In the game Impulse, researchers combined human coding of video with educational data mining to create a set of automated detectors of students' implicit understanding of Newtonian mechanics. For Quantum Spectre, an optics puzzle game, human coding of Interaction Networks was used to identify common student errors. Findings show that several of our measures of student implicit learning within these games were significantly correlated with improvements in external postassessments. Methods and detailed findings were different for each type of game. These results suggest GBLA shows promise for future work such as adaptive games and in-class, data-driven formative assessments, but design of the assessment mechanics must be carefully crafted for each game.     

Customization in educational computer games and its effect on learning: Experimental study with primary school children

The instructional effects of customization features in child learning games have rarely been examined. This value-added study addresses the existing gap with regards to user-initiated cosmetic customization of environment elements (i.e., non-avatar customization). Participants (N = 143; M_age = 9.41) studied a biological topic for about 20 min: either using the experimental version of a learning game with customization features, or from a control version without them. Null results were found as concerns between-group differences: both for motivation-related variables and learning outcome measures. These findings indicate that user-initiated cosmetic customization features can be omitted by game designers, especially in settings where children are assigned specific instructional materials from which to study.     

Game immersion experience: its hierarchical structure and impact on game‐based science learning

Many studies have shown the positive impact of serious educational games (SEGs) on learning outcomes. However, there still exists insufficient research that delves into the impact of immersive experience in the process of gaming on SEG‐based science learning. The dual purpose of this study was to further explore this impact. One purpose was to develop and validate an innovative measurement, the Game Immersion Questionnaire (GIQ), and to further verify the hierarchical structure of game immersion by construct validity approaches, including exploratory factor analysis (EFA) (n = 257) and confirmatory factor analysis (CFA) (n = 1044). The second purpose was to investigate the impact of game immersion on science learning through SEG play (n = 260). Overall, the results supported the internal structure of the GIQ with good reliability and validity, and the inter factor bivariate correlations for each construct indicated a high internal consistency. Players did learn from playing an SEG, and game immersion experience did lead to higher gaming performance. Moreover, players' gaming performance plays a role in mediating the effect of immersion on science learning outcomes through SEG play. However, as players became more emotionally and subjectively attached to the game, the science learning outcomes were not definitively reliable.     

Implicit learning as a design strategy for learning games: Alert Hockey

Concussion education and prevention for youth hockey players has been an issue of recent concern amongst sport medicine practitioners and hockey’s administrative bodies. This article details the assessment of a sports-action hockey video game that aims to reduce the aggressive and negligent behaviours that can lead to concussions. The game, termed Alert Hockey, was designed to modify game playing behaviour by embedding an implicit teaching mechanism within the gameplay. In Alert Hockey, participants were expected to learn by simply playing to win, in contrast to playing to learn. We studied learning in an experimental simulated environment where the possibility to win the game was exaggerated as a consequence of desirable safety behaviours (positive learning group) and effectively reduced as a consequence of undesirable (negative learning group) behaviour. The positive learning group significantly improved their mean score on a composite behavioural indicator compared with no significant change amongst control group participants. The results demonstrate that implicit learning embedded in a sports-action game can lead to changes in game-play behaviour.     

Playing monotone games to understand learning behaviors

We deal with a special class of games against nature which correspond to subsymbolic learning problems where we know a local descent direction in the error landscape but not the amount gained at each step of the learning procedure. Namely, Alice and Bob play a game where the probability of victory grows monotonically by unknown amounts with the resources each employs. For a fixed effort on Alice’s part Bob increases his resources on the basis of the results of the individual contests (victory, tie or defeat). Quite unlike the usual ones in game theory, his aim is to stop as soon as the defeat probability goes under a given threshold with high confidence. We adopt such a game policy as an archetypal remedy to the general overtraining threat of learning algorithms. Namely, we deal with the original game in a computational learning framework analogous to the Probably Approximately Correct formulation. Therein, a wise use of a special inferential mechanism (known as twisting argument) highlights relevant statistics for managing different trade-offs between observability and controllability of the defeat probability. With similar statistics we discuss an analogous trade-off at the basis of the stopping criterion of subsymbolic learning procedures. As a conclusion, we propose a principled stopping rule based solely on the behavior of the training session, hence without distracting examples into a test set.     

A meta-analysis of research on digital game-based science learning

This meta-analysis investigates the relative effectiveness of game-based science learning against other instructional methods (Gameplay design) as well as against science game variants enriched with mechanisms (Game-mechanism design). An overall medium effect size for Gameplay design (k = 14, N_es = 14, g_RE = 0.646, p = .000), and an overall small-to-medium effect size for Game-mechanism design (k = 12, N_es = 13, adjusted g_RE = 0.270, p = .001) are reported. Further, the results of subgroup analyses suggest that students across educational levels all significantly benefit from game-based science learning although there is no significant difference between the subgroup mean effects. Further, learning and gaming mechanisms play equal roles significantly increasing students' scientific knowledge gains. With these promising results, however, high variance within the subgroups of educational levels and those of gaming mechanisms indicate that gaming mechanisms should be developed with care to meet students' different needs in different educational levels.     

Exploring selfish reinforcement learning in repeated games with stochastic rewards

In this paper we introduce a new multi-agent reinforcement learning algorithm, called exploring selfish reinforcement learning (ESRL). ESRL allows agents to reach optimal solutions in repeated non-zero sum games with stochastic rewards, by using coordinated exploration. First, two ESRL algorithms for respectively common interest and conflicting interest games are presented. Both ESRL algorithms are based on the same idea, i.e. an agent explores by temporarily excluding some of the local actions from its private action space, to give the team of agents the opportunity to look for better solutions in a reduced joint action space. In a latter stage these two algorithms are transformed into one generic algorithm which does not assume that the type of the game is known in advance. ESRL is able to find the Pareto optimal solution in common interest games without communication. In conflicting interest games ESRL only needs limited communication to learn a fair periodical policy, resulting in a good overall policy. Important to know is that ESRL agents are independent in the sense that they only use their own action choices and rewards to base their decisions on, that ESRL agents are flexible in learning different solution concepts and they can handle both stochastic, possible delayed rewards and asynchronous action selection. A real-life experiment, i.e. adaptive load-balancing of parallel applications is added.     

The effects of computer games on primary school students’ achievement and motivation in geography learning

The implementation of a computer game for learning about geography by primary school students is the focus of this article. Researchers designed and developed a three-dimensional educational computer game. Twenty four students in fourth and fifth grades in a private school in Ankara, Turkey learnt about world continents and countries through this game for three weeks. The effects of the game environment on students’ achievement and motivation and related implementation issues were examined through both quantitative and qualitative methods. An analysis of pre and post achievement tests showed that students made significant learning gains by participating in the game-based learning environment. When comparing their motivations while learning in the game-based learning environment and in their traditional school environment, it was found that students demonstrated statistically significant higher intrinsic motivations and statistically significant lower extrinsic motivations learning in the game-based environment. In addition, they had decreased focus on getting grades and they were more independent while participating in the game-based activities. These positive effects on learning and motivation, and the positive attitudes of students and teachers suggest that computer games can be used as an ICT tool in formal learning environments to support students in effective geography learning.