Machine learning in digital games: a survey

Artificial intelligence for digital games constitutes the implementation of a set of algorithms and techniques from both traditional and modern artificial intelligence in order to provide solutions to a range of game dependent problems. However, the majority of current approaches lead to predefined, static and predictable game agent responses, with no ability to adjust during game-play to the behaviour or playing style of the player. Machine learning techniques provide a way to improve the behavioural dynamics of computer controlled game agents by facilitating the automated generation and selection of behaviours, thus enhancing the capabilities of digital game artificial intelligence and providing the opportunity to create more engaging and entertaining game-play experiences. This paper provides a survey of the current state of academic machine learning research for digital game environments, with respect to the use of techniques from neural networks, evolutionary computation and reinforcement learning for game agent control.     

Game immersion experience: its hierarchical structure and impact on game‐based science learning

Many studies have shown the positive impact of serious educational games (SEGs) on learning outcomes. However, there still exists insufficient research that delves into the impact of immersive experience in the process of gaming on SEG‐based science learning. The dual purpose of this study was to further explore this impact. One purpose was to develop and validate an innovative measurement, the Game Immersion Questionnaire (GIQ), and to further verify the hierarchical structure of game immersion by construct validity approaches, including exploratory factor analysis (EFA) (n = 257) and confirmatory factor analysis (CFA) (n = 1044). The second purpose was to investigate the impact of game immersion on science learning through SEG play (n = 260). Overall, the results supported the internal structure of the GIQ with good reliability and validity, and the inter factor bivariate correlations for each construct indicated a high internal consistency. Players did learn from playing an SEG, and game immersion experience did lead to higher gaming performance. Moreover, players' gaming performance plays a role in mediating the effect of immersion on science learning outcomes through SEG play. However, as players became more emotionally and subjectively attached to the game, the science learning outcomes were not definitively reliable.     

The format of problem representation for in‐game learning supports

A promising method to support game‐based learning is to facilitate learners' externalization of cognitive and metacognitive processes. Externalizing Problem Representation (EPR) refers to a cognitive behaviour in which a learner constructs her own representations overtly. The purpose of this study is to investigate whether learning supports promoting EPR enhance qualitative understanding and quantitative proficiency in ratios and proportional relationships in a learning game (i.e., E‐Rebuild) context. Specifically, this study investigated the effects of representation format in problem representation on qualitative understanding and quantitative proficiency in a learning game context. The results of this study indicate that (a) symbolic learning supports better facilitate comprehension of math concepts and their relations than iconic learning supports in video game contexts, (b) symbolic learning supports better facilitate players' reflection for implicit understanding and promote their math problem‐solving skills, (c) participants in the symbolic learning support group increased significantly in qualitative understanding but not in quantitative proficiency after gameplay, and (d) participants in the iconic learning support group experienced significant growth in quantitative proficiency but not in qualitative understanding after gameplay.     

Evaluation of serious game for changing students' behaviour in bullying situation

The aim of the study was to test the effectiveness of a serious game in improving the helping behaviours of bystanders in bullying situations by using a multimethod approach. Students (N=345, aged 12 to 14) participated in an experimental study with one experimental and two control groups. Before and after playing the game, all students completed an online questionnaire about their knowledge, experiences and behaviours related to bullying, and their behaviours in everyday social situations. For the experimental group, players' reactions during game-play were recorded. Results showed that students' social behaviour displayed before gaming experience determined their playing activities. Besides, most of the students who declared in pre-test that they do not help the victim, although they think they should, chose helping behaviour in all bullying situations in by stander role in the game. Results indicate weak positive effect of the game on students' behavioural choices.     

Cognitive consequences of making computer-based learning activities more game-like

Some students (base group) played the Circuit Game, a 10-level computer-based learning activity intended to help students learn how electrical circuits work. Other students (competition group) played the same game but with competition features added - including a score bar showing performance on each level, the opportunity to earn one ticket per level if a performance criterion is met, and the opportunity to win a prize based on the number of tickets earned. On a retention test given after the game, the competition group remembered significantly more than the base group (d = 0.47). On an embedded transfer test constituting the final level of the game, the groups did not differ significantly. However, on the transfer test there was a significant gender by group interaction in which men performed worse in the competition group than the base group (d = −0.54) and women performed better in the competition group than the base group (d = 0.24). Overall, adding game-like features to a computer-based learning activity caused students to pay attention to game details but did not motivate students - particularly men - to learn more deeply.     

Reaching pareto-optimality in prisoner’s dilemma using conditional joint action learning

We consider the learning problem faced by two self-interested agents repeatedly playing a general-sum stage game. We assume that the players can observe each other’s actions but not the payoffs received by the other player. The concept of Nash Equilibrium in repeated games provides an individually rational solution for playing such games and can be achieved by playing the Nash Equilibrium strategy for the single-shot game in every iteration. Such a strategy, however can sometimes lead to a Pareto-Dominated outcome for games like Prisoner’s Dilemma. So we prefer learning strategies that converge to a Pareto-Optimal outcome that also produces a Nash Equilibrium payoff for repeated two-player, n-action general-sum games. The Folk Theorem enable us to identify such outcomes. In this paper, we introduce the Conditional Joint Action Learner (CJAL) which learns the conditional probability of an action taken by the opponent given its own actions and uses it to decide its next course of action. We empirically show that under self-play and if the payoff structure of the Prisoner’s Dilemma game satisfies certain conditions, a CJAL learner, using a random exploration strategy followed by a completely greedy exploitation technique, will learn to converge to a Pareto-Optimal solution. We also show that such learning will generate Pareto-Optimal payoffs in a large majority of other two-player general sum games. We compare the performance of CJAL with that of existing algorithms such as WOLF-PHC and JAL on all structurally distinct two-player conflict games with ordinal payoffs.     

Goal setting outcomes: Examining the role of goal interaction in influencing the experience and learning outcomes of video game play for earthquake preparedness

Video game goals are important features of video games. Player’s interaction with goals can not only shape the gaming experience by evoking cognitive and affective reactions in players, but also lead to learning outcomes. However, there are few empirical studies on the effects of interacting with game goals, and no previous research has manipulated goal setting. In two experimental studies, participants were randomly assigned to one of the following five conditions: self-set goal repetitive play, assigned goal repetitive play, no-set goal repetitive play, no-set goal single play, and no play. Results show that playing earthquake preparedness video games generates significant learning outcomes; playing repeatedly with self-set goals yields greater learning compared to playing once with no-set goals or not playing; and cognitive reactions mediates the relation between goal interaction and learning. Implications of the results for the design and evaluation of future video games for learning are explored.     

Learning immunology in a game: Learning outcomes,the use of player characters,immersion experiences and visual attention distributions

This study contributed to the current body of literature on game-based learning by investigating the way playing an educational game, Humunology, affected learning about the immune system and examining further the association between game immersion and visual attention distribution. A total of 79 undergraduate and graduate students participated, and data were collected both in situ and ex situ. The results showed that the students learned through playing Humunology, and the analyses of the use of player characters indicated that the game design facilitated gameplay behaviours that are consistent with the science content. The use of the eye-tracking method also revealed that students who were more immersed in playing Humunology paid more attention to areas related to player characters. The interpretations and limitations are discussed further.     

“Games are made for fun”: Lessons on the effects of concept maps in the classroom use of computer games

Does using a computer game improve students’ motivation to learn classroom material? The current study examined students’ motivation to learn history concepts while playing a commercial, off-the-shelf computer game, Civilization III. The study examined the effect of using conceptual scaffolds to accompany game play. Students from three ninth-grade classrooms were assigned to one of three groups: one group used an expert generated concept map, one group constructed their own concept maps, and a control group used no map. It was predicted that the use of concept maps would enhance the educational value of the game playing activity, in particular students’ motivational levels; however, the opposite happened. Students who used a concept map showed lower motivation on the task relative to their baseline motivation for regular classroom instruction. In contrast, the levels of motivation in playing the game, for students in the control group, met or exceeded their levels of motivation during regular classroom instruction. These results suggest that using a conceptual scaffold can decrease students’ motivation to learn classroom material through game play, perhaps because conceptual maps can (a) focus students’ attention on the difficulty of learning the concepts and on the extrinsic rewards for playing the game and (b) make game play less autonomous, less creative, and less active. All of these can negate the primary property that provides playing its principal potential pedagogical power: fun.     

Improving the effectiveness of English vocabulary review by integrating ARCS with mobile game‐based learning

Memorizing English vocabulary is often considered uninteresting, and a lack of motivation exists during learning activities. Moreover, most vocabulary practice systems automatically select words from articles and do not provide integrated model methods for students. Therefore, this study constructed a mobile game‐based English vocabulary practice system, which imitates the popular block elimination game and combines test items of article, difficulty, and teacher model in accordance with curriculum objectives and demands. The participants were first‐year students and classified into traditional group (Group T) and game‐based group (Group G), and learning effectiveness and motivation were analysed in this study. According to statistical analysis results, students who used the game‐based vocabulary practice system, which could captivate their attention and interest, had higher learning effectiveness and provided positive feedback of learning motivation. Furthermore, English teachers could select teaching materials that are consistent with the features, knowledge, and cultural background of learners to improve the relevance dimension of learning motivation.     

A morphable template framework for robot learning by demonstration: Integrating one-shot and incremental learning approaches

Robot learning by demonstration is key to bringing robots into daily social environments to interact with and learn from human and other agents. However, teaching a robot to acquire new knowledge is a tedious and repetitive process and often restrictive to a specific setup of the environment. We propose a template-based learning framework for robot learning by demonstration to address both generalisation and adaptability. This novel framework is based upon a one-shot learning model integrated with spectral clustering and an online learning model to learn and adapt actions in similar scenarios. A set of statistical experiments is used to benchmark the framework components and shows that this approach requires no extensive training for generalisation and can adapt to environmental changes flexibly. Two real-world applications of an iCub humanoid robot playing the tic-tac-toe game and soldering a circuit board are used to demonstrate the relative merits of the framework.     

Understanding student simultaneous smartphone use in learning settings: A conceptual framework

Smartphone use in learning settings is a common behaviour amongst college students. Building on the theory of consumerism, self-efficacy and addictive behaviours, the current study developed a three-component conceptual framework to understand college students' smartphone use in organizational as well as self-directed learning settings. One thousand and thirty-three students in Shenzhen completed an online survey that measured their frequency of using smartphones in learning settings, motivation for using smartphones in learning settings, behavioural control self-efficacy, smartphone use time, problematic smartphone use, multitask habit and impulsiveness. Regression and configurational analytical approaches were used to examine linear and nonlinear relationships. Results support the use of the proposed framework to explain students' smartphone use behaviour in learning settings.     

Assessing implicit science learning in digital games

Building on the promise shown in game-based learning research, this paper explores methods for Game-Based Learning Assessments (GBLA) using a variety of educational data mining techniques (EDM). GBLA research examines patterns of behaviors evident in game data logs for the measurement of implicit learning—the development of unarticulated knowledge that is not yet expressible on a test or formal assessment. This paper reports on the study of two digital games showing how the combination of human coding with EDM has enabled researchers to measure implicit learning of Physics. In the game Impulse, researchers combined human coding of video with educational data mining to create a set of automated detectors of students' implicit understanding of Newtonian mechanics. For Quantum Spectre, an optics puzzle game, human coding of Interaction Networks was used to identify common student errors. Findings show that several of our measures of student implicit learning within these games were significantly correlated with improvements in external postassessments. Methods and detailed findings were different for each type of game. These results suggest GBLA shows promise for future work such as adaptive games and in-class, data-driven formative assessments, but design of the assessment mechanics must be carefully crafted for each game.     

Enhancing formal e‐learning with edutainment on social networks

This paper reports on the investigation of the possibilities of enhancing the formal e‐learning process by harnessing the potential of informal game‐based learning on social networks. The goal of the research is to improve the outcomes of the formal learning process through the design and implementation of an educational game on a social network and its integration with a learning management system (LMS) of an educational institution. As a proof of concept, a Facebook educational game that enables students to learn and test their knowledge was developed. Furthermore, the game was integrated with the Moodle LMS and the evaluation was performed within the e‐learning system at the Faculty of Organizational Sciences, University of Belgrade. The results show that the application of social network edutainment in an e‐learning ecosystem has a positive impact on both the students' results and their satisfaction with the learning process as a whole.     

My‐Mini‐Pet: a handheld pet‐nurturing game to engage students in arithmetic practices

In the last decade, more and more games have been developed for handheld devices. Furthermore, the popularity of handheld devices and increase of wireless computing can be taken advantage of to provide students with more learning opportunities. Games also could bring promising benefits – specifically, motivating students to learn/play, sustaining their interest, reflecting their learning/playing status, and facilitating the learning/playing progress. However, most of these have been designed for entertainment rather than education. Hence, in this study we incorporate game elements into a learning environment. The My‐Mini‐Pet system is a handheld pet‐nurturing game environment, in which students learn with an animal learning companion, their My‐Mini‐Pet. Three design strategies are adopted. First, the pet‐nurturing strategy, which simulates the relationship between the pet and its owner, the My‐Mini‐Pet becomes a motivator/sustainer of learning. Second, the pet appearance‐changing strategy, which externalizes the learning status of the student. In other words, the My‐Mini‐Pet plays the role of a reflector. Third, the pet feedback strategy, which links the behaviours of the student and his/her pet, the My‐Mini‐Pet acts as a facilitator of learning. A pilot study was also conducted to preliminarily investigate the effectiveness and experiences of the strategies on allowing the student to understand arithmetic practices. The results showed that the strategy was effective, encouraging the students to engage in learning activities. Furthermore, the game attracted the students’ attention and stimulated discussion between peers. Some implications about the further developments are also discussed.     

The instructor's gaze guidance in video lectures improves learning

Instructor behaviour is known to affect learning performance, but it is unclear which specific instructor behaviours can optimize learning. We used eye‐tracking technology and questionnaires to test whether the instructor's gaze guidance affected learners' visual attention, social presence, and learning performance, using four video lectures: declarative knowledge with and without the instructor's gaze guidance and procedural knowledge with and without the instructor's gaze guidance. The results showed that the instructor's gaze guidance not only guided learners to allocate more visual attention to corresponding learning content but also increased learners' sense of social presence and learning. Furthermore, the link between the instructor's gaze guidance and better learning was especially strong for participants with a high sense of social connection with the instructor when they learned procedural knowledge. The findings lead to a strong recommendation for educational practitioners: Instructors should provide gaze guidance in video lectures for better learning performance.     

Impasse-driven learning in the context of video games

This study was designed to elucidate the problem-solving skills used by frequent and infrequent video game players to negotiate impasses encountered while playing a novel video game. All participants were instructed to think aloud while playing a video game for 20 consecutive minutes. Comments made were then used to make inferences about the problem-solving skills that participants used to resolve impasses encountered during the game. Findings showed that frequent players made significantly greater reference to insight and game strategies than infrequent players. After reaching an impasse, all players also were most likely to comment on their game progress and potential game strategies to use. Over the course of game play, all participants showed increasing emphasis on their problem-solving skills as evidenced through their greater mention of insight, game strategies, and goal comments.     

Teaching a pet-robot to understand user feedback through interactive virtual training tasks

In this paper, we present a human-robot teaching framework that uses “virtual” games as a means for adapting a robot to its user through natural interaction in a controlled environment. We present an experimental study in which participants instruct an AIBO pet robot while playing different games together on a computer generated playfield. By playing the games and receiving instruction and feedback from its user, the robot learns to understand the user’s typical way of giving multimodal positive and negative feedback. The games are designed in such a way that the robot can reliably predict positive or negative feedback based on the game state and explore its user’s reward behavior by making good or bad moves. We implemented a two-staged learning method combining Hidden Markov Models and a mathematical model of classical conditioning to learn how to discriminate between positive and negative feedback. The system combines multimodal speech and touch input for reliable recognition. After finishing the training, the system was able to recognize positive and negative reward with an average accuracy of 90.33%.     

Successful implementation of user-centered game based learning in higher education: An example from civil engineering

Goal

The use of an online game for learning in higher education aims to make complex theoretical knowledge more approachable. Permanent repetition will lead to a more in-depth learning.

Objective

To gain insight into whether and to what extent, online games have the potential to contribute to student learning in higher education.

Experimental setting

The online game was used for the first time during a lecture on Structural Concrete at Master’s level, involving 121 seventh semester students.

Methods

Pre-test/post-test experimental control group design with questionnaires and an independent online evaluation.

Results

The minimum learning result of playing the game was equal to that achieved with traditional methods. A factor called “joy” was introduced, according to [Nielsen, J. (2002): User empowerment and the fun factor. In Jakob Nielsen’s Alertbox, July 7, 2002. Available from http://www.useit.com/alertbox/20020707.html.], which was amazingly high.

Conclusion

The experimental findings support the efficacy of game playing. Students enjoyed this kind of e-learning.     

元强化学习综述

强化学习在游戏对弈、系统控制等领域内表现出良好的性能,如何使用少量样本快速学习新任务是强化学习中亟需解决的问题。目前的有效解决方法是将元学习应用在强化学习中,由此所产生的元强化学习日益成为强化学习领域中的研究热点。为了帮助后续研究人员快速并全面了解元强化学习领域,根据近年来的元强化学习文献对研究方法进行梳理,将其归纳成基于循环网络的元强化学习、基于上下文的元强化学习、基于梯度的元强化学习、基于分层的元强化学习和离线元强化学习,对五种类型的研究方法进行对比分析,简要阐述了元强化学习的基本理论和面临的挑战,最后基于当前研究现状讨论了元强化学习的未来发展前景。