Iterative augmentation of a medical training simulator: Effects of affective metacognitive scaffolding

Experiential training simulators are gaining increasing popularity for job-related training due to their potential to engage and motivate adult learners. They are designed to provide learning experiences that are directly connected to users' work environments and support self-regulated learning. Nevertheless, learners often fail to transfer the knowledge gained in the simulated environment to real-world contexts. The EU-funded ImREAL project aimed to bridge that gap by developing a suite of intelligent services designed to enhance existing training simulators. This paper presents work that was a subset of this research project, reporting the iterative development and evaluation of a scaffolding service, which was integrated into a simulator for training medical students to perform diagnostic interviews. The study comprises three evaluation phases, comparing the pure simulator to a first version with metacognitive scaffolding and then to a final version with affective metacognitive scaffolding and enriched user modelling. The scaffolding service provides the learner with metacognitive prompts; affective elements are realized by an integrated affect reporting tool and affective prompts. Using a mixed-method approach by analysing questionnaires (N = 106) and log-data (N = 426), the effects of the services were investigated with respect to real-world relevance, self-regulated learning support, learning experience, and integration. Despite some limitations, the outcomes of this study demonstrate the usefulness of affective metacognitive scaffolding in the context of experiential training simulators; significant post-simulation increases in perceived relevance of the simulator, reflective note-taking, overall motivation, and feeling of success could be identified. Perceived usability and flow of the simulation increased, whereas overall workload and frustration decreased. However, low response rates to specific functions of the simulation point to a need to further investigate how to raise users' awareness and understanding of the provided tools, to encourage interaction with the services, and to better convey the benefits of using them. Thus, future challenges concern not so much technological developments for personalizing learning experiences, but rather new ways to change user attitudes towards an open approach to learning systems that enables them to benefit from all offered features.