Methodology for evaluating a novel education technology: a case study of handheld video games in Chile

Many school systems, in both the developed and developing world, are implementing educational technology to assist in student learning. However, there is no clear consensus on how to evaluate these new technologies. This paper proposes a comprehensive methodology for estimating the value of a new educational technology in three steps: benefit analysis, through the administration of a well-designed experiment; cost analysis, which incorporates costs to weigh against the benefits; and feasibility analysis, which introduces real-world concerns that may affect the ability to actually implement the technology. To illustrate the methodology, a case study from Chile is used where portable educational video games were introduced into first and second grade classrooms with the aim of improving learning in mathematics and language. This paper demonstrates the importance of all three steps in the evaluation process and provides a framework for future analyses.     

On the usability and likeability of virtual reality games for education: The case of VR-ENGAGE

Educational software games aim at increasing the students’ motivation and engagement while they learn. However, if software games are targeted to school classrooms they have to be usable and likeable by all students. Usability of virtual reality games may be a problem because these games tend to have complex user interfaces so that they are more attractive. Moreover, if the games acquire an educational content they may lose the attractiveness and appeal that they have on users who are familiar with commercial games. Consequently, likeability may also be questioned. In this paper, we address the issue of usability and likeability of a virtual reality game that is meant to teach students geography. We describe the evaluation experiments conducted, which involved three categories of students in terms of their level of game-playing expertise: novice, intermediate and expert game players. The evaluation results showed that the game was indeed usable and likeable but there was scope for usability and likeability improvement so that the educational benefits may be maximised for all categories of students. The evaluation studies reported in this paper, revealed important issues about further research on virtual reality educational games.     

Potential pedagogical benefits and drawbacks of multimedia use in the English language classroom equipped with interactive whiteboard technology

This paper analyses the process of multimedia integration in English language classrooms equipped with interactive whiteboard (IWB) technology, and offers insights into the theoretical underpinnings of multimedia use in language learning from the perspective of cognitive learning theory. The data discussed here are drawn from a study carried out as part of a PhD research programme at Lancaster University (UK). The study was conducted within an interpretative research paradigm, and data were collected and analysed according to a qualitative approach. In the first part, the paper discusses some perceived pedagogical benefits of adopting a multimedia-oriented approach in the IWB-based classroom. Secondly, it discusses a variety of potential problems related to the use of multimedia resources in the language classroom in question. Finally, the paper draws upon the literature on multimedia learning to address the potential pedagogical implications of these research findings.     

Investigating acceptance toward mobile learning to assist individual knowledge management: Based on activity theory approach

Mobile devices could facilitate human interaction and access to knowledge resources anytime and anywhere. With respect to wide application possibilities of mobile learning, investigating learners’ acceptance towards it is an essential issue. Based on activity theory approach, this research explores positive factors for the acceptance of m-learning systems. In the research, we developed an m-learning system for learners’ knowledge management and invited 152 participants who knew how to use the m-learning system then report on their experience. The results show that enhancing learners’ satisfaction, encouraging learners’ autonomy, empowering system functions, and enriching interaction and communication activities have a significant positive influence on the acceptance of m-learning systems.     

Learning science in a virtual reality application: The impacts of animated-virtual actors’ visual complexity

As the technology in computer graphics advances, Animated-Virtual Actors (AVAs) in Virtual Reality (VR) applications become increasingly rich and complex. Cognitive Theory of Multimedia Learning (CTML) suggests that complex visual materials could hinder novice learners from attending to the lesson properly. On the other hand, previous studies have shown that visual complexity correlates with presence and may increase the perceived affective quality of the virtual world, towards an optimal experience or flow. Increasing these in VR applications may promote enjoyment and higher cognitive engagement for better learning outcomes. While visually complex materials could be motivating and pleasing to attend to, would they affect learning adversely? We developed a series of VR presentations to teach second-year psychology students about the navigational behaviour of Cataglyphis ants with flat, cartoon, or lifelike AVAs. To assess learning outcomes, we used Program Ratings, which measured perception of learning and perceived difficulty, and retention and transfer tests. The results from 200 students did not reveal any significant differences in presence, perceived affective quality, or learning outcomes as a function of the AVA’s visual complexity. While the results showed positive correlations between presence, perceived affective quality and perception of learning, none of these correlates with perceived difficulty, retention, or transfer scores. Nevertheless, our simulation produced significant improvements on retention and transfer scores in all conditions. We discuss possible explanations and future research directions.     

The influence of collaboration styles to children’s cognitive performance in digital problem-solving game “William Adventure”: A comparative case study

Since a large variety of digital games have been used in many fields for educational purposes, their real functions in learning have caught much attention as well. This study first defines learning characteristics of problem-solving digital games and their corresponding cognitive levels, then designs and develops a problem-solving game in accordance to the criteria. Tasks in the game context are inter-related to each other so that players need to critically and creatively think about problem solutions. Learners’ task analyses are performed to observe four elementary learners’ gaming paths, behaviors and cognitive activities, individually and collaboratively. System documentation, video recording, researcher observation, and interviews are conducted to analyze learners’ learning strategies and their cognitive performance during the gaming process.     

A web-based formative assessment tool for Masters students: A pilot study

The use of web-based learning and assessment tools is growing in tertiary institutions around the world. To date, very few papers have reported the development and evaluation of a web-based formative assessment tool for postgraduate students. The aim of the present paper was to report on the development and evaluation of an online formative assessment tool for this student group. The web-based formative assessment tool was evaluated by a sample of undergraduate students, postgraduate students and academic staff within a psychology department in order to determine the suitability and sensitivity of the tool. The results of this pilot test suggest that the development of such a tool is both appropriate and feasible for Masters students studying psychology.     

Improving literacy and metacognition with electronic portfolios: Teaching and learning with ePEARL

Can an electronic portfolio that is both a multimedia container for student work and a tool to support key learning processes have a positive impact on the literacy practices and self-regulated learning skills of students? This article presents the findings of a yearlong study conducted in three Canadian provinces during the 2007–2008 school year initially involving 32 teachers and 388 students. Due to varying levels of implementation our final data set included 14 teachers and 296 students. Using a non-equivalent pre-test/post-test design, we found that grade 4–6 students who were in classrooms where the teacher provided regular and appropriate use of the electronic portfolio tool ePEARL (i.e., medium–high implementation condition, n = 7 classrooms and 121 students), compared to control students (n = 7 classrooms and 175 students) who did not use ePEARL, showed significant improvements (p < .05) in their writing skills on a standardized literacy measure (i.e., the constructed response subtest of the Canadian Achievement Test-4th ed.) and certain metacognitive skills measured via student self-report. The results of this study indicate that teaching with ePEARL has positive impacts on students’ literacy and self-regulated learning skills when the tool is used regularly and integrated into classroom instruction.