Computer-generated log-file analyses as a window into students' minds? A showcase study based on the PISA 2012 assessment of problem solving

This paper aims at showcasing the potential of log-file analyses by capitalizing on the computer-based assessment of complex problem solving (CPS) in the 2012 cycle of the Programme for International Student Assessment (PISA). We analyzed log-file data from the CPS item Climate Control encompassing N = 16,219 students from 44 countries and economies. In Research Question 1, we related the application of an optimal exploration strategy (i.e., vary-one-thing-at-at-time, VOTAT) to performance on this specific problem and to overall level of CPS proficiency, both on an individual and on the country level. In Research Question 2, we identified several groups of students with different levels of non-mastery on a fine-grained level. Results indicated that (1) the VOTAT strategy was strongly related to performance on the Climate Control item as well as to overall problem solving proficiency, and (2) that there were different levels of non-mastery that ranged from applying no systematic strategic behavior to actually applying VOTAT but still failing to solve the item. On the backdrop of these results, we discuss implications and future potentials of log-file analyses in educational large-scale assessments for researchers, teachers, and policy makers.     

Towards a conceptual framework for assessing the effectiveness of digital game-based learning

In recent years, interest has grown in the systematic assessment of the effectiveness of digital game-based learning (DGBL). A conceptual framework describing what effectiveness means in the context of DGBL and which are its subcomponents has hitherto been lacking however. Hence, the goal of this paper is to propose a conceptualization and operationalization of effectiveness rooted in social-cognitive theory. In order to identify desired outcomes and be able to operationalize effectiveness, focus groups were organized with three stakeholder groups following a user requirements analysis methodology.Results indicate that three categories of desired outcomes can be distinguished: learning, motivational and efficiency outcomes. For the different outcomes, different subcomponents can be extracted which can be organized hierarchically. Learning outcomes that are seen as relevant to the effectiveness of DGBL are 1) increased interest in the subject matter, 2) improvement in objective performance (e.g., in a test), and 3) transfer, referring to the player's ability to apply acquired knowledge or skills to real-world situations. Relevant motivational outcomes concern 1) enjoyment, the extent to which playing the game evoked an enjoyable experience, and 2) increased motivation to learn using DGBL. Efficiency outcomes relevant to DGBL effectiveness, finally, are related to 1) time management and 2) cost-effectiveness. Overall, it can be stated that a DGBL intervention is effective when it achieves similar or higher scores compared to other instructional methods in relation to any of the above mentioned outcomes without significantly (in the common, not the statistical sense) diminishing any of the others.     

Researching haptics in higher education: The complexity of developing haptics virtual learning systems and evaluating its impact on students’ learning

hapTEL, an interdisciplinary project funded by two UK research councils from 2007 to 2011, involves a large interdisciplinary team (with undergraduate and post-graduate student participants) which has been developing and evaluating a virtual learning system within an HE healthcare education setting, working on three overlapping strands. Strand 1 involves the technical development and evaluation of the hapTEL workstation which simulates clinical conditions for dental training including haptics (sense of touch). Strand 2 involves examining the traditional undergraduate curriculum and how this could benefit from the use of haptics. Strand 3 is concerned with the educational evaluation of the impact of the work carried out within Strands 1 and 2. Two theoretical frameworks (Entwistle, (1987) and Webb and Cox (2004)) have been used to identify as many factors as possible which could affect the impact of Technology Enhanced Learning (TEL) on the quality of the learning achieved. These frameworks have formed a foundation for measuring the impact of TEL on curriculum change, teachers’ pedagogical practices, students’ learning and on institutional practices. A range of quantitative and qualitative methods were designed, piloted and evaluated in order to measure the impact of TEL on teaching and learning; and to have a rich and robust data set which also addresses the variables in the frameworks. The results from using these frameworks show that institutional and departmental factors should be considered when evaluating the impact of TEL in higher education and that these had a major influence on the design and curriculum integration of the hapTEL systems. We have also shown that by involving the end users from the beginning enabled not only an enhancement of the students’ learning experiences but also a modification to the traditional curriculum itself and the successful integration of TEL within a very traditional undergraduate higher education dental curriculum. The conclusions from this paper confirm earlier reviews of researching TEL that technology integration is extremely complex and the related research requires a comprehensive approach of both quantitative and qualitative methods if one is to take account of the range of variables identified by theoretical frameworks. Finally, repeating the range of empirical investigations for a second year enables researchers to validate the effectiveness of the methods used in the initial year and thereby maximise the reliability and generalisability of the research outcomes.     

Design and usability testing of a learning and plagiarism avoidance tutorial system for paraphrasing and citing in English: A case study

The prevalence of information and communication technology (ICT) has considerably converted the means of/for publication and circulation, as well as transforming academia and English pedagogy. However, with the availability and convenience of online resources, one of the critical issues emerged is that non-native English speakers are constantly accused of committing textual plagiarism: either intentionally or unintentionally. While many writing tools and plagiarism detectors are available to help solve the problem, none of them were customized for the great population of Chinese learners of English. Accordingly, DWright—a Chinese-interfaced online writing tutorial for paraphrasing and citing English—was developed in the hope of ensuring academic integrity through the avoidance of textual plagiarism.The purpose of this study was to evaluate the design and efficacy of DWright regarding plagiarism prevention and improvement of English writing. A questionnaire and semi-structured interview were administrated to participants to test DWright and its technology acceptance model (TAM) to identify the effect of perceived usefulness, usability and user attitudes of DWright. Results show that users of DWright were in high agreement regarding the content effectiveness of all DWright-based tasks, indicating the reading activities, multiple-choice exercises and paraphrasing practices were effective to help users enhance writing knowledge and skills to avoid plagiarism. Furthermore, perceived usefulness and system usability affected DWright user attitudes significantly and positively, which mirrors their attitudes toward continued use of DWright. In this study, DWright met its users' needs by extending their knowledge to avoid plagiarism while simultaneously enhancing their paraphrasing and writing skills. The conclusion suggests that tutorial designers, content experts and subject teachers should support effective communication to improve content usefulness, so as to help users achieve their writing goals with a research proven learning and plagiarism avoidance tutorial system.     

Digital game-based learning: Impact of instructions and feedback on motivation and learning effectiveness

Although many studies have investigated the effects of digital game-based learning (DGBL) on learning and motivation, its benefits have never been systematically demonstrated. In our first experiment, we sought to identify the conditions under which DGBL is most effective, by analyzing the effects of two different types of instructions (learning instruction vs. entertainment instruction). Results showed that the learning instruction elicited deeper learning than the entertainment one, without impacting negatively on motivation. In our second experiment, we showed that if learners are given regular feedback about their performance, the entertainment instruction results in deep learning. These two experiments demonstrate that a serious game environment can promote learning and motivation, providing it includes features that prompt learners to actively process the educational content.     

The learning effects of computer simulations in science education

This article reviews the (quasi)experimental research of the past decade on the learning effects of computer simulations in science education. The focus is on two questions: how use of computer simulations can enhance traditional education, and how computer simulations are best used in order to improve learning processes and outcomes. We report on studies that investigated computer simulations as a replacement of or enhancement to traditional instruction. In particular, we consider the effects of variations in how information is visualized, how instructional support is provided, and how computer simulations are embedded within the lesson scenario. The reviewed literature provides robust evidence that computer simulations can enhance traditional instruction, especially as far as laboratory activities are concerned. However, in most of this research the use of computer simulations has been approached without consideration of the possible impact of teacher support, the lesson scenario, and the computer simulation’s place within the curriculum.     

Collaborative environments for the learning of design: a model and a case study in Domotics

Design plays a central role in a range of subjects at different educational levels. Students have to acquire the knowledge necessary for the execution of tasks that enable them to construct an artefact or model that can be tested by simulation and that satisfies some requirements and verifies some constraints. They achieve this by means of a design process. In some design domains there is a lack of teaching tools from a learner-centred perspective. Moreover, when these domains are complex, the design problems that the students have to solve during their learning process require the design activity to be carried out in group. In response to this situation, we have developed a design model and a collaborative learning method. Using this conceptual framework, we have built a collaborative environment for the learning of domotical¹ design by means of complex problem solving, with an emphasis on synchronous collaboration for work distribution, discussion, design in shared surfaces and simulation. This environment has already been evaluated and used in real teaching experiences.     

The atomic intrinsic integration approach: A structured methodology for the design of games for the conceptual understanding of physics

Computer simulations combined with games have been successfully used to teach conceptual physics. However, there is no clear methodology for guiding the design of these types of games. To remedy this, we propose a structured methodology for the design of conceptual physics games that explicitly integrates the principles of the intrinsic integration approach for designing instructional games (Habgood & Ainsworth, 2011) with an atomic analysis of the structure of games (Cook, 2007; Cousins, 2005; Koster, 2005). To test this approach, we redesigned an existing game to teach electrostatics and compared the educational effectiveness of the original and redesigned versions. Our studies also compared an endogenous fantasy version of the game with a non-fantasy version.     

The effect of multiple scaffolding tools on students’ understanding,consideration of different perspectives,and misconceptions of a complex problem

This study investigated the effectiveness of multiple scaffolding tools in helping students understand a complex problem. In order to support students with this task, a multimedia learning environment was developed based on the cognitive flexibility theory (CFT) and scaffolding through computer-based tools. Seventy-nine 10th-grade students in an urban high school participated in this study. A quasi-experimental method was used to compare the effectiveness of different scaffolding tools within this learning environment. Scaffolding used in conjunction with CFT principles did not affect students’ ability to consider multiple perspectives or their numbers of misconceptions. On the other hand, scaffolding tools had varying effects on students’ problem understanding, and a significant interaction was found between the different scaffolding tools used. This study raised questions about the effectiveness of combining multiple scaffolding tools within a multimedia environment.     

The interactions between problem solving and conceptual change: System dynamic modelling as a platform for learning

This study examines the interactions between problem solving and conceptual change in an elementary science class where students build system dynamic models as a form of problem representations. Through mostly qualitative findings, we illustrate the interplay of three emerging intervening conditions (epistemological belief, structural knowledge and domain knowledge), the choice of learning strategy and the learning outcomes through a theoretical model.     

Impacts and preferences study for e-HO as a holistic learning environment complementary to e-Learning

We present in this study the significance and impacts of an innovative e-HO as a holistic and horizontal platform complementary to e-Learning to help realize transdisciplinary learning and foster integration of knowledge in higher education. A comprehensive investigation of a survey conducted among 647 university students for e-HO is presented with detailed studies on students’ preferences toward e-HO with respect to various demographic variables including gender, access frequency, grade level, discipline and access route concerning the five scales extracted in this particular survey. No significant gender difference surfaced, whereas access frequency turned out to be the most influential and decisive variable that has resulted in significant differences on all the scales involved, especially the characteristic ‘Leisure climate’. Further significant differences appeared on different scales with distinct natures for different demographic variables other than gender and access frequency, indicating the multidimensional characteristics of the whole endeavor and that care must be taken whenever a learning environment like this is constructed in higher education.     

Blended learning applied to the study of Mechanical Couplings in engineering

This paper begins with a brief introduction to blended learning (BL), recognising the important contributions to learning that can be obtained from the use of methods which combine the New Information and Communications Technologies (NICT) with more traditional methods. The paper goes onto describe the bases, tasks and methodology for developing an online tool, using the most up-to-date technologies. Moreover, it presents an application about Mechanical Couplings (MC) developed for students of Mechanical Engineering in accordance with the described methodology. Among other things, this tool enables students to see simulations of coupling assembly, to consult data bases about the technical characteristics of very different existing couplings, to calculate and choose the right coupling for a specific application of power drive between machine shafts and to carry out self-evaluation tests.     

Taking a signal: A review of gesture-based computing research in education

This study used content analysis of journal articles from 2001 to 2013 to explore the characteristics and trends of empirical research on gesture-based computing in education. Among the 3018 articles retrieved from 5 academic databases by a comprehensive search, 59 articles were identified manually and then analyzed. The distribution and trends analyzed were research methods, study disciplines, learning content, technology used, and intended settings of the gesture-based learning systems. Furthermore, instructional interventions were also analyzed based on the learning context or the sub-education domain to which they belonged to ascertain if any instructional intervention was applied in these systems. It was found that experimental design research is the most commonly used method (72.9%) followed by design-based research (20.3%). The findings indicate that Nintendo Wii is the gesture-based device that is the most often used (40%), while the domain in which the technology is most frequently used is special education (42.4%). The same trend was also found in a further analysis which identified that the domain which uses Wii the most is special education (70%). Among all the identified learning topics, motor skills learning has the highest percentage (44%). When grouping these topics into three domains of knowledge (procedural, conceptual, and both), the result demonstrates that both procedural and conceptual type of knowledge are equally distributed in the gesture-based learning studies. Finally, a comparison of instructional intervention of gesture-based learning systems in different sub-education domains is reported.     

Using Hofstede’s cultural dimensions to interpret cross-cultural blended teaching and learning

This article reflects on the cross-cultural communicative experiences of professors from South Africa and students from Sudan, during a two-year Internet-supported Masters’ course in Computers in Education. Four of Hofstede’s cultural dimensions were considered as categories of interpretation. The purpose of the research was to determine the extent to which Hofstede’s static quantitative, research could be used as a basis for an essentially qualitative dynamic interpretation. While Hofstede’s work focuses on cultural differences, this article tries to uncover what commonalities were constructed in the process. It was found that in this case, dimensions such as power distance and uncertainty avoidance tended to amplify each other, while together they resulted in a movement away from individualism towards collectivism. Accommodating across cultures did not mean that one should move into the other culture. Three elements seem to play a role when cultures meet: Reduction of communicative uncertainty, construction of shared meaning, and appropriate use of technology. More research should be conducted to uncover the elements that are common to cultures because emphasising commonality seems more useful than trying to overcome differences.     

The effects of instructional support and learner interests when learning using computer simulations

Within the scope of this study, the effectiveness of two kinds of instructional support was evaluated with regard to the learner’s interests. Two versions of a simulation program about the respiratory chain were developed, differing only in the kind of tasks provided for instructional support: One version contained problem-solving tasks, the other one contained worked-out examples. The focus was on the learner’s interest in the subject and in computers. The first goal of the study was to find to what extent computer simulations incorporating the different kinds of instructional support have positive effects on situational subject-interest. The second goal was to evaluate the interactions between the learner’s interests and the instructional support with regard to the learning results (subdivided into factual knowledge and understanding). Simulations with worked-out examples were shown to have positive effects on the learner’s situational interest in the subject. This was not found to be the case in simulations with problem-solving tasks. Regardless of the kind of instructional support, learners with little interest in the subject were able to achieve significant gains in factual knowledge. However, improvement in understanding was dependent on the kind of instructional support.     

The effects of modern mathematics computer games on mathematics achievement and class motivation

This study examined the effects of a computer game on students' mathematics achievement and motivation, and the role of prior mathematics knowledge, computer skill, and English language skill on their achievement and motivation as they played the game. A total of 193 students and 10 teachers participated in this study. The teachers were randomly assigned to experimental and control groups. A mixed method of quantitative and interviews were used with Multivariate Analysis of Co-Variance to analyze the data.     

Exploring Newtonian mechanics in a conceptually-integrated digital game: Comparison of learning and affective outcomes for students in Taiwan and the United States

This study investigates the potential of a digital game that overlays popular game-play mechanics with formal physics representations and terminology to support explicit learning and exploration of Newtonian mechanics. The analysis compares test data, survey data, and observational data collected during implementations in Taiwan and the United States with students in grades 7–9. Results demonstrate learning on some core disciplinary measures and high levels of learner engagement, indicating the potential benefits of this genre of conceptually-integrated games, but also suggesting that further research and development will be needed to more fully harness this potential. Encouragingly, striking similarities were observed across the two countries in terms of learning and engagement, suggesting that this genre of learning games may prove suitable for engaging students in active exploration of core science concepts across multiple countries.     

Scaffolding mentalizing via a play-&-learn game for preschoolers

Educational software in which the student takes the role of teacher and instructs a digital tutee – a so-called teachable agent – has repeatedly proven to have positive effects for school children's learning. In a study with 39 preschoolers aged 3:9 to 6:3, we explored the conditions under which children this young would benefit as well from this kind of educational software. We specifically investigated to what extent children of this age group would be able to reason about and reflect upon the actions of their digital tutee, and to what extent they would enjoy an educational game centered around instructing and helping a digital tutee. Results revealed that preschoolers were quite capable of reasoning and reflecting upon their digital tutee. This was rather surprising, since the results of a standardized false-belief test did not indicate this level of mentalizing capabilities in the group as a whole. The results also indicated that the preschoolers, like older children, were interested and engaged in this kind of game that involves instructing a digital tutee.     

“Games are made for fun”: Lessons on the effects of concept maps in the classroom use of computer games

Does using a computer game improve students’ motivation to learn classroom material? The current study examined students’ motivation to learn history concepts while playing a commercial, off-the-shelf computer game, Civilization III. The study examined the effect of using conceptual scaffolds to accompany game play. Students from three ninth-grade classrooms were assigned to one of three groups: one group used an expert generated concept map, one group constructed their own concept maps, and a control group used no map. It was predicted that the use of concept maps would enhance the educational value of the game playing activity, in particular students’ motivational levels; however, the opposite happened. Students who used a concept map showed lower motivation on the task relative to their baseline motivation for regular classroom instruction. In contrast, the levels of motivation in playing the game, for students in the control group, met or exceeded their levels of motivation during regular classroom instruction. These results suggest that using a conceptual scaffold can decrease students’ motivation to learn classroom material through game play, perhaps because conceptual maps can (a) focus students’ attention on the difficulty of learning the concepts and on the extrinsic rewards for playing the game and (b) make game play less autonomous, less creative, and less active. All of these can negate the primary property that provides playing its principal potential pedagogical power: fun.