The effect of the modality principle to support learning with virtual reality: An eye-tracking and electrodermal activity study

Background

Virtual reality (VR) is considered a promising approach to support learning. An instructional design is essential to optimize cognitive processes. Studies show that VR has unique instructional and pedagogical requirements.

Objectives

To evaluate the effectiveness and applicability of the modality principle, which was previously validated in 2D classic multimedia, for learning with VR. The modality principle states that multimedia information presented as spoken narration is superior to on-screen text.

Methods

A prospective experimental study with two compared conditions of instruction: VR-based learning guided by on-screen text (n = 34) versus spoken narration (n = 28). Students' cognitive learning experiences were captured by eye-tracking and electrodermal activity (EDA). In addition, students' knowledge was evaluated using a pre–post knowledge test.

Results and Conclusions

Overall, there was no significant difference in knowledge retention between the participants who learned with on-screen text compared to spoken narration. However, results from the eye-tracking analysis showed that students who learned with the on-screen text devoted longer visual attention toward important learning activity areas of interest, suggesting a better ability to discern between relevant and irrelevant information. Conversely, students who learned with the spoken narration expressed significantly more EDA peak responses, proposing a higher cognitive load.

Implications

This study outlines that while learning with VR was effective, the modality principle might not apply to learning with VR. Moreover, the analysis of the learning process suggests even an inverse effect, favouring the provision of instructional scaffolds as on-screen text. Future research should evaluate this effect on long-term knowledge retention.     

多媒体学习软件的人机界面设计研究

包含丰富的教学材料、华丽的色彩搭配、新颖的动画效果的多媒体学习软件界面,不仅不能提升学习者的学习成效,反而可能分散学习者的注意力,将学习者的注意力吸引到学习材料的形式上,而忽略了学习材料所表达的信息。为此,文章依据认知心理学理论和一些认知心理学的实验研究成果,对多媒体学习软件人机界面的布局、图文材料、听觉材料的形式选择和呈现方式进行了设计研究。     

Designing multimedia learning environments using animated pedagogical agents: factors and issues

Animated pedagogical agents (APAs) are known to possess great potential in supporting learning because of their ability to simulate a real classroom learning environment. But research in this area has produced mixed results. The reason for this remains puzzling. This paper is written with two purposes: (1) to examine some recent research and organize the findings in terms of classroom characteristics, and (2) to discuss and reveal any uncovered issues pertaining to the findings and provide input whenever possible. A framework formed by using APA characteristics, APA presentation, and learners' characteristics is used to analyse past research findings. The findings from the analysis reveal that because APAs are regarded as social members similar to humans, they are more effective in engaging learners in environments that require social communication and interactions. They therefore produce more definitive results in terms of affective gain and group learning. But such conditions also impose greater demand on designers to create more complex learning environments that can provide interactions with several agents and yet maintain a mode of communication that is pedagogically effective. The challenges for creating such environment include using an agent's gestures to duplicate its speech in instruction, which is usually uncommon in human practice, overcoming the needs to use input–output interface for communication and taking into consideration the possible influence of the learners' characteristics such as their sensory preference. This paper suggests that APAs' application in instruction should be seen in the light of affordances and be designed within its own practical limits.     

Individual differences in visuospatial working memory capacity influence the modality effect

The cognitive theory of multimedia learning postulates learning information in a dual-modality design is more effective than in a single modality, which is known as the modality effect. Research has found that the modality effect supports problem-solving learning, but not retention-based learning. This divergence in findings can be explained by the mediating effects of individual differences in visuospatial working memory. The current study confirmed the modality effect, showing superiority of dual-modality over single modality. Results revealed a significant interaction between visuospatial working memory and modality by retention, but no such interaction was obtained for problem solving. It was found that learners with low visuospatial working memory scored higher on retention in dual modality than these in single modality. Interestingly, no differences were found for learners with high visuospatial working memory on retention in both single- and dual-modality conditions. The findings may explain for whom the modality effect works in relation to retention. Finally, the analysis showed no association between modality conditions and subtypes of cognitive load, that is, extraneous and germane cognitive load.     

Using pop-up windows to improve multimedia learning

Abstract The aim of the present study is to evaluate the effects on learning of the spatial integration of textual information incorporated into illustrations in the form of pop‐up windows that are opened by the user. Three groups of students viewed illustrated texts depicting the functioning of the heart and the replication of the AIDS virus either with textual information presented below the picture, with textual information integrated within the picture, or with textual information integrated within the picture with pop‐up windows. The results showed that the integrated formats were more effective than the separated format when we tested the retention of textual information, comprehension and the matching of textual elements to the appropriate illustrated elements. Furthermore, they indicated that the group working with pop‐up windows performed better than the integrated groups on the retention test for illustrated information as well as when asked to find correct solutions to problems in a comprehension test. Consequently, these findings support the use of pop‐up windows in learning with this kind of illustrated explanatory text.     

Modality and cueing in multimedia learning: Examining cognitive and perceptual explanations for the modality effect

The purpose of this study was to investigate the effects of modality (written text vs. spoken text) and visual cueing (low cueing vs. high cueing) on the learning and mental effort of participants studying a computer-based static diagram at their own pace. Participants were randomly assigned to four versions of the computer-based materials formed into a 2 × 2 factorial design by crossing modality with cueing. The results revealed a reverse modality effect, wherein participants studying written text outperformed those studying spoken text on tests of free recall, matching, comprehension, and spatial recall, but not mental effort. Information cueing did not significantly affect either performance or mental effort. These findings are discussed in the context of two popular explanations of the modality effect: the cognitive resources explanation and the perceptual resources explanation. The results were best explained from a perceptual resources viewpoint.     

Saying the wrong thing: improving learning with multimedia by including misconceptions

Abstract   In this study, 364 first-year physics students were randomly assigned to one of four online multimedia treatments on Newton's First and Second Laws of Motion: (1) the 'Exposition', a concise lecture-style presentation; (2) the 'Extended Exposition', the Exposition with additional interesting information; (3) the 'Refutation', the Exposition with common misconceptions explicitly stated and refuted; or (4) the 'Dialogue', a student–tutor discussion of the same material as in the Refutation. Students were tested using questions from mechanics conceptual inventories before and after watching the multimedia treatments. Results show the Refutation and Dialogue produced the greatest learning gains, with effect sizes of 0.79 and 0.83, respectively, compared with the Exposition. Students with low prior knowledge benefited most, however high prior knowledge learners were not disadvantaged by the misconception-based approach. The findings suggest that online multimedia can be greatly improved, promoting conceptual change in students with all levels of experience, by including a discussion of misconceptions.     

化工原理多媒体课件的开发应用

介绍了用多媒体开发工具Authorware开发的化工原理多媒体CAI教学课件，并且阐述了制作中的一些关键技术问题。该软件的应用有效地提高了教学质量。     

The modality effect tested in children in a user‐paced multimedia environment

The modality learning effect proposes that learning is enhanced when information is presented in both the visual and the auditory domains (e.g. pictures and spoken information) compared with presenting information solely in the visual channel (e.g. pictures and written text). Most of the evidence for this effect comes from adults in a laboratory setting. Therefore, we tested the modality effect with 80 children in the highest grade of elementary school in a naturalistic setting. In a between‐subjects design, the children either saw representational pictures with speech or representational pictures with text. Retention and transfer knowledge was tested at three moments: immediately after the intervention, one day after and after one week. The present study did not find any evidence for a modality effect in children when the lesson was learner‐paced. Instead, we found a reversed modality effect directly after the intervention for retention. A reversed modality effect was also found for the transfer questions one day later. This effect was robust, even when controlling for individual differences.     

Does the modality principle hold for different media? A test of the method‐affects‐learning hypothesis

Abstract Does the modality of instructional messages affect learning? How does it affect different media? In this paper, I offer an answer to these questions by first proposing a theoretical framework from which effective instructional methods can be derived. Then, I report a set of studies where one method, the modality principle, was tested across different media and found to produce significant learning benefits. Based on the findings and those of a recent meta‐analysis of modality effects, I suggest future research directions to examine method and media contributions to learning.     

The role of spatial descriptions in learning from multimedia

In the reported experiment we investigated how spatial information conveyed in an expository text influenced multimedia learning. It was based on a 2 × 2-design with the degree of spatial information given in the text (high vs. low spatial text) and the presentation format (written text-only vs. written text + animation) as between-subjects factors. As dependent variables learning outcomes as well as self-reported cognitive load were assessed. The results revealed that there was a multimedia effect with regard to learning outcomes only for low spatial text, but not for high spatial text. Moreover, the cognitive load measures showed an overall multimedia effect irrespective of the degree of spatial information conveyed by the text (i.e., higher cognitive load ratings in the text-only conditions). These results can be explained as a special instance of the redundancy effect as well as a consequence of processing interference within visuo-spatial working memory.     

How flipped learning based on the cognitive theory of multimedia learning affects students' academic achievements

The study aims to identify the effect of a flipped classroom approach designed according to the cognitive theory of multimedia learning on the academic achievements of eighth‐grade students (aged 14 years) in Saudi Arabia in computer science. To this end, a quasi‐experimental design was used, with a sample of 67 students; 33 students were assigned to the experimental group, whereas 34 comprised the control group. The experimental group was subjected to the flipped classroom approach, whereas the control group was given direct instruction. To measure student achievements, an instrument that measures cognitive skills based on Revised Bloom's taxonomy levels was designed. Findings revealed a positive effect on the experimental group's achievement levels with respect to Bloom's higher order thinking skills, that is, applying, analysing, and evaluating. No difference was found between the two groups in terms of academic achievements at the remembering and understanding levels of Bloom's taxonomy. Moreover, learners with low prior knowledge showed a higher improvement in academic achievements compared with those with high prior knowledge. This corresponds to the assumption of the cognitive theory of multimedia learning that learners with low prior knowledge would benefit from its principles more than learners with high prior knowledge.     

Presentations for databases in multimedia environments

The problem of presenting database query results has not been investigated sufficiently. This work proposes an approach to identify effective presentations for the results of database queries. This will be done by considering different kinds of information, such as user goals, the characteristics of the data to be presented, and the relationships among them. Our approach is based on the definition of composition operations which, on the basis of elementary presentation types, allow us to obtain designs of complex final presentations. Some effectiveness criteria are considered in order to identify those presentations which best match the user's goals and cognitive abilities. We consider multimedia environments at both the data and presentation level.     

A multimedia speech learning system for the hearing impaired

The lack (or limited amount) of audio feedback for many persons with hearing disabilities greatly reduces the quality of speech development and use. Current methodologies for the teaching speech techniques are relatively primitive, and generally require the active participation of a therapist. This makes essential practice difficult, time consuming, and expensive for the student. However, using multimedia technology, it is possible to amplify the efforts of the teachers and students for improving the quality of speech. This paper presents a multimedia visualization system that augments the limited hearing capabilities of hearing impaired persons using visual and tactile feedback approaches and provides alternate representations of sound for increasing speech capabilities and reduced learning time. The proposed system has the capability of not only decreasing the learning time for many persons, but can also be adopted for teaching young children correct speech patterns through the use of speech learning games based on the system.     

基于多媒体技术的认知功能评估系统之研究

基于多媒体技术,结合MSP430,设计了一款交互式的认知功能评估系统,该系统主要由MSP430微控制器、电磁感应模块、3轴传感器、LCD和无线通信模块组成。开发了包括命名、记忆、注意、抽象、定向、执行的6类游戏来评估大脑认知功能。实验中使用科技接受模型对系统进行评估（满分为7分,从“非常不同意”到“非常同意”分数依次增加）,其中使用意愿得分6.52分、娱乐性得分6.45分。实验结果表明：正常老人和轻度认知障碍老人相比,游戏执行时间更短（平均相差27.26 s）、准确率更高（平均相差4.38%）、数字记忆广度更好（平均相差0.7分）。     

Learning about a serious disease: When a personalized message is harmful unless you are happy

According to the personalization principle, addressing learners by means of a personalized compared to a nonpersonalized message can foster learning. Interestingly, though, a recent study found that the personalization principle can invert for aversive contents. The present study investigated whether the negative effect of a personalized message for an aversive content can be compensated when learners are in a happy mood. It was hypothesized that the negative effect of a personalized compared to a nonpersonalized message would only be observable for participants in a sad mood, while for participants in a happy mood a personalized message should be beneficial. A 2 × 2 between-subject design with mood (happy vs. sad) and personalization (personalized vs. nonpersonalized message) was used (N = 125 University students). Mood was experimentally varied prior to learning. Learning outcomes were measured by a retention and a transfer test. Results were essentially in line with the assumption: For participants in the sad mood condition, a negative effect of a personalized message was observable for retention and transfer. For participants in the happy mood condition, a positive effect of personalized message was observable for retention, but no effect for transfer. Note that the manipulation check measure for the mood induction procedure did not detect differences between conditions; this may be due to a shortcoming of the used measure (as indicated by an additional evaluation study). The study emphasizes the importance to consider the inherent emotional content of a topic, such as its aversive nature, since the emotional content of a topic can be a boundary condition for design principles in multimedia learning. The study also highlights the complex interplay of externally induced and inherently arising emotions.