Integrating written text and graphics as a desirable difficulty in long-term multimedia learning

Many principles for the design of multimedia learning materials share the recommendation to facilitate processing. One prominent example is the modality principle, according to which pictures should be presented with auditory rather than visual texts. Research on desirable difficulties, however, indicates that – unlike short-term learning – long-term learning benefits when processing is more demanding and therefore more effortful. In a classroom experiment (Experiment 1) and in a laboratory study (Experiment 2), we tested whether the modality principle serves long-term learning. In a multimedia presentation on the formation of lightning, we varied the text modality (oral vs. written) and the delay between learning and test (retention and transfer performance tested immediately after instruction vs. one week later). In the immediate tests, there was either an auditory advantage (Experiment 1) or no difference (Experiment 2). However, when learning was tested after a delay, the combined processing of written text and animations led to better transfer performance than oral text and animations in both experiments. This suggests that written text presentation serves as a desirable difficulty that supports long-term learning.     

Does modality play a role? Visual‐verbal cognitive style and multimedia learning

The study presented in this paper aimed to examine the effect of visual and verbal cognitive style on learning from different types of visualization and modalities of explanatory text. Learning materials in the form of either computer‐based animation or a series of static pictures with written or spoken explanations were presented to 197 students. We found that a more developed visual cognitive style was related to a better learning outcome, when learning from a combination of static pictures and written text. Higher developed visualizers achieved poorer learning outcomes when learning with an animation and written text. The results are partially in line with an ability‐as‐compensator effect and the expertise reversal effect. Additionally, we found a modality effect as the versions with spoken text provided better results on learning outcome than the versions with written text regardless of the prominence of visual cognitive style. No significant interaction effects were found regarding verbal cognitive style.     

Fostering multimedia learning of science: Exploring the role of an animated agent’s image

Research suggests that students learn better when studying a picture coupled with narration rather than on-screen text in a computer-based multimedia learning environment. Moreover, combining narration with the visual presence of an animated pedagogical agent may also encourage students to process information deeper than narration or on-screen text alone. The current study was designed to evaluate three effects among students learning about the human cardiovascular system: the modality effect (narration vs. on-screen text), the embodied agent effect (narration + agent vs. on-screen text), and the image effect (narration + agent vs. narration). The results of this study document large and significant embodied agent and image effects on the posttest (particularly retention items) but surprisingly no modality effect was found. Overall, the results suggest that incorporating an animated pedagogical agent – programmed to coordinate narration with gaze and pointing – into a science-focused multimedia learning environment can foster learning.     

A comparison of temporal speech and text cueing in educational multimedia

This research focused on the prediction that children in their school setting would learn more from educational multimedia when critical information was presented as spoken instead of textual cues. Analyses of a study (n= 42) showed that 12‐year‐olds did not learn any more from temporal speech cueing than from temporal text cueing. The findings suggest that multimedia learning for children is a different kind of learning experience than for adults or older adolescents. The results indicate underdeveloped executive control of the referential connections in the children's working memory between reading screen text while listening to spoken cues, and between watching on‐screen animations play while listening to spoken cues. Further study is warranted. Implications may be derived for educational multimedia research in school settings.     

Do Hebrew electronic books differ from Dutch electronic books? A replication of a Dutch content analysis

Abstract This replication study of Hebrew versus Dutch electronic books for young children was based on De Jong & Bus's content analysis, which explored whether e‐books are appropriate supports for young children's literacy development. Our criteria for analysing 43 Hebrew e‐books for young children included book processing, multimedia in pictures, multimedia connected to printed or spoken text, interactivity of pictures, interactive legibility, printed text quality, and congruence to story content. Like Dutch e‐books, currently available, Hebrew e‐books have no uniform components design and are unsatisfactory supports for children's literacy. Although most Hebrew e‐books did not include hidden hot‐spots that could be activated, when included, their congruence with the storyline was good. Recommendations for CD e‐storybook designers and practical implications for educators are discussed.     

A sequenced multimodal learning approach to support students' development of conceptual learning

Virtual learning environments can now be enriched not only with visual and auditory information, but also with tactile and kinesthetic feedback. However, the way to successfully integrate haptic feedback on a multimodal learning environment is still unclear. This study aims to provide guidelines on how visuohaptic simulations can be implemented effectively, thus the research question is: Under what conditions do visual and tactile information support students' development of conceptual learning of force‐related concepts? Participants comprised 170 undergraduate students of a Midwestern University enrolled in a physics for elementary education class. Four experiments were conducted using four different configurations of multimodal learning environments: Visual feedback only, haptic force feedback only, visual and haptic force feedback at the same time, and sequenced modality of haptic feedback first and visual feedback second. Our results suggest that haptic force feedback has the potential to enrich learning when compared with visual only environments. Also, haptic and visual modalities interact better when sequenced one after another rather than presented simultaneously. Finally, exposure to virtual learning environments enhanced by haptic forced feedback was a positive experience, but the ease of use and ease of interpretation was not so evident.     

Examining learning from text and pictures for different task types: Does the multimedia effect differ for conceptual,causal, and procedural tasks?

The multimedia effect (ME) is a well-researched effect in the field of learning and instruction. In this article, two views that explain the ME are compared. The outcome-oriented view focuses on the beneficial effect of text and pictures on mental representations, whereas the process-oriented view focuses on the beneficial effect of text and pictures for information processing. To contrast these views, the ME sizes for different task types were compared (i.e., conceptual, causal, procedural tasks). Whereas the outcome-oriented view predicts no differences in ME size, the process-oriented view predicts that the ME is largest in causal tasks, smaller in procedural tasks, and smallest in conceptual tasks. Sixty-five students learnt with text only or with text and pictures. Task type and information source (i.e., whether the text, picture, or text and picture provided the answer to a post-test question) were varied within subjects. The results showed that, in line with the process-oriented view, the ME was smaller for conceptual tasks than for procedural tasks. Contrary to the expectations, the ME was larger in procedural tasks than in causal tasks. Moreover, the pattern of results varied with information source. Research and practical implications are described, so that pictures can be deployed optimally.     

An empirical evaluation of multi-media based learning of a procedural task

The present study investigated the effects of multi-media modules and their combinations on the learning of procedural tasks. In the experiment, 72 participants were classified as having either low- or high spatial ability based on their spatial ability test. They were randomly assigned to one of the six experimental conditions in a 2 × 3 factorial design with verbal modality (on-screen text procedure vs. auditory procedure) and the format of visual representation (static visual representation vs. static visual representation with motion cues vs. animated visual representation). After they completed their learning session, the ability to perform the procedural task was directly measured in a realistic setting. The results revealed that: (1) in the condition of static visual representation, the high spatial ability group outperformed the low spatial ability group, (2) for the low spatial ability participants, the animated visual representation group outperformed the static visual representation group, however, the static visual representation with motion cues group did not outperform the static visual representation group, (3) the use of animated visual representation helped participants with low spatial ability more than those with high spatial ability, and (4) a modality effect was found for the measure of satisfaction when viewing the animated visual representation. Since the participants with low spatial ability benefited from the use of animation, the results might support an idea that people are better able to retrieve the procedural information by viewing animated representation. The findings also might reflect a preference for the auditory mode of presentation with greater familiarity with the type of visual representation.     

The effect of text messaging on 9‐ and 10‐year‐old children's reading,spelling and phonological processing skills

This paper reports on an intervention study that considered the impact of text messaging on 9‐ to 10‐year‐old children's literacy skills. One hundred and fourteen children who had never owned a mobile phone before were recruited and randomly allocated to either the intervention or control conditions. All children were pre‐ and post‐tested on a range of reading, spelling and phonological awareness measures. Children in the intervention group were given access to a mobile phone (enabled for text messaging only) for weekends and during half‐term break for a 10‐week period. It was found that there were no significant differences between the two groups of children in terms of their literacy attainment during that period. However, within the mobile phone group, there was evidence that use of text abbreviations was positively related to gains in literacy skills. Moreover, after controlling for individual differences in IQ, and the children's performance at pre‐test, textism usage was able to account for a significant amount of variance in post‐test spelling scores. These results show that text messaging does not adversely affect the development of literacy skills within this age group, and that the children's use of textisms when text messaging is positively related to improvement in literacy skills, especially spelling.     

The influence of text modality on learning with static and dynamic visualizations

In this study we investigated the influence of text modality on learning with static and dynamic visualizations in a dynamic domain, namely the physical principles underlying fish locomotion. A 2 × 2-design with type of visualization (static vs. dynamic) and text modality (spoken vs. written) as independent variables was used. Concerning learning outcomes, it was hypothesized that (1) learners presented with dynamic visualizations would outperform learners presented with static visualizations, (2) learners presented with spoken text would outperform learners presented with written text, and (3) an interaction between type of visualization and modality would occur: the superiority of dynamic over static visualizations was expected to be more pronounced for spoken compared to written text. Subjective cognitive load measures were assessed and expected to mirror the aforementioned pattern of learning outcomes in accordance with Cognitive Load Theory (i.e., higher extraneous cognitive load (ECL) related to lower learning outcomes). For transfer tasks, the first two hypotheses could be confirmed. However, there was no interaction. Moreover, ECL was rated higher by subjects when learning with static compared to dynamic visualizations, but there were no differences for ECL with respect to the text modality. The results are discussed within the framework of Cognitive Load Theory.     

The design and efficacy of a robot‐mediated visual motor program for children learning disabilities

This study investigated the effects of a robot‐mediated visual motor program on improving the visual motor skills of children ages 5 to 11 years with learning disabilities and visual motor delays. It also compared the effects of a robotic program with traditional occupational therapy and tested the reliability of the computer‐generated scores. A randomized experimental design was used with 25 students assigned to either a robotic program group or traditional occupational therapy. The primary outcome measure was the Developmental Test of Visual Motor Integration (VMI). Non‐parametric statistics evaluated the effects of intervention and compared the two methods. The correlation between the computer‐based scoring methods and the VMI was tested. The children who received the robotic program demonstrated significant gains in visual motor performance. The correlation between the computer‐based scoring and the VMI was moderate, but not significant.     

The effect of the modality principle to support learning with virtual reality: An eye-tracking and electrodermal activity study

Background

Virtual reality (VR) is considered a promising approach to support learning. An instructional design is essential to optimize cognitive processes. Studies show that VR has unique instructional and pedagogical requirements.

Objectives

To evaluate the effectiveness and applicability of the modality principle, which was previously validated in 2D classic multimedia, for learning with VR. The modality principle states that multimedia information presented as spoken narration is superior to on-screen text.

Methods

A prospective experimental study with two compared conditions of instruction: VR-based learning guided by on-screen text (n = 34) versus spoken narration (n = 28). Students' cognitive learning experiences were captured by eye-tracking and electrodermal activity (EDA). In addition, students' knowledge was evaluated using a pre–post knowledge test.

Results and Conclusions

Overall, there was no significant difference in knowledge retention between the participants who learned with on-screen text compared to spoken narration. However, results from the eye-tracking analysis showed that students who learned with the on-screen text devoted longer visual attention toward important learning activity areas of interest, suggesting a better ability to discern between relevant and irrelevant information. Conversely, students who learned with the spoken narration expressed significantly more EDA peak responses, proposing a higher cognitive load.

Implications

This study outlines that while learning with VR was effective, the modality principle might not apply to learning with VR. Moreover, the analysis of the learning process suggests even an inverse effect, favouring the provision of instructional scaffolds as on-screen text. Future research should evaluate this effect on long-term knowledge retention.     

Investigating an application of speech‐to‐text recognition: a study on visual attention and learning behaviour

One major drawback of previous research on speech‐to‐text recognition (STR) is that most findings showing the effectiveness of STR for learning were based upon subjective evidence. Very few studies have used eye‐tracking techniques to investigate visual attention of students on STR‐generated text. Furthermore, not much attention was paid to learning differences, such as learning ability, learning style preferences and gender, to use STR texts. Therefore, this study carried out one experiment. Firstly, participants' visual attention on STR‐generated text was investigated by employing eye‐tracking technique. Secondly, how differently effective STR‐generated texts can be to influence participants' learning achievement was tested. Thirdly, this study compared visual attention and learning behaviour with the different characteristics of participants, such as learning ability, learning style preferences and gender, to use STR‐generated texts. Finally, this study explored students' perceptions regarding usefulness of STR‐generated texts for learning. This paper discusses results, research findings and implications along with conclusions and several suggestions for future development and research.     

Cognitive load and instructionally supported learning with provided and learner-generated visualizations

This study investigated, whether learning from science texts can be enhanced by providing learners with different forms of visualizations (pictures) in addition to text. One-hundred-two 9th and 10th graders read a computer-based text on chemical processes of washing and answered questions on cognitive load (mental effort, perceived difficulty) and comprehension (retention, transfer, drawing). Instruction varied according to a 2 × 2-factorial design with ‘learner-generated pictures’ (yes, no) and ‘provided pictures’ (yes, no) as factors. Results indicate positive main effects of provided pictures on all three comprehension measures and negative main effects on both cognitive load measures. Additional analyses revealed a mediation effect of perceived difficulty on retention and transfer, that is learning with provided pictures decreased cognitive load and enhanced comprehension. Furthermore, results show a positive main effect of learner-generated pictures on drawing and mental effort, but no mediation effect. Taken together, computer-based learning with provided pictures enhances comprehension as it seems to promote active processing while reducing extraneous cognitive processing. Learners, generating pictures, however, seem to have less cognitive resources available for essential and generative processing, resulting in reduced comprehension. These results are in line with cognitive load theory, cognitive theories of multimedia learning, and generative theories of learning.     

Modality and cueing in multimedia learning: Examining cognitive and perceptual explanations for the modality effect

The purpose of this study was to investigate the effects of modality (written text vs. spoken text) and visual cueing (low cueing vs. high cueing) on the learning and mental effort of participants studying a computer-based static diagram at their own pace. Participants were randomly assigned to four versions of the computer-based materials formed into a 2 × 2 factorial design by crossing modality with cueing. The results revealed a reverse modality effect, wherein participants studying written text outperformed those studying spoken text on tests of free recall, matching, comprehension, and spatial recall, but not mental effort. Information cueing did not significantly affect either performance or mental effort. These findings are discussed in the context of two popular explanations of the modality effect: the cognitive resources explanation and the perceptual resources explanation. The results were best explained from a perceptual resources viewpoint.     

Exploring the relation between visualizer-verbalizer cognitive styles and performance with visual or verbal learning material

A student might find a certain representational format (e.g., diagram, text) more attractive than other formats for learning. Computer technology offers opportunities to adjust the formats used in learning environments to the preferences of individual learners. The question addressed in the current study was: does the match between a student’s preference regarding the format of learning materials have any relation with performance when learning with a specific format? For example, do learners with a preference for visual materials indeed perform better with visual learning materials? In a study with a pre-test post-test design, 48 participants were randomly assigned to one of two conditions. Both conditions completed a mathematical learning task about combinatorics and probability theory. In one condition learning materials were mainly diagram-based in the other condition they were mainly text-based. Afterward, the relations between cognitive style (visualizers-verbalizers), cognitive abilities (e.g., spatial and verbal ability), and learning performance were examined. The findings showed that cognitive style and learning outcomes were unrelated, for example, learners with a preference for visual materials do not necessarily perform better with visual learning materials. Learning results seem to be influenced by cognitive ability (in particular spatial visualization) and the extent to which a format affords cognitive processing, rather than a match between used and preferred format. It is argued that students should not choose on the basis of their preference, because it might lead them to selecting a format that is less effective for learning.     

Early reading intervention by means of a multicomponent reading game

This study examined the effects of an intervention with a multicomponent reading game on the development of reading skills in 60 Dutch primary school children with special educational needs. The game contains evidence‐based reading exercises and is based on principles of applied gaming. Using a multiple baseline approach, we tested children's word, pseudoword and text reading fluency, as well as their reading motivation, at three test occasions. The results indicated that the short intervention (9 × 15 min) enhanced children's pseudoword reading fluency as well as their text reading fluency. Interestingly, the early intervention group showed a sustained intervention effect (i.e., also during retention), which shows that using this reading game has a long‐term effect on early text reading development. Intervention did not affect reading motivation, which suggests that the multicomponent reading game can be used to facilitate early reading development without compromising reading motivation.     

基于跨模态实体信息融合的神经机器翻译方法

现有多模态机器翻译(Multi-modal machine translation, MMT)方法将图片与待翻译文本进行句子级别的语义融合. 这些方法存在视觉信息作用不明确和模型对视觉信息不敏感等问题, 并进一步造成了视觉信息与文本信息无法在翻译模型中充分融合语义的问题. 针对这些问题, 提出了一种跨模态实体重构(Cross-modal entity reconstruction, CER)方法. 区别于将完整的图片输入到翻译模型中, 该方法显式对齐文本与图像中的实体, 通过文本上下文与一种模态的实体的组合来重构另一种模态的实体, 最终达到实体级的跨模态语义融合的目的, 通过多任务学习方法将CER模型与翻译模型结合, 达到提升翻译质量的目的. 该方法在多模态翻译数据集的两个语言对上取得了最佳的翻译准确率. 进一步的分析实验表明, 该方法能够有效提升模型在翻译过程中对源端文本实体的忠实度.     

Learning cell biology with close-up views or connecting lines: Evidence for the structure mapping effect

Different visualization strategies for structuring non-hierarchical learning tasks in hypermedia are used until now. Do these strategies differ in learning effectiveness? In the present study, a hypermedia learning environment about cell biology was investigated. The graphical properties of a content module were created in two different ways, either depicting close-up views of the cell organelles or with connecting lines between the cell organelles and their respective technical terms within the graphic of the whole cell. Students’ post test performance indicate that connecting lines fostered auditory recall, that is remembering and understanding of narrative information, more efficiently than close-up views. In the case of visual recall, that is identification of electron microscopic pictures depicting cell structures, there was no difference in performance between these two visualization techniques. Transfer performance depended on an interaction between visualization technique and students’ spatial ability: Only students with high spatial ability benefited from the connecting lines variant. With respect to auditory recall and transfer performance, the present study supports the structure mapping effect as proposed by the integrated model of text and picture comprehension. Especially students with low spatial ability seem to be prone to an illusion of knowing.     

The impact of multimedia effect on science learning: Evidence from eye movements

This study examined how middle school students constructed their understanding of the mitosis and meiosis processes at a molecular level through multimedia learning materials presented in different interaction and sensory modality modes. A two (interaction modes: animation/simulation) by two (sensory modality modes: narration/on-screen text) factorial design was employed. The dependent variables included subjects’ pre-test, post-test, and retention-test scores, showing their understanding of mitosis and meiosis process at molecular level, as well as data of subjects’ eye-movement behavior. Results showed the group that received animation with narration allocated a greater amount of visual attention (number of fixations, total inspection time, and mean fixation duration) than the group that received animation with on-screen text, in both pictorial area and area of interest, which is consistent with students’ immediate and long-term retained learning of the processes of mitosis and meiosis. The group that received simulation with on-screen text allocated a greater amount of visual attention than the group that received simulation with narration, consistent with students’ immediate and retained learning. The group that received simulation with on-screen text also allocated a greater amount of visual attention than the group that received animation with on-screen text, consistent with students’ immediate and retained learning. This study adds empirical evidence of a direct correlation between the length of eye fixation behavior and the depth of learning. Moreover, it provides insight into the multimedia effect on students’ cognitive process through the use of eye fixation behavior evidence.