Examining learning from text and pictures for different task types: Does the multimedia effect differ for conceptual,causal, and procedural tasks?

The multimedia effect (ME) is a well-researched effect in the field of learning and instruction. In this article, two views that explain the ME are compared. The outcome-oriented view focuses on the beneficial effect of text and pictures on mental representations, whereas the process-oriented view focuses on the beneficial effect of text and pictures for information processing. To contrast these views, the ME sizes for different task types were compared (i.e., conceptual, causal, procedural tasks). Whereas the outcome-oriented view predicts no differences in ME size, the process-oriented view predicts that the ME is largest in causal tasks, smaller in procedural tasks, and smallest in conceptual tasks. Sixty-five students learnt with text only or with text and pictures. Task type and information source (i.e., whether the text, picture, or text and picture provided the answer to a post-test question) were varied within subjects. The results showed that, in line with the process-oriented view, the ME was smaller for conceptual tasks than for procedural tasks. Contrary to the expectations, the ME was larger in procedural tasks than in causal tasks. Moreover, the pattern of results varied with information source. Research and practical implications are described, so that pictures can be deployed optimally.     

A comparison of temporal speech and text cueing in educational multimedia

This research focused on the prediction that children in their school setting would learn more from educational multimedia when critical information was presented as spoken instead of textual cues. Analyses of a study (n= 42) showed that 12‐year‐olds did not learn any more from temporal speech cueing than from temporal text cueing. The findings suggest that multimedia learning for children is a different kind of learning experience than for adults or older adolescents. The results indicate underdeveloped executive control of the referential connections in the children's working memory between reading screen text while listening to spoken cues, and between watching on‐screen animations play while listening to spoken cues. Further study is warranted. Implications may be derived for educational multimedia research in school settings.     

Fostering multimedia learning of science: Exploring the role of an animated agent’s image

Research suggests that students learn better when studying a picture coupled with narration rather than on-screen text in a computer-based multimedia learning environment. Moreover, combining narration with the visual presence of an animated pedagogical agent may also encourage students to process information deeper than narration or on-screen text alone. The current study was designed to evaluate three effects among students learning about the human cardiovascular system: the modality effect (narration vs. on-screen text), the embodied agent effect (narration + agent vs. on-screen text), and the image effect (narration + agent vs. narration). The results of this study document large and significant embodied agent and image effects on the posttest (particularly retention items) but surprisingly no modality effect was found. Overall, the results suggest that incorporating an animated pedagogical agent – programmed to coordinate narration with gaze and pointing – into a science-focused multimedia learning environment can foster learning.     

Integrating textual and pictorial information via pop-up windows: an experimental study

Following previous research in cognitive psychology, this paper deals with the effect of the spatial display of textpicture information on the user's cognitive processes. Two experiments were carried out to compare three displays on a computer screen: 'split' display (text and picture information displayed in separate areas on the screen), 'integrated' display (text information close to the part of the picture to which it refers), and 'pop-up' display (text information integrated in pop-up fields which appeared only via the user's action). In both experiments, the results showed that the integrated display and to a greater extent the pop-up display led to higher performances for an equal or lower learning time. Thus, these experiments reinforce the hypothesis that material where text and picture are integrated improves learning, especially if text information appears in pop-up fields. Results are discussed from a theoretical and a practical point of view.     

An empirical assessment of a multimedia executive support system

An experiment was conducted to assess the use of `Multimedia' in ESS (executive support system) and its potential impact on the effectiveness of information analysis. Data was collected from 40 managers and/or professional employees in three business organizations. The primary task was to analyze information, stored in an ESS prototype, in order to identify some possible threats and opportunities. The results of the study suggest that multimedia may not be an appropriate presentation format for analytical tasks. Managers and professional employees who used a multimedia ESS identified fewer threats and opportunities than those who used a text-based system. Contrary to the results of several previous studies, the use of multimedia did not improve information retention. During the post-experiment interview, subjects indicated that the use of sound annotations had adversely affected their ability to analyze information. They also suggested that some multimedia components, such as animated charts or animated text blocks, were unnecessary.     

Using eye movements to model the sequence of text–picture processing for multimedia comprehension

This study used eye movement modeling examples (EMME) to support students' integrative processing of verbal and graphical information during the reading of an illustrated text. EMME consists of a replay of eye movements of a model superimposed onto the materials that are processed for accomplishing the task. Specifically, the study investigated the effects of modeling the temporal sequence of text and picture processing as shown in various replays of a model's gazes. Eighty‐four 7th graders were randomly assigned to one of the four experimental conditions: text‐first processing sequence (text‐first EMME), picture‐first processing sequence (picture‐first EMME), picture‐last processing sequence (picture‐last EMME) and no‐EMME (control). Online and offline measures were used. Eye movement indices indicate that only readers in the picture‐first EMME condition spent significantly longer processing the picture and showed stronger integrative processing of verbal and graphical information than students in the no‐EMME condition. Moreover, readers in all EMME conditions outperformed those in the control condition for recall. However, for learning and transfer, only readers in the picture‐first EMME condition were significantly superior to readers of the control condition. Furthermore, both the frequency and duration of integrative processing of verbal and graphical information mediated the effect of condition on learning outcomes.     

Integrating textual and pictorial information via pop-up windows: An experimental study

Following previous research in cognitive psychology, this paper deals with the effect of the spatial display of textpicture information on the user's cognitive processes. Two experiments were carried out to compare three displays on a computer screen: 'split' display (text and picture information displayed in separate areas on the screen), 'integrated' display (text information close to the part of the picture to which it refers), and 'pop-up' display (text information integrated in pop-up fields which appeared only via the user's action). In both experiments, the results showed that the integrated display and to a greater extent the pop-up display led to higher performances for an equal or lower learning time. Thus, these experiments reinforce the hypothesis that material where text and picture are integrated improves learning, especially if text information appears in pop-up fields. Results are discussed from a theoretical and a practical point of view.     

Interaction of textual and graphical information in locating web page widgets

Current models of web navigation focus only on the influence of textual information and ignore the role of graphical information. We studied the differential role of text and graphics in identifying web page widgets classified into two kinds: textual and graphical. Four different versions of web pages were created by systematically removing textual and graphical information from each page. The task of the participants was to locate either textual or graphical widgets on the displayed web page. Results show that for any widget, the task-completion time and the number of clicks were significantly less in web pages with graphics than in those with no graphics. This demonstrates the importance of graphical information. However, textual information is also important because performance in locating graphical widgets under no-graphics conditions was better when text was present than with no text. Since, for identifying graphical widgets, text and graphics interact and complement each other, we conclude that cognitive models on web navigation should include the role of graphical information next to textual information.     

Cognitive and affective effects of seductive details in multimedia learning

The present study integrates cognitive and affective aspects of media processing in order to make an argument for reexamining the current cold cognition perspective in multimedia research in favor of a more integrative perspective. The Cognitive-Affective-Theory-of-Learning-with-Media (CATLM) assumes that students need to become motivated to make full use of their cognitive resources. Therefore, and even though seductive details (sds) are additional interesting but unnecessary pieces of information that do not conform with the coherence principle, their possible motivational role should not be dismissed. Using a 2 × 3-experimental design, participants (N = 123) were asked to learn about biology with multimedia instruction that manipulated modality (text vs. narration) and presence of seductive details (no-sds vs. textual-sds vs. narrated-sds). Results of variance analyses show a modality effect. In addition, moderated mediation analyses with the moderator modality and mediator situational interest confirm the affective mediation assumption with the following two conditional effects. A direct detrimental effect of seductive details on learning performance under the text-condition and an indirect compensatory effect under the narration-condition were shown.     

The Effects of Different Presentation Modalities on a Person Identification Task: A Study of Wireless Multimedia PDA Use in a Security Context

The convergence of affordable multimedia capabilities with local and wide area wireless data networking on handheld devices is a particularly important trend. Using off-the-shelf technology, we conducted a simple experiment that required participants to engage in a person-identification task while posing as security agents in a field setting. Participants attempted to identify a target person in a public setting under different conditions with or without the aid of video information distributed to them via wireless-enabled multimedia PDAs. We developed a research framework and expectations based on previous research in the areas of the dual processing theory of working memory, information richness theory, and the role of video in grounding. Our findings suggest that dual modes of presentation (i.e., phonological and visuospatial information) simultaneously incorporated by a specific presentation significantly influence user perception and enhance task performance.     

The effect of picture attributes of online ordering pages on visual attention and user experience

The popularity of online ordering services is increasing daily, with food pictures playing a critical role in influencing user attention and experience. In this study, 58 participants were enrolled in a 2 (saturation: higher vs. lower) × 3 (Area of interest: category vs. picture vs. text) within-subject experiment using large and small pictures to evaluate the impact of picture attributes on user responses. A combination of eye-tracking technology and a questionnaire survey was used for the measurement. The results suggest that for larger food pictures, the picture area received more attention than the category and text areas when pictures were highly saturated, whereas the category information was more appealing than other areas when pictures were low saturated, and the user experience of the overall page was higher than in other cases. For smaller food pictures, the picture area attracted initial attention, whereas the text area gained more fixation than the other areas. The findings of this study provide empirical suggestions for designing online interfaces for ordering services and webpages.     

Homogeneous and heterogeneous multiple representations in equation-solving problems: An eye-tracking study

Multiple external representations (MERs) play an important role in the learning field of mathematics. Whereas the cognitive theory of multimedia learning and the integrative text and picture comprehension model assume that the heterogeneous combination of symbolic and analogous representations fosters learning; the design, functions, and tasks framework holds that learning benefits depend on the specific functions of MERs. The current paper describes a conceptual replication study of one of the few studies comparing single representations, heterogeneous, and homogeneous MERs in the context of mathematics learning. In a balanced incomplete block design, the participants were provided single representations (a graphic, text, or formula) or a heterogeneous (e.g., text + graphic) or homogeneous (text + formula) combination of these to solve linear system of equations problems. In accordance with previous research, performance was superior in conditions providing MERs compared to single-representation conditions. Moreover, heterogeneous MERs led to time savings over homogeneous MERs which triggered an increase in cognitive load. Contrary to previous research, text was the least fixated representation whereas the graphical representation proved to be most beneficial. With regard to practical implications, experts should be fostered through more challenging homogeneous MERs whereas novices should be supported through the accessible graphic contained in heterogeneous MERs.     

Effects of VDT and paper presentation on consumption and production of information: Psychological and physiological factors

Two experiments were performed to investigate the influence of VDT (video display terminals) and paper presentation of text on consumption of information (Study 1) measured in the form of convergent production and production of information (Study 2) measured in form of divergent production. The READ test of reading comprehension was used as the convergent task whereas the “Headlines” test was used as the divergent task. Several other factors pertaining to performance were also studied including the PANAS test of positive and negative affect, the STH test of stress, tiredness and hunger, the TRI (Technology Readiness Inventory) and the SE test of stress and energy.The results show that performance in the VDT presentation condition where inferior to that of the Paper presentation condition for both consumption and production of information. Concomitantly, participants in the VDT presentation condition of the consumption of information study reported higher levels of experienced stress and tiredness whereas the participants in the VDT presentation condition of production of information study reported only slightly higher levels of stress.Although the results are discussed in both physiological and psychological terms arguments are made that the incremental effects of VDT text presentation stem mainly from dual-task effects of fulfilling the assignment and working with the computer resulting in a higher cognitive workload.     

基于多传递影响力的社交媒体谣言检测方法

社交媒体谣言检测是当前研究的热点问题,现有方法多数通过获取大量用户属性学习用户特征,但不适用于谣言的早期检测,忽略了用户之间的潜在关系对信息传播的影响。提出一种基于多传递影响力的谣言检测方法,根据源微博及其对应转发(评论)之间的关系构建文本信息传播图,并通过图卷积神经网络来捕获、学习文本信息的传播特征。利用文本信息和用户传播过程中的影响力,丰富可用于谣言检测早期的检测信息。将存在转发关系的用户构成用户影响力传播图,构建一种用户节点影响力学习方法,获取用户节点影响力,以增强用户特征信息。在此基础上,将文本特征与用户特征融合以进行谣言检测,从而提升检测效果。在3个真实社交媒体数据集上的实验结果表明,该方法在谣言自动检测以及早期检测的效果都有显著提升,与目前最好的基准方法相比,在微博、Twitter15、Twitter16数据集上的正确率分别提高了2.8%、6.9%和3.4%。     

TSPT：基于预训练的三阶段复合式文本摘要模型

抽取式方法从源文本中抽取句子,会造成信息冗余;生成式方法可以生成非源文词,会产生语法问题,自然性差。BERT作为一种双向Transformer模型,在自然语言理解任务上展现了优异的性能,但在文本生成任务的应用有待探索。针对以上问题,提出一种基于预训练的三阶段复合式文本摘要模型（TSPT）,结合抽取式方法和生成式方法,将源本文经过预训练产生的双向上下文信息词向量由sigmoid函数获取句子得分抽取关键句,在摘要生成阶段将关键句作为完形填空任务重写,生成最终摘要。实验结果表明,该模型在CNN/Daily Mail数据集中取得了良好效果。     

Can eye-tracking data be measured to assess product design?: Visual attention mechanism should be considered

Eye tracking probes user's perception of real-time reaction to products, while conventional methods (i.e. interviews, focus group, questionnaires and so on) have generally failed because they depend on users' willingness and competency to describe how they feel when they are exposed to a product. Two tasks were designed to explore the indexes of eye movement that can reflect user experience of product, and analyse the attention captured by product attributes and goal-oriented. In task one, participants just browsed two smart phone pictures and evaluated the whole user experience. Binary choices were used in task two to ask participants to select the smart phone picture with higher user experience and then click the mouse. The results showed that in the browsing task, participants had shorter time to first fixation for the smart phone picture with higher level of user experience than the lower. And pupil dilated significantly when participants browse smart phone picture with lower level of user experience. In goal-oriented task, participants' attentions were dominated by visual perception of task driven, mainly reflected on longer fixation time and larger pupil diameter when looking at the smart phone with higher level of user experience. These results support the notion that we cannot assess product design just by several eye-movement indexes without considering the effects of visual attention mechanism.Relevance to industryThe appearance of product plays an important role to attract user's attention and stimulate their intention to experience. And vision is the main channel for users to obtain product information. Hence a thorough research on the inherent mechanism of vision perception can provide technical support for product designers, which in turn can attract more consumers to experience the product, even buy it. Moreover, the seller can find out the real buyers and predict their desired products by tracking user's eyes.     

Motivation and interface design: maximising learning opportunities

Abstract Instructional multimedia is creating a revolution in universities. Academic staff are being urged to transfer instruction to a format suitable for either CD/ROM and/or the Internet, principally the World Wide Web. Many, it seems, simply take their existing course materials, add image and sound without proper consideration of the nature of the medium in which they are seeking to instruct and present the product to their students. In essence, it appears they ignore the need to motivate their students to work with their instructional multimedia materials. It is a contention of this paper that the user-interface to instructional multimedia is strategically important: if it is poorly designed students will not be intrinsically motivated to make use of the product or to learn with it. Interfaces that motivate learners are realistic, easy to use, challenging and engaging. Superior interfaces have some of the elements of a game: they provide the user with a functional model of task, content and processes; they encourage exploration and engagement; and they demonstrate cognisance of design considerations such as interactivity, functionality, learner control and cognition.     

多媒体技术与计算机课程课堂教学分析

作为一种先进的、新兴的辅助教学方式,近些年来,多媒体技术逐渐开始被广泛地应用到课堂教学的过程当中,计算机的课堂教学可以充分借鉴其他学科应用多媒体技术的成功经验,将计算机课程的特点和多媒体技术的应用充分结合起来,提高计算机课程的课堂教学效果。     

How attitude strength biases information processing and evaluation on the web

In this study we examined the influence of attitude strength on the processing and evaluation of sources and information on the Web. Seventy-nine German university students participated in an eye-tracking study in which they read information from eight pre-selected websites from different sources on the controversial topic of organic foods. Results showed that participants who felt strongly about the topic (i.e., those with strong prior attitude strength) scrutinized website logos of attitude-inconsistent websites shorter and judged the credibility of attitude-inconsistent websites lower. They also included more attitude-consistent information in an essay task than participants with weaker prior attitudes. Participants who felt less strongly about the topic (i.e., those with weaker prior attitudes) fixated longer on text from attitude-inconsistent websites than from attitude-consistent websites. By contrast, for participants with strong prior attitudes the time spent on the texts from attitude-consistent websites and attitude-inconsistent websites did not differ significantly. The results show that prior attitudes can bias evaluation and processing of information in different ways. Even though participants were not fully biased during initial information processing, they were so when evaluating the information and presenting it in an essay task.