多通道用户界面

随着近年来计算机技术和INTERNET技术的飞速发展,用户界面由于其对最终用户使用的影响及对计算机推广应用的影响,已经越来越显现出它的重要性.多通道用户界面使用多种交互设备,通过多种交互方式的协作和互补,提高了交互效率,增进了交互自然性.首先介绍了多通道用户界面的概念以及各种交互技术,最后介绍了多通道用户界面的发展前景.     

Multimodal medical image fusion based on nonsubsampled shearlet transform and convolutional sparse representation

Multimedia Tools and Applications - Multimodal medical image fusion technology can assist doctors diagnose diseases accurately and efficiently. However the multi-scale decomposition based image...     

Multimodal emotion recognition based on manifold learning and convolution neural network

Multimedia Tools and Applications - Multimodal emotion recognition task based on physiological signals is becoming a research hotspot. Traditional methods need to design and extract a series of...     

Multimodal speaker clustering in full length movies

Multimedia Tools and Applications - Multimodal clustering/diarization tries to answer the question ”who spoke when” by using audio and visual information. Diarizationconsists of two...     

Adaptive fuzzy genetic algorithm for multi biometric authentication

Multimedia Tools and Applications - Biometric Authentication (BA) has turn out to be presently as key problem in privacy and security. Multimodal biometric system specializes in enhancing...     

Multimodal person discovery in broadcast TV: lessons learned from MediaEval 2015

Multimedia Tools and Applications - We describe the “Multimodal Person Discovery in Broadcast TV” task of MediaEval 2015 benchmarking initiative. Participants were asked to return the...     

Learning shared embedding representation of motion and text using contrastive learning

Artificial Life and Robotics - Multimodal learning of motion and text tries to find the correspondence between skeletal time-series data acquired by motion capture and the text that describes the...     

System for multimodal data acquisition for human action recognition

Multimedia Tools and Applications - Multimodal data is being used more widely for human action recognition nowadays due to the progress of machine learning methods and the development of new types...     

Multimodal data modeling for efficiency assessment of social priority based urban bus route transportation system using GIS and data envelopment analysis

Multimedia Tools and Applications - Multimodal data modeling is fast growing area of research. It may used to combine the information from the different sources. The research interest in multimodal...     

融合多模态深度游走与偏差校准因子的推荐模型

曝光偏差严重影响协同过滤模型的推荐精度,导致预测结果偏离用户的真实兴趣,而现有模型对曝光偏差的建模能力有限,甚至放大偏差。为此,提出融合多模态深度游走与偏差校准因子（MmDW-BC）的推荐模型。首先,引入项目多模态属性特征作为项目图的连接边,从而缓解低曝光项目交互数据稀疏的问题;在此基础上,构建图嵌入模块——多模态深度游走（MmDW）将项目多模态信息融入嵌入向量,以获取丰富的节点表示;最后,基于校准策略设计新的偏差校准推荐算法进行用户偏好预测。将提出的模型应用于Amazon和ML-1M数据集上,实验结果验证所提模型明确考虑曝光偏差来提升推荐精度的必要性和有效性。     

基于多任务学习的时序多模态情感分析模型

针对时序多模态情感分析中存在的单模态特征表示和跨模态特征融合问题,结合多头注意力机制,提出一种基于多任务学习的情感分析模型。首先,使用卷积神经网络（CNN）、双向门控循环神经网络（BiGRU）和多头自注意力（MHSA）实现了对时序单模态的特征表示;然后,利用多头注意力实现跨模态的双向信息融合;最后,基于多任务学习思想,添加额外的情感极性分类和情感强度回归任务作为辅助,从而提升情感评分回归主任务的综合性能。实验结果表明,相较于多模态分解模型,所提模型的二分类准确度指标在CMU-MOSEI和CMU-MOSI多模态数据集上分别提高了7.8个百分点和3.1个百分点。该模型适用于多模态场景下的情感分析问题,能够为商品推荐、股市预测、舆情监控等应用提供决策支持。     

Intelligent video and audio applications for learning enhancement

The role of computers in school education is briefly discussed. Multimodal interfaces development history is shortly reviewed. Examples of applications of multimodal interfaces for learners with special educational needs are presented, including interactive electronic whiteboard based on video image analysis, application for controlling computers with facial expression and speech stretching audio interface representing audio modality. Intelligent and adaptive algorithms applications to the developed multimodal interfaces are discussed.     

多模态生物特征识别技术进展综述

近年来,生物特征识别已经成为一种最具潜力的身份认证技术之一.主要对多模态生物特征识别技术进行概括和总结,同时也介绍了现存若干用于身份识别和验证的多模态生物特征识别系统.经调查研究显示,多模态生物特征识别技术由于在身份认证和识别过程中考虑了个体的多种生理或行为特征,因而表现出了较单生物特征更高的可靠性和安全性,并已成为生物特征识别技术未来发展的趋势之一.     

人机交互的若干关键技术

人机交互(Human-ComputerInteraction)是研究人、计算机以及它们相互影响的技术。人机结合以人为主,将是未来计算机系统的特点,实现人机高效合作将是新一代人机界面的主要目的。多通道用户界面、计算机支持的协同工作、三维人机交互等是实现高效自然的人机交互的关键技术。     

Extending multimedia languages to support multimodal user interactions

Historically, the Multimedia community research has focused on output modalities, through studies on timing and multimedia processing. The Multimodal Interaction community, on the other hand, has focused on user-generated modalities, through studies on Multimodal User Interfaces (MUI). In this paper, aiming to assist the development of multimedia applications with MUIs, we propose the integration of concepts from those two communities in a unique high-level programming framework. The framework integrates user modalities —both user-generated (e.g., speech, gestures) and user-consumed (e.g., audiovisual, haptic)— in declarative programming languages for the specification of interactive multimedia applications. To illustrate our approach, we instantiate the framework in the NCL (Nested Context Language) multimedia language. NCL is the declarative language for developing interactive applications for Brazilian Digital TV and an ITU-T Recommendation for IPTV services. To help evaluate our approach, we discuss a usage scenario and implement it as an NCL application extended with the proposed multimodal features. Also, we compare the expressiveness of the multimodal NCL against existing multimedia and multimodal languages, for both input and output modalities.

     

Using constraint propagation in MAGPIE: a multi-agent approach

Multimodal presentation planning must solve the constraints imposed by the structure of the discourse. In this paper, we discuss a multi-agent implementation of the design and realization layers of the Standard Reference Model (SRM) for Intelligent Multimedia Presentation Systems (IMMPS), where design constraints for multimodal presentations are distributed over a number of agents. Thus, a constraint propagation mechanism is required so that the distributed constraints can be satisfied cooperatively by these agents.     

基于特征融合的多模态身份识别方法研究

研究了多模态身份识别问题,结合人脸和掌纹两种不同生理特征,提出了基于特征融合的多模态身份识别方法。对人脸和掌纹图像分别进行Gabor小波、二维主元变换(2DPCA)提取图像特征,根据新的权重算法,结合两种模态的特征,利用最邻近分类器进行分类识别。在AMP、ORL人脸库和Poly-U掌纹图像库中的实验结果表明,两种模态的融合能更多地给出决策分析所需的特征信息相比传统的单一模态的人脸或掌纹识别具有较高的识别率,更具安全性和准确性。     

DMRFNet: Deep Multimodal Reasoning and Fusion for Visual Question Answering and explanation generation

Visual Question Answering (VQA), which aims to answer questions in natural language according to the content of image, has attracted extensive attention from artificial intelligence community. Multimodal reasoning and fusion is a central component in recent VQA models. However, most existing VQA models are still insufficient to reason and fuse clues from multiple modalities. Furthermore, they are lack of interpretability since they disregard the explanations. We argue that reasoning and fusing multiple relations implied in multimodalities contributes to more accurate answers and explanations. In this paper, we design an effective multimodal reasoning and fusion model to achieve fine-grained multimodal reasoning and fusion. Specifically, we propose Multi-Graph Reasoning and Fusion (MGRF) layer, which adopts pre-trained semantic relation embeddings, to reason complex spatial and semantic relations between visual objects and fuse these two kinds of relations adaptively. The MGRF layers can be further stacked in depth to form Deep Multimodal Reasoning and Fusion Network (DMRFNet) to sufficiently reason and fuse multimodal relations. Furthermore, an explanation generation module is designed to justify the predicted answer. This justification reveals the motive of the model’s decision and enhances the model’s interpretability. Quantitative and qualitative experimental results on VQA 2.0, and VQA-E datasets show DMRFNet’s effectiveness.     

From signals to knowledge: A conceptual model for multimodal learning analytics

Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science.     

多模态情感分析研究综述

多模态情感分析现已成为自然语言处理领域的核心研究课题之一.文中首先介绍多模态情感分析的研究背景,归纳该领域最新的两类子课题,即叙述式多模态情感分析与交互式多模态情感分析.再分别以这两类子课题为出发点,梳理概括相应的发展脉络,最后,总结目前该领域中存在的交互建模的科学问题,探讨未来发展方向.