融合触、听、视觉的多通道认知和交互模型

以触觉相关的多通道交互研究为立足点,结合经典的多通道假设和最新的认知理论,提出一种融合触觉、听觉、视觉的多通道信息认知加工模型,并就计算机端的信息处理过程提出了多通道交互的分层处理模型,分析了相应的多通道整合方法.该模型对交互界面与程序主体功能定义不同的实现路径,区分交互设备和交互信息处理过程,有利于从不同角度对交互界面的研究工作进行简化,避免高耦合度带来的冗余工作量;实现了一个融合触、听、视觉交互的实例.实例结果表明,利用文中模型能够降低多通道交互研究的分析难度并提高实验效率.     

电子沙盘应用的多通道交互系统设计与实现

多通道交互技术是人机交互研究领域的一个重要方面。将多通道交互技术应用于电子沙盘中,为指挥人员提供自然、高效的交互方式,必将大大提高指挥人员的指挥效率。文中采用模块化的思想描述以电子沙盘为平台的多通道交互系统架构,重点阐述了任务分析验证和多通道整合方法。其中任务分析与验证主要实现基于原子操作的交互任务模板构建和操作安全验证;多通道整合则基于任务模板最优匹配方法实现桌面协作机制。并基于上述设计实现了一个态势研讨原型系统。     

面向儿童的多通道交互系统

设计和实现了一个基于笔和语音的面向儿童的多通道三维交互系统.系统中包含了基于笔和语音的交互信息整合框架,用来整合儿童输入的笔和语音信息.同时,系统中定义了一些基于笔和语音的交互技术,可以支持孩子们以自然的方式,通过笔和语音同系统进行交互.用笔来勾画三维的场景和小动物等实体,同时用笔和语音同场景和场景中的实体进行一定的交互.     

虚拟城市规划中的多通道整合

将多通道交互用于虚拟环境,研究自然和谐的交互方式已经成为虚拟现实的一个重要研究方向.文中以城市规划为应用背景,融合跟踪器、语音和笔输入,提出概率合一的任务制导多通道整合,并辅以上下文语义;以此为基础,有针对性地设计了多通道交互技术,并依据多通道交互的特性分别进行阐述,实现了自然、高效的多通道虚拟城市规划.     

多通道整合的相关问题及算法

多通道界面旨在充分利用一个以上的感觉和动作通道来捕捉用户的意向，提高人机交互的自然性和高效性，其核心问题就是通道整合的问题。该文针对该问题提出了一个基于分层的任务模型的整合算法，该算法处理中涉及到程序运行状态表示、任务结构的设计和交互原语相关性表示等问题。针对以上3个问题展开讨论，在此基础上得到最后的算法。     

多通道延时遥交互研究综述

多通道遥交互(Multimodal Tele-interaction,MMTI)旨在通过使用多种交互设备和协作方式,并利用多交互通道间的互补特性,以便有效传达和理解用户交互信息,提高交互效率,增进交互自然性,最终使用户能够以“预期的想法”完成遥交互任务。近年来,随着多通道遥交互的发展,人们对深空、深海和远程医疗的探索和开发不断增强,由于通信时延的约束,多通道遥交互面临着交互异步和通道缺失等问题,对用户行为、心理和认知等人素特性产生了根本影响,切断和阻碍了交互通道的连续性、实时性和自然性,降低了交互的用户体验,并进一步影响了系统的有效性,因此迫切需要对大时延约束下的多通道遥交互技术进行研究。分析了国内外研究现状,给出了遥交互的一个定义,讨论了遥交互研究问题和关键技术(包括时延、异步和缺失问题),讨论了其关键应用领域,最后展望了遥交互的发展趋势和研究挑战。     

基于函数截获的、面向图形应用的远程交互分布式设备框架

随着对交互和高效利用资源的需求的增加,用户渴望与远程多通道图形应用程序进行实时可视化和交互.由于大多数此类应用程序通过除鼠标和键盘外的其他设备进行交互,因此必须支持这些设备的远程交互.目前,这种远程交互可以通过修改应用程序的源代码实现.当源代码不可访问时,该方法不可行.提出一个基于函数调用截获的、轻量级的分布式框架,在不修改应用程序的源代码的情况下实现这些设备的远程交互.基于该框架,实现了分布式DirectInput系统.实验结果表明通过集成该系统与远程交互系统,在局域网上,用户能对远程的、由手柄驱动的多通道图形应用程序进行实时可视化和交互.     

任务制导的多通道分层整合模型及其算法

多通道整合是多通道交互系统的核心,旨在对来自不同通道的信息进行合一化处理,最终向计算机正确传达用户意图。围绕多通道整合问题,通过分析多通道输入信息流的特点,提出了一种任务制导的、分层的整合模型及相应算法,改变了通道间输入信息对时间的依赖方式,使人机交互更为自然和高效,该整合机制与宿主系统及交互设备无关,具有良好的通用性。     

移动环境下神经功能评价多通道交互模型

随着平板电脑、智能手机、智能手表等智能移动设备的普及,利用便携的智能移动设备随时随地进行健康评价受到了国内外学者的广泛关注.人机交互特有的多通道、交互式、人机协同的计算能够有效地提高移动环境下神经功能评价的准确度.然而,目前很少有研究对人机交互在这一应用场景上发挥的作用进行过充分讨论,更没有形成统一的多通道交互模型.为此,首先分析了目前移动设备上主流的神经功能评价方法,归纳总结出了一套适用该应用场景的交互原语和交互任务.然后,在此基础上提出了移动环境下神经功能评价多通道人机交互模型——MINA（multimodal human-computer interaction model for nerve function assessment in mobile environment),并对该模型的移动医学评价和多通道融合特点进行了分析.最后,依据此模型给出神经系统疾病检测的应用实例.实践证明,MINA能够较好地指导交互式神经功能评价应用的设计和开发,多通道融合的方式能够有效地提高医学评价的准确度.     

手持移动计算中的人机交互技术研究

叙述了在手持移动计算设备上的基于上下文感知的多通道交互模型、多通道汉字输入、移动计算中的小屏幕网络浏览策略等方面的研究成果。实现了一个智能导游系统TGH,对这些模型和方法进行了应用和测试。最后总结了手持移动计算中人机交互技术研究的一些结论。     

基于任务的多通道整合设计与实例

多通道人机交互技术的研究旨在为计算机用户提供一个方便高效的人机交互环境。     

Multimodal integration-a statistical view

We present a statistical approach to developing multimodal recognition systems and, in particular, to integrating the posterior probabilities of parallel input signals involved in the multimodal system. We first identify the primary factors that influence multimodal recognition performance by evaluating the multimodal recognition probabilities. We then develop two techniques, an estimate approach and a learning approach, which are designed to optimize accurate recognition during the multimodal integration process. We evaluate these methods using Quickset, a speech/gesture multimodal system, and report evaluation results based on an empirical corpus collected with Quickset. From an architectural perspective, the integration technique presented offers enhanced robustness. It also is premised on more realistic assumptions than previous multimodal systems using semantic fusion. From a methodological standpoint, the evaluation techniques that we describe provide a valuable tool for evaluating multimodal systems     

A self-organized artificial neural network architecture for sensory integration with applications to letter-phoneme integration

The multimodal self-organizing network (MMSON), an artificial neural network architecture carrying out sensory integration, is presented here. The architecture is designed using neurophysiological findings and imaging studies that pertain to sensory integration and consists of interconnected lattices of artificial neurons. In this artificial neural architecture, the degree of recognition of stimuli, that is, the perceived reliability of stimuli in the various subnetworks, is included in the computation. The MMSON's behavior is compared to aspects of brain function that deal with sensory integration. According to human behavioral studies, integration of signals from sensory receptors of different modalities enhances perception of objects and events and also reduces time to detection. In neocortex, integration takes place in bimodal and multimodal association areas and result, not only in feedback-mediated enhanced unimodal perception and shortened reaction time, but also in robust bimodal or multimodal percepts. Simulation data from the presented artificial neural network architecture show that it replicates these important psychological and neuroscientific characteristics of sensory integration.     

Animated interface agent applying atms-based multimodal input interpretation

Two requirements should be met in order to develop a practical multimodal interface system , i . e ., ( 1 ) integration of delayed arrival of data and ( 2 ) elimination of ambiguity in recognition results of each modality . This paper presents an efficient and generic methodology for interpretation of multimodal input to satisfy these requirements . The proposed methodology can integrate delayed - arrival data satisfactorily and efficiently interpret multimodal input that contains ambiguity . In the input interpretation the multimodal interpretation process is regarded as hypothetical reasoning , and the control mechanismof interpretation is formalized by applying the assumption - based truth maintenance system ( ATMS ). The proposed method is applied to an interface agent system that accepts multimodal input consisting of voice and direct indication gesture on a touch display . The systemcommunicates to the user through a human - like interface agent's three - dimensional motion image with facial expressions , gestures , and a synthesized voice .     

计算机手势输入及其在人机交互技术中的应用

本文简介计算机手势识别技术的基本手段,方法和技术,讨论手势作为人机交互通道所具有的特点,并介绍作者在多通道用户界面的研究中实现的对手势的整合技术。     

Type-1 and type-2 fuzzy inference systems as integration methods in modular neural networks for multimodal biometry and its optimization with genetic algorithms

We describe in this paper a comparative study between fuzzy inference systems as methods of integration in modular neural networks for multimodal biometry. These methods of integration are based on techniques of type-1 fuzzy logic and type-2 fuzzy logic. Also, the fuzzy systems are optimized with simple genetic algorithms with the goal of having optimized versions of both types of fuzzy systems. First, we considered the use of type-1 fuzzy logic and later the approach with type-2 fuzzy logic. The fuzzy systems were developed using genetic algorithms to handle fuzzy inference systems with different membership functions, like the triangular, trapezoidal and Gaussian; since these algorithms can generate fuzzy systems automatically. Then the response integration of the modular neural network was tested with the optimized fuzzy systems of integration. The comparative study of the type-1 and type-2 fuzzy inference systems was made to observe the behavior of the two different integration methods for modular neural networks for multimodal biometry.