融合动作特征的多模态情绪识别

近年来,利用计算机技术实现基于多模态数据的情绪识别成为自然人机交互和人工智能领域重要 的研究方向之一。利用视觉模态信息的情绪识别工作通常都将重点放在脸部特征上,很少考虑动作特征以及融合 动作特征的多模态特征。虽然动作与情绪之间有着紧密的联系,但是从视觉模态中提取有效的动作信息用于情绪 识别的难度较大。以动作与情绪的关系作为出发点,在经典的 MELD 多模态情绪识别数据集中引入视觉模态的 动作数据,采用 ST-GCN 网络模型提取肢体动作特征,并利用该特征实现基于 LSTM 网络模型的单模态情绪识别。 进一步在 MELD 数据集文本特征和音频特征的基础上引入肢体动作特征,提升了基于 LSTM 网络融合模型的多 模态情绪识别准确率,并且结合文本特征和肢体动作特征提升了上下文记忆模型的文本单模态情绪识别准确率, 实验显示虽然肢体动作特征用于单模态情绪识别的准确度无法超越传统的文本特征和音频特征,但是该特征对于 多模态情绪识别具有重要作用。基于单模态和多模态特征的情绪识别实验验证了人体动作中含有情绪信息,利用 肢体动作特征实现多模态情绪识别具有重要的发展潜力。     

Facial expression recognition and tracking for intelligent human-robot interaction

For effective interaction between humans and socially adept, intelligent service robots, a key capability required by this class of sociable robots is the successful interpretation of visual data. In addition to crucial techniques like human face detection and recognition, an important next step for enabling intelligence and empathy within social robots is that of emotion recognition. In this paper, an automated and interactive computer vision system is investigated for human facial expression recognition and tracking based on the facial structure features and movement information. Twenty facial features are adopted since they are more informative and prominent for reducing the ambiguity during classification. An unsupervised learning algorithm, distributed locally linear embedding (DLLE), is introduced to recover the inherent properties of scattered data lying on a manifold embedded in high-dimensional input facial images. The selected person-dependent facial expression images in a video are classified using the DLLE. In addition, facial expression motion energy is introduced to describe the facial muscle’s tension during the expressions for person-independent tracking for person-independent recognition. This method takes advantage of the optical flow which tracks the feature points’ movement information. Finally, experimental results show that our approach is able to separate different expressions successfully.     

基于姿态识别的机器人人机交互系统设计

针对现有遥机器人人机交互系统依赖于视频信息使操控性能不稳定的问题,设计并实现一种基于姿态传感器的人机交互系统。利用姿态传感器获取人的姿态信息并对其进行解算,通过基于D‐S证据理论的信息融合法对姿态进行判定,使用语音确定动作指令的发送,对复杂动作进行组合,简化操作,通过“双虚拟模型”获取远端机器人的姿态,对运动轨迹进行修正。实验结果表明,该系统具有高稳定、高准确率及易操作的特点。     

基于交互注意力机制的多模态情感识别算法

在多模态语音情感识别中,现有的研究通过提取大量特征来识别情感,但过多的特征会导致关键特征被淹没在相对不重要特征里,造成关键信息遗漏.为此提出了一种模型融合方法,通过两种注意力机制来寻找可能被遗漏的关键特征.本方法在IEMOCAP数据集上的四类情感识别准确率相比现有文献有明显提升;在注意力机制可视化下,两种注意力机制分别找到了互补且对人类情感识别重要的关键信息,从而证明了所提方法相比传统方法的优越性.     

Controller design for human-robot interaction

Many robotics tasks require a robot to share the same workspace with humans. In such settings, it is important that the robot performs in such a way that does not cause distress to humans in the workspace. In this paper, we address the problem of designing robot controllers which minimize the stress caused by the robot while performing a given task. We present a novel, data-driven algorithm which computes human-friendly trajectories. The algorithm utilizes biofeedback measurements and combines a set of geometric controllers to achieve human friendliness. We evaluate the comfort level of the human using a Galvanic Skin Response (GSR) sensor. We present results from a human tracking task, in which the robot is required to stay within a specified distance without causing high stress values.

AR-based interaction for human-robot collaborative manufacturing

Industrial standards define safety requirements for Human-Robot Collaboration (HRC) in industrial manufacturing. The standards particularly require real-time monitoring and securing of the minimum protective distance between a robot and an operator. This paper proposes a depth-sensor based model for workspace monitoring and an interactive Augmented Reality (AR) User Interface (UI) for safe HRC. The AR UI is implemented on two different hardware: a projector-mirror setup and a wearable AR gear (HoloLens). The workspace model and UIs are evaluated in a realistic diesel engine assembly task. The AR-based interactive UIs provide 21–24% and 57–64% reduction in the task completion and robot idle time, respectively, as compared to a baseline without interaction and workspace sharing. However, user experience assessment reveal that HoloLens based AR is not yet suitable for industrial manufacturing while the projector-mirror setup shows clear improvements in safety and work ergonomics.     

Pre-collision safety strategies for human-robot interaction

Safe planning and control is essential to bringing human-robot interaction into common experience. This paper presents an integrated human−robot interaction strategy that ensures the safety of the human participant through a coordinated suite of safety strategies that are selected and implemented to anticipate and respond to varying time horizons for potential hazards and varying expected levels of interaction with the user. The proposed planning and control strategies are based on explicit measures of danger during interaction. The level of danger is estimated based on factors influencing the impact force during a human-robot collision, such as the effective robot inertia, the relative velocity and the distance between the robot and the human. A second key requirement for improving safety is the ability of the robot to perceive its environment, and more specifically, human behavior and reaction to robot movements. This paper also proposes and demonstrates the use of human monitoring information based on vision and physiological sensors to further improve the safety of the human robot interaction. A methodology for integrating sensor-based information about the user's position and physiological reaction to the robot into medium and short-term safety strategies is presented. This methodology is verified through a series of experimental test cases where a human and an articulated robot respond to each other based on the human's physical and physiological behavior.

Multimodal emotion recognition based on manifold learning and convolution neural network

Multimedia Tools and Applications - Multimodal emotion recognition task based on physiological signals is becoming a research hotspot. Traditional methods need to design and extract a series of...     

Multimodal information fusion application to human emotion recognition from face and speech

A multimedia content is composed of several streams that carry information in audio, video or textual channels. Classification and clustering multimedia contents require extraction and combination of information from these streams. The streams constituting a multimedia content are naturally different in terms of scale, dynamics and temporal patterns. These differences make combining the information sources using classic combination techniques difficult. We propose an asynchronous feature level fusion approach that creates a unified hybrid feature space out of the individual signal measurements. The target space can be used for clustering or classification of the multimedia content. As a representative application, we used the proposed approach to recognize basic affective states from speech prosody and facial expressions. Experimental results over two audiovisual emotion databases with 42 and 12 subjects revealed that the performance of the proposed system is significantly higher than the unimodal face based and speech based systems, as well as synchronous feature level and decision level fusion approaches.     

A force reflected exoskeleton-type masterarm for human-robot interaction

Two human-robot interactions, including a haptic interaction and a teleoperated interaction, are explored with a new exoskeleton-type masterarm, in which the electric brakes with the torque sensor beams are used for force reflection. In the haptic interaction with virtual environment, the masterarm is used as a haptic device and tested to examine how the resistant torque of the electric brake for the force reflection is implemented in contact regime prior to conducting the teleoperated interaction. Two types of virtual environments, a rigid wall with high stiffness (hard contact with 10 [KN/m]) and a soft wall with low stiffness (soft contact with 0.1 [N/m]), are integrated with the masterarm for the haptic interaction. In hard contact, large force is fed back to the human operator, and makes the human operator hardly move. The electric brake with the torque sensor beam can detect the torque and its direction so that it allows free motion as well as contact motion by releasing or holding the movement of the operator. The experimental results show how the electric brake is switched from contact to free regime to allow the operator to move freely, especially when the operator intends to move toward the free regime in contact. In soft contact, the force applied to the human operator can be increased or decreased proportionally to the torque amount sensed by the torque sensor beam, thus the operator can feel the contact force proportional to the amount of the deformation during the contact. Finally, the masterarm is integrated with the humanoid robot, CENTAUR developed at Korea Institute of Science and Technology to conduct a pick-and-place task through the teleoperated interaction. It is examined that the CENTAUR as a slave robot can follow the movement of the operator.     

人机交互中的个性化情感模型

人与机器人的交互过程中,情感因素的引入能够使人机交流更加自然和谐.因此,完整的人工情感模型的建立是首要解决的问题.基于情感能量理论基础,首先,提出了心境自发转移和刺激转移模型.其次,结合情绪自发转移的马尔可夫链模型和刺激转移的HMM模型,将心境和情绪的自发和刺激转移过程统一在一个框架下.最后,将完整的人工情感模型软件化并应用于儿童玩伴机器人上,在接受非结构化环境与用户的信息输入后,个性化的情感软件模块产生输出,实现针对儿童用户的玩伴机器人个性化交互,通过应用验证了该模型的有效性.     

面向情绪调节的多模态人机交互技术

目的 现代社会存在心理问题的人日趋增多,及时调节其负面情绪对社会和谐稳定具有重要现实意义。传统的情绪调节方法需要花费大量人力,为此提出一种面向情绪调节的多模态人机交互方法,识别用户情绪,采用文本对话和体感交互实现对用户情绪的调节。方法 综合运用了表情识别、文本对话和手势来实现对用户情绪的识别,构建了具有情绪表现力的智能体。用户的表情识别采用支持向量机方法,文本对话采用基于规则和融入情绪因素的Seq2Seq模型实现。设计了聊天、过生日和互动游戏（打篮球）等交互剧情,采用手势和肢体动作来辅助交互。为了更好地体现情绪调节的作用,在互动游戏中设计了强化学习算法,可根据用户的情绪反馈来自动调整游戏难度,最终让用户情绪调整到积极状态。结果 通过实验发现,采用单模态交互很难感知到用户的背景信息,因此对用户的情绪识别可能出现误判。而采用多模态的人机交互方式,可以通过文本对话了解用户的背景信息,从而对用户情绪的识别更为合理。在多模态的人机交互剧情中,用户能以更自然的方式实现情景互动,智能体在多模态交互中更能发挥情绪调节作用。结论 本文提出一种基于多模态人机交互的情绪调节方法,该方法不需要昂贵的硬件设备,便于推广普及,为消极情绪的调节提供了一种可计算方案。     

Control sharing in human-robot team interaction

The interaction between humans and robot teams is highly relevant in many application domains, for example in collaborative manufacturing, search and rescue, and logistics. It is well-known that humans and robots have complementary capabilities: Humans are excellent in reasoning and planning in unstructured environments, while robots are very good in performing tasks repetitively and precisely. In consequence, one of the key research questions is how to combine human and robot team decision making and task execution capabilities in order to exploit their complementary skills. From a controls perspective this question boils down to how control should be shared among them. This article surveys advances in human-robot team interaction with special attention devoted to control sharing methodologies. Additionally, aspects affecting the control sharing design, such as human behavior modeling, level of autonomy and human-machine interfaces are identified. Open problems and future research directions towards joint decision making and task execution in human-robot teams are discussed.     

实现人身和机器人交流的神经振动子控制算法

为实现人身和机器人交流时的运动同步,首先提出了一类非线性多关节神经振动子运动控制算法,其输入为机器人和人相互作用所产生的关节扭矩信号,输出为机器人关节期望角度;然后对具有代表性的二关节神经振动子控制算法中各参数的耦合特性进行了分析;最后,基于7自由度机器人臂平台对该神经振动子控制算法的有效性进行实验.实验结果表明,该控制算法能够实现人和机器人相互运动的同步,通过调节神经振动子的增益参数,同步的程度能够被改变.     

Human-object integrated assembly intention recognition for context-aware human-robot collaborative assembly

Human-robot collaborative (HRC) assembly combines the advantages of robot's operation consistency with human's cognitive ability and adaptivity, which provides an efficient and flexible way for complex assembly tasks. In the process of HRC assembly, the robot needs to understand the operator's intention accurately to assist the collaborative assembly tasks. At present, operator intention recognition considering context information such as assembly objects in a complex environment remains challenging. In this paper, we propose a human-object integrated approach for context-aware assembly intention recognition in the HRC, which integrates the recognition of assembly actions and assembly parts to improve the accuracy of the operator's intention recognition. Specifically, considering the real-time requirements of HRC assembly, spatial-temporal graph convolutional networks (ST-GCN) model based on skeleton features is utilized to recognize the assembly action to reduce unnecessary redundant information. Considering the disorder and occlusion of assembly parts, an improved YOLOX model is proposed to improve the focusing capability of network structure on the assembly parts that are difficult to recognize. Afterwards, taking decelerator assembly tasks as an example, a rule-based reasoning method that contains the recognition information of assembly actions and assembly parts is designed to recognize the current assembly intention. Finally, the feasibility and effectiveness of the proposed approach for recognizing human intentions are verified. The integration of assembly action recognition and assembly part recognition can facilitate the accurate operator's intention recognition in the complex and flexible HRC assembly environment.     

Model reference adaptive impedance control for physical human-robot interaction

This paper presents a novel enhanced human-robot interaction system based on model reference adaptive control. The presented method delivers guaranteed stability and task performance and has two control loops. A robot-specific inner loop, which is a neuroadaptive controller, learns the robot dynamics online and makes the robot respond like a prescribed impedance model. This loop uses no task information, including no prescribed trajectory. A task-specific outer loop takes into account the human operator dynamics and adapts the prescribed robot impedance model so that the combined human-robot system has desirable characteristics for task performance. This design is based on model reference adaptive control, but of a nonstandard form. The net result is a controller with both adaptive impedance characteristics and assistive inputs that augment the human operator to provide improved task performance of the human-robot team. Simulations verify the performance of the proposed controller in a repetitive point-to-point motion task. Actual experimental implementations on a PR2 robot further corroborate the effectiveness of the approach.     

Resampling methods for meta-model validation with recommendations for evolutionary computation

Meta-modeling has become a crucial tool in solving expensive optimization problems. Much of the work in the past has focused on finding a good regression method to model the fitness function. Examples include classical linear regression, splines, neural networks, Kriging and support vector regression. This paper specifically draws attention to the fact that assessing model accuracy is a crucial aspect in the meta-modeling framework. Resampling strategies such as cross-validation, subsampling, bootstrapping, and nested resampling are prominent methods for model validation and are systematically discussed with respect to possible pitfalls, shortcomings, and specific features. A survey of meta-modeling techniques within evolutionary optimization is provided. In addition, practical examples illustrating some of the pitfalls associated with model selection and performance assessment are presented. Finally, recommendations are given for choosing a model validation technique for a particular setting.