Expressing reactive emotion based on multimodal emotion recognition for natural conversation in human–robot interaction*

Human–human interaction consists of various nonverbal behaviors that are often emotion-related. To establish rapport, it is essential that the listener respond to reactive emotion in a way that makes sense given the speaker's emotional state. However, human–robot interactions generally fail in this regard because most spoken dialogue systems play only a question-answer role. Aiming for natural conversation, we examine an emotion processing module that consists of a user emotion recognition function and a reactive emotion expression function for a spoken dialogue system to improve human–robot interaction. For the emotion recognition function, we propose a method that combines valence from prosody and sentiment from text by decision-level fusion, which considerably improves the performance. Moreover, this method reduces fatal recognition errors, thereby improving the user experience. For the reactive emotion expression function, the system's emotion is divided into emotion category and emotion level, which are predicted using the parameters estimated by the recognition function on the basis of distributions inferred from human–human dialogue data. As a result, the emotion processing module can recognize the user's emotion from his/her speech, and expresses a reactive emotion that matches. Evaluation with ten participants demonstrated that the system enhanced by this module is effective to conduct natural conversation.     

Meaningful conversation with mobile robots

We describe an implementation integrating a complete spoken dialogue system with a mobile robot, which a human can direct to specific locations, ask for information about its status and supply information about its environment. The robot uses an internal map for navigation, and communicates its current orientation and accessible locations to the dialogue system using a topological map as interface. We focus on linguistic and inferential aspects of the human–robot communication process. The result is a novel approach using a principled semantic theory combined with techniques from automated deduction applied to a mobile robot platform. Due to the abstract level of the dialogue system, it is easily portable to other environments or applications.     

Robot, asker of questions

Collaborative control is a teleoperation system model based on human–robot dialogue. With this model, the robot asks questions to the human in order to obtain assistance with cognition and perception. This enables the human to function as a resource for the robot and help to compensate for limitations of autonomy. To understand how collaborative control influences human–robot interaction, we performed a user study based on contextual inquiry (CI). The study revealed that: (1) dialogue helps users understand problems encountered by the robot and (2) human assistance is a limited resource that must be carefully managed.     

Data-driven models for timing feedback responses in a Map Task dialogue system

Traditional dialogue systems use a fixed silence threshold to detect the end of users’ turns. Such a simplistic model can result in system behaviour that is both interruptive and unresponsive, which in turn affects user experience. Various studies have observed that human interlocutors take cues from speaker behaviour, such as prosody, syntax, and gestures, to coordinate smooth exchange of speaking turns. However, little effort has been made towards implementing these models in dialogue systems and verifying how well they model the turn-taking behaviour in human–computer interactions. We present a data-driven approach to building models for online detection of suitable feedback response locations in the user's speech. We first collected human–computer interaction data using a spoken dialogue system that can perform the Map Task with users (albeit using a trick). On this data, we trained various models that use automatically extractable prosodic, contextual and lexico-syntactic features for detecting response locations. Next, we implemented a trained model in the same dialogue system and evaluated it in interactions with users. The subjective and objective measures from the user evaluation confirm that a model trained on speaker behavioural cues offers both smoother turn-transitions and more responsive system behaviour.     

INTEGRATION OF A VOICE RECOGNITION SYSTEM IN A SOCIAL ROBOT

Human–robot interaction (HRI) ^{1 1}HRI can be defined as the study of humans, robots, and the ways in which they influence each other. is one of the main fields in the study and research of robotics. Within this field, dialogue systems and interaction by voice play an important role. When speaking about human–robot natural dialogue we assume that the robot has the capability to accurately recognize what the human wants to transmit verbally and even its semantic meaning, but this is not always achieved. In this article we describe the steps and requirements that we went through in order to endow the personal social robot Maggie, developed at the University Carlos III of Madrid, with the capability of understanding the natural language spoken by any human. We have analyzed the different possibilities offered by current software/hardware alternatives by testing them in real environments. We have obtained accurate data related to the speech recognition capabilities in different environments, using the most modern audio acquisition systems and analyzing not so typical parameters such as user age, gender, intonation, volume, and language. Finally, we propose a new model to classify recognition results as accepted or rejected, based on a second automatic speech recognition (ASR) opinion. This new approach takes into account the precalculated success rate in noise intervals for each recognition framework, decreasing the rate of false positives and false negatives.     

Testing the performance of spoken dialogue systems by means of an artificially simulated user

This paper proposes a new technique to test the performance of spoken dialogue systems by artificially simulating the behaviour of three types of user (very cooperative, cooperative and not very cooperative) interacting with a system by means of spoken dialogues. Experiments using the technique were carried out to test the performance of a previously developed dialogue system designed for the fast-food domain and working with two kinds of language model for automatic speech recognition: one based on 17 prompt-dependent language models, and the other based on one prompt-independent language model. The use of the simulated user enables the identification of problems relating to the speech recognition, spoken language understanding, and dialogue management components of the system. In particular, in these experiments problems were encountered with the recognition and understanding of postal codes and addresses and with the lengthy sequences of repetitive confirmation turns required to correct these errors. By employing a simulated user in a range of different experimental conditions sufficient data can be generated to support a systematic analysis of potential problems and to enable fine-grained tuning of the system.     

An Assessment of Written/Interactive Dialogue for Information Retrieval Applications

For the foreseeable future, natural language access to databases and other information systems will require the exchange of written messages between the system and user. Such written/interactive dialogue is unique, having the qualities of both written text and spoken discourse yet, also, differing significantly from both. In the present study, we used "Wizard of Oz" techniques to elicit written/interactive dialogue for information retrieval purposes. Our objectives in doing this were (a) to assess the general nature and prevalence of linguistic and dialogue phenomena within the written/interactive register and (b) to determine the impact of user interface shortcuts, such as precanned messages and patterned output, on both the complexity of written/interactive dialogue and general measures of user satisfaction. Our findings indicate that written/interactive dialogue for information retrieval would be very feasible. In spite of slow system response times, subjective reactions from the users were positive, the size of the lexicon used in the dialogues was small, the dialogues decomposed readily into hierarchical structures, and the number and distribution of anaphors were also rather reasonable. Two independent variables were also manipulated: (a) the degree of prefamiliarization given to participants about the base of travel information they would be accessing (i.e., the primer variable) and (b) the degree of constraint on the Wizard's ability to formulate natural language responses to the user (i.e., the natural language output variable). Failure to provide either a primer or a realistic, human natural language output made the dialogues more complex in a number of different ways.     

Evaluating the Robot Personality and Verbal Behavior of Domestic Robots Using Video-Based Studies

Robots are increasingly being used in domestic environments and should be able to interact with inexperienced users. Human–human interaction and human–computer interaction research findings are relevant, but often limited because robots are different from both humans and computers. Therefore, new human–robot interaction (HRI) research methods can inform the design of robots suitable for inexperienced users. A video-based HRI (VHRI) methodology was here used to carry out a multi-national HRI user study for the prototype domestic robot BIRON (BIelefeld RObot companioN). Previously, the VHRI methodology was used in constrained HRI situations, while in this study HRIs involved a series of events as part of a 'hometour' scenario. Thus, the present work is the first study of this methodology in extended HRI contexts with a multi-national approach. Participants watched videos of the robot interacting with a human actor and rated two robot behaviors (Extrovert and Introvert). Participants' perceptions and ratings of the robot's behaviors differed with regard to both verbal interactions and person following by the robot. The study also confirms that the VHRI methodology provides a valuable means to obtain early user feedback, even before fully working prototypes are available. This can usefully guide the future design work on robots, and associated verbal and non-verbal behaviors.     

Human-Robot Interaction Through Gesture-Free Spoken Dialogue

We present an approach to human-robot interaction through gesture-free spoken dialogue. Our approach is based on passive knowledge rarefication through goal disambiguation, a technique that allows a human operator to collaborate with a mobile robot on various tasks through spoken dialogue without making bodily gestures. A key assumption underlying our approach is that the operator and the robot share a common set of goals. Another key idea is that language, vision, and action share common memory structures.We discuss how our approach achieves four types of human-robot interaction: command, goal disambiguation, introspection, and instruction-based learning. We describe the system we developed to implement our approach and present experimental results.     

Interaction With Robots: Physical Constraints on the Interpretation of Demonstrative Pronouns

This study investigated what effect physical constraints have on the interpretation of demonstrative pronouns when a user navigates a robot. For this investigation, a robot navigation environment called Spondia-II was develope, and an experiment conducted. It is known that the interpretation of demonstrative pronouns requires information about not only the situation (or context) but also the speaker's viewpoint during a dialogue. The results of the experiment suggest that physical constraints do affect the user's viewpoint, especially when a user utters a demonstrative pronoun while navigating the robot. In actual fact, the user alters the use of demonstrative pronouns according to the change in the user's viewpoint. It is also suggested that the user and the robot share the same viewpoint during the physical interaction.     

A spoken language interface with a mobile robot

We describe a spoken dialogue interface with a mobile robot, which a human can direct to specific locations, ask for information about its status, and supply information about its environment. The robot uses an internal map for navigation, and communicates its current orientation and accessible locations to the dialogue system. In this article, we focus on linguistic and inferential aspects of the human–robot communication process. This work was conducted at ICCS, School of Informatics, University of Edinburgh, Edinburgh, Scotland, UK This work was presented in part at the 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006     

对话意图及语音识别错误对交互体验的影响

在自然人机对话中,由于环境噪声、方言口音等因素带来的语音识别错误以及语义分析的不充分等原因,计算机在理解用户交互意图时出现偏差,使得计算机对要反馈的话题出现错误,造成人机对话进程的断裂.以面向咖啡为主题的漫谈式人机对话为例,将对话中断分为3种情况：话题反馈不当引起中断、话题正确情况下的模糊反馈不当和精确反馈不当引起中断.根据用户与计算机对话的记录分析比较上述3种情况下人机对话进程断裂情况.统计数据结果表明,话题反馈不当带来的对话中断最为明显,在对话进程断裂情况中达到了60.1%的比例;在话题反馈正确情况下,模糊回答不当和精确回答不当带来的话题中断比例分别为22.2%和21.6%;在语音识别错误情况下,语义分析会带来数量更大的反馈错误.实验数据分析结果表明,在语音识别错误情况下,根据上下文信息提高计算机对用户话题反馈的准确率,能够有效降低人机对话的中断,提高人机对话的自然度.该工作为自然人机对话的意图分类重要性提供了数据分析和实验论证.     

A novel path planning and control framework for passive resistance therapy with a robot manipulator

In this article, we present a paradigm for safe path generation and control for a robotic manipulator such that it provides programmable passive resistance therapy to patients with deficits in the upper extremities. When the patient applies an interaction force at the robot's end-effector, a dynamic path generator time parameterises any therapist-specified contour in the robot's workspace–thus, the robot mimics the dynamics of a passive impedance whose anisotropy vector can be continuously reconfigured. The proposed algorithm is easily implementable because it is robust to uncertainty in the robot dynamics. Moreover, the proposed strategy also guarantees user safety by maintaining the net flow of energy during the human robot interaction from the user towards the manipulator.     

Control of data processing systems by voice commands

Abstract

A spoken dialogue between a user and a computer system has to be governed by the system because of: (i) the limited capabilities of present word-recognition apparatus; and (ii) the limited possibilities of the system for ‘understanding’ its user. The user's part of the dialogue therefore needs to be unobtrusively controlled by the system by carefully phrased and timed prompts. Short pauses in these prompts enable the experienced user to make shortcuts through the dialogue, without forsaking complete explanations for the inexperienced user. The user is also able to control the system by utilizing other pauses in the system utterances for corrective words or protests in case of incorrect recognition.     

Assessing the effects of building social intelligence in a robotic interface for the home

This paper reports an exploration of the concept of social intelligence in the context of designing home dialogue systems for an Ambient Intelligence home. It describes a Wizard of Oz experiment involving a robotic interface capable of simulating several human social behaviours. Our results show that endowing a home dialogue system with some social intelligence will: (a) create a positive bias in the user's perception of technology in the home environment, (b) enhance user acceptance for the home dialogue system, and (c) trigger social behaviours by the user in relation to the home dialogue system.     

Evaluation of Methods for the Determination of Factors Inducing Fatigue in Man at Work

The aim of the study was to determine the influence of textual feedback on the content and outcome of spoken interaction with a natural language dialogue system. More specifically, the assumption that textual feedback could disrupt spoken interaction was tested in a human–computer dialogue situation. In total, 48 adult participants, familiar with the system, had to find restaurants based on simple or difficult scenarios using a real natural language service system in a speech-only (phone), speech plus textual dialogue history (multimodal) or text-only (web) modality. The linguistic contents of the dialogues differed as a function of modality, but were similar whether the textual feedback was included in the spoken condition or not. These results add to burgeoning research efforts on multimodal feedback, in suggesting that textual feedback may have little or no detrimental effect on information searching with a real system.

Statement of Relevance: The results suggest that adding textual feedback to interfaces for human–computer dialogue could enhance spoken interaction rather than create interference. The literature currently suggests that adding textual feedback to tasks that depend on the visual sense benefits human–computer interaction. The addition of textual output when the spoken modality is heavily taxed by the task was investigated.     

Sensors and actuators for 'humanoid' robots

Though much research is still necessary before a personal robot with anthropomorphic features becomes a full reality, accessible and appealing to the general user, present technology is able to offer many solutions to the different problems arising in the development of sensors and actuators, which are a key factor in the achievement of the final goal. However, the challenge of obtaining a humanoid robot combining adequate technical performance with a 'behavior' appealing for a real 'companion' to the human user requires a truly interdisciplinary effort involving not only technical disciplines, but also the contribution of human sciences. This paper presents the motivations for the development of sensors and actuators for humanoid robots, with particular emphasis on service robotics and on the concept of a personal robot. A brief survey of existing technologies is reported and a few examples of present implementations are described. We also present the main features of a real personal robot whose technical and aesthetical characteristics are purposively designed to match those desirable for a 'humanoid' robot.     

The MATCH corpus: a corpus of older and younger users’ interactions with spoken dialogue systems

We present the MATCH corpus, a unique data set of 447 dialogues in which 26 older and 24 younger adults interact with nine different spoken dialogue systems. The systems varied in the number of options presented and the confirmation strategy used. The corpus also contains information about the users’ cognitive abilities and detailed usability assessments of each dialogue system. The corpus, which was collected using a Wizard-of-Oz methodology, has been fully transcribed and annotated with dialogue acts and “Information State Update” (ISU) representations of dialogue context. Dialogue act and ISU annotations were performed semi-automatically. In addition to describing the corpus collection and annotation, we present a quantitative analysis of the interaction behaviour of older and younger users and discuss further applications of the corpus. We expect that the corpus will provide a key resource for modelling older people’s interaction with spoken dialogue systems.     

Robot selection expert ‘Rose’

One of the major problems facing the robot user in the future will be his choice of the optimum robot for a particular task. What is needed is a highly automated robot selection system which will eliminate the human decision-making process. The system presented will be used when a robot is being considered to replace a human at a particular task, while the rest of the workplace remains fixed. The purpose of this paper is twofold; firstly, to demonstrate the knowledge required in making an optimum robot selection, and secondly, to provide a tutoral in designing an expert system using EXPERT. The paper will provide (1) the data base, (2) rules for transforming that data base, and (3) the control strategy that is necessary in implementing an expert system to perform the aforementioned task. The system will query the user as to the characteristics of the desired robot and the expert system will choose an optimum robot from the choices in the data base. The user will construct the environment in which the robot will be working by using 3-D modeling techniques. The user will choose from a menu and place the various objects which the robot will have to conform to. Thus, constraints such as maximum space available, can be stripped out of the 3-D drawing rather than having the expert system query the user for dimensions. One very good feature of such a system is that as new robots are developed their specifications can be added to the data base very easily.