Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot

In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.     

Accelerating Robot Development Through Integral Analysis of Human–Robot Interaction

The development of interactive robots is a complicated process, involving a plethora of psychological, technical, and contextual influences. To design a robot capable of operating ldquointelligentlyrdquo in everyday situations, one needs a profound understanding of human-robot interaction (HRI). We propose an approach based on integral analysis of multimodal data to pursue this understanding and support interdisciplinary research and development in the field of robotics. To adopt this approach, a software tool named interaction debugger was developed that features user-friendly navigation, browsing, searching, viewing, and annotation of data; it enables fine-grained inspection of the HRI. In four case studies, we demonstrated how our analysis approach aids the development process of interactive robots.     

Analysis of Humanoid Appearances in Human–Robot Interaction

Identifying the extent to which the appearance of a humanoid robot affects human behavior toward it is important. We compared participant impressions of and behaviors toward two real humanoid robots in simple human-robot interactions. These two robots, which have different appearances but are controlled to perform the same recorded utterances and motions, are adjusted by a motion-capturing system. We conducted an experiment with 48 human participants who individually interacted with the two robots and also with a human for reference. The results revealed that different appearances did not affect participant verbal behaviors, but they did affect such nonverbal behaviors as distance and delay of response. These differences are explained by two factors: impressions and attributions.     

Remote research methods for Human–AI–Robot Teaming

This study focuses on methodological adaptations and considerations for remote research on Human–AI–Robot Teaming (HART) amidst the COVID-19 pandemic. Themes and effective remote research methods were explored. Central issues in remote research were identified, such as challenges in attending to participants' experiences, coordinating experimenter teams remotely, and protecting privacy and confidentiality. Instances of experimental design overcoming these challenges were identified in methods for recruitment and onboarding, training, team task scenarios, and measurement. Three case studies are presented in which interactive in-person testbeds for HART were rapidly redesigned to function remotely. Although COVID-19 may have temporarily constrained experimental design, future HART studies may adopt remote research methods to expand the research toolkit.     

A Two-Month Field Trial in an Elementary School for Long-Term Human–Robot Interaction

Interactive robots participating in our daily lives should have the fundamental ability to socially communicate with humans. In this paper, we propose a mechanism for two social communication abilities: forming long-term relationships and estimating friendly relationships among people. The mechanism for long-term relationships is based on three principles of behavior design. The robot we developed, Robovie, is able to interact with children in the same way as children do. Moreover, the mechanism is designed for long-term interaction along the following three design principles: (1) it calls children by name using radio frequency identification tags; (2) it adapts its interactive behaviors for each child based on a pseudo development mechanism; and (3) it confides its personal matters to the children who have interacted with the robot for an extended period of time. Regarding the estimation of friendly relationships, the robot assumes that people who spontaneously behave as a group together are friends. Then, by identifying each person in the interacting group around the robot, it estimates the relationships between them. We conducted a two-month field trial at an elementary school. An interactive humanoid robot, Robovie, was placed in a classroom at the school. The results of the field trial revealed that the robot successfully continued interacting with many children for two months, and seemed to have established friendly relationships with them. In addition, it demonstrated reasonable performance in identifying friendships among children. We believe that these results demonstrate the potential of current interactive robots to establish social relationships with humans in our daily lives.     

Cost-Based Anticipatory Action Selection for Human–Robot Fluency

A crucial skill for fluent action meshing in human team activity is a learned and calculated selection of anticipatory actions. We believe that the same holds for robotic teammates, if they are to perform in a similarly fluent manner with their human counterparts. In this work, we describe a model for human-robot joint action, and propose an adaptive action selection mechanism for a robotic teammate, which makes anticipatory decisions based on the confidence of their validity and their relative risk. We conduct an analysis of our method, predicting an improvement in task efficiency compared to a purely reactive process. We then present results from a study involving untrained human subjects working with a simulated version of a robot using our system. We show a significant improvement in best-case task efficiency when compared to a group of users working with a reactive agent, as well as a significant difference in the perceived commitment of the robot to the team and its contribution to the team's fluency and success. By way of explanation, we raise a number of fluency metric hypotheses, and evaluate their significance between the two study conditions. [All rights reserved Elsevier].     

Improving the Human–Robot Interface Through Adaptive Multispace Transformation

Teleoperation is essential for applications in which, despite the availability of a precise geometrical definition of the working area, a task cannot be explicitly programmed. This paper describes a method of assisted teleoperation that improves the execution of such tasks in terms of ergonomics, precision, and reduction of execution time. The relationships between the operating spaces corresponding to the human-robot interface triangle are analyzed. The proposed teleoperation aid is based on applying adaptive transformations between these spaces.     

Context‐Centric Speech‐Based Human–Computer Interaction

This paper describes research that addresses the problem of dialog management from a strong, context‐centric approach. We further present a quantitative method of measuring the importance of contextual cues when dealing with speech‐based human–computer interactions. It is generally accepted that using context in conjunction with a human input, such as spoken speech, enhances a machine's understanding of the user's intent as a means to pinpoint an adequate reaction. For this work, however, we present a context‐centric approach in which the use of context is the primary basis for understanding and not merely an auxiliary process. We employ an embodied conversation agent that facilitates the seamless engagement of a speech‐based information‐deployment entity by its human end user. This dialog manager emphasizes the use of context to drive its mixed‐initiative discourse model. A typical, modern automatic speech recognizer (ASR) was incorporated to handle the speech‐to‐text translations. As is the nature of these ASR systems, the recognition rate is consistently less than perfect, thus emphasizing the need for contextual assistance. The dialog system was encapsulated into a speech‐based embodied conversation agent platform for prototyping and testing purposes. Experiments were performed to evaluate the robustness of its performance, namely through measures of naturalness and usefulness, with respect to the emphasized use of context. The contribution of this work is to provide empirical evidence of the importance of conversational context in speech‐based human–computer interaction using a field‐tested context‐centric dialog manager.     

机器人情感交互模型研究

为了实现机器人与人的和谐交互,该文给出了基于Multi-agent的情感机器人结构模型,提出了机器人情感交互模型构建方法,介绍了基于灰色系统的情感模型和情感关联模型,构造了机器人学习模型,实现了情感机器人交互系统,结果表明机器人能够和人进行有智能和情感的交互。     

Brian Shackel’s contribution to the written history of Human–Computer Interaction

In 1997, Brian Shackel published the article “Human–Computer Interaction – Whence and Whither?” In this early foray into historical reflection on the field, past work is covered with a focus on identifying European contributions, issues of particular contemporary interest are explored, and a set of 10-year predictions are offered. In this essay, from a vantage-point of an additional decade of history, insights of lasting value that Professor Shackel was uniquely positioned to glean are identified. His work is placed in the broader context now available, and an always-useful reminder of the difficulty of anticipating future events is provided.     

Human–battery interaction on mobile phones

Mobile phone users have to deal with limited battery lifetime through a reciprocal process we call human–battery interaction. We conducted three user studies in order to understand human–battery interaction and discover the problems in existing designs that prevent users from effectively dealing with the limited battery lifetime. The studies include a large-scale international survey, two long-term field trials including quantitative battery logging and qualitative inquiries, and structured interviews with twenty additional mobile phone users. We evaluated various aspects of human–battery interaction, including charging behavior, battery indicators, user interfaces for power-saving settings, user knowledge, and user reaction. We find that mobile phone users can be categorized into two types regarding human–battery interaction and often have inadequate knowledge regarding phone power characteristics. We provide qualitative and quantitative evidence that problems in state-of-the-art user interfaces have led to under-utilized power-saving settings, under-utilized battery energy, and dissatisfied users. Our findings provide insights into improving mobile phone design for users to effectively deal with the limited battery lifetime. Our work is the first to systematically address human–battery interaction on mobile phones and is complementary to the extensive research on energy-efficient design for a longer battery lifetime.     

Human–computer interaction viewed as pseudo-communication

Semiotics is considered fundamental to an understanding of human–computer interaction, and of all computer artifacts. Informatics should therefore be viewed as technical semiotics (or semiotics engineering). In particular, interaction between human and computer is characterized by features of communication, a sort of communication, however, that lacks decisive communicative features. It must be identified as a process of pseudo-communication. Interaction is viewed as the coupling of two autonomous processes: a sign process (carried out by the human user) and a signal process (carried out by the computer). Software appears as a semiotic entity in a duplicate way: calculated and calculating, i.e. both as a result and agent of calculations. This dialectics characterizes the class of signs on the computer medium. Problems of software design (functionality and usability design) are specific problems of the coupling of sign and signal processes.     

Full-Body Compliant Human–Humanoid Interaction: Balancing in the Presence of Unknown External Forces

This paper proposes an effective framework of human-humanoid robot physical interaction. Its key component is a new control technique for full-body balancing in the presence of external forces, which is presented and then validated empirically. We have adopted an integrated system approach to develop humanoid robots. Herein, we describe the importance of replicating human-like capabilities and responses during human-robot interaction in this context. Our balancing controller provides gravity compensation, making the robot passive and thereby facilitating safe physical interactions. The method operates by setting an appropriate ground reaction force and transforming these forces into full-body joint torques. It handles an arbitrary number of force interaction points on the robot. It does not require force measurement at interested contact points. It requires neither inverse kinematics nor inverse dynamics. It can adapt to uneven ground surfaces. It operates as a force control process, and can therefore, accommodate simultaneous control processes using force-, velocity-, or position-based control. Forces are distributed over supporting contact points in an optimal manner. Joint redundancy is resolved by damping injection in the context of passivity. We present various force interaction experiments using our full-sized bipedal humanoid platform, including compliant balance, even when affected by unknown external forces, which demonstrates the effectiveness of the method.     

Human‐Robot Interaction in Assisted Personal Services: Factors Influencing Distances That Humans Will Accept between Themselves and an Approaching Service Robot

In the provision of robot‐assisted personal services, the ergonomic design of the phase when a robot approaches a human being is important for the efficacy, efficiency, and safety of human‐robot interaction. For this reason, an empirical user study was conducted, using a repeated measures fractional factorial design. The participants (n  =  30) were approached 30 times by the 1.45‐m‐tall service robot Care‐O‐bot® 3 under different conditions to investigate what influences the distances that humans will accept between themselves and an approaching robot. The results show that participant body position, robot speed, and speed profile significantly influence the accepted distances. Furthermore, an effect of habituation was found for the repeated approaches. Significant effects of participant age and gender as well as robot appearance on the accepted distance were not found.     

A Review of Web‐Based Dietary Interventions: From the Human–Computer Interaction Practitioners' Perspective

Despite the popularity of web‐based dietary interventions, there are few evidence‐based, practical guidelines that help human–computer interaction (HCI) practitioners design new dietary intervention systems. We suspect that a lack of such guidelines is partly due to a chasm between two major research domains, healthcare and HCI. We believe that technologies developed in HCI are not used and evaluated by healthcare researchers, so we fail to accumulate experiences to develop guidelines. To assess the gap, we carefully selected 86 papers that employed and evaluated various web‐based dietary interventions in both fields and analyzed general characteristics, behavior change strategies, intervention media, and research outcomes used in each paper. Through this review, we reaffirmed our belief about the discrepancies between healthcare and HCI, and additional findings helped us offer some suggestions to close the gap. We also identified several interesting patterns among behavior change strategies, intervention media, and outcomes that provide potential topics for future research. © 2012 Wiley Periodicals, Inc.     

情感机器人的情感模型研究

本文建立了仿人类情感的情感模型,在情感模型中建立了三维情感空间,采用马尔可夫过程来描述情感状态的变化转移过程,并提出了性格矩阵,对比分析了不同性格人对相同刺激的反应.应用D-S证据理论融合来自视觉、声音及其他方面的情感信息,分析两种因素同时作用时引起情感状态的转移规律.最后将情感模型应用于情感机器人系统,使机器人可以根据外界刺激产生情感,并做出相应的表情.实验结果证明了情感模型的有效性.     

A Real-Time Approach to the Spotting, Representation, and Recognition of Hand Gestures for Human–Computer Interaction

Aiming at the use of hand gestures for human–computer interaction, this paper presents a real-time approach to the spotting, representation, and recognition of hand gestures from a video stream. The approach exploits multiple cues including skin color, hand motion, and shape. Skin color analysis and coarse image motion detection are joined to perform reliable hand gesture spotting. At a higher level, a compact spatiotemporal representation is proposed for modeling appearance changes in image sequences containing hand gestures. The representation is extracted by combining robust parameterized image motion regression and shape features of a segmented hand. For efficient recognition of gestures made at varying rates, a linear resampling technique for eliminating the temporal variation (time normalization) while maintaining the essential information of the original gesture representations is developed. The gesture is then classified according to a training set of gestures. In experiments with a library of 12 gestures, the recognition rate was over 90%. Through the development of a prototype gesture-controlled panoramic map browser, we demonstrate that a vocabulary of predefined hand gestures can be used to interact successfully with applications running on an off-the-shelf personal computer equipped with a home video camera.     

Human–automation teams and adaptable control for future air traffic management

The design and management of human–automation teams for future air traffic systems require an understanding of principles of cognitive systems engineering, allocation of function and team adaptation. The current article proposes a framework of human–automation team adaptable control that incorporates adaptable automation [Oppermann, R., Simm, H., 1994. Adaptability: user-initiated individualization. In: Oppermann, R. (Ed.), Adaptive User Support: Ergonomic Design of Manually and Automatically Adaptable Software. Lawrence Erlbaum Associates, Hillsdale, NJ, pp. 14–64] with an Extended Control Model of Joint Cognitive System functioning [Hollnagel, E., Nåbo, A., Lau, I., 21–24 July 2003. A systemic model for driver-in-control. In: Paper Presented at the Second International Driving Symposium on Human Factors in Driver Assessment, Training, and Vehicle Design, Public Policy Center, University of Iowa, Park City, UT] nested within a dynamic view of team adaptation [Burke, C.S., Stagl, K.C., Salas, E., Pierce, L., Kendall, D., 2006. Understanding team adaptation: a conceptual analysis and model. Journal of Applied Psychology 91, 1189–1207]. Modeling the temporal dynamics of the coordination of human–automation teams under conditions of Free Flight requires an appreciation of the episodic, cyclical nature of team processes from transition to action phases, along with the distinction of team processes from emergent states [Marks, M.A., Mathieu, J.E., Zaccaro, S.J., 2001. A temporally based framework and taxonomy of team processes. Academy of Management Review 26, 356–376]. The conceptual framework of human–automation team adaptable control provides a basis for future research and design.

Relevance to industry

The current article provides a conceptual framework to direct future investigations to determine the optimal design and management of Human–automation teams for Free Flight-based air traffic management systems.