RUNA: a multimodal command language for home robot users

This article describes a multimodal command language for home robot users, and a robot system which interprets users’ messages in the language through microphones, visual and tactile sensors, and control buttons. The command language comprises a set of grammar rules, a lexicon, and nonverbal events detected in hand gestures, readings of tactile sensors attached to the robots, and buttons on the controllers in the users’ hands. Prototype humanoid systems which immediately execute commands in the language are also presented, along with preliminary experiments of faceto-face interactions and teleoperations. Subjects unfamiliar with the language were able to command humanoids and complete their tasks with brief documents at hand, given a short demonstration beforehand. The command understanding system operating on PCs responded to multimodal commands without significant delay. This work was presented in part at the 13th International Symposium on Artificial Life and Robotics, Oita, Japan, January 31–February 2, 2008     

Speech recognition for command entry in multimodal interaction

Two experiments investigated the cognitive efficiency of using speech recognition in combination with the mouse and keyboard for a range of word processing tasks. The first experiment examined the potential of this multimodal combination to increase performance by engaging concurrent multiple resources. Speech and mouse responses were compared when using menu and direct (toolbar icon) commands, making for a fairer comparison than in previous research which has been biased against the mouse. Only a limited basis for concurrent resource use was found, with speech leading to poorer task performance with both command types. Task completion times were faster with direct commands for both speech and mouse responses, and direct commands were preferred. In the second experiment, participants were free to choose command type, and nearly always chose to use direct commands with both response modes. Speech performance was again worse than mouse, except for tasks which involved a large amount of hand and eye movement, or where direct speech wasused but mouse commands were made via menus. In both experiments recognition errors were low, and although they had some detrimental effect on speech use, problems in combining speech and manual modes were highlighted. Potential verbal interference effects when using speech are discussed.     

An optimizer for a relational database command language

This paper describes the optimizer included in the ICL personal data system (PDS), an interactive relational database system designed for users with little or no computer experience. The optimizer transforms the selection criteria in a command into an equivalent form which can be used to obey the command more quickly. The optimization method and the factorization procedure it uses are described in detail.     

Hand waving in command spaces: a framework for operating home appliances

In this paper, hand waving in a ‘command space’ is proposed as a new framework for operating home appliances. A command space is associated with the operation of a home appliance; an operation is retrieved by waving a hand in the command space. A home appliance operation system with the function of multiplexing command space modules and integrating multiple commands is constructed. With the proposed framework and system, home appliance operation is possible only by hand waving, an intuitive gesture for a human. Experiments were conducted that verified that complex operations, such as scheduling the recording of a TV program, can be performed with the system. The usability of the constructed system is evaluated using Brooke's System Usability Scale (SUS). The average SUS score was 66.8, which indicates that the subjects had relatively positive impression on the system in spite of long operation time.     

An experimental evaluation of delimiters in a command language syntax

This article examines the effects of delimiters on the cognitive processes involved in command construction and interpretation. After a theoretical analysis of the impact of delimiters on commands, two techniques of delimiter usage were examined: single-level delimiters (space) and two-level hierarchical delimiters (space and comma). Using a within-subject experimental design with 20 subjects, we found that hierarchical delimiters improved the ability of individuals to distinguish fields and identify the grammatical correctness of the commands. This study suggests that the ability to orgainize elements of commands mentally is enhanced when they consist of fields separated by one delimiter and items within fields by another delimiter. Other factors, not examined in this study, should have an impact on production. They may include the name of the command, the order in which operands are presented, and the experience level of the user.     

Commanding a humanoid to move objects in a multimodal language

This article describes a study on a humanoid robot that moves objects at the request of its users. The robot understands commands in a multimodal language which combines spoken messages and two types of hand gesture. All of ten novice users directed the robot using gestures when they were asked to spontaneously direct the robot to move objects after learning the language for a short period of time. The success rate of multimodal commands was over 90%, and the users completed their tasks without trouble. They thought that gestures were preferable to, and as easy as, verbal phrases to inform the robot of action parameters such as direction, angle, step, width, and height. The results of the study show that the language is fairly easy for nonexperts to learn, and can be made more effective for directing humanoids to move objects by making the language more sophisticated and improving our gesture detector.     

Designing multimodal tracks for mobile users in unfamiliar urban environments

While there have been many studies of navigation design guidelines for integrating maps in mobile devices, we argue that the research focus should turn more towards the design of multimodal navigation aids, which would reflect the impact of typical human wayfinding behaviour. Therefore, we suggest an interdisciplinary design approach, i.e. building on urban design practice that focuses on supporting the navigation of pedestrians in urban areas. The design implications are explored in this study with the subjects being pedestrians accomplishing wayfinding tasks in an unfamiliar, urban area. The main areas that this paper contributes to are: investigating the design implications of multimodal navigation aids; evaluating these in the context of mobile wayfinding tasks; and reflecting the results according to the user's wayfinding strategies and spatial anxiety. It is concluded that through designing multimodal tracks into mobile navigation applications we can help users to find their way in unfamiliar, urban environments.     

GeeAir: a universal multimodal remote control device for home appliances

In this paper, we present a handheld device called GeeAir for remotely controlling home appliances via a mixed modality of speech, gesture, joystick, button, and light. This solution is superior to the existing universal remote controllers in that it can be used by the users with physical and vision impairments in a natural manner. By combining diverse interaction techniques in a single device, the GeeAir enables different user groups to control home appliances effectively, satisfying even the unmet needs of physically and vision-impaired users while maintaining high usability and reliability. The experiments demonstrate that the GeeAir prototype achieves prominent performance through standardizing a small set of verbal and gesture commands and introducing the feedback mechanisms.     

User study of a life-supporting humanoid directed in a multimodal language

This article describes a user study of a life-supporting humanoid directed in a multimodal language, and discusses the results. Twenty inexperienced users commanded the humanoid in a computer-simulated remote home environment in the multimodal language by pressing keypad buttons and speaking to the robot. The results show that they comprehended the language well and were able to give commands successfully. They often chose a press-button action in place of verbal phrases to specify a direction, speed, length, angle, and/or temperature value, and preferred multimodal commands to spoken commands. However, they did not think that it was very easy to give commands in the language. This article discusses the results and points out both strong and weak points of the language and our robot.     

Spoken language and multimodal applications for electronic realities

We use the term electronic reality (ER) to encompass a broad class of concepts that mix real-world metaphors and computer interfaces. In our definition, electronic reality includes notions such as virtual reality, augmented reality, computer interactions with physical devices, interfaces that enhance 2D media such as paper or maps, and social interfaces where computer avatars engage humans in various forms of dialogue. One reason for bringing real-world metaphors to computer interfaces is that people already know how to navigate and interact with the world around them. Every day, people interact with each other, with pets, and sometimes with physical objects by using a combination of expressivemodalities, such as spoken words, lone of voice, pointing and gesturing, facial expressions, and body language. In contrast, when people typically interact with computers or appliances, interactions are unimodal, with a single method of communication such as the click of a mouse or a set of keystrokes serving to express intent. In this article, we describe our efforts to apply multimodal and spoken language interfaces to a number of ER applications, with the goal of creating an even more realistic or natural experience for the end user.     

A user study of command strategies for mobile robot teleoperation

This article presents a user study of mobile robot teleoperation. Performance of speed, position and combined command strategies in combination with text, visual and haptic feedback information were evaluated by experiments. Two experimental tasks were designed as follows: positioning of mobile robot and navigation in complex environment. Time for task completion and motion accuracy were measured and compared for different command strategies and types of feedback. Role of haptic, text and visual feedback information in combination with described command strategies is outlined.     

Two-handed gesture recognition and fusion with speech to command a robot

Assistance is currently a pivotal research area in robotics, with huge societal potential. Since assistant robots directly interact with people, finding natural and easy-to-use user interfaces is of fundamental importance. This paper describes a flexible multimodal interface based on speech and gesture modalities in order to control our mobile robot named Jido. The vision system uses a stereo head mounted on a pan-tilt unit and a bank of collaborative particle filters devoted to the upper human body extremities to track and recognize pointing/symbolic mono but also bi-manual gestures. Such framework constitutes our first contribution, as it is shown, to give proper handling of natural artifacts (self-occlusion, camera out of view field, hand deformation) when performing 3D gestures using one or the other hand even both. A speech recognition and understanding system based on the Julius engine is also developed and embedded in order to process deictic and anaphoric utterances. The second contribution deals with a probabilistic and multi-hypothesis interpreter framework to fuse results from speech and gesture components. Such interpreter is shown to improve the classification rates of multimodal commands compared to using either modality alone. Finally, we report on successful live experiments in human-centered settings. Results are reported in the context of an interactive manipulation task, where users specify local motion commands to Jido and perform safe object exchanges.     

Involving users in the development of a modeling language for customer journeys

Software and Systems Modeling - Although numerous methods for handling the technical aspects of developing domain-specific modeling languages (DSMLs) have been formalized, user needs and usability...     

Using a natural language interface with casual users

Although there is much controversy about the merits of natural language interfaces, little empirical research has been conducted on the use of natural language interfaces for database access, especially for casual users. In this work casual users were observed while interacting with a real-life database using a natural language interface, Intellect.Results show that natural language is an efficient and powerful means for expressing requests. This is especially true for users with a good knowledge of the database contents regardless of training or previous experience with computers. Users generally have a positive attitude towards natural language. The majority of errors users make are directly related to restrictions in the vocabulary. However, feedback helps users understand the language limitations and learn how to avoid or recover from errors. Natural language processing technology is developed enough to handle the limited domain of discourse associated with a database; it is simple enough to support casual users with a general knowledge of the database contents; and it is flexible enough to assist problem-solving behaviour.     

Multiple robot programming using a concurrent logic language

Advances in robotics has led to the cooperation of multiple robots among themselves and with their industrial automation environment. Efficient interaction with industrial robots thus becomes one of the key factors in the successful utilization of this modern equipment. When multiple manipulators have to be coordinated, there is a need for a new programming approach that facilitates and encompasses the needs of concurrency, synchronization, timing, and communication. Most robot languages have been developed with little attention being given to the integration of the robot with its environment. Currently, there is a gap between the robot capabilities, the task definition environment, and language facilities supplied to use robots.This paper analyzes the needs and then establishes that a concurrent logic programming approach is a step towards achieving a multi-robot knowledgeable task programming. In particular, the FCP dialect of concurrent Prolog is demonstrated, and analyzed.This research is partially supported by the Paul Ivanier Center for research in robots and production management.     

A spoken language interface with a mobile robot

We describe a spoken dialogue interface with a mobile robot, which a human can direct to specific locations, ask for information about its status, and supply information about its environment. The robot uses an internal map for navigation, and communicates its current orientation and accessible locations to the dialogue system. In this article, we focus on linguistic and inferential aspects of the human–robot communication process. This work was conducted at ICCS, School of Informatics, University of Edinburgh, Edinburgh, Scotland, UK This work was presented in part at the 11th International Symposium on Artificial Life and Robotics, Oita, Japan, January 23–25, 2006