Adaptive visual gesture recognition for human–robot interaction using a knowledge-based software platform

In human–human communication we can adapt or learn new gestures or new users using intelligence and contextual information. Achieving natural gesture-based interaction between humans and robots, the system should be adaptable to new users, gestures and robot behaviors. This paper presents an adaptive visual gesture recognition method for human–robot interaction using a knowledge-based software platform. The system is capable of recognizing users, static gestures comprised of the face and hand poses, and dynamic gestures of face in motion. The system learns new users, poses using multi-cluster approach, and combines computer vision and knowledge-based approaches in order to adapt to new users, gestures and robot behaviors. In the proposed method, a frame-based knowledge model is defined for the person-centric gesture interpretation and human–robot interaction. It is implemented using the frame-based Software Platform for Agent and Knowledge Management (SPAK). The effectiveness of this method has been demonstrated by an experimental human–robot interaction system using a humanoid robot ‘Robovie’.     

工业机器人实验多媒体教学软件的开发和应用

根据“机器人技术”课程的内容和有关实验教学的基本要求,开发了用于工业机器人实验的多媒体教学软件。软件的编制基于广东工业大学FMOC中心的硬件环境,做到了多媒体技术与实验设备的有机结合。     

Learning about natural human–robot interaction styles

If we are to achieve natural human–robot interaction, we may need to complement current vision and speech interfaces. Touch may provide us with an extra tool in this quest. In this paper we demonstrate the role of touch in interaction between a robot and a human. We show how infrared sensors located on robots can be easily used to detect and distinguish human interaction, in this case interaction with individual children. This application of infrared sensors potentially has many uses; for example, in entertainment or service robotics. This system could also benefit therapy or rehabilitation, where the observation and recording of movement and interaction is important. In the long term, this technique might enable robots to adapt to individuals or individual types of user.     

Lessons learned from Xavier

We have been running an experiment in web-based interaction with an autonomous indoor mobile robot. The robot, called Xavier, can accept commands to travel to different offices in our building, broadcasting camera images as it travels. The experiment, which was originally designed to test a new navigation algorithm, has proven very successful. This article describes the autonomous robot system, the web-based interfaces, and how they communicate with the robot. It highlights lessons learned during this experiment in web-based robotics and includes recommendations for putting future mobile robots on the web     

User evaluation of a novel eye-based control modality for robot-assisted object retrieval

In recent years, great efforts have been put into the investigation of assistive robotics. However, rapid growth in robot functionality has caused control of these robots to become more complex. Natural interaction interfaces between the user and robot systems are highly desired to promote the usability of these robots in practical applications. This paper presents a novel intuitive gaze-based system, which enables a user to effectively command an assistive robot with his/her gaze. This system has the potential to help those, who have limited mobility and/or are confined in bed, with their daily living. Using this system, the object that needs to be manipulated can be directly identified and localized by simply looking at it by the user. Then the assistive robot is activated to perform appropriate manipulation of the object. In the experiments, participants were able to successfully accomplish the object retrieval tasks using the presented system. Moreover, results from subjective evaluation showed high usability of the presented system.     

Multi-Modal Interaction of Human and Home Robot in the Context of Room Map Generation

In robotics, the idea of human and robot interaction is receiving a lot of attention lately. In this paper, we describe a multi-modal system for generating a map of the environment through interaction of a human and home robot. This system enables people to teach a newcomer robot different attributes of objects and places in the room through speech commands and hand gestures. The robot learns about size, position, and topological relations between objects, and produces a map of the room based on knowledge learned through communication with the human. The developed system consists of several sections including: natural language processing, posture recognition, object localization and map generation. This system combines multiple sources of information and model matching to detect and track a human hand so that the user can point toward an object of interest and guide the robot to either go near it or to locate that object's position in the room. The positions of objects in the room are located by monocular camera vision and depth from focus method.     

Object capture with a camera-mobile robot system

This article describes a mobile robot/camera system that is both simple and inexpensive enough for an undergraduate engineering student to construct. It provides an excellent first introduction to hands-on robotics, enabling the capture of small objects with the robot. It is scalable and can lead the interested student further into many diverse areas of robotics research. This system was carefully designed and arranged components so that the system can work with straightforward algorithms, off-the-shelf hardware, and minimal programming. Creating a working robot system not only introduces the beginning roboticist to many interesting problems in robotics but also creates a testbed to begin exploring them.     

Playing it safe [human-friendly robots]

We have presented a new actuation concept for human-friendly robot design, referred to as DM/sup 2/. The new concept of DM/sup 2/ was demonstrated on a two-degree-of-freedom prototype robot arm that we designed and built to validate our approach. The new actuation approach substantially reduces the impact loads associated with uncontrolled manipulator collision by relocating the major source of actuation effort from the joint to the base of the manipulator. The emerging field of human-centered robotics focuses on application such as medical robotics and service robotics, which require close interaction between robotic manipulation systems and human beings, including direct human-manipulator contact. As a result, this system must consider the requirements of safety. To achieve safety we must employ multiple strategies involving all aspects of manipulator design.     

Social and collaborative aspects of interaction with a service robot

To an increasing extent, robots are being designed to become a part of the lives of ordinary people. This calls for new models of the interaction between humans and robots, taking advantage of human social and communicative skills. Furthermore, human–robot relationships must be understood in the context of use of robots, and based on empirical studies of humans and robots in real settings. This paper discusses social aspects of interaction with a service robot, departing from our experiences of designing a fetch-and-carry robot for motion-impaired users in an office environment. We present the motivations behind the design of the Cero robot, especially its communication paradigm. Finally, we discuss experiences from a recent usage study, and research issues emerging from this work. A conclusion is that addressing only the primary user in service robotics is unsatisfactory, and that the focus should be on the setting, activities and social interactions of the group of people where the robot is to be used.     

Vision-based short range interaction between a personal service robot and a user

Interaction between a personal service robot and a human user is contingent on being aware of the posture and facial expression of users in the home environment. In this work, we propose algorithms to robustly and efficiently track the head, facial gestures, and the upper body movements of a user. The face processing module consists of 3D head pose estimation, modeling nonrigid facial deformations, and expression recognition. Thus, it can detect and track the face, and classify expressions under various poses, which is the key for human–robot interaction. For body pose tracking, we develop an efficient algorithm based on bottom-up techniques to search in a tree-structured 2D articulated body model, and identify multiple pose candidates to represent the state of current body configuration. We validate these face and body modules in varying experiments with different datasets, and the experimental results are reported. The implementation of both modules can run in real-time, which meets the requirement for real-world human–robot interaction task. These two modules have been ported onto a real robot platform by the Electronics and Telecommunications Research Institute.     

Design and experiments on a personal robotic assistant

The design and development of the 'personal robot, i.e. a robot capable of living together with and assisting human beings, is perhaps the main challenge for robotics research and technology in the third Millennium. One of the most intriguing issues in the design of such robot is the definition and implementation of adequate schemes for human-robot interaction. In fact, achieving this goal requires multidisciplinary efforts, and the integration of engineering analysis with psychological, anthropological, ethological and product design considerations. In this paper, the basic concepts to be followed in the design of human-robot interaction are discussed. with reference to a concrete example: the MOVAID personal robot for assistance to disabled and elderly people at home. The MOVAID system was designed taking into careful account as a primary design goal the need for interaction with the human user. A prototype of the MOVAID system has been developed and then successfully validated with disabled end-users. The results of these tests indicate that a personal robot is technically feasible, and is acceptable to users in terms of helpfulness, easiness of use and pleasure of interaction.     

Compliant actuator designs

In the growing fields of wearable robotics, rehabilitation robotics, prosthetics, and walking k robots, variable stiffness actuators (VSAs) or adjustable compliant actuators are being designed and implemented because of their ability to minimize large forces due to shocks, to safely interact with the user, and their ability to store and release energy in passive elastic elements. This review article describes the state of the art in the design of actuators with adaptable passive compliance. This new type of actuator is not preferred for classical position-controlled applications such as pick and place operations but is preferred in novel robots where safe human- robot interaction is required or in applications where energy efficiency must be increased by adapting the actuator's resonance frequency. The working principles of the different existing designs are explained and compared. The designs are divided into four groups: equilibrium-controlled stiffness, antagonistic-controlled stiffness, structure-controlled stiffness (SCS), and mechanically controlled stiffness.     

Teaching a pet-robot to understand user feedback through interactive virtual training tasks

In this paper, we present a human-robot teaching framework that uses “virtual” games as a means for adapting a robot to its user through natural interaction in a controlled environment. We present an experimental study in which participants instruct an AIBO pet robot while playing different games together on a computer generated playfield. By playing the games and receiving instruction and feedback from its user, the robot learns to understand the user’s typical way of giving multimodal positive and negative feedback. The games are designed in such a way that the robot can reliably predict positive or negative feedback based on the game state and explore its user’s reward behavior by making good or bad moves. We implemented a two-staged learning method combining Hidden Markov Models and a mathematical model of classical conditioning to learn how to discriminate between positive and negative feedback. The system combines multimodal speech and touch input for reliable recognition. After finishing the training, the system was able to recognize positive and negative reward with an average accuracy of 90.33%.     

Some considerations of user interface switching to support the weaker user

We have already proposed a new concept of “universal multimedia access” intended to narrow the digital divide by providing appropriate multimedia expressions according to users’ (mental and physical) abilities, computer facilities, and network environments. In this article, we redefine the switching functions for our new concept of universal design-based multimedia access, and discuss its user interface for supporting users in accordance with their abilities, computer facilities, and network environments.     

Human-robot coexistence and interaction in open industrial cells

Recent research results on human–robot interaction and collaborative robotics are leaving behind the traditional paradigm of robots living in a separated space inside safety cages, allowing humans and robot to work together for completing an increasing number of complex industrial tasks. In this context, safety of the human operator is a main concern. In this paper, we present a framework for ensuring human safety in a robotic cell that allows human–robot coexistence and dependable interaction. The framework is based on a layered control architecture that exploits an effective algorithm for online monitoring of relative human–robot distance using depth sensors. This method allows to modify in real time the robot behavior depending on the user position, without limiting the operative robot workspace in a too conservative way. In order to guarantee redundancy and diversity at the safety level, additional certified laser scanners monitor human–robot proximity in the cell and safe communication protocols and logical units are used for the smooth integration with an industrial software for safe low-level robot control. The implemented concept includes a smart human-machine interface to support in-process collaborative activities and for a contactless interaction with gesture recognition of operator commands. Coexistence and interaction are illustrated and tested in an industrial cell, in which a robot moves a tool that measures the quality of a polished metallic part while the operator performs a close evaluation of the same workpiece.     

Rewriting rules to permeate complex similarity and fuzzy queries within a relational database system

In recent years, the availability of complex data repositories (e.g., multimedia, genomic, semistructured databases) has paved the way to new potentials as to data querying. In this scenario, similarity and fuzzy techniques have proven to be successful principles for effective data retrieval. However, most proposals are domain specific and lack of a general and integrated approach to deal with generalized complex queries, i.e., queries where multiple conditions are expressed, possibly on complex as well as on traditional data. To overcome such limitations, much work has been devoted to the development of middleware systems to support query processing on multiple repositories. On a similar line, We present a formal framework to permeate complex similarity and fuzzy queries within a relational database system. As an example, we focus on multimedia data, which is represented in an integrated view with common database data. We have designed an application layer that relies on an algebraic query language, extended with MM-tailored operators, and that maps complex similarity and fuzzy queries to standard SQL statements that can be processed by a relational database system, exploiting standard facilities of modern extensible RDBMS. To show the applicability of our proposal, we implemented a prototype that provides the user with rich query capabilities, ranging from traditional database queries to complex queries gathering a mixture of Boolean, similarity, and fuzzy predicates on the data.     

Human-assistance robotic system based on distributed computing technology

Population aging is becoming an increasingly pressing problem for society. Thus, the timely development of supporting systems to improve care cost and the quality of life of the elderly is becoming particularly important. Because of the dispersal of the elderly and disabled, a standard software architecture that can implement network-distributed software sharing and improve the cost of writing and maintaining software is in high demand. This paper proposes using distributing computing technology CORBA to integrate network-distributed software and robotic systems for supporting the aged or disabled. Using CORBA as a communication architecture, we implemented a networkdistributed human-assistance robotic system. We developed a hardware base including a robot arm and an omnidirectional mobile robot, developed key technologies such as localization of a mobile robot, real-time recognition, real-time multimedia, and implemented several CORBA servers including a Web user management server, a task-level robot arm control server, a live image and video feedback server, a mobile robot control server, and an indoor Global Positioning System server. With the user controlling a robot arm with 5 d.o.f. to cooperate with an omnidirectional mobile robot, the developed system can provide basic services to support the aged and disabled.     

Vision-based remote control of cellular robots

This paper describes the development and design of a vision-based remote controlled cellular robot. Cellular robots have numerous applications in industrial problems where simple inexpensive robots can be used to perform different tasks that involve covering a large working space. As a methodology, the robots are controlled based on the visual input from one or more cameras that monitor the working area. As a result, a robust control of the robot trajectory is achieved without depending on the camera calibration. The remote user simply specifies a target point in the image to indicate the robot final position.

We describe the complete system at various levels: the visual information processing, the robot characteristics and the closed loop control system design, including the stability analysis when the camera location is unknown. Results are presented and discussed.

In our opinion, such a system may have a wide spectrum of applications in industrial robotics and may also serve as an educational testbed for advanced students in the fields of vision, robotics and control.     

Network Distributed Multi-Functional Robotic System Supporting the Elderly and Disabled People

We have been developing a network distributed Human-Assistance Robotic System in order to improve care cost and the QoL (Quality of Life) of the elderly people in the population-aging society. We developed multi-functional robotic system and implemented several CORBA application servers to provide some basic services to aid the aged or disabled. A novel method of localization of mobile robot with a camera and RFID (Radio Frequency Identification) technology is proposed as it is inexpensive, flexible and easy to use in practical environment. A video/audio conference system is also developed to improve the interaction among the users, switch robot manipulating privilege with the help of a centralized user management server, and enable a web-user to get a better understanding of what is going on in the local environment. Considering multi-type user of the developed, we have implemented multiple HRIs (Human Robot Interfaces) that enable different user to control robot systems easily.     

Robust face tracking control of a mobile robot using self‐tuning Kalman filter and echo state network

This paper presents a novel design of face tracking algorithm and visual state estimation for a mobile robot face tracking interaction control system. The advantage of this design is that it can track a user's face under several external uncertainties and estimate the system state without the knowledge about target's 3D motion‐model information. This feature is helpful for the development of a real‐time visual tracking control system. In order to overcome the change in skin color due to light variation, a real‐time face tracking algorithm is proposed based on an adaptive skin color search method. Moreover, in order to increase the robustness against colored observation noise, a new visual state estimator is designed by combining a Kalman filter with an echo state network‐based self‐tuning algorithm. The performance of this estimator design has been evaluated using computer simulation. Several experiments on a mobile robot validate the proposed control system. Copyright © 2010 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society