Bilateral remote teaching and autonomous task execution with task progress feedback

This paper describes an approach to estimating the progress in a task executed by a humanoid robot and to synthesizing motion based on the current progress so that the robot can achieve the task. The robot observes a human performing whole body motion for a specific task, and encodes these motions into a hidden Markov model (HMM). The current observation is compared with the motion generated by the HMM, and the task progress can be estimated during the robot performing the motion. The robot subsequently uses the estimate of the task progress to generate a motion appropriate to the current situation with the feedback rule. We constructed a bilateral remote control system with humanoid robot HRP-4 and haptic device Novint Falcon, and we made the humanoid robot push a button. Ten trial motions of pushing a button were recorded for the training data. We tested our proposed approach on the autonomous execution of the pushing motion by the humanoid robot, and confirmed the effectiveness of our task progress feedback method.     

Planning and executing navigation among movable obstacles

This paper explores autonomous locomotion, reaching, grasping and manipulation for the domain of navigation among movable obstacles (NAMO). The robot perceives and constructs a model of an environment filled with various fixed and movable obstacles, and automatically plans a navigation strategy to reach a desired goal location. The planned strategy consists of a sequence of walking and compliant manipulation operations. It is executed by the robot with online feedback. We give an overview of our NAMO system, as well as provide details of the autonomous planning, online grasping and compliant hand positioning during dynamically stable walking. Finally, we present results of a successful implementation running on the humanoid robot HRP-2.     

Whole-Body Motion Generation Integrating Operator's Intention and Robot's Autonomy in Controlling Humanoid Robots

This paper introduces a framework for whole-body motion generation integrating operator's control and robot's autonomous functions during online control of humanoid robots. Humanoid robots are biped machines that usually possess multiple degrees of freedom (DOF). The complexity of their structure and the difficulty in maintaining postural stability make the whole-body control of humanoid robots fundamentally different from fixed-base manipulators. Getting hints from human conscious and subconscious motion generations, the authors propose a method of generating whole-body motions that integrates the operator's command input and the robot's autonomous functions. Instead of giving commands to all the joints all the time, the operator selects only the necessary points of the humanoid robot's body for manipulation. This paper first explains the concept of the system and the framework for integrating operator's command and autonomous functions in whole-body motion generation. Using the framework, autonomous functions were constructed for maintaining postural stability constraint while satisfying the desired trajectory of operation points, including the feet, while interacting with the environment. Finally, this paper reports on the implementation of the proposed method to teleoperate two 30-DOF humanoid robots, HRP-1S and HRP-2, by using only two 3-DOF joysticks. Experiments teleoperating the two robots are reported to verify the effectiveness of the proposed method.     

A sensor for dynamic tactile information with applications in human–robot interaction and object exploration

We present a novel tactile sensor, which is applied for dextrous grasping with a simple robot gripper. The hardware novelty consists of an array of capacitive sensors, which couple to the object by means of little brushes of fibers. These sensor elements are very sensitive (with a threshold of about 5 mN) but robust enough not to be damaged during grasping. They yield two types of dynamical tactile information corresponding roughly to two types of tactile sensor in the human skin. The complete sensor consists of a foil-based static force sensor, which yields the total force and the center of the two-dimensional force distribution and is surrounded by an array of the dynamical sensor elements. One such sensor has been mounted on each of the two gripper jaws of our humanoid robot and equipped with the necessary read-out electronics and a CAN bus interface. We describe applications to guiding a robot arm on a desired trajectory with negligible force, reflective grip improvement, and tactile exploration of objects to create a shape representation and find stable grips, which are applied autonomously on the basis of visual recognition.     

Force feedback exploiting tactile and proximal force/torque sensing

The paper addresses the problem of measuring whole-body dynamics for a multiple-branch kinematic chain in presence of unknown external wrenches. The main result of the paper is to give a methodology for computing whole body dynamics by aligning a model of the system dynamics with the measurements coming from the available sensors. Three primary sources of information are exploited: (1) embedded force/torque sensors, (2) embedded inertial sensors, (3) distributed tactile sensors (i.e. artificial skin). In order to cope with external wrenches applied at continuously changing locations, we model the kinematic chain with a graph which dynamically adapts to the contact locations. Classical pre-order and post-order traversals of this dynamically evolving graph allow computing whole-body dynamics and estimate external wrenches. Theoretical results have been implemented in an open-source software library (iDyn) released under the iCub project. Experimental results on the iCub humanoid robot show the effectiveness of the proposed approach.     

Designing Synergistic Walking of a Whole-Body Humanoid Driven by Pneumatic Artificial Muscles: An Empirical Study

Our body consists of many body parts that are compliantly connected with each other by muscles and ligaments, and their behavior emerges out of the synergy of the whole-body dynamics. Such synergistic behavior generation is supposed to contribute to human adaptive movement such as walking. This paper describes designing synergistic walking of a whole-body humanoid robot whose joints are driven by artificial pneumatic muscles antagonistically. We propose to take an incremental design approach to deal with the complicated dynamics of the system. As a result, we can determine control parameters that govern whole-body behavior. We experimentally demonstrate that the humanoid walks stably with a simple limit-cycle controller.     

Dynamic perception after visually guided grasping by a human-like autonomous robot

We will explore dynamic perception following the visually guided grasping of several objects by a human-like autonomous robot. This competency serves for object categorization. Physical interaction with the hand-held object gives the neural network of the robot the rich, coherent and multi-modal sensory input. Multi-layered self-organizing maps are designed and examined in static and dynamic conditions. The results of the tests in the former condition show its capability of robust categorization against noise. The network also shows better performance than a single-layered map does. In the latter condition we focus on shaking behavior by moving only the forearm of the robot. In some combinations of grasping style and shaking radius the network is capable of categorizing two objects robustly. The results show that the network capability to achieve the task largely depends on how to grasp and how to move the objects. These results together with a preliminary simulation are promising toward the self-organization of a high degree of autonomous dynamic object categorization.     

Humanoid robotics platforms developed in HRP

This paper presents humanoid robotics platform that consists of a humanoid robot and an open architecture software platform developed in METI’s Humanoid Robotics Project (HRP). The final version of the robot, called HRP-2, has 1540 mm height, 58 kg weight and 30 degrees of the freedom. The software platform includes a dynamics simulator and motion controllers of the robot for biped locomotion, falling and getting up motions. The platform has been used to develop various applications and is expected to initiate more humanoid robotics research.     

Stepping over obstacles with humanoid robots

The wide potential applications of humanoid robots require that the robots can walk in complex environments and overcome various obstacles. To this end, we address the problem of humanoid robots stepping over obstacles in this paper. We focus on two aspects, which are feasibility analysis and motion planning. The former determines whether a robot can step over a given obstacle, and the latter discusses how to step over, if feasible, by planning appropriate motions for the robot. We systematically examine both of these aspects. In the feasibility analysis, using an optimization technique, we cast the problem into global optimization models with nonlinear constraints, including collision-free and balance constraints. The solutions to the optimization models yield answers to the possibility of stepping over obstacles under some assumptions. The presented approach for feasibility provides not only a priori knowledge and a database to implement stepping over obstacles, but also a tool to evaluate and compare the mobility of humanoid robots. In motion planning, we present an algorithm to generate suitable trajectories of the feet and the waist of the robot using heuristic methodology, based on the results of the feasibility analysis. We decompose the body motion of the robot into two parts, corresponding to the lower body and upper body of the robot, to meet the collision-free and balance constraints. This novel planning method is adaptive to obstacle sizes, and is, hence, oriented to autonomous stepping over by humanoid robots guided by vision or other range finders. Its effectiveness is verified by simulations and experiments on our humanoid platform HRP-2.     

A bio-inspired predictive sensory-motor coordination scheme for robot reaching and preshaping

This paper presents a sensory-motor coordination scheme for a robot hand-arm-head system that provides the robot with the capability to reach an object while pre-shaping the fingers to the required grasp configuration and while predicting the tactile image that will be perceived after grasping. A model for sensory-motor coordination derived from studies in humans inspired the development of this scheme. A peculiar feature of this model is the prediction of the tactile image. The implementation of the proposed scheme is based on a neuro-fuzzy module that, after a learning phase, starting from visual data, calculates the position and orientation of the hand for reaching, selects the best-suited hand configuration, and predicts the tactile feedback. The implementation of the scheme on a humanoid robot allowed experimental validation of its effectiveness in robotics and provided perspectives on applications of sensory predictions in robot motor control.     

morph3: a compact-size humanoid robot system capable of acrobatic behavior

morph3 is a compact-size humanoid robot that is able to generate acrobatic motions such as somersaults and gymnastic motion. A new mechanism and control system, i. e. a real-time joint compliance control system and tactile sensing shells on its body, make acrobatic motion possible. In this paper, the design concepts and construction of morph3 are described.     

A new paradigm of humanoid robot motion programming based on touch interpretation

Most humanoid soccer robot teams design the basic movements of their robots, like walking and kicking, off-line and manually. Once these motions are considered satisfactory, they are stored in the robot’s memory and played according to a high level behavioral strategy. Much time is spent in the development of the movements, and despite the significant progress made in humanoid soccer robots, the interfaces employed for the development of motions are still quite primitive. In order to accelerate development, an intuitive instruction method is desired. We propose the development of robot motions through physical interaction. In this paper we propose a ”teaching by touching” approach; the human operator teaches a motion by directly touching the robot’s body parts like a dance instructor. Teaching by directly touching is intuitive for instructors. However, the robot needs to interpret the instructor’s intention since tactile communication can be ambiguous. This paper presents a method to learn the interpretation of the touch meaning and investigates, through experiments, a general (shared among different users) and intuitive touch manner.     

From human to humanoid locomotion—an inverse optimal control approach

The purpose of this paper is to present inverse optimal control as a promising approach to transfer biological motions to robots. Inverse optimal control helps (a) to understand and identify the underlying optimality criteria of biological motions based on measurements, and (b) to establish optimal control models that can be used to control robot motion. The aim of inverse optimal control problems is to determine—for a given dynamic process and an observed solution—the optimization criterion that has produced the solution. Inverse optimal control problems are difficult from a mathematical point of view, since they require to solve a parameter identification problem inside an optimal control problem. We propose a pragmatic new bilevel approach to solve inverse optimal control problems which rests on two pillars: an efficient direct multiple shooting technique to handle optimal control problems, and a state-of-the art derivative free trust region optimization technique to guarantee a match between optimal control problem solution and measurements. In this paper, we apply inverse optimal control to establish a model of human overall locomotion path generation to given target positions and orientations, based on newly collected motion capture data. It is shown how the optimal control model can be implemented on the humanoid robot HRP-2 and thus enable it to autonomously generate natural locomotion paths.     

Evaluation of active wearable assistive devices with human posture reproduction using a humanoid robot*

This study proposes a quantitative evaluation method for assessing active wearable assistive devices that can efficiently support the human body. We utilize a humanoid robot to simulate human users wearing assistive devices owing to various advantages offered by the robot such as quantitative torque measurement from sensors and highly repeatable motion. In this study, we propose a scheme for estimating the supportive torques supplied by a device called stationary torque replacement. To validate the reliability of this evaluation method by using a humanoid robot, we conducted measurements of human muscular activity during assisted motion. Analysis of the measured muscle activity revealed that a humanoid robot closely simulates the actual usage of assistive devices. Finally, we showed the feasibility of the proposed evaluation method through an experiment with the humanoid robot platform HRP-4 and the Muscle Suit active assistive device. With the proposed method, the supportive effects of the assistive device could be measured quantitatively in terms of the static supportive torque acting directly on the body of a simulated human user.     

HRP-2W: A humanoid platform for research on support behavior in daily life environments

We introduce a concept of a real-world-oriented humanoid robot that can support humans’ activities in daily life. In such environments, robots have to watch humans, understand their behavior, and support their daily life tasks. In particular, these robots must be capable of such real-world behavior as handling tableware and delivering daily commodities by hand. We developed a humanoid robot, HRP-2W, which has an upper body of HRP-2 [K. Kaneko, F. Kanehiro, S. Kajita, H. Hirukawa, T. Kawasaki, M. Hirata, K. Akachi, T. Isozumi, Humanoid Robot HRP-2, in: Proceedings of the 2004 IEEE International Conference on Robotics & Automation, 2004, pp. 1083–1090] and a wheel module instead of legs, as a research platform to fulfill this aim. We also developed basic software configuration in order to integrate our platform with other research groups. Through experiments, we demonstrated the feasibility of the humanoid robot platform and the potential of the software architecture.     

Implementation and stability analysis of prioritized whole-body compliant controllers on a wheeled humanoid robot in uneven terrains

In this work, we implement the floating base prioritized whole-body compliant control framework described in Sentis et al. (IEEE Transactions on Robotics 26(3):483–501, 2010) on a wheeled humanoid robot maneuvering in sloped terrains. We then test it for a variety of compliant whole-body behaviors including balance and kinesthetic mobility on irregular terrain, and Cartesian hand position tracking using the co-actuated (i.e. two joints are simultaneously actuated with one motor) robot’s upper body. The implementation serves as a hardware proof for a variety of whole-body control concepts that had previously been developed and tested in simulation. First, behaviors of two and three priority tasks are implemented and successfully executed on the humanoid hardware. In particular, first and second priority tasks are linearized in the task space through model feedback and then controlled through task accelerations. Postures, on the other hand, are shown to be asymptotically stable when using prioritized whole-body control structures and then successfully tested in the real hardware. To cope with irregular terrains, the base is modeled as a six degree of freedom floating system and the wheels are characterized through contact and rolling constraints. Finally, center of mass balance capabilities using whole-body compliant control and kinesthetic mobility are implemented and tested in the humanoid hardware to climb terrains with various slopes.     

Humanoids skill learning based on real-time human motion imitation using Kinect

In this paper, a novel framework which enables humanoid robots to learn new skills from demonstration is proposed. The proposed framework makes use of real-time human motion imitation module as a demonstration interface for providing the desired motion to the learning module in an efficient and user-friendly way. This interface overcomes many problems of the currently used interfaces like direct motion recording, kinesthetic teaching, and immersive teleoperation. This method gives the human demonstrator the ability to control almost all body parts of the humanoid robot in real time (including hand shape and orientation which are essential to perform object grasping). The humanoid robot is controlled remotely and without using any sophisticated haptic devices, where it depends only on an inexpensive Kinect sensor and two additional force sensors. To the best of our knowledge, this is the first time for Kinect sensor to be used in estimating hand shape and orientation for object grasping within the field of real-time human motion imitation. Then, the observed motions are projected onto a latent space using Gaussian process latent variable model to extract the relevant features. These relevant features are then used to train regression models through the variational heteroscedastic Gaussian process regression algorithm which is proved to be a very accurate and very fast regression algorithm. Our proposed framework is validated using different activities concerned with both human upper and lower body parts and object grasping also.     

Pivoting based manipulation by a humanoid robot

In this paper we address whole-body manipulation of bulky objects by a humanoid robot. We adopt a “pivoting” manipulation method that allows the humanoid to displace an object without lifting, but by the support of the ground contact. First, the small-time controllability of pivoting is demonstrated. On its basis, an algorithm for collision-free pivoting motion planning is established taking into account the naturalness of motion as nonholonomic constraints. Finally, we present a whole-body motion generation method by a humanoid robot, which is verified by experiments.     

Learning to Acquire Whole-Body Humanoid Center of Mass Movements to Achieve Dynamic Tasks

This paper presents a novel approach for acquiring dynamic whole-body movements on humanoid robots focused on learning a control policy for the center of mass (CoM). In our approach, we combine both a model-based CoM controller and a model-free reinforcement learning (RL) method to acquire dynamic whole-body movements in humanoid robots. (i) To cope with high dimensionality, we use a model-based CoM controller as a basic controller that derives joint angular velocities from the desired CoM velocity. The balancing issue can also be considered in the controller. (ii) The RL method is used to acquire a controller that generates the desired CoM velocity based on the current state. To demonstrate the effectiveness of our approach, we apply it to a ball-punching task on a simulated humanoid robot model. The acquired whole-body punching movement was also demonstrated on Fujitsu's Hoap-2 humanoid robot.     

Digital twin-enabled grasp outcomes assessment for unknown objects using visual-tactile fusion perception

Humans can instinctively predict whether a given grasp will be successful through visual and rich haptic feedback. Towards the next generation of smart robotic manufacturing, robots must be equipped with similar capabilities to cope with grasping unknown objects in unstructured environments. However, most existing data-driven methods take global visual images and tactile readings from the real-world system as input, making them incapable of predicting the grasp outcomes for cluttered objects or generating large-scale datasets. First, this paper proposes a visual-tactile fusion method to predict the results of grasping cluttered objects, which is the most common scenario for grasping applications. Concretely, the multimodal fusion network (MMFN) uses the local point cloud within the gripper as the visual signal input, while the tactile signal input is the images provided by two high-resolution tactile sensors. Second, collecting data in the real world is high-cost and time-consuming. Therefore, this paper proposes a digital twin-enabled robotic grasping system to collect large-scale multimodal datasets and investigates how to apply domain randomization and domain adaptation to bridge the sim-to-real transfer gap. Finally, extensive validation experiments are conducted in physical and virtual environments. The experimental results demonstrate the effectiveness of the proposed method in assessing grasp stability for cluttered objects and performing zero-shot sim-to-real policy transfer on the real robot with the aid of the proposed migration strategy.