Human Tracking System Integrating Sound and Face Localization Using an Expectation-Maximization Algorithm in Real Environments

We have developed a human tracking system for use by robots that integrate sound and face localization. Conventional systems usually require many microphones and/or prior information to localize several sound sources. Moreover, they are incapable of coping with various types of background noise. Our system, the cross-power spectrum phase analysis of sound signals obtained with only two microphones, is used to localize the sound source without having to use prior information such as impulse response data. An expectation-maximization (EM) algorithm is used to help the system cope with several moving sound sources. The problem of distinguishing whether sounds are coming from the front or back is also solved with only two microphones by rotating the robot's head. A developed method that uses facial skin colors classified by another EM algorithm enables the system to detect faces in various poses. It can compensate for the error in the sound localization for a speaker and also identify noise signals entering from undesired directions by detecting a human face. A developed probability-based method is used to integrate the auditory and visual information in order to produce a reliable tracking path in real-time. Experiments using a robot showed that our system can localize two sounds at the same time and track a communication partner while dealing with various types of background noise.     

Sound and Visual Tracking for Humanoid Robot

Mobile robots capable of auditory perception usually adopt the stop-perceive-act principle to avoid sounds made during moving due to motor noise. Although this principle reduces the complexity of the problems involved in auditory processing for mobile robots, it restricts their capabilities of auditory processing. In this paper, sound and visual tracking are investigated to compensate each other's drawbacks in tracking objects and to attain robust object tracking. Visual tracking may be difficult in case of occlusion, while sound tracking may be ambiguous in localization due to the nature of auditory processing. For this purpose, we present an active audition system for humanoid robot. The audition system of the highly intelligent humanoid requires localization of sound sources and identification of meanings of the sound in the auditory scene. The active audition reported in this paper focuses on improved sound source tracking by integrating audition, vision, and motor control. Given the multiple sound sources in the auditory scene, SIG the humanoid actively moves its head to improve localization by aligning microphones orthogonal to the sound source and by capturing the possible sound sources by vision. The system adaptively cancels motor noises using motor control signals. The experimental result demonstrates the effectiveness of sound and visual tracking.     

Improved binaural sound localization and tracking for unknown time-varying number of speakers

A method based on the generalized cross-correlation (GCC) method weighted by the phase transform (PHAT) has been developed for binaural sound source localization (SSL) and tracking of multiple sound sources. Accurate binaural audition is important for applying inexpensive and widely applicable auditory capabilities to robots and systems. Conventional SSL based on the GCC-PHAT method is degraded by low resolution of the time difference of arrival estimation, by the interference created when the sound waves arrive at a microphone from two directions around the robot head, and by impaired performance when there are multiple speakers. The low-resolution problem is solved by using a maximum-likelihood-based SSL method in the frequency domain. The multipath interference problem is avoided by incorporating a new time delay factor into the GCC-PHAT method with assuming a spherical robot head. The performance when there are multiple speakers was improved by using a multisource speech tracking method consisting of voice activity detection (VAD) and K-means clustering. The standard K-means clustering algorithm was extended to enable tracking of an unknown time-varying number of speakers by adding two additional steps that increase the number of clusters automatically and eliminate clusters containing incorrect direction estimations. Experiments conducted on the SIG-2 humanoid robot show that this method outperforms the conventional SSL method; it reduces localization errors by 18.1° on average and by over 37° in the side directions. It also tracks multiple speakers in real time with tracking errors below 4.35°.     

非时间参考的移动机器人路径跟踪控制

基于非时间参考的思想提出了一种移动机器人路径跟踪控制方法.首先选择移动机器人实际路径在某参考系下的x轴投影作为非时间参考量,并针对一类几何路径的跟踪建立移动机器人非时间参考的运动学模 型,据此设计以恒定速度跟踪期望路径的控制律,然后在此基础上给出跟踪任意几何路径的分段控制策略.此跟踪 控制系统所采用的参考量为非时间量􀁯,摆脱了时间因素的影响,因此能提高移动机器人在不确定环境中的跟踪能 力.仿真和物理实验表明了控制方法的有效性.􀁱     

基于Lyapunov方法和快速终端滑模的轨迹跟踪控制

针对移动机器人的运动学模型,提出一种具有全局渐近稳定性的跟踪控制器。该跟踪控制器的设计分为两部分：第一部分是采用全局快速终端滑动模态的思想设计了角速度的控制律,用来渐近镇定移动机器人跟踪的前向角误差;第二部分是采用Lyapunov方法设计了线速度的控制律,用来渐近镇定移动机器人跟踪的平面坐标误差。采用Lyapunov稳定性定理,证明了移动机器人在满足这些控制律条件下,实现了对参考轨迹的全局渐近跟踪。实验结果表明移动机器人能够有效地跟踪期望轨迹,有利于在实际应用中推广。     

Reliability-based camera handoff for cooperative tracking with multiple pan-tilt cameras

We present a new handoff method for multiple pan-tilt cameras for mobile robot tracking in an indoor environment. Camera handoff is an important step to consistently maintain the visibility of a mobile robot with maximized object tracking accuracy. First, we propose a method to estimate the position of a mobile robot using single pan-tilt camera. Then, the concept of position reliability is defined to quantitatively evaluate the accuracy of position estimation and tracking ability of individual pan-tilt cameras. Position reliability is used to decide when to trigger handoff and who to response handoff in the proposed handoff algorithm. Experimental results demonstrate that four pan-tilt cameras can systematically track a mobile robot in an indoor environment using the proposed method.     

Discovery of sound sources by an autonomous mobile robot

In this work, we describe an autonomous mobile robotic system for finding, investigating, and modeling ambient noise sources in the environment. The system has been fully implemented in two different environments, using two different robotic platforms and a variety of sound source types. Making use of a two-step approach to autonomous exploration of the auditory scene, the robot first quickly moves through the environment to find and roughly localize unknown sound sources using the auditory evidence grid algorithm. Then, using the knowledge gained from the initial exploration, the robot investigates each source in more depth, improving upon the initial localization accuracy, identifying volume and directivity, and, finally, building a classification vector useful for detecting the sound source in the future.     

Multisensor-Based Human Detection and Tracking for Mobile Service Robots

One of fundamental issues for service robots is human-robot interaction. In order to perform such a task and provide the desired services, these robots need to detect and track people in the surroundings. In this paper, we propose a solution for human tracking with a mobile robot that implements multisensor data fusion techniques. The system utilizes a new algorithm for laser-based leg detection using the onboard laser range finder (LRF). The approach is based on the recognition of typical leg patterns extracted from laser scans, which are shown to also be very discriminative in cluttered environments. These patterns can be used to localize both static and walking persons, even when the robot moves. Furthermore, faces are detected using the robot's camera, and the information is fused to the legs' position using a sequential implementation of unscented Kalman filter. The proposed solution is feasible for service robots with a similar device configuration and has been successfully implemented on two different mobile platforms. Several experiments illustrate the effectiveness of our approach, showing that robust human tracking can be performed within complex indoor environments.     

A system decomposition method for region tracking control of a non-holonomic mobile robot with dynamic parameter uncertainties

The tracking control problem of non-holonomic mobile robot systems has been extensively investigated in the past decades, however, most of the existing control strategies were developed specifically for the fixed-point tracking. This technical note focuses on the region tracking control for a non-holonomic mobile robot system with parameter uncertainties in the robot dynamics. With the system decomposition and adaptive control method, some restrictions imposed on the angular and linear velocities of the non-holonomic mobile robot in recent literature are removed, enabling to track dynamic trajectories with any values of the angular and line velocities. The proposed adaptive control scheme can simultaneously solve both the regulation and region tracking problems of a non-holonomic mobile robot with one passive wheel and two actuated wheels. By utilizing the designed control laws, the mobile robot system is able to globally reach inside a moving region specified by potential functions whose path can be a circular curve, a straight line, or sinusoidal curve, by using a single adaptive controller. Since the dynamic region can be specified arbitrarily small, the fixed-point tracking can be regarded as a special case of region tracking studied in this paper. Compared with the traditional fixed-point tracking, region tracking has more flexibility and better robustness. Numerical results are presented to show the effectiveness of the designed strategy.     

基于主动视觉和超声的机器人目标跟踪系统

运动目标跟踪技术是未知环境下移动机器人研究领域的一个重要研究方向。该文提出了一种基于主动视觉和超声信息的移动机器人运动目标跟踪设计方法,利用一台SONY EV-D31彩色摄像机、自主研制的摄像机控制模块、图像采集与处理单元等构建了主动视觉系统。移动机器人采用了基于行为的分布式控制体系结构,利用主动视觉锁定运动目标,通过超声系统感知外部环境信息,能在未知的、动态的、非结构化复杂环境中可靠地跟踪运动目标。实验表明机器人具有较高的鲁棒性,运动目标跟踪系统运行可靠。     

基于决策树的羽流追踪机器人自主决策方法研究

针对无法获得可靠羽流流向信息不利于实现羽流追踪的问题，提出了一种基于决策树的羽流追踪移动机器人自主决策方法。该方法通过移动机器人两侧的浓度传感器采集到的浓度信息，利用追踪的行为规则建立决策树模型，获得行为决策信息，使机器人高效地追踪到羽流并精确地定位。由于浓度变化关系蕴含了羽流的流向及流速信息，从而取代了传统方法中流向及流速传感器。在扩散环境下，通过移动机器人羽流追踪实验，实现了良好的源定位效果。     

Mobile robot self-localisation using occupancy histograms and a mixture of Gaussian location hypotheses

The topic of mobile robot self-localisation is often divided into the sub-problems of global localisation and position tracking. Both are now well understood individually, but few mobile robots can deal simultaneously with the two problems in large, complex environments. In this paper, we present a unified approach to global localisation and position tracking which is based on a topological map augmented with metric information. This method combines a new scan matching technique, using histograms extracted from local occupancy grids, with an efficient algorithm for tracking multiple location hypotheses over time. The method was validated with experiments in a series of real world environments, including its integration into a complete navigating robot. The results show that the robot can localise itself reliably in large, indoor environments using minimal computational resources.     

Decentralized Coordinated Tracking with Mixed Discrete–Continuous Decisions

In this paper, we study the problem of dynamically positioning a team of mobile robots for target tracking. We treat the coordination of mobile robots for target tracking as a joint team optimization to minimize uncertainty in target state estimates over a fixed horizon. The optimization is inherently a function of both the positioning of robots in continuous space and the assignment of robots to targets in discrete space. Thus, the robot team must make decisions over discrete and continuous variables. In contrast to methods that decouple target assignments and robot positioning, our approach avoids the strong assumption that a robot's utility for observing a target is independent of other robots’ observations. We formulate the optimization as a mixed integer nonlinear program and apply integer relaxation to develop an approximate solution in decentralized form. We demonstrate our coordinated multirobot tracking algorithm both in simulation and using a pair of mobile robotic sensor platforms to track moving pedestrians. Our results show that coupling target assignment and robot positioning realizes coordinated behaviors that are not possible with decoupled methods.     

Robust face tracking control of a mobile robot using self‐tuning Kalman filter and echo state network

This paper presents a novel design of face tracking algorithm and visual state estimation for a mobile robot face tracking interaction control system. The advantage of this design is that it can track a user's face under several external uncertainties and estimate the system state without the knowledge about target's 3D motion‐model information. This feature is helpful for the development of a real‐time visual tracking control system. In order to overcome the change in skin color due to light variation, a real‐time face tracking algorithm is proposed based on an adaptive skin color search method. Moreover, in order to increase the robustness against colored observation noise, a new visual state estimator is designed by combining a Kalman filter with an echo state network‐based self‐tuning algorithm. The performance of this estimator design has been evaluated using computer simulation. Several experiments on a mobile robot validate the proposed control system. Copyright © 2010 John Wiley and Sons Asia Pte Ltd and Chinese Automatic Control Society     

Auditory scene reproduction for tele-operated robot systems

ABSTRACT

In a tele-operated robot environment, reproducing auditory scenes and conveying 3D spatial information of sound sources are inevitable in order to make operators feel more realistic tele-presence. In this paper, we propose a tele-presence robot system that enables reproduction and manipulation of auditory scenes. This tele-presence system is carried out on the basis of 3D information about where targeted humans are speaking, and matching with the operator's head orientation. We employed multiple microphone arrays and human tracking technologies to localize and separate voices around a robot. In the operator site, separated sound sources are rendered using head-related transfer functions (HRTF) according to the sound sources' spatial positions and the operator's head orientation that is being tracked real time. Two-party and three-party interaction experiments indicated that the proposed system has significantly higher accuracy when perceiving direction of sounds and gains higher subjective scores in sense of presence and listenability, compared to a baseline system which uses stereo binaural sounds obtained by two microphones mounted on the humanoid robot's ears.     

Simultaneous Localization of a Mobile Robot and Multiple Sound Sources Using a Microphone Array

Sound source localization is an important function in robot audition. Most existing works perform sound source localization using static microphone arrays. This work proposes a framework that simultaneously localizes the mobile robot and multiple sound sources using a microphone array on the robot. First, an eigenstructure-based generalized cross-correlation method for estimating time delays between microphones under multi-source environments is described. Using the estimated time delays, a method to compute the farfield source directions as well as the speed of sound is proposed. In addition, the correctness of the sound speed estimate is utilized to eliminate spurious sources, which greatly enhances the robustness of sound source detection. The arrival angles of the detected sound sources are used as observations in a bearing-only simultaneous localization and mapping procedure. As the source signals are not persistent and there is no identification of the signal content, data association is unknown and it is solved using the FastSLAM algorithm. The experimental results demonstrate the effectiveness of the proposed method.     

Trajectory tracking optimization of mobile robot using artificial immune system

In this paper, an optimization method that provides quick response using artificial immune system, is proposed and applied to a mobile robot for trajectory tracking. The study focuses on the immune theory to derive a quick optimization method that puts emphasis on immunity feedback using memory cells by the expansion and suppression of the test group rather than to derive a specific mathematical model of the artificial immune system. Various trajectories were selected in mobile environment to evaluate the performance of the proposed artificial immune system. The global inputs to the mobile robot are reference position and reference velocity, which are time variables. The global output of mobile robot is a current position. The tracking controller makes position error to be converged to zero. In order to reduce position error, compensation velocities on the track of trajectory are necessary. Input variables of fuzzy are position errors in every sampling time. The output values of fuzzy are compensation velocities. Immune algorithm is implemented to adjust the scaling factor of fuzzy automatically. The results of the computer simulation proved the system to be efficient and effective for tracing the trajectory to the final destination by the mobile robot.

     

The Tracking of a Moving Object by a Mobile Robot Following the Object’s Sound

The tracking of a moving object with a mobile robot has been implemented based on the detected sound from the moving object using a microphone array. The difference between the travel times of the sound source to each of the three microphones mounted to the robot has been used to calculate the distance and orientation of the sound source. The cross-correlations between the received signals have been used to detect the individual sound signal from the object and to calculate the time difference between two signals. This provides reliable and precise time differences among the sound signals arrived at the microphones compared to the conventional method. In order to determine the tracking direction to the sound source, Fuzzy rules have been applied; the results are used for real-time control of the mobile robot. The efficiency of the proposed algorithm has been demonstrated through real-world experiments and compared to the conventional approach.     

基于迭代学习控制的移动机器人轨迹跟踪控制

为提高移动机器人对特定轨迹的重复跟踪能力,提出了采用开闭环PD型迭代学习控制算法对移动机器人进行轨迹跟踪控制的方法。建立了包含外界干扰的非完整约束条件下的轮式移动机器人运动学模型,给出了系统的控制算法和控制结构。仿真结果表明,采用开闭环PD型迭代学习控制算法对轨迹跟踪是可行有效的,收敛速度优于其他迭代学习算法。     

Position estimation of a mobile robot using images of a moving target in intelligent space with distributed sensors

Latest advances in hardware technology and state-of-the-art of mobile robots and artificial intelligence research can be employed to develop autonomous and distributed monitoring systems. A mobile service robot requires the perception of its present position to co-exist with humans and support humans effectively in populated environments. To realize this, a robot needs to keep track of relevant changes in the environment. This paper proposes localization of a mobile robot using images recognized by distributed intelligent networked devices in intelligent space (ISpace) in order to achieve these goals. This scheme combines data from the observed position, using dead-reckoning sensors, and the estimated position, using images of moving objects, such as a walking human captured by a camera system, to determine the location of a mobile robot. The moving object is assumed to be a point-object and projected onto an image plane to form a geometrical constraint equation that provides position data of the object based on the kinematics of the ISpace. Using the a priori known path of a moving object and a perspective camera model, the geometric constraint equations that represent the relation between image frame coordinates for a moving object and the estimated robot's position are derived. The proposed method utilizes the error between the observed and estimated image coordinates to localize the mobile robot, and the Kalman filtering scheme is used for the estimation of the mobile robot location. The proposed approach is applied for a mobile robot in ISpace to show the reduction of uncertainty in determining the location of a mobile robot, and its performance is verified by computer simulation and experiment.