Supervised semantic labeling of places using information extracted from sensor data

Indoor environments can typically be divided into places with different functionalities like corridors, rooms or doorways. The ability to learn such semantic categories from sensor data enables a mobile robot to extend the representation of the environment facilitating interaction with humans. As an example, natural language terms like “corridor” or “room” can be used to communicate the position of the robot in a map in a more intuitive way. In this work, we first propose an approach based on supervised learning to classify the pose of a mobile robot into semantic classes. Our method uses AdaBoost to boost simple features extracted from sensor range data into a strong classifier. We present two main applications of this approach. Firstly, we show how our approach can be utilized by a moving robot for an online classification of the poses traversed along its path using a hidden Markov model. In this case we additionally use as features objects extracted from images. Secondly, we introduce an approach to learn topological maps from geometric maps by applying our semantic classification procedure in combination with a probabilistic relaxation method. Alternatively, we apply associative Markov networks to classify geometric maps and compare the results with a relaxation approach. Experimental results obtained in simulation and with real robots demonstrate the effectiveness of our approach in various indoor environments.     

基于场景图谱的室内移动机器人目标搜索

在移动机器人执行日常家庭任务时,首先需要其能够在环境中避开障碍物,自主地寻找到房间中的物体。针对移动机器人如何有效在室内环境下对目标物体进行搜索的问题,提出了一种基于场景图谱的室内移动机器人目标搜索,其框架结合了导航地图、语义地图和语义关系图谱。在导航地图的基础上建立了包含地标物体位置信息的语义地图,机器人可以轻松对地标物体进行寻找。对于动态的物体,机器人根据语义关系图中物体之间的并发关系,优先到关系强度比较高的地标物体旁寻找。通过物理实验展示了机器人在语义地图和语义关系图的帮助下可以实现在室内环境下有效地寻找到目标,并显著地减少了搜索的路径长度,证明了该方法的有效性。     

Probabilistic state estimation of dynamic objects with a moving mobile robot

Mobile service robots are designed to operate in dynamic and populated environments. To plan their missions and to perform them successfully, mobile robots need to keep track of relevant changes in the environment. For example, office delivery or cleaning robots must be able to estimate the state of doors or the position of waste-baskets in order to deal with the dynamics of the environment. In this paper we present a probabilistic technique for estimating the state of dynamic objects in the environment of a mobile robot. Our method matches real sensor measurements against expected measurements obtained by a sensor simulation to efficiently and accurately identify the most likely state of each object even if the robot is in motion. The probabilistic approach allows us to incorporate the robot’s uncertainty in its position into the state estimation process. The method has been implemented and tested on a real robot. We present different examples illustrating the efficiency and robustness of our approach.     

Robust and Efficient Feature Tracking for Indoor Navigation

Robust feature tracking is a requirement for many computer vision tasks such as indoor robot navigation. However, indoor scenes are characterized by poorly localizable features. As a result, indoor feature tracking without artificial markers is challenging and remains an attractive problem. We propose to solve this problem by constraining the locations of a large number of nondistinctive features by several planar homographies which are strategically computed using distinctive features. We experimentally show the need for multiple homographies and propose an illumination-invariant local-optimization scheme for motion refinement. The use of a large number of nondistinctive features within the constraints imposed by planar homographies allows us to gain robustness. Also, the lesser computation cost in estimating these nondistinctive features helps to maintain the efficiency of the proposed method. Our local-optimization scheme produces subpixel accurate feature motion. As a result, we are able to achieve robust and accurate feature tracking.     

Flexible view recognition for indoor navigation based on Gabor filters and support vector machines

Real-world scenes are hard to segment into (relevant) objects and (irrelevant) background. In this paper, we argue for view-based vision, which does not use segmentation, and demonstrate a practical approach for recognizing textured objects and scenes in office environments.

A small set of Gabor filters is used to preprocess texture combinations from input images. The impulse responses of the filters are transformed into feature vectors that are fed to support vector machines. Pairwise feature comparisons decide the classification of views.

We validate the approach on a robot platform using three different types of target objects and indoor scenes: people, doorways, and written signs. The general-purpose system can run in real time, and that recognition accuracies of practical utility are obtained.     

Active Scene Understanding via Online Semantic Reconstruction

We propose a novel approach to robot‐operated active understanding of unknown indoor scenes, based on online RGBD reconstruction with semantic segmentation. In our method, the exploratory robot scanning is both driven by and targeting at the recognition and segmentation of semantic objects from the scene. Our algorithm is built on top of a volumetric depth fusion framework and performs real‐time voxel‐based semantic labeling over the online reconstructed volume. The robot is guided by an online estimated discrete viewing score field (VSF) parameterized over the 3D space of 2D location and azimuth rotation. VSF stores for each grid the score of the corresponding view, which measures how much it reduces the uncertainty (entropy) of both geometric reconstruction and semantic labeling. Based on VSF, we select the next best views (NBV) as the target for each time step. We then jointly optimize the traverse path and camera trajectory between two adjacent NBVs, through maximizing the integral viewing score (information gain) along path and trajectory. Through extensive evaluation, we show that our method achieves efficient and accurate online scene parsing during exploratory scanning.     

Motion strategies for exploration and map building under uncertainty with multiple heterogeneous robots

In this paper, we present a multi-robot exploration strategy for map building. We consider an indoor structured environment and a team of robots with different sensing and motion capabilities. We combine geometric and probabilistic reasoning to propose a solution to our problem. We formalize the proposed solution using stochastic dynamic programming (SDP) in states with imperfect information. Our modeling can be considered as a partially observable Markov decision process (POMDP), which is optimized using SDP. We apply the dynamic programming technique in a reduced search space that allows us to incrementally explore the environment. We propose realistic sensor models and provide a method to compute the probability of the next observation given the current state of the team of robots based on a Bayesian approach. We also propose a probabilistic motion model, which allows us to take into account errors (noise) on the velocities applied to each robot. This modeling also allows us to simulate imperfect robot motions, and to estimate the probability of reaching the next state given the current state. We have implemented all our algorithms and simulations results are presented.     

移动机器人基于三维激光测距的室内场景认知

研究了移动机器人在室内三维环境中的场景认知问题.室内场景框架具有结构化特性,而室 内多样化的物体则难以进行模型化表述. 本文利用区域扩张算法进行平面特征的提取,并根据平面属性及其相互间的空间关系,完成室 内场景框架的辨识.为了借鉴图像处理领域的物体识别方法, 本文提出一种基于Bearing Angle模型的激光测距数据表述方法,从而将三维点云数据转换为二维Bearing Angle图. 同一类物体中的个体形态具有多样性,同时观测视角也导致激光测距数据的显著差异.针对这些 问题,采用一种基于Gentleboost算法的有监督学习方法, 并利用物体碎片及其相对于物体中心的位置作为特征,从而完成室内场景中的物体认知. 利用室内场景框架辨识结果在Bearing Angle图中进行天棚、地面、墙壁、房门等区域的标记,并利用所产生的语义信息去除错误的认知结果,从而有助于提高识别率. 利用实际机器人平台所获得的实验结果验证了所提方法的有效性.     

Robot semantic mapping through human activity recognition: A wearable sensing and computing approach

Semantic information can help robots understand unknown environments better. In order to obtain semantic information efficiently and link it to a metric map, we present a new robot semantic mapping approach through human activity recognition in a human–robot coexisting environment. An intelligent mobile robot platform called ASCCbot creates a metric map while wearable motion sensors attached to the human body are used to recognize human activities. Combining pre-learned models of activity–furniture correlation and location–furniture correlation, the robot determines the probability distribution of the furniture types through a Bayesian framework and labels them on the metric map. Computer simulations and real experiments demonstrate that the proposed approach is able to create a semantic map of an indoor environment effectively.     

Extracting semantic indoor maps from occupancy grids

The primary challenge for any autonomous system operating in realistic, rather unconstrained scenarios is to manage the complexity and uncertainty of the real world. While it is unclear how exactly humans and other higher animals master these problems, it seems evident, that abstraction plays an important role. The use of abstract concepts allows us to define the system behavior on higher levels. In this paper we focus on the semantic mapping of indoor environments. We propose a method to extract an abstracted floor plan from typical grid maps using Bayesian reasoning. The result of this procedure is a probabilistic generative model of the environment defined over abstract concepts. It is well suited for higher-level reasoning and communication purposes. We demonstrate the effectiveness of the approach using real-world data.     

A generative framework for fast urban labeling using spatial and temporal context

This paper introduces a multi-level classification framework for the semantic annotation of urban maps as provided by a mobile robot. Environmental cues are considered for classification at different scales. The first stage considers local scene properties using a probabilistic bag-of-words classifier. The second stage incorporates contextual information across a given scene (spatial context) and across several consecutive scenes (temporal context) via a Markov Random Field (MRF). Our approach is driven by data from an onboard camera and 3D laser scanner and uses a combination of visual and geometric features. By framing the classification exercise probabilistically we take advantage of an information-theoretic bail-out policy when evaluating class-conditional likelihoods. This efficiency, combined with low order MRFs resulting from our two-stage approach, allows us to generate scene labels at speeds suitable for online deployment. We demonstrate the virtue of considering such spatial and temporal context during the classification task and analyze the performance of our technique on data gathered over almost 17 km of track through a city.     

Avoiding Non-Manhattan Obstacles Based on Projection of Spatial Corners in Indoor Environment

Monocular vision-based navigation is a considerable ability for a home mobile robot. However, due to diverse disturbances, helping robots avoid obstacles, especially non-Manhattan obstacles, remains a big challenge. In indoor environments, there are many spatial right-corners that are projected into two dimensional projections with special geometric configurations. These projections, which consist of three lines, might enable us to estimate their position and orientation in 3D scenes. In this paper, we present a method for home robots to avoid non-Manhattan obstacles in indoor environments from a monocular camera. The approach first detects non-Manhattan obstacles. Through analyzing geometric features and constraints, it is possible to estimate posture differences between orientation of the robot and non-Manhattan obstacles. Finally according to the convergence of posture differences, the robot can adjust its orientation to keep pace with the pose of detected non-Manhattan obstacles, making it possible avoid these obstacles by itself. Based on geometric inferences, the proposed approach requires no prior training or any knowledge of the camera’s internal parameters, making it practical for robots navigation. Furthermore, the method is robust to errors in calibration and image noise. We compared the errors from corners of estimated non-Manhattan obstacles against the ground truth. Furthermore, we evaluate the validity of convergence of differences between the robot orientation and the posture of non-Manhattan obstacles. The experimental results showed that our method is capable of avoiding non-Manhattan obstacles, meeting the requirements for indoor robot navigation.      

Vision for mobile robot navigation: a survey

Surveys the developments of the last 20 years in the area of vision for mobile robot navigation. Two major components of the paper deal with indoor navigation and outdoor navigation. For each component, we have further subdivided our treatment of the subject on the basis of structured and unstructured environments. For indoor robots in structured environments, we have dealt separately with the cases of geometrical and topological models of space. For unstructured environments, we have discussed the cases of navigation using optical flows, using methods from the appearance-based paradigm, and by recognition of specific objects in the environment     

3D model based tracking for omnidirectional vision: A new spherical approach

The current work addresses the problem of 3D model tracking in the context of monocular and stereo omnidirectional vision in order to estimate the camera pose. To this end, we track 3D objects modeled by line segments because the straight line feature is often used to model the environment. Indeed, we are interested in mobile robot navigation using omnidirectional vision in structured environments. In the case of omnidirectional vision, 3D straight lines are projected as conics in omnidirectional images. Under certain conditions, these conics may have singularities.In this paper, we present two contributions. We, first, propose a new spherical formulation of the pose estimation withdrawing singularities, using an object model composed of lines. The theoretical formulation and the validation on synthetic images thus show that the new formulation clearly outperforms the former image plane one. The second contribution is the extension of the spherical representation to the stereovision case. We consider in the paper a sensor which combines a camera and four mirrors. Results in various situations show the robustness to illumination changes and local mistracking. As a final result, the proposed new stereo spherical formulation allows us to localize online a robot indoor and outdoor whereas the classical formulation fails.     

Relational Model for Robotic Semantic Navigation in Indoor Environments

The emergence of service robots in our environment raises the need to find systems that help the robots in the task of managing the information from human environments. A semantic model of the environment provides the robot with a representation closer to the human perception, and it improves its human-robot communication system. In addition, a semantic model will improve the capabilities of the robot to carry out high level navigation tasks. This paper presents a semantic relational model that includes conceptual and physical representation of objects and places, utilities of the objects, and semantic relation among objects and places. This model allows the robot to manage the environment and to make queries about the environment in order to do plans for navigation tasks. In addition, this model has several advantages such as conceptual simplicity and flexibility of adaptation to different environments. To test the performance of the proposed semantic model, the output for the semantic inference system is associate to the geometric and topological information of objects and places in order to do the navigation tasks.     

Living in a partially structured environment: How to bypass the limitations of classical reinforcement techniques

In this paper, we propose an unsupervised neural network allowing a robot to learn sensory-motor associations with a delayed reward. The robot task is to learn the “meaning” of pictograms in order to “survive” in a maze. First, we introduce a new neural conditioning rule probabilistic conditioning rule (PCR) allowing us to test hypotheses (associations between visual categories and movements) during a given time span. Second, we describe a real maze experiment with our mobile robot. We propose a neural architecture overcoming the difficulty to build visual categories dynamically while associating them to movements. Third, we propose to use our algorithm on a simulation in order to test it exhaustively. We give the results for different kinds of mazes and we compare our system to an adapted version of the Q-learning algorithm. Finally, we conclude by showing the limitations of approaches that do not take into account the intrinsic complexity of a reasoning based on image recognition.     

Localization of a mobile robot based on an ultrasonic sensor using dynamic obstacles

In this article, we propose a localization scheme for a mobile robot based on the distance between the robot and moving objects. This method combines the distance data obtained from ultrasonic sensors in a mobile robot, and estimates the location of the mobile robot and the moving object. The movement of the object is detected by a combination of data and the object’s estimated position. Then, the mobile robot’s location is derived from the a priori known initial state. We use kinematic modeling that represents the movement of a robot and an object. A Kalman-filtering algorithm is used for addressing estimation error and measurement noise. Throughout the computer simulation experiments, the performance is verified. Finally, the results of experiments are presented and discussed. The proposed approach allows a mobile robot to seek its own position in a weakly structured environment. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007     

Simultaneous localization and map building for mobile robotnavigation

Inexpensive ultrasonic sensors, incremental encoders, and grid-based probabilistic modeling are used for improved robot navigation in indoor environments. For model-building, range data from ultrasonic sensors are constantly sampled and a map is built and updated immediately while the robot is travelling through the workspace. The local world model is based on the concept of an occupancy grid. The world model extracted from the range data is based on the geometric primitive of line segments. For the extraction of these features, methods such as the Hough transform and clustering are utilized. The perceived local world model along with dead-reckoning and ultrasonic sensor data are combined using an extended Kalman filter in a localization scheme to estimate the current position and orientation of the mobile robot, which is subsequently fed to the map-building algorithm. Implementation issues and experimental results with the Nomad 150 mobile robot in a real-world indoor environment (office space) are presented