Meaningful conversation with mobile robots

We describe an implementation integrating a complete spoken dialogue system with a mobile robot, which a human can direct to specific locations, ask for information about its status and supply information about its environment. The robot uses an internal map for navigation, and communicates its current orientation and accessible locations to the dialogue system using a topological map as interface. We focus on linguistic and inferential aspects of the human–robot communication process. The result is a novel approach using a principled semantic theory combined with techniques from automated deduction applied to a mobile robot platform. Due to the abstract level of the dialogue system, it is easily portable to other environments or applications.     

Building 3D semantic maps for mobile robots using RGB-D camera

The wide availability of affordable RGB-D sensors changes the landscape of indoor scene analysis. Years of research on simultaneous localization and mapping (SLAM) have made it possible to merge multiple RGB-D images into a single point cloud and provide a 3D model for a complete indoor scene. However, these reconstructed models only have geometry information, not including semantic knowledge. The advancements in robot autonomy and capabilities for carrying out more complex tasks in unstructured environments can be greatly enhanced by endowing environment models with semantic knowledge. Towards this goal, we propose a novel approach to generate 3D semantic maps for an indoor scene. Our approach creates a 3D reconstructed map from a RGB-D image sequence firstly, then we jointly infer the semantic object category and structural class for each point of the global map. 12 object categories (e.g. walls, tables, chairs) and 4 structural classes (ground, structure, furniture and props) are labeled in the global map. In this way, we can totally understand both the object and structure information. In order to get semantic information, we compute semantic segmentation for each RGB-D image and merge the labeling results by a Dense Conditional Random Field. Different from previous techniques, we use temporal information and higher-order cliques to enforce the label consistency for each image labeling result. Our experiments demonstrate that temporal information and higher-order cliques are significant for the semantic mapping procedure and can improve the precision of the semantic mapping results.     

基于视觉惯性里程计与语义信息的无人机SLAM方法研究

室内环境中存在丰富的语义信息,可以使机器人更好地理解环境,提高机器人位姿估计的准确性。虽然语义信息在机器人同时定位与地图构建(SLAM)领域得到了深入研究和广泛应用,但是在环境准确感知、语义特征提取和语义信息利用等方面还存在着很多困难。针对上述难点,提出了一种基于视觉惯性里程计算法与语义信息相结合的新方法,该方法通过视觉惯性里程计来估计机器人的状态,通过校正估计,构建从语义检测中提取的几何表面的稀疏语义地图;通过将检测到的语义对象的几何信息与先前映射的语义信息相关联来解决视觉惯性里程计和惯性测量单元的累积误差问题。在室内环境中对装备RGB-D深度视觉和激光雷达的无人机进行验证实验,结果表明,该方法比视觉惯性里程计算法取得了更好的结果。应用结合语义信息和视觉惯性里程计的SLAM算法表现出很好的鲁棒性和准确性,该方法能提高无人机导航精度,实现无人机智能自主导航。     

激光—相机系统语义栅格建图和路径规划

目的 SLAM（simultaneous localization and mapping）是移动机器人在未知环境进行探索、感知和导航的关键技术。激光SLAM测量精确，便于机器人导航和路径规划，但缺乏语义信息。而视觉SLAM的图像能提供丰富的语义信息，特征区分度更高，但其构建的地图不能直接用于路径规划和导航。为了实现移动机器人构建语义地图并在地图上进行路径规划，本文提出一种语义栅格建图方法。方法 建立可同步获取激光和语义数据的激光-相机系统，将采集的激光分割数据与目标检测算法获得的物体包围盒进行匹配，得到各物体对应的语义激光分割数据。将连续多帧语义激光分割数据同步融入占据栅格地图。对具有不同语义类别的栅格进行聚类，得到标注物体类别和轮廓的语义栅格地图。此外，针对语义栅格地图发布导航任务，利用路径搜索算法进行路径规划，并对其进行改进。结果 在实验室走廊和办公室分别进行了语义栅格建图的实验，并与原始栅格地图进行了比较。在语义栅格地图的基础上进行了路径规划，并采用了语义赋权算法对易移动物体的路径进行对比。结论 多种环境下的实验表明本文方法能获得与真实环境一致性较高、标注环境中物体类别和轮廓的语义栅格地图，且实验硬件结构简单、成本低、性能良好，适用于智能化机器人的导航和路径规划。     

基于QRcode技术的家庭半未知环境语义地图构建

根据家庭环境中机器人特有的服务任务和人机交互要求,结合大物品上粘贴的基于QR code技术的自相似人工物标,构建半未知环境的以房间为单位的语义地图。首先基于谱聚类算法构建具有房间分割功能的拓扑地图。再利用QR code中存储的物品信息,建立物品信息库和物品归属关联图,最终获得包含物品信息描述、房间功能描述及物品-房间归属关系的语义地图。该地图为家庭环境中物品的搜寻、管理和机器人服务提供完备的、拟人化的信息。仿真实验表明,基于语义地图机器人能理解人的语义命令,生成合理的服务路径,实现功能驱动的导航。     

Holography map for home robot: an object-oriented approach

The environment map plays an important role in robot service, so it should contain not only appearance information about the whole service environment, but also their profoundness. The key contribution of the paper is the presentation of a novel semantic map, namely, a holography map composed of robot, family persons, operable items, local environments, as well as locations and path sections for home service robot cognizing its surroundings and providing services. Inspired by the object-oriented approach, the holography map is divided into three hierarchies of item-room-home and in detail 13 classes of objects. The design and storage of the object-oriented holography map are described comprehensively, and construction of the map is introduced. The execution of robot service based on the object-oriented holography map is discussed briefly. Experiments on real service robot demonstrate that the object-oriented holography map is nearer to human thinking and applicable to indoor robot service tasks.     

Integrating fuzzy topological maps and fuzzy geometric maps for behavior‐based robots

In behavior‐based robots, planning is necessary to elaborate abstract plans that resolve complex navigational tasks. Usually maps of the environment are used to plan the robot motion and to resolve the navigational tasks. Two types of maps have been mainly used: metric and topological maps. Both types present advantages and weakness so that several integration approaches have been proposed in literature. However, in many approaches the integration is conducted to build a global representation model, and the planning and navigational techniques have not been fitted to profit from both kinds of information. We propose the integration of topological and metric models into a hybrid deliberative‐reactive architecture through a path planning algorithm based on A^* and a hierarchical map with two levels of abstraction. The hierarchical map contains the required information to take advantage of both kinds of modeling. On one hand, the topological model is based on a fuzzy perceptual model that allows the robot to classify the environment in distinguished places, and on the other hand, the metric map is built using regions of possibility with the shape of fuzzy segments, which are used later to build fuzzy grid‐based maps. The approach allows the robot to decide on the use of the most appropriate model to navigate the world depending on minimum‐cost and safety criteria. Experiments in simulation and in a real office‐like environment are shown for validating the proposed approach integrated into the navigational architecture. © 2002 Wiley Periodicals, Inc.     

基于场景图谱的室内移动机器人目标搜索

在移动机器人执行日常家庭任务时,首先需要其能够在环境中避开障碍物,自主地寻找到房间中的物体。针对移动机器人如何有效在室内环境下对目标物体进行搜索的问题,提出了一种基于场景图谱的室内移动机器人目标搜索,其框架结合了导航地图、语义地图和语义关系图谱。在导航地图的基础上建立了包含地标物体位置信息的语义地图,机器人可以轻松对地标物体进行寻找。对于动态的物体,机器人根据语义关系图中物体之间的并发关系,优先到关系强度比较高的地标物体旁寻找。通过物理实验展示了机器人在语义地图和语义关系图的帮助下可以实现在室内环境下有效地寻找到目标,并显著地减少了搜索的路径长度,证明了该方法的有效性。     

A semantically-informed multirobot system for exploration of relevant areas in search and rescue settings

Coordinated multirobot exploration involves autonomous discovering and mapping of the features of initially unknown environments by using multiple robots. Autonomously exploring mobile robots are usually driven, both in selecting locations to visit and in assigning them to robots, by knowledge of the already explored portions of the environment, often represented in a metric map. In the literature, some works addressed the use of semantic knowledge in exploration, which, embedded in a semantic map, associates spatial concepts (like ‘rooms’ and ‘corridors’) with metric entities, showing its effectiveness in improving the total area explored by robots. In this paper, we build on these results and propose a system that exploits semantic information to push robots to explore relevant areas of initially unknown environments, according to a priori information provided by human users. Discovery of relevant areas is significant in some search and rescue settings, in which human rescuers can instruct robots to search for victims in specific areas, for example in cubicles if a disaster happened in an office building during working hours. We propose to speed up the exploration of specific areas by using semantic information both to select locations to visit and to determine the number of robots to allocate to those locations. In this way, for example, more robots could be assigned to a candidate location in a corridor, so the attached rooms can be explored faster. We tested our semantic-based multirobot exploration system within a reliable robot simulator and we evaluated its performance in realistic search and rescue indoor settings with respect to state-of-the-art approaches.     

Absolute localization for a mobile robot using place cells

This paper describes a method for absolute localization and environment recognition for an autonomous, sonar-equipped robot. The addition of an auto-associative memory to previously developed non-neural map making software results in a system that is capable of recognizing its environment and its position within the environment using remembered features and room geometry. In the prior system the robot used sonar to construct a metric map of an environment, but the map information had to be reconstructed each time the robot returned to an environment. We evaluated the system with a task that requires memory of the position of a goal that is not directly detectable by sonar.     

Intelligent and Efficient Strategy for Unstructured Environment Sensing Using Mobile Robot Agents

In field environments it is not usually possible to provide robots in advance with valid geometric models of its task and environment. The robot or robot teams need to create these models by scanning the environment with its sensors. Here, an information-based iterative algorithm to plan the robot's visual exploration strategy is proposed to enable it to most efficiently build 3D models of its environment and task. The method assumes mobile robot (or vehicle) with vision sensors mounted at a manipulator end-effector (eye-in-hand system). This algorithm efficiently repositions the systems' sensing agents using an information theoretic approach and fuses sensory information using physical models to yield a geometrically consistent environment map. This is achieved by utilizing a metric derived from Shannon's information theory to determine optimal sensing poses for the agent(s) mapping a highly unstructured environment. This map is then distributed among the agents using an information-based relevant data reduction scheme. This method is particularly well suited to unstructured environments, where sensor uncertainty is significant. Issues addressed include model-based multiple sensor data fusion, and uncertainty and vehicle suspension motion compensation. Simulation results show the effectiveness of this algorithm.     

语义地图导航的智能化服务机器人研究

基于语义地图导航的智能化服务机器人系统,以搭载深度相机和二维激光雷达的差分轮智能车的形式呈现。系统以树莓派作为主控制器,利用OpenCR控制电机驱动和辅助驾驶传感器,通过ROS机器人操作系统将机器人控制器、智能信息处理模块和各类传感器融合成一个有机整体。所构建的机器人系统在服务过程中,通过感知周围环境、收集数据完成静态、动态物体的辨识,然后结合二维地图数据完成目标检测和语义地图的构建。为了提升系统智能化程度,配合深度推理神经计算棒,并结合Mobile Net网络架构下的SSD目标实时检测算法,提高了视觉感知的实时程度以及准确性。     

基于语义建图的室内机器人实时场景分类

针对室内环境下的机器人场景识别问题,重点研究了场景分类策略的自主性、实时性和准确性,提出了一种语义建图方法.映射深度信息构建二维栅格地图,自主规划场景识别路径;基于卷积网络建立场景分类模型,实时识别脱离特定训练;利用贝叶斯框架融合先验知识,修正了错误分类并完成语义建图.实验结果表明:机器人能够进行全局自主探索,实时判断场景类别,并创建满足要求的语义地图.同时,实际路径规划中,机器人可以根据语义信息改善导航行为,验证了方法的可行性.     

基于机器人服务任务导向的室内未知环境地图构建

针对室内移动机器人的服务任务,提出一种包括全局语义层、区域规划层、局部空间层的3 级室内环 境地图,使机器人不仅掌握面向导航的环境平面结构,而且还了解局部复杂空间的3 维栅格地图及描述房间和物品 功能及关联、归属关系的语义信息．首先,机器人依据视觉获得的深度信息及QR 码标签提供的物品操作功能信息 进行3 维栅格地图和物品功能图的构建,形成局域层空间描述．其次,基于贝叶斯估计算法构建区域层的2 维栅格 地图,同时形成无向加权图,构成区域规划层．最后,基于谱聚类算法构建具有房间分割功能的拓扑地图,结合物 品功能图,获得房间功能及房间之间关联关系、物品与房间归属关系等的语义信息,形成全局语义拓扑地图．仿真 试验表明,环境地图的3 级结构适用于室内机器人的服务任务,可以理解人的语义命令,生成合理的服务路径,并 确保机器人在复杂环境中安全运行．     

Concrete Defects Inspection and 3D Mapping Using CityFlyer Quadrotor Robot

The concrete aging problem has gained more attention in recent years as more bridges and tunnels in the United States lack proper maintenance. Though the Federal Highway Administration requires these public concrete structures to be inspected regularly, on-site manual inspection by human operators is time-consuming and labor-intensive. Conventional inspection approaches for concrete inspection, using RGB image-based thresholding methods, are not able to determine metric information as well as accurate location information for assessed defects for conditions. To address this challenge, we propose a deep neural network (DNN) based concrete inspection system using a quadrotor flying robot (referred to as CityFlyer) mounted with an RGB-D camera. The inspection system introduces several novel modules. Firstly, a visual-inertial fusion approach is introduced to perform camera and robot positioning and structure 3D metric reconstruction. The reconstructed map is used to retrieve the location and metric information of the defects. Secondly, we introduce a DNN model, namely AdaNet, to detect concrete spalling and cracking, with the capability of maintaining robustness under various distances between the camera and concrete surface. In order to train the model, we craft a new dataset, i.e., the concrete structure spalling and cracking (CSSC) dataset, which is released publicly to the research community. Finally, we introduce a 3D semantic mapping method using the annotated framework to reconstruct the concrete structure for visualization. We performed comparative studies and demonstrated that our AdaNet can achieve 8.41% higher detection accuracy than ResNets and VGGs. Moreover, we conducted five field tests, of which three are manual hand-held tests and two are drone-based field tests. These results indicate that our system is capable of performing metric field inspection, and can serve as an effective tool for civil engineers.      

Relational Model for Robotic Semantic Navigation in Indoor Environments

The emergence of service robots in our environment raises the need to find systems that help the robots in the task of managing the information from human environments. A semantic model of the environment provides the robot with a representation closer to the human perception, and it improves its human-robot communication system. In addition, a semantic model will improve the capabilities of the robot to carry out high level navigation tasks. This paper presents a semantic relational model that includes conceptual and physical representation of objects and places, utilities of the objects, and semantic relation among objects and places. This model allows the robot to manage the environment and to make queries about the environment in order to do plans for navigation tasks. In addition, this model has several advantages such as conceptual simplicity and flexibility of adaptation to different environments. To test the performance of the proposed semantic model, the output for the semantic inference system is associate to the geometric and topological information of objects and places in order to do the navigation tasks.     

Environment model adaptation for mobile robot exploration

In this work, we propose a methodology to adapt a mobile robot’s environment model during exploration as a means of decreasing the computational complexity associated with information metric evaluation and consequently increasing the speed at which the system is able to plan actions and travel through an unknown region given finite computational resources. Recent advances in exploration compute control actions by optimizing information-theoretic metrics on the robot’s map. These metrics are generally computationally expensive to evaluate, limiting the speed at which a robot is able to explore. To reduce computational cost, we propose keeping two representations of the environment: one full resolution representation for planning and collision checking, and another with a coarse resolution for rapidly evaluating the informativeness of planned actions. To generate the coarse representation, we employ the Principal of Relevant Information from rate distortion theory to compress a robot’s occupancy grid map. We then propose a method for selecting a coarse representation that sacrifices a minimal amount of information about expected future sensor measurements using the Information Bottleneck Method. We outline an adaptive strategy that changes the robot’s environment representation in response to its surroundings to maximize the computational efficiency of exploration. On computationally constrained systems, this reduction in complexity enables planning over longer predictive horizons, leading to faster navigation. We simulate and experimentally evaluate mutual information based exploration through cluttered indoor environments with exploration rates that adapt based on environment complexity leading to an order-of-magnitude increase in the maximum rate of exploration in contrast to non-adaptive techniques given the same finite computational resources.     

Smooth task switching through behaviour competition

Navigation in large-scale environments is composed of different local tasks. To achieve smooth switching between these tasks and thus a continuous control signal, usually a precise map of the environment and an exact pose estimate of the robot are needed. Both are hard to fulfil for experiments in real-world settings. We present a system that shows how one can relax the need for accurate metric models of the environment while at the same time achieving smooth task switching. To facilitate this scheme the dynamical systems approach is used, which incorporates behaviour coordination through competition in a dynamic framework. Feature detectors use sonar data to provide means for local navigation. This ability combined with a simple topological map constitutes a complete navigation system for large-scale office environments. Experiments showed that a Scout robot using this scheme is able to successfully navigate through our whole institute. Through the use of the dynamic behaviour coordination, switching between the navigational tasks occurs in a smooth manner leading to continuous control of the platform.