Real-time Automated Visual Inspection using Mobile Robots

We present a framework to perform novelty detection using visual input in which a mobile robot first learns a model of normality in its operating environment and later uses this to highlight uncommon visual features that may appear. This ability is of great importance for both robotic exploration and inspection tasks, because it enables the robot to allocate computational and attentional resources efficiently to those features which are novel. At the heart of the proposed system is the image encoding mechanism which uses local colour statistics from regions selected by a biologically-inspired model of visual attention. Our approach works in real-time with a wide, unrestricted field of view and is robust to image transformations. Experiments conducted in an engineered scenario demonstrate the efficiency and functionality of our method.      

基于单目视觉的同时定位与建图算法研究综述*

与传统基于激光传感器的同时定位与建图（SLAM）方法相比,基于图像视觉传感器SLAM方法能廉价的获得更多环境信息,帮助移动机器人提高智能性。不同于用带深度信息的3D传感器研究SLAM问题,单目视觉SLAM算法用二维图像序列在线构建三维环境地图并实现实时定位。针对多种单目视觉SLAM算法进行对比研究,分析了近10年来流行的单目视觉定位算法的主要思路及其分类,指出基于优化方法正取代滤波器方法成为主流方法。从初始化、位姿估计、地图创建、闭环检测等功能组件的角度分别总结了当下流行的各种单目视觉 SLAM 或Odometry系统的工作原理和关键技术,阐述它们的工作过程和性能特点。总结了近年最新单目视觉定位算法的设计思路,最后概括指出本领域的研究热点与发展趋势。     

Dynamic visual servo control of a 4-axis joint tool to track image trajectories during machining complex shapes

A large part of the new generation of computer numerical control systems has adopted an architecture based on robotic systems. This architecture improves the implementation of many manufacturing processes in terms of flexibility, efficiency, accuracy and velocity. This paper presents a 4-axis robot tool based on a joint structure whose primary use is to perform complex machining shapes in some non-contact processes. A new dynamic visual controller is proposed in order to control the 4-axis joint structure, where image information is used in the control loop to guide the robot tool in the machining task. In addition, this controller eliminates the chaotic joint behavior which appears during tracking of the quasi-repetitive trajectories required in machining processes. Moreover, this robot tool can be coupled to a manipulator robot in order to form a multi-robot platform for complex manufacturing tasks. Therefore, the robot tool could perform a machining task using a piece grasped from the workspace by a manipulator robot. This manipulator robot could be guided by using visual information given by the robot tool, thereby obtaining an intelligent multi-robot platform controlled by only one camera.     

Visual attention servo control for task-specific robotic applications

This paper proposes a visual attention servo control (VASC) method which uses the Gaussian mixture model (GMM) for task-specific applications of mobile robots. In particular, low dimensional bias feature template is obtained using GMM to get an efficient attention process. An image-based visual servo (IBVS) controller is used to search for a desired object in a scene through an attention system which forms a task-specific state representation of the environment. First, task definition and object representation in semantic memory (SM) are proposed, and bias feature template is obtained using GMM deduction for features from high dimension to low dimension. Second, the features intensity, color, size and orientation are extracted to build the feature set. Mean shift method is used to segment the visual scene into discrete proto-objects. Given a task-specific object, top-down bias attention is evaluated to generate the saliency map by combining with the bottom-up saliency-based attention. Third, a visual attention servo controller is developed to integrate the IBVS controller and the attention system for robotic cognitive control. A rule-based arbitrator is proposed to switch between the episodic memory (EM)-based controller and the IBVS controller depending on whether the robot obtains the desired attention point on the image. Finally, the proposed method is evaluated on task-specific object detection under different conditions and visual attention servo tasks. The obtained results validate the applicability and usefulness of the developed method for robotics.     

Recognition and path planning strategy for autonomous navigation in the elevator environment

This paper presents a robust and reliable method for a mobile robot to get on/off an elevator in a multistory building. Getting on/off the elevator requires the robot to perform two different tasks: a recognition task and a navigation task. First, we propose a recognition algorithm for the elevator buttons and status so that the robot reacts flexibly to the current elevator status. We first apply an adaptive threshold to the current image in order to get a binary image. Then we extract the candidates of the buttons and the floor number after preliminary filtering. Ambiguous candidates are rejected using an artificial neural network, and a matching method is applied to finally recognize the call buttons, destination floor buttons, moving direction and current location of the elevator. Second, we suggest a path planning algorithm to navigate into and out of the elevator without any collision. By constructing an occupancy grid map and computing a target function, we find the best position for the robot to get on the elevator. Then we plan an optimal path to the best position using a potential field method. Experiments were carried out in several simulated and real environments including empty, crowd and blocked scenarios. The approach presented here has been found to allow the robot to navigate in the elevator without collisions.     

Learning multiple experiences useful visual features for active maps localization in crowded environments

Crowded urban environments are composed of different types of dynamic and static elements. Learning and classification of features is a major task in solving the localization problem in such environments. This work presents a gradual learning methodology to learn the useful features using multiple experiences. The usefulness of an observed element is evaluated by a scoring mechanism which uses two scores – reliability and distinctiveness. The visual features thus learned are used to partition the visual map into smaller regions. The robot is efficiently localized in such a partitioned environment using two-level localization. The concept of active map (AM) is proposed here, which is a map that represents one partition of the environment in which there is a high probability of the robot existing. High-level localization is used to track the mode of the AMs using discrete Bayes filter. Low-level localization uses a bag-of-words model to retrieve images and accurately localize the robot. The pose of the robot is the one retrieved from the AM that has maximum a posteriori. Experiments have been conducted on a unique highly crowded data-set collected from Indian roads. The results support the proposed method due to speed and localization accuracy.     

Image feature command generation of contour following tasks for SCARA robots employing Image-Based Visual Servoing—A PH-spline approach

Recently, visual servoing has been widely employed in industrial robots and has become an invaluable asset to enhance the functionality of the robot. However, the issue of image feature command generation in a visual servoing task receives little attention. In a contour following task that adopts Image-Based Visual Servoing (IBVS), it is crucial to perform motion planning on the desired image trajectory. Without proper motion planning, not only may the discrepancy between the target position and the current position on the image plane not converge, but also the flexibility of exploiting visual servoing for applications such as contour following will be limited. In order to cope with the aforementioned problem, this paper proposes a PH-spline based motion planning approach for systems that adopt IBVS. In particular, the exterior contour of an object is represented by a PH quantic spline. With proper acceleration/deceleration motion planning, a PH quantic spline interpolator is constructed to generate desired image feature commands so that IBVS can be applied to handle contour following problems of an object without a known geometric model. Furthermore, this paper also develops a depth estimation algorithm for the eye-to-hand camera structure, providing a convenient way to estimate the depth value that is essential in computing image Jacobian. Experimental results of several contour following tasks verify the effectiveness of the proposed approach.     

基于遗传算法的变电站巡检机器人任务路径规划方法研究

近年来,随着变电站巡检机器人在变电站中的广泛使用,巡检机器人路径规划问题越来越成为亟待解决的问题。巡检机器人在已知的拓扑地图中标记了待执行巡检任务的停靠点,不同任务需要从初始点出发经过不同的一系列停靠点再返回初始点,如何规划路径是机器人面临的问题。首先分析了路径规划面临的问题,然后通过分析拓扑地图的特征,对地图进行等价简化,再对问题进行建模使用遗传算法求解巡检任务路径规划的近似最优解。通过仿真实验证明,提出的基于遗传算法的路径规划方法是可行有效的,为变电站巡检机器人任务路径规划提供了一种有效方法。     

基于场景理解与改进型BUG算法的移动机器人避障

针对现有移动机器人在视觉避障上存在的局限,将深度学习算法和路径规划技术相结合,提出了一种基于深层卷积神经网络和改进Bug算法的机器人避障方法;该方法采用多任务深度卷积神经网络提取道路图像特征,实现图像分类和语义分割任务;其次,基于语义分割结果构建栅格地图,并将图像分类结果与改进的Bug算法相结合,搜索出最优避障路径;同时,为降低冗余计算,设计了特征对比结构来对避免对重复计算的特征信息,保障机器人在实际应用中实时性;通过实验结果表明,所提方法有效的平衡了多视觉任务的精度与效率,并能准确规划出安全的避障路径,辅助机器人完成导航避障。     

采用自然语言的移动机器人任务编程

本文在机器人具备基本运动技能的基础上［1］，采用基于指令教导的学习方法．通 过自然语言教会机器人完成抽象化任务，并以程序体方式保存所学知识，也即通过自然语言 对话自动生成程序流．通过让机器人完成导航等任务，验证所提自然语言编程方法的可行性 ．     

Design and Implementation of a Novel Outdoor Road-Cleaning Robot

In order to clean unknown outdoor environments, we devised a novel outdoor cleaning robot using mainly on-board vision-based auto-navigation in this paper. The track-driven and cleaning mechanisms of the robot are designed for cleaning tasks in outdoor rough terrain. A single image sensing module is exploited for clean-region detection and three ultrasonic ranging modules are used for obstacle avoidance. The cleaning task is performed autonomously after the boustrophedon path planning is completed in the grid-cell map. The map is obtained from transforming the two-dimensional images captured by the image sensor. The robot also self-localizes based on the deduced boundary lines at the end of each cleaning stage to continue its unfinished cleaning work. The experimental results prove that our outdoor cleaning robot performs well in general outdoor environments.     

Automatic singularity avoidance using joint variations in robottask modification

The research community has addressed the need to improve the flexibility of robot systems. Two particular concepts that have resulted from this research are off-line programming and modular tooling. These concepts are directed at allowing the robot system to be used to perform a variety of tasks with minimal setup time and to allow easy replication of an application. Both of these concepts require that the robot system have the ability to measure the position and orientation of features in the workspace. These measurements can then be used to perform coordinate transformations on each of the task data points. These modified task data points then, theoretically, facilitate the performance of the task by the robot system without human intervention     

Automatic robotic tasks in unstructured environments using an image path tracker

This paper describes a new method to perform automatic tasks with a robot in an unstructured environment. A task to replace a blown light bulb in a streetlamp is described to show that this method works properly. In order to perform this task correctly, the robot is positioned by tracking secure previously defined paths. The robot, using an eye-in-hand configuration on a visual servoing scheme and a force sensor, is able to interact with its environment due to the fact that the path tracking is performed with time-independent behaviour. The desired path is expressed in the image space. However, the proposed method obtains a correct tracking not only in the image, but also in the 3D space. This method solves the problems of the previously proposed time-independent tracking systems based on visual servoing, such as the specification of the desired tracking velocity, less oscillating behaviour and a correct tracking in the 3D space when high velocities are used. The experiments shown in this paper demonstrate the necessity of time-independent behaviour in tracking and the correct performance of the system.     

A screening tool for occupations requiring a high level of attentional performance

Vigilance tasks with a high level of attentional demand are becoming more frequent in daily work. High levels of attentional demand are required in critical systems, such as air traffic control, crowd monitoring, visual quality inspection, or sports.In a previous study, we introduced an ecologically valid test for screening visual attention in the general working population. The test requires participants to complete a visual search task at a moderate level of difficulty. In the present work, we increased the test's level of difficulty to enable accurate screening of high levels of attentional skills. To establish reference data, attentional skills were recorded in 60 participants, ranging in age from 19 to 40, performing the test at both, moderate and high levels of difficulty.Increasing the difficulty level of the test resulted in a lower test performance score. Furthermore, an increased level of difficulty reduced the ceiling effects caused by highly-skilled participants. In the present report, we provide normative performance score data and evidence for the reliability of the developed test. Adapting the level of difficulty enables our attention test to effectively screen for attention in occupations requiring moderate or elevated attentional skills.Relevance to industryDemands for visual attention are increasing in a variety of industries. Therefore, screening for attentional performance is becoming more important in accident prevention and quality of work. A previously-reported ecologic test for screening visual attention at a moderate level of attention was adapted to enable screening at an elevated level of attentional demands.     

Reduction of state space in reinforcement learning by sensor selection

Much research has been conducted on the application of reinforcement learning to robots. Learning time is a matter of concern in reinforcement learning. In reinforcement learning, information from sensors is projected on to a state space. A robot learns the correspondence between each state and action in state space and determines the best correspondence. When the state space is expanded according to the number of sensors, the number of correspondences learnt by the robot is increased. Therefore, learning the best correspondence becomes time consuming. In this study, we focus on the importance of sensors for a robot to perform a particular task. The sensors that are applicable to a task differ for different tasks. A robot does not need to use all installed sensors to perform a task. The state space should consist of only those sensors that are essential to a task. Using such a state space consisting of only important sensors, a robot can learn correspondences faster than in the case of a state space consisting of all installed sensors. Therefore, in this paper, we propose a relatively fast learning system in which a robot can autonomously select those sensors that are essential to a task and a state space for only such important sensors is constructed. We define the measure of importance of a sensor for a task. The measure is the coefficient of correlation between the value of each sensor and reward in reinforcement learning. A robot determines the importance of sensors based on this correlation. Consequently, the state space is reduced based on the importance of sensors. Thus, the robot can efficiently learn correspondences owing to the reduced state space. We confirm the effectiveness of our proposed system through a simulation.     

Visual processing and classification of items on a moving conveyor: a selective perception approach

Many industrial applications require some sort of automated visual processing and classification of items placed on a moving conveyor. In this paper, we present a selective perception based approach to visual processing. The novelty of this approach is that instead of processing the whole image, only areas that are deemed ‘‘interesting’’ and hence calling for attention are analyzed. The attentional sequences thus constructed can then be used for a variety of tasks including shape determination. Since only a small portion of the whole image is processed, visual processing can be real-time and flexible without requiring special hardware. Two different applications based on this approach are described. In a defective item detection task, we explain in detail how attentional sequences can be used. As a second application, the approach has been implemented in an automated remote controller sorter in a TV manufacturing plant—thus confirming its practical applicability.     

满足复杂要求的机器人最优巡回控制系统设计与实现

本文结合线性时序逻辑理论与模糊控制方法,设计并实现了一种满足复杂任务需求的移动机器人巡回控制系统,它既能够针对复杂时序任务进行路径规划,又能够对机器人进行模糊控制实现路径跟踪.首先,基于线性时序逻辑理论,确定能够满足复杂巡回任务需求的全局最优路径.接着,根据所获得的最优路径,采用模糊控制方法设计轨迹跟踪控制器,使其通过实时位姿反馈对机器人进行路径跟踪控制.仿真结果验证了移动机器人巡回控制系统的有效性.最后,基于E-Puck移动机器人构建了能够满足复杂任务需求的移动机器人巡回控制实验系统.基于所提出的最优巡回路径规划算法和模糊控制器设计方法,通过图像处理、数据通信、算法加载等软件模块的实现完成了满足复杂任务需求的移动机器人巡回控制.     

Path planning using a Multiclass Support Vector Machine

In this paper, a new path planning algorithm for unstructured environments based on a Multiclass Support Vector Machine (MSVM) is presented. Our method uses as its input an aerial image or an unfiltered auto-generated map of the area in which the robot will be moving. Given this, the algorithm is able to generate a graph showing all of the safe paths that a robot can follow. To do so, our algorithm takes advantage of the training stage of a MSVM, making it possible to obtain the set of paths that maximize the distance to the obstacles while minimizing the effect of measurement errors, yielding paths even when the input data are not sufficiently clear. The method also ensures that it is able to find a path, if it exists, and it is fully adaptable to map changes over time. The functionality of these features was assessed using tests, divided into simulated results and real-world tests. For the latter, four different scenarios were evaluated involving 500 tests each. From these tests, we concluded that the method presented is able to perform the tasks for which it was designed.     

Visual servo walking control for humanoids with finite-time convergence and smooth robot velocities

In this paper, we address the problem of humanoid locomotion guided from information of a monocular camera. The goal of the robot is to reach a desired location defined in terms of a target image, i.e., a positioning task. The proposed approach allows us to introduce a desired time to complete the positioning task, which is advantageous in contrast to the classical exponential convergence. In particular, finite-time convergence is achieved while generating smooth robot velocities and considering the omnidirectional waking capability of the robot. In addition, we propose a hierarchical task-based control scheme, which can simultaneously handle the visual positioning and the obstacle avoidance tasks without affecting the desired time of convergence. The controller is able to activate or inactivate the obstacle avoidance task without generating discontinuous velocity references while the humanoid is walking. Stability of the closed loop for the two task-based control is demonstrated theoretically even during the transitions between the tasks. The proposed approach is generic in the sense that different visual control schemes are supported. We evaluate a homography-based visual servoing for position-based and image-based modalities, as well as for eye-in-hand and eye-to-hand configurations. The experimental evaluation is performed with the humanoid robot NAO.