Towards autonomous robotic servicing: Using an integrated hand-arm-eye system for manipulating unknown objects

Executing complex robotic tasks including dexterous grasping and manipulation requires a combination of dexterous robots, intelligent sensors and adequate object information processing. In this paper, vision has been integrated into a highly redundant robotic system consisting of a tiltable camera and a three-fingered dexterous gripper both mounted on a puma-type robot arm. In order to condense the image data of the robot working space acquired from the mobile camera, contour image processing is used for offline grasp and motion planning as well as for online supervision of manipulation tasks. The performance of the desired robot and object motions is controlled by a visual feedback system coordinating motions of hand, arm and eye according to the specific requirements of the respective situation. Experiences and results based on several experiments in the field of service robotics show the possibilities and limits of integrating vision and tactile sensors into a dexterous hand-arm-eye system being able to assist humans in industrial or servicing environments.     

结合机器视觉的采摘机械手的定位仿真研究

对水果采摘机械手空间定位机理进行了研究,分析了双目立体视觉系统的定位误差并建立视觉误差补偿机制,利用虚拟机械手开发软件和CCD视觉硬件构建了仿真系统,通过双目立体视觉获取空间位置数据映射到虚拟环境下引导机械手进行模拟采摘。该系统利用多领域知识融合实现了采摘机构与视觉关联精确定位的仿真,能有效地指导实际作业环境中采摘机械手精确定位的优化设计。     

Steps in the development of a robotic scrub nurse

This paper describes how a robotic scrub nurse has been developed to assist human scrub nurses (also called delivery scrub nurses) during surgical interventions. The developed system in this work (nicknamed Quirubot) is equipped with a speech recognition module to recognize the requested surgical instrument; Quirubot locates this element on a storage tray using computer vision and pattern recognition and picks the element from the tray placing it on an interchange tray where the human scrub nurse can finally use this tool. Moreover, to implement the manipulation of surgical instruments, an electromagnetic gripper has been adapted to pick and place them. Fanuc LR Mate 200iB robot has been electrically modified to add a new connector devoted to control the electromagnetic gripper. Nowadays Quirubot can identify up to 27 surgical instruments and more than 82 spoken instructions; the advantages of the system have been demonstrated using a group of experienced and intermediate skilled scrub nurses in a open abdominal aortic aneurysm surgery simulation (experiments performed in a robotics lab and an operating room). After this study, the use of a robotic scrub nurse is justified in an nBio operating room to improve the system up to obtaining a more robust version to be used at the Hospital Universitario de Sant Joan d’Alacant.     

A field‐tested robotic harvesting system for iceberg lettuce

Agriculture provides an unique opportunity for the development of robotic systems; robots must be developed which can operate in harsh conditions and in highly uncertain and unknown environments. One particular challenge is performing manipulation for autonomous robotic harvesting. This paper describes recent and current work to automate the harvesting of iceberg lettuce. Unlike many other produce, iceberg is challenging to harvest as the crop is easily damaged by handling and is very hard to detect visually. A platform called Vegebot has been developed to enable the iterative development and field testing of the solution, which comprises of a vision system, custom end effector and software. To address the harvesting challenges posed by iceberg lettuce a bespoke vision and learning system has been developed which uses two integrated convolutional neural networks to achieve classification and localization. A custom end effector has been developed to allow damage free harvesting. To allow this end effector to achieve repeatable and consistent harvesting, a control method using force feedback allows detection of the ground. The system has been tested in the field, with experimental evidence gained which demonstrates the success of the vision system to localize and classify the lettuce, and the full integrated system to harvest lettuce. This study demonstrates how existing state‐of‐the art vision approaches can be applied to agricultural robotics, and mechanical systems can be developed which leverage the environmental constraints imposed in such environments.     

Modeling, analysis, and performance evaluation of a robotic grippersystem for limp material handling

This paper presents an analytical model of a flat surfaced robotic gripper designed to automate the process of reliable, rapid and distortion-free limp material handling. The designed gripper prototype is integrated with an industrial robot manipulator. The gripper geometry and its grasp stability are justified. Performance of the overall system is experimentally tested, based on a set of industry dictated operational constraints. It is found that the gripper system has high reliability, grasp stability, and that it is capable of rapid rates of manipulation.     

Development,integration, and field evaluation of an autonomous citrus-harvesting robot

Citrus harvesting is a labor-intensive and time-intensive task. As the global population continues to age, labor costs are increasing dramatically. Therefore, the citrus-harvesting robot has attracted considerable attention from the business and academic communities. However, robotic harvesting in unstructured and natural citrus orchards remains a challenge. This study aims to address some challenges faced in commercializing citrus-harvesting robots. We present a fully integrated, autonomous, and innovative solution for citrus-harvesting robots to overcome the harvesting difficulties derived from the natural growth characteristics of citrus. This solution uses a fused simultaneous localization and mapping algorithm based on multiple sensors to perform high-precision localization and navigation for the robot in the field orchard. Besides, a novel visual method for estimating fruit poses is proposed to cope with the randomization of citrus growth orientations. Further, a new end-effector is designed to improve the success and conformity rate of citrus stem cutting. Finally, a fully autonomous harvesting robot system has been developed and integrated. Field evaluations showed that the robot could harvest citrus continuously with an overall success rate of 87.2% and an average picking time of 10.9 s/fruit. These efforts provide a solid foundation for the future commercialization of citrus-harvesting robots.     

An open-source multi-DOF articulated robotic educational platform for autonomous object manipulation

This research presents an autonomous robotic framework for academic, vocational and training purpose. The platform is centred on a 6 Degree Of Freedom (DOF) serial robotic arm. The kinematic and dynamic models of the robot have been derived to facilitate controller design. An on-board camera to scan the arm workspace permits autonomous applications development. The sensory system consists of position feedback from each joint of the robot and a force sensor mounted at the arm gripper. External devices can be interfaced with the platform through digital and analog I/O ports of the robot controller. To enhance the learning outcome for beginners, higher level commands have been provided. Advanced users can tailor the platform by exploiting the open-source custom-developed hardware and software architectures. The efficacy of the proposed platform has been demonstrated by implementing two experiments; autonomous sorting of objects and controller design. The proposed platform finds its potential to teach technical courses (like Robotics, Control, Electronics, Image-processing and Computer vision) and to implement and validate advanced algorithms for object manipulation and grasping, trajectory generation, path planning, etc. It can also be employed in an industrial environment to test various strategies prior to their execution on actual manipulators.     

Experimental validation and field performance metrics of a hybrid mobile robot mechanism

This paper presents the experimental validation and field testing of a novel hybrid mobile robot (HMR) system using a complete physical prototype. The mobile robot system consists of a hybrid mechanism whereby the locomotion platform and manipulator arm are designed as one entity to support both locomotion and manipulation symbiotically and interchangeably. The mechanical design is briefly described along with the related control hardware architecture based on an embedded onboard wireless communication network between the robot's subsystems, including distributed onboard power using Li‐ion batteries. The paper focuses on demonstrating through extensive experimental results the qualitative and quantitative field performance improvements of the mechanical design and how it significantly enhances mobile robot functionality in terms of the new operative locomotion and manipulation capabilities that it provides. In terms of traversing challenging obstacles, the robot was able to surmount cylindrical obstacles up to 0.6‐m diameter; cross ditches with at least 0.635‐m width; climb and descend step obstacles up to 0.7‐m height; and climb and descend stairs of different materials (wood, metal, concrete, plastic plaster, etc.), different stair riser and run sizes, and inclinations up to 60 deg. The robot also demonstrated the ability to manipulate objects up to 61 kg before and after flipping over, including pushing capacity of up to 61 kg when lifting objects from underneath. The above‐mentioned functions are critical in various challenging applications, such as search and rescue missions, military and police operations, and hazardous site inspections. © 2010 Wiley Periodicals, Inc.     

一种基于随动视觉的排爆机器人自主抓取系统

在传统基于固定视觉的排爆机器人抓取系统中,相机视觉易被遮挡且不能保证拍摄清晰度。基于随动视觉技术,提出一种将深度相机置于机械手末端并随机械手运动的排爆机器人自主抓取系统。利用深度相机计算目标物体的三维坐标,采用坐标转换方法将目标物体的位置坐标信息实时转换至机器人全局坐标系,并研究相机坐标系、机器人全局坐标系与末端执行器手爪工具坐标系三者的动态映射关系,实现排爆机器人的自主抓取。实验结果表明,与传统固定视觉方法相比,随动视觉方法可在误差2cm内,使得机器人机械手爪准确到达目标物体所在位置,且当机器人距离目标物体100cm~150cm时,抓取效果最佳。     

Development of a sweet pepper harvesting robot

This paper presents the development, testing and validation of SWEEPER, a robot for harvesting sweet pepper fruit in greenhouses. The robotic system includes a six degrees of freedom industrial arm equipped with a specially designed end effector, RGB‐D camera, high‐end computer with graphics processing unit, programmable logic controllers, other electronic equipment, and a small container to store harvested fruit. All is mounted on a cart that autonomously drives on pipe rails and concrete floor in the end‐user environment. The overall operation of the harvesting robot is described along with details of the algorithms for fruit detection and localization, grasp pose estimation, and motion control. The main contributions of this paper are the integrated system design and its validation and extensive field testing in a commercial greenhouse for different varieties and growing conditions. A total of 262 fruits were involved in a 4‐week long testing period. The average cycle time to harvest a fruit was 24 s. Logistics took approximately 50% of this time (7.8 s for discharge of fruit and 4.7 s for platform movements). Laboratory experiments have proven that the cycle time can be reduced to 15 s by running the robot manipulator at a higher speed. The harvest success rates were 61% for the best fit crop conditions and 18% in current crop conditions. This reveals the importance of finding the best fit crop conditions and crop varieties for successful robotic harvesting. The SWEEPER robot is the first sweet pepper harvesting robot to demonstrate this kind of performance in a commercial greenhouse.     

采摘机械臂自适应输入整形控制研究

采摘机械臂在夹住柔性果茎后运输果实时,执行器末端的加减速运动使得果实在移动过程中产生摆动,易引发掉落,进而导致采摘失败.本文以单个西红柿作为负载,将果茎近似为柔性连杆.由于每一个果实的质量是不同的,因此,针对机械臂抓取可变柔性负载移动过程中的振动抑制问题,提出了自适应输入整形控制方法.当系统模型由于负载的不确定性发生变化后,传统的输入整形算法无法抑制柔性连杆移动过程中产生的振动.因此采用自适应输入整形算法,实时计算脉冲的幅值和时间.构造二次性能指标函数,通过对机械臂移动的加速度和负载的摆角实时数据进行迭代运算,达到零残余振动的目的.仿真实验结果表明,在变负载情况下,自适应输入整形算法有良好的末端振动抑制能力,获得满意的控制效果.     

Artificial neural networks for coordination and control: The portability of experiential representations

It is time to locate connectionist representation theory in the new wave of robotics research. The utility of representations developed in artificial neural networks (ANNs) during learning has been demonstrated in cognitive science research since the 1980s. The research reported here puts learned representations to work in a decentered control task, the disembodied arm problem, in which a mobile robot operates an arm fixed to a table to pick up objects. There is no physical linkage between the arm and the robot and so the robot's point of view must be decentered. This is done by developing a modular Artificial Neural Net system in three stages: (i) a classifier net is trained with laser scan data to output transformationally invariant position classes; (ii) an arm net is trained for picking up objects; (iii) an inter net is trained to communicate and coordinate the sensing and acting. The completed system is shown to create new nonsymbolic transformationally invariant representations in order to perform the effective generalization of decentered viewpoints.     

基于主动式全景视觉的移动机器人障碍物检测

针对现有的移动机器人视觉系统计算资源消耗大、实时性能欠佳、检测范围受限等问题,提出一种基于主动式全景视觉传感器(AODVS)的移动机器人障碍物检测方法。首先,将单视点的全方位视觉传感器(ODVS)和由配置在1个平面上的4个红色线激光组合而成的面激光发生器进行集成,通过主动全景视觉对移动机器人周边障碍物进行检测;其次,移动机器人中的全景智能感知模块根据面激光发生器投射到周边障碍物上的激光信息,通过视觉处理方法解析出移动机器人周边障碍物的距离和方位等信息;最后,基于上述信息采用一种全方位避障策略,实现移动机器人的快速避障。实验结果表明,基于AODVS的障碍物检测方法能在实现快速高效避障的同时,降低对移动机器人的计算资源的要求。     

基于OpenCV的嵌入式自动捡乒乓球系统设计

为了解决在乒乓球比赛和训练中人工手动捡球的问题,从机器视觉的角度出发,综合嵌入式、图像处理和测量控制等技术,基于第三方库OpenCV设计了一种自动捡乒乓球系统。系统是以Contex-A8平台为控制核心,首先通过对云台和USB摄像头的控制获取视频信号;然后通过霍夫圆变换和SVM分类器算法识别出视频中的乒乓球,并根据质心计算出其与系统的相对位置;最后将视频信号中抽象出的位置信息传递给机械运动控制终端,控制小车和机械手将乒乓球捡起,并送到指定位置。实验结果表明本系统能够正确的检测和识别乒乓球,并对其进行定位捡取,效果良好,提高了拾取乒乓球的效率,减轻工作人员的负担。     

Neural Network-Based Vision for Precise Control of a Walking Robot

This article describes a connectionist vision system for the precise control of a robot designed to walk on the exterior of the space station. The network learns to use video camera input to determine the displacement of the robot's gripper relative to a hole in which the gripper must be inserted. Once trained, the network's output is used to control the robot, with a resulting factor of five fewer missed gripper insertions than occur when the robot walks without sensor feedback. The neural network visual feedback techniques described could also be applied in domains such as manufacturing, where precise robot positioning is required in an uncertain environment.     

Improvements to and large‐scale evaluation of a robotic kiwifruit harvester

The growing popularity of kiwifruit orchards in New Zealand is increasing the already high demand for seasonal labourers. A novel robotic kiwifruit harvester has been designed and built to help meet this demand [H. A. Williams et al. Biosystems Eng. 181 (2019), pp. 140–156]. Early evaluations of the platform have shown good results with the system capable of harvesting 51.0% of 1,456 kiwifruit in four bays with an average cycle‐time of 5.5 s/fruit. However, the harvester's high cycle‐time, and high fruit loss rate at 23.4%, prevent it from being commercially viable. This paper presents two new developments for the harvesting system, a new vision system and two new gripper variations designed to overcome the harvester's previous limitations. Furthermore, we have designed and conducted the largest real‐world evaluation of a robotic fruit harvesting system published to date with over 12,000 kiwifruit involved. The results of this trial demonstrated that our kiwifruit harvester is one of the most effective selective fruit harvesters in the world capable of successfully harvesting 86.0% of reachable fruit, and 55.8% of all kiwifruit with a cycle‐time of 2.78 s/fruit.     

Robotic harvesting of the occluded fruits with a precise shape and position reconstruction approach

Occlusion is one of the key factors affecting the success rate of vision-based fruit-picking robots. It is important to accurately locate and grasp the occluded fruit in field applications, However, there is yet no universal and effective solution. In this paper, a high-precision estimation method of spatial geometric features of occluded targets based on deep learning and multisource images is presented, enabling the selective harvest robot to envision the whole target fruit as if its occlusions do not exist. First, RGB, depth and infrared images are acquired. And pixel-level matched RGB-D-I fusion images are obtained by image registration. Second, aiming at the problem of detecting the occluded tomatoes in the greenhouse, an extended Mask-RCNN network is designed to extract the target tomato. The target segmentation accuracy is improved by 7.6%. Then, for partially occluded tomatoes, a shape and position restoration method is used to recover the obscured tomato. This algorithm can extract tomato radius and centroid coordinates directly from the restored depth image. The mean Intersection over Union is 0.895, and the centroid position error is 0.62 mm for the occluded rate under 25% and the illuminance between 1 and 12 KLux. And hereby a dual-arm robotic harvesting system is improved to achieve a picking time of 11 s per fruit, an average gripping accuracy of 8.21 mm, and an average picking success rate of 73.04%. The proposed approach realizes a high-fidelity geometrics reconstruction instead of mere image style restoration, which endows the robot with the ability to see through obstacles in the field scenes and improves its operational success rate in its result.