Development of a sweet pepper harvesting robot

This paper presents the development, testing and validation of SWEEPER, a robot for harvesting sweet pepper fruit in greenhouses. The robotic system includes a six degrees of freedom industrial arm equipped with a specially designed end effector, RGB‐D camera, high‐end computer with graphics processing unit, programmable logic controllers, other electronic equipment, and a small container to store harvested fruit. All is mounted on a cart that autonomously drives on pipe rails and concrete floor in the end‐user environment. The overall operation of the harvesting robot is described along with details of the algorithms for fruit detection and localization, grasp pose estimation, and motion control. The main contributions of this paper are the integrated system design and its validation and extensive field testing in a commercial greenhouse for different varieties and growing conditions. A total of 262 fruits were involved in a 4‐week long testing period. The average cycle time to harvest a fruit was 24 s. Logistics took approximately 50% of this time (7.8 s for discharge of fruit and 4.7 s for platform movements). Laboratory experiments have proven that the cycle time can be reduced to 15 s by running the robot manipulator at a higher speed. The harvest success rates were 61% for the best fit crop conditions and 18% in current crop conditions. This reveals the importance of finding the best fit crop conditions and crop varieties for successful robotic harvesting. The SWEEPER robot is the first sweet pepper harvesting robot to demonstrate this kind of performance in a commercial greenhouse.     

Localization of spherical fruits for robotic harvesting

The orange picking robot (OPR) is a project for developing a robot that is able to harvest oranges automatically. One of the key tasks in this robotic application is to identify the fruit and to measure its location in three dimensions. This should be performed using image processing techniques which must be sufficiently robust to cope with variations in lighting conditions and a changing environment. This paper describes the image processing system developed so far to guide automatic harvesting of oranges, which here has been integrated in the first complete full-scale prototype OPR. Received: 16 April 2000 / Accepted: 19 December 2000     

Autonomous robotic capture of non-cooperative target using visual servoing and motion predictive control

This paper presents a framework for autonomous capture operation of a non-cooperative mobile target in a 3-dimensional workspace using a robotic manipulator with visual servoing. The visual servoing with an eye-in-hand configuration is based on motion predictive control using Kalman filter for the on-line state and parameter estimation of the target. A transitional decision making process is developed to guide the robotic manipulator between the different phases of the capture operation by employing a custom metric that translates visual misalignments between the end-effector and the target into a guidance measurement. These phases include the target acquisition and approach stage and the alignment and capture phase. Experiments have been carried out on a custom designed and built robotic manipulator with 6 degrees of freedom. The objective is to evaluate the performance of the proposed motion predictive control scheme for the autonomous capturing task and to demonstrate the robustness of the proposed control scheme in the presence of noise and unexpected disturbances in vision system, sensory-motor coordination and constraints for the execution in real environments. Experimental results of the visual servoing control scheme integrated with the motion predictive Kalman filter indicate the feasibility and applicability of the proposed control scheme. It shows that when the target motion is properly predicted, the tracking and capture performance has been improved significantly.     

Incremental visual servo control of robotic manipulator for autonomous capture of non-cooperative target

This paper develops a new autonomous incremental visual servo control law for the robotic manipulator to capture a non-cooperative target, where the control input is the incremental joint angle to avoid the multiple solutions in the existing inverse kinematics. The position and motion of the non-cooperative target are estimated by an eye-to-hand vision system in real time by integrated photogrammetry and extended Kalman filter. The estimated position and motion of the target are fed into the newly developed position-based visual servo control law to drive the manipulator incrementally towards the dynamically predicted interception point between trajectories of the end effector and the target. To validate the proposed approach, a hardware-in-the-loop simulation has been conducted where the position and motion of the target is estimated by a real eye-to-hand camera and fed into the simulation of the robotic manipulator. The simulation results show the proposed incremental visual servo control law is stable and able to avoid the multiple solutions in the total inverse kinematics.     

Mechanical design and control of inflatable robotic arms for high positioning accuracy

In recent years, inflatable robotic arms have been developed so that physical contact with humans and working environments could be performed safety. In industry, there is a growing demand for safe industrial robots to collaborate with people and work in various environments. In general, however, the positioning accuracy of inflatable robotic arms has not been discussed. This paper proposes an inflatable link structure and a non-inflatable joint structure that could realize a high positioning accuracy for such robotic arms. This paper experimentally demonstrates that these structures can improve positioning accuracy. In addition to these structures, the joint torque characteristics of the inflatable robotic arms were investigated. In order to perform accurate motion control, a visual feedback control method was introduced for inflatable robotic arms. The mechanism and control system used in this paper can improve the positioning accuracy performance of inflatable robots. A 2-DOF inflatable robotic arm and a camera system were found to be able to achieve a high positioning accuracy (i.e. less than 1 mm).     

Design of an autonomous agricultural robot

This paper presents a state-of-the-art review in the development of autonomous agricultural robots including guidance systems, greenhouse autonomous systems and fruit-harvesting robots. A general concept for a field crops robotic machine to selectively harvest easily bruised fruit and vegetables is designed. Future trends that must be pursued in order to make robots a viable option for agricultural operations are focused upon.A prototype machine which includes part of this design has been implemented for melon harvesting. The machine consists of a Cartesian manipulator mounted on a mobile chassis pulled by a tractor. Two vision sensors are used to locate the fruit and guide the robotic arm toward it. A gripper grasps the melon and detaches it from the vine. The real-time control hardware architecture consists of a blackboard system, with autonomous modules for sensing, planning and control connected through a PC bus. Approximately 85% of the fruit are successfully located and harvested.     

Development,integration, and field evaluation of an autonomous citrus-harvesting robot

Citrus harvesting is a labor-intensive and time-intensive task. As the global population continues to age, labor costs are increasing dramatically. Therefore, the citrus-harvesting robot has attracted considerable attention from the business and academic communities. However, robotic harvesting in unstructured and natural citrus orchards remains a challenge. This study aims to address some challenges faced in commercializing citrus-harvesting robots. We present a fully integrated, autonomous, and innovative solution for citrus-harvesting robots to overcome the harvesting difficulties derived from the natural growth characteristics of citrus. This solution uses a fused simultaneous localization and mapping algorithm based on multiple sensors to perform high-precision localization and navigation for the robot in the field orchard. Besides, a novel visual method for estimating fruit poses is proposed to cope with the randomization of citrus growth orientations. Further, a new end-effector is designed to improve the success and conformity rate of citrus stem cutting. Finally, a fully autonomous harvesting robot system has been developed and integrated. Field evaluations showed that the robot could harvest citrus continuously with an overall success rate of 87.2% and an average picking time of 10.9 s/fruit. These efforts provide a solid foundation for the future commercialization of citrus-harvesting robots.     

基于机器视觉的圣女果采摘机械臂的动作研究

针对当前圣女果采摘机器人无法保证带蒂采摘的问题,提出一种通过机器视觉进行圣女果姿态分析进而生成特定的机械臂采摘动作的方法;该方法通过模拟人手采摘流程,能够使得机械臂末端执行器到达采摘位置时与圣女果的果蒂方向保持一致;整体系统包括对成熟圣女果的目标检测,测距算法的实现,圣女果方向识别以及机械臂动作生成;根据轮廓拟合算法的思想进行算法改进,实现针对圣女果的更加精确、稳定的方向识别算法,从而获得机械臂末端执行器与圣女果果蒂方向一致的目标位姿,进而实现相应机械臂采摘动作的生成;多次实验表明,改进后的对于圣女果方向的识别算法相较于传统轮廓拟合算法而言误差角度更小,对于不同姿态圣女果的方向识别更具稳定性,因此更加适用于实际采摘流程中根据圣女果姿态生成机械臂的特定采摘动作。     

Performance improvements of a sweet pepper harvesting robot in protected cropping environments

Using robots to harvest sweet peppers in protected cropping environments has remained unsolved despite considerable effort by the research community over several decades. In this paper, we present the robotic harvester, Harvey, designed for sweet peppers in protected cropping environments that achieved a 76.5% success rate on 68 fruit (within a modified scenario) which improves upon our prior work which achieved 58% on 24 fruit and related sweet pepper harvesting work which achieved 33% on 39 fruit (for their best tool in a modified scenario). This improvement was primarily achieved through the introduction of a novel peduncle segmentation system using an efficient deep convolutional neural network, in conjunction with three‐dimensional postfiltering to detect the critical cutting location. We benchmark the peduncle segmentation against prior art demonstrating an improvement in performance with a $F_1$ score of 0.564 compared to 0.302. The robotic harvester uses a perception pipeline to detect a target sweet pepper and an appropriate grasp and cutting pose used to determine the trajectory of a multimodal harvesting tool to grasp the sweet pepper and cut it from the plant. A novel decoupling mechanism enables the gripping and cutting operations to be performed independently. We perform an in‐depth analysis of the full robotic harvesting system to highlight bottlenecks and failure points that future work could address.     

A vision-centered multi-sensor fusing approach to self-localization and obstacle perception for robotic cars

Most state-of-the-art robotic cars’ perception systems are quite different from the way a human driver understands traffic environments. First, humans assimilate information from the traffic scene mainly through visual perception, while the machine perception of traffic environments needs to fuse information from several different kinds of sensors to meet safety-critical requirements. Second, a robotic car requires nearly 100% correct perception results for its autonomous driving, while an experienced human driver works well with dynamic traffic environments, in which machine perception could easily produce noisy perception results. In this paper, we propose a vision-centered multi-sensor fusing framework for a traffic environment perception approach to autonomous driving, which fuses camera, LIDAR, and GIS information consistently via both geometrical and semantic constraints for efficient self-localization and obstacle perception. We also discuss robust machine vision algorithms that have been successfully integrated with the framework and address multiple levels of machine vision techniques, from collecting training data, efficiently processing sensor data, and extracting low-level features, to higher-level object and environment mapping. The proposed framework has been tested extensively in actual urban scenes with our self-developed robotic cars for eight years. The empirical results validate its robustness and efficiency.     

Integrated design of an end effector for a visual servoing algorithm

An algorithm is presented for using a robot system with a single camera to position in threedimensional space a slender object for insertion into a hole; for example, an electrical pin-type termination into a connector hole. The algorithm relies on a control-configured end effector to achieve the required horizontal translations and rotational motion, and it does not require camera calibration. A force sensor in each fingertip is integrated with the vision system to allow the robot to teach itself new reference points when different connectors and pins are used. Variability in the grasped orientation and position of the pin can be accommodated with the sensor system. Performance tests show that the system is feasible. More work is needed to determine more precisely the effects of lighting levels and lighting direction.     

Development and characterization of a multi-camera 2D-vision system for enhanced performance of a drink serving robotic cell

A 2D-vision system is integrated into a drink-serving robotic cell, to enhance its flexibility. Two videocameras are used in a hybrid configuration scheme. The former is rigidly mounted on the robot end effector, the latter is fixed to the workplace. The robot cell is based on two Denso robots that interoperate to simulate real human tasks. Blob analysis, template matching and edge detection algorithms cooperate with motion procedures for fast object recognition and flexible adaptation to the environment. The paper details the system workflow, with particular emphasis to the vision procedures. The experimental results show their performance in terms of flexibility and robustness against defocusing, lighting conditions and noise.     

Combinatorial Optimization and Performance Analysis of a Multi-arm Cartesian Robotic Fruit Harvester—Extensions of Graph Coloring

A mobile melon robotic harvester consisting of multiple Cartesian manipulators, each with three degrees of freedom, is being developed. In order to design an optimal robot in terms of number of arms, manipulator capabilities, and robot speed, a method of allocating the fruits to be picked by each manipulator in a way that yields the maximum harvest has been developed. Such a method has already been devised for a multi-arm robot with 2DOF each. The maximum robotic harvesting problem was shown there to be an example of the maximum k-colorable subgraph problem (MKCSP) on an interval graph. However, for manipulators with 3DOF, the additional longitudinal motion results in variable intervals. To overcome this issue, we devise a new model based on the color-dependent interval graph (CDIG). This enables the harvest by multiple robotic arms to be modeled as a modified version of the MKCSP. Based on previous research, we develop a greedy algorithm that solves the problem in polynomial time, and prove its optimality using induction. As with the multi-arm 2DOF robot, when simulated numerous times on a field of randomly distributed fruits, the algorithm yields a nearly identical percentage of fruit harvested for given robot parameters. The results of the probabilistic analysis developed for the 2DOF robot was modified to yield a formula for the expected harvest ratio of the 3DOF robot. The significance of this method is that it enables selecting the most efficient actuators, number of manipulators, and robot forward velocity for maximal robotic fruit harvest.     

Towards autonomous robotic servicing: Using an integrated hand-arm-eye system for manipulating unknown objects

Executing complex robotic tasks including dexterous grasping and manipulation requires a combination of dexterous robots, intelligent sensors and adequate object information processing. In this paper, vision has been integrated into a highly redundant robotic system consisting of a tiltable camera and a three-fingered dexterous gripper both mounted on a puma-type robot arm. In order to condense the image data of the robot working space acquired from the mobile camera, contour image processing is used for offline grasp and motion planning as well as for online supervision of manipulation tasks. The performance of the desired robot and object motions is controlled by a visual feedback system coordinating motions of hand, arm and eye according to the specific requirements of the respective situation. Experiences and results based on several experiments in the field of service robotics show the possibilities and limits of integrating vision and tactile sensors into a dexterous hand-arm-eye system being able to assist humans in industrial or servicing environments.     

Vision-Based Path Generation Method for a Robot-Based Arc Welding System

In this paper a vision-based integrated method intended for path generation for a robot-based arc welding system, is presented. The described system is composed of the recently developed pseudo stereovision system (PSVS) or an ordinary stereovision system and the related software. A desired path can be generated, using a part or the entire edge of an image captured from a scene of the robotic environment, a line manually designed in the image, a combination of lines of the previous cases or lines belonging in successive images captured from different scenes. A user can initially process images selecting by means of pull down menus a variety of filters, edge detection methods and operations. Then the desired path as a combination of lines is selected from images. Applying our correspondence algorithm, corresponding edges can be found. Finally, a number of successive path points are calculated by means of the proposed path point calculation algorithm. In on line operation, the vision system mounted on the end-effector can capture images with the desired best view (welding view) of a scene by moving or rotating (using push buttons) the end effector of the robotic manipulator – PUMA 761. Other facilities of the described system are the selection of a variety of colors and shapes, histogram view, desired magnification, system information and automatic execution of user-selected operations. The graphical user interface is developed in Visual C++, it runs in a personal computer and communicates with the robotic manipulator (PUMA 761) through ALTER communication port.     

A pipelined image analysis system using custom integrated circuits

The requirements of a computer vision system in relation to industrial applications are studied. It is concluded that for the iconic image-processing part, a pipeline system built around a custom integrated circuit is preferred. This custom integrated circuit makes it possible to realize four basic operations in a compact way: shape recognition, mask generation, programmable image delay, and subsample filtering. An example that has been processed by a hardware realization of such a pipeline system is provided     

A neuro-genetic-simulated annealing approach to the inverse kinematics solution of robots: a simulation based study

In this study, a hybrid intelligent solution system including neural networks, genetic algorithms and simulated annealing has been proposed for the inverse kinematics solution of robotic manipulators. The main purpose of the proposed system is to decrease the end effector error of a neural network based inverse kinematics solution. In the designed hybrid intelligent system, simulated annealing algorithm has been used as a genetic operator to decrease the process time of the genetic algorithm to find the optimum solution. Obtained best solution from the neural network has been included in the initial solution of genetic algorithm with randomly produced solutions. The end effector error has been reduced micrometer levels after the implementation of the hybrid intelligent solution system.     

Nuclear sample management using a plutonium dissolution robotic system

At Los Alamos National Laboratory, a unique sample management robotic system has been developed that automates plutonium metal acid dissolutions, the aliquoting of resulting solutions to be analyzed using several analytical chemistry methods, and automates the barcode labeling of the solution containment vial providing positive solution identification. This system is based upon a commercial laboratory robot from Zymark, Inc. that has been enhanced by the addition of custom robotic workstations engineered and fabricated at our facility. This system will significantly reduce personnel radiation exposure and eliminate the redundancy of each analytical area producing separate dissolved plutonium solutions, which are used in chemical analyses.     

3D‐vision based detection,localization, and sizing of broccoli heads in the field

This paper describes a 3D vision system for robotic harvesting of broccoli using low‐cost RGB‐D sensors, which was developed and evaluated using sensory data collected under real‐world field conditions in both the UK and Spain. The presented method addresses the tasks of detecting mature broccoli heads in the field and providing their 3D locations relative to the vehicle. The paper evaluates different 3D features, machine learning, and temporal filtering methods for detection of broccoli heads. Our experiments show that a combination of Viewpoint Feature Histograms, Support Vector Machine classifier, and a temporal filter to track the detected heads results in a system that detects broccoli heads with high precision. We also show that the temporal filtering can be used to generate a 3D map of the broccoli head positions in the field. Additionally, we present methods for automatically estimating the size of the broccoli heads, to determine when a head is ready for harvest. All of the methods were evaluated using ground‐truth data from both the UK and Spain, which we also make available to the research community for subsequent algorithm development and result comparison. Cross‐validation of the system trained on the UK dataset on the Spanish dataset, and vice versa, indicated good generalization capabilities of the system, confirming the strong potential of low‐cost 3D imaging for commercial broccoli harvesting.     

Evolvable Hardware in Evolutionary Robotics

In recent decades the research on Evolutionary Robotics (ER) has developed rapidly. This direction is primarily concerned with the use of evolutionary computing techniques in the design of intelligent and adaptive controllers for robots. Meanwhile, much attention has been paid to a new set of integrated circuits named Evolvable Hardware (EHW), which is capable of reconfiguring its architectures unlimited time based on artificial evolution techniques. This paper surveys the application of evolvable hardware in evolutionary robotics. The evolvable hardware is an emerging research field concerning the development of evolvable robot controllers at the hardware level to adapt to dynamic changes in environments. The context of evolvable hardware and evolutionary robotics is reviewed, and a few representative experiments in the field of robotic hardware evolution are presented. As an alternative to conventional robotic controller designs, the potentialities and limitations of the EHW-based robotic system are discussed and summarized.