Automated crop plant detection based on the fusion of color and depth images for robotic weed control

Robotic weeding enables weed control near or within crop rows automatically, precisely and effectively. A computer‐vision system was developed for detecting crop plants at different growth stages for robotic weed control. Fusion of color images and depth images was investigated as a means of enhancing the detection accuracy of crop plants under conditions of high weed population. In‐field images of broccoli and lettuce were acquired 3–27 days after transplanting with a Kinect v2 sensor. The image processing pipeline included data preprocessing, vegetation pixel segmentation, plant extraction, feature extraction, feature‐based localization refinement, and crop plant classification. For the detection of broccoli and lettuce, the color‐depth fusion algorithm produced high true‐positive detection rates (91.7% and 90.8%, respectively) and low average false discovery rates (1.1% and 4.0%, respectively). Mean absolute localization errors of the crop plant stems were 26.8 and 7.4 mm for broccoli and lettuce, respectively. The fusion of color and depth was proved beneficial to the segmentation of crop plants from background, which improved the average segmentation success rates from 87.2% (depth‐based) and 76.4% (color‐based) to 96.6% for broccoli, and from 74.2% (depth‐based) and 81.2% (color‐based) to 92.4% for lettuce, respectively. The fusion‐based algorithm had reduced performance in detecting crop plants at early growth stages.     

Robust joint stem detection and crop‐weed classification using image sequences for plant‐specific treatment in precision farming

Conventional farming still relies on large quantities of agrochemicals for weed management which have several negative side‐effects on the environment. Autonomous robots offer the potential to reduce the amount of chemicals applied, as robots can monitor and treat each plant in the field individually and thereby circumventing the uniform chemical treatment of the whole field. Such agricultural robots need the ability to identify individual crops and weeds in the field using sensor data and must additionally select effective treatment methods based on the type of weed. For example, certain types of weeds can only be effectively treated mechanically due to their resistance to herbicides, whereas other types can be treated trough selective spraying. In this article, we present a novel system that provides the necessary information for effective plant‐specific treatment. It estimates the stem location for weeds, which enables the robots to perform precise mechanical treatment, and at the same time provides the pixel‐accurate area covered by weeds for treatment through selective spraying. The major challenge in developing such a system is the large variability in the visual appearance that occurs in different fields. Thus, an effective classification system has to robustly handle substantial environmental changes including varying weed pressure, various weed types, different growth stages, changing visual appearance of the plants and the soil. Our approach uses an end‐to‐end trainable fully convolutional network that simultaneously estimates plant stem positions as well as the spatial extent of crop plants and weeds. It jointly learns how to detect the stems and the pixel‐wise semantic segmentation and incorporates spatial information by considering image sequences of local field strips. The jointly learned feature representation for both tasks furthermore exploits the crop arrangement information that is often present in crop fields. This information is considered even if it is only observable from the image sequences and not a single image. Such image sequences, as typically provided by robots navigating over the field along crop rows, enable our approach to robustly estimate the semantic segmentation and stem positions despite the large variations encountered in different fields. We implemented and thoroughly tested our approach on images from multiple farms in different countries. The experiments show that our system generalizes well to previously unseen fields under varying environmental conditions—a key capability to deploy such systems in the real world. Compared to state‐of‐the‐art approaches, our approach generalizes well to unseen fields and not only substantially improves the stem detection accuracy, that is, distinguishing crop and weed stems, but also improves the semantic segmentation performance.     

A semi-supervised system for weed mapping in sunflower crops using unmanned aerial vehicles and a crop row detection method

This paper presents a system for weed mapping, using imagery provided by unmanned aerial vehicles (UAVs). Weed control in precision agriculture is based on the design of site-specific control treatments according to weed coverage. A key component is precise and timely weed maps, and one of the crucial steps is weed monitoring, by ground sampling or remote detection. Traditional remote platforms, such as piloted planes and satellites, are not suitable for early weed mapping, given their low spatial and temporal resolutions. Nonetheless, the ultra-high spatial resolution provided by UAVs can be an efficient alternative. The proposed method for weed mapping partitions the image and complements the spectral information with other sources of information. Apart from the well-known vegetation indexes, which are commonly used in precision agriculture, a method for crop row detection is proposed. Given that crops are always organised in rows, this kind of information simplifies the separation between weeds and crops. Finally, the system incorporates classification techniques for the characterisation of pixels as crop, soil and weed. Different machine learning paradigms are compared to identify the best performing strategies, including unsupervised, semi-supervised and supervised techniques. The experiments study the effect of the flight altitude and the sensor used. Our results show that an excellent performance is obtained using very few labelled data complemented with unlabelled data (semi-supervised approach), which motivates the use of weed maps to design site-specific weed control strategies just when farmers implement the early post-emergence weed control.     

Computer vision based methods for detecting weeds in lawns

In this paper, two methods for detecting weeds in lawns using computer vision techniques are proposed. The first is based on an assumption about the differences in statistical values between the weed and grass areas in edge images and using Bayes classifier to discriminate them. The second also uses the differences in texture between both areas in edge images but instead applies only simple morphology operators. Correct weed detection rates range from 77.70 to 82.60% for the first method and from 89.83 to 91.11% for the second method. From the results, the methods show the robustness against lawn color change. In addition, the proposed methods together with a chemical weeding system as well as a non-chemical weeding system based on pulse high voltage discharge are simulated and the efficiency of the overall systems are evaluated theoretically. With a chemical based system, more than 72% of the weeds can be destroyed with a herbicide reduction rate of 90–94% for both methods. For the latter weeding system, killed weed rate varies from 58 to 85%.     

Image-based recognition framework for robotic weed control systems

In this paper, we introduce a novel and efficient image-based weed recognition system for the weed control problem of Broad-leaved Dock (Rumex obtusifolius L.). Our proposed weed recognition system is developed using a framework, that allows the examination of the affects for various image resolutions in detection and recognition accuracy. Moreover, it includes state-of-the-art object/image categorization processes such as feature detection and extraction, codebook learning, feature encoding, image representation and classification. The efficiency of those processes have been improved and optimized by introducing methodologies, techniques and system parameters specially tailored for the goal of weed recognition. Through an exhaustive optimization process, which is presented as our experimental evaluation, we conclude to a weed recognition system that uses an image input resolution of 200 ×150, SURF features over dense feature extraction, an optimized Gaussian Mixture Model based codebook combined with Fisher encoding, using a two level image representation. The resulting image representation vectors are classified using a linear classifier. This system is experimentally shown to yield state-of-the-art recognition accuracy of 89.09% in the examined dataset. Our proposed system is also experimentally shown to comply with the specifications of the examined applications since it provides low false-positive results of 4.38%. As a result, the proposed framework can be efficiently used in weed control robots for precision farming applications.     

Crop recognition under weedy conditions based on 3D imaging for robotic weed control

A 3D time‐of‐flight camera was applied to develop a crop plant recognition system for broccoli and green bean plants under weedy conditions. The developed system overcame the previously unsolved problems caused by occluded canopy and illumination variation. An efficient noise filter was developed to remove the sparse noise points in 3D point cloud space. Both 2D and 3D features including the gradient of amplitude and depth image, surface curvature, amplitude percentile index, normal direction, and neighbor point count in 3D space were extracted and found effective for recognizing these two types of plants. Separate segmentation algorithms were developed for each of the broccoli and green bean plant in accordance with their 3D geometry and 2D amplitude characteristics. Under the experimental condition where the crops were heavily infested by various types of weed plants, detection rates over 88.3% and 91.2% were achieved for broccoli and green bean plant leaves, respectively. Additionally, the crop plants were segmented out with nearly complete shape. Moreover, the algorithms were computationally optimized, resulting in an image processing speed of over 30 frames per second.     

Uncertainty analysis and evaluation of a complex,multi-specific weed dynamics model with diverse and incomplete data sets

Weed dynamics models are needed to test prospective cropping systems but are rarely evaluated with independent data (“validated”). Here, we evaluated the FlorSys model which quantifies the effects of cropping systems and pedoclimate on multispecific weed dynamics with a daily time step. We adapted existing validation methodologies and uncertainty analyses to account for multi-specific, multi-annual and diverse outputs, focusing on missing input data, incomplete and imprecise weed time series. Field data ranged from entirely monitored cropping system trials to annual snapshots recorded on farm fields by the French Biovigilance-Flore network. FlorSys satisfactorily predicted weed seed bank, plant densities and crop yields, at daily and multi-annual scales, at well monitored sites. It overestimated plant biomass and underestimated total flora density. Missing processes (photoperiod dependency in flowering, crop:weed competition for nitrogen) and inadequately predicted scenarios (weed dynamics in untilled fields, floras with summer-emerging species) were identified. Guidelines for model use were proposed.     

Delay‐range‐dependent control of nonlinear time‐delay systems under input saturation

This paper describes a delay‐range‐dependent local state feedback controller synthesis approach providing estimation of the region of stability for nonlinear time‐delay systems under input saturation. By employing a Lyapunov–Krasovskii functional, properties of nonlinear functions, local sector condition and Jensen's inequality, a sufficient condition is derived for stabilization of nonlinear systems with interval delays varying within a range. Novel solutions to the delay‐range‐dependent and delay‐dependent stabilization problems for linear and nonlinear time‐delay systems, respectively, subject to input saturation are derived as specific scenarios of the proposed control strategy. Also, a delay‐rate‐independent condition for control of nonlinear systems in the presence of input saturation with unknown delay‐derivative bound information is established. And further, a robust state feedback controller synthesis scheme ensuring L₂ gain reduction from disturbance to output is devised to address the problem of the stabilization of input‐constrained nonlinear time‐delay systems with varying interval lags. The proposed design conditions can be solved using linear matrix inequality tools in connection with conventional cone complementary linearization algorithms. Simulation results for an unstable nonlinear time‐delay network and a large‐scale chemical reactor under input saturation and varying interval time‐delays are analyzed to demonstrate the effectiveness of the proposed methodology. Copyright © 2015 John Wiley & Sons, Ltd.     

Encrypted polynomial control based on tailored two‐party computation

Encrypted control enables confidential controller evaluations in cloud‐based or networked control systems. Roughly speaking, an encrypted controller is a modified control algorithms that is capable of computing encrypted control actions based on encrypted system states. Encrypted control has been realized using different tools from cryptography such as homomorphic encryption, secret sharing, and multiparty computation. However, the vast majority of existing encrypted controllers is linear or makes intensive use of encrypted linear operations. In this article, we present a novel and flexible method for the encrypted implementation of arbitrary polynomial controllers. Technically, our approach builds on tailored two‐party computation combined with secret sharing and additively homomorphic encryption.     

综合颜色和形态特征的小麦田杂草识别方法

利用机器视觉技术把杂草精确识别出来是精细农业领域研究的热点问题之一。针对杂草与小麦叶子交叠的情况，提出一种综合颜色和形态特征的方法进行杂草识别。在L*a*b*颜色空间，选取a*作为特征量并用改进的最大类间方差法进行阈值分割获得植物图像；在HSI颜色空间，利用多层的同质性分割算法分离小麦与杂草；结合形态学特征开闭运算滤波及二值逻辑与运算获得杂草图像；模拟化学除草系统，从理论上评价整个系统的识别效率。实验结果表明：杂草正确识别率高达92.6%以上，且除草剂的使用量减少超过72.4%。     

Multi-spectral vision system for weed detection

The proposed online system distinguishes crop from weeds based on multi-spectal reflectance gathered with an imaging spectrograph. Under field conditions, up to 86% of the vegetation samples (80% of crop, 91% of weed) were recognized herbicide reductions of up to 90%.     

ADAPTIVE DYNAMIC MATRIX CONTROL FOR NETWORK‐BASED CONTROL SYSTEMS WITH RANDOM DELAYS

Random transfer delays in network‐based control systems (NCSs) degrade the control performance and can even destabilize the control system. To address this problem, the adaptive dynamic matrix control (DMC) algorithm is proposed. The control algorithm is derived by applying the philosophy behind DMC to a discrete time‐delay model. A method to estimate the network‐induced delays is also presented to facilitate implementation of the control algorithm. Finally, an NCS platform based on the TrueTime simulator is constructed. With it, the adaptive DMC algorithm is compared with the conventional DMC algorithm under different network conditions. Simulation results show that the proposed adaptive DMC algorithm can respond to various network conditions adaptively and achieve better control performance for NCSs with random transfer delays.     

Bounded Consensus Algorithms for Multi‐Agent Systems in Directed Networks

In this paper, the consensus problem is investigated via bounded controls for the multi‐agent systems with or without communication. Based on the nested saturation method, the saturated control laws are designed to solve the consensus problem. Under the designed saturated control laws, the transient performance of the closed‐loop system can be improved by tuning the saturation level. First of all, asymptotical consensus algorithms with bounded control inputs are proposed for the multi‐agent systems with or without communication delays. Under these consensus algorithms, the states’ consensus can be achieved asymptotically. Then, based on a kind of novel nonlinear saturation functions, bounded finite‐time consensus algorithms are further developed. It is shown that the states’ consensus can be achieved in finite time. Finally, two examples are given to verify the efficiency of the proposed methods.     

Decentralized event‐triggered circle formation control for multiagent systems via synchronous periodic event detection

This paper investigates circle formation problem of multiagent systems over a kind of strongly connected and weight‐unbalanced directed graphs. To solve the concerned problem, decentralized periodic event‐triggered algorithms subject to or not to time delays are proposed, which have the advantages of decreasing the overall burden of the network in terms of finite communication and control input updates. In such algorithms, each agent independently evaluates whether the locally sampled information of itself should be broadcasted to or not to its neighbors. Furthermore, another advantage of our proposed algorithms is to automatically exclude Zeno behavior, which should be seriously considered in a variety of event‐based network systems. Sufficient conditions on circle formation control are derived under which the states of all agents can be ensured to converge to some desired equilibrium point. Simulation results are given to validate the effectiveness of the proposed methods.     

Selecting patterns and features for between- and within- crop-row weed mapping using UAV-imagery

This paper approaches the problem of weed mapping for precision agriculture, using imagery provided by Unmanned Aerial Vehicles (UAVs) from sunflower and maize crops. Precision agriculture referred to weed control is mainly based on the design of early post-emergence site-specific control treatments according to weed coverage, where one of the most important challenges is the spectral similarity of crop and weed pixels in early growth stages. Our work tackles this problem in the context of object-based image analysis (OBIA) by means of supervised machine learning methods combined with pattern and feature selection techniques, devising a strategy for alleviating the user intervention in the system while not compromising the accuracy. This work firstly proposes a method for choosing a set of training patterns via clustering techniques so as to consider a representative set of the whole field data spectrum for the classification method. Furthermore, a feature selection method is used to obtain the best discriminating features from a set of several statistics and measures of different nature. Results from this research show that the proposed method for pattern selection is suitable and leads to the construction of robust sets of data. The exploitation of different statistical, spatial and texture metrics represents a new avenue with huge potential for between and within crop-row weed mapping via UAV-imagery and shows good synergy when complemented with OBIA. Finally, there are some measures (specially those linked to vegetation indexes) that are of great influence for weed mapping in both sunflower and maize crops.     

Classifying and clustering malicious advertisement uniform resource locators using deep learning

Malicious online advertisement detection has attracted increasing attention in recent years in both academia and industry. The existing advertising blocking systems are vulnerable to the evolution of new attacks and can cause time latency issues by analyzing web content or querying remote servers. This article proposes a lightweight detection system for advertisement Uniform resource locators (URLs) detection, depending only on lexical‐based features. Deep learning algorithms are used for online advertising classification. After optimizing the deep neural network architecture, our proposed approach can achieve satisfactory results with false negative rate as low as 1.31%. We also design a novel unsupervised method for data clustering. With the implementation of AutoEncoder for feature preprocessing and t‐distributed stochastic neighbor embedding for clustering and visualization, our model outperforms other dimensionality reduction algorithms by generating clear clusterings for different URL families.     

Delay compensation for linear systems with both state and distinct input delays

This paper is concerned with the stabilization of linear systems with both state and distinct input delays. Nested predictor feedback controllers are designed to predict the future states such that the distinct input delays that can be arbitrarily large yet bounded are compensated completely. It is shown that the compensated closed‐loop system possesses the same characteristic equation as the closed‐loop system without distinct input delays. Both continuous‐time and discrete‐time time‐delay systems are studied in this paper. Moreover, the safe implementation problem for the continuous‐time nested predictor feedback controller is solved via adding input filters. Three numerical examples show the effectiveness of the proposed approaches.     

Global output feedback regulation of nonlinear time delay systems subject to sensor measurement faults

This article studied the global output feedback regulation problem for a class of uncertain nonlinear time delay systems subject to unknown measurement faults on sensors. Different from the existing works, we consider the unknown time‐varying delays on the system states and relax their conservative condition on nonlinear functions. By introducing two novel time‐varying gains, a new global output feedback regulation algorithm is proposed, which ensures control parameters can be chosen flexibly. The proposed linear‐like controller is independent of the unknown time‐varying delays. Moreover, it has a simple structure, which is convenient for the implementation in practice. Based on the Lyapunov stability theory, it is strictly proved that all signals of the resulting closed‐loop system are globally bounded with the designed controller. Finally, a simulation example is presented to illustrate the effectiveness of the proposed output feedback regulation algorithm.     

Large scale economic dispatch of power systems using oppositional invasive weed optimization

This paper presents an evolutionary hybrid algorithm of invasive weed optimization (IWO) merged with oppositional based learning to solve the large scale economic load dispatch (ELD) problems. The oppositional invasive weed optimization (OIWO) is based on the colonizing behavior of weed plants and empowered by quasi opposite numbers. The proposed OIWO methodology has been developed to minimize the total generation cost by satisfying several constraints such as generation limits, load demand, valve point loading effect, multi-fuel options and transmission losses. The proposed algorithm is tested and validated using five different test systems. The most important merit of the proposed methodology is high accuracy and good convergence characteristics and robustness to solve ELD problems. The simulation results of the proposed OIWO algorithm show its applicability and superiority when compared with the results of other tested algorithms such as oppositional real coded chemical reaction, shuffled differential evolution, biogeography based optimization, improved coordinated aggregation based PSO, quantum-inspired particle swarm optimization, hybrid quantum mechanics inspired particle swarm optimization, modified shuffled frog leaping algorithm with genetic algorithm, simulated annealing based optimization and estimation of distribution and differential evolution algorithm.     

基于图像处理技术的杂草特征提取方法研究

利用计算机视觉技术将杂草从背景中识别出来进行定位喷洒农药已成为精细农业研究的热点。选取颜色空间OHTA中I＇2分量作为特征量;利用基于遗传算法的自动阈值选取方法对特征分量巧进行阈值分割初步分离杂草与小麦;通过颜色聚类和形态滤波获得准确的杂草区域。实验结果表明：直接在彩色空间进行分割,可提高彩色图像的分割效果,利用该方法获得的杂草平均正确识别率达到90．47％。