Automated crop plant detection based on the fusion of color and depth images for robotic weed control

Robotic weeding enables weed control near or within crop rows automatically, precisely and effectively. A computer‐vision system was developed for detecting crop plants at different growth stages for robotic weed control. Fusion of color images and depth images was investigated as a means of enhancing the detection accuracy of crop plants under conditions of high weed population. In‐field images of broccoli and lettuce were acquired 3–27 days after transplanting with a Kinect v2 sensor. The image processing pipeline included data preprocessing, vegetation pixel segmentation, plant extraction, feature extraction, feature‐based localization refinement, and crop plant classification. For the detection of broccoli and lettuce, the color‐depth fusion algorithm produced high true‐positive detection rates (91.7% and 90.8%, respectively) and low average false discovery rates (1.1% and 4.0%, respectively). Mean absolute localization errors of the crop plant stems were 26.8 and 7.4 mm for broccoli and lettuce, respectively. The fusion of color and depth was proved beneficial to the segmentation of crop plants from background, which improved the average segmentation success rates from 87.2% (depth‐based) and 76.4% (color‐based) to 96.6% for broccoli, and from 74.2% (depth‐based) and 81.2% (color‐based) to 92.4% for lettuce, respectively. The fusion‐based algorithm had reduced performance in detecting crop plants at early growth stages.     

3维自适应最小误差阈值分割法

为解决现有算法无法有效解决被多种噪声干扰的图像的阈值分割问题,本文提出了3维最小误差阈值法。该方法充分考虑图像像元点之间的灰度相关信息,结合图像灰度、均值和中值信息,构造出3维观测空间。然后基于相对熵定义出3维最佳阈值判别式。同时为了提高算法的处理速度,提出基于分解的快速实现方法,将3维阈值的求解分解成三个1维阈值的求解,其时间复杂度降为O(L),空间复杂度降为S(L)。实验结果分析表明,在不同噪声环境及非均匀光照条件下,尤其对多种噪声干扰的图像,与现有方法相比,文中算法均取得了更好的分割效果。     

基于叶片识别的侵入式室内植物建模

由于室内植物的叶片存在大量自遮挡,为了得到植物的完整三维信息,往往需要用户手动裁剪、扫描和配准叶片.针对该问题,提出了利用实例分割网络进行叶片识别并选取被裁剪叶片的方法.通过总结植物叶片形状与分布特征,建立虚拟植物模型,渲染大量带叶片轮廓信息的图片来训练实例分割网络,避免了耗费大量精力标注真实植物叶片.进一步提出了自动...     

一种快速有效的彩色图像边缘检测方法

在传统的灰度图像的边缘检测算子的基础上,对其进行改进。充分利用了彩色图像的颜色信息,将算法从灰度空间转换到RGB颜色空间。提出了彩色图像的高斯-拉普拉斯边缘检测算子,同时采用滤波来抑制噪声以及非极大值抑制来细化边缘,算法简单易于实现。实验结果表明,算法能有效地提取出彩色图像的边缘信息。     

3D‐vision based detection,localization, and sizing of broccoli heads in the field

This paper describes a 3D vision system for robotic harvesting of broccoli using low‐cost RGB‐D sensors, which was developed and evaluated using sensory data collected under real‐world field conditions in both the UK and Spain. The presented method addresses the tasks of detecting mature broccoli heads in the field and providing their 3D locations relative to the vehicle. The paper evaluates different 3D features, machine learning, and temporal filtering methods for detection of broccoli heads. Our experiments show that a combination of Viewpoint Feature Histograms, Support Vector Machine classifier, and a temporal filter to track the detected heads results in a system that detects broccoli heads with high precision. We also show that the temporal filtering can be used to generate a 3D map of the broccoli head positions in the field. Additionally, we present methods for automatically estimating the size of the broccoli heads, to determine when a head is ready for harvest. All of the methods were evaluated using ground‐truth data from both the UK and Spain, which we also make available to the research community for subsequent algorithm development and result comparison. Cross‐validation of the system trained on the UK dataset on the Spanish dataset, and vice versa, indicated good generalization capabilities of the system, confirming the strong potential of low‐cost 3D imaging for commercial broccoli harvesting.     

Robust joint stem detection and crop‐weed classification using image sequences for plant‐specific treatment in precision farming

Conventional farming still relies on large quantities of agrochemicals for weed management which have several negative side‐effects on the environment. Autonomous robots offer the potential to reduce the amount of chemicals applied, as robots can monitor and treat each plant in the field individually and thereby circumventing the uniform chemical treatment of the whole field. Such agricultural robots need the ability to identify individual crops and weeds in the field using sensor data and must additionally select effective treatment methods based on the type of weed. For example, certain types of weeds can only be effectively treated mechanically due to their resistance to herbicides, whereas other types can be treated trough selective spraying. In this article, we present a novel system that provides the necessary information for effective plant‐specific treatment. It estimates the stem location for weeds, which enables the robots to perform precise mechanical treatment, and at the same time provides the pixel‐accurate area covered by weeds for treatment through selective spraying. The major challenge in developing such a system is the large variability in the visual appearance that occurs in different fields. Thus, an effective classification system has to robustly handle substantial environmental changes including varying weed pressure, various weed types, different growth stages, changing visual appearance of the plants and the soil. Our approach uses an end‐to‐end trainable fully convolutional network that simultaneously estimates plant stem positions as well as the spatial extent of crop plants and weeds. It jointly learns how to detect the stems and the pixel‐wise semantic segmentation and incorporates spatial information by considering image sequences of local field strips. The jointly learned feature representation for both tasks furthermore exploits the crop arrangement information that is often present in crop fields. This information is considered even if it is only observable from the image sequences and not a single image. Such image sequences, as typically provided by robots navigating over the field along crop rows, enable our approach to robustly estimate the semantic segmentation and stem positions despite the large variations encountered in different fields. We implemented and thoroughly tested our approach on images from multiple farms in different countries. The experiments show that our system generalizes well to previously unseen fields under varying environmental conditions—a key capability to deploy such systems in the real world. Compared to state‐of‐the‐art approaches, our approach generalizes well to unseen fields and not only substantially improves the stem detection accuracy, that is, distinguishing crop and weed stems, but also improves the semantic segmentation performance.     

RECODE: an image‐based collision detection algorithm

Object interactions are ubiquitous in interactive computer graphics, 3D object motion simulations, virtual reality and robotics applications. Most collision detection algorithms are based on geometrical object‐space interference tests. Some algorithms have employed an image‐space approach to the collision detection problem. In this paper we demonstrate an image‐space collision detection process that allows substantial computational savings during the image‐space interference test. This approach makes efficient use of the graphics rendering hardware for real‐time complex object interactions. Copyright © 1999 John Wiley & Sons, Ltd.     

Evaluation of visual saliency analysis algorithms in noisy images

Most existing visual saliency analysis algorithms assume that the input image is clean and does not have any disturbances. However, this situation is not always the case. In this paper, we provide an extensive evaluation of visual saliency analysis algorithms in noisy images. We analyze the noise immunity of saliency analysis algorithms by evaluating the performances of the algorithms in noisy images with increasing noise scales and by studying the effects of applying different denoising methods before performing saliency analysis. We use 10 state-of-the-art saliency analysis algorithms and 7 typical image denoising methods on 4 eye fixation datasets and 2 salient object detection datasets. Our experiments show that the performances of saliency analysis algorithms decrease with increasing image noise scales in general. An exception is that the nonlinear features (NF) integrated algorithm shows good noise immunity. We also find that image denoising methods can greatly improve the noise immunity of the algorithms. Our results show that the combination of NF and Median denoising method works best on eye fixation datasets and the combination of saliency optimization (SO) and color block-matching and 3D filtering (C-BM3D) method works best on salient object detection datasets. The combination of SO and Average denoising method works best for applications wherein time efficiency is a major concern for both types of datasets.     

基于形状特征的叶片图像识别算法比较研究

植物是生命的主要形态之一,其种类已达40多万种,对其进行分类识别在生物多样性保护,生态农业,生物安全中有着重要的意义。不同的种类的植物一般有着不同的叶片形状,因此叶片的形状特征在植物分类中扮演着重要的角色。作为计算机视觉的一个重要应用的植物叶片图像识别,近些年来受到了学者们的关注,产生了大量的研究成果。但由于植物种类巨大,叶片图像存在的类内差异大、类间差异小和叶片的自遮挡等问题等诸多问题,使得叶片图像的识别仍然是目前计算机视觉应用研究的一个热点。对近些年来的基于形状特征的叶片图像识别算法进行了综述和比较,对现有的算法进行了分类,对目前各类最先进的识别算法进行了分析和比较。此外,还介绍了常用的叶片图像测试集和性能评估方法,并将各类算法进行了实验结果的比较研究。研究工作既为现有的植物叶片识别算法的实际应用提供了指导,又为今后进一步研究新的高性能的识别算法提出了努力的方向。     

局部均值噪声估计的盲3维滤波降噪算法

目的 图像在获取和传输的过程中很容易受到噪声的干扰,图像降噪作为众多图像处理系统的预处理模块在过去数十年中得到了广泛的研究。在已提出的降噪算法中,往往采用加性高斯白噪声模型AWGN（additive white Gaussian noise）为噪声建模,噪声水平（严重程度）由方差参数控制。经典的BM3D 3维滤波算法属于非盲降噪（non-blind denoising algorithm）算法,在实际使用中需要由人工评估图像噪声水平并设置参数,存在着噪声评估值随机性大而导致无法获得最佳降噪效果的问题。为此,提出了一种新的局部均值噪声估计（LME）算法并作为BM3D算法的前置预处理模块。方法 本文专注于利用基于自然统计规律（NSS）的图像质量感知特征和局部均值估计技术构建图像噪声水平预测器,并通过它高效地获得噪声图像中准确的噪声水平值。关于自然场景统计方面的研究表明,无失真的自然场景图像在空域或者频率域上具有显著的统计规律,一旦受到噪声干扰会产生规律性的偏移,可以提取这些特征值作为反映图像质量好坏的图像质量感知特征。另外,局部均值估计因其简单而高效率的预测特性被采用。具体实现上,在具有广泛代表性且未受噪声干扰图像集合上添加不同噪声水平的高斯噪声构建失真图像集合,然后利用小波变换对这些失真图像进行不同尺度和不同方向的分解,再用广义高斯分布模型（GGD）提取子带滤波系数的统计信息构成描述图像失真程度的特征矢量,最后用每幅失真图像上所提取的特征矢量及对其所施加的高斯噪声水平值构成了失真特征矢量库。在降噪阶段,用相同的特征提取方法提取待降噪的图像的特征矢量并在失真特征矢量库中检索出与之类似的若干特征矢量及它们所对应的噪声水平值,然后用局部均值法估计出待降噪图像中高斯噪声大小作为经典BM3D算法的输入参数。结果 改进后的BM3D算法转换为盲降噪算法,称为BM3D-LME（block-matching and 3D filtering based on local means estimation）算法。准确的噪声估计对于诸如图像降噪,图像超分辨率和图像分割等图像处理任务非常重要。已经验证了所提出噪声水平估计算法的准确性、鲁棒性和有效性。结论 相对人工进行噪声估计,LME算法能够准确、快速地估算出任意待降噪图像中的噪声大小。配合BM3D算法使用后,有效提高了它的实际降噪效果并扩大它的应用范围。     

A two-stage fuzzy multi-objective framework for segmentation of 3D MRI brain image data

Segmentation of Magnetic Resonance Imaging (MRI) brain image data has a significant impact on the computer guided medical image diagnosis and analysis. However, due to limitation of image acquisition devices and other related factors, MRI images are severely affected by the noise and inhomogeneity artefacts which lead to blurry edges in the intersection of the intra-organ soft tissue regions, making the segmentation process more difficult and challenging. This paper presents a novel two-stage fuzzy multi-objective framework (2sFMoF) for segmenting 3D MRI brain image data. In the first stage, a 3D spatial fuzzy c-means (3DSpFCM) algorithm is introduced by incorporating the 3D spatial neighbourhood information of the volume data to define a new local membership function along with the global membership function for each voxel. In particular, the membership functions actually define the underlying relationship between the voxels of a close cubic neighbourhood and image data in 3D image space. The cluster prototypes thus obtained are fed into a 3D modified fuzzy c-means (3DMFCM) algorithm, which further incorporates local voxel information to generate the final prototypes. The proposed framework addresses the shortcomings of the traditional FCM algorithm, which is highly sensitive to noise and may stuck into a local minima. The method is validated on a synthetic image volume and several simulated and in-vivo 3D MRI brain image volumes and found to be effective even in noisy data. The empirical results show the supremacy of the proposed method over the other FCM based algorithms and other related methods devised in the recent past.     

Practical implementation of a depth‐feeling‐enhanced two‐plane electro‐floating display system using three‐dimensional integral images

Abstract— Although there are numerous types of floating‐image display systems which can project three‐dimensional (3‐D) images into real space through a convex lens or a concave mirror, most of them provide only one image plane in space to the observer; therefore, they lack an in‐depth feeling. In order to enhance a real 3‐D feeling of floating images, a multi‐plane floating display is required. In this paper, a novel two‐plane electro‐floating display system using 3‐D integral images is proposed. One plane for the object image is provided by an electro‐floating display system, and the other plane for the background image is provided with the 3‐D integral imaging system. Consequently, the proposed two‐plane electro‐floating display system, having a 3‐D background, can provide floated images in front of background integral images resulting in a different perspective to the observer. To show the usefulness of the proposed system, experiments were carried out and their results are presented. In addition, the prototype was practically implemented and successfully tested.     

Projective Blue‐Noise Sampling

We propose projective blue‐noise patterns that retain their blue‐noise characteristics when undergoing one or multiple projections onto lower dimensional subspaces. These patterns are produced by extending existing methods, such as dart throwing and Lloyd relaxation, and have a range of applications. For numerical integration, our patterns often outperform state‐of‐the‐art stochastic and low‐discrepancy patterns, which have been specifically designed only for this purpose. For image reconstruction, our method outperforms traditional blue‐noise sampling when the variation in the signal is concentrated along one dimension. Finally, we use our patterns to distribute primitives uniformly in 3D space such that their 2D projections retain a blue‐noise distribution.     

Early detection of nutrient and biotic stress in Phaseolus vulgaris

Prerequisites for optimal, high crop yield are disease‐free growth and an equilibrated supply of nutrients. Early signatures of stress‐altered physiology, before appearance of symptoms in the visible spectrum, allow timely treatment. Early detection of stress development was carried out on phaseolus vulgaris bean infected with the agriculturally important grey mould pathogen and under conditions of magnesium deficiency, limiting photosynthesis. During stress development, bean plants were monitored by time‐lapse imaging with thermal, video and chlorophyll fluorescence cameras, mounted on a gantry robot system. For early detection of grey mould infection, chlorophyll fluorescence imaging proved to be the most sensitive. This technique detected magnesium deficiency at least three days before visual symptoms appeared. Further development of non‐contact technology for plant health monitoring will help to achieve optimal productivity in greenhouse and field cultures. Associated establishment of a stress catalogue based on early symptoms will allow swift diagnosis.     

圆形植物识别定位的HSV模型研究

圆形植物的识别定位是植物自动修剪设备的核心,针对圆形植物图像,采用传统图像分割方法因存在噪声难以完成有效分割,提出基于HSV模型的图像分割方法。该方法利用色调和饱和度分量对圆形植物图像进行图像分割处理,对分割图像进行形态学处理得到理想的圆形植物目标区域,并进行图像标定得出圆形植物的中心点坐标和半径尺寸。使用该方法可以有效地对圆形植物进行识别定位。     

Multi-modality imagery database for plant phenotyping

Among many applications of machine vision, plant image analysis has recently began to gain more attention due to its potential impact on plant visual phenotyping, particularly in understanding plant growth, assessing the quality/performance of crop plants, and improving crop yield. Despite its importance, the lack of publicly available research databases containing plant imagery has substantially hindered the advancement of plant image analysis. To alleviate this issue, this paper presents a new multi-modality plant imagery database named “MSU-PID,” with two distinct properties. First, MSU-PID is captured using four types of imaging sensors, fluorescence, infrared, RGB color, and depth. Second, the imaging setup and the variety of manual labels allow MSU-PID to be suitable for a diverse set of plant image analysis applications, such as leaf segmentation, leaf counting, leaf alignment, and leaf tracking. We provide detailed information on the plants, imaging sensors, calibration, labeling, and baseline performances of this new database.     

Sketch‐Based Editing Tools for Tumour Segmentation in 3D Medical Images

In the past years sophisticated automatic segmentation algorithms for various medical image segmentation problems have been developed. However, there are always cases where automatic algorithms fail to provide an acceptable segmentation. In these cases the user needs efficient segmentation editing tools, a problem which has not received much attention in research. We give a comprehensive overview on segmentation editing for three‐dimensional (3D) medical images. For segmentation editing in two‐dimensional (2D) images, we discuss a sketch‐based approach where the user modifies the segmentation in the contour domain. Based on this 2D interface, we present an image‐based as well as an image‐independent method for intuitive and efficient segmentation editing in 3D in the context of tumour segmentation in computed tomography (CT). Our editing tools have been evaluated on a database containing 1226 representative liver metastases, lung nodules and lymph nodes of different shape, size and image quality. In addition, we have performed a qualitative evaluation with radiologists and technical experts, proving the efficiency of our tools.     

Integrating an urban green space typology into procedural 3D visualization for collaborative planning

Urban green spaces offer multiple ecosystem services (ES), which provide a variety of benefits to human well-being. Yet in urban planning they are not taken into account systematically. Recently new tools have been developed integrating ES into procedural modeling and visualization to raise stakeholder awareness for the explicit ES trade-offs that have to be made. These tools yet do not allow fast and comprehensive integration of ES provision in urban environments. In this paper we show how urban green space typologies can be linked to ES provision for facilitating collaboration between stakeholders of different backgrounds. Based on a generic typology green spaces were mapped and linked with information on potentially provided ES and their parameters. Further, pattern designs of the green space types were described with a form-based code. Both the map of green space types and the pattern designs were integrated into the parametric modeling and visualization chain of Esri CityEngine resulting in 3D visualizations of the green space patterns and correlating ES indicators. The green space typology allows for integrating different kinds of knowledge from both science and practice communities. The procedural model enables rapid interactive visualization of urban patterns and calculation of simple indicator values on the provision of ES. The simple approach for mapping green space types with low data requirements and the generic green space design patterns allow for transferability to other places and application to large areas. The developed approach is simple and fast yet comprehensive to communicate the vital importance of all green space types within the urban environment.