Automated crop plant detection based on the fusion of color and depth images for robotic weed control

Robotic weeding enables weed control near or within crop rows automatically, precisely and effectively. A computer‐vision system was developed for detecting crop plants at different growth stages for robotic weed control. Fusion of color images and depth images was investigated as a means of enhancing the detection accuracy of crop plants under conditions of high weed population. In‐field images of broccoli and lettuce were acquired 3–27 days after transplanting with a Kinect v2 sensor. The image processing pipeline included data preprocessing, vegetation pixel segmentation, plant extraction, feature extraction, feature‐based localization refinement, and crop plant classification. For the detection of broccoli and lettuce, the color‐depth fusion algorithm produced high true‐positive detection rates (91.7% and 90.8%, respectively) and low average false discovery rates (1.1% and 4.0%, respectively). Mean absolute localization errors of the crop plant stems were 26.8 and 7.4 mm for broccoli and lettuce, respectively. The fusion of color and depth was proved beneficial to the segmentation of crop plants from background, which improved the average segmentation success rates from 87.2% (depth‐based) and 76.4% (color‐based) to 96.6% for broccoli, and from 74.2% (depth‐based) and 81.2% (color‐based) to 92.4% for lettuce, respectively. The fusion‐based algorithm had reduced performance in detecting crop plants at early growth stages.     

Robust joint stem detection and crop‐weed classification using image sequences for plant‐specific treatment in precision farming

Conventional farming still relies on large quantities of agrochemicals for weed management which have several negative side‐effects on the environment. Autonomous robots offer the potential to reduce the amount of chemicals applied, as robots can monitor and treat each plant in the field individually and thereby circumventing the uniform chemical treatment of the whole field. Such agricultural robots need the ability to identify individual crops and weeds in the field using sensor data and must additionally select effective treatment methods based on the type of weed. For example, certain types of weeds can only be effectively treated mechanically due to their resistance to herbicides, whereas other types can be treated trough selective spraying. In this article, we present a novel system that provides the necessary information for effective plant‐specific treatment. It estimates the stem location for weeds, which enables the robots to perform precise mechanical treatment, and at the same time provides the pixel‐accurate area covered by weeds for treatment through selective spraying. The major challenge in developing such a system is the large variability in the visual appearance that occurs in different fields. Thus, an effective classification system has to robustly handle substantial environmental changes including varying weed pressure, various weed types, different growth stages, changing visual appearance of the plants and the soil. Our approach uses an end‐to‐end trainable fully convolutional network that simultaneously estimates plant stem positions as well as the spatial extent of crop plants and weeds. It jointly learns how to detect the stems and the pixel‐wise semantic segmentation and incorporates spatial information by considering image sequences of local field strips. The jointly learned feature representation for both tasks furthermore exploits the crop arrangement information that is often present in crop fields. This information is considered even if it is only observable from the image sequences and not a single image. Such image sequences, as typically provided by robots navigating over the field along crop rows, enable our approach to robustly estimate the semantic segmentation and stem positions despite the large variations encountered in different fields. We implemented and thoroughly tested our approach on images from multiple farms in different countries. The experiments show that our system generalizes well to previously unseen fields under varying environmental conditions—a key capability to deploy such systems in the real world. Compared to state‐of‐the‐art approaches, our approach generalizes well to unseen fields and not only substantially improves the stem detection accuracy, that is, distinguishing crop and weed stems, but also improves the semantic segmentation performance.     

Optimization of scale and parametrization for terrain segmentation: An application to soil-landscape modeling

This paper presents a procedure to optimize parametrization and scale for terrain-based environmental modeling. The workflow was exemplified on crop yield data, which is assumed to represent a proxy for soil productivity. Focal mean statistics were used to generate different scale levels of terrain derivatives by increasing the neighborhood size in calculation. The degree of association between each terrain derivative and crop yield values was established iteratively for all scale levels through correlation analysis. The first peak of correlation indicated the scale level to be further retained. To select the best combination of terrain parameters that explains the variation of crop yield, we ran stepwise multiple regressions with appropriately scaled terrain parameters as independent variables. These techniques proved that the mean curvature, filtered over a neighborhood of 55 m, together with slope, made up the optimal combination to account for patterns of soil productivity.To illustrate the importance of scale, we compared the regression results of unfiltered and filtered mean curvature vs. crop yield. The comparison shows an improvement of R² from a value of 0.01 when the curvature was not filtered, to 0.16 when the curvature was filtered within 55×55 m neighborhood size.The results were further used in an object-based image analysis environment to create terrain objects containing aggregated values of both terrain derivatives and crop yield. Hence, we introduce terrain segmentation as an alternative method for generating scale levels in terrain-based environmental modeling, besides existing per-cell methods. At the level of segments, R² improved up to a value of 0.47.     

Fusion of FISH image analysis methods of HER2 status determination in breast cancer

The paper presents an application of several methods of markers localization in fluorescent in situ hybridization images in order to determine HER2 status of the breast cancer samples. The applied methods implement different mathematical foundations of image processing and perform the analysis in an independent way. The paper proposes their fusion to obtain better results of spot recognition. The integration process of many methods takes into account different points of view on the classification problem and takes the best from each of them in developing the final solution. Thanks to this the improvement of the performance of the system is achieved. The efficiency of the developed system in the form of sensitivity and specificity has been verified on the examples of analysis of many FISH images. They confirm the advantage of using several methods of image processing combined in the form of an ensemble.     

条件生成对抗网络的低照度图像增强方法

目的 在日常的图像采集工作中,由于场景光照条件差或设备的补光能力不足,容易产生低照度图像。为了解决低照度图像视觉感受差、信噪比低和使用价值低（难以分辨图像内容）等问题,本文提出一种基于条件生成对抗网络的低照度图像增强方法。方法 本文设计一个具备编解码功能的卷积神经网络（CNN）模型作为生成模型,同时加入具备二分类功能的CNN作为判别模型,组成生成对抗网络。在模型训练的过程中,以真实的亮图像为条件,依靠判别模型监督生成模型以及结合判别模型与生成模型间的相互博弈,使得本文网络模型具备更好的低照度图像增强能力。在本文方法使用过程中,无需人工调节参数,图像输入模型后端到端处理并输出结果。结果 将本文方法与现有方法进行比较,利用本文方法增强的图像在亮度、清晰度以及颜色还原度等方面有了较大的提升。在峰值信噪比、直方图相似度和结构相似性等图像质量评价指标方面,本文方法比其他方法的最优值分别提高了0.7 dB、3.9%和8.2%。在处理时间上,本文方法处理图像的速度远远超过现有的传统方法,可达到实时增强的要求。结论 通过实验比较了本文方法与现有方法对于低照度图像的处理效果,表明本文方法具有更优的处理效果,同时具有更快的处理速度。     

A novel super-resolution method of PolSAR images based on target decomposition and polarimetric spatial correlation

The polarimetric synthetic aperture radar (PolSAR) is becoming more and more popular in remote-sensing research areas. However, due to system limitations, such as bandwidth of the signal and the physical dimension of antennas, the resolution of PolSAR images cannot be compared with those of optical remote-sensing images. Super-resolution processing of PolSAR images is usually desired for PolSAR image applications, such as image interpretation and target detection. Usually, in a PolSAR image, each resolution contains several different scattering mechanisms. If these mechanisms can be allocated to different parts within one resolution cell, details of the images can be enhanced, which that means the resolution of the images is improved. In this article, a novel super-resolution algorithm for PolSAR images is proposed, in which polarimetric target decomposition and polarimetric spatial correlation are both taken into consideration. The super-resolution method, based on polarimetric spatial correlation (SRPSC), can make full use of the polarimetric spatial correlation to allocate different scattering mechanisms of PolSAR images. The advantage of SRPSC is that the phase information can be preserved in the processed PolSAR images. The proposed methods are demonstrated with the German Aerospace Center (DLR) Experimental SAR (E-SAR) L-band full polarized images of the Oberpfaffenhofen Test Site Area in Germany, obtained on 30 September 2000. The experimental results of the SRPSC confirms the effectiveness of the proposed methods.¹     

Automatic expert system based on images for accuracy crop row detection in maize fields

This paper proposes an automatic expert system for accuracy crop row detection in maize fields based on images acquired from a vision system. Different applications in maize, particularly those based on site specific treatments, require the identification of the crop rows. The vision system is designed with a defined geometry and installed onboard a mobile agricultural vehicle, i.e. submitted to vibrations, gyros or uncontrolled movements. Crop rows can be estimated by applying geometrical parameters under image perspective projection. Because of the above undesired effects, most often, the estimation results inaccurate as compared to the real crop rows. The proposed expert system exploits the human knowledge which is mapped into two modules based on image processing techniques. The first one is intended for separating green plants (crops and weeds) from the rest (soil, stones and others). The second one is based on the system geometry where the expected crop lines are mapped onto the image and then a correction is applied through the well-tested and robust Theil–Sen estimator in order to adjust them to the real ones. Its performance is favorably compared against the classical Pearson product–moment correlation coefficient.     

Selecting patterns and features for between- and within- crop-row weed mapping using UAV-imagery

This paper approaches the problem of weed mapping for precision agriculture, using imagery provided by Unmanned Aerial Vehicles (UAVs) from sunflower and maize crops. Precision agriculture referred to weed control is mainly based on the design of early post-emergence site-specific control treatments according to weed coverage, where one of the most important challenges is the spectral similarity of crop and weed pixels in early growth stages. Our work tackles this problem in the context of object-based image analysis (OBIA) by means of supervised machine learning methods combined with pattern and feature selection techniques, devising a strategy for alleviating the user intervention in the system while not compromising the accuracy. This work firstly proposes a method for choosing a set of training patterns via clustering techniques so as to consider a representative set of the whole field data spectrum for the classification method. Furthermore, a feature selection method is used to obtain the best discriminating features from a set of several statistics and measures of different nature. Results from this research show that the proposed method for pattern selection is suitable and leads to the construction of robust sets of data. The exploitation of different statistical, spatial and texture metrics represents a new avenue with huge potential for between and within crop-row weed mapping via UAV-imagery and shows good synergy when complemented with OBIA. Finally, there are some measures (specially those linked to vegetation indexes) that are of great influence for weed mapping in both sunflower and maize crops.     

基于多时相Sentinel-2影像的黑河中游玉米种植面积提取研究

玉米是黑河中游种植面积最大的农作物,生长期需水量大、蒸散量高.准确获取玉米种植面积对该区域农作物种植结构调整、水资源合理规划有重要参考意义.基于2019年4月至9月Sentinel-2多时相影像,采用随机森林算法开展了黑河中游玉米种植面积提取研究.研究方法分为两类—直接提取法和两步提取法.进一步探讨了多时间信息量对玉米...     

Hidden Markov Models for crop recognition in remote sensing image sequences

This work proposes a Hidden Markov Model (HMM) based technique to classify agricultural crops. The method uses HMM to relate the varying spectral response along the crop cycle with plant phenology, for different crop classes, and recognizes different agricultural crops by analyzing their spectral profiles over a sequence of images. The method assigns each image segment to the crop class whose corresponding HMM delivers the highest probability of emitting the observed sequence of spectral values. Experimental analysis was conducted upon a set of 12 co-registered and radiometrically corrected LANDSAT images of region in southeast Brazil, of approximately 124.100 ha, acquired between 2002 and 2004. Reference data was provided by visual classification, validated through extensive field work. The HMM-based method achieved 93% average class accuracy in the identification of the correct crop, being, respectively, 10% and 26% superior to multi-date and single-date alternative approaches applied to the same data set.     

基于级联卷积神经网络的作物病害叶片分割

针对传统卷积神经网络在作物病害叶片图像中分割精度低的问题,提出一种基于级联卷积神经网络（Cascade Convolutional Neural Network,CCNN）的作物病害叶片图像分割方法。该网络由区域病斑检测网络和区域病斑分割网络组成。基于传统VGG16模型构建区域病斑检测网络（Regional Detection Network,RD-net）,利用全局池化层代替全连接层,由此减少模型参数,实现叶片病斑区域精确定位。基于Encoder-Decoder模型结构建立区域分割网络（Regional Segmentation Network,RS-net）,并利用多尺度卷积核提高原始卷积核的局部感受野,对病斑区域精确分割。在不同环境下的病害叶片图像上进行分割实验,分割精度为87.04%、召回率为78.31%、综合评价指标值为88.22%、单幅图像分割速度为0.23?s。实验结果表明该方法能够满足不同环境下的作物病害叶片图像分割需求,可为进一步的作物病害识别方法研究提供参考。     

CHAIR: automatic image registration based on correlation and Hough transform

Automatic image registration is a process related to several application fields: remote sensing, medicine and computer vision, among others. Particularly in the field of remote sensing, the ever-increasing number of available satellite images requires automatic image registration methods, capable of correctly aligning a new image. An automatic image registration method – CHAIR (correlation- and Hough transform-based method of automatic image registration) – is proposed, the key concept of which relies on the ‘correlation image’ produced in both the horizontal and vertical directions. In particular, the computation of the distance of an identified diagonal brighter strip in the correlation image (through the Hough transform) to an offset (the main diagonal) allows for the determination of translational shifts and consequently control points. The set of obtained control points allows for the correction of several types of distortions. The geometric correction quality achieved by CHAIR was objectively evaluated through measures recently proposed, which allow for a more complete assessment of the obtained results. The CHAIR performance was evaluated on both synthetic and real data, with different spatial resolutions and spectral contents. CHAIR has been shown to be able to correctly align two images with a subpixel accuracy, having a priori a ‘gold standard’ image covering a considerable part of the image to be registered, and has also been shown to work for images of different sensors and/or different spectral bands, situations where traditional correlation methods often yield low and smooth peaks on the correlation surface. It is also able to account for elevation differences and to some extent for rotation and scale effects. Furthermore, it has been shown to have potential for registering synthetic aperture radar (SAR) with optical images.     

Multisource ERS-1 and optical data for vegetal cover assessment and monitoring in a semi-arid region of Algeria

This paper highlights advantages of using Synthetic Aperture Radar (SAR) data combined with multispectral data to improve vegetal cover assessment and monitoring in a semi-arid region of southern Algeria. We present a number of preprocessing and processing techniques using multidate optical data analysis alone and SAR ERS-1 and Landsat Thematic Mapper (TM) data integration due to aspects of radar image enhancement techniques and the study of roughness of different types of vegetation in steppic regions. Image data integration has become a valuable approach to integrate multisource satellite data. It has been found that image data from different spectral domains (visible, near-infrared, microwave) provides datasets with complementarity information content and can be used to improve the spatial resolution of satellite images. In this communication, we present a part of the cooperation research project which deals with fusing ERS-1 SAR geocoded images with Landsat TM data, investigating different combinations of integration and classification techniques. The methodology consists of several steps: (1) Speckle noise reduction by comparative performance of different filtering algorithms. Several filtering algorithms were implemented and tested with different window sizes, iterations and parameters. (2) Geometric superposition and geocoding of optical images regarding SAR ERS-1 image and resampling at unique resolution of 25 m. (3) Application of different numerical combinations of integration techniques and unsupervized classifications such as the Forgy method, the MacQueen method and other methods. The results were compared with vegetal cover mapping from aerial photographs of the region of Foum Redad in the south of the Saharian Atlas. The combinations proposed above allow us to distinguish different themes in the arid and semi-arid regions in the south of the Saharian Atlas using a colour composite image and show a good correlation between different types of land cover and land use and radar backscattering level in the SAR data which corresponds essentially to the roughness of the soil surface.     

利用航空成像光谱数据进行冬小麦产量预测

以国产成像光谱仪PHI(Pushbroom Hyperspectral Imaget)所获遥感影像数据为基础，根据田间冬小麦单产遥感研究试验数据建立了研究区不同时相冬小麦单产预测模型，实现了利用航空高光谱遥感数据对研究区小麦产量的整体预测；对试验区土壤氮素水平与不同时相冬小麦预测产量以及试验区实测产量进行了初步分析，分析结果显示：土壤氮素分布的差异性对小麦的产量有明显影响。     

Multispectral processing for color particle image velocimetry

Since the adoption of digital video cameras and cross-correlation methods for particle image velocimetry (PIV), the use of color images has largely been abandoned. Recently, however, with the re-emergence of color-based stereo and volumetric techniques, and the extensive use of color microscopy, color imaging for PIV has again become relevant. In this work, we explore the potential advantages of color PIV processing by developing and proposing new methods for handling multi-color images. The first method uses cross-correlation of every color channel independently to build a color vector cross-correlation plane. The vector cross-correlation can then be searched for one or more peaks corresponding to either the average displacement of several flow components using a color ensemble operation, or for the individual motion of colored particles, each with a different behavior. In the latter case, linear unmixing is used on the correlation plane to separate each known particle type as captured by the different color channels. The second method introduces the use of quaternions to encode the color data, and the cross-correlation is carried out simultaneously on all colors. The resulting correlation plane can be searched either for a single peak, corresponding to the mean flow or for multiple peaks, with velocity phase separation to determine which velocity corresponds to which particle type. Each of these methods was tested using synthetic images simulating the color recording of noisy particle fields both with and without the use of a Bayer filter and demosaicing operation. It was determined that for single-phase flow, both color methods decreased random errors by approximately a factor of two due to the noise signal being uncorrelated between color channels, while maintaining similar bias errors as compared to traditional monochrome PIV processing. In multi-component flows, the color vector correlation technique was able to successfully resolve displacements of two distinct yet coupled flow components with errors similar to traditional grayscale PIV processing of a single phase. It should be noted that traditional PIV processing is bound to fail entirely under such processing conditions. In contrast, the quaternion methods frequently failed to properly identify the correct velocity and phase and showed significant cross talk in the measurements between particle types. Finally, the color vector method was applied to experimental color images of a microchannel designed for contactless dielectrophoresis particle separation, and good results were obtained for both instantaneous and ensemble PIV processing. However, in both the synthetic color images that were generated using a Bayer filter and the experimental data, a significant peak-locking effect with a period of two pixels was observed. This effect is attributed to the inherent architecture of the Bayer filter. In order to mitigate this detrimental artifact, it is suggested that improved image interpolation or demosaicing algorithms tuned for use in PIV be developed and applied on the color images before processing, or that cameras that do not use a Bayer filter and therefore do not require a demosaicing algorithm be used for color PIV.     

基于深度学习的白菜田杂草分割

田间除草在农业生产中具有重要意义,传统杂草识别方式具有效率低或者局限性大的缺点。为此提出将Mask R-CNN算法应用到自然光照下杂草幼苗和白菜幼苗的图像识别。实验选取常见杂草幼苗和白菜幼苗图像作为训练集训练网络,经测试集测试得到81%的合格率。与传统阈值分割算法对比,Mask R-CNN在不同环境下都能精确地识别出杂草幼苗,解决了传统图像算法在复杂光照和叶片遮挡环境下图像难以分割的问题,并且避免了分类器设置工作。     

A semi-supervised system for weed mapping in sunflower crops using unmanned aerial vehicles and a crop row detection method

This paper presents a system for weed mapping, using imagery provided by unmanned aerial vehicles (UAVs). Weed control in precision agriculture is based on the design of site-specific control treatments according to weed coverage. A key component is precise and timely weed maps, and one of the crucial steps is weed monitoring, by ground sampling or remote detection. Traditional remote platforms, such as piloted planes and satellites, are not suitable for early weed mapping, given their low spatial and temporal resolutions. Nonetheless, the ultra-high spatial resolution provided by UAVs can be an efficient alternative. The proposed method for weed mapping partitions the image and complements the spectral information with other sources of information. Apart from the well-known vegetation indexes, which are commonly used in precision agriculture, a method for crop row detection is proposed. Given that crops are always organised in rows, this kind of information simplifies the separation between weeds and crops. Finally, the system incorporates classification techniques for the characterisation of pixels as crop, soil and weed. Different machine learning paradigms are compared to identify the best performing strategies, including unsupervised, semi-supervised and supervised techniques. The experiments study the effect of the flight altitude and the sensor used. Our results show that an excellent performance is obtained using very few labelled data complemented with unlabelled data (semi-supervised approach), which motivates the use of weed maps to design site-specific weed control strategies just when farmers implement the early post-emergence weed control.     

Al-Thocb HSC: a harmony search algorithm for automated calibration of industrial equipment

Calibration is an essential task for setting up camera parameters, especially when cameras are used for industrial applications like object recognition and picking that require a fine-grained location of the observed object. However, this process is time-consuming and requires specific image processing skills, which are not always available: an operator often needs to use the equipment rapidly without costly setup operations. The calibration system needs a coherent set of images of a given model, called a mire, positioned in different ways. In this paper, we propose to automate and to optimize the calibration system by eliminating the requirement for the user to select a suitable set of images. Thus, an optimized calibration can be obtained in a minimum of time. First, we propose to retrieve the set of points of each input image in order to avoid a renewed search at each calibration. Second, we define Al-Thocb, a Harmony Search Calibration algorithm, based on Harmony Search Optimization. The algorithm optimizes the selection of the best images. The satisfaction criterion is defined by a fitness function based on the projection error. The method allows to retrieve coherent camera parameters with no need for specific user skills. It also significantly improves the accuracy of calibration through the use of the reprojection error as fitness function. To demonstrate the applicability of Al-Thocb, we evaluate the accuracy and the responsiveness of the proposed algorithm and compare it to other existing methods.     

多通道图像EMD及应用

针对现有的经验模态分解方法（Empirical Mode Decomposition,EMD）对多通道图像（如彩色图像）进行分解时通常忽略各通道图像之间相关性的问题,提出了一种多通道图像EMD方法。该方法采用双拉普拉斯算子插值得到图像上下包络,并建立一个整体筛分停止准则进行筛分来考虑各通道图像相关性,能够将多通道图像自适应分解为数目不多的内蕴模态函数（Intrinsic Mode Function,IMF）分量和一个余量,其中内蕴模态函数分量体现了原始图像不同尺度的特征信息,余量体现了图像的整体变化趋势。该方法可以应用在图像锐化、夜景图像增强等图像分析和处理领域。实验结果显示该方法能够取得较好的效果。