The effect of data augmentation and network simplification on the image‐based detection of broccoli heads with Mask R‐CNN

In current practice, broccoli heads are selectively harvested by hand. The goal of our work is to develop a robot that can selectively harvest broccoli heads, thereby reducing labor costs. An essential element of such a robot is an image‐processing algorithm that can detect broccoli heads. In this study, we developed a deep learning algorithm for this purpose, using the Mask Region‐based Convolutional Neural Network. To be applied on a robot, the algorithm must detect broccoli heads from any cultivar, meaning that it can generalize on the broccoli images. We hypothesized that our algorithm can be generalized through network simplification and data augmentation. We found that network simplification decreased the generalization performance, whereas data augmentation increased the generalization performance. In data augmentation, the geometric transformations (rotation, cropping, and scaling) led to a better image generalization than the photometric transformations (light, color, and texture). Furthermore, the algorithm was generalized on a broccoli cultivar when 5% of the training images were images of that cultivar. Our algorithm detected 229 of the 232 harvestable broccoli heads from three cultivars. We also tested our algorithm on an online broccoli data set, which our algorithm was not previously trained on. On this data set, our algorithm detected 175 of the 176 harvestable broccoli heads, proving that the algorithm was successfully generalized. Finally, we performed a cost‐benefit analysis for a robot equipped with our algorithm. We concluded that the robot was more profitable than the human harvest and that our algorithm provided a sufficient basis for robot commercialization.     

Crop recognition under weedy conditions based on 3D imaging for robotic weed control

A 3D time‐of‐flight camera was applied to develop a crop plant recognition system for broccoli and green bean plants under weedy conditions. The developed system overcame the previously unsolved problems caused by occluded canopy and illumination variation. An efficient noise filter was developed to remove the sparse noise points in 3D point cloud space. Both 2D and 3D features including the gradient of amplitude and depth image, surface curvature, amplitude percentile index, normal direction, and neighbor point count in 3D space were extracted and found effective for recognizing these two types of plants. Separate segmentation algorithms were developed for each of the broccoli and green bean plant in accordance with their 3D geometry and 2D amplitude characteristics. Under the experimental condition where the crops were heavily infested by various types of weed plants, detection rates over 88.3% and 91.2% were achieved for broccoli and green bean plant leaves, respectively. Additionally, the crop plants were segmented out with nearly complete shape. Moreover, the algorithms were computationally optimized, resulting in an image processing speed of over 30 frames per second.     

Exact and heuristic algorithms for balancing transfer lines when a set of available spindle heads is given

A balancing problem for paced tandem transfer lines with several spindle heads at each station is considered. A spindle head executes a block of operations. The set of all available spindle heads as well as the operations executed by each spindle head, the spindle head times and costs are known. There are operations with several spindle head candidates. The problem at the line design stage consists in the choice of spindle heads from the given set and their assignment to workstations. The goal is to minimize the line cost while satisfying the precedence, inclusion and exclusion constraints. An exact algorithm based on a mixed integer programming approach is developed. Two types of new heuristic algorithms are also suggested. One of them step‐by‐step assigns randomly spindle heads to a current workstation. The second uses depth‐first search techniques. Experimental results are reported.     

Consistent Video Filtering for Camera Arrays

Visual formats have advanced beyond single‐view images and videos: 3D movies are commonplace, researchers have developed multi‐view navigation systems, and VR is helping to push light field cameras to mass market. However, editing tools for these media are still nascent, and even simple filtering operations like color correction or stylization are problematic: naively applying image filters per frame or per view rarely produces satisfying results due to time and space inconsistencies. Our method preserves and stabilizes filter effects while being agnostic to the inner working of the filter. It captures filter effects in the gradient domain, then uses input frame gradients as a reference to impose temporal and spatial consistency. Our least‐squares formulation adds minimal overhead compared to naive data processing. Further, when filter cost is high, we introduce a filter transfer strategy that reduces the number of per‐frame filtering computations by an order of magnitude, with only a small reduction in visual quality. We demonstrate our algorithm on several camera array formats including stereo videos, light fields, and wide baselines.     

CubeALS:新的三维协同过滤推荐算法

随着互联网的快速发展,人们对个性化网页搜索、个性化广告投放、个性化社会标注等三维推荐服务的需求越来越紧迫。这些三维立方体数据高度稀疏,且与二维推荐系统相比三维推荐系统中对象之间的关系更加复杂。为了更好地模拟三维对象之间的关系并解决三维数据高度稀疏的问题,提出了一种新的三维协同过滤推荐算法CubeALS(cube alternating least squares)。该算法对三维协同过滤推荐算法CubeSVD(cube singular value decomposition)进行了改进,尝试使用不同于SVD的算法进行矩阵分解。在真实的个性化社会标注数据集上的实验结果表明,与CubeSVD算法相比,CubeALS的性能得到了显著提高。     

Driving on Point Clouds: Motion Planning,Trajectory Optimization,and Terrain Assessment in Generic Nonplanar Environments

We present a practical approach to global motion planning and terrain assessment for ground robots in generic three‐dimensional (3D) environments, including rough outdoor terrain, multilevel facilities, and more complex geometries. Our method computes optimized six‐dimensional trajectories compliant with curvature and continuity constraints directly on unordered point cloud maps, omitting any kind of explicit surface reconstruction, discretization, or topology extraction. We assess terrain geometry and traversability on demand during motion planning, by fitting robot‐sized planar patches to the map and analyzing the local distribution of map points. Our motion planning approach consists of sampling‐based initial trajectory generation, followed by precise local optimization according to a custom cost measure, using a novel, constraint‐aware trajectory optimization paradigm. We embed these methods in a complete autonomous navigation system based on localization and mapping by means of a 3D laser scanner and iterative closest point matching, suitable for both static and dynamic environments. The performance of the planning and terrain assessment algorithms is evaluated in offline experiments using recorded and simulated sensor data. Finally, we present the results of navigation experiments in three different environments—rough outdoor terrain, a two‐level parking garage, and a dynamic environment, demonstrating how the proposed methods enable autonomous navigation in complex 3D terrain.     

An autonomous strawberry‐harvesting robot: Design,development, integration,and field evaluation

This paper presents an autonomous robot capable of picking strawberries continuously in polytunnels. Robotic harvesting in cluttered and unstructured environment remains a challenge. A novel obstacle‐separation algorithm was proposed to enable the harvesting system to pick strawberries that are located in clusters. The algorithm uses the gripper to push aside surrounding leaves, strawberries, and other obstacles. We present the theoretical method to generate pushing paths based on the surrounding obstacles. In addition to manipulation, an improved vision system is more resilient to lighting variations, which was developed based on the modeling of color against light intensity. Further, a low‐cost dual‐arm system was developed with an optimized harvesting sequence that increases its efficiency and minimizes the risk of collision. Improvements were also made to the existing gripper to enable the robot to pick directly into a market punnet, thereby eliminating the need for repacking. During tests on a strawberry farm, the robots first‐attempt success rate for picking partially surrounded or isolated strawberries ranged from 50% to 97.1%, depending on the growth situations. Upon an additional attempt, the pick success rate increased to a range of 75–100%. In the field tests, the system was not able to pick a target that was entirely surrounded by obstacles. This failure was attributed to limitations in the vision system as well as insufficient dexterity in the grippers. However, the picking speed improved upon previous systems, taking just 6.1 s for manipulation operation in the one‐arm mode and 4.6 s in the two‐arm mode.     

Breast cancer detection using K‐nearest neighbors data mining method obtained from the bow‐tie antenna dataset

Breast cancer, has been a significant cancer type for women on the society. Early diagnosis and timely medical treatment are important key factors spreading to the other tissues and permitting long‐time survival of patients. Since the existing methods have several serious shortfalls, microwave imaging method for the diagnosis of early stage tumors has been interested by different scientific research groups in terms of moderating endogenous the electrical property difference between healthy tissue and malignancies. In this article, both an ultra‐wideband bow‐tie antenna with enhanced bandwidth and a 3D breast model which has different electrical properties which are permittivity and conductivity is created in simulation tool to solve electromagnetic field values. Return loss, VSWR, and radiation pattern characteristics, which are significant antenna parameters, are simulated and obtained whether the antenna possess an efficient characteristic or not. Electric field values over the breast tissue in which there is a tumor or not tumor are evaluated. In this article, above‐mentioned values of frequency bandwidth, dielectric constant of antenna's substrate, electric field, and tumor information were consisted in their dataset. This dataset obtained from the Bow‐Tie Antenna was used to detect the breast cancer with one of the data mining method, which is K‐Nearest Neighbor Algorithm.     

Real-Time Accurate 3D Head Tracking and Pose Estimation with Consumer RGB-D Cameras

We demonstrate how 3D head tracking and pose estimation can be effectively and efficiently achieved from noisy RGB-D sequences. Our proposal leverages on a random forest framework, designed to regress the 3D head pose at every frame in a temporal tracking manner. One peculiarity of the algorithm is that it exploits together (1) a generic training dataset of 3D head models, which is learned once offline; and, (2) an online refinement with subject-specific 3D data, which aims for the tracker to withstand slight facial deformations and to adapt its forest to the specific characteristics of an individual subject. The combination of these works allows our algorithm to be robust even under extreme poses, where the user’s face is no longer visible on the image. Finally, we also propose another solution that utilizes a multi-camera system such that the data simultaneously acquired from multiple RGB-D sensors helps the tracker to handle challenging conditions that affect a subset of the cameras. Notably, the proposed multi-camera frameworks yields a real-time performance of approximately 8 ms per frame given six cameras and one CPU core, and scales up linearly to 30 fps with 25 cameras.     

不均匀簇无线传感器网络高效节能通讯协议

把无线传感器网络中的节点组织成簇能够有效的利用节点有限能量获得较长生命周期。传统随机成簇策略因忽视簇头负载差异而使能耗较高。针对这一问题,提出了基于不均匀簇的无线传感器网络高效节能数据通讯协议（Unequal Cluster size Protocol,UCP）在成簇时将离基站较近的簇分配少的簇内节点,而将离基站较远的簇分配更多的簇内节点。这样,离基站较远的簇头节点相比离基站较近的簇头节点在数据聚合时会消耗更多的能量,但是由于离基站近的簇不仅要聚合簇内节点的数据,而且还要转发离基站较远的簇头聚合后的数据,相比之下,能量的消耗会趋近平衡,而总体上每轮消耗的能量会更少。实验结果显示,UCP性能比MIT提出的LEACH协议性能好13%-16%。     

Stereo vision during adverse weather — Using priors to increase robustness in real-time stereo vision

Stereo vision can deliver a dense 3D reconstruction of the environment in real-time for driver assistance as well as autonomous driving. Semi-Global Matching (SGM) is a popular method of choice for solving this task which is already in use for production vehicles. Despite the enormous progress in the field and the high level of performance of modern stereo methods, one key challenge remains: robust stereo vision in automotive scenarios during rain, snow and darkness. Under these circumstances, current methods generate strong temporal noise, many disparity outliers and false positives on object level. These problems are addressed in this work by regularizing stereo vision via prior information. We formulate a temporal prior and a scene prior, which we apply to SGM in order to overcome the deficiencies. The temporal prior integrates knowledge from the previous disparity map to exploit the high temporal correlation, the scene prior exploits knowledge of a representative traffic scene. Using these priors, the object detection rate improves significantly on a driver assistance dataset of 3000 frames including bad weather while reducing the rate of erroneous object detections. We also outperform the ECCV Robust Vision Challenge 2012 winner, iSGM, on this dataset. In addition, results are presented for the KITTI dataset, even showing improvements under good weather conditions when exploiting the temporal prior.We also show that the temporal and scene priors are easy and efficient to implement on a hybrid CPU/reconfigurable hardware platform. The use of these priors can be extended to other application areas such as mobile robotics.     

Evaluation of jerkiness of moving three‐dimensional images produced by high‐density directional display

Abstract— The jerkiness of moving three‐dimensional (3‐D) images produced by a high‐density directional display was studied. Under static viewing conditions in which subjects' heads did not move, jerkiness was more noticeable when moving 3‐D images were displayed in front of the display screen and was less noticeable when moving 3‐D images were displayed behind the screen. We found that the perception of jerkiness depended on the visual angular velocities of moving 3‐D images. Under dynamic viewing conditions in which subjects' heads were forced to move, when moving 3‐D images were displayed in front of the screen, jerkiness was less noticeable when the subjects' heads and 3‐D images moved in opposite directions and was more noticeable when they moved in the same direction. When moving 3‐D images were displayed behind the screen, jerkiness was less noticeable when subjects' heads and 3‐D images moved in the same direction and was more noticeable when they moved in opposite directions.     

基于多模态融合的三维模型检索算法研究

为了获得更好的三维模型检索分类性能,基于深度学习模型研究了多模态信息融合对三维模型的特征描述,在训练步骤提出相关性损失函数来指导不同模态之间的训练,提取更稳健的特征向量;最后将融合特征应用于三维模型的检索和分类,在ModelNet40数据集上进行了三维模型分类任务和检索任务评估。实验结果及与现有方法进行的对比证明了该方法的优越性,为三维模型检索分类领域提供了一种新的思路。     

Automated crop plant detection based on the fusion of color and depth images for robotic weed control

Robotic weeding enables weed control near or within crop rows automatically, precisely and effectively. A computer‐vision system was developed for detecting crop plants at different growth stages for robotic weed control. Fusion of color images and depth images was investigated as a means of enhancing the detection accuracy of crop plants under conditions of high weed population. In‐field images of broccoli and lettuce were acquired 3–27 days after transplanting with a Kinect v2 sensor. The image processing pipeline included data preprocessing, vegetation pixel segmentation, plant extraction, feature extraction, feature‐based localization refinement, and crop plant classification. For the detection of broccoli and lettuce, the color‐depth fusion algorithm produced high true‐positive detection rates (91.7% and 90.8%, respectively) and low average false discovery rates (1.1% and 4.0%, respectively). Mean absolute localization errors of the crop plant stems were 26.8 and 7.4 mm for broccoli and lettuce, respectively. The fusion of color and depth was proved beneficial to the segmentation of crop plants from background, which improved the average segmentation success rates from 87.2% (depth‐based) and 76.4% (color‐based) to 96.6% for broccoli, and from 74.2% (depth‐based) and 81.2% (color‐based) to 92.4% for lettuce, respectively. The fusion‐based algorithm had reduced performance in detecting crop plants at early growth stages.     

Pixel History Linear Models for Real‐Time Temporal Filtering

We propose a new real‐time temporal filtering and antialiasing (AA) method for rasterization graphics pipelines. Our method is based on Pixel History Linear Models (PHLM), a new concept for modeling the history of pixel shading values over time using linear models. Based on PHLM, our method can predict per‐pixel variations of the shading function between consecutive frames. This combines temporal reprojection with per‐pixel shading predictions in order to provide temporally coherent shading, even in the presence of very noisy input images. Our method can address both spatial and temporal aliasing problems under a unique filtering framework that minimizes filtering error through a recursive least squares algorithm. We demonstrate our method working with a commercial deferred shading engine for rasterization and with our own OpenGL deferred shading renderer. We have implemented our method in GPU and it has shown significant reduction of temporal flicker in very challenging scenarios including foliage rendering, complex non‐linear camera motions, dynamic lighting, reflections, shadows and fine geometric details. Our approach, based on PHLM, avoids the creation of visible ghosting artifacts and it reduces the filtering overblur characteristic of temporal deflickering methods. At the same time, the results are comparable to state‐of‐the‐art real‐time filters in terms of temporal coherence.     

Development of a 3D HUD using a tunable bandpass filter for wavelength multiplexing

A 3D stereoscopic head‐up display using a tunable bandpass filter to perform left and right image spectral separation is presented. Using a single filter reduces the size and the cost of the head‐up display optical engine and enables each spectral band to be accurately tuned. Experiments performed on the first prototype demonstrate the ability to continuously tune the bandpass frequency on 30‐nm range while keeping a 20‐nm bandwidth. Such a system avoids the use of a bulky and costly rotating wheel and enables the use of holographic optical elements known to be wavelength selective.     

基于RNN-LSTM的磨矿系统故障诊断技术

目前磨矿系统故障诊断多为人为判断，效率低、准确率低、成本高且容易造成人员伤亡.传统方法对高维度和时间相关性较大的样本数据集分类能力较差，针对以上问题，提出一种基于RNN-LSTM（Recurrent Neural Network-Long Short-Term Memory）的深度学习方法，实现磨矿系统故障的智能化诊断.该方法通过将数据集"分批处理"分别输入到LSTM单元网络中，提取数据集在时间维度上的相关性，并比较分析前后时刻的输入特征向量实现对故障分类.通过分别对RNN-LSTM深度学习网络与基于自编码分类方法进行实验对比验证，得出结论：在时间相关性较强的高维度数据集中基于RNN-LSTM深度方法辨识效果明显优于基于自编码方法的分类器，最终网络对于故障诊断的错误率低至3%.     

GeoNet: An open source software for the automatic and objective extraction of channel heads,channel network,and channel morphology from high resolution topography data

Extracting hydrologic and geomorphic features from high resolution topography data is a challenging and computationally demanding task. We illustrate the new capabilities and features of GeoNet, an open source software for the extraction of channel heads, channel networks, and channel morphology from high resolution topography data. The method has been further developed and includes a median filtering operation to remove roads in engineered landscapes and the calculation of hillslope lengths to inform the channel head identification procedure. The software is now available in both MATLAB and Python, allowing it to handle datasets larger than the ones previously analyzed. We present the workflow of GeoNet using three different test cases; natural high relief, engineered low relief, and urban landscapes. We analyze default and user-defined parameters, provide guidance on setting parameter values, and discuss the parameter effect on the extraction results. Metrics on computational time versus dataset size are also presented. We show the ability of GeoNet to objectively and accurately extract channel features in terrains of various characteristics.     

3‐D crosstalk and luminance uniformity from angular luminance profiles of multiview autostereoscopic 3‐D displays

Abstract— Autostereoscopic 3‐D display technologies enable a more immersive media experience by adding real depth to the visual content. However, the method used for the creation of a sensation of depth or stereo illusion contains several display design and content‐related issues that need to be carefully considered to maintain sufficient image quality. Conventionally, methods used for 3‐D image‐quality evaluations have been based on subjective testing. Optical measurements, in addition to subjective testing, can be used as an efficient tool for 3‐D display characterization. Objective characterization methods for autostereoscopic displays have been developed. How parameters affecting stereo image quality can be defined and measured, and how their effect on the stereo image quality can be evaluated have been investigated. Developed characterization methods are based on empirically gathered data. In this paper, previously presented methodology for two‐view displays is extended to cover autostereoscopic multiview displays. A distinction between displays where the change in content occurs in clear steps when the user moves in front of the display, and displays where the apparent movement of the objects is more continuous as a function of the head movement is made. Definitions for 3‐D luminance and luminance uniformity, which are equally important, as well as 3‐D crosstalk, which is the dominant factor in the evaluations of the perceived 3‐D image quality, is focused upon.