3D‐vision based detection,localization, and sizing of broccoli heads in the field

This paper describes a 3D vision system for robotic harvesting of broccoli using low‐cost RGB‐D sensors, which was developed and evaluated using sensory data collected under real‐world field conditions in both the UK and Spain. The presented method addresses the tasks of detecting mature broccoli heads in the field and providing their 3D locations relative to the vehicle. The paper evaluates different 3D features, machine learning, and temporal filtering methods for detection of broccoli heads. Our experiments show that a combination of Viewpoint Feature Histograms, Support Vector Machine classifier, and a temporal filter to track the detected heads results in a system that detects broccoli heads with high precision. We also show that the temporal filtering can be used to generate a 3D map of the broccoli head positions in the field. Additionally, we present methods for automatically estimating the size of the broccoli heads, to determine when a head is ready for harvest. All of the methods were evaluated using ground‐truth data from both the UK and Spain, which we also make available to the research community for subsequent algorithm development and result comparison. Cross‐validation of the system trained on the UK dataset on the Spanish dataset, and vice versa, indicated good generalization capabilities of the system, confirming the strong potential of low‐cost 3D imaging for commercial broccoli harvesting.     

Patch‐Collaborative Spectral Point‐Cloud Denoising

We present a new framework for point cloud denoising by patch‐collaborative spectral analysis. A collaborative generalization of each surface patch is defined, combining similar patches from the denoised surface. The Laplace–Beltrami operator of the collaborative patch is then used to selectively smooth the surface in a robust manner that can gracefully handle high levels of noise, yet preserves sharp surface features. The resulting denoising algorithm competes favourably with state‐of‐the‐art approaches, and extends patch‐based algorithms from the image processing domain to point clouds of arbitrary sampling. We demonstrate the accuracy and noise‐robustness of the proposed algorithm on standard benchmark models as well as range scans, and compare it to existing methods for point cloud denoising.     

Out‐of‐Core Simplification and Crack‐Free LOD Volume Rendering for Irregular Grids

We propose a novel out‐of‐core simplification and level‐of‐detail (LOD) volume rendering algorithm for large irregular grids represented as tetrahedral meshes. One important feature of our algorithm is that it creates a space decomposition as required by I/O‐efficient simplification and volume rendering, and simplifies both the internal and boundary portions of the sub‐volumes progressively by edge collapses using the (extended) quadric error metric, while ensuring any selected LOD mesh to be crack‐free (i.e., any neighboring sub‐volumes in the LOD have consistent boundaries, and all the cells in the LOD do not have negative volumes), with all computations performed I/O‐ejficiently. This has been an elusive goal for out‐of‐core progressive meshes and LOD visualization, and our novel solution achieves this goal with a theoretical guarantee to be crack‐free for tetrahedral meshes. As for selecting a desirable LOD mesh for volume rendering, our technique supports selective refinement LODs (where different places can have different error bounds), in addition to the basic uniform LODs (where the error bound is the same in all places). The proposed scalar‐value range and view‐dependent selection queries for selective refinement are especially effective in producing images of the highest quality with a much faster rendering speed. The experiments demonstrate the efficacy of our new technique.     

On the consistency of rule bases based on lattice‐valued first‐order logic LF(X)

The consistency of a rule base is an essential issue for rule‐based intelligent information processing. Due to the uncertainty inevitably included in the rule base, it is necessary to verify the consistency of the rule base while investigating, designing, and applying a rule‐based intelligent system. In the framework of the lattice‐valued first‐order logic system LF(X), which attempts to handle fuzziness and incomparability, this article focuses on how to verify and increase the consistency degree of the rule base in the intelligent information processing system. First, the representations of eight kinds of rule bases in LF(X) as the generalized clause set forms based on these rule bases' nonredundant generalized Skolem standard forms are presented. Then an α‐automated reasoning algorithm in LF(X), also used as an automated simplification algorithm, is proposed. Furthermore, the α‐consistency and the α‐simplification theories of the rule base in LF(X) are formulated, and especially the coherence between these two theories is proved. Therefore, the verification of the α‐consistency of the rule base, often an infinity problem that is difficult to solve, can be transformed into a finite and achievable α‐simplification problem. Finally, an α‐simplification stepwise search algorithm for verifying the consistency of the rule base as well as a kind of filtering algorithm for increasing the consistency level of the rule base are proposed. © 2006 Wiley Periodicals, Inc. Int J Int Syst 21: 399–424, 2006.     

Vision‐based Localization and Robot‐centric Mapping in Riverine Environments

This paper presents a vision‐based localization and mapping algorithm developed for an unmanned aerial vehicle (UAV) that can operate in a riverine environment. Our algorithm estimates the three‐dimensional positions of point features along a river and the pose of the UAV. By detecting features surrounding a river and the corresponding reflections on the water's surface, we can exploit multiple‐view geometry to enhance the observability of the estimation system. We use a robot‐centric mapping framework to further improve the observability of the estimation system while reducing the computational burden. We analyze the performance of the proposed algorithm with numerical simulations and demonstrate its effectiveness through experiments with data from Crystal Lake Park in Urbana, Illinois. We also draw a comparison to existing approaches. Our experimental platform is equipped with a lightweight monocular camera, an inertial measurement unit, a magnetometer, an altimeter, and an onboard computer. To our knowledge, this is the first result that exploits the reflections of features in a riverine environment for localization and mapping.     

Evaluation of jerkiness of moving three‐dimensional images produced by high‐density directional display

Abstract— The jerkiness of moving three‐dimensional (3‐D) images produced by a high‐density directional display was studied. Under static viewing conditions in which subjects' heads did not move, jerkiness was more noticeable when moving 3‐D images were displayed in front of the display screen and was less noticeable when moving 3‐D images were displayed behind the screen. We found that the perception of jerkiness depended on the visual angular velocities of moving 3‐D images. Under dynamic viewing conditions in which subjects' heads were forced to move, when moving 3‐D images were displayed in front of the screen, jerkiness was less noticeable when the subjects' heads and 3‐D images moved in opposite directions and was more noticeable when they moved in the same direction. When moving 3‐D images were displayed behind the screen, jerkiness was less noticeable when subjects' heads and 3‐D images moved in the same direction and was more noticeable when they moved in opposite directions.     

Querying time series data based on similarity

We study similarity queries for time series data where similarity is defined, in a fairly general way, in terms of a distance function and a set of affine transformations on the Fourier series representation of a sequence. We identify a safe set of transformations supporting a wide variety of comparisons and show that this set is rich enough to formulate operations such as moving average and time scaling. We also show that queries expressed using safe transformations can efficiently be computed without prior knowledge of the transformations. We present a query processing algorithm that uses the underlying multidimensional index built over the data set to efficiently answer similarity queries. Our experiments show that the performance of this algorithm is competitive to that of processing ordinary (exact match) queries using the index, and much faster than sequential scanning. We propose a generalization of this algorithm for simultaneously handling multiple transformations at a time, and give experimental results on the performance of the generalized algorithm     

Smooth Image Sequences for Data‐driven Morphing

Smoothness is a quality that feels aesthetic and pleasing to the human eye. We present an algorithm for finding “as‐smooth‐as‐possible” sequences in image collections. In contrast to previous work, our method does not assume that the images show a common 3D scene, but instead may depict different object instances with varying deformations, and significant variation in lighting, texture, and color appearance. Our algorithm does not rely on a notion of camera pose, view direction, or 3D representation of an underlying scene, but instead directly optimizes the smoothness of the apparent motion of local point matches among the collection images. We increase the smoothness of our sequences by performing a global similarity transform alignment, as well as localized geometric wobble reduction and appearance stabilization. Our technique gives rise to a new kind of image morphing algorithm, in which the in‐between motion is derived in a data‐driven manner from a smooth sequence of real images without any user intervention. This new type of morph can go far beyond the ability of traditional techniques. We also demonstrate that our smooth sequences allow exploring large image collections in a stable manner.     

Pattern‐Based Quadrangulation for N‐Sided Patches

We propose an algorithm to quadrangulate an N‐sided patch (2 ≤ N ≤ 6) with prescribed numbers of edge subdivisions at its boundary. Our algorithm is guaranteed to succeed for arbitrary valid input, which is proved using a canonical simplification of the input and a small set of topological patterns that are sufficient for supporting all possible cases. Our algorithm produces solutions with minimal number of irregular vertices by default, but it also allows the user to choose other feasible solutions by solving a set of small integer linear programs. We demonstrate the effectiveness of our algorithm by integrating it into a sketch‐based quad remeshing system. A reference C++ implementation of our algorithm is provided as a supplementary material.     

Scheduling model‐to‐model transformations with continuations

Model transformations are at the heart of model‐driven engineering because they allow the automation of diverse kinds of model manipulations. Transformation scheduling is a key issue in the design and implementation of many transformation languages. This paper reports our results using continuations as the underlying technique for building a scheduling mechanism implicitly driven by data dependence among transformation rules. To support our experiments, we have built a proof‐of‐concept model transformation language, which is also reported here. First, we motivate the problem by analyzing the scheduling mechanism of current model transformation languages. Then, we introduce the notion of continuation, showing its applicability to model transformations. Afterwards, we present our approach, notably explaining how dependence is specified and giving the scheduling algorithm. We also analyze the lazy resolution of rules and how to deal with collection operations. The approach is validated by an implementation that targets the Java Virtual Machine and by running of the performance benchmarks that show its efficiency and scalability. Besides, we discuss how it can be applied to other existing transformation languages and present several applicability scenarios. Copyright © 2013 John Wiley & Sons, Ltd.     

Making in‐Front‐of Cars Transparent: Sharing First‐Person‐Views via Dashcam

Visual obstruction caused by a preceding vehicle is one of the key factors threatening driving safety. One possible solution is to share the first‐person‐view of the preceding vehicle to unveil the blocked field‐of‐view of the following vehicle. However, the geometric inconsistency caused by the camera‐eye discrepancy renders view sharing between different cars a very challenging task. In this paper, we present a first‐person‐perspective image rendering algorithm to solve this problem. Firstly, we contour unobstructed view as the transferred region, then by iteratively estimating local homography transformations and performing perspective‐adaptive warping using the estimated transformations, we are able to locally adjust the shape of the unobstructed view so that its perspective and boundary could be matched to that of the occluded region. Thus, the composited view is seamless in both the perceived perspective and photometric appearance, creating an impression as if the preceding vehicle is transparent. Our system improves the driver's visibility and thus relieves the burden on the driver, which in turn increases comfort. We demonstrate the usability and stability of our system by performing its evaluation with several challenging data sets collected from real‐world driving scenarios.     

Data‐Parallel Decompression of Triangle Mesh Topology

We propose a lossless, single‐rate triangle mesh topology codec tailored for fast data‐parallel GPU decompression. Our compression scheme coherently orders generalized triangle strips in memory. To unpack generalized triangle strips efficiently, we propose a novel parallel and scalable algorithm. We order vertices coherently to further improve our compression scheme. We use a variable bit‐length code for additional compression benefits, for which we propose a scalable data‐parallel decompression algorithm. For a set of standard benchmark models, we obtain (min: 3.7, med: 4.6, max: 7.6) bits per triangle. Our CUDA decompression requires only about 15% of the time it takes to render the model even with a simple shader.     

Distributed stereo vision‐based 6D localization and mapping for multi‐robot teams

Joint simultaneous localization and mapping (SLAM) constitutes the basis for cooperative action in multi‐robot teams. We designed a stereo vision‐based 6D SLAM system combining local and global methods to benefit from their particular advantages: (1) Decoupled local reference filters on each robot for real‐time, long‐term stable state estimation required for stabilization, control and fast obstacle avoidance; (2) Online graph optimization with a novel graph topology and intra‐ as well as inter‐robot loop closures through an improved submap matching method to provide global multi‐robot pose and map estimates; (3) Distribution of the processing of high‐frequency and high‐bandwidth measurements enabling the exchange of aggregated and thus compacted map data. As a result, we gain robustness with respect to communication losses between robots. We evaluated our improved map matcher on simulated and real‐world datasets and present our full system in five real‐world multi‐robot experiments in areas of up 3,000 m² (bounding box), including visual robot detections and submap matches as loop‐closure constraints. Further, we demonstrate its application to autonomous multi‐robot exploration in a challenging rough‐terrain environment at a Moon‐analogue site located on a volcano.     

Proxy‐guided Image‐based Rendering for Mobile Devices

VR headsets and hand‐held devices are not powerful enough to render complex scenes in real‐time. A server can take on the rendering task, but network latency prohibits a good user experience. We present a new image‐based rendering (IBR) architecture for masking the latency. It runs in real‐time even on very weak mobile devices, supports modern game engine graphics, and maintains high visual quality even for large view displacements. We propose a novel server‐side dual‐view representation that leverages an optimally‐placed extra view and depth peeling to provide the client with coverage for filling disocclusion holes. This representation is directly rendered in a novel wide‐angle projection with favorable directional parameterization. A new client‐side IBR algorithm uses a pre‐transmitted level‐of‐detail proxy with an encaging simplification and depth‐carving to maintain highly complex geometric detail. We demonstrate our approach with typical VR / mobile gaming applications running on mobile hardware. Our technique compares favorably to competing approaches according to perceptual and numerical comparisons.     

Generalized As‐Similar‐As‐Possible Warping with Applications in Digital Photography

Discrete conformal mappings of planar triangle meshes, also known as the As‐Similar‐As‐Possible (ASAP) mapping, involve the minimization of a quadratic energy function, thus are very easy to generate and are popular in image warping scenarios. We generalize this classical mapping to the case of quad meshes, taking into account the mapping of the interior of the quad, and analyze in detail the most common case ‐ the unit grid mesh. We show that the generalization, when combined with barycentric coordinate mappings between the source and target polygons, spawns an entire family of new mappings governed by quadratic energy functions, which allow to control quite precisely various effects of the mapping. This approach is quite general and applies also to arbitrary planar polygon meshes. As an application of generalized ASAP mappings of the unit grid mesh, we demonstrate how they can be used to warp digital photographs to achieve a variety of effects. One such effect is modifying the perspective of the camera that took a given photograph (without moving the camera). A related, but more challenging, effect is re‐photography ‐ warping a contemporary photograph in order to reproduce the camera view present in a vintage photograph of the same scene ‐ taken many years before with a different camera from a different viewpoint. We apply the generalized ASAP mapping to these images, discretized to a unit grid. Using a quad mesh (as opposed to a triangle mesh) permits biasing towards affine maps of the unit squares. This allows the introduction of an As‐Affine‐As‐Possible (AAAP) mapping for a good approximation of the homographies present in these warps, achieving quite accurate results. We demonstrate the advantages of the AAAP mapping on a variety of synthetic and real‐world examples.     

Cloud‐based Real‐time Outsourcing Localization for a Ground Mobile Robot in Large‐scale Outdoor Environments

Cloud robotics is the application of cloud computing concepts to robotic systems. It utilizes modern cloud computing infrastructure to distribute computing resources and datasets. Cloud‐based real‐time outsourcing localization architecture is proposed in this paper to allow a ground mobile robot to identify its location relative to a road network map and reference images in the cloud. An update of the road network map is executed in the cloud, as is the extraction of the robot‐terrain inclination (RTI) model as well as reference image matching. A particle filter with a network‐delay‐compensation localization algorithm is executed on the mobile robot based on the local RTI model and the recognized location both of which are sent from the cloud. The proposed methods are tested in different challenging outdoor scenarios with a ground mobile robot equipped with minimal onboard hardware, where the longest trajectory was 13.1 km. Experimental results show that this method could be applicable to large‐scale outdoor environments for autonomous robots in real time.     

On-line motion synthesis and adaptation using a trajectory database

Autonomous robots cannot be programmed in advance for all possible situations. Instead, they should be able to generalize the previously acquired knowledge to operate in new situations as they arise. A possible solution to the problem of generalization is to apply statistical methods that can generate useful robot responses in situations for which the robot has not been specifically instructed how to respond. In this paper we propose a methodology for the statistical generalization of the available sensorimotor knowledge in real-time. Example trajectories are generalized by applying Gaussian process regression, using the parameters describing a task as query points into the trajectory database. We show on real-world tasks that the proposed methodology can be integrated into a sensory feedback loop, where the generalization algorithm is applied in real-time to adapt robot motion to the perceived changes of the external world.     

Visual‐inertial curve simultaneous localization and mapping: Creating a sparse structured world without feature points

We present a simultaneous localization and mapping (SLAM) algorithm that uses Bézier curves as static landmark primitives rather than feature points. Our approach allows us to estimate the full six degrees of freedom pose of a robot while providing a structured map that can be used to assist a robot in motion planning and control. We demonstrate how to reconstruct the three‐dimensional (3D) location of curve landmarks from a stereo pair and how to compare the 3D shape of curve landmarks between chronologically sequential stereo frames to solve the data association problem. We also present a method to combine curve landmarks for mapping purposes, resulting in a map with a continuous set of curves that contain fewer landmark states than conventional point‐based SLAM algorithms. We demonstrate our algorithm's effectiveness with numerous experiments, including comparisons to existing state‐of‐the‐art SLAM algorithms.     

Efficient Multi‐image Correspondences for On‐line Light Field Video Processing

Light field videos express the entire visual information of an animated scene, but their shear size typically makes capture, processing and display an off‐line process, i. e., time between initial capture and final display is far from real‐time. In this paper we propose a solution for one of the key bottlenecks in such a processing pipeline, which is a reliable depth reconstruction possibly for many views. This is enabled by a novel correspondence algorithm converting the video streams from a sparse array of off‐the‐shelf cameras into an array of animated depth maps. The algorithm is based on a generalization of the classic multi‐resolution Lucas‐Kanade correspondence algorithm from a pair of images to an entire array. Special inter‐image confidence consolidation allows recovery from unreliable matching in some locations and some views. It can be implemented efficiently in massively parallel hardware, allowing for interactive computations. The resulting depth quality as well as the computation performance compares favorably to other state‐of‐the art light field‐to‐depth approaches, as well as stereo matching techniques. Another outcome of this work is a data set of light field videos that are captured with multiple variants of sparse camera arrays.