Robust real-time ground plane motion compensation from a moving vehicle

One method to detect obstacles from a vehicle moving on a planar road surface is the analysis of motion-compensated difference images. In this contribution, a motion compensation algorithm is presented, which computes the required image-warping parameters from an estimate of the relative motion between camera and ground plane. The proposed algorithm estimates the warping parameters from displacements at image corners and image edges. It exploits the estimated confidence of the displacements to cope robustly with outliers. Knowledge about camera calibration, measuremts from odometry, and the previous estimate are used for motion prediction and to stabilize the estimation process when there is not enough information available in the measured image displacements. The motion compensation algorithm has been integrated with modules for obstacle detection and lane tracking. This system has been integrated in experimental vehicles and runs in real time with an overall cycle of 12.5 Hz on low-cost standard hardware. Received: 23 April 1998 / Accepted: 25 August 1999     

Straight lines have to be straight

Most algorithms in 3D computer vision rely on the pinhole camera model because of its simplicity, whereas video optics, especially low-cost wide-angle or fish-eye lenses, generate a lot of non-linear distortion which can be critical. To find the distortion parameters of a camera, we use the following fundamental property: a camera follows the pinhole model if and only if the projection of every line in space onto the camera is a line. Consequently, if we find the transformation on the video image so that every line in space is viewed in the transformed image as a line, then we know how to remove the distortion from the image. The algorithm consists of first doing edge extraction on a possibly distorted video sequence, then doing polygonal approximation with a large tolerance on these edges to extract possible lines from the sequence, and then finding the parameters of our distortion model that best transform these edges to segments. Results are presented on real video images, compared with distortion calibration obtained by a full camera calibration method which uses a calibration grid. Received: 27 December 1999 / Accepted: 8 November 2000     

Real-time multiple vehicle detection and tracking from a moving vehicle

Abstract. A real-time vision system has been developed that analyzes color videos taken from a forward-looking video camera in a car driving on a highway. The system uses a combination of color, edge, and motion information to recognize and track the road boundaries, lane markings and other vehicles on the road. Cars are recognized by matching templates that are cropped from the input data online and by detecting highway scene features and evaluating how they relate to each other. Cars are also detected by temporal differencing and by tracking motion parameters that are typical for cars. The system recognizes and tracks road boundaries and lane markings using a recursive least-squares filter. Experimental results demonstrate robust, real-time car detection and tracking over thousands of image frames. The data includes video taken under difficult visibility conditions. Received: 1 September 1998 / Accepted: 22 February 2000     

基于单目视觉的道路图像理解综述

基于视觉的道路图像提供了车辆运行局部环境的丰富的信息,对这些序列图像的分析理解在车辆辅助驾驶、自动导引车、室外移动机器人等多领域得到了广泛的应用。详细介绍了道路检测系统以及道路偏离预警等系统中对道路图像进行处理、分析、理解的各种方法,阐述了道路识别与跟踪的技术和方法,并进行了比较。最后给出了该领域今后的研究方向和发展趋势。     

Parametric Model of the Perspective Projection of a Road with Applications to Lane Keeping and 3D Road Reconstruction

This paper describes an algorithm for the computation of the 3D structure of a road and of the motion state of an in-vehicle CCD camera from the lane borders on the image plane. A parametric model of the projective projection of the lane borders on the image plane is introduced and a fast and reliable algorithm to compute its parameters is described. The physical significance of the model is demonstrated by showing that the model parameters completely determine the position of the vehicle inside the lane, its heading direction, and the local structure of the road. The conditions of applicability of the model are also given and the results of its application to a sequence of images taken during a test run discussed. The algorithm is suited to real time applications and indeed an implementation on the dedicated hardware of the mobile laboratory MOBLAB runs at a frame rate of 12 images per second.     

结构化道路车道线快速检测的一种改进算法

基于视觉导航的高速智能车,提出一种改进的道路快速检测算法。用改进的水平均值投影法划分道路和背景区域,结合边缘检测算子和最大类间方差法(大津算法),构成双阈值法对道路区域图像进行二值化处理,利用先验知识改进的霍夫变换,在路面存在阴影和噪声干扰的条件下,能准确地检测车道标识线;对动态预测划分感兴趣区域,采用菱形搜索法进行车道线跟踪,融合初始检测和后续跟踪两层算法循环处理道路图像序列。实车试验表明,算法具有良好的实时性和鲁棒性,满足智能车高速行驶要求。     

基于PER-APF算法的无人驾驶汽车换道轨迹规划

针对传统人工势场算法在解决无人驾驶汽车换道轨迹规划过程中存在的不足,提出一种基于势能重构人工势场 (Potential Energy Reconstruction- Artificial Potential Field, PER-APF) 的无人驾驶汽车换道轨迹规划算法。首先,建立了具有斥力区分的道路边界约束条件和多约束换道轨迹规划模型,通过判断障碍车辆与道路边沿的距离来保证换道过程的安全性与有效性;其次,提出了基于势能重构的改进APF算法,通过构建虚拟区域以及重构物理势能力场,有效的解决了目标不可达以及局部最优问题。仿真结果表明,所设计的PER-APF算法能够快速有效地为无人驾驶汽车规划一条安全合理的换道轨迹。     

Iterative model-based binarization algorithm for cheque images

Binarization of document images with poor contrast, strong noise, complex patterns, and variable modalities in the gray-scale histograms is a challenging problem. A new binarization algorithm has been developed to address this problem for personal cheque images. The main contribution of this approach is optimizing the binarization of a part of the document image that suffers from noise interference, referred to as the Target Sub-Image (TSI), using information easily extracted from another noise-free part of the same image, referred to as the Model Sub-Image (MSI). Simple spatial features extracted from MSI are used as a model for handwriting strokes. This model captures the underlying characteristics of the writing strokes, and is invariant to the handwriting style or content. This model is then utilized to guide the binarization in the TSI. Another contribution is a new technique for the structural analysis of document images, which we call “Wavelet Partial Reconstruction” (WPR). The algorithm was tested on 4,200 cheque images and the results show significant improvement in binarization quality in comparison with other well-established algorithms. Received: October 10, 2001 / Accepted: May 7, 2002 This research was supported in part by NCR and NSERC's industrial postgraduate scholarship No. 239464. A simplified version of this paper has been presented at ICDAR 2001 [3].     

Lane detection by orientation and length discrimination

This paper describes a novel lane detection algorithm for visual traffic surveillance applications under the auspice of intelligent transportation systems. Traditional lane detection methods for vehicle navigation typically use spatial masks to isolate instantaneous lane information from on-vehicle camera images. When surveillance is concerned, complete lane and multiple lane information is essential for tracking vehicles and monitoring lane change frequency from overhead cameras, where traditional methods become inadequate. The algorithm presented in this paper extracts complete multiple lane information by utilizing prominent orientation and length features of lane markings and curb structures to discriminate against other minor features. Essentially, edges are first extracted from the background of a traffic sequence, then thinned and approximated by straight lines. From the resulting set of straight lines, orientation and length discriminations are carried out three-dimensionally with the aid of two-dimensional (2-D) to three-dimensional (3-D) coordinate transformation and K-means clustering. By doing so, edges with strong orientation and length affinity are retained and clustered, while short and isolated edges are eliminated. Overall, the merits of this algorithm are as follows. First, it works well under practical visual surveillance conditions. Second, using K-means for clustering offers a robust approach. Third, the algorithm is efficient as it only requires one image frame to determine the road center lines. Fourth, it computes multiple lane information simultaneously. Fifth, the center lines determined are accurate enough for the intended application.     

Visual routines for real-time monitoring of vehicle behavior

Low-cost, purely visual modules are presented, which are able to reliably perform real-time computation of kinematic quantities relevant for the navigation of commercial vehicles moving along highways or motorways. Taking as input a video stream of b/w images coming from a standard camera mounted aboard the vehicle, the modules return instantaneous measurements for the speed of the vehicle, its lateral position in the lane, and the distance from the preceding vehicle. The modules, which are designed to work in real-time on standard PC platforms, have been tested under typical working conditions. Preliminary results are reported and discussed. Received: 10 March 1997 / Accepted: 24 November 1997     

Image-based animation of facial expressions

We present a novel technique for creating realistic facial animations given a small number of real images and a few parameters for the in-between images. This scheme can also be used for reconstructing facial movies where the parameters can be automatically extracted from the images. The in-between images are produced without ever generating a three-dimensional model of the face. Since facial motion due to expressions are not well defined mathematically our approach is based on utilizing image patterns in facial motion. These patterns were revealed by an empirical study which analyzed and compared image motion patterns in facial expressions. The major contribution of this work is showing how parameterized “ideal” motion templates can generate facial movies for different people and different expressions, where the parameters are extracted automatically from the image sequence. To test the quality of the algorithm, image sequences (one of which was taken from a TV news broadcast) were reconstructed, yielding movies hardly distinguishable from the originals. Published online: 2 October 2002 Correspondence to: A. Tal Work has been supported in part by the Israeli Ministry of Industry and Trade, The MOST Consortium     

A sweeping algorithm for cluster hair rendering

projection-based sweeping algorithm , is developed to render hair images using the cluster model. Compared with a previous ray-tracing algorithm, the new algorithm offers several significant improvements: (1) it avoids the complex and expensive computation of the ray–cluster intersection; (2) it eliminates the time-consuming density-value evaluations at the sample points along a ray during ray tracing; (3) it allows more flexible variation of the density map along the trajectory to enhance the natural appearance of hair; and (4) it allows easier coarse/fine control over the density map for multi-resolution rendering. As a result, the new algorithm significantly improves rendering efficiency and hair image quality. Compared with the existing hardware-supported, texture-mapping projection methods, the new method allows more sophisticated shading models, accurate shadow calculation, and uniform handling of arbitrary slice orientations with accurate opacity accumulation. Published online: 22 May 2003     

Estimations of previewed road curvatures and vehicular motion by a vision-based data fusion scheme

In this study, a new framework of vision-based estimation is developed using some data fusion schemes to obtain previewed road curvatures and vehicular motion states based on the scene viewed from an in-vehicle camera. The previewed curvatures are necessary for the guidance of an automatically steering vehicle, and the desired vehicular motion variables, including lateral deviation, heading angle, yaw rate, and sideslip angle, are also required for proper control of the vehicular lateral motion via steering. In this framework, physical relationships of previewed curvatures among consecutive images, motion variables in terms of image features searched at various levels in the image plane, and dynamic correlation among vehicular motion variables are derived as bases of data fusion to enhance the accuracy of estimation. The vision-based measurement errors are analyzed to determine the fusion gains based on the technique of a Kalman filter such that the measurements from the image plane and predictions of physical models can be properly integrated to obtain reliable estimations. Off-line experimental works using real road scenes are performed to verify the whole framework for image sensing.     

Mobile robot navigation and scene modeling using stereo fish-eye lens system

We present an autonomous mobile robot navigation system using stereo fish-eye lenses for navigation in an indoor structured environment and for generating a model of the imaged scene. The system estimates the three-dimensional (3D) position of significant features in the scene, and by estimating its relative position to the features, navigates through narrow passages and makes turns at corridor ends. Fish-eye lenses are used to provide a large field of view, which images objects close to the robot and helps in making smooth transitions in the direction of motion. Calibration is performed for the lens-camera setup and the distortion is corrected to obtain accurate quantitative measurements. A vision-based algorithm that uses the vanishing points of extracted segments from a scene in a few 3D orientations provides an accurate estimate of the robot orientation. This is used, in addition to 3D recovery via stereo correspondence, to maintain the robot motion in a purely translational path, as well as to remove the effects of any drifts from this path from each acquired image. Horizontal segments are used as a qualitative estimate of change in the motion direction and correspondence of vertical segment provides precise 3D information about objects close to the robot. Assuming detected linear edges in the scene as boundaries of planar surfaces, the 3D model of the scene is generated. The robot system is implemented and tested in a structured environment at our research center. Results from the robot navigation in real environments are presented and discussed. Received: 25 September 1996 / Accepted: 20 October 1996     

A local method for contour matching and its parallel implementation

This paper presents a local approach for matching contour segments in an image sequence. This study has been primarily motivated by work concerned with the recovery of 3D structure using active vision. The method to recover the 3D structure of the scene requires to track in real-time contour segments in an image sequence. Here, we propose an original and robust approach that is ideally suited for this problem. It is also of more general interest and can be used in any context requiring matching of line boundaries over time. This method only involves local modeling and computation of moving edges dealing “virtually” with a contour segment primitive representation. Such an approach brings robustness to contour segmentation instability and to occlusion, and easiness for implementation. Parallelism has also been investigated using an SIMD-based real-time image-processing system. This method has been validated with experiments on several real-image sequences. Our results show quite satisfactory performance and the algorithm runs in a few milliseconds. Received: 11 December 1996 / Accepted: 8 August 1997     

Robust and efficient map-to-image registration with line segments

This work investigates map-to-image registration for planar scenes in the context of robust parameter estimation. Registration is posed as the problem of estimating a projective transformation which optimally aligns transformed model line segments from a map with data line segments extracted from an image. Matching and parameter estimation is solved simultaneously by optimizing an objective function which is based on M-estimators, and depends on overlap and the weighted orthogonal distance between transformed model segments and data segments. An extensive series of registration experiments was conducted to test the performance of the proposed parameter estimation algorithm. More than 200 000 registration experiments were run with different objective functions for 12 aerial images and randomly corrupted maps distorted by randomly selected projective transformations. Received: 10 August 2000 / Accepted: 29 January 2001     

A new and fast approach for DPIV using an incompressible affine flow model

Traditional digital particle image velocimetry (DPIV) methods are previously based on area-correlation. Though proven to be very time-consuming and error prone, it has been widely adopted because it is conceptually simple, and easy to implement, and also because there are few alternatives. This paper provides a non-correlative, conceptually new, fast and efficient approach for DPIV which takes the nature of flow into consideration. An incompressible affine flow model (IAFM) is introduced to describe a flow that incorporates rational constraint directly into the computation. This IAFM, combining with a modified optical flow method – named total optical flow computation, provides a linear system solution to DPIV. Experimental results on real images demonstrate our method to be a very promising approach for DPIV. Received: 23 March 1998 / Accepted: 1 September 1999     

基于Mask R-CNN与改进Criminisi的沥青路面车道线移除方法

在对沥青路面病害图像进行自动分类时,含车道线的图像数量较多易造成干扰。 为此,提出一种车道线移除方法以降低其对分类的影响,首先基于 Mask R-CNN 网络训练出复 杂背景下车道线区域的检测模型,通过该模型自动获取车道线区域 mask;然后利用 mask 将车 道线区域全部移除得到破损图像;最后用改进的 Criminisi 图像修复方法对破损图像进行样本块 填充。实验表明,采用 Mask R-CNN 方法对 400 张不同环境下的路面图像进行检测,其漏检率 和误检率分别为 0.50%和 7.87%。在保证图像修复质量的基础上,改进的 Criminisi 方法在修复 速度上比改进前提升约 4~5 倍。同等条件下,采用 VGG 分类模型对比验证,经该算法移除车 道线后的新数据集表现更优。     

Reconstructing a textured CAD model of an urban environment using vehicle-borne laser range scanners and line cameras

Abstract. In this paper, a novel method is presented for generating a textured CAD model of an outdoor urban environment using a vehicle-borne sensor system. In data measurement, three single-row laser range scanners and six line cameras are mounted on a measurement vehicle, which has been equipped with a GPS/INS/Odometer-based navigation system. Laser range and line images are measured as the vehicle moves forward. They are synchronized with the navigation system so they can be geo-referenced to a world coordinate system. Generation of the CAD model is conducted in two steps. A geometric model is first generated using the geo-referenced laser range data, where urban features, such as buildings, ground surfaces, and trees are extracted in a hierarchical way. Different urban features are represented using different geometric primitives, such as a planar face, a triangulated irregular network (TIN), and a triangle. The texture of the urban features is generated by projecting and resampling line images onto the geometric model. An outdoor experiment is conducted, and a textured CAD model of a real urban environment is reconstructed in a full automatic mode.     

Adaptive image-smoothing using a coplanar matrix and its application to document image binarization

Abstract. For document images corrupted by various kinds of noise, direct binarization images may be severely blurred and degraded. A common treatment for this problem is to pre-smooth input images using noise-suppressing filters. This article proposes an image-smoothing method used for prefiltering the document image binarization. Conceptually, we propose that the influence range of each pixel affecting its neighbors should depend on local image statistics. Technically, we suggest using coplanar matrices to capture the structural and textural distribution of similar pixels at each site. This property adapts the smoothing process to the contrast, orientation, and spatial size of local image structures. Experimental results demonstrate the effectiveness of the proposed method, which compares favorably with existing methods in reducing noise and preserving image features. In addition, due to the adaptive nature of the similar pixel definition, the proposed filter output is more robust regarding different noise levels than existing methods. Received: October 31, 2001 / October 09, 2002 Correspondence to:L. Fan (e-mail: fanlixin@ieee.org)