Robust real-time ground plane motion compensation from a moving vehicle

One method to detect obstacles from a vehicle moving on a planar road surface is the analysis of motion-compensated difference images. In this contribution, a motion compensation algorithm is presented, which computes the required image-warping parameters from an estimate of the relative motion between camera and ground plane. The proposed algorithm estimates the warping parameters from displacements at image corners and image edges. It exploits the estimated confidence of the displacements to cope robustly with outliers. Knowledge about camera calibration, measuremts from odometry, and the previous estimate are used for motion prediction and to stabilize the estimation process when there is not enough information available in the measured image displacements. The motion compensation algorithm has been integrated with modules for obstacle detection and lane tracking. This system has been integrated in experimental vehicles and runs in real time with an overall cycle of 12.5 Hz on low-cost standard hardware. Received: 23 April 1998 / Accepted: 25 August 1999     

A model-driven approach for real-time road recognition

This article describes a method designed to detect and track road edges starting from images provided by an on-board monocular monochromic camera. Its implementation on specific hardware is also presented in the framework of the VELAC project. The method is based on four modules: (1) detection of the road edges in the image by a model-driven algorithm, which uses a statistical model of the lane sides which manages the occlusions or imperfections of the road marking – this model is initialized by an off-line training step; (2) localization of the vehicle in the lane in which it is travelling; (3) tracking to define a new search space of road edges for the next image; and (4) management of the lane numbers to determine the lane in which the vehicle is travelling. The algorithm is implemented in order to validate the method in a real-time context. Results obtained on marked and unmarked road images show the robustness and precision of the method. Received: 18 November 2000 / Accepted: 7 May 2001     

Tracking the human arm using constraint fusion and multiple-cue localization

Abstract. The use of hand gestures provides an attractive means of interacting naturally with a computer-generated display. Using one or more video cameras, the hand movements can potentially be interpreted as meaningful gestures. One key problem in building such an interface without a restricted setup is the ability to localize and track the human arm robustly in video sequences. This paper proposes a multiple-cue localization scheme combined with a tracking framework to reliably track the dynamics of the human arm in unconstrained environments. The localization scheme integrates the multiple cues of motion, shape, and color for locating a set of key image features. Using constraint fusion, these features are tracked by a modified extended Kalman filter that exploits the articulated structure of the human arm. Moreover, an interaction scheme between tracking and localization is used for improving the estimation process while reducing the computational requirements. The performance of the localization/tracking framework is validated with the help of extensive experiments and simulations. These experiments include tracking with calibrated stereo camera and uncalibrated broadcast video. Received: 19 January 2001 / Accepted: 27 December 2001 Correspondence to: R. Sharma     

夜晚车辆异常事件分析

夜晚车道模型是车辆跟踪和车辆行为分析的基础,但是当高速公路或者城市道路光线较暗时,很难通过车道检测的方法来建立车道模型,夜晚车辆快速行驶或相邻帧车辆之间重叠度较低时无法实现准确跟踪。针对此类问题提出了一种基于学习的车道模型建立方法和基于多帧的最佳匹配跟踪方法。首先利用自动多阈值分割方法提取场景中光亮的目标;其次,利用车灯的相关特征移除非车灯光亮区域;接着,利用空间信息把车灯聚类成一个车辆目标,利用多帧的最佳匹配跟踪方法进行跟踪;最后利用车辆跟踪参数与车道模型的融合对夜晚车辆异常事件进行分析。实验结果表明,该算法能够准确地检测出夜晚车辆换道、逆向行驶、交通拥挤、停车等异常事件,并且有很强的鲁棒性。     

Visual routines for real-time monitoring of vehicle behavior

Low-cost, purely visual modules are presented, which are able to reliably perform real-time computation of kinematic quantities relevant for the navigation of commercial vehicles moving along highways or motorways. Taking as input a video stream of b/w images coming from a standard camera mounted aboard the vehicle, the modules return instantaneous measurements for the speed of the vehicle, its lateral position in the lane, and the distance from the preceding vehicle. The modules, which are designed to work in real-time on standard PC platforms, have been tested under typical working conditions. Preliminary results are reported and discussed. Received: 10 March 1997 / Accepted: 24 November 1997     

车道模型的高速公路车辆异常行为检测方法

针对高速公路上车辆的逆行、停车、轨迹异常等事件的检测问题,提出了一种基于车道模型知识的自底上向的车辆异常检测方法。首先由车道线的连续性、共线性的感知搜索出车道线和消失点,自动建立车道模型;然后在多车辆检测与跟踪时,通过目标运动位置预测和KLT特征点跟踪的方法建立表示目标区域交叠关系图,依据后验关系通过对图中节点对应目标区域的合并与拆分实现目标与实际车辆的一一对应,建立可靠的跟踪轨迹;最后基于消失点的坐标变换,计算车辆实际位置与速度,采用轨迹滑动窗口方法判断目标运动趋势并计算窗口内轨迹点平均速度,判断车辆的异常行为。实验结果表明,所提方法在不同天气、不同车流量环境中均有80%以上的事件检测率,同时算法简单,具有很好的实时性,能够适应实际高速公路智能检测设备的需求。     

Ego-Lane Analysis System (ELAS): Dataset and algorithms

Decreasing costs of vision sensors and advances in embedded hardware boosted lane related research – detection, estimation, tracking, etc. – in the past two decades. The interest in this topic has increased even more with the demand for advanced driver assistance systems (ADAS) and self-driving cars. Although extensively studied independently, there is still need for studies that propose a combined solution for the multiple problems related to the ego-lane, such as lane departure warning (LDW), lane change detection, lane marking type (LMT) classification, road markings detection and classification, and detection of adjacent lanes (i.e., immediate left and right lanes) presence. In this paper, we propose a real-time Ego-Lane Analysis System (ELAS) capable of estimating ego-lane position, classifying LMTs and road markings, performing LDW and detecting lane change events. The proposed vision-based system works on a temporal sequence of images. Lane marking features are extracted in perspective and Inverse Perspective Mapping (IPM) images that are combined to increase robustness. The final estimated lane is modeled as a spline using a combination of methods (Hough lines with Kalman filter and spline with particle filter). Based on the estimated lane, all other events are detected. To validate ELAS and cover the lack of lane datasets in the literature, a new dataset with more than 20 different scenes (in more than 15,000 frames) and considering a variety of scenarios (urban road, highways, traffic, shadows, etc.) was created. The dataset was manually annotated and made publicly available to enable evaluation of several events that are of interest for the research community (i.e., lane estimation, change, and centering; road markings; intersections; LMTs; crosswalks and adjacent lanes). Moreover, the system was also validated quantitatively and qualitatively on other public datasets. ELAS achieved high detection rates in all real-world events and proved to be ready for real-time applications.     

视频图像的车辆速度实时检测算法研究

在监控视频图像中因拍摄角度的问题,导致图像中的车道线并不是竖直的。针对在此图像中选取对应块,实现车辆速度实时检测的问题,分析了车身的特征及图像中车道线的倾斜角度,选择车灯作为对应块,选取车灯存在的候选区域;然后根据车灯的对称性强度筛选车灯带,实现车灯的准确定位,并把车灯移动的距离通过摄像机标定转换到实际坐标中,从而实现车辆速度的实时检测。实验结果证明该方法运行速度快,定位准确。因此可以广泛的应用于红绿灯路口环境中实现视频图像的车速实时检测。     

Query by video clip

Typical digital video search is based on queries involving a single shot. We generalize this problem by allowing queries that involve a video clip (say, a 10-s video segment). We propose two schemes: (i) retrieval based on key frames follows the traditional approach of identifying shots, computing key frames from a video, and then extracting image features around the key frames. For each key frame in the query, a similarity value (using color, texture, and motion) is obtained with respect to the key frames in the database video. Consecutive key frames in the database video that are highly similar to the query key frames are then used to generate the set of retrieved video clips. (ii) In retrieval using sub-sampled frames, we uniformly sub-sample the query clip as well as the database video. Retrieval is based on matching color and texture features of the sub-sampled frames. Initial experiments on two video databases (basketball video with approximately 16,000 frames and a CNN news video with approximately 20,000 frames) show promising results. Additional experiments using segments from one basketball video as query and a different basketball video as the database show the effectiveness of feature representation and matching schemes.     

Design and FPGA implementation of dual-stage lane detection,based on Hough transform and localized stripe features

Robust, accurate and real-time detection of lane boundaries based on embedded hardware is an essential element of several driver assistant systems. In this work we present an FPGA-based dual-stage lane detection algorithm to cope with real world challenges such as cast shadows, occlusion of lane markers, brightness variations, wear, etc. In first stage, Sobel operator and adaptive threshold are used to extract lane edges, followed by Hough transform to extract the road markers. Second stage of the algorithm operates on original grayscale image and identifies stripe features near several candidate points with highest probabilities to find the landmarks. These extracted features are then used to detect the lane boundaries with high accuracy. Experimental results based on FPGA platform under various road conditions obtained from various datasets indicate that our algorithm can process about 60 frames per second for 720 pixels video input. Lane detection accuracy of 94.3% is achieved in average which may reach up to 97.8% in low congested highway during daylight.     

An Internet accessible telepresence

The US Vice President, Al Gore, in a speech on the information superhighway, suggested that it could be used to remotely control a nuclear reactor. We do not have enough confidence in computer software, hardware, or networks to attempt this experiment, but have instead built a Internet-accessible, remote-controlled model car that provides a race driver's view via a video camera mounted on the model car. The remote user can see live video from the car, and, using a mouse, control the speed and direction of the car. The challenge was to build a car that could be controlled by novice users in narrow corridors, and that would work not only with the full motion video that the car natively provides, but also with the limited size and frame rate video available over the Internet multicast backbone. We have built a car that has been driven from a site 50 miles away over a 56-kbps IP link using format video at as little as one frame per second and at as low as pixels resolution. We also built hardware to control the car, using a slightly modified voice grade channel videophone. Our experience leads us to believe that it is now possible to put together readily available hardware and software components to build a cheap and effective telepresence.     

Shooting a smooth video with a shaky camera

Abstract. The image sequence in a video taken by a moving camera may suffer from irregular perturbations because of irregularities in the motion of the person or vehicle carrying the camera. We show how to use information in the image sequence to correct the effects of these irregularities so that the sequence is smoothed, i.e., is approximately the same as the sequence that would have been obtained if the motion of the camera had been smooth. Our method is based on the fact that the irregular motion is almost entirely rotational, and that the rotational image motion can be detected and corrected if a distant object, such as the horizon, is visible. Received: 14 February 2001 / Accepted: 11 February 2002 Correspondence to: A. Rosenfeld     

TELEPORT – Towards immersive copresence

TELEPORT is an experimental teleconferencing system with the goal of enabling small groups of people, although geographically separated, to meet as if face to face. The innovative features of the system include the use of full-wall display surfaces, “merging” of real and virtual environments, viewer tracking, and real-time compositing of live video with synthetic backgrounds.     

基于单目视觉的障碍物检测方法研究

为了解决车辆的碰撞问题,提出了一种基于单目视觉的车道标识线二维重建及障碍物检测与跟踪的方法.主要采用最大类间方差法对初始图像进行分割,给出一种新的车道标识线特征点搜索策略,用统计模型的方法获取车道线参数.利用障碍物(前方车辆)在道路中留下的阴影,检测车辆边缘并确认车辆存在,运用卡尔曼滤波方法跟踪车辆具体位置.所设计的方法能够准确地检测跟踪车道线及前方障碍物.实验结果表明:方法满足车辆安全行驶的前提需要,具有较高的准确性和实时性.     

Extraction of special effects caption text events from digital video

Abstract. The popularity of digital video is increasing rapidly. To help users navigate libraries of video, algorithms that automatically index video based on content are needed. One approach is to extract text appearing in video, which often reflects a scene's semantic content. This is a difficult problem due to the unconstrained nature of general-purpose video. Text can have arbitrary color, size, and orientation. Backgrounds may be complex and changing. Most work so far has made restrictive assumptions about the nature of text occurring in video. Such work is therefore not directly applicable to unconstrained, general-purpose video. In addition, most work so far has focused only on detecting the spatial extent of text in individual video frames. However, text occurring in video usually persists for several seconds. This constitutes a text event that should be entered only once in the video index. Therefore it is also necessary to determine the temporal extent of text events. This is a non-trivial problem because text may move, rotate, grow, shrink, or otherwise change over time. Such text effects are common in television programs and commercials but so far have received little attention in the literature. This paper discusses detecting, binarizing, and tracking caption text in general-purpose MPEG-1 video. Solutions are proposed for each of these problems and compared with existing work found in the literature. Received: January 29, 2002 / Accepted: September 13, 2002 D. Crandall is now with Eastman Kodak Company, 1700 Dewey Avenue, Rochester, NY 14650-1816, USA; e-mail: david.crandall@kodak.com S. Antani is now with the National Library of Medicine, 8600 Rockville Pike, Bethesda, MD 20894, USA; e-mail: antani@nlm.nih.gov Correspondence to: David Crandall     

Semi-automatic extraction of road networks by least squares interlaced template matching in urban areas

Semi-automatic extraction of roads is greatly needed to accelerate the acquisition and updating of geodata. In fact, most roads are often seriously impacted by various types of noise, such as the occlusion of vehicles and the shadow of trees on very-high-resolution (VHR) remotely sensed imagery, which makes most of the existing road trackers ineffective. Fortunately, lane markings are less frequently disturbed than other parts of the road surface, and they provide a unique clue for road tracking on VHR images. In this paper, a semi-automatic method is proposed to extract roads with lane markings in urban areas. First, an operator detects a road segment and selects three seed points, which indicate the starting point, the direction and the width of a road, and lane markings near the seed points are automatically detected. Subsequently, an interlaced reference template of the selected road, composed of a few profiles of the road surface and the detected rectangular templates of lane markings on the road surface, is constructed. The reference template is then convolved with the image, and least squares template matching is employed to track the road axis. To complete the task, the process is then repeated. Various types of images are used for test, and the results show that our method is capable of robustly extracting roads from VHR imagery because the special configuration of the reference template can decrease side effects such as the occlusion of vehicles and the shadow of trees as much as possible.     

基于SLIC和改进区域生长的非结构化道路识别

非结构化道路一般没有车道标识线且道路边界模糊,区分道路区域与背景区域难度较大。针对现有非结构化道路识别方法存在全像素域计算分类处理实时性差、易受噪声数据干扰等问题,提出一种基于SLIC（simple linear iterative clustering）超像素分割和改进区域生长算法的非结构化道路识别方法。利用均匀化初始聚类中心的SLIC算法生成低分辨率超像素特征图。在此基础上,利用聚类算法与邻域搜索算法自适应选择种子点,并引入CIEDE2000色差理论作为区域生长法生长准则,初步确定道路区域。根据道路连续一致特点,优化超像素级生长图并映射轮廓区域至原图,获得道路最终区域。基于数据集及真实场景的实验结果表明,该方法具有较高的识别率和抗干扰能力。     

高速公路中的行车道检测和车辆跟踪

提出了一种有效的高速公路检测和多车辆跟踪视觉系统.该系统主要模块包括道路检测、基于二维模型的车辆跟踪器(近距离车辆的矩形模型和远距离车辆的U形模型)、启发式车辆检测、系统协调器.在系统中,跟踪器的动态产生和终止优化了系统的计算资源.另外,系统利用鲁棒性估计技术提高了道路检测的性能.车辆的跟踪是采用三参数状态空间的多边形拟合技术来实现的.本文采用了PETS2001提供的图像序列测试了系统有效性,在Pentium Ⅲ 450MHz PC 上系统的处理速度为平均每帧12毫秒.