Intelligent visual surveillance — A survey

Detection, tracking, and understanding of moving objects of interest in dynamic scenes have been active research areas in computer vision over the past decades. Intelligent visual surveillance (IVS) refers to an automated visual monitoring process that involves analysis and interpretation of object behaviors, as well as object detection and tracking, to understand the visual events of the scene. Main tasks of IVS include scene interpretation and wide area surveillance control. Scene interpretation aims at detecting and tracking moving objects in an image sequence and understanding their behaviors. In wide area surveillance control task, multiple cameras or agents are controlled in a cooperative manner to monitor tagged objects in motion. This paper reviews recent advances and future research directions of these tasks. This article consists of two parts: The first part surveys image enhancement, moving object detection and tracking, and motion behavior understanding. The second part reviews wide-area surveillance techniques based on the fusion of multiple visual sensors, camera calibration and cooperative camera systems.     

Passivity-based Visual Motion Observer with Panoramic Camera for Pose Control

This paper considers the vision-based estimation and pose control with a panoramic camera via passivity approach. First, a hyperbolic projection of a panoramic camera is presented. Next, using standard body-attached coordinate frames (the world frame, mirror frame, camera frame and object frame), we represent the body velocity of the relative rigid body motion (position and orientation). After that, we propose a visual motion observer to estimate the relative rigid body motion from the measured camera data. We show that the estimation error system with a panoramic camera has the passivity which allows us to prove stability in the sense of Lyapunov. The visual motion error system which consists of the estimation error system and the pose control error system preserves the passivity. After that, stability and L ₂-gain performance analysis for the closed-loop system are discussed via Lyapunov method and dissipative systems theory, respectively. Finally, simulation and experimental results are shown in order to confirm the proposed method.     

Visual Servoing Based on Structure From Controlled Motion or on Robust Statistics

This paper focuses on the way to achieve accurate visual servoing tasks when the shape of the object being observed as well as the desired image are unknown. More precisely, we want to control the camera orientation with respect to the tangent plane at a certain object point corresponding to the center of a region of interest. We also want to observe this point at the principal point to fulfil a fixation task. A 3-D reconstruction phase must, therefore, be performed during the camera motion. Our approach is then close to the structure-from-motion problem. The reconstruction phase is based on the measurement of the 2-D motion in a region of interest and on the measurement of the camera velocity. Since the 2-D motion depends on the shape of the objects being observed, we introduce a unified motion model to cope both with planar and nonplanar objects. However, since this model is only an approximation, we propose two approaches to enlarge its domain of validity. The first is based on active vision, coupled with a 3-D reconstruction based on a continuous approach, and the second is based on statistical techniques of robust estimation, coupled with a 3-D reconstruction based on a discrete approach. Theoretical and experimental results compare both approaches.     

Autonomous selection and modification of camera configurations using visibility and manipulability measures

In this article, we present a camera control method in which the selection of an optimal camera position and the modification of camera configurations are accomplished according to changes in the surroundings. For the autonomous selection and modification of camera configurations during tasks, we consider the camera's visibility and the manipulator's manipulability. The visibility constraint guarantees that the whole of a target object can be “viewed” with no occlusions by the surroundings, and the manipulability constraint guarantees avoidance of the singular position of the manipulator and rapid modification of the camera position. By considering visibility and manipulability constraints simultaneously, we determine the optimal camera position and modify the camera configuration such that visual information for the target object can be obtained continuously during tasks. The results of simulations and experiments show that the active camera system with an eye‐in‐hand configuration can modify its configuration autonomously according to the motion of the surroundings by applying the proposed camera control method. © 2002 Wiley Periodicals, Inc.     

Unmanned aerial vehicles UAVs attitude,height, motion estimation and control using visual systems

This paper presents an implementation of an aircraft pose and motion estimator using visual systems as the principal sensor for controlling an Unmanned Aerial Vehicle (UAV) or as a redundant system for an Inertial Measure Unit (IMU) and gyros sensors. First, we explore the applications of the unified theory for central catadioptric cameras for attitude and heading estimation, explaining how the skyline is projected on the catadioptric image and how it is segmented and used to calculate the UAV’s attitude. Then we use appearance images to obtain a visual compass, and we calculate the relative rotation and heading of the aerial vehicle. Additionally, we show the use of a stereo system to calculate the aircraft height and to measure the UAV’s motion. Finally, we present a visual tracking system based on Fuzzy controllers working in both a UAV and a camera pan and tilt platform. Every part is tested using the UAV COLIBRI platform to validate the different approaches, which include comparison of the estimated data with the inertial values measured onboard the helicopter platform and the validation of the tracking schemes on real flights.     

RSLAM: A System for Large-Scale Mapping in Constant-Time Using Stereo

Large scale exploration of the environment requires a constant time estimation engine. Bundle adjustment or pose relaxation do not fulfil these requirements as the number of parameters to solve grows with the size of the environment. We describe a relative simultaneous localisation and mapping system (RSLAM) for the constant-time estimation of structure and motion using a binocular stereo camera system as the sole sensor. Achieving robustness in the presence of difficult and changing lighting conditions and rapid motion requires careful engineering of the visual processing, and we describe a number of innovations which we show lead to high accuracy and robustness. In order to achieve real-time performance without placing severe limits on the size of the map that can be built, we use a topo-metric representation in terms of a sequence of relative locations. When combined with fast and reliable loop-closing, we mitigate the drift to obtain highly accurate global position estimates without any global minimisation. We discuss some of the issues that arise from using a relative representation, and evaluate our system on long sequences processed at a constant 30–45 Hz, obtaining precisions down to a few meters over distances of a few kilometres.     

1-Point-RANSAC Structure from Motion for Vehicle-Mounted Cameras by Exploiting Non-holonomic Constraints

This paper presents a new method to estimate the relative motion of a vehicle from images of a single camera. The computational cost of the algorithm is limited only by the feature extraction and matching process, as the outlier removal and the motion estimation steps take less than a fraction of millisecond with a normal laptop computer. The biggest problem in visual motion estimation is data association; matched points contain many outliers that must be detected and removed for the motion to be accurately estimated. In the last few years, a very established method for removing outliers has been the “5-point RANSAC” algorithm which needs a minimum of 5 point correspondences to estimate the model hypotheses. Because of this, however, it can require up to several hundreds of iterations to find a set of points free of outliers. In this paper, we show that by exploiting the nonholonomic constraints of wheeled vehicles it is possible to use a restrictive motion model which allows us to parameterize the motion with only 1 point correspondence. Using a single feature correspondence for motion estimation is the lowest model parameterization possible and results in the two most efficient algorithms for removing outliers: 1-point RANSAC and histogram voting. To support our method we run many experiments on both synthetic and real data and compare the performance with a state-of-the-art approach. Finally, we show an application of our method to visual odometry by recovering a 3 Km trajectory in a cluttered urban environment and in real-time.     

Geometric particle swarm optimization for robust visual ego-motion estimation via particle filtering

Conventional particle filtering-based visual ego-motion estimation or visual odometry often suffers from large local linearization errors in the case of abrupt camera motion. The main contribution of this paper is to present a novel particle filtering-based visual ego-motion estimation algorithm that is especially robust to the abrupt camera motion. The robustness to the abrupt camera motion is achieved by multi-layered importance sampling via particle swarm optimization (PSO), which iteratively moves particles to higher likelihood region without local linearization of the measurement equation. Furthermore, we make the proposed visual ego-motion estimation algorithm in real-time by reformulating the conventional vector space PSO algorithm in consideration of the geometry of the special Euclidean group SE(3), which is a Lie group representing the space of 3-D camera poses. The performance of our proposed algorithm is experimentally evaluated and compared with the local linearization and unscented particle filter-based visual ego-motion estimation algorithms on both simulated and real data sets.     

面向室内场景的主被动融合视觉定位系统

目的 视觉定位旨在利用易于获取的RGB图像对运动物体进行目标定位及姿态估计。室内场景中普遍存在的物体遮挡、弱纹理区域等干扰极易造成目标关键点的错误估计,严重影响了视觉定位的精度。针对这一问题,本文提出一种主被动融合的室内定位系统,结合固定视角和移动视角的方案优势,实现室内场景中运动目标的精准定位。方法 提出一种基于平面先验的物体位姿估计方法,在关键点检测的单目定位框架基础上,使用平面约束进行3自由度姿态优化,提升固定视角下室内平面中运动目标的定位稳定性。基于无损卡尔曼滤波算法设计了一套数据融合定位系统,将从固定视角得到的被动式定位结果与从移动视角得到的主动式定位结果进行融合,提升了运动目标的位姿估计结果的可靠性。结果 本文提出的主被动融合室内视觉定位系统在iGibson仿真数据集上的平均定位精度为2～3 cm,定位误差在10 cm内的准确率为99%;在真实场景中平均定位精度为3～4 cm,定位误差在10 cm内的准确率在90%以上,实现了cm级的定位精度。结论 提出的室内视觉定位系统融合了被动式和主动式定位方法的优势,能够以较低设备成本实现室内场景中高精度的目标定位结果,并在遮挡、目标...     

双舵轮AGV视觉里程计的设计方法

针对双舵轮AGV在地面崎岖不平和轮胎打滑的情况下编码器失效的问题.本文提出一种使用价格低廉的RGB-D相机做视觉里程计的方案,避免了双舵轮AGV直接运动学建模导致里程计航迹推算累积误差过大的问题.本文采用ORB算子对图像进行特征提取和匹配,使用ICP的方法进行位姿估计.然后在Linux+ROS平台下搭建视觉里程计,并且和激光雷达数据融合,通过粒子滤波算法进行定位.最后,分别在不同环境下对比了编码器和视觉里程计的定位效果,并验证了整个系统的鲁棒性.     

基于最大后验概率的单视频时间超分辨率重建算法

任何视频摄像设备均具有一定的时间分辨率限制,时间分辨率不足会造成视频中存在运动模糊和运动混叠现象。针对这一问题常用的解决方法是空间去模糊和时间插值,然而这些方法无法从根本上解决问题。提出一种基于最大后验概率(MAP)的单视频时间超分辨率重建方法,该方法通过重建约束来确定条件概率模型,然后利用视频自身具有的时间自相似先验信息得到先验信息模型,最后求得基于最大后验概率的估计值,即通过对单个低时间分辨率视频重建来得到高时间分辨率视频,从而有效解决由于相机曝光时间过长所造成的“运动模糊”和相机帧率不足引起的“运动混叠”现象。通过理论分析与实验,证明了所提方法的有效性。     

Visual odometry for ground vehicle applications

We present a system that estimates the motion of a stereo head, or a single moving camera, based on video input. The system operates in real time with low delay, and the motion estimates are used for navigational purposes. The front end of the system is a feature tracker. Point features are matched between pairs of frames and linked into image trajectories at video rate. Robust estimates of the camera motion are then produced from the feature tracks using a geometric hypothesize‐and‐test architecture. This generates motion estimates from visual input alone. No prior knowledge of the scene or the motion is necessary. The visual estimates can also be used in conjunction with information from other sources, such as a global positioning system, inertia sensors, wheel encoders, etc. The pose estimation method has been applied successfully to video from aerial, automotive, and handheld platforms. We focus on results obtained with a stereo head mounted on an autonomous ground vehicle. We give examples of camera trajectories estimated in real time purely from images over previously unseen distances (600 m) and periods of time. © 2006 Wiley Periodicals, Inc.     

Planar-Based Visual Inertial Navigation: Observability Analysis and Motion Estimation

In this paper, we address the problem of ego-motion estimation by fusing visual and inertial information. The hardware consists of an inertial measurement unit (IMU) and a monocular camera. The camera provides visual observations in the form of features on a horizontal plane. Exploiting the geometric constraint of features on the plane into visual and inertial data, we propose a novel closed form measurement model for this system. Our first contribution in this paper is an observability analysis of the proposed planar-based visual inertial navigation system (VINS). In particular, we prove that the system has only three unobservable states corresponding to global translations parallel to the plane, and rotation around the gravity vector. Hence, compared to general VINS, an advantage of using features on the horizontal plane is that the vertical translation along the normal of the plane becomes observable. As the second contribution, we present a state-space formulation for the pose estimation in the analyzed system and solve it via a modified unscented Kalman filter (UKF). Finally, the findings of the theoretical analysis and 6-DoF motion estimation are validated by simulations as well as using experimental data.     

A two‐stage sampling for robust feature matching

For any visual feature‐based SLAM (simultaneous localization and mapping) solutions, to estimate the relative camera motion between two images, it is necessary to find “correct” correspondence between features extracted from those images. Given a set of feature correspondents, one can use a n‐point algorithm with robust estimation method, to produce the best estimate to the relative camera pose. The accuracy of a motion estimate is heavily dependent on the accuracy of the feature correspondence. Such a dependency is even more significant when features are extracted from the images of the scenes with drastic changes in viewpoints and illuminations and presence of occlusions. To make a feature matching robust to such challenging scenes, we propose a new feature matching method that incrementally chooses a five pairs of matched features for a full DoF (degree of freedom) camera motion estimation. In particular, at the first stage, we use our 2‐point algorithm to estimate a camera motion and, at the second stage, use this estimated motion to choose three more matched features. In addition, we use, instead of the epipolar constraint, a planar constraint for more accurate outlier rejection. With this set of five matching features, we estimate a full DoF camera motion with scale ambiguity. Through the experiments with three, real‐world data sets, our method demonstrates its effectiveness and robustness by successfully matching features (1) from the images of a night market where presence of frequent occlusions and varying illuminations, (2) from the images of a night market taken by a handheld camera and by the Google street view, and (3) from the images of a same location taken daytime and nighttime.     

基于线特征与内区域特征的视觉伺服解耦控制

针对多自由度机械臂快速趋近任意四边形态目标的视觉伺服控制难题，提出了结合线特征与内区域特征的机器人视觉伺服解耦控制方法.构建了目标内区域特征以指导相机的平移运动速率，利用目标的线特征给出相机的旋转角速率，并通过引入内区域特征的矢量补偿和质心坐标的位置补偿，实现了平移和旋转控制的部分解耦.最后，对机器人视觉伺服控制系统进行了稳定性分析.仿真验证结果表明所提方法能控制相机以较快而平滑的动作收敛到期望位姿，且在相机光轴与目标平面近似垂直的条件下能较好地克服深度估计造成的不确定性问题.     

Self Calibration of the Fixation Movement of a Stereo Camera Head

In this article we show how an active stereo camera head can be made to autonomously learn to fixate objects in space. During fixatio n, the system performs an initial and a correction saccade. In the learning phase the correction saccade is controlled by a crude prewired algorithm, in analogy to a mechanism surmised to exist in the brainstem. A vector-based neural network serves as the adaptive component in our system. A self-organizing fovea improves dramatically the convergence of the learning algorithm and the accuracy of the fixation. As a possible application we describe the visuo-motor coordination of the camera head with an anthropomorphic robot arm.     

Self Calibration of the Fixation Movement of a Stereo Camera Head

In this article we show how an active stereo camera head can be made to autonomously learn to fixate objects in space. During fixation, the system performs an initial and a correction saccade. In the learning phase the correction saccade is controlled by a crude pre-wired algorithm, in analogy to a mechanism surmised to exist in the brainstem. A vector-based neural network serves as the adaptive component in our system. A self-organizing fovea improves dramatically the convergence of the learning algorithm and the accuracy of the fixation. As a possible application we describe the visuo-motor coordination of the camera head with an anthropomorphic robot arm.     

Performance evaluation of 1‐point‐RANSAC visual odometry

Monocular visual odometry is the process of computing the egomotion of a vehicle purely from images of a single camera. This process involves extracting salient points from consecutive image pairs, matching them, and computing the motion using standard algorithms. This paper analyzes one of the most important steps toward accurate motion computation, which is outlier removal. The random sample consensus (RANSAC) has been established as the standard method for model estimation in the presence of outliers. RANSAC is an iterative method, and the number of iterations necessary to find a correct solution is exponential in the minimum number of data points needed to estimate the model. It is therefore of utmost importance to find the minimal parameterization of the model to estimate. For unconstrained motion [six degrees of freedom (DoF)] of a calibrated camera, this would be five correspondences. In the case of planar motion, the motion model complexity is reduced (three DoF) and can be parameterized with two points. In this paper we show that when the camera is installed on a nonholonomic wheeled vehicle, the model complexity reduces to two DoF and therefore the motion can be parameterized with a single‐point correspondence. Using a single‐feature correspondence for motion estimation is the lowest model parameterization possible and results in the most efficient algorithm for removing outliers, which we call 1‐point RANSAC. To support our method, we run many experiments on both synthetic and real data and compare the performance with state‐of‐the‐art approaches and with different vehicles, both indoors and outdoors. © 2011 Wiley Periodicals, Inc.     

Structure from controlled motion

This paper deals with the recovery of 3D information using a single mobile camera in the context of active vision. First, we propose a general revisited formulation of the structure-from-known-motion issue. Within the same formalism, we handle various kinds of 3D geometrical primitives such as points, lines, cylinders, spheres, etc. We also aim at minimizing effects of the different measurement errors which are involved in such a process. More precisely, we mathematically determine optimal camera configurations and motions which lead to a robust and accurate estimation of the 3D structure parameters. We apply the visual servoing approach to perform these camera motions using a control law in closed-loop with respect to visual data. Real-time experiments dealing with 3D structure estimation of points and cylinders are reported. They demonstrate that this active vision strategy can very significantly improve the estimation accuracy     

Linearized motion estimation for articulated planes

In this paper, we describe the explicit application of articulation constraints for estimating the motion of a system of articulated planes. We relate articulations to the relative homography between planes and show that these articulations translate into linearized equality constraints on a linear least-squares system, which can be solved efficiently using a Karush-Kuhn-Tucker system. The articulation constraints can be applied for both gradient-based and feature-based motion estimation algorithms and to illustrate this, we describe a gradient-based motion estimation algorithm for an affine camera and a feature-based motion estimation algorithm for a projective camera that explicitly enforces articulation constraints. We show that explicit application of articulation constraints leads to numerically stable estimates of motion. The simultaneous computation of motion estimates for all of the articulated planes in a scene allows us to handle scene areas where there is limited texture information and areas that leave the field of view. Our results demonstrate the wide applicability of the algorithm in a variety of challenging real-world cases such as human body tracking, motion estimation of rigid, piecewise planar scenes, and motion estimation of triangulated meshes.