Can You See Me Now? Sensor Positioning for Automated and Persistent Surveillance

Most existing camera placement algorithms focus on coverage and/or visibility analysis, which ensures that the object of interest is visible in the camera's field of view (FOV). However, visibility, which is a fundamental requirement of object tracking, is insufficient for automated persistent surveillance. In such applications, a continuous consistently labeled trajectory of the same object should be maintained across different camera views. Therefore, a sufficient uniform overlap between the cameras' FOVs should be secured so that camera handoff can successfully and automatically be executed before the object of interest becomes untraceable or unidentifiable. In this paper, we propose sensor-planning methods that improve existing algorithms by adding handoff rate analysis. Observation measures are designed for various types of cameras so that the proposed sensor-planning algorithm is general and applicable to scenarios with different types of cameras. The proposed sensor-planning algorithm preserves necessary uniform overlapped FOVs between adjacent cameras for an optimal balance between coverage and handoff success rate. In addition, special considerations such as resolution and frontal-view requirements are addressed using two approaches: 1) direct constraint and 2) adaptive weights. The resulting camera placement is compared with a reference algorithm published by Erdem and Sclaroff. Significantly improved handoff success rates and frontal-view percentages are illustrated via experiments using indoor and outdoor floor plans of various scales.      

Camera handoff with adaptive resource management for multi-camera multi-object tracking

Camera handoff is a crucial step to obtain a continuously tracked and consistently labeled trajectory of the object of interest in multi-camera surveillance systems. Most existing camera handoff algorithms concentrate on data association, namely consistent labeling, where images of the same object are identified across different cameras. However, there exist many unsolved questions in developing an efficient camera handoff algorithm. In this paper, we first design a trackability measure to quantitatively evaluate the effectiveness of object tracking so that camera handoff can be triggered timely and the camera to which the object of interest is transferred can be selected optimally. Three components are considered: resolution, distance to the edge of the camera’s field of view (FOV), and occlusion. In addition, most existing real-time object tracking systems see a decrease in the frame rate as the number of tracked objects increases. To address this issue, our handoff algorithm employs an adaptive resource management mechanism to dynamically allocate cameras’ resources to multiple objects with different priorities so that the required minimum frame rate is maintained. Experimental results illustrate that the proposed camera handoff algorithm can achieve a substantially improved overall tracking rate by 20% in comparison with the algorithm presented by Khan and Shah.     

Inference topology of distributed camera networks with multiple cameras

This paper proposes an inference method to construct the topology of a camera network with overlapping and non-overlapping fields of view for a commercial surveillance system equipped with multiple cameras. It provides autonomous object detection, tracking and recognition in indoor or outdoor urban environments. The camera network topology is estimated from object tracking results among and within FOVs. The merge-split method is used for object occlusion in a single camera and an EM-based approach for extracting the accurate object feature to track moving people and establishing object correspondence across multiple cameras. The appearance of moving people and the transition time between entry and exit zones is measured to track moving people across blind regions of multiple cameras with non-overlapping FOVs. Our proposed method graphically represents the camera network topology, as an undirected weighted graph using the transition probabilities and 8-directional chain code. The training phase and the test were run with eight cameras to evaluate the performance of our method. The temporal probability distribution and the undirected weighted graph are shown in the experiments.     

An efficient continuous tracking system in real-time surveillance application

A tracking object must present a proper field of view (FOV) in a multiple active camera surveillance system; its clarity can facilitate smooth processing by the surveillance system before further processing, such as face recognition. However, when pan–tilt–zoom (PTZ) cameras are used, the tracking object can be brought into the FOV by adjusting its intrinsic parameters; consequently, selection of the best-performing camera is critical. Performance is determined by the relative positions of the camera and the tracking objects, image quality, lighting and how much of the front side of the object faces the camera. In a multi-camera surveillance system, both camera hand-off and camera assignment play an important role in automated and persistent tracking, which are typical surveillance requirements. This study investigates the use of automatic methods for tracking an object across cameras in a surveillance network using PTZ cameras. An automatic, efficient continuous tracking scheme is developed. The goal is to determine the decision criteria for hand-off using Sight Quality Indication (SQI) (which includes information on the position of the object and the proportion of the front of object faces the camera), and to perform the camera handoff task in a manner that optimizes the vision effect associated with monitoring. Experimental results reveal that the proposed algorithm can be efficiently executed, and the handoff method for feasible and continuously tracking active objects under real-time surveillance.     

Reliability-based camera handoff for cooperative tracking with multiple pan-tilt cameras

We present a new handoff method for multiple pan-tilt cameras for mobile robot tracking in an indoor environment. Camera handoff is an important step to consistently maintain the visibility of a mobile robot with maximized object tracking accuracy. First, we propose a method to estimate the position of a mobile robot using single pan-tilt camera. Then, the concept of position reliability is defined to quantitatively evaluate the accuracy of position estimation and tracking ability of individual pan-tilt cameras. Position reliability is used to decide when to trigger handoff and who to response handoff in the proposed handoff algorithm. Experimental results demonstrate that four pan-tilt cameras can systematically track a mobile robot in an indoor environment using the proposed method.     

基于全景视觉的监控系统设计

设计了一个基于全景视觉的多摄像机监控网络。全景相机视野广,可以实现大范围的目标检测与跟踪。云台摄像机视角具有一定的自由度,可以捕捉目标的高分辨率图像。将全景相机与云台相机相互配合,通过多传感器的数据融合,分层次的跟踪算法及多相机调度算法,实现了大范围的多个运动目标的检测与跟踪,并能捕获目标的清晰图像。实验验证了该系统的有效性和合理性。     

基于场景模型的多摄像机目标跟踪算法

在多个相机组成的视频监视系统中,当目标物移出某一相机的视野而进入下一个时,如何实现相机的交接,实现目标物的继续跟踪是监视系统中要解决的关键问题。针对该问题,提出了一种基于位置比较的多摄像机运动目标跟踪方法。为获得目标物的位置,建立多个相机与目标物世界坐标之间映射关系的场景模型,并根据目标物出现在不同相机之间的视野边界线上的瞬间时刻的位置来给出重叠视野的边界线。由此可对任意角度摆放的多个具有重叠视野的相机之间运行的目标物进行接力跟踪。该方法可以适应多个目标物同时进入场景的情况,实验结果表明,该方法具有较高的鲁棒性,能够满足视频跟踪的实时性要求。     

Bayesian-competitive consistent labeling for people surveillance

This paper presents a novel and robust approach to consistent labeling for people surveillance in multi-camera systems. A general framework scalable to any number of cameras with overlapped views is devised. An off-line training process automatically computes ground-plane homography and recovers epipolar geometry. When a new object is detected in any one camera, hypotheses for potential matching objects in the other cameras are established. Each of the hypotheses is evaluated using a prior and likelihood value. The prior accounts for the positions of the potential matching objects, while the likelihood is computed by warping the vertical axis of the new object on the field of view of the other cameras and measuring the amount of match. In the likelihood, two contributions (forward and backward) are considered so as to correctly handle the case of groups of people merged into single objects. Eventually, a maximum-a-posteriori approach estimates the best label assignment for the new object. Comparisons with other methods based on homography and extensive outdoor experiments demonstrate that the proposed approach is accurate and robust in coping with segmentation errors and in disambiguating groups.     

多摄像机目标跟踪系统设计

针对目前大多数基于视频的目标检测、跟踪系统中,仅对单摄像机视频进行目标分析时存在视野有限、无法处理目标遮挡等问题,搭建了一个基于多摄像机的目标检测与跟踪系统,并给出了系统的总体设计方案;详细阐述了多摄像机系统主要模块如视频采集、目标检测与跟踪、多机通讯、数据管理和实时用户交互等的设计思路,针对多摄像机之间的目标交接、云台控制、多机实时通讯、数据编码等关键技术进行了深入分析研究,提出了相应的解决方法,实现了多摄像机之间的目标交接、连续检测与跟踪等功能。     

A Catadioptric and Pan-Tilt-Zoom Camera Pair Object Tracking System for UAVs

Unmanned aerial vehicles (UAVs) are seeing widespread use in military, scientific, and civilian sectors in recent years. As the mission demands increase, these systems are becoming more complicated. Omnidirectional camera is a vision sensor that can captures 360° view in a single frame. In recent years omnidirectional camera usage has experienced a remarkable increase in many fields, where many innovative research has been done. Although, it is very promising, employment of omnidirectional cameras in UAVs is quite new. In this paper, an innovative sensory system is proposed, that has an omnidirectional imaging device and a pan tilt zoom (PTZ) camera. Such a system combines the advantages of both of the camera systems. The system can track any moving object within its 360° field of view and provide detailed images of it. The detection of the moving object has been accomplished by an adaptive background subtraction method implemented on the lowered resolution images of the catadioptric camera. A novel algorithm has also been developed to estimate the relative distance of the object with respect to the UAV, using tracking information of both of the cameras. The algorithms are implemented on an experimental system to validate the approach.     

Probabilistic-topological calibration of widely distributed camera networks

We propose a method for estimating the topology of distributed cameras, which can provide useful information for multi-target tracking in a wide area, without object identification among the fields of view (FOVs) of the cameras. In our method, each camera first detects objects in its observed images independently in order to obtain the positions/times where/when the objects enter/exit its FOV. Each obtained data is tentatively paired with all other data detected before the data is observed. A transit time between each paired data and their x–y coordinates are then computed. Based on classifying the distribution of the transit times and the x–y coordinates, object routes between FOVs can be detected. The classification is achieved by simple and robust vector quantization. The detected routes are then categorized to acquire the probabilistic-topological information of distributed cameras. In addition, offline tracking of observed objects can be realized by means of the calibration process. Experiments demonstrated that our method could automatically estimate the topological relationships of the distributed cameras and the object transits among them.     

Consistent labeling of tracked objects in multiple cameras with overlapping fields of view

We address the issue of tracking moving objects in an environment covered by multiple uncalibrated cameras with overlapping fields of view, typical of most surveillance setups. In such a scenario, it is essential to establish correspondence between tracks of the same object, seen in different cameras, to recover complete information about the object. We call this the problem of consistent labeling of objects when seen in multiple cameras. We employ a novel approach of finding the limits of field of view (FOV) of each camera as visible in the other cameras. We show that, if the FOV lines are known, it is possible to disambiguate between multiple possibilities for correspondence. We present a method to automatically recover these lines by observing motion in the environment, Furthermore, once these lines are initialized, the homography between the views can also be recovered. We present results on indoor and outdoor sequences containing persons and vehicles.     

Extrinsic calibration of heterogeneous cameras by line images

The extrinsic calibration refers to determining the relative pose of cameras. Most of the approaches for cameras with non-overlapping fields of view (FOV) are based on mirror reflection, object tracking or rigidity constraint of stereo systems whereas cameras with overlapping FOV can be calibrated using structure from motion solutions. We propose an extrinsic calibration method within structure from motion framework for cameras with overlapping FOV and its extension to cameras with partially non-overlapping FOV. Recently, omnidirectional vision has become a popular topic in computer vision as an omnidirectional camera can cover large FOV in one image. Combining the good resolution of perspective cameras and the wide observation angle of omnidirectional cameras has been an attractive trend in multi-camera system. For this reason, we present an approach which is applicable to heterogeneous types of vision sensors. Moreover, this method utilizes images of lines as these features possess several advantageous characteristics over point features, especially in urban environment. The calibration consists of a linear estimation of orientation and position of cameras and optionally bundle adjustment to refine the extrinsic parameters.     

Distributed wide-area multi-object tracking with non-overlapping camera views

We present a distributed system for wide-area multi-object tracking across disjoint camera views. Every camera in the system performs multi-object tracking, and keeps its own trackers and trajectories. The data from multiple features are exchanged between adjacent cameras for object matching. We employ a probabilistic Petri Net-based approach to account for the uncertainties of the vision algorithms (such as unreliable background subtraction, and tracking failure) and to incorporate the available domain knowledge. We combine appearance features of objects as well as the travel-time evidence for target matching and consistent labeling across disjoint camera views. 3D color histogram, histogram of oriented gradients, local binary patterns, object size and aspect ratio are used as the appearance features. The distribution of the travel time is modeled by a Gaussian mixture model. Multiple features are combined by the weights, which are assigned based on the reliability of the features. By incorporating the domain knowledge about the camera configurations and the information about the received packets from other cameras, certain transitions are fired in the probabilistic Petri net. The system is trained to learn different parameters of the matching process, and updated online. We first present wide-area tracking of vehicles, where we used three non-overlapping cameras. The first and the third cameras are approximately 150 m apart from each other with two intersections in the blind region. We also present an example of applying our method to a people-tracking scenario. The results show the success of the proposed method. A comparison between our work and related work is also presented.     

Spatial-temporal consistent labeling of tracked pedestrians across non-overlapping camera views

Tracking people across multiple cameras with non-overlapping views is a challenging task, since their observations are separated in time and space and their appearances may vary significantly. This paper proposes a Bayesian model to solve the consistent labeling problem across multiple non-overlapping camera views. Significantly different from related approaches, our model assumes neither people are well segmented nor their trajectories across camera views are estimated. We formulate a spatial-temporal probabilistic model in the hypothesis space that consists the potentially matched objects between the exit field of view (FOV) of one camera and the entry FOV of another camera. A competitive major color spectrum histogram representation (CMCSHR) for appearance matching between two objects is also proposed. The proposed spatial-temporal and appearance models are unified by a maximum-a-posteriori (MAP) Bayesian model. Based on this Bayesian model, when a detected new object corresponds to a group hypothesis (more than one object), we further develop an online method for online correspondence update using optimal graph matching (OGM) algorithm. Experimental results on three different real scenarios validate the proposed Bayesian model approach and the CMCSHR method. The results also show that the proposed approach is able to address the occlusion problem/group problem, i.e. finding the corresponding individuals in another camera view for a group of people who walk together into the entry FOV of a camera.     

一种基于投影不变量的目标交接改进方法

在多摄像机视频监控系统中,图像之间的视点对应以及目标的交接是重要的研究内容。不需要标定摄像机的参数,该文提出了一种利用尺度不变特征变换（SIFT：scale-invariant features transform）及融合颜色信息的投影不变量实现目标交接的方法。利用SIFT方法自动生成图像间匹配的特征点对,并由此生成视野分界线,然后利用融合颜色信息的投影不变量方法完成对多摄像机之间目标身份的确认。     

多摄像机视野分界线快速自动生成算法*

针对摄像机视野分界线是一种有效地解决多摄像机人体跟踪目标交接问题的工具,提出了一种基于同步视频的摄像机视野分界线快速自动生成算法,利用视野分界线和目标中心到视野分界线的距离实现多摄像机之间的目标交接.该算法不依赖摄像机的标定信息和目标颜色信息.为验证该算法的有效性,设计搭建了一个室内视觉有重叠区域的多摄像机人体跟踪系统.实验结果表明,该算法具有易实现、实时、准确率高的优点.     

监控系统中的多摄像机协同

描述了一个用于室内场合对多个目标进行跟踪的分布式监控系统.该系统由多个廉价的固定镜头的摄像机构成,具有多个摄像机处理模块和一个中央模块用于协调摄像机间的跟踪任务.由于每个运动目标有可能被多个摄像机同时跟踪,因此如何选择最合适的摄像机对某一目标跟踪,特别是在系统资源紧张时,成为一个问题.提出的新算法能根据目标与摄像机之间的距离并考虑到遮挡的情况,把目标分配给相应的摄像机,因此在遮挡出现时,系统能把遮挡的目标分配给能看见目标并距离最近的那个摄像机.实验表明该系统能协调好多个摄像机进行目标跟踪,并处理好遮挡问题.     

监控系统中的多摄像头协同算法

给出了一个分布式多摄像头监控系统结构,描述了一种使用路径模型和目标外形变化估计的数据融合方案,提出了一种新的在分布式监控系统中的多摄像头协同算法(PDA,PriorityandDistanceAlgorithm)。提出的算法基于多摄像头数据融合结果,根据任务优先级、目标与摄像头之间的距离及目标的可见性情况,分配摄像头给目标,其特点是使系统中有高优先级并距离摄像头最近的可见目标优先分配摄像头。实验结果表明提出算法能协同多摄像头可靠地跟踪人。     

Multi-view structure-from-motion for hybrid camera scenarios

We describe a pipeline for structure-from-motion (SfM) with mixed camera types, namely omnidirectional and perspective cameras. For the steps of this pipeline, we propose new approaches or adapt the existing perspective camera methods to make the pipeline effective and automatic. We model our cameras of different types with the sphere camera model. To match feature points, we describe a preprocessing algorithm which significantly increases scale invariant feature transform (SIFT) matching performance for hybrid image pairs. With this approach, automatic point matching between omnidirectional and perspective images is achieved. We robustly estimate the hybrid fundamental matrix with the obtained point correspondences. We introduce the normalization matrices for lifted coordinates so that normalization and denormalization can be performed linearly for omnidirectional images. We evaluate the alternatives of estimating camera poses in hybrid pairs. A weighting strategy is proposed for iterative linear triangulation which improves the structure estimation accuracy. Following the addition of multiple perspective and omnidirectional images to the structure, we perform sparse bundle adjustment on the estimated structure by adapting it to use the sphere camera model. Demonstrations of the end-to-end multi-view SfM pipeline with the real images of mixed camera types are presented.