Camera handoff and placement for automated tracking systems with multiple omnidirectional cameras

In a multi-camera surveillance system, both camera handoff and placement play an important role in generating an automated and persistent object tracking, typical of most surveillance requirements. Camera handoff should comprise three fundamental components, time to trigger handoff process, the execution of consistent labeling, and the selection of the next optimal camera. In this paper, we design an observation measure to quantitatively formulate the effectiveness of object tracking so that we can trigger camera handoff timely and select the next camera appropriately before the tracked object falls out of the field of view (FOV) of the currently observing camera. In the meantime, we present a novel solution to the consistent labeling problem in omnidirectional cameras. A spatial mapping procedure is proposed to consider both the noise inherent to the tracking algorithms used by the system and the lens distortion introduced by omnidirectional cameras. This does not only avoid the tedious process, but also increases the accuracy, to obtain the correspondence between omnidirectional cameras without human interventions. We also propose to use the Wilcoxon Signed-Rank Test to improve the accuracy of trajectory association between pairs of objects. In addition, since we need a certain amount of time to successfully carry out the camera handoff procedure, we introduce an additional constraint to optimally reserve sufficient cameras’ overlapped FOVs for the camera placement. Experiments show that our proposed observation measure can quantitatively formulate the effectiveness of tracking, so that camera handoff can smoothly transfer objects of interest. Meanwhile, our proposed consistent labeling approach can perform as accurately as the geometry-based approach without tedious calibration processes and outperform Calderara’s homography-based approach. Our proposed camera placement method exhibits a significant increase in the camera handoff success rate at the cost of slightly decreased coverage, as compared to Erdem and Sclaroff’s method without considering the requirement on overlapped FOVs.     

Camera handoff with adaptive resource management for multi-camera multi-object tracking

Camera handoff is a crucial step to obtain a continuously tracked and consistently labeled trajectory of the object of interest in multi-camera surveillance systems. Most existing camera handoff algorithms concentrate on data association, namely consistent labeling, where images of the same object are identified across different cameras. However, there exist many unsolved questions in developing an efficient camera handoff algorithm. In this paper, we first design a trackability measure to quantitatively evaluate the effectiveness of object tracking so that camera handoff can be triggered timely and the camera to which the object of interest is transferred can be selected optimally. Three components are considered: resolution, distance to the edge of the camera’s field of view (FOV), and occlusion. In addition, most existing real-time object tracking systems see a decrease in the frame rate as the number of tracked objects increases. To address this issue, our handoff algorithm employs an adaptive resource management mechanism to dynamically allocate cameras’ resources to multiple objects with different priorities so that the required minimum frame rate is maintained. Experimental results illustrate that the proposed camera handoff algorithm can achieve a substantially improved overall tracking rate by 20% in comparison with the algorithm presented by Khan and Shah.     

多摄像机视野分界线快速自动生成算法*

针对摄像机视野分界线是一种有效地解决多摄像机人体跟踪目标交接问题的工具,提出了一种基于同步视频的摄像机视野分界线快速自动生成算法,利用视野分界线和目标中心到视野分界线的距离实现多摄像机之间的目标交接.该算法不依赖摄像机的标定信息和目标颜色信息.为验证该算法的有效性,设计搭建了一个室内视觉有重叠区域的多摄像机人体跟踪系统.实验结果表明,该算法具有易实现、实时、准确率高的优点.     

Surveillance camera scheduling: a virtual vision approach

We present a surveillance system, comprising wide field-of-view (FOV) passive cameras and pan/tilt/zoom (PTZ) active cameras, which automatically captures high-resolution videos of pedestrians as they move through a designated area. A wide-FOV static camera can track multiple pedestrians, while any PTZ active camera can capture high-quality videos of one pedestrian at a time. We formulate the multi-camera control strategy as an online scheduling problem and propose a solution that combines the information gathered by the wide-FOV cameras with weighted round-robin scheduling to guide the available PTZ cameras, such that each pedestrian is observed by at least one PTZ camera while in the designated area. A centerpiece of our work is the development and testing of experimental surveillance systems within a visually and behaviorally realistic virtual environment simulator. The simulator is valuable as our research would be more or less infeasible in the real world given the impediments to deploying and experimenting with appropriately complex camera sensor networks in large public spaces. In particular, we demonstrate our surveillance system in a virtual train station environment populated by autonomous, lifelike virtual pedestrians, wherein easily reconfigurable virtual cameras generate synthetic video feeds. The video streams emulate those generated by real surveillance cameras monitoring richly populated public spaces.A preliminary version of this paper appeared as [1].     

Can You See Me Now? Sensor Positioning for Automated and Persistent Surveillance

Most existing camera placement algorithms focus on coverage and/or visibility analysis, which ensures that the object of interest is visible in the camera's field of view (FOV). However, visibility, which is a fundamental requirement of object tracking, is insufficient for automated persistent surveillance. In such applications, a continuous consistently labeled trajectory of the same object should be maintained across different camera views. Therefore, a sufficient uniform overlap between the cameras' FOVs should be secured so that camera handoff can successfully and automatically be executed before the object of interest becomes untraceable or unidentifiable. In this paper, we propose sensor-planning methods that improve existing algorithms by adding handoff rate analysis. Observation measures are designed for various types of cameras so that the proposed sensor-planning algorithm is general and applicable to scenarios with different types of cameras. The proposed sensor-planning algorithm preserves necessary uniform overlapped FOVs between adjacent cameras for an optimal balance between coverage and handoff success rate. In addition, special considerations such as resolution and frontal-view requirements are addressed using two approaches: 1) direct constraint and 2) adaptive weights. The resulting camera placement is compared with a reference algorithm published by Erdem and Sclaroff. Significantly improved handoff success rates and frontal-view percentages are illustrated via experiments using indoor and outdoor floor plans of various scales.      

多摄像机目标跟踪系统设计

针对目前大多数基于视频的目标检测、跟踪系统中,仅对单摄像机视频进行目标分析时存在视野有限、无法处理目标遮挡等问题,搭建了一个基于多摄像机的目标检测与跟踪系统,并给出了系统的总体设计方案;详细阐述了多摄像机系统主要模块如视频采集、目标检测与跟踪、多机通讯、数据管理和实时用户交互等的设计思路,针对多摄像机之间的目标交接、云台控制、多机实时通讯、数据编码等关键技术进行了深入分析研究,提出了相应的解决方法,实现了多摄像机之间的目标交接、连续检测与跟踪等功能。     

Systems and algorithms for autonomous and scalable crowd surveillance using robotic PTZ cameras assisted by a wide-angle camera

We report an autonomous surveillance system with multiple pan-tilt-zoom (PTZ) cameras assisted by a fixed wide-angle camera. The wide-angle camera provides large but low resolution coverage and detects and tracks all moving objects in the scene. Based on the output of the wide-angle camera, the system generates spatiotemporal observation requests for each moving object, which are candidates for close-up views using PTZ cameras. Due to the fact that there are usually much more objects than the number of PTZ cameras, the system first assigns a subset of the requests/objects to each PTZ camera. The PTZ cameras then select the parameter settings that best satisfy the assigned competing requests to provide high resolution views of the moving objects. We propose an approximation algorithm to solve the request assignment and the camera parameter selection problems in real time. The effectiveness of the proposed system is validated in both simulation and physical experiment. In comparison with an existing work using simulation, it shows that in heavy traffic scenarios, our algorithm increases the number of observed objects by over 210%.     

基于全景视觉的监控系统设计

设计了一个基于全景视觉的多摄像机监控网络。全景相机视野广,可以实现大范围的目标检测与跟踪。云台摄像机视角具有一定的自由度,可以捕捉目标的高分辨率图像。将全景相机与云台相机相互配合,通过多传感器的数据融合,分层次的跟踪算法及多相机调度算法,实现了大范围的多个运动目标的检测与跟踪,并能捕获目标的清晰图像。实验验证了该系统的有效性和合理性。     

一种基于投影不变量的目标交接改进方法

在多摄像机视频监控系统中,图像之间的视点对应以及目标的交接是重要的研究内容。不需要标定摄像机的参数,该文提出了一种利用尺度不变特征变换（SIFT：scale-invariant features transform）及融合颜色信息的投影不变量实现目标交接的方法。利用SIFT方法自动生成图像间匹配的特征点对,并由此生成视野分界线,然后利用融合颜色信息的投影不变量方法完成对多摄像机之间目标身份的确认。     

基于场景模型的多摄像机目标跟踪算法

在多个相机组成的视频监视系统中,当目标物移出某一相机的视野而进入下一个时,如何实现相机的交接,实现目标物的继续跟踪是监视系统中要解决的关键问题。针对该问题,提出了一种基于位置比较的多摄像机运动目标跟踪方法。为获得目标物的位置,建立多个相机与目标物世界坐标之间映射关系的场景模型,并根据目标物出现在不同相机之间的视野边界线上的瞬间时刻的位置来给出重叠视野的边界线。由此可对任意角度摆放的多个具有重叠视野的相机之间运行的目标物进行接力跟踪。该方法可以适应多个目标物同时进入场景的情况,实验结果表明,该方法具有较高的鲁棒性,能够满足视频跟踪的实时性要求。     

Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking

Pan–tilt–zoom (PTZ) cameras are well suited for object identification and recognition in far-field scenes. However, the effective use of PTZ cameras is complicated by the fact that a continuous online camera calibration is needed and the absolute pan, tilt and zoom values provided by the camera actuators cannot be used because they are not synchronized with the video stream. So, accurate calibration must be directly extracted from the visual content of the frames. Moreover, the large and abrupt scale changes, the scene background changes due to the camera operation and the need of camera motion compensation make target tracking with these cameras extremely challenging. In this paper, we present a solution that provides continuous online calibration of PTZ cameras which is robust to rapid camera motion, changes of the environment due to varying illumination or moving objects. The approach also scales beyond thousands of scene landmarks extracted with the SURF keypoint detector. The method directly derives the relationship between the position of a target in the ground plane and the corresponding scale and position in the image and allows real-time tracking of multiple targets with high and stable degree of accuracy even at far distances and any zoom level.     

Consistent labeling of tracked objects in multiple cameras with overlapping fields of view

We address the issue of tracking moving objects in an environment covered by multiple uncalibrated cameras with overlapping fields of view, typical of most surveillance setups. In such a scenario, it is essential to establish correspondence between tracks of the same object, seen in different cameras, to recover complete information about the object. We call this the problem of consistent labeling of objects when seen in multiple cameras. We employ a novel approach of finding the limits of field of view (FOV) of each camera as visible in the other cameras. We show that, if the FOV lines are known, it is possible to disambiguate between multiple possibilities for correspondence. We present a method to automatically recover these lines by observing motion in the environment, Furthermore, once these lines are initialized, the homography between the views can also be recovered. We present results on indoor and outdoor sequences containing persons and vehicles.     

监控系统中的多摄像机协同

描述了一个用于室内场合对多个目标进行跟踪的分布式监控系统.该系统由多个廉价的固定镜头的摄像机构成,具有多个摄像机处理模块和一个中央模块用于协调摄像机间的跟踪任务.由于每个运动目标有可能被多个摄像机同时跟踪,因此如何选择最合适的摄像机对某一目标跟踪,特别是在系统资源紧张时,成为一个问题.提出的新算法能根据目标与摄像机之间的距离并考虑到遮挡的情况,把目标分配给相应的摄像机,因此在遮挡出现时,系统能把遮挡的目标分配给能看见目标并距离最近的那个摄像机.实验表明该系统能协调好多个摄像机进行目标跟踪,并处理好遮挡问题.     

Probabilistic-topological calibration of widely distributed camera networks

We propose a method for estimating the topology of distributed cameras, which can provide useful information for multi-target tracking in a wide area, without object identification among the fields of view (FOVs) of the cameras. In our method, each camera first detects objects in its observed images independently in order to obtain the positions/times where/when the objects enter/exit its FOV. Each obtained data is tentatively paired with all other data detected before the data is observed. A transit time between each paired data and their x–y coordinates are then computed. Based on classifying the distribution of the transit times and the x–y coordinates, object routes between FOVs can be detected. The classification is achieved by simple and robust vector quantization. The detected routes are then categorized to acquire the probabilistic-topological information of distributed cameras. In addition, offline tracking of observed objects can be realized by means of the calibration process. Experiments demonstrated that our method could automatically estimate the topological relationships of the distributed cameras and the object transits among them.     

Intelligent visual surveillance — A survey

Detection, tracking, and understanding of moving objects of interest in dynamic scenes have been active research areas in computer vision over the past decades. Intelligent visual surveillance (IVS) refers to an automated visual monitoring process that involves analysis and interpretation of object behaviors, as well as object detection and tracking, to understand the visual events of the scene. Main tasks of IVS include scene interpretation and wide area surveillance control. Scene interpretation aims at detecting and tracking moving objects in an image sequence and understanding their behaviors. In wide area surveillance control task, multiple cameras or agents are controlled in a cooperative manner to monitor tagged objects in motion. This paper reviews recent advances and future research directions of these tasks. This article consists of two parts: The first part surveys image enhancement, moving object detection and tracking, and motion behavior understanding. The second part reviews wide-area surveillance techniques based on the fusion of multiple visual sensors, camera calibration and cooperative camera systems.     

视觉监控应用中多传感器协作的人脸检测系统

提出了一种新颖的由两个可控摄像机组成的多传感器视觉监控系统,旨在实现户外环境下的实时跟踪与特征化运动目标.特别地,该系统利用一个在多个缩放级别上可操作的移动摄像机在连续视频帧中自动获取与跟踪人脸.配合它的是一架能执行自动目标跟踪与分类的固定广域摄像机.     

Dynamic calibration of pan-tilt-zoom cameras for traffic monitoring.

Pan-tilt-zoom (PTZ) cameras have been widely used in recent years for monitoring and surveillance applications. These cameras provide flexible view selection as well as a wider observation range. This makes them suitable for vision-based traffic monitoring and enforcement systems. To employ PTZ cameras for image measurement applications, one first needs to calibrate the camera to obtain meaningful results. For instance, the accuracy of estimating vehicle speed depends on the accuracy of camera calibration and that of vehicle tracking results. This paper presents a novel calibration method for a PTZ camera overlooking a traffic scene. The proposed approach requires no manual operation to select the positions of special features. It automatically uses a set of parallel lane markings and the lane width to compute the camera parameters, namely, focal length, tilt angle, and pan angle. Image processing procedures have been developed for automatically finding parallel lane markings. Interesting experimental results are presented to validate the robustness and accuracy of the proposed method.     

Panoramic Gaussian Mixture Model and large-scale range background substraction method for PTZ camera-based surveillance systems

In this paper, we present a novel approach for constructing a large-scale range panoramic background model that provides fast registration of the observed frame and localizes the foreground targets with arbitrary camera direction and scale in a Pan–tilt–zoom (PTZ) camera-based surveillance system. Our method consists of three stages. (1) In the first stage, a panoramic Gaussian mixture model (PGMM) of the PTZ camera’s field of view is generated off-line for later use in on-line foreground detection. (2) In the second stage, a multi-layered correspondence ensemble is generated off-line from frames captured at different scales which is used by the correspondence propagation method to register observed frames online to the PGMM. (3) In the third stage, foreground is detected and the PGMM is updated. The proposed method has the capacity to deal with the PTZ camera’s ability to cover a wide field of view (FOV) and large-scale range. We demonstrate the advantages of the proposed PGMM background subtraction method by incorporating it with a tracking system for surveillance applications.     

Persistent people tracking and face capture using a PTZ camera

Pan–tilt–zoom (PTZ) camera is a powerful tool in far-field scenarios. However, most of the current PTZ surveillance systems require manual intervention to move the camera to the desired position. In this paper, we address the problem of persistent people tracking and face capture in uncontrolled scenarios using a single PTZ camera, which could prove most helpful in forensic applications. The system first detects and tracks pedestrians in zoomed-out mode. Then, according to a scheduler, the system selects a person to zoom in. In the zoomed-in mode, we detect a set of face images and solve the face–face association and face–person association problems. The system then zooms back out where tracking is continued as people re-appear in the view. The person–person association module associates the people on the schedule list with the people in the current view. The detected faces are associated with the corresponding people and trajectories. Due to the dynamic nature of our problem, e.g. the field of view of the camera changes because of the pan/tilt/zoom movement of the camera, all of the processes including receiving images from the camera and processing must be done in real time. To the best of our knowledge, the proposed method is the first to address the association of face images to people and trajectories using a single PTZ camera. Extensive experiments in challenging indoor and outdoor uncontrolled conditions demonstrate the effectiveness of the proposed system.     

An adaptive focus-of-attention model for video surveillance and monitoring

In current video surveillance systems, commercial pan/tilt/zoom (PTZ) cameras typically provide naive (or no) automatic scanning functionality to move a camera across its complete viewable field. However, the lack of scene-specific information inherently handicaps these scanning algorithms. We address this issue by automatically building an adaptive, focus-of-attention, scene-specific model using standard PTZ camera hardware. The adaptive model is constructed by first detecting local human activity (i.e., any translating object with a specific temporal signature) at discrete locations across a PTZ camera’s entire viewable field. The temporal signature of translating objects is extracted using motion history images (MHIs) and an original, efficient algorithm based on an iterative candidacy-classification-reduction process to separate the target motion from noise. The target motion at each location is then quantified and employed in the construction of a global activity map for the camera. We additionally present four new camera scanning algorithms which exploit this activity map to maximize a PTZ camera’s opportunity of observing human activity within the camera’s overall field of view. We expect that these efficient and effective algorithms are implementable within current commercial camera systems.