Camera handoff with adaptive resource management for multi-camera multi-object tracking

Camera handoff is a crucial step to obtain a continuously tracked and consistently labeled trajectory of the object of interest in multi-camera surveillance systems. Most existing camera handoff algorithms concentrate on data association, namely consistent labeling, where images of the same object are identified across different cameras. However, there exist many unsolved questions in developing an efficient camera handoff algorithm. In this paper, we first design a trackability measure to quantitatively evaluate the effectiveness of object tracking so that camera handoff can be triggered timely and the camera to which the object of interest is transferred can be selected optimally. Three components are considered: resolution, distance to the edge of the camera’s field of view (FOV), and occlusion. In addition, most existing real-time object tracking systems see a decrease in the frame rate as the number of tracked objects increases. To address this issue, our handoff algorithm employs an adaptive resource management mechanism to dynamically allocate cameras’ resources to multiple objects with different priorities so that the required minimum frame rate is maintained. Experimental results illustrate that the proposed camera handoff algorithm can achieve a substantially improved overall tracking rate by 20% in comparison with the algorithm presented by Khan and Shah.     

An efficient continuous tracking system in real-time surveillance application

A tracking object must present a proper field of view (FOV) in a multiple active camera surveillance system; its clarity can facilitate smooth processing by the surveillance system before further processing, such as face recognition. However, when pan–tilt–zoom (PTZ) cameras are used, the tracking object can be brought into the FOV by adjusting its intrinsic parameters; consequently, selection of the best-performing camera is critical. Performance is determined by the relative positions of the camera and the tracking objects, image quality, lighting and how much of the front side of the object faces the camera. In a multi-camera surveillance system, both camera hand-off and camera assignment play an important role in automated and persistent tracking, which are typical surveillance requirements. This study investigates the use of automatic methods for tracking an object across cameras in a surveillance network using PTZ cameras. An automatic, efficient continuous tracking scheme is developed. The goal is to determine the decision criteria for hand-off using Sight Quality Indication (SQI) (which includes information on the position of the object and the proportion of the front of object faces the camera), and to perform the camera handoff task in a manner that optimizes the vision effect associated with monitoring. Experimental results reveal that the proposed algorithm can be efficiently executed, and the handoff method for feasible and continuously tracking active objects under real-time surveillance.     

Systems and algorithms for autonomous and scalable crowd surveillance using robotic PTZ cameras assisted by a wide-angle camera

We report an autonomous surveillance system with multiple pan-tilt-zoom (PTZ) cameras assisted by a fixed wide-angle camera. The wide-angle camera provides large but low resolution coverage and detects and tracks all moving objects in the scene. Based on the output of the wide-angle camera, the system generates spatiotemporal observation requests for each moving object, which are candidates for close-up views using PTZ cameras. Due to the fact that there are usually much more objects than the number of PTZ cameras, the system first assigns a subset of the requests/objects to each PTZ camera. The PTZ cameras then select the parameter settings that best satisfy the assigned competing requests to provide high resolution views of the moving objects. We propose an approximation algorithm to solve the request assignment and the camera parameter selection problems in real time. The effectiveness of the proposed system is validated in both simulation and physical experiment. In comparison with an existing work using simulation, it shows that in heavy traffic scenarios, our algorithm increases the number of observed objects by over 210%.     

基于全景视觉的监控系统设计

设计了一个基于全景视觉的多摄像机监控网络。全景相机视野广,可以实现大范围的目标检测与跟踪。云台摄像机视角具有一定的自由度,可以捕捉目标的高分辨率图像。将全景相机与云台相机相互配合,通过多传感器的数据融合,分层次的跟踪算法及多相机调度算法,实现了大范围的多个运动目标的检测与跟踪,并能捕获目标的清晰图像。实验验证了该系统的有效性和合理性。     

Fast image motion segmentation for surveillance applications

Wireless, battery-powered camera networks are becoming of increasing interest for surveillance and monitoring applications. The computational power of these platforms is often limited in order to reduce energy consumption. In addition, many embedded processors do not have floating point support in hardware. Among the visual tasks that a visual sensor node may be required to perform, motion analysis is one of the most basic and relevant. Events of interest are usually characterized by the presence of moving objects or persons. Knowledge of the direction of motion and velocity of a moving body may be used to take actions such as sending an alarm or triggering other camera nodes in the network.We present a fast algorithm for identifying moving areas in an image. The algorithm is efficient and amenable to implementation in fixed point arithmetic. Once the moving blobs in an image have been precisely localized, the average velocity vector can be computed using a small number of floating point operations. Our procedure starts by determining an initial labeling of image blocks based on local differential analysis. Then, belief propagation is used to impose spatial coherence and to resolve aperture effect inherent in texture less areas. A detailed analysis of the computational cost of the algorithm and of the provisions that must be taken in order to avoid overflow with 32-bit words is included.     

基于HEIV模型的摄像机一维标定

多摄像机系统广泛应用于文化创意产业,其高精度标定是迫切需要解决的一个关键问题. 新近出现的摄像机一维标定方法能够克服标定物自身遮挡,特别适合标定多摄像机系统. 然而,现有的摄像机一维标定研究主要集中在降低一维标定物的运动约束,而标定精度较低的问题未受到应有的关注. 本文提出一种基于变量含异质噪声 （Heteroscedastic error-in-variables,HEIV）模型的高精度摄像机一维标定方法. 首先,推导出摄像机一维标定的计算模型;其次,利用该计算模型详细分析了一维标定中的噪声,得出摄像机一维标定可以视为一个HEIV问题的结论;最后给出了基于HEIV模型的摄像机一维标定算法. 与现有的算法相比,该方法可以显著改善一维标定的精度,并且受初始值影响小,收敛速度快. 实验结果验证了该方法的正确性和可行性.     

A LSS-based registration of stereo thermal–visible videos of multiple people using belief propagation

In this paper, we propose a novel stereo method for registering foreground objects in a pair of thermal and visible videos of close-range scenes. In our stereo matching, we use Local Self-Similarity (LSS) as similarity metric between thermal and visible images. In order to accurately assign disparities to depth discontinuities and occluded Region Of Interest (ROI), we have integrated color and motion cues as soft constraints in an energy minimization framework. The optimal disparity map is approximated for image ROIs using a Belief Propagation (BP) algorithm. We tested our registration method on several challenging close-range indoor video frames of multiple people at different depths, with different clothing, and different poses. We show that our global optimization algorithm significantly outperforms the existing state-of-the art method, especially for disparity assignment of occluded people at different depth in close-range surveillance scenes and for relatively large camera baseline.     

Balancing target acquisition and target tracking in existing resource constrained stationary camera systems

Safety security concerns drive the need to visually monitor large, complex environments (such as college campuses and office parks). Operators use networks of cameras to surveil areas in a dual fashion: first, identify events or targets of interest and second, monitor newly identified events and targets until a determination of subsequent action is made. An intelligent camera motion algorithm can assist operators with surveilling large areas by leveraging the region of interest configuration. This article compares the usefulness of camera motion planning by minimizing the time a space goes unseen (time based) against minimizing the size of contiguous unviewable space (space based). In addition, an autonomous, multi-camera, cooperative surveillance algorithm that operates a network of pan–tilt–zoom cameras to monitor a region of interest is presented. Coverage is accomplished via a novel algorithm that combines both time and space-based heuristics without an exhaustive search for the optimal parameters. This approach not only cooperatively manages camera coverage, but introduces a mechanism for balancing resource assignment between coverage to acquire new targets and tracking of previously discovered targets. Other benefits include a user-interface mechanism and occlusion awareness for expansion to cluttered urban environments. Experimental results are presented and compared for target acquisition and tracking in a typical urban campus environment. Results show that average linear uncovered length (ALUL) better predicts target acquisition performance. In addition, the use of ALUL for selection of camera panning angles improves target tracking and surveillance performance for both static and dynamic surveillance systems.     

监控系统中的多摄像头协同算法

给出了一个分布式多摄像头监控系统结构,描述了一种使用路径模型和目标外形变化估计的数据融合方案,提出了一种新的在分布式监控系统中的多摄像头协同算法(PDA,PriorityandDistanceAlgorithm)。提出的算法基于多摄像头数据融合结果,根据任务优先级、目标与摄像头之间的距离及目标的可见性情况,分配摄像头给目标,其特点是使系统中有高优先级并距离摄像头最近的可见目标优先分配摄像头。实验结果表明提出算法能协同多摄像头可靠地跟踪人。     

Intelligent visual surveillance — A survey

Detection, tracking, and understanding of moving objects of interest in dynamic scenes have been active research areas in computer vision over the past decades. Intelligent visual surveillance (IVS) refers to an automated visual monitoring process that involves analysis and interpretation of object behaviors, as well as object detection and tracking, to understand the visual events of the scene. Main tasks of IVS include scene interpretation and wide area surveillance control. Scene interpretation aims at detecting and tracking moving objects in an image sequence and understanding their behaviors. In wide area surveillance control task, multiple cameras or agents are controlled in a cooperative manner to monitor tagged objects in motion. This paper reviews recent advances and future research directions of these tasks. This article consists of two parts: The first part surveys image enhancement, moving object detection and tracking, and motion behavior understanding. The second part reviews wide-area surveillance techniques based on the fusion of multiple visual sensors, camera calibration and cooperative camera systems.     

Improved assignment with ant colony optimization for multi-target tracking

Detecting and tracking ground targets is crucial in military intelligence in battlefield surveillance. Once targets have been detected, the system used can proceed to track them where tracking can be done using Ground Moving Target Indicator (GMTI) type indicators that can observe objects moving in the area of interest. However, when targets move close to each other in formation as a convoy, then the problem of assigning measurements to targets has to be addressed first, as it is an important step in target tracking. With the increasing computational power, it became possible to use more complex association logic in tracking algorithms. Although its optimal solution can be proved to be an NP hard problem, the multidimensional assignment enjoyed a renewed interest mostly due to Lagrangian relaxation approaches to its solution. Recently, it has been reported that randomized heuristic approaches surpassed the performance of Lagrangian relaxation algorithm especially in dense problems. In this paper, impelled from the success of randomized heuristic methods, we investigate a different stochastic approach, namely, the biologically inspired ant colony optimization to solve the NP hard multidimensional assignment problem for tracking multiple ground targets.     

An Efficient Approach for Dynamic Calibration of Multiple Cameras

In this paper, we propose a new algorithm for dynamic calibration of multiple cameras. Based on the mapping between a horizontal plane in the 3-D space and the 2-D image plane on a panned and tilted camera, we utilize the displacement of feature points and the epipolar-plane constraint among multiple cameras to infer the changes of pan and tilt angles for each camera. This algorithm does not require a complicated correspondence of feature points. It can be applied to surveillance systems with wide-range coverage. It also allows the presence of moving objects in the captured scenes while performing dynamic calibration. The sensitivity analysis of our algorithm with respect to measurement errors and fluctuations in previous estimations is also discussed. The efficiency and feasibility of this approach has been demonstrated in some experiments over real scenery.     

Backpack: Detection of People Carrying Objects Using Silhouettes

We describe a video-rate surveillance algorithm for determining whether people are carrying objects or moving unencumbered from a stationary camera. The contribution of the paper is the shape analysis algorithm that both determines whether a person is carrying an object and segments the object from the person so that it can be tracked, e.g., during an exchange of objects between two people. As the object is segmented, an appearance model of the object is constructed. The method combines periodic motion estimation with static symmetry analysis of the silhouettes of a person in each frame of the sequence. Experimental results demonstrate robustness and real-time performance of the proposed algorithm.     

Detection algorithm for color dynamic images by multiple surveillance cameras under low luminance conditions based on fuzzy corresponding map

An objects detection algorithm for color dynamic images from two cameras is proposed for a surveillance system under low illumination. It provides automatic calculation of a fuzzy corresponding map and color similarity for lower luminance conditions, which detects little chromatic regions in CCD camera images under lower illumination and presents regions with a possibility of occlusion situation. Experimental detection results for two dynamic images from real surveillance cameras in a downtown area in Japan under low luminance conditions show that the proposed algorithm has 15% improved accuracy compared with the independent detection algorithm in the same false alarm rate, which occlusion regions are correctly presented. Moreover, implementability for severe surveillance situation is discussed. The proposed algorithm is being considered for use in a low cost surveillance system at a relatively poor security downtown (shopping mall) area in Japan.     

具有并行约束目标的多主体系统动态角色分配

多主体系统已成为建模和开发大型复杂分布式信息系统的一种理想范型.很多基于主体技术的系统要求支持动态角色分配,而已有动态角色分配算法忽略了目标之间的约束对角色分配的影响.首先,提出一个具有并行约束目标的多主体系统动态角色分配模型,引入多个角色分配管理者主体共同承担角色分配的计算任务,避免因单个主体可能造成的计算瓶颈.然后,基于并行约束目标结构图,给出目标集划分算法.并设计实现了角色分配算法,分析了该算法的时间复杂度.最后,实验研究了角色分配算法的执行时间,表明理论分析与实验结果一致.基于目标集合划分对角色分配的计算任务进行分割,使得各个角色分配管理者主体的计算结果无需进行合并再进行并行约束检查.     

Multi-object detection and tracking by stereo vision

This paper presents a new stereo vision-based model for multi-object detection and tracking in surveillance systems. Unlike most existing monocular camera-based systems, a stereo vision system is constructed in our model to overcome the problems of illumination variation, shadow interference, and object occlusion. In each frame, a sparse set of feature points are identified in the camera coordinate system, and then projected to the 2D ground plane. A kernel-based clustering algorithm is proposed to group the projected points according to their height values and locations on the plane. By producing clusters, the number, position, and orientation of objects in the surveillance scene can be determined for online multi-object detection and tracking. Experiments on both indoor and outdoor applications with complex scenes show the advantages of the proposed system.     

3D measures exploitation for a monocular semi-supervised fall detection system

Falls have been reported as the leading cause of injury-related visits to emergency departments and the primary etiology of accidental deaths in elderly. Thus, the development of robust home surveillance systems is of great importance. In this article, such a system is presented, which tries to address the fall detection problem through visual cues. The proposed methodology utilizes a fast, real-time background subtraction algorithm, based on motion information in the scene and pixels intensity, capable to operate properly in dynamically changing visual conditions, in order to detect the foreground object. At the same time, it exploits 3D space’s measures, through automatic camera calibration, to increase the robustness of fall detection algorithm which is based on semi-supervised learning approach. The above system uses a single monocular camera and is characterized by minimal computational cost and memory requirements that make it suitable for real-time large scale implementations.     

Automated and coupled services of advanced smart surveillance systems toward green IT: tracking,retrieval and digital evidence

Green Security is a new research field defining and investigating security solutions using an energy-aware perspective. Growing efforts and interests for an intelligent or smart surveillance system which is capable of automatically detecting and tracking target objects is in the spotlight in the security community. So far, these technologies are mainly aimed at single camera applications and are evolving with the demand for wide-area surveillance systems currently. However, the tracking techniques used on a single camera have limitations in providing effective crime prevention and countermeasures when an incident occurs since an object is not linked to other cameras. In addition, the use of multi-camera systems for wide-area surveillance not only produces large amounts of video data to be stored, but also have more technical requirements in the interrelation between cameras or server. It require a considerable amount of time, manpower and energy in multi-camera tracking and back-tracking of objects. Therefore, we propose the advanced smart surveillance system for wide-areas which is capable of the automated tracking and retrieval of target object and digital evidence-video collection. Furthermore, we considered the multiple-camera environment with non-overlapping views which includes more constraint conditions by various light changes. This system enables real-time object tracking, fast post-retrieval and selective digital evidence collection with economy of time, manpower, memory devices, and energy consumption. Also, this system is more energy-efficient since our schemes are organically connected to each other.     

An adaptive focus-of-attention model for video surveillance and monitoring

In current video surveillance systems, commercial pan/tilt/zoom (PTZ) cameras typically provide naive (or no) automatic scanning functionality to move a camera across its complete viewable field. However, the lack of scene-specific information inherently handicaps these scanning algorithms. We address this issue by automatically building an adaptive, focus-of-attention, scene-specific model using standard PTZ camera hardware. The adaptive model is constructed by first detecting local human activity (i.e., any translating object with a specific temporal signature) at discrete locations across a PTZ camera’s entire viewable field. The temporal signature of translating objects is extracted using motion history images (MHIs) and an original, efficient algorithm based on an iterative candidacy-classification-reduction process to separate the target motion from noise. The target motion at each location is then quantified and employed in the construction of a global activity map for the camera. We additionally present four new camera scanning algorithms which exploit this activity map to maximize a PTZ camera’s opportunity of observing human activity within the camera’s overall field of view. We expect that these efficient and effective algorithms are implementable within current commercial camera systems.     

基于视觉的增强现实运动跟踪算法

增强现实系统不仅具有虚拟现实的特点同时具有虚实结合的新特性，为实现虚拟物体与真实物体间的完善结合，必须实时地动态跟踪摄像与真实物体间的相对位置和方向，建立观测模,墼是而通过动态三维显示技术迅速地将虚拟物体添加到真实物体之上，然而目前大多数增强现实系统的注册对象均匀静物体，运动物体的注册跟踪尚很少有人涉足。该算法通过标志点的光流场估计真实环境中运动物体的运动参数，根据透视投影原理和刚体的运动特性确定摄像机与运动物体间的相对位置和方向，实现增强现实系统的运动目标跟踪注册。该算法构架简单、实时性强，易于实现，扩展了增强现实系统的应用范围。