Time-Delayed Correlation Analysis for Multi-Camera Activity Understanding

We propose a novel approach to understanding activities from their partial observations monitored through multiple non-overlapping cameras separated by unknown time gaps. In our approach, each camera view is first decomposed automatically into regions based on the correlation of object dynamics across different spatial locations in all camera views. A new Cross Canonical Correlation Analysis (xCCA) is then formulated to discover and quantify the time delayed correlations of regional activities observed within and across multiple camera views in a single common reference space. We show that learning the time delayed activity correlations offers important contextual information for (i) spatial and temporal topology inference of a camera network; (ii) robust person re-identification and (iii) global activity interpretation and video temporal segmentation. Crucially, in contrast to conventional methods, our approach does not rely on either intra-camera or inter-camera object tracking; it thus can be applied to low-quality surveillance videos featured with severe inter-object occlusions. The effectiveness and robustness of our approach are demonstrated through experiments on 330 hours of videos captured from 17 cameras installed at two busy underground stations with complex and diverse scenes.     

视觉监控应用中多传感器协作的人脸检测系统

提出了一种新颖的由两个可控摄像机组成的多传感器视觉监控系统,旨在实现户外环境下的实时跟踪与特征化运动目标.特别地,该系统利用一个在多个缩放级别上可操作的移动摄像机在连续视频帧中自动获取与跟踪人脸.配合它的是一架能执行自动目标跟踪与分类的固定广域摄像机.     

Automatic object extraction and reconstruction in active video

A new method of video object extraction is proposed to automatically extract the object of interest from actively acquired videos. Traditional video object extraction techniques often operate under the assumption of homogeneous object motion and extract various parts of the video that are motion consistent as objects. In contrast, the proposed active video object extraction (AVOE) approach assumes that the object of interest is being actively tracked by a non-calibrated camera under general motion and classifies the possible movements of the camera that result in the 2D motion patterns as recovered from the image sequence. Consequently, the AVOE method is able to extract the single object of interest from the active video. We formalize the AVOE process using notions from Gestalt psychology. We define a new Gestalt factor called “shift and hold” and present 2D object extraction algorithms. Moreover, since an active video sequence naturally contains multiple views of the object of interest, we demonstrate that these views can be combined to form a single 3D object regardless of whether the object is static or moving in the video.     

Surveillance camera scheduling: a virtual vision approach

We present a surveillance system, comprising wide field-of-view (FOV) passive cameras and pan/tilt/zoom (PTZ) active cameras, which automatically captures high-resolution videos of pedestrians as they move through a designated area. A wide-FOV static camera can track multiple pedestrians, while any PTZ active camera can capture high-quality videos of one pedestrian at a time. We formulate the multi-camera control strategy as an online scheduling problem and propose a solution that combines the information gathered by the wide-FOV cameras with weighted round-robin scheduling to guide the available PTZ cameras, such that each pedestrian is observed by at least one PTZ camera while in the designated area. A centerpiece of our work is the development and testing of experimental surveillance systems within a visually and behaviorally realistic virtual environment simulator. The simulator is valuable as our research would be more or less infeasible in the real world given the impediments to deploying and experimenting with appropriately complex camera sensor networks in large public spaces. In particular, we demonstrate our surveillance system in a virtual train station environment populated by autonomous, lifelike virtual pedestrians, wherein easily reconfigurable virtual cameras generate synthetic video feeds. The video streams emulate those generated by real surveillance cameras monitoring richly populated public spaces.A preliminary version of this paper appeared as [1].     

Automated camera sabotage detection for enhancing video surveillance systems

Surveillance cameras are vital source of information in crime investigations. A surveillance video must be recorded with correct field of view and be of good quality, otherwise, it may not be suitable for investigation or analysis purposes. Perpetrators may tamper the recorded video or the physical device itself, in order to conceal their dubious activities. Generally, surveillance systems are unmanned due to limitations of manual monitoring. Automatic detection of camera tamper events is crucial for timely operator intervention. We propose a new method for detecting video camera tampering events like occlusion, defocus and displacement. The features used are edge information, frame count, foreground objects’ coverage area and its static nature. Effectiveness of our method is tested through experimentation on public datasets. The results obtained are encouraging with high detection and low false alarm rates. The proposed method automatically detects routine problems with cameras like dirt on camera lens, fog and smoke.

     

Automated and coupled services of advanced smart surveillance systems toward green IT: tracking,retrieval and digital evidence

Green Security is a new research field defining and investigating security solutions using an energy-aware perspective. Growing efforts and interests for an intelligent or smart surveillance system which is capable of automatically detecting and tracking target objects is in the spotlight in the security community. So far, these technologies are mainly aimed at single camera applications and are evolving with the demand for wide-area surveillance systems currently. However, the tracking techniques used on a single camera have limitations in providing effective crime prevention and countermeasures when an incident occurs since an object is not linked to other cameras. In addition, the use of multi-camera systems for wide-area surveillance not only produces large amounts of video data to be stored, but also have more technical requirements in the interrelation between cameras or server. It require a considerable amount of time, manpower and energy in multi-camera tracking and back-tracking of objects. Therefore, we propose the advanced smart surveillance system for wide-areas which is capable of the automated tracking and retrieval of target object and digital evidence-video collection. Furthermore, we considered the multiple-camera environment with non-overlapping views which includes more constraint conditions by various light changes. This system enables real-time object tracking, fast post-retrieval and selective digital evidence collection with economy of time, manpower, memory devices, and energy consumption. Also, this system is more energy-efficient since our schemes are organically connected to each other.     

User-centric privacy awareness in video surveillance

Especially in urban environments, video cameras have become omnipresent. Supporters of video surveillance argue that it is an excellent tool for many applications including crime prevention and law enforcement. While this is certainly true, it must be questioned if sufficient efforts are made to protect the privacy of monitored people. Privacy concerns are often set aside when compared to public safety and security. One reaction to this situation is emerging: community-based efforts where citizens register and map surveillance cameras in their environment. Our study is inspired by this idea and proposes a user-specific and location-aware privacy awareness system. Using conventional smartphones, users not only can contribute to the camera maps, but also use community-collected data to be alerted of potential privacy violations. In our model, we define different levels of privacy awareness. For the highest level, we present a mechanism that allows users to directly interact with specially designed, trustworthy cameras. These cameras provide direct feedback about the tasks that are executed by the camera and how privacy-sensitive data is handled. A hardware security chip that is integrated into the camera is used to ensure authenticity, integrity and freshness of the provided camera status information.     

A general framework for managing and processing live video data with privacy protection

Though a large body of existing work on video surveillance focuses on image and video processing techniques, few address the usability of such systems, and in particular privacy issues. This study fuses concepts from stream processing and content-based image retrieval to construct a privacy-preserving framework for rapid development and deployment of video surveillance applications. Privacy policies, instantiated to as privacy filters, may be applied both granularly and hierarchically. Privacy filters are granular as they are applicable to specific objects appearing in the video streams. They are hierarchal because they can be specified at specific objects in the framework (e.g., users, cameras) and are combined such that the disseminated video stream adheres to the most stringent aspect specified in the cascade of all privacy filters relevant to a video stream or query. To support this privacy framework, we extend our Live Video Database Model with an informatics-based approach to object recognition and tracking and add an intrinsic privacy model that provides a level of privacy protection not previously available for real-time streaming video data. The proposed framework also provides a formal approach to implement and enforce privacy policies that are verifiable, an important step towards privacy certification of video surveillance systems through a standardized privacy specification language.     

An adaptive focus-of-attention model for video surveillance and monitoring

In current video surveillance systems, commercial pan/tilt/zoom (PTZ) cameras typically provide naive (or no) automatic scanning functionality to move a camera across its complete viewable field. However, the lack of scene-specific information inherently handicaps these scanning algorithms. We address this issue by automatically building an adaptive, focus-of-attention, scene-specific model using standard PTZ camera hardware. The adaptive model is constructed by first detecting local human activity (i.e., any translating object with a specific temporal signature) at discrete locations across a PTZ camera’s entire viewable field. The temporal signature of translating objects is extracted using motion history images (MHIs) and an original, efficient algorithm based on an iterative candidacy-classification-reduction process to separate the target motion from noise. The target motion at each location is then quantified and employed in the construction of a global activity map for the camera. We additionally present four new camera scanning algorithms which exploit this activity map to maximize a PTZ camera’s opportunity of observing human activity within the camera’s overall field of view. We expect that these efficient and effective algorithms are implementable within current commercial camera systems.     

Video super-resolution using controlled subpixel detector shifts

Video cameras must produce images at a reasonable frame-rate and with a reasonable depth of field. These requirements impose fundamental physical limits on the spatial resolution of the image detector. As a result, current cameras produce videos with a very low resolution. The resolution of videos can be computationally enhanced by moving the camera and applying super-resolution reconstruction algorithms. However, a moving camera introduces motion blur, which limits super-resolution quality. We analyze this effect and derive a theoretical result showing that motion blur has a substantial degrading effect on the performance of super-resolution. The conclusion is that, in order to achieve the highest resolution motion blur should be avoided. Motion blur can be minimized by sampling the space-time volume of the video in a specific manner. We have developed a novel camera, called the "jitter camera," that achieves this sampling. By applying an adaptive super-resolution algorithm to the video produced by the jitter camera, we show that resolution can be notably enhanced for stationary or slowly moving objects, while it is improved slightly or left unchanged for objects with fast and complex motions. The end result is a video that has a significantly higher resolution than the captured one.     

Distributed interactive video arrays for event capture and enhanced situational awareness

Video surveillance activity has dramatically increased over the past few years. Earlier work dealt mostly with single stationary cameras, but the recent trend is toward active multicamera systems. Such systems offer several advantages over single camera systems - multiple overlapping views for obtaining 3D information and handling occlusions, multiple nonoverlapping cameras for covering wide areas, and active pan-tilt-zoom (PTZ) cameras for observing object details. To address these issues, we have developed a multicamera video surveillance approach, called distributed interactive video array. The DIVA framework provides multiple levels of semantically meaningful information ("situational" awareness) to match the needs of multiple remote observers. We have designed DIVA-based systems that can track and identify vehicles and people, monitor perimeters and bridges, and analyze activities. A new video surveillance approach employing a large-scale cluster of video sensors demonstrates the promise of multicamera arrays for homeland security.     

Enabling video privacy through computer vision

Closed-circuit television cameras used today for surveillance sometimes enable privacy intrusion. The authors' privacy console manages operator access to different versions of video-derived data according to access-control lists. Additionally, their PrivacyCam is a smart camera that produces a video stream with privacy-intrusive information already removed.     

Structural similarity-based object tracking in multimodality surveillance videos

This paper addresses the problem of object tracking in video sequences for surveillance applications by using a recently proposed structural similarity-based image distance measure. Multimodality surveillance videos pose specific challenges to tracking algorithms, due to, for example, low or variable light conditions and the presence of spurious or camouflaged objects. These factors often cause undesired luminance and contrast variations in videos produced by infrared sensors (due to varying thermal conditions) and visible sensors (e.g., the object entering shadowy areas). Commonly used colour and edge histogram-based trackers often fail in such conditions. In contrast, the structural similarity measure reflects the distance between two video frames by jointly comparing their luminance, contrast and spatial characteristics and is sensitive to relative rather than absolute changes in the video frame. In this work, we show that the performance of a particle filter tracker is improved significantly when the structural similarity-based distance is applied instead of the conventional Bhattacharyya histogram-based distance. Extensive evaluation of the proposed algorithm is presented together with comparisons with colour, edge and mean-shift trackers using real-world surveillance video sequences from multimodal (infrared and visible) cameras.     

运动目标三维轨迹可视化与关联分析方法

随着治安监控系统的普及,越来越多的监控摄像头被安装在各个交通道路和公共场所中,每天都产生大量的监控视频.如今,监控视频分析工作主要是采用人工观看的方式来排查异常,以这种方式来分析视频内容耗费大量的人力和时间.目前,关于视频分析方面的研究大多是针对目标个体的异常行为检测和追踪,缺乏针对对象之间的关联关系的分析,对视频中的一些对象和场景之间的关联关系等还没有较为有效的表示和分析方法.针对这一现状,提出一种基于运动目标三维轨迹的关联视频可视分析方法来辅助人工分析视频,首先对视频资料进行预处理,获取各个目标对象的运动轨迹信息,由于二维轨迹难以处理轨迹的自相交、循环运动和停留等现象,并且没有时间信息就难以对同一空间内多个对象轨迹进行的关联性分析,于是结合时间维度对轨迹进行三维化扩展.该方法支持草图交互方式来操作,在分析过程中进行添加草图注释来辅助分析.可结合场景和对象的时空关系对轨迹进行关联性计算,得出对象及场景之间的关联模型,通过对对象在各个场景出现状况的统计,结合人工预先设定的规则,可实现对异常行为报警,辅助用户决策.     

An efficient continuous tracking system in real-time surveillance application

A tracking object must present a proper field of view (FOV) in a multiple active camera surveillance system; its clarity can facilitate smooth processing by the surveillance system before further processing, such as face recognition. However, when pan–tilt–zoom (PTZ) cameras are used, the tracking object can be brought into the FOV by adjusting its intrinsic parameters; consequently, selection of the best-performing camera is critical. Performance is determined by the relative positions of the camera and the tracking objects, image quality, lighting and how much of the front side of the object faces the camera. In a multi-camera surveillance system, both camera hand-off and camera assignment play an important role in automated and persistent tracking, which are typical surveillance requirements. This study investigates the use of automatic methods for tracking an object across cameras in a surveillance network using PTZ cameras. An automatic, efficient continuous tracking scheme is developed. The goal is to determine the decision criteria for hand-off using Sight Quality Indication (SQI) (which includes information on the position of the object and the proportion of the front of object faces the camera), and to perform the camera handoff task in a manner that optimizes the vision effect associated with monitoring. Experimental results reveal that the proposed algorithm can be efficiently executed, and the handoff method for feasible and continuously tracking active objects under real-time surveillance.     

3-D recovery of human motion by mobile stereo cameras

A method is described which recovers the 3-D shape of deformable objects, particularly human motions, from mobile stereo images. In the proposed technique, camera calibration is not required when taking images. Existing optical 3-D modeling systems must employ calibrated cameras that are set at fixed positions. This inevitably puts constraints on the range of the movement of an object. In the proposed method, multiple mobile cameras take images of a deformable object moving freely, and its 3-D model is reconstructed from the video image streams obtained. The advantages of the proposed method include the fact that the cameras employed are calibration-free, and that the image-taking cameras can move freely. The theory is described, and the performance is shown by an experiment on 3-D human motion modeling in an outdoor environment. The accuracy of the 3-D model obtained is evaluated and a discussion is given. This work was presented in part at the 10th International Symposium on Artificial Life and Robotics, Oita, Japan, February 4–6, 2005     

Anonymous subject identification and privacy information management in video surveillance

The widespread deployment of surveillance cameras has raised serious privacy concerns, and many privacy-enhancing schemes have been recently proposed to automatically redact images of selected individuals in the surveillance video for protection. Of equal importance are the privacy and efficiency of techniques to first, identify those individuals for privacy protection and second, provide access to original surveillance video contents for security analysis. In this paper, we propose an anonymous subject identification and privacy data management system to be used in privacy-aware video surveillance. The anonymous subject identification system uses iris patterns to identify individuals for privacy protection. Anonymity of the iris-matching process is guaranteed through the use of a garbled-circuit (GC)-based iris matching protocol. A novel GC complexity reduction scheme is proposed by simplifying the iris masking process in the protocol. A user-centric privacy information management system is also proposed that allows subjects to anonymously access their privacy information via their iris patterns. The system is composed of two encrypted-domain protocols: The privacy information encryption protocol encrypts the original video records using the iris pattern acquired during the subject identification phase; the privacy information retrieval protocol allows the video records to be anonymously retrieved through a GC-based iris pattern matching process. Experimental results on a public iris biometric database demonstrate the validity of our framework.     

移动远程视频监控系统

为了扩大单个摄像头的视频监控范围及灵活性, 设计了一种可远程操控的移动视频监控系统. 该系统由四个模块组成, 基于Arduino系统的智能车搭载有摄像头, 接收用户指令, 用于移动视频的采集; 嵌入式Linux系统通过V4L2接口实现对视频数据的实时采集, 一方面将数据通过网络发送至转发服务器, 另一方面将来自用户的控制指令转发至智能车; 服务器则用于转发视频至客户端以及转发用户控制指令至Linux系统; 基于Android的移动端呈现监控视频并提供用户控制界面. 与现有系统相比, 该系统可使用单摄像头实现无死角监控.     

Bayesian-competitive consistent labeling for people surveillance

This paper presents a novel and robust approach to consistent labeling for people surveillance in multi-camera systems. A general framework scalable to any number of cameras with overlapped views is devised. An off-line training process automatically computes ground-plane homography and recovers epipolar geometry. When a new object is detected in any one camera, hypotheses for potential matching objects in the other cameras are established. Each of the hypotheses is evaluated using a prior and likelihood value. The prior accounts for the positions of the potential matching objects, while the likelihood is computed by warping the vertical axis of the new object on the field of view of the other cameras and measuring the amount of match. In the likelihood, two contributions (forward and backward) are considered so as to correctly handle the case of groups of people merged into single objects. Eventually, a maximum-a-posteriori approach estimates the best label assignment for the new object. Comparisons with other methods based on homography and extensive outdoor experiments demonstrate that the proposed approach is accurate and robust in coping with segmentation errors and in disambiguating groups.     

Outdoor Human Motion Capture by Simultaneous Optimization of Pose and Camera Parameters

We present a method for capturing the skeletal motions of humans using a sparse set of potentially moving cameras in an uncontrolled environment. Our approach is able to track multiple people even in front of cluttered and non‐static backgrounds, and unsynchronized cameras with varying image quality and frame rate. We completely rely on optical information and do not make use of additional sensor information (e.g. depth images or inertial sensors). Our algorithm simultaneously reconstructs the skeletal pose parameters of multiple performers and the motion of each camera. This is facilitated by a new energy functional that captures the alignment of the model and the camera positions with the input videos in an analytic way. The approach can be adopted in many practical applications to replace the complex and expensive motion capture studios with few consumer‐grade cameras even in uncontrolled outdoor scenes. We demonstrate this based on challenging multi‐view video sequences that are captured with unsynchronized and moving (e.g. mobile‐phone or GoPro) cameras.