A robust framework for joint background/foreground segmentation of complex video scenes filmed with freely moving camera

This paper explores a robust region-based general framework for discriminating between background and foreground objects within a complex video sequence. The proposed framework works under difficult conditions such as dynamic background and nominally moving camera. The originality of this work lies essentially in our use of the semantic information provided by the regions while simultaneously identifying novel objects (foreground) and non-novel ones (background). The information of background regions is exploited to make moving objects detection more efficient, and vice-versa. In fact, an initial panoramic background is modeled using region-based mosaicing in order to be sufficiently robust to noise from lighting effects and shadowing by foreground objects. After the elimination of the camera movement using motion compensation, the resulting panoramic image should essentially contain the background and the ghost-like traces of the moving objects. Then, while comparing the panoramic image of the background with the individual frames, a simple median-based background subtraction permits a rough identification of foreground objects. Joint background-foreground validation, based on region segmentation, is then used for a further examination of individual foreground pixels intended to eliminate false positives and to localize shadow effects. Thus, we first obtain a foreground mask from a slow-adapting algorithm, and then validate foreground pixels (moving visual objects + shadows) by a simple moving object model built by using both background and foreground regions. The tests realized on various well-known challenging real videos (across a variety of domains) show clearly the robustness of the suggested solution. This solution, which is relatively computationally inexpensive, can be used under difficult conditions such as dynamic background, nominally moving camera and shadows. In addition to the visual evaluation, spatial-based evaluation statistics, given hand-labeled ground truth, has been used as a performance measure of moving visual objects detection.     

Camera cooperation for achieving visual attention

In this paper we address the problem of establishing a computational model for visual attention using cooperation between two cameras. More specifically we wish to maintain a visual event within the field of view of a rotating and zooming camera through the understanding and modeling of the geometric and kinematic coupling between a static camera and an active camera. The static camera has a wide field of view thus allowing panoramic surveillance at low resolution. High-resolution details may be captured by a second camera, provided that it looks in the right direction. We derive an algebraic formulation for the coupling between the two cameras and we specify the practical conditions yielding a unique solution. We describe a method for separating a foreground event (such as a moving object) from its background while the camera rotates. A set of outdoor experiments shows the two-camera system in operation.     

双摄像机协同人脸鹰眼检测与定位方法

现代视频监控系统需要获取大范围场景中感兴趣目标的清晰图像,这在目标距离较远并且不断移动时单纯采用摄像机调焦方式通常有一定的困难。为了获取宽范围监控场景中远距离行人的主要面部特征,采用广角静止—窄视场运动双摄像机协同工作方式可以同时获得远距离目标的全局和细节信息。首先采用改进的Codebook背景减法从广角摄像机中检测运动目标,然后指引运动摄像机近距离跟踪观察;若行人停止运动,则利用运动摄像机对其进行放大,然后从中检测人脸,并将人脸置于视野中心放大得到清晰图像。当行人再次运动时,广角相机将初始位置再次传递给运动摄像机,由其再对行人进行跟踪。通过实验室内和室外真实场景的实验表明,广角相机的检测算法具有一定的鲁棒性,运动相机能跟踪放大行人人脸图像,算法运行速度满足实时性要求。     

一种基于特征匹配的目标入侵检测方法

视频监控中常用云台摄像机监控视场较大的区域.对于云台摄像机跟随拍摄的情况,提出了一种基于特征匹配的目标入侵检测方法.通过提取的尺度不变特征变换(SIFT)特征点对,将当前图像和全景图像进行匹配,从而得到当前图像和全景图像投影关系,再将当前图像的坐标系变换到全景图像下,最后运用差分法,找到入侵目标.实验结果表明,即使当前图像与全景图像存在尺度、缩放、形变等差异,通过本方法也可正确地检测出入侵目标.     

An efficient continuous tracking system in real-time surveillance application

A tracking object must present a proper field of view (FOV) in a multiple active camera surveillance system; its clarity can facilitate smooth processing by the surveillance system before further processing, such as face recognition. However, when pan–tilt–zoom (PTZ) cameras are used, the tracking object can be brought into the FOV by adjusting its intrinsic parameters; consequently, selection of the best-performing camera is critical. Performance is determined by the relative positions of the camera and the tracking objects, image quality, lighting and how much of the front side of the object faces the camera. In a multi-camera surveillance system, both camera hand-off and camera assignment play an important role in automated and persistent tracking, which are typical surveillance requirements. This study investigates the use of automatic methods for tracking an object across cameras in a surveillance network using PTZ cameras. An automatic, efficient continuous tracking scheme is developed. The goal is to determine the decision criteria for hand-off using Sight Quality Indication (SQI) (which includes information on the position of the object and the proportion of the front of object faces the camera), and to perform the camera handoff task in a manner that optimizes the vision effect associated with monitoring. Experimental results reveal that the proposed algorithm can be efficiently executed, and the handoff method for feasible and continuously tracking active objects under real-time surveillance.     

PTZ自主跟踪中的全景视频生成

提出了一种在单PTZ摄像机自主跟踪过程中生成全景视频的方法。该方法在自主跟踪目标的同时,生成目标在大场景上运动的全景视频,可应用于PTZ摄像机监控场所。该方 法将全景视频看作全景背景图像和当前目标图像的叠加：首先利用Mean Shift跟踪方法逐帧获取目标区域图像并保存;然后利用相邻两帧视频图像的竖直方向投影匹配和Harris角 点匹配结果合成全景背景,与传统的配准方法相比,大大降低了匹配运算的复杂度,使全景背景的生成能够实时进行,并记录每帧图像到背景图像的变换参数;最后逐帧将目标区 域图像变换到背景图像上得到全景视频。本文方法与传统的全景视频生成方法相比,无需人工控制摄像机的转动,也无需手工对齐视频帧,整个过程全部自动完成。     

基于教室监控视频的学生位置检测和人脸图像捕获算法

实现一个基于课堂监控视频的学生位置检测和学生人脸图像获取系统。本系统由一个定焦全景摄像机和一个PTZ(平移(Pan)、倾斜(Tilt)、变焦(Zoom))摄像机组成。首先利用全景摄像机获得教室全景图像，针对实际课堂环境中的光线突变，提出基于帧间差的异常光线排除算法，实现异常光线监测和动态空教室图像检测与存储；使用HR网络结构对全景图像进行人脸检测，得到人脸检测框集合；针对非约束环境中学生因姿势变化和人脸遮挡、全景图像分辨率低等因素引起人脸信息缺失而导致人脸检测漏检问题，提出基于人体头肩特征的加权运动目标检测算法，得到目标检测框，提高人脸信息缺失的学生位置检测率；针对多种检测框的大量冗余，提出多种检测框加权融合算法，有效减少检测框的重复，得到学生人物检测框集合。然后，将学生人物检测框包含的位置信息传递至PTZ摄像机控制子系统，使PTZ逐个聚焦目标学生，捕获到清晰的学生人脸图像，为后续的人脸识别提供高质量的图像。     

Persistent people tracking and face capture using a PTZ camera

Pan–tilt–zoom (PTZ) camera is a powerful tool in far-field scenarios. However, most of the current PTZ surveillance systems require manual intervention to move the camera to the desired position. In this paper, we address the problem of persistent people tracking and face capture in uncontrolled scenarios using a single PTZ camera, which could prove most helpful in forensic applications. The system first detects and tracks pedestrians in zoomed-out mode. Then, according to a scheduler, the system selects a person to zoom in. In the zoomed-in mode, we detect a set of face images and solve the face–face association and face–person association problems. The system then zooms back out where tracking is continued as people re-appear in the view. The person–person association module associates the people on the schedule list with the people in the current view. The detected faces are associated with the corresponding people and trajectories. Due to the dynamic nature of our problem, e.g. the field of view of the camera changes because of the pan/tilt/zoom movement of the camera, all of the processes including receiving images from the camera and processing must be done in real time. To the best of our knowledge, the proposed method is the first to address the association of face images to people and trajectories using a single PTZ camera. Extensive experiments in challenging indoor and outdoor uncontrolled conditions demonstrate the effectiveness of the proposed system.     

Continuous localization and mapping of a pan–tilt–zoom camera for wide area tracking

Pan–tilt–zoom (PTZ) cameras are well suited for object identification and recognition in far-field scenes. However, the effective use of PTZ cameras is complicated by the fact that a continuous online camera calibration is needed and the absolute pan, tilt and zoom values provided by the camera actuators cannot be used because they are not synchronized with the video stream. So, accurate calibration must be directly extracted from the visual content of the frames. Moreover, the large and abrupt scale changes, the scene background changes due to the camera operation and the need of camera motion compensation make target tracking with these cameras extremely challenging. In this paper, we present a solution that provides continuous online calibration of PTZ cameras which is robust to rapid camera motion, changes of the environment due to varying illumination or moving objects. The approach also scales beyond thousands of scene landmarks extracted with the SURF keypoint detector. The method directly derives the relationship between the position of a target in the ground plane and the corresponding scale and position in the image and allows real-time tracking of multiple targets with high and stable degree of accuracy even at far distances and any zoom level.     

Dynamic 3D surface reconstruction and motion modeling from a pan–tilt–zoom camera

3D surface reconstruction and motion modeling has been integrated in several industrial applications. Using a pan–tilt–zoom (PTZ) camera, we present an efficient method called dynamic 3D reconstruction (D3DR) for recovering the 3D motion and structure of a freely moving target. The proposed method estimates the PTZ measurements to keep the target in the center of the field of view (FoV) of the camera with the same size. Feature extraction and tracking approach are used in the imaging framework to estimate the target's translation, position, and distance. A selection strategy is used to select keyframes that show significant changes in target movement and directly update the recovered 3D information. The proposed D3DR method is designed to work in a real-time environment, not requiring all frames captured to be used to update the recovered 3D motion and structure of the target. Using fewer frames minimizes the time and space complexity required. Experimental results conducted on real-time video streams using different targets to prove the efficiency of the proposed method. The proposed D3DR has been compared to existing offline and online 3D reconstruction methods, showing that it uses less execution time than the offline method and uses an average of 49.6% of the total number of frames captured.     

基于全景视觉的监控系统设计

设计了一个基于全景视觉的多摄像机监控网络。全景相机视野广,可以实现大范围的目标检测与跟踪。云台摄像机视角具有一定的自由度,可以捕捉目标的高分辨率图像。将全景相机与云台相机相互配合,通过多传感器的数据融合,分层次的跟踪算法及多相机调度算法,实现了大范围的多个运动目标的检测与跟踪,并能捕获目标的清晰图像。实验验证了该系统的有效性和合理性。     

由抖动图象序列建立自然环境三维真实感全景模型

由真实环境中的现场图象进行三维环境建模是目前国际上研究的热点问题。本文依据合理的运动模型，提出和实现了由包含抖动的摄像机运动下的图象序列建立３Ｄ环境全景模型的两步法。首先通过运动滤波和运动分解获得运动稳定的图象序列，然后采用无特征提取的时空纹理方向精确估计、深度边界确定和遮挡恢复算法，建立全局自然景物的真实感三维环境模型。提出了２种三维全景图象的表示方法，即非阵列方式深度分层区域表示和阵列方式的深度分层布景表示，可用于机器人全局定位的自然路标提取和真实环境虚拟再现的图象合成。该研究推广和结合了外极面图象的方法和全景图象的方法，放宽了对运动的要求，从而可使该种方法适用于室外颠簸的道路环境。和现有运动分层方法相比，避免了该类方法迭代过程中的局部最小化问题，并具有计算和存储效率高，适应性强，算法鲁棒性好的优点。     

Aligning windows of live video from an imprecise pan-tilt-zoom camera into a remote panoramic display for remote nature observation

A pan-tilt-zoom (PTZ) robotic camera can provide a detailed live video of selected areas of interest within a large potential viewing field. The selective coverage is ideal for nature observation applications where power and bandwidth are often limited. To provide the spatial context for human observers, it is desirable to insert the live video into a large spherical panoramic display representing the entire viewing field. Accurate alignment of the video stream within the panoramic display is difficult due to imprecise pan-tilt values and rapid changes in camera configurations. Common image alignment algorithms are computationally expensive for real time applications. We are interested in designing algorithms that fit low power computation platform and hence can be implemented inside the PTZ camera in the future. We present a sampling-based constant-time image alignment algorithm based on spherical projection and projection-invariant selective sampling that accurately registers paired images at 100 frames per second on a simulated embedded platform. The alignment accuracy actually is better than existing methods when high rotational difference is involved. Experiments suggest that the new alignment algorithm is faster than existing algorithms by 1,471.6 times when aligning a six-mega-pixel image pair.     

基于CHMM的背景差算法

为了解决背景差算法在前景提取的过程中对光照变化的敏感性和提取的前景中容易产生椒盐噪声的问题,提出了一种基于耦合隐马尔科夫模型的背景差方法.对像素的马尔科夫性进行了分析,并对像素建立耦合隐马尔科夫模型,通过时间统计的方法统计了像素隐含状态的转移概率,通过实验的方法选取了合适的前景标准差和背景标准差,利用Viterbi算法来求解耦合隐马尔科夫模型的最优隐含状态问题,运用该算法对一段交通监控视频进行分析,表明了该算法能够有效的抑制光照变化的影响,并且能够在一定程度上抑制前景噪声的出现.     

基于统计分析的视频目标检测

针对如何从包含大量冗余信息的视频中快速检测目标的问题,提出了一种基于统计分析的目标检测方法。该方法采用改进的直方图均衡化算法对图像做预处理;通过曼哈顿距离计算图像帧之间的差值,并对差值做进一步处理;采用迭代的方法,从图像帧差值中求取阈值,利用阈值判断前景帧和背景帧;在背景帧基础上建立背景模型,通过卡方值判断前景点和背景点;最后利用形态学还原物体真实形状,实现目标的准确检测。实验表明,该方法能快速准确地检测目标,可应用于视频监控的目标检测。     

一种大视差场景下的实时视频拼接系统

随着虚拟现实技术和视频直播技术的发展,全景视频直播受到广泛的关注。传统的全景产品在处理大视差场景时易发生图像畸变,且难以兼顾实时性要求。为了解决这些问题,设计一种两路视频拼接系统。首先建立拼接背景模型。然后结合ORB(Oriented FAST and Rotated BRIEF)和基于移动DLT(Direct Linear Transformation)的APAP（As-Projective-As-Possible）算法完成图像配准,并改进一种基于最小能量检测的方法寻找最优接缝以避免移动前景引起的鬼影和错位。最后通过拼接模型计算阶段所得的参数索引表,对重叠区域进行融合完成视频帧的实时拼接。实验结果表明该系统可以处理大视差场景,具有良好的实时拼接效果。     

Surveillance camera scheduling: a virtual vision approach

We present a surveillance system, comprising wide field-of-view (FOV) passive cameras and pan/tilt/zoom (PTZ) active cameras, which automatically captures high-resolution videos of pedestrians as they move through a designated area. A wide-FOV static camera can track multiple pedestrians, while any PTZ active camera can capture high-quality videos of one pedestrian at a time. We formulate the multi-camera control strategy as an online scheduling problem and propose a solution that combines the information gathered by the wide-FOV cameras with weighted round-robin scheduling to guide the available PTZ cameras, such that each pedestrian is observed by at least one PTZ camera while in the designated area. A centerpiece of our work is the development and testing of experimental surveillance systems within a visually and behaviorally realistic virtual environment simulator. The simulator is valuable as our research would be more or less infeasible in the real world given the impediments to deploying and experimenting with appropriately complex camera sensor networks in large public spaces. In particular, we demonstrate our surveillance system in a virtual train station environment populated by autonomous, lifelike virtual pedestrians, wherein easily reconfigurable virtual cameras generate synthetic video feeds. The video streams emulate those generated by real surveillance cameras monitoring richly populated public spaces.A preliminary version of this paper appeared as [1].     

摄像机运动情况下的运动对象检测

在监控应用中,由于场景是已知的,因此可以使用背景减去法检测运动对象.当摄像机进行扫描和倾斜运动时,需要使用多个图像帧才能完整地表示监控场景.如何组织和索引这些背景帧属于摄像机跟踪问题.提出一种无需摄像机标定的背景帧索引和访问方法.这一方法需要使用图像配准技术估计图像初始运动参数.提出一种屏蔽外点的图像配准算法,综合利用线性回归和稳健回归快速估计初始运动参数.为了快速计算连续帧之间的运动参数,提出一种基于四参数模型的优化算法.利用非参数背景维护模型抑制虚假运动象素.室内和户外实验结果表明本文方法是有效的.     

一种适应户外光照变化的背景建模及目标检测方法

针对户外视频监控存在光照变化这一问题, 提出一个用于准确完成目标检测的实时背景建模框架. 考虑到目标检测的准确性要求, 建立基于帧间像素亮度差统计直方图的像素亮度扰动阈值. 在此基础上, 针对背景建模的实时性要求, 提出一种基于自回归背景模型的参数快速更新方法. 鉴于不同光照变化的适应性要求, 定义对光照变化不敏感的背景纹理模型. 上述模型统称为自回归--纹理 (Auto regression and texture, ART) 模型, 该模型适应于户外光照变化. 基于该模型构建像素亮度和纹理置信区间用于目标检测. 实验结果表明, 该框架能适应和实时跟踪户外背景的光照变化, 并对目标进行准确检测.     

采用视频拼图方法构建高分辨率全景视频监控系统

与普通视频监控系统只能实现单向监控不同,全景视频监控系统可以实现360°全向监控。设计并实现了一种嵌入式高分辨率全景视频监控系统KD-PVS。重点介绍了KD-PVS中多个摄像头的空间位置设计、视频图像变换与拼接算法。KD-PVS通过对多个摄像头采集的视频进行实时变换与拼接以生成全景视频。该系统可方便应用于金融系统、仓库、监狱和移动监控等多种场合,尤其适用于室内监控。