复杂环境下基于信息融合的视频实时抠像算法

针对复杂场景下传统的视频抠像算法对目标物体与背景纹理相似或边界不清晰的图像分割困难的问题,提出了一种基于视觉传感器和激光雷达信息相融合的视频实时抠像算法。该算法从原始激光雷达点云数据中获取感兴趣区域深度信息,并作为先验知识融合到改进的谱抠图算法,创建感兴趣区域深度抠图拉普拉斯矩阵,通过聚类算法最优迭代得出抠像结果,并运用导向滤波器对抠像结果进行后处理。实验证明,对比于融合深度信息的传统算法和没有融合其他信息的算法,该算法降低了欠分割率、提高了运行效率,抠像目标的边缘信息也更加饱满、清晰、平滑。     

Thermo-key: human region segmentation from video

We focus on the segmentation of human regions, an area that has had a significant amount of research. Our approach, is based on invisible thermal information that we can measure without any estimation. In fact, we measure infrared light radiation of the thermal information for keys hence we call our proposed system thermo-key. A color and thermal combination camera measures the temperature distribution of its field of view. This system segments the human region from the video sequence - captured by the color camera in real time - with high robustness against lighting and background conditions.     

A Progressive Tri‐level Segmentation Approach for Topology‐Change‐Aware Video Matting

Previous video matting approaches mostly adopt the “binary segmentation + matting” strategy, i.e., first segment each frame into foreground and background regions, then extract the fine details of the foreground boundary using matting techniques. This framework has several limitations due to the fact that binary segmentation is employed. In this paper, we propose a new supervised video matting approach. Instead of applying binary segmentation, we explicitly model segmentation uncertainty in a novel tri‐level segmentation procedure. The segmentation is done progressively, enabling us to handle difficult cases such as large topology changes, which are challenging to previous approaches. The tri‐level segmentation results can be naturally fed into matting techniques to generate the final alpha mattes. Experimental results show that our system can generate high quality results with less user inputs than the state‐of‐theart methods.     

面向RGBD图像的标记分水岭分割

目的 针对分水岭分割算法中存在的过分割现象及现有基于RGB图像分割方法的局限,提出了一种基于RGB图像和深度图像(RGBD)的标记分水岭分割算法。方法 本文使用物体表面几何信息来辅助进行图像分割,定义了一种深度梯度算子和一种法向量梯度算子来衡量物体表面几何信息的变化。通过生成深度梯度图像和法向量梯度图像,与彩色梯度图像进行融合,实现标记图像的提取。在此基础上,使用极小值标定技术对彩色梯度图像进行修正,然后使用分水岭算法进行图像分割。结果 在纽约大学提供的NYU2数据集上进行实验,本文算法有效抑制了过分割现象,将分割区域从上千个降至数十个,且获得了与人工标定的分割结果更接近的分割效果,分割的准确率也比只使用彩色图像进行分割提高了10%以上。结论 本文算法普遍适用于RGBD图像的分割问题,该算法加入了物体表面几何信息的使用,提高了分割的准确率,且对颜色纹理相似的区域获得了较好的分割结果。     

Adaptive propagation matting based on transparency of image

Image matting is an essential technique in many image and video editing applications. Although many matting methods have been proposed, it is still a challenge for most to obtain satisfactory matting results in the transparent foreground region of an image. To solve this problem, this paper proposes a novel matting algorithm, i.e. adaptive transparency-based propagation matting (ATPM) algorithm. ATPM algorithm considers image matting from a new slant. We pay attention to the transparencies of the input images and creatively assign them into three categories (highly transparent, strongly transparent and little transparent) according to the transparencies of the foreground objects in the images. Our matting model can make relevant adjustment in terms of the transparency types of the input images. Moreover, many current matting methods do not perform well when the foreground and background regions have similar color distributions. Our method adds texture as an additional feature to effectively discriminate the foreground and background regions. Experimental results on the benchmark dataset show that our method gets high-quality matting results for images of three transparency types, especially provides more accurate results for highly transparent images comparing with the state-of-the-art methods.     

Combining color and depth for enhanced image segmentation and retargeting

As depth cameras become more popular, pixel depth information becomes easier to obtain. This information can clearly enhance many image processing applications. However, combining depth and color information is not straightforward as these two signals can have different noise characteristics, differences in resolution, and their boundaries do not generally agree. We present a technique that combines depth and color image information from real devices in synergy. In particular, we focus on combining them to improve image segmentation. We use color information to fill and clean depth and use depth to enhance color image segmentation. We demonstrate the utility of the combined segmentation for extracting layers and present a novel image retargeting algorithm for layered images.     

Kinect based real-time synthetic aperture imaging through occlusion

Real-time and high performance occluded object imaging is a big challenge to many computer vision applications. In recent years, camera array synthetic aperture theory proves to be a potential powerful way to solve this problem. However, due to the high cost of complex system hardware, the severe blur of occluded object imaging, and the slow speed of image processing, the exiting camera array synthetic aperture imaging algorithms and systems are difficult to apply in practice. In this paper, we present a novel handheld system to handle those challenges. The objective of this work is to design a convenient system for real-time high quality object imaging even under severe occlusion. The main characteristics of our work include: (1) To the best of our knowledge, this is the first real-time handheld system for seeing occluded object in synthetic imaging domain using color and depth images. (2) A novel sequential synthetic aperture imaging framework is designed to achieve seamless interaction among multiple novel modules, and this framework includes object probability generation, virtual camera array generation, and sequential synthetic aperture imaging. (3) In the virtual camera array generation module, based on the integration of color and depth information, a novel feature set iterative optimization algorithm is presented, which can improve the robustness and accuracy of camera pose estimation even in dynamic occlusion scene. Experimental results in challenging scenarios demonstrate the superiority of our system both in robustness and efficiency compared against the state-of-the-art algorithms.     

Scene-adaptive accurate and fast vertical crowd counting via joint using depth and color information

Reliable and real-time crowd counting is one of the most important tasks in intelligent visual surveillance systems. Most previous works only count passing people based on color information. Owing to the restrictions of color information influences themselves for multimedia processing, they will be affected inevitably by the unpredictable complex environments (e.g. illumination, occlusion, and shadow). To overcome this bottleneck, we propose a new algorithm by multimodal joint information processing for crowd counting. In our method, we use color and depth information together with a ordinary depth camera (e.g. Microsoft Kinect). Specifically, we first detect each head of the passing or still person in the surveillance region with adaptive modulation ability to varying scenes on depth information. Then, we track and count each detected head on color information. The characteristic advantage of our algorithm is that it is scene adaptive, which means the algorithm can be applied into all kinds of different scenes directly without additional conditions. Based on the proposed approach, we have built a practical system for robust and fast crowd counting facing complicated scenes. Extensive experimental results show the effectiveness of our proposed method.     

Bilayer segmentation of webcam videos using tree-based classifiers

This paper presents an automatic segmentation algorithm for video frames captured by a (monocular) webcam that closely approximates depth segmentation from a stereo camera. The frames are segmented into foreground and background layers that comprise a subject (participant) and other objects and individuals. The algorithm produces correct segmentations even in the presence of large background motion with a nearly stationary foreground. This research makes three key contributions: First, we introduce a novel motion representation, referred to as "motons," inspired by research in object recognition. Second, we propose estimating the segmentation likelihood from the spatial context of motion. The estimation is efficiently learned by random forests. Third, we introduce a general taxonomy of tree-based classifiers that facilitates both theoretical and experimental comparisons of several known classification algorithms and generates new ones. In our bilayer segmentation algorithm, diverse visual cues such as motion, motion context, color, contrast, and spatial priors are fused by means of a conditional random field (CRF) model. Segmentation is then achieved by binary min-cut. Experiments on many sequences of our videochat application demonstrate that our algorithm, which requires no initialization, is effective in a variety of scenes, and the segmentation results are comparable to those obtained by stereo systems.     

Integrating tracking with fine object segmentation

We present a novel method for on-line, joint object tracking and segmentation in a monocular video captured by a possibly moving camera. Our goal is to integrate tracking and fine segmentation of a single, previously unseen, potentially non-rigid object of unconstrained appearance, given its segmentation in the first frame of an image sequence as the only prior information. To this end, we tightly couple an existing kernel-based object tracking method with Random Walker-based image segmentation. Bayesian inference mediates between tracking and segmentation, enabling effective data fusion of pixel-wise spatial and color visual cues. The fine segmentation of an object at a certain frame provides tracking with reliable initialization for the next frame, closing the loop between the two building blocks of the proposed framework. The effectiveness of the proposed methodology is evaluated experimentally by comparing it to a large collection of state of the art tracking and video-based object segmentation methods on the basis of a data set consisting of several challenging image sequences for which ground truth data is available.     

Fully automatic person segmentation in unconstrained video using spatio-temporal conditional random fields

The segmentation of objects and people in particular is an important problem in computer vision. In this paper, we focus on automatically segmenting a person from challenging video sequences in which we place no constraint on camera viewpoint, camera motion or the movements of a person in the scene. Our approach uses the most confident predictions from a pose detector as a form of anchor or keyframe stick figure prediction which helps guide the segmentation of other more challenging frames in the video. Since even state of the art pose detectors are unreliable on many frames –especially given that we are interested in segmentations with no camera or motion constraints –only the poses or stick figure predictions for frames with the highest confidence in a localized temporal region anchor further processing. The stick figure predictions within confident keyframes are used to extract color, position and optical flow features. Multiple conditional random fields (CRFs) are used to process blocks of video in batches, using a two dimensional CRF for detailed keyframe segmentation as well as 3D CRFs for propagating segmentations to the entire sequence of frames belonging to batches. Location information derived from the pose is also used to refine the results. Importantly, no hand labeled training data is required by our method. We discuss the use of a continuity method that reuses learnt parameters between batches of frames and show how pose predictions can also be improved by our model. We provide an extensive evaluation of our approach, comparing it with a variety of alternative grab cut based methods and a prior state of the art method. We also release our evaluation data to the community to facilitate further experiments. We find that our approach yields state of the art qualitative and quantitative performance compared to prior work and more heuristic alternative approaches.     

自动分割视频序列中运动物体的新算法：多特征联合方法

视频编码标准MPEG-4增加了适于多种应用的基于视频内容的功能,为了支持这一功能和提高编码效率,MPEG-4将视频序列中的每一帧分解成视频对象面(VOP);另外,由于基于内容的视频检索和视频监控系统均期望用分割出的关键视频对象紧致地表示一个序列,同时由于视频分割技术在模式识别、计算机视觉等领域也得到了广泛的应用,因此,分割视频运动物体并跟踪运动物体的变化变得至关重要.为了对视频中运动物体进行有效的分割,在帧差图象的基础上,采用Canny边缘检测和随机信号的高阶矩检测相结合的方法,来自动分割视频序列的前景区域和背景区域,并在前景区域中应用区域生长法进行颜色分割,以精确提取运动物体的边缘;还利用边缘和颜色特征来对分割出的运动物体建立模板,用于解决非刚体运动中局部暂时停止运动的情况.实验结果表明,此方法可以有效地分割运动物体,并能跟踪运动物体的变化.     

Key observation selection-based effective video synopsis for camera network

Nowadays, tremendous amount of video is captured endlessly from increased numbers of video cameras distributed around the world. Since needless information is abundant in the raw videos, making video browsing and retrieval is inefficient and time consuming. Video synopsis is an effective way to browse and index such video, by producing a short video representation, while keeping the essential activities of the original video. However, video synopsis for single camera is limited in its view scope, while understanding and monitoring overall activity for large scenarios is valuable and demanding. To solve the above issues, we propose a novel video synopsis algorithm for partially overlapping camera network. Our main contributions reside in three aspects: First, our algorithm can generate video synopsis for large scenarios, which can facilitate understanding overall activities. Second, for generating overall activity, we adopt a novel unsupervised graph matching algorithm to associate trajectories across cameras. Third, a novel multiple kernel similarity is adopted in selecting key observations for eliminating content redundancy in video synopsis. We have demonstrated the effectiveness of our approach on real surveillance videos captured by our camera network.     

Human segmentation by geometrically fusing visible-light and thermal imageries

From depth sensors to thermal cameras, the increased availability of camera sensors beyond the visible spectrum has created many exciting applications. Most of these applications require combining information from these hyperspectral cameras with a regular RGB camera. Information fusion from multiple heterogeneous cameras can be a very complex problem. They can be fused at different levels from pixel to voxel or even semantic objects, with large variations in accuracy, communication, and computation costs. In this paper, we propose a system for robust segmentation of human figures in video sequences by fusing visible-light and thermal imageries. Our system focuses on the geometric transformation between visual blobs corresponding to human figures observed at both cameras. This approach provides the most reliable fusion at the expense of high computation and communication costs. To reduce the computational complexity of the geometric fusion, an efficient calibration procedure is first applied to rectify the two camera views without the complex procedure of estimating the intrinsic parameters of the cameras. To geometrically register different blobs at the pixel level, a blob-to-blob homography in the rectified domain is then computed in real-time by estimating the disparity for each blob-pair. Precise segmentation is finally achieved using a two-tier tracking algorithm and a unified background model. Our experimental results show that our proposed system provides significant improvements over existing schemes under various conditions.     

An effective algorithm to detect both smoke and flame using color and wavelet analysis

Fire detection is an important task in many applications. Smoke and flame are two essential symbols of fire in images. In this paper, we propose an algorithm to detect smoke and flame simultaneously for color dynamic video sequences obtained from a stationary camera in open space. Motion is a common feature of smoke and flame and usually has been used at the beginning for extraction from a current frame of candidate areas. The adaptive background subtraction has been utilized at a stage of moving detection. In addition, the optical flow-based movement estimation has been applied to identify a chaotic motion. With the spatial and temporal wavelet analysis, Weber contrast analysis and color segmentation, we achieved moving blobs classification. Real video surveillance sequences from publicly available datasets have been used for smoke detection with the utilization of our algorithm. We also have conducted a set of experiments. Experiments results have shown that our algorithm can achieve higher detection rate of 87% for smoke and 92% for flame.     

一种基于TOF相机与CCD相机的联合标定算法研究

针对基于Time-of-Flight(TOF)相机的彩色目标三维重建需标定CCD相机与TOF相机联合系统的几何参数,在研究现有的基于彩色图像和TOF深度图像标定算法的基础上,提出了一种基于平面棋盘模板的标定方法。拍摄了固定在平面标定模板上的彩色棋盘图案在不同角度下的彩色图像和振幅图像,改进了Harris角点提取,根据棋盘格上角点与虚拟像点的共轭关系,建立了相机标定系统模型,利用Levenberg-Marquardt算法求解,进行了标定实验。获取了TOF与CCD相机内参数,并利用像平面之间的位姿关系估计两相机坐标系的相对姿态,最后进行联合优化,获取了相机之间的旋转矩阵与平移向量。实验结果表明,提出的算法优化了求解过程,提高了标定效率,能够获得较高的精度。     

Video repairing under variable illumination using cyclic motions

This paper presents a complete system capable of synthesizing a large number of pixels that are missing due to occlusion or damage in an uncalibrated input video. These missing pixels may correspond to the static background or cyclic motions of the captured scene. Our system employs user-assisted video layer segmentation, while the main processing in video repair is fully automatic. The input video is first decomposed into the color and illumination videos. The necessary temporal consistency is maintained by tensor voting in the spatio-temporal domain. Missing colors and illumination of the background are synthesized by applying image repairing. Finally, the occluded motions are inferred by spatio-temporal alignment of collected samples at multiple scales. We experimented on our system with some difficult examples with variable illumination, where the capturing camera can be stationary or in motion.     

Fast video segmentation algorithm with shadow cancellation, global motion compensation, and adaptive threshold techniques

Automatic video segmentation plays an important role in real-time MPEG-4 encoding systems. Several video segmentation algorithms have been proposed; however, most of them are not suitable for real-time applications because of high computation load and many parameters needed to be set in advance. This paper presents a fast video segmentation algorithm for MPEG-4 camera systems. With change detection and background registration techniques, this algorithm can give satisfying segmentation results with low computation load. The processing speed of 40 QCIF frames per second can be achieved on a personal computer with an 800 MHz Pentium-III processor. Besides, it has shadow cancellation mode, which can deal with light changing effect and shadow effect. A fast global motion compensation algorithm is also included in this algorithm to make it applicable in slight moving camera situations. Furthermore, the required parameters can be decided automatically, which can enhance the proposed algorithm to have adaptive threshold ability. It can be integrated into MPEG-4 videophone systems and digital cameras.     

一种基于视频结构的场景分割方法

场景是视频结构中的语义单元。因此基于场景的视频分割,将会对视频的内容分析、检索和浏览提供有益的帮助。提出了一种新的场景分割算法,它利用流形学习理论获得视频的结构特征,然后用马尔科夫链蒙特卡罗方法(Markov chain Monte Carlo,MCMC)动态地进行模拟采样,寻找场景边界的最大后验概率分布,完成场景的自动分割。通过发掘视频结构的内在特征并考虑它的局部信息,使算法能够适合不同类型的视频数据。实验结果也证明了该方法的有效性。     

Fuzzy segmentation of video shots using hybrid color spaces and motion information

Video segmentation can be defined as the process of partitioning video into spatio-temporal objects that are homogeneous in some feature space, with the choice of features being very important to the success of the segmentation process. Fuzzy segmentation is a semi-automatic region-growing segmentation algorithm that assigns to each element in an image a grade of membership in an object. In this paper, we propose an extension of the multi-object fuzzy segmentation algorithm to segment pre-acquired color video shots. The color features are selected from channels belonging to different color models using two different heuristics: one that uses the correlations between the color channels to maximize the amount of information used in the segmentation process, and one that chooses the color channels based on the separation of the clusters formed by the seed spels for all possible color spaces. Motion information is also incorporated into the segmentation process by making use of dense optical flow maps. We performed experiments on synthetic videos, with and without noise, as well as on some real videos. The experiments show promising results, with the segmentations of real videos produced using hybrid color spaces being more accurate than the ones produced using three other color models. We also show that our method compares favorably to a state-of-the art video segmentation algorithm.