Analysis and Hardware Architecture Design of Global Motion Estimation

Global motion estimation and compensation (GME/GMC) is an important video processing technique and has been applied to many applications including video segmentation, sprite/mosaic generation, and video coding. In MPEG-4 Advanced Simple Profile (ASP), GME/GMC is adopted to compensate camera motions. Since GME is important, many GME algorithms have been proposed. These algorithms have two common characteristics, huge computation complexity and ultra large memory bandwidth. Hence for realtime applications, a hardware accelerator of GME is required. However, there are many hardware design challenges of GME like irregular memory access and huge memory bandwidth, and only few hardware architectures have been proposed. In this paper, we first analyzed three typical algorithms of GME, and a fast GME algorithm is proposed. By using temporal prediction and skipping the redundant computation, 91% memory bandwidth and 80% iterations are saved, while the performance is kept, compared to Gradient Descent in MPEG-4 Verification Model. Based on our proposed algorithm, a hardware architecture of GME is also presented. A new scheduling, Reference-Based Scheduling, is developed to solve the irregular memory access problem. An interleaved memory arrangement is applied to satisfy the memory access requirement of interpolation. The total gate count of hardware implementation is 131 K with Artisan 0.18 um cell library, and the internal memory size is about 7.9 Kb. Its processing ability is MPEG-4 ASP@L3, which is 352×288 with 30 fps, at 30 MHz.

Fast perceptual region tracking with coding-depth sensitive access for stream transcoding

Object-based bit allocation can result in significant improvement in the perceptual quality of extremely compressed video. However, real-time video object detection in large format high fidelity video is computationally daunting. Most algorithms begin with extensive use of classical bit analysis, and thus remain computationally heavy. Based on some recent results in human visual perception, in this paper, we present an experimental visual region tracking algorithm particularly designed for perceptual stream transcoding. This exploits the cue order observed in human visual perception to achieve very high computation speed as well as tracking efficiency. Rather than begin processing from pixel level or using any pixel level processing at all, it employs high level motion cue and block shape cue analysis to identify signatures of various relative movements between object of interest, scene background and the camera on the motion vector set, and from there it identifies objects. It then uses predictive filters to track the regions. The result is a fast yet highly effective perceptual region tracking algorithm that can operate in stream rate and track regions of perceptually significant object despite camera movements such as zoom, panning and translation. The technique is not specific to any special class of objects. We have implemented this algorithm in a live ISO-13818/MPEG-2 perceptual transcoder. In this paper, we share the performance of this implementation. This fast object-aware video rate transcoder is particularly suitable for live streaming and can convert a regular stream into a perceptually coded video stream.     

背景渐变的视频对象分割算法研究及实现

提出了一种有效的背景渐变的视频对象分割算法.首先将前一帧分成前景和背景两部分,然后采用灰度投影匹配算法对当前帧进行全局运动估计和补偿,将当前帧与上一帧进行差分运算,便可得到差分图像.通过对差分图像进行二值化处理,得到运动模板并与前景信息进行相与计算,再结合当前帧信息便可得到运动目标.在TI公司的TMS320DM642芯片上验证了该算法,实验结果表明该算法不仅对亮度变化和环境变化具有鲁棒性,而且可独立、精确地分割出运动目标.     

运动目标的自动分割与跟踪

该文提出了一种对视频序列中的运动目标进行自动分割的算法。该算法分析图像在L U V空间中的局部变化,同时使用运动信息来把目标从背景中分离出来。首先根据图像的局部变化,使用基于图论的方法把图像分割成不同的区域。然后,通过度量合成的全局运动与估计的局部运动之间的偏差来检测出运动的区域,运动的区域通过基于区域的仿射运动模型来跟踪到下一帧。为了提高提取的目标的时空连续性,使用Hausdorff跟踪器对目标的二值模型进行跟踪。对一些典型的MPEG-4测试序列所进行的评估显示了该算法的优良性能。     

Moving object detection and tracking from video captured by moving camera

This paper presents an effective method for the detection and tracking of multiple moving objects from a video sequence captured by a moving camera without additional sensors. Moving object detection is relatively difficult for video captured by a moving camera, since camera motion and object motion are mixed. In the proposed method, the feature points in the frames are found and then classified as belonging to foreground or background features. Next, moving object regions are obtained using an integration scheme based on foreground feature points and foreground regions, which are obtained using an image difference scheme. Then, a compensation scheme based on the motion history of the continuous motion contours obtained from three consecutive frames is applied to increase the regions of moving objects. Moving objects are detected using a refinement scheme and a minimum bounding box. Finally, moving object tracking is achieved using a Kalman filter based on the center of gravity of a moving object region in the minimum bounding box. Experimental results show that the proposed method has good performance.     

Time efficient ghost removal for motion detection in visual surveillance systems

Segmentation of foreground objects is an important and essential task for many systems that aim to carry out motion tracking, object classification, event detection and is used in applications such as traffic monitoring and analysis, access control to special areas, human and vehicle identification and the detection of anomalous behaviour. The most common approach for detecting moving objects is background subtraction, in which each frame of a video sequence is compared against a background model. A large number of background subtraction algorithms have been proposed [1], but problems remain for moving object identification under certain conditions. One of the toughest problems in background subtraction is caused by the detection of false objects when an object that belongs to the background (e.g. after staying stationary for some time) starts to move away. This generates what are called `ghosts?. It is important to address the problem because ghost objects will adversely affect many tasks such as object classification, tracking and event analysis (e.g. abandoned item detection). This Letter focuses on the problem of ghost identification and elimination. We used a state-of-the-art industrial tracker which includes basic background subtraction and object tracking. Then we included our ghost detection algorithm into the basic tracker to identify and eliminate ghosts. Finally, we systematically evaluated and compared performance on urban traffic video sequences.     

基于优选特征轨迹的全分辨率视频稳定

该文提出一种基于优选特征轨迹的视频稳定算法。首先,采用改进的Harris角点检测算子提取特征点,通过K-Means聚类算法剔除前景特征点。然后,利用帧间特征点的空间运动一致性减少错误匹配和时间运动相似性实现长时间跟踪,从而获取有效特征轨迹。最后,建立同时包含特征轨迹平滑度与视频质量退化程度的目标函数计算视频序列的几何变换集以平滑特征轨迹获取稳定视频。针对图像扭曲产生的空白区,由当前帧定义区与参考帧的光流作引导来腐蚀,并通过图像拼接填充仍属于空白区的像素。经仿真验证,该文方法稳定的视频,空白区面积仅为Matsushita方法的33%左右,对动态复杂场景和多个大运动前景均具有较高的有效性并可生成内容完整的视频,既提高了视频的视觉效果,又减轻了费时的边界修复任务。     

基于运动矢量多级分析的视频全局运动估计

基于运动矢量场的视频全局运动估计相较于基于像素的估计方法具有较低的计算复杂度,因而广泛应用于视频分割及视频压缩等领域中。然而噪声和前景目标等外点区域的存在,降低了全局运动估计的准确性。为了提高全局运动估计的准确度,该文提出一种基于运动矢量多级分析的全局运动估计算法,该算法根据局部运动与全局运动的运动特性差异自适应地滤除前景目标区域,由邻域矢量间相似性度量检测出纹理平滑周期区域,最后滤除孤立的噪声区域,由滤波得到的内点区域求解全局运动参数。实验结果表明,该方法能有效地滤除外点区域,提高全局运动估计的准确性。     

Multi-view video based multiple objects segmentation using graph cut and spatiotemporal projections

In this paper, we present an automatic algorithm to segment multiple objects from multi-view video. The Initial Interested Objects (IIOs) are automatically extracted in the key view of the initial frame based on the saliency model. Multiple objects segmentation is decomposed into several sub-segmentation problems, and solved by minimizing the energy function using binary label graph cut. In the proposed novel energy function, the color and depth cues are integrated with the data term, which is then modified with background penalty with occlusion reasoning. In the smoothness term, foreground contrast enhancement is developed to strengthen the moving objects boundary, and at the same time attenuates the background contrast. To segment the multi-view video, the coarse predictions of the other views and the successive frame are projected by pixel-based disparity and motion compensation, respectively, which exploits the inherent spatiotemporal consistency. Uncertain band along the object boundary is shaped based on activity measure and refined with graph cut, resulting in a more accurate Interested Objects (IOs) layer across all views of the frames. The experiments are implemented on a couple of multi-view videos with real and complex scenes. Excellent subjective results have shown the robustness and efficiency of the proposed algorithm.     

Layered moving-object segmentation for stereoscopic video using motion and depth information

A novel layered stereoscopic moving-object segmentation method is proposed in this paper by exploiting both motion information and depth information to extract moving objects for each depth layer with high accuracy on their shape boundary. By taking a higher-order statistics on two frame-difference fields across three adjacent frames, the computed motion information are used to conduct change detection and generate one motion mask that consists of all the moving objects from all the depth layers involved at each view. It would be highly desirable, and challenging, to further differentiate them according to their residing depth layer to achieve layered segmentation. For that, multiple depth-layer masks are generated using our proposed disparity estimation method, one for each depth layer. By intersecting the motion mask and one depth-layer mask at any given layer-of-interest, the moving objects associated with the corresponding layer are then extracted. All the above-mentioned processes are repeatedly performed along the video sequence with a sliding window of three frames at a time. For demonstration, only the foreground and the background layers are considered in this paper, while the proposed method is generic and can be straightforwardly extended to more layers, once the corresponding depth-layer masks are made available. Experimental results have shown that the proposed layered moving-object segmentation method is able to segment the foreground and background moving objects separately, with high accuracy on their shape boundary. In addition, the required computational load is considered fairly inexpensive, since our design methodology is to generate masks and perform intersections for extracting the moving objects for each depth layer.     

Semi-automatic 2D-to-3D video conversion based on background sprite generation

This paper presents a technique for semi-automatic 2D-to-3D stereo video conversion, which is known to provide user intervention in assigning foreground/background depths for key frames and then get depth maps for non-key frames via automatic depth propagation. Our algorithm treats foreground and background separately. For foregrounds, kernel pixels are identified and then used as the seeds for graph-cut segmentation for each non-key frame independently, resulting in results not limited by objects’ motion activity. For backgrounds, all video frames, after foregrounds being removed, are integrated into a common background sprite model (BSM) based on a relay-frame-based image registration algorithm. Users can then draw background depths for BSM in an integrated manner, thus reducing human efforts significantly. Experimental results show that our method is capable of retaining more faithful foreground depth boundaries (by 1.6–2.7 dB) and smoother background depths than prior works. This advantage is helpful for 3D display and 3D perception.     

Adaptive coding of moving objects for very low bit rates

This paper describes an object-based video coding system with new ideas in both the motion analysis and source encoding procedures. The moving objects in a video are extracted by means of a joint motion estimation and segmentation algorithm based on the Markov random field (MRF) model. The two important features of the presented technique are the temporal linking of the objects, and the guidance of the motion segmentation with spatial color information. This facilitates several aspects of an object-based coder. First, a new temporal updating scheme greatly reduces the bit rate to code the object boundaries without resorting to crude lossy approximations. Next, the uncovered regions can be extracted and encoded in an efficient manner by observing their revealed contents. The objects are classified adaptively as P objects or I objects and encoded accordingly. Subband/wavelet coding is applied in encoding the object interiors. Simulations at very low bit rates yielded comparable performance in terms of reconstructed PSNR to the H.263 coder. The object-based coder produced visually more pleasing video with less blurriness and devoid of block artifacts, thus confirming the advantages of object-based coding at very low bit-rates     

网格运动分析与形态学滤波相结合的视频对象分割

提出了一种基于二维网格运动分析与改进形态学滤波空域自动分割策略相结合的视频对象时空分割算法。该算法首先利用高阶统计方法对视频图像的二维网格表示进行运动分析，快速得到前景对象区域，通过后处理有效获得前景对象运动检测掩膜。然后，用一种结合交变序列重建滤波算法和自适应阈值判别算法的改进分水岭分割策略有效获得前景对象的精确边缘。最后，用区域基时空融合算法将时域分割结果和空域分割结果结合起来提取出边缘精细的视频对象。实验结果表明，本算法综合了多种算法的优点，主客观分割效果理想。     

Semantic video fingerprinting and retrieval using face information

The management of large video databases, especially those containing motion picture and television data, is a major contemporary challenge. A very significant tool for this management is the ability to retrieve those segments that are perceptually similar to a query segment. Another similar but equally important task is determining if a query segment is a (possibly modified) copy of part of a video in the database. The basic way to perform these two tasks is to characterize each video segment with a unique representation called a signature. Using semantic information for the construction of the signatures is a good way to ensure robustness in retrieval and fingerprinting. Here a ubiquitous semantic feature, namely the existence and identity of human faces, will be used to construct the signature. A fast algorithm has been developed to quickly and robustly perform these two tasks on very large video databases. The prerequisite face recognition was performed by a commercial system. Having verified the basic efficacy of our algorithm on a database of real video from motion pictures and television series, we then proceed to further explore its performance in an artificial digital video database, which was created using a probabilistic model of the video creation process. This enabled us to explore variations in performance based on parameters that were impossible to control in a real video database. Furthermore, the suitability of the proposed approach for very large databases was tested using (artificial) data corresponding to hundreds or thousands of hours of video.     

基于GA的压缩域中全局运动估计及在字幕遮挡区域恢复中的应用

文章提出了一种直接使用压缩域中的运动矢量进行全局运动估计的方法,并用遗传算法优化输入运动矢量与全局运动参数所产生运动矢量的平均匹配误差.最终的实验结果表明本文的方法能够很好的估计出全局运动的参数.并且提出了一种运用全局/局部运动信息进行视频中字幕遮挡区域的恢复的方法.实验结果表明该方法取得了较好的视觉效果.     

基于运动目标的De-Interlace算法

本文提出一种新颖的基于运动目标的De-interlace算法.该算法以实际的运动目标作为操作对象,引入一种较精确的运动目标提取方法,并采用免疫克隆选择算法加速匹配目标的搜索过程.新算法融合了运动补偿、中值滤波、Weave、Bob等De-interlace方法.与流行的基于运动块补偿的De-interlace算法相比,新算法更适应复杂的视频序列,不仅可以处理平移运动,还适用于旋转、尺度变换等复杂运动情况.实验结果表明新算法的整体性能优于基于块匹配的方法.     

Video inpainting under constrained camera motion.

A framework for inpainting missing parts of a video sequence recorded with a moving or stationary camera is presented in this work. The region to be inpainted is general: it may be still or moving, in the background or in the foreground, it may occlude one object and be occluded by some other object. The algorithm consists of a simple preprocessing stage and two steps of video inpainting. In the preprocessing stage, we roughly segment each frame into foreground and background. We use this segmentation to build three image mosaics that help to produce time consistent results and also improve the performance of the algorithm by reducing the search space. In the first video inpainting step, we reconstruct moving objects in the foreground that are "occluded" by the region to be inpainted. To this end, we fill the gap as much as possible by copying information from the moving foreground in other frames, using a priority-based scheme. In the second step, we inpaint the remaining hole with the background. To accomplish this, we first align the frames and directly copy when possible. The remaining pixels are filled in by extending spatial texture synthesis techniques to the spatiotemporal domain. The proposed framework has several advantages over state-of-the-art algorithms that deal with similar types of data and constraints. It permits some camera motion, is simple to implement, fast, does not require statistical models of background nor foreground, works well in the presence of rich and cluttered backgrounds, and the results show that there is no visible blurring or motion artifacts. A number of real examples taken with a consumer hand-held camera are shown supporting these findings.     

Unsupervised video object segmentation with distractor-aware online adaptation

Unsupervised video object segmentation is a crucial application in video analysis when there is no prior information about the objects. It becomes tremendously challenging when multiple objects occur and interact in a video clip. In this paper, a novel unsupervised video object segmentation approach via distractor-aware online adaptation (DOA) is proposed. DOA models spatiotemporal consistency in video sequences by capturing background dependencies from adjacent frames. Instance proposals are generated by the instance segmentation network for each frame and they are grouped by motion information as positives or hard negatives. To adopt high-quality hard negatives, the block matching algorithm is then applied to preceding frames to track the associated hard negatives. General negatives are also introduced when there are no hard negatives in the sequence. The experimental results demonstrate these two kinds of negatives are complementary. Finally, we conduct DOA using positive, negative, and hard negative masks to update the foreground and background segmentation. The proposed approach achieves state-of-the-art results on two benchmark datasets, the DAVIS 2016 and the Freiburg-Berkeley motion segmentation (FBMS)-59.     

采用低秩与加权稀疏分解的视频前景检测算法

传统的鲁棒主成分分析模型能较好地解决视频前景检测问题.但是,若该模型的假设条件不能满足,算法性能会变差.针对此问题,本文提出了一种低秩与加权稀疏分解模型,通过对前景矩阵加权以增强其稀疏性.在建立加权矩阵的过程中,采用光流法获取每帧的运动矢量,以区分真实运动区域.其次,进一步提出一种增强模型,通过将加权矩阵作用于观测矩阵及背景矩阵,防止前景与背景的错误分离.实验结果表明,在无噪和有噪的情况下,提出的算法均能有效地分离监控视频中的前景和背景.     

基于运动矢量对消和差分原理的快速全局运动估计

该文针对全局运动估计计算复杂度过高的问题,提出了一种基于运动矢量对消和差分原理的快速全局运动估计方法。基于不同象限运动矢量对之间存在的对称抵消特性,首先估计出平移运动参数分量;再使用运动矢量对的差分原理,并结合一种置信判断的策略,估计出变换运动参数分量。在5组几何全局运动模型和真实视频序列上的实验结果表明,与基于迭代最小二乘(ILS)和基于偏导数(PD)的估计算法相比,该方法在保证估计精度的情况下,所需的平均计算时间只要它们的50％左右。