Figure-ground segmentation from occlusion.

Layered video representations are increasingly popular; see [2] for a recent review. Segmentation of moving objects is a key step for automating such representations. Current motion segmentation methods either fail to segment moving objects in low-textured regions or are computationally very expensive. This paper presents a computationally simple algorithm that segments moving objects, even in low-texture/low-contrast scenes. Our method infers the moving object templates directly from the image intensity values, rather than computing the motion field as an intermediate step. Our model takes into account the rigidity of the moving object and the occlusion of the background by the moving object. We formulate the segmentation problem as the minimization of a penalized likelihood cost function and present an algorithm to estimate all the unknown parameters: the motions, the template of the moving object, and the intensity levels of the object and of the background pixels. The cost function combines a maximum likelihood estimation term with a term that penalizes large templates. The minimization algorithm performs two alternate steps for which we derive closed-form solutions. Relaxation improves the convergence even when low texture makes it very challenging to segment the moving object from the background. Experiments demonstrate the good performance of our method.     

Space-time segmentation using level set active contours applied to myocardial gated SPECT

This paper presents a new variational method for the segmentation of a moving object against a still background, over a sequence of [two-dimensional or three-dimensional (3-D)] image frames. The method is illustrated in application to myocardial gated single photon emission computed tomography (SPECT) data, and incorporates a level set framework to handle topological changes while providing closed boundaries. The key innovation is the introduction of a geometrical constraint into the derivation of the Euler-Lagrange equations, such that the segmentation of each individual frame can be interpreted as a closed boundary of an object (an isolevel of a set of hyper-surfaces) while integrating information over the entire sequence. This results in the definition of an evolution velocity normal to the object boundary. Applying this method to 3-D myocardial gated SPECT sequences, the left ventricle endocardial and epicardial limits can be computed in each frame. This space-time segmentation method was tested on simulated and clinical 3-D myocardial gated SPECT sequences and the corresponding ejection fractions were computed.     

Two-dimensional mesh-based mosaic representation for manipulationof video objects with occlusion

We present a two-dimensional (2-D) mesh-based mosaic representation, consisting of an object mesh and a mosaic mesh for each frame and a final mosaic image, for video objects with mildly deformable motion in the presence of self and/or object-to-object (external) occlusion. Unlike classical mosaic representations where successive frames are registered using global motion models, we map the uncovered regions in the successive frames onto the mosaic reference frame using local affine models, i.e., those of the neighboring mesh patches. The proposed method to compute this mosaic representation is tightly coupled with an occlusion adaptive 2-D mesh tracking procedure, which consist of propagating the object mesh frame to frame, and updating of both object and mosaic meshes to optimize texture mapping from the mosaic to each instance of the object. The proposed representation has been applied to video object rendering and editing, including self transfiguration, synthetic transfiguration, and 2-D augmented reality in the presence of self and/or external occlusion. We also provide an algorithm to determine the minimum number of still views needed to reconstruct a replacement mosaic which is needed for synthetic transfiguration. Experimental results are provided to demonstrate both the 2-D mesh-based mosaic synthesis and two different video object editing applications on real video sequences.     

An automatic segmentation method for moving objects based on the spatial-temporal information of video

The new MPEG-4 video coding standard enables content-based functions. In order to support the new standard, frames should be decomposed into Video Object Planes （VOP）, each VOP representing a moving object. This paper proposes an image segmentation method to separate moving objects from image sequences. The proposed method utilizes the spatial-temporal information. Spatial segmentation is applied to divide each image into connected areas and to find pre~：ise object boundaries of moving objects. To locate moving objects in image sequences, two consecutive image frames in the temporal direction are examined and a hypothesis testing is performed with Neyman-Pearson criterion. Spatial segmentation produces a spatial segmentation mask, and temporal segmentation yields a change detection mask that indicates moving objects and the background. Then spatial-temporal merging can be used to get the final results. This method has been tested on several images. Experimental results show that this segmentation method is efficient.     

Unsupervised video object segmentation with distractor-aware online adaptation

Unsupervised video object segmentation is a crucial application in video analysis when there is no prior information about the objects. It becomes tremendously challenging when multiple objects occur and interact in a video clip. In this paper, a novel unsupervised video object segmentation approach via distractor-aware online adaptation (DOA) is proposed. DOA models spatiotemporal consistency in video sequences by capturing background dependencies from adjacent frames. Instance proposals are generated by the instance segmentation network for each frame and they are grouped by motion information as positives or hard negatives. To adopt high-quality hard negatives, the block matching algorithm is then applied to preceding frames to track the associated hard negatives. General negatives are also introduced when there are no hard negatives in the sequence. The experimental results demonstrate these two kinds of negatives are complementary. Finally, we conduct DOA using positive, negative, and hard negative masks to update the foreground and background segmentation. The proposed approach achieves state-of-the-art results on two benchmark datasets, the DAVIS 2016 and the Freiburg-Berkeley motion segmentation (FBMS)-59.     

Model-based temporal object verification using video

An approach to model-based dynamic object verification and identification using video is proposed. From image sequences containing the moving object, we compute its motion trajectory. Then we estimate its three-dimensional (3-D) pose at each time step. Pose estimation is formulated as a search problem, with the search space constrained by the motion trajectory information of the moving object and assumptions about the scene structure. A generalized Hausdorff (1962) metric, which is more robust to noise and allows a confidence interpretation, is suggested for the matching procedure used for pose estimation as well as the identification and verification problem. The pose evolution curves are used to assist in the acceptance or rejection of an object hypothesis. The models are acquired from real image sequences of the objects. Edge maps are extracted and used for matching. Results are presented for both infrared and optical sequences containing moving objects involved in complex motions     

基于运动跟踪匹配技术的视频对象提取

本文提出一种自动分割VOP的技术。其方法是：先对初始帧使用形态运动滤波技术提取出初始运动对象的二值轮廓模型,并在后继帧中使用豪斯道夫对象跟踪器跟踪运动以对象模型;而为了适应对象的形状变化,本文使用活动轮廓模型（snake）技术对运动心合匹配;最后根据一系列精确的二值轮廓引导提取运动对象序列。实验结果表明,我们的算法可有效地提取视频对象平面。     

Video inpainting under constrained camera motion.

A framework for inpainting missing parts of a video sequence recorded with a moving or stationary camera is presented in this work. The region to be inpainted is general: it may be still or moving, in the background or in the foreground, it may occlude one object and be occluded by some other object. The algorithm consists of a simple preprocessing stage and two steps of video inpainting. In the preprocessing stage, we roughly segment each frame into foreground and background. We use this segmentation to build three image mosaics that help to produce time consistent results and also improve the performance of the algorithm by reducing the search space. In the first video inpainting step, we reconstruct moving objects in the foreground that are "occluded" by the region to be inpainted. To this end, we fill the gap as much as possible by copying information from the moving foreground in other frames, using a priority-based scheme. In the second step, we inpaint the remaining hole with the background. To accomplish this, we first align the frames and directly copy when possible. The remaining pixels are filled in by extending spatial texture synthesis techniques to the spatiotemporal domain. The proposed framework has several advantages over state-of-the-art algorithms that deal with similar types of data and constraints. It permits some camera motion, is simple to implement, fast, does not require statistical models of background nor foreground, works well in the presence of rich and cluttered backgrounds, and the results show that there is no visible blurring or motion artifacts. A number of real examples taken with a consumer hand-held camera are shown supporting these findings.     

Contour image sequence compression through motion analysis and hybrid coding method

This paper presents a motion analysis algorithm (MAA) and a hybrid coding method for contour image sequence compression. The contour image sequence consists of objects moving and rotating in a 3-D world with occlusion, shape, and size variations from frame to frame. The MAA separates the moving image sequence into several object-oriented subsequences (OOSs). In each OOS, the component is either stationary or moves smoothly, and the motion parameters can be easily estimated. The first and last frames of OOS are key frames, and the others are in-between frames. The key frames are unpredictable, and the entire frames need to be encoded. The in-between frames are compensable, and they are encoded by the motion parameter coding. The hybrid coder uses vectorgraph coding to remove spatial redundancy of the key frames and motion parameter coding to reduce the temporal redundancy of the OOSs. The motion parameters are encoded as combinations of 2-D translation, 2-D rotation, and scaling. There are many applications for contour image sequence compression. The cartoon image sequence (a sequence of line drawing sketches) and the high-frame-rate videophone for sign language transmission are good examples. Experiments show that our method encodes the contour image sequence at a very high compression ratio without losing intelligibility.     

基于时空曲线演化的多视频运动对象分割算法

多视频对象由于其运动的复杂性,在分割提取过程中有较大的难度.本文提出了一种基于时空曲线演化的多视频对象自动分割方法,首先根据视频序列帧间(时间域)和帧内(空间域)信息的不同特点,建立基于全局和局部特征的能量模型,并由此导出基于level sets方法的曲线演化方程;然后用视频序列的连继两帧帧差得到初始的视频对象,分别进行时间和空间曲线演化跟踪,提取多个视频对象;当对象因运动而发生相互遮挡现象时,利用基于Bayes最小错误概率决策法则的判断方法,分割遮挡对象和显露对象.实验结果表明,本文提出算法的分割效果在空间准确度上比COST211算法提高30-50%,比最佳的帧差分割算法提高5-10%.     

空域视频场景监视中运动对象的实时检测与跟踪技术

本文分析了空域视频场景中运动对象实时检测、跟踪系统的模型。提出了一种在运动背景下实时检测与跟踪视频运动目标的技术。该方法首先进行背景的全局运动参数估计,并对背景进行补偿校正,将补偿校正后的相邻两帧进行差分检测。然后利用假设检验从差分图像中提取运动区域,利用遗传学方法在指定区域内确定最优分割门限,提取视频运动对象及其特征;最后利用线性预测器对目标进行匹配跟踪。在基于高速DSP的系统平台上的实验结果表明该方法取得了很好的效果。     

Robust bilayer video segmentation by adaptive propagation of global shape and local appearance

Segmenting semantic objects of interest from video has long been an active research topic, with a wide range of potential applications. In this paper, we present a bilayer video segmentation method robust to abrupt motion and change in appearance for both the foreground and background. Specifically, based on a few manually segmented keyframes, the proposed method propagates the global shape of the foreground as priors to adjacent frames by applying branch-and-mincut [1], which jointly estimates what is optimal among a set of shapes along with its pose and the corresponding segmentation in the current image. Based on this preliminary segmentation we determine two types of local regions likely to have erroneous results, and apply a probabilistic framework where shape and appearance cues are adaptively emphasized for local refinement. With each successive frame segmentation, the set of shapes applied as priors are incrementally updated. Experimental results support the robustness of the proposed method for obstacles such as background clutter, motion, and appearance changes, from only a small number of user segmented keyframes.     

Moving object detection and tracking from video captured by moving camera

This paper presents an effective method for the detection and tracking of multiple moving objects from a video sequence captured by a moving camera without additional sensors. Moving object detection is relatively difficult for video captured by a moving camera, since camera motion and object motion are mixed. In the proposed method, the feature points in the frames are found and then classified as belonging to foreground or background features. Next, moving object regions are obtained using an integration scheme based on foreground feature points and foreground regions, which are obtained using an image difference scheme. Then, a compensation scheme based on the motion history of the continuous motion contours obtained from three consecutive frames is applied to increase the regions of moving objects. Moving objects are detected using a refinement scheme and a minimum bounding box. Finally, moving object tracking is achieved using a Kalman filter based on the center of gravity of a moving object region in the minimum bounding box. Experimental results show that the proposed method has good performance.     

运动检测算法的研究和仿真实现

帧间差法和背景差法都是重要的运动检测方法,其核心问题在于如何得到准确的运动对象.针对该问题,本文提出一种结合帧间差和背景差的自动分割算法.该算法通过累积的帧差信息构建出可靠的背景,再将背景与当前帧比较,进而提取出视频运动对象.本文运用了最大类间方差法OTSU(又名"大津法")来获得自适应阈值,能更准确地对背景差图像进行阈值化分割,克服了传统固定阈值容易失效的问题.还采用了形态滤波的方法,对二值图像进行去噪,填充空洞.     

Estimation of depth fields suitable for video compression based on3-D structure and motion of objects

Intensity prediction along motion trajectories removes temporal redundancy considerably in video compression algorithms. In three-dimensional (3-D) object-based video coding, both 3-D motion and depth values are required for temporal prediction. The required 3-D motion parameters for each object are found by the correspondence-based E-matrix method. The estimation of the correspondences-two-dimensional (2-D) motion field-between the frames and segmentation of the scene into objects are achieved simultaneously by minimizing a Gibbs energy. The depth field is estimated by jointly minimizing a defined distortion and bit-rate criterion using the 3-D motion parameters. The resulting depth field is efficient in the rate-distortion sense. Bit-rate values corresponding to the lossless encoding of the resultant depth fields are obtained using predictive coding; prediction errors are encoded by a Lempel-Ziv algorithm. The results are satisfactory for real-life video scenes.     

Motion analysis and segmentation through spatio-temporal slices processing

This paper presents new approaches in characterizing and segmenting the content of video. These approaches are developed based upon the pattern analysis of spatio-temporal slices. While traditional approaches to motion sequence analysis tend to formulate computational methodologies on two or three adjacent frames, spatio-temporal slices provide rich visual patterns along a larger temporal scale. We first describe a motion computation method based on a structure tensor formulation. This method encodes visual patterns of spatio-temporal slices in a tensor histogram, on one hand, characterizing the temporal changes of motion over time, on the other hand, describing the motion trajectories of different moving objects. By analyzing the tensor histogram of an image sequence, we can temporally segment the sequence into several motion coherent subunits, in addition, spatially segment the sequence into various motion layers. The temporal segmentation of image sequences expeditiously facilitates the motion annotation and content representation of a video, while the spatial decomposition of image sequences leads to a prominent way of reconstructing background panoramic images and computing foreground objects.     

Automatic Segmentation of Moving Objects in Video Sequences for Indoor and Outdoor Applications

1　IntroductionAutomaticsegmentationofmovingobjectsfromvideosequencesisadifficultandchallengingproblemincomputervisionsystems.Ithasmanyapplicationssuchasvideosurveillance,trafficmonitoring ,peopletrackingandvideocommunication[1～4] .Italsoplaysanimportantroleinsupportingcontent basedimagecoding,especiallyaftertheemergenceofthevideocodingstandardMPEG 4[5～ 1 4 ] .Therearealotofresearchworksonmovingob jectssegmentationandextraction .Thesealgorithmscanberoughlyclassifiedintotwocategories:inter …     

Background removal of multiview images by learning shape priors.

Image-based rendering has been successfully used to display 3-D objects for many applications. A well-known example is the object movie, which is an image-based 3-D object composed of a collection of 2-D images taken from many different viewpoints of a 3-D object. In order to integrate image-based 3-D objects into a chosen scene (e.g., a panorama), one has to meet a hard challenge--to efficiently and effectively remove the background from the foreground object. This problem is referred to as multiview images (MVIs) segmentation. Another task requires MVI segmentation is image-based 3-D reconstruction using multiview images. In this paper, we propose a new method for segmenting MVI, which integrates some useful algorithms, including the well-known graph-cut image segmentation and volumetric graph-cut. The main idea is to incorporate the shape prior into the image segmentation process. The shape prior introduced into every image of the MVI is extracted from the 3-D model reconstructed by using the volumetric graph cuts algorithm. Here, the constraint obtained from the discrete medial axis is adopted to improve the reconstruction algorithm. The proposed MVI segmentation process requires only a small amount of user intervention, which is to select a subset of acceptable segmentations of the MVI after the initial segmentation process. According to our experiments, the proposed method can provide not only good MVI segmentation, but also provide acceptable 3-D reconstructed models for certain less-demanding applications.     

运动目标的自动分割与跟踪

该文提出了一种对视频序列中的运动目标进行自动分割的算法。该算法分析图像在L U V空间中的局部变化,同时使用运动信息来把目标从背景中分离出来。首先根据图像的局部变化,使用基于图论的方法把图像分割成不同的区域。然后,通过度量合成的全局运动与估计的局部运动之间的偏差来检测出运动的区域,运动的区域通过基于区域的仿射运动模型来跟踪到下一帧。为了提高提取的目标的时空连续性,使用Hausdorff跟踪器对目标的二值模型进行跟踪。对一些典型的MPEG-4测试序列所进行的评估显示了该算法的优良性能。     

用于视频对象平面生成的运动对象自动分割

新的视频编码标准MPEG-4具有基于内容的功能。它把图像序列分解成视频对象平面(VOP),每个VOP代表一个运动对象。文中提出了一种提取运动对象的新的视频序列分割算法,算法的核心是一个对象跟踪器,它利用Hausdorff距离将对象的二维二值模型与后续帧进行匹配,然后采用一种新的基于运动相连成分的模型刷新方法对模型的每一帧进行刷新。初始的模型自动产生,再利用滤波技术滤除静止背景,最后,利用二值模型从序列中提取出VOP。