Motion Competition: A Variational Approach to Piecewise Parametric Motion Segmentation

We present a novel variational approach for segmenting the image plane into a set of regions of parametric motion on the basis of two consecutive frames from an image sequence. Our model is based on a conditional probability for the spatio-temporal image gradient, given a particular velocity model, and on a geometric prior on the estimated motion field favoring motion boundaries of minimal length.Exploiting the Bayesian framework, we derive a cost functional which depends on parametric motion models for each of a set of regions and on the boundary separating these regions. The resulting functional can be interpreted as an extension of the Mumford-Shah functional from intensity segmentation to motion segmentation. In contrast to most alternative approaches, the problems of segmentation and motion estimation are jointly solved by continuous minimization of a single functional. Minimizing this functional with respect to its dynamic variables results in an eigenvalue problem for the motion parameters and in a gradient descent evolution for the motion discontinuity set.We propose two different representations of this motion boundary: an explicit spline-based implementation which can be applied to the motion-based tracking of a single moving object, and an implicit multiphase level set implementation which allows for the segmentation of an arbitrary number of multiply connected moving objects.Numerical results both for simulated ground truth experiments and for real-world sequences demonstrate the capacity of our approach to segment objects based exclusively on their relative motion.     

Hierarchical Estimation and Segmentation of Dense Motion Fields

In this paper we present a comprehensive energy-based framework for the estimation and the segmentation of the apparent motion in image sequences. The robust cost functions and the associated hierarchical minimization techniques that we propose mix efficiently non-parametric (dense) representations, local interacting parametric representations, and global non-interacting parametric representations related to a partition into regions. Experimental comparisons, both on synthetic and real images, demonstrate the merit of the approach on different types of photometric and kinematic contents ranging from moving rigid objects to moving fluids.     

Moving object detection and segmentation in urban environments from a moving platform

This paper proposes an effective approach to detect and segment moving objects from two time-consecutive stereo frames, which leverages the uncertainties in camera motion estimation and in disparity computation. First, the relative camera motion and its uncertainty are computed by tracking and matching sparse features in four images. Then, the motion likelihood at each pixel is estimated by taking into account the ego-motion uncertainty and disparity in computation procedure. Finally, the motion likelihood, color and depth cues are combined in the graph-cut framework for moving object segmentation. The efficiency of the proposed method is evaluated on the KITTI benchmarking datasets, and our experiments show that the proposed approach is robust against both global (camera motion) and local (optical flow) noise. Moreover, the approach is dense as it applies to all pixels in an image, and even partially occluded moving objects can be detected successfully. Without dedicated tracking strategy, our approach achieves high recall and comparable precision on the KITTI benchmarking sequences.     

视频图像序列运动参数估计与动态拼接

本文采用多重分层叠代算法来估计全局运动参数，并提出应用于动态拼接的运动分割新方法，实现既有摄像机运动又有物体运动的视频图像序列自动拼接。我们的方法基本步骤如下：首先进行全局运动参数的初始估计，并且在分层叠代过程中进行区域分类，得到初始运动模板。接着空间分割原始图像，先根据图像的空间属性由底向上分层合并图像空间区域，再利用视频图像时间属性进一步向上合并，得到图像空间分割结果。然后结合初始运动模板和图像空间分割结果，采用区域分类新方法重新对图像空间分割结果的每个区域进行分类。然后根据分类结果逐步精确求解全局运动参数。最后进行图像合成，得到全景拼接图像。我们的方法利用了多重分层叠代的优点，并且充分考虑到视频图像空间和时间上的属性，实现了运动物体和覆盖背景的精确分割，避免了遮挡问题对全局运动参数估计精度的影响。而且在图像合成时我们解决了拼接图可能产生模糊或某些区域不连续等问题。实验结果表明我们的方法实现了动态视频图像序列高质量的全景拼接。     

Motion estimation,segmentation and separation,using hypercomplex phase correlation,clustering techniques and graph-based optimization

The problem of estimating the motions in image sequences has been extensively studied. However, most of the proposed motion estimation approaches deal only with the luminance component of the images, although the color information is also important. The usual way to apply these techniques in color image sequences is to process each color channel separately. As proposed by Ell, Sangwine and Moxey, a more sophisticated approach is to handle the color channels in a “holistic” manner, using quaternions. We present a motion estimation approach which is based on the Hypercomplex (quaternionic) Phase Correlation of Moxey et al. We show that the quaternionic approach is computationally more efficient and more robust to sensor noise, compared with the corresponding three-channels-separately approach. With the assumption of smoothly time-varying velocities we propose the application of a weighted fuzzy c-means clustering procedure to the obtained velocity estimates. This renders the estimation more robust. A methodology in the hypercomplex Fourier transform domain for separating the moving objects/layers is also presented. An energy-minimization-based approach for the spatial assignment of the velocities and the creation of segmentation maps, is given. We furthermore show how to apply the motion estimation approach locally in space for the extraction of dense motion fields. Our experimental results and comparisons with state-of-the-art methodologies verify the effectiveness of the proposed approach.     

Retargeting images and video for preserving information saliency.

A nonphotorealistic algorithm for retargeting images adapts large images so that important objects in the image are still recognizable when displayed at a lower target resolution. Unlike existing image manipulation techniques such as cropping and scaling, the retargeting algorithm can handle multiple important objects in an image. To identify the important objects in an image, we must first segment the image. We use mean-shift image segmentation to decompose an image into homogeneous regions.     

Object-based image labeling through learning by example and multi-level segmentation

We propose a method for automatic extraction and labeling of semantically meaningful image objects using “learning by example” and threshold-free multi-level image segmentation. The proposed method scans through images, each of which is pre-segmented into a hierarchical uniformity tree, to seek and label objects that are similar to an example object presented by the user. By representing images with stacks of multi-level segmentation maps, objects can be extracted in the segmentation map level with adequate detail. Experiments have shown that the proposed multi-level image segmentation results in significant reduction in computation complexity for object extraction and labeling (compared to a single fine-level segmentation) by avoiding unnecessary tests of combinations in finer levels. The multi-level segmentation-based approach also achieves better accuracy in detection and labeling of small objects.     

基于非线性最小二乘的图像自适应SR重建以及运动估计

图像超分辨率（SR）重建是利用数字信号处理技术由一系列低分辨率观测图像得到高分辨率图像。为了扩展SR技术的应用范围,提出了一种同时进行图像超分辨率重建和全局运动估计的方法。该方法首先基于最大后验概率(MAP)给出了图像SR重建和运动估计框架,该框架不仅考虑了前后两次迭代所得的HR图像差值对最终重建图像的影响,而且引入了不同LR图像对重建图像的重要性权值,使得算法具有自适应性;然后将总体框架转换为图像SR重建模型和运动估计模型;最后基于非线性最小二乘法对模型进行优化求解,得出了SR重建图像及其全局运动域。实验表明,该方法不仅图像重建效果良好,并有着良好的收敛性。     

结合几何划分技术和最大期望值/最大边缘概率算法的彩色图像分割

本文针对基于区域和统计的彩色图像分割方法进行研究,提出了一种结合Voronoi划分技术、最大期望值(Expectation Maximization, EM)和最大边缘概率(Maximization of the Posterior Marginal, MPM)算法的彩色图像分割方法。首先利用Voronoi几何划分将图像域划分成不同的子区域,并假设每个子区域内的像素强度满足独立同一的概率分布,在此基础上建立彩色图像模型;利用上述模型,在贝叶斯理论架构下建模图像分割问题,然后结合EM/MPM算法进行图像分割。该方法将基于像素的MRF模型扩展到基于区域的MRF,并且能同时有效的获取模型参数估计和基于区域的彩色图像最优分割。采用本文算法,分别对真实彩色图像和合成彩色图像进行了分割实验,定性和定量的测试结果验证了本文方法的有效性、可靠性和准确性。     

Segmentation and Interpretation of Multicolored Objects with Highlights

This paper presents a segmentation system, based on a general framework for segmentation, that returns not only regions that correspond to coherent surfaces in an image, but also low-level interpretations of those regions' physical characteristics. This system is valid for images of piecewise uniform dielectric objects with highlights, moving it beyond the capabilities of previous physics-based segmentation algorithms which assume uniformly colored objects. This paper presents a summary of the complete system and focuses on two extensions of it that demonstrate its interpretive capacity and applicability to more complex scenes. The first extension provides interpretations of a scene by reasoning about the likelihood of different physical characteristics of simple image regions. The second extension allows the system to handle highlights within the general framework for segmentation. The resulting segmentations and interpretations more closely match our perceptions of objects since the resulting regions correspond to coherent surfaces, even when those surfaces have multiple colors and highlights.     

Range image segmentation based on differential geometry: a hybridapproach

The authors describe a hybrid approach to the problem of image segmentation in range data analysis, where hybrid refers to a combination of both region- and edge-based considerations. The range image of 3-D objects is divided into surface primitives which are homogeneous in their intrinsic differential geometric properties and do not contain discontinuities in either depth of surface orientation. The method is based on the computation of partial derivatives, obtained by a selective local biquadratic surface fit. Then, by computing the Gaussian and mean curvatures, an initial region-gased segmentation is obtained in the form of a curvature sign map. Two additional initial edge-based segmentations are also computed from the partial derivatives and depth values, namely, jump and roof-edge maps. The three image maps are then combined to produce the final segmentation. Experimental results obtained for both synthetic and real range data of polyhedral and curved objects are given     

Two-Dimensional Compact Variational Mode Decomposition

Decomposing multidimensional signals, such as images, into spatially compact, potentially overlapping modes of essentially wavelike nature makes these components accessible for further downstream analysis. This decomposition enables space–frequency analysis, demodulation, estimation of local orientation, edge and corner detection, texture analysis, denoising, inpainting, or curvature estimation. Our model decomposes the input signal into modes with narrow Fourier bandwidth; to cope with sharp region boundaries, incompatible with narrow bandwidth, we introduce binary support functions that act as masks on the narrow-band mode for image recomposition. \(L^1\) and TV terms promote sparsity and spatial compactness. Constraining the support functions to partitions of the signal domain, we effectively get an image segmentation model based on spectral homogeneity. By coupling several submodes together with a single support function, we are able to decompose an image into several crystal grains. Our efficient algorithm is based on variable splitting and alternate direction optimization; we employ Merriman–Bence–Osher-like threshold dynamics to handle efficiently the motion by mean curvature of the support function boundaries under the sparsity promoting terms. The versatility and effectiveness of our proposed model is demonstrated on a broad variety of example images from different modalities. These demonstrations include the decomposition of images into overlapping modes with smooth or sharp boundaries, segmentation of images of crystal grains, and inpainting of damaged image regions through artifact detection.     

A neural paradigm for time-varying motion segmentation

This paper proposes a new neural algorithm to perform the segmentation of an observed scene into regions corresponding to different moving objects by analyzing a time-varying images sequence.The method consists of a classification step,where the motion of small patches is characterized through an optimization approach,and a segmentation step merging meighboring patches characterized by the same motion.Classification of motion is performed without optical flow computation,but considering only the spatial and temporal image gradients into an appropriate energy function minimized with a Hopfield-like neural network giving as output directly the 3D motion parameter estimates.Network convergence is accelerated by integrating the quantitative estimation of motion parameters with a qualitative estimate of dominant motion using the geometric theory of differential equations.     

Linear color segmentation and its implementation

A framework for color image segmentation is presented, which combines color histogram analysis and region merging approach. Its main goal is to segment an image at material boundaries (i.e., discontinuities of reflectance properties) while ignoring spatial color inhomogeneities of uniformly pigmented (colored) objects, caused by accidents of illumination and viewing geometry. Theoretical examination of light spectrum transformations upon light reflection from material surfaces and upon interaction with a sensor system shows that in a wide variety of viewed scenes (even containing interreflections and highlight areas) uniformly pigmented objects are projected to the color space of the sensor as planar, linear, or point-like clusters, depending on lighting and viewing conditions and object geometry. To detect such clusters in the color space, three methods are suggested: Generalized Hough Transform method, gradient descent method, and eigenvectors method. A framework algorithm of color segmentation based on region merging approach is developed, which can use any of these methods. Testing this algorithm with both artificially generated and real images shows quite reliable results.     

Motion estimation of elastic articulated objects from points and contours with volume invariable constraint

This paper presents a model of elastic articulated objects based on revolving conic surface and a method of model-based motion estimation. The model includes 3D object skeleton and deformable surfaces that can represent the deformation of human body surfaces. In each limb, surface deformation is represented by adjusting one or two deformation parameters. Then, the 3D deformation parameters are determined by corresponding 2D image points and contours with volume invariable constraint from a sequence of stereo images. The 3D motion parameters are estimated based on the 3D model. The algorithm presented in this paper includes model-based parameter estimation of motion and parameter determination of deformable surfaces.     

A Novel Self-Organizing Neural Network for Motion Segmentation

Many computer vision techniques, above all for structure from motion problems, require a segmentation of the images captured by one or more cameras. This paper deals with the segmentation based on the motion information, but can be easily extended to other cases (color, texture and so on). A new neural network, the EXIN Segmentation Neural Network (EXIN SNN) is here introduced. It is incremental, self-organizing and considers its task as the solution of a pattern recognition problem. This original approach overcomes the limits of the traditional segmentation techniques, namely the need of a spatial support for the image objects and the translation parallel to the image plane for the objects in the scene. Examples are given both for synthetic and real images.     

NeTra: A toolbox for navigating large image databases

We present here an implementation of NeTra, a prototype image retrieval system that uses color, texture, shape and spatial location information in segmented image regions to search and retrieve similar regions from the database. A distinguishing aspect of this system is its incorporation of a robust automated image segmentation algorithm that allows object- or region-based search. Image segmentation significantly improves the quality of image retrieval when images contain multiple complex objects. Images are segmented into homogeneous regions at the time of ingest into the database, and image attributes that represent each of these regions are computed. In addition to image segmentation, other important components of the system include an efficient color representation, and indexing of color, texture, and shape features for fast search and retrieval. This representation allows the user to compose interesting queries such as “retrieve all images that contain regions that have the color of object A, texture of object B, shape of object C, and lie in the upper of the image”, where the individual objects could be regions belonging to different images. A Java-based web implementation of NeTra is available at http://vivaldi.ece.ucsb.edu/Netra.     

Occlusion-Aware 3D Morphable Models and an Illumination Prior for Face Image Analysis

Faces in natural images are often occluded by a variety of objects. We propose a fully automated, probabilistic and occlusion-aware 3D morphable face model adaptation framework following an analysis-by-synthesis setup. The key idea is to segment the image into regions explained by separate models. Our framework includes a 3D morphable face model, a prototype-based beard model and a simple model for occlusions and background regions. The segmentation and all the model parameters have to be inferred from the single target image. Face model adaptation and segmentation are solved jointly using an expectation–maximization-like procedure. During the E-step, we update the segmentation and in the M-step the face model parameters are updated. For face model adaptation we apply a stochastic sampling strategy based on the Metropolis–Hastings algorithm. For segmentation, we apply loopy belief propagation for inference in a Markov random field. Illumination estimation is critical for occlusion handling. Our combined segmentation and model adaptation needs a proper initialization of the illumination parameters. We propose a RANSAC-based robust illumination estimation technique. By applying this method to a large face image database we obtain a first empirical distribution of real-world illumination conditions. The obtained empirical distribution is made publicly available and can be used as prior in probabilistic frameworks, for regularization or to synthesize data for deep learning methods.     

基于块运动矢量加权的Snakes模型及在序列图像分割中的应用

经典的Snakes模型具有开放的、统一的架构，在此基础上，为了分割复杂背景的序列图像，产生了各种改进的Snakes模型，但都存在着不足：计算量大、需要先验知识、易受光流计算精度影响等。针对这些缺点，提出了块运动矢量加权的Snakes模型，可以用于复杂背景序列图像的分割。这种模型以图像中的边缘信息为分割的最终依据，结合块运动估计的结果，增强了序列图像分割的鲁棒性。根据运动场估计的结果在该模型中所起的作用，提出了边缘优先的块运动估计算法，大大减少了计算量。用块运动矢量加权的Snakes模型分割复杂背景序列图像，取得了好的分割结果。     

A generic knowledge-guided image segmentation and labeling system using fuzzy clustering algorithms

Segmentation of an image into regions and the labeling of the regions is a challenging problem. In this paper, an approach that is applicable to any set of multifeature images of the same location is derived. Our approach applies to, for example, medical images of a region of the body; repeated camera images of the same area; and satellite images of a region. The segmentation and labeling approach described here uses a set of training images and domain knowledge to produce an image segmentation system that can be used without change on images of the same region collected over time. How to obtain training images, integrate domain knowledge, and utilize learning to segment and label images of the same region taken under any condition for which a training image exists is detailed. It is shown that clustering in conjunction with image processing techniques utilizing an iterative approach can effectively identify objects of interest in images. The segmentation and labeling approach described here is applied to color camera images and two other image domains are used to illustrate the applicability of the approach.