Shape Based Detection and Top-Down Delineation Using Image Segments

We introduce a segmentation-based detection and top-down figure-ground delineation algorithm. Unlike common methods which use appearance for detection, our method relies primarily on the shape of objects as is reflected by their bottom-up segmentation. Our algorithm receives as input an image, along with its bottom-up hierarchical segmentation. The shape of each segment is then described both by its significant boundary sections and by regional, dense orientation information derived from the segment’s shape using the Poisson equation. Our method then examines multiple, overlapping segmentation hypotheses, using their shape and color, in an attempt to find a “coherent whole,” i.e., a collection of segments that consistently vote for an object at a single location in the image. Once an object is detected, we propose a novel pixel-level top-down figure-ground segmentation by “competitive coverage” process to accurately delineate the boundaries of the object. In this process, given a particular detection hypothesis, we let the voting segments compete for interpreting (covering) each of the semantic parts of an object. Incorporating competition in the process allows us to resolve ambiguities that arise when two different regions are matched to the same object part and to discard nearby false regions that participated in the voting process. We provide quantitative and qualitative experimental results on challenging datasets. These experiments demonstrate that our method can accurately detect and segment objects with complex shapes, obtaining results comparable to those of existing state of the art methods. Moreover, our method allows us to simultaneously detect multiple instances of class objects in images and to cope with challenging types of occlusions such as occlusions by a bar of varying size or by another object of the same class, that are difficult to handle with other existing class-specific top-down segmentation methods.     

SAR图像的最优分割方法

根据SAR图像的概率密度函数获得图像的拟然函数,然后将似然函数和边界约束方程结合起来,提出适合于SAR图像分割的代价函数,其中边界约束方程引入邻域结构信息来保证区域边界的规则性,通过使代价函数最小来获得图像的最优分割。算法首先将原图分割成一定大小的块状区域作为初始分割,每一区域代表一个类别;然后随机调整相邻两个区域之间的像素,通过比较代价函数的变化,利用模拟退火算法确定接受该调整的概率。模拟退火是一种求解全局最优的算法,当温度趋向于0时,它可以获得使代价函数最小的SAR图像的分割。最后,利用基于相似性的融合方法对分割进行后期处理,将相似的较小的区域融合成较大的区域,使得分割更合理。我们将该算法应用到一些SAR测试图像上,获得了比较满意的结果。     

基于高阶CRF模型的图像语义分割

图像语义分割方法大多基于点对条件随机场模型, 不能定位到单个目标, 并且难以利用全局形状特征, 造成误识。针对这些问题, 提出一种新的高阶条件随机场模型, 将基于全局形状特征的目标检测结果和点对条件随机场模型统一在一个概率模型框架中, 同时完成图像分割、目标检测与识别的任务。利用目标检测器和前背景分割算法获取图像中目标区域, 在目标区域上定义新的高阶能量项。新的高阶条件随机场模型就是高阶能量项和点对条件随机场模型的加权混合模型, 其最优解即为图像语义分割结果。在MSRC-21类数据库上进行的实验验证了该模型能够显著提升图像语义分割性能, 并定位到单个目标。     

Mumford-Shah on the Move: Region-Based Segmentation on Deforming Manifolds with Application to 3-D Reconstruction of Shape and Appearance from Multi-View Images

We address the problem of estimating the shape and appearance of a scene made of smooth Lambertian surfaces with piecewise smooth albedo. We allow the scene to have self-occlusions and multiple connected components. This class of surfaces is often used as an approximation of scenes populated by man-made objects. We assume we are given a number of images taken from different vantage points. Mathematically this problem can be posed as an extension of Mumford and Shah’s approach to static image segmentation to the segmentation of a function defined on a deforming surface. We propose an iterative procedure to minimize a global cost functional that combines geometric priors on both the shape of the scene and the boundary between smooth albedo regions. We carry out the numerical implementation in the level set framework.     

Prior-based Segmentation and Shape Registration in the Presence of Perspective Distortion

Challenging object detection and segmentation tasks can be facilitated by the availability of a reference object. However, accounting for possible transformations between the different object views, as part of the segmentation process, remains difficult. Recent statistical methods address this problem by using comprehensive training data. Other techniques can only accommodate similarity transformations. We suggest a novel variational approach to prior-based segmentation, using a single reference object, that accounts for planar projective transformation. Generalizing the Chan-Vese level set framework, we introduce a novel shape-similarity measure and embed the projective homography between the prior shape and the image to segment within a region-based segmentation functional. The proposed algorithm detects the object of interest, extracts its boundaries, and concurrently carries out the registration to the prior shape. We demonstrate prior-based segmentation on a variety of images and verify the accuracy of the recovered transformation parameters.     

Stereoscopic Segmentation

We cast the problem of multiframe stereo reconstruction of a smooth shape as the global region segmentation of a collection of images of the scene. Dually, the problem of segmenting multiple calibrated images of an object becomes that of estimating the solid shape that gives rise to such images. We assume that the radiance of the scene results in piecewise homogeneous image statistics. This simplifying assumption covers Lambertian scenes with constant albedo as well as fine homogeneous textures, which are known challenges to stereo algorithms based on local correspondence. We pose the segmentation problem within a variational framework, and use fast level set methods to find the optimal solution numerically. Our algorithm does not work in the presence of strong photometric features, where traditional reconstruction algorithms do. It enjoys significant robustness to noise under the assumptions it is designed for.     

Real-time landing place assessment in man-made environments

We propose a novel approach to the real-time landing site detection and assessment in unconstrained man-made environments using passive sensors. Because this task must be performed in a few seconds or less, existing methods are often limited to simple local intensity and edge variation cues. By contrast, we show how to efficiently take into account the potential sites’ global shape, which is a critical cue in man-made scenes. Our method relies on a new segmentation algorithm and shape regularity measure to look for polygonal regions in video sequences. In this way, we enforce both temporal consistency and geometric regularity, resulting in very reliable and consistent detections. We demonstrate our approach for the detection of landable sites such as rural fields, building rooftops and runways from color and infrared monocular sequences significantly outperforming the state-of-the-art.     

<Emphasis Type="Italic">Skeleton Search</Emphasis>: Category-Specific Object Recognition and Segmentation Using a Skeletal Shape Model

We describe a top-down object detection and segmentation approach that uses a skeleton-based shape model and that works directly on real images. The approach is based on three components. First, we propose a fragment-based generative model for shape that is based on the shock graph and has minimal dependency among its shape fragments. The model is capable of generating a wide variation of shapes as instances of a given object category. Second, we develop a progressive selection mechanism to search among the generated shapes for the category instances that are present in the image. The search begins with a large pool of candidates identified by a dynamic programming (DP) algorithm and progressively reduces it in size by applying series of criteria, namely, local minimum criterion, extent of shape overlap, and thresholding of the objective function to select the final object candidates. Third, we propose the Partitioned Chamfer Matching (PCM) measure to capture the support of image edges for a hypothesized shape. This measure overcomes the shortcomings of the Oriented Chamfer Matching and is robust against spurious edges, missing edges, and accidental alignment between the image edges and the shape boundary contour. We have evaluated our approach on the ETHZ dataset and found it to perform well in both object detection and object segmentation tasks.     

The Shape Boltzmann Machine: A Strong Model of Object Shape

A good model of object shape is essential in applications such as segmentation, detection, inpainting and graphics. For example, when performing segmentation, local constraints on the shapes can help where object boundaries are noisy or unclear, and global constraints can resolve ambiguities where background clutter looks similar to parts of the objects. In general, the stronger the model of shape, the more performance is improved. In this paper, we use a type of deep Boltzmann machine (Salakhutdinov and Hinton, International Conference on Artificial Intelligence and Statistics, 2009) that we call a Shape Boltzmann Machine (SBM) for the task of modeling foreground/background (binary) and parts-based (categorical) shape images. We show that the SBM characterizes a strong model of shape, in that samples from the model look realistic and it can generalize to generate samples that differ from training examples. We find that the SBM learns distributions that are qualitatively and quantitatively better than existing models for this task.     

Perceptual grouping of segmented regions in color images

Image segmentation is often the first yet important step of an image understanding system. However, general-purpose image segmentation algorithms that do not rely on specific object models still cannot produce perceptually coherent segmentation of regions at a level comparable to humans. Over-segmentation and under-segmentation have plagued the research community in spite of many significant advances in the field. Therefore, grouping of segmented region plays a significant role in bridging image segmentation and high-level image understanding. In this paper, we focused on non-purposive grouping (NPG), which is built on general expectations of a perceptually desirable segmentation as opposed to any object specific models, such that the grouping algorithm is applicable to any image understanding application. We propose a probabilistic model for the NPG problem by defining the regions as a Markov random field (MRF). A collection of energy functions is used to characterize desired single-region properties and pair-wise region properties. The single-region properties include region area, region convexity, region compactness, and color variances in one region. The pair-wise properties include color mean differences between two regions; edge strength along the shared boundary; color variance of the cross-boundary area; and contour continuity between two regions. The grouping process is implemented by a greedy method using a highest confidence first (HCF) principle. Experiments have been performed on hundreds of color photographic images to show the effectiveness of the grouping algorithm using a set of fixed parameters.     

Unsupervised segmentation of unknown objects in complex environments

This paper presents a novel object segmentation approach for highly complex indoor scenes. Our approach starts with a novel algorithm which partitions the scene into distinct regions whose boundaries accurately conform to the physical object boundaries in the scene. Next, we propose a novel perceptual grouping algorithm based on local cues (e.g., 3D proximity, co-planarity, and shape convexity) to merge these regions into object hypotheses. Our extensive experimental evaluations demonstrate that our object segmentation results are superior compared to the state-of-the-art methods.     

Optimizing Object Decomposition to Reduce Visual Artifacts in 3D Printing

We propose a method for the automatic segmentation of 3D objects into parts which can be individually 3D printed and then reassembled by preserving the visual quality of the final object. Our technique focuses on minimizing the surface affected by supports, decomposing the object into multiple parts whose printing orientation is automatically chosen. The segmentation reduces the visual impact on the fabricated model producing non-planar cuts that adapt to the object shape. This is performed by solving an optimization problem that balances the effects of supports and cuts, while trying to place both in occluded regions of the object surface. To assess the practical impact of the solution, we show a number of segmented, 3D printed and reassembled objects.     

Efficient Graph-Based Image Segmentation

This paper addresses the problem of segmenting an image into regions. We define a predicate for measuring the evidence for a boundary between two regions using a graph-based representation of the image. We then develop an efficient segmentation algorithm based on this predicate, and show that although this algorithm makes greedy decisions it produces segmentations that satisfy global properties. We apply the algorithm to image segmentation using two different kinds of local neighborhoods in constructing the graph, and illustrate the results with both real and synthetic images. The algorithm runs in time nearly linear in the number of graph edges and is also fast in practice. An important characteristic of the method is its ability to preserve detail in low-variability image regions while ignoring detail in high-variability regions.     

Elastic shapes models for improving segmentation of object boundaries in synthetic aperture sonar images

We present a variational framework for naturally incorporating prior shape knowledge in guidance of active contours for boundary extraction in images. This framework is especially suitable for images collected outside the visible spectrum, where boundary estimation is difficult due to low contrast, low resolution, and presence of noise and clutter. Accordingly, we illustrate this approach using the segmentation of various objects in synthetic aperture sonar (SAS) images of underwater terrains. We use elastic shape analysis of planar curves in which the shapes are considered as elements of a quotient space of an infinite dimensional, non-linear Riemannian manifold. Using geodesic paths under the elastic Riemannian metric, one computes sample mean and covariances of training shapes in each classes and derives statistical models for capturing class-specific shape variability. These models are then used as shape priors in a variational setting to solve for Bayesian estimation of desired contours as follows. In traditional active contour models curves are driven towards minimum of an energy composed of image and smoothing terms. We introduce an additional shape term based on shape models of relevant shape classes. The minimization of this total energy, using iterated gradient-based updates of curves, leads to an improved segmentation of object boundaries. This is demonstrated using a number of shape classes in two large SAS image datasets.     

全局孤立性和局部同质性图表示的随机游走显著目标检测算法

目前的显著性检测算法主要依赖像素间的相互对比,缺乏对显著目标自身特性的分析理解. 依据显著目标是显眼、紧凑和完整的思路,提出一种基于目标全局孤立性和局部同质性的 随机游走显著目标检测算法,将视觉显著性检测公式化为马尔科夫随机游走问题. 首先将输入图像进行分块,根据像素块之间颜色特征和方向特征的相似性确定边的权重, 从而构建图模型;然后通过全连通图搜索提取全局特性,突出全局较孤立的区域; 同时通过k-regular图搜索提取局部特性,增强局部较均匀的区域;最后将全局特性和局部 特性相结合得到显著图,进而确定感兴趣区域位置. 实验结果表明,相比于其他两种具有代表性的算法,所提方法检测结果更加准确、合理, 证明该算法切实可行.     

Multiview stereo via volumetric Graph-Cuts and occlusion robust photo-consistency

This paper presents a volumetric formulation for the multi-view stereo problem which is amenable to a computationally tractable global optimisation using Graph-cuts. Our approach is to seek the optimal partitioning of 3D space into two regions labelled as "object" and "empty" under a cost functional consisting of the following two terms: (1) A term that forces the boundary between the two regions to pass through photo-consistent locations and (2) a ballooning term that inflates the "object" region. To take account of the effect of occlusion on the first term we use an occlusion robust photo-consistency metric based on Normalised Cross Correlation, which does not assume any geometric knowledge about the reconstructed object. The globally optimal 3D partitioning can be obtained as the minimum cut solution of a weighted graph.     

Shape and View Independent Reflectance Map from Multiple Views

We consider the problem of estimating the 3D shape and reflectance properties of an object made of a single material from a set of calibrated views. To model the reflectance, we propose to use the View Independent Reflectance Map (VIRM), which is a representation of the joint effect of the diffuse+specular Bidirectional Reflectance Distribution Function (BRDF) and the environment illumination. The object shape is parameterized using a triangular mesh. We pose the estimation problem as minimizing the cost of matching input images, and the images synthesized using the shape and VIRM estimates. We show that by enforcing a constant value of VIRM as a global constraint, we can minimize the cost function by iterating between the VIRM and shape estimation. Experimental results on both synthetic and real objects show that our algorithm can recover both the 3D shape and the diffuse/specular reflectance information. Our algorithm does not require the light sources to be known or calibrated. The estimated VIRM can be used to predict the appearances of objects with the same material from novel viewpoints and under transformed illumination. The support of National Science Foundation under grant ECS 02-25523 is gratefully acknowledged. Tianli Yu was supported in part by a Beckman Institute Graduate Fellowship.     

Entropy based region selection for moving object detection

This article addresses a problem of moving object detection by combining two kinds of segmentation schemes: temporal and spatial. It has been found that consideration of a global thresholding approach for temporal segmentation, where the threshold value is obtained by considering the histogram of the difference image corresponding to two frames, does not produce good result for moving object detection. This is due to the fact that the pixels in the lower end of the histogram are not identified as changed pixels (but they actually correspond to the changed regions). Hence there is an effect on object background classification. In this article, we propose a local histogram thresholding scheme to segment the difference image by dividing it into a number of small non-overlapping regions/windows and thresholding each window separately. The window/block size is determined by measuring the entropy content of it. The segmented regions from each window are combined to find the (entire) segmented image. This thresholded difference image is called the change detection mask (CDM) and represent the changed regions corresponding to the moving objects in the given image frame. The difference image is generated by considering the label information of the pixels from the spatially segmented output of two image frames. We have used a Markov Random Field (MRF) model for image modeling and the maximum a posteriori probability (MAP) estimation (for spatial segmentation) is done by a combination of simulated annealing (SA) and iterated conditional mode (ICM) algorithms. It has been observed that the entropy based adaptive window selection scheme yields better results for moving object detection with less effect on object background (mis) classification. The effectiveness of the proposed scheme is successfully tested over three video sequences.     

面向RGBD图像的标记分水岭分割

目的 针对分水岭分割算法中存在的过分割现象及现有基于RGB图像分割方法的局限,提出了一种基于RGB图像和深度图像(RGBD)的标记分水岭分割算法。方法 本文使用物体表面几何信息来辅助进行图像分割,定义了一种深度梯度算子和一种法向量梯度算子来衡量物体表面几何信息的变化。通过生成深度梯度图像和法向量梯度图像,与彩色梯度图像进行融合,实现标记图像的提取。在此基础上,使用极小值标定技术对彩色梯度图像进行修正,然后使用分水岭算法进行图像分割。结果 在纽约大学提供的NYU2数据集上进行实验,本文算法有效抑制了过分割现象,将分割区域从上千个降至数十个,且获得了与人工标定的分割结果更接近的分割效果,分割的准确率也比只使用彩色图像进行分割提高了10%以上。结论 本文算法普遍适用于RGBD图像的分割问题,该算法加入了物体表面几何信息的使用,提高了分割的准确率,且对颜色纹理相似的区域获得了较好的分割结果。