Dynamic Shape and Appearance Modeling via Moving and Deforming Layers

We propose a model of the shape, motion and appearance of a scene, seen through a sequence of images, that captures occlusions, scene deformations, unconstrained viewpoint variations and changes in its radiance. This model is based on a collection of overlapping layers that can move and deform, each supporting an intensity function that can change over time. We discuss the generality and limitations of this model in relation to existing ones such as traditional optical flow or motion segmentation, layers, deformable templates and deformotion. We then illustrate how this model can be used for inference of shape, motion, deformation and appearance of the scene from a collection of images. The layering structure allows for automatic inpainting of partially occluded regions. We illustrate the model on synthetic and real sequences where existing schemes fail, and show how suitable choices of constants in the model yield existing schemes, from optical flow to motion segmentation and inpainting.     

New Possibilities in Image Diffusion and Sharpening via High-Order Sobolev Gradient Flows

We study the use of high-order Sobolev gradients for PDE-based image smoothing and sharpening, extending our previous work on this problem. In particular, we study the gradient descent equation on the heat equation energy functional obtained by modifying the usual metric on the space of images, which is the L ² metric, to a weighted H ^k Sobolev metric. We present existence and uniqueness results which show that the Sobolev diffusion PDE are well-posed both in the forward and backward direction. Furthermore, we perform a Fourier analysis on the scale space generated by the Sobolev PDE and show that as the order of the Sobolev metric tends to infinity, the Sobolev gradients converge to a Gaussian smoothed L ² gradient. We then present experimental results which exploit the theoretical stability results by applying the various Sobolev gradient flows in the backward direction for image sharpening effects. Furthermore, we show that as the Sobolev order is increased, the sharpening effects become more global in nature and more immune to noise.     

Local or global minima: flexible dual-front active contours

Most variational active contour models are designed to find local minima of data-dependent energy functionals with the hope that reasonable initial placement of the active contour will drive it toward a "desirable" local minimum as opposed to an undesirable configuration due to noise or complex image structure. As such, there has been much research into the design of complex region-based energy functionals that are less likely to yield undesirable local minima when compared to simpler edge-based energy functionals whose sensitivity to noise and texture is significantly worse. Unfortunately, most of these more "robust" region-based energy functionals are applicable to a much narrower class of imagery compared to typical edge-based energies due to stronger global assumptions about the underlying image data. Devising new implementation algorithms for active contours that attempt to capture more global minimizers of already proposed image-based energies would allow us to choose an energy that makes sense for a particular class of energy without concern over its sensitivity to local minima. Such implementations have been proposed for capturing global minima. However, sometimes the completely-global minimum is just as undesirable as a minimum that is too local. In this paper, we propose a novel, fast, and flexible dual front implementation of active contours, motivated by minimal path techniques and utilizing fast sweeping algorithms, which is easily manipulated to yield minima with variable "degrees" of localness and globalness. By simply adjusting the size of active regions, the ability to gracefully move from capturing minima that are more local (according to the initial placement of the active contour/surface) to minima that are more global allows this model to more easily obtain "desirable" minimizers (which often are neither the most local nor the most global). Experiments on various 2D and 3D images and comparisons with some active contour models and region-growing methods are also given to illustrate the properties of this model and its performance in a variety of segmentation applications.     

A variational approach to problems in calibration of multiple cameras

This paper addresses the problem of calibrating camera parameters using variational methods. One problem addressed is the severe lens distortion in low-cost cameras. For many computer vision algorithms aiming at reconstructing reliable representations of 3D scenes, the camera distortion effects will lead to inaccurate 3D reconstructions and geometrical measurements if not accounted for. A second problem is the color calibration problem caused by variations in camera responses that result in different color measurements and affects the algorithms that depend on these measurements. We also address the extrinsic camera calibration that estimates relative poses and orientations of multiple cameras in the system and the intrinsic camera calibration that estimates focal lengths and the skew parameters of the cameras. To address these calibration problems, we present multiview stereo techniques based on variational methods that utilize partial and ordinary differential equations. Our approach can also be considered as a coordinated refinement of camera calibration parameters. To reduce computational complexity of such algorithms, we utilize prior knowledge on the calibration object, making a piecewise smooth surface assumption, and evolve the pose, orientation, and scale parameters of such a 3D model object without requiring a 2D feature extraction from camera views. We derive the evolution equations for the distortion coefficients, the color calibration parameters, the extrinsic and intrinsic parameters of the cameras, and present experimental results.     

Integral invariants for shape matching

For shapes represented as closed planar contours, we introduce a class of functionals which are invariant with respect to the Euclidean group and which are obtained by performing integral operations. While such integral invariants enjoy some of the desirable properties of their differential counterparts, such as locality of computation (which allows matching under occlusions) and uniqueness of representation (asymptotically), they do not exhibit the noise sensitivity associated with differential quantities and, therefore, do not require presmoothing of the input shape. Our formulation allows the analysis of shapes at multiple scales. Based on integral invariants, we define a notion of distance between shapes. The proposed distance measure can be computed efficiently and allows warping the shape boundaries onto each other; its computation results in optimal point correspondence as an intermediate step. Numerical results on shape matching demonstrate that this framework can match shapes despite the deformation of subparts, missing parts and noise. As a quantitative analysis, we report matching scores for shape retrieval from a database.     

An Eulerian PDE approach for computing tissue thickness

We outline an Eulerian framework for computing the thickness of tissues between two simply connected boundaries that does not require landmark points or parameterizations of either boundary. Thickness is defined as the length of correspondence trajectories, which run from one tissue boundary to the other, and which follow a smooth vector field constructed in the region between the boundaries. A pair of partial differential equations (PDEs) that are guided by this vector field are then solved over this region, and the sum of their solutions yields the thickness of the tissue region. Unlike other approaches, this approach does not require explicit construction of any correspondence trajectories. An efficient, stable, and computationally fast solution to these PDEs is found by careful selection of finite differences according to an upwinding condition. The behavior and performance of our method is demonstrated on two simulations and two magnetic resonance imaging data sets in two and three dimensions. These experiments reveal very good performance and show strong potential for application in tissue thickness visualization and quantification.     

Stochastic differential equations and geometric flows

In previous years, curve evolution, applied to a single contour or to the level sets of an image via partial differential equations, has emerged as an important tool in image processing and computer vision. Curve evolution techniques have been utilized in problems such as image smoothing, segmentation, and shape analysis. We give a local stochastic interpretation of the basic curve smoothing equation, the so called geometric heat equation, and show that this evolution amounts to a tangential diffusion movement of the particles along the contour. Moreover, assuming that a priori information about the shapes of objects in an image is known, we present modifications of the geometric heat equation designed to preserve certain features in these shapes while removing noise. We also show how these new flows may be applied to smooth noisy curves without destroying their larger scale features, in contrast to the original geometric heat flow which tends to circularize any closed curve.     

A nonparametric statistical method for image segmentation using information theory and curve evolution.

In this paper, we present a new information-theoretic approach to image segmentation. We cast the segmentation problem as the maximization of the mutual information between the region labels and the image pixel intensities, subject to a constraint on the total length of the region boundaries. We assume that the probability densities associated with the image pixel intensities within each region are completely unknown a priori, and we formulate the problem based on nonparametric density estimates. Due to the nonparametric structure, our method does not require the image regions to have a particular type of probability distribution and does not require the extraction and use of a particular statistic. We solve the information-theoretic optimization problem by deriving the associated gradient flows and applying curve evolution techniques. We use level-set methods to implement the resulting evolution. The experimental results based on both synthetic and real images demonstrate that the proposed technique can solve a variety of challenging image segmentation problems. Futhermore, our method, which does not require any training, performs as good as methods based on training.     

A hybrid Eulerian-Lagrangian approach for thickness, correspondence, and gridding of annular tissues.

We present a novel approach to efficiently compute thickness, correspondence, and gridding of tissues between two simply connected boundaries. The solution of Laplace's equation within the tissue region provides a harmonic function whose gradient flow determines the correspondence trajectories going from one boundary to the other. The proposed method uses and expands upon two recently introduced techniques in order to compute thickness and correspondences based on these trajectories. Pairs of partial differential equations are efficiently computed within an Eulerian framework and combined with a Lagrangian approach so that correspondences trajectories are partially constructed when necessary. Examples are presented in order to compare the performance of this method with those of the pure Lagrangian and pure Eulerian approaches. Results show that the proposed technique takes advantage of both the speed of the Eulerian approach and the accuracy of the Lagrangian approach.     

Coarse-to-fine segmentation and tracking using Sobolev active contours

Recently proposed Sobolev active contours introduced a new paradigm for minimizing energies defined on curves by changing the traditional cost of perturbing a curve and thereby redefining their gradients. Sobolev active contours evolve more globally and are less attracted to certain intermediate local minima than traditional active contours, and it is based on a well-structured Riemannian metric. In this paper, we analyze Sobolev active contours using scale-space analysis in order to understand their evolution across different scales. This analysis shows an extremely important and useful behavior of Sobolev contours, namely, that they move successively from coarse to increasingly finer scale motions in a continuous manner. This property illustrates that one justification for using the Sobolev technique is for applications where coarse-scale deformations are preferred over fine scale deformations. Along with other properties to be discussed, the coarse-to-fine observation reveals that Sobolev active contours are, in particular, ideally suited for tracking algorithms that use active contours. We will also justify our assertion that the Sobolev metric should be used over the traditional metric for active contours in tracking problems by experimentally showing how a variety of active contour based tracking methods can be significantly improved merely by evolving the active contour according to the Sobolev method.