A Variational Framework for Retinex

Retinex theory addresses the problem of separating the illumination from the reflectance in a given image and thereby compensating for non-uniform lighting. This is in general an ill-posed problem. In this paper we propose a variational model for the Retinex problem that unifies previous methods. Similar to previous algorithms, it assumes spatial smoothness of the illumination field. In addition, knowledge of the limited dynamic range of the reflectance is used as a constraint in the recovery process. A penalty term is also included, exploiting a-priori knowledge of the nature of the reflectance image. The proposed formulation adopts a Bayesian view point of the estimation problem, which leads to an algebraic regularization term, that contributes to better conditioning of the reconstruction problem.Based on the proposed variational model, we show that the illumination estimation problem can be formulated as a Quadratic Programming optimization problem. An efficient multi-resolution algorithm is proposed. It exploits the spatial correlation in the reflectance and illumination images. Applications of the algorithm to various color images yield promising results.     

Estimating Facial Reflectance Properties Using Shape-from-Shading

In this paper we show how to estimate facial surface reflectance properties (a slice of the BRDF and the albedo) in conjunction with the facial shape from a single image. The key idea underpinning our approach is to iteratively interleave the two processes of estimating reflectance properties based on the current shape estimate and updating the shape estimate based on the current estimate of the reflectance function. For frontally illuminated faces, the reflectance properties can be described by a function of one variable which we estimate by fitting a curve to the scattered and noisy reflectance samples provided by the input image and estimated shape. For non-frontal illumination, we fit a smooth surface to the scattered 2D reflectance samples. We make use of a novel statistical face shape constraint which we term ‘model-based integrability’ which we use to regularise the shape estimation. We show that the method is capable of recovering accurate shape and reflectance information from single grayscale or colour images using both synthetic and real world imagery. We use the estimated reflectance measurements to render synthetic images of the face in varying poses. To synthesise images under novel illumination, we show how to fit a parametric model of reflectance to the estimated reflectance function.     

Elastic shapes models for improving segmentation of object boundaries in synthetic aperture sonar images

We present a variational framework for naturally incorporating prior shape knowledge in guidance of active contours for boundary extraction in images. This framework is especially suitable for images collected outside the visible spectrum, where boundary estimation is difficult due to low contrast, low resolution, and presence of noise and clutter. Accordingly, we illustrate this approach using the segmentation of various objects in synthetic aperture sonar (SAS) images of underwater terrains. We use elastic shape analysis of planar curves in which the shapes are considered as elements of a quotient space of an infinite dimensional, non-linear Riemannian manifold. Using geodesic paths under the elastic Riemannian metric, one computes sample mean and covariances of training shapes in each classes and derives statistical models for capturing class-specific shape variability. These models are then used as shape priors in a variational setting to solve for Bayesian estimation of desired contours as follows. In traditional active contour models curves are driven towards minimum of an energy composed of image and smoothing terms. We introduce an additional shape term based on shape models of relevant shape classes. The minimization of this total energy, using iterated gradient-based updates of curves, leads to an improved segmentation of object boundaries. This is demonstrated using a number of shape classes in two large SAS image datasets.     

Neural computation approach for developing a 3D shapereconstruction model

The shape from shading problem refers to the well-known fact that most real images usually contain specular components and are affected by unknown reflectivity. In this paper, these limitations are addressed and a new neural-based 3D shape reconstruction model is proposed. The idea behind this approach is to optimize a proper reflectance model by learning the parameters of the proposed neural reflectance model. In order to do this, new neural-based reflectance models are presented. The feedforward neural network (FNN) model is able to generalize the diffuse term, while the RBF model is able to generalize the specular term. A hybrid structure of FNN-based and RBF-based models is also presented because most real surfaces are usually neither Lambertian models nor ideally specular models. Experimental results, including synthetic and real images, are presented to demonstrate the performance of our approach given different specular effects, unknown illuminate conditions, and different noise environments.     

A mathematical model for computer image tracking

A mathematical model using an operator formulation for a moving object in a sequence of images is presented. Time-varying translation and rotation operators are derived to describe the motion. A variational estimation algorithm is developed to track the dynamic parameters of the operators. The occlusion problem is alleviated by using a predictive Kalman filter to keep the tracking on course during severe occlusion. The tracking algorithm (variational estimation in conjunction with Kalman filter) is implemented to track moving objects with occasional occlusion in computer-simulated binary images.     

Combining medium and coarse spatial resolution satellite data to improve the estimation of sub-pixel NDVI time series

The potential of multitemporal coarse spatial resolution remotely sensed images for vegetation monitoring is reduced in fragmented landscapes, where most of the pixels are composed of a mixture of different surfaces. Several approaches have been proposed for the estimation of reflectance or NDVI values of the different land-cover classes included in a low resolution mixed pixel. In this paper, we propose a novel approach for the estimation of sub-pixel NDVI values from multitemporal coarse resolution satellite data. Sub-pixel NDVIs for the different land-cover classes are calculated by solving a weighted linear system of equations for each pixel of a coarse resolution image, exploiting information about within-pixel fractional cover derived from a high resolution land-use map. The weights assigned to the different pixels of the image for the estimation of sub-pixel NDVIs of a target pixel i are calculated taking into account both the spatial distance between each pixel and the target and their spectral dissimilarity estimated on medium-resolution remote-sensing images acquired in different periods of the year. The algorithm was applied to daily and 16-day composite MODIS NDVI images, using Landsat-5 TM images for calculation of weights and accuracy evaluation.Results showed that application of the algorithm provided good estimates of sub-pixel NDVIs even for poorly represented land-cover classes (i.e., with a low total cover in the test area). No significant accuracy differences were found between results obtained on daily and composite MODIS images. The main advantage of the proposed technique with respect to others is that the inclusion of the spectral term in weight calculation allows an accurate estimate of sub-pixel NDVI time series even for land-cover classes characterized by large and rapid spatial variations in their spectral properties.     

Learning parametric specular reflectance model by radial basis function network

For the shape from shading problem, it is known that most real images usually contain specular components and are affected by unknown reflectivity. In the paper, these limitations are addressed and a neural-based specular reflectance model is proposed. The idea of this method is to optimize a proper specular model by learning the parameters of a radial basis function network and to recover the object shape by the variational approach with this resulting model. The obtained results are very encouraging and the performance is demonstrated by using the synthetic and real images in the case of different specular effects and noisy environments.     

Stereoscopic Scene Flow Computation for 3D Motion Understanding

Building upon recent developments in optical flow and stereo matching estimation, we propose a variational framework for the estimation of stereoscopic scene flow, i.e., the motion of points in the three-dimensional world from stereo image sequences. The proposed algorithm takes into account image pairs from two consecutive times and computes both depth and a 3D motion vector associated with each point in the image. In contrast to previous works, we partially decouple the depth estimation from the motion estimation, which has many practical advantages. The variational formulation is quite flexible and can handle both sparse or dense disparity maps. The proposed method is very efficient; with the depth map being computed on an FPGA, and the scene flow computed on the GPU, the proposed algorithm runs at frame rates of 20 frames per second on QVGA images (320×240 pixels). Furthermore, we present solutions to two important problems in scene flow estimation: violations of intensity consistency between input images, and the uncertainty measures for the scene flow result.     

本征图像分解的可变尺度局部分析与集成方法

针对自然图像与磁共振图像,提出本征图像分解的统一的数学模型与算法,解决这两类图像中的重要问题:1)自然图像的光照和反射图像的估计,2)磁共振图像中的偏移场估计与分割.文中数学模型只需要一个基本的假设,即观察到的图像可近似为两个具有不同特性的本征图像的乘积:一个光滑的图像,简称为S-图像;一个近似为分片常量的图像,简称为L-图像.为了充分利用本征图像的特性,提出可变尺度局部分析与集成的方法.由于S-图像的光滑性,使用低阶泰勒展开式或更一般的光滑基函数的线性组合以局部逼近.得到的局部光滑逼近可通过整个感兴趣区域(ROI)的局部区域覆盖及其对应的单位分解扩展成整个ROI上的光滑图像,同时得到图像分割结果和L-图像.实验表明,文中方法对图像的两个本征因子的假设较弱,适用于更广泛的图像.目前方法已在磁共振图像及自然图像中进行测试,得到较优结果.     

Reflectance sharing: predicting appearance from a sparse set of images of a known shape

Three-dimensional appearance models consisting of spatially varying reflectance functions defined on a known shape can be used in analysis-by-synthesis approaches to a number of visual tasks. The construction of these models requires the measurement of reflectance, and the problem of recovering spatially varying reflectance from images of known shape has drawn considerable interest. To date, existing methods rely on either: 1) low-dimensional (e.g., parametric) reflectance models, or 2) large data sets involving thousands of images (or more) per object. Appearance models based on the former have limited accuracy and generality since they require the selection of a specific reflectance model a priori, and while approaches based on the latter may be suitable for certain applications, they are generally too costly and cumbersome to be used for image analysis. We present an alternative approach that seeks to combine the benefits of existing methods by enabling the estimation of a nonparametric spatially varying reflectance function from a small number of images. We frame the problem as scattered-data interpolation in a mixed spatial and angular domain, and we present a theory demonstrating that the angular accuracy of a recovered reflectance function can be increased in exchange for a decrease in its spatial resolution. We also present a practical solution to this interpolation problem using a new representation of reflectance based on radial basis functions. This representation is evaluated experimentally by testing its ability to predict appearance under novel view and lighting conditions. Our results suggest that since reflectance typically varies slowly from point to point over much of an object's surface, we can often obtain a nonparametric reflectance function from a sparse set of images. In fact, in some cases, we can obtain reasonable results in the limiting case of only a single input image.     

Visual learning and recognition of 3-d objects from appearance

The problem of automatically learning object models for recognition and pose estimation is addressed. In contrast to the traditional approach, the recognition problem is formulated as one of matching appearance rather than shape. The appearance of an object in a two-dimensional image depends on its shape, reflectance properties, pose in the scene, and the illumination conditions. While shape and reflectance are intrinsic properties and constant for a rigid object, pose and illumination vary from scene to scene. A compact representation of object appearance is proposed that is parametrized by pose and illumination. For each object of interest, a large set of images is obtained by automatically varying pose and illumination. This image set is compressed to obtain a low-dimensional subspace, called the eigenspace, in which the object is represented as a manifold. Given an unknown input image, the recognition system projects the image to eigenspace. The object is recognized based on the manifold it lies on. The exact position of the projection on the manifold determines the object's pose in the image.A variety of experiments are conducted using objects with complex appearance characteristics. The performance of the recognition and pose estimation algorithms is studied using over a thousand input images of sample objects. Sensitivity of recognition to the number of eigenspace dimensions and the number of learning samples is analyzed. For the objects used, appearance representation in eigenspaces with less than 20 dimensions produces accurate recognition results with an average pose estimation error of about 1.0 degree. A near real-time recognition system with 20 complex objects in the database has been developed. The paper is concluded with a discussion on various issues related to the proposed learning and recognition methodology.     

基于支持向量回归的光谱反射率重建方法

提出了一种基于支持向量回归和小框架核的光谱反射率重建方法。光谱反射率重建是光学研究的一个重要问题,其目的是通过各种成像设备所获取的与设备相关的RGB三色值重建出物体本身固有的与设备和光照都无关的光谱反射率。回归方法已经在这一领域取得了广泛应用,如基于多项式模型的正则化最小二乘方法、基于核的正则化最小二乘方法等。提出了一种新的光谱反射率重建方法,这种方法采用了一种可以减弱样本不规则噪音影响的小框架核函数,并将其用于支持向量回归来重建光谱反射率函数。实验表明,新方法可以提高光谱反射率重建的精度和稳定性。     

A non-stationary image prior combination in super-resolution

A new Bayesian Super-Resolution (SR) image registration and reconstruction method is proposed. The new method utilizes a prior distribution based on a general combination of spatially adaptive, or non-stationary, image filters, which includes an adaptive local strength parameter able to preserve both image edges and textures. With the application of variational techniques, the proposed method allows for the automatic estimation of all problem unknowns. An experimental comparison between state of the art methods and the proposed SR approach has been performed on both synthetic and real images.     

An Adaptive Gaussian Mixture Model for Non-rigid Image Registration

This paper presents a new model based on statistical and variational methods for non-rigid image registration. It can be viewed as an improvement of the intensity-based model whose dissimilarity term is based on minimization of the so-called sum of squared difference(SSD). In the proposed model, it is assumed that the residue of two images can be described as a mixture of Gaussian distributions. Then we incorporate the features of variational regularization methods and expectation-maximization(EM) algorithm, and propose the new model. The novelty is the introduction of two weighting functions and some control parameters in dissimilarity term. The weighting functions could identify low and high contrast objects of the residue automatically and effectively, and the control parameters help to improve the robustness of the model to the choice of regularization parameters. By the introduced parameters and weighting functions, the algorithm could locally adjust the behavior of deformation in different contrast regions. Numerical experimental results of 2D synthetic and 3D MR brain images demonstrate the efficiency and accuracy of the proposed approach compared with other methods.     

基于变分的CT图像环形伪影校正

有效去除CT图像中环形伪影是医学图像处理领域的一个重要研究方向,现有的方法在去除环形伪影的同时,对CT图像的边缘及细节保留存在困难和挑战.本文采用变分优化的思想,将环形伪影的去除问题建模为一个能量最小化问题,来缓解保持图像信息和去除伪影之间的矛盾,提出了一种后处理的伪影校正算法.根据环形伪影产生机理和特性表现构造有针对性的变分模型,一是从环形伪影的几何特性入手,设计更为合理的梯度保真形式,增强模型对图像细节信息的保护;二是从环形伪影的边缘特性入手,构建具有伪影辨识能力的相对全变分正则项,降低模型对图像结构性信息的影响.基于构造的变分模型,采用高效的优化求解算法,实现环形伪影的有效去除.对比实验表明,无论在视觉观察还是定量分析方面,本文算法均体现出了较好的性能.     

Color and illuminant voting

A geometric-vision approach to color constancy and illuminant estimation is presented in this paper. We show a general framework, based on ideas from the generalized probabilistic Hough transform, to estimate the illuminant and reflectance of natural images. Each image pixel “votes” for possible illuminants and the estimation is based on cumulative votes. The framework is natural for the introduction of physical constraints in the color constancy problem. We show the relationship of this work to previous algorithms for color constancy and present examples     

Joint Estimation of Shape and Reflectance using Multiple Images with Known Illumination Conditions

We propose a generative model based method for recovering both the shape and the reflectance of the surface(s) of a scene from multiple images, assuming that illumination conditions and cameras calibration are known in advance. Based on a variational framework and via gradient descents, the algorithm minimizes simultaneously and consistently a global cost functional with respect to both shape and reflectance. The motivations for our approach are threefold. (1) Contrary to previous works which mainly consider specific individual scenarios, our method applies indiscriminately to a number of classical scenarios; in particular it works for classical stereovision, multiview photometric stereo and multiview shape from shading. It works with changing as well as static illumination. (2) Our approach naturally combines stereo, silhouette and shading cues in a single framework. (3) Moreover, unlike most previous methods dealing with only Lambertian surfaces, the proposed method considers general dichromatic surfaces. We verify the method using various synthetic and real data sets.     

Intrinsic Image Decomposition Using Multi‐Scale Measurements and Sparsity

Automatic decomposition of intrinsic images, especially for complex real‐world images, is a challenging under‐constrained problem. Thus, we propose a new algorithm that generates and combines multi‐scale properties of chromaticity differences and intensity contrast. The key observation is that the estimation of image reflectance, which is neither a pixel‐based nor a region‐based property, can be improved by using multi‐scale measurements of image content. The new algorithm iteratively coarsens a graph reflecting the reflectance similarity between neighbouring pixels. Then multi‐scale reflectance properties are aggregated so that the graph reflects the reflectance property at different scales. This is followed by a L₀ sparse regularization on the whole reflectance image, which enforces the variation in reflectance images to be high‐frequency and sparse. We formulate this problem through energy minimization which can be solved efficiently within a few iterations. The effectiveness of the new algorithm is tested with the Massachusetts Institute of Technology (MIT) dataset, the Intrinsic Images in the Wild (IIW) dataset, and various natural images.     

Reflectance Parameter Estimation and its Application to Surface Inspection

In this paper, we present a reflectance parameter estimation technique by using range and brightness and its relation, i.e. reflectance function. Because the reflectance function is quite complex and nonlinear, the parameter estimation is not straightforward. Therefore, we choose a coarse-to-fine approach to estimate the reflectance parameters. In the coarse step, the surface toughness is coarsely estimated by applying the partial linear method to the simplified Torrance-Sparrow reflectance model. Then the genetic algorithm is applied to the Wolff's reflectance model for more accurate estimation. In order to extend the dynamic range of CCD of laser finder, in this paper, we introduce the pseudo-brightness. The proposed reflectance parameter estimation algorithm is tested on the synthesized and real data. The results show that the estimated parameter using the synthesized data is very accurate. We also apply the proposed algorithm to inspect the flaws on shiny surfaces, which would be a promising method to discriminate between a wide range of surfaces.