A novel iterative shape from focus algorithm based on combinatorial optimization

Shape from focus (SFF) is a technique to estimate the depth and 3D shape of an object from a sequence of images obtained at different focus settings. In this paper, the SFF is presented as a combinatorial optimization problem. The proposed algorithm tries to find the combination of pixel frames which produces maximum focus measure computed over pixels lying on those frames. To reduce the high computational complexity, a local search method is proposed. After the estimate of the initial depth map solution of an object, the neighborhood is defined, and an intermediate image volume is generated from the neighborhood. The updated depth map solution is found from the intermediate image volume. This update process of the depth map solution continues until the amount of improvement is negligible. The results of the proposed SFF algorithm have shown significant improvements in both the accuracy of the depth map estimation and the computational complexity, with respect to the existing SFF methods.     

Reliability measure for shape-from-focus

Shape-from-focus (SFF) is a passive technique widely used in image processing for obtaining depth-maps. This technique is attractive since it only requires a single monocular camera with focus control, thus avoiding correspondence problems typically found in stereo, as well as more expensive capturing devices. However, one of its main drawbacks is its poor performance when the change in the focus level is difficult to detect. Most research in SFF has focused on improving the accuracy of the depth estimation. Less attention has been paid to the problem of providing quality measures in order to predict the performance of SFF without prior knowledge of the recovered scene. This paper proposes a reliability measure aimed at assessing the quality of the depth-map obtained using SFF. The proposed reliability measure (the R-measure) analyzes the shape of the focus measure function and estimates the likelihood of obtaining an accurate depth estimation without any previous knowledge of the recovered scene. The proposed R-measure is then applied for determining the image regions where SFF will not perform correctly in order to discard them. Experiments with both synthetic and real scenes are presented.     

3D shape recovery from image focus using kernel regression in eigenspace

Shape from focus (SFF) is one of the optical passive methods for three dimensional (3D) shape recovery of an object from its two dimensional (2D) images. The focus measure plays important role in SFF algorithms. Mostly, conventional focus measures are based on gradient, so their performance is restricted under noisy conditions. Moreover, SFF methods also suffer from loss of focus information due to discreteness. This paper introduces a new SFF method based on principal component analysis (PCA) and kernel regression. The focus values are computed through PCA by considering a sequence of small 3D neighborhood for each object point. We apply unsupervised regression through Nadaraya and Watson Estimate (NWE) on depth values to get a refined 3D shape of the object. It reduces the effect of noise within a small surface area as well as approximates the accurate 3D shape by exploiting the depth dependencies in the neighborhood. Performance of the proposed scheme is investigated in the presence of different types of noises and textured areas. Experimental results demonstrate effectiveness of the proposed approach.     

Optimizing image focus for 3D shape recovery through genetic algorithm

Three-dimensional information of objects is advantageous and widely used in multimedia systems and applications. Shape form focus (SFF) is a passive optical technique that reconstructs 3D shape of an object using a sequence of images with varying focus settings. In this paper, we propose an optimization of the focus measure. First, Wiener filter is applied for noise reduction from the image sequence. At the second stage, genetic algorithm (GA) is applied for focus measure optimization. GA finds the maximum focus measurement under a fitness criterion. Finally, 3D shape of the object is determined by maximizing focus measure along the optical direction. The proposed method is tested with image sequences of simulated and real objects. The performance of the proposed technique is analyzed through statistical criteria such as root mean square error (RMSE) and correlation. Comparative analysis shows the effectiveness of the proposed method.     

Consideration of illumination effects and optimization of window size for accurate calculation of depth map for 3D shape recovery

Obtaining an accurate and precise depth map is the ultimate goal for 3D shape recovery. For depth map estimation, one of the most vital parts is the initial selection of the focus measure and processing the images with the selected focus measure. Although, many focus measures have been proposed in the literature but not much attention has been paid to the factors affecting those focus measures as well as the manner the images are processed with those focus measures. In this paper, for accurate calculation of depth map, we consider the effects of illumination on the depth map as well as the selection of the window size for application of the focus measures. The resulting depth map can further be used in techniques and algorithms leading to recovery of three-dimensional structure of the object which is required in many high-level vision applications. It is shown that the illumination effects can directly result in incorrect estimation of depth map if proper window size is not selected during focus measure computation. Further, it is shown that the images need some kind of pre-processing to enhance the dark regions and shadows in the image. For this purpose, an adaptive enhancement algorithm is proposed for pre-processing. In this paper, we prove that without such pre-processing for image enhancement and without the use of proper window size for the estimation of depth maps, it is not possible to obtain the accurate depth map.     

Lighting virtual objects in a single image via coarse scene understanding

Achieving convincing visual consistency between virtual objects and a real scene mainly relies on the lighting effects of virtual-real composition scenes. The problem becomes more challenging in lighting virtual objects in a single real image. Recently,scene understanding from a single image has made great progress. The estimated geometry,semantic labels and intrinsic components provide mostly coarse information,and are not accurate enough to re-render the whole scene. However,carefully integrating the estimated coarse information can lead to an estimate of the illumination parameters of the real scene. We present a novel method that uses the coarse information estimated by current scene understanding technology to estimate the parameters of a ray-based illumination model to light virtual objects in a real scene. Our key idea is to estimate the illumination via a sparse set of small 3D surfaces using normal and semantic constraints. The coarse shading image obtained by intrinsic image decomposition is considered as the irradiance of the selected small surfaces. The virtual objects are illuminated by the estimated illumination parameters. Experimental results show that our method can convincingly light virtual objects in a single real image,without any pre-recorded 3D geometry,reflectance,illumination acquisition equipment or imaging information of the image.     

Object detection,shape recovery,and 3D modelling by depth-encoded hough voting

Detecting objects, estimating their pose, and recovering their 3D shape are critical problems in many vision and robotics applications. This paper addresses the above needs using a two stages approach. In the first stage, we propose a new method called DEHV – Depth-Encoded Hough Voting. DEHV jointly detects objects, infers their categories, estimates their pose, and infers/decodes objects depth maps from either a single image (when no depth maps are available in testing) or a single image augmented with depth map (when this is available in testing). Inspired by the Hough voting scheme introduced in [1], DEHV incorporates depth information into the process of learning distributions of image features (patches) representing an object category. DEHV takes advantage of the interplay between the scale of each object patch in the image and its distance (depth) from the corresponding physical patch attached to the 3D object. Once the depth map is given, a full reconstruction is achieved in a second (3D modelling) stage, where modified or state-of-the-art 3D shape and texture completion techniques are used to recover the complete 3D model. Extensive quantitative and qualitative experimental analysis on existing datasets [2], [3], [4] and a newly proposed 3D table-top object category dataset shows that our DEHV scheme obtains competitive detection and pose estimation results. Finally, the quality of 3D modelling in terms of both shape completion and texture completion is evaluated on a 3D modelling dataset containing both in-door and out-door object categories. We demonstrate that our overall algorithm can obtain convincing 3D shape reconstruction from just one single uncalibrated image.     

Outlier deletion based improvement on the StOMP algorithm for sparse solution of large-scale underdetermined problems

StOMP algorithm is well suited to large-scale underdetermined applications in sparse vector estimations. It can reduce computation complexity and has some attractive asymptotical statistical properties.However,the estimation speed is at the cost of accuracy violation. This paper suggests an improvement on the StOMP algorithm that is more efficient in finding a sparse solution to the large-scale underdetermined problems. Also,compared with StOMP,this modified algorithm can not only more accurately estimate parameters for the distribution of matched filter coefficients,but also improve estimation accuracy for the sparse vector itself. Theoretical success boundary is provided based on a large-system limit for approximate recovery of sparse vector by modified algorithm,which validates that the modified algorithm is more efficient than StOMP. Actual computations with simulated data show that without significant increment in computation time,the proposed algorithm can greatly improve the estimation accuracy.     

Structure from Motion: Beyond the Epipolar Constraint

The classic approach to structure from motion entails a clear separation between motion estimation and structure estimation and between two-dimensional (2D) and three-dimensional (3D) information. For the recovery of the rigid transformation between different views only 2D image measurements are used. To have available enough information, most existing techniques are based on the intermediate computation of optical flow which, however, poses a problem at the locations of depth discontinuities. If we knew where depth discontinuities were, we could (using a multitude of approaches based on smoothness constraints) accurately estimate flow values for image patches corresponding to smooth scene patches; but to know the discontinuities requires solving the structure from motion problem first. This paper introduces a novel approach to structure from motion which addresses the processes of smoothing, 3D motion and structure estimation in a synergistic manner. It provides an algorithm for estimating the transformation between two views obtained by either a calibrated or uncalibrated camera. The results of the estimation are then utilized to perform a reconstruction of the scene from a short sequence of images.The technique is based on constraints on image derivatives which involve the 3D motion and shape of the scene, leading to a geometric and statistical estimation problem. The interaction between 3D motion and shape allows us to estimate the 3D motion while at the same time segmenting the scene. If we use a wrong 3D motion estimate to compute depth, we obtain a distorted version of the depth function. The distortion, however, is such that the worse the motion estimate, the more likely we are to obtain depth estimates that vary locally more than the correct ones. Since local variability of depth is due either to the existence of a discontinuity or to a wrong 3D motion estimate, being able to differentiate between these two cases provides the correct motion, which yields the least varying estimated depth as well as the image locations of scene discontinuities. We analyze the new constraints, show their relationship to the minimization of the epipolar constraint, and present experimental results using real image sequences that indicate the robustness of the method.     

基于高斯-柯西混合模型的单幅散焦图像深度恢复方法

单幅图像场景深度的获取一直是计算机视觉领域的一个难题。使用高斯分布函数或柯西分布函数近似点扩散函数模型(PSF),再根据图像边缘处散焦模糊量的大小与场景深度之间的关系估算出深度信息,是一种常用的方法。真实世界中图像模糊的缘由千变万化,高斯分布函数以及柯西分布函数并不一定是最佳的近似模型,并且传统的方法对于图像存在阴影、边缘不明显以及深度变化比较细微的区域的深度恢复结果不够准确。为了提取更为精确的深度信息,提出一种利用高斯-柯西混合模型近似PSF的方法;然后对散焦图像进行再模糊处理,得到两幅散焦程度不同的图像;再通过计算两幅散焦图像边缘处梯度的比值估算出图像边缘处的散焦模糊量,从而得到稀疏深度图;最后使用深度扩展法得到场景的全景深度图。通过大量真实图像的测试,说明新方法能够从单幅散焦图像中恢复出完整、可靠的深度信息,并且其结果优于目前常用的两种方法。     

基于改进暗通道和导向滤波的单幅图像去雾算法

针对单幅雾霾图像中包含的大面积天空或白色物体等区域暗通道先验失效和导向滤波去雾方法去雾不彻底的问题, 提出了一种基于改进暗通道和导向滤波的单幅图像去雾算法.首先基于暗通道引入了混合暗通道, 然后对混合暗通道进行映射处理, 从而得到大气耗散函数粗估计值; 利用导向滤波方法优化大气耗散函数粗估计值, 进而求解环境光值和初始传输图; 利用全变差正则化方法对初始传输图进行优化, 以解决其平滑性较差的问题.实验结果表明, 本文算法得到的去雾图像具有较高的清晰度, 对于大面积天空或白色物体区域也能实现良好的去雾效果.     

3D遮挡模型引导的光场图像深度获取

目的 光场相机可以通过单次曝光同时从多个视角采样单个场景,在深度估计领域具有独特优势。消除遮挡的影响是光场深度估计的难点之一。现有方法基于2D场景模型检测各视角遮挡状态,但是遮挡取决于所采样场景的3D立体模型,仅利用2D模型无法精确检测,不精确的遮挡检测结果将降低后续深度估计精度。针对这一问题,提出了3D遮挡模型引导的光场图像深度获取方法。方法 向2D模型中的不同物体之间添加前后景关系和深度差信息,得到场景的立体模型,之后在立体模型中根据光线的传输路径推断所有视角的遮挡情况并记录在遮挡图（occlusion map）中。在遮挡图引导下,在遮挡和非遮挡区域分别使用不同成本量进行深度估计。在遮挡区域,通过遮挡图屏蔽被遮挡视角,基于剩余视角的成像一致性计算深度;在非遮挡区域,根据该区域深度连续特性设计了新型离焦网格匹配成本量,相比传统成本量,该成本量能够感知更广范围的色彩纹理,以此估计更平滑的深度图。为了进一步提升深度估计的精度,根据遮挡检测和深度估计的依赖关系设计了基于最大期望（exception maximization,EM）算法的联合优化框架,在该框架下,遮挡图和深度图通过互相引导的方式相继提升彼此精度。结果 实验结果表明,本文方法在大部分实验场景中,对于单遮挡、多遮挡和低对比度遮挡在遮挡检测和深度估计方面均能达到最优结果。均方误差（mean square error,MSE）对比次优结果平均降低约19.75%。结论 针对遮挡场景的深度估计,通过理论分析和实验验证,表明3D遮挡模型相比传统2D遮挡模型在遮挡检测方面具有一定优越性,本文方法更适用于复杂遮挡场景的深度估计。     

基于联合卷积自编码网络的多聚焦图像融合方法

受相机景深的限制,单次成像无法对不同景深的内容全部清晰成像.多聚焦图像融合技术可以将不同聚焦层次的图像融合为一幅全聚焦的图像,其中如何得到准确的聚焦映射是多聚焦图像融合中的关键问题.对此,利用卷积神经网络强大的特征提取能力,设计具有公共分支和私有分支的联合卷积自编码网络以学习多源图像的特征,公共分支学习多幅图像之间的公共特征,每幅图像的私有分支学习该图像区别于其他图像的私有特征.基于私有特征计算图像的活动测度,得到图像聚焦区域映射,据此设计融合规则以融合两幅多聚焦图像,最终得到全聚焦的融合图像.在公开数据集上的对比实验结果显示:主观评测上,所提出的方法能够较好地融合聚焦区域,视觉效果自然清晰;客观指标上,该方法在多个评价指标上优于对比方法.     

基于场景对象注意与深度图融合的深度估计

现有单目深度估计算法主要从单幅图像中获取立体信息,存在相邻深度边缘细节模糊、明显的对象缺失问题。提出一种基于场景对象注意机制与加权深度图融合的单目深度估计算法。通过特征矩阵相乘的方式计算特征图任意两个位置之间的相似特征向量,以快速捕获长距离依赖关系,增强用于估计相似深度区域的上下文信息,从而解决自然场景中对象深度信息不完整的问题。基于多尺度特征图融合的优点,设计加权深度图融合模块,为具有不同深度信息的多视觉粒度的深度图赋予不同的权值并进行融合,融合后的深度图包含深度信息和丰富的场景对象信息,有效地解决细节模糊问题。在KITTI数据集上的实验结果表明,该算法对目标图像预估时σ<1.25的准确率为0.879,绝对相对误差、平方相对误差和对数均方根误差分别为0.110、0.765和0.185,预测得到的深度图具有更加完整的场景对象轮廓和精确的深度信息。     

Multiple Light Source Estimation in a Single Image

Many high‐level image processing tasks require an estimate of the positions, directions and relative intensities of the light sources that illuminated the depicted scene. In image‐based rendering, augmented reality and computer vision, such tasks include matching image contents based on illumination, inserting rendered synthetic objects into a natural image, intrinsic images, shape from shading and image relighting. Yet, accurate and robust illumination estimation, particularly from a single image, is a highly ill‐posed problem. In this paper, we present a new method to estimate the illumination in a single image as a combination of achromatic lights with their 3D directions and relative intensities. In contrast to previous methods, we base our azimuth angle estimation on curve fitting and recursive refinement of the number of light sources. Similarly, we present a novel surface normal approximation using an osculating arc for the estimation of zenith angles. By means of a new data set of ground‐truth data and images, we demonstrate that our approach produces more robust and accurate results, and show its versatility through novel applications such as image compositing and analysis.     

基于Shape from Shading的医学图像三维重建

通过对由医疗成像设备获取的二维灰度图像进行形状重建,得到的三维立体原型能帮助医学诊断人员确诊病情。介绍了Shape from Shading的实现原理和扫描电镜成像系统的简单构成,提出了一种基于线性逼近的用于解决SEM反射映射函数的实现方法,并将之应用于红血细胞的三维图像重构,得到的细胞图形非常接近其真实形状。     

Texture Mapping Real‐World Objects with Hydrographics

In the digital world, assigning arbitrary colors to an object is a simple operation thanks to texture mapping. However, in the real world, the same basic function of applying colors onto an object is far from trivial. One can specify colors during the fabrication process using a color 3D printer, but this does not apply to already existing objects. Paint and decals can be used during post‐fabrication, but they are challenging to apply on complex shapes. In this paper, we develop a method to enable texture mapping of physical objects, that is, we allow one to map an arbitrary color image onto a three‐dimensional object. Our approach builds upon hydrographics, a technique to transfer pigments printed on a sheet of polymer onto curved surfaces. We first describe a setup that makes the traditional water transfer printing process more accurate and consistent across prints. We then simulate the transfer process using a specialized parameterization to estimate the mapping between the planar color map and the object surface. We demonstrate that our approach enables the application of detailed color maps onto complex shapes such as 3D models of faces and anatomical casts.     

Simple Reconstruction of Tree Branches from a Single Range Image

3D modeling of trees in real environments is a challenge in computer graphics and computer vision, since the geometric shape and topological structure of trees are more complex than conventional artificial objects. In this paper, we present a multi-process approach that is mainly performed in 2D space to faithfully construct a 3D model of the trunk and main branches of a real tree from a single range image. The range image is first segmented into patches by jump edge detection based on depth discontinuity. Coarse skeleton points and initial radii are then computed from the contour of each patch. Axis directions are estimated using cylinder fitting in the neighborhood of each coarse skeleton point. With the help of axis directions, skeleton nodes and corresponding radii are computed. Finally, these skeleton nodes are hierarchically connected, and improper radii are modified based on plant knowledge. 3D models generated from single range images of real trees demonstrate the effectiveness of our method. The main contributions of this paper are simple reconstruction by virtue of image storage order of single scan and skeleton computation based on axis directions.     

Continuous Generalized Procrustes analysis

Two-dimensional shape models have been successfully applied to solve many problems in computer vision, such as object tracking, recognition, and segmentation. Typically, 2D shape models are learned from a discrete set of image landmarks (corresponding to projection of 3D points of an object), after applying Generalized Procustes Analysis (GPA) to remove 2D rigid transformations. However, the standard GPA process suffers from three main limitations. Firstly, the 2D training samples do not necessarily cover a uniform sampling of all the 3D transformations of an object. This can bias the estimate of the shape model. Secondly, it can be computationally expensive to learn the shape model by sampling 3D transformations. Thirdly, standard GPA methods use only one reference shape, which can might be insufficient to capture large structural variability of some objects.To address these drawbacks, this paper proposes continuous generalized Procrustes analysis (CGPA). CGPA uses a continuous formulation that avoids the need to generate 2D projections from all the rigid 3D transformations. It builds an efficient (in space and time) non-biased 2D shape model from a set of 3D model of objects. A major challenge in CGPA is the need to integrate over the space of 3D rotations, especially when the rotations are parameterized with Euler angles. To address this problem, we introduce the use of the Haar measure. Finally, we extended CGPA to incorporate several reference shapes. Experimental results on synthetic and real experiments show the benefits of CGPA over GPA.     

Estimating Facial Reflectance Properties Using Shape-from-Shading

In this paper we show how to estimate facial surface reflectance properties (a slice of the BRDF and the albedo) in conjunction with the facial shape from a single image. The key idea underpinning our approach is to iteratively interleave the two processes of estimating reflectance properties based on the current shape estimate and updating the shape estimate based on the current estimate of the reflectance function. For frontally illuminated faces, the reflectance properties can be described by a function of one variable which we estimate by fitting a curve to the scattered and noisy reflectance samples provided by the input image and estimated shape. For non-frontal illumination, we fit a smooth surface to the scattered 2D reflectance samples. We make use of a novel statistical face shape constraint which we term ‘model-based integrability’ which we use to regularise the shape estimation. We show that the method is capable of recovering accurate shape and reflectance information from single grayscale or colour images using both synthetic and real world imagery. We use the estimated reflectance measurements to render synthetic images of the face in varying poses. To synthesise images under novel illumination, we show how to fit a parametric model of reflectance to the estimated reflectance function.