Improved estimation of defocus blur and spatial shifts in spatial domain: a homotopy-based approach

This paper presents a homotopy-based algorithm for the recovery of depth cues in the spatial domain. The algorithm specifically deals with defocus blur and spatial shifts, that is 2D motion, stereo disparities and/or zooming disparities. These cues are estimated from two images of the same scene acquired by a camera evolving in time and/or space. We show that they can be simultaneously computed by resolving a system of equations using a homotopy method. The proposed algorithm is tested using synthetic and real images. The results confirm that the use of a homotopy method leads to a dense and accurate estimation of depth cues. This approach has been integrated into an application for relief estimation from remotely sensed images.     

Depth from Defocus Estimation in Spatial Domain

This paper presents an algorithm for a dense computation of the difference in blur between two images. The two images are acquired by varying the intrinsic parameters of the camera. The image formation system is assumed to be passive. Estimation of depth from the blur difference is straightforward. The algorithm is based on a local image decomposition technique using the Hermite polynomial basis. We show that any coefficient of the Hermite polynomial computed using the more blurred image is a function of the partial derivatives of the other image and the blur difference. Hence, the blur difference is computed by resolving a system of equations. The resulting estimation is dense and involves simple local operations carried out in the spatial domain. The mathematical developments underlying estimation of the blur in both 1D and 2D images are presented. The behavior of the algorithm is studied for constant images, step edges, line edges, and junctions. The selection of its parameters is discussed. The proposed algorithm is tested using synthetic and real images. The results obtained are accurate and dense. They are compared with those obtained using an existing algorithm.     

利用运动线索的单目深度测量

目的 传统的单目视觉深度测量方法具有设备简单、价格低廉、运算速度快等优点,但需要对相机进行复杂标定,并且只在特定的场景条件下适用。为此,提出基于运动视差线索的物体深度测量方法,从图像中提取特征点,利用特征点与图像深度的关系得到测量结果。方法 对两幅图像进行分割,获取被测量物体所在区域;然后采用本文提出的改进的尺度不变特征变换SIFT（scale-invariant feature transtorm）算法对两幅图像进行匹配,结合图像匹配和图像分割的结果获取被测量物体的匹配结果;用Graham扫描法求得匹配后特征点的凸包,获取凸包上最长线段的长度;最后利用相机成像的基本原理和三角几何知识求出图像深度。结果 实验结果表明,本文方法在测量精度和实时性两方面都有所提升。当图像中的物体不被遮挡时,实际距离与测量距离之间的误差为2.60%,测量距离的时间消耗为1.577 s;当图像中的物体存在部分遮挡时,该方法也获得了较好的测量结果,实际距离与测量距离之间的误差为3.19%,测量距离所需时间为1.689 s。结论 利用两幅图像上的特征点来估计图像深度,对图像中物体存在部分遮挡情况具有良好的鲁棒性,同时避免了复杂的摄像机标定过程,具有实际应用价值。     

结合同场景立体图对的高质量深度图像重建

目的 越来越多的应用依赖于对场景深度图像准确且快速的观测和分析,如机器人导航以及在电影和游戏中对虚拟场景的设计建模等.飞行时间深度相机等直接的深度测量设备可以实时的获取场景的深度图像,但是由于硬件条件的限制,采集的深度图像分辨率比较低,无法满足实际应用的需要.通过立体匹配算法对左右立体图对之间进行匹配获得视差从而得到深度图像是计算机视觉的一种经典方法,但是由于左右图像之间遮挡以及无纹理区域的影响,立体匹配算法在这些区域无法匹配得到正确的视差,导致立体匹配算法在实际应用中存在一定的局限性.方法 结合飞行时间深度相机等直接的深度测量设备和立体匹配算法的优势,提出一种新的深度图像重建方法.首先结合直接的深度测量设备采集的深度图像来构造自适应局部匹配权值,对左右图像之间的局部窗立体匹配过程进行约束,得到基于立体匹配算法的深度图像;然后基于左右检测原理将采集到的深度图像和匹配得到的深度图像进行有效融合;接着提出一种局部权值滤波算法,来进一步提高深度图像的重建质量.结果 实验结果表明,无论在客观指标还是视觉效果上,本文提出的深度图像重建算法较其他立体匹配算法可以得到更好的结果.其中错误率比较实验表明,本文算法较传统的立体匹配算法在深度重建错误率上可以提升10％左右.峰值信噪比实验结果表明,本文算法在峰值信噪比上可以得到10 dB左右的提升.结论 提出的深度图像重建方法通过结合高分辨率左右立体图对和初始的低分辨率深度图像,可以有效地重建高质量高分辨率的深度图像.     

定位图像匹配尺度与区域的摄像机位姿实时跟踪

目的 提出一种定位图像匹配尺度及区域的有效算法,通过实现当前屏幕图像特征点与模板图像中对应尺度下部分区域中的特征点匹配,实现摄像机对模板图像的实时跟踪,解决3维跟踪算法中匹配精度与效率问题。方法 在预处理阶段,算法对模板图像建立多尺度表示,各尺度下的图像进行区域划分,在每个区域内采用ORB（oriented FAST and rotated BRIEF）方法提取特征点并生成描述子,由此构建图像特征点的分级分区管理模式。在实时跟踪阶段,对于当前摄像机获得的图像,首先定位该图像所对应的尺度范围,在相应尺度范围内确定与当前图像重叠度大的图像区域,然后将当前图像与模板图像对应的尺度与区域中的特征点集进行匹配,最后根据匹配点对计算摄像机的位姿。结果 利用公开图像数据库（stanford mobile visual search dataset）中不同分辨率的模板图像及更多图像进行实验,结果表明,本文算法性能稳定,配准误差在1个像素左右;系统运行帧率总体稳定在2030 帧/s。结论 与多种经典算法对比,新方法能够更好地定位图像匹配尺度与区域,采用这种局部特征点匹配的方法在配准精度与计算效率方面比现有方法有明显提升,并且当模板图像分辨率较高时性能更好,特别适合移动增强现实应用。     

A LSS-based registration of stereo thermal–visible videos of multiple people using belief propagation

In this paper, we propose a novel stereo method for registering foreground objects in a pair of thermal and visible videos of close-range scenes. In our stereo matching, we use Local Self-Similarity (LSS) as similarity metric between thermal and visible images. In order to accurately assign disparities to depth discontinuities and occluded Region Of Interest (ROI), we have integrated color and motion cues as soft constraints in an energy minimization framework. The optimal disparity map is approximated for image ROIs using a Belief Propagation (BP) algorithm. We tested our registration method on several challenging close-range indoor video frames of multiple people at different depths, with different clothing, and different poses. We show that our global optimization algorithm significantly outperforms the existing state-of-the art method, especially for disparity assignment of occluded people at different depth in close-range surveillance scenes and for relatively large camera baseline.     

面向RGBD图像的标记分水岭分割

目的 针对分水岭分割算法中存在的过分割现象及现有基于RGB图像分割方法的局限,提出了一种基于RGB图像和深度图像(RGBD)的标记分水岭分割算法。方法 本文使用物体表面几何信息来辅助进行图像分割,定义了一种深度梯度算子和一种法向量梯度算子来衡量物体表面几何信息的变化。通过生成深度梯度图像和法向量梯度图像,与彩色梯度图像进行融合,实现标记图像的提取。在此基础上,使用极小值标定技术对彩色梯度图像进行修正,然后使用分水岭算法进行图像分割。结果 在纽约大学提供的NYU2数据集上进行实验,本文算法有效抑制了过分割现象,将分割区域从上千个降至数十个,且获得了与人工标定的分割结果更接近的分割效果,分割的准确率也比只使用彩色图像进行分割提高了10%以上。结论 本文算法普遍适用于RGBD图像的分割问题,该算法加入了物体表面几何信息的使用,提高了分割的准确率,且对颜色纹理相似的区域获得了较好的分割结果。     

Rainbow Flash Camera: Depth Edge Extraction Using Complementary Colors

We present a novel color multiplexing method for extracting depth edges in a scene. It has been shown that casting shadows from different light positions provides a simple yet robust cue for extracting depth edges. Instead of flashing a single light source at a time as in conventional methods, our method flashes all light sources simultaneously to reduce the number of captured images. We use a ring light source around a camera and arrange colors on the ring such that the colors form a hue circle. Since complementary colors are arranged at any position and its antipole on the ring, shadow regions where a half of the hue circle is occluded are colorized according to the orientations of depth edges, while non-shadow regions where all the hues are mixed have a neutral color in the captured image. Thus the colored shadows in the single image directly provide depth edges and their orientations in an ideal situation. We present an algorithm that extracts depth edges from a single image by analyzing the colored shadows. We also present a more robust depth edge extraction algorithm using an additional image captured by rotating the hue circle with \(180^\circ \) to compensate for scene textures and ambient lights. We compare our approach with conventional methods for various scenes using a camera prototype consisting of a standard camera and 8 color LEDs. We also demonstrate a bin-picking system using the camera prototype mounted on a robot arm.     

Non-uniform image deblurring using an optical computing system

Removing non-uniform blur caused by camera shaking is troublesome because of its high computational cost. We analyze the efficiency bottlenecks of a non-uniform deblurring algorithm and propose an efficient optical computation deblurring framework that implements the time-consuming and repeatedly required modules, i.e., non-uniform convolution and perspective warping, by light transportation. Specifically, the non-uniform convolution and perspective warping are optically computed by a hybrid system that is composed of an off-the-shelf projector and a camera mounted on a programmable motion platform. Benefitting from the high speed and parallelism of optical computation, our system has the potential to accelerate existing non-uniform motion deblurring algorithms significantly. To validate the effectiveness of the proposed approach, we also develop a prototype system that is incorporated into an iterative deblurring framework to effectively address the image blur of planar scenes that is caused by 3D camera rotation around the x-, y- and z-axes. The results show that the proposed approach has a high efficiency while obtaining a promising accuracy and has a high generalizability to more complex camera motions.     

结合图像识别的移动增强现实系统设计与应用

目的 在移动互联网时代下,移动增强现实应用得到越来越快的发展。然而户外场景中存在许多相似结构的建筑,且手机的存储和计算能力有限,因此应用多集中于室内小范围环境,对于室外大规模复杂场景的适应性较弱。对此,建立一套基于云端图像识别的移动增强现实系统。方法 为解决相似特征的误匹配问题,算法中将重力信息加入到SURF和BRISK特征描述中去,构建Gravity-SURF和Gravity-BRISK特征描述。云端系统对增强信息进行有效管理,采用基于Gravity-SURF特征的VLAD方法对大规模图像进行识别;在智能终端上的应用中呈现识别图像的增强信息,并利用识别图像的Gravity-BRISK特征和光流结合的方法对相机进行跟踪,采用Unity3D渲染引擎实时绘制3维模型。结果 在包含重力信息的4 000幅户外图像的数据库中进行实验。采用结合重力信息的特征描述算法,能够增强具有相似特征的描述符的区分性,并提高匹配正确率。图像识别算法的识别率能达到88%以上,识别时间在420 ms左右;光流跟踪的RMS误差小于1.2像素,帧率能达到23 帧/s。结论 本文针对室外大规模复杂场景建立的基于图像识别的移动增强现实系统,能方便对不同应用的增强现实数据进行管理。系统被应用到谷歌眼镜和新闻领域上,不局限于单一的应用领域。结果表明,识别算法和跟踪注册算法能够满足系统的精度和实时性要求。     

Implicit and explicit camera calibration: theory and experiments

By implicit camera calibration, we mean the process of calibrating a camera without explicitly computing its physical parameters. Implicit calibration can be used for both three-dimensional (3-D) measurement and generation of image coordinates. In this paper, we present a new implicit model based on the generalized projective mappings between the image plane and two calibration planes. The back-projection and projection processes are modelled separately to ease the computation of distorted image coordinates from known world points. A set of constraints of perspectivity is derived to relate the transformation parameters of the two calibration planes. Under the assumption of the radial distortion model, we present a computationally efficient method for explicitly correcting the distortion of image coordinates in frame buffer without involving the computation of camera position and orientation. By combining with any linear calibration techniques, this method makes explicit the camera physical parameters. Extensive experimental comparison of our methods with the classic photogrammetric method and Tsai's (1986) method in the aspects of 3-D measurement (both absolute and relative errors), the prediction of image coordinates, and the effect of the number of calibration points, is made using real images from 15 different depth values     

Accurate and speedy computation of image Legendre moments for computer vision applications

A novel algorithm that permits the fast and accurate computation of the Legendre image moments is introduced in this paper. The proposed algorithm is based on the block representation of an image and on a new image representation scheme, the Image Slice Representation (ISR) method. The ISR method decomposes a gray-scale image as an expansion of several two-level images of different intensities (slices) and thus enables the partial application of the well-known Image Block Representation (IBR) algorithm to each image component. Moreover, using the resulted set of image blocks, the Legendre moments’ computation can be accelerated through appropriate computation schemes. Extensive experiments prove that the proposed methodology exhibits high efficiency in calculating Legendre moments on gray-scale, but furthermore on binary images. The newly introduced algorithm is suitable for the computation of the Legendre moments for pattern recognition and computer vision applications, where the images consist of objects presented in a scene.     

Image Sequence Stabilization in Real Time

This paper describes a method ofstabilizingimage sequences obtained by a camera carried by a ground vehicle. The motion of the vehicle can usually be regarded as consisting of a desired smooth motion combined with an undesired non-smooth motion that includes impulsive or high-frequency components. The goal of the stabilization process is to correct the images so that they are approximately the same as the images that would have been obtained if the motion of the vehicle had been smooth.We analyse the smooth and non-smooth motions of a ground vehicle and show that only the rotational components of the non-smooth motion have significant perturbing effects on the images. We show how to identify image points at which rotational image flow is dominant, and how to use such points to estimate the vehicle's rotation. Finally, we describe an algorithm that fits smooth (ideally, piecewise constant) rotational motions to these estimates; the residual rotational motion can then be used to correct the images. We have obtained good results for several image sequences obtained from a camera carried by a ground vehicle moving across bumpy terrain.     

A Variational Framework for Structure from Motion in Omnidirectional Image Sequences

We address the problem of depth and ego-motion estimation from omnidirectional images. We propose a correspondence-free structure-from-motion problem for sequences of images mapped on the 2-sphere. A novel graph-based variational framework is first proposed for depth estimation between pairs of images. The estimation is cast as a TV-L1 optimization problem that is solved by a fast graph-based algorithm. The ego-motion is then estimated directly from the depth information without explicit computation of the optical flow. Both problems are finally addressed together in an iterative algorithm that alternates between depth and ego-motion estimation for fast computation of 3D information from motion in image sequences. Experimental results demonstrate the effective performance of the proposed algorithm for 3D reconstruction from synthetic and natural omnidirectional images.     

Efficient and accurate computation of geometric moments on gray-scale images

A novel algorithm that permits the fast and accurate computation of geometric moments on gray-scale images is presented in this paper. The proposed algorithm constitutes an extension of the IBR algorithm, introduced in the past, which was applicable only for binary images. A new image representation scheme, the ISR (intensity slice representation), which represents a gray-scale image as an expansion of several two-level images of different intensity values, enables the partially application of the IBR algorithm to each image component. Moreover, using the resulted set of image blocks, the geometric moments’ computation can be accelerated through appropriate computation schemes.     

Efficient Multi‐image Correspondences for On‐line Light Field Video Processing

Light field videos express the entire visual information of an animated scene, but their shear size typically makes capture, processing and display an off‐line process, i. e., time between initial capture and final display is far from real‐time. In this paper we propose a solution for one of the key bottlenecks in such a processing pipeline, which is a reliable depth reconstruction possibly for many views. This is enabled by a novel correspondence algorithm converting the video streams from a sparse array of off‐the‐shelf cameras into an array of animated depth maps. The algorithm is based on a generalization of the classic multi‐resolution Lucas‐Kanade correspondence algorithm from a pair of images to an entire array. Special inter‐image confidence consolidation allows recovery from unreliable matching in some locations and some views. It can be implemented efficiently in massively parallel hardware, allowing for interactive computations. The resulting depth quality as well as the computation performance compares favorably to other state‐of‐the art light field‐to‐depth approaches, as well as stereo matching techniques. Another outcome of this work is a data set of light field videos that are captured with multiple variants of sparse camera arrays.     

An improved image analogy method based on adaptive CUDA-accelerated neighborhood matching framework

The image analogy framework is especially useful to synthesize appealing images for non-homogeneous input and gives users creative control over the synthesized results. However, the traditional framework did not adaptively employ the searching strategy based on neighborhood’s different textural contents. Besides, the synthesis speed is slow due to intensive computation involved in neighborhood matching. In this paper we present a CUDA-based neighborhood matching algorithm for image analogy. Our algorithm adaptively applies the global search of the exact L ₂ nearest neighbor and k-coherence search strategies during synthesis according to different textural features of images, which is especially usefully for non-homogeneous textures. To consistently implement the above two search strategies on GPU, we adopt the fast k nearest neighbor searching algorithm based on CUDA. Such an acceleration greatly reduces the time of the pre-process of k-coherence search and the synthesis procedure of the global search, which makes possible the adjustment of important synthesis parameters. We further adopt synthesis magnification to get the final high-resolution synthesis image for running efficiency. Experimental results show that our algorithm is suitable for various applications of the image analogy framework and takes full advantage of GPU’s parallel processing capability to improve synthesis speed and get satisfactory synthesis results.     

Realtime Object-aware Monocular Depth Estimation in Onboard Systems

This paper proposes the object depth estimation in real-time, using only a monocular camera in an onboard computer with a low-cost GPU. Our algorithm estimates scene depth from a sparse feature-based visual odometry algorithm and detects/tracks objects’ bounding box by utilizing the existing object detection algorithm in parallel. Both algorithms share their results, i.e., feature, motion, and bounding boxes, to handle static and dynamic objects in the scene. We validate the scene depth accuracy of sparse features with KITTI and its ground-truth depth map made from LiDAR observations quantitatively, and the depth of detected object with the Hyundai driving datasets and satellite maps qualitatively. We compare the depth map of our algorithm with the result of (un-) supervised monocular depth estimation algorithms. The validation shows that our performance is comparable to that of monocular depth estimation algorithms which train depth indirectly (or directly) from stereo image pairs (or depth image), and better than that of algorithms trained with monocular images only, in terms of the error and the accuracy. Also, we confirm that our computational load is much lighter than the learning-based methods, while showing comparable performance.

     

Multi-scale fidelity measure for image fusion quality assessment

Image fusion is considered an effective enhancing methodology widely included in high-quality imaging systems. Nevertheless, like other enhancing techniques, output quality assessment is made within small sample subjective evaluation studies which are very limited in predicting the human-perceived quality of general image fusion outputs. Simple, blind, universal and perceptual-like methods for assessing composite image quality are still a challenge, partially solved only in particular applications. In this paper, we propose a fidelity measure, called MS-Q_W with two major characteristics related to natural image statistics framework: A multi-scale computation and a structural similarity score. In our experiments, we correlate the scores of our measure with subjective ratings and state of the art measures included in the 2015 Waterloo IVC multi-exposure fusion (MEF) image database. We also use the measure to rank correctly the classical general fusion methods included in the Image Fusion Toolbox for medical, infra-red and multi-focus image examples. Moreover, we study the scores variability and statistical discrimination power with the TNO night vision database using the Friedman test. Finally, we define a new leave one out procedure based on our fidelity measure that selects the best subset of images (within a collection of distorted and unregistered cell phone type images) that provides a defect-free composite output. We exemplify the procedure with the fusion of a collection of images from Latour and Van Dongen paintings suffering from glass highlights and speckle noise, among other artifacts. The proposed multiscale quality measure MS-Q_W demonstrates improvement over the previous single-scale similarity measures towards a fidelity assessment between quantitative image fusion quality metrics and human perceptual qualitative scores.