DIBR中基于平面扫描的深度重建方法

提出一种 DIBR 中基于平面扫描法的深度重建方法,与立体深度重建算法和基于图像的视觉壳算法不同,本文进一步改进平面扫描算法,无需任何场景的几何先验知识,而是利用每个像素点的深度信息合成真实场景的虚拟视点.当输入图像映射至相互平行的虚拟深度平面时,采用"动态判决方法"来计算像素间的色彩一致度;并在虚拟视合成中采用了基于视向权重策略的新视点重建方法.本文算法获取的深度信息更为精确,虚拟新视点的质量得到较大提高.     

Image-based rendering of complex scenes from a multi-camera rig

Image-based rendering is a method to synthesise novel views from a set of given real images. Two methods to extrapolate novel views of complex scenes with occlusions and large depth discontinuities from images of a moving uncalibrated multi-camera rig are described. The real camera viewpoints are calibrated from the image data and dense depth maps are estimated for each real view. Novel views are synthesised from this representation with view-dependent image-based rendering techniques at interactive rates. Since the 3D scene geometry is available in this approach, it is well suited for mixed reality applications where synthetic 3D objects are seamlessly embedded in the novel view.     

Hybrid dual stream blender for wide baseline view synthesis

Free navigation of a scene requires warping some reference views to some desired target viewpoint and blending them to synthesize a virtual view. Convolutional Neural Networks (ConvNets) based methods can learn both the warping and blending tasks jointly. Such methods are often designed for moderate inter-camera baseline distance and larger kernels are required for warping if the baseline distance increases. Algorithmic methods can in principle deal with large baselines, however the synthesized view suffers from artifacts near disoccluded pixels. We present a hybrid approach where first, reference views are algorithmically warped to the target position and then are blended via a ConvNet. Preliminary view warping allows reducing the size of the convolutional kernels and thus the learnable parameters count. We propose a residual encoder–decoder for image blending with a Siamese encoder to further keep the parameters count low. We also contribute a hole inpainting algorithm to fill the disocclusions in the warped views. Our view synthesis experiments on real multiview sequences show better objective image quality than state-of-the-art methods due to fewer artifacts in the synthesized images.     

多孔径压缩编码超分辨率大视场成像方法

多孔径成像是一种融合了仿生复眼视觉的新型成像方法,具有小型化、大视场、高分辨率等多种优势,但由于每个子孔径对应的单元图像分辨率过低,导致其成像质量和视场角的提升十分有限。为了进一步提高成像分辨率和探测视场,基于压缩感知理论设计随机编码模板,并紧贴子孔径放置对入射光场进行调制,通过单次曝光记录编码后的低分辨率单元图像阵列,利用稀疏优化算法,重构所有低分辨率单元图像获得超分辨率大视场图像。理论分析和仿真实验证明了该方法的有效性。该方法不仅能兼顾大视场高分辨率成像,而且大大缩小系统等效焦距,具有薄层结构,体积小而重量轻,可为微光机电一体化系统的研制设计提供借鉴。     

Reduced Reference Quality Metric for Synthesized Virtual Views in 3DTV

Multi‐view video plus depth (MVD) has been widely used owing to its effectiveness in three‐dimensional data representation. Using MVD, color videos with only a limited number of real viewpoints are compressed and transmitted along with captured or estimated depth videos. Because the synthesized views are generated from decoded real views, their original reference views do not exist at either the transmitter or receiver. Therefore, it is challenging to define an efficient metric to evaluate the quality of synthesized images. We propose a novel metric—the reduced‐reference quality metric. First, the effects of depth distortion on the quality of synthesized images are analyzed. We then employ the high correlation between the local depth distortions and local color characteristics of the decoded depth and color images, respectively, to achieve an efficient depth quality metric for each real view. Finally, the objective quality metric of the synthesized views is obtained by combining all the depth quality metrics obtained from the decoded real views. The experimental results show that the proposed quality metric correlates very well with full reference image and video quality metrics.     

基于逆向映射的空洞填补方法

为了有效填补虚拟视点图像中的公共空洞,提出 了一种基于逆向映射的空洞填补方法。 首先利用深度图像绘制(DIBR)技术将左、右参考视点映射到虚拟视点位置,利用图像膨胀方 法将映射的虚拟视图中的空 洞区域进行扩大,以消除虚拟视点图像中的伪影瑕疵;然后,提取出膨胀后空洞区域的边界 ,并将其逆映 射到原始的参考图像中,根据空洞与边界的相对位置,选取原始图像中相对位置上的像素来 填充虚拟视图 中的空洞区域;最后,将空洞填补之后的左、右视点映射的虚拟视图进行融合获得最终的虚 拟视图。实验 证明,本文方法有效解决了传统空洞填补方法容易将前景像素填充到背景区域的问题,能 够获得较好的视觉观看效果和较高的客观峰值信噪比(PSNR)值。     

Improved interview video error concealment on whole frame packet loss

An improved DIBR-based (Depth image based rendering) whole frame error concealment method for multiview video with depth is designed. An optimal reference view selection is first proposed. The paper further includes three modified parts for the DIBRed pixels. First, the missing 1-to-1 pixels are concealed by the pixels from another view. The light differences between views are taken care of by the information of the motion vector of the projected coordination and a reverse DIBR procedure. Second, the generation of the many-to-1 pixels is improved via their depth information. Third, the hole pixels are found using the estimated motion vectors derived efficiently from a weighted function of the neighboring available motion vectors and their distance to the target hole pixel. The experimental results show that, compared to the state-of-the-art method, the combined system of the four proposed methods is superior and improves the performance by 5.53 dB at maximum.     

Reconstructing dense light field from array of multifocus images for novel view synthesis.

This paper presents a novel method for synthesizing a novel view from two sets of differently focused images taken by an aperture camera array for a scene consisting of two approximately constant depths. The proposed method consists of two steps. The first step is a view interpolation to reconstruct an all-in-focus dense light field of the scene. The second step is to synthesize a novel view by a light-field rendering technique from the reconstructed dense light field. The view interpolation in the first step can be achieved simply by linear filters that are designed to shift different object regions separately, without region segmentation. The proposed method can effectively create a dense array of pin-hole cameras (i.e., all-in-focus images), so that the novel view can be synthesized with better quality.     

Inverse tensor transfer with applications to novel view synthesis and multi-baseline stereo

A new tensor transfer-based novel view synthesis (NVS) method is proposed in this paper. As opposed to conventional tensor transfer methods which transfer the pixel from the real input views to the virtual novel view, our method operates inversely in the sense that it transfers a pixel from the novel view image back to the real images. This inverse tensor-transfer approach offers a simple mechanism for associating corresponding image points across multiple views, resulting in geometrically consistent pixel chains across the input images. A colour consistency metric is used to choose the most likely colour for a pixel in the novel image by analysing the spread of colours in each of the possible pixel chains. By emphasizing colour consistency, rather than depth, our method avoids the need to precompute a dense depth map (which is essential for most conventional transfer methods), therefore alleviating many common problems with conventional methods. Experiments involving NVS on real images give promising results. The synthesized novel view image is not only photo-realistic but also has the right geometric relationship with respect to the other views.Since this method avoids explicit depth map computation, we further investigate its applicability to the multi-baseline stereo matching problem (MBS). By using this inverse transfer idea, we are able to handle non-ideally configured MBS in a natural and efficient way. The new MBS algorithm has been used for stereo vision navigation.     

柱面全景图像视图合成中的关键问题研究

图像校正和插值是基于柱面全景图像视图合成中的关键问题。首先分析了极线采样方法在柱面图像校正像素保持率和像素有效率上的优势,通过计算极线的数量、范围及起止位置,有效保留了源图像的信息,减少了以往算法中由于使用单应性映射而导致的弱采样和过采样效应;其次,在新视点插值上,根据柱面全景图像的成像模型,推导了合成视图上像素点的位置和颜色计算公式。最后给出了基于仿真图像和实际图像的视图合成结果。      

A multilinear constraint on dichromatic planes for illumination estimation.

A new multilinear constraint on the color of the scene illuminant based on the dichromatic reflection model is proposed. The formulation avoids the problem, common to previous dichromatic methods, of having to first identify pixels corresponding to the same surface material. Once pixels from two or more materials have been identified, their corresponding dichromatic planes can be intersected to yield the illuminant color. However, it is not always easy to determine which pixels from an arbitrary region of an image belong to which dichromatic plane. The image region may cover an area of the scene encompassing several different materials and, hence, pixels from several different dichromatic planes. The new multilinear constraint accounts for this multiplicity of materials and provides a mechanism for choosing the most plausible illuminant from a finite set of candidate illuminants. The performance of this new method is tested on a database of real images.     

Various density light field image coding based on distortion minimization interpolation

In recent years, the light field (LF) as a new imaging modality has attracted wide interest. The large data volume of LF images poses great challenge to LF image coding, and the LF images captured by different devices show significant differences in angular domain. In this paper we propose a view prediction framework to handle LF image coding with various sampling density. All LF images are represented as view arrays. We first partition the views into reference view (RV) set and intermediate view (IV) set. The RVs are rearranged into a pseudo sequence and directly compressed by a video encoder. Other views are then predicted by the RVs. To exploit the four dimensional signal structure, we propose the linear approximation prior (LAP) to reveal the correlation among LF views and efficiently remove the LF data redundancy. Based on the LAP, a distortion minimization interpolation (DMI) method is used to predict IVs. To robustly handle the LF images with different sampling density, we propose an Iteratively Updating depth image based rendering (IU-DIBR) method to extend our DMI. Some auxiliary views are generated to cover the target region and then the DMI calculates reconstruction coefficients for the IVs. Different view partition patterns are also explored. Extensive experiments on different types LF images also valid the efficiency of the proposed method.     

软分割约束边缘保持插值的半自动2D转3D

半自动2D转3D将用户标注的稀疏深度转换成稠密深度,是解决3D片源不足的主要手段之一.针对现有方法利用硬分割增强深度边缘引入误差的问题,提出像素点与超像素深度一致性约束的边缘保持插值方法.首先,建立像素点深度和超像素深度传播的能量模型,通过像素点与所属超像素间深度差异的约束项将二者关联起来;其次,利用矩阵表示形式将两个能量模型的最优化转换成一个稀疏线性方程组的求解问题.通过超像素提供的约束项,可避免深度传播穿过低对比度边缘区域,从而能保持对象边缘.实验结果表明,本文方法对象边缘处深度恢复的准确性优于融合图割的随机游走方法,PSNR改善了1.5dB以上.     

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Light Field (LF) image angular super-resolution aims to synthesize a high angular resolution LF image from a low angular resolution one, and is drawing increased attention because of its wide applications. In order to reconstruct a high angular resolution LF image, many learning based LF image angular super-resolution methods have been proposed. However, most existing methods are based on LF Epipolar Plane Image or Epipolar Plane Image volume representation, which underuse the LF image structure. The LF view spatial correlation and neighboring LF views angular correlations which can reflect LF image structure are not fully explored, which reduces LF angular super-resolution quality. In order to alleviate this problem, this paper introduces an Epipolar Plane Image Volume Stack (EPI-VS) representation for LF angular super-resolution. The EPI-VS is constituted by arranging all LF views in a raster order, which benefits in exploring LF view spatial correlation and neighboring LF views angular correlations. Based on such representation, we further propose an LF angular super-resolution network. 3D convolutions are applied in the whole super-resolution network to better accommodate the input EPI-VS data and allow information propagation between two spatial and one directional dimensions of EPI-VS data. Extensive experiments on synthetic and real-world LF scenes demonstrate the effectiveness of the proposed network. Moreover, we also illustrate the superiority of our network by applying it in scene depth estimation task.     

Geometric and colorimetric error compensation for multi-view images

In general, excessive colorimetric and geometric errors in multi-view images induce visual fatigue to users. Various works have been proposed to reduce these errors, but conventional works have only been available for stereoscopic images while requiring cumbersome additional tasks, and often showing unstable results. In this paper, we propose an effective multi-view image refinement algorithm. The proposed algorithm analyzes such errors in multi-view images from sparse correspondences and compensates them automatically. While the conventional works transform every view to compensate geometric errors, the proposed method transforms only the source views with consideration of a reference view. Therefore this approach can be extended regardless of the number of views. In addition, we also employ uniform view intervals to provide consistent depth perception among views. We correct color inconsistency among views from the correspondences by considering importance and channel properties. Various experimental results show that the proposed algorithm outperforms conventional approaches and generates more visually comfortable multi-view images.     

Geometry-based distributed scene representation with omnidirectional vision sensors

This paper addresses the problem of efficient representation of scenes captured by distributed omnidirectional vision sensors. We propose a novel geometric model to describe the correlation between different views of a 3-D scene. We first approximate the camera images by sparse expansions over a dictionary of geometric atoms. Since the most important visual features are likely to be equivalently dominant in images from multiple cameras, we model the correlation between corresponding features in different views by local geometric transforms. For the particular case of omnidirectional images, we define the multiview transforms between corresponding features based on shape and epipolar geometry constraints. We apply this geometric framework in the design of a distributed coding scheme with side information, which builds an efficient representation of the scene without communication between cameras. The Wyner-Ziv encoder partitions the dictionary into cosets of dissimilar atoms with respect to shape and position in the image. The joint decoder then determines pairwise correspondences between atoms in the reference image and atoms in the cosets of the Wyner-Ziv image in order to identify the most likely atoms to decode under epipolar geometry constraints. Experiments demonstrate that the proposed method leads to reliable estimation of the geometric transforms between views. In particular, the distributed coding scheme offers similar rate-distortion performance as joint encoding at low bit rate and outperforms methods based on independent decoding of the different images.     

A robust blind watermarking algorithm for depth-image-based rendering 3D images

The protection of 3D contents from illegal distribution has attracted considerable attention and depth-image-based rendering (DIBR) is proved to be a promising technology for 3D image and video displaying. In this paper, we propose a new digital watermarking scheme for DIBR 3D images based on feature regions and ridgelet transform (RT). In this scheme, the center view and the depth map are made available at the content provider side. After selecting the reference points of the center view, we construct the feature regions for watermark embedding. Considering the sparse image representation and directional sensitivity of the RT, the watermark bits are embedded into the amplitudes of the ridgelet coefficients of the most energetic direction. The virtual left and virtual right views are generated from the watermarked center view and the associated depth map at the content consumer side. The watermarked view has good perceptual quality under both the objective and subjective image quality evaluations. The embedded watermark can be detected blindly with low bit error rate (BER) from the watermarked center view, the synthesized left and right views even when the views are distorted and distributed separately. The experimental results demonstrate that the proposed scheme exhibits good performance in terms of robustness against various image processing attacks. Meanwhile, our method can be robust to common DIBR processing, such as depth image variation, baseline distance adjustment and different rendering conditions. Furthermore, compared with other related and state-of-the-art methods, the proposed algorithm shows higher accuracy in watermark extraction.     

Binocular transfer methods for point-feature tracking of image sequences

Image transfer is a method for projecting a 3D scene from two or more reference images. Typically, the correspondences of target points to be transferred and the reference points must be known over the reference images. We present two new transfer methods that eliminate the target point correspondence requirement. We show that five reference points matched across two reference images are sufficient to linearly resolve transfer under affine projection using two views instead of three views as needed by other techniques. Furthermore, given the correspondences of any four of the five reference points in any other view, we can transfer a target point to a third view from any one of the two original reference views. To improve the robustness of the affine projection method, we incorporate an orthographic camera model. A factorization method is applied to the reference points matched over two reference views. Experiments with real image sequences demonstrate the application of both methods for motion tracking.     

A Virtual Walk along Street Based on View Interpolation

Image morphing is a powerful tool for visual effect. In this paper, a view interpolation algorithm is proposed to simulate a virtual walk along a street from start position to end position. To simulate a virtual walking view needs to create new appearing scene in the vision-vanishing point and disappearing scene beyond the scope of view. To attain these two aims we use two enhanced position parameters to match pixels of source images and target images. One enhanced position parameter is the angular coordinates of pixels. Another enhanced position parameter is the distances from pixels to the vision-vanishing point. According to the parameter values, pixels beyond the scope of view can be ＂moved＂ out in linear interpolation. Result demonstrates the validity of the algorithm. Another advantage of this algorithm is that the enhanced position parameters are based on real locations and walking distances, so it is also an approach to online virtual tour by satellite maps of virtual globe applications such as Google Earth.     

Underwater image enhancement by wavelength compensation and dehazing

Light scattering and color change are two major sources of distortion for underwater photography. Light scattering is caused by light incident on objects reflected and deflected multiple times by particles present in the water before reaching the camera. This in turn lowers the visibility and contrast of the image captured. Color change corresponds to the varying degrees of attenuation encountered by light traveling in the water with different wavelengths, rendering ambient underwater environments dominated by a bluish tone. No existing underwater processing techniques can handle light scattering and color change distortions suffered by underwater images, and the possible presence of artificial lighting simultaneously. This paper proposes a novel systematic approach to enhance underwater images by a dehazing algorithm, to compensate the attenuation discrepancy along the propagation path, and to take the influence of the possible presence of an artifical light source into consideration. Once the depth map, i.e., distances between the objects and the camera, is estimated, the foreground and background within a scene are segmented. The light intensities of foreground and background are compared to determine whether an artificial light source is employed during the image capturing process. After compensating the effect of artifical light, the haze phenomenon and discrepancy in wavelength attenuation along the underwater propagation path to camera are corrected. Next, the water depth in the image scene is estimated according to the residual energy ratios of different color channels existing in the background light. Based on the amount of attenuation corresponding to each light wavelength, color change compensation is conducted to restore color balance. The performance of the proposed algorithm for wavelength compensation and image dehazing (WCID) is evaluated both objectively and subjectively by utilizing ground-truth color patches and video downloaded from the Youtube website. Both results demonstrate that images with significantly enhanced visibility and superior color fidelity are obtained by the WCID proposed.