Texture aided depth frame interpolation

Recent development of depth acquiring technique has accelerated the progress of 3D video in the market. Utilizing the acquired depth, arbitrary view frames can be generated based on depth image based rendering (DIBR) technique in free viewpoint video system. Different from texture video, depth sequence is mainly utilized for virtual view generation rather than viewing. Inspired by this, a depth frame interpolation scheme using texture information is proposed in this paper. The proposed scheme consists of a texture aided motion estimation (TAME) and texture aided motion compensation (TAMC) to fully explore the correlation between depth and the accompanying textures. The optimal motion vectors in TAME and the best interpolation weights in TAMC are respectively selected taking the geometric mapping relationship between depth and the accompanying texture frames into consideration. The proposed scheme is able to not only maintain the temporal consistency among interpolated depth sequence but also improve the quality of virtual frames generated by interpolated depth. Besides, it can be easily applied to arbitrary motion compensation based frame interpolation scheme. Experimental results demonstrate that the proposed depth frame interpolation scheme is able to improve the quality of virtual view texture frames in both subjective and objective criterions compared with existing schemes.     

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.     

Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

Optimal depth recovery using image guided TGV with depth confidence for high-quality view synthesis

This paper presents a new depth image recovery method for RGB-D sensors giving a complete, sharp, and accurate object shape from a noisy boundary depth map. The proposed method uses the image guided Total Generalized Variation (TGV) with the depth confidence. A new directional hole filling method of view synthesis is also investigated to produce natural texture in hole regions whereas reducing blurring effect and preventing distortion. Thus, a high-quality image view can be achieved. Experimental results show that the proposed method yields higher quality recovered depth maps and synthesized image views than other previous methods.     

基于Kinect的虚拟视点生成技术

传统的多视点生成方法是基于多相机阵列系统的关键技术。现提出了基于Kinect的多视点成像计算方法。首先对Kinect的深度图使用三边滤波器进行平滑,根据修复好的深度图配合彩色图,利用DIBR技术生成多个存在空缺信息的彩色视点;最后结合彩色图的纹理结构信息和深度图的背景信息对有丢失信息的彩色图进行修复。实验结果表明,文中提出的深度修复方法能够有效地修补Kinect的深度图,生成的虚拟视点图在3DTV上效果明显,立体视觉效果显著。     

基于JNDD模型面向虚拟视绘制的快速深度图编码

为降低深度数据的编码复杂度并保证重建虚拟视点的质量,提出了一种基于JNDD模型面向虚拟视点绘制的快速深度图编码算法,引入最小可觉深度差模型,将深度图划分为对绘制失真敏感的竖直边缘区域与失真难以被人眼察觉的平坦区域,并相应地为编码过程中的宏块模式选择设计了两种搜索策略。实验结果表明,与JM编码方案相比,本文所提出的方法在保证虚拟视质量与编码码率基本不变的前提下,显著降低了编码复杂度,有助于在三维视频系统中提高深度编码模块的编码速度。     

View generation for three-dimensional scenes from video sequences

This paper focuses on the representation and view generation of three-dimensional (3-D) scenes. In contrast to existing methods that construct a full 3-D model or those that exploit geometric invariants, our representation consists of dense depth maps at several preselected viewpoints from an image sequence. Furthermore, instead of using multiple calibrated stationary cameras or range scanners, we derive our depth maps from image sequences captured by an uncalibrated camera with only approximately known motion. We propose an adaptive matching algorithm that assigns various confidence levels to different regions in the depth maps. Nonuniform bicubic spline interpolation is then used to fill in low confidence regions in the depth maps. Once the depth maps are computed at preselected viewpoints, the intensity and depth at these locations are used to reconstruct arbitrary views of the 3-D scene. Specifically, the depth maps are regarded as vertices of a deformable 2-D mesh, which are transformed in 3-D, projected to 2-D, and rendered to generate the desired view. Experimental results are presented to verify our approach.     

Improved interview video error concealment on whole frame packet loss

An improved DIBR-based (Depth image based rendering) whole frame error concealment method for multiview video with depth is designed. An optimal reference view selection is first proposed. The paper further includes three modified parts for the DIBRed pixels. First, the missing 1-to-1 pixels are concealed by the pixels from another view. The light differences between views are taken care of by the information of the motion vector of the projected coordination and a reverse DIBR procedure. Second, the generation of the many-to-1 pixels is improved via their depth information. Third, the hole pixels are found using the estimated motion vectors derived efficiently from a weighted function of the neighboring available motion vectors and their distance to the target hole pixel. The experimental results show that, compared to the state-of-the-art method, the combined system of the four proposed methods is superior and improves the performance by 5.53 dB at maximum.     

Disparity field and depth map coding for multiview 3D image generation

In the present paper techniques are examined for the coding of the depth map and disparity fields for stereo or multiview image communication applications. It is assumed that both the left and right channels of the multiview image sequence are coded using block- or object-based methods. A dynamic programming algorithm is used to estimate a disparity field between each stereo image pair. Depth is then estimated and occlusions are optionally detected, based on the estimated disparity fields. Spatial interpolation techniques are examined based on the disparity/depth information and the detection of occluded regions using either stereoscopic or trinocular camera configurations. It is seen that the presence of a third camera at the transmitter site improves the estimation of disparities, the detection of occlusions and the accuracy of the resulting spatial interpolation at the receiver. Various disparity field and depth map coding techniques are then proposed and evaluated, with emphasis given to the quality of the resulting intermediate images at the receiver site. Block-based and wireframe modeling techniques are examined for the coding of isolated depth or disparity map information. Further, 2D and 3D motion compensation techniques are evaluated for the coding of sequences of depth or disparity maps. The motion fields needed may be available as a byproduct of block-based or object-based coding of the intensity images. Experimental results are given for the evaluation of the performance of the proposed coding and spatial interpolation methods.     

DIBR中基于平面扫描的深度重建方法

提出一种 DIBR 中基于平面扫描法的深度重建方法,与立体深度重建算法和基于图像的视觉壳算法不同,本文进一步改进平面扫描算法,无需任何场景的几何先验知识,而是利用每个像素点的深度信息合成真实场景的虚拟视点.当输入图像映射至相互平行的虚拟深度平面时,采用"动态判决方法"来计算像素间的色彩一致度;并在虚拟视合成中采用了基于视向权重策略的新视点重建方法.本文算法获取的深度信息更为精确,虚拟新视点的质量得到较大提高.     

Stereoscopic image generation based on depth images for 3D TV

A depth-image-based rendering system for generating stereoscopic images is proposed. One important aspect of the proposed system is that the depth maps are pre-processed using an asymmetric filter to smoothen the sharp changes in depth at object boundaries. In addition to ameliorating the effects of blocky artifacts and other distortions contained in the depth maps, the smoothing reduces or completely removes newly exposed (disocclusion) areas where potential artifacts can arise from image warping which is needed to generate images from new viewpoints. The asymmetric nature of the filter reduces the amount of geometric distortion that might be perceived otherwise. We present some results to show that the proposed system provides an improvement in image quality of stereoscopic virtual views while maintaining reasonably good depth quality.     

基于逆向映射的空洞填补方法

为了有效填补虚拟视点图像中的公共空洞,提出 了一种基于逆向映射的空洞填补方法。 首先利用深度图像绘制(DIBR)技术将左、右参考视点映射到虚拟视点位置,利用图像膨胀方 法将映射的虚拟视图中的空 洞区域进行扩大,以消除虚拟视点图像中的伪影瑕疵;然后,提取出膨胀后空洞区域的边界 ,并将其逆映 射到原始的参考图像中,根据空洞与边界的相对位置,选取原始图像中相对位置上的像素来 填充虚拟视图 中的空洞区域;最后,将空洞填补之后的左、右视点映射的虚拟视图进行融合获得最终的虚 拟视图。实验 证明,本文方法有效解决了传统空洞填补方法容易将前景像素填充到背景区域的问题,能 够获得较好的视觉观看效果和较高的客观峰值信噪比(PSNR)值。     

3D reconstruction with auto-selected keyframes based on depth completion correction and pose fusion

Dense 3D reconstruction is required for robots to safely navigate or perform advanced tasks. The accurate depth information of the image and its pose are the basis of 3D reconstruction. The resolution of depth maps obtained by LIDAR and RGB-D cameras is limited, and traditional pose calculation methods are not accurate enough. In addition, if each image is used for dense 3D reconstruction, the dense point clouds will increase the amount of calculation. To address these issues, we propose a 3D reconstruction system. Specifically, we propose a depth network of contour and gradient attention, which is used to complete and correct depth maps to obtain high-resolution and high-quality depth maps. Then, we propose a method of fusion of traditional algorithms and deep learning for pose estimation to obtain accurate localization results. Finally, we adopt the method of autonomous selection of keyframes to reduce the number of keyframes, the surfel-based geometric reconstruction is performed to reconstruct the dense 3D environment. On the TUM RGB-D, ICL-NIUM, and KITTI datasets, our method significantly improves the quality of the depth maps, the localization results, and the effect of 3D reconstruction. At the same time, we have also accelerated the speed of 3D reconstruction.     

A new method for inpainting of depth maps from time-of-flight sensors based on a modified closing by reconstruction algorithm

Time-of-Flight (ToF) sensors are popular devices that extract 3D information from a scene but result to be susceptible to noise and loss of data creating holes and gaps in the boundaries of the objects. The most common approaches to tackling this problem are supported by color images with good results, however, not all ToF devices produce color information. Mathematical morphology provides operators that can manage the problem of noise in single depth frames. In this paper, a new method for the filtering of single depth maps, when no color image is available, is presented, based on a modification to the morphological closing by reconstruction algorithm. The proposed method eliminates noise, emphasizing a high contour preservation, and it is compared, both qualitative and quantitatively, with other state-of-the-art filters. The proposed method represents an improvement to the closing by reconstruction algorithm that can be applied for filter depth maps of ToF devices.     

An effective error concealment scheme for heavily corrupted H.264/AVC videos based on Kalman filtering

Error concealment at the decoder side is an economical approach to ensuring an acceptable and stable video quality in case of packet erasure or loss, and thus, it has attracted considerable research and application interest. Relevant techniques usually employ the spatial or temporal correlation to recover the motion vectors (MVs) of the missing blocks, and interpolation, extrapolation, or boundary-matching schemes are usually effective. However, for heavily corrupted sequences, e.g., with block loss rate beyond 50 %, most methods might perform less satisfactorily. Inspired by the tracking efficiency of Kalman filter (KF), in the present work, we adopted it to predict the missing MVs, and the unpredicted ones (minority) were restored complementarily using a modified bilinear motion field interpolation (MFI) method. Since the KF prediction is independent of the loss rate, the present framework proves to be robust for heavily corrupted videos. Experimental results on typical sequences reveal that the proposed algorithm outperforms the boundary-matching algorithm embedded in the H.264/AVC reference code, the MFI and the MV extrapolation techniques in the literature.     

Virtual view synthesis using joint information from multi-view

Immersive media has attracted widespread attention with the development of virtual reality. Three Degree of Freedom Plus media greatly enhances the user experience by allowing users’ head motion and viewpoint switching within a certain range. Due to the limitation of panoramic video acquisition and transmission, it is impossible to obtain videos from any viewpoint directly. Virtual view synthesis is the general solution to this problem. However, existing algorithms do not adequately consider the pixel correlation between multiple views. Thus, we propose a virtual view synthesis algorithm using joint information from multi-view panoramic videos to further explore the pixel correlation. Specifically, sub-pixels from different reference views in the virtual view are obtained by performing multi-view three-dimensional image warping. Dedicated area division and interpolation methods are then designed to improve the synthesized quality. Experimental results show that the proposed algorithm outperforms the state-of-the-art virtual view synthesis algorithms in performance and efficiency.     

Adaptive Learning Based View Synthesis Prediction for Multi-View Video Coding

In the applications of Free View TV, pre-estimated depth information is available to synthesize the intermediate views as well as to assist multi-view video coding. Existing view synthesis prediction schemes generate virtual view picture only from interview pictures. However, there are many types of signal mismatches caused by depth errors, camera heterogeneity or illumination difference across views and these mismatches decrease the prediction capability of virtual view picture. In this paper, we propose an adaptive learning based view synthesis prediction algorithm to enhance the prediction capability of virtual view picture. This algorithm integrates least square prediction with backward warping to synthesize the virtual view picture, which not only utilizes the adjacent views information but also the temporal decoded information to adaptively learn the prediction coefficients. Experiments show that the proposed method reduces the bitrates by up to 18 % relative to the multi-view video coding standard, and about 11 % relative to the conventional view synthesis prediction method.     

Using 3D face priors for depth recovery

For repairing inaccurate depth measurements from commodity RGB-D sensors, existing depth recovery methods primarily rely on low-level and rigid prior information. However, as the depth quality deteriorates, the recovered depth maps become increasingly unreliable, especially for non-rigid objects. Thus, additional high-level and non-rigid information is needed to improve the recovery quality. Taking as a starting point the human face that is the primary prior available in many high-level tasks, in this paper, we incorporate face priors into the depth recovery process. In particular, we propose a joint optimization framework that consists of two main steps: transforming the face model for better alignment and applying face priors for improved depth recovery. Face priors from both sparse and dense 3D face models are studied. By comparing with the baseline method on benchmark datasets, we demonstrate that the proposed method can achieve up to 23.8% improvement in depth recovery with more accurate face registrations, bringing inspirations to both non-rigid object modeling and analysis.     

基于双目立体视觉的场景分割方法

提出一种基于双目立体视觉的场景分割方法:首先根据双目立体视觉系统提供的左右视图进行三维场景重构,得到场景的几何深度图,同时利用左视图进行RGB颜色空间到CIELab均匀颜色空间的转换以得到颜色信息;然后将颜色与几何信息构造生成六维向量;最后再将六维向量给到聚类算法中进行分割并对分割的伪影进行消除,得到最终的分割结果.对Middlebury数据集样本场景baby 2实验了6种立体视觉算法和3种聚类技术的不同组合进行的场景分割,从实验结果来看,不同的组合应用所提方法都比传统方法具有更好的分割效果.     

Stereoscopic view synthesis based on region-wise rendering and sparse representation

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D conversion process. One of the main problems in DIBR is how to reduce holes that occur on the generated virtual view images. In this paper, we make two main contributions to deal with the problem. Firstly, a region-wise rendering framework, which divides the original image regions into three special classes and renders each with optimal adaptive process respectively, is introduced. Then, a novel sparse representation-based inpainting method, which can yield visually satisfactory results with less computational complexity for high quality 2D to 3D conversion, is proposed. Numerical experimental results demonstrate the good performance of the proposed methods.