Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

Stereoscopic view synthesis based on region-wise rendering and sparse representation

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D conversion process. One of the main problems in DIBR is how to reduce holes that occur on the generated virtual view images. In this paper, we make two main contributions to deal with the problem. Firstly, a region-wise rendering framework, which divides the original image regions into three special classes and renders each with optimal adaptive process respectively, is introduced. Then, a novel sparse representation-based inpainting method, which can yield visually satisfactory results with less computational complexity for high quality 2D to 3D conversion, is proposed. Numerical experimental results demonstrate the good performance of the proposed methods.     

Optimal depth recovery using image guided TGV with depth confidence for high-quality view synthesis

This paper presents a new depth image recovery method for RGB-D sensors giving a complete, sharp, and accurate object shape from a noisy boundary depth map. The proposed method uses the image guided Total Generalized Variation (TGV) with the depth confidence. A new directional hole filling method of view synthesis is also investigated to produce natural texture in hole regions whereas reducing blurring effect and preventing distortion. Thus, a high-quality image view can be achieved. Experimental results show that the proposed method yields higher quality recovered depth maps and synthesized image views than other previous methods.     

A robust blind watermarking algorithm for depth-image-based rendering 3D images

The protection of 3D contents from illegal distribution has attracted considerable attention and depth-image-based rendering (DIBR) is proved to be a promising technology for 3D image and video displaying. In this paper, we propose a new digital watermarking scheme for DIBR 3D images based on feature regions and ridgelet transform (RT). In this scheme, the center view and the depth map are made available at the content provider side. After selecting the reference points of the center view, we construct the feature regions for watermark embedding. Considering the sparse image representation and directional sensitivity of the RT, the watermark bits are embedded into the amplitudes of the ridgelet coefficients of the most energetic direction. The virtual left and virtual right views are generated from the watermarked center view and the associated depth map at the content consumer side. The watermarked view has good perceptual quality under both the objective and subjective image quality evaluations. The embedded watermark can be detected blindly with low bit error rate (BER) from the watermarked center view, the synthesized left and right views even when the views are distorted and distributed separately. The experimental results demonstrate that the proposed scheme exhibits good performance in terms of robustness against various image processing attacks. Meanwhile, our method can be robust to common DIBR processing, such as depth image variation, baseline distance adjustment and different rendering conditions. Furthermore, compared with other related and state-of-the-art methods, the proposed algorithm shows higher accuracy in watermark extraction.     

基于Kinect的虚拟视点生成技术

传统的多视点生成方法是基于多相机阵列系统的关键技术。现提出了基于Kinect的多视点成像计算方法。首先对Kinect的深度图使用三边滤波器进行平滑,根据修复好的深度图配合彩色图,利用DIBR技术生成多个存在空缺信息的彩色视点;最后结合彩色图的纹理结构信息和深度图的背景信息对有丢失信息的彩色图进行修复。实验结果表明,文中提出的深度修复方法能够有效地修补Kinect的深度图,生成的虚拟视点图在3DTV上效果明显,立体视觉效果显著。     

Real-Time Refinement of Kinect Depth Maps using Multi-Resolution Anisotropic Diffusion

In this paper, we present a novel real-time algorithm to refine depth maps generated by low-cost commercial depth sensors like the Microsoft Kinect. The Kinect sensor falls under the category of RGB-D sensors that can generate a high resolution depth map and color image of a scene. They are relatively inexpensive and are commercially available off-the-shelf. However, owing to their low complexity, there are several artifacts that one encounters in the depth map like holes, mis-alignment between the depth map and color image and lack of sharp object boundaries in the depth map. This is a potential problem in applications that require the color image to be projected in 3-D using the depth map. Such applications depend heavily on the depth map and thus the quality of the depth map is of vital importance. In this paper, a novel multi-resolution anisotropic diffusion based algorithm is presented that accepts a Kinect generated depth map and color image and computes a dense depth map in which the holes have been filled and the edges of the objects are sharpened and aligned with the objects in the color image. The proposed algorithm also ensures that regions in the depth map where the depth is properly estimated are not filtered and ensures that the depth values in the final depth map are the same values that existed in the original depth map. Experimental results are provided to demonstrate the improvement in the quality of the depth map and also execution time results are provided to prove that the proposed method can be executed in real-time.     

3D-HEVC中深度图编码技术研究进展

3D-HEVC是为了满足3D视频和自由视点视频的高效编码而最新制定的视频编码标准,它要求同时编码几个视点的纹理视频和深度图.完全采用传统的技术来编码深度图会使得深度图内部锐利边界处产生伪影效应,为此,一些新的针对于深度图的编码工具被开发.详细介绍了这些编码工具,同时介绍了编码深度图时所使用的率失真优化方法.     

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.     

Depth dithering based on texture edge-assisted classification

Development in low-cost depth acquisition devices pushes the relevant applications forward, which requires higher quality of the depth maps. In this paper, a novel depth de-noising algorithm is proposed, which dithers the depth map according to its spatial and temporal depth classification around the extracted texture edges. After roughly filling the holes in the depth map by joint bilateral filter, the depth map is partitioned into edge region and non-edge region according to the extracted texture edges. The depth values on each side of the mapped texture edges are dithered adapting to depth classification. Joint spatio-temporal dithering approach is proposed to align the depth edges with the texture edges and to suppress the depth artifacts. Experimental results demonstrate that compared with existing depth de-noising approaches, the proposed algorithm provides much sharper and clearer depth edges and it improves the spatial and temporal consistency of the processed depth, which further benefits the applications requiring the depth of high quality, e.g., 3D rendering.     

Hole‐Filling Methods Using Depth and Color Information for Generating Multiview Images

This paper presents new hole‐filling methods for generating multiview images by using depth image based rendering (DIBR). Holes appear in a depth image captured from 3D sensors and in the multiview images rendered by DIBR. The holes are often found around the background regions of the images because the background is prone to occlusions by the foreground objects. Background‐oriented priority and gradient‐oriented priority are also introduced to find the order of hole‐filling after the DIBR process. In addition, to obtain a sample to fill the hole region, we propose the fusing of depth and color information to obtain a weighted sum of two patches for the depth (or rendered depth) images and a new distance measure to find the best‐matched patch for the rendered color images. The conventional method produces jagged edges and a blurry phenomenon in the final results, whereas the proposed method can minimize them, which is quite important for high fidelity in stereo imaging. The experimental results show that, by reducing these errors, the proposed methods can significantly improve the hole‐filling quality in the multiview images generated.     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

INPAINTING ALGORITHM FOR KINECT DEPTH MAP BASED ON FOREGROUND SEGMENTATION简

The depth information of the scene camera, and depth extraction is a key technology indicates the distance between the object and the in 3D video system. The emergence of Kinect makes the high resolution depth map capturing possible. However, the depth map captured by Kinect can not be directly used due to the existing holes and noises, which needs to be repaired. We propose a texture combined inpainting algorithm in this paper. Firstly, the foreground is segmented combined with the color characteristics of the texture image to repair the foreground of the depth map. Secondly, region growing is used to determine the match region of the hole in the depth map, and to accurately position the match region according to the texture information. Then the match region is weighted to fill the hole. Finally, a Gaussian filter is used to remove the noise in the depth map. Experimental results show that the proposed method can effectively repair the holes existing in the original depth map and get an accurate and smooth depth map, which can be used to render a virtual image with good quality.     

基于JNDD模型面向虚拟视绘制的快速深度图编码

为降低深度数据的编码复杂度并保证重建虚拟视点的质量,提出了一种基于JNDD模型面向虚拟视点绘制的快速深度图编码算法,引入最小可觉深度差模型,将深度图划分为对绘制失真敏感的竖直边缘区域与失真难以被人眼察觉的平坦区域,并相应地为编码过程中的宏块模式选择设计了两种搜索策略。实验结果表明,与JM编码方案相比,本文所提出的方法在保证虚拟视质量与编码码率基本不变的前提下,显著降低了编码复杂度,有助于在三维视频系统中提高深度编码模块的编码速度。     

面向虚拟视点图像绘制的深度图编码算法

针对不同区域对绘制虚拟视点图像质量产生不同的影响,以及深度估计不准确导致时域抖动效应影响压缩效率的问题,提出了一种面向虚拟视点图像绘制的深度图压缩算法。通过彩色图像帧差、深度图边缘提取等相关处理过程,提取深度图的静态区域、边缘区域以及动态区域。对深度图边缘区域使用了较低的量化系数,以提高深度图边缘区域编码质量;根据深度图各个区域的编码模式特点,仅对部分编码模式而不是所有模式进行率失真优化搜索,以提高深度图的编码速度;对于深度图P帧的静态区域,合理地采用了SKIP模式,以消除由于深度估计算法的局限性导致时域抖动效应对深度图压缩的影响。实验结果表明,与传统的H.264编码方案相比,本文方案在传输码流大小基本不变的前提下提高了最终虚拟视点图像边缘区域的绘制质量,其余区域主观质量相近,而深度图编码时间则节省了约77～87%。     

Depth image enlargement using an evolutionary approach

Accurate depth map at high resolution is required in many 3D video display concepts. Given a low-resolution depth map, this paper studies how to enhance its resolution with a registered high-resolution color image. The idea of the proposed approach is that pixels with similar color values and small distances should have similar depth values. Therefore, the known depth values in input depth map can be propagated to estimate the unknown depth values of their neighboring pixels with similar color values and small distances in high-resolution depth map. Different from conventional approaches, the proposed approach utilizes the ant colony optimization (ACO) technique to dispatch artificial ants moving on a coupled graph, which consists of a depth map and a color image. Then these artificial ants propagate the known depth information from the observed low-resolution depth map to its up-sampled counterpart. Experimental results show that the proposed approach achieves high-resolution depth map with more desirable quality than that of several conventional approaches.     

Fast depth map mode decision based on depth–texture correlation and edge classification for 3D-HEVC

The 3D extension of High Efficiency Video Coding (3D-HEVC) has been adopted as the emerging 3D video coding standard to support the multi-view video plus depth map (MVD) compression. In the joint model of 3D-HEVC design, the exhaustive mode decision is required to be checked all the possible prediction modes and coding levels to find the one with least rate distortion cost in depth map coding. Furthermore, new coding tools (such as depth-modeling mode (DMM) and segment-wise depth coding (SDC)) are exploited for the characteristics of depth map to improve the coding efficiency. These achieve the highest possible coding efficiency to code depth map, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In this paper, we propose a fast depth map mode decision algorithm for 3D-HEVC by jointly using the correlation of depth map-texture video and the edge information of depth map. Since the depth map and texture video represent the same scene at the same time instant (they have the same motion characteristics), it is not efficient to use all the prediction modes and coding levels in depth map coding. Therefore, we can skip some specific prediction modes and depth coding levels rarely used in corresponding texture video. Meanwhile, the depth map is mainly characterized by sharp object edges and large areas of nearly constant regions. By fully exploiting these characteristics, we can skip some prediction modes which are rarely used in homogeneity regions based on the edge classification. Experimental results show that the proposed algorithm achieves considerable encoding time saving while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.     

Key-frame-based depth propagation for semi-automatic stereoscopic video conversion

In this paper, we propose a key-frame-based bi-directional depth propagation algorithm for semi-automatic 2D-to-3D stereoscopic video conversion. First, key-frames are identified from each video shot based on color motion-compensation errors to prevent high-motion content between any pair of consecutive key frames. Depths for key-frames are manually assigned or rendered by popular computer tools, and then bi-directionally propagated to non-key-frames there between. Our depth propagation algorithm is featured of a multi-pass error correcting procedure for each frame to prevent depth artifacts from being further propagated to adjacent frames. Our proposed algorithm is advantageous in solving the background occlusion/dis-occlusion problem that degrades the performances of traditional depth propagation algorithms. Experimental results show that our scheme is capable of achieving better results against three prior algorithms in view of the qualities of the estimated depth map (e.g., dis-occluded background and object boundaries) and the synthesized stereo views.     

Stereoscopic image generation based on depth images for 3D TV

A depth-image-based rendering system for generating stereoscopic images is proposed. One important aspect of the proposed system is that the depth maps are pre-processed using an asymmetric filter to smoothen the sharp changes in depth at object boundaries. In addition to ameliorating the effects of blocky artifacts and other distortions contained in the depth maps, the smoothing reduces or completely removes newly exposed (disocclusion) areas where potential artifacts can arise from image warping which is needed to generate images from new viewpoints. The asymmetric nature of the filter reduces the amount of geometric distortion that might be perceived otherwise. We present some results to show that the proposed system provides an improvement in image quality of stereoscopic virtual views while maintaining reasonably good depth quality.     

Macroblock Level Bits Allocation for Depth Maps in 3-D Video Coding

For 3-D videos, one commonly used representation method is texture videos plus depth maps for several selected viewpoints, whereas the other viewpoints are synthesized based on the available texture videos and depth maps with the depth-image-based rendering (DIBR) technique. As both the quality of the texture videos and depth maps will affect the quality of the synthesized views, bits allocation for the depth maps become indispensable. The existing bits allocation approaches are either inaccurate or requiring pre-encoding and analyzing in temporal dimension, making them unsuitable for the real-time applications. Motivated by the fact that different regions of the depth maps have different impacts on the synthesized image quality, a real-time macroblock level bits allocation approach is proposed, where different macroblocks of the depth maps are encoded with different quantization parameters and coding modes. As the bits allocation granularity is fine, the R-D performance of the proposed approach outperforms other bits allocation approaches significantly, while no additional pre-encoding delay is caused. Specifically, it can save more than 10% overall bit rate comparing with Morvan’s full search approach, while maintaining the same synthesized view quality.     

Texture aided depth frame interpolation

Recent development of depth acquiring technique has accelerated the progress of 3D video in the market. Utilizing the acquired depth, arbitrary view frames can be generated based on depth image based rendering (DIBR) technique in free viewpoint video system. Different from texture video, depth sequence is mainly utilized for virtual view generation rather than viewing. Inspired by this, a depth frame interpolation scheme using texture information is proposed in this paper. The proposed scheme consists of a texture aided motion estimation (TAME) and texture aided motion compensation (TAMC) to fully explore the correlation between depth and the accompanying textures. The optimal motion vectors in TAME and the best interpolation weights in TAMC are respectively selected taking the geometric mapping relationship between depth and the accompanying texture frames into consideration. The proposed scheme is able to not only maintain the temporal consistency among interpolated depth sequence but also improve the quality of virtual frames generated by interpolated depth. Besides, it can be easily applied to arbitrary motion compensation based frame interpolation scheme. Experimental results demonstrate that the proposed depth frame interpolation scheme is able to improve the quality of virtual view texture frames in both subjective and objective criterions compared with existing schemes.