High efficiency depth image-based rendering with simplified inpainting-based hole filling

Hole and crack filling is the most important issue in depth-image-based rendering (DIBR) algorithms for generating virtual view images when only one view image and one depth map are available. This paper proposes a priority patch inpainting algorithm for hole filling in DIBR algorithms by generating multiple virtual views. A texture-based interpolation method is applied for crack filling. Then, an inpainting-based algorithm is applied patch by patch for hole filling. A prioritized method for selecting the critical patch is also proposed to reduce computation time. Finally, the proposed method is realized on the compute unified device architecture parallel computing platform which runs on a graphics processing unit. Simulation results show that the proposed algorithm is 51-fold faster for virtual view synthesis and achieves better virtual view quality compared to the traditional DIBR algorithm which contains depth preprocessing, warping, and hole filling.     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

Discontinuity-based registration of depth and video data in depth image based rendering

Depth image based rendering (DIBR) has been proposed to create content for 3D-TV. In DIBR, stereoscopic images are created from monoscopic images and associated depth data. Since for most of the available video content sensor depth data are lacking, methods to create artificial depth data for video content have been developed. Yet artificial as well as sensor depth data may contain misalignments with respect to video data. Misaligned depth data are a source of artifacts observable in rendered 3D views. We show that by using an edge-based registration method, the spatial alignment of depth and video data can be improved, leading to an alleviation of the observed artifacts.     

基于卷积神经网络的虚拟视点空洞填补

为了提高虚拟视点绘制中空洞填补的质量,本文提 出一种基于卷积神经网络的虚拟视点空洞填补算 法,包括图像预处理、特征提取和空洞填补三部分。首先,对双视点虚拟视点图像进行预处 理,即利用深 度图像绘制(Depth Image Based Rendering,DIBR)技术得到空洞填补前的图像和空洞掩膜 ;然后,通过卷积 神经网络对虚拟视点图像提取多维特征,针对空洞区域,使用结构相似性损失函数优化网络 ;最后,利用 提取的特征填补虚拟视点中的空洞。实验结果表明,本文算法能较好地保持虚拟视点图像前 景和背景的锐 利边缘,主观视觉感知效果佳,同时,对不同场景测试序列的客观评价指标PS NR和SSIM都能取得满意结果,该方法能有效地填补虚拟视点中的空洞。     

基于光场结构特性与多视点匹配的深度估计

针对现有光场图像深度估计技术无法均衡地对主要对象和背景进行深度估计的问题,提出了一种基于光场结构特性与多视点匹配的深度估计方法。该方法在光场结构特性引导的深度估计的基础上,为了实现光场图像深度变化区域的平滑过渡,同时又考虑光场图像具有多视点子孔径图像阵列的特点,采用多视点匹配优化光场图像深度估计。在马尔可夫随机域中,基于光场结构特性构建深度估计平滑项,同时联合多视点匹配构建深度估计数据项,并进行全局深度迭代优化,从而有效平衡对象深度边界和背景深度估计,提高光场图像深度估计的性能。实验结果表明,所提出的方法能够得到更加清晰的深度边界,同时可以修正背景中不准确的深度值,获得高质量的深度估计结果。     

Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

Improved interview video error concealment on whole frame packet loss

An improved DIBR-based (Depth image based rendering) whole frame error concealment method for multiview video with depth is designed. An optimal reference view selection is first proposed. The paper further includes three modified parts for the DIBRed pixels. First, the missing 1-to-1 pixels are concealed by the pixels from another view. The light differences between views are taken care of by the information of the motion vector of the projected coordination and a reverse DIBR procedure. Second, the generation of the many-to-1 pixels is improved via their depth information. Third, the hole pixels are found using the estimated motion vectors derived efficiently from a weighted function of the neighboring available motion vectors and their distance to the target hole pixel. The experimental results show that, compared to the state-of-the-art method, the combined system of the four proposed methods is superior and improves the performance by 5.53 dB at maximum.     

Depth map misalignment correction and dilation for DIBR view synthesis

The quality of the synthesized views by Depth Image Based Rendering (DIBR) highly depends on the accuracy of the depth map, especially the alignment of object boundaries of texture image. In practice, the misalignment of sharp depth map edges is the major cause of the annoying artifacts at the disoccluded regions of the synthesized views. Conventional smooth filter approach blurs the depth map to reduce the disoccluded regions. The drawbacks are the degradation of 3D perception of the reconstructed 3D videos and the destruction of the texture in background regions. Conventional edge preserving filter utilizes the color image information in order to align the depth edges with color edges. Unfortunately, the characteristics of color edges and depth edges are very different which causes annoying boundaries artifacts in the synthesized virtual views. Recent solution of reliability-based approach uses reliable warping information from other views to fill the holes. However, it is not suitable for the view synthesis in video-plus-depth based DIBR applications. In this paper, a new depth map preprocessing approach is proposed. It utilizes Watershed color segmentation method to correct the depth map misalignment and then the depth map object boundaries are extended to cover the transitional edge regions of color image. This approach can handle the sharp depth map edges lying inside or outside the object boundaries in 2D sense. The quality of the disoccluded regions of the synthesized views can be significantly improved and unknown depth values can also be estimated. Experimental results show that the proposed method achieves superior performance for view synthesis by DIBR especially for generating large baseline virtual views.     

采用图像修复的基于深度图像复制

在传统的基于深度图像复制(DIBR)的基础上提出一种基于图像修复的DIBR方法,将预处理深度图像和图像修复算法相结合来填补三维图像映射后的空洞。与传统方法相比更加灵活,本文方法仅需传输一路参考图像序列,从而有效降低DIBR系统的传输带宽。实验结果证明,本文所提出方法是有效的。     

An efficient depth image-based rendering with depth reliability maps for view synthesis

In this paper, an efficient depth image-based rending (DIBR) with depth reliability maps (DRM) is proposed to improve the quality of synthesized images. First, a DRM-based occlusion-aware approach is developed to obtain a segmentation mask, which can explicitly indicate where the information in an intermediate image should be blended preferably. Next, an improved weight model for view creation is introduced to enhance the quality of synthesized images. Finally, a distance and depth-based sub-pixel weighted (DDSPW) algorithm is presented to solve the visibility and resampling problems. Experimental results demonstrate that the treated DIBR schemes have better performance for view synthesis than the other three methods through the subjective visual perception and objective assessments in terms of peak signal to noise ratio and structural similarity index.     

Stereoscopic view synthesis based on region-wise rendering and sparse representation

Depth image-based rendering (DIBR), which is used to render virtual views with a color image and the corresponding depth map, is one of the key techniques in the 2D to 3D conversion process. One of the main problems in DIBR is how to reduce holes that occur on the generated virtual view images. In this paper, we make two main contributions to deal with the problem. Firstly, a region-wise rendering framework, which divides the original image regions into three special classes and renders each with optimal adaptive process respectively, is introduced. Then, a novel sparse representation-based inpainting method, which can yield visually satisfactory results with less computational complexity for high quality 2D to 3D conversion, is proposed. Numerical experimental results demonstrate the good performance of the proposed methods.     

零视差点重组在三维图像数字水印中的应用

在基于深度图像渲染（DIBR）水印方案中,利用深度图像和中央彩色图像通过像素平移渲染出左眼视图和右眼视图,嵌入到中央图像的水印信息会受到严重破坏,无法在左眼视图和右眼视图中正确提取水印信息。本文提出一种DIBR中零视差点的三维图像数字水印方法,利用深度值确定视差值,标定视差为零的点,重新组合成新的载体图像,进行水印嵌入。由于视差为零的像素点在生成左右眼视图时没有发生水平移动,所以此方法克服了DIBR过程中的像素平移带来的攻击。实验证明此方法能得到和二维水印一致的效果并且对于高斯噪声和JPEG压缩都具有良好的鲁棒性。      

基于背景光修正成像模型的水下图像复原

光在水下传播时由于受到水体吸收和散射作用的影响,导致水下图像质量严重退化。为了有效去除色偏和模糊,改善水下图像质量,该文提出一种基于背景光修正成像模型的水下图像复原方法。该方法基于对雾天图像的观察,提出了水下图像背景光偏移假设,并基于此建立背景光修正成像模型;随后使用单目深度估计网络获得场景深度的估计,并结合背景光修正的水下成像模型,利用非线性最小二乘拟合获得水下偏移分量的估计值从而实现水下图像去水;最后优化去水后的含雾图像的透射率,并结合修正后的背景光实现图像复原。实验结果表明,该文方法在恢复水下图像颜色和去除散射光方面效果良好。     

A Novel DIBR 3D Image Watermarking Algorithm Resist to Geometrical Attacks

A histogram shape based method is utilized to design watermarking algorithm to protect the Depthimage- based rendering (DIBR) 3D images. To make the watermarking method robust to common attacks, much more suitable pixel groups are selected for watermark embedding which are insensitive to geometric attacks. Taking into account the adjusting of baseline distance in DIBR process might affect the watermarking extraction, a postprocessing is proposed after DIBR process to make the proposed watermarking method much more robust to DIBR process. As the experimental results shown, the proposed method is much more robust to the geometric attacks and combined attacks compared with existing methods. In addition, the proposed watermarking method also has good robustness to the adjusting of baseline distance and depth image blurred.     

Multi-layer and Multi-scale feature aggregation for DIBR-Synthesized image quality assessment

Depth-Image-Based Rendering (DIBR) is one of the main fundamental techniques for generating new viewpoints in 3D video applications such as multi-viewpoint video (MVV), free viewpoint video (FVV) and virtual reality (VR). Due to the imperfections of color images, depth maps or texture restoration techniques, several types of distortions occur in synthesized views. However, most of related works evaluated the quality of DIBR-synthesized views by only detecting a specific type of distortion, such as stretching, black holes, blurring, etc., which were unable to accurately evaluate the quality of DIBR-synthesized views. In this paper, a new no-reference image quality assessment method is proposed to evaluate the quality of DIBR-synthesized images by combining multi-layer and multi-scale features of images. To be specific, the distortions introduced by different stages of virtual viewpoint synthesis are first analyzed, and then multi-layer and multi-scale features are extracted to estimate the degree of texture and structure distortions. As a result, individual quality scores associated with two types of distortions (e.g., structural distortion and texture distortion) are aggregated to an overall image quality. Experimental results on two publicly available DIBR datasets show that the method has better performance than the state-of-the-art models.Index Terms: image quality assessment, DIBR-synthesized image, distortion correction, BIQA.     

构造全局背景的虚拟视点合成算法

虚拟视点合成技术是三维电视、多视点视频等诸多领域中的关键技术。目前,基于深度图像的绘制(DIBR)受到了广泛的关注,而这一技术的主要难点在于如何降低映射过程中产生的空洞。为解决这个问题,提高合成图像的质量,同时降低所需参考视点的数量,本文提出了一种基于全局背景图像的虚拟视点合成算法。首先,我们利用视频序列的帧间互补信息生成一张全局背景图像,并以此为辅助图像,去填补从参考视点映射到虚拟目标视点的各帧图像。与传统方法相比,本文算法能使用空洞区域对应的真实像素进行填充,提高了图像的质量。最后,对填补后仍余留的空洞采用图像修复算法进行处理,得到最终的虚拟视点图像。实验结果表明,这个算法优于传统算法,并且只利用了一个参考视点。      

Depth Similarity Enhanced Image Summarization Algorithm for Hole-Filling in Depth Image-Based Rendering

In free viewpoint video （FVV） and 3DTV, the depth image-based rendering method has been put forward for rendering virtual view video based on multi-view video plus depth （MVD） format. However, the projection with slightly different perspective turns the covered background regions into hole regions in the rendered video. This paper presents a depth enhanced image summarization generation model for the hole-filling via exploiting the texture fidelity and the geometry consistency between the hole and the remaining nearby regions. The texture fidelity and the geometry consistency are enhanced by drawing texture details and pixel-wise depth information into the energy cost of similarity measure correspondingly. The proposed approach offers significant improvement in terms of 0.2dB PSNR gain, 0.06 SSIM gain and subjective quality enhancement for the hole-filling images in virtual viewpoint video.     

View-spatial–temporal post-refinement for view synthesis in 3D video systems

Depth image based rendering is one of key techniques to realize view synthesis for three-dimensional television and free-viewpoint television, which provide high quality and immersive experiences to end viewers. However, artifacts of rendered images, including holes caused by occlusion/disclosure and boundary artifacts, may degrade the subjective and objective image quality. To handle these problems and improve the quality of rendered images, we present a novel view-spatial–temporal post-refinement method for view synthesis, in which new hole-filling and boundary artifact removal techniques are proposed. In addition, we propose an optimal reference frame selection algorithm for a better trade-off between the computational complexity and rendered image quality. Experimental results show that the proposed method can achieve a peak signal-to-noise ratio gain of 0.94 dB on average for multiview video test sequences when compared with the benchmark view synthesis reference software. In addition, the subjective quality of the rendered image is also improved.     

Recent MPEG Standardization Activities on 3D Video Coding

The Moving Picture Experts Group(MPEG) has been developing a 3D video(3DV) coding standard for depth-based 3DV data representations,especially for multiview video plus depth(MVD) format.With MVD,depth-image-based rendering(DIBR) is used to synthesize virtual views that are based on a few transmitted pairs of texture and depth data.In this paper,we discuss ongoing 3DV standardization and summarize coding tools proposed in the responses to MPEG’s call for proposals on 3DV coding.     

A bilateral attention based generative adversarial network for DIBR 3D image watermarking

This paper presents a bilateral attention based generative adversarial network (BAGAN) for depth-image-based rendering (DIBR) 3D image watermarking to protect the image copyright. Convolutional block operations are employed to extract main image features for robust watermarking, but embedding watermark into some features will degrade image quality much. To relieve this kind of image distortion, the bilateral attention module (BAM) is utilized by mining correlations of the center view and the depth map to compute attention of the 3D image for guiding watermark to distribute over different image regions. Since a modality gap exists between the center view and the depth map, a cross-modal feature fusion module (CMFFM) is designed for BAM to bridge the cross-view gap. Because the depth map has lots of flat background information including many redundant features, to prune them, the depth redundancy elimination module (DREM) is used for cross-view feature fusion. In the decoder, two extractors with the same structure are built to recover watermark from the center view and the synthesized view, respectively. In addition, the discriminator is supposed to build a competitive relationship with the encoder to increase the image quality. The noise sub-network is used to train different image attacks for robustness. Extensive experimental results have demonstrated that the proposed BAGAN can obtain higher watermarking invisibility and robustness compared with existing DIBR 3D watermarking methods. Ablation experiments have also proven the effectiveness of DREM, CMFFM and BAM on BAGAN.