Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline

Content production for stereoscopic 3D-TV displays has become mature in the past years while huge progress has also been achieved in the improvement of the image quality of glasses-free auto-stereoscopic displays and light-field displays. Concerning the latter two display families, the content production workflow is less elaborated and more complex, as the number of required views not only differs considerably but is also likely to increase in the near future. As a co-existence of all 3D display families can be expected for the next years, one aims to establish an efficient content production workflow which yields to high quality content for all 3D-TV displays.Against this background we present a real-time capable multi-view video plus depth (MVD) content production workflow based on a four-camera rig with mixed narrow and wide baseline. Results show the suitability of the approach to simultaneously produce high quality MVD4 and native stereoscopic 3D content.     

Improved interview video error concealment on whole frame packet loss

An improved DIBR-based (Depth image based rendering) whole frame error concealment method for multiview video with depth is designed. An optimal reference view selection is first proposed. The paper further includes three modified parts for the DIBRed pixels. First, the missing 1-to-1 pixels are concealed by the pixels from another view. The light differences between views are taken care of by the information of the motion vector of the projected coordination and a reverse DIBR procedure. Second, the generation of the many-to-1 pixels is improved via their depth information. Third, the hole pixels are found using the estimated motion vectors derived efficiently from a weighted function of the neighboring available motion vectors and their distance to the target hole pixel. The experimental results show that, compared to the state-of-the-art method, the combined system of the four proposed methods is superior and improves the performance by 5.53 dB at maximum.     

采用图像修复的基于深度图像复制

在传统的基于深度图像复制(DIBR)的基础上提出一种基于图像修复的DIBR方法,将预处理深度图像和图像修复算法相结合来填补三维图像映射后的空洞。与传统方法相比更加灵活,本文方法仅需传输一路参考图像序列,从而有效降低DIBR系统的传输带宽。实验结果证明,本文所提出方法是有效的。     

Semi-automatic 2D-to-3D video conversion based on background sprite generation

This paper presents a technique for semi-automatic 2D-to-3D stereo video conversion, which is known to provide user intervention in assigning foreground/background depths for key frames and then get depth maps for non-key frames via automatic depth propagation. Our algorithm treats foreground and background separately. For foregrounds, kernel pixels are identified and then used as the seeds for graph-cut segmentation for each non-key frame independently, resulting in results not limited by objects’ motion activity. For backgrounds, all video frames, after foregrounds being removed, are integrated into a common background sprite model (BSM) based on a relay-frame-based image registration algorithm. Users can then draw background depths for BSM in an integrated manner, thus reducing human efforts significantly. Experimental results show that our method is capable of retaining more faithful foreground depth boundaries (by 1.6–2.7 dB) and smoother background depths than prior works. This advantage is helpful for 3D display and 3D perception.     

基于非局部随机游走和运动补偿的2D转3D优化方法

D转3D技术可以从2D资源中获取深度信息,以满足3D显示对3D内容的需求。针 对2D转 3D深度估计中的深度优化问题,提出一种基于非局部随机游走(NRW)和运动补偿的 深度优化算 法。本文方法在采用NRW和移动双边滤波(SBF)获得关键帧和非关键帧深度图的基础上,为 了锐化非关 键帧深度序列对象边界,结合纹理信息利用NRW算法优化深度图,同时又考虑相邻帧间的时 域信息,采 用运动补偿的方法对非关键帧深度序列进行优化,获得高质量的深度视频序列。实验结果表 明,本文方法可以得到对象边界更加准确的深度视频估计结果。     

分布式2D视频到3D视频转换系统

2D-to-3D video conversion is a feasible way to generate 3D programs for the current 3DTV industry. However, for large-scale 3D video production, current systems are no longer ade-quate in terms of the time and labor required for conversion. In this paper, we introduce a distributed 2D-to-3D video conversion system that includes a 2D-to-3D video conversion module, architecture of the parallel computation on the cloud, and 3D video cod-ing in the system. The system enables coope-ration among multiple users in the simultane-ous completion of their conversion tasks so that the conversion efficiency is greatly pro-moted. In the experiments, we evaluate the system based on criteria related to both time consumption and video coding performance.     

基于最大可容忍深度失真模型的低复杂度深度视频编码

在保证虚拟视点绘制质量的前提下,如何降低深 度视频的编码复杂度是一个亟需解决的问题。本文提出了一 种基于最大可容忍深度失真(MTDD,maximum tolerable d epth distortion)模型的低复杂度深度编码算法,将MTDD模型引 入 到率失真(RD)代价函数,通过对编码单元(CU)进行失真度量判决 ,如果对当前CU进 行编码不会导致可察觉的绘制失真,则失真度量只考虑深度编码失真,否则失真度量同时考 虑深度编码失真和视点 合成失真(VSD),从而降低编码复杂度。实验结果表明,所 提出的算法在不 降低虚拟视点绘制质量和不增加编码码率的情况下,能显著降低深度视频的编码复杂度。     

A depth perception and visual comfort guided computational model for stereoscopic 3D visual saliency

With the emerging development of three-dimensional (3D) related technologies, 3D visual saliency modeling is becoming particularly important and challenging. This paper presents a new depth perception and visual comfort guided saliency computational model for stereoscopic 3D images. The prominent advantage of the proposed model is that we incorporate the influence of depth perception and visual comfort on 3D visual saliency computation. The proposed saliency model is composed of three components: 2D image saliency, depth saliency and visual comfort based saliency. In the model, color saliency, texture saliency and spatial compactness are computed respectively and fused to derive 2D image saliency. Global disparity contrast is considered to compute depth saliency. Particularly, we train a visual comfort prediction function to distinguish stereoscopic image pair as high comfortable stereo viewing (HCSV) or low comfortable stereo viewing (LCSV), and devise different computational rules to generate a visual comfort based saliency map. The final 3D saliency map is obtained by using a linear combination and enhanced by a “saliency-center bias” model. Experimental results show that the proposed 3D saliency model outperforms the state-of-the-art models on predicting human eye fixations and visual comfort assessment.     

基于FPGA的三维视频系统实时深度估计

深度估计是基于视频加深度图像的三维视频系统中前端预处理的核心技术,其主要技术难题包括准确性、实时处理和大分辨率深度图获取等。本文提出一种实时深度估计的硬件实现方案,主要解决处理速度问题,并兼顾了准确性和大分辨率问题。本方案采用单片FPGA实现深度估计,其中采用census变换与SAD(Sum of Absolute Differences)混合的算法进行逐点匹配得到稠密深度图。硬件设计充分利用FPGA的大规模并行能力,并采用流水线设计提高数据通路的数据吞吐量,提升整个设计的时钟频率。实验表明,所提出的方案可实现全高清(1 920×1 080)分辨率视频实时深度估计。为了支持大分辨率图像并能观测距离相机较近的物体深度,本文方案视差搜索范围可以达到240pixels,帧率最高可达69.6fps,达到了实时和高清的处理目的。     

A bilateral attention based generative adversarial network for DIBR 3D image watermarking

This paper presents a bilateral attention based generative adversarial network (BAGAN) for depth-image-based rendering (DIBR) 3D image watermarking to protect the image copyright. Convolutional block operations are employed to extract main image features for robust watermarking, but embedding watermark into some features will degrade image quality much. To relieve this kind of image distortion, the bilateral attention module (BAM) is utilized by mining correlations of the center view and the depth map to compute attention of the 3D image for guiding watermark to distribute over different image regions. Since a modality gap exists between the center view and the depth map, a cross-modal feature fusion module (CMFFM) is designed for BAM to bridge the cross-view gap. Because the depth map has lots of flat background information including many redundant features, to prune them, the depth redundancy elimination module (DREM) is used for cross-view feature fusion. In the decoder, two extractors with the same structure are built to recover watermark from the center view and the synthesized view, respectively. In addition, the discriminator is supposed to build a competitive relationship with the encoder to increase the image quality. The noise sub-network is used to train different image attacks for robustness. Extensive experimental results have demonstrated that the proposed BAGAN can obtain higher watermarking invisibility and robustness compared with existing DIBR 3D watermarking methods. Ablation experiments have also proven the effectiveness of DREM, CMFFM and BAM on BAGAN.     

Efficient multiple-example based super-resolution for symmetric mixed resolution stereoscopic video coding

In this paper, we first propose a new symmetric mixed resolution stereoscopic video coding (SMRSVC) model which can provide clear bitrate-reduction and visual merits. Based on the newly proposed SMRSVC model, we then propose a quality-efficient multiple-example based super-resolution method. In the proposed super-resolution method, the four block examples selected from the forward and backward key-frames, the reference super-resolved frame, and the interview super-resolved frame are referred so as to effectively fuse the high frequency component of the super-resolved current block of the downsampled non-key-frame, and then an enhanced super-resolved non-key-frame is followed. Based on six test stereoscopic video sequences, the experimental results demonstrate that besides the bitrate-saving effect, the proposed super-resolution method for the proposed SMRSVC model also has better quality performance in terms of six well-known quality metrics when compared with several state-of-the-art methods for the previous asymmetric resolution stereoscopic video coding model and the SMRSVC model.     

基于逆向映射的空洞填补方法

为了有效填补虚拟视点图像中的公共空洞,提出 了一种基于逆向映射的空洞填补方法。 首先利用深度图像绘制(DIBR)技术将左、右参考视点映射到虚拟视点位置,利用图像膨胀方 法将映射的虚拟视图中的空 洞区域进行扩大,以消除虚拟视点图像中的伪影瑕疵;然后,提取出膨胀后空洞区域的边界 ,并将其逆映 射到原始的参考图像中,根据空洞与边界的相对位置,选取原始图像中相对位置上的像素来 填充虚拟视图 中的空洞区域;最后,将空洞填补之后的左、右视点映射的虚拟视图进行融合获得最终的虚 拟视图。实验 证明,本文方法有效解决了传统空洞填补方法容易将前景像素填充到背景区域的问题,能 够获得较好的视觉观看效果和较高的客观峰值信噪比(PSNR)值。     

结合预处理的深度视频帧内快速编码算法

多视点彩色加深度(MVD)视频是三维(3D)视频的 主流格式。在3D高效视频编码中,深度视频帧内编码 具有较高的编码复杂度;深度估计软件获取的深度视频由于不够准确会使深度图平坦 区域纹理增加, 从而进一步增加帧内编码复杂度。针对以上问题,本文提出了一种联合深度处理的深度视频 帧内低复杂度 编码算法。首先,在编码前对深度视频进行预处理,减少由于深度图不准确而出现的纹理信 息;其次,运 用反向传播神经网络(BPNN,backpropagation neural network)预测最大编码单元 (LCU,la rgest coding unit)的最大划分深度;最后联合深度视频的边缘信 息及对应的彩色LCU最大划分深度进行CU提前终止划分和快速模式选取。实验结果表明, 本文算法在保证 虚拟视点质量的前提下,BDBR下降0.33% ,深度视频编码时间平均节省50.63%。     

用于多视视频加深度的错误隐藏算法

提出了一种针对多视视频加深度(MVD)的错误隐藏( EC)算法。算法充分利用MVD特有的深度 信息及当前丢失宏块周围正确解码的宏块信息,将丢失宏块分为3类不同属性的宏块。针对3类宏块的特 点,分别提出了基于候选运动矢量修正(CMVR)、基于深度的外边界匹配(DOBMA)以及自适应 权 值的EC(AWEC)等模式。实验表明,本文提出的算法在保证相同的视频主客观质量情况下, 能够快速有效地实现EC。     

Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model

Joint video/depth rate allocation is an important optimization problem in 3D video coding. To address this problem, this paper proposes a distortion model to evaluate the synthesized view without access to the captured original view. The proposed distortion model is an additive model that accounts for the video-coding-induced distortion and the depth-quantization-induced distortion, as well as the inherent geometry distortion. Depth-quantization-induced distortion not only considers the warping error distortion, which is described by a piecewise linear model with the video power spectral property, but also takes into account the warping error correlation distortion between two sources reference views. Geometry distortion is approximated from that of the adjacent view synthesis. Based on the proposed distortion model, a joint rate allocation method is proposed to seek the optimal trade-off between video bit-rate and depth bit-rate for maximizing the view synthesis quality. Experimental results show that the proposed distortion model is capable of approximately estimating the actual distortion for the synthesized view, and that the proposed rate allocation method can almost achieve the identical rate allocation performance as the full-search method at less computational cost. Moreover, the proposed rate allocation method consumes less computational cost than the hierarchical-search method at high bit-rates while providing almost the equivalent rate allocation performance.     

Scalable 3D video streaming over P2P networks with playback length changeable chunk segmentation

3D video distribution over P2P networks has been thought as a promising way for 3D video entering home. The convergence of scalable 3D video coding and P2P streaming can provide diverse 3D experiences for heterogeneous clients with high distribution efficiencies. However, the conventional chunk segmentation and scheduling algorithms originally aiming at the non-scalable 2D video streaming are not very efficient for scalable 3D video streaming over P2P networks due to the particular data characteristics of scalable 3D video. Based on this motivation, this paper first presents a playback length changeable 3D video chunk segmentation (PLC3DCS) algorithm to provide different error resilience strengths to video and depth as well as layers with different importance levels in the 3D video transmission. Then, a hybrid-priority based chunk scheduling (HPS) algorithm is proposed to be tied in with the proposed chunk segmentation algorithm to further promote the overall 3D video P2P streaming performance. The simulation results show that the proposed PLC3DCS algorithm with the corresponding HPS can increase the success delivery rates of chunks with more important levels, and further improve the user’s quality of 3D experience.     

Estimation of accurate effective loss rate for FEC video transmission

We propose an algorithm for adjusting data transmission parameters, such as the packet size and the code rate of forward error correction (FEC), to obtain maximum video quality under dynamic channel conditions. When determining transmission parameters, it is essential to calculate an accurate effective loss rate that reflects FEC recovery failures and over-deadline packets. To this end, we analyze the delays caused by FEC coding and the potential packet size variations. In our analysis, we consider the effect of delayed transmission of video packets incurred by the parity packets as well as the encoder and decoder buffers. With the analysis reflecting the delay effect, we are able to accurately estimate the delay patterns of all video packets. Based on the analysis results, we establish an accurate model for estimating the effective loss rate. Simulations show that the proposed effective loss rate model accurately estimates the effective loss rate and significantly improves the reconstructed video quality at the receiver.