Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

Real-time generation of multi-view video plus depth content using mixed narrow and wide baseline

Content production for stereoscopic 3D-TV displays has become mature in the past years while huge progress has also been achieved in the improvement of the image quality of glasses-free auto-stereoscopic displays and light-field displays. Concerning the latter two display families, the content production workflow is less elaborated and more complex, as the number of required views not only differs considerably but is also likely to increase in the near future. As a co-existence of all 3D display families can be expected for the next years, one aims to establish an efficient content production workflow which yields to high quality content for all 3D-TV displays.Against this background we present a real-time capable multi-view video plus depth (MVD) content production workflow based on a four-camera rig with mixed narrow and wide baseline. Results show the suitability of the approach to simultaneously produce high quality MVD4 and native stereoscopic 3D content.     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

Improved interview video error concealment on whole frame packet loss

An improved DIBR-based (Depth image based rendering) whole frame error concealment method for multiview video with depth is designed. An optimal reference view selection is first proposed. The paper further includes three modified parts for the DIBRed pixels. First, the missing 1-to-1 pixels are concealed by the pixels from another view. The light differences between views are taken care of by the information of the motion vector of the projected coordination and a reverse DIBR procedure. Second, the generation of the many-to-1 pixels is improved via their depth information. Third, the hole pixels are found using the estimated motion vectors derived efficiently from a weighted function of the neighboring available motion vectors and their distance to the target hole pixel. The experimental results show that, compared to the state-of-the-art method, the combined system of the four proposed methods is superior and improves the performance by 5.53 dB at maximum.     

采用图像修复的基于深度图像复制

在传统的基于深度图像复制(DIBR)的基础上提出一种基于图像修复的DIBR方法,将预处理深度图像和图像修复算法相结合来填补三维图像映射后的空洞。与传统方法相比更加灵活,本文方法仅需传输一路参考图像序列,从而有效降低DIBR系统的传输带宽。实验结果证明,本文所提出方法是有效的。     

分布式2D视频到3D视频转换系统

2D-to-3D video conversion is a feasible way to generate 3D programs for the current 3DTV industry. However, for large-scale 3D video production, current systems are no longer ade-quate in terms of the time and labor required for conversion. In this paper, we introduce a distributed 2D-to-3D video conversion system that includes a 2D-to-3D video conversion module, architecture of the parallel computation on the cloud, and 3D video cod-ing in the system. The system enables coope-ration among multiple users in the simultane-ous completion of their conversion tasks so that the conversion efficiency is greatly pro-moted. In the experiments, we evaluate the system based on criteria related to both time consumption and video coding performance.     

A depth perception and visual comfort guided computational model for stereoscopic 3D visual saliency

With the emerging development of three-dimensional (3D) related technologies, 3D visual saliency modeling is becoming particularly important and challenging. This paper presents a new depth perception and visual comfort guided saliency computational model for stereoscopic 3D images. The prominent advantage of the proposed model is that we incorporate the influence of depth perception and visual comfort on 3D visual saliency computation. The proposed saliency model is composed of three components: 2D image saliency, depth saliency and visual comfort based saliency. In the model, color saliency, texture saliency and spatial compactness are computed respectively and fused to derive 2D image saliency. Global disparity contrast is considered to compute depth saliency. Particularly, we train a visual comfort prediction function to distinguish stereoscopic image pair as high comfortable stereo viewing (HCSV) or low comfortable stereo viewing (LCSV), and devise different computational rules to generate a visual comfort based saliency map. The final 3D saliency map is obtained by using a linear combination and enhanced by a “saliency-center bias” model. Experimental results show that the proposed 3D saliency model outperforms the state-of-the-art models on predicting human eye fixations and visual comfort assessment.     

Efficient multiple-example based super-resolution for symmetric mixed resolution stereoscopic video coding

In this paper, we first propose a new symmetric mixed resolution stereoscopic video coding (SMRSVC) model which can provide clear bitrate-reduction and visual merits. Based on the newly proposed SMRSVC model, we then propose a quality-efficient multiple-example based super-resolution method. In the proposed super-resolution method, the four block examples selected from the forward and backward key-frames, the reference super-resolved frame, and the interview super-resolved frame are referred so as to effectively fuse the high frequency component of the super-resolved current block of the downsampled non-key-frame, and then an enhanced super-resolved non-key-frame is followed. Based on six test stereoscopic video sequences, the experimental results demonstrate that besides the bitrate-saving effect, the proposed super-resolution method for the proposed SMRSVC model also has better quality performance in terms of six well-known quality metrics when compared with several state-of-the-art methods for the previous asymmetric resolution stereoscopic video coding model and the SMRSVC model.     

基于逆向映射的空洞填补方法

为了有效填补虚拟视点图像中的公共空洞,提出 了一种基于逆向映射的空洞填补方法。 首先利用深度图像绘制(DIBR)技术将左、右参考视点映射到虚拟视点位置,利用图像膨胀方 法将映射的虚拟视图中的空 洞区域进行扩大,以消除虚拟视点图像中的伪影瑕疵;然后,提取出膨胀后空洞区域的边界 ,并将其逆映 射到原始的参考图像中,根据空洞与边界的相对位置,选取原始图像中相对位置上的像素来 填充虚拟视图 中的空洞区域;最后,将空洞填补之后的左、右视点映射的虚拟视图进行融合获得最终的虚 拟视图。实验 证明,本文方法有效解决了传统空洞填补方法容易将前景像素填充到背景区域的问题,能 够获得较好的视觉观看效果和较高的客观峰值信噪比(PSNR)值。     

Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model

Joint video/depth rate allocation is an important optimization problem in 3D video coding. To address this problem, this paper proposes a distortion model to evaluate the synthesized view without access to the captured original view. The proposed distortion model is an additive model that accounts for the video-coding-induced distortion and the depth-quantization-induced distortion, as well as the inherent geometry distortion. Depth-quantization-induced distortion not only considers the warping error distortion, which is described by a piecewise linear model with the video power spectral property, but also takes into account the warping error correlation distortion between two sources reference views. Geometry distortion is approximated from that of the adjacent view synthesis. Based on the proposed distortion model, a joint rate allocation method is proposed to seek the optimal trade-off between video bit-rate and depth bit-rate for maximizing the view synthesis quality. Experimental results show that the proposed distortion model is capable of approximately estimating the actual distortion for the synthesized view, and that the proposed rate allocation method can almost achieve the identical rate allocation performance as the full-search method at less computational cost. Moreover, the proposed rate allocation method consumes less computational cost than the hierarchical-search method at high bit-rates while providing almost the equivalent rate allocation performance.     

Scalable 3D video streaming over P2P networks with playback length changeable chunk segmentation

3D video distribution over P2P networks has been thought as a promising way for 3D video entering home. The convergence of scalable 3D video coding and P2P streaming can provide diverse 3D experiences for heterogeneous clients with high distribution efficiencies. However, the conventional chunk segmentation and scheduling algorithms originally aiming at the non-scalable 2D video streaming are not very efficient for scalable 3D video streaming over P2P networks due to the particular data characteristics of scalable 3D video. Based on this motivation, this paper first presents a playback length changeable 3D video chunk segmentation (PLC3DCS) algorithm to provide different error resilience strengths to video and depth as well as layers with different importance levels in the 3D video transmission. Then, a hybrid-priority based chunk scheduling (HPS) algorithm is proposed to be tied in with the proposed chunk segmentation algorithm to further promote the overall 3D video P2P streaming performance. The simulation results show that the proposed PLC3DCS algorithm with the corresponding HPS can increase the success delivery rates of chunks with more important levels, and further improve the user’s quality of 3D experience.     

Estimation of accurate effective loss rate for FEC video transmission

We propose an algorithm for adjusting data transmission parameters, such as the packet size and the code rate of forward error correction (FEC), to obtain maximum video quality under dynamic channel conditions. When determining transmission parameters, it is essential to calculate an accurate effective loss rate that reflects FEC recovery failures and over-deadline packets. To this end, we analyze the delays caused by FEC coding and the potential packet size variations. In our analysis, we consider the effect of delayed transmission of video packets incurred by the parity packets as well as the encoder and decoder buffers. With the analysis reflecting the delay effect, we are able to accurately estimate the delay patterns of all video packets. Based on the analysis results, we establish an accurate model for estimating the effective loss rate. Simulations show that the proposed effective loss rate model accurately estimates the effective loss rate and significantly improves the reconstructed video quality at the receiver.     

Cross-layer optimized authentication and error control for wireless 3D medical video streaming over LTE

3D video for tele-medicine applications is gradually gaining momentum since the 3D technology can provide precise location information. However, the weak link for 3D video streaming is the necessary wireless link of the communication system. Neglecting the wireless impairments can severely degrade the performance of 3D video streaming that communicates complex critical medical data. In this paper, we propose systematic methodology for ensuring high performance of the 3D medical video streaming system. First, we present a recursive end-to-end distortion estimation approach for MVC (multiview video coding)-based 3D video streaming over error-prone networks by considering the 3D inter-view prediction. Then, based on the previous model, we develop a cross-layer optimization scheme that considers the LTE wireless physical layer (PHY). In this optimization, the authentication requirements of 3D medical video are also taken into account. The proposed cross-layer optimization approach jointly controls and manages the authentication, video coding quantization of 3D video, and the modulation and channel coding scheme (MCS) of the LTE wireless PHY to minimize the end-to-end video distortion. Experimental results show that the proposed approach can provide superior 3D medical video streaming performance in terms of peak signal-to-noise ratio (PSNR) when compared to state-of-the-art approaches that include joint source-channel optimized streaming with multi-path hash-chaining based-authentication, and also conventional video streaming with single path hash-chaining-based authentication.     

An adaptive quantization algorithm without side information for depth map coding

This paper presents a novel block-adaptive quantization scheme for efficient bit allocation without side information in depth map coding. Since the type of distortion in a depth map causes different effects in terms of the visual artifacts in a synthesized view, the proposed method adaptively assigns the number of bits according to the characteristics of the corresponding texture block. I have studied the details of the depth map and its rendered view distortion, modeled these analytically, and then proposed a new rate and distortion model for depth map coding. Finally, I derived a simple closed-form solution based on my proposed rate and distortion model, which determines the block-adaptive quantization parameter without any side information. Experimental results show that the proposed scheme can achieve coding gains of more than 0.6% and 1.4% for quarter- and full-resolution depth maps, respectively, in a multi-view-plus-depth 3D system.     

标准数字视盘(DVD)技术及发展前景(连载Ⅰ)

ＤＶＤ是近年来正在开发的数字化高技术产品，它集光、电、声、象于一体，广泛应用于电影、动画制作、音乐、多媒体等方面，其先进的数据压缩技术、广播级的图象质量、超高密度的光盘存储以及高性价比将给视听产业注入新的活力，形成国际范围的大市场。叙述了ＤＶＤ目前的发展概况，分析了与之相关的关键技术，简要介绍了ＤＶＤ所采用的ＭＰＥＧ－２压缩技术标准，最后还阐述了对国内开发ＤＶＤ的一些看法以及ＤＶＤ的前景展望。     

Window-based rate control for video quality optimization with a novel INTER-dependent rate-distortion model

Most model-based rate control schemes use independent rate-distortion (R–D) models at macroblock (MB) level to represent the relationship among bit rate, distortion and encoding complexity. However the correlations between frames (INTER-dependency) are not well considered for distortion, bit allocation and quantization parameter (QP) decision. In this paper, a novel INTER-dependent R–D model is proposed based on the theoretical analysis of the relationship between the predicted residual of one frame and the distortion of its reference frame. To achieve both bit rate accuracy and consistent video quality, a window-based rate control scheme with two sliding windows is introduced. One window is to group certain previously encoded frames and current frame to control the bit rate and buffer delay; the other is to group certain future encoding frames to optimize the fluctuation of video quality. Furthermore, the optimization of Lagrange multiplier is also discussed under the INTER-dependent situation. Experimental results demonstrate that the proposed window-based rate control scheme with INTER-dependent R–D model can achieve accurate target bit rate and improve PSNR performance, meanwhile the variation of PSNR is the smallest compared with other three benchmark algorithms. This one-pass rate control scheme is highly practical for the real-time video coding applications.     

机载圆周SAR成像技术研究

机载圆周合成孔径雷达(CSAR)作为一种新兴的成像模式,具有全方位观测、高空间分辨率和可三维成像等优点。随着CSAR成像技术的不断发展,现已逐渐成为对重点区域实施精确观测的有效手段之一。该文重点阐述了作者所在研究团队近年来在机载CSAR成像技术方面完成的研究工作,包括机载CSAR成像模型,空间分辨率评估,CSAR二维成像,基于单圆周CSAR的目标三维图像重构和多基线CSAR(HoloSAR)三维成像等技术,并给出了P, X两个频段机载CSAR的实测数据处理结果。已取得的研究成果证明了机载CSAR成像的有效性和实用性。该文主要内容基于作者2019年8月16日在“雷达学报第五届青年科学家论坛”上的学术报告。