新一代基于HEVC的3D视频编码技术

HEVC标准出台后,新一代基于HEVC的多视点加深度编码也将正式推出。基于HEVC的3D视频编码作为HEVC标准的扩展部分,主要面向立体电视和自由立体视频。从该编码方式的基本结构出发,较全面地介绍了视频编码方式、深度图编码方式和对深度图的编码控制三个方面的关键技术,包括视点间运动预测、深度图建模模式和视点合成优化等技术。     

Comparative study of frame-compatible stereo 3D services and a novel method for spatial interleaving using HEVC

There are a number of ways to realize frame-compatible stereo 3D services, which includes the spatial and temporal interleaving based methods. We first present the detailed analysis of these methods with respect to compression efficiency, backward compatibility and the ability to re-use existing infrastructure. Simulations have been set up to benchmark the Rate–Distortion (R–D) performance of these options with the state-of-the-art Multiview Video Coding (MVC) and results show that temporal interleaving based method performs better when compared to spatial interleaving in both H.264/AVC as well as in HEVC compression schemes, with MVC still outperforms both the methods. In the case of the spatial interleaving method using HEVC, we propose to encapsulate the two views in different tiles with filters turned off across the tile boundary. Simulation results show that the proposed method, when compared to “no-tiles” case, improves decoder performance by approximately 50% with no reduction in R–D performance.     

An adaptive quantization algorithm without side information for depth map coding

This paper presents a novel block-adaptive quantization scheme for efficient bit allocation without side information in depth map coding. Since the type of distortion in a depth map causes different effects in terms of the visual artifacts in a synthesized view, the proposed method adaptively assigns the number of bits according to the characteristics of the corresponding texture block. I have studied the details of the depth map and its rendered view distortion, modeled these analytically, and then proposed a new rate and distortion model for depth map coding. Finally, I derived a simple closed-form solution based on my proposed rate and distortion model, which determines the block-adaptive quantization parameter without any side information. Experimental results show that the proposed scheme can achieve coding gains of more than 0.6% and 1.4% for quarter- and full-resolution depth maps, respectively, in a multi-view-plus-depth 3D system.     

Multiview coding and error correction coding for 3D video over noisy channels

We consider the joint source–channel coding problem of stereo video transmitted over AWGN and flat Rayleigh fading channels. Multiview coding (MVC) is used to encode the source, as well as a type of spatial scalable MVC. Our goal is to minimize the total number of bits, which is the sum of the number of source bits and the number of forward error correction bits, under the constraints that the quality of the left and right views must each be greater than predetermined PSNR thresholds at the receiver. We first consider symmetric coding, for which the quality thresholds are equal. Following binocular suppression theory, we also consider asymmetric coding, for which the quality thresholds are unequal. The optimization problem is solved using both equal error protection (EEP) and a proposed unequal error protection (UEP) scheme. An estimate of the expected end-to-end distortion of the two views is formulated for a packetized MVC bitstream over a noisy channel. The UEP algorithm uses these estimates for packet rate allocation. Results for various scenarios, including non-scalable/scalable MVC, symmetric/asymmetric coding, and UEP/EEP, are provided for both AWGN and flat Rayleigh fading channels. The UEP bit savings compared to EEP are given, and the performances of different scenarios are compared for a set of stereo video sequences.     

Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

HEVC-based 3D holoscopic video coding using self-similarity compensated prediction

Holoscopic imaging, also known as integral, light field, and plenoptic imaging, is an appealing technology for glassless 3D video systems, which has recently emerged as a prospective candidate for future image and video applications, such as 3D television. However, to successfully introduce 3D holoscopic video applications into the market, adequate coding tools that can efficiently handle 3D holoscopic video are necessary. In this context, this paper discusses the requirements and challenges for 3D holoscopic video coding, and presents an efficient 3D holoscopic coding scheme based on High Efficiency Video Coding (HEVC). The proposed 3D holoscopic codec makes use of the self-similarity (SS) compensated prediction concept to efficiently explore the inherent correlation of the 3D holoscopic content in Intra- and Inter-coded frames, as well as a novel vector prediction scheme to take advantage of the peculiar characteristics of the SS prediction data. Extensive experiments were conducted, and have shown that the proposed solution is able to outperform HEVC as well as other coding solutions proposed in the literature. Moreover, a consistently better performance is also observed for a set of different quality metrics proposed in the literature for 3D holoscopic content, as well as for the visual quality of views synthesized from decompressed 3D holoscopic content.     

面向3D呈现的有损和无损混合深度视频编码

针对多个深度视频流提出实时的压缩方法和评估 方案,采用一种有损无损混合的编码 方法,可以在图像质量和压缩率之间提供一种有效的平衡控制。行程编码(RLE)用于无损 压 缩,用来保存深度图像素的高位;像素低10bits保存在YUV图像的Y通道,直接使用×264编码。 实验表明,所提方法可以在平均时间小于12ms的前提下同时编解码 多个深度图。在实时传输 中,通过动态调节质量控制级别,可以获取4∶1到20∶ 1的压缩率。在压缩率10∶1的情况下,主观解码3D重构 效果与原始图几乎完全一致。     

多视点与深度视频编码技术研究综述

多视点与深度视频(MVD)可以有效表示自由视点视频,减少了需传输视点的个数,该表达形式正受到越来越多的关注,如何对MVD进行高效编码尤为重要.介绍了自由视点视频的发展现状及国内外研究概况,详细讨论了深度图像的高效编码技术及多视点与深度联合编码技术,并对多视点与深度视频编码技术进行了总结及展望.     

Exploiting temporal information and adaptive thresholding for fast mode decision in the H264 video coding standard

A new algorithm for fast mode decision in the H264 video coding standard is presented in this paper. The algorithm exploits temporal information and adaptive thresholding and can provide significant computational savings with similar or even better Rate Distortion performance as compared to accepted standard contributions and other techniques found in the literature.     

Early merge mode decision for texture coding in 3D-HEVC

As the upcoming 3D video coding standard, high efficiency video coding (HEVC) based 3D video coding (3D-HEVC) has been drafted. In 3D-HEVC, the computational complexity of mode decision process is significantly high due to exhaustive modes’ checks for coding units (CU) derived from recursive quad-tree partitioning. In this paper, we propose an early merge mode decision method for complexity reduction of dependent texture views. First, inter-view correlation and hierarchical depth correlation of coding modes are separately analyzed for B frame and P frame. Then, conditions to early determine merge mode coded CUs are derived based on the correlations. All of the early determined CUs only check merge modes in the mode decision process, which brings considerable complexity reduction. Experimental results demonstrate that the proposed method can achieve average 20.4% of encoding time saving for dependent texture views with negligible rate distortion performance loss.     

Edge-preserving Intra mode for efficient depth map coding based on H.264/AVC

Depth-image-based-rendering (DIBR) algorithms for 3D video communication systems based on the “multi-view video plus depth” format are very sensitive to the accuracy of depth information. Specifically, edge regions in the depth data should be preserved in the coding/decoding process to ensure good view synthesis performance, which directly affects the overall system performance. This paper proposes a novel scheme for edge-aware Intra depth compression based on the H.264/AVC framework enabled on both Intra (I) and Inter (P) slices. The proposed scheme includes a new Intra mode specifically targeted to depth macroblocks with arbitrarily shaped edges, which are typically not predicted well by the standard Intra modes of H.264/AVC and result in high rate–distortion costs. The proposed algorithm segments edge macroblocks into two regions each approximated by a flat surface. A binary mask identifying the two regions is defined and encoded by means of context-coding with adaptive template selection. As a novel contribution, the proposed mode allows exploiting the correlation with causal neighboring edge macroblocks to improve the performance of context-coding of binary masks and allow significant bit rate savings. The proposed method has been exhaustively compared with different state-of-the-art algorithms for edge-aware depth coding and the results highlight significant improvements in most of the cases, both in terms of reconstructed depth quality, view synthesis performance, and overall texture plus depth rate–distortion performance.     

Mode dependent down-sampling and interpolation scheme for high efficiency video coding

In this paper, a mode dependent down-sampling and interpolation scheme is proposed to improve the coding efficiency of the intra prediction module. In the proposed method, we elaborately design the down-sampling structures and interpolation schemes for each directional intra prediction mode by minimizing the spatial prediction distance. The sampled pixels are predicted with a traditional directional intra prediction scheme, and the non-sampled pixels are predicted from the interpolation of their neighboring reconstructed sampling pixels. Both the residuals of the sampled and non-sampled pixels are encoded at last. Experimental results show that the proposed method achieves an average 7.52% bitrate reduction relative to KTA reference software. Since the down-sampling structure and interpolation method is only related to the intra mode, there is no additional overhead at the encoder.     

3D visual experience oriented cross-layer optimized scalable texture plus depth based 3D video streaming over wireless networks

3D video streaming over the mobile Internet generally incurs the inferior 3D visual experience due to the time-varying characteristics of wireless channel. The conventional video streaming optimization methods generally neglect the harmony among different networking protocol layers. This paper proposes a cross-layer optimized texture plus depth based scalable 3D video streaming method to improve the expected 3D visual experience of the user by systematically considering the application layer texture-video/depth/FEC bit-rate allocation, MAC layer multi-channel allocation, and physical layer modulation and channel coding scheme (MCS) selection. In the cross-layer optimization, a networking-related 3D visual experience model which fuses the overlapped retinal view visual quality and depth sensation with mimicking human vision system is established to predict the 3D visual experience under the specific parameter configurations of different protocol layers. The efficiency and effectiveness of the proposed cross-layer optimized 3D video streaming method has been validated by subjective and objective experimental results.     

Disparity field and depth map coding for multiview 3D image generation

In the present paper techniques are examined for the coding of the depth map and disparity fields for stereo or multiview image communication applications. It is assumed that both the left and right channels of the multiview image sequence are coded using block- or object-based methods. A dynamic programming algorithm is used to estimate a disparity field between each stereo image pair. Depth is then estimated and occlusions are optionally detected, based on the estimated disparity fields. Spatial interpolation techniques are examined based on the disparity/depth information and the detection of occluded regions using either stereoscopic or trinocular camera configurations. It is seen that the presence of a third camera at the transmitter site improves the estimation of disparities, the detection of occlusions and the accuracy of the resulting spatial interpolation at the receiver. Various disparity field and depth map coding techniques are then proposed and evaluated, with emphasis given to the quality of the resulting intermediate images at the receiver site. Block-based and wireframe modeling techniques are examined for the coding of isolated depth or disparity map information. Further, 2D and 3D motion compensation techniques are evaluated for the coding of sequences of depth or disparity maps. The motion fields needed may be available as a byproduct of block-based or object-based coding of the intensity images. Experimental results are given for the evaluation of the performance of the proposed coding and spatial interpolation methods.     

Adaptive intra-refresh for low-delay error-resilient video coding

Low-delay and error-resilient video coding is critical for real-time video communication over wireless networks. Intra-refresh coding, which embeds intra coded regions into inter frames can achieve a relatively smooth bit-rate and terminate the error propagation caused by the transmission loss. In this paper, we proposed a novel linear model for the intra-refresh cycle-size selection adapting to the network packet loss rates and the motions in the video content. We also analyze issues in designing the intra-refresh coding pattern and the refresh order, and propose a strategy which can adapt to different cycle-size and obtain better R–D performance compared with traditional random intra-refresh and vertical-partition intra-refresh. Experimental results show that the linear cycle-size selection model works effectively, where a 3 dB improvement can be achieved compared with a fixed cycle-size. Also, with the proposed intra-refresh order, a 2.0% bitrate reduction is obtained in average compared with the vertical-partition intra-refresh.     

基于H.264编码算法的高速CCD视频压缩系统设计

为了满足航空大面阵CCD相机视频数据高速、实时传输和存储的要求,本文设计了一种基于H.264视频编码算法的压缩系统。整个压缩系统分为CCD前端、视频压缩、视频显示、视频压缩码流存储以及压缩分析单元,视频压缩单元采用高性能视频专用DSP处理器TMS320DM642,软件平台采用在CCS3.1上使用C语言实现H.264压缩算法。为了使压缩算法高效快速的运行,本文使用了DSP/BIOS资源来管理软硬件工作。 为了高速交互数据,采用了EDMA高速搬运数据策略,进而保证了数据实时传输的需要。实验结果表明,本文提出的压缩系统可以稳定正常的工作,具有良好压缩性能,在压缩比40:1~10:1范围内,平均信噪比高于35dB,满足了航空CCD相机应用的需求。     

Dynamic macroblock wavefront parallelism for parallel video coding

Wavefront parallelism is effective for parallel video encoding thanks to its merits of low latency, no quality loss and high degree of parallelism. In traditional video encoders, macroblock row wavefront (MRW) parallelism was widely adopted. However the performance of MRW is limited by workload unbalance and computing resource unbalance among multiple work threads. This paper proposes a new dynamic macroblock wavefront (DMW) parallelism to alleviate the limitations of MRW. In DMW, the available macroblocks (MBs) are scheduled to work threads MB by MB other than MB row by MB row; and after one MB is encoded by a work thread, the MB on its right (if available) has the highest priority to be scheduled to avoid synchronization delay. Experimental results demonstrate that video encoder with traditional MRW can be accelerated by more than 10% with our proposed DMW. When hyper-threading is used, the advantage of DMW is more prominent.     

Motion adaptive video coding scheme for time‐varying network

Adaptive video coding algorithms can encode a video stream dynamically on the basis of the amount of bandwidth available in a network. In this paper, a novel approach for adaptive video coding based on look‐up tables has been proposed. On the basis of the network conditions, the proposed codec estimates quantization parameter (QP) and also the spatial resolution for a group of pictures from the look‐up table. Then, QP for each frame is estimated on the basis of the motion content in each frame. More motion leads to more burden of bits while encoding the motion vectors. Also, quantization noise is less prominent in high‐motioned frames than frames possessing low motion information because of the temporal masking phenomenon of the human eyes. So, the main trick of the proposed scheme is to assign higher QP to the frames having higher motion than those having less motion. This method also reduces the requirement of excess bits for encoding of motion vectors having higher motion content. Additionally, QP is adjusted on the basis of the buffer availability in order to avoid bit loss error. Experimental results show the efficacy of the proposed codec. Copyright © 2013 John Wiley & Sons, Ltd.     

Joint video/depth rate allocation for 3D video coding based on view synthesis distortion model

Joint video/depth rate allocation is an important optimization problem in 3D video coding. To address this problem, this paper proposes a distortion model to evaluate the synthesized view without access to the captured original view. The proposed distortion model is an additive model that accounts for the video-coding-induced distortion and the depth-quantization-induced distortion, as well as the inherent geometry distortion. Depth-quantization-induced distortion not only considers the warping error distortion, which is described by a piecewise linear model with the video power spectral property, but also takes into account the warping error correlation distortion between two sources reference views. Geometry distortion is approximated from that of the adjacent view synthesis. Based on the proposed distortion model, a joint rate allocation method is proposed to seek the optimal trade-off between video bit-rate and depth bit-rate for maximizing the view synthesis quality. Experimental results show that the proposed distortion model is capable of approximately estimating the actual distortion for the synthesized view, and that the proposed rate allocation method can almost achieve the identical rate allocation performance as the full-search method at less computational cost. Moreover, the proposed rate allocation method consumes less computational cost than the hierarchical-search method at high bit-rates while providing almost the equivalent rate allocation performance.     

基于FPGA的三维视频系统实时深度估计

深度估计是基于视频加深度图像的三维视频系统中前端预处理的核心技术,其主要技术难题包括准确性、实时处理和大分辨率深度图获取等。本文提出一种实时深度估计的硬件实现方案,主要解决处理速度问题,并兼顾了准确性和大分辨率问题。本方案采用单片FPGA实现深度估计,其中采用census变换与SAD(Sum of Absolute Differences)混合的算法进行逐点匹配得到稠密深度图。硬件设计充分利用FPGA的大规模并行能力,并采用流水线设计提高数据通路的数据吞吐量,提升整个设计的时钟频率。实验表明,所提出的方案可实现全高清(1 920×1 080)分辨率视频实时深度估计。为了支持大分辨率图像并能观测距离相机较近的物体深度,本文方案视差搜索范围可以达到240pixels,帧率最高可达69.6fps,达到了实时和高清的处理目的。