Fast motion and disparity estimation for HEVC based 3D video coding

The emerging international standard for high efficiency video coding (HEVC) based 3D video coding (3D-HEVC) is an extension of HEVC. In the test model of 3D-HEVC, variable size motion estimation (ME) and disparity estimation (DE) are both employed to select the best coding mode for each treeblock in the encoding process. This technique achieves the highest possible coding efficiency, but it brings extremely high computational complexity which limits 3D-HEVC from practical applications. In this paper, a fast ME/DE algorithm based on inter-view and spatial correlations is proposed to reduce 3D-HEVC computational complexity. Since the multi-view videos represent the same scene with similar characteristic, there is a high correlation among the coding information from inter-view prediction. Besides, the homogeneous regions in texture video have a strong spatial correlation, and thus spatially neighboring treeblocks have similar coding information. Therefore, we can determine ME search range and skip some specific ME and DE rarely used in the previously coded view frames and spatially neighboring coding unit. Experimental results demonstrate that the proposed algorithm can significantly reduce computational complexity of 3D-HEVC encoding while maintaining almost the same rate-distortion performance.     

Fast depth map mode decision based on depth–texture correlation and edge classification for 3D-HEVC

The 3D extension of High Efficiency Video Coding (3D-HEVC) has been adopted as the emerging 3D video coding standard to support the multi-view video plus depth map (MVD) compression. In the joint model of 3D-HEVC design, the exhaustive mode decision is required to be checked all the possible prediction modes and coding levels to find the one with least rate distortion cost in depth map coding. Furthermore, new coding tools (such as depth-modeling mode (DMM) and segment-wise depth coding (SDC)) are exploited for the characteristics of depth map to improve the coding efficiency. These achieve the highest possible coding efficiency to code depth map, but also bring a significant computational complexity which limits 3D-HEVC from real-time applications. In this paper, we propose a fast depth map mode decision algorithm for 3D-HEVC by jointly using the correlation of depth map-texture video and the edge information of depth map. Since the depth map and texture video represent the same scene at the same time instant (they have the same motion characteristics), it is not efficient to use all the prediction modes and coding levels in depth map coding. Therefore, we can skip some specific prediction modes and depth coding levels rarely used in corresponding texture video. Meanwhile, the depth map is mainly characterized by sharp object edges and large areas of nearly constant regions. By fully exploiting these characteristics, we can skip some prediction modes which are rarely used in homogeneity regions based on the edge classification. Experimental results show that the proposed algorithm achieves considerable encoding time saving while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.     

Multi-view video coding with view interpolation prediction for 2D camera arrays

An efficient compression algorithm for multi-view video sequences, which are captured by two-dimensional (2D) camera arrays, is proposed in this work. First, we propose a novel prediction structure, called three-dimensional hierarchical B prediction (3DHBP), which can efficiently reduce horizontal inter-view redundancies, vertical inter-view redundancies, and temporal redundancies in multi-view videos. Second, we develop a view interpolation scheme based on the bilateral disparity estimation. The interpolation scheme yields high quality view frames by adapting disparity estimation and compensation procedures using the information in neighboring frames. Simulation results demonstrate that the proposed multi-view video coding algorithm provides significantly better rate–distortion (R–D) performance than the conventional algorithm, by employing the 3DHBP structure and using interpolated view frames as additional reference frames.     

应用于多视点视频压缩的多参考B帧快速编码算法

基于H.264平台就多视点视频压缩中多参考帧技术存在计算复杂度高的现状,根据多视点视频序列特性,提出了一种针对多参考B帧的快速多视点视频编码压缩算法。新算法改进了多参考帧列表策略,并结合多视点视频序列的时空域相关性特性,提出了适用于多视点视频编码的参考帧列表调整策略。此外,还针对多种块模式编码引入了块模式预判策略。新算法在保证率失真性能的基础上,有效地降低了计算复杂度,提高了多视点视频编码速度。虽然本文是基于M-Picture编码结构实现了新算法,但它同样适用于其它应用多参考B帧预测技术的多视点视频编码结构中。     

基于3D视频的错误隐藏技术分析

随着3D视频产品的逐步问世,3D视频技术逐渐成为当前研究的热点。由于3D视频编码是利用多个视点的空间、时间及视点间的相关性进行压缩编码,导致当3D视频遭遇不可靠信道差错时,其受损的影响比传统的2D视频更严重。因此,研究基于3D视频的差错控制技术十分必要,针对3D视频的3种视频格式,初步探讨了目前已经出现或正在研究中的错误隐藏技术。     

Advanced side information creation techniques and framework for Wyner–Ziv video coding

Recently, several distributed video coding (DVC) solutions based on the distributed source coding (DSC) paradigm have appeared in the literature. Wyner–Ziv (WZ) video coding, a particular case of DVC where side information is made available at the decoder, enable to achieve a flexible distribution of the computational complexity between the encoder and decoder, promising to fulfill novel requirements from applications such as video surveillance, sensor networks and mobile camera phones. The quality of the side information at the decoder has a critical role in determining the WZ video coding rate-distortion (RD) performance, notably to raise it to a level as close as possible to the RD performance of standard predictive video coding schemes. Towards this target, efficient motion search algorithms for powerful frame interpolation are much needed at the decoder. In this paper, the RD performance of a Wyner–Ziv video codec is improved by using novel, advanced motion compensated frame interpolation techniques to generate the side information. The development of these type of side information estimators is a difficult problem in WZ video coding, especially because the decoder only has available some reference, decoded frames. Based on the regularization of the motion field, novel side information creation techniques are proposed in this paper along with a new frame interpolation framework able to generate higher quality side information at the decoder. To illustrate the RD performance improvements, this novel side information creation framework has been integrated in a transform domain turbo coding based Wyner–Ziv video codec. Experimental results show that the novel side information creation solution leads to better RD performance than available state-of-the-art side information estimators, with improvements up to 2 dB; moreover, it allows outperforming H.264/AVC Intra by up to 3 dB with a lower encoding complexity.     

基于AVS+实时编码的多核并行视频编码算法

面向3D高清的国家广电行业标准AVS+较大地提高了编码效率,但也增加了编码复杂度,并行编码技术是解决高复杂度编码器的有效方法。该文改变了传统视频编码的反馈环,提出一种新的视频并行编码框架,有效地实现了编码的并行化;并将该框架应用于AVS+实时编码器中,实现了一种基于AVS+实时编码的并行算法。实验结果表明,该算法充分发挥多核处理器的运算能力,在编码性能损失极小,编码延迟可控的条件下极大地提高了视频编码速率。     

嵌入式平台下的安全媒体传输系统

Security video communication is a challenging task, especially for wireless video applications. An efficient security multimedia system on embedded platform is designed. By analyzing the hardware architecture and resource, the efficient DSP-based H.264/AVC coding is studied by efficient video coding techniques and system optimizing implementation.To protect the confidentiality and integrity of media information, a novel security mechanism is presented, which includes user identify authentication and a perceptual video encryption algorithm based on exploiting the special feature of entropy coding in H.264. Experimental results show that the proposed hardware framework has high performance and achieves a better balance between security and efficiency. The proposed security mechanism can achieve high security and low complexity cost, and has a little effect on the compression ratio and transmission bandwidth. What’s more, encoding and encryption at the same time, the performance of data process can meet real-time application.     

Coding-oriented multi-view video color correction

Color inconsistency between views is an important problem to be solved in multi-view video applications, such as free viewpoint television and other three-dimensional video systems. In this paper, by combining with multi-view video coding, a coding-oriented multi-view video color correction method is proposed. We first separate foreground and background in first Group Of Pictures （GOP） by using SKIP coding mode. Then by transferring means and standard deviations in backgrounds, color correction is performed for each frame in GOP, and multi-view video coding is performed and used to renew the backgrounds. Experimental results ances in color correction and multi-view video show the proposed method can obtain better performcoding.     

基于多核稀疏编码的三维人体姿态估计

为了准确有效的重构多视角图像中的三维人体姿态,该文提出一种基于多核稀疏编码的人体姿态估计算法.首先,针对连续帧姿态估计的歧义问题,该文设计了一种用于表达多视角图像的HA-SIFT描述子,其中,人体局部拓扑、肢体相对位置及外观信息被同时编码;然后,在多核学习框架下建立同时考虑特征空间内在流形结构与姿态空间几何信息的目标函数,并在希尔伯特空间优化目标函数以更新稀疏编码、过完备字典与多核权值;最后,利用姿态字典原子的线性组合来估计对应未知输入的三维人体姿态.实验结果表明,与核稀疏编码、Laplace稀疏编码及Bayesian稀疏编码相比,文本方法具有更高的估计精度.     

基于3D ResNet-LSTM的多视角人体动作识别方法

在基于视频图像的动作识别中,由于固定视角相机所获取的不同动作视频存在视角差异,会造成识别准确率降低等问题。使用多视角视频图像是提高识别准确率的方法之一,提出基于三维残差网络（3D Residual Network,3D ResNet）和长短时记忆（Long Short-term Memory,LSTM）网络的多视角人体动作识别算法,通过3D ResNet学习各视角动作序列的融合时空特征,利用多层LSTM网络继续学习视频流中的长期活动序列表示并深度挖掘视频帧序列之间的时序信息。在NTU RGB+D 120数据集上的实验结果表明,该模型对多视角视频序列动作识别的准确率可达83.2%。     

3D多视点立体显示及其关键技术

作为基于 DTV/HDTV 的二维(2D)显示之后的下一代视频显示技术,三维(3D)多视点立体显示已成为国际上的研究热点之一.为建立多视点立体显示系统,阐述了相关的关键技术,包括:光场表示模型和光场获取系统、高效的与现行视频标准兼容的多视点编码和传输方法、解码端任意位置视点的高效绘制方法、3D显示技术以及多视点自由立体显示.针对上述关键技术,分析了当前国际上的发展趋势及存在的问题,同时提出了一种基于交互式自由立体显示的 3D 视频处理系统的解决方案.     

An efficient rate–distortion optimization method for dependent view in MV-HEVC based on inter-view dependency

Rate–distortion optimization (RDO) is utilized to select the optimal coding parameters in multi-view video coding (MVC), which employs a Lagrange multiplier to balance the relationship between the distortion and the bitrate. In this paper, an efficient RDO method for the dependent view (DV) in multi-view video (MVV) is proposed based on inter-view dependency. First of all, by investigating the sources of the distortion in the DV, a new distortion model for the DV is established. In addition, based on the proposed distortion model, an efficient Lagrangian multiplier decision for B frame is proposed by considering the inter-view dependency. Finally, the optimized Lagrangian multiplier for P frame is designed using the scaling factor which is deduced to have a linear relationship with the disparity between I frame and P frame. Experiment results demonstrate that compared with the original HTM-16.0 encoder, the proposed overall method reduces 12.19% BD-rate for the DV on average, bringing 0.40 dB BD-PSNR gain.     

基于多视点伪序列的光场图像压缩

近年来，作为一种能够提供更富有沉浸感的多媒体媒质，光场图像（Light Field Image，LFI）引起广泛的关注。针对光场图像数据量巨大的问题，本文提出了一种基于多视点伪序列的光场图像高效压缩方案。在编码端，所提方法首先将光场相机捕获得到的原始光场图像根据相机的微透镜阵列分解成子孔径图像。接着根据子孔径图像存在较强视点内和视点间相关性，选取部分子孔径图像进行多视点伪序列构建，基于MV-HEVC设计适用于多视点伪序列的预测编码结构进行编码。在解码端，所提方法基于已解码多视点伪序列通过视频帧插值方法重建出未编码传输的子孔径视图，从而重建出全部光场图像。实验结果表明本文所提算法优于现有基于视差引导稀疏编码的光场图像压缩方法，BD-rate平均节约18.5%，BD-PSNR平均提高1.28dB。      

Fast color correction for multi-view video by modeling spatio-temporal variation

In multi-view video, a number of cameras capture the same scene from different viewpoints. Color variations between the camera views may deteriorate the performance of multi-view video coding or virtual view rendering. In this paper, a fast color correction method for multi-view video is proposed by modeling spatio-temporal variation. In the proposed method, multi-view keyframes are defined to establish the spatio-temporal relationships for accurate and fast implementation. For keyframes, accurate color correction is performed based on spatial color discrepancy model that disparity estimation is used to find correspondence points between views, and linear regression is performed on these sets of points to find the optimal correction coefficients. For non-keyframes, fast color correction is performed based on temporal variations model that time-invariant regions are detected to reflect the change trends of correction coefficients. Experimental results show that compared with other methods, the proposed method can promote the correction speed greatly without noticeable quality degradation, and obtain higher coding performance.     

Efficient coding of experimental holograms using speckle denoising

Lossy compression for Experimental Holograms (EH) and Computer-Generated Holograms (CGH) using standardized coding solutions is a highly efficient process provided that these solutions can be applied to the object plane. This compression efficiency reveals to be more relevant in CGH. Speckle noise mainly affects reconstructed EH, and to less extent reconstructed CGH. In the current work, the reduction of speckle noise of EH is proposed to improve the coding efficiency of the hologram compression scheme. The compression scheme defines a base layer where a 2D version of the object is coded with an image codec standard. When speckle noise reduction is performed before any compression, efficient compression is obtained for both CGH and EH. Since speckle noise reduction is performed only on amplitude data, without affecting the phase information of the reconstructed hologram, it is still possible to render 3D features such as depth map, multi-view or to recover holographic interference patterns for further 3D visualization.     

Adaptive Learning Based View Synthesis Prediction for Multi-View Video Coding

In the applications of Free View TV, pre-estimated depth information is available to synthesize the intermediate views as well as to assist multi-view video coding. Existing view synthesis prediction schemes generate virtual view picture only from interview pictures. However, there are many types of signal mismatches caused by depth errors, camera heterogeneity or illumination difference across views and these mismatches decrease the prediction capability of virtual view picture. In this paper, we propose an adaptive learning based view synthesis prediction algorithm to enhance the prediction capability of virtual view picture. This algorithm integrates least square prediction with backward warping to synthesize the virtual view picture, which not only utilizes the adjacent views information but also the temporal decoded information to adaptively learn the prediction coefficients. Experiments show that the proposed method reduces the bitrates by up to 18 % relative to the multi-view video coding standard, and about 11 % relative to the conventional view synthesis prediction method.     

基于多视点视频特征的3D-HEVC深度视频信息隐藏算法

为了更安全地传输秘密信息和保护3D视频,提 出基于多视点视频特征的3D-HEVC深度视频信息隐 藏算法。首先结合彩色视频的纹理特征,考虑深度视频不同区域对绘制视点的质量和编码效 率的影响,对 深度视频进行区域分割。然后针对不同的区域,采用不同的方式调制最大编码单元(LCU) 的QP值嵌入 秘密信息。最后,用修改后的QP值进行编码压缩,传输视频信息。实验仿真结果表明,相 比于原始HTM13.0, 本算法编码重建深度视频绘制视点质量的PSNR平均下降0.0015dB, 码率平均增加0.035%。本文算法 能较好地保证绘制视点的质量,对视频流的码率影响较小,且能实现秘密信息的盲提取。     

Sum-of-gradient based fast intra coding in 3D-HEVC for depth map sequence (SOG-FDIC)

As the latest video coding standard for multi-view plus depth video, 3D-HEVC yields high coding efficiency but at the cost of heavy computational complexity. To reduce the computational complexity, a fast intra coding algorithm based on sum-of-gradient criterion for depth map coding in 3D-HEVC, named SOG-FDIC, is proposed in this paper. Based on the observation that DMM modes and smaller partitioning sizes are rarely used in flat region, sum of gradient is presented to determine whether the current block belongs to the flat region so as to skip unnecessary checking of DMMs and smaller partitioning sizes. Experimental results show that the proposed algorithm can save about 21.8% coding time while keeping almost the same coding efficiency and the reconstructed video quality of depth maps and synthesized views, compared with the original 3D-HEVC. Moreover, it has been verified that the proposed method outperforms the state-of-the-art methods.     

多视点纹理加深度编码的联合码率控制方法

码率控制技术是多视点视频编码和传输中一个关键的问题。为了提高三维(3D)视频的整体显示质量,包括虚拟视点质量和编码视点质量,提出一种多视点纹理加深度编码的联合码率控制方法。该算法研究了纹理和深度的关系,采用基于模型方法确定最优的纹理和深度之间的码率比例。根据各个视点编码结果的统计规律,不同的视频序列采用不同的视点间比特分配比例。实验结果表明,与目前流行的多视点码率控制算法相比,该算法在计算复杂度基本保持不变的情况下,平均码率控制误差在0.6%以内,客观质量PSNR最高可提高0.65 dB。