Interactive 3-D Video Representation and Coding Technologies

Interactivity in the sense of being able to explore and navigate audio-visual scenes by freely choosing viewpoint and viewing direction, is an important key feature of new and emerging audio-visual media. This paper gives an overview of suitable technology for such applications, with a focus on international standards, which are beneficial for consumers, service providers, and manufacturers. We first give a general classification and overview of interactive scene representation formats as commonly used in computer graphics literature. Then, we describe popular standard formats for interactive three-dimensional (3-D) scene representation and creation of virtual environments, the virtual reality modeling language (VRML), and the MPEG-4 BInary Format for Scenes (BIFS) with some examples. Recent extensions to MPEG-4 BIFS, the Animation Framework eXtension (AFX), providing advanced computer graphics tools, are explained and illustrated. New technologies mainly targeted at reconstruction, modeling, and representation of dynamic real world scenes are further studied. The user shall be able to navigate photorealistic scenes within certain restrictions, which can be roughly defined as 3-D video. Omnidirectional video is an extension of the planar two-dimensional (2-D) image plane to a spherical or cylindrical image plane. Any 2-D view in any direction can be rendered from this overall recording to give the user the impression of looking around. In interactive stereo two views, one for each eye, are synthesized to provide the user with an adequate depth cue of the observed scene. Head motion parallax viewing can be supported in a certain operating range if sufficient depth or disparity data are delivered with the video data. In free viewpoint video, a dynamic scene is captured by a number of cameras. The input data are transformed into a special data representation that enables interactive navigation through the dynamic scene environment.     

面向三维高效视频编码的深度图错误隐藏

基于多视点视频序列视点内、视点间存在的相关性,并结合视点间运动矢量共享技术,该文提出一种面向3维高效视频编码中深度序列传输丢包的错误隐藏算法。首先,根据3D高效视频编码(3D-HEVC)的分层B帧预测(HBP)结构和深度图纹理特征,将深度图丢失块分成运动块和静止块;然后,对于受损运动块,使用结合纹理结构的外边界匹配准则来选择相对最优的运动/视差矢量进行基于位移矢量补偿的错误掩盖,而对受损静止块采用参考帧直接拷贝进行快速错误隐藏;最后,使用参考帧拆分重组来获取新的运动/视差补偿块对修复质量较差的重建块进行质量提升。实验结果表明：相较于近年提出的对比算法,该文算法隐藏后的深度帧平均峰值信噪比(PSNR)能提升0.25～2.03 dB,结构相似度测量值(SSIM)能提升0.001～0.006,且修复区域的主观视觉质量与原始深度图更接近。     

Convex Optimization-Based Bit Allocation for Video Coding

This paper presents a new convex optimization-based bit allocation framework for video coding. Motivated by the statistical analysis of the relationship between the rate and the corresponding distortion in typical video sequences, an improved empirical rate-distortion model with more flexibility in explaining the experimental observations is proposed. The convexity and monotonicity properties of the proposed model allow us to formulate the bit allocation problem as a convex programming problem, which can be solved using interior-point methods. Coupled with the region of interest (ROI) functionality and the technique of MB classification, the proposed bit allocation scheme is then applied to the problem of frame-level bit allocation for various video coding standards employing motion compensated hybrid DCT/DPCM technique. Similar bit allocation scheme is also developed for MPEG-4 video object coding at the object level. The relevant model parameters are determined using previously encoded data by means of linear regression. Simulation results show that the proposed algorithms can achieve considerably better PSNR performance than the conventional approaches for the tested video sequences, at the expenses of slight increase in coding complexity.     

Bit Allocation for Interframe Video Coding Systems

In this work, we present a novel approach to the bit allocation problem that aims to minimize overall distortion subject to a bit rate constraint. The optimal solution can be found by the Lagrangian method with dynamic programming. However, the optimal bit allocation for block‐based interframe coding is practically unattainable because of the interframe dependency of macroblocks caused by motion compensation. To reduce the computational burden while maintaining a result close to the optimum, i.e., near optimum, we propose an alternative method. First, we present a partitioned form of the bit allocation problem: a “frame‐level problem” and “one‐frame macroblock‐level problems.” We show that the solution to this new form is also the solution to the conventional bit allocation problem. Further, we propose a bit allocation algorithm using a “two‐phase optimization technique” with an interframe dependency model and a rate‐distortion model.     

Implementation of Multiple Macroblock Mode Overlapped Block Motion Compensation for Wavelet Video Coding

This paper proposes a hybrid video coding framework based on discrete wavelet transform and multiple macroblock mode overlapped block motion compensation (OBMC). Studying the neighborhood in multiple macroblock mode situations, we set rules for the selection of neighboring blocks in OBMC. Based on experimental results, the raised cosine window and the best neighboring blocks are chosen for their outstanding performance. The multiple macroblock mode OBMC has a good performance in reducing the blocking effect, and the coding efficiency is improved. The proposed scheme can provide a 1 dB coding gain compared to MPEG-4, and its performance is close to the latest H.264 standard. From a subjective point of view, the new framework is also better than the standards.     

3D视频生成与编码的时间一致性深度图估计

In this paper, we propose a new algorithm for temporally consistent depth map estimation to generate three-dimensional video. The proposed algorithm adaptively computes the matching cost using a temporal weighting function, which is obtained by block-based moving object detection and motion estima-tion with variable block sizes. Experimental results show that the proposed algorithm im-proves the temporal consistency of the depth video and reduces by about 38% both the flic-kering artefact in the synthesized view and the number of coding bits for depth video coding.     

关于组合多针图复原物体3D表面形状的研究

光度立体系统很易确定物体表面任意点方向及相对深度，但不能确定绝对深度，为确定绝对深度，本文提出的算法首先利用双目光度立体系统的每个摄象机分别获得一幅针图，并求出左、右两针图内对应区域的视差。然后，经适当组合及精确的匹配，重建３－Ｄ物体表面深度。这一方法对进一步开发３－Ｄ表面深度重建及复原景物结构探索性研究有着十分重要的意义。     

多视点纹理加深度编码的联合码率控制方法

码率控制技术是多视点视频编码和传输中一个关键的问题。为了提高三维(3D)视频的整体显示质量,包括虚拟视点质量和编码视点质量,提出一种多视点纹理加深度编码的联合码率控制方法。该算法研究了纹理和深度的关系,采用基于模型方法确定最优的纹理和深度之间的码率比例。根据各个视点编码结果的统计规律,不同的视频序列采用不同的视点间比特分配比例。实验结果表明,与目前流行的多视点码率控制算法相比,该算法在计算复杂度基本保持不变的情况下,平均码率控制误差在0.6%以内,客观质量PSNR最高可提高0.65 dB。     

采用灰度共生矩阵进行深度预判的3D-HEVC深度图帧内快速编码算法

针对3D视频的3D-HEVC编码标准以多视点纹理视频和深度视频格式进行编码,其深度图编码仍延续纹理视频编码的模式和编码尺寸遍历选择,使得3D-HEVC的编码复杂度居高不下。本文针对深度图帧内预测编码,采用灰度共生矩阵对深度图中的CTU进行计算,统计并分析其矩阵中非零值个数与CTU分割深度的关系,根据非零值个数分布规律,设定阈值,使得帧内编码时可以预判编码模块的分割深度,从而选择性跳过部分不同深度CU的帧内预测过程。经过HTM16.0测试平台的检验,本算法在全帧内编码模式下,测试序列合成视点比特率仅增加0.08%的同时,平均节省了16.8%的编码时间,与其他同类较新算法在HTM16.0平台上的性能比较也有一定的优势。      

基于深度信息的立体视频错误隐藏方法

针对双视加深度结构的立体视频的网络传输,该文提出了解码端的错误检测和错误隐藏技术。结合深度信息对发生错误宏块的编码模式进行估计,在错误宏块周围相邻宏块的编码模式不一致时,选择邻近宏块中深度与待重建的错误宏块深度最为接近的宏块的编码模式作为其编码模式。然后,根据编码模式选择用视点内或视点间的相关性信息重建错误宏块。实验结果表明采用该方法重建的图像客观质量峰值信噪比(PSNR)较传统的边界匹配错误隐藏方法提高了0.8-6.0 dB,并且主观质量也有提升。     

多视点与深度视频编码技术研究综述

多视点与深度视频(MVD)可以有效表示自由视点视频,减少了需传输视点的个数,该表达形式正受到越来越多的关注,如何对MVD进行高效编码尤为重要.介绍了自由视点视频的发展现状及国内外研究概况,详细讨论了深度图像的高效编码技术及多视点与深度联合编码技术,并对多视点与深度视频编码技术进行了总结及展望.     

一种低延迟的3维高效视频编码中深度建模模式编码器

为了更好地对3D视频中深度图进行编码,该文将3维高效视频编码(3D-HEVC)标准新引入了深度建模模式(DMMs),新模式在提高了编码质量的同时改进了原有算法的复杂度。在设计DMM-1编码器电路时,传统架构电路的编码周期均较长,只能满足较低分辨率和帧率的视频实时编码要求。为了进一步提高3D-HEVC中DMM-1编码器的性能,该文对DMM-1算法架构进行了研究,针对其中楔形块评估无数据相关性的特点,提出了一种5级流水线架构的DMM-1编码器硬件电路,以期能够降低一个深度块编码所需的编码周期,并使用Verilog HDL进行实现。实验表明：该架构与Sanchez等人(2017年)的工作相比,以电路门数增加约1568门为代价,可减少至少52.3%的编码周期。     

关于融合多针图确定表面绝对深度的误差分析与研究

本文利用ＢＰＳ系统首先产生景物的一对表面方向图，然后：１．分割表面方向图为与景物内物体对应的孤立区域；２．匹配两针图内对应区域间的视差；３．确定物体的３Ｄ表面绝对深度，最后，着重指出怎样识别以下几种主要误差：（１）表面方向测量误差；（２）重建深度误差；（３）极线与扫描线非平行误差等，并且讨论对这些误差进行定量分析与校正的方法。     

Fast Depth Video Coding Method Using Adaptive Edge Classification

In this paper, we propose a fast-mode decision algorithm for both intra and inter prediction in depth videos. The proposed algorithm can reduce the computational complexity of the depth video coding. According to the depth variation, depth video can be classified into depth-continuity and depth-discontinuity regions. From experiments, we determine a threshold value for classifying these regions. Since the depth-continuity region has an imbalance in the mode distribution, we limit the mode candidates to reduce the complexity of the mode decision process. Experimental results show that our proposed algorithm reduces the encoding time by up to 78 % and 97 % for the intra and inter frames, respectively, compared to JMVC 8.3, with negligible PSNR loss and bitrate increase.     

3D-HEVC深度图像帧内编码单元划分快速算法

针对3维高性能视频编码(3D-HEVC)中深度图像帧内编码单元(Coding Unit, CU)划分复杂度高的问题,该文提出一种基于角点和彩色图像的自适应快速CU划分算法。首先利用角点算子,并根据量化参数选取一定数目的角点,以此进行CU的预划分;然后联合彩色图像的CU划分对预划分的CU深度级进行调整;最后依据调整后的CU深度级,缩小当前CU的深度级范围。实验结果表明,与原始3D-HEVC的算法相比,该文所提算法平均减少了约63%的编码时间;与只基于彩色图像的算法相比,该文的算法减少了约13%的编码时间,同时降低了约3%的平均比特率,有效地提高了编码效率。     

可伸缩性视频编码的码率分配算法

本文研究了渐进的精细的可伸缩性(Progressive Fine Granular Scalable,PFGS)视频编码的增强层码流的码率分配问题,它是由PFGS各帧的增强层码流长度不同引起的.我们首先建立了PFGS增强层的多帧联合的率失真(rate-distortion,R-D)函数,并通过MPEG-4标准测试序列检验了它的正确性.然后利用联合率失真函数给出了局部最优的码率分配算法,并进行了简化使得它的复杂性极低,非常适合流化视频(streaming video)服务器使用.实验结果显示本文提出的算法不仅使得各帧图像的解码质量比较平衡,而且将PFGS的编码效率提高了将近0.5dB.     

离散多频音调制中的子信道比特分配

讨论了ＡＤＳＬ传输中最有吸引力的一种调制技术－－离散多频音调制，给出了其中的关键技术即各子信道比特数的分配算法，并给出了简化算法例子。     

H.264并行视频编码的分配机制的研究

H.264作为新的视频压缩标准,采用了许多先进的算法,提供了很好的视频压缩质量。获得出色压缩效果和质量的代价是压缩编码算法复杂度的增加。为了寻求更高的编码速度,集群并行计算被运用到H.264的视频编码计算中。本文结合并行算法设计,通过对H.264视频编码的研究,分析了H.264可实现并行计算的任务单元选择;采用了片（Slice）级的H.264并行编码计算方法,并对其性能进行了测试和分析。     

Resource Allocation for Error Resilient Video Coding Over AWGN Using Optimization Approach

The number of slices for error resilient video coding is jointly optimized with 802.11a-like media access control and the physical layers with automatic repeat request and rate compatible punctured convolutional code over additive white gaussian noise channel as well as channel times allocation for time division multiple access. For error resilient video coding, the relation between the number of slices and coding efficiency is analyzed and formulated as a mathematical model. It is applied for the joint optimization problem, and the problem is solved by a convex optimization method such as the primal-dual decomposition method. We compare the performance of a video communication system which uses the optimal number of slices with one that codes a picture as one slice. From numerical examples, end-to-end distortion of utility functions can be significantly reduced with the optimal slices of a picture especially at low signal-to-noise ratio.      

基于帧内区域的细粒度可扩展视频编码最优码率分配算法

MPEG-4 FGS是MPEG-4中精细可扩展性编码方法,由于采用帧8*8交织的位平面编码结构,无法支持帧内区域级的动态码率分配,不适合基于内容的增强和流化传输,针对MPEG-4 FGS这编码限制,本文根据MPEG-7的内容描述标准,提出了一种基于帧内区域的细粒度可扩展A-FGS编码,实现了实时的帧内区域流化编码,并就该编码提出了一种最优码率分配算法.利用帧内区域8*8宏块高位平面非零系数很少的特点,提出了动态码率分配算法,使在中等码率范围内编码效率提高了0.5dB左右.