多视点纹理加深度编码的联合码率控制方法

码率控制技术是多视点视频编码和传输中一个关键的问题。为了提高三维(3D)视频的整体显示质量,包括虚拟视点质量和编码视点质量,提出一种多视点纹理加深度编码的联合码率控制方法。该算法研究了纹理和深度的关系,采用基于模型方法确定最优的纹理和深度之间的码率比例。根据各个视点编码结果的统计规律,不同的视频序列采用不同的视点间比特分配比例。实验结果表明,与目前流行的多视点码率控制算法相比,该算法在计算复杂度基本保持不变的情况下,平均码率控制误差在0.6%以内,客观质量PSNR最高可提高0.65 dB。     

基于Kinect的实时深度提取与多视绘制算法

提出了一种基于Kinect的实时深度提取算法和单纹理+深度的多视绘制方法。在采集端,使用Kinect提取场景纹理和深度,并针对Kinect输出深度图的空洞提出一种快速修复算法。在显示端,针对单纹理+深度的基于深度图像的绘制(DIBR,depth image based rendering)绘制产生的大空洞,采用一种基于背景估计和前景分割的绘制方法。实验结果表明,本文方法可实时提取质量良好的深度图,并有效修复了DIBR绘制过程中产生的大空洞,得到质量较好的多路虚拟视点图像。以所提出的深度获取和绘制算法为核心,实现了一种基于深度的立体视频系统,最终的虚拟视点交织立体显示的立体效果良好,进一步验证了本文算法的有效性。本文系统可用于实景的多视点立体视频录制与播放。     

Virtual view synthesis using joint information from multi-view

Immersive media has attracted widespread attention with the development of virtual reality. Three Degree of Freedom Plus media greatly enhances the user experience by allowing users’ head motion and viewpoint switching within a certain range. Due to the limitation of panoramic video acquisition and transmission, it is impossible to obtain videos from any viewpoint directly. Virtual view synthesis is the general solution to this problem. However, existing algorithms do not adequately consider the pixel correlation between multiple views. Thus, we propose a virtual view synthesis algorithm using joint information from multi-view panoramic videos to further explore the pixel correlation. Specifically, sub-pixels from different reference views in the virtual view are obtained by performing multi-view three-dimensional image warping. Dedicated area division and interpolation methods are then designed to improve the synthesized quality. Experimental results show that the proposed algorithm outperforms the state-of-the-art virtual view synthesis algorithms in performance and efficiency.     

自由视点视频系统中虚拟视合成及校正方法

根据多视点视频中的几何约束关系,提出了一种基于图像的新视点重建方法,能够根据所选择方位重建虚拟视;针对虚拟视中合成误差,提出了基于平面扫描的校正方法,最终获得了主观质量和客观质量均优于传统方法的虚拟视。     

View upsampling optimization for mixed resolution 3D video coding

3D video is composed out of two or more, temporally synchronized, 2D video streams acquired at different camera poses and accompanied by geometrical information. In a mixed resolution 3D video stream, a subset of views is coded at reduced resolution. It has been shown in the literature that subjective quality of mixed resolution 3D video is close to that of full resolution 3D video. In order to improve the coding gain in mixed resolution coding scenario we present a new depth encoding method called view upsampling optimization. A novel depth distortion metric based on the performance of the depth-based super resolution is also presented. Finally, to improve the quality of the decoded video an improved depth-based super resolution method that uses view synthesis quality mapping is used for upsampling of low resolution views. The simulations, performed with the recently standardized MVC+D encoder, show that the proposed solution combined with the state of the art view synthesis distortion outperforms the anchor MVC+D coding scheme by 14.5 % of dBR on average for the total coded bitrate and by 17 % of dBR on average for the synthesized views.     

基于最大可容忍深度失真模型的低复杂度深度视频编码

在保证虚拟视点绘制质量的前提下,如何降低深 度视频的编码复杂度是一个亟需解决的问题。本文提出了一 种基于最大可容忍深度失真(MTDD,maximum tolerable d epth distortion)模型的低复杂度深度编码算法,将MTDD模型引 入 到率失真(RD)代价函数,通过对编码单元(CU)进行失真度量判决 ,如果对当前CU进 行编码不会导致可察觉的绘制失真,则失真度量只考虑深度编码失真,否则失真度量同时考 虑深度编码失真和视点 合成失真(VSD),从而降低编码复杂度。实验结果表明,所 提出的算法在不 降低虚拟视点绘制质量和不增加编码码率的情况下,能显著降低深度视频的编码复杂度。     

基于多视点视频特征的3D-HEVC深度视频信息隐藏算法

为了更安全地传输秘密信息和保护3D视频,提 出基于多视点视频特征的3D-HEVC深度视频信息隐 藏算法。首先结合彩色视频的纹理特征,考虑深度视频不同区域对绘制视点的质量和编码效 率的影响,对 深度视频进行区域分割。然后针对不同的区域,采用不同的方式调制最大编码单元(LCU) 的QP值嵌入 秘密信息。最后,用修改后的QP值进行编码压缩,传输视频信息。实验仿真结果表明,相 比于原始HTM13.0, 本算法编码重建深度视频绘制视点质量的PSNR平均下降0.0015dB, 码率平均增加0.035%。本文算法 能较好地保证绘制视点的质量,对视频流的码率影响较小,且能实现秘密信息的盲提取。     

Color calibration of multi-view video plus depth for advanced 3D video

Multi-view video plus depth (MVD) format is considered as the next-generation standard for advanced 3D video systems. MVD consists of multiple color videos with a depth value associated with each texture pixel. Relying on this representation and by using depth-image-based rendering techniques, new viewpoints for multi-view video applications can be generated. However, since MVD is captured from different viewing angles with different cameras, significant illumination and color differences can be observed between views. These color mismatches degrade the performance of view rendering algorithms by introducing visible artifacts leading to a reduced view synthesis quality. To cope with this issue, we propose an effective method for correcting color inconsistencies in MVD. Firstly, to avoid occlusion problems and allow performing correction in the most accurate way, we consider only the overlapping region when calculating the color mapping function. These common regions are determined using a reliable feature matching technique. Also, to maintain the temporal coherence, correction is applied on a temporal sliding window. Experimental results show that the proposed method reduces the color difference between views and improves view rendering process providing high-quality results.     

A motion vector prediction method for multi-view video coding

This paper proposes a new motion vector (MV) prediction method in multi-view video coding (MVC). In order to exploit the information in adjacent views, inter-view MVs as well as temporal MVs are used in conventional MVC. Since the inter-view MVs are usually uncorrelated with the temporal MVs and most neighboring partitions have temporal MVs only, the conventional DPCM coding gain of inter-view MV is very low and thus the inter-view MVs are seldom selected. In order to increase the probability of inter-view MV selection, we define a virtual inter-view MV which can be generated from temporal MVs. Then, an inter-view MV is predicted using these neighboring virtual inter-view MVs, leading to less prediction error than using the temporal MVs. As a result, bit-rates are decreased by up to 9% for the view-temporal prediction structure.     

Fast color correction for multi-view video by modeling spatio-temporal variation

In multi-view video, a number of cameras capture the same scene from different viewpoints. Color variations between the camera views may deteriorate the performance of multi-view video coding or virtual view rendering. In this paper, a fast color correction method for multi-view video is proposed by modeling spatio-temporal variation. In the proposed method, multi-view keyframes are defined to establish the spatio-temporal relationships for accurate and fast implementation. For keyframes, accurate color correction is performed based on spatial color discrepancy model that disparity estimation is used to find correspondence points between views, and linear regression is performed on these sets of points to find the optimal correction coefficients. For non-keyframes, fast color correction is performed based on temporal variations model that time-invariant regions are detected to reflect the change trends of correction coefficients. Experimental results show that compared with other methods, the proposed method can promote the correction speed greatly without noticeable quality degradation, and obtain higher coding performance.     

一种用于深度图编码的虚拟视失真估计模型

多视视频加深度(MVD,multi-view video plus depth)的3D视频格式中,深度图提供视频的场景几何信息,其不在终端成像显示而是通过基于深度图像的绘制(DIBR)技术用于绘制虚拟视图像。在深度图的压缩编码过程中,深度图的失真会引起绘制的虚拟视图像的失真。深度图用于绘制而不用于显示的特性使得准确估计深度图绘制的虚拟视失真可以提高深度图编码的率失真性能。本文分析了不同的深度图失真引起的不同的虚拟视失真,进而提出一种估计深度图失真引起虚拟视失真的指数模型,并将模型用于深度图编码的率失真优化(RDO)中。实验结果表明,本文提出的模型可以准确估计深度图失真引起的虚拟视失真,提高深度图编码性能,相比于HTM的VSO可以降低约10%的编码时间,并且虚拟视质量略优于HTM。     

采用灰度共生矩阵进行深度预判的3D-HEVC深度图帧内快速编码算法

针对3D视频的3D-HEVC编码标准以多视点纹理视频和深度视频格式进行编码,其深度图编码仍延续纹理视频编码的模式和编码尺寸遍历选择,使得3D-HEVC的编码复杂度居高不下。本文针对深度图帧内预测编码,采用灰度共生矩阵对深度图中的CTU进行计算,统计并分析其矩阵中非零值个数与CTU分割深度的关系,根据非零值个数分布规律,设定阈值,使得帧内编码时可以预判编码模块的分割深度,从而选择性跳过部分不同深度CU的帧内预测过程。经过HTM16.0测试平台的检验,本算法在全帧内编码模式下,测试序列合成视点比特率仅增加0.08%的同时,平均节省了16.8%的编码时间,与其他同类较新算法在HTM16.0平台上的性能比较也有一定的优势。      

构造全局背景的虚拟视点合成算法

虚拟视点合成技术是三维电视、多视点视频等诸多领域中的关键技术。目前,基于深度图像的绘制(DIBR)受到了广泛的关注,而这一技术的主要难点在于如何降低映射过程中产生的空洞。为解决这个问题,提高合成图像的质量,同时降低所需参考视点的数量,本文提出了一种基于全局背景图像的虚拟视点合成算法。首先,我们利用视频序列的帧间互补信息生成一张全局背景图像,并以此为辅助图像,去填补从参考视点映射到虚拟目标视点的各帧图像。与传统方法相比,本文算法能使用空洞区域对应的真实像素进行填充,提高了图像的质量。最后,对填补后仍余留的空洞采用图像修复算法进行处理,得到最终的虚拟视点图像。实验结果表明,这个算法优于传统算法,并且只利用了一个参考视点。      

Multi-view video coding with view interpolation prediction for 2D camera arrays

An efficient compression algorithm for multi-view video sequences, which are captured by two-dimensional (2D) camera arrays, is proposed in this work. First, we propose a novel prediction structure, called three-dimensional hierarchical B prediction (3DHBP), which can efficiently reduce horizontal inter-view redundancies, vertical inter-view redundancies, and temporal redundancies in multi-view videos. Second, we develop a view interpolation scheme based on the bilateral disparity estimation. The interpolation scheme yields high quality view frames by adapting disparity estimation and compensation procedures using the information in neighboring frames. Simulation results demonstrate that the proposed multi-view video coding algorithm provides significantly better rate–distortion (R–D) performance than the conventional algorithm, by employing the 3DHBP structure and using interpolated view frames as additional reference frames.     

Unified depth intra coding for 3D video extension of HEVC

With the development of high-efficiency video coding (HEVC), the newest video coding standard, 3D video extension of HEVC (3D-HEVC) has been actively investigated. Since 3D-HEVC uses multi-view texture and depth data for input, various coding tools have been added to HEVC. In 3D-HEVC, on top of the existing 35 HEVC intra modes, eight additional modes exist, which are specifically for depth coding. In this paper, we propose a unified depth intra coding method that incorporates such depth intra modes into the regular intra mode set. In particular, the most rarely used HEVC intra modes are replaced by depth intra modes. As a result, binarization for depth intra modes is removed. Furthermore, the most probable mode selection procedure is modified to consider the elimination of several angular intra modes. The proposed method is implemented and tested on 3D video HEVC test model version 7.0. Simulation results report 2.2 % synthesis gain under all-intra configuration.     

DIBR中基于平面扫描的深度重建方法

提出一种 DIBR 中基于平面扫描法的深度重建方法,与立体深度重建算法和基于图像的视觉壳算法不同,本文进一步改进平面扫描算法,无需任何场景的几何先验知识,而是利用每个像素点的深度信息合成真实场景的虚拟视点.当输入图像映射至相互平行的虚拟深度平面时,采用"动态判决方法"来计算像素间的色彩一致度;并在虚拟视合成中采用了基于视向权重策略的新视点重建方法.本文算法获取的深度信息更为精确,虚拟新视点的质量得到较大提高.     

Color correction algorithm based on camera characteristics for multi-view video coding

Various types of multi-view camera systems have been proposed for capturing three dimensional scenes. Yet, color distributions among multi-view images remain inconsistent in most cases, degrading multi-view video coding performance. In this paper, we propose a color correction algorithm based on the camera characteristics to effectively solve such a problem. Initially, we model camera characteristics and estimate their coefficients by means of correspondences between views. To consider occlusion in multi-view images, correspondences are extracted via feature-based matching. During coefficient estimation with nonlinear regression, we remove outliers in the extracted correspondences. Consecutively, we generate lookup tables for each camera using the model and estimated coefficients. Such tables are employed for fast color converting in the final color correction process. The experimental results show that our algorithm enhances coding efficiency with gains of up to 0.9 and 0.8 dB for luminance and chrominance components, respectively. Further, the method also improves subjective viewing quality and reduces color distance between views.     

Fast intra mode decision for depth coding in 3D-HEVC

The emergent 3D High Efficiency Video Coding (3D-HEVC) is an extension of the High Efficiency Video Coding (HEVC) standard for the compression of the multi-view texture videos plus depth maps format. Since depth maps have different statistical properties compared to texture video, various new intra tools have been added to 3D-HEVC depth coding. In current 3D-HEVC, new intra tools are utilized together with the conventional HEVC intra prediction modes for depth coding. This technique achieves the highest possible coding efficiency, but leads to an extremely high computational complexity which limits 3D-HEVC from practical applications. In this paper, we propose a fast intra mode decision algorithm for depth coding in 3D-HEVC. The basic idea of the proposed algorithm is to utilize the depth map characteristics to predict the current depth prediction mode and skip some specific depth intra modes rarely used in 3D-HEVC depth coding. Based on this analysis, two fast intra mode decision strategies are proposed including reduction of the number of conventional intra prediction modes, and simplification of depth modeling modes (DMMs). Experimental results demonstrate that the proposed algorithm can save 30 % coding runtime on average while maintaining almost the same rate-distortion (RD) performance as the original 3D-HEVC encoder.     

High quality virtual view synthesis based on corrected surface mapping and image fusion

Perceptual errors introduced by current virtual view creation approaches for free viewpoint video often result in severe distortions in synthesised images. A novel virtual view synthesis algorithm is proposed based on corrected region based surface mapping and an image fusion method to reduce perceptual errors. The results over multi-view image sets imply that the proposed method improves the objective and subjective performance simultaneously, compared with previous dominant multiple image view synthesis approaches.     

Generating Virtual Images for Multi-view Video

As for the research in the Free view televi-sion (FTV) and stereo video areas, the accurate multi-view video capture is a expensive precondition. We present a virtual multi-view video capture system based on OpenGL programming. By using this system, a researcher may cre-ate arbitrary 3D scenes, and capture images with depth maps by setting up multiple virtual cameras, whose in-trinsic and extrinsic parameters can be imported from or exported to the outside. We conducted several experiments within it to perform camera calibration, depth estimation and view-points synthesis. The system has been demon-strated to be a powerful tool for evaluation of stereo video algorithms.