Adaptive Learning Based View Synthesis Prediction for Multi-View Video Coding

In the applications of Free View TV, pre-estimated depth information is available to synthesize the intermediate views as well as to assist multi-view video coding. Existing view synthesis prediction schemes generate virtual view picture only from interview pictures. However, there are many types of signal mismatches caused by depth errors, camera heterogeneity or illumination difference across views and these mismatches decrease the prediction capability of virtual view picture. In this paper, we propose an adaptive learning based view synthesis prediction algorithm to enhance the prediction capability of virtual view picture. This algorithm integrates least square prediction with backward warping to synthesize the virtual view picture, which not only utilizes the adjacent views information but also the temporal decoded information to adaptively learn the prediction coefficients. Experiments show that the proposed method reduces the bitrates by up to 18 % relative to the multi-view video coding standard, and about 11 % relative to the conventional view synthesis prediction method.     

Multiview depth coding based on combined color/depth segmentation

In this paper, a new coding method for multiview depth video is presented. Considering the smooth structure and sharp edges of depth maps, a segmentation based approach is proposed. This allows further preserving the depth contours thus introducing fewer artifacts in the depth perception of the video. To reduce the cost associated with partition coding, an approximation of the depth partition is built using the decoded color view segmentation. This approximation is refined by sending some complementary information about the relevant differences between color and depth partitions. For coding the depth content of each region, a decomposition into orthogonal basis is used in this paper although similar decompositions may be also employed. Experimental results show that the proposed segmentation based depth coding method outperforms H.264/AVC and H.264/MVC by more than 2 dB at similar bitrates.     

Edge-preserving Intra mode for efficient depth map coding based on H.264/AVC

Depth-image-based-rendering (DIBR) algorithms for 3D video communication systems based on the “multi-view video plus depth” format are very sensitive to the accuracy of depth information. Specifically, edge regions in the depth data should be preserved in the coding/decoding process to ensure good view synthesis performance, which directly affects the overall system performance. This paper proposes a novel scheme for edge-aware Intra depth compression based on the H.264/AVC framework enabled on both Intra (I) and Inter (P) slices. The proposed scheme includes a new Intra mode specifically targeted to depth macroblocks with arbitrarily shaped edges, which are typically not predicted well by the standard Intra modes of H.264/AVC and result in high rate–distortion costs. The proposed algorithm segments edge macroblocks into two regions each approximated by a flat surface. A binary mask identifying the two regions is defined and encoded by means of context-coding with adaptive template selection. As a novel contribution, the proposed mode allows exploiting the correlation with causal neighboring edge macroblocks to improve the performance of context-coding of binary masks and allow significant bit rate savings. The proposed method has been exhaustively compared with different state-of-the-art algorithms for edge-aware depth coding and the results highlight significant improvements in most of the cases, both in terms of reconstructed depth quality, view synthesis performance, and overall texture plus depth rate–distortion performance.     

A Framework for Representation and Processing of Multi-view Video Using the Concept of Layered Depth Image

The multi-view video is a collection of multiple videos, capturing the same scene at different viewpoints. Since it contains more affluent information than a single video, it can be applied to various applications, such as 3DTV, free viewpoint TV, surveillance, sports matches, and so on. However, the data size of the multi-view video linearly increases as the number of cameras, therefore it is necessary to develop an effective framework to represent, process, and transmit those huge amounts of data. In recent, multi-view video coding is getting lots of attention as efficient video coding technologies are being developed. Although most of multi-view video coding algorithms are based on the state-of-the-art H.264/AVC video coding technology, they do not utilize rich 3-D information. In this paper, we propose a new framework using the concept of layered depth image (LDI), one of the efficient image-based rendering techniques, to efficiently represent and process multi-view video data. We describe how to represent natural multi-view video based on the LDI approach and the overall framework to process those converted data.     

Adaptive depth truncation filter for MVC based compressed depth image

In multiview video plus depth (MVD) format, virtual views are generated from decoded texture videos with corresponding decoded depth images through depth image based rendering (DIBR). 3DV-ATM is a reference model for the H.264/AVC based multiview video coding (MVC) and aims at achieving high coding efficiency for 3D video in MVD format. Depth images are first downsampled then coded by 3DV-ATM. However, sharp object boundary characteristic of depth images does not well match with the transform coding based nature of H.264/AVC in 3DV-ATM. Depth boundaries are often blurred with ringing artifacts in the decoded depth images that result in noticeable artifacts in synthesized virtual views. This paper presents a low complexity adaptive depth truncation filter to recover the sharp object boundaries of the depth images using adaptive block repositioning and expansion for increasing the depth values refinement accuracy. This new approach is very efficient and can avoid false depth boundary refinement when block boundaries lie around the depth edge regions and ensure sufficient information within the processing block for depth layers classification. Experimental results demonstrate that the sharp depth edges can be recovered using the proposed filter and boundary artifacts in the synthesized views can be removed. The proposed method can provide improvement up to 3.25 dB in the depth map enhancement and bitrate reduction of 3.06% in the synthesized views.     

应用于多视点视频压缩的多参考B帧快速编码算法

基于H.264平台就多视点视频压缩中多参考帧技术存在计算复杂度高的现状,根据多视点视频序列特性,提出了一种针对多参考B帧的快速多视点视频编码压缩算法。新算法改进了多参考帧列表策略,并结合多视点视频序列的时空域相关性特性,提出了适用于多视点视频编码的参考帧列表调整策略。此外,还针对多种块模式编码引入了块模式预判策略。新算法在保证率失真性能的基础上,有效地降低了计算复杂度,提高了多视点视频编码速度。虽然本文是基于M-Picture编码结构实现了新算法,但它同样适用于其它应用多参考B帧预测技术的多视点视频编码结构中。     

An efficient retinex-like brightness normalization method for coding camera flashes and strong brightness variation in videos

Conventional hybrid video coding systems rely on the assumption that the brightness is constant. This does not take inter-frame brightness variations into consideration during motion estimation and compensation processes. Under the influence of inter-frame lighting variations like camera flashes, video motion activities are not accurately estimated and the pixel prediction is poor which directly increases the bits for prediction error coding. In this paper, we propose an efficient algorithm based on the retinex-like system which allows inter-frame brightness being normalized before applying the conventional motion estimation and compensation. Experimental results show that our approach is superior to all similar approaches in the literature and demonstrate that our proposed system is very robust against the inter-frame brightness variations. Further experimental works have been done using the verification models of the MPEG-4 and the H.264 on sequences with brightness variations, results of which show that our proposed system outperforms these coding systems, including the weighted prediction feature in H.264, which were specifically designed for this purpose.     

Selective VS-MRF-ME and intra coding in H.264 based on spatiotemporal continuity of motion field

The new video coding standard, H.264 uses variable size motion estimation (VS-ME), multiple reference frame motion estimation (MRF-ME) and spatial-based intra prediction with selectable block size in inter frame coding. These tools have achieved significant coding efficiency compared to coding a macroblock (MB) only based on motion-compensation in regular size with single reference frame. However, these new features also give rise to an exhaustive computation in the coding procedure since there are so many combinations of coding modes and reference frames to be tried. In this paper, a fast motion estimation algorithm based on the selective VS-MRF-ME and intra prediction is proposed to reduce H.264 coding computational complexity. The basic idea of the method is to utilize the spatiotemporal property of motion field in predicting where VS-MRF-ME and intra prediction are needed, and only in these regions VS-MRF-ME and intra coding are enabled. The motion field is generated by motion vectors from 16×16 motion estimation on the nearest reference frame. Simulation results show that the proposed algorithm can save 50% computational complexity on average, with negligible loss of coding efficiency.     

Reconstruction of lost depth data in multiview video-plus-depth communications using geometric transforms

This paper addresses depth data recovery in multiview video-plus-depth communications affected by transmission errors and/or packet loss. The novel aspects of the proposed method rely on the use of geometric transforms and warping vectors, capable of capturing complex motion and view-dependent deformations, which are not efficiently handled by traditional motion and/or disparity compensation methods. By exploiting the geometric nature of depth information, a region matching approach combined with depth contour reconstruction is devised to achieve accurate interpolation of arbitrary shapes within lost regions of depth maps. The simulation results show that, for different packet loss rates, up to 20%, the depth maps recovered by the proposed method produce virtual views with better quality than existing methods based on motion information and spatial interpolation. An average PSNR gain of 1.48 dB is obtained in virtual views synthesised from depth maps using the proposed method.     

多视点纹理加深度编码的联合码率控制方法

码率控制技术是多视点视频编码和传输中一个关键的问题。为了提高三维(3D)视频的整体显示质量,包括虚拟视点质量和编码视点质量,提出一种多视点纹理加深度编码的联合码率控制方法。该算法研究了纹理和深度的关系,采用基于模型方法确定最优的纹理和深度之间的码率比例。根据各个视点编码结果的统计规律,不同的视频序列采用不同的视点间比特分配比例。实验结果表明,与目前流行的多视点码率控制算法相比,该算法在计算复杂度基本保持不变的情况下,平均码率控制误差在0.6%以内,客观质量PSNR最高可提高0.65 dB。     

Free-viewpoint depth image based rendering

In 3D TV research, one approach is to employ multiple cameras for creating a 3D multi-view signal with the aim to make interactive free-viewpoint selection possible in 3D TV media. This paper explores a new rendering algorithm that enables to compute a free-viewpoint between two reference views from existing cameras. A unique property is that we perform forward warping for both texture and depth simultaneously. Advantages of our rendering are manyfold. First, resampling artifacts are filled in by inverse warping. Second, disocclusions are processed while omitting warping of edges at high discontinuities. Third, our disocclusion inpainting approach explicitly uses depth information. We obtain an average PSNR gain of 3 dB and 4.5 dB for the ‘Breakdancers’ and ‘Ballet’ sequences, respectively, compared recently published results. Moreover, experiments are performed using compressed video from surrounding cameras. The overall system quality is dominated by rendering quality and not by coding.     

基于Kinect的虚拟视点生成技术

传统的多视点生成方法是基于多相机阵列系统的关键技术。现提出了基于Kinect的多视点成像计算方法。首先对Kinect的深度图使用三边滤波器进行平滑,根据修复好的深度图配合彩色图,利用DIBR技术生成多个存在空缺信息的彩色视点;最后结合彩色图的纹理结构信息和深度图的背景信息对有丢失信息的彩色图进行修复。实验结果表明,文中提出的深度修复方法能够有效地修补Kinect的深度图,生成的虚拟视点图在3DTV上效果明显,立体视觉效果显著。     

一种基于H.264/AVC压缩域的GOP级视频场景转换检测算法

本文提出了一种基于H.264/AVC压缩域的GOP(Group of Pictures)级视频场景转换检测算法.该算法利用H.264/AVC基本档次码流中的帧内预测模式、运动矢量和宏块编码类型等可用信息,提出了基于子块的色度模式差异、累积运动值和累积帧内宏块数等三个判决准则,然后综合利用这三个判决准则,提出了一种GOP级的视频场景转换检测算法.实验结果表明,与现有的一个COP级场景检测算法对比,本文提出的GOP级视频场景转换检测算法可以获得更好的检测性能.     

快速帧内预测模式选择新方法

为了提高编码性能,H.264采用了RDO(率失真优化),但与此同时带来的是计算复杂度的增加.本文着重分析了H.264中快速帧内预测模式选择的问题.为了降低帧内预测模式选择的计算复杂度,提出了一种高效的快速帧内预测模式选择算法.首先,对Pan的基于边缘方向直方图的快速算法进行了改进,同时提出了基于参考象素特征的4×4快速帧内预测模式选择算法,并将两者进行了结合.实验结果表明,本文算法和H.264校验模型JM61相比,I帧编码时间降低61%~69%,而PSNR基本保持不变,输出码率仅略有增加;与Pan的基于边缘方向直方图快速算法相比,本文算法的I帧编码时间降低12%~33%,PSNR和输出码率均基本保持不变.     

一种基于H.264/AVC的鲁棒性视频水印算法的研究

依据 H.264视频编码标准提出一种视频水印方案。结合H.264帧间预测编码的特点以及人类视觉系统原理,选择P帧宏块中的MV作为水印信息嵌入的对象,对MV进行量化处理,得到两个互相独立的区域。根据水印信息的不同,修改MV,使其指向不同的区域,对其采用扩频技术来实现水印嵌入,以提高水印的鲁棒性。实验结果表明,新方案能够抵抗重编码压缩、加载噪声、旋转等多种攻击,鲁棒性良好。     

Multi-View Video Coding Based on Vector Estimation and Weighted Disparity Interpolation

Aiming at fully exploiting the temporal and spatial redundancy and answering the “Call for Proposals” for multi-view video coding (MVC) issued by MPEG, a MVC scheme based on vector field estimation and weighted disparity interpolation is presented. By extending the loop constraint to multi-view images for a parallel camera model and proposing the novel “vector field estimation” scheme, the temporal and spatial redundancy is significantly reduced. Also, weighted disparity interpolation is performed to predict adjacent disparity vectors. Experimental results over multi-view image sets imply that the coding efficiency is improved about 0.2–0.5 dB compared with previous coding approaches such as H.264/AVC simulcast and JMVM.     

一种新的视点间预测结构

多视点视频编码除应具有较高的编码效率外,还应该包括后向兼容性、时间随机访问和视点可分级性等,这些都主要取决于所采用的预测结构。目前所提供的多视点视频编码(Joint Multi-view Video Coding, JMVC)采用固定的视点间预测结构,难以适应复杂情况的多视点视频编码。该文综合考虑编码效率和用户随机访问等因素,根据多视点视频相关性分析自适应调整视点间预测结构,以获得较好的编码综合性能。试验结果表明,与JMVC相比,该文的方法在提高编码效率的同时,有较好的随机访问性能。     

H.264中的运动估计和帧内预测算法研究

 针对一些算法过早的确定搜索方向,容易陷入局部最优点缺失搜索准确度的情况,提出一种新的基于搜索方向预测的运动估计算法.实验结果表明,与单一搜索图形相比,该算法具有更高的搜索精度和搜索速度.帧内预测技术作为影响帧编码效率的关键,一直被广泛研究,为此,本文提出了一种快速的帧内预测算法,对宏块划分提出了新的"模板"宏块对比算法思想,并在已有的抽样算法和搜索窗算法的基础上,提出了改进方法.实验结果证明,算法在不降低图像质量的基础上,编码速度平均提高80%.     

基于H.264的内容自适应分数像素运动估计算法

分数像素精确运动估计的改进是整个运动估计模块优化的关键,本文提出了基于H.264的内容自适应分数像素运动估计算法。首先,提出基于平坦区域宏块预测的无效分数像素运动矢量(MV)搜索省略算法(SMBP);然后,改进H.264采用的基于中心的分数像素搜索算法(CBFPS),提出基于预测矢量的增强型菱形模板(EDSP)搜索算法。实验结果表明,内容自适应分数像素运动估计算法比分数像素全搜索算法(FFPS)在峰值信噪比(PSNR)有微小降低(0.095～0.209 dB)的情况下,平均减少了75.6%的分数像素搜索点,整个运动估计模块平均节省了38.5%的计算量。