Distributed Monoview and Multiview Video Coding

Growing percentage of the world population now uses image and video coding technologies on a regular basis. These technologies are behind the success and quick deployment of services and products such as digital pictures, digital television, DVDs, and Internet video communications. Today's digital video coding paradigm represented by the ITU-T and MPEG standards mainly relies on a hybrid of block- based transform and interframe predictive coding approaches. In this coding framework, the encoder architecture has the task to exploit both the temporal and spatial redundancies present in the video sequence, which is a rather complex exercise. As a consequence, all standard video encoders have a much higher computational complexity than the decoder (typically five to ten times more complex), mainly due to the temporal correlation exploitation tools, notably the motion estimation process. This type of architecture is well-suited for applications where the video is encoded once and decoded many times, i.e., one-to-many topologies, such as broadcasting or video-on-demand, where the cost of the decoder is more critical than the cost of the encoder.     

Introducing skip mode in distributed video coding

Although it was proven in the 1970s already by Wyner and Ziv and Slepian and Wolf that, under certain conditions, the same rate–distortion boundaries exist for distributed video coding (DVC) systems as for traditional predicting systems, until now no practical DVC system has been developed that even comes close to the performance of state-of-the-art video codecs such as H.264/AVC in terms of rate–distortion. Some important factors for this are the lower accuracy of the motion estimation performed at the decoder, the inaccurate modeling of the correlation between the side information and the original frame, and the absence in most state-of-the-art DVC systems of anything conceptually similar to the notion of skipped macroblocks in predictive coding systems.This paper proposes an extension of a state-of-the-art transform domain residual DVC system with an implementation of skip mode. The skip mode has an impact at two different places: in the turbo decoder, more specifically the soft input, soft output (SISO) convolutional decoder, and in the puncturing of the parity bits. Results show average bitrate gains up to 39% (depending on the sequence) achieved by combining both approaches.Furthermore, a hybrid video codec is presented where the motion estimation task is shifted back to the encoder. This results in a drastic increase in encoder complexity, but also in a drastic performance gain in terms of rate–distortion, with average bitrate savings up to 60% relative to the distributed video codec. In the hybrid video codec, smaller but still important average bitrate gains are achieved by implementing skip mode: up to 24%.     

Advanced side information creation techniques and framework for Wyner–Ziv video coding

Recently, several distributed video coding (DVC) solutions based on the distributed source coding (DSC) paradigm have appeared in the literature. Wyner–Ziv (WZ) video coding, a particular case of DVC where side information is made available at the decoder, enable to achieve a flexible distribution of the computational complexity between the encoder and decoder, promising to fulfill novel requirements from applications such as video surveillance, sensor networks and mobile camera phones. The quality of the side information at the decoder has a critical role in determining the WZ video coding rate-distortion (RD) performance, notably to raise it to a level as close as possible to the RD performance of standard predictive video coding schemes. Towards this target, efficient motion search algorithms for powerful frame interpolation are much needed at the decoder. In this paper, the RD performance of a Wyner–Ziv video codec is improved by using novel, advanced motion compensated frame interpolation techniques to generate the side information. The development of these type of side information estimators is a difficult problem in WZ video coding, especially because the decoder only has available some reference, decoded frames. Based on the regularization of the motion field, novel side information creation techniques are proposed in this paper along with a new frame interpolation framework able to generate higher quality side information at the decoder. To illustrate the RD performance improvements, this novel side information creation framework has been integrated in a transform domain turbo coding based Wyner–Ziv video codec. Experimental results show that the novel side information creation solution leads to better RD performance than available state-of-the-art side information estimators, with improvements up to 2 dB; moreover, it allows outperforming H.264/AVC Intra by up to 3 dB with a lower encoding complexity.     

Content-adaptive motion estimation algorithm for coarse-grain SVC

A joint model of scalable video coding (SVC) uses exhaustive mode and motion searches to select the best prediction mode and motion vector for each macroblock (MB) with high coding efficiency at the cost of computational complexity. If major characteristics of a coding MB such as the complexity of the prediction mode and the motion property can be identified and used in adjusting motion estimation (ME), one can design an algorithm that can adapt coding parameters to the video content. This way, unnecessary mode and motion searches can be avoided. In this paper, we propose a content-adaptive ME for SVC, including analyses of mode complexity and motion property to assist mode and motion searches. An experimental analysis is performed to study interlayer and spatial correlations in the coding information. Based on the correlations, the motion and mode characteristics of the current MB are identified and utilized to adjust each step of ME at the enhancement layer including mode decision, search-range selection, and prediction direction selection. Experimental results show that the proposed algorithm can significantly reduce the computational complexity of SVC while maintaining nearly the same rate distortion performance as the original encoder.     

PRISM: A video coding paradigm with motion estimation at the decoder.

We describe PRISM, a video coding paradigm based on the principles of lossy distributed compression (also called source coding with side information or Wyner-Ziv coding) from multiuser information theory. PRISM represents a major departure from conventional video coding architectures (e.g., the MPEGx, H.26x families) that are based on motion-compensated predictive coding, with the goal of addressing some of their architectural limitations. PRISM allows for two key architectural enhancements: (1) inbuilt robustness to "drift" between encoder and decoder and (2) the feasibility of a flexible distribution of computational complexity between encoder and decoder. Specifically, PRISM enables transfer of the computationally expensive video encoder motion-search module to the video decoder. Based on this capability, we consider an instance of PRISM corresponding to a near reversal in codec complexities with respect to today's codecs (leading to a novel light encoder and heavy decoder paradigm), in this paper. We present encouraging preliminary results on real-world video sequences, particularly in the realm of transmission losses, where PRISM exhibits the characteristic of rapid recovery, in contrast to contemporary codecs. This renders PRISM as an attractive candidate for wireless video applications.     

Rate-distortion driven decoder-side bitplane mode decision for distributed video coding

Distributed video coding (DVC) features simple encoders but complex decoders, which lies in contrast to conventional video compression solutions such as H.264/AVC. This shift in complexity is realized by performing motion estimation at the decoder side instead of at the encoder, which brings a number of problems that need to be dealt with. One of these problems is that, while employing different coding modes yields significant coding gains in classical video compression systems, it is still difficult to fully exploit this in DVC without increasing the complexity at the encoder side. Therefore, in this paper, instead of using an encoder-side approach, techniques for decoder-side mode decision are proposed. A rate-distortion model is derived that takes into account the position of the side information in the quantization bin. This model is then used to perform mode decision at the coefficient level and bitplane level. Average rate gains of 13–28% over the state-of-the-art DISCOVER codec are reported, for a GOP of size four, for several test sequences.     

A generalized model for Kalman filtering of motion information in video compression

In this paper, we tackle the problem of motion estimation in video compression. Since Full Search Algorithms (FSA) present the disadvantage of adding a high computational burden to the encoder, fast search techniques have been used in conjunction with predictive filtering, in such a way to guarantee an acceptable quality with an affordable complexity. The aim of this work is to propose a novel framework for Kalman filtering of motion information in compressed video sequences. The merits of our new framework are twofold: First, using an appropriate formulation of the system equations, several shortcomings inherent with former models in the literature are greatly counteracted. Secondly, it is constructed using a generalized structure in such a way to enclose a large variety of prediction models. Therefore, it can adapt to different types of motion activities in video sequences, without the need for a different formulation in each prediction model, as was the case in previous studies. Furthermore, we propose an adaptive motion compensation technique that permits an additional improvement to the decoded video quality. Our framework permits a considerable gain in the average performance compared to previous models and even to the FSA technique.     

Reduced decoder complexity and latency in pixel-domain Wyner–Ziv video coders

In some video coding applications, it is desirable to reduce the complexity of the video encoder at the expense of a more complex decoder. Wyner–Ziv (WZ) video coding is a new paradigm that aims to achieve this. To allocate a proper number of bits to each frame, most WZ video coding algorithms use a feedback channel, which allows the decoder to request additional bits when needed. However, due to these multiple bit requests, the complexity and the latency of WZ video decoders increase massively. To overcome these problems, in this paper we propose a rate allocation (RA) algorithm for pixel-domain WZ video coders. This algorithm estimates at the encoder the number of bits needed for the decoding of every frame while still keeping the encoder complexity low. Experimental results show that, by using our RA algorithm, the number of bit requests over the feedback channel—and hence, the decoder complexity and the latency—are significantly reduced. Meanwhile, a very near-to-optimal rate-distortion performance is maintained. This work has been partially supported by the Spanish Ministry of Education and Science and the European Commission (FEDER) under grant TEC2005-07751-C02-01. A. Pižurica is a postdoctoral research fellow of FWO, Flanders.     

Overlapped block motion compensation: an estimation-theoreticapproach

We present an estimation-theoretic analysis of motion compensation that, when used with fields of block-based motion vectors, leads to the development of overlapped block algorithms with improved compensation accuracy. Overlapped block motion compensation (OBMC) is formulated as a probabilistic linear estimator of pixel intensities given the limited block motion information available to the decoder. Although overlapped techniques have been observed to reduce blocking artifacts in video coding, this analysis establishes for the first time how (and why) OBMC can offer substantial reductions in prediction error as well, even with no change in the encoder's search and no extra side information. Performance can be further enhanced with the use of state variable conditioning in the compensation process. We describe the design of optimized windows for OBMC. We also demonstrate how, with additional encoder complexity, a motion estimation algorithm optimized for OBMC offers further significant gains in compensation accuracy. Overall mean-square prediction improvements in the range of 16 to 40% (0.8 to 2.2 dB) are demonstrated     

Side information creation for efficient Wyner–Ziv video coding: Classifying and reviewing

Video coding technologies have played a major role in the explosion of large market digital video applications and services. In this context, the very popular MPEG-x and H-26x video coding standards adopted a predictive coding paradigm, where complex encoders exploit the data redundancy and irrelevancy to ‘control’ much simpler decoders. This codec paradigm fits well applications and services such as digital television and video storage where the decoder complexity is critical, but does not match well the requirements of emerging applications such as visual sensor networks where the encoder complexity is more critical. The Slepian–Wolf and Wyner–Ziv theorems brought the possibility to develop the so-called Wyner–Ziv video codecs, following a different coding paradigm where it is the task of the decoder, and not anymore of the encoder, to (fully or partly) exploit the video redundancy. Theoretically, Wyner–Ziv video coding does not incur in any compression performance penalty regarding the more traditional predictive coding paradigm (at least for certain conditions). In the context of Wyner–Ziv video codecs, the so-called side information, which is a decoder estimate of the original frame to code, plays a critical role in the overall compression performance. For this reason, much research effort has been invested in the past decade to develop increasingly more efficient side information creation methods. This paper has the main objective to review and evaluate the available side information methods after proposing a classification taxonomy to guide this review, allowing to achieve more solid conclusions and better identify the next relevant research challenges. After classifying the side information creation methods into four classes, notably guess, try, hint and learn, the review of the most important techniques in each class and the evaluation of some of them leads to the important conclusion that the side information creation methods provide better rate-distortion (RD) performance depending on the amount of temporal correlation in each video sequence. It became also clear that the best available Wyner–Ziv video coding solutions are almost systematically based on the learn approach. The best solutions are already able to systematically outperform the H.264/AVC Intra, and also the H.264/AVC zero-motion standard solutions for specific types of content.     

计算能力可伸缩的运动估计率失真优化

不同硬件设备具有不同的计算能力,能否在任意给定计算能力约束下达到最好的编码效率,是当前视频编码研究领域的一个极具挑战性问题.同时,随着分块结构越来越灵活的编码标准不断出现（如：HEVC,H.264等）,运动估计不得不反复地应用在大小不同的各种分块上,导致其对编码总体计算复杂度的影响愈加重要.在此背景下,本文提出了一种针对运动估计的计算能力可伸缩（Complexity scalable）优化算法.我们通过对运动估计过程中预测失真度和计算复杂度的变化规律建模,发现根据各宏块的特性设置不同的预测失真度阈值可以优化地分配计算资源.而该阈值的大小则恰恰是各宏块的最小预测失真度加上一个由复杂度约束统一决定的偏移量.有鉴于此,我们进一步构造了计算能力可伸缩的优化运动估计算法,在不增加额外计算量的前提下,快速地得到各个宏块所对应的优化阈值,并完成运动估计.通过实验分析,该算法不仅具备自动适应不同计算复杂度约束的能力,而且在任意给定的复杂度约束下,都能提供优化的编码性能.     

Deep chroma prediction of Wyner–Ziv frames in distributed video coding of wireless capsule endoscopy video

Compression of captured video frames is crucial for saving the power in wireless capsule endoscopy (WCE). A low complexity encoder is desired to limit the power consumption required for compressing the WCE video. Distributed video coding (DVC) technique is best suitable for designing a low complexity encoder. In this technique, frames captured in RGB colour space are converted into YCbCr colour space. Both Y and CbCr representing luma and chroma components of the Wyner–Ziv (WZ) frames are processed and encoded in existing DVC techniques proposed for WCE video compression. In the WCE video, consecutive frames exhibit more similarity in texture and colour properties. The proposed work uses these properties to present a method for processing and encoding only the luma component of a WZ frame. The chroma components of the WZ frame are predicted by an encoder–decoder based deep chroma prediction model at the decoder by matching luma and texture information of the keyframe and WZ frame. The proposed method reduces the computations required for encoding and transmitting of WZ chroma component. The results show that the proposed DVC with a deep chroma prediction model performs better when compared to motion JPEG and existing DVC systems for WCE at the reduced encoder complexity.     

Low‐Complexity MPEG‐4 Shape Encoding towards Realtime Object‐Based Applications

Although frame‐based MPEG‐4 video services have been successfully deployed since 2000, MPEG‐4 video coding is now facing great competition in becoming a dominant player in the market. Object‐based coding is one of the key functionalities of MPEG‐4 video coding. Realtime object‐based video encoding is also important for multimedia broadcasting for the near future. Object‐based video services using MPEG‐4 have not yet made a successful debut due to several reasons. One of the critical problems is the coding complexity of object‐based video coding over frame‐based video coding. Since a video object is described with an arbitrary shape, the bitstream contains not only motion and texture data but also shape data. This has introduced additional complexity to the decoder side as well as to the encoder side. In this paper, we have analyzed the current MPEG‐4 video encoding tools and proposed efficient coding technologies that reduce the complexity of the encoder. Using the proposed coding schemes, we have obtained a 56 percent reduction in shape‐coding complexity over the MPEG‐4 video reference software (Microsoft version, 2000 edition).     

An image feature-based method to efficiently determine inter-coding depth in HEVC

The quad-tree based picture partition scheme in High Efficiency Video Coding (HEVC) results in a more substantial increase in computational complexity than those incurred by its predecessor video coding standards because of the need in this scheme to determine the best coding unit (CU) partitions. In this paper, we propose a method to effectively reduce the computational complexity of inter-prediction coding in the HEVC standard. The relative displacement of the largest coding unit (LCU) at the corresponding position between adjacent frames is tested through optical flow (motion estimation). The texture intensity of the LCU at the given time is tested if the condition that determines the coding depth in advance cannot be satisfied. The depth of the coding unit (CU) can be determined in advance beyond the xCompressCU function by using our proposed method, which does not require the calculation of the rate-distortion (RD) cost for each level of depth, and thus reduces the circular traversal times of the xCompressCU function. Experimental results proved that our proposed method is effective, as it reduced the computational complexity of an encoder by 53.2% on average, and had a slight influence on coding performance.     

Early SKIP mode decision for MVC using inter-view correlation

In the joint multiview video model (JMVM) proposed by JVT, the variable block-size motion estimation (ME) and disparity estimation (DE) have been employed to determine the best coding mode for each macroblock (MB). These give a high coding efficiency for multiview video coding (MVC), however, they cause a very high computational complexity in encoding system. This paper proposes to reduce the complexities of the ME and DE processes with an early SKIP mode decision algorithm based on the analysis of prediction mode distribution regarding the corresponding MBs in the neighbor view. In this method, the mode decision procedures of most of MBs can be early terminated, and thus much of computation for ME and DE can be greatly reduced. Simulation results demonstrate that our algorithm can achieve computational saving of 46–57% (depending on the tested sequences) with no significant loss of rate-distortion performance.     

一种空间域Wyner-Ziv视频编码系统的性能改进算法

分布式视频编码是建立在Slepian-Wolf和Wyner-Ziv信息编码理论基础上的全新视频编码框架,具有编码复杂度低,编码效率较高,抗误码性能好的特点.本文首先简单介绍了一种典型的分布式视频编码实现方案——空间域Wyner-Ziv视频编码,随后提出一种空间域Wyner-Ziv视频编码系统的性能改进算法,该算法在不增加编码复杂度的基础上,在解码端利用双向运动估计预测获取更高质量的边信息,同时采用基于Huber-Markov随机场约束的联合迭代解码算法重建图像.实验结果表明,在相同的输出码流情况下,本文改进算法在解码端重建图像的峰值信噪比与空间域Wyner-Ziv视频编码算法相比平均提高2dB,并且主观效果有所改善.     

Efficient Inter Prediction Mode Decision Method for Fast Motion Estimation in High Efficiency Video Coding

High Efficiency Video Coding (HEVC) is the most recent video coding standard to achieve a higher coding performance than the previous H.264/AVC. In order to accomplish this improved coding performance, HEVC adopted several advanced coding tools; however, these cause heavy computational complexity. Similar to previous video coding standards, motion estimation (ME) of HEVC requires the most computational complexity; this is because ME is conducted for three inter prediction modes — namely, uniprediction in list 0, uniprediction in list 1, and biprediction. In this paper, we propose an efficient inter prediction mode (EIPM) decision method to reduce the complexity of ME. The proposed EIPM method computes the priority of all inter prediction modes and performs ME only on a selected inter prediction mode. Experimental results show that the proposed method reduces computational complexity arising from ME by up to 51.76% and achieves near similar coding performance compared to HEVC test model version 10.1.     

H.264/AVC中基于全零块检测的运动估计快速算法

全零块检测是面向低比特率的视频编码器常用优化方法之一.特别是与运动估计相结合,可以有效的减少编码器的计算复杂性.本文根据H.264/AVC中整数变换的特点,给出了相应的全零块检测门限,提出了一种基于全零块检测的运动搜索提前中止准则.针对H.264/AVC多编码模式的特点,进一步将全零块检测用于H.264/AVC中多种编码模式的选择,有效的提高了运动估计的效率.利用这种方法,在有效减少编码器的计算复杂性,提高H.264/AVC软件编码器编码效率的同时,可以保持比特率和图像质量基本不变.     

Error-resilient video transmission using long-term memorymotion-compensated prediction

Long-term memory prediction extends the spatial displacement vector utilized in hybrid video coding by a variable time delay, permitting the use of more than one reference frame for motion compensation. This extension leads to improved rate-distortion performance. However, motion compensation in combination with transmission errors leads to temporal error propagation that occurs when the reference frames at the coder and decoder differ. In this paper, we present a framework that incorporates an estimated error into rate-constrained motion estimation and mode decision. Experimental results with a Rayleigh fading channel show that long-term memory prediction significantly outperforms the single-frame prediction H.263-based anchor. When a feedback channel is available, the decoder can inform the encoder about successful or unsuccessful transmission events by sending positive (ACK) or negative (NACK) acknowledgments. This information is utilized for updating the error estimates at the encoder. Similar concepts, such as the ACK and NACK mode known from the H.263 standard, are unified into a general framework providing superior transmission performance