Efficient error resilient algorithm for H.264/AVC: mobility management in wireless video streaming

The H.264/AVC standard introduces enhanced error robustness capabilities enabling resilient and reliable transmission of compressed video signals over wireless lossy packet networks. Those robustness capabilities are achieved by integrating some new error resilience tools that are essential for a proper delivery of real-time video services. Those tools include the Intra Refreshing (IR), Arbitrary Slice Ordering (ASO), Sequence Picture Parameter Sets (PPS), Redundant Slices (RS) tools and Flexible Macroblock Ordering (FMO). This paper presents an error resilient algorithm in wireless H.264/AVC streaming. The proposed method merges Reference Frame Selection (RFS), Intra Redundancy Slice and Adaptive Intra Refreshment techniques in order to prevent temporal error propagation in error-phone wireless video streaming. The coding standards only specify the decoding process and the bitstream syntax to allow considerable flexibility for the designers to optimize the encoder for coding performance improvement and complexity reduction. Performance evaluations demonstrate that the proposed encoding algorithm outperforms the conventional H.264/AVC standard. Both subjective and objective visual quality comparative study has been also carried out in order to validate the proposed approach. The proposed method can be used and integrated into H264/AVC without violating the standard.     

无线环境中H.264/AVC结构的改进

研究无线环境的视频传输是3G发展的关键.无线视频压缩编码不仅要求高压缩性能还要求适应无线网络的特征,这使得H.264/AVC成为目前惟一可选的编解码标准.文章介绍了H.264/AVC中适合用于无线视频传输的错误掩盖和抗误码的方法,讨论了基于无线网络如何应用这些方法,展望了无线环境中H.264/AVC需要的进一步改进.     

H.264/AVC中基于全零块检测的运动估计快速算法

全零块检测是面向低比特率的视频编码器常用优化方法之一.特别是与运动估计相结合,可以有效的减少编码器的计算复杂性.本文根据H.264/AVC中整数变换的特点,给出了相应的全零块检测门限,提出了一种基于全零块检测的运动搜索提前中止准则.针对H.264/AVC多编码模式的特点,进一步将全零块检测用于H.264/AVC中多种编码模式的选择,有效的提高了运动估计的效率.利用这种方法,在有效减少编码器的计算复杂性,提高H.264/AVC软件编码器编码效率的同时,可以保持比特率和图像质量基本不变.     

SSIM-based error-resilient rate-distortion optimization of H.264/AVC video coding for wireless streaming

The SSIM-based rate-distortion optimization (RDO) has been verified to be an effective tool for H.264/AVC to promote the perceptual video coding performance. However, the current SSIM-based RDO is not efficient for improving the perceptual quality of the video streaming application over the error-prone network, because it does not consider the transmission induced distortion in the encoding process. In this paper, a SSIM-based error-resilient RDO scheme for H.264/AVC is proposed to improve the wireless video streaming performance. Firstly, with the help of the SSE-based RDO, we present a low-complexity Lagrange multiplier decision method for the SSIM-based RDO video coding in the error-free environment. Then, the SSIM-based decoding distortion of the user end is estimated at the encoder and is correspondingly introduced into the RDO to involve the transmission induced distortion into the encoding process. Further, the Lagrange multiplier is theoretically derived to optimize the encoding mode selection in the error-resilient RDO process. Experimental results show that the proposed SSIM-based error-resilient RDO can obtain superior perceptual video quality (more structural information) to the traditional SSE-based error-resilient RDO for wireless video streaming at the same bit rate condition.     

Error resilient video coding techniques

We review error resilience techniques for real-time video transport over unreliable networks. Topics covered include an introduction to today's protocol and network environments and their characteristics, encoder error resilience tools, decoder error concealment techniques, as well as techniques that require cooperation between encoder, decoder, and the network. We provide a review of general principles of these techniques as well as specific implementations adopted by the H.263 and MPEG-4 video coding standards. The majority of the article is devoted to the techniques developed for block-based hybrid coders using motion-compensated prediction and transform coding. A separate section covers error resilience techniques for shape coding in MPEG-4     

A joint encoder–decoder error control framework for stereoscopic video coding

Stereoscopic video coding (SSVC) plays an important role in various 3D video applications. In SSVC, robust stereoscopic video transmission over error-prone networks is still a challenge problem to be solved. In this paper, we propose a joint encoder–decoder error control framework for SSVC, where error-resilient source coding, transmission network conditions, and error concealment scheme are jointly considered to achieve better error robustness performance. The proposed joint encoder–decoder error control framework includes two parts: an error concealment algorithm at the decoder side and a rate–distortion optimized error resilience algorithm at the encoder side. For error concealment at the decoder side, an overlapped block motion and disparity compensation based error concealment scheme is proposed to adaptively utilize inter-view correlations and temporal correlations. For error resilience at the encoder side, first, the inter-view refreshment is proposed for SSVC to suppress error propagations. Then, an end-to-end distortion model for SSVC is derived, which jointly considers the transmission network conditions, inter-view refreshment, and error concealment tools at the decoder side. Finally, based on the derived end-to-end distortion model, the rate–distortion optimized error resilience algorithm is presented to adaptively select inter-view, inter- or intra-coding for SSVC. The experimental results show that the proposed joint encoder–decoder error control framework has superior error robustness performance for stereoscopic video transmission over error-prone networks.     

视频编码参数对目标识别性能影响的研究

国内外研究人员对图像目标分类识别和视频编码传输问题都分别进行了大量研究,但是对于视频编码参数对目标识别性能影响的定量关系,还没有公开的文献报导。针对这一问题,该文选择典型的目标识别算法可变部件模型(DPM)和最常用的视频编码方法H.264/AVC作用测试对象,通过设计的编码和检测实验,研究了码率和分辨率参数对视频目标识别性能的影响,并拟合了识别性能随码率和分辨率变化的函数关系。通过选取编码器合适的码率和分辨率工作参数,可以获得信道带宽与视频目标识别性能的折中,为设计不同视频应用的编码优化目标函数提供了依据。     

Side information creation for efficient Wyner–Ziv video coding: Classifying and reviewing

Video coding technologies have played a major role in the explosion of large market digital video applications and services. In this context, the very popular MPEG-x and H-26x video coding standards adopted a predictive coding paradigm, where complex encoders exploit the data redundancy and irrelevancy to ‘control’ much simpler decoders. This codec paradigm fits well applications and services such as digital television and video storage where the decoder complexity is critical, but does not match well the requirements of emerging applications such as visual sensor networks where the encoder complexity is more critical. The Slepian–Wolf and Wyner–Ziv theorems brought the possibility to develop the so-called Wyner–Ziv video codecs, following a different coding paradigm where it is the task of the decoder, and not anymore of the encoder, to (fully or partly) exploit the video redundancy. Theoretically, Wyner–Ziv video coding does not incur in any compression performance penalty regarding the more traditional predictive coding paradigm (at least for certain conditions). In the context of Wyner–Ziv video codecs, the so-called side information, which is a decoder estimate of the original frame to code, plays a critical role in the overall compression performance. For this reason, much research effort has been invested in the past decade to develop increasingly more efficient side information creation methods. This paper has the main objective to review and evaluate the available side information methods after proposing a classification taxonomy to guide this review, allowing to achieve more solid conclusions and better identify the next relevant research challenges. After classifying the side information creation methods into four classes, notably guess, try, hint and learn, the review of the most important techniques in each class and the evaluation of some of them leads to the important conclusion that the side information creation methods provide better rate-distortion (RD) performance depending on the amount of temporal correlation in each video sequence. It became also clear that the best available Wyner–Ziv video coding solutions are almost systematically based on the learn approach. The best solutions are already able to systematically outperform the H.264/AVC Intra, and also the H.264/AVC zero-motion standard solutions for specific types of content.     

Video coding with H.264/AVC: tools, performance, and complexity

H.264/AVC, the result of the collaboration between the ISO/IEC Moving Picture Experts Group and the ITU-T Video Coding Experts Group, is the latest standard for video coding. The goals of this standardization effort were enhanced compression efficiency, network friendly video representation for interactive (video telephony) and non-interactive applications (broadcast, streaming, storage, video on demand). H.264/AVC provides gains in compression efficiency of up to 50% over a wide range of bit rates and video resolutions compared to previous standards. Compared to previous standards, the decoder complexity is about four times that of MPEG-2 and two times that of MPEG-4 Visual Simple Profile. This paper provides an overview of the new tools, features and complexity of H.264/AVC.     

Advanced side information creation techniques and framework for Wyner–Ziv video coding

Recently, several distributed video coding (DVC) solutions based on the distributed source coding (DSC) paradigm have appeared in the literature. Wyner–Ziv (WZ) video coding, a particular case of DVC where side information is made available at the decoder, enable to achieve a flexible distribution of the computational complexity between the encoder and decoder, promising to fulfill novel requirements from applications such as video surveillance, sensor networks and mobile camera phones. The quality of the side information at the decoder has a critical role in determining the WZ video coding rate-distortion (RD) performance, notably to raise it to a level as close as possible to the RD performance of standard predictive video coding schemes. Towards this target, efficient motion search algorithms for powerful frame interpolation are much needed at the decoder. In this paper, the RD performance of a Wyner–Ziv video codec is improved by using novel, advanced motion compensated frame interpolation techniques to generate the side information. The development of these type of side information estimators is a difficult problem in WZ video coding, especially because the decoder only has available some reference, decoded frames. Based on the regularization of the motion field, novel side information creation techniques are proposed in this paper along with a new frame interpolation framework able to generate higher quality side information at the decoder. To illustrate the RD performance improvements, this novel side information creation framework has been integrated in a transform domain turbo coding based Wyner–Ziv video codec. Experimental results show that the novel side information creation solution leads to better RD performance than available state-of-the-art side information estimators, with improvements up to 2 dB; moreover, it allows outperforming H.264/AVC Intra by up to 3 dB with a lower encoding complexity.     

End-to-end optimized TCP-friendly rate control for real-time video streaming over wireless multi-hop networks

Rate control is an important issue in video streaming applications. The most popular rate control scheme over wired networks is TCP-Friendly Rate Control (TFRC), which is designed to provide optimal transport service for unicast multimedia delivery based on the TCP Reno’s throughput equation. It assumes perfect link quality, treating network congestion as the only reason for packet losses. Therefore, when used in wireless environment, it suffers significant performance degradation because of packet losses arising from time-varying link quality. Most current research focuses on enhancing the TFRC protocol itself, ignoring the tightly coupled relation between the transport layer and other network layers. In this paper, we propose a new approach to address this problem, integrating TFRC with the application layer and the physical layer to form a holistic design for real-time video streaming over wireless multi-hop networks. The proposed approach can achieve the best user-perceived video quality by jointly optimizing system parameters residing in different network layers, including real-time video coding parameters at the application layer, packet sending rate at the transport layer, and modulation and coding scheme at the physical layer. The problem is formulated and solved as to find the optimal combination of parameters to minimize the end-to-end expected video distortion constrained by a given video playback delay, or to minimize the video playback delay constrained by a given end-to-end video distortion. Experimental results have validated 2–4 dB PSNR performance gain of the proposed approach in wireless multi-hop networks by using H.264/AVC and NS-2.     

A high performance and low complexity sampling-based intra coding method for H.264/AVC

The H.264/AVC video coding standard can achieves higher compression performance than previous video coding standards, such as MPEG-2, MPEG-4, and H.263. Especially, in order to obtain the high coding performance in intra pictures, the H.264/AVC encoder employs various directional spatial prediction modes and the rate-distortion (RD) optimization technique inducing high computational complexity. For further improvement in the coding performance with the low computational complexity, we introduce a sampling-based intra coding method. The proposed method generates two sub-images, which are defined as a sampled sub-image and a prediction error sub-image in this paper, from an original image through horizontal or vertical sampling and prediction processes, and then each sub-image is encoded with different intra prediction modes, quantization parameters, and scanning patterns. Experimental results demonstrate that the proposed method significantly improves the intra coding performance and reduces the encoding complexity with the smaller number of the RD cost calculation process.     

Low-complexity video coding via power-rate-distortion optimization

Wireless multimedia sensor networks (WMSNs) have been potentially applicable for several emerging applications. The resources, i.e., power and bandwidth available to visual sensors in a WMSN are, however, very limited. Hence, it is important but challenging to achieve efficient resource allocation and optimal video data compression while maximizing the overall network lifetime. In this paper, a power-rate-distortion (PRD) optimized resource-scalable low-complexity multiview video encoding scheme is proposed. In our video encoder, both the temporal and interview information can be exploited based on the comparisons of extracted media hashes without performing motion and disparity estimations, which are known to be time-consuming. We present a PRD model to characterize the relationship between the available resources and the RD performance of our encoder. More specifically, an RD function in terms of the percentages for different coding modes of blocks and the target bit rate under the available resource constraints is derived for optimal coding mode decision. The major goal here is to design a PRD model to optimize a “motion estimation-free” low-complexity video encoder for applications with resource-limited devices, instead of designing a general-purpose video codec to compete compression performance against current compression standards (e.g., H.264/AVC). Analytic results verify the accuracy of our PRD model, which can provide a theoretical guideline for performance optimization under limited resource constraints. Simulation results on joint RD performance and power consumption (measured in terms of encoding time) demonstrate the applicability of our video coding scheme for WMSNs.     

结合空时域下采样与重建的H.264/AVC压缩性能优化

为提高H.264/AVC标准在带宽资源严重受限时的压缩效率,采用空时域相结合的编码思路,提出了一种基于运动检测的自适应抽帧方法,并结合空域下采样与重建研究了一种改进的H.264/AVC压缩性能优化框架。在编码端,原视频先空域下采样以减少空间分辨率,然后根据视频运动特征,采用不同抽帧模式自适应地降低帧率,再经H.264/AVC编码,有效降低了编码码率。在解码端,解码视频则采用与抽帧模式相对应的运动估计与补偿插帧方法重建出抽取帧,再利用超分辨率重建技术将视频恢复到原空间分辨率。实验结果表明,所提方法在低码率段的视频压缩性能优于H.264/AVC标准编解码及相关文献方法。     

Optimal mode selection and synchronization for robust videocommunications over error-prone networks

We describe an effective method for increasing error resilience of video transmission over bit error prone networks. Rate-distortion optimized mode selection and synchronization marker insertion algorithms are introduced. The resulting video communication system takes into account the channel condition and the error concealment method used by the decoder, to optimize video coding mode selection and placement of synchronization markers in the compressed bit stream. The effects of mismatch between the parameters used by the encoder and the parameters associated with the actual channel condition and the decoder error concealment method are evaluated. Results for the binary symmetric channel and wideband code division multiple access mobile network models are presented in order to illustrate the advantages of the proposed method     

Novel prediction schemes for error resilient video coding

A conventional video codec uses encoder reconstruction of previous frames for motion compensated prediction. This is designed to minimize the encoder prediction error and assumes error free transmission. In this paper we use a modified prediction mechanism both at the encoder and decoder and propose techniques to improve the error resilience of H.264/AVC when transmitted over error prone networks. In our schemes we provide greater emphasis on Intra pixels during the formation of the reference frame used for prediction, thereby achieving better resilience. We also incorporate leaky prediction to further improve the robustness. We apply leaky prediction selectively at a macroblock level based on a simple mean square error metric in order to reduce the bit-rate penalty. Substantial performance gains have been observed in simulations. The effectiveness of using leaky prediction can be observed in medium and fast moving video sequences.     

Perceptually driven video error protection using a distributed source coding approach

In video communication systems, the video signals are typically compressed and sent to the decoder through an error-prone transmission channel that may corrupt the compressed signal, causing the degradation of the final decoded video quality. In this context, it is possible to enhance the error resilience of typical predictive video coding schemes using as inspiration principles and tools from an alternative video coding approach, the so-called Distributed Video Coding (DVC), based on the Distributed Source Coding (DSC) theory. Further improvements in the decoded video quality after error-prone transmission may also be obtained by considering the perceptual relevance of the video content, as distortions occurring in different regions of a picture have a different impact on the user's final experience. In this context, this paper proposes a Perceptually Driven Error Protection (PDEP) video coding solution that enhances the error resilience of a state-of-the-art H.264/AVC predictive video codec using DSC principles and perceptual considerations. To increase the H.264/AVC error resilience performance, the main technical novelties brought by the proposed video coding solution are: (i) design of an improved compressed domain perceptual classification mechanism; (ii) design of an improved transcoding tool for the DSC-based protection mechanism; and (iii) integration of a perceptual classification mechanism in an H.264/AVC compliant codec with a DSC-based error protection mechanism. The performance results obtained show that the proposed PDEP video codec provides a better performing alternative to traditional error protection video coding schemes, notably Forward Error Correction (FEC)-based schemes.     

Requantization transcoding for H.264/AVC video coding

In this paper, efficient solutions for requantization transcoding in H.264/AVC are presented. By requantizing residual coefficients in the bitstream, different error components can appear in the transcoded video stream. Firstly, a requantization error is present due to successive quantization in encoder and transcoder. In addition to the requantization error, the loss of information caused by coarser quantization will propagate due to dependencies in the bitstream. Because of the use of intra prediction and motion-compensated prediction in H.264/AVC, both spatial and temporal drift propagation arise in transcoded H.264/AVC video streams. The spatial drift in intra-predicted blocks results from mismatches in the surrounding prediction pixels as a consequence of requantization. In this paper, both spatial and temporal drift components are analyzed. As is shown, spatial drift has a determining impact on the visual quality of transcoded video streams in H.264/AVC. In particular, this type of drift results in serious distortion and disturbing artifacts in the transcoded video stream. In order to avoid the spatially propagating distortion, we introduce transcoding architectures based on spatial compensation techniques. By combining the individual temporal and spatial compensation approaches and applying different techniques based on the picture and/or macroblock type, overall architectures are obtained that provide a trade-off between computational complexity and rate-distortion performance. The complexity of the presented architectures is significantly reduced when compared to cascaded decoder–encoder solutions, which are typically used for H.264/AVC transcoding. The reduction in complexity is particularly large for the solution which uses spatial compensation only. When compared to traditional solutions without spatial compensation, both visual and objective quality results are highly improved.