Hybrid model-and-object-based real-time conversational video coding

Bandwidth-constrained real-time conversational video communications (such as mobile teleconferencing) require video codecs with good rate-distortion characteristics at low bit-rates and modest computational complexity. While target-specific object-based and model-based coding methods have been proposed for low bit-rate conversational video coding, difficulties in generalization and high computational complexity hinder their practical utilization. In this paper, we propose a low bit-rate coding method for typical conversational video by combining two-dimensional model-based coding of face regions and object-based coding of non-face head-shoulder regions, achieving high-quality face reconstruction and low overall bit-rate with real-time encoding capability. Experiments on typical conversational test sequences confirm that, compared to other conversational video codecs, our model-and-object-based coding method offers superior rate-distortion performance at low bit-rates.     

两种实时、低码率视频编码的运动估计搜索算法

先提出实时、低码率视频编码对运动估计搜索算法在运算时间和准确度上的要求，总结归纳了钻石搜索法和两步法的具体算法，通过与几种经典算法作比较，说明这两种算法更优，更适合用于实时，低码率视频编码，最后，以H．263的实现为例，说明了钻石搜索算法的具体实现。     

Joint on-the-fly network coding/video quality adaptation for real-time delivery

This paper introduces a redundancy adaptation algorithm based on an on-the-fly erasure network coding scheme named Tetrys in the context of real-time video transmission. The algorithm exploits the relationship between the redundancy ratio used by Tetrys and the gain or loss in encoding bit rate from changing a video quality parameter called the Quantization Parameter (QP). Our evaluations show that with equal or less bandwidth occupation, the video protected by Tetrys with redundancy adaptation algorithm obtains a PSNR gain up to or more than 4 dB compared to the video without Tetrys protection. We demonstrate that the Tetrys redundancy adaptation algorithm performs well with the variations of both loss pattern and delay induced by the networks. We also show that Tetrys with the redundancy adaptation algorithm outperforms traditional block-based FEC codes with and without redundancy adaptation.     

Adaptive video coding control for real-time H.264/AVC encoder

In this paper, we present a new adaptive video coding control for real-time H.264/AVC encoding system. The main techniques include: (1) the initial quantization parameter (QP) decision scheme is based on Laplacian of Gaussian (LoG) operators; (2) the MB-level QP calculation is based on the spatio-temporal correlation, in which the computation is less than the quadratic model used by H.264/AVC; (3) the adaptive GOP structure is proposed, in which the I-frame is adaptively replaced by an enhancement P-frame to improve the coding efficiency; (4) the scene change is detected with the complexity of adjacent inter-frames and the appropriate QP is re-calculated for the scene-change frame. The proposed algorithm is not only to save the computational complexity but also to improve coding quality. Compared to the JM12.4 reference under various sequences testing, the proposed algorithm can decrease coding time by 64.5% and improve PSNR by 1.52 dB while keeping the same bit-rate.     

Context-based entropy coding in AVS video coding standard

In this paper, two context-based entropy coding schemes for AVS Part-2 video coding standard are presented. One is Context-based 2D Variable Length Coding (C2DVLC) as a low complexity entropy coding scheme for AVS Part-2 Jizhun profile. C2DVLC uses multiple 2D-VLC tables to exploit the statistical features of DCT coefficients for higher coding efficiency. Exponential–Golomb codes are applied in C2DVLC to code the pairs of the run-length of zero coefficients and the non-zero coefficients for lower storage requirement. The other is Context-based Binary Arithmetic Coding (CBAC) as an enhanced entropy coding scheme for AVS Part-2 Jiaqiang profile. CBAC utilizes all previously coded coefficient magnitudes in a DCT block for context modeling. This enables adaptive arithmetic coding to exploit the redundancy of the high-order Markov process in DCT domain with a few contexts. In addition, a context weighting technique is used to further improve CBAC's coding efficiency. Moreover, CBAC is designed to be compatible to C2DVLC in coding elements which simplifies the implementations. The experimental results demonstrate that both C2DVLC and CBAC can achieve comparable or even slightly higher coding performance when compared to Context-Adaptive Variable Length Coding (CAVLC) in H.264/AVC baseline profile and Context-Based Adaptive Binary Arithmetic Coding (CABAC) in H.264/AVC main profile respectively.     

Developments in model-based video coding

This paper reports on current developments in the area of model-based video coding, a technique which shows promise of achieving very large bit-rate reductions for moving images. After an introduction and historical review, advances are summarized in several areas, among them improved 3D tracking of the human head and of facial expressions, the use of muscle-driven model animation with skin synthesis, techniques for luminance compensation, and switched coders. Bit rates ranging from 64 kb/s down to about 1 kb/s have been obtained using head-and-shoulder video sequences. Problems with model-based methods are identified and future developments in both CBR and VBR transmission discussed     

Perceptual-based distributed video coding

In this paper, we propose a perceptual-based distributed video coding (DVC) technique. Unlike traditional video codecs, DVC applies video prediction process at the decoder side using previously received frames. The predicted video frames (i.e., side information) contain prediction errors. The encoder then transmits error-correcting parity bits to the decoder to reconstruct the video frames from side information. However, channel codes based on i.i.d. noise models are not always efficient in correcting video prediction errors. In addition, some of the prediction errors do not cause perceptible visual distortions. From perceptual coding point of view, there is no need to correct such errors. This paper proposes a scheme for the decoder to perform perceptual quality analysis on the predicted side information. The decoder only requests parity bits to correct visually sensitive errors. More importantly, with the proposed technique, key frames can be encoded at higher rates while still maintaining consistent visual quality across the video sequence. As a result, even the objective PSNR measure of the decoded video sequence will increase too. Experimental results show that the proposed technique improves the R-D performance of a transform domain DVC codec both subjectively and objectively. Comparisons with a well-known DVC codec show that the proposed perceptual-based DVC coding scheme is very promising for distributed video coding framework.     

Multilayer subband-based video coding

A motion compensated interframe subband coding algorithm suitable for a wide range of video coding applications is described. In this approach the spectrum of each frame of video signal is first decomposed into smaller frequency bands where each can then be coded accordingly. For the best performance a combination of hybrid DCT/DPCM (discrete cosine transform/differential pulse code modulation), interframe DPCM, and intraframe PCM was considered. To preserve its hierarchical structure each band is coded independently of higher frequency bands but can share information with the lower bands. A simulation was carried out for HDTV sequences     

Rate-distortion-constrained subband video coding

This paper introduces a subband video coding algorithm for operation over a continuum of rates from very low to very high. The key elements of the system are statistical rate-distortion-constrained motion estimation and compensation, multistage residual quantization, high order statistical modeling, and arithmetic coding. The method is unique in that it provides an improved mechanism for dynamic spatial and temporal coding. Motion vectors are determined in a nontraditional way, using a rate-distortion cost criterion. This results in a smoother and more consistent motion field, relative to that produced by conventional block matching algorithms. Control over the system computational complexity and performance may be exercised easily     

Constant quality video coding using video content analysis

In the literature, several rate control techniques have been proposed to aim at the optimal quality of digitally encoded video under given bit budget, channel rate and buffer size constraints. Typically, these approaches are group-of-picture (GOP) based. For longer, heterogeneous sequences, they become unacceptably complex or struggle with model mismatches. In this paper, an off-line segment-based rate control approach is proposed for controlling the distortion variation across successive shots of a video sequence when encoding with single-layer (MPEG-4 baseline, MPEG-4 AVC) and scalable (wavelet) video codecs. Consistent quality is achieved by optimally distributing the available bits among the different segments, based on efficient rate-distortion (R-D) modelling of each segment. The individual segments are defined based on shot segmentation and activity analysis techniques. The algorithm is formulated for three different distribution models: download, progressive download and streaming. The results indicate that the proposed technique improves the quality consistency significantly, while the processing overhead compared to classical two-pass variable bit-rate (VBR) encoding is limited.     

On entropy minimization for near-lossless differential coding

In this letter we investigate near-lossless signal coding for which the reconstruction signal is required to have an absolute error in each component bounded by a tolerance τ. For differential coding no practical algorithm is known that computes an optimal reconstruction, i.e., one for which the sequence of consecutive differences has minimal entropy. In this letter we give the optimal lower bound for the entropy of the differences of the reconstruction and, in addition, present tighter bounds for some special cases     

Fast coding unit partitioning algorithms for versatile video coding intra coding

Versatile video coding (VVC) is the newest video compression standard. It adopts quadtree with nested multi-type tree (QT-MTT) to encode square or rectangular coding units (CUs). The QT-MTT coding structure is more flexible for encoding video texture, but it is also accompanied by many time-consuming algorithms. So, this work proposes fast algorithms to determine horizontal or vertical split for binary or ternary partition of a 32 × 32 CU in the VVC intra coding to replace the rate-distortion optimization (RDO) process, which is time-consuming. The proposed fast algorithms are actually a two-step algorithm, including feature analysis method and deep learning method. The feature analysis method is based on variances of pixels, and the deep learning method applies the convolution neural networks (CNNs) for classification. Experimental results show that the proposed method can reduce encoding time by 28.94% on average but increase Bjontegaard delta bit rate (BDBR) by about 0.83%.     

Layered Wyner-Ziv video coding.

Following recent theoretical works on successive Wyner-Ziv coding (WZC), we propose a practical layered Wyner-Ziv video coder using the DCT, nested scalar quantization, and irregular LDPC code based Slepian-Wolf coding (or lossless source coding with side information at the decoder). Our main novelty is to use the base layer of a standard scalable video coder (e.g., MPEG-4/H.26L FGS or H.263+) as the decoder side information and perform layered WZC for quality enhancement. Similar to FGS coding, there is no performance difference between layered and monolithic WZC when the enhancement bitstream is generated in our proposed coder. Using an H.26L coded version as the base layer, experiments indicate that WZC gives slightly worse performance than FGS coding when the channel (for both the base and enhancement layers) is noiseless. However, when the channel is noisy, extensive simulations of video transmission over wireless networks conforming to the CDMA2000 1X standard show that H.26L base layer coding plus Wyner-Ziv enhancement layer coding are more robust against channel errors than H.26L FGS coding. These results demonstrate that layered Wyner-Ziv video coding is a promising new technique for video streaming over wireless networks.     

视频编码预处理算法研究

针对现有降噪采用时空域滤波、去块效应采用环路滤波/后处理这种分而治之方案的缺点,通过分析降噪的基本技术、低码率视频应用中块效应产生的原因和去块效应的常用方法,从滤除不需要或相对不重要的高频离散余弦变换(DCT,discrete cosine transform)系数角度提出了降噪和去块效应可同时处理的设想。新提出的预处理算法是根据图像纹理特性、运动情况和码率约束等,在图像的不同区域采用不同强度的自适应双边滤波,这样不仅去噪,滤除不重要的细节以便于高效压缩,避免块效应的产生,同时也可起到一定程度的码率控制效果。     

Three-dimensional subband coding of video

We describe and show the results of video coding based on a three-dimensional (3-D) spatio-temporal subband decomposition. The results include a 1-Mbps coder based on a new adaptive differential pulse code modulation scheme (ADPCM) and adaptive bit allocation. This rate is useful for video storage on CD-ROM. Coding results are also shown for a 384-kbps rate that are based on ADPCM for the lowest frequency band and a new form of vector quantization (geometric vector quantization (GVQ)) for the data in the higher frequency bands. GVQ takes advantage of the inherent structure and sparseness of the data in the higher bands. Results are also shown for a 128-kbps coder that is based on an unbalanced tree-structured vector quantizer (UTSVQ) for the lowest frequency band and GVQ for the higher frequency bands. The results are competitive with traditional video coding techniques and provide the motivation for investigating the 3-D subband framework for different coding schemes and various applications.     

Reconfigurable video coding on multicore

The current monolithic and lengthy scheme behind the standardization and the design of new video coding standards is becoming inappropriate to satisfy the dynamism and changing needs of the video coding community. Such scheme and specification formalism does not allow the clear commonalities between the different codecs to be shown, at the level of the specification nor at the level of the implementation. Such a problem is one of the main reasons for the typically long interval elapsing between the time a new idea is validated until it is implemented in consumer products as part of a worldwide standard. The analysis of this problem originated a new standard initiative within the International Organization for Standardization (ISO)/International Electrotechnical Commission (IEC) Moving Pictures Experts Group (MPEG) committee, namely Reconfigurable Video Coding (RVC). The main idea is to develop a video coding standard that overcomes many shortcomings of the current standardization and specification process by updating and progressively incrementing a modular library of components. As the name implies, flexibility and reconfigurability are new attractive features of the RVC standard. Besides allowing for the definition of new codec algorithms, such features, as well as the dataflow-based specification formalism, open the way to define video coding standards that expressly target implementations on platforms with multiple cores. This article provides an overview of the main objectives of the new RVC standard, with an emphasis on the features that enable efficient implementation on platforms with multiple cores. A brief introduction to the methodologies that efficiently map RVC codec specifications to multicore platforms is accompanied with an example of the possible breakthroughs that are expected to occur in the design and deployment of multimedia services on multicore platforms.     

Error resilient video coding techniques

We review error resilience techniques for real-time video transport over unreliable networks. Topics covered include an introduction to today's protocol and network environments and their characteristics, encoder error resilience tools, decoder error concealment techniques, as well as techniques that require cooperation between encoder, decoder, and the network. We provide a review of general principles of these techniques as well as specific implementations adopted by the H.263 and MPEG-4 video coding standards. The majority of the article is devoted to the techniques developed for block-based hybrid coders using motion-compensated prediction and transform coding. A separate section covers error resilience techniques for shape coding in MPEG-4     

Low complexity distributed video coding

ContextConventional video encoding is a computationally intensive process that requires a lot of computing resources, power and memory. Such codecs cannot be deployed in remote sensors that are constrained in terms of power, memory and computational capabilities. For such applications, distributed video coding might hold the answer.ObjectiveIn this paper, we propose a distributed video coding (DVC) architecture that adheres to the principles of DVC by shifting the computational complexity from the encoder to the decoder and caters to low-motion scenarios like video conferencing and surveillance of hallways and buildings.MethodThe architecture presented is block-based and introduces a simple yet effective classification scheme that aims at maximizing the use of skip blocks to exploit temporal correlation between consecutive frames. In addition to the skip blocks, a dynamic GOP size control algorithm is proposed that instantaneously alters the GOP size in response to the video statistics without causing any latency and without the need to buffer additional frames at the encoder. To facilitate real-time video delivery and consumption, iterative channel codes like low density parity check codes and turbo codes are not used and in their place a Bose–Chaudhuri–Hocquenghem (BCH) code with encoder rate control is used.ResultsIn spite of reducing the complexity and eliminating the feedback channel, the proposed architecture can match and even surpass the performance of current DVC systems making it a viable solution as a codec for low-motion scenarios.ConclusionWe conclude that the proposed architecture is a suitable solution for applications that require real-time, low bit rate video transmission but have constrained resources and cannot support the complex conventional video encoding solutions.Practical implicationsThe practical implications of the proposed DVC architecture include deployment in remote video sensors like hallway and building surveillance, video conferencing, video sensors that are deployed in remote regions (wildlife surveillance applications), and capsule endoscopy.     

面向HEVC的恰可察觉编码失真模型

为进一步提高现有视频编码技术的压缩效率及解 码重建图像的主观视觉感知质量,在现有人眼恰可 察觉失真(JND,just noticeable distortion)模型的基础上, 提出了恰可察觉编码失真(JNCD,just noticeable coding dist ortion)模型。首先,通过主观实验,对恰可察觉梯 度幅值差异(JNGD,just noticeable gradient difference)进 行了研究,分析其变化规律并建立JNGD模型。使用全变 分(TV,total variation)方法将图像分解为结构图和纹理图后,分别求 取其梯度信息得到结构梯度图和纹理梯度图, 利用JNGD模型分别滤除结构梯度图和纹理梯度图中的人眼不可察觉的梯度幅值 ;其后,分析了人眼感知对于不同 梯度幅值的编码失真敏感性,设计了梯度幅值与JNCD值的主观实验,得到两者的关系模型; 最后,考虑人眼对图 像中的边缘、平坦和纹理3类区域失真感知程度的差异性,利用滤波后的结构梯度和纹理梯 度信息将图像划分为上 述3类区域,最终建立整幅图像的JNCD模型。为验证本文提出的JNCD模型的可靠性,在高效 视频编码(HEVC)标准测试平台上进行的模型验证结果表明,在本模 型指导下的编码其解码重建 图像获得了较好的主观视觉效果,可为人眼视觉感知冗余的分析及感知编码的改 进提供依据。