Motion Estimation Based on Spline Interpolation in H.264/AVC Video Coder for Videoconferencing Application: Performance Versus Computation Load

The powerful H.264/AVC video coder involves a large encoding computational cost than the existing video standards due mainly to the motion-compensated estimation scheme based on a full search of multiple reference frames in the sequence. This strategy decreases the residual errors of the predicted frames and may improve the performance of the video coder. However a great number of computations are usually wasted without improving significantly the quality of the decoded video mostly in videoconferencing applications. To reduce the encoding computational load and preserve the performance of the video coder, this paper proposes to substitute the motion-compensated estimation method implemented in H.264/AVC by a temporal spline interpolation. Simulations on several test sequences show that important encoding saving times are achieved with a competitive quality of the decoded video compared to the exhaustive search of multiple reference frames in the H.264/AVC video coder.     

A Novel SAO-based Filtering Technique for Reduction in Temporal Flickering Artifacts in H.265/HEVC

This paper presents a novel filtering technique based on sample adaptive offset (SAO) in H.265/high-efficiency video coding (HEVC) for reduction in the temporal flickering artifacts and improving the coding performance. SAO is a newly introduced technique for in-loop filtering in H.265/HEVC, which derives the offsets independently for each frame in the spatial domain without considering temporal frame correlation. As a result, the temporal distortion artifacts which will have a negative effect on the subjective quality, such as flickering artifacts, cannot be effectively addressed. In this paper, the rate-distortion optimization of the newly developed SAO method, referred to as Inter-SAO, is performed on the residual samples between adjacent frames. Inter-SAO and SAO in the reference software of H.265/HEVC (i.e., the test model HM) are then combined to form the novel in-loop filter-based method, denoted as 3D-SAO filtering method, where both spatial information and temporal information are effectively utilized to reduce the overall distortion in reconstructed videos. Compared with the SAO in HM, 3D-SAO has demonstrated its advanced performance for flickering artifacts suppression. Furthermore, 3D-SAO improves the coding efficiency compared with the SAO in HM with a performance gain of up to 0.91 dB in \(\Delta PSNR\), 1.74 dB in \(\Delta PSPNR\) and 7.33 % in BD-rate reduction.     

Edge oriented block motion estimation for video coding

Intensity-based block motion estimation and compensation algorithms are widely used to exploit temporal redundancies in video coding, although they suffer from several drawbacks. One of the problems is that blocks located on boundaries of moving objects are not estimated accurately. It causes poor motion-compensated prediction along the moving edges to which the human visual system is very sensitive. By considering the characteristics of block motions for typical image sequences, an intelligent classifier is proposed to separate blocks containing moving edges to improve on conventional intensity-based block matching approaches. The motion vectors of these blocks are computed using edge matching techniques, so that the motion-compensated frames are tied more closely to the physical features. The proposed method can then make use of this accurate motion information for edge blocks to compute the remaining non-edged blocks. Consequently, a fast and efficient block motion estimation algorithm is developed. Experimental results show that this approach gives a significant improvement in accuracy for motion-compensated frames and computational complexity, in comparison with the traditional intensity-based block motion estimation methods     

Nonlinear quantisation for pixel domain distributed video coding

A nonlinear quantisation algorithm for pixel domain distributed video codec (DVC) is proposed. A residual signal is generated at the encoder considering the Wyner-Ziv frame to be encoded and adjacent reference frames and this residual signal is quantised using a nonlinear quantiser. The proposed algorithm is simulated for a number of test video sequences and the results depict a significant improvement of rate distortion performance, by reducing the bit rate while keeping the same PSNR when compared with available pixel domain DVC codec that uses a linear quantiser.     

Adaptive reversible video watermarking based on motion-compensated prediction error expansion with pixel selection

In this study, a motion-compensated prediction error expansion-based adaptive reversible video watermarking algorithm is proposed. Blocks of motion-compensated frames are classified as smooth and non-smooth according to their prediction errors. Unlike the current reversible video watermarking methods that apply a single watermarking strategy to all blocks, the proposed method uses two different strategies for smooth and non-smooth blocks. This adaptive strategy is shown to increase watermarking capacity. In addition, an approach is suggested to detect those pixels causing high distortion in the watermarked video and they are not used in watermarking to limit the distortion occurring in the original video. Simulations show that the proposed method is superior to existing methods in terms of capacity and distortion.     

Rate-Distortion Optimized Motion-Compensated Prediction for Packet Loss Resilient Video Coding

A rate-distortion optimized motion-compensated prediction method for robust video coding is proposed. Contrasting methods from the conventional literature, the proposed approach uses the expected reconstructed distortion after transmission, instead of the displaced frame difference in motion estimation. Initially, the end-to-end reconstructed distortion is estimated through a recursive per-pixel estimation algorithm. Then the total bit rate for motion-compensated encoding is predicted using a suitable rate distortion model. The results are fed into the Lagrangian optimization at the encoder to perform motion estimation. Here, the encoder automatically finds an optimized motion compensated prediction by estimating the best tradeoff between coding efficiency and end-to-end distortion. Finally, rate-distortion optimization is applied again to estimate the macroblock mode. This process uses previously selected optimized motion vectors and their corresponding reference frames. It also considers intraprediction. Extensive computer simulations in lossy channel environments were conducted to assess the performance of the proposed method. Selected results for both single and multiple reference frames settings are described. A comparative evaluation using other conventional techniques from the literature was also conducted. Furthermore, the effects of mismatches between the actual channel packet loss rate and the one assumed at the encoder side have been evaluated and reported in this paper     

Level-set-based motion estimation algorithm for multiple reference frame motion estimation

Motion estimation (ME) has a variety of applications in image processing, pattern recognition, target tracking, and video compression. In modern video compression standards such as H.264/AVC and HEVC, multiple reference frame ME (MRFME) is adopted to reduce the temporal redundancy between successive frames in a video sequence. In MRFME, the motion search process is conducted using additional reference frames, thereby obtaining better prediction signal as compared to single reference frame ME (SRFME). However, its high computational complexity makes it difficult to be utilized in real-world applications. In order to reduce the computational complexity of MRFME, this paper proposes a level-set-based ME algorithm (LSME) without any penalty in the rate-distortion (RD) performance. First, the proposed algorithm partitions the motion search space into multiple level sets based on a rate constraint. The proposed algorithm then controls the ME process on the basis of the predetermined level sets. Experimental results show that the proposed algorithm reduces the ME time by up to 83.46% as compared to the conventional full search (FS) algorithm.     

Image sequence coding using quadtree-based block-matching motioncompensation and classified vector quantisation

The authors propose a new image sequence coding algorithm based on two crucial methods: quadtree segmentation and classified vector quantisation (CVQ). Overall coding rates are efficiently lowered by quadtree segmentation while visual quality is well preserved by a CVQ method. A moving-block extraction technique is employed to greatly improve the coding efficiency in the interframe coding mode. A quadtree efficiently segments the stationary background regions of interframe differential signals with various large-sized blocks, and the moving regions are extracted from the smallest blocks of 4×4 size during the growth of the quadtree. These moving regions are motion-compensated using a block-matching method based on 4×4 blocks and the residual signals of the motion-compensated moving regions are coded by CVQ. The stationary regions are simply replenished from the previous frame. The proposed coding scheme is effective for coding the sequential signals of video telephony or video conferencing at low bit rates     

一种新型的无损视频压缩算法

多次使用有损压缩技术压缩数字视频,会导致视频的质量大幅下降,从而降低了数字视频的可再利用率,为此设计了一种新型的无损视频压缩算法。 该算法的特点在于:(1)提出了一种改进的基于上下文树的算术编码来压缩运动补偿后的误差帧。(2)针对改进的算术编码,优化基于宏块的运动估计与补偿算法,以提高无损视频压缩算法的压缩率。对于压缩运动补偿后误差帧的算法,与静态图像无损压缩算法JPEG-LS和CALIC相比表明,该文设计的无损视频压缩算法的压缩率超过JPEG-LS算法最高为23.3%,超过CALIC算法最高为19.3%。     

Homography-based block motion estimation for video coding of PTZ cameras

Due to the constrained movement of pan-tilt-zoom (PTZ) cameras, two frames in the video sequences captured by such cameras can be geometrically related by a relationship (homography). This geometric relationship is helpful for reducing the spatial redundancy in video coding. In this paper, by exploiting the homography between two frames with optical flow tracking algorithm, we propose a novel homography-based search (HBS) algorithm for block motion estimation in coding the sequences captured by PTZ cameras. In addition, adaptive thresholds are adopted in our method to classify different kinds of blocks. Compared with other traditional fast algorithms, the proposed HBS algorithm is proved to be more efficient for the sequences captured by PTZ cameras. And compared to our previous work in ICME (Cui et al., 2011), which only deals with pan-tilt (PT) camera and calculates the homography with mechanical devices, in this extended work we compute the homography by using information on images instead.     

Multiview video compression with 1-D transforms

Many alternative transforms have been developed recently for improved compression of images, intra prediction residuals or motion-compensated prediction residuals. In this paper, we propose alternative transforms for multiview video coding. We analyze the spatial characteristics of disparity-compensated prediction residuals, and the analysis results show that many regions have 1-D signal characteristics, similar to previous findings for motion-compensated prediction residuals. Signals with such characteristics can be transformed more efficiently with transforms adapted to these characteristics and we propose to use 1-D transforms in the compression of disparity-compensated prediction residuals in multiview video coding. To show the compression gains achievable from using these transforms, we modify the reference software (JMVC) of the multiview video coding amendment to H.264/AVC so that each residual block can be transformed either with a 1-D transform or with the conventional 2-D Discrete Cosine Transform. Experimental results show that coding gains ranging from about 1–15% of Bjontegaard-Delta bitrate savings can be achieved.     

DVFS-aware motion estimation design scheme based on bandwidth–rate–distortion optimization in application processor systems

In this paper, a novel dynamic voltage–frequency scaling-aware (DVFS-aware) bandwidth- efficient motion estimation (ME) scheme is presented for mobile application processor (AP) systems. Under volatile operating performance conditions due to the power management mechanism, we model the coding bandwidth (BW) and coding performance for the video processor as a convex function of the working frequency. In this paper, we present a bandwidth–rate–distortion (B–R–D) optimized framework that will guarantee the smallest possible rate–distortion cost among coding BW constraints applied in video coding design. By formulating the coding bandwidth-constrained ME problem as an optimization problem, known convex optimization theory can be applied to yield optimal resource-constrained compression. Using varied CIF (352×288)- and HP (1280×720)-sized video sequences with different motion activities over our proposed DVFS-aware video coding approach, the excellent results in terms of coding performance and coding bandwidth savings are obtained. With negligible quality loss, the proposed scheme under coding BW constraints achieves 45–65% coding BW usage reduction over HD-sized 30 frame/s video coding.     

Fast motion and disparity estimation for HEVC based 3D video coding

The emerging international standard for high efficiency video coding (HEVC) based 3D video coding (3D-HEVC) is an extension of HEVC. In the test model of 3D-HEVC, variable size motion estimation (ME) and disparity estimation (DE) are both employed to select the best coding mode for each treeblock in the encoding process. This technique achieves the highest possible coding efficiency, but it brings extremely high computational complexity which limits 3D-HEVC from practical applications. In this paper, a fast ME/DE algorithm based on inter-view and spatial correlations is proposed to reduce 3D-HEVC computational complexity. Since the multi-view videos represent the same scene with similar characteristic, there is a high correlation among the coding information from inter-view prediction. Besides, the homogeneous regions in texture video have a strong spatial correlation, and thus spatially neighboring treeblocks have similar coding information. Therefore, we can determine ME search range and skip some specific ME and DE rarely used in the previously coded view frames and spatially neighboring coding unit. Experimental results demonstrate that the proposed algorithm can significantly reduce computational complexity of 3D-HEVC encoding while maintaining almost the same rate-distortion performance.     

Motion estimation in low-delay hierarchical p-frame coding using motion vector composition

Low-delay hierarchical prediction structure is currently adopted in various new video coding standards. The only hurdle of this structure is the need of motion estimation in distant reference frames. To maintain high coding efficiency, a large search range for motion estimation can improve the coding efficiency in distant reference pictures. Computational complexity will thus be increased dramatically. In this paper, a fast motion estimation scheme for a low-delay hierarchical P-frame structure is proposed. The proposed scheme adopts a motion vector composition strategy to expedite the motion estimation process for distant reference frames in the hierarchical P structure. In addition, a motion vector composition algorithm is tailor-made with the proposed hierarchical P coding scheme to further improve the coding efficiency. Simulation results show that the proposed scheme can deliver a remarkable complexity savings and coding efficiency improvement on coding a frame in low temporal layers of the hierarchical P structure.     

一种基于立体视邻接帧时空相关性的最小代价函数帧估计算法

为了在立体视频序列编码中获得高的压缩率,需要对立体视频序列中一个视的序列按传统方法进行独立编码;另一个视的序列中,只对其中一些参考帧(I帧或P帧)按视差补偿预测的方法进行编码,其余帧不进行编码和传输,而在解码端用立体视帧估计的方法得到重建.本文提出了一种基于立体视中邻接帧在图像、视差场和运动矢量场之间高度相关性的方法.对于因遮挡而缺乏估计的区域,则结合了图像强度的连续性和运动,视差矢量的分布特性,构造了代价方程并估计出该部分的运动矢量及强度值.实验证明,重建出来的帧图像在视觉和信噪比意义上均具有较好的效果.     

Temporal image sequence prediction using motion field interpolation

A new method for motion-compensated temporal prediction of image sequences is proposed. Motion vector fields in natural scenes should possess two basic properties. First, the field should be smoothly varying within moving objects to compensate for nonrigid or rotational motion, and scaling of objects. Second, the field should be discontinuous along the boundaries of the objects. In the proposed method the motion vector field is modelled using finite element methods and interpolated using adaptive interpolators to satisfy the above-stated requirements. This is particularly important when only very sparse estimates of motion vector fields are available in the decoder due to bit-rate constraints limiting the amount of overhead information that can be transmitted. The proposed prediction method can be applied for low-bit-rate video coding in conventional codecs based on motion-compensated prediction and transform coding, as well as in model-based codecs. The performance of the proposed method is compared with standard motion-compensated prediction based on block matching. It is shown that for simple video telephony scenes a reduction of more than 30% in the energy of the prediction error can be achieved with an unchanged number of transmitted motion vectors and with only a modest increase in computational complexity. When implemented in an H.261 codec the new prediction method can improve the peak SNR 1–2 dB producing a significant visual improvement.     

Unconstrained motion compensated temporal filtering (UMCTF) for efficient and flexible interframe wavelet video coding

We introduce an efficient and flexible framework for temporal filtering in wavelet-based scalable video codecs called unconstrained motion compensated temporal filtering (UMCTF). UMCTF allows for the use of different filters and temporal decomposition structures through a set of controlling parameters that may be easily modified during the coding process, at different granularities and levels. The proposed framework enables the adaptation of the coding process to the video content, network and end-device characteristics, allows for enhanced scalability, content-adaptivity and reduced delay, while improving the coding efficiency as compared to state-of-the-art motion-compensated wavelet video coders. Additionally, a mechanism for the control of the distortion variation in video coding based on UMCTF employing only the predict step is proposed. The control mechanism is formulated by expressing the distortion in an arbitrary decoded frame, at any temporal level in the pyramid, as a function of the distortions in the reference frames at the same temporal level. All the different scenarios proposed in the paper are experimentally validated through a coding scheme that incorporates advanced features (such as rate-distortion optimized variable block-size multihypothesis prediction and overlapped block motion compensation). Experiments are carried out to determine the relative efficiency of different UMCTF instantiations, as well as to compare against the current state-of-the-art in video coding.     

The Efficiency of Motion-Compensating Prediction for Hybrid Coding of Video Sequences

Performance bounds for generalized hybrid coding of video sequences with motion-compensating prediction are derived based on rate-distortion theory. It is shown that the spatial power spectrum of the motion-compensated prediction error can be calculated from the signal power spectrum and the displacement estimation error p.d.f.. A spatial Wiener filter can improve the efficiency of motion-compensating prediction. Memoryless encoding of the motion-compensated prediction error and intraframe encoding of the motion-compensated prediction error are compared. An evaluation of the rate-distortion functions for a typical videoconference sampling format shows that for integer pel accuracy of the displacement estimate the additional gain by motion-compensating prediction over pure intraframe coding is limited to ∼ 0.8 bits/sample in moving areas. Required accuracies of the displacement estimate for a gain of motion-compensating interframe coding over intraframe coding are given.     

Video coding using fast geometry-adaptive partitioning and an elastic motion model

Effective motion-compensated prediction is the key to high-performance video coding. To ensure continuous improvement of video coders, emerging motion-compensation technologies will need to be successfully integrated into future standards. Higher order elastic motion models and geometry-adaptive block partitioning are such advanced techniques that are good candidates for integration into future generations of video coders. However, it is vital that these techniques are additive in performance, non-interfering and maintain justifiable complexity. In this paper, we propose an efficient block-partitioning scheme that incorporates both geometry-adaptive partitioning and an elastic motion model as extensions to the standard motion estimation procedure. Our experiments suggest that geometric partitioning in combination with the use of an elastic motion model can provide enhanced performance, although the increased complexity is of some concern for real-time applications.