Fuzzy quantization based bit transform for low bit-resolution motion estimation

This study proposes a novel fuzzy quantization based bit transform for low bit-resolution motion estimation. We formalize the procedure of bit resolution reduction by two successive steps, namely interval partitioning and interval mapping. The former is a many-to-one mapping which determines motion estimation performance, while the latter is a one-to-one mapping. To gain a reasonable interval partitioning, we propose a non-uniform quantization method to compute coarse thresholds. They are then refined by using a membership function to solve the mismatch of pixel values near threshold caused by camera noise, coding distortion, etc. Afterwards, we discuss that the sum of absolute difference (SAD) is one of the fast matching metrics suitable for low bit-resolution motion estimation in the sense of mean squared errors. A fuzzy quantization based low bit-resolution motion estimation algorithm is consequently proposed. Our algorithm not only can be directly employed in video codecs, but also be applied to other fast or complexity scalable motion estimation algorithms. Extensive experimental results show that the proposed algorithm can always achieve good motion estimation performances for video sequences with various characteristics. Compared with one-bit transform, multi-thresholding two-bit transform, and adaptive quantization based two-bit transform, our bit transform separately gains 0.98 dB, 0.42 dB, and 0.24 dB improvement in terms of average peak signal-to-noise ratio, with less computational cost as well.     

Packet prioritization for H.264/AVC video with cyclic intra-refresh line

Insertion of a cyclic intra-refresh line is a lightweight way of mitigating spatio-temporal error propagation in a video stream transmitted over a mobile network. This paper presents low-complexity yet effective prioritization based on slice position within a video frame relative to the cyclic refresh line. Two prioritization schemes are compared. The first is a region-based method, while the second, which is packet-based, improves packet classification. Experimental results indicate that, the packet-based scheme can achieve video quality gains of up to 4 dB, compared to when the scheme is not used. The proposed schemes require no decoder modifications and do not introduce an increase in bitrate or in computational complexity.     

Adaptive intra-refresh for low-delay error-resilient video coding

Low-delay and error-resilient video coding is critical for real-time video communication over wireless networks. Intra-refresh coding, which embeds intra coded regions into inter frames can achieve a relatively smooth bit-rate and terminate the error propagation caused by the transmission loss. In this paper, we proposed a novel linear model for the intra-refresh cycle-size selection adapting to the network packet loss rates and the motions in the video content. We also analyze issues in designing the intra-refresh coding pattern and the refresh order, and propose a strategy which can adapt to different cycle-size and obtain better R–D performance compared with traditional random intra-refresh and vertical-partition intra-refresh. Experimental results show that the linear cycle-size selection model works effectively, where a 3 dB improvement can be achieved compared with a fixed cycle-size. Also, with the proposed intra-refresh order, a 2.0% bitrate reduction is obtained in average compared with the vertical-partition intra-refresh.     

Advanced side information creation techniques and framework for Wyner–Ziv video coding

Recently, several distributed video coding (DVC) solutions based on the distributed source coding (DSC) paradigm have appeared in the literature. Wyner–Ziv (WZ) video coding, a particular case of DVC where side information is made available at the decoder, enable to achieve a flexible distribution of the computational complexity between the encoder and decoder, promising to fulfill novel requirements from applications such as video surveillance, sensor networks and mobile camera phones. The quality of the side information at the decoder has a critical role in determining the WZ video coding rate-distortion (RD) performance, notably to raise it to a level as close as possible to the RD performance of standard predictive video coding schemes. Towards this target, efficient motion search algorithms for powerful frame interpolation are much needed at the decoder. In this paper, the RD performance of a Wyner–Ziv video codec is improved by using novel, advanced motion compensated frame interpolation techniques to generate the side information. The development of these type of side information estimators is a difficult problem in WZ video coding, especially because the decoder only has available some reference, decoded frames. Based on the regularization of the motion field, novel side information creation techniques are proposed in this paper along with a new frame interpolation framework able to generate higher quality side information at the decoder. To illustrate the RD performance improvements, this novel side information creation framework has been integrated in a transform domain turbo coding based Wyner–Ziv video codec. Experimental results show that the novel side information creation solution leads to better RD performance than available state-of-the-art side information estimators, with improvements up to 2 dB; moreover, it allows outperforming H.264/AVC Intra by up to 3 dB with a lower encoding complexity.     

Dictionary learning based reconstruction for distributed compressed video sensing

Distributed compressed video sensing (DCVS) is a framework that integrates both compressed sensing and distributed video coding characteristics to achieve a low-complexity video coding. However, how to design an efficient reconstruction by leveraging more realistic signal models that go beyond simple sparsity is still an open challenge. In this paper, we propose a novel “undersampled” correlation noise model to describe compressively sampled video signals, and present a maximum-likelihood dictionary learning based reconstruction algorithm for DCVS, in which both the correlation and sparsity constraints are included in a new probabilistic model. Moreover, the signal recovery in our algorithm is performed during the process of dictionary learning, instead of being employed as an independent task. Experimental results show that our proposal compares favorably with other existing methods, with 0.1–3.5 dB improvements in the average PSNR, and a 2–9 dB gain for non-key frames when key frames are subsampled at an increased rate.     

Correlation based universal image/video coding loss recovery

Coding artifacts are annoying in highly compressed signals. Most of the existing artifact reduction methods are designed for one specific type of artifacts, codecs, and bitrates, which are complex and exclusive for one type of artifact reduction. Since both the compressed image/video and the coding error contain information of the original signal, they are highly correlated. Therefore, we try to recover some lost data based on the correlation between the compressed signal and the coding error, and introduce a novel and universal artifact reduction method. Firstly, according to the spatial correlation among pixels, a pixel-adaptive anisotropic filter is designed to reconstruct the distorted signal. Next, a globally optimal filter is designed to further recover the coding loss. Experimental results demonstrate that within an extensive range of bitrates, the proposed method achieves about 0.8 dB, 0.45 dB, 0.3 dB, and 0.2 dB on average of PSNR improvement for JPEG, MPEG4, H.264/AVC, and HEVC compressed signals, respectively.     

A 3D-DCT video encoder using advanced coding techniques for low power mobile device

The three-dimensional discrete cosine transform (3D-DCT) has been researched as an alternative to existing dominant video standards based on motion estimation and compensation. Since it does not need to search macro block for inter/intra prediction, 3D-DCT has great advantages for complexity. However, it has not been developed well because of poor video quality while video standards such as H.263(+) and HEVC have been blooming. In this paper, we propose a new 3D-DCT video coding as a new video solution for low power mobile technologies such as Internet of Things (IoT) and Drone. We focus on overcoming drawbacks reported in previous research. We build a complete 3D-DCT video coding system by adopting existing advanced techniques and devising new coding algorithms to improve overall performance of 3D-DCT. Experimental results show proposed 3D-DCT outperforms H.264 low power profiles while offering less complexity. From GBD-PSNR, proposed 3D-DCT provides better performance by average 4.6 dB.     

Rate-distortion optimization of scalable video codecs

In this paper joint optimization of layers in the layered video coding is investigated. Through theoretical analysis and simulations, it is shown that, due to higher interactions between the layers in a SNR scalable codec, this type of layering technique benefits most from joint optimization of the layers. A method for joint optimization is then proposed, and its compression efficiency is contrasted against the separate optimization and an optimized single layer coder. It is shown that, in joint optimization of SNR scalable coders when the quantization step size of the enhancement layer is larger than half the step size of the base layer, an additional improvement is gained by not sending the enhancement zero valued quantized coefficients, provided they are quantized at the base-layer. This will result in a non-standard bitstream syntax and as an alternative for standard syntax, one may skip the inter coded enhancement macroblocks. Through extensive tests it is shown that while separate optimization of SNR coders is inferior to single layer coder by more than 2 dB, with joint optimization this gap is reduced to 0.3–0.5 dB. We have shown that through joint optimization quality of the base layer video is also improved over the separate optimization. It is also shown that spatial scalability like SNR scalability does benefit from joint optimization, though not being able to exploit the relation between the quantizer step sizes. The amount of improvement depends on the interpolation artifacts of upsampled base-layer and the residual quantization distortion of this layer. Hence, the degree of improvement depends on image contents as well as the bit rate budget. Simulation results show that joint optimization of spatial scalable coders is about 0.5–1 dB inferior to the single layer optimized coder, where its separate optimization counterpart like SNR scalability is more than 2 dB worse.     

Inter-layer correlation-based adaptive bit allocation for enhancement layer in scalable high efficiency video coding

To facilitate the diversity of network and end-user devices, bit allocation technology must be combined with scalable high efficiency video coding (SHVC) to achieve continuous bitrate variation. However, to date, the multilayer coding feature in SHVC has not been fully utilized. In this paper, a bit allocation algorithm is proposed for the enhancement layer in SHVC. The algorithm includes two parts: a bit allocation method for the initial frame and another for subsequent frames. The bit allocation method for the initial frame consists of a bit allocation factor model constructed based on investigation of the influence of the initial target bitrate on the overall coding performance. For the bit allocation of the subsequent frames, alternate GOP coding is designed and implemented to fully exploit the inter layer correlation. In addition, an adaptive frame layer bit allocation ratio model is deduced according to the rate distortion optimization theory. Experimental results show that the proposed algorithm can improve PSNR by 0.44 dB and 0.41 dB under low delay and random-access configurations, respectively, and achieve high bitrate control accuracy. The algorithm also outperforms the state-of-the-art algorithms in term of PSNR improvement.     

SIFT-flow-based color correction for multi-view video

During the multi-view video acquisition, color variation across the views tends to be incurred due to different camera positions, orientations, and local lighting conditions. Such color variation will inevitably deteriorate the performance of the follow-up multi-view video processing, such as multi-view video coding (MVC). To address this problem, an effective color correction algorithm, called the SIFT flow-based color correction (SFCC), is proposed in this paper. First, the SIFT-flow technique is used to establish point-to-point correspondences across all the views of the multi-view video. The average color is then computed based on those identified common corresponding points and used as the reference color. By minimizing the energy of the difference yielded between the color of those identified common corresponding points in each view with respect to the reference color, the color correction matrix for each view can be obtained and used to correct its color. Experimental results have shown that the proposed SFCC algorithm is able to effectively eliminate the color variation inherited in multi-view video. By further exploiting the developed SFCC algorithm as a pre-processing for the MVC, extensive simulation results have shown that the coding efficiency of the color-corrected multi-view video can be greatly improved (on average, 0.85 dB, 1.27 dB and 1.63 dB gain for Y, U, and V components, respectively), compared with that of the original multi-view video without color correction.     

White and narrow band image compressor based on a new color space for capsule endoscopy

In this paper, we present the design of a low power and hardware efficient image compressor integrated circuit for wireless capsule endoscopy application. The proposed compression algorithm supports dual-band imaging, that is, works on both white-band imaging (WBI) and narrow-band imaging (NBI). The scheme uses a novel color-space and simple predictive coding for optimized performance. Based on the nature of the narrow- and white-band endoscopic images and video sequences, several sub-sampling schemes are introduced. The proposed dual-band compressor is designed in such as way that it can easily be interfaced with any commercial low power image sensor that outputs RGB image pixels in a raster scan fashion, eliminating the need of large buffer memory and temporary storage. Both NBI and WBI reconstructed images have been verified by medical doctors for acceptability. Compared to other designs targeted to video capsule endoscopy, the proposed algorithm performs strongly with a compression ratio of 80.4% (for WBI) and 79.2% (for NBI), and a high reconstruction peak-signal-to-noise-ratio (over 43.7 dB for both bands). The results of the fabricated chip are also presented.     

Object-based stereo video compression using fractals and shape-adaptive DCT

Based on the classical fractal video compression method, an improved object-based stereo video compression scheme with Shape-Adaptive DCT is proposed in this paper. Firstly, we use more effective macroblock partition scheme instead of classical quadtree partition scheme; thus reducing the block searching strategy. The stereo fractal video coding is proposed which matches the macroblock with two reference frames in left and right view results in increasing compression ratio and reducing bit rate when transmitting compressed stereo data. The stereo codec combines the Motion Compensation Prediction (MCP) and Disparity Compensation Prediction (DCP). Fractal coding is adopted and each object is encoded independently by a prior video segmentation alpha plane, which is defined exactly as in MPEG-4. The testing results with the nature monocular and stereo video sequences provide promising performances at low bit rate coding. We believe it will be a powerful and efficient technique for the object-based monocular and stereo video sequences coding.     

Hardware implementation and validation of the fast variable block size motion estimation architecture for H.264/AVC

Block matching motion estimation is the heart of video coding system. It leads to a high compression ratio, whereas it is time consuming and calculation intensive. Many fast search block matching motion estimation algorithms have been developed in order to minimize search positions and speed up computation but they do not take into account how they can be effectively implemented by hardware. In this paper, we propose an efficient hardware architecture of the fast line diamond parallel search (LDPS) algorithm with variable block size motion estimation (VBSME) for H.264/AVC video coding system. The design is described in VHDL language, synthesized to Altera Stratix III FPGA and to TSMC 0.18 μm standard-cells. The throughput of the hardware architecture reaches a processing rate up to 78 millions of pixels per second at 83.5 MHz frequency clock and uses only 28 kgates when mapped to standard-cells. Finally, a system on a programmable chip (SoPC) implementation and validation of the proposed design as an IP core is presented using the embedded video system.     

New rate-distortion modeling and efficient rate control for H.264/AVC video coding

Rate control (RC) is crucial in controlling compression bit rates and qualities for networked video applications. In this paper, we propose a new rate-distortion (R-D) model and an efficient rate control scheme for H.264/AVC video coding, which elegantly resolve the inter-dependency problem between rate-distortion optimization and rate control by eliminating the need of coding complexity prediction for an inter-frame. The objective is to achieve accurate bit rate, obtain optimal video quality while reducing quality variations and simultaneously handling buffer fullness effectively. The proposed algorithm encapsulates a number of new features, including a coding complexity measure for intra-frames, a rate-distortion model, an accurate quantization parameter (QP) estimation for intra-frames, an incremental quantization parameter calculation method for inter-frames, a proportional+integral+derivative (PID) buffer controller, and an intelligent bit-allocation-balancing technique. Our experimental results demonstrate that the proposed scheme outperforms the JVT-G012 solution by providing accurate rate regulation, effectively reducing frame skipping, and finally improving coding quality by up to 1.80 dB.     

Just noticeable distortion model and its applications in video coding

We explore a new perceptually-adaptive video coding (PVC) scheme for hybrid video compression, in order to achieve better perceptual coding quality and operational efficiency. A new just noticeable distortion (JND) estimator for color video is first devised in the image domain. How to efficiently integrate masking effects together is a key issue of JND modelling. We integrate spatial masking factors with the nonlinear additivity model for masking (NAMM). The JND estimator applies to all color components and accounts for the compound impact of luminance masking, texture masking and temporal masking. Extensive subjective viewing confirms that it is capable of determining a more accurate visibility threshold that is close to the actual JND bound in human eyes. Secondly, the image-domain JND profile is incorporated into hybrid video encoding via the JND-adaptive motion estimation and residue filtering process. The scheme works with any prevalent video coding standards and various motion estimation strategies. To demonstrate the effectiveness of the proposed scheme, it has been implemented in the MPEG-2 TM5 coder and demonstrated to achieve average improvement of over 18% in motion estimation efficiency, 0.6 dB in average peak signal-to perceptual-noise ratio (PSPNR) and most remarkably, 0.17 dB in the objective coding quality measure (PSNR) on average. Theoretical explanation is presented for the improvement on the objective coding quality measure. With the JND-based motion estimation and residue filtering process, hybrid video encoding can be more efficient and the use of bits is optimized for visual quality.     

Motion Compensated Three-Dimensional Frequency Selective Extrapolation for improved error concealment in video communication

During transmission of video data over error-prone channels the risk of getting severe image distortions due to transmission errors is ubiquitous. To deal with image distortions at decoder side, error concealment is applied. This article presents Motion Compensated Three-Dimensional Frequency Selective Extrapolation, a novel spatio-temporal error concealment algorithm. The algorithm uses fractional-pel motion estimation and compensation as initial step, being followed by the generation of a model of the distorted signal. The model generation is conducted by an enhanced version of Three-Dimensional Frequency Selective Extrapolation, an existing error concealment algorithm. Compared to this existent algorithm, the proposed one yields an improvement in concealment quality of up to 1.64 dB PSNR. Altogether, the incorporation of motion compensation and the improved model generation extends the already high extrapolation quality of the underlying Frequency Selective Extrapolation, resulting in a gain of more than 3 dB compared to other well-known error concealment algorithms.     

Overview of the MVC + D 3D video coding standard

3D video services are emerging in various application domains including cinema, TV broadcasting, Blu-ray discs, streaming and smartphones. A majority of the 3D video content in market is still based on stereo video, which is typically coded with the multiview video coding (MVC) extension of the Advanced Video Coding (H.264/AVC) standard or as frame-compatible stereoscopic video. However, the 3D video technologies face challenges as well as opportunities to support more demanding application scenarios, such as immersive 3D telepresence with numerous views and 3D perception adaptation for heterogeneous 3D devices and/or user preferences. The Multiview Video plus Depth (MVD) format enables depth-image-based rendering (DIBR) of additional viewpoints in the decoding side and hence helps in such advanced application scenarios. This paper reviews the MVC + D standard, which specifies an MVC-compatible MVD coding format.     

A wavelet implementation of the pioneering block-based disparity compensated predictive coding algorithm for stereo image pair compression

In this paper, we propose a novel, discrete wavelet transform (DWT) domain implementation of our previously proposed, pioneering block-based disparity compensated predictive coding algorithm for stereo image compression. Under the present research context we perform predictive coding in the form of pioneering block search in the sub-band domain. The resulting transform domain predictive error image is subsequently converted to a so-called wavelet-block representation, before being quantized and entropy coded by a JPEG-like CODEC. We show that the proposed novel implementation is able to effectively transfer the inherent advantages of DWT-based image coding technology to efficient stereo image pair compression. At equivalent bit rates, the proposed algorithm achieves peak signal to noise ratio gains of up to 5.5 dB, for reconstructed predicted images, as compared to traditional and state of the art DCT and DWT-based predictive coding algorithms.     

一种面向HEVC的并行码率控制算法

视频通信时通常带宽有限,为了能在规定的目标码率下获得尽可能高质量的解码图像,需要在视频编码时进行码率控制.目前针对HEVC的并行编码以及对应码率控制已成为研究热点,现有并行结构下的平均比特率控制算法受到帧间依赖性的约束,待编码帧无法及时获得与其并行编码帧的实际比特数,因此本文算法通过预测并行帧的实际比特数来进行码率控制,并在此基础提出了自适应调整帧层量化参数补偿值.仿真结果表明,相比已有算法,前者减少码率误差约为3.38％,后者可提高PSNR约为0.204 dB同时减少约0.3％的码率误差.     

Improved macroblock level rate control for the spatially scalable extension of H.264/AVC

An improved rate control algorithm, designed for scalable video coders incorporating interlayer prediction, is proposed. Firstly, a Rate Distortion (RD) model for interlayer prediction involving the spatial enhancement layers is devised. An optimised Mean Absolute Difference (MAD) prediction model for the spatial enhancement layers that considers both the MAD from the spatial base layer in the same frame and the MAD from the corresponding macroblock in previous frames is also proposed. Simulation results show that the resulting algorithm produces accurate rate control with an average bit rate error of less than 0.26%. Compared with the JVT-W043 default rate control algorithm of the JSVM, the proposed algorithm improves the average PSNR by up to 0.53 dB or reduces the bit rate by an average of 10.95%. Furthermore, the proposed algorithm can be combined with the existing rate control scheme for H.264/AVC, resulting in further improvements.