Complexity-based intra frame rate control by jointing inter-frame correlation for high efficiency video coding

Rate control is of great significance for the High Efficiency Video Coding (HEVC). Due to the high efficiency and low complexity, the R-lambda model has been applied to the HEVC as the default rate control algorithm. However, the video content complexity, which can help improve the code efficiency and rate control performance, is not fully considered in the R-lambda model. To address this problem, an intra-frame rate control algorithm, which aims to provide improved and smooth video quality, is developed in this paper by jointly taking into consideration the frame-level content complexity between the encoded intra frames and the encoded inter frame, as well as the CTU-level complexity among different CTUs in texture–different regions for intra-frame. Firstly, in order to improve the rate control efficiency, this paper introduces a new prediction measure of content complexity for CTUs of intra-frame by jointly considering the inter-frame correlations between encoding intra frame and previous encoded inter frames as well as correlations between encoding intra frame and previous encoded intra frame. Secondly, a frame-level complexity-based bit-allocation-balancing method, by jointly considering the inter-frame correlation between intra frame and previous encoded inter frame, is brought up so that the smoothness of the visual quality can be improved between adjacent inter- and intra-frames. Thirdly, a new region-division and complexity-based CTU-level bit allocation method is developed to improve the objective quality and to reduce PSNR fluctuation among CTUs in intra-frame. In the end, related model parameters are updated during the encoding process to increase rate control accuracy. As a result, as can be seen from the extensive experimental results that compared with the state-of-the-art schemes, the video quality can be significantly improved. More specifically, up to 10.5% and on average 5.2% BD-Rate reduction was achieved compared to HM16.0 and up to 2.7% and an average of 2.0% BD-Rate reduction was achieved compared to state-of-the-art algorithm. Besides, a superior performance in enhancing the smoothness of quality can be achieved, which outperforms the state-of-the-art algorithms in term of flicker measurement, frame and CTU-wise PSNR, as well as buffer fullness.     

Wyner–Ziv-based bidirectionally decodable video coding

In this paper, we propose a novel Wyner–Ziv-based video compression scheme which supports encoding a new type of inter frame called ‘M-frame’. Different from traditional multi-hypothesis inter frames, the M-frame is specially compressed with its two neighbor frames as reference at the encoder, but can be identically reconstructed by using any one of them as prediction at the decoder. Based on this, the proposed Wyner–Ziv-based bidirectionally decodable video compression scheme supports decoding the frames in a video stream in both temporal order and reverse order. Unlike the other schemes which support reverse playback, our scheme achieves the reversibility with low extra cost of storage and bandwidth. In error-resilient test, our scheme outperforms H.264 based schemes up to 3.5 dB at same bit rate. The proposed scheme also provides more flexibility for stream switching.     

Improved macroblock level rate control for the spatially scalable extension of H.264/AVC

An improved rate control algorithm, designed for scalable video coders incorporating interlayer prediction, is proposed. Firstly, a Rate Distortion (RD) model for interlayer prediction involving the spatial enhancement layers is devised. An optimised Mean Absolute Difference (MAD) prediction model for the spatial enhancement layers that considers both the MAD from the spatial base layer in the same frame and the MAD from the corresponding macroblock in previous frames is also proposed. Simulation results show that the resulting algorithm produces accurate rate control with an average bit rate error of less than 0.26%. Compared with the JVT-W043 default rate control algorithm of the JSVM, the proposed algorithm improves the average PSNR by up to 0.53 dB or reduces the bit rate by an average of 10.95%. Furthermore, the proposed algorithm can be combined with the existing rate control scheme for H.264/AVC, resulting in further improvements.     

一种无帧内失真漂移的HEVC视频自适应隐写算法

随着数字视频的普及,HEVC(high efficiency video coding)视频隐写受到了越来越多的关注。残差系数域隐写具有高安全性和低码率增长的特点,然而,视频编码过程中重构的系数不满足块间相互独立的性质,导致修改视频残差系数往往会带来较为严重的失真漂移。本文提出了一种无帧内失真漂移的HEVC视频自适应隐写算法。首先,根据帧内预测方向和多系数修改的方法选择合适的载体,进而避免修改系数引起的失真漂移。然后,设计一种包含块内失真和修改载体后的系数分布的失真函数,指导校验网格编码(syndrome trellis codes, STC)对嵌入失真小的载体进行修改。最后,根据最小化失真代价将消息嵌入到满足特定条件的4×4亮度块的量化正弦变换(quantized discrete sinc transform, QDST)系数中。实验结果表明,所提出的视频隐写算法能够有效地避免帧内失真漂移,在获得良好的视觉感知质量的同时,能保证隐写算法的安全性。     

FITD: Fast Intra Transcoding from H.264/AVC to high efficiency video coding based on DCT coefficients and prediction modes

In the literatures, the designs of H.264 to High Efficiency Video Coding (HEVC) transcoders mostly focus on inter transcoding. In this paper, a fast intra transcoding system from H.264 to HEVC based on discrete cosine transform (DCT) coefficients and intra prediction modes, called FITD, is proposed by using the intra information retrieved from an H.264 decoder for transcoding. To design effective transcoding strategies, FITD not only refers block size of intra prediction and intra prediction modes, but also effectively uses the DCT coefficients to help a transcoder to predict the complexity of the blocks. We successfully use DCT coefficients as well as intra prediction information embedded in H.264 bitstreams to predict the coding depth map for depth limitation and early termination to simplify HEVC re-encoding process. After a HEVC encoder gets the prediction of a certain CU size from depth map, if it reaches the predicted depth, the HEVC encoder will stop the next CU branch. As a result, the numbers of CU branches and predictions in HEVC re-encoder will be substantially reduced to achieve fast and precise intra transcoding. The experimental results show that the FITD is 1.7–2.5 times faster than the original HEVC in encoding intra frames, while the bitrate is only increased to 3% or less and the PSNR degradation is also controlled within 0.1 dB. Compared to the previous H.264 to HEVC transcoding approaches, FITD clearly maintains the better trade-off between re-encoding speed and video quality.     

Prediction of transmission distortion for wireless video communication: Algorithm and application

In this paper, we develop algorithms for estimating transmission distortion in wireless video communication systems. By leveraging the analytical results obtained in our previous works, we design low complexity algorithms that are capable of estimating transmission distortion accurately. We also extend our algorithm for pixel-level transmission distortion estimation to pixel-level end-to-end distortion estimation. Furthermore, we apply our pixel-level end-to-end distortion estimation algorithm to prediction mode decision in H.264 encoder. Experimental results show that (1) our transmission distortion estimation algorithm is more accurate and more robust against inaccurate channel estimation than existing distortion estimation algorithms; (2) our mode decision algorithm achieves remarkable PSNR gain over the existing algorithms for prediction mode decision in H.264 encoder, e.g., an average PSNR gain of 1.44 dB for ‘foreman’ sequence when Packet Error Probability (PEP) equals 5%.     

Enhanced pipelined architecture of H.264/AVC intra prediction

This paper presents a high-performance encoder for H.264/AVC intra prediction. Due to long data dependency loop of intra 4×4 prediction and complex algorithms, improving encoding speed turns into a stumbling block we have to face. To solve this problem, we first propose a pipelined method in and between macro blocks with new block processing order to accelerate the encoding speed. Benefiting from the pipelined method, reconstructed pixels of up-right blocks are available for two blocks in a macro block which could not take advantage of reconstructed pixels of up-right blocks in JM. So diagonal down left mode and vertical left mode are effective for these two blocks, which ultimately achieves a better bit-rate. Secondly, all 4×4 mode formula sharing method is proposed to reduce the redundancy of predicting formulas. Thirdly, streamlined reconstruction method is applied to improve the performance of reconstruction. CAVLC encoder with three parallel units is proposed to improve entropy coding speed significantly. As a result, it takes 268 cycles to encode a macro block. The experimental results indicate that synthesized into a 0.18 µm CMOS cell library, the new architecture only requires about 238K gates and it is able to encode 1080pHD video sequences at 30 frames per second (fps), at the operating frequency of 56 MHz.     

Hardware implementation of a spatio-temporal average filter for real-time denoising of fluoroscopic images

An electronic system for the real-time denoising of fluoroscopic images is proposed in this paper. Fluoroscopic devices use X-rays to obtain real-time moving images of patients and support many surgical interventions and a variety of diagnostic procedures. In order to avoid risks for the patient, X-ray intensity has to be kept acceptably low during the clinical applications. This implies that fluoroscopic images are corrupted by large quantum noise (Poisson-distributed). Real-time noise reduction can offer a better visual perception to doctors and possible further reductions of the dose.The proposed circuit implements a spatio-temporal filter optimized for the removal of the quantum noise while preserving video edges and the prompt response of the image to the introduction of new features in the field. The filter incorporates information on the dependence of the standard deviation of the noise on the local brightness of the image and performs a conditioned average operation.The proposed circuit is implemented on FPGA (Field Programmable Gate Array) device allowing the real time elaboration of video streams composed by frames with 1024×1024 pixel and uses an external DDR2 (Double Data Rate 2) memory for the storage and the reuse of the fluoroscopic frames needed by the filter.When implemented on StratixIV-GX70 FPGA the circuit is able to process up to 49 fps (frames per second) while using 80% of the logic resources of the FPGA.     

Distortion free image-in-image communication with implementation in FPGA

The proliferation of the digitized media (audio, image and video) introduces a challenging problem for data transmission in the network environment. In this paper, a novel, simple and low cost algorithm that serves the purpose of distortion free covert image-in-image communication is proposed. Its very large scale integration (VLSI) implementation using field programmable gate array (FPGA) is also developed. A binary equivalent message signal is developed first from the combination of the auxiliary gray scale image information and the carrier gray scale image (original) using channel coding and spatial bi-phase modulation scheme. The auxiliary image information is then decoded from the distorted/distortion free version of the original image using binary message under certain noise constraint. Implementation of the proposed low cost algorithm can be speeded up significantly by hardware realization. The developed hardware design allows data transmission at the rate of 4.706 Mbits/s at 80 MHz clock frequency.     

Novel intra prediction via position-dependent filtering

This paper presents a novel intra prediction algorithm, named position-dependent filtering (PDF), to improve the intra prediction accuracy. Different from the existing schemes where the samples along one prediction direction are predicted with the same set of filtering coefficients, in the proposed PDF, position-dependent filtering coefficients are employed, i.e., different sets of filtering coefficients are pre-defined for samples with different coordinates in one coding block. For each intra prediction mode, the set of linear filtering coefficients for each position within one block is obtained from off-line training using the least square method. Moreover, to further reduce the algorithm complexity, a simplified PDF (sPDF) is proposed. In sPDF, only a subset of reference samples are used for prediction and the others are discarded because of the minor contribution to intra prediction. The proposed algorithm has been implemented in the latest ITU-T VCEG KTA software. Experimental results demonstrate that, compared with the original KTA with new intra coding tool enabled, up to 0.53 dB of average coding gain is achieved by the proposed method, while applicable computational complexity is retained for practical video codecs.     

Dictionary learning based reconstruction for distributed compressed video sensing

Distributed compressed video sensing (DCVS) is a framework that integrates both compressed sensing and distributed video coding characteristics to achieve a low-complexity video coding. However, how to design an efficient reconstruction by leveraging more realistic signal models that go beyond simple sparsity is still an open challenge. In this paper, we propose a novel “undersampled” correlation noise model to describe compressively sampled video signals, and present a maximum-likelihood dictionary learning based reconstruction algorithm for DCVS, in which both the correlation and sparsity constraints are included in a new probabilistic model. Moreover, the signal recovery in our algorithm is performed during the process of dictionary learning, instead of being employed as an independent task. Experimental results show that our proposal compares favorably with other existing methods, with 0.1–3.5 dB improvements in the average PSNR, and a 2–9 dB gain for non-key frames when key frames are subsampled at an increased rate.     

An automatic region-based video sequence codec based on fractal compression

In order to improve the performance of fractal video coding, we explore a novel fractal video sequences codec with automatic region-based functionality. To increase the quality of decoding image, intra frame coding, deblocking loop filter and sub-pixel block matching are applied to the codec. An efficient searching algorithm is used to increase the compression ratio and encoding speed. Automatic region-based fractal video sequences coding reduces coding stream greatly. Experimental results indicate that the proposed algorithm is more robust, and provides much less encoding time and bitrate while maintaining the quality of decompression image than the conventional CPM/NCIM method and other related references. We compare the proposed algorithm with three algorithms in Refs. [24], [25], [26], and the results of all these four algorithms are compared with H.264. The bitrate of the proposed algorithm is decreased by 0.11% and the other algorithms are increased by 4.29%, 6.85% and 11.62%, respectively. The average PSNR degradations of the four algorithms are 0.71 dB, 0.48 dB, 0.48 dB and 0.75 dB. So the bitrate of the proposed algorithm is decreased and the other algorithms are increased. At the meantime the compression time is reduced greatly, about 79.19% on average. The results indicate that, on average, the proposed automatic region-based fractal video sequences coding system can save compression time 48.97% and bitrate 52.02% with some image quality degradation in comparison with H.264, since they are all above 32 dB and the human eyes are insensitive to the differences.     

Hardware implementation and validation of the fast variable block size motion estimation architecture for H.264/AVC

Block matching motion estimation is the heart of video coding system. It leads to a high compression ratio, whereas it is time consuming and calculation intensive. Many fast search block matching motion estimation algorithms have been developed in order to minimize search positions and speed up computation but they do not take into account how they can be effectively implemented by hardware. In this paper, we propose an efficient hardware architecture of the fast line diamond parallel search (LDPS) algorithm with variable block size motion estimation (VBSME) for H.264/AVC video coding system. The design is described in VHDL language, synthesized to Altera Stratix III FPGA and to TSMC 0.18 μm standard-cells. The throughput of the hardware architecture reaches a processing rate up to 78 millions of pixels per second at 83.5 MHz frequency clock and uses only 28 kgates when mapped to standard-cells. Finally, a system on a programmable chip (SoPC) implementation and validation of the proposed design as an IP core is presented using the embedded video system.     

Rate-distortion optimized unequal loss protection for video transmission over packet erasure channels

Video transmission over networks often suffers from packet loss due to network congestions and stringent end-to-end delay constraints. In this paper, we develop a Rate-Distortion optimized Unequal Loss Protection (RD-ULP) scheme to combat packet loss. Based on packet-level transmission distortion modeling, we estimate the amount of contribution of each video packet to the reconstructed video quality, which defines the priority level of each packet. Unequal amounts of protection are then allocated to different video packets according to their priority levels and the dynamic channel conditions. The proposed RD-ULP resource allocation problem is formulated as a constrained nonlinear optimization problem. An optimization algorithm based on Particle Swarm Optimization (PSO) is then developed to solve the optimal resource allocation problem. Our extensive experimental results demonstrate the effectiveness of the proposed RD-ULP scheme, which outperforms existing methods by up to 2 dB in the reconstructed video quality.     

A Fast Intra Skip Detection Algorithm for H.264/AVC Video Encoding

A fast intra skip detection algorithm based on the rate‐distortion (RD) cost for an inter frame (P‐slices) is proposed for H.264/AVC video encoding. In the H.264/AVC coding standard, a robust rate‐distortion optimization technique is used to select the best coding mode and reference frame for each macroblock (MB). There are three types of intra predictions according to profiles. These are 16×16 and 4×4 intra predictions for luminance and an 8×8 intra prediction for chroma. For the high profile, an 8×8 intra prediction has been added for luminance. The 4×4 prediction mode has 9 prediction directions with 4 directions for 16×16 and 8×8 luma, and 8×8 chrominance. In addition to the inter mode search procedure, an intra mode search causes a significant increase in the complexity and computational load for an inter frame. To reduce the computational load of the intra mode search at the inter frame, the RD costs of the neighborhood MBs for the current MB are used and we propose an adaptive thresholding scheme for the intra skip extraction. We verified the performance of the proposed scheme through comparative analysis of experimental results using joint model reference software. The overall encoding time was reduced up to 32% for the IPPP sequence type and 35% for the IBBPBBP sequence type.     

Real-time and rate–distortion optimized video streaming with TCP

In this paper we explore the use of a new rate–distortion metric for optimizing real-time Internet video streaming with the transmission control protocol (TCP). We lay out the groundwork by developing a simple model that characterizes the expected latency for packets send with TCP-Reno. Subsequently, we develop an analytical model of the expected video distortion at the decoder with respect to the expected latency for TCP, the packetization mechanism, and the error-concealment method used at the decoder. Characterizing the duo protocol/channel more accurately, we obtain a better estimate of the expected distortion and the available channel rate. This better knowledge is exploited with the design of a new algorithm for rate–distortion optimized encoding mode selection for video streaming with TCP. Experimental results for real-time video streaming depict improvement in PSNR in the range of 2 dB over metrics that do not consider the behavior of the transport protocol.     

End-to-end optimized TCP-friendly rate control for real-time video streaming over wireless multi-hop networks

Rate control is an important issue in video streaming applications. The most popular rate control scheme over wired networks is TCP-Friendly Rate Control (TFRC), which is designed to provide optimal transport service for unicast multimedia delivery based on the TCP Reno’s throughput equation. It assumes perfect link quality, treating network congestion as the only reason for packet losses. Therefore, when used in wireless environment, it suffers significant performance degradation because of packet losses arising from time-varying link quality. Most current research focuses on enhancing the TFRC protocol itself, ignoring the tightly coupled relation between the transport layer and other network layers. In this paper, we propose a new approach to address this problem, integrating TFRC with the application layer and the physical layer to form a holistic design for real-time video streaming over wireless multi-hop networks. The proposed approach can achieve the best user-perceived video quality by jointly optimizing system parameters residing in different network layers, including real-time video coding parameters at the application layer, packet sending rate at the transport layer, and modulation and coding scheme at the physical layer. The problem is formulated and solved as to find the optimal combination of parameters to minimize the end-to-end expected video distortion constrained by a given video playback delay, or to minimize the video playback delay constrained by a given end-to-end video distortion. Experimental results have validated 2–4 dB PSNR performance gain of the proposed approach in wireless multi-hop networks by using H.264/AVC and NS-2.     

Packet prioritization for H.264/AVC video with cyclic intra-refresh line

Insertion of a cyclic intra-refresh line is a lightweight way of mitigating spatio-temporal error propagation in a video stream transmitted over a mobile network. This paper presents low-complexity yet effective prioritization based on slice position within a video frame relative to the cyclic refresh line. Two prioritization schemes are compared. The first is a region-based method, while the second, which is packet-based, improves packet classification. Experimental results indicate that, the packet-based scheme can achieve video quality gains of up to 4 dB, compared to when the scheme is not used. The proposed schemes require no decoder modifications and do not introduce an increase in bitrate or in computational complexity.     

Video compressed sensing reconstruction based on structural group sparsity and successive approximation estimation model

The existing video compressed sensing (CS) algorithms for inconsistent sampling ignore the joint correlations of video signals in space and time, and their reconstruction quality and speed need further improvement. To balance reconstruction quality with computational complexity, we introduce a structural group sparsity model for use in the initial reconstruction phase and propose a weight-based group sparse optimization algorithm acting in joint domains. Then, a coarse-to-fine optical flow estimation model with successive approximation is introduced for use in the interframe prediction stage to recover non-key frames through alternating optical flow estimation and residual sparse reconstruction. Experimental results show that, compared with the existing algorithms, the proposed algorithm achieves a peak signal-to-noise ratio gain of 1–3 dB and a multi-scale structural similarity gain of 0.01–0.03 at a low time complexity, and the reconstructed frames not only have good edge contours but also retain textural details.