Early termination strategy of HEVC based on statistical models

High efficiency video coding (HEVC) uses half of the bitrate compared to H.264/advanced video coding(AVC) for encoding the same sequence with similar quality. Because of the advanced hierarchical structures of coding units (CUs), predicting units (PUs), and transform units (TUs), HEVC can better adapt when encoding full high definition (HD) and ultra high definition (UHD) videos. At the expense of encoding efficiency, the complexity of HEVC sharply increases compared to H.264/AVC, mainly due to its quad-tree structure that splits pictures. In this study, the probability distribution, which is generated by a rate distortion optimizing (RDO) cost, is analyzed. Then, an early terminating method is proposed to decrease the complexity of the HEVC based on probability distributions. The experiment shows that the coding time is reduced by 44.9% for HEVC intra coding, at the cost of a 0.61% increase in the Bjøntegaard delta rate (BD-rate), on average.     

结合卷积神经网络的HEVC帧内编码压缩改进算法

近年来,卷积网络深度学习已在图像处理、目标检测等领域取得巨大成功。受其启发,将卷积神经网络(CNN)应用于传统视频压缩标准已成为一个新的研究热点。本文提出一种集成卷积神经网络的高效视频编码(HEVC)压缩改进算法,将下采样过程、HEVC的编解码过程、上采样及质量增强过程集成为一体。为高效提取视频帧的结构特征,在所提压缩算法中集成了两个卷积神经网络。提出了一种下采CNN(DwSCNN)代替双三次下采,在有效降低分辨率的同时保留细节信息,得到更为紧凑的低分辨率视频序列,将此低分辨率视频序列通过HEVC帧内编码进行进一步的数据量压缩,通过提出一个质量增强CNN(PPCNN)来改善解码后恢复到原始分辨率的降质视频序列。实验结果显示,本文压缩改进算法在低码率段与标准HEVC相比,能达到更好的质量重建,并且在接近一致的PSNR值时,能节省39.46%的时间和11.04%的比特率,本文算法的视频压缩性能优于HEVC标准算法和相关文献方法。     

A DCT/DST-based error propagation-free data hiding algorithm for HEVC intra-coded frames

Currently, two error propagation-free discrete cosine transform (DCT)-based data hiding algorithms, one by Ma et al. and the other by Lin et al., were presented for H.264/AVC intra-coded frames. However, the state-of-the-art video codec, high efficiency video coding (HEVC), adopts both integer DCT and discrete sine transform (DST) such that the previous DCT-based algorithms cannot fully utilize available capacity for data hiding in HEVC. This paper presents the first DCT/DST-based data hiding algorithm for HEVC intra-coded frames where the block DCT and DST coefficient characteristics are investigated to locate the transformed coefficients that can be perturbed without propagating errors to neighboring blocks. Experimental results confirm the merits of the proposed algorithm in providing the intra-frame error propagation-free advantage, the quality improvement for marked images, the compression power inherited from HEVC, and the superiority of embedding capacity for low bitrate coding when compared with the previous two algorithms for H.264/AVC.     

SSIM-based Error Resilient Video Coding Over Packet-Switched Networks

The visual quality is a critical factor in prediction video coding over packet-switched networks. However, the traditional MSE-based error resilient video coding cannot correlate well with the perceptual characteristics of the human visual system (HVS). This paper proposes a structural similarity (SSIM) based error resilient video coding scheme to improve the visual quality of compressed videos over packet-switched networks. In the proposed scheme, a SSIM-based end-to-end distortion model is developed to estimate the perceptual distortion due to quantization, error concealment, and error propagation. Based on this model, an adaptive mode selection strategy is presented to enhance the communication robustness of compressed videos. Experiments show that the proposed scheme significantly improves the visual quality for H.264/AVC video coding over packet-switched networks.     

FITD: Fast Intra Transcoding from H.264/AVC to high efficiency video coding based on DCT coefficients and prediction modes

In the literatures, the designs of H.264 to High Efficiency Video Coding (HEVC) transcoders mostly focus on inter transcoding. In this paper, a fast intra transcoding system from H.264 to HEVC based on discrete cosine transform (DCT) coefficients and intra prediction modes, called FITD, is proposed by using the intra information retrieved from an H.264 decoder for transcoding. To design effective transcoding strategies, FITD not only refers block size of intra prediction and intra prediction modes, but also effectively uses the DCT coefficients to help a transcoder to predict the complexity of the blocks. We successfully use DCT coefficients as well as intra prediction information embedded in H.264 bitstreams to predict the coding depth map for depth limitation and early termination to simplify HEVC re-encoding process. After a HEVC encoder gets the prediction of a certain CU size from depth map, if it reaches the predicted depth, the HEVC encoder will stop the next CU branch. As a result, the numbers of CU branches and predictions in HEVC re-encoder will be substantially reduced to achieve fast and precise intra transcoding. The experimental results show that the FITD is 1.7–2.5 times faster than the original HEVC in encoding intra frames, while the bitrate is only increased to 3% or less and the PSNR degradation is also controlled within 0.1 dB. Compared to the previous H.264 to HEVC transcoding approaches, FITD clearly maintains the better trade-off between re-encoding speed and video quality.     

ViSTRA2: Video coding using spatial resolution and effective bit depth adaptation

We present a new video compression framework (ViSTRA2) which exploits adaptation of spatial resolution and effective bit depth, down-sampling these parameters at the encoder based on perceptual criteria, and up-sampling at the decoder using a deep convolution neural network. ViSTRA2 has been integrated with the reference software of both the HEVC (HM 16.20) and VVC (VTM 4.0.1), and evaluated under the Joint Video Exploration Team Common Test Conditions using the Random Access configuration. Our results show consistent and significant compression gains against HM and VVC based on Bjønegaard Delta measurements, with average BD-rate savings of 12.6% (PSNR) and 19.5% (VMAF) over HM and 5.5% (PSNR) and 8.6% (VMAF) over VTM.     

Fast algorithms for designing nearly optimal lookup tables for complexity control of the H.264 encoder

The H.264/AVC standard significantly improves video compression performance over earlier standards at the cost of increased complexity. In this paper, we present two offline algorithms for generating a lookup table of parameter settings that can be used by a complexity control algorithm for controlling the speed of the H.264 encoder. Our algorithms to design the lookup table of parameter settings take fewer than 2% of the number of encodings required by an exhaustive search of all possible parameter settings and find parameter settings that offer high peak signal-to-noise ratio (PSNR) with low encoding time at a given bitrate. Our parameter settings are fairly robust over different videos and bitrates. We focus on low-resolution videos at bitrates less than 300 kb/s. We compare the performance of our algorithms to both exhaustive search and a multiobjective optimization algorithm. Our parameter settings improve the average encoding speed over the default parameter setting of the x264 encoder on both PC and cell phone platforms by up to 37.4 and 94.1%, respectively, with PSNR difference of up to 0.3 dB.     

A new distortion measure for motion estimation in motion-compensated hybrid video coding

In the video coding standards MPEG-x and H.26x, a motion-compensated prediction technique is used for enhancing the coding performance of bitrate reduction or peak signal to noise ratio (PSNR) improvement. This technique takes advantage of the correlation between consecutive frames in the time domain, which is relatively higher than that between adjacent blocks in the spatial domain. In order to utilize the correlation between consecutive frames, the conventional video coding standards have used the motion estimation (ME) and compensation technique, where the Sum of the Absolute Differences (SAD) is usually used as the distortion measure. The ME estimates the reference block that could minimize the residual signal between the current and reference blocks. However, the SAD is not appropriate to the specific sequences that have global or local illumination changes. In addition, the high-resolution video sequences have higher spatial correlation than the low-resolution video sequences in general. Therefore, a new distortion measure that can consider spatial and temporal correlation simultaneously may be helpful to enhance the coding performance. The proposed distortion measure searches for a reference block that minimizes the motion-compensated residual signal when the DC-component is predicted. In our proposed algorithm, the maximum BD-rate improvement is up to 13.6% for illumination-changed video sequences, and the average BD-rate improvement is 6.6% for various high-resolution video sequences in the baseline profile.     

采用双网络结构的压缩视频超分辨率重建

在实际应用中,为了节省带宽和方便存储,图像和视频通常被下采样和压缩,而降质的图像与视频无法满足人们的实际需求。针对这一问题,采用了一种双网络结构的超分辨率重建方法,首先建立下采视频与压缩后的低分辨率视频的映射关系,然后建立质量增强的压缩视频与原始视频的映射关系,最终在输出端可以得到质量提升的视频帧。在网络中,采用密集残差块来提取压缩视频中丰富的局部分层特征,并结合全局残差学习恢复视频中的高频信息。在压缩环节,采用高性能视频编码来验证所提算法的有效性。实验结果表明,相比于主流的视频编码标准和先进的超分辨率重建算法,所提方法能有效提升编码视频的率失真性能。     

基于神经网络的量化参数预测研究

在视频有损压缩编码中,量化是为达到压缩视频的目的,将原信号值映射到较小取值范围的过程。一个合适的量化参数(QP)可以在增加压缩效率的同时平衡视频质量和编码复杂度。现有的码率控制算法对运动剧烈及存在场景切换的连续帧控制不精确,视频PSNR值波动大。本文提出一种基于神经网络的视频量化参数选择,利用已编码帧信息对当前帧的量化参数计算进行调整。神经网络模型选取当前帧的目标比特、当前帧的MAD和前一帧编码的QP和实际比特数作为输入,输出值经过反归一化及取整处理对视频进行量化。实验表明,编码时使用该方法进行与H.264/AVC的标准参考软件JM14.2相比,在保证码率精确度和平均PSNR值基本不变的情况下,减小了PSNR的波动。     

A GOP-level fuzzy rate control algorithm for high-delay applications of HEVC

The high -efficiency video coding (HEVC) standard was introduced for high-resolution video contents suitable for many high-delay applications. The bit rate of compressed video is controlled in almost all digital video applications according to the practical constraints such as available channel bandwidth and allowed delay or buffering constraint. In high-delay video applications such as video broadcasting and video streaming applications, variable-bit-rate videos can provide a higher visual quality than constant-bit-rate videos. In this paper, a rate control algorithm (RCA) for high-delay applications of the HEVC standard with buffering constraint is proposed. A fuzzy controller and a virtual buffer are used in the proposed RCA. The fuzzy controller is designed to minimize the fluctuations of quantization parameter (QP) while the buffering constraint is obeyed. It computes a base QP for each group of pictures (GOP) to prevent unnecessary fluctuations of QP at GOP level and thereafter provide a higher visual quality. Experimental results show that not only the bit rate and but also buffering constraints are fully maintained but also the objective quality of compressed video is well preserved. Moreover, the proposed RCA provides smooth QP and peak signal-to-noise ratio close to constant QP case for encoded videos that means high subjective quality.     

基于多尺度特征残差学习卷积神经网络的视频超分辨率方法

本文提出了一种基于多尺度特征残差学习卷积神经网络的视频超分辨率方法,考虑到视频帧间的时空相关性,所提的方法采用由双三次插值预处理后的连续五帧视频作为卷积神经网络的输入,经由网络重建中间帧作为输出,依次按顺序重建直至获得整个高分辨率视频。本文所提出的卷积神经网络主要由多尺度特征提取、残差学习、亚像素卷积层、残差连接(skip-connection)四大部分组成,通过对视频的多尺度特征的提取获得更丰富的不同尺度特征和残差学习达到较好地恢复高频信息的目的。本文采用峰值信噪比（PSNR）和结构相似性指数（SSIM）作为损失函数优化网络。实验结果表明,本方法在平均评价指标上较其他方法均有一定的提升（PSNR +3.151dB,SSIM +0.102）,从主观评价上看可以有效地减少视频边缘模糊的现象。      

Smart selective encryption of H.264/AVC videos using confidentiality metrics

In the field of video protection, selective encryption (SE) is a scheme which ensures the visual security of a video by encrypting only a small part of the data. This paper presents a new SE algorithm for H.264/AVC videos in context-adaptive variable-length coding mode. This algorithm controls the amount of encrypted alternative coefficients (ACs) of the integer transform in the entropic encoder. Two visual quality measures, the peak signal-to-noise ratio (PSNR) and the structural similarity (SSIM), are used to measure the visual confidentiality level of each video frame and to control the amount of encrypted ACs. Moreover, a new psychovisual metric to measure the flickering is introduced, the so-called temporal structural similarity (TSSIM). This method can be applied on intra and inter frame video sequences. Several experimental results show the efficiency of the proposed method.     

A Novel SAO-based Filtering Technique for Reduction in Temporal Flickering Artifacts in H.265/HEVC

This paper presents a novel filtering technique based on sample adaptive offset (SAO) in H.265/high-efficiency video coding (HEVC) for reduction in the temporal flickering artifacts and improving the coding performance. SAO is a newly introduced technique for in-loop filtering in H.265/HEVC, which derives the offsets independently for each frame in the spatial domain without considering temporal frame correlation. As a result, the temporal distortion artifacts which will have a negative effect on the subjective quality, such as flickering artifacts, cannot be effectively addressed. In this paper, the rate-distortion optimization of the newly developed SAO method, referred to as Inter-SAO, is performed on the residual samples between adjacent frames. Inter-SAO and SAO in the reference software of H.265/HEVC (i.e., the test model HM) are then combined to form the novel in-loop filter-based method, denoted as 3D-SAO filtering method, where both spatial information and temporal information are effectively utilized to reduce the overall distortion in reconstructed videos. Compared with the SAO in HM, 3D-SAO has demonstrated its advanced performance for flickering artifacts suppression. Furthermore, 3D-SAO improves the coding efficiency compared with the SAO in HM with a performance gain of up to 0.91 dB in \(\Delta PSNR\), 1.74 dB in \(\Delta PSPNR\) and 7.33 % in BD-rate reduction.     

Content analysis based smart macroblock rearrangement for error resilience in wireless video transmission

Transmission of compressed video is very sensitive to channel error. The use of resynchronization marks is an efficient method in improving the error resilient performance of a video stream in error-prone environment. In this paper, a smart macroblock rearrangement (SMR) method is proposed to enhance the performance of the resynchronization mark insertion technique for intracoded frames in wireless video transmission. The proposed method makes use of content analysis to rearrange and encode macroblocks between adjacent resynchronization marks. Experiments using the SMR method in both H.263+ and H.264 codecs show that the proposed method outperforms some existing algorithms in both PSNR performance and visual quality.     

不同量化比特数对超高清图像质量损伤的影响研究

通过挑选比较严格的超高清视频测试序列,采用x264压缩编码软件,计算量化比特数为10和8的各序列在不同码率下的PSNR值,以此来研究不同量化比特数对超高清图像质量损伤的影响。由实验数据得出,同一序列在同一码率下,量化比特数为10相比量化比特数为8的序列的PSNR高0.7 dB左右,也即量化比特数为10的序列相对量化比特数为8的序列码率节省5%~20%左右。     

Quadtree-based non-local Kuan’s filtering in video compression

Transform coding has been widely used in video coding standards, such as H.264 advanced video coding (H.264/AVC) and high efficiency video coding (HEVC). But the coded video sequences suffer from annoying coding artifacts, such as blocking and ringing artifacts. In this paper, we propose the quadtree-based non-local Kuan’s (QNLK) filter to suppress the quantization noise optimally and improve the objective and subjective quality of the reconstructed frame simultaneously. The proposed filter takes advantage of the non-local Kuan’s (NLK) filter to restore the quantized signal in transform domain. Restored coefficients are then projected onto designed quantization constraint sets (QCS). Quadtree-based signaling strategy is used at the end of QNLK for adaptive filtering on/off control. Experimental results of QNLK show that the proposed method achieves significant objective coding gain and visual quality improvement, compared with both H.264/AVC high profile and HEVC.     

Adaptive transform skipping for improved coding of motion compensated residuals

New generations of video compression algorithms, such as those included in the under development High Efficiency Video Coding (HEVC) standard, provide substantially higher compression compared to their ancestors. The gain is achieved by improved prediction of pixels, both within a frame and between frames. Novel coding tools that contribute to the gain provide highly uncorrelated prediction residuals for which classical frequency decomposition methods, such as the discrete cosine transform, may not be able to supply a compact representation with few significant coefficients. To further increase the compression gains, this paper proposes transform skip modes which allow skipping one or both 1-D constituent transforms (i.e., vertical and horizontal), which is more suitable for sparse residuals. The proposed transform skip mode is tested in the HEVC codec and is able to provide bitrate reductions of up to 10% at the same objective quality when compared with the application of 2-D block transforms only. Moreover, the proposed transform skip mode outperforms the full transform skip currently investigated for possible adoption in the HEVC standard.     

Fast inter-prediction mode decision algorithm for HEVC

The high efficiency video coding (HEVC) is superior to the previous video coding standards in compression performance, while the computational complexity is introduced simultaneously. The complexity increases mainly due to the novel flexible partitioning scheme that allows the inter-prediction mode partition split via exhaustive rate-distortion optimization (RDO). In this paper, a fast inter-prediction mode decision algorithm is proposed, which contains adaptive threshold determination based on quantization parameter and fast inter-prediction mode partition decision. The proposed algorithm utilizes the edge information of the partition to simplify the RDO and then accelerates coding time of inter-prediction mode decision structure for the original HEVC encoder. The experimental results show that the proposed algorithm can achieve 39.5 % coding time reduction with just 1.97 % bitrate increase on average under random access condition and 35.2 % coding time reduction with just 1.89 % bitrate increase on average under low-delay B condition, compared to the original HEVC encoder.     

基于模式对应与机器学习的HEVC降分辨率转码算法

HEVC是ITU-T VCEG 继H.264之后所制定的新的视频编码标准,它提高了视频的编码效率,在相同视频质量的前提下,压缩比与H.264相比提高了一倍。另外,随着4G网络的兴起和智能手机的普及,移动终端成为人们观看网上视频的一大主流平台。但是,网络中存储的视频分辨率普遍要大于移动终端屏幕分辨率,为解决这个问题,本文开展了针对HEVC的降分辨率转码研究工作,利用高分辨率视频的编码信息,通过模式对应来简化低分辨率视频的编码模式的计算过程,并采用机器学习的方法来确定降分辨率时的组块阈值,以提高模式对应的准确性。实验结果表明,提出的算法与Trivial transcoder相比,在保持PSNR和比特率几乎不变的同时,编码时间平均节省了60%左右。