Improving video quality by predicting inter-frame residuals based on an additive 3D-CNN model

Video compression is essential for uploading videos to online platforms which usually have bandwidth limitations. However, the compression reduces the visual quality. To overcome this problem, the visual quality of the low bitrate compressed videos for various standards, including H.264 and HEVC in decoders, needs to be improved. Accordingly, this paper proposes a novel method for improving video quality based on 3D convolutional neural networks (CNNs). This method is totally compatible with the encoders of video compression standards, i.e., H.264, VVC, and HEVC, and can be implemented easily. In particular, the proposed neural network model receives five frames of the low bitrate compressed video as input and subsequently predicts the compression error of frames using the first and fifth frames. Finally, it reconstructs an improved version of the frame with high quality. The CNN is an Additive (3D) model that can predict the eliminated inter-frame redundancies resulting from compression. Our goal is to increase the peak signal to noise ratio (PSNR) and structural index similarity (SSIM) of the luminance (Y) and chrominance (U, V) frames in the video. Additive 3D-CNN achieves an average of 12.4%, 9.9% and 5% BD-rate increases for LP, LB and RA for the Y component. The results indicate that the new proposed algorithm outperforms the previous methods in terms of PSNR, SSIM, and BD-rate.