图像超分辨率重建研究综述

图像超分辨率重建（Super-resolution Reconstruction,SR）是由一张或多张低分辨率图像得到高分辨率图像的过程.近年来,SR技术不断发展,在许多领域被广泛应用.本文在回顾SR技术发展历史的基础上,全面综述了SR技术在各个时期的代表性方法,重点介绍了基于深度学习的图像超分辨率工作.我们从模型类型、网络结构、信息传递方式等方面对各种算法进行了详细评述,并对比了其优缺点.最后探讨了图像超分辨率技术未来的发展方向.     

Super-resolution image reconstruction: a technical overview

A new approach toward increasing spatial resolution is required to overcome the limitations of the sensors and optics manufacturing technology. One promising approach is to use signal processing techniques to obtain an high-resolution (HR) image (or sequence) from observed multiple low-resolution (LR) images. Such a resolution enhancement approach has been one of the most active research areas, and it is called super resolution (SR) (or HR) image reconstruction or simply resolution enhancement. In this article, we use the term "SR image reconstruction" to refer to a signal processing approach toward resolution enhancement because the term "super" in "super resolution" represents very well the characteristics of the technique overcoming the inherent resolution limitation of LR imaging systems. The major advantage of the signal processing approach is that it may cost less and the existing LR imaging systems can be still utilized. The SR image reconstruction is proved to be useful in many practical cases where multiple frames of the same scene can be obtained, including medical imaging, satellite imaging, and video applications. The goal of this article is to introduce the concept of SR algorithms to readers who are unfamiliar with this area and to provide a review for experts. To this purpose, we present the technical review of various existing SR methodologies which are often employed. Before presenting the review of existing SR algorithms, we first model the LR image acquisition process.     

Optimized video compression with residual split attention and swin-block artifact contraction

Research in video compression has seen significant advancement in the last several years. However, the existing deep learning-based algorithms continue to be plagued by erroneous motion compression and ineffective motion compensation architectures, resulting in compression errors with a lower rate–distortion trade-off. To overcome these challenges, we present an end-to-end purely deep learning-based video compression method through a set of primary operations (e.g., motion estimation, motion compression, motion compensation, residual compression, and artifact contraction) differently. A deep residual attention split (DRAS) block is introduced for motion compression networks to pay more attention to certain image regions to create more effective features for the decoder while boosting the rate–distortion optimization (RDO) efficiency. A channel residual block (CRB) is proposed in motion compensation to yield a more accurate predicted frame, potentially improving the residual frame. To mitigate the compression errors, an artifact contraction module (ACM) by residual swin convolution UNet block is included in this model to improve the reconstruction quality. To improve the final frame, a buffer is added to fine-tune the previous reference frames. These modules combine with a loss function by assessing the trade-off and enhancing the decoded video quality. A comprehensive ablation study demonstrates the effectiveness of the proposed blocks and modules for video compression. Experimental results show the competitive performance of the proposed method on four benchmark datasets.     

生成式对抗神经网络的多帧红外图像超分辨率重建

生成式对抗神经网络在约束图像生成表现出了巨大潜力,使得其适合运用于图像超分辨率重建。但是使用生成式对抗神经网络重建后的超分辨率图像存在过度平滑,缺少高频细节信息的缺点。针对单帧图像超分辨率重建方法不能有效利用图像序列间的时间-空间相关性的问题,提出了一种基于生成式对抗神经网络的多帧红外图像超分辨率重建方法（M-GANs）。首先,对低分辨率图像序列进行运动补偿;其次,使用权值表示卷积层对运动补偿后的图像序列进行权值转换计算;最后,将其输入生成式对抗重建网络,输出重建后的高分辨率图像。实验结果表明:文中方法在主观及客观评价中均优于当前代表性的超分辨率重建方法。     

A nonlinear least square technique for simultaneous image registration and super-resolution.

This paper proposes a new algorithm to integrate image registration into image super-resolution (SR). Image SR is a process to reconstruct a high-resolution (HR) image by fusing multiple low-resolution (LR) images. A critical step in image SR is accurate registration of the LR images or, in other words, effective estimation of motion parameters. Conventional SR algorithms assume either the estimated motion parameters by existing registration methods to be error-free or the motion parameters are known a priori. This assumption, however, is impractical in many applications, as most existing registration algorithms still experience various degrees of errors, and the motion parameters among the LR images are generally unknown a priori. In view of this, this paper presents a new framework that performs simultaneous image registration and HR image reconstruction. As opposed to other current methods that treat image registration and HR reconstruction as disjoint processes, the new framework enables image registration and HR reconstruction to be estimated simultaneously and improved progressively. Further, unlike most algorithms that focus on the translational motion model, the proposed method adopts a more generic motion model that includes both translation as well as rotation. An iterative scheme is developed to solve the arising nonlinear least squares problem. Experimental results show that the proposed method is effective in performing image registration and SR for simulated as well as real-life images.     

采用双网络结构的压缩视频超分辨率重建

在实际应用中,为了节省带宽和方便存储,图像和视频通常被下采样和压缩,而降质的图像与视频无法满足人们的实际需求。针对这一问题,采用了一种双网络结构的超分辨率重建方法,首先建立下采视频与压缩后的低分辨率视频的映射关系,然后建立质量增强的压缩视频与原始视频的映射关系,最终在输出端可以得到质量提升的视频帧。在网络中,采用密集残差块来提取压缩视频中丰富的局部分层特征,并结合全局残差学习恢复视频中的高频信息。在压缩环节,采用高性能视频编码来验证所提算法的有效性。实验结果表明,相比于主流的视频编码标准和先进的超分辨率重建算法,所提方法能有效提升编码视频的率失真性能。     

基于聚类和高斯过程回归的超分辨率重建

在样本学习的思想框架下,针对图像超分辨率问题的研究,提出了数据聚类和高斯过程回归相结合的解决方法.使用K-means对数据进行聚类,在各类中利用高斯过程回归对样本库中高低分辨率图像之间的对应关系进行学习.根据得到的学习模型对需要处理的低分辨率图像所对应的高分辨率图像进行预测,有效地利用了高低分辨率图像之间的统计特性.实验结果表明该方法可以较好地改善超分辨率重建效果.     

A multi-stage spatio-temporal adaptive network for video super-resolution

Video super-resolution aims at restoring the spatial resolution of the reference frame based on consecutive input low-resolution (LR) frames. Existing implicit alignment-based video super-resolution methods commonly utilize convolutional LSTM (ConvLSTM) to handle sequential input frames. However, vanilla ConvLSTM processes input features and hidden states independently in operations and has limited ability to handle the inter-frame temporal redundancy in low-resolution fields. In this paper, we propose a multi-stage spatio-temporal adaptive network (MS-STAN). A spatio-temporal adaptive ConvLSTM (STAC) module is proposed to handle input features in low-resolution fields. The proposed STAC module utilizes the correlation between input features and hidden states in the ConvLSTM unit and modulates the hidden states adaptively conditioned on fused spatio-temporal features. A residual stacked bidirectional (RSB) architecture is further proposed to fully exploit the processing ability of the STAC unit. The proposed STAC and RSB architecture promote the vanilla ConvLSTM’s ability to exploit the inter-frame correlations, thus improving the reconstruction quality. Furthermore, different from existing methods that only aggregate features from the temporal branch once at a specified stage of the network, the proposed network is organized in a multi-stage manner. The corresponding temporal correlation in features at different stages can be fully exploited. Experimental results on Vimeo-90K-T and UMD10 datasets show that the proposed method has comparable performance with current video super-resolution methods. The code is available at https://github.com/yhjoker/MS-STAN.     

A state-space super-resolution approach for video reconstruction

The main objective of super-resolution video reconstruction is to make use of a set of low-resolution image frames to produce their respective counterparts with higher resolution. The conventional two-equation-based Kalman filter only considers the information from the previously reconstructed high-resolution frame and the currently observed low-resolution frame for producing each high-resolution frame. It has been observed that the information inherited in the previously observed low-resolution frame could be beneficial on the reconstruction of the super-resolution video. For that, an extra observation equation is incorporated into the framework of the conventional two-equation-based Kalman filtering in this paper to establish a three-equation-based state-space approach as a more generalized framework. The closed-form solution is mathematically derived, and extensive simulations using both artificially degraded and real-life image sequences are conducted to demonstrate its superior performance. Furthermore, a unified theoretical analysis is provided to analyze the relationship between the proposed framework and two existing super-resolution approaches, the sliding-window-based Bayesian estimation approach and the conventional two-equation-based Kalman filtering, respectively.     

基于Zernike矩的空间视频序列时空自适应性超分辨率重建模型

Video Super-Resolution (SR) reconstruc-tion produces video sequences with High Resolu-tion (HR) via the fusion of several Low-Resolution (LR) video frames. Traditional methods rely on the accurate estimation of subpixel motion, which con-strains their applicability to video sequences with relatively simple motions such as global translation. We propose an efficient iterative spatio-temporal a-daptive SR reconstruction model based on Zernike Moment (ZM), which is effective for spatial video sequences with arbitrary motion. The model uses re-gion correlation judgment and self-adaptive thresh-old strategies to improve the effect and time effi-ciency of the ZM-based SR method. This leads to better mining of non-local self-similarity and local structural regularity, and is robust to noise and rota-tion. An efficient iterative curvature-based interpo-lation scheme is introduced to obtain the initial HR estimation of each LR video frame. Experimental results both on spatial and standard video sequences demonstrate that the proposed method outperforms existing methods in terms of both subjective visual and objective quantitative evaluations, and greatly improves the time efficiency.     

多级FFD融合超分辨率重建的视频人脸识别

针对视频人脸识别中由于人脸畸变、表情变化等非刚性变化导致无法精确配准和重建的问题,提出一种基于多级自由变形配准的超分辨率重建算法。首先,利用低分辨率FFD网格全局配准,引入边缘配准度量到差平方总和准则;然后,将全局配准后的图像和基准图像划分成一系列对应子图对,使用高分辨率FFD网格对相关系数小的子图对进行局部配准;最后,采用凸集投影算法对多帧低分辨率图像重建SR人脸图像,并利用支持向量机分类器完成人脸识别。在标准视频库Choke Point和自己搜集的人脸视频库上的实验结果表明,在人脸畸变和表情变化很大的情况下,本文算法也能够精确配准和重建人脸图像,相比其它几种视频人脸识别算法,本文算法取得了更好的识别效果。     

Superresolution video reconstruction with arbitrary samplinglattices and nonzero aperture time

Printing from an NTSC source and conversion of NTSC source material to high-definition television (HDTV) format are some of the applications that motivate superresolution (SR) image and video reconstruction from low-resolution (LR) and possibly blurred sources. Existing methods for SR image reconstruction are limited by the assumptions that the input LR images are sampled progressively, and that the aperture time of the camera is zero, thus ignoring the motion blur occurring during the aperture time. Because of the observed adverse effects of these assumptions for many common video sources, this paper proposes (i) a complete model of video acquisition with an arbitrary input sampling lattice and a nonzero aperture time, and (ii) an algorithm based on this model using the theory of projections onto convex sets to reconstruct SR still images or video from an LR time sequence of images. Experimental results with real video are provided, which clearly demonstrate that a significant increase in the image resolution can be achieved by taking the motion blurring into account especially when there exists large interframe motion.     

结合卷积神经网络的HEVC帧内编码压缩改进算法

近年来,卷积网络深度学习已在图像处理、目标检测等领域取得巨大成功。受其启发,将卷积神经网络(CNN)应用于传统视频压缩标准已成为一个新的研究热点。本文提出一种集成卷积神经网络的高效视频编码(HEVC)压缩改进算法,将下采样过程、HEVC的编解码过程、上采样及质量增强过程集成为一体。为高效提取视频帧的结构特征,在所提压缩算法中集成了两个卷积神经网络。提出了一种下采CNN(DwSCNN)代替双三次下采,在有效降低分辨率的同时保留细节信息,得到更为紧凑的低分辨率视频序列,将此低分辨率视频序列通过HEVC帧内编码进行进一步的数据量压缩,通过提出一个质量增强CNN(PPCNN)来改善解码后恢复到原始分辨率的降质视频序列。实验结果显示,本文压缩改进算法在低码率段与标准HEVC相比,能达到更好的质量重建,并且在接近一致的PSNR值时,能节省39.46%的时间和11.04%的比特率,本文算法的视频压缩性能优于HEVC标准算法和相关文献方法。     

基于自适应字典稀疏表示的超分辨率算法

基于学习的超分辨率算法通过一组训练样例来学习一个字典,并从该字典中合成低分辨率图像中丢失的高频信息,最终得到相应的高分辨率图像。介绍了几种常用的基于学习的超分辨率算法,并提出了一种新的算法:基于自适应字典稀疏表示的超分辨率算法。实验结果表明,该方法在主观与客观上均具有较好的重建效果。     

Super resolution image reconstruction through bregman iteration using morphologic regularization

Multiscale morphological operators are studied extensively in the literature for image processing and feature extraction purposes. In this paper, we model a nonlinear regularization method based on multiscale morphology for edge-preserving super resolution (SR) image reconstruction. We formulate SR image reconstruction as a deblurring problem and then solve the inverse problem using Bregman iterations. The proposed algorithm can suppress inherent noise generated during low-resolution image formation as well as during SR image estimation efficiently. Experimental results show the effectiveness of the proposed regularization and reconstruction method for SR image.     

PSO-based Fusion Method for Video Super-Resolution

In this study, video super-resolution using particle swarm optimization (PSO) is proposed to super-resolve low-resolution (LR) frames. The proposed super-resolution method consists of three main modules, i.e., supersampling, spatio-temporal classification, and frame fusion using PSO. In the proposed method, the LR frames are super-resolved to high-resolution frames through the fusion of four full-resolution frames. One of four full-resolution frames is obtained using direct spatial interpolation, and the other three are obtained using motion compensation with given reference frames. The essence of the proposed method is the spatio-temporal classification mechanism that exploits the temporal variation between frames and the spatial energy inside the frame. Using the classification results, PSO is used to determine the optimal weights for frame fusion. Simulation results show that the proposed fusion method successfully improves the perceptual quality and the average peak signal-to-noise ratio (PSNR) in super-resolved frames.     

Extraction of high-resolution frames from video sequences

The human visual system appears to be capable of temporally integrating information in a video sequence in such a way that the perceived spatial resolution of a sequence appears much higher than the spatial resolution of an individual frame. While the mechanisms in the human visual system that do this are unknown, the effect is not too surprising given that temporally adjacent frames in a video sequence contain slightly different, but unique, information. This paper addresses the use of both the spatial and temporal information present in a short image sequence to create a single high-resolution video frame. A novel observation model based on motion compensated subsampling is proposed for a video sequence. Since the reconstruction problem is ill-posed, Bayesian restoration with a discontinuity-preserving prior image model is used to extract a high-resolution video still given a short low-resolution sequence. Estimates computed from a low-resolution image sequence containing a subpixel camera pan show dramatic visual and quantitative improvements over bilinear, cubic B-spline, and Bayesian single frame interpolations. Visual and quantitative improvements are also shown for an image sequence containing objects moving with independent trajectories. Finally, the video frame extraction algorithm is used for the motion-compensated scan conversion of interlaced video data, with a visual comparison to the resolution enhancement obtained from progressively scanned frames.     

Reference guided image super-resolution via efficient dense warping and adaptive fusion

Due to the limited improvement of single-image based super-resolution (SR) methods in recent years, the reference based image SR (RefSR) methods, which super-resolve the low-resolution (LR) input with the guidance of similar high-resolution (HR) reference images are emerging. There are two main challenges in RefSR, i.e. reference image warping and exploring the guidance information from the warped references. For reference warping, we propose an efficient dense warping method to deal with large displacements, which is much faster than traditional patch (or texture) matching strategy. For the SR process, since different reference images complement each other, and have different similarities with the LR image, we further propose a similarity based feature fusion strategy to take advantage of the most similar reference regions. The SR process is realized by an encoder–decoder network and trained with pixel-level reconstruction loss, degradation loss and feature-level perceptual loss. Extensive experiments on three benchmark datasets demonstrate that the proposed method outperforms state-of-the-art SR methods in both subjective and objective measurements.     

Image super-resolution with sparse neighbor embedding

Until now, neighbor-embedding-based (NE) algorithms for super-resolution (SR) have carried out two independent processes to synthesize high-resolution (HR) image patches. In the first process, neighbor search is performed using the Euclidean distance metric, and in the second process, the optimal weights are determined by solving a constrained least squares problem. However, the separate processes are not optimal. In this paper, we propose a sparse neighbor selection scheme for SR reconstruction. We first predetermine a larger number of neighbors as potential candidates and develop an extended Robust-SL0 algorithm to simultaneously find the neighbors and to solve the reconstruction weights. Recognizing that the k-nearest neighbor (k-NN) for reconstruction should have similar local geometric structures based on clustering, we employ a local statistical feature, namely histograms of oriented gradients (HoG) of low-resolution (LR) image patches, to perform such clustering. By conveying local structural information of HoG in the synthesis stage, the k-NN of each LR input patch is adaptively chosen from their associated subset, which significantly improves the speed of synthesizing the HR image while preserving the quality of reconstruction. Experimental results suggest that the proposed method can achieve competitive SR quality compared with other state-of-the-art baselines.     

Improved hybrid layered image compression using deep learning and traditional codecs

Recently deep learning-based methods have been applied in image compression and achieved many promising results. In this paper, we propose an improved hybrid layered image compression framework by combining deep learning and the traditional image codecs. At the encoder, we first use a convolutional neural network (CNN) to obtain a compact representation of the input image, which is losslessly encoded by the FLIF codec as the base layer of the bit stream. A coarse reconstruction of the input is obtained by another CNN from the reconstructed compact representation. The residual between the input and the coarse reconstruction is then obtained and encoded by the H.265/HEVC-based BPG codec as the enhancement layer of the bit stream. Experimental results using the Kodak and Tecnick datasets show that the proposed scheme outperforms the state-of-the-art deep learning-based layered coding scheme and traditional codecs including BPG in both PSNR and MS-SSIM metrics across a wide range of bit rates, when the images are coded in the RGB444 domain.