一种稳健的多视频时空超分辨率重建算法

为了提高视频序列的时间分辨率和空间分辨率,该文提出了一种基于一阶范数和时空总变分正则化方法的时空超分辨率视频重建算法。该算法利用同一场景的多个具有子像素空间位移偏差和子帧率时间偏差的低分辨率视频序列,重建得到一个高时空分辨率视频序列。在求解过程中不需要直接构造大型矩阵,大大降低了对内存的要求。实验结果表明该算法是有效的,且对成像模型估计误差具有一定的鲁棒性。     

超分辨率重建算法综述

随着高分辨率移动设备和超高清电视的发展,对已有的低分辨率视频进行超分辨率上采样成为最近的一个研究热点.对已有的超分辨率重建算法根据输入输出方式的不同,分为多图像超分辨率重建、单图像超分辨率重建、视频超分辨率重建三大类,综述了其中每类算法的发展情况及常用算法,并对不同算法的特点分析比较.随后讨论了多图像超分辨率重建和单图像超分辨率重建方法对视频超分辨率重建方法的影响,最后展望了超分辨率重建算法的进一步发展.     

一种基于子像素配准视频超分辨率重建方法

为了提高视频的空间分辨率,提出了一种利用帧间运动信息进行超分辨率重建的方法。对于整个视频的重建,提出了一种基于滑动窗的分段重建模型。在每一个滑动窗中,首先对相邻帧进行子像素级精度的运动配准;然后通过迭代反投影算法进行超分辨率重建。在配准算法中,提出了一种基于四参数刚体变换模型的配准方法,通过迭代求解和高斯金字塔图像模型由粗及精地进行运动估计。分别对模拟图像及实拍彩色视频进行重建,实验结果表明,该配准算法具有较高的精度,重建算法取得了较高的峰值信噪比(PSNR)值,重建视频具有更好的视觉效果和更高的分辨率能力,可被广泛应用于在帧间主要存在平移和旋转运动的视频序列的超分辨率重建。     

超分辨率重构

利用低分辨率的图像序列来估计高分辨率图像的方法称为超分辨率图像重建,逐步成为当前科研热点。本文通过POCS视频图像重建算法为例,阐述了超分辨率的概念应用场合及基本策略和分类,并对超分辨率的重构方法和前景进行了展望。     

超分辨率重构

利用低分辨率的图像序列来估计高分辨率图像的方法称为超分辨率图像重建,逐步成为当前科研热点.本文通过POCS视频图像重建算法为例,阐述了超分辨率的概念、应用场合及基本策略和分类,并对超分辨率的重构方法和前景进行了展望.     

图像超分辨率研究综述

介绍了图像超分辨率技术的概念及来源,综述了超分辨率技术在国内外发展的概况,澄清了图像超分辨率重建和超分辨率复原两个概念,对图像超分辨率的方法进行了分类对比,并对图像超分辨率的发展进行了展望.     

基于超分辨率重建的强压缩深度伪造视频检测

经典的深度伪造(DeepFake)视频检测方法一般使用卷积神经网络进行检测,但在强压缩深度伪造换脸视频数据集上表现较差,并会对真实数据做出大量误检测。针对这个问题,该文提出一种基于超分辨率重建的强压缩深度伪造视频检测方法。该方法基于深度神经网络检测模型,通过融入超分辨率重建技术,恢复强压缩视频所损失的空间与时间信息,进而提升对强压缩视频的检测准确率。使用FaceForensics++及DFDC数据集进行实验,针对强压缩的深度伪造视频,该方法较ResNet50提高了单帧以及视频的测试准确率,有效缓解强压缩真实视频的误检测问题。     

基于超分辨率重建的强压缩深度伪造视频检测

经典的深度伪造(DeepFake)视频检测方法一般使用卷积神经网络进行检测,但在强压缩深度伪造换脸视频数据集上表现较差,并会对真实数据做出大量误检测.针对这个问题,该文提出一种基于超分辨率重建的强压缩深度伪造视频检测方法.该方法基于深度神经网络检测模型,通过融入超分辨率重建技术,恢复强压缩视频所损失的空间与时间信息,进而提升对强压缩视频的检测准确率.使用FaceForensics++及DFDC数据集进行实验,针对强压缩的深度伪造视频,该方法较ResNet50提高了单帧以及视频的测试准确率,有效缓解强压缩真实视频的误检测问题.     

视频监控图像的超分辨率复原研究

超分辨率复原技术是一种可用于提高图像细节辨识能力的有效方法。其在视频监控领域可望得到广泛应用。超分辨率图像处理技术通过融合多帧相似的低分辨率图像达到提高图像细节的目的。从而降低对监控视频采集硬件与后端辅助处理系统的要求,提高对特定目标的辨析能力。本文重点介绍了在视频监控领域较为实用的凸集投影算法、最大后验概率估计算法、基于对象的超分辨率复原方法、基于示例学习与多类预测器的超分辨率复原方法。对以上超分辨率复原方法实现流程的优缺点与其在视频图像监控领域的应用方法进行了相应分析。分析了超分辨率视频监控图像复原常用的基于块匹配与光流的对象运动估计方法。对超分辨率复原重建图像质量的评估标准也进行了相应讨论。     

基于L曲线的自适应时空正则化视频超分辨率重建

为了解决视频超分辨率重建的病态问题,以得到良好的重建效果,提出了一种新颖的视频超分辨率重建算法。在算法中引入了时空联合正则化算子,通过视频帧本身的空间平滑信息和视频相邻帧的帧间相关先验信息的引入,提高了解的质量;同时,为了选择合适的时空正则化系数,提出了基于L曲线的自适应时空正则化系数计算方法,可以自适应地计算合适的正则化系数。通过对模拟图像序列和真实视频序列的实验结果表明,算法能得到较为精确的解,重建出具有良好视觉效果的高分辨率视频。     

结合空时域下采样与重建的H.264/AVC压缩性能优化

为提高H.264/AVC标准在带宽资源严重受限时的压缩效率,采用空时域相结合的编码思路,提出了一种基于运动检测的自适应抽帧方法,并结合空域下采样与重建研究了一种改进的H.264/AVC压缩性能优化框架。在编码端,原视频先空域下采样以减少空间分辨率,然后根据视频运动特征,采用不同抽帧模式自适应地降低帧率,再经H.264/AVC编码,有效降低了编码码率。在解码端,解码视频则采用与抽帧模式相对应的运动估计与补偿插帧方法重建出抽取帧,再利用超分辨率重建技术将视频恢复到原空间分辨率。实验结果表明,所提方法在低码率段的视频压缩性能优于H.264/AVC标准编解码及相关文献方法。     

A spatiotemporal super-resolution algorithm for a hybrid stereo video system

This paper presents a spatiotemporal super-resolution method to enhance both the spatial resolution and the frame rate in a hybrid stereo video system. In this system, a scene is captured by two cameras to form two videos, including a low spatial resolution with high-frame-rate video and a high spatial resolution with low-frame-rate video. For the low-spatial-resolution video, the low-resolution frames are spatially super-resolved by the high-resolution video via the stereo matching, the bilateral overlapped block motion estimation, and the adaptive overlapped block motion compensation algorithms, while for the low-frame-rate video, those missed frames are interpolated using the high-resolution frames obtained by fusing the disparity compensation and the motion compensation frame rate up-conversion. Experimental results demonstrate that the proposed mixed spatiotemporal super-resolution method has a more significant contribution to both the subjective and objective qualities than the pure spatial super-resolution or the frame rate up-conversion.     

Learning from EPI-Volume-Stack for Light Field image angular super-resolution

Light Field (LF) image angular super-resolution aims to synthesize a high angular resolution LF image from a low angular resolution one, and is drawing increased attention because of its wide applications. In order to reconstruct a high angular resolution LF image, many learning based LF image angular super-resolution methods have been proposed. However, most existing methods are based on LF Epipolar Plane Image or Epipolar Plane Image volume representation, which underuse the LF image structure. The LF view spatial correlation and neighboring LF views angular correlations which can reflect LF image structure are not fully explored, which reduces LF angular super-resolution quality. In order to alleviate this problem, this paper introduces an Epipolar Plane Image Volume Stack (EPI-VS) representation for LF angular super-resolution. The EPI-VS is constituted by arranging all LF views in a raster order, which benefits in exploring LF view spatial correlation and neighboring LF views angular correlations. Based on such representation, we further propose an LF angular super-resolution network. 3D convolutions are applied in the whole super-resolution network to better accommodate the input EPI-VS data and allow information propagation between two spatial and one directional dimensions of EPI-VS data. Extensive experiments on synthetic and real-world LF scenes demonstrate the effectiveness of the proposed network. Moreover, we also illustrate the superiority of our network by applying it in scene depth estimation task.     

高速超分辨率图像处理系统

为了对遥感图像进行实时脱机的超分辨率增强处理,设计了一种基于现场可编程逻辑门阵列(FPGA)和数字信号处理器(DSP)的超分辨率图像处理系统,系统在视频预处理部分利用FPGA来实现对视频的降采样处理,在超分辨率处理部分分别用FPGA和DSP来完成对视频源的两种经典超分辨率算法的硬件实现.达到了对视频图像进行高速实时超分辨率处理的目的.     

A multi-stage spatio-temporal adaptive network for video super-resolution

Video super-resolution aims at restoring the spatial resolution of the reference frame based on consecutive input low-resolution (LR) frames. Existing implicit alignment-based video super-resolution methods commonly utilize convolutional LSTM (ConvLSTM) to handle sequential input frames. However, vanilla ConvLSTM processes input features and hidden states independently in operations and has limited ability to handle the inter-frame temporal redundancy in low-resolution fields. In this paper, we propose a multi-stage spatio-temporal adaptive network (MS-STAN). A spatio-temporal adaptive ConvLSTM (STAC) module is proposed to handle input features in low-resolution fields. The proposed STAC module utilizes the correlation between input features and hidden states in the ConvLSTM unit and modulates the hidden states adaptively conditioned on fused spatio-temporal features. A residual stacked bidirectional (RSB) architecture is further proposed to fully exploit the processing ability of the STAC unit. The proposed STAC and RSB architecture promote the vanilla ConvLSTM’s ability to exploit the inter-frame correlations, thus improving the reconstruction quality. Furthermore, different from existing methods that only aggregate features from the temporal branch once at a specified stage of the network, the proposed network is organized in a multi-stage manner. The corresponding temporal correlation in features at different stages can be fully exploited. Experimental results on Vimeo-90K-T and UMD10 datasets show that the proposed method has comparable performance with current video super-resolution methods. The code is available at https://github.com/yhjoker/MS-STAN.     

基于空域变换的超分辨成像系统

传统的超分辨成像系统易受外界环境影响,抗噪效果差,输出的超分辨图像清晰度较低,基于此设计基于空域变换的超分辨成像系统,系统先将激光集中在孔径光栏上射出,通过聚光镜汇聚激光,根据分光镜和辅助镜将激光聚焦在显微镜焦平面上,依照物镜将激光束转变成平型光,使平行光能够均匀照射测量样品上,采用显微镜和辅助物镜处理从样品表面反射的带有样品信息的激光,最后通过分光镜反射带有信息的激光,反射光束在CCD上显示成像,核心处理器是FPGAWie的图像采集模块,采集CCD传输的视频图像后,通过通用并行接口将采集的视频图像数据传输到图像处理模块,该模块通过空域变换方法处理图像,获取超分辨图像,最终传输到计算机上进行显示。经过实验分析发现,该系统图像稳定性和图像清晰度平均值分别是98.538%和99.19%,干扰信噪比为16 d B时,该系统成像时间最短为6.57 ms,说明该系统成像清晰度高、抗干扰性能优。     

采用边缘Gibbs现象抑制的红外图像POCS重构算法

在研究和分析凸集投影(Projection Onto Convex Sets,POCS)法的基础上,提出了一种改进的图像超分辨率重构算法.该算法充分利用空域确定模型,通过平滑降噪处理,经运动估计进行配准;突出数字图像细节信息的同时,有针对性地修改点扩散函数(Point Spread Function,PSF)取值,通过有效抑制边缘Gibbs现象获得最佳质量的高分辨率重构图像.对重构后的红外图像质量进行了定量评价.结果表明,图像质量取得了预期的效果.该技术在红外目标识别与跟踪、红外侦察与反侦察、舰船红外目标特性研究、高清数字图像处理、旧视频翻录和生物信息提取与识别等方面具有重要的应用价值.     

Patch-based spatio-temporal super-resolution for video with non-rigid motion

This paper presents a novel approach for spatio-temporal video super-resolution. Whereas the task of synthesizing high-frequency information on the spatial domain can be accomplished without introducing arbitrary priors on the image model (beyond the assumption of local cross-scale self-similarity), the high degree of temporal aliasing in standard-frame-rate video requires applying motion compensation in order to correctly interpolate the video sequence along the temporal axis. As the experimental results show, the proposed technique is capable of augmenting both the frame-rate of a video sequence (by any real up-converting factor) and its effective spatial resolution (by any rational magnification factor). The presented method is suitable for parallelized computing environments, such as GPUs, and provides an output quality comparable to state-of-the-art methods in both temporal interpolation and spatial super-resolution.