视频帧间分组及超分辨率重建的自适应性研究

由于视频场景变化较快、配准误差、噪声、低分辨率图像数量不足等原因,会使传统基于压缩感知的采用视频帧固定分组形式的视频编解码器的重构效果较差,同时也使超分辨率重建出现病态问题。为解决这些问题,文章提出一种基于压缩感知的自适应帧图像分组的视频编解码器,同时又在超分辨率重建算法中提出了L曲线的自适应时空正则化系数计算方法,可以自适应地计算正规化系数。由实验结果表明,该算法能够很好地解决上述问题从而重构出视觉效果良好的视频帧图像。     

无人机侦察视频超分辨率重建方法

目的 无人机摄像资料的分辨率直接影响目标识别与信息获取,所以摄像分辨率的提高具有重大意义。为了改善无人机侦察视频质量,针对目前无人机摄像、照相数据的特点,提出一种无人机侦察视频超分辨率重建方法。方法 首先提出基于AGAST-Difference与Fast Retina Keypoint (FREAK)的特征匹配算法对视频目标帧与相邻帧之间配准,然后提出匹配区域搜索方法找到目标帧与航片的对应关系,利用航片对视频帧进行高频补偿,最后采用凸集投影方法对补偿后视频帧进行迭代优化。结果 基于AGAST-Difference与FREAK的特征匹配算法在尺度、旋转、视点等变化及运行速度上存在很大优势,匹配区域搜索方法使无人机视频的高频补偿连续性更好,凸集投影迭代优化提高了重建的边缘保持能力,与一种简单有效的视频序列超分辨率复原算法相比,本文算法重建质量提高约4 dB,运行速度提高约5倍。结论 提出了一种针对无人机的视频超分辨率重建方法,分析了无人机视频超分辨率问题的核心所在,并且提出基于AGAST-Difference与FREAK的特征匹配算法与匹配区域搜索方法来解决图像配准与高频补偿问题。实验结果表明,本文算法强化了重建图像的一致性与保真度,特别是对图像边缘细节部分等效果极为明显,且处理速度更快。     

基于非局部均值和总变分最小化的单视频超分辨率算法

传统的基于重建的单视频超分辨率方法能够获得较好的重建效果。然而,已有算法没有充分利用视频内的帧间、帧内相关性,重建效果仍有待提升。针对这一问题,提出了一种新的单视频超分辨率算法。为充分利用帧内相关性,采用非局部均值模型表征帧内非局部结构特性,采用总变分模型表征帧内局部结构特性;为了探索帧间相关性,采用光流法进行帧间预测。最后,为了求解所建立的优化问题,提出了基于split-Bregman方法的快速迭代算法。实验结果表明,与同类算法相比,所提算法在主、客观质量上均有相应的提升。     

基于注意力融合网络的视频超分辨率重建

基于深度学习的视频超分辨率方法主要关注视频帧内和帧间的时空关系,但以往的方法在视频帧的特征对齐和融合方面存在运动信息估计不精确、特征融合不充分等问题。针对这些问题,采用反向投影原理并结合多种注意力机制和融合策略构建了一个基于注意力融合网络（AFN）的视频超分辨率模型。首先,在特征提取阶段,为了处理相邻帧和参考帧之间的多种运动,采用反向投影结构来获取运动信息的误差反馈;然后,使用时间、空间和通道注意力融合模块来进行多维度的特征挖掘和融合;最后,在重建阶段,将得到的高维特征经过卷积重建出高分辨率的视频帧。通过学习视频帧内和帧间特征的不同权重,充分挖掘了视频帧之间的相关关系,并利用迭代网络结构采取渐进的方式由粗到精地处理提取到的特征。在两个公开的基准数据集上的实验结果表明,AFN能够有效处理包含多种运动和遮挡的视频,与一些主流方法相比在量化指标上提升较大,如对于4倍重建任务,AFN产生的视频帧的峰值信噪比（PSNR）在Vid4数据集上比帧循环视频超分辨率网络（FRVSR）产生的视频帧的PSNR提高了13.2%,在SPMCS数据集上比动态上采样滤波视频超分辨率网络（VSR-DUF）产生的视频帧的PSNR提高了15.3%。     

基于纹理特征的图像自动配准方法研究

精确的图像配准是超分辨率重建取得成功的前提。为此,针对大多数图像配准算法存在精度低、抗噪声能力差等缺点,提出一种基于纹理特征的图像自动配准方法。首先用Canny边缘检测算子提取裂缝纹理图像,然后用Photoshop图层工具,通过对待配准帧关于参考帧作变换,得到空间几何变换参数,最后用Matlab编程实现待配准帧的缩放、旋转和平移。对配准后的酸蚀岩板裂缝图像,进行超分辨率重建,结果证明,该方法具有抗噪声能力强、配准精度高、易于编程实现等优点,取得了较好的效果。     

基于卷积网络的帧率提升算法研究*

基于运动补偿的帧率提升算法是目前主要的帧率提升方法。为减小内插帧中的块效应、孔洞和遮挡问题,提高插值帧质量,本文提出一种基于卷积神经网络（convolutional neural network）的自学习帧率提升（frame rate up-conversion）方法。卷积神经网络用于利用两相邻帧预测待插值帧。在卷积神经网络的训练阶段,我们假设高帧率视频是存在的,网络参数由高帧率视频与低帧率视频训练而来。最后视频数据以低帧率视频加网络参数的形式传输,在接收端就可以利用卷积神经网络重建高帧率视频。实质上,我们这样做是通过增加视频发布者的负担以提供给视频接受者更多便利。对于视频点播网站来说,这是提升用户体验的重要因素。实验表明,我们的方案相对于传统的基于运动补偿的帧率提升算法,平均PSNR提升至少0.6 DB,取得较大程度提升。并且,我们的方法是基于全局的帧预测方法,可以有效避免快效应、孔洞和遮挡问题。     

基于深度学习特征匹配的视频超分辨率方法

视频复原的目标是从给定的退化视频序列中把潜在的高质量视频复原出来.现有的视频复原方法主要集中在如何有效地找到相邻帧之间的运动信息,然后利用运动信息建立相邻帧之间的匹配.与这些方法不同,文中提出了基于深度学习特征匹配的方法来解决视频超分辨率问题.首先,通过深度卷积神经网络计算出相邻帧之间的运动信息;然后,采用一个浅层深度卷积神经网络从输入的视频帧中提取特征,基于估计到的运动信息,将浅层深度卷积神经网络提取到的特征匹配到中间视频帧对应的特征中,并将得到的特征进行有效融合;最后,采用一个深度卷积神经网络重建视频帧.大量的实验结果验证了基于深度学习特征匹配的方法能有效地解决视频超分辨率问题.与现有的基于视频帧匹配的方法相比,所提方法在现有的公开视频超分辨率数据集上取得了较好的效果.     

基于光流残差的视频超分辨率重建算法

随着卷积神经网络的发展,视频超分辨率算法取得了显著的成功。因为帧与帧之间的依赖关系比较复杂,所以传统方法缺乏对复杂的依赖关系进行建模的能力,难以对视频超分辨率重建的过程进行精确地运动估计和补偿。因此提出一个基于光流残差的重建网络,在低分辨率空间使用密集残差网络得到相邻视频帧的互补信息,通过金字塔的结构来预测高分辨率视频帧的光流,通过亚像素卷积层将低分辨率的视频帧变成高分辨率视频帧,并将高分辨率的视频帧与预测的高分辨率光流进行运动补偿,将其输入到超分辨率融合网络来得到更好的效果,提出新的损失函数训练网络,能够更好地对网络进行约束。在公开数据集上的实验结果表明,重建效果在峰值信噪比、结构相似度、主观视觉的效果上均有提升。     

基于卷积神经网络的视频图像超分辨率重建方法

为了进一步增强视频图像超分辨率重建的效果,研究利用卷积神经网络的特性进行视频图像的空间分辨率重建,提出了一种基于卷积神经网络的视频图像重建模型。采取预训练的策略用于重建模型参数的初始化,同时在多帧视频图像的空间和时间维度上进行训练,提取描述主要运动信息的特征进行学习,充分利用视频帧间图像的信息互补进行中间帧的重建。针对帧间图像的运动模糊,采用自适应运动补偿加以处理,对通道进行优化输出得到高分辨率的重建图像。实验表明,重建视频图像在平均客观评价指标上均有较大提升（PSNR +0.4 dB / SSIM +0.02）,并且有效减少了图像在主观视觉效果上的边缘模糊现象。与其他传统算法相比,在图像评价的客观指标和主观视觉效果上均有明显的提升,为视频图像的超分辨率重建提供了一种基于卷积神经网络的新颖架构,也为进一步探索基于深度学习的视频图像超分辨率重建方法提供了思路。     

基于SIFT的POCS图像超分辨率重建

针对传统的POCS图像超分辨率重建算法中广泛使用的基于改进的Keren配准算法,对于序列帧间存在剪切和非均匀尺度变换现象时,很难做到精确的亚像素级配准,文中讨论了一种基于SIFT算法的POCS序列图像超分辨率重建算法。首先利用SIFT算法提取序列帧与参考帧间的SIFT关键点对,随后选取匹配关键点对,通过RANSAC去除误配点的同时估算出六参数仿射变换参数,最后使用POCS重建算法得到最终的重建结果。实验结果表明：该方法能有效地解决因运动估计不准而引起的重建图像效果不好的问题,特别是在序列帧间存在剪切和非均匀尺度变换现象时,重建效果明显好于传统的POCS算法,具有更强适应性。     

An adaptive image interpolation algorithm for image/video processing

Image interpolation is one of the key technologies in image/video processing. In this study, a new adaptive image interpolation algorithm is proposed. The objective of the proposed approach is to recover up-sampled image frames from the corresponding decimated (low-resolution) image frames. In the proposed approach, within each iteration, two proposed nonlinear filters are utilized to iteratively generate high-frequency components lost within the decimation procedure. Finally, a post-processing procedure is adopted to reduce the blocking artifacts within the interpolated images. Based on the experimental results obtained in this study, in terms of the average PSNR_p (peak signal-to-noise ratio) in dB and subjective measure of the quality of the interpolated images, the interpolation results by the proposed approach are better than that by three existing interpolation approaches for comparison.     

Image super resolution using distributed locality sensitive hashing for manifold learning

In this paper we propose a distributed locality sensitive hashing based framework for image super resolution exploiting computational and storage efficiency of cloud. Now days huge multimedia data is available on the cloud which can be utilized using store anywhere and excess anywhere model. It may be noted that super resolution is required for consumer electronics display devices due to various reasons. The propose framework exploits the image correlation for image super resolution using locality sensitive hashing (LSH) for manifold learning. In our work we have exploited the benefits of manifold learning for image super resolution, which in-turn is a highly time complex operation. The time complexity is involved due to finding the approximate nearest neighbors from trillion of image patches for locally linear embedding (LLE) operation. In our approach it is mitigated by using a distributed framework which internally uses hash tables for mapping of patches in the target image from a database of internet picture collection. The proposed framework for super resolution provides promising results in comparison to existing approaches.

     

基于光流法的迭代反投影超分辨率重构算法

为了提高重构图像或者视频的分辨率．提出把新型的基于光流法的图像配准算法应用于迭代反投影（IBP）超分辨率算法中。在所提出的方法中．基于光流法的图像配准算法用来提高图像配准的准确性。首先,为了得到像素级别的运动矢量．基于光流法的图像配准算法被用于估计图像间的运动矢量。以得到更加准确的运动矢量矩阵。接着,利用所获得的运动矢量矩阵结合迭代反投影算法重构高分辨率的图像。同时．由于基于光流法的图像配准能够很好地估计视频图像间的运动．所提出的方法同样适用于视频图像的超分辨。实验结果表明．提出的方法对于图像或者视频的超分辨率效果．在主观效果和客观评价上都有一定的提升。     

基于结构张量的视频超分辨率算法

针对传统正则化超分辨率（SR）重建模型中,正则化参数选择过大会使重建结果模糊,导致边缘和纹理等细节丢失,选择过小模型去噪能力又不足的问题,提出一种基于结构张量的双正则化参数的视频超分辨率重建算法。首先,利用局部结构张量对图像进行平滑区域和边缘的检测;然后,利用差异曲率对全变分（TV）进行先验信息加权;最后,对平滑区域和边缘采用不同的正则化参数进行超分辨率重建。实验数据显示提出的算法将峰值信噪比（PSNR）提高了0.033～0.11 dB,具有较好的重建效果。实验结果表明：该算法能够有效地提升低分辨率（LR）视频帧重建效果,可应用于低分辨率视频增强、车牌识别和视频监控中感兴趣目标增强等方面。     

Super resolution land cover mapping of hyperspectral images using the deep image prior-based approach

ABSTRACT

Due to the instantaneous field-of-view (IFOV) of the sensor and diversity of land cover types, some pixels, usually named mixed pixels, contain more than one land cover type. Soft classification can predict the portion of each land cover type in mixed pixels in the absence of spatial distribution. The spatial distribution information in mixed pixels can be solved by super resolution mapping (SRM). Typically, SRM involves two steps: soft class value estimation, which is similar to the image super resolution of image restoration, and land cover allocation. A new SRM approach utilizes a deep image prior (DIP) strategy combined with a super resolution convolutional neural network (SRCNN) to estimate fine resolution fraction images for each land cover type; then, a simple and efficient classifier is used to allocate subpixel land cover types under the constraint of the generated fine fraction images. The proposed approach can use prior information of input images to update network parameters and no longer require training data. Experiments on three different cases demonstrate that the subpixel classification accuracy of the proposed DIP-based SRM approach is significantly better than the three conventional SRM approaches and a transfer learning-based neural network SRM approach. In addition, the DIP-SRM approach performs very robustly about small-area objects within multiple land cover types and significantly reduces soft classification uncertainty. The results of this paper provide an extension for utilizing SRCNN to address SRM issues in hyperspectral images.     

Super-resolution Reconstruction of Remote Sensing Images by Using Empirical Mode Decomposition and Compressed Sensing

Super resolution (SR) of remote sensing images is significant for improving accuracy of target identification and for image fusing.Conventional fusion-based methods inevitably result in distortion of spectral information,a feasible solution to the problem is the single-image based super resolution.In this work,we proposed a single-image based approach to super resolution of multiband remote sensing images.The method combines the EMD (Empirical Mode Decomposition),compressed sensing and PCA to dictionary learning and super resolution reconstruction of remote sensing color image.First,the original image is decomposed into a series of IMFs(Intrinsic Mode Function) according to their frequency component by using EMD,and the super resolution is implemented only on IMF1,which includes high-frequency component;then the K-SVD algorithm is used to learn and obtain overcomplete dictionaries,and the MOP (Orthogonal Matching Pursuit) algorithm is used to reconstruct the IMF1;Finally,the up-scaled IMF1 is combined with other IMFs to acquire the super resolution of original image.For a multiband image reconstruction,a PCA transform is first implemented on multiband image,and the PC1 is adopted for learning to get overcomplete dictionaries,the obtained dictionaries is then used to super-resolution reconstruction of each multi-spectral band.The Geoeye-1 panchromatic and multi-spectral images are used as experimental data to demonstrate the effectiveness of the proposed algorithm.The results show that the proposed method is workable to exhibit the detail within the images.     

Motion block based video super resolution

This paper presents a new video super resolution technique, based on the motion and static areas of the low resolution video frames. In order to separate the motion and static blocks, a block motion estimation method is performed between a reference and its neighboring frames. Among the motion blocks, the occluded blocks are identified using an adaptive threshold applied on each block individually. Structure-adaptive normalized convolution (SANC) reconstruction method is used to generate the high resolution static and motion blocks where discrete wavelet transform (DWT) based interpolation is used to produce the high resolution occluded blocks. The static and motion blocks are combined into a high resolution frame. Finally, a sharpening process is performed on the high resolution frame in order to generate the super resolved high resolution output frame. The experimental results show that the proposed technique provides significantly better qualitative visual results as well as quantitative higher PSNR than the state of the art video super resolution algorithms.