Hyperspectral image super-resolution combining with deep learning and spectral unmixing

In recent years, hyperspectral image super-resolution has attracted the attention of many researchers and has become a hot topic in the field of computer vision. However, it is difficult to obtain high-resolution images due to imaging hardware devices. At present, many existing hyperspectral image super-resolution methods have not achieved good results. In this paper, we propose a hyperspectral image super-resolution method combining with deep residual convolutional neural network (DRCNN) and spectral unmixing. Firstly, the spatial resolution of the image is enhanced by learning a priori knowledge of natural images. The DRCNN reconstructs high spatial resolution hyperspectral images by concatenating multiple residual blocks, each containing two convolutional layers. Secondly, the spectral features of low-resolution and high-resolution hyperspectral images are linked by spectral unmixing. This approach aims to obtain the endmember matrix and the abundance matrix. The final reconstruction result is obtained by multiplying the endmember matrix and the abundance matrix. In addition, in order to improve the visual effect of the reconstructed image, the total variation regularity is used to impose constraints on the abundance matrix to enhance the relationship between the pixels. The experimental results of remote sensing data based on ground facts show that the proposed method has good performance and preserves spatial information and spectral information without the need for auxiliary images.     

基于压缩卷积神经网络的图像超分辨率算法

为了有效提高深度图像的分辨率,文中借鉴经典SqueezeNet网络结构,提出一种基于Fire Module的卷积神经网络模型。该算法实现了直接从低分辨率图像到高分辨率图像的映射和转化,其中Fire Module作为网络的非线性映射模块,在减少参数的同时可学习图像的深层特征。为了避免插值预处理,在网络的输出层引入反卷积层,实现3倍上采样和高分辨率图像的输出。实验表明,采用该基于Fire Module的卷积神经网络模型的反卷积算法得到的超分辨率图像细节更加丰富,客观指标PSNR值和SSIM值的评价也明显优于其他算法。     

基于自注意力深度网络的图像超分辨率重建方法

针对现有图像超分辨重建方法难以充分重建图像的细节信息且易出现重建的图像缺乏层次的问题,提出一种基于自注意力深度网络的图像超分辨重建方法。以深度神经网络为基础,通过提取低分辨率图像特征,建立低分辨率图像特征到高分辨率图像特征的非线性映射,重建高分辨率图像。在进行非线性映射时,引入自注意力机制,获取图像中全部像素间的依赖关系,利用图像的全局特征指导图像重建,增强图像层次。在训练深度神经网络时,使用图像像素级损失和感知损失作为损失函数,以强化网络对图像细节信息的重建能力。在3类数据集上的对比测试结果表明,所提方法能够提升图像超分辨重建结果的细节信息,且重建图像的视觉效果更好。     

Variational method for super-resolution optical flow

The motion fields in an image sequence observed by a car-mounted imaging system depend on the positions in the imaging plane. Since the motion displacements in the regions close to the camera centre are small, for accurate optical flow computation in this region, we are required to use super-resolution of optical flow fields. We develop an algorithm for super-resolution optical flow computation. Super-resolution of images is a technique for recovering a high-resolution image from a low-resolution image and/or image sequence. Optical flow is the appearance motion of points on the image. Therefore, super-resolution optical flow computation yields the appearance motion of each point on the high-resolution image from a sequence of low-resolution images. We combine variational super-resolution and variational optical flow computation in super-resolution optical flow computation. Our method directly computes the gradient and spatial difference of high-resolution images from those of low-resolution images, without computing any high-resolution images used as intermediate data for the computation of optical flow vectors of the high-resolution image.     

基于卷积稀疏自编码的图像超分辨率重建

针对卷积稀疏编码算法中特征映射的准确性的问题,为了进一步提高图像超分辨率重建的的质量,文中提出一种基于卷积稀疏自编码的图像超分辨率重建算法。该算法首先在预训练阶段利用稀疏自编码器对输入高低分辨率图像分别进行训练,得到对应的图像稀疏特征表示;然后再由卷积神经网络根据得到的稀疏系数共同训练相应的滤波器及特征映射函数并更新到最优解;最后由高分辨率滤波器和对应的稀疏表示系数卷积求和,得到高分辨率重建图像估计。实验结果显示,改进算法的峰值信噪比（PSNR）结果较卷积稀疏编码算法提高了近0.1 dB,有效提高了重建图像的质量。     

基于门控卷积神经网络的图像超分辨重建算法北大核心CSCD

近年来,卷积神经网络被广泛应用于图像超分辨率领域。针对基于卷积神经网络的超分辨率算法存在图像特征提取不充分,参数量大和训练难度大等问题,本文提出了一种基于门控卷积神经网络(gated convolutional neural network, GCNN)的轻量级图像超分辨率重建算法。首先,通过卷积操作对原始低分辨率图像进行浅层特征提取。之后,通过门控残差块(gated residual block, GRB)和长短残差连接充分提取图像特征,其高效的结构也能加速网络训练过程。GRB中的门控单元(gated unit, GU)使用区域自注意力机制提取输入特征图中的每个特征点权值,紧接着将门控权值与输入特征逐元素相乘作为GU输出。最后,使用亚像素卷积和卷积模块重建出高分辨率图像。在Set14、BSD100、Urban100和Manga109数据集上进行实验,并和经典方法进行对比,本文算法有更高的峰值信噪比(peak signal-to-noise ratio,PSNR)和结构相似性(structural similarity,SSIM),重建出的图像有更清晰的轮廓边缘和细节信息。     

基于稀疏正则化结合NLS的超分辨率图像重建

为了保持高光谱(HS)超分辨率重建过程中的频谱一致性和边缘锐度,提出一种基于空间谱结合非局部相似性的超分辨率重建算法。首先,使用HS图像生成模型,采用稀疏正则化解决全色(PAN)图像和HS图像重建的病态问题求逆;然后分析了从高空间分辨率到低空间分辨率数据生成的丰度系数映射;最后利用非局部相似性,设计空间谱联合正则化项。实验结果表明,本文算法重建图像在PSNR,SSIM和FSIM方面明显高于其他优秀算法,在SAM和ERGAS方面明显低于其他优秀算法,在光谱失真方面丢失最少,仅有2%-3%,低于其他算法30%左右,且重建效果更加清晰自然。     

Polyphase back-projection filtering for image resolution enhancement

The method for reconstruction and restoration of super-resolution images from sets of low-resolution images presented is an extension of the algorithm proposed by Irani and Peleg (1991). After estimating the projective transformation parameters between the image sequence frames, the observed data are transformed into a sequence with only quantised sub-pixel translations. The super-resolution reconstruction is an iterative process, in which a high-resolution image is initialised and iteratively improved. The improvement is achieved by back-projecting the errors between the translated low-resolution images and the respective images obtained by simulating the imaging system. The imaging system's point-spread function (PSF) and the back-projection function are first estimated with a resolution higher than that of the super-resolution image. The two functions are then decimated so that two banks of polyphase filters are obtained. The use of the polyphase filters allows exploitation of the input data without any smoothing and/or interpolation operations. The presented experimental results show that the resolution improvement is better than the results obtained with Irani and Peleg's algorithm.     

基于光谱稀疏模型的高光谱压缩感知重构

提出了一种基于光谱稀疏化的压缩感知采样与重构模型,通过从训练样本中构建光谱稀疏字典提升光谱稀疏化效果,同时在重构时兼顾空间图像的全变分约束进一步提升重构精度.对200波段AVIRIS高光谱场景进行压缩感知重构的实验表明,利用构建的光谱稀疏字典与传统的DCT字典和Haar小波字典相比光谱稀疏化效果明显提升,同时在25%采样下基于光谱稀疏字典几乎无差别重构出了高光谱图像,同样条件下在空间和光谱的精度与现有常用方法相比有较大的提升.     

基于聚类和高斯过程回归的超分辨率重建

在样本学习的思想框架下,针对图像超分辨率问题的研究,提出了数据聚类和高斯过程回归相结合的解决方法.使用K-means对数据进行聚类,在各类中利用高斯过程回归对样本库中高低分辨率图像之间的对应关系进行学习.根据得到的学习模型对需要处理的低分辨率图像所对应的高分辨率图像进行预测,有效地利用了高低分辨率图像之间的统计特性.实验结果表明该方法可以较好地改善超分辨率重建效果.     

Image super-resolution reconstruction based on sparse representation and deep learning

Super-resolution reconstruction technology has important scientific significance and application value in the field of image processing by performing image restoration processing on one or more low-resolution images to improve image spatial resolution. Based on the SCSR algorithm and VDSR network, in order to further improve the image reconstruction quality, an image super-resolution reconstruction algorithm combined with multi-residual network and multi-feature SCSR(MRMFSCSR) is proposed. Firstly, at the sparse reconstruction stage, according to the characteristics of image blocks, our algorithm extracts the contour features of non-flat blocks by NSCT transform, extracts the texture features of flat blocks by Gabor transform, then obtains the reconstructed high-resolution (HR) images by using sparse models. Secondly, according to improve the VDSR deep network and introduce the feature fusion idea, the multi-residual network structure (MR) is designed. The reconstructed HR image obtained by the sparse reconstruction stage is used as the input of the MR network structure to optimize the high-frequency detail residual information. Finally, we can obtain a higher quality super-resolution image compared with the SCSR algorithm and the VDSR algorithm.     

Singular value decomposition based fusion for super-resolution image reconstruction

In this paper, we address a super-resolution problem of generating a high-resolution image from low-resolution images. The proposed super-resolution method consists of three steps: image registration, singular value decomposition (SVD)-based image fusion and interpolation. The contribution of this work is two-fold. First we customize an image registration approach using Scale Invariant Feature Transform (SIFT), Belief Propagation and Random Sampling Consensus (RANSAC) for super-resolution. Second, we propose SVD-based fusion to integrate the important features from the low-resolution images. The proposed image registration and fusion steps effectively maintain the important features and greatly improve the super-resolution results. Results, for a variety of image examples, show that the proposed method successfully generates high-resolution images from low-resolution images.     

Super-resolution reconstruction of hyperspectral images.

Hyperspectral images are used for aerial and space imagery applications, including target detection, tracking, agricultural, and natural resource exploration. Unfortunately, atmospheric scattering, secondary illumination, changing viewing angles, and sensor noise degrade the quality of these images. Improving their resolution has a high payoff, but applying super-resolution techniques separately to every spectral band is problematic for two main reasons. First, the number of spectral bands can be in the hundreds, which increases the computational load excessively. Second, considering the bands separately does not make use of the information that is present across them. Furthermore, separate band super-resolution does not make use of the inherent low dimensionality of the spectral data, which can effectively be used to improve the robustness against noise. In this paper, we introduce a novel super-resolution method for hyperspectral images. An integral part of our work is to model the hyperspectral image acquisition process. We propose a model that enables us to represent the hyperspectral observations from different wavelengths as weighted linear combinations of a small number of basis image planes. Then, a method for applying super resolution to hyperspectral images using this model is presented. The method fuses information from multiple observations and spectral bands to improve spatial resolution and reconstruct the spectrum of the observed scene as a combination of a small number of spectral basis functions.     

基于支撑向量机的盲超分辨率图像复原算法

本文提出了一种基于支撑向量机的盲超分辨率图像复原算法.首先采用Sobel算子和局部方差从训练图像中提取能够表征模糊参数信息的特征向量,并利用支撑向量机建立特征向量与对应的候选参数的映射关系,然后通过建立的模型对不同光照条件下的低分辨率图像进行参数辨识,最后根据辨识出的模糊参数融合不同光照条件下的低分辨率图像同时实现了图像动态范围和空间分辨率的增强.为了实现低分辨率图像间的亚像素配准,还提出了一种基于Retinex的亚像素运动估计算法.仿真结果表明与传统算法相比,无论从主观视觉还是定量描述上本文算法均具有较好的效果.     

基于正则化稀疏表示的图像超分辨率算法

为了从单幅低分辨率(LR)图像恢复出高分辨率(H R)图像,提出了一种应用正则化稀疏表示和基于机器学习 的超分辨率(SR)图像恢复算法。构造了一种基于稀疏表示的SR凸变模型,为了提高 恢复效果,针对模型 提出了两种稀疏正则化约束条件,一是将分类效果更好的图表拉普拉斯作为正则化约束条件 ,从而找到与 输入LR图像块在结构上最接近的学习样本;另一种是针对冗余的学习样本进行约 束,保证了图像边 缘的锐利。将输入的每一块LR图像应用正则化稀疏表示,经过学习得到与之对应的HR图像块 , 最终得到整幅HR图像。试验结果表明,算法恢复出的HR图像峰值信噪比(PSNR )值较双三次插值算法最高提升约2dB,主观目视清晰、边缘锐利。     

基于多维度特征遥感图像分类方法的研究北大核心CSCD

为了解决传统高光谱图像分类方法精度低、计算成本高及未能充分利用空-谱信息的问题,本文提出一种基于多维度并行卷积神经网络(multidimensional parallel convolutional neural network,3D-2D-1D PCNN)的高光谱图像分类方法。首先,该算法利用不同维度卷积神经网络(convolutional neural network,CNN)提取高光谱图像信息中的空-谱特征、空间特征及光谱特征;之后,采用相同并行卷积层将组合后的空-谱特征、空间特征及光谱特征进行特征融合;最后,通过线性分类器对高光谱图像信息进行精准分类。本文所提方法不仅可以提取高光谱图像中更深层次的空间特征和光谱特征信息,同时能够将光谱图像不同维度的特征进行融合,减小计算成本。在Indian Pines、Pavia Center和Pavia University数据集上对本文算法和4种传统算法进行对比实验,结果表明,本文算法均得到最优结果,分类精度分别达到了99.210%、99.755%和99.770%。     

Multi-resolution analysis techniques and nonlinear PCA for hybrid pansharpening applications

Hyperspectral images have a higher spectral resolution (i.e., a larger number of bands covering the electromagnetic spectrum), but a lower spatial resolution with respect to multispectral or panchromatic acquisitions. For increasing the capabilities of the data in terms of utilization and interpretation, hyperspectral images having both high spectral and spatial resolution are desired. This can be achieved by combining the hyperspectral image with a high spatial resolution panchromatic image. These techniques are generally known as pansharpening and can be divided into component substitution (CS) and multi-resolution analysis (MRA) based methods. In general, the CS methods result in fused images having high spatial quality but the fused images suffer from spectral distortions. On the other hand, images obtained using MRA techniques are not as sharp as CS methods but they are spectrally consistent. Both substitution and filtering approaches are considered adequate when applied to multispectral and PAN images, but have many drawbacks when the low-resolution image is a hyperspectral image. Thus, one of the main challenges in hyperspectral pansharpening is to improve the spatial resolution while preserving as much as possible of the original spectral information. An effective solution to these problems has been found in the use of hybrid approaches, combining the better spatial information of CS and the more accurate spectral information of MRA techniques. In general, in a hybrid approach a CS technique is used to project the original data into a low dimensionality space. Thus, the PAN image is fused with one or more features by means of MRA approach. Finally the inverse projection is used to obtain the enhanced image in the original data space. These methods, permit to effectively enhance the spatial resolution of the hyperspectral image without relevant spectral distortions and on the same time to reduce the computational load of the entire process. In particular, in this paper we focus our attention on the use of Nonlinear Principal Component Analysis (NLPCA) for the projection of the image into a low dimensionality feature space. However, if on one hand the NLPCA has been proved to better represent the intrinsic information of hyperspectral images in the feature space, on the other hand an analysis of the impact of different fusion techniques applied to the nonlinear principal components in order to define the optimal framework for the hybrid pansharpening has not been carried out yet. More in particular, in this paper we analyze the overall impact of several widely used MRA pansharpening algorithms applied in the nonlinear feature space. The results obtained on both synthetic and real data demonstrate that an accurate selection of the pansharpening method can lead to an effective improvement of the enhanced hyperspectral image in terms of spectral quality and spatial consistency, as well as a strong reduction in the computational time.     

Integrating the Missing Information Estimation into Multi-frame Super-Resolution

Multi-frame super-resolution image reconstruction aims to restore a high-resolution image by fusing a set of low-resolution images. The low-resolution images are usually subject to some degradation, such as warping, blurring, down-sampling, or noising, which causes substantial information loss in the low-resolution images, especially in the texture regions. The missing information is not well estimated using existing traditional methods. In this paper, having analyzed the observation model describing the degradation process starting with a high-resolution image and moving to the low-resolution images, we propose a more reasonable observation model that integrates the missing information into the super-resolution reconstruction. Our approach is fully formulated in a Bayesian framework using the Kullback–Leibler divergence. In this way, the missing information is estimated simultaneously with the high-resolution image, motion parameters, and hyper-parameters. Our proposed estimation of the missing information improves the quality of the reconstructed image. Experimental results presented in this paper show improved performance compared with that of existing traditional methods.     

多孔径压缩编码超分辨率大视场成像方法

多孔径成像是一种融合了仿生复眼视觉的新型成像方法,具有小型化、大视场、高分辨率等多种优势,但由于每个子孔径对应的单元图像分辨率过低,导致其成像质量和视场角的提升十分有限。为了进一步提高成像分辨率和探测视场,基于压缩感知理论设计随机编码模板,并紧贴子孔径放置对入射光场进行调制,通过单次曝光记录编码后的低分辨率单元图像阵列,利用稀疏优化算法,重构所有低分辨率单元图像获得超分辨率大视场图像。理论分析和仿真实验证明了该方法的有效性。该方法不仅能兼顾大视场高分辨率成像,而且大大缩小系统等效焦距,具有薄层结构,体积小而重量轻,可为微光机电一体化系统的研制设计提供借鉴。     

Stochastic super-resolution image reconstruction

The objective of super-resolution (SR) imaging is to reconstruct a single higher-resolution image based on a set of lower-resolution images that were acquired from the same scene to overcome the limitations of image acquisition process for facilitating better visualization and content recognition. In this paper, a stochastic Markov chain Monte Carlo (MCMC) SR image reconstruction approach is proposed. First, a Bayesian inference formulation, which is based on the observed low-resolution images and the prior high-resolution image model, is mathematically derived. Second, to exploit the MCMC sample-generation technique for the stochastic SR image reconstruction, three fundamental issues are observed as follows. First, since the hyperparameter value of the prior image model controls the degree of regularization and intimately affects the quality of the reconstructed high-resolution image, how to determine an optimal hyperparameter value for different low-resolution input images becomes a very challenging task. Rather than exploiting the exhaustive search, an iterative updating approach is developed in this paper by allowing the value of hyperparameter being simultaneously updated in each sample-generation iteration. Second, the samples generated during the so-called burn-in period (measured in terms of the number of samples initially generated) of the MCMC-based sample-generation process are considered unreliable and should be discarded. To determine the length of the burn-in period for each set of low-resolution input images, a time-period bound in closed form is mathematically derived. Third, image artifacts could be incurred in the reconstructed high-resolution image, if the number of samples (counting after the burn-in period) generated by the MCMC-based sample-generation process is insufficient. For that, a variation-sensitive bilateral filter is proposed as a ‘complementary’ post-processing scheme, to improve the reconstructed high-resolution image quality, when the number of samples is insufficient. Extensive simulation results have clearly shown that the proposed stochastic SR image reconstruction method consistently yields superior performance.