Comprehensive review on fusion techniques for spatial information enhancement in hyperspectral imagery

The volume of data grows with the advent of multiple types of remote sensing sensors and in order to extract the most useful information there is a need to combine the data gathered from the different sources. The widely used panchromatic and multispectral imageries in many applications offer decimetric and metric spatial resolution. However, the spectral resolution of these images is poor. Hyperspectral imaging has unique characteristics of providing very fine spectral resolution in a large number of bands with decametric spatial resolution and found to be highly useful for a wide span of application areas that requires high spectral resolution. The fusion of spectral and spatial information provides an effective way of enhancing the spatial quality of hyperspectral imagery as well as a method for preserving spectral quality. This fusion process is not a trivial task as always there has been a tradeoff between the preservation of spatial and spectral quality information as in the original sources of fusion. In this paper, a review on hyperspectral pansharpening and hyperspectral multispectral fusion based approaches has been reported. The widely adopted quantitative and qualitative performance measures to verify the fusion results are highlighted. In addition, the challenges in existing fusion techniques have also been discussed.     

Practical remote sensing image fusion method based on guided filter and improved SML in the NSST domain

Due to the different characteristics of image modality, the panchromatic (PAN) and multispectral (MS) images include complementary and redundancy information in the spatial and spectral resolutions. Image fusion is an effective way to integrate the source PAN and MS images to obtain high-resolution MS image. In this paper, a novel remote sensing image fusion scheme in non-subsample Shearlet transform (NSST) domain is presented. An enhancement strategy is designed to solve the insufficiency of spatial detail in multiresolution analysis (MRA)-based methods after the intensity–hue–saturation (IHS) color space transform. Then, in the NSST fusion process, a guided filter-based low-frequency coefficient fusion rule and an improved sum-modified-Laplacian (SML)-based high-frequency coefficient fusion rule are proposed. The final fused image can be obtained through the inverse NSST transform and inverse IHS transform. Two different groups of satellite dataset are utilized to evaluate the fusion performance. The experiment results demonstrate that the proposed approach can achieve more spatial details and less spectral distortion compared with the existing methods regarding both the visual quality and the objective measurements.     

High spectral quality pansharpening approach based on MTF-matched filter banks

Pansharpening consists in merging a low-resolution multispectral image (MS) with a high spatial resolution panchromatic image (PAN) to produce a high resolution pansharpened MS image. It consists in enhancing spatially the low-resolution MS image by injecting the missing details provided by the high-resolution PAN image. In this paper, we propose a novel pansharpening approach based on decomposition/reconstruction processing using low-pass and high-pass filter banks. On the one hand, the low-pass approximation (taking into account the imaging system modulation transfer function MTF) of the pansharpened MS image is assumed to be equal to the original MS image in order to preserve the spectral quality. On the other hand, the high-pass filter allowing us to extract the high-frequency PAN details is designed as complementary filter to the low-pass one in order to provide perfect reconstruction in the ideal case. Quantitative assessment performed on reduced and full-resolution images are used to validate the proposed technique and compare it to state-of-art. Experimental results using Pléaides and GeoEye-1 data show that our proposed fusion schema outperforms the pre-existing methods visually as well as quantitatively.     

Estimation of the Number of Decomposition Levels for a Wavelet-Based Multiresolution Multisensor Image Fusion

The wavelet-based scheme for the fusion of multispectral (MS) and panchromatic (PAN) imagery has become quite popular due to its ability to preserve the spectral fidelity of the MS imagery while improving its spatial quality. This is important if the resultant imagery is used for automatic classification. Wavelet-based fusion results depend on the number of decomposition levels applied in the wavelet transform. Too few decomposition levels result in poor spatial quality fused images. On the other hand, too many levels reduce the spectral similarity between the original MS and the pan-sharpened images. If the shift-invariant wavelet transform is applied, each excessive decomposition level results in a large computational penalty. Thus, the choice of the number of decomposition levels is significant. In this paper, PAN and MS image pairs with different resolution ratios were fused using the shift-invariant wavelet transform, and the optimal decomposition levels were determined for each resolution ratio. In general, it can be said that the fusion of images with larger resolution ratios requires a higher number of decomposition levels. This paper provides the practitioner an understanding of the tradeoffs associated with the computational demand and the spatial and spectral quality of the wavelet-based fusion algorithm as a function of the number of decomposition levels     

Generalized IHS‐Based Satellite Imagery Fusion Using Spectral Response Functions

Image fusion is a technical method to integrate the spatial details of the high‐resolution panchromatic (HRP) image and the spectral information of low‐resolution multispectral (LRM) images to produce high‐resolution multispectral images. The most important point in image fusion is enhancing the spatial details of the HRP image and simultaneously maintaining the spectral information of the LRM images. This implies that the physical characteristics of a satellite sensor should be considered in the fusion process. Also, to fuse massive satellite images, the fusion method should have low computation costs. In this paper, we propose a fast and efficient satellite image fusion method. The proposed method uses the spectral response functions of a satellite sensor; thus, it rationally reflects the physical characteristics of the satellite sensor to the fused image. As a result, the proposed method provides high‐quality fused images in terms of spectral and spatial evaluations. The experimental results of IKONOS images indicate that the proposed method outperforms the intensity‐hue‐saturation and wavelet‐based methods.     

Introduction of sensor spectral response into image fusion methods. Application to wavelet-based methods

Usual image fusion methods inject features from a high spatial resolution panchromatic sensor into every low spatial resolution multispectral band trying to preserve spectral signatures and improve spatial resolution to that of the panchromatic sensor. The objective is to obtain the image that would be observed by a sensor with the same spectral response (i.e., spectral sensitivity and quantum efficiency) as the multispectral sensors and the spatial resolution of the panchromatic sensor. But in these methods, features from electromagnetic spectrum regions not covered by multispectral sensors are injected into them, and physical spectral responses of the sensors are not considered during this process. This produces some undesirable effects, such as resolution overinjection images and slightly modified spectral signatures in some features. The authors present a technique which takes into account the physical electromagnetic spectrum responses of sensors during the fusion process, which produces images closer to the image obtained by the ideal sensor than those obtained by usual wavelet-based image fusion methods. This technique is used to define a new wavelet-based image fusion method.     

基于滤波器组的遥感图像融合方法及其性能分析研究

提出了一种基于滤波器组的图像融合方法,用以融合高空间分辨率全色图像和低空间分辨率的多光谱图像．在高空间分辨率全色图像经过多通道滤波器组分解的基础上,用多光谱图像直接替换全色图像低频子图像的方式进行融合处理;最后对替代后的子图像进行滤波器组重构得到融合后的图像．实验结果表明,通过调整滤波器组的通道个数,该方法能够使融合图像中空间信息和多光谱信息获得更好地折衷．     

基于CNN的QuickBird遥感影像融合研究

为了充分利用QuickBird多光谱影像的光谱信息和全色影像的高空间分辨率信息,提出了一种基于细胞神经网络（CNN）的QuickBird多光谱和全色遥感影像融合方法.实验结果表明,相对于传统的影像融合算法,提出的方法具有更好的性能.     

联合多流融合和多尺度学习的卷积神经网络遥感图像融合方法

为尽可能保持原始低分辨率多光谱(LRMS)图像光谱信息的同时,显著提高融合后的多光谱图像的空间分辨率,该文提出一种联合多流融合和多尺度学习的卷积神经网络遥感图融合方法.首先将原始MS图像输入频谱特征提取子网得到其光谱特征,然后分别将通过梯度算子处理全色图像得到的梯度信息和通过卷积后的全色图像与得到的光谱特征图在通道上拼...     

一种超高分辨率遥感图像融合新算法

该文针对超高分辨率的全色光图像和多光谱图像的融合,提出了一种基于对应分析的图像融合新算法。该算法在对多光谱数据进行对应分析的基础上,利用冗余小波变换提取出全色光图像的空间细节信息并将其融入到成分空间。实验分别采用IKONOS和QuickBird数据,融合结果的目视效果与客观评价表明,相比现有同类融合方法,该方法能够在提高空间分辨率的同时更好地保持光谱特性,有效地减少了色彩失真的现象。     

Image Fusion Processing for IKONOS 1-m Color Imagery

Many image fusion techniques have been developed. However, most existing fusion processes produce color distortion in 1-m fused IKONOS images due to nonsymmetrical spectral responses of IKONOS imagery. Here, we proposed a fusion process to minimize this spectral distortion in IKONOS 1-m color images. The 1-m fused image is produced from a 4-m multispectral (MS) and 1-m panchromatic (PAN) image, maintaining the relations of spectral responses between PAN and each band of the MS images. To obtain this relation, four spectral weighting parameters are added with the pixel value of each band of the original MS image. Then, each pixel value is updated using a steepest descent method to reflect the maximum spectral response on the fused image. Comparison among the proposed technique and existing processes [intensity hue saturation (IHS) image fusion, Brovey transform, principal component analysis, fast IHS image fusion] has been done. Our proposed technique has succeeded to generate 1-m fused images where spectral distortion has been reduced significantly, although some block distortions appeared at the edge of the fused images. To remove this block distortion, we also proposed a sharpening process using a wavelet transform, which removed block distortion without significant change in the color of the entire image.     

基于Contourlet系数局部特征的选择性遥感图像融合算法

为了使融合后的多光谱图像在显著提高空间分辨率的同时,尽可能多地保持原始多光谱特性,提出了一种基于Contourlet变换系数局部特征的选择性遥感图像融合方法。根据多光谱和全色图像融合过程中Contourlet变换后的低频和高频部分融合目的的不同,对得到的近似和各层各方向的细节分量分别运用窗口邻域移动模板逐一计算相应区域Contourlet系数阵的不同局部特征量,然后选择适当的准则,对图像的近似和细节分量分别应用不同的策略在Contourlet系数域内进行选择性融合,通过Contourlet和亮度-色调-饱和度(IHS)逆变换得到融合的高分辨率多光谱图像。采用Landsat TM多光谱和SPOT全色图像进行的融合实验结果表明:提出的算法在显著提高空间分辨率的同时,又能很好地保持原始图像的光谱特征,并优于传统的融合方法。     

基于非亚采样Contourlet和SWT的多光谱图像和全色图像的融合算法

该文研究了多尺度几何分析工具非亚采样Contourlet变换(NSCT),提出一种新的全色图像和多光谱图像融合的方法。该方法首先对全色图像和进行过IHS变换的多光谱图像的亮度分量进行NSCT变换,对于二者的低频近似系数再进行平稳小波变换(SWT)并融合,进一步提高融合图像的空间信息量,对于高频细节系数,采用基于局部平均梯度的方法进行融合,经过逆NSCT得到融合图像。实验结果表明,该文提出的方法在保留多光谱图像的光谱信息的同时,增强了融合图像的空间细节表现能力,提高了信息量,并且优于传统的基于IHS变换、小波变换、双树复小波变换及Contourlet变换的融合方法,该方法是有效可行的。     

基于稀疏正则化结合NLS的超分辨率图像重建

为了保持高光谱(HS)超分辨率重建过程中的频谱一致性和边缘锐度,提出一种基于空间谱结合非局部相似性的超分辨率重建算法。首先,使用HS图像生成模型,采用稀疏正则化解决全色(PAN)图像和HS图像重建的病态问题求逆;然后分析了从高空间分辨率到低空间分辨率数据生成的丰度系数映射;最后利用非局部相似性,设计空间谱联合正则化项。实验结果表明,本文算法重建图像在PSNR,SSIM和FSIM方面明显高于其他优秀算法,在SAM和ERGAS方面明显低于其他优秀算法,在光谱失真方面丢失最少,仅有2%-3%,低于其他算法30%左右,且重建效果更加清晰自然。     

基于sc-NMF的高光谱图像融合

将高光谱图像与全色图像融合,所得融合数据对于后续的其它高光谱图像处理非常有帮助。区别于传统方法,针对高光谱图像特点,引入了光谱约束项,改进并建立基于光谱约束的非负矩阵分解（spectral-constrained nonnegative matrix factorization,sc-NMF）。改进后,该模型首先在光谱约束前提下,对高光谱图像进行非负矩阵分解,对分解所得基底进行增强,再重建高光谱图像。这样,所得到的融合图像在空间细节和光谱保持性均有比较好的效果。最后,进行了仿真和实际数据的实验验证,通过主观和客观的评价结果,所改进的融合方法性能较好,比传统方法更适用于高光谱图像融合。     

一种自适应模糊密度赋值的小波变换遥感图像融合算法

该文提出了一种基于模糊积分的多光谱图像和全色遥感图像融合算法。该算法首先由多光谱图像和全色图像的小波系数来获取模糊积分的信度函数,再依据小波系数的局部窗口内的方差自适应的构造模糊密度,然后由模糊积分计算融合后的小波系数,最后进行小波逆变换,得到融合图像。实验表明,采用新的基于模糊积分的融合方法,只要选择适当的模糊密度,就可以使得融合图像在提高空间细节质量的同时,相比其它融合算法能够具有更好的光谱质量。     

Super Resolution of Multispectral Images using ℓ1 Image Models and Interband Correlations

In this paper we propose a novel super-resolution based algorithm for the pansharpening of multispectral images. Within the Bayesian formulation, the proposed methodology incorporates prior knowledge on the expected characteristics of multispectral images; that is, it imposes smoothness within each band by means of the energy associated with the ℓ1 norm of vertical and horizontal first order differences of image pixel values and also takes into account the correlation among the bands of the multispectral image. The observation process is modeled using the sensor characteristics of both panchromatic and multispectral images. The method is tested on real and synthetic images, compared with other pansharpening methods, and the quality of the results assessed both qualitatively and quantitatively.     

Fusion of Multispectral and Panchromatic Images Using a Restoration-Based Method

Many remote-sensing satellites can obtain images in multispectral and panchromatic bands. By fusing low-resolution multispectral and high-resolution panchromatic images, one can obtain high-resolution multispectral images. In this paper, an image fusion algorithm based on image restoration is proposed to combine multispectral and panchromatic images. For remote-sensing satellites, the wavelength of the panchromatic band usually covers the wavelengths of the multispectral bands. This relationship between the two kinds of images is useful for fusion. In our approach, the low-resolution multispectral images are first resampled to the scale of the high-resolution panchromatic image. The relationship between these two kinds of images is then used to restore the resampled multispectral images. That is, the resampled multispectral images are modeled as the noisy blurred versions of the ideal multispectral images, and the high-resolution panchromatic image is modeled as a linear combination of the ideal multispectral images plus the observation noise. The ideal high-resolution multispectral images are then estimated based on the panchromatic and the resampled multispectral images. A closed-form solution of the fused images is derived here. Experiments show that the proposed fusion algorithm works effectively in integrating multispectral and panchromatic images.     

Synthesis of Multispectral Images to High Spatial Resolution: A Critical Review of Fusion Methods Based on Remote Sensing Physics

Our framework is the synthesis of multispectral images (MS) at higher spatial resolution, which should be as close as possible to those that would have been acquired by the corresponding sensors if they had this high resolution. This synthesis is performed with the help of a high spatial but low spectral resolution image: the panchromatic (Pan) image. The fusion of the Pan and MS images is classically referred as pan-sharpening. A fused product reaches good quality only if the characteristics and differences between input images are taken into account. Dissimilarities existing between these two data sets originate from two causes-different times and different spectral bands of acquisition. Remote sensing physics should be carefully considered while designing the fusion process. Because of the complexity of physics and the large number of unknowns, authors are led to make assumptions to drive their development. Weaknesses and strengths of each reported method are raised and confronted to these physical constraints. The conclusion of this critical survey of literature is that the choice in the assumptions for the development of a method is crucial, with the risk to drastically weaken fusion performance. It is also shown that the Amelioration de la Resolution Spatiale par Injection de Structures concept prevents from introducing spectral distortion into fused products and offers a reliable framework for further developments.     

Hyperspectral image quality based on convolutional network of multi-scale depth

Hyperspectral imagery has been widely used in military and civilian research fields such as crop yield estimation, mineral exploration, and military target detection. However, for the limited imaging equipment and the complex imaging environment of hyperspectral images, the spatial resolution of hyperspectral images is still relatively low, which limits the application of hyperspectral images. So, studying the data characteristics of hyperspectral images deeply and improving the spatial resolution of hyperspectral images is an important prerequisite for accurate interpretation and wide application of hyperspectral images. The purpose of this paper is to deal with super-resolution of the hyperspectral image quickly and accurately, and maintain the spectral characteristics of the hyperspectral image, makes the spectral separability of the substrate in the original image remains unchanged after super-resolution processing. This paper first learns the mapping relationship between the spectral difference of low-resolution hyperspectral image and the spectral difference of the corresponding high-resolution hyperspectral image based on multiple scale convolutional neural network, Thus, apply this mapping relationship to the input low-resolution hyperspectral image generally, getting the corresponding high resolution spectral difference. Constrained space by using the image of reconstructed spectral difference, this requires the low-resolution hyperspectral image generated by the reconstructed image is to be close to the input low-resolution hyperspectral image in space, so that the whole process becomes a closed circulation system where the low-resolution hyperspectral image generation of high-resolution hyperspectral images, then back to low-resolution hyperspectral images. This innovative design further enhances the super-resolution performance of the algorithm. The experimental results show that the hyperspectral image super-resolution method based on convolutional neural network improves the input image spatial information, and the super-resolution performance of the model is above 90%, which can maintain the spectral information well.