Enhancing multispectral imagery spatial resolution using optimized adaptive PCA and high-pass modulation

ABSTRACT

The pan-sharpening scheme combines high-resolution panchromatic imagery (HRPI) data and low-resolution multispectral imagery (LRMI) data to get a single merged high-resolution multispectral image (HRMI). The pan-sharpened image has extensive information that will promote the efficiency of image analysis methods. Pan-sharpening technique is considered as a pixel-level fusion scheme utilized for enhancing LRMI using HRPI while keeping LRMI spectral information. In this article, an efficient optimized integrated adaptive principal component analysis (APCA) and high-pass modulation (HPM) pan-sharpening method is proposed to get excellent spatial resolution within fused image with minimal spectral distortion. The proposed method is adjusted with multi-objective optimizationto determine the optimal window size and σfor the Gaussian low-pass filter (GLPF) and gain factor utilized for adding the high-pass details extracted from the HRPI to the LRMI principlecomponent of maximum correlation. Optimization results show that if the spatial resolution ratio of HRPI to LRMI is 0.50, then a GLPF of 5 × 5 window size and σ = 1.640 yields HRMI with low spectral distortion and high spatial quality. If the HRPI/LRMI spatial resolution ratio is 0.25, then a GLPF of 7 × 7 window size and σ = 1.686 yields HRMI with low spectral distortion and high spatial quality. Simulation tests demonstrated that the proposed optimized APCA–HPM fusion scheme gives adjustment between spectral quality and spatial quality and has small computational and memory complexity.     

Smoothing Filter-based Intensity Modulation: A spectral preserve image fusion technique for improving spatial details

Image fusion techniques are widely used to integrate a lower spatial resolution multispectral image with a higher spatial resolution panchromatic image, such as Thematic Mapper (TM) multispectral band and SPOT Panchromatic images. However, the existing techniques either cannot avoid distorting the image spectral properties or involve complicated and time-consuming frequency decomposition and re-construction processing. A simple spectral preserve fusion technique: the Smoothing Filter-based Intensity Modulation (SFIM) has thus been developed based on a simplified solar radiation and land surface reflection model. By using a ratio between a higher resolution image and its low pass filtered (with a smoothing filter) image, spatial details can be modulated to a co-registered lower resolution multispectral image without altering its spectral properties and contrast. The technique can be applied to improve spatial resolution for either colour composites or individual bands. The fidelity to spectral property and the spatial textural quality of SFIM are convincingly demonstrated by an image fusion experiment using TM and SPOT Panchromatic images of south-east Spain. The visual evaluation and statistical analysis compared with HSI and Brovey transform techniques confirmed that SFIM is a superior fusion technique for improving spatial detail of multispectral images with their spectral properties reliably preserved.     

A wavelet transform method to merge Landsat TM and SPOT panchromatic data

To take advantage of the high spectral resolution of Landsat TM images and the high spatial resolution of SPOT panchromatic images (SPOT PAN), we present a wavelet transform method to merge the two data types. In a pyramidal fashion, each TM reflective band or SPOT PAN image was decomposed into an orthogonal wavelet representation at a given coarser resolution, which consisted of a low frequency approximation image and a set of high frequency, spatially-oriented detail images. Band-by-band, the merged images were derived by performing an inverse wavelet transform using the approximation image from each TM band and detail images from SPOT PAN. The spectral and spatial features of the merged results of the wavelet methods were compared quantitatively with those of intensity-hue-saturation (IHS), principal component analysis (PCA), and the Brovey transform. It was found that multisensor data merging is a trade-off between the spectral information from a low spatial-high spectral resolution sensor and the spatial structure from a high spatial-low spectral resolution sensor. With the wavelet merging method, it is easy to control this trade-off. Experiments showed that the simultaneous best spectral and spatial quality can only be achieved with wavelet transform methods, compared with the three other approaches examined.     

基于多进制小波的多源遥感影像融合

首先介绍了遥感影像融合的一般理论和方法，然后在讨论多进制小波理论和影像特征的基础上，提出了一种基于特征的多进制小波变换的影像融合算法，该算法根据待融合影像分辨率之比来确定采用几进制小波，将待融合的高分辨率影像进行多进制小波变换，然后把高分辨影像经小波变换后获得的低频成分和低分辨率影像依据一定的关系进行相互转换，以形成新的高分辨影像的低频成分，经过多进制小波逆变换获得到融合后的影像，最大限度地利用了待融合影像的信息，防止了影像信息的丢失，通过对具体影像的清晰度和空间分辨率，融合后的影像最大限度地保留了待融合影像的光谱信息，同时提高了待融合影像的清晰度和空间分辨率，给出了SPOT全色影像与SPOT多光谱影像，SPOT全色影像与TM影像的融合结果，并与其他方法进行了比较，从而证明了本方法的优越性和自适应能力。     

Information fusion techniques for change detection from multi-temporal remote sensing images

In order to investigate the impacts of different information fusion techniques on change detection, a sequential fusion strategy combining pan-sharpening with decision level fusion is introduced into change detection from multi-temporal remotely sensed images. Generally, change map from multi-temporal remote sensing images using any single method or single kind of data source may contain a number of omission/commission errors, degrading the detection accuracy to a great extent. To take advantage of the merits of multi-resolution image and multiple information fusion schemes, the proposed procedure consists of two steps: (1) change detection from pan-sharpened images, and (2) final change detection map generation by decision level fusion. Impacts of different fusion techniques on change detection results are evaluated by unsupervised similarity metric and supervised accuracy indices. Multi-temporal QuickBird and ALOS images are used for experiments. The experimental results demonstrate the positive impacts of different fusion strategies on change detection. Especially, pan-sharpening techniques improve spatial resolution and image quality, which effectively reduces the omission errors in change detection; and decision level fusion integrates the change maps from spatially enhanced fusion datasets and can well reduce the commission errors. Therefore, the overall accuracy of change detection can be increased step by step by the proposed sequential fusion framework.     

波段自适应细节注入的高分五号与Sentinel-2遥感影像空谱融合

目的 针对当前空谱融合方法应用到高光谱图像融合时,出现的空间细节信息提升明显但光谱失真,或者光谱保真度高但空间细节信息提升不足的问题,本文提出一种波段自适应细节注入的高分五号（GF-5）高光谱图像（30 m）与Sentinel-2多光谱图像（10 m）的遥感影像空谱融合方法。方法 首先,为了解决两个多波段图像不便于直接融合的问题,提出一种波段自适应的融合策略,对多光谱图像波谱范围以外的高光谱图像波段,以相关系数为标准将待融合图像进行分组。其次,针对传统Gram-Schmidt （GS）融合方法用平均权重系数模拟低分辨率图像造成的光谱失真问题,使用最小均方误差估计计算线性拟合系数,再将拟合图像作为第1分量进行GS正变换,提升融合图像的光谱保真度。最后,为了能同时注入更多的空间细节信息,通过非下采样轮廓波变换将拟合图像、空间细节信息图像和多光谱图像的空间、光谱信息融入到重构的高空间分辨率图像中,再将其与其他GS分量一起进行逆变换,最终得到10 m分辨率的GF-5融合图像。结果 通过与当前用于高光谱图像空谱融合的典型方法比较,本文方法对于受时相影响较小的城镇区域,在提升空间分辨率的同时有较好的光谱保真度,且不会出现噪点;对于受时相变化影响大的植被密集区域,本文方法融合图像有较好的清晰度和地物细节信息,且没有噪点出现。本文方法的CC （correlation coefficient）、ERGAS （erreur relative globale adimensionnelle de synthèse）和SAM （spectral angle mapper）相比于传统GS方法分别提升8%、26%和28%,表明本文方法的光谱保真度大大提高。结论 本文方法的结果空间上没有噪点且光谱曲线与原始光谱曲线基本保持一致,是一种兼具高空间分辨率和高光谱保真度的高光谱图像融合方法。     

基于自适应加权的MODIS和SPOT影像融合研究

针对MODIS和SPOT影像融合问题，提出了一种基于自适应加权的亮度相关矩多分辨率影像融合方法。该方法首先对SPOT影像进行小波分解，将MODIS影像由RGB颜色空间变换到IHS颜色空间；然后，根据强度分量和SPOT影像低频分量加权后的局部均值和方差来定义影像亮度相关矩，以选择不同策略进行融合；最后，通过IHS逆变换和小波逆变换来得到包含更多信息和有效特征的融合影像。试验结果证明，该方法得到的融合影像，在保留地物光谱信息和提高空间分辨率上都具有很好的效果。     

Pan-sharpening via conditional invertible neural network

In the realm of conventional deep-learning-based pan-sharpening approaches, there has been an ongoing struggle to harmonize the input panchromatic (PAN) and multi-spectral (MS) images across varied channels. Existing methods have often been stymied by spectral distortion and an inadequate texture representation. To address these limitations, we present an innovative constraint-based image generation strategy tailored for the pan-sharpening task. Our method employs a multi-scale conditional invertible neural network, named PSCINN, which is capable of converting the ground truth MS image into a downscaled MS image and a latent variable, all under the guidance of the PAN image. Subsequently, the resampled latent variable, obtained from a prior distribution, and the low-resolution MS image are harnessed to predict the pan-sharpened image in an information-preserving manner, with the PAN image providing essential guidance during the reversion process. Furthermore, we meticulously architect a conditional invertible block to construct a Jacobian Determinant for the spectral information recovery. This structure effectively pre-processes the conditioning PAN image into practical texture information, thereby preventing the spectral information in the pan-sharpened result from potential contamination. The proposed PSCINN outperforms existing state-of-the-art pan-sharpening methodologies, both in terms of objective and subjective results. Post-treatment experiments underscore a substantial enhancement in the perceived quality attributed to our method. The source code for PSCINN will be accessible at https://github.com/jiaming-wang/PSCINN.     

Pan-sharpening based on common saliency feature analysis and multiscale spatial information extraction for multiple remote sensing images

ABSTRACT

The requirements of spectral and spatial quality differ from region to region in remote sensing images. The employment of saliency in pan-sharpening methods is an effective approach to fulfil this kind of demands. Common saliency feature analysis, which considers the mutual information between multiple images, can ensure the consistency and accuracy when assigning saliency to regions in different images. Thus, we propose a pan-sharpening method based on common saliency feature analysis and multiscale spatial information extraction for multiple remote sensing images. Firstly, we extract spatial information by the guided filter and accurate intensity component estimation. Then, a common saliency feature analysis method based on global contrast calculation and intensity feature extraction is designed to obtain preliminary pixel-wise saliency estimation, which is subsequently integrated with text-featured based compensation to generate adaptive injection gains. The introduction of common saliency feature analysis guarantees that the same pan-sharpening strategy will be applied to regions with similar features in multiple images. Finally, the injection gains are used to implement the detail injection. Our proposal satisfies diverse needs of spatial and spectral information for different regions in the single image and guarantees that regions with similar features in different images are treated consistently in the process of pan-sharpening. Both visual and quantitative results demonstrate that our method has better performance in guaranteeing consistency in multiple images, improving spatial quality and preserving spectral fidelity.     

A regularized pan-sharpening approach based on self-similarity and Gabor prior

In this article, we propose a new regularization-based approach for pan-sharpening based on the concepts of self-similarity and Gabor prior. The given low spatial resolution (LR) and high spectral resolution multi-spectral (MS) image is modelled as degraded and noisy version of the unknown high spatial resolution (HR) version. Since this problem is ill-posed, we use regularization to obtain the final solution. In the proposed method, we first obtain an initial HR approximation of the unknown pan-sharpened image using self-similarity and sparse representation (SR) theory. Using self-similarity, we obtain the HR patches from the given LR observation by searching for matching patches in its coarser resolution, thereby obtaining LR–HR pairs. An SR framework is used to obtain the patch pairs for which no matches are available for the patches in LR observation. The entire set of matched HR patches constitutes initial HR approximation (initial estimate) to the final pan-sharpened image which is used to estimate the degradation matrix as used in our model. A regularization framework is then used to obtain the final solution in which we propose to use a new prior which we refer as Gabor prior that extracts the bandpass details from the registered panchromatic (Pan) image. In addition, we also include Markov random field (MRF) smoothness prior that preserves the smoothness in the final pan-sharpened image. MRF parameter is derived using the initial estimate image. The final cost function consists of data fitting term and two prior terms corresponding to Gabor and MRF. Since the derived cost function is convex, simple gradient-based method is used to obtain the final solution. The efficacy of the proposed method is evaluated by conducting the experiments on degraded as well as on un-degraded datasets of three different satellites, i.e., Ikonos-2, Quickbird, and Worldview-2. The results are compared on the basis of traditional measures as well as recently proposed quality with no reference (QNR) measure, which does not require the reference image.     

自适应权重注入机制遥感图像融合

目的 遥感图像融合是将一幅高空间分辨率的全色图像和对应场景的低空间分辨率的多光谱图像,融合成一幅在光谱和空间两方面都具有高分辨率的多光谱图像。为了使融合结果在保持较高空间分辨率的同时减轻光谱失真现象,提出了自适应的权重注入机制,并针对上采样图像降质使先验信息变得不精确的问题,提出了通道梯度约束和光谱关系校正约束。方法 使用变分法处理遥感图像融合问题。考虑传感器的物理特性,使用自适应的权重注入机制向多光谱图像各波段注入不同的空间信息,以处理多光谱图像波段间的差异,避免向多光谱图像中注入过多的空间信息导致光谱失真。考虑到上采样的图像是降质的,采用局部光谱一致性约束和通道梯度约束作为先验信息的约束,基于图像退化模型,使用光谱关系校正约束更精确地保持融合结果的波段间关系。结果 在Geoeye和Pleiades卫星数据上同6种表现优异的算法进行对比实验,本文提出的模型在2个卫星数据上除了相关系数CC（correlation coefficient）和光谱角映射SAM（spectral angle mapper）评价指标表现不够稳定,偶尔为次优值外,在相对全局误差ERGAS（erreur relative globale adimensionnelle de synthèse）、峰值信噪比PSNR（peak signal-to-noise ratio）、相对平均光谱误差RASE（relative average spectral error）、均方根误差RMSE（root mean squared error）、光谱信息散度SID（spectral information divergence）等评价指标上均为最优值。结论 本文模型与对比算法相比,在空间分辨率提升和光谱保持方面都取得了良好效果。     

提升细节捕捉能力的非下采样轮廓波变换

针对传统NSCT(非下采样轮廓波变换)算法中NSP(多尺度分解方法)对细节信息捕捉能力较差及利用其进行图像融合得到的融合图像出现细节丢失问题,提出改进的NSCT算法。不同于传统NSCT算法,该算法首先采用细节捕捉能力较强的非下采样形态学小波分解替代NSP分解,实现对源图像的多尺度分解,将源图像分解成水平高频、垂直高频、对角高频和低频4部分;然后利用NDFB(非下采样的方向性滤波器)对高频部分进行多方向分解得到一系列高频信息,实现改进型NSCT分解。实验结果表明,该算法的细节捕捉能力较传统算法好,在相同融合规则下其图像融合效果更好,各项融合指标值均有所提高,其中平均梯度提高了10%,且易于实现,可广泛用于多分辨率图像融合,是一种有效的融合图像算法。     

The joint use of IHS transform and redundant wavelet decomposition for fusing multispectral and panchromatic images

Intensity hue saturation (IHS) and wavelet decomposition are two distinct fusion methods used for enhancing the spatial resolution of multispectral images by exploiting a high-resolution panchromatic image. In this paper, a combination of the IHS transform and redundant wavelet decomposition is proposed as a general method for fusing multisensor images. The principle consists of transforming low-resolution multispectral images into IHS independent components. The low-resolution intensity component is fused with the high-resolution panchromatic image in the redundant wavelet domain through an appropriate model. Subsequently, the high-resolution intensity produced is substituted to the low-resolution intensity. High spatial resolution multispectral images are then obtained through an inverse IHS transformation. SPOT images are used to illustrate the superiority of this approach over the IHS fuser in terms of preservation of spectral properties.     

Research on fundus image registration and fusion method based on nonsubsampled contourlet and adaptive pulse coupled neural network

We present a registration and fusion method of fluorescein fundus angiography image and color fundus image which combines Nonsubsampled Contourlet (NSCT) and adaptive Pulse Coupled Neural Network (PCNN). Firstly, we register two images by Speeded Up Robust Features (SURF) feature points, the nearest neighbor and the next nearest neighbor distance ratio method to eliminate the spatial difference between the source images. Secondly, we use Random Sample Consensus (RANSAC) algorithm to achieve precise matching of feature points. Then, according to the transformation parameters obtained by RANSAC algorithm, we perform spatial transformation on the floating image to complete the registration. Finally, we obtain the low-frequency sub-band and high-frequency sub-band of the image to be fused by NSCT decomposition. The low-frequency sub-band is fused by the regional energy. The high-frequency sub-bands are studied using a simplified-PCNN model and the Particle Swarm Optimization algorithm. The link strength of the simplified-PCNN is an improved Laplacian energy and the images are fused based on the number of times the pixels are ignited. The proposed method has higher average gradient (AG) value and information entropy (IE) value and lower relative global dimensional synthesis error (ERGAS) than the existing fusion methods of the fundus image. The fusion image can accurately synthesize the image information, clarify the performance of the details, and has better spectral quality in the spectral range. The image of fused provides an effective reference for the clinical diagnosis of fundus diseases.

     

Fusion of multispectral and panchromatic satellite sensor imagery based on tailored filtering in the Fourier domain

A new methodology for fusing satellite sensor imagery, based on tailored filtering in the Fourier domain is proposed. Finite‐duration Impulse Response (FIR) filters have been designed through an objective criterion, which depends on source image characteristics only. The designed filters allow a weighted fusion of the information contained in a fine spatial resolution image (PAN) and in a multispectral image (MULTI), respectively, establishing a trade‐off between spatial and spectral quality of the resulting fused image. This new technique has been tested with Landsat Enhanced Thematic Mapper Plus (ETM+) imagery. Spatial and spectral quality of the fused images was compared with the results provided by Mallat's Wavelet algorithm. The images fused by the proposed method were characterized by a spatial resolution very close to the PAN image, and by the spectral resolution of the MULTI image.     

基于改进NSCT和IHS变换的遥感影像融合方法

目的 为了增强多光谱和全色影像融合质量,提出基于脉冲耦合神经网络（PCNN）的非下采样Contoulet变换（NSCT）和IHS变换相结合的融合方法。方法 先对多光谱图像进行IHS变换提取亮度I分量,采用主成分分析增强I分量得到新的I+分量;然后通过NSCT变换分别对I+分量和全色图像进行分解,并采用边缘梯度信息激励的PCNN得到融合图像的低频和高频分量;最后进行NSCT逆变换、IHS逆变换得到融合图像。结果 利用资源一号02C卫星数据进行实验,结果表明该算法在保留光谱信息的同时提高了图像空间分辨率,获得了较好的融合效果。结论 结合NSCT和IHS变换的融合方法在视觉效果和客观评价指标上都优于常用的图像融合方法。     

小波变换在SPOT5全色与多光谱图像融合中的应用

离散小波变换可以将图像分解成为一系列具有不同分辨率特征、频率特征和方向特性的子带信号,并且将图像的光谱特征和空间特征分离,从而为不同分辨率的遥感图像融合提供了有利条件。采用小波变换融合方法对SPOT5全色与多光谱图像进行融合处理,以提高图像的空间信息质量和光谱质量为目的。通过对融合结果图像进行主观和客观的综合评价,可得出融合前后图像灰度均值变化很小,灰度标准差、熵和清晰度三者变化趋势一致;当小波分解层数为3时,融合图像的光谱质量与空间质量之间达到较好的平衡,同时融合图像的视觉效果良好。     

基于非子采样Contourlet变换的遥感图像融合算法

针对人类视觉特性, 以及全色高分辨图像和多光谱遥感自身的特点, 提出一种非下采样 Contourlet (NSCT) 域的图像融合新策略. NSCT 具有好的多分辨、移不变和多方向等特性, 能对图像中的边缘和围线信息给出渐近最优表示. 为了更好地保持空间分辨率和颜色分量, 引入基于 LHS 变换的亮度成分叠加策略. 实验结果表明: 本文提出的融合方法在提高空间分辨率的同时较好地保持了光谱信息. 与传统的 PCA 方法、基于 IHS 的融合方法、基于小波加权的融合方法, 以及同样采用本文的融合策略、分别基于小波变换和基于 Contourlet 变换的融合策略相比较, 本文方法在视觉效果和客观衡量指标两方面都有所改善.     

基于 MTF 和变分的全色与多光谱图像融合模型

Pan-sharpening将高分辨率图像全色(Panchromatic, Pan)波段的空间细节注入多光谱(Multispectral, MS)波段, 以生成同时具有高光谱和高空间分辨率的多光谱图像. 为改善融合效果, 需要考虑多光谱和全色波段的调制传输函数(Modulation transfer function, MTF). 本文提出了一个新的基于MTF和变分的Pan-sharpening模型. 该模型的能量泛函包括两项, 第1项为细节注入项, 基于高通滤波器从Pan波段中提取细节信息并注入融合图像;第2项为光谱保真项, 基于MTF设计多孔小波的低通滤波器以保持MS波段的多光谱信息. 在QuickBird、IKONOS和GeoEye数据集上的融合结果表明, 该模型可以生成同时具有高空间和高光谱质量的融合图像, 融合效果优于AWLP、IHS_BT、HPM-CC-PSF、NAWL、快速变分等算法.     

A regularised model-based pan-sharpening method for remote sensing images with local dissimilarities

Combining the spectral information of a low-resolution multispectral (LRMS) image and the spatial information of a high-resolution panchromatic (HRP) image to generate a high-resolution multispectral (HRMS) image has become an important and interesting issue. Local dissimilarities between the LRMS image and the HRP image affect the performance of the pan-sharpening technique. This paper presents a model-based pan-sharpening method with global and nonlocal spatial similarity regularisers to reduce the effects of the local dissimilarities. The degraded model relating the LRMS image to the unknown HRMS image is employed as the data-fitting term to keep spectral fidelity. Two spatial similarity constraints are utilized to further enhance the spatial resolution of the unknown HRMS image. The first regularisation term is under the assumption that the high-pass component of each HRMS band has the similar geometry structure with the adjusted high-pass component of the HRP image. A modulation matrix is constructed to reduce the contrast differences. Moreover, nonlocal self-similarity characteristic of the high-pass component extracted from each HRMS band is considered as another regulariser, which is an effective structural prior to improve the local spatial quality of the HRMS image. The weights of nonlocal similarity model are learned from the high-pass component of available HRP image. Experiments conducted on QuickBird and IKONOS data validate that the proposed pan-sharpening method can achieve better performance compared with several traditional and state-of-the-art pan-sharpening algorithms in terms of quantitative evaluation and visual analysis.