Multispectral image adaptive pansharpening based on wavelet transformation and NMDB approaches

In remote sensing, satellite images acquired from sensors provide either high spectral or high spatial resolution. The pansharpening framework is applied to remote-sensing systems to enhance the spatial quality of coarse-resolution multispectral (MS) images using information from panchromatic imagery. A multidecomposition pansharpening approach combining MS and panchromatic (PAN) images is proposed in this paper in order to bring the resolution of the low-resolution MS imagery up to that of the panchromatic images. In particular, multilevel wavelet decomposition is applied to the luminance-chrominance (YUV) space transformation (taking into account the red green and blue (RGB) bands) or extended-YUV transformation (taking into account the near infrared (NIR) band in addition to RGB) of the original MS channels, where geometrical details from the panchromatic image are introduced into the MS ones. Our approach contains a preprocessing step that consists of homogenizing the luminance, Y, and the panchromatic image reflectance, which are, respectively, a value integrated over a wavelength spectrum and simply a linear combination of some values in the same spectrum. Hence, as the panchromatic image reflectance and luminance reflectance correspond to different measurements, they do not correspond to the same physical information, which results in a difference between their histograms. Therefore, simple histogram matching is traditionally applied to panchromatic data to fit it to the luminance to avoid colour distortion after fusion. However, as the transformation concerns just the details of the panchromatic and MS images, a new scheme for matching the images which ignores the divergence between their approximations and maximizes the resemblance between their details is proposed in this work. After that, the fusion approach is applied, and in contrast to the original approach where the details of the fused MS luminance are set equal to the PAN luminance, we propose an adaptive approach in which just a part of the PAN details proportional to the similarity between the luminance and lowered PAN image is taken. Indeed, high-resolution geometrical details cannot be similar if the low-resolution details are not in good agreement. Besides, as the agreement between PAN and MS images depends on the occupation class, we have created a segmentation map and then computed separately the correlation in each region. Finally, the evaluation is done based on QuickBird and Pleiades-1A data sets showing rural and suburban areas. When compared to recent methods, our approach provides better results.     

形态学滤波和改进PCNN的NSST域多光谱与全色图像融合

目的 全色图像的空间细节信息增强和多光谱图像的光谱信息保持通常是相互矛盾的,如何能够在这对矛盾中实现最佳融合效果一直以来都是遥感图像融合领域的研究热点与难点。为了有效结合光谱信息与空间细节信息,进一步改善多光谱与全色图像的融合质量,提出一种形态学滤波和改进脉冲耦合神经网络（PCNN）的非下采样剪切波变换（NSST）域多光谱与全色图像融合方法。方法 该方法首先分别对多光谱和全色图像进行非下采样剪切波变换;对二者的低频分量采用形态学滤波和高通调制框架（HPM）进行融合,将全色图像低频子带的细节信息注入到多光谱图像低频子带中得到融合后的低频子带;对二者的高频分量则采用改进脉冲耦合神经网络的方法进行融合,进一步增强融合图像中的空间细节信息;最后通过NSST逆变换得到融合图像。结果 仿真实验表明,本文方法得到的融合图像细节信息清晰且光谱保真度高,视觉效果上优势明显,且各项评价指标与其他方法相比整体上较优。相比于5种方法中3组融合结果各指标平均值中的最优值,清晰度和空间频率分别比NSCT-PCNN方法提高0.5%和1.0%,光谱扭曲度比NSST-PCNN方法降低4.2%,相关系数比NSST-PCNN方法提高1.4%,信息熵仅比NSST-PCNN方法低0.08%。相关系数和光谱扭曲度两项指标的评价结果表明本文方法相比于其他5种方法能够更好地保持光谱信息,清晰度和空间频率两项指标的评价结果则展示了本文方法具有优于其他对比方法的空间细节注入能力,信息熵指标虽不是最优值,但与最优值非常接近。结论 分析视觉效果及各项客观评价指标可以看出,本文方法在提高融合图像空间分辨率的同时,很好地保持了光谱信息。综合来看,本文方法在主观与客观方面均具有优于亮度色调饱和度（IHS）法、主成分分析（PCA）法、基于非负矩阵分解（CNMF）、基于非下采样轮廓波变换和脉冲耦合神经网络（NSCT-PCNN）以及基于非下采样剪切波变换和脉冲耦合神经网络（NSST-PCNN）5种经典及现有流行方法的融合效果。     

改进的基于稀疏表示的全色锐化算法

为了更有效地结合高分辨率全色（PAN）图像细节信息和低分辨率多光谱（MS）图像光谱信息，提出了一种改进的全色锐化算法。首先，对低分辨率MS图像的强度通道进行下采样再上采样获取其低频成分；其次，用强度通道减去低频成分获取其高频成分，在获取到的高低频成分中进行随机采样来构建字典；然后，用构建好的过完备字典对高分辨率PAN图像进行分块分解以获取高频信息；最后，将分解出的高频信息注入到低分辨率MS图像中以重建高分辨率MS图像。经多组实验后发现，所提出的算法在主观上保留了光谱信息，并注入了大量的空间细节信息。对比结果表明，相比其他诸如基于成分替换算法、基于多分辨率分析算法、基于稀疏表示算法，所提算法重建出来的高分辨率MS图像更加清晰，且在相关系数等多种客观评价指标上优于对比算法。     

An image fusion method based on object-oriented classification

In image fusion of different spatial resolution multispectral (MS) and panchromatic (PAN) images, a spectrally mixed MS pixel superimposes multiple mixed PAN pixels and multiple pure PAN pixels. This verifies that with increased spatial resolution in imaging, a low spatial resolution spectrally mixed subpixel may be unmixed to be a pure pixel. However, spectral unmixing of mixed MS subpixels is rarely considered in current remote-sensing image fusion methods, resulting in blurred fused images. In the image fusion method proposed in this article, such spectral unmixing is realized. In this method, the MS and PAN images are jointly segmented into image objects, image objects are classified to obtain a classification map of the PAN image and each MS subpixel is fused to be a pixel matching the class of the corresponding PAN pixel. Tested on spatially degraded IKONOS MS and PAN images with a significant spatial resolution ratio of 8:1, the fusion method offered fused images with high spectral quality and deblurred visualization.     

Across-scale process similarity based interpolation for image super-resolution

A pivotal step in image super-resolution techniques is interpolation, which aims at generating high resolution images without introducing artifacts such as blurring and ringing. In this paper, we propose a technique that performs interpolation through an infusion of high frequency signal components computed by exploiting ‘process similarity’. By ‘process similarity’, we refer to the resemblance between a decomposition of the image at a resolution to the decomposition of the image at another resolution. In our approach, the decompositions generating image details and approximations are obtained through the discrete wavelet (DWT) and stationary wavelet (SWT) transforms. The complementary nature of DWT and SWT is leveraged to get the structural relation between the input image and its low resolution approximation. The structural relation is represented by optimal model parameters obtained through particle swarm optimization (PSO). Owing to process similarity, these parameters are used to generate the high resolution output image from the input image. The proposed approach is compared with six existing techniques qualitatively and in terms of PSNR, SSIM, and FSIM measures, along with computation time (CPU time). It is found that our approach is the fastest in terms of CPU time and produces comparable results.     

PAN‐sharpening of very high resolution multispectral images using genetic algorithms

A novel image fusion method is presented, suitable for sharpening of multispectral (MS) images by means of a panchromatic (PAN) observation. The method is based on redundant multiresolution analysis (MRA); the MS bands expanded to the finer scale of the PAN band are sharpened by adding the spatial details from the MRA representation of the PAN data. As a direct, unconditioned injection of PAN details gives unsatisfactory results, a new injection model is proposed that provides the optimum injection by maximizing a global quality index of the fused product. To this aim, a real‐valued genetic algorithm (GA) has been defined and tested on Quickbird data. The optimum GA injection is driven by an index function capable of measuring different types of possible distortions in the fused images. Fusion tests are carried out on spatially degraded data to objectively compare the proposed scheme to the most promising state‐of‐the‐art image fusion methods, and on full‐resolution image data to visually assess the performance of the proposed genetic image fusion method.     

基于PST相位约束和稀疏表示的MS和PAN影像融合算法

在基于多光谱(MS)影像和全色(PAN)遥感影像融合中,提高融合影像质量的一个关键问题是如何有效提取PAN影像的纹理特征信息,并有针对性地对MS影像进行信息注入.因此,文中提出基于相位拉伸变换(PST)相位约束的MS和PAN影像稀疏融合算法.首先对MS和PAN影像进行高斯滤波.对于中低频信息,基于PST相位差对影像中边缘和纹理区域的敏感性,通过高频信息PST的相位差获得融合权重约束.对于高频信息,通过学习PAN影像的高频信息获得训练字典,并利用字典对MS和PAN影像的高频信息进行稀疏表示和融合,提高融合高频信息的准确度.算法在一定程度上克服传统融合方法对边缘纹理区域融合效果较差和光谱信息扭曲等现象,取得更好的融合效果.大量仿真实验验证算法的有效性.     

灰色关联分析与模糊推理边缘检测图像融合法

为解决融合图像不同程度的光谱失真问题,提出了一种结合灰色关联分析、模糊推理和IHS变换的图像融合算法。首先通过灰色关联分析和模糊推理算出全色图像的边缘点和非边缘点,得到丰富的边缘信息,然后对多光谱图像进行IHS变换,以亮度分量为依据对全色图像进行直方图匹配,再基于边缘信息对亮度分量和直方图匹配后的全色图像进行线性加权,最后通过IHS逆变换得到融合图像。为验证本文方法的有效性,与5种常用方法比较,从视觉和定量两方面进行评价,且采用降尺度评价和全分辨率评价。结果表明,该方法得到的融合图像比其他5种方法更优越。本文方法不仅提高了遥感图像的空间分辨率,也较好保留了多光谱图像的光谱信息。     

Pan-sharpening via conditional invertible neural network

In the realm of conventional deep-learning-based pan-sharpening approaches, there has been an ongoing struggle to harmonize the input panchromatic (PAN) and multi-spectral (MS) images across varied channels. Existing methods have often been stymied by spectral distortion and an inadequate texture representation. To address these limitations, we present an innovative constraint-based image generation strategy tailored for the pan-sharpening task. Our method employs a multi-scale conditional invertible neural network, named PSCINN, which is capable of converting the ground truth MS image into a downscaled MS image and a latent variable, all under the guidance of the PAN image. Subsequently, the resampled latent variable, obtained from a prior distribution, and the low-resolution MS image are harnessed to predict the pan-sharpened image in an information-preserving manner, with the PAN image providing essential guidance during the reversion process. Furthermore, we meticulously architect a conditional invertible block to construct a Jacobian Determinant for the spectral information recovery. This structure effectively pre-processes the conditioning PAN image into practical texture information, thereby preventing the spectral information in the pan-sharpened result from potential contamination. The proposed PSCINN outperforms existing state-of-the-art pan-sharpening methodologies, both in terms of objective and subjective results. Post-treatment experiments underscore a substantial enhancement in the perceived quality attributed to our method. The source code for PSCINN will be accessible at https://github.com/jiaming-wang/PSCINN.     

Pansharpening by exploiting sharpness of the spatial structure

High-frequency injection (HFI)-based methods are proved to be powerful in pansharpening multispectral (MS) images. In this article, based on one of the low-rank and sparse (LRS) decomposition algorithms, i.e. Go Decomposition (GoDec), a HFI-based pansharpening method exploiting spatial structure sharpness of both MS images and a low-frequency panchromatic (PAN) image component is proposed. The spectral and spatial measure of local perceived sharpness (S₃) is employed to estimate sharpness of the corresponding MS and low-frequency PAN component blocks, and the sharper one is used to construct the spatial structure of the sharpened MS images. Experimental results with QuickBird, IKONOS and WorldView-2 data demonstrate that the proposed method is comparable with or even better than other popular methods.     

Two improvement schemes of PAN modulation fusion methods for spectral distortion minimization

Fusion of panchromatic (PAN) and multispectral (MS) images is one of the most promising issues in remote sensing. PAN modulation fusion methods are usually based on an assumption that a ratio of two different‐resolution versions of an MS band is equal to a ratio of two different‐resolution versions of a PAN image. In such fusion methods, image haze is rarely taken into account, and it may produce serious spectral distortion in synthetic images. In this paper, assuming that the previous ratio relationship only holds for haze‐free images, two relevant improvement schemes are proposed to better express the ratio relationship of haze‐included images. In a test on a spatially degraded IKONOS dataset, the first scheme synthesizes an image with minimum spectral distortion, and the second modifies several current PAN modulation fusion methods and generates high‐quality synthetic products. The experiment results confirm that image haze can seriously impact the quality of fused images obtained by using PAN modulation fusion methods, and it should be taken into account in relevant image fusion.     

A genetic algorithm solution to the gram-schmidt image fusion

ABSTRACT

There is no such thing as ‘the best image fusion method’ in terms of both spectral and spatial fidelity. This fact encourages the researchers to develop more advanced approaches in order to optimally transfer the spatial details without distorting the colour content. Component substitution (CS)-based image fusion methods have been proven to produce sharper images but suffer from colour distortion. The aim of this study was to modify the CS-based Gram-Schmidt (GS) fusion method with the aid of the Genetic Algorithm (GA) to further improve its colour preservation performance. The GA was used to estimate a weight for each multispectral (MS) band. The obtained band weights were used to generate a low-resolution panchromatic (PAN) band, which plays a significant role in the performance of the GS method. The performance of the proposed approach was compared not only against the conventional GS, but also against widely-used CS-based, multiresolution analysis (MRA)-based and colour-based (CB) image fusion methods. The results indicated that the proposed GA-based approach produced spectrally and spatially superior results compared to the other methods used.     

Fusion of multispectral and panchromatic images based on support value transform and adaptive principal component analysis

In this paper we combined the projection-substitution with ARSIS (French acronym for “Amélioration de la Résolution Spatiale par Injection de Structures”, i.e., Improving Spatial Resolution by Structure Injection) concept assumption for fusion of panchromatic (PAN) and multispectral (MS) images. Firstly support value filter (SVF) is used to establish a new multiscale model (MSM), support vector transform (SVT), and adaptive principal component analysis (APCA) is then employed to select the principal components of MS images by means of a statistical measure of the correlation between MS and PAN images; secondly, a local approach is used to check whether a structure should appear in the new principal component and PAN high frequency structures are transformed by high resolution interband structure model (HRIBSM) before inserting in the MS modalities. Because SVT is an undecimated, dyadic and aliasing transform with shift-invariant property, the fused image can avoid ringing effects suffered from sampling. Additionally, the ARSIS concept can make full use of the remote sensing physics to reduce the spatial and spectrum distortion in the structure injection. Texture extraction is also employed to avoid the spectral distortion caused by the mistaken injection of low-pass components into the MS images. Experimental results including visual and numerical evaluation also proves the superiority of the proposed method to its counterparts.     

Multispectral and multiresolution image fusion using particle swarm optimization

Multispectral and multiresolution image fusion is important for many multimedia and remote sensing applications, such as video surveillance, medical imaging, and satellite imaging. For the commercial satellite ??IKONOS,?? spatial resolutions of high-resolution panchromatic (PAN) and low-resolution multispectral (MS) satellite images are 1?m and 4?m, respectively. To cope with color distortion and blocking artifacts in fused images, in this study, a multispectral and multiresolution image fusion approach using PSO is proposed. The pixels of fused images in the training set are classified into several categories based on the characteristics of low-resolution MS images. Then, the smooth parameters of spatial and spectral responses between the high-resolution PAN and low-resolution MS images are determined by PSO. All the pixels within each category are normalized by its own smooth parameter so that color distortion and blocking artifacts can be greatly reduced. Based on the experimental results obtained in this study, the overall visual quality of the fused images by the proposed approach is better than that by three comparison approaches, whereas the correlation coefficients, ?? _PAN , for the fused images by the proposed approach are greater than that by three comparison approaches.     

Remote sensing image fusion based on Bayesian linear estimation

A new remote sensing image fusion method based on statistical parameter estimation is proposed in this paper. More specially, Bayesian linear estimation (BLE) is applied to observation models between remote sensing images with different spa- tial and spectral resolutions. The proposed method only estimates the mean vector and covariance matrix of the high-resolution multispectral (MS) images, instead of assuming the joint distribution between the panchromatic (PAN) image and low-resolution multispectral image. Furthermore, the proposed method can enhance the spatial resolution of several principal components of MS images, while the traditional Principal Component Analysis (PCA) method is limited to enhance only the first principal component. Experimental results with real MS images and PAN image of Landsat ETM demonstrate that the proposed method performs better than traditional methods based on statistical parameter estimation, PCA-based method and wavelet-based method.     

Pan-sharpening using a guided filter

Pan-sharpening aims to integrate the spatial details of a high-resolution panchromatic (Pan) image with the spectral information of low-resolution multispectral (MS) images to produce high-resolution MS images. The key is to appropriately estimate the missing spatial details of the MS images while preserving their spectral contents. However, many existing methods extract the spatial details from the Pan image without fully considering the structures of the MS images, resulting in spectral distortion due to redundant detail injection. A guided filter can transfer the structures of the MS images into the intensity component or the low-pass approximation of the Pan image. Using the guided filter, we propose two novel pan-sharpening methods to reduce the redundant details among the MS and Pan images. Specifically, we extract the missing spatial details of the MS images by minimizing the difference between the Pan image and its corresponding filtering output, with the help of the MS images. Two different ways of using the MS images as guided images lead to two proposed methods, which can be grouped into component substitution (CS) family. Extensive experimental results over three data sets collected by different satellite sensors demonstrate the effectiveness of the proposed methods.     

MTF driven adaptive multiscale bilateral filtering for pansharpening

The recently proposed Bilateral Filter Luminance Proportional (BFLP) method extracts the high-frequency details from panchromatic (Pan) image via a multiscale bilateral filter and adds them proportionally to the multispectral (MS) image. Although this approach seems similar to other multiresolution (MRA) based schemes such as Additive Wavelet proportional Luminance (AWLP) or Generalized Laplacian (GLP) methods, multiscale bilateral filter obtains the detail planes to be injected to MS image by the combination of two Gaussian kernels controlling the transfer of details and performing successively in spatial and range domains, thus it has two parameters to be defined, namely spatial and range parameters. Since the parameter determination step considerably affects the efficiency of the method, in this paper we propose a single parameter bilateral filter by approximating the Gaussian kernel with the bicubic kernel of à trous wavelet transform (ATWT) or modulation transfer function (MTF). Moreover, we adopt an adaptive injection scheme where the range parameter is determined adaptively so as to follow the statistics of the images to be fused. The pansharpening results are compared with ATWT-based methods, as well as some state-of-the-art methods and BFLP. The visual and quantitative comparisons for Système Pour l’Observation de la Terre 7 (SPOT 7) and Pléiades 1A images, field studies supported with UAV (Unmanned Aerial Vehicle) images and digitization results of the chosen areas in Istanbul Technical University (ITU) Maslak campus confirm the superiority of the proposed detail injection approach.     

基于 MTF 和变分的全色与多光谱图像融合模型

Pan-sharpening将高分辨率图像全色(Panchromatic, Pan)波段的空间细节注入多光谱(Multispectral, MS)波段, 以生成同时具有高光谱和高空间分辨率的多光谱图像. 为改善融合效果, 需要考虑多光谱和全色波段的调制传输函数(Modulation transfer function, MTF). 本文提出了一个新的基于MTF和变分的Pan-sharpening模型. 该模型的能量泛函包括两项, 第1项为细节注入项, 基于高通滤波器从Pan波段中提取细节信息并注入融合图像;第2项为光谱保真项, 基于MTF设计多孔小波的低通滤波器以保持MS波段的多光谱信息. 在QuickBird、IKONOS和GeoEye数据集上的融合结果表明, 该模型可以生成同时具有高空间和高光谱质量的融合图像, 融合效果优于AWLP、IHS_BT、HPM-CC-PSF、NAWL、快速变分等算法.     

Querying color images using user-specified wavelet features

In this paper, an image retrieval method based on wavelet features is proposed. Due to the superiority in multiresolution analysis and spatial-frequency localization, the discrete wavelet transform (DWT) is used to extract wavelet features (i.e., approximations, horizontal details, vertical details, and diagonal details) at each resolution level. During the feature-extraction process, each image is first transformed from the standard RGB color space to the YUV space for the purpose of efficiency and ease of extracting the features based on color tones; then each component (i.e., Y, U, and V) of the image is further transformed to the wavelet domain. In the image database establishing phase, the wavelet coefficients of each image are stored; in the image retrieving phase, the system compares the wavelet coefficients of the Y, U, and V components of the query image with those of the images in the database, based on the weight factors adjusted by users, and find out good matches. To benefit from the user–machine interaction, a friendly graphic user interface (GUI) for fuzzy cognition is developed, allowing users to easily adjust weights for each feature according to their preferences. In our experiment, 1000 test images are used to demonstrate the effectiveness of our system. Te-Wei Chiang received the B.S. degree in electrical engineering from Tatung Institute of Technology, Taiwan, in 1990, and the Ph.D. degree in computer science from the National Taiwan University in 1997. From 1999 to 2000, he was a postdoctoral fellow in the Institute of Information Science, Academia Sinica, Taiwan. He is an assistant professor in the Department of Accounting Information Systems, Chihlee Institute of Technology, Taiwan. His current research interests include pattern recognition, optical character recognition, content-based image retrieval, and information retrieval. Tienwei Tsai received the M.S. degree in computer engineering from Pennsylvania State University and the Ph.D. degree in computer engineering from Tatung University, Taiwan. He is an associate professor in the Department of Information Management at Chihlee Institute of Technology, Taiwan. His current research interests include image retrieval, information retrieval, data mining, and intelligent systems.     

Multi-spectral and panchromatic images fusion based on PCA and fractional spline wavelet

A fusion method of panchromatic and multi-spectral (MS) images based on fractional spline wavelet and principal component analysis (PCA) transformation is proposed. The family of fractional spline wavelets outperforms traditional wavelets in obtaining the texture information of images due to its fractional order of approximation, which is beneficial for image fusion. Fusion experiments taking the instance of two image data sets collected by Landsat-TM and IKONOS are carried out in this article. The mean universal quality indices (MUQs) of the three red, green, blue (RGB) channels of the proposed method are 0.8866 and 0.9259 compared to the traditional PCA fusion method with MUQ = 0.8279 and 0.8496. From the visual evaluation and numerical analysis, it can be confirmed that the proposed method is a superior technique for enhancing the spatial details of MS images with their spectral properties reliably preserved.