Spatio-spectral fusion of satellite images based on dictionary-pair learning

This paper proposes a novel spatial and spectral fusion method for satellite multispectral and hyperspectral (or high-spectral) images based on dictionary-pair learning. By combining the spectral information from sensors with low spatial resolution but high spectral resolution (LSHS) and the spatial information from sensors with high spatial resolution but low spectral resolution (HSLS), this method aims to generate fused data with both high spatial and spectral resolution. Based on the sparse non-negative matrix factorization technique, this method first extracts spectral bases of LSHS and HSLS images by making full use of the rich spectral information in LSHS data. The spectral bases of these two categories data then formulate a dictionary-pair due to their correspondence in representing each pixel spectra of LSHS data and HSLS data, respectively. Subsequently, the LSHS image is spatial unmixed by representing the HSLS image with respect to the corresponding learned dictionary to derive its representation coefficients. Combining the spectral bases of LSHS data and the representation coefficients of HSLS data, fused data are finally derived which are characterized by the spectral resolution of LSHS data and the spatial resolution of HSLS data. The experiments are carried out by comparing the proposed method with two representative methods on both simulation data and actual satellite images, including the fusion of Landsat/ETM+ and Aqua/MODIS data and the fusion of EO-1/Hyperion and SPOT5/HRG multispectral images. By visually comparing the fusion results and quantitatively evaluating them in term of several measurement indices, it can be concluded that the proposed method is effective in preserving both the spectral information and spatial details and performs better than the comparison approaches.     

波段自适应细节注入的高分五号与Sentinel-2遥感影像空谱融合

目的 针对当前空谱融合方法应用到高光谱图像融合时,出现的空间细节信息提升明显但光谱失真,或者光谱保真度高但空间细节信息提升不足的问题,本文提出一种波段自适应细节注入的高分五号（GF-5）高光谱图像（30 m）与Sentinel-2多光谱图像（10 m）的遥感影像空谱融合方法。方法 首先,为了解决两个多波段图像不便于直接融合的问题,提出一种波段自适应的融合策略,对多光谱图像波谱范围以外的高光谱图像波段,以相关系数为标准将待融合图像进行分组。其次,针对传统Gram-Schmidt （GS）融合方法用平均权重系数模拟低分辨率图像造成的光谱失真问题,使用最小均方误差估计计算线性拟合系数,再将拟合图像作为第1分量进行GS正变换,提升融合图像的光谱保真度。最后,为了能同时注入更多的空间细节信息,通过非下采样轮廓波变换将拟合图像、空间细节信息图像和多光谱图像的空间、光谱信息融入到重构的高空间分辨率图像中,再将其与其他GS分量一起进行逆变换,最终得到10 m分辨率的GF-5融合图像。结果 通过与当前用于高光谱图像空谱融合的典型方法比较,本文方法对于受时相影响较小的城镇区域,在提升空间分辨率的同时有较好的光谱保真度,且不会出现噪点;对于受时相变化影响大的植被密集区域,本文方法融合图像有较好的清晰度和地物细节信息,且没有噪点出现。本文方法的CC （correlation coefficient）、ERGAS （erreur relative globale adimensionnelle de synthèse）和SAM （spectral angle mapper）相比于传统GS方法分别提升8%、26%和28%,表明本文方法的光谱保真度大大提高。结论 本文方法的结果空间上没有噪点且光谱曲线与原始光谱曲线基本保持一致,是一种兼具高空间分辨率和高光谱保真度的高光谱图像融合方法。     

Multispectral images fusion by a joint multidirectional and multiresolution representation

This paper proposes a new fusion method that permits an adequate selection of information extracted from source images to obtain fused images with good spatial and spectral quality simultaneously. This method is based on a joint multiresolution multidirectional representation of the source images using a single directional spatial frequency low pass filter bank of low computational complexity, defined in the Fourier domain. The source images correspond to those captured by the IKONOS satellite (panchromatic and multispectral). The results obtained indicate that the proposed method provides, in a simple manner, objective control over the trade‐off between high spatial and spectral quality of the fused images.     

Combining the spectral PCA and spatial PCA fusion methods by an optimal filter

High correlation among the neighboring pixels, both spectrally and spatially in a multispectral image makes it indispensable to use relevant data transformation approaches, before performing image fusion. The principal component analysis (PCA) method has been a popular choice for the spectral transformation. To propose a new consistent data transformation method in spatial domain, this paper applies the PCA transform to the spatial information of the neighboring pixels. Owing to the fact that the coefficients of PCA are obtained from statistical properties of data, they are adaptive and robust. Then, a new hybrid algorithm is proposed combining the spectral PCA and spatial PCA methods, by an optimal filter to make the synthesized result more similar to what the corresponding multisensors would observe at the high-resolution level. The evaluation of the pan-sharpened images, using global validation indexes, reveals that the proposed approach improves the fusion quality compared with six state of the art fusion methods.     

基于多通道空间光谱全变差的衍射光谱图像复原算法

在衍射成像光谱仪成像过程中,准焦谱段成像会受到其他离焦谱段的干扰而产生模糊.现有的重构算法只利用了图像空间信息,并且对于此类不适定反问题的复原效果不佳.因此,提出了一种基于多通道空间光谱全变差的正则化方法来重构衍射光谱图像.首先根据衍射光谱成像原理构建退化光谱图像的观测模型,然后在最大后验概率框架下结合空间和光谱先验信息建立复原模型.该方法充分利用衍射光谱图像的局部空间平滑性和局部光谱平滑性,并使用交替方向乘子法对模型进行有效的优化.大量实验表明,与其他的衍射光谱图像重构方法相比,此复原模型在平均峰值信噪比、平均结构相似度、平均光谱角距离和视觉质量方面都具有一定的优越性.此外,对于多通道模糊重叠且受噪声干扰的病态问题,该模型能够在保证求解速度的情况下抑制噪声,保留边缘信息,减缓锯齿状光谱失真的情况.     

Hyperspectral image super-resolution with spectral–spatial network

ABSTRACT

The super-resolution problem for hyperspectral images is currently one of the most challenging topics in remote sensing. Increasingly effective methods have been presented to solve this ill-posed problem under certain circumstances. In this article, we propose a new approach named the spectral–spatial network (SSN), which can effectively increase spatial resolution while keeping spectral information. The SSN consists of two sections: a spatial section and a spectral section that contribute to enhancing spatial resolution and preserving spectral information, respectively. The spatial section is proposed to learn end-to-end mapping between single-band images, from low-resolution and high-resolution hyperspectral images. In this section, we enhance the traditional sub-pixel convolutional layer by adding a maximum variance principle that can realize nonlinear fitting through piecewise linearization. The spectral section aims to fine-tune spectral caves to keep the spectral signature with a spectral angle error loss function. In order to make the SSN converge quickly, we also develop a corresponding three-step training method. The experimental results on two databases, with both indoor and outdoor scenes, show that our proposed method performs better than the existing state-of-the-art methods.     

Remote-sensing image fusion using sparse representation with sub-dictionaries

Remote-sensing image fusion aims to obtain a multispectral (MS) image with a high spatial resolution, which integrates spatial information from the panchromatic (Pan) image and with spectral information from the MS image. Sparse representation (SR) has been recently used in remote-sensing image fusion method, and can obtain superior results to many traditional methods. However, the main obstacle is that the dictionary is generated from high resolution MS images (HRMS), which are difficult to acquire. In this article, a new SR-based remote-sensing image fusion method with sub-dictionaries is proposed. The image fusion problem is transformed into a restoration problem under the observation model with the sparsity constraint, so the fused HRMS image can then be reconstructed by a trained dictionary. The proposed dictionary for image fusion is composed of several sub-dictionaries, each of which is constructed from a source Pan image and its corresponding MS images. Therefore, the dictionary can be constructed without other HRMS images. The fusion results from QuickBird and IKONOS remote-sensing images demonstrate that the proposed method gives higher spatial resolution and less spectral distortion compared with other widely used and the state-of-the-art remote-sensing image fusion methods.     

A probabilistic SVM approach for hyperspectral image classification using spectral and texture features

New hyperspectral sensors can collect a large number of spectral bands, which provide a capability to distinguish various objects and materials on the earth. However, the accurate classification of these images is still a big challenge. Previous studies demonstrate the effectiveness of combination of spectral data and spatial information for better classification of hyperspectral images. In this article, this approach is followed to propose a novel three-step spectral–spatial method for classification of hyperspectral images. In the first step, Gabor filters are applied for texture feature extraction. In the second step, spectral and texture features are separately classified by a probabilistic Support Vector Machine (SVM) pixel-wise classifier to estimate per-pixel probability. Therefore, two probabilities are obtained for each pixel of the image. In the third step, the total probability is calculated by a linear combination of the previous probabilities on which a control parameter determines the efficacy of each one. As a result, one pixel is assigned to one class which has the highest total probability. This method is performed in multivariate analysis framework (MAF) on which one pixel is represented by a d-dimensional vector, d is the number of spectral or texture features, and in functional data analysis (FDA) on which one pixel is considered as a continuous function. The proposed method is evaluated with different training samples on two hyperspectral data. The combination parameter is experimentally obtained for each hyperspectral data set as well as for each training samples. This parameter adjusts the efficacy of the spectral versus texture information in various areas such as forest, agricultural or urban area to get the best classification accuracy. Experimental results show high performance of the proposed method for hyperspectral image classification. In addition, these results confirm that the proposed method achieves better results in FDA than in MAF. Comparison with some state-of-the-art spectral–spatial classification methods demonstrates that the proposed method can significantly improve classification accuracies.     

Integrated fusion framework based on semicoupled sparse tensor factorization for spatio-temporal–spectral fusion of remote sensing images

Remote sensing image fusion is considered a cost effective method for handling the tradeoff between the spatial, temporal and spectral resolutions of current satellite systems. However, most current fusion methods concentrate on fusing images in two domains among the spatial, temporal and spectral domains, and a few efforts have been made to comprehensively explore the relationships of spatio-temporal–spectral features. In this study, we propose a novel integrated spatio-temporal–spectral fusion framework based on semicoupled sparse tensor factorization to generate synthesized frequent high-spectral and high-spatial resolution images by blending multisource observations. Specifically, the proposed method regards the desired high spatio-temporal–spectral resolution images as a four-dimensional tensor and formulates the integrated fusion problem as the estimation of the core tensor and the dictionary along each mode. The high-spectral correlation across the spectral domain and the high self-similarity (redundancy) features in the spatial and temporal domains are jointly exploited using the low dimensional and sparse core tensors. In addition, assuming that the sparse coefficients in the core tensors across the observed and desired image spaces are not strictly the same, we formulate the estimation of the core tensor and the dictionaries as a semicoupled sparse tensor factorization of available heterogeneous spatial, spectral and temporal remote sensing observations. Finally, the proposed method can exploit the multicomplementary spatial, temporal and spectral information of any combination of remote sensing data based on this single unified model. Experiments on multiple data types, including spatio-spectral, spatio-temporal, and spatio-temporal–spectral data fusion, demonstrate the effectiveness and efficiency of the proposed method.     

Fusion of multispectral and panchromatic satellite sensor imagery based on tailored filtering in the Fourier domain

A new methodology for fusing satellite sensor imagery, based on tailored filtering in the Fourier domain is proposed. Finite‐duration Impulse Response (FIR) filters have been designed through an objective criterion, which depends on source image characteristics only. The designed filters allow a weighted fusion of the information contained in a fine spatial resolution image (PAN) and in a multispectral image (MULTI), respectively, establishing a trade‐off between spatial and spectral quality of the resulting fused image. This new technique has been tested with Landsat Enhanced Thematic Mapper Plus (ETM+) imagery. Spatial and spectral quality of the fused images was compared with the results provided by Mallat's Wavelet algorithm. The images fused by the proposed method were characterized by a spatial resolution very close to the PAN image, and by the spectral resolution of the MULTI image.     

Hyperspectral image classification using a spectral–spatial random walker method

This article proposes a spectral–spatial method for classification of hyperspectral images (HSIs) by modifying traditional random walker (RW). The proposed method consists of suggesting two main modifications. First, to construct a spatial edge weighting function, low-frequency edge weighting function is proposed. In this function, the detail weights are removed. Second, to enhance the classification accuracy, a fusion of spectral and spatial Laplacian matrix in RW is suggested. This fusion can improve the classification performances compared to traditional RW using only spatial Laplacian matrix. In comparison with some of the state-of-the-art RW and spectral–spatial classifier methods, the experimental results of the proposed method (spectral–spatial RW) show that the proposed method significantly increases the classification accuracy of HSI.     

Multi spectral image fusion by deep convolutional neural network and new spectral loss function

In this paper we propose a new multispectral image fusion architecture. The proposed method includes two steps related to two neural networks. First the extracted spatial information, from panchromatic (Pan) image, is injected to upsampled multi-spectral (MS) image. In this step, the method employed a deep convolution neural network (DCNN) to estimate the spatial information of the MS image, according to multi-resolution analysis (MRA) scheme. This DCNN is trained by the low-spatial resolution version of Pan as an input, and by the spatial information as the target. This trained DCNN is called ‘Fusion network (FN)’. The FN, adaptively, estimates the spatial information of the MS images, and operates as an injection gain in the MRA scheme. In the second step, the spectral compensation is performed on the fused MS image. For this purpose, we used a novel loss function for this DCNN, to reduce the spectral distortion in the fused images, and simultaneously maintain the spatial information. This network is called ‘Spectral compensation network (SCN)’. Finally, the proposed method is compared to the several state-of-the-art methods on three datasets, using both full-reference and reduced reference criterion. The experimental results show that the proposed method can achieve competitive performance in both spatial and spectral information.     

PoNet: A universal physical optimization-based spectral super-resolution network for arbitrary multispectral images

Spectral super-resolution is a very important technique to obtain hyperspectral images from only multispectral images, which can effectively solve the high acquisition cost and low spatial resolution of hyperspectral images. However, in practice, multispectral channels or images captured by the same sensor are often with different spatial resolutions, which brings a severe challenge to spectral super-resolution. This paper proposed a universal spectral super-resolution network based on physical optimization unfolding for arbitrary multispectral images, including single-resolution and cross-scale multispectral images. Furthermore, two new strategies are proposed to make full use of the spectral information, namely, cross-dimensional channel attention and cross-depth feature fusion. Experimental results on five data sets show superiority and stability of PoNet addressing any spectral super-resolution situations.     

A spatial–spectral kernel-based approach for the classification of remote-sensing images

Classification of remotely sensed images with very high spatial resolution is investigated. The proposed method deals with the joint use of the spatial and the spectral information provided by the remote-sensing images. A definition of an adaptive neighborhood system is considered. Based on morphological area filtering, the spatial information associated with each pixel is modeled as the set of connected pixels with an identical gray value (flat zone) to which the pixel belongs: The pixel's neighborhood is characterized by the vector median value of the corresponding flat zone. The spectral information is the original pixel's value, be it a scalar or a vector value. Using kernel methods, the spatial and spectral information are jointly used for the classification through a support vector machine formulation. Experiments on hyperspectral and panchromatic images are presented and show a significant increase in classification accuracies for peri-urban area: For instance, with the first data set, the overall accuracy is increased from 80% with a conventional support vectors machines classifier to 86% with the proposed approach. Comparisons with other contextual methods show that the method is competitive.     

A novel adaptive fast IHS transform fusion method driven by regional spectral characteristics for Gaofen-2 imagery

ABSTRACT

With the increasing diversity of applications based on the Gaofen-2 satellite imagery, broadly applicable methods to generate high quality fused images is a significant problem to investigate. To obtain an image with high spatial and spectral resolutions from given panchromatic (Pan) images and multispectral (MS) images, most existing fusion algorithms adopt a unified strategy for the whole image. However, regions have distinct characteristics that impact the spatial and spectral resolution processing, on account of their varying regional features. In this article, to satisfy the diverse needs of different regions, a novel fast IHS (Intensity-Hue-Saturation) transform fusion method driven by regional spectral characteristics is proposed to fuse Gaofen-2 imagery. First, by the fast IHS transform framework, the original intensity component is obtained from the upsampled MS imagery. Then, numerous independent regions of upsampled MS imagery are generated by a novel superpixel merging strategy, and the spectral characteristics of these regions are utilized for generating a fusion factor. Next, to acquire a new fused intensity component, the fusion factor is applied to guide the injection of details in the fusion procedure. This fusion factor adapts the method to meet the spatial and spectral resolution needs for each region. Finally, the difference between the new fused intensity component and the original one is regarded as the detail that needs to be injected; these are added equally to the different bands of the upsampled MS imagery to yield the final fused multispectral image. In comparison with other classical algorithms, the visual and statistical analysis reveal that our proposed method can provide better results in improving spatial detail and preserving spectral information.     

Covariance intersection based image fusion technique with application to pansharpening in remote sensing

Image fusion of multi-spectral images and panchromatic images has been widely applied to imaging sensors. Multi-spectral images are rich in spectral information whereas panchromatic images have relatively higher spatial resolution. In this paper, we consider the image fusion as an estimation problem, that is to estimate the ideal scene of multi-spectral images at the resolution of panchromatic images. We propose a method of combining the covariance intersection (CI) principle with the expectation maximization (EM) algorithm to develop a novel image fusion approach. In contrast to other fusion methods, the proposed scheme takes cross-correlation among data sources into account, and thus provides consistent and accurate estimates through convex combinations. Since the covariance information is usually unknown in practice, the EM method is employed to provide a maximum likelihood estimate (MLE) of the covariance matrix. Real multi-spectral and panchromatic images are used to evaluate the effectiveness of the proposed EM-CI method. The proposed algorithm is found to preserve both the spectral information of the multi-spectral image and the high spatial resolution information of the panchromatic image more effectively than the conventional image fusion techniques.     

Spatial–spectral ant colony optimization for hyperspectral image classification

Hyperspectral satellite images contain a lot of information in terms of spectral behaviour of objects and this information can be extracted by several mechanisms including image classification. Traditional spectral information-based methods of hyperspectral image classification are generally followed by spatial information-driven post-processing techniques such as relaxation labelling and Markov Random Field. Spectral or spatial information alone may lead to different results depending upon scene captured. An algorithm which can incorporate influence of both spectral and spatial features is needed to address this problem. In this article, an ant colony optimisation-based hyperspectral image classification technique is proposed. This method exploits both spatial and spectral features. Five standard hyperspectral data sets have been used to validate the proposed method and comparisons with other approaches have been carried out. It was observed that the proposed method yielded a significant improvement in classification accuracy. For the instance, nearly 10% increase in accuracy was observed when compared to Support Vector Machine for Indian pines, Botswana, and Salinas images.     

An image fusion method based on object-oriented classification

In image fusion of different spatial resolution multispectral (MS) and panchromatic (PAN) images, a spectrally mixed MS pixel superimposes multiple mixed PAN pixels and multiple pure PAN pixels. This verifies that with increased spatial resolution in imaging, a low spatial resolution spectrally mixed subpixel may be unmixed to be a pure pixel. However, spectral unmixing of mixed MS subpixels is rarely considered in current remote-sensing image fusion methods, resulting in blurred fused images. In the image fusion method proposed in this article, such spectral unmixing is realized. In this method, the MS and PAN images are jointly segmented into image objects, image objects are classified to obtain a classification map of the PAN image and each MS subpixel is fused to be a pixel matching the class of the corresponding PAN pixel. Tested on spatially degraded IKONOS MS and PAN images with a significant spatial resolution ratio of 8:1, the fusion method offered fused images with high spectral quality and deblurred visualization.     

Pansharpening by exploiting sharpness of the spatial structure

High-frequency injection (HFI)-based methods are proved to be powerful in pansharpening multispectral (MS) images. In this article, based on one of the low-rank and sparse (LRS) decomposition algorithms, i.e. Go Decomposition (GoDec), a HFI-based pansharpening method exploiting spatial structure sharpness of both MS images and a low-frequency panchromatic (PAN) image component is proposed. The spectral and spatial measure of local perceived sharpness (S₃) is employed to estimate sharpness of the corresponding MS and low-frequency PAN component blocks, and the sharper one is used to construct the spatial structure of the sharpened MS images. Experimental results with QuickBird, IKONOS and WorldView-2 data demonstrate that the proposed method is comparable with or even better than other popular methods.     

Improving forest type classification using the vegetation local difference index

A spatial feature extraction method was applied to increase the accuracy of land-cover classification of forest type information extraction. Traditional spatial feature extraction applications use high-resolution images. However, improving the classification accuracy is difficult when using medium-resolution images, such as a 30 m resolution Enhanced Thematic Mapper Plus (ETM+) image. In this study, we demonstrated a novel method that used the vegetation local difference index (VLDI) derived from the normalized difference vegetation index (NDVI), which were calculated based on the topographically corrected ETM+ image, to delineate spatial features. A simple maximum likelihood classifier and two different ways to use spatial information were introduced in this study as the frameworks to incorporate both spectral and spatial information for analysis. The results of the experiments, where Landsat ETM+ and digital elevation model (DEM) images, together with ground truth data acquired in the study area were used, show that combining the spatial information extracted from medium-resolution images and spectral information improved both classification accuracy and visual qualities. Moreover, the use of spatial information extracted through the proposed method greatly improved the classification performance of particular forest types, such as sparse woodlands.