遥感图像全色锐化的卷积神经网络方法研究进展

随着计算机科学、遥感科学和大数据科学等领域的迅速发展，基于卷积神经网络的方法在图像处理、计算机视觉等任务上发挥着越来越重要的作用。而在遥感图像全色锐化领域，卷积神经网络由于其优秀的融合效果，已得到研究学者的广泛关注并有大量的研究成果产生。尽管如此，依然有一些亟待解决的问题，例如缺乏全色锐化数据集的仿真细节描述、公平公开的训练—测试数据集、简单易懂的统一代码编写框架等。对此，本文主要从以下几方面回顾当前遥感图像全色锐化问题在卷积神经网络方面的一些进展，并针对前述问题发布相关数据集和代码编写框架。1)详细介绍7种典型的基于卷积神经网络的全色锐化方法，并在统一数据集上进行公平比较(包括与典型传统方法的比较);2)详细介绍训练—测试数据集的仿真细节，并发布相关卫星(如WorldView-3, QuickBird, GaoFen2,WorldView-2)的全色锐化训练—测试数据集；3)针对本文介绍的7种基于卷积神经网络的方法，发布基于Pytorch深度学习库的Python代码统一编写框架，便于后来初学者入门、开展研究以及公平比较；4)发布统一的全色锐化传统—深度学习方法MATLAB测试软件包，...

CNN-based remote sensing pan-sharpening: a critical review

The aim of pan-sharpening is focused on data acquisition and processing captured from multiple remote sensing satellites in terms of machine learning and signal processing techniques. To produce a high-quality multispectral image, its objective is oriented to fuse a low spatial resolution-based multispectral (MS) image and a high spatial resolution-based panchromatic (PAN) image. Machine learning (ML) technique is being developed to deal with it. Pan-sharpening, a recent multispectral image-based image fusion technique, has been concerned about in terms of machine learning, which is featured by its medium/low spatial resolution. In recent years, remote sensing (RS) science, big data science and convolutional neural networks based (CNNs-based) techniques (a sub-category of ML), have promoted image processing, computer vision and its contexts. Thanks to the priority of RS images-based pan-sharpening of fusion, CNN-oriented researches have been developing dramatically, but some challenging issues are required to be handled, such as the detailed introduction to typical pan-sharpening CNNs, data simulation, feasible and open training-test datasets, simple and easy-to-understand unified code writing framework, etc. So, we mainly review the growth of CNNs for RS-based pan-sharpening from 5 aspects as following: 1) detailed introduction of 7 CNN-based pan-sharpening methods and comparative analysis are given out under the datasets-coordinated; 2) the simulation of training-test datasets is introduced in details. We intend to release the pan-sharpening datasets of related satellites (such as WorldView-3, QuickBird, GaoFen2, WorldView-2 satellites); 3) for all 7 CNN-based methods introduced, Python codes-based on Pytorch library is released in a framework-integrated to facilitate the completeness and feasibility further; 4) a pan-sharpening-unified MATLAB software package is demonstrated for the test of deep-learning and traditional approaches, which is beneficial to conduct balancing tests; and 5) future research direction is predicted as well. Please click the link for project details: https://liangjiandeng.github.io/PanCollection.html. First, CNN-based techniques have proven to have good performance under the similar situations. But, the performance has decreased when a complete different situation is employed (such as in contrast with existing contemporary methods), a limitation of these approaches are challenged. Next, the fine-tuning technique has demonstrated its effectiveness in resolving the problem mentioned above, ensuring better performance for these complicated test scenarios. Compared to the quickest speed conventional methods, the computing load of the studied CNN-based algorithms can be determined as an appropriate manner during the testing phase. Finally, we recommend that a novel design of CNN-based pan-sharpening methods. The skip connection operation can aid ML-based methods in achieving a faster convergence when it is focused on the analyzed CNN-based pan-sharpening approaches. Instead, a stronger feature extraction and learning can be supported by the design of multi-scaled architectures (including the bidirectional structure even). Furthermore, the capacity of the generalized networks can be improved using a fine-tuning technique and learning in a customized domain (instead of the original image domain). However, there are still some problems to be resolved. The computational weight problem is challenged to create networks with fewer parameters so as to guarantee network performance (even achieving a quick convergence). Furthermore, the recent advancements in CNN for pan-sharpening have a restricted capacity for generalization. A lower resolution-based initial vision is beneficial to provide labels for network training. As a result, loss functions-based techniques (unsupervised) have been developed in evaluating similarities at full resolution. The future research is potential to be developed for full resolution further.