BoVSG: bag of visual SubGraphs for remote sensing scene classification

ABSTRACT

Remote sensing scene classification is gaining much more interest in the recent few years for many strategic fields such as security, land cover and land use monitoring. Several methods have been proposed in the literature and they can be divided into three main classes based on the features used: handcrafted features, features obtained by unsupervised learning and those obtained from deep learning. Handcrafted features are generally time consuming and suboptimal. Unsupervised learning based features which have been proposed later gave better results but their performances are still limited because they mainly rely on shallow networks and are not able to extract powerful features. Deep learning based features are recently investigated and gave interesting results. But, they cannot be usually used because of the scarcity of labelled remote sensing images and are also computationally expensive. Most importantly, whatever kind of feature is used, the neighbourhood information of them is ignored. In this paper, we propose a novel remote sensing scene representation and classification approach called Bag of Visual SubGraphs (BoVSG). First, each image is segmented into superpixels in order to summarize the image content while retaining relevant information. Then, the superpixels from all images are clustered according to their colour and texture features and a random label is assigned to each cluster that probably corresponds to some material or land cover type. Thus superpixels belonging to the same cluster have the same label. Afterwards, each image is modelled with a graph where nodes correspond to labelled superpixels and edges model spatial neighbourhoods. Finally, each image is represented by a histogram of the most frequent subgraphs corresponding to land cover adjacency patterns. This way, local spatial relations between the nodes are also taken into account. Resultant feature vectors are classified using standard classification algorithms. The proposed approach is tested on three popular datasets and its performance outperforms state-of-the-art methods, including deep learning methods. Besides its accuracy, the proposed approach is computationally much less expensive than deep learning methods.     

Growing random forest on deep convolutional neural networks for scene categorization

Breakthrough performances have been achieved in computer vision by utilizing deep neural networks. In this paper we propose to use random forest to classify image representations obtained by concatenating multiple layers of learned features of deep convolutional neural networks for scene classification. Specifically, we first use deep convolutional neural networks pre-trained on the large-scale image database Places to extract features from scene images. Then, we concatenate multiple layers of features of the deep neural networks as image representations. After that, we use random forest as the classifier for scene classification. Moreover, to reduce feature redundancy in image representations we derived a novel feature selection method for selecting features that are suitable for random forest classification. Extensive experiments are conducted on two benchmark datasets, i.e. MIT-Indoor and UIUC-Sports. Obtained results demonstrated the effectiveness of the proposed method. The contributions of the paper are as follows. First, by extracting multiple layers of deep neural networks, we can explore more information of image contents for determining their categories. Second, we proposed a novel feature selection method that can be used to reduce redundancy in features obtained by deep neural networks for classification based on random forest. In particular, since deep learning methods can be used to augment expert systems by having the systems essentially training themselves, and the proposed framework is general, which can be easily extended to other intelligent systems that utilize deep learning methods, the proposed method provide a potential way for improving performances of other expert and intelligent systems.     

基于内容相关性的场景图像分类方法

场景图像分类是计算机视觉领域中的一个基本问题.提出一种基于内容相关性的场景图像分类方法.首先从图像上提取视觉单词.并把图像表示成视觉单词的词频矢量;然后利用产生式模型来学习训练集合中包含的主题,和每一幅图像所包含的相关主题;最后用判定式分类器进行多类学习.提出的方法利用logistic正态分布对主题的相关性进行建模.使得学习得到的类别的主题分布更准确.并且在学习过程中不需要对图像内容进行人工标注.还提出了一种新的局部区域描述方法,它结合了局部区域的梯度信息和彩色信息.在自然场景图像集合和人造场景图像集合上实验了提出的方法,它相对于传统方法取得了更好的结果.     

基于多尺度稀疏表示的场景分类

提出一种基于多尺度稀疏表示的场景分类框架。首先从图像中提取多个尺度的局部特征,然后利用稀疏编码为每个尺度的特征单独学习相应的过完备字典;在图像表示阶段,为图像各尺度上的局部特征依据与其对应尺度的字典进行编码,并按照空间金字塔表示方法和特征各维最大汇总(max pooling)对各尺度上的特征编码分别汇总;最后将不同尺度上汇总的特征串接,形成对图像最终描述的全局向量。在三个常用标准场景库上的分类结果表明,提出的算法由于利用了不同尺度特征间的互补关系,与采用单尺度特征的方法相比,性能有了显著提升。     

多尺度深度特征融合的变化检测

目的 图像的变化检测是视觉领域的一个重要问题，传统的变化检测对光照变化、相机位姿差异过于敏感，使得在真实场景中检测结果较差。鉴于卷积神经网络（convolutional neural networks，CNN）可以提取图像中的深度语义特征，提出一种基于多尺度深度特征融合的变化检测模型，通过提取并融合图像的高级语义特征来克服检测噪音。方法 使用VGG（visual geometry group）16作为网络的基本模型，采用孪生网络结构，分别从参考图像和查询图像中提取不同网络层的深度特征。将两幅图像对应网络层的深度特征拼接后送入一个编码层，通过编码层逐步将高层与低层网络特征进行多尺度融合，充分结合高层的语义和低层的纹理特征，检测出准确的变化区域。使用卷积层对每一个编码层的特征进行运算产生对应尺度的预测结果。将不同尺度的预测结果融合得到进一步细化的检测结果。结果 与SC_SOBS（SC-self-organizing background subtraction）、SuBSENSE（self-balanced sensitivity segmenter）、FGCD（fine-grained change detection）和全卷积网络（fully convolutional network，FCN）4种检测方法进行对比。与性能第2的模型FCN相比，本文方法在VL_CMU_CD（visual localization of Carnegie Mellon University for change detection）数据集中，综合评价指标F1值和精度值分别提高了12.2%和24.4%；在PCD（panoramic change detection）数据集中，F1值和精度值分别提高了2.1%和17.7%；在CDnet（change detection net）数据集中，F1值和精度值分别提高了8.5%和5.8%。结论 本文提出的基于多尺度深度特征融合的变化检测方法，利用卷积神经网络的不同网络层特征，有效克服了光照和相机位姿差异，在不同数据集上均能得到较为鲁棒的变化检测结果。     

Detection of mitotic cells in breast cancer histopathological images using deep versus handcrafted features

One of the most important processes in the diagnosis of breast cancer, which is the leading mortality rate in women, is the detection of the mitosis stage at the cellular level. In literature, many studies have been proposed on the computer-aided diagnosis (CAD) system for detecting mitotic cells in breast cancer histopathological images. In this study, comparative evaluation of conventional and deep learning based feature extraction methods for automatic detection of mitosis in histopathological images are focused. While various handcrafted features are extracted with textural/spatial, statistical and shape-based methods in conventional approach, the convolutional neural network structure proposed on the deep learning approach aims to create an architecture that extracts the features of small cellular structures such as mitotic cells. Mitosis detection/counting is an important process that helps us assess how aggressive or malignant the cancer’s spread is. In the proposed study, approximately 180,000 non-mitotic and 748 mitotic cells are extracted for the evaluations. It is obvious that the classification stage cannot be performed properly due to the imbalanced numbers of mitotic and non-mitotic cells extracted from histopathological images. Hence, the random under-sampling boosting (RUSBoost) method is exploited to overcome this problem. The proposed framework is tested on mitosis detection in breast cancer histopathological images dataset provided from the International Conference on Pattern Recognition (ICPR) 2014 contest. In the results obtained with the deep learning approach, 79.42% recall, 96.78% precision and 86.97% F-measure values are achieved more successfully than handcrafted methods. A client/server-based framework has also been developed as a secondary decision support system for use by pathologists in hospitals. Thus, it is aimed that pathologists will be able to detect mitotic cells in various histopathological images more easily through necessary interfaces.

     

局部Gist特征匹配核的场景分类

针对场景分类任务中全局Gist特征粒度较为粗糙的问题,提出一种基于稠密网格的局部Gist特征描述,利用空间金字塔结构加入空间信息,通过引入RGB颜色空间加入颜色信息,并基于词汇包(BOW)模型设计一种高效匹配核来度量局部特征间的相似性,核化特征匹配过程,使用线性SVM完成场景分类.实验考察了不同尺度、方向、粒度和不同匹配核的局部Gist特征以及训练样本集的大小对分类结果的影响,并通过在OT场景图像集上与全局Gist特征和稠密SIFT特征的场景分类结果进行比较,充分说明了本文特征构造方法和分类模型的有效性.     

用于多光谱和高光谱图像融合的联合自注意力Transformer

目的 将高光谱图像和多光谱图像进行融合，可以获得具有高空间分辨率和高光谱分辨率的光谱图像，提升光谱图像的质量。现有的基于深度学习的融合方法虽然表现良好，但缺乏对多源图像特征中光谱和空间长距离依赖关系的联合探索。为有效利用图像的光谱相关性和空间相似性，提出一种联合自注意力的Transformer网络来实现多光谱和高光谱图像融合超分辨。方法 首先利用联合自注意力模块，通过光谱注意力机制提取高光谱图像的光谱相关性特征，通过空间注意力机制提取多光谱图像的空间相似性特征，将获得的联合相似性特征用于指导高光谱图像和多光谱图像的融合；随后，将得到的融合特征输入到基于滑动窗口的残差Transformer深度网络中，探索融合特征的长距离依赖信息，学习深度先验融合知识；最后，特征通过卷积层映射为高空间分辨率的高光谱图像。结果 在CAVE和Harvard光谱数据集上分别进行了不同采样倍率下的实验，实验结果表明，与对比方法相比，本文方法从定量指标和视觉效果上，都取得了更好的效果。本文方法相较于性能第二的方法EDBIN （enhanced deep blind iterative network），在CAVE数据集上峰值信噪比提高了0.5 dB，在Harvard数据集上峰值信噪比提高了0.6 dB。结论 本文方法能够更好地融合光谱信息和空间信息，显著提升高光谱融合超分图像的质量。     

顾及目标关联的自然场景文本检测

目的 目前基于卷积神经网络（CNN）的文本检测方法对自然场景中小尺度文本的定位非常困难。但自然场景图像中文本目标与其他目标存在很强的关联性,即自然场景中的文本通常伴随特定物体如广告牌、路牌等同时出现,基于此本文提出了一种顾及目标关联的级联CNN自然场景文本检测方法。方法 首先利用CNN检测文本目标及包含文本的关联物体目标,得到文本候选框及包含文本的关联物体候选框;再扩大包含文本的关联物体候选框区域,并从原始图像中裁剪,然后以该裁剪图像作为CNN的输入再精确检测文本候选框;最后采用非极大值抑制方法融合上述两步生成的文本候选框,得到文本检测结果。结果 本文方法能够有效地检测小尺度文本,在ICDAR-2013数据集上召回率、准确率和F值分别为0.817、0.880和0.847。结论 本文方法顾及自然场景中文本目标与包含文本的物体目标的强关联性,提高了自然场景图像中小尺度文本检测的召回率。     

残差密集空间金字塔网络的城市遥感图像分割

目的 遥感图像语义分割是根据土地覆盖类型对图像中每个像素进行分类,是遥感图像处理领域的一个重要研究方向。由于遥感图像包含的地物尺度差别大、地物边界复杂等原因,准确提取遥感图像特征具有一定难度,使得精确分割遥感图像比较困难。卷积神经网络因其自主分层提取图像特征的特点逐步成为图像处理领域的主流算法,本文将基于残差密集空间金字塔的卷积神经网络应用于城市地区遥感图像分割,以提升高分辨率城市地区遥感影像语义分割的精度。方法 模型将带孔卷积引入残差网络,代替网络中的下采样操作,在扩大特征图感受野的同时能够保持特征图尺寸不变;模型基于密集连接机制级联空间金字塔结构各分支,每个分支的输出都有更加密集的感受野信息;模型利用跳线连接跨层融合网络特征,结合网络中的高层语义特征和低层纹理特征恢复空间信息。结果 基于ISPRS （International Society for Photogrammetry and Remote Sensing） Vaihingen地区遥感数据集展开充分的实验研究,实验结果表明,本文模型在6种不同的地物分类上的平均交并比和平均F₁值分别达到69.88%和81.39%,性能在数学指标和视觉效果上均优于SegNet、pix2pix、Res-shuffling-Net以及SDFCN （symmetrical dense-shortcut fully convolutional network）算法。结论 将密集连接改进空间金字塔池化网络应用于高分辨率遥感图像语义分割,该模型利用了遥感图像不同尺度下的特征、高层语义信息和低层纹理信息,有效提升了城市地区遥感图像分割精度。     

Learning natural scene categories by selective multi-scale feature extraction

Natural scene categorization from images represents a very useful task for automatic image analysis systems. In the literature, several methods have been proposed facing this issue with excellent results. Typically, features of several types are clustered so as to generate a vocabulary able to describe in a multi-faceted way the considered image collection. This vocabulary is formed by a discrete set of visual codewords whose co-occurrence and/or composition allows to classify the scene category. A common drawback of these methods is that features are usually extracted from the whole image, actually disregarding whether they derive properly from the natural scene to be classified or from foreground objects, possibly present in it, which are not peculiar for the scene. As quoted by perceptual studies, objects present in an image are not useful to natural scene categorization, indeed bringing an important source of clutter, in dependence of their size.     

各向异性导向滤波的红外与可见光图像融合

目的 针对红外与可见光图像融合时易产生边缘细节信息丢失、融合结果有光晕伪影等问题，同时为充分获取多源图像的重要特征，将各向异性导向滤波和相位一致性结合，提出一种红外与可见光图像融合算法。方法 首先，采用各向异性导向滤波从源图像获得包含大尺度变化的基础图和包含小尺度细节的系列细节图；其次，利用相位一致性和高斯滤波计算显著图，进而通过对比像素显著性得到初始权重二值图，再利用各向异性导向滤波优化权重图，达到去除噪声和抑制光晕伪影；最后，通过图像重构得到融合结果。结果 从主客观两个方面，将所提方法与卷积神经网络（convolutional neural network，CNN）、双树复小波变换（dual-tree complex wavelet transform，DTCWT）、导向滤波（guided filtering，GFF）和各向异性扩散（anisotropic diffusion，ADF）等4种经典红外与可见光融合方法在TNO公开数据集上进行实验对比。主观分析上，所提算法结果在边缘细节、背景保存和目标完整度等方面均优于其他4种方法；客观分析上，选取互信息（mutual information，MI）、边缘信息保持度（degree of edge information，Q^AB/F）、熵（entropy，EN）和基于梯度的特征互信息（gradient based feature mutual information，FMI_gradient）等4种图像质量评价指数进行综合评价。相较于其他4种方法，本文算法的各项指标均有一定幅度的提高，MI平均值较GFF提高了21.67%，Q^AB/F平均值较CNN提高了20.21%，EN平均值较CNN提高了5.69%，FMI_gradient平均值较GFF提高了3.14%。结论 本文基于各向异性导向滤波融合算法可解决原始导向滤波存在的细节"光晕"问题，有效抑制融合结果中伪影的产生，同时具有尺度感知特性，能更好保留源图像的边缘细节信息和背景信息，提高了融合结果的准确性。     

结合Gabor滤波器和扩展LTP算子的无监督纹理图像分割

目的 现实中的纹理往往具有类型多样、形态多变、结构复杂等特点,直接影响到纹理图像分割的准确性。传统的无监督纹理图像分割算法具有一定的局限性,不能很好地提取稳定的纹理特征。本文提出了基于Gabor滤波器和改进的LTP（local ternary pattern）算子的针对复杂纹理图像的纹理特征提取算法。方法 利用Gabor滤波器和扩展LTP算子分别提取相同或相似纹理模式的纹理特征和纹理的差异性特征,并将这些特征融入到水平集框架中对纹理图像进行分割。结果 通过实验表明,对纹理方向及尺度变化较大的图像、复杂背景下的纹理图像以及弱纹理模式的图像,本文方法整体分割结果明显优于传统的Gabor滤波器、结构张量、拓展结构张量、局部相似度因子等纹理分割方法得到的结果。同时,将本文方法与基于LTP的方法进行对比,分割结果依然更优。在量化指标方面,将本文方法与各种无监督的纹理分割方法就分割准确度进行对比,结果表明,在典型的纹理图像上,本文方法准确度达到97%以上,高于其他方法的分割准确度。结论 提出了一种结合Gabor滤波器和扩展LTP算子的无监督多特征的纹理图像分割方法,能够较好地提取相似纹理模式的特征和纹理的差异性特征,且这些纹理特征可以很好地融合到水平集框架中,对真实世界复杂纹理图像能够得到良好的分割效果。     

基于多尺度图匹配核的场景单字识别方法（英文）

Recognizing characters extracted from natural scene images is quite challenging due to the high degree of intraclass variation. In this paper, we propose a multi-scale graph-matching based kernel for scene character recognition. In order to capture the inherently distinctive structures of characters, each image is represented by several graphs associated with multi-scale image grids. The similarity between two images is thus defined as the optimum energy by matching two graphs(images), which finds the best match for each node in the graph while also preserving the spatial consistency across adjacent nodes. The computed similarity is suitable to construct a kernel for support vector machine(SVM). Multiple kernels acquired by matching graphs with multi-scale grids are combined so that the final kernel is more robust. Experimental results on challenging Chars74k and ICDAR03-CH datasets show that the proposed method performs better than the state of the art methods.     

融合分割先验的多图像目标语义分割

目的 在序列图像或多视角图像的目标分割中，传统的协同分割算法对复杂的多图像分割鲁棒性不强，而现有的深度学习算法在前景和背景存在较大歧义时容易导致目标分割错误和分割不一致。为此，提出一种基于深度特征的融合分割先验的多图像分割算法。方法 首先，为了使模型更好地学习复杂场景下多视角图像的细节特征，通过融合浅层网络高分辨率的细节特征来改进PSPNet-50网络模型，减小随着网络的加深导致空间信息的丢失对分割边缘细节的影响。然后通过交互分割算法获取一至两幅图像的分割先验，将少量分割先验融合到新的模型中，通过网络的再学习来解决前景/背景的分割歧义以及多图像的分割一致性。最后通过构建全连接条件随机场模型，将深度卷积神经网络的识别能力和全连接条件随机场优化的定位精度耦合在一起，更好地处理边界定位问题。结果 本文采用公共数据集的多图像集进行了分割测试。实验结果表明本文算法不但可以更好地分割出经过大量数据预训练过的目标类，而且对于没有预训练过的目标类，也能有效避免歧义的区域分割。本文算法不论是对前景与背景区别明显的较简单图像集，还是对前景与背景颜色相似的较复杂图像集，平均像素准确度（PA）和交并比（IOU）均大于95%。结论 本文算法对各种场景的多图像分割都具有较强的鲁棒性，同时通过融入少量先验，使模型更有效地区分目标与背景，获得了分割目标的一致性。     

基于多尺度图匹配核的场景单字识别方法

由于自然场景中的文字具有较大的类内间距, 因此识别场景文字具有很大的挑战性. 本文提出了一种基于多尺度图匹配核的场景单字识别方法. 为了利用字符特有的结构特征, 将每幅图像表示为基于不同网格划分的无向图, 通过计算两个无向图之间图匹配的最优能量值来得到两幅图像的相似度, 由于图匹配在计算每个节点的最佳匹配节点时也考虑了相邻节点之间的空间位置约束, 因此可以应对具有一定形变的文字. 通过图匹配得到的两幅图像之间的相似度很适合用来构造支持向量机的核矩阵. 本文将不同尺度网格划分下得到的核矩阵进行多核融合, 使得最终得到的核矩阵更加地鲁棒. 在国际公开场景文字识别数据集Chars74k和ICDAR03-CH上的实验结果表明, 本方法取得了高于国际上已发表的其他方法的单字识别率.     

基于结构和纹理特征融合的场景图像分类

利用整体结构特征和局部纹理特征的优势,采用两级分类器对场景图像进行分类。第1级分类器利用全局结构信息得到候选类别,并通过分类结果判定相似类别对;第2级分类器则利用局部纹理信息区分相似类别,采用分类器的级联综合利用场景图像的整体结构信息和局部纹理信息。实验结果表明,该方法能够做到不同场景类别鲁棒分类,有效区分相似场景类别,提高场景图像的分类准确率。     

选择性卷积特征融合的花卉图像分类

目的 针对花卉图像标注样本缺乏、标注成本高、传统基于深度学习的细粒度图像分类方法无法较好地定位花卉目标区域等问题,提出一种基于选择性深度卷积特征融合的无监督花卉图像分类方法。方法 构建基于选择性深度卷积特征融合的花卉图像分类网络。首先运用保持长宽比的尺寸归一化方法对花卉图像进行预处理,使得图像的尺寸相同,且目标不变形、不丢失图像细节信息;之后运用由ImageNet预训练好的深度卷积神经网络VGG-16模型对预处理的花卉图像进行特征学习,根据特征图的响应值分布选取有效的深度卷积特征,并将多层深度卷积特征进行融合;最后运用softmax分类层进行分类。结果 在Oxford 102 Flowers数据集上做了对比实验,将本文方法与传统的基于深度学习模型的花卉图像分类方法进行对比,本文方法的分类准确率达85.55%,较深度学习模型Xception高27.67%。结论 提出了基于选择性卷积特征融合的花卉图像分类方法,该方法采用无监督的方式定位花卉图像中的显著区域,去除了背景和噪声部分对花卉目标的干扰,提高了花卉图像分类的准确率,适用于处理缺乏带标注的样本时的花卉图像分类问题。