Feature-level and decision-level fusion of noncoincidently sampledsensors for land mine detection

We present and compare methods for feature-level (predetection) and decision-level (postdetection) fusion of multisensor data. This study emphasizes fusion techniques that are suitable for noncommensurate data sampled at noncoincident points. Decision-level fusion is most convenient for such data, but it is suboptimal in principle, since targets not detected by all sensors will not obtain the full benefits of fusion. A novel algorithm for feature-level fusion of noncommensurate, noncoincidently sampled data is described, in which a model is fitted to the sensor data and the model parameters are used as features. Formulations for both feature-level and decision-level fusion are described, along with some practical simplifications. A closed-form expression is available for feature-level fusion of normally distributed data and this expression is used with simulated data to study requirements for sample position accuracy in multisensor data. The performance of feature-level and decision-level fusion algorithms are compared for experimental data acquired by a metal detector, a ground-penetrating radar, and an infrared camera at a challenging test site containing surrogate mines. It is found that fusion of binary decisions does not perform significantly better than the best available sensor. The performance of feature-level fusion is significantly better than the individual sensors, as is decision-level fusion when detection confidence information is also available (“soft-decision” fusion)     

兼顾特征级和决策级融合的场景分类

针对单一特征在场景分类中精度不高的问题,借鉴信息融合的思想,提出了一种兼顾特征级融合和决策级融合的分类方法。首先,提取图像的尺度不变特征变换词包(SIFT-BoW)、Gist、局部二值模式(LBP)、Laws纹理以及颜色直方图五种特征。然后,将每种特征单独对场景进行分类得到的结果以Dezert-Smarandache理论(DSmT)推理的方式在决策级进行融合,获得决策级融合下的分类结果;同时,将五种特征串行连接实现特征级融合并进行分类,得到特征级融合下的分类结果。最后,将特征级和决策级的分类结果进行自适应的再次融合完成场景分类。在决策级融合中,为解决DSmT推理过程中基本信度赋值(BBA)构造困难的问题,提出一种利用训练样本构造后验概率矩阵来完成基本信度赋值的方法。在21类遥感数据集上进行分类实验,当训练样本和测试样本各为50幅时,分类精度达到88.61%,较单一特征中的最高精度提升了12.27个百分点,同时也高于单独进行串行连接的特征级融合或DSmT推理的决策级融合的分类精度。     

AI-enabled and multimodal data driven smart health monitoring of wind power systems: A case study

The development of AI has enabled the fault detection of industrial components to be achieved through the combination with deep learning. A detection method combined with deep learning has also emerged for the fault detection of fan blades, such as models based on neural networks using the appearance or sound of the blades. However, the detection model obtained from a single data type often has limitations, such as low accuracy and overfitting. This is also the problem with fan blade detection. In contrast, multimodal data fusion detection models are often more stable. The modality diversity of blade diagnosis is strong, and it can be achieved from multiple modalities such as image, sound, and vibration. To improve the accuracy of fault diagnosis of fan blades, this article proposes a multimodal double-layer detection system (MTDS) based on decision-level and feature-level fusion. The system includes a wind turbine simulation platform and a multimodal detection system. It mainly obtains different modal data of the simulated wind turbine from the image, sound, and vibration signals, including blade images through unmanned aerial vehicle photography, blade vibration signals through electronic vibrators, and blade sound signals through microphones. The highly correlated sound and vibration modal data are fused at the feature level, and a detection model based on the sound and vibration mixed mode is implemented using a sound-vibration-CNN (SV-CNN) proposed in this case. Then, a detection model of the image mode is trained based on the blade image using a Convolution Block Attention Module ResNet (CBAM-ResNet) network. Finally, the detection input of the two modal models is fed into a perceptron to obtain the final prediction result, and the decision-level fusion is implemented to achieve fan blade detection based on multimodal, namely the implementation of MTDS.     

基于多特征融合的医学图像识别研究*

以医学图像为研究对象,针对任何一类特征都不能很好地表达医学图像的缺点以及进一步提高医学图像的识别率,提出了一种基于特征级数据融合与决策级数据融合相结合的分类方法。实验结果表明,采用特征级数据融合,融合后的特征可以较好地表达医学图像,且减少了后期分类的计算量;采用决策级数据融合,取得了比单个分类器更高的识别率。     

语种辨识的多特征信息应用

提取动态的高层语言学特征建立了改进的语种相关的、联合的GMM-LM语种辨识方案。该方案减小了不同语种的高斯混合模型和语言模型之间的相关性,也降低了训练的复杂度。还提出了基于特征提取层和判决层融合技术的语种辨识系统。该系统利用了不同类型的特征对区分不同语种的贡献来增加不同语种语料之间的差异,并使相同语种的语料之间的差异减小。实验表明,设计的语种辨识系统具有较好的扩展性;基于特征提取层和判决层的融合系统能够有效地提高系统识别率。     

Gender recognition from visible and thermal infrared facial images

Most present research of gender recognition focuses on visible facial images, which are sensitive to illumination changes. In this paper, we proposed hybrid methods for gender recognition by fusing visible and thermal infrared images. First, the active appearance model is used to extract features from visible images, as well as local binary pattern features and several statistical temperature features are extracted from thermal infrared images. Then, feature selection is performed by using the F-test statistic. Third, we propose using Bayesian Networks to perform explicit and implicit fusion of visible and thermal infrared image features. For explicit fusion, we propose two Bayesian Networks to perform decision-level and feature-level fusion. For implicit fusion, we propose using features from one modality as privileged information to improve gender recognition by another modality. Finally, we evaluate the proposed methods on the Natural Visible and Infrared facial Expression spontaneous database and the Equinox face database. Experimental results show that both feature-level and decision-level fusion improve the gender recognition performance, compared to that achieved from one modality. The proposed implicit fusion methods successfully capture the role of privileged information of one modality, thus enhance the gender recognition from another modality.     

An adaptive PNN-DS approach to classification using multi-sensor information fusion

In this paper, an adaptive neural network approach to classification which combines modified probabilistic neural network and D-S evidence theory (PNN-DS) is proposed. It attempts to deal with the drawbacks of information uncertainty and imprecision using single classification algorithm. This PNN-DS approach firstly adopts a modified probabilistic neural network (PNN) to obtain posteriori probabilities and make a primary classification decision in feature-level fusion. Then posteriori probabilities are transformed to masses noting the evidence of the D-S evidential theory. Finally advanced D-S evidential theory is utilized to gain more accurate classification results in the last decision-level fusion. In order to implement PNN-DS, covariance matrices are firstly employed in the modified PNN module to replace the singular smoothing factor in the PNN’s kernel function, and linear function is utilized in the pattern of summation layer. Secondly, the whole scheme of the proposed approach is explained in depth. Thirdly, three classification experiments are carried out on the proposed approach and a large amount of comparable analyses are done to demonstrate the effectiveness and robustness of the proposed approach. Experiments reveal that the PNN-DS outperforms BPNN-DS, which provides encouraging results in terms of classification accuracy and the speed of learning convergence.

     

An adaptive PNN-DS approach to classification using multi-sensor information fusion

In this paper, an adaptive neural network approach to classification which combines modified probabilistic neural network and D-S evidence theory (PNN-DS) is proposed. It attempts to deal with the drawbacks of information uncertainty and imprecision using single classification algorithm. This PNN-DS approach firstly adopts a modified PNN to obtain posteriori probabilities and make a primary classification decision in feature-level fusion. Then posteriori probabilities are transformed to masses noting the evidence of the D-S evidential theory. Finally advanced D-S evidential theory is utilized to gain more accurate classification results in the last decision-level fusion. In order to implement PNN-DS, covariance matrices are firstly employed in the modified PNN module to replace the singular smoothing factor in the PNN’s kernel function, and linear function is utilized in the pattern of summation layer. Secondly, the whole scheme of the proposed approach is explained in depth. Thirdly, three classification experiments are carried out on the proposed approach and a large amount of comparable analyses are done to demonstrate the effectiveness and robustness of the proposed approach. Experiments reveal that the PNN-DS outperforms BPNN-DS, which provides encouraging results in terms of classification accuracy and the speed of learning convergence.     

Fuzzy integral based information fusion for classification of highly confusable non-speech sounds

Acoustic event classification may help to describe acoustic scenes and contribute to improve the robustness of speech technologies. In this work, fusion of different information sources with the fuzzy integral (FI), and the associated fuzzy measure (FM), are applied to the problem of classifying a small set of highly confusable human non-speech sounds. As FI is a meaningful formalism for combining classifier outputs that can capture interactions among the various sources of information, it shows in our experiments a significantly better performance than that of any single classifier entering the FI fusion module. Actually, that FI decision-level fusion approach shows comparable results to the high-performing SVM feature-level fusion and thus it seems to be a good choice when feature-level fusion is not an option. We have also observed that the importance and the degree of interaction among the various feature types given by the FM can be used for feature selection, and gives a valuable insight into the problem.     

基于数据融合的遥感图象处理技术

简要地回顾了数据融合技术产生，发展的必然性，以及学者们提出的几种相关定义；尽可能详细地分析了数据融合的框架结构。包括像素层，特征层和决策层三层数据融合，并重点分析了各个数据融合层中的融合方法，以及这些方法在遥感图象处理中的应用，由于数据融合与遥感图象分类，目标检测，变化检测，目标识别的密切相关性，还对数据融合与这些应用的结合作了一定的分析。最后给出了结论和展望。     

Fusion of voice signal information for detection of mild laryngeal pathology

Detection of mild laryngeal disorders using acoustic parameters of human voice is the main objective in this study. Observations of sustained phonation (audio recordings of vocalized /a/) are labeled by clinical diagnosis and rated by severity (from 0 to 3). Research is exclusively constrained to healthy (severity 0) and mildly pathological (severity 1) cases – two the most difficult classes to distinguish between.Comprehensive voice signal characterization and information fusion constitute the approach adopted here. Characterization is obtained through diverse feature set, containing 26 feature subsets of varying size, extracted from the voice signal. Usefulness of feature-level and decision-level fusion is explored using support vector machine (SVM) and random forest (RF) as basic classifiers. For both types of fusion we also investigate the influence of feature selection on model accuracy. To improve the decision-level fusion we introduce a simple unsupervised technique for ensemble design, which is based on partitioning the feature set by k-means clustering, where the parameter k controls the size and diversity of the prospective ensemble.All types of the fusion resulted in an evident improvement over the best individual feature subset. However, none of the types, including fusion setups comprising feature selection, proved to be significantly superior over the rest. The proposed ensemble design by feature set decomposition discernibly enhanced decision-level and significantly outperformed feature-level fusion. Ensemble of RF classifiers, induced from a cluster-based partitioning of the feature set, achieved equal error rate of 13.1 ± 1.8% in the detection of mildly pathological larynx. This is a very encouraging result, considering that detection of mild laryngeal disorder is a more challenging task than a common discrimination between healthy and a wide spectrum of pathological cases.     

基于自适应遗传算法的多传感器多目标静态数据关联

利用多传感器跟踪多目标技术中最重要的问题是目标关联问题,而常见的关联算法要私计算量大,要私实际动用中效果不理想。本语文提出了利用自适应遗传算法来解决在传感人、检测空域中目标个数未知情况下,单平台多传感器数据融合系统对目标进行检测时的静态数据关联问题。实验结果表明,这种算法具有很高的关联成功率,并且提高了多传感器数据融合系统的检测概率。     

Fusion of visible and thermal images for facial expression recognition

Most present research into facial expression recognition focuses on the visible spectrum, which is sensitive to illumination change. In this paper, we focus on integrating thermal infrared data with visible spectrum images for spontaneous facial expression recognition. First, the active appearance model AAM parameters and three defined head motion features are extracted from visible spectrum images, and several thermal statistical features are extracted from infrared (IR) images. Second, feature selection is performed using the F-test statistic. Third, Bayesian networks BNs and support vector machines SVMs are proposed for both decision-level and feature-level fusion. Experiments on the natural visible and infrared facial expression (NVIE) spontaneous database show the effectiveness of the proposed methods, and demonstrate thermal IR images’ supplementary role for visible facial expression recognition.     

GeoAI in terrain analysis: Enabling multi-source deep learning and data fusion for natural feature detection

In this paper we report on a new GeoAI research method which enables deep machine learning from multi-source geospatial data for natural feature detection. In particular, a multi-source, deep learning-based object detection pipeline was developed. This pipeline introduces three new features: First, strategies of both data-level fusion (i.e., channel expansion on convolutional neural networks) and feature-level fusion were integrated into the object detection model to allow simultaneous machine learning from multi-source data, including remote sensing imagery and Digital Elevation Model (DEM) data. Second, a new data fusion strategy was developed to blend DEM data and its derivatives to create a new, fused data source with enriched information content and image features. The model has also enabled deep learning by combining both the proposed data fusion and feature-level fusion strategies to yield a much-improved detection result. Third, two different sets of data augmentation techniques were applied to the multi-source training data to further improve the model performance. A series of experiments were conducted to verify the effectiveness of the proposed strategies in multi-source deep learning.     

多传感器数据融合的通讯复杂性问题

通讯复杂性理论是一个计算抽象模型,它关心的是系统内部之间的数据通讯量的大小。多传感器数据融合是指多个传感器跟踪多个目标,是一种多层次的、多方面的处理过程,这个过程是对多源数据进行检测、结合、相关、估计和组合以达到精确跟踪不同目标的目的。该文把复杂性秩理论应用到多传感器融合,以获得多传感器之间数据融合所需的最小的通讯条件。     

多模态情绪识别研究综述

本文针对多模态情绪识别这一新兴领域进行综述。首先从情绪描述模型及情绪诱发方式两个方面对情绪识别的研究基础进行了综述。接着针对多模态情绪识别中的信息融合这一重难点问题,从数据级融合、特征级融合、决策级融合、模型级融合4种融合层次下的主流高效信息融合策略进行了介绍。然后从多种行为表现模态混合、多神经生理模态混合、神经生理与行为表现模态混合这3个角度分别列举具有代表性的多模态混合实例,全面合理地论证了多模态相较于单模态更具情绪区分能力和情绪表征能力,同时对多模态情绪识别方法转为工程技术应用提出了一些思考。最后立足于情绪识别研究现状的分析和把握,对改善和提升情绪识别模型性能的方式和策略进行了深入的探讨与展望。     

物联网信息融合技术及存在的问题研究

针对物联网信息融合问题进行了简要阐述, 指出物联网信息融合技术分为数据级融合、特征级融合和决策级融合三个层次。从融合技术的原理、适用领域以及技术之间差异性等方面对不同类型技术进行了较为全面的评述。在此基础上, 指出了物联网信息融合过程中存在的问题和挑战, 分析了有待进一步研究的方向。     

A multilevel information fusion approach for visual quality inspection

For visual quality inspection systems to be applicable in industrial settings, it is mandatory that they are highly flexible, robust and accurate. In order to improve these characteristics a multilevel information fusion approach is presented. A first fusion step at the feature-level enables the system to learn from an undefined number of potential defects which might be segmented from the images. This allows for the quality control operators to label the data at the image-level and the sub-image-level, and use this information during the learning process. Additionally, the operators are allowed to provide a confidence measure for their labelling. The additional information obtained from the increased flexibility of the operator inputs allows to build more accurate classifiers. A second fusion step at the decision-level combines the classifications of different classifiers, making the system more accurate and more robust with respect to the classification method chosen. The experimental results, using various artificial and real-world visual quality inspection data sets, show that each of these fusion approaches can significantly improve the classification accuracy. If both information fusion approaches are combined the accuracy increases even further, significantly outperforming each of the fusion approaches on their own.     

Reinforcement Learning Based Data Fusion Method for Multi-Sensors

In order to improve detection system robustness and reliability, multi-sensors fusion is used in modern air combat. In this paper, a data fusion method based on reinforcement learning is developed for multi-sensors. Initially, the cubic B-spline interpolation is used to solve time alignment problems of multi-source data. Then, the reinforcement learning based data fusion (RLBDF) method is proposed to obtain the fusion results. With the case that the priori knowledge of target is obtained, the fusion accuracy reinforcement is realized by the error between fused value and actual value. Furthermore, the Fisher information is instead used as the reward if the priori knowledge is unable to be obtained. Simulations results verify that the developed method is feasible and effective for the multi-sensors data fusion in air combat.