基于多模态信息融合的新闻图像人脸标注

针对传统新闻图像中人脸标注方法主要依赖人脸相似度信息,分辨噪声和非噪声人脸能力以及非噪声人脸标注能力较差的问题,提出一种基于多模态信息融合的新闻图像人脸标注方法。首先根据人脸和姓名的共现关系,利用改进的K近邻算法,获得基于人脸相似度信息的人脸姓名匹配度;然后,分别从图像中提取人脸大小和位置的信息对人脸重要程度进行表征,从文本中提取姓名位置信息对姓名重要程度进行表征;最后,使用反向传播神经网络来融合上述信息完成人脸标签的推理,并提出一个标签修正策略来进一步改善标注结果。在Label Yahoo! News数据集上的测试效果表明,所提方法的标注准确率、精度和召回率分别达到了77.11%、73.58%和78.75%,与仅基于人脸相似度的算法相比,具有较好的分辨噪声和非噪声人脸能力以及非噪声人脸标注能力。     

3D shape recognition based on multi-modal information fusion

The classification and retrieval of 3D models have been widely used in the field of multimedia and computer vision. With the rapid development of computer graphics, different algorithms corresponding to different representations of 3D models have achieved the best performance. The advances in deep learning also encourage various deep models for 3D feature representation. For multi-view, point cloud, and PANORAMA-view, different models have shown significant performance on 3D shape classification. However, There’s not a way to consider utilizing the fusion information of multi-modal for 3D shape classification. In our opinion, We propose a novel multi-modal information fusion method for 3D shape classification, which can fully utilize the advantage of different modal to predict the label of class. More specifically, the proposed can effectively fuse more modal information. it is easy to utilize in other similar applications. We have evaluated our framework on the popular dataset ModelNet40 for the classification task on 3D shape. Series experimental results and comparisons with state-of-the-art methods demonstrate the validity of our approach.

     

Usability evaluation of multi-modal biometric verification systems

As a result of the evolution in the field of biometrics, a new breed of techniques and methods for user identity recognition and verification has appeared based on the recognition and verification of several biometric features considered unique to each individual. Signature and voice characteristics, facial features, and iris and fingerprint patterns have all been used to identify a person or just to verify that the person is who he/she claims to be. Although still relatively new, these new technologies have already reached a level of development that allows its commercialization. However, there is a lack of studies devoted to the evaluation of these technologies from a user-centered perspective. This paper is intended to promote user-centered design and evaluation of biometric technologies. Towards this end, we have developed a platform to perform empirical evaluations of commercial biometric identity verification systems, including fingerprint, voice and signature verification. In this article, we present an initial empirical study in which we evaluate, compare and try to get insights into the factors that are crucial for the usability of these systems.     

基于特征融合的多模态身份识别方法研究

研究了多模态身份识别问题,结合人脸和掌纹两种不同生理特征,提出了基于特征融合的多模态身份识别方法。对人脸和掌纹图像分别进行Gabor小波、二维主元变换(2DPCA)提取图像特征,根据新的权重算法,结合两种模态的特征,利用最邻近分类器进行分类识别。在AMP、ORL人脸库和Poly-U掌纹图像库中的实验结果表明,两种模态的融合能更多地给出决策分析所需的特征信息相比传统的单一模态的人脸或掌纹识别具有较高的识别率,更具安全性和准确性。     

Robust fusion of uncertain information.

A technique is presented to combine n data points, each available with point-dependent uncertainty, when only a subset of these points come from N < n sources, where N is unknown. We detect the significant modes of the underlying multivariate probability distribution using a generalization of the nonparametric mean shift procedure. The number of detected modes automatically defines N, while the belonging of a point to the basin of attraction of a mode provides the fusion rule. The robust data fusion algorithm was successfully applied to two computer vision problems: estimating the multiple affine transformations, and range image segmentation.     

Performance evaluation of score level fusion in multimodal biometric systems

In a multimodal biometric system, the effective fusion method is necessary for combining information from various single modality systems. In this paper the performance of sum rule-based score level fusion and support vector machines (SVM)-based score level fusion are examined. Three biometric characteristics are considered in this study: fingerprint, face, and finger vein. We also proposed a new robust normalization scheme (Reduction of High-scores Effect normalization) which is derived from min-max normalization scheme. Experiments on four different multimodal databases suggest that integrating the proposed scheme in sum rule-based fusion and SVM-based fusion leads to consistently high accuracy. The performance of simple sum rule-based fusion preceded by our normalization scheme is comparable to another approach, likelihood ratio-based fusion [8] (Nandakumar et al., 2008), which is based on the estimation of matching scores densities. Comparison between experimental results on sum rule-based fusion and SVM-based fusion reveals that the latter could attain better performance than the former, provided that the kernel and its parameters have been carefully selected.     

An accurate multi-modal biometric identification system for person identification via fusion of face and finger print

World Wide Web - Internet of things (IoT) have entirely revolutionized the industry. However, the cyber-security of IoT enabled cyber-physical systems is still one of the main challenges. The...     

Cognitive high level information fusion

Fusion of sensor and communication data currently can only be performed at a late processing stage after sensor and textual information are formulated as logical statements at appropriately high level of abstraction. Contrary to this it seems, the human mind integrates sensor and language signals seamlessly, before signals are understood, at pre-conceptual level. Learning of conceptual contents of the surrounding world depends on language and vice versa. The paper describes a mathematical technique for such integration. It combines fuzzy dynamic logic with dual cognitive-language models. The paper briefly discusses relationships between the proposed mathematical technique, working of the mind and applications to understanding-based search engines.     

Robust multi-modal medical image fusion via anisotropic heat diffusion guided low-rank structural analysis

This paper proposes a novel and robust multi-modal medical image fusion method, which is built upon a novel framework comprising multi-scale image decomposition based on anisotropic heat kernel design, scale-aware salient information extraction based on low-rank analysis, and scale-specific fusion rules. Our framework respects multi-scale structure features, while being robust to complex noise perturbation. First, anisotropic heat kernel is computed by constructing an image pyramid and embedding multi-level image properties into 2D manifolds in a divide-and-conquer way, consequently, multi-scale structure-preserving image decomposition can be accommodated. Second, to extract meaningfully scale-aware salient information, we conduct low-rank analysis over the image layer groups obtained in the first step, and employ the low-rank components to form the scale space of the salient features, wherein the underlying noise can be synchronously decoupled in a natural way. Third, to better fuse the complementary salient information extracted from multi-modal images, we design an S-shaped weighting function to fuse the large-scale layers, and employ the maximum selection principle to handle the small-scale layers. Moreover, we have conducted extensive experiments on MRI and PET/SPECT images. The comprehensive and quantitative comparisons with state-of-the-art methods demonstrate the informativeness, accuracy, robustness, and versatility of our novel approach.     

Robust image recognition by fusion of contextual information

This paper studies the fusion of contextual information in pattern recognition, with applications to biomedical image identification. In the real world there are cases where the identity of an object is ambiguous if the classification is based only on its own features. It is helpful to reduce the ambiguity by utilizing extra information, referred to as context, provided by accompanying objects. We investigate two techniques that incorporate context. The first approach, based on compound Bayesian theory, incorporates context by fusing the measurements of all objects under consideration. It is an optimal strategy in terms of achieving minimum set-by-set error probability. The second approach fuses the measurements of an object with explicitly extracted context. Its linear computational complexity makes it more tractable than the first approach, which requires exponential computation. These two techniques are applied to two medical applications: white blood cell image classification and microscopic urinalysis. It is demonstrated that superior classification performances are achieved by using context. In our particular applications, it reduces overall classification error, as well as false positive and false negative diagnosis rates.     

Robust enhancement and centroid-based concealment of fingerprint biometric data into audio signals

Multimedia Tools and Applications - Numerous issues are raised through the enhancement, transmission and storage of biometric data due to its high sensitivity and extremely crucial purpose....     

Towards multi-modal causability with Graph Neural Networks enabling information fusion for explainable AI

AI is remarkably successful and outperforms human experts in certain tasks, even in complex domains such as medicine. Humans on the other hand are experts at multi-modal thinking and can embed new inputs almost instantly into a conceptual knowledge space shaped by experience. In many fields the aim is to build systems capable of explaining themselves, engaging in interactive what-if questions. Such questions, called counterfactuals, are becoming important in the rising field of explainable AI (xAI). Our central hypothesis is that using conceptual knowledge as a guiding model of reality will help to train more explainable, more robust and less biased machine learning models, ideally able to learn from fewer data. One important aspect in the medical domain is that various modalities contribute to one single result. Our main question is “How can we construct a multi-modal feature representation space (spanning images, text, genomics data) using knowledge bases as an initial connector for the development of novel explanation interface techniques?”. In this paper we argue for using Graph Neural Networks as a method-of-choice, enabling information fusion for multi-modal causability (causability – not to confuse with causality – is the measurable extent to which an explanation to a human expert achieves a specified level of causal understanding). The aim of this paper is to motivate the international xAI community to further work into the fields of multi-modal embeddings and interactive explainability, to lay the foundations for effective future human–AI interfaces. We emphasize that Graph Neural Networks play a major role for multi-modal causability, since causal links between features can be defined directly using graph structures.     

A novel biometric system for signature verification based on score level fusion approach

Multimedia Tools and Applications - The active modality of handwriting is broadly related to signature verification in the context of biometric user authentication systems. Signature verification...     

A comprehensive overview of biometric fusion

The performance of a biometric system that relies on a single biometric modality (e.g., fingerprints only) is often stymied by various factors such as poor data quality or limited scalability. Multibiometric systems utilize the principle of fusion to combine information from multiple sources in order to improve recognition accuracy whilst addressing some of the limitations of single-biometric systems. The past two decades have witnessed the development of a large number of biometric fusion schemes. This paper presents an overview of biometric fusion with specific focus on three questions: what to fuse, when to fuse, and how to fuse. A comprehensive review of techniques incorporating ancillary information in the biometric recognition pipeline is also presented. In this regard, the following topics are discussed: (i) incorporating data quality in the biometric recognition pipeline; (ii) combining soft biometric attributes with primary biometric identifiers; (iii) utilizing contextual information to improve biometric recognition accuracy; and (iv) performing continuous authentication using ancillary information. In addition, the use of information fusion principles for presentation attack detection and multibiometric cryptosystems is also discussed. Finally, some of the research challenges in biometric fusion are enumerated. The purpose of this article is to provide readers a comprehensive overview of the role of information fusion in biometrics.     

An improved MPGA-ACO-BP algorithm and comprehensive evaluation system for intelligence workshop multi-modal data fusion

The digital economy is a new economic form taking data as an important production factor and digital and intelligent technology as a driving force for transformation. The core idea is to extract and fuse the knowledge implicit in data and transform it into intelligence to drive the transformation of traditional manufacturing industries, and one of its key technologies is multi-modal data fusion. In this paper, an improved MPGA-ACO-BP algorithm is proposed, and combined with an improved entropy-weighted TOPSIS method comprehensive evaluation system, which effectively solves the problem of “data scale inconsistency” between modal data leading to difficult model fusion and fusion accuracy. Finally, the validity of the theory and methods of this paper are verified using the example of multi-modal data fusion tool wear prediction in an intelligence workshop. By distilling the corresponding evaluation metrics inductively, the improved comprehensive evaluation system in this paper can also be extended to different production control scenarios to provide them with the corresponding integration information, which has a certain practical value.     

分层信息融合的物体级显著性检测

目的 显著性检测是基于对人类视觉的研究,用来帮助计算机传感器感知世界的重要研究手段。现有显著性检测方法大多仅能检测出人类感兴趣的显著点或区域,无法突出对象整体的显著性以及无法区分对象不同层次的显著性。针对上述问题,提出一种基于分层信息融合的物体级显著性检测方法。方法 与当前大多数方法不同,本文同时运用了中级别超像素和物体级别区域两种不同层次的结构信息来获取对象的显著图。首先,将图像分割为中级别的超像素,利用自下而上的方法构造初始显著图;然后通过谱聚类方法将中级别的超像素聚类成物体级的区域,并运用自上而下的先验来调整初始先验图;最后,通过热核扩散过程,将超像素级别上的显著性扩散到物体级的区域上,最终获得一致的均匀的物体级显著性图。结果 在MSRA1000标准数据库上与其他16种相关算法在准确率-召回率曲线及F度量等方面进行了定量比较,检测的平均精度和F-检验分数比其他算法高出5%以上。结论 通过多层次信息融合最终生成的显著图,实现了突出对象整体显著性以及区分不同对象显著性的目标。本文方法同样适用于多目标的显著性检测。     

Problems of automatic fusion of biometric identificators

We consider methods and algorithm of fusion of finger and facial biometrics. Suggested methods allows designing multimodal biometric systems with automatic training ability. We have carried out experiments that shows efficiency of developed methods and algorithms.     

Segmentation fusion based on neighboring information for MR brain images

In this paper, we study on how to boost image segmentation algorithms. First of all, a novel fusion scheme is proposed to combine different segmentations with mutual information to reduce misclassified pixels and obtain an accurate segmentation. As the class label of each pixel depends on the pixel’s gray level and neighbors’ labels, the fusion scheme takes both spatial and intensity information of pixels into account. Then, a detail thresholding segmentation case is designed using the proposed fusion scheme. In the case, the local Laplacian filter is used to get the smoothed version of original image. To accelerate segmentation, a discrete curve evolution based Otsu method is employed to segment the original image and its smoothed version to get two different segmentation maps. The fusion scheme is used to fuse the two maps to get the final segmentation result. Experiments on medical MR-T2 brain images are conducted to demonstrate the effectiveness of the proposed segmentation fusion method. The experimental results indicate that the proposed algorithm can improve segmentation accuracy and it is superior to other multilevel thresholding methods.     

Face and palmprint pixel level fusion and Kernel DCV-RBF classifier for small sample biometric recognition

Recently, multi-modal biometric fusion techniques have attracted increasing atove the recognition performance in some difficult biometric problems. The small sample biometric recognition problem is such a research difficulty in real-world applications. So far, most research work on fusion techniques has been done at the highest fusion level, i.e. the decision level. In this paper, we propose a novel fusion approach at the lowest level, i.e. the image pixel level. We first combine two kinds of biometrics: the face feature, which is a representative of contactless biometric, and the palmprint feature, which is a typical contacting biometric. We perform the Gabor transform on face and palmprint images and combine them at the pixel level. The correlation analysis shows that there is very small correlation between their normalized Gabor-transformed images. This paper also presents a novel classifier, KDCV-RBF, to classify the fused biometric images. It extracts the image discriminative features using a Kernel discriminative common vectors (KDCV) approach and classifies the features by using the radial base function (RBF) network. As the test data, we take two largest public face databases (AR and FERET) and a large palmprint database. The experimental results demonstrate that the proposed biometric fusion recognition approach is a rather effective solution for the small sample recognition problem.