A multimodal biometric test bed for quality-dependent, cost-sensitive and client-specific score-level fusion algorithms

This paper presents a test bed, called the Biosecure DS2 score-and-quality database, for evaluating, comparing and benchmarking score-level fusion algorithms for multimodal biometric authentication. It is designed to benchmark quality-dependent, client-specific, cost-sensitive fusion algorithms. A quality-dependent fusion algorithm is one which attempts to devise a fusion strategy that is dependent on the biometric sample quality. A client-specific fusion algorithm, on the other hand, exploits the specific score characteristics of each enrolled user in order to customize the fusion strategy. Finally, a cost-sensitive fusion algorithm attempts to select a subset of biometric modalities/systems (at a specified cost) in order to obtain the maximal generalization performance. To the best of our knowledge, the BioSecure DS2 data set is the first one designed to benchmark the above three aspects of fusion algorithms. This paper contains some baseline experimental results for evaluating the above three types of fusion scenarios.     

On estimating performance indices for biometric identification

This paper investigates an information theoretic approach for formulating performance indices for the biometric authentication. Firstly, we formulate the constrained capacity, as a performance index for biometric authentication system for the finite number of users. Like Shannon capacity, constrained capacity is formulated using signal to noise ratio which is estimated from known statistics of users’ biometric information in the database. Constrained capacity of a user and of biometric system is fixed, given the database and the matching function. Experimental analysis using real palmprint and hand geometry images illustrates use of constrained capacity to estimate: (i) performance gains from the cohort information, (ii) the effective number of user-specific cohorts for a user and for the biometric system, (iii) information content of biometric features, and (iv) the performance of score level fusion rules for multimodal biometric system. Secondly, this paper investigates a rate-distortion framework for formulating false random correspondence probability as performance of a generic biometric. Our analysis concludes that constrained capacity can be a promising addition to performance of a biometric system. Similarly, individuality expressed as false random correspondence probability can be the performance index of a biometric trait.     

Score normalization in multimodal biometric systems

Multimodal biometric systems consolidate the evidence presented by multiple biometric sources and typically provide better recognition performance compared to systems based on a single biometric modality. Although information fusion in a multimodal system can be performed at various levels, integration at the matching score level is the most common approach due to the ease in accessing and combining the scores generated by different matchers. Since the matching scores output by the various modalities are heterogeneous, score normalization is needed to transform these scores into a common domain, prior to combining them. In this paper, we have studied the performance of different normalization techniques and fusion rules in the context of a multimodal biometric system based on the face, fingerprint and hand-geometry traits of a user. Experiments conducted on a database of 100 users indicate that the application of min–max, z-score, and tanh normalization schemes followed by a simple sum of scores fusion method results in better recognition performance compared to other methods. However, experiments also reveal that the min–max and z-score normalization techniques are sensitive to outliers in the data, highlighting the need for a robust and efficient normalization procedure like the tanh normalization. It was also observed that multimodal systems utilizing user-specific weights perform better compared to systems that assign the same set of weights to the multiple biometric traits of all users.     

结合并行融合的序列化多模态生物特征识别系统框架

针对多模态生物特征识别系统并行融合模式中使用方便性和使用效率方面的问题,在现有序列化多模态生物特征识别系统的基础上,提出了一种结合并行融合和序列化融合的多生物特征识别系统框架。框架中首先采用步态、人脸与指纹三种生物特征的不同组合方式以加权相加的得分级融合算法进行的识别过程;其次,利用在线的半监督学习技术提高弱特征的识别性能,从而进一步增强系统的使用方便性和识别可靠性。理论分析和实验结果表明,在此框架下,随使用时间的推移,系统能够通过在线学习提高弱分类器的性能,用户的使用方便性和系统的识别精度都得到了进一步提升。     

Fusion of palm-phalanges print with palmprint and dorsal hand vein

To ensure the high performance of a biometric system, various unimodal systems are combined to evade their constraints to form a multimodal biometric system. Here, a multimodal personal authentication system using palmprint, dorsal hand vein pattern and a novel biometric modality “palm-phalanges print” is presented. Firstly, we have collected a new anterior hand database of 50 individuals with 500 images at the institute referred to as NSIT Palmprint Database 1.0 by using NSIT palmprint device. Then from these anterior hand images, database for palmprint and palm-phalanges is created. In this biometric system, the individuals do not have to undergo the distress of using two different sensors since the palmprint and palm-phalanges print features can be captured from the same image, using NSIT palmprint device, at the same time. For dorsal hand vein, Bosphorus Hand Vein Database is used because of the stability and uniqueness of hand vein patterns. We propose fusion of three different biometric modalities which includes palmprint (PP), palm-phalanges print (PPP) and dorsal hand vein (DHV) and perform score level fusion of PP-PPP, PP-DHV, PPP-DHV and PP-PPP-DHV strategies. Lastly, we use K-nearest neighbor, support vector machine and random forest to validate the matching stage. The results proved the validity of our proposed modality and show that multimodal fusion has an edge over unimodal fusion.     

Group sparse representation based classification for multi-feature multimodal biometrics

Multimodal biometrics technology consolidates information obtained from multiple sources at sensor level, feature level, match score level, and decision level. It is used to increase robustness and provide broader population coverage for inclusion. Due to the inherent challenges involved with feature-level fusion, combining multiple evidences is attempted at score, rank, or decision level where only a minimal amount of information is preserved. In this paper, we propose the Group Sparse Representation based Classifier (GSRC) which removes the requirement for a separate feature-level fusion mechanism and integrates multi-feature representation seamlessly into classification. The performance of the proposed algorithm is evaluated on two multimodal biometric datasets. Experimental results indicate that the proposed classifier succeeds in efficiently utilizing a multi-feature representation of input data to perform accurate biometric recognition.     

Performance evaluation of score level fusion in multimodal biometric systems

In a multimodal biometric system, the effective fusion method is necessary for combining information from various single modality systems. In this paper the performance of sum rule-based score level fusion and support vector machines (SVM)-based score level fusion are examined. Three biometric characteristics are considered in this study: fingerprint, face, and finger vein. We also proposed a new robust normalization scheme (Reduction of High-scores Effect normalization) which is derived from min-max normalization scheme. Experiments on four different multimodal databases suggest that integrating the proposed scheme in sum rule-based fusion and SVM-based fusion leads to consistently high accuracy. The performance of simple sum rule-based fusion preceded by our normalization scheme is comparable to another approach, likelihood ratio-based fusion [8] (Nandakumar et al., 2008), which is based on the estimation of matching scores densities. Comparison between experimental results on sum rule-based fusion and SVM-based fusion reveals that the latter could attain better performance than the former, provided that the kernel and its parameters have been carefully selected.     

Multimodal Biometric Score Fusion Using Gaussian Mixture Model and Monte Carlo Method

Multimodal biometric fusion is gaining more attention among researchers in recent days. As multimodal biometric system consolidates the information from multiple biometric sources, the effective fusion of information obtained at score level is a challenging task. In this paper, we propose a framework for optimal fusion of match scores based on Gaussian Mixture Model (GMM) and Monte Carlo sampling based hypothesis testing. The proposed fusion approach has the ability to handle: 1) small size of match scores as is more commonly encountered in biometric fusion, and 2) arbitrary distribution of match scores which is more pronounced when discrete scores and multimodal features are present. The proposed fusion scheme is compared with well established schemes such as Likelihood Ratio (LR) method and weighted SUM rule. Extensive experiments carried out on five different multimodal biometric databases indicate that the proposed fusion scheme achieves higher performance as compared with other contemporary state of art fusion techniques.     

Score level fusion of multimodal biometrics using triangular norms

A multimodal biometric system that alleviates the limitations of the unimodal biometric systems by fusing the information from the respective biometric sources is developed. A general approach is proposed for the fusion at score level by combining the scores from multiple biometrics using triangular norms (t-norms) due to Hamacher, Yager, Frank, Schweizer and Sklar, and Einstein product. This study aims at tapping the potential of t-norms for multimodal biometrics. The proposed approach renders very good performance as it is quite computationally fast and outperforms the score level fusion using the combination approach (min, mean, and sum) and classification approaches like SVM, logistic linear regression, MLP, etc. The experimental evaluation on three databases confirms the effectiveness of score level fusion using t-norms.     

Overview of the combination of biometric matchers

Biometric identity verification refers to technologies used to measure human physical or behavioral characteristics, which offer a radical alternative to passports, ID cards, driving licenses or PIN numbers in authentication. Since biometric systems present several limitations in terms of accuracy, universality, distinctiveness, acceptability, methods for combining biometric matchers have attracted increasing attention of researchers with the aim of improving the ability of systems to handle poor quality and incomplete data, achieving scalability to manage huge databases of users, ensuring interoperability, and protecting user privacy against attacks. The combination of biometric systems, also known as “biometric fusion”, can be classified into unimodal biometric if it is based on a single biometric trait and multimodal biometric if it uses several biometric traits for person authentication.The main goal of this study is to analyze different techniques of information fusion applied in the biometric field. This paper overviews several systems and architectures related to the combination of biometric systems, both unimodal and multimodal, classifying them according to a given taxonomy. Moreover, we deal with the problem of biometric system evaluation, discussing both performance indicators and existing benchmarks.As a case study about the combination of biometric matchers, we present an experimental comparison of many different approaches of fusion of matchers at score level, carried out on three very different benchmark databases of scores. Our experiments show that the most valuable performance is obtained by mixed approaches, based on the fusion of scores. The source code of all the method implemented for this research is freely available for future comparisons¹.After a detailed analysis of pros and cons of several existing approaches for the combination of biometric matchers and after an experimental evaluation of some of them, we draw our conclusion and suggest some future directions of research, hoping that this work could be a useful start point for newer research.     

Multibiometric fusion strategy and its applications: A review

The unimodal biometric based system faced several inherent problems like lack of uniqueness, intra-class variation, non-universality, noisy data (presence of dirt on the sensor), restricted degree of freedom, unacceptable error rate, failure-to-enroll and spoofing attack. Multibiometric is one of the best choices to overcome these problems. Multibiometric fusion plays an important role to enhance the overall performance of the system, in which two or more individual biometric are combined together to form a better performance system. The proper use of fusion strategy is very important in the multibiometric system because it can affect the overall performance and accuracy level of the systems. In designing a multibiometric based system we can use various methods and fusion strategies to combine information from multiple sources. This paper is an in-depth study on multibiometric (multimodal, multialgorithm, multi-sample, multi-sensor and multi-instance) fusion strategy and its different applications. In addition, this paper also discusses the different methodology used in a fusion process (Sensor, Feature, Score, Decision, Rank) of multibiometric systems from last three decades and examines the methods used, to explore their successes and failure.     

FEATURES-LEVEL FUSION OF FACE AND HANDWRITTEN SIGNATURE IN MULTIMODAL BIOMETRIC IDENTIFICATION SYSTEM

Biometrics is an emerging tool used to identify humans by their physical and/or behavioral characteristics. This article presents a novel neural network–based approach for features-level fusion in a multimodal biometric identification system by combining both physical (human face) and behavioral (handwritten signature) traits. A single biometrics system has the weakness of providing neither 100% identification nor a 0% false accept rate (FAR)/false reject rate (FRR). One solution to this is to combine different biometrics together to get a multimodal biometric identification system. Moreover, a multimodal system is also robust in providing security against spoof attacks. Images of 32 × 32 pixels are used to eliminate bulk storage and processing requirements.     

Biometric template protection with DCT-based watermarking

In this paper we have addressed a solution of two big issues in design of multimodal system: template protection and fusion strategy. A robust biometric watermarking algorithm is proposed for biometric template protection. The fingerprint feature vector and iris features are used as watermark. Proposed DCT-based watermarking technique embeds watermark in low-frequency AC coefficients of selected 8 $\times $ 8 DCT smoother blocks. Blocks are classified based on human visual system. The robustness of the proposed algorithm is compared with the few state-of-art literature when watermarked image is subjected to possible channel attacks. Decision level fusion strategy is used to improve the overall performance of multimodal system. That is achieved by conditionally limiting the threshold of the fingerprint system to a maximum value, obtained by projecting 50 % of the cross over error rate on to the FRR curve of the iris system.     

A 3D face and hand biometric system for robust user-friendly authentication

A complete authentication system based on fusion of 3D face and hand biometrics is presented and evaluated in this paper. The system relies on a low cost real-time sensor, which can simultaneously acquire a pair of depth and color images of the scene. By combining 2D and 3D facial and hand geometry features, we are able to provide highly reliable user authentication robust to appearance and environmental variations. The design of the proposed system addresses two basic requirements of biometric technologies: dependable performance under real-world conditions along with user convenience. Experimental evaluation on an extensive database recorded in a real working environment demonstrates the superiority of the proposed multimodal scheme against unimodal classifiers in the presence of numerous appearance and environmental variations, thus making the proposed system an ideal solution for a wide range of real-world applications, from high-security to personalization of services and attendance control.     

Selection of optimized features for fusion of palm print and finger knuckle‐based person authentication

The impact of digital technology in biometrics is much more efficient at interpreting data than humans, which results in completely replacement of manual identification procedures in forensic science. Because the single modality‐based biometric frameworks limit performance in terms of accuracy and anti‐spoofing capabilities due to the presence of low quality data, therefore, information fusion of more than one biometric characteristic in pursuit of high recognition results can be beneficial. In this article, we present a multimodal biometric system based on information fusion of palm print and finger knuckle traits, which are least associated to any criminal investigation as evidence yet. The proposed multimodal biometric system might be useful to identify the suspects in case of physical beating or kidnapping and establish supportive scientific evidences, when no fingerprint or face information is present in photographs. The first step in our work is data preprocessing, in which region of interest of palm and finger knuckle images have been extracted. To minimize nonuniform illumination effects, we first normalize the detected circular palm or finger knuckle and then apply line ordinal pattern (LOP)‐based encoding scheme for texture enrichment. The nondecimated quaternion wavelet provides denser feature representation at multiple scales and orientations when extracted over proposed LOP encoding and increases the discrimination power of line and ridge features. To best of our knowledge, this first attempt is a combination of backtracking search algorithm and 2D²LDA has been employed to select the dominant palm and knuckle features for classification. The classifiers output for two modalities are combined at unsupervised rank level fusion rule through Borda count method, which shows an increase in performance in terms of recognition and verification, that is, 100% (correct recognition rate), 0.26% (equal error rate), 3.52 (discriminative index), and 1,262 m (speed).     

Multimodal biometrics: Empirical study of performance-throughput trade-off

The paper briefly describes results of empirical study on performance (as measured by ROC) and throughput (as measured by number of matches per sec) of multimodal biometrics. We use cascaded multimodal biometric identification. Experiments show that cascaded multimodal biometric fusion improves both throughput and performance.     

A comprehensive overview of biometric fusion

The performance of a biometric system that relies on a single biometric modality (e.g., fingerprints only) is often stymied by various factors such as poor data quality or limited scalability. Multibiometric systems utilize the principle of fusion to combine information from multiple sources in order to improve recognition accuracy whilst addressing some of the limitations of single-biometric systems. The past two decades have witnessed the development of a large number of biometric fusion schemes. This paper presents an overview of biometric fusion with specific focus on three questions: what to fuse, when to fuse, and how to fuse. A comprehensive review of techniques incorporating ancillary information in the biometric recognition pipeline is also presented. In this regard, the following topics are discussed: (i) incorporating data quality in the biometric recognition pipeline; (ii) combining soft biometric attributes with primary biometric identifiers; (iii) utilizing contextual information to improve biometric recognition accuracy; and (iv) performing continuous authentication using ancillary information. In addition, the use of information fusion principles for presentation attack detection and multibiometric cryptosystems is also discussed. Finally, some of the research challenges in biometric fusion are enumerated. The purpose of this article is to provide readers a comprehensive overview of the role of information fusion in biometrics.     

Database, protocols and tools for evaluating score-level fusion algorithms in biometric authentication

Fusing the scores of several biometric systems is a very promising approach to improve the overall system's accuracy. Despite many works in the literature, it is surprising that there is no coordinated effort in making a benchmark database available. It should be noted that fusion in this context consists not only of multimodal fusion, but also intramodal fusion, i.e., fusing systems using the same biometric modality but different features, or same features but using different classifiers. Building baseline systems from scratch often prevents researchers from putting more efforts in understanding the fusion problem. This paper describes a database of scores taken from experiments carried out on the XM2VTS face and speaker verification database. It then proposes several fusion protocols and provides some state-of-the-art tools to evaluate the fusion performance.     

基于多模态融合的城市道路场景视频描述模型研究

城市道路视频描述存在仅考虑视觉信息而忽视了同样重要的音频信息的问题,多模态融合算法是解决此问题的方案之一。针对现有基于Transformer的多模态融合算法都存在着模态之间融合性能低、计算复杂度高的问题,为了提高多模态信息之间的交互性,提出了一种新的基于Transformer的视频描述模型多模态注意力瓶颈视频描述（multimodal attention bottleneck for video captioning,MABVC）。首先使用预训练好的I3D和VGGish网络提取视频的视觉和音频特征并将提取好的特征输入到Transformer模型当中,然后解码器部分分别训练两个模态的信息再进行多模态的融合,最后将解码器输出的结果经过处理生成人们可以理解的文本描述。在通用数据集MSR-VTT、MSVD和自建数据集BUUISE上进行对比实验,通过评价指标对模型进行验证。实验结果表明,基于多模态注意力融合的视频描述模型在各个指标上都有明显提升。该模型在交通场景数据集上依旧能够取得良好的效果,在智能驾驶行业具有很大的应用前景。     

A supervised method to discriminate between impostors and genuine in biometry

In this paper, we describe a supervised technique that allows to develop a more robust biometric system with respect to those based directly on the similarities of the biometric matchers or on the similarities normalised by the unconstrained cohort normalisation.In order to discriminate between genuine and impostors a quadratic discriminant classifier is trained using four features: the similarities of the biometric matcher; the similarities of the biometric matcher after the unconstrained cohort normalisation (UCN); the average scores among the test pattern and the users that belong to the background model; the difference between the user-specific threshold and the user-independent threshold.The proposed technique is validated by extensive experiments carried out on several biometric datasets (palm, finger, 2D and 3D faces, and ear). The experimental results demonstrate that the capabilities provided by our supervised method can significantly improve the performance of a standard biometric matcher or the performance of the standard UCN.