The effects of combining classifiers with the same training statistics using Bayesian decision rules

Multiple classifier systems (MCS) are attracting increasing interest in the field of pattern recognition and machine learning. Recently, MCS are also being introduced in the remote sensing field where the importance of classifier diversity for image classification problems has not been examined. In this article, Satellite Pour l'Observation de la Terre (SPOT) IV panchromatic and multispectral satellite images are classified into six land cover classes using five base classifiers: contextual classifier, k-nearest neighbour classifier, Mahalanobis classifier, maximum likelihood classifier and minimum distance classifier. The five base classifiers are trained with the same feature sets throughout the experiments and a posteriori probability, derived from the confusion matrix of these base classifiers, is applied to five Bayesian decision rules (product rule, sum rule, maximum rule, minimum rule and median rule) for constructing different combinations of classifier ensembles. The performance of these classifier ensembles is evaluated for overall accuracy and kappa statistics. Three statistical tests, the McNemar's test, the Cochran's Q test and the Looney's F-test, are used to examine the diversity of the classification results of the base classifiers compared to the results of the classifier ensembles. The experimental comparison reveals that (a) significant diversity amongst the base classifiers cannot enhance the performance of classifier ensembles; (b) accuracy improvement of classifier ensembles can only be found by using base classifiers with similar and low accuracy; (c) increasing the number of base classifiers cannot improve the overall accuracy of the MCS and (d) none of the Bayesian decision rules outperforms the others.     

Neural-network classifiers for recognizing totally unconstrainedhandwritten numerals

Artificial neural networks have been recognized as a powerful tool for pattern classification problems, but a number of researchers have also suggested that straightforward neural-network approaches to pattern recognition are largely inadequate for difficult problems such as handwritten numeral recognition. In this paper, we present three sophisticated neural-network classifiers to solve complex pattern recognition problems: multiple multilayer perceptron (MLP) classifier, hidden Markov model (HMM)/MLP hybrid classifier, and structure-adaptive self-organizing map (SOM) classifier. In order to verify the superiority of the proposed classifiers, experiments were performed with the unconstrained handwritten numeral database of Concordia University, Montreal, Canada. The three methods have produced 97.35%, 96.55%, and 96.05% of the recognition rates, respectively, which are better than those of several previous methods reported in the literature on the same database.     

Fuzzy learning vector quantization for hyperspectral coastal vegetation classification

Artificial neural networks (ANNs) may be of significant value in extracting vegetation type information in complex vegetation mapping problems, particularly in coastal wetland environments. Unsupervised, self-organizing ANNs have not been employed as frequently as supervised ANNs for vegetation mapping tasks, and further remote sensing research involving fuzzy ANNs is also needed. In this research, the utility of a fuzzy unsupervised ANN, specifically a fuzzy learning vector quantization (FLVQ) ANN, was investigated in the context of hyperspectral AVIRIS image classification. One key feature of the neural approach is that unlike conventional hyperspectral data processing methods, endmembers for a given scene, which can be difficult to determine with confidence, are not required for neural analysis. The classification accuracy of FLVQ was comparable to a conventional supervised multi-layer perceptron, trained with backpropagation (MLP) (KHAT (Kˆ) accuracy: 82.82% and 84.66%, respectively; normalized accuracy: 74.60% and 75.85%, respectively), with no significant difference at the 95% confidence level. All neural algorithms in the experiment yielded significantly higher classification accuracies than the conventional endmember-based hyperspectral mapping method assessed (i.e., matched filtering, where Kˆ accuracy = 61.00% and normalized accuracy = 57.96%). FLVQ was also dramatically more computationally efficient than the baseline supervised and unsupervised ANN algorithms tested, including the MLP and the Kohonen self-organizing map (SOM), respectively. The 400-neuron FLVQ network required only 3.6% of the computation time used by the MLP network, and only 5.9% of the MLP time was used by the 588-neuron FLVQ network. In addition, the 400-neuron FLVQ used only 16.7% of the time used by the 400-neuron SOM for model development.     

Genetic classifiers for remotely sensed images: Comparison with standard methods

In this article the effectiveness of some recently developed genetic algorithm-based pattern classifiers was investigated in the domain of satellite imagery which usually have complex and overlapping class boundaries. Landsat data, SPOT image and IRS image are considered as input. The superiority of these classifiers over k-NN rule, Bayes' maximum likelihood classifier and multilayer perceptron (MLP) for partitioning different landcover types is established. Results based on producer's accuracy (percentage recognition score), user's accuracy and kappa values are provided. Incorporation of the concept of variable length chromosomes and chromosome discrimination led to superior performance in terms of automatic evolution of the number of hyperplanes for modelling the class boundaries, and the convergence time. This non-parametric classifier requires very little a priori information, unlike k-NN rule and MLP (where the performance depends heavily on the value of k and the architecture, respectively), and Bayes' maximum likelihood classifier (where assumptions regarding the class distribution functions need to be made).     

A modified probabilistic neural network for partial volume segmentation in brain MR image.

A modified probabilistic neural network (PNN) for brain tissue segmentation with magnetic resonance imaging (MRI) is proposed. In this approach, covariance matrices are used to replace the singular smoothing factor in the PNN's kernel function, and weighting factors are added in the pattern of summation layer. This weighted probabilistic neural network (WPNN) classifier can account for partial volume effects, which exist commonly in MRI, not only in the final result stage, but also in the modeling process. It adopts the self-organizing map (SOM) neural network to overly segment the input MR image, and yield reference vectors necessary for probabilistic density function (pdf) estimation. A supervised "soft" labeling mechanism based on Bayesian rule is developed, so that weighting factors can be generated along with corresponding SOM reference vectors. Tissue classification results from various algorithms are compared, and the effectiveness and robustness of the proposed approach are demonstrated.     

基于可见光与红外数据融合的地形分类

针对单传感器地形分类效果不佳的问题,提出一种基于可见光与红外数据融合的地形分类方法。分别对可见光图像与红外图像提取特征,使用最近邻分类器和最小距离分类器进行后验概率估计,将来自不同特征、不同分类器的后验概率加权组合,通过散度计算得到特征的权重,实验确定分类器的权重,并在最小距离的后验概率估计中,使用马氏距离代替欧氏距离。实验结果表明,该方法对水泥路和沙子路的识别率分别达到99.33%和96.67%,均高于同类方法。     

基于条件对数似然函数导数的贝叶斯网络分类器优化算法

通常基于鉴别式学习策略训练的贝叶斯网络分类器有较高的精度,但在具有冗余边的网络结构之上鉴别式参数学习算法的性能受到一定的限制.为了在实际应用中进一步提高贝叶斯网络分类器的分类精度,该文定量描述了网络结构与真实数据变量分布之间的关系,提出了一种不存在冗余边的森林型贝叶斯网络分类器及其相应的FAN学习算法(Forest-Augmented Naive Bayes Algorithm),FAN算法能够利用对数条件似然函数的偏导数来优化网络结构学习.实验结果表明常用的限制性贝叶斯网络分类器通常存在一些冗余边,其往往会降低鉴别式参数学习算法的性能;森林型贝叶斯网络分类器减少了结构中的冗余边,更加适合于采用鉴别式学习策略训练参数;应用条件对数似然函数偏导数的FAN算法在大多数实验数据集合上提高了分类精度.     

Using the self organizing map for clustering of text documents

An increasing number of computational and statistical approaches have been used for text classification, including nearest-neighbor classification, naïve Bayes classification, support vector machines, decision tree induction, rule induction, and artificial neural networks. Among these approaches, naïve Bayes classifiers have been widely used because of its simplicity. Due to the simplicity of the Bayes formula, the naïve Bayes classification algorithm requires a relatively small number of training data and shorter time in both the training and classification stages as compared to other classifiers. However, a major short coming of this technique is the fact that the classifier will pick the highest probability category as the one to which the document is annotated too. Doing this is tantamount to classifying using only one dimension of a multi-dimensional data set. The main aim of this work is to utilize the strengths of the self organizing map (SOM) to overcome the inadvertent dimensionality reduction resulting from using only the Bayes formula to classify. Combining the hybrid system with new ranking techniques further improves the performance of the proposed document classification approach. This work describes the implementation of an enhanced hybrid classification approach which affords a better classification accuracy through the utilization of two familiar algorithms, the naïve Bayes classification algorithm which is used to vectorize the document using a probability distribution and the self organizing map (SOM) clustering algorithm which is used as the multi-dimensional unsupervised classifier.     

ECG arrhythmia classification based on optimum-path forest

An important tool for the heart disease diagnosis is the analysis of electrocardiogram (ECG) signals, since the non-invasive nature and simplicity of the ECG exam. According to the application, ECG data analysis consists of steps such as preprocessing, segmentation, feature extraction and classification aiming to detect cardiac arrhythmias (i.e., cardiac rhythm abnormalities). Aiming to made a fast and accurate cardiac arrhythmia signal classification process, we apply and analyze a recent and robust supervised graph-based pattern recognition technique, the optimum-path forest (OPF) classifier. To the best of our knowledge, it is the first time that OPF classifier is used to the ECG heartbeat signal classification task. We then compare the performance (in terms of training and testing time, accuracy, specificity, and sensitivity) of the OPF classifier to the ones of other three well-known expert system classifiers, i.e., support vector machine (SVM), Bayesian and multilayer artificial neural network (MLP), using features extracted from six main approaches considered in literature for ECG arrhythmia analysis. In our experiments, we use the MIT-BIH Arrhythmia Database and the evaluation protocol recommended by The Association for the Advancement of Medical Instrumentation. A discussion on the obtained results shows that OPF classifier presents a robust performance, i.e., there is no need for parameter setup, as well as a high accuracy at an extremely low computational cost. Moreover, in average, the OPF classifier yielded greater performance than the MLP and SVM classifiers in terms of classification time and accuracy, and to produce quite similar performance to the Bayesian classifier, showing to be a promising technique for ECG signal analysis.     

Using class-based feature selection for the classification of hyperspectral data

The rapid advances in hyperspectral sensing technology have made it possible to collect remote-sensing data in hundreds of bands. However, the data-analysis methods that have been successfully applied to multispectral data are often limited in achieving satisfactory results for hyperspectral data. The major problem is the high dimensionality, which deteriorates the classification due to the Hughes Phenomenon. In order to avoid this problem, a large number of algorithms have been proposed, so far, for feature reduction. Based on the concept of multiple classifiers, we propose a new schema for the feature selection procedure. In this framework, instead of using feature selection for whole classes, we adopt feature selection for each class separately. Thus different subsets of features are selected at the first step. Once the feature subsets are selected, a Bayesian classifier is trained on each of these feature subsets. Finally, a combination mechanism is used to combine the outputs of these classifiers. Experiments are carried out on an Airborne Visible/Infrared Imaging Spectroradiometer (AVIRIS) data set. Encouraging results have been obtained in terms of classification accuracy, suggesting the effectiveness of the proposed algorithms.     

Improving classical contextual classifications

This paper shows some combinations of classifiers that achieve high accuracy classifications. Traditionally the maximum likelihood classification is used as an initial classification for a contextual classifier. We show that by using different non-parametric spectral classifiers to obtain the initial classification, we can significatively improve the accuracy of the classification with a reasonable computational cost. In this work we propose the use of different spectral classifications as initial maps for a contextual classifier (ICM) in order to obtain some interesting combinations of spectral-contextual classifiers for remote sensing image classification with an acceptable trade-off between the accuracy of the final classification and the computational effort required.     

Combining magnitude and shape features for hyperspectral classification

The spectral angle mapper (SAM) and maximum likelihood classification (MLC) are two traditional classifiers for hyperspectral classification. This paper presents two methods to combine magnitude and shape features, one for each classifier. As the magnitude and shape features are complementary, combining both features can improve the classification accuracy. First, magnitude features are represented by the spectral radiance vector, whereas shape features are represented by the spectral gradient vector. Then, in SAM, each feature vector generates a spectral angle for each class. The two generated angles are added together to obtain a single similarity, which is used for the final classification. Similarly, in MLC, after the dimensionality reduction using Fisher's linear discriminant (FLD), each feature vector in the new feature space generates a likelihood. The two generated likelihoods are multiplied to obtain a single value, which is adopted for the final classification. Experimental results on an Airborne Visible/Infrared Imaging Spectrometer (AVIRIS) data set demonstrate that the proposed methods outperform the methods with a single feature set.     

Probabilistic neural network for breast cancer classification

Among cancers, breast cancer causes second most number of deaths in women. To reduce the high number of unnecessary breast biopsies, several computer-aided diagnosis systems have been proposed in the last years. These systems help physicians in their decision to perform a breast biopsy on a suspicious lesion seen in a mammogram or to perform a short-term follow-up examination instead. In clinical diagnosis, the use of artificial intelligent techniques as neural networks has shown great potential in this field. In this paper, three classification algorithms, multi-layer perceptron (MLP), radial basis function (RBF) and probabilistic neural networks (PNN), are applied for the purpose of detection and classification of breast cancer. Decision making is performed in two stages: training the classifiers with features from Wisconsin Breast Cancer database and then testing. The performance of the proposed structure is evaluated in terms of sensitivity, specificity, accuracy and ROC. The results revealed that PNN was the best classifiers by achieving accuracy rates of 100 and 97.66 % in both training and testing phases, respectively. MLP was ranked as the second classifier and was capable of achieving 97.80 and 96.34 % classification accuracy for training and validation phases, respectively, using scaled conjugate gradient learning algorithm. However, RBF performed better than MLP in the training phase, and it has achieved the lowest accuracy in the validation phase.     

Performance‐enhanced modified self‐organising map for iris data classification

Biometric systems are widely used in applications such as forensics and military. Biometric authentication is a challenging and complex task. These biometric systems must be accurate for practical applications. In this era of artificial intelligence, artificial neural network‐based classifiers are widely used in biometric‐based systems. However, most of the artificial neural network‐based classifiers are less accurate and computationally complex. In this work, two modified self‐organising map (SOM) networks are proposed for iris image classification to improve the performance measures. Particle swarm optimization technique is used in the training process of conventional SOM. The experiments are carried out with conventional and modified classifiers. The proposed modified classifiers provide better performance than the conventional SOM classifier.     

一种基于图像区域系综分类的室外场景理解方法

多层感知机分类器是一种有效的数据分类方法，但其分类性能受训练样本空间的限制。通过多层感知机分类器系综提高室外场景理解中图像区域的分类性能，提出了一种自动识别室外场景图像中多种景物所属概念类别的方法。该方法首先提取图像分割区域的低层视觉特征，然后基于系综分类方法建立区域视觉特征和语义类别的对应关系，通过合并相同标注区域，确定图像中景物的高层语义。对包含5种景物的150幅图像进行测试，识别率达到了87％。与基于多层感知机方法的实验结果相比，本文提出的方法取得了更好的性能，这表明该方法适合于图像区域分类。此外，系综方法还可以推广到其他的分类问题。     

Multiscale Bayesian texture segmentation using neural networks and Markov random fields

This paper presents a wavelet-based texture segmentation method using multilayer perceptron (MLP) networks and Markov random fields (MRF) in a multi-scale Bayesian framework. Inputs and outputs of MLP networks are constructed to estimate a posterior probability. The multi-scale features produced by multi-level wavelet decompositions of textured images are classified at each scale by maximum a posterior (MAP) classification and the posterior probabilities from MLP networks. An MRF model is used in order to model the prior distribution of each texture class, and a factor, which fuses the classification information through scales and acts as a guide for the labeling decision, is incorporated into the MAP classification of each scale. By fusing the multi-scale MAP classifications sequentially from coarse to fine scales, our proposed method gets the final and improved segmentation result at the finest scale. In this fusion process, the MRF model serves as the smoothness constraint and the Gibbs sampler acts as the MAP classifier. Our texture segmentation method was applied to segmentation of gray-level textured images. The proposed segmentation method shows better performance than texture segmentation using the hidden Markov trees (HMT) model and the HMTseg algorithm, which is a multi-scale Bayesian image segmentation algorithm.     

A Multiscale Approach to Automatic Medical Image Segmentation Using Self-Organizing Map

In this paper,a new medical image classification scheme is proposed using selforganizing map(SOM)combined with multiscale technique.It addresses the problem of the handling of edge pixels in the traditional multiscale SOM classifiers.First,to solve the difficulty in manual selection of edge pixels,a multiscale edge detection algorithm based on wavelet transform is proposed.Edge pixels detected are then selected into the training set as a new class and a multiscale SOM classifier is trained using this training set.In this new scheme,the SOM classifier can perform both the classification on the entire image and the edge detection simultaneously.On the other hand,the misclassification of the traditional multiscale SOM classifier in regions near edges is graeatly reduced and the correct classification is improved at the same time.     

Double-layer bayesian classifier ensembles based on frequent itemsets

Numerous models have been proposed to reduce the classification error of Na¨ ve Bayes by weakening its attribute independence assumption and some have demonstrated remarkable error performance. Considering that ensemble learning is an effective method of reducing the classification error of the classifier, this paper proposes a double-layer Bayesian classifier ensembles (DLBCE) algorithm based on frequent itemsets. DLBCE constructs a double-layer Bayesian classifier (DLBC) for each frequent itemset the new instance contained and finally ensembles all the classifiers by assigning different weight to different classifier according to the conditional mutual information. The experimental results show that the proposed algorithm outperforms other outstanding algorithms.     

Analysis of classification accuracy for pre-filtered multichannel remote sensing data

Noise is one of the main factors degrading the quality of original multichannel remote sensing data and its presence influences classification efficiency, object detection, etc. Thus, pre-filtering is often used to remove noise and improve the solving of final tasks of multichannel remote sensing. Recent studies indicate that a classical model of additive noise is not adequate enough for images formed by modern multichannel sensors operating in visible and infrared bands. However, this fact is often ignored by researchers designing noise removal methods and algorithms. Because of this, we focus on the classification of multichannel remote sensing images in the case of signal-dependent noise present in component images. Three approaches to filtering of multichannel images for the considered noise model are analysed, all based on discrete cosine transform in blocks. The study is carried out not only in terms of conventional efficiency metrics used in filtering (MSE) but also in terms of multichannel data classification accuracy (probability of correct classification, confusion matrix). The proposed classification system combines the pre-processing stage where a DCT-based filter processes the blocks of the multichannel remote sensing image and the classification stage. Two modern classifiers are employed, radial basis function neural network and support vector machines. Simulations are carried out for three-channel image of Landsat TM sensor. Different cases of learning are considered: using noise-free samples of the test multichannel image, the noisy multichannel image and the pre-filtered one. It is shown that the use of the pre-filtered image for training produces better classification in comparison to the case of learning for the noisy image. It is demonstrated that the best results for both groups of quantitative criteria are provided if a proposed 3D discrete cosine transform filter equipped by variance stabilizing transform is applied. The classification results obtained for data pre-filtered in different ways are in agreement for both considered classifiers. Comparison of classifier performance is carried out as well. The radial basis neural network classifier is less sensitive to noise in original images, but after pre-filtering the performance of both classifiers is approximately the same.     

A comparative study of classifier ensembles for bankruptcy prediction

The aim of bankruptcy prediction in the areas of data mining and machine learning is to develop an effective model which can provide the higher prediction accuracy. In the prior literature, various classification techniques have been developed and studied, in/with which classifier ensembles by combining multiple classifiers approach have shown their outperformance over many single classifiers. However, in terms of constructing classifier ensembles, there are three critical issues which can affect their performance. The first one is the classification technique actually used/adopted, and the other two are the combination method to combine multiple classifiers and the number of classifiers to be combined, respectively. Since there are limited, relevant studies examining these aforementioned disuses, this paper conducts a comprehensive study of comparing classifier ensembles by three widely used classification techniques including multilayer perceptron (MLP) neural networks, support vector machines (SVM), and decision trees (DT) based on two well-known combination methods including bagging and boosting and different numbers of combined classifiers. Our experimental results by three public datasets show that DT ensembles composed of 80–100 classifiers using the boosting method perform best. The Wilcoxon signed ranked test also demonstrates that DT ensembles by boosting perform significantly different from the other classifier ensembles. Moreover, a further study over a real-world case by a Taiwan bankruptcy dataset was conducted, which also demonstrates the superiority of DT ensembles by boosting over the others.