LDA/SVM driven nearest neighbor classification

Nearest neighbor (NN) classification relies on the assumption that class conditional probabilities are locally constant. This assumption becomes false in high dimensions with finite samples due to the curse of dimensionality. The NN rule introduces severe bias under these conditions. We propose a locally adaptive neighborhood morphing classification method to try to minimize bias. We use local support vector machine learning to estimate an effective metric for producing neighborhoods that are elongated along less discriminant feature dimensions and constricted along most discriminant ones. As a result, the class conditional probabilities can be expected to be approximately constant in the modified neighborhoods, whereby better classification performance can be achieved. The efficacy of our method is validated and compared against other competing techniques using a number of datasets.     

Adaptive quasiconformal kernel nearest neighbor classification

Nearest neighbor classification assumes locally constant class conditional probabilities. This assumption becomes invalid in high dimensions due to the curse-of-dimensionality. Severe bias can be introduced under these conditions when using the nearest neighbor rule. We propose an adaptive nearest neighbor classification method to try to minimize bias. We use quasiconformal transformed kernels to compute neighborhoods over which the class probabilities tend to be more homogeneous. As a result, better classification performance can be expected. The efficacy of our method is validated and compared against other competing techniques using a variety of data sets.     

Large margin nearest neighbor classifiers

The nearest neighbor technique is a simple and appealing approach to addressing classification problems. It relies on the assumption of locally constant class conditional probabilities. This assumption becomes invalid in high dimensions with a finite number of examples due to the curse of dimensionality. Severe bias can be introduced under these conditions when using the nearest neighbor rule. The employment of a locally adaptive metric becomes crucial in order to keep class conditional probabilities close to uniform, thereby minimizing the bias of estimates. We propose a technique that computes a locally flexible metric by means of support vector machines (SVMs). The decision function constructed by SVMs is used to determine the most discriminant direction in a neighborhood around the query. Such a direction provides a local feature weighting scheme. We formally show that our method increases the margin in the weighted space where classification takes place. Moreover, our method has the important advantage of online computational efficiency over competing locally adaptive techniques for nearest neighbor classification. We demonstrate the efficacy of our method using both real and simulated data.     

Discriminant adaptive nearest neighbor classification

Nearest neighbour classification expects the class conditional probabilities to be locally constant, and suffers from bias in high dimensions. We propose a locally adaptive form of nearest neighbour classification to try to ameliorate this curse of dimensionality. We use a local linear discriminant analysis to estimate an effective metric for computing neighbourhoods. We determine the local decision boundaries from centroid information, and then shrink neighbourhoods in directions orthogonal to these local decision boundaries, and elongate them parallel to the boundaries. Thereafter, any neighbourhood-based classifier can be employed, using the modified neighbourhoods. The posterior probabilities tend to be more homogeneous in the modified neighbourhoods. We also propose a method for global dimension reduction, that combines local dimension information. In a number of examples, the methods demonstrate the potential for substantial improvements over nearest neighbour classification     

一种新的自适应尺度近邻分类器

基于多类别监督学习,提出了一种局部自适应最近邻分类器。此方法使用椭球聚类学习方法估计有效尺度,用于拉长特征不明显的维,并限制特征重要的维。在修正的领域中,类条件概率按预期近似为常数,从而得到更好的分类性能。实验结果显示,对多类问题,这是一种有效且鲁棒的分类方法。     

Improving nearest neighbor classification with cam weighted distance

Nearest neighbor (NN) classification assumes locally constant class conditional probabilities, and suffers from bias in high dimensions with a small sample set. In this paper, we propose a novel cam weighted distance to ameliorate the curse of dimensionality. Different from the existing neighborhood-based methods which only analyze a small space emanating from the query sample, the proposed nearest neighbor classification using the cam weighted distance (CamNN) optimizes the distance measure based on the analysis of inter-prototype relationship. Our motivation comes from the observation that the prototypes are not isolated. Prototypes with different surroundings should have different effects in the classification. The proposed cam weighted distance is orientation and scale adaptive to take advantage of the relevant information of inter-prototype relationship, so that a better classification performance can be achieved. Experiments show that CamNN significantly outperforms one nearest neighbor classification (1-NN) and k-nearest neighbor classification (k-NN) in most benchmarks, while its computational complexity is comparable with that of 1-NN classification.     

用于文本分类的改进KNN算法

最近邻分类器是假定局部的类条件概率不变,而这个假定在高维特征空间中无效。因此在高维特征空间中使用k最近邻分类器,不对特征权重进行修正就会引起严重的偏差。本文采用灵敏度法,利用前馈神经网络获得初始特征权重并进行二次降维。在初始权重下,根据样本间相似度采用SS树方法将训练样本划分成若干小区域,以此寻找待分类样本的近似k₀个最近邻,并根据近似k₀个最近邻和Chi-square距离原理计算新权重,搜索出新的k个最近邻。此方法在付出较小时间代价的情况下,在文本分离中可获得较好的分类精度的提高。     

On the calculation of control transition probabilities in a program

In the above examples we have considered some basic program constructs. We have assumed that the distribution functions of the needed variables are known at the entry point of each program construct. We have shown how the corresponding distribution functions change during the execution of the program construct. Based on these changing distribution functions the transition probabilities connected to the arcs of the control-flow graph can be calculated locally. Combining these constructs to programs does not give trouble: the needed probabilities are obtained as products of appropriate conditional probabilities. For a large class of programs, the distribution functions can be calculated passing through the control-flow graph in a step-by-step fashion.     

Nonparametric supervised learning by linear interpolation with maximum entropy

Nonparametric neighborhood methods for learning entail estimation of class conditional probabilities based on relative frequencies of samples that are "near-neighbors" of a test point. We propose and explore the behavior of a learning algorithm that uses linear interpolation and the principle of maximum entropy (LIME). We consider some theoretical properties of the LIME algorithm: LIME weights have exponential form; the estimates are consistent; and the estimates are robust to additive noise. In relation to bias reduction, we show that near-neighbors contain a test point in their convex hull asymptotically. The common linear interpolation solution used for regression on grids or look-up-tables is shown to solve a related maximum entropy problem. LIME simulation results support use of the method, and performance on a pipeline integrity classification problem demonstrates that the proposed algorithm has practical value.     

Incremental learning with sample queries

The classical theory of pattern recognition assumes labeled examples appear according to unknown underlying class conditional probability distributions where the pattern classes are picked randomly in a passive manner according to their a priori probabilities. This paper presents experimental results for an incremental nearest-neighbor learning algorithm which actively selects samples from different pattern classes according to a querying rule as opposed to the a priori probabilities. The amount of improvement of this query-based approach over the passive batch approach depends on the complexity of the Bayes rule     

Enhancing multi-label classification by modeling dependencies among labels

In this paper, we propose a novel framework for multi-label classification, which directly models the dependencies among labels using a Bayesian network. Each node of the Bayesian network represents a label, and the links and conditional probabilities capture the probabilistic dependencies among multiple labels. We employ our Bayesian network structure learning method, which guarantees to find the global optimum structure, independent of the initial structure. After structure learning, maximum likelihood estimation is used to learn the conditional probabilities among nodes. Any current multi-label classifier can be employed to obtain the measurements of labels. Then, using the learned Bayesian network, the true labels are inferred by combining the relationship among labels with the labels? estimates obtained from a current multi-labeling method. We further extend the proposed multi-label classification method to deal with incomplete label assignments. Structural Expectation-Maximization algorithm is adopted for both structure and parameter learning. Experimental results on two benchmark multi-label databases show that our approach can effectively capture the co-occurrent and the mutual exclusive relation among labels. The relation modeled by our approach is more flexible than the pairwise or fixed subset labels captured by current multi-label learning methods. Thus, our approach improves the performance over current multi-label classifiers. Furthermore, our approach demonstrates its robustness to incomplete multi-label classification.     

A decision fusion method using an algorithm for fusion of correlated probabilities

This paper proposes a new decision fusion method accounting for conditional dependence (correlation) between land-cover classifications from multi-sensor data. The dependence structure between different classification results is calculated and used as weighting parameters for the subsequent fusion scheme. An algorithm for fusion of correlated probabilities (FCP) is adopted to fuse the prior probability, conditional probability, and obtained weighting parameters to generate a posterior probability for each class. A maximum posterior probability rule is then used to combine the posterior probabilities generated for each class to produce the final fusion result. The proposed FCP-based decision fusion method is assessed in land-cover classification over two study areas. The experimental results demonstrate that the proposed decision fusion method outperformed the existing decision fusion methods that do not take into account the correlation or dependence. The proposed decision fusion method can also be applied to other applications with different sensor data.     

Adaptive hybrid learning for neural networks

A robust locally adaptive learning algorithm is developed via two enhancements of the Resilient Propagation (RPROP) method. Remaining drawbacks of the gradient-based approach are addressed by hybridization with gradient-independent Local Search. Finally, a global optimization method based on recursion of the hybrid is constructed, making use of tabu neighborhoods to accelerate the search for minima through diversification. Enhanced RPROP is shown to be faster and more accurate than the standard RPROP in solving classification tasks based on natural data sets taken from the UCI repository of machine learning databases. Furthermore, the use of Local Search is shown to improve Enhanced RPROP by solving the same classification tasks as part of the global optimization method.     

Locally strong coherence and inference with lower–upper probabilities

 We introduce an operational way to reduce the spatial complexity in inference processes based on conditional lower–upper probabilities assessments. To reach such goal we must suitably exploit zero probabilities taking account of logical conditions characterizing locally strong coherence. We actually re-formulate for conditional lower–upper probabilities the notion of locally strong coherence already introduced for conditional precise probabilities. Thanks to the characterization, we avoid to build all atoms, so that several real problems become feasible. In fact, the real complexity problem is connected to the number of atoms. Since for an inferential process with lower–upper probabilities several sequences of constraints must be fulfilled, our simplification can have either a “global” or a “partial” effect, being applicable to all or just to some sequences. The whole procedure has been implemented by XLisp-Stat language. A comparison with other approaches will be done by an example. The contribution of L. Galli has been essentially addressed to some aspects of the algorithm's implementation.     

The use of prior probabilities in maximum likelihood classification of remotely sensed data

The expected distribution of classes in a final classification map can be used to improve classification accuracies. Prior information is incorporated through the use of prior probabilities—that is, probabilities of occurrence of classes which are based on separate, independent knowledge concerning the area to be classified. The use of prior probabilities in a classification system is sufficiently versatile to allow (1) prior weighting of output classes based on their anticipated sizes; (2) the merging of continuously varying measurements (multispectral signatures) with discrete collateral information datasets (e.g., rock type, soil type); and (3) the construction of time-sequential classification systems in which an earlier classification modifies the outcome of a later one. The prior probabilities are incorporated by modifying the maximum likelihood decision rule employed in a Bayesian-type classifier to calculate a posteriori probabilities of class membership which are based not only on the resemblance of a pixel to the class signature, but also on the weight of the class which is estimated for the final output classification. In the merging of discrete collateral information with continuous spectral values into a single classification, a set of prior probabilities (weights) is estimated for each value which the discrete collateral variable may assume (e.g., each rock type or soil type). When maximum likelihood calculations are performed, the prior probabilities appropriate to the particular pixel are used in classification. For time-sequential classification, the prior classification of a pixel indexes a set of appropriate conditional probabilities reflecting either the confidence of the investigator in the prior classification or the extent to which the prior class identified is likely to change during the time period of interest.     

Attribute weighted Naive Bayes classifier using a local optimization

The Naive Bayes classifier is a popular classification technique for data mining and machine learning. It has been shown to be very effective on a variety of data classification problems. However, the strong assumption that all attributes are conditionally independent given the class is often violated in real-world applications. Numerous methods have been proposed in order to improve the performance of the Naive Bayes classifier by alleviating the attribute independence assumption. However, violation of the independence assumption can increase the expected error. Another alternative is assigning the weights for attributes. In this paper, we propose a novel attribute weighted Naive Bayes classifier by considering weights to the conditional probabilities. An objective function is modeled and taken into account, which is based on the structure of the Naive Bayes classifier and the attribute weights. The optimal weights are determined by a local optimization method using the quasisecant method. In the proposed approach, the Naive Bayes classifier is taken as a starting point. We report the results of numerical experiments on several real-world data sets in binary classification, which show the efficiency of the proposed method.     

Fast design of reduced-complexity nearest-neighbor classifiersusing triangular inequality

We propose a method of designing a reduced complexity nearest-neighbor classifier with near-minimal computational complexity from a given nearest-neighbor classifier that has high input dimensionality and a large number of class vectors. We applied our method to the classification problem of handwritten numerals in the NIST database. If the complexity of the RCNN classifier is normalized to that of the given classifier, the complexity of the derived classifier is 62 percent, 2 percent higher than that of the optimal classifier. This was found using the exhaustive search     

Sensitivity analysis in discrete Bayesian networks

This paper presents an efficient computational method for performing sensitivity analysis in discrete Bayesian networks. The method exploits the structure of conditional probabilities of a target node given the evidence. First, the set of parameters which is relevant to the calculation of the conditional probabilities of the target node is identified. Next, this set is reduced by removing those combinations of the parameters which either contradict the available evidence or are incompatible. Finally, using the canonical components associated with the resulting subset of parameters, the desired conditional probabilities are obtained. In this way, an important saving in the calculations is achieved. The proposed method can also be used to compute exact upper and lower bounds for the conditional probabilities, hence a sensitivity analysis can be easily performed. Examples are used to illustrate the proposed methodology     

Edit distance-based kernel functions for structural pattern classification

A common approach in structural pattern classification is to define a dissimilarity measure on patterns and apply a distance-based nearest-neighbor classifier. In this paper, we introduce an alternative method for classification using kernel functions based on edit distance. The proposed approach is applicable to both string and graph representations of patterns. By means of the kernel functions introduced in this paper, string and graph classification can be performed in an implicit vector space using powerful statistical algorithms. The validity of the kernel method cannot be established for edit distance in general. However, by evaluating theoretical criteria we show that the kernel functions are nevertheless suitable for classification, and experiments on various string and graph datasets clearly demonstrate that nearest-neighbor classifiers can be outperformed by support vector machines using the proposed kernel functions.     

Decentralized adaptive synchronization of an uncertain complex delayed dynamical network

In this paper, we investigate the locally and globally adaptive synchronization problem for an uncertain complex dynamical network with time-varying coupling delays based on the decentralized control. The coupling terms here are bounded by high-order polynomials with known gains that are ubiquitous in a large class of complex dynamical networks. We generalize the usual technology of searching for an appropriate coordinates transformation to change the network dynamics into a series of decoupled lower-dimensional systems. Several adaptive synchronization criteria are derived by constructing the Lyapunov-Krasovskii functional and Barbalat lemma, and the proposed criteria are simple in form and convenient for the practical engineering design. Numerical simulations illustrated by a nearest-neighbor coupling network verify the effectiveness of the proposed synchronization scheme.