On the algorithmic implementation of stochastic discrimination

Stochastic discrimination is a general methodology for constructing classifiers appropriate for pattern recognition. It is based on combining arbitrary numbers of very weak components, which are usually generated by some pseudorandom process, and it has the property that the very complex and accurate classifiers produced in this way retain the ability, characteristic of their weak component pieces, to generalize to new data. In fact, it is often observed, in practice, that classifier performance on test sets continues to rise as more weak components are added, even after performance on training sets seems to have reached a maximum. This is predicted by the underlying theory, for even though the formal error rate on the training set may have reached a minimum, more sophisticated measures intrinsic to this method indicate that classifier performance on both training and test sets continues to improve as complexity increases. We begin with a review of the method of stochastic discrimination as applied to pattern recognition. Through a progression of examples keyed to various theoretical issues, we discuss considerations involved with its algorithmic implementation. We then take such an algorithmic implementation and compare its performance, on a large set of standardized pattern recognition problems from the University of California Irvine, and Statlog collections, to many other techniques reported on in the literature, including boosting and bagging. In doing these studies, we compare our results to those reported in the literature by the various authors for the other methods, using the same data and study paradigms used by them. Included in the paper is an outline of the underlying mathematical theory of stochastic discrimination and a remark concerning boosting, which provides a theoretical justification for properties of that method observed in practice, including its ability to generalize     

Representational accuracy of stochastic neural populations.

Fisher information is used to analyze the accuracy with which a neural population encodes D stimulus features. It turns out that the form of response variability has a major impact on the encoding capacity and therefore plays an important role in the selection of an appropriate neural model. In particular, in the presence of baseline firing, the reconstruction error rapidly increases with D in the case of Poissonian noise but not for additive noise. The existence of limited-range correlations of the type found in cortical tissue yields a saturation of the Fisher information content as a function of the population size only for an additive noise model. We also show that random variability in the correlation coefficient within a neural population, as found empirically, considerably improves the average encoding quality. Finally, the representational accuracy of populations with inhomogeneous tuning properties, either with variability in the tuning widths or fragmented into specialized subpopulations, is superior to the case of identical and radially symmetric tuning curves usually considered in the literature.     

Strive for accuracy—improvement of predictions

Finite-difference approximations to the governing equations of fluid motinos are discussed for viscous three-dimensional flows for Reynolds numbers of several hundred, for three-dimensional incompressible and compressible boundary layers and for inviscid near-sonic supersonic flows. It is shown how the boundary conditions of the problems chosen influence the solutions. If certain conditions to be discussed in the following are not met, neither convergence nor uniqueness of the solution can be guaranteed. Comparison of the predictions described herein with experimental data of the recent literature asserts the validity of the solutions given. The examples chosen include: Viscous flows through biological vessels with bends and bifurcations, formation of Taylor-Görtler vortices in spherical gaps, three-dimensional viscous displacement effect on wings in transonic flow and front and embedded shocks in flow fields with Mach numbers slightly above one.     

On the convergence and improvement of stochastic normalized gradient descent

Non-convex models,like deep neural networks,have been widely used in machine learning applications.Training non-convex models is a difficult task owing to the s...     

On the accuracy of the stochastic simulation of infinite-horizonLQG control systems

The simulation accuracy of the infinite-horizon sampled-data linear-quadratic Gaussian (LQG) control system is investigated. The effect of the proper selection of the noise statistics, their dependence on the integration routines, and their effects on robustness recovery are addressed. Estimation error variances for the LQG control are compared     

Data preprocessing techniques for classification without discrimination

Recently, the following Discrimination-Aware Classification Problem was introduced: Suppose we are given training data that exhibit unlawful discrimination; e.g., toward sensitive attributes such as gender or ethnicity. The task is to learn a classifier that optimizes accuracy, but does not have this discrimination in its predictions on test data. This problem is relevant in many settings, such as when the data are generated by a biased decision process or when the sensitive attribute serves as a proxy for unobserved features. In this paper, we concentrate on the case with only one binary sensitive attribute and a two-class classification problem. We first study the theoretically optimal trade-off between accuracy and non-discrimination for pure classifiers. Then, we look at algorithmic solutions that preprocess the data to remove discrimination before a classifier is learned. We survey and extend our existing data preprocessing techniques, being suppression of the sensitive attribute, massaging the dataset by changing class labels, and reweighing or resampling the data to remove discrimination without relabeling instances. These preprocessing techniques have been implemented in a modified version of Weka and we present the results of experiments on real-life data.     

Comparison of different classification algorithms for underwater target discrimination

Classification of underwater targets from the acoustic backscattered signals is considered. Several different classification algorithms are tested and benchmarked not only for their performance but also to gain insight to the properties of the feature space. Results on a wideband 80-kHz acoustic backscattered data set collected for six different objects are presented in terms of the receiver operating characteristic (ROC) and robustness of the classifiers wrt reverberation.     

Dual criterion stochastic optimal control problem for robustness improvement

The solution of the dual criterion linear quadratic stochastic optimal control problem is obtained by following a Wiener type of solution procedure. A stabilizing solution is guaranteed by parameterizing the controller using the Desoer fractional representation approach. The dual criterion includes sensitivity and complementary sensitivity weighting terms which provide a means of varying the robustness characteristics of the multivariable system.     

Analysis of classification accuracy for pre-filtered multichannel remote sensing data

Noise is one of the main factors degrading the quality of original multichannel remote sensing data and its presence influences classification efficiency, object detection, etc. Thus, pre-filtering is often used to remove noise and improve the solving of final tasks of multichannel remote sensing. Recent studies indicate that a classical model of additive noise is not adequate enough for images formed by modern multichannel sensors operating in visible and infrared bands. However, this fact is often ignored by researchers designing noise removal methods and algorithms. Because of this, we focus on the classification of multichannel remote sensing images in the case of signal-dependent noise present in component images. Three approaches to filtering of multichannel images for the considered noise model are analysed, all based on discrete cosine transform in blocks. The study is carried out not only in terms of conventional efficiency metrics used in filtering (MSE) but also in terms of multichannel data classification accuracy (probability of correct classification, confusion matrix). The proposed classification system combines the pre-processing stage where a DCT-based filter processes the blocks of the multichannel remote sensing image and the classification stage. Two modern classifiers are employed, radial basis function neural network and support vector machines. Simulations are carried out for three-channel image of Landsat TM sensor. Different cases of learning are considered: using noise-free samples of the test multichannel image, the noisy multichannel image and the pre-filtered one. It is shown that the use of the pre-filtered image for training produces better classification in comparison to the case of learning for the noisy image. It is demonstrated that the best results for both groups of quantitative criteria are provided if a proposed 3D discrete cosine transform filter equipped by variance stabilizing transform is applied. The classification results obtained for data pre-filtered in different ways are in agreement for both considered classifiers. Comparison of classifier performance is carried out as well. The radial basis neural network classifier is less sensitive to noise in original images, but after pre-filtering the performance of both classifiers is approximately the same.     

Some statistical bounds for the accuracy of distance-based pattern classification

Before implementing a pattern recognition algorithm, a rational step is to estimate its validity by bounding the probability of error. The ability to make such an estimate impacts crucially on the satisfactoriness of the particular features used, on the number of samples required to train and test the system and on the overall paradigm. This study develops statistical upper and lower bounds for estimating the probability of error, in the one-dimensional case. The bounds are distribution-free except for requiring the existence of the relevant statistics and can be evaluated easily by hand or by computer. Many of the results are also applicable to other problems involving the estimation of an arbitrary distribution of a random variable. Some multidimensional generalizations may be feasible.     

Binary classification by stochastic neural nets

We classify points in R(d) (feature vector space) by functions related to feedforward artificial neural networks. These functions, dubbed "stochastic neural nets", arise in a natural way from probabilistic as well as from statistical considerations. The probabilistic idea is to define a classifying bit locally by using the sign of a hidden state-dependent noisy linear function of the feature vector as a new (d+1)th coordinate of the vector. This (d+1)-dimensional distribution is approximated by a mixture distribution. The statistical idea is that the approximating mixtures, and hence the a posteriori class probability functions (stochastic neural nets) defined by them, can be conveniently trained either by maximum likelihood or by a Bayes criterion through the use of an appropriate expectation-maximization algorithm.     

On stochastic Riccati equations for the stochastic LQR problem

This paper studies an approximation of stochastic Riccati equations for stochastic LQR problems some of which may be even with indefinite control weight costs.     

Triply stochastic gradient method for large-scale nonlinear similar unlabeled classification

Machine Learning - Similar unlabeled (SU) classification is pervasive in many real-world applications, where only similar data pairs (two data points have the same label) and unlabeled data points...     

A method for improving the accuracy of data mining classification algorithms

In this paper we introduce a method called CL.E.D.M. (CLassification through ELECTRE and Data Mining), that employs aspects of the methodological framework of the ELECTRE I outranking method, and aims at increasing the accuracy of existing data mining classification algorithms. In particular, the method chooses the best decision rules extracted from the training process of the data mining classification algorithms, and then it assigns the classes that correspond to these rules, to the objects that must be classified. Three well known data mining classification algorithms are tested in five different widely used databases to verify the robustness of the proposed method.     

Potential improvement of classifier accuracy by using fuzzymeasures

Typical digit recognizers classify an unknown digit pattern by computing its distance from the cluster centers in a feature space. In this paper, we propose a methodology that has many salient aspects. First, the classification rule is dependent on the “difficulty” of the unknown sample. Samples “far” from the center, which tend to fall on the boundaries of classes are error prone and, hence, “difficult”. An “overlapping zone” is defined in the feature space to identify such difficult samples. A table is precomputed to facilitate an efficient lookup of the class corresponding to all the points in the overlapping zone. The lookup function itself is defined by a modification of the KNN rule. A characteristic function defining the new boundaries is computed using the topology of the set of samples in the overlapping zones. Our two-pronged approach uses different classification schemes with the “difficult” and “easy” samples. The method described has improved the performance of the gradient structural concavity digit recognizer described by Favata et al. (1996)     

On stochastic approximation algorithms for classes of PAC learning problems.

The classical stochastic approximation methods are shown to yield algorithms to solve several formulations of the PAC learning problem defined on the domain [0,1](d). Under some smoothness conditions on the probability measure functions, simple algorithms to solve some PAC learning problems are proposed based on networks of nonpolynomial units (e.g. artificial neural networks). Conditions on the sizes of the samples required to ensure the error bounds are derived using martingale inequalities.     

On stochastic controllability for nonlinear systems

Both the stochastic ε-controllability and the stochastic controllability with probability one are first defined. Second, by using a stochastic Lyapunov-like approach, several theorems are developed which give sufficient conditions for the stochastic controllability defined for an important class of nonlinear stochastic systems. A theorem of stochastic uncontrollability is also presented, giving sufficient conditions for stochastic uncontrollability for a class of nonlinear systems. Finally, the relation between the deterministic controllability and the stochastic one is comparatively discussed.     

Texture discrimination based upon an assumed stochastic texture model

A new approach to texture discrimination is described. This approach is based upon an assumed stochastic model for texture in imagery and is an approximation to the statistically optimum maximum likelihood classifier. The construction and properties of the stochastic texture model are described and a digital filtering implementation of the resulting maximum likelihood texture discriminant is provided. The efficacy of this approach is demonstrated through experimental results obtained with simulated texture data. A comparison is provided with more conventional texture discriminants under identical conditions. The implications to texture discrimination in realworld imagery are discussed.     

Fields as limit functions of stochastic discrimination and their adaptability

For a particular type of elementary function, stochastic discrimination is shown to have an analytic limit function. Classifications can be performed directly by this limit function instead of by a sampling procedure. The limit function has an interpretation in terms of fields that originate from the training examples of a classification problem. Fields depend on the global configuration of the training points. The classification of a point in input space is known when the contributions of all fields are summed. Two modifications of the limit function are proposed. First, for nonlinear problems like high-dimensional parity problems, fields can be quantized. This leads to classification functions with perfect generalization for high-dimensional parity problems. Second, fields can be provided with adaptable amplitudes. The classification corresponding to a limit function is taken as an initialization; subsequently, amplitudes are adapted until an error function for the test set reaches minimal value. It is illustrated that this increases the performance of stochastic discrimination. Due to the nature of the fields, generalization improves even if the amplitude of every training example is adaptable.     

A meta-heuristic approach for improving the accuracy in some classification algorithms

Current classification algorithms usually do not try to achieve a balance between fitting and generalization when they infer models from training data. Furthermore, current algorithms ignore the fact that there may be different penalty costs for the false-positive, false-negative, and unclassifiable types. Thus, their performance may not be optimal or may even be coincidental. This paper proposes a meta-heuristic approach, called the Convexity Based Algorithm (CBA), to address these issues. The new approach aims at optimally balancing the data fitting and generalization behaviors of models when some traditional classification approaches are used. The CBA first defines the total misclassification cost (TC) as a weighted function of the three penalty costs and the corresponding error rates as mentioned above. Next it partitions the training data into regions. This is done according to some convexity properties derivable from the training data and the traditional classification method to be used in conjunction with the CBA. Next the CBA uses a genetic approach to determine the optimal levels of fitting and generalization. The TC is used as the fitness function in this genetic approach. Twelve real-life datasets from a wide spectrum of domains were used to better understand the effectiveness of the proposed approach. The computational results indicate that the CBA may potentially fill in a critical gap in the use of current or future classification algorithms.