Probabilistic Modeling and Recognition of 3-D Objects

This paper introduces a uniform statistical framework for both 3-D and 2-D object recognition using intensity images as input data. The theoretical part provides a mathematical tool for stochastic modeling. The algorithmic part introduces methods for automatic model generation, localization, and recognition of objects. 2-D images are used for learning the statistical appearance of 3-D objects; both the depth information and the matching between image and model features are missing for model generation. The implied incomplete data estimation problem is solved by the Expectation Maximization algorithm. This leads to a novel class of algorithms for automatic model generation from projections. The estimation of pose parameters corresponds to a non-linear maximum likelihood estimation problem which is solved by a global optimization procedure. Classification is done by the Bayesian decision rule. This work includes the experimental evaluation of the various facets of the presented approach. An empirical evaluation of learning algorithms and the comparison of different pose estimation algorithms show the feasibility of the proposed probabilistic framework.     

决定图框架下本体学习算法的稳定性分析

传统的本体算法采用启发式的方法来计算语义相似度,而随着本体处理数据量的日益增大,越来越多的机器学习方法被用于本体函数的获取。稳定性是本体学习算法的必要条件,它要求在本体样本集做轻微改动的情况下不会对得到的最优本体函数产生本质的改变。文中研究了在本体样本集的依赖关系由图结构决定的框架下,本体学习算法的稳定性和对应的统计学特征。首先对传统的PO和LTO一致稳定性条件进行分析;其次在大样本情况下扩展一致稳定性条件,提出Pk和LkO一致稳定性并得到相关的理论结果;最后把替换本体样本和删除本体样本两种样本进行变换组合,提出在大本体样本前提下的组合一致稳定性概念,并利用统计学习理论的方法得到一般结果。此外,在各类稳定性条件下,对满足m-独立条件的本体学习算法的广义界进行了讨论。     

Quality assessment of individual classifications in machine learning and data mining

Although in the past machine learning algorithms have been successfully used in many problems, their serious practical use is affected by the fact that often they cannot produce reliable and unbiased assessments of their predictions' quality. In last few years, several approaches for estimating reliability or confidence of individual classifiers have emerged, many of them building upon the algorithmic theory of randomness, such as (historically ordered) transduction-based confidence estimation, typicalness-based confidence estimation, and transductive reliability estimation. Unfortunately, they all have weaknesses: either they are tightly bound with particular learning algorithms, or the interpretation of reliability estimations is not always consistent with statistical confidence levels. In the paper we describe typicalness and transductive reliability estimation frameworks and propose a joint approach that compensates the above-mentioned weaknesses by integrating typicalness-based confidence estimation and transductive reliability estimation into a joint confidence machine. The resulting confidence machine produces confidence values in the statistical sense. We perform series of tests with several different machine learning algorithms in several problem domains. We compare our results with that of a proprietary method as well as with kernel density estimation. We show that the proposed method performs as well as proprietary methods and significantly outperforms density estimation methods. Matjaž Kukar is currently Assistant Professor in the Faculty of Computer and Information Science at University of Ljubljana. His research interests include machine learning, data mining and intelligent data analysis, ROC analysis, cost-sensitive learning, reliability estimation, and latent structure analysis, as well as applications of data mining in medical and business problems.     

线性支持向量机优化问题的极大熵方法

支特向量机是一种新的机器学习方法，已成功地应用于模式分类、回归分析和密度估计等问题中．本文依据统计学习理论和最优化理论建立了线性支特向量机的无约束优化模型，并给出了一种有效的近似解法一极大熵方法,为求解支持向量机优化问题提供了一种新途径,本文方法特别易于计算机实现。数值实验结果表明了模型和算法的可行性和有效性．     

Support vector machine applications in the field of hydrology: A review

In the recent few decades there has been very significant developments in the theoretical understanding of Support vector machines (SVMs) as well as algorithmic strategies for implementing them, and applications of the approach to practical problems. SVMs introduced by Vapnik and others in the early 1990s are machine learning systems that utilize a hypothesis space of linear functions in a high dimensional feature space, trained with optimization algorithms that implements a learning bias derived from statistical learning theory. This paper reviews the state-of-the-art and focuses over a wide range of applications of SVMs in the field of hydrology. To use SVM aided hydrological models, which have increasingly extended during the last years; comprehensive knowledge about their theory and modelling approaches seems to be necessary. Furthermore, this review provides a brief synopsis of the techniques of SVMs and other emerging ones (hybrid models), which have proven useful in the analysis of the various hydrological parameters. Moreover, various examples of successful applications of SVMs for modelling different hydrological processes are also provided.     

Adaptive algorithms with filtered regressor and filtered error

This paper presents a unified framework for the analysis of several discrete time adaptive parameter estimation algorithms, including RML with nonvanishing stepsize, several ARMAX identifiers, the Landau-style output error algorithms, and certain others for which no stability proof has yet appeared. A general algorithmic form is defined, incorporating a linear time-varying regressor filter and a linear time-varying error filter. Local convergence of the parameters in nonideal (or noisy) environments is shown via averaging theory under suitable assumptions of persistence of excitation, small stepsize, and passivity. The excitation conditions can often be transferred to conditions on external signals, and a small stepsize is appropriate in a wide range of applications. The required passivity is demonstrated for several special cases of the general algorithm. The first and third authors were supported by NSF Grants ECS-8506149, INT-8513400, and MIP-8608787. Research done while at the School of Electrical Engineering, Cornell University, Ithaca, New York 14853, U.S.A.     

支持向量机训练和实现算法综述

支持向量机是在统计学习理论基础上发展起来的一种新的机器学习方法,支持向量机已成为目前研究的热点,并在模式识别、回归分析、函数估计等领域有了广泛的应用。该文在介绍了支持向量机的目前研究、应用状况和新进展的基础上,对支持向量机训练和实现算法进行了综述,最后指出了进一步研究和应用亟待解决的一些问题。     

Unified development of algorithms used for linear predictive coding of speech signals

The requirement for low data rate voice transmission has resulted in a large number of algorithms being proposed for speech digitization at data rates of 2·4–4 kilobits/sec. Many of the proposed algorithms are quite complicated and have their origin in disciplines generally considered to be outside of the realm of the speech researcher or communication system designer. Additionally, the algorithms have been developed and presented in highly varying notation using various theoretical approaches. The result is a confusing array of equations, algorithms, and numerical analysis procedure. It is the goal of this paper to alleviate this problem by providing a unified tutorial development of the various algorithms used and proposed for speech data compression.Classical least squares estimation theory is used as the focal point of the discussion since it forms the basis for several of the more familiar speech digitization algorithms. The remainder of the algorithms, whether they have their basis in stochastic estimation theory or statistical regression theory, are related back to the more familiar least squares approach. The speech digitization techniques discussed are the covariance method, the autocorrelation method, the PARCOR method, a priori analysis, the sequential least squares method, the Kalman filter approach, the stochastic approximation method, and the general linear regression model. An effort has been made to provide sufficient theoretical background to establish the algorithm relationships without stressing mathematical rigor.     

An introduction to kernel-based learning algorithms

This paper provides an introduction to support vector machines, kernel Fisher discriminant analysis, and kernel principal component analysis, as examples for successful kernel-based learning methods. We first give a short background about Vapnik-Chervonenkis theory and kernel feature spaces and then proceed to kernel based learning in supervised and unsupervised scenarios including practical and algorithmic considerations. We illustrate the usefulness of kernel algorithms by discussing applications such as optical character recognition and DNA analysis.     

Probability estimation for multi-class classification using AdaBoost

It is a general viewpoint that AdaBoost classifier has excellent performance on classification problems but could not produce good probability estimations. In this paper we put forward a theoretical analysis of probability estimation model and present some verification experiments, which indicate that AdaBoost can be used for probability estimation. With the theory, we suggest some useful measures for using AdaBoost algorithms properly. And then we deduce a probability estimation model for multi-class classification by pairwise coupling. Unlike previous approximate methods, we provide an analytical solution instead of a special iterative procedure. Moreover, a new problem that how to get a robust prediction with classifier scores is proposed. Experiments show that the traditional predict framework, which chooses one with the highest score from all classes as the prediction, is not always good while our model performs well.     

Approximate string matching for music analysis

Approximate pattern matching algorithms have become an important tool in computer assisted music analysis and information retrieval. The number of different problem formulations has greatly expanded in recent years, not least because of the subjective nature of measuring musical similarity. From an algorithmic perspective, the complexity of each problem depends crucially on the exact definition of the difference between two strings. We present an overview of advances in approximate string matching in this field focusing on new measures of approximation.     

Factorial hidden Markov models and the generalized backfitting algorithm

Previous researchers developed new learning architectures for sequential data by extending conventional hidden Markov models through the use of distributed state representations. Although exact inference and parameter estimation in these architectures is computationally intractable, Ghahramani and Jordan (1997) showed that approximate inference and parameter estimation in one such architecture, factorial hidden Markov models (FHMMs), is feasible in certain circumstances. However, the learning algorithm proposed by these investigators, based on variational techniques, is difficult to understand and implement and is limited to the study of real-valued data sets. This chapter proposes an alternative method for approximate inference and parameter estimation in FHMMs based on the perspective that FHMMs are a generalization of a well-known class of statistical models known as generalized additive models (GAMs; Hastie & Tibshirani, 1990). Using existing statistical techniques for GAMs as a guide, we have developed the generalized backfitting algorithm. This algorithm computes customized error signals for each hidden Markov chain of an FHMM and then trains each chain one at a time using conventional techniques from the hidden Markov models literature. Relative to previous perspectives on FHMMs, we believe that the viewpoint taken here has a number of advantages. First, it places FHMMs on firm statistical foundations by relating them to a class of models that are well studied in the statistics community, yet it generalizes this class of models in an interesting way. Second, it leads to an understanding of how FHMMs can be applied to many different types of time-series data, including Bernoulli and multinomial data, not just data that are real valued. Finally, it leads to an effective learning procedure for FHMMs that is easier to understand and easier to implement than existing learning procedures. Simulation results suggest that FHMMs trained with the generalized backfitting algorithm are a practical and powerful tool for analyzing sequential data.     

Toward the border between neural and Markovian paradigms

A new tendency in the design of modern signal processing methods is the creation of hybrid algorithms. This paper gives an overview of different signal processing algorithms situated halfway between Markovian and neural paradigms. A new systematic way to classify these algorithms is proposed. Four specific classes of models are described. The first one is made up of algorithms based upon either one of the two paradigms, but including some parts of the other one. The second class includes algorithms proposing a parallel or sequential cooperation of two independent Markovian and neural parts. The third class tends to show Markov models (MMs) as a special case of neural networks (NNs), or conversely NNs as a special case of MMs. These algorithms concentrate mainly on bringing together respective learning methods. The fourth class of algorithms are hybrids, neither purely Markovian nor neural. They can be seen as belonging to a more general class of models, presenting features from both paradigms. The first two classes essentially include models with structural modifications, while two later classes propose algorithmic modifications. For the sake of clarity, only main mathematical formulas are given. Specific applications are intentionally avoided to give a wider view of the subject. The references provide more details for interested readers.     

Nonnegative matrix factorization with the Itakura-Saito divergence: with application to music analysis

This letter presents theoretical, algorithmic, and experimental results about nonnegative matrix factorization (NMF) with the Itakura-Saito (IS) divergence. We describe how IS-NMF is underlaid by a well-defined statistical model of superimposed gaussian components and is equivalent to maximum likelihood estimation of variance parameters. This setting can accommodate regularization constraints on the factors through Bayesian priors. In particular, inverse-gamma and gamma Markov chain priors are considered in this work. Estimation can be carried out using a space-alternating generalized expectation-maximization (SAGE) algorithm; this leads to a novel type of NMF algorithm, whose convergence to a stationary point of the IS cost function is guaranteed. We also discuss the links between the IS divergence and other cost functions used in NMF, in particular, the Euclidean distance and the generalized Kullback-Leibler (KL) divergence. As such, we describe how IS-NMF can also be performed using a gradient multiplicative algorithm (a standard algorithm structure in NMF) whose convergence is observed in practice, though not proven. Finally, we report a furnished experimental comparative study of Euclidean-NMF, KL-NMF, and IS-NMF algorithms applied to the power spectrogram of a short piano sequence recorded in real conditions, with various initializations and model orders. Then we show how IS-NMF can successfully be employed for denoising and upmix (mono to stereo conversion) of an original piece of early jazz music. These experiments indicate that IS-NMF correctly captures the semantics of audio and is better suited to the representation of music signals than NMF with the usual Euclidean and KL costs.     

Algorithms as co-workers: Human algorithm role interactions in algorithmic work

In algorithmic work, algorithms execute operational and management tasks such as work allocation, task tracking and performance evaluation. Humans and algorithms interact with one another to accomplish work so that the algorithm takes on the role of a co-worker. Human–algorithm interactions are characterised by problematic issues such as absence of mutually co-constructed dialogue, lack of transparency regarding how algorithmic outputs are generated, and difficulty of over-riding algorithmic directive – conditions that create lack of clarity for the human worker. This article examines human–algorithm role interactions in algorithmic work. Drawing on the theoretical framing of organisational roles, we theorise on the algorithm as role sender and the human as the role taker. We explain how the algorithm is a multi-role sender with entangled roles, while the human as role taker experiences algorithm-driven role conflict and role ambiguity. Further, while the algorithm records all of the human's task actions, it is ignorant of the human's cognitive reactions – it undergoes what we conceptualise as ‘broken loop learning’. The empirical context of our study is algorithm-driven taxi driving (in the United States) exemplified by companies such as Uber. We draw from data that include interviews with 15 Uber drivers, a netnographic study of 1700 discussion threads among Uber drivers from two popular online forums, and analysis of Uber's web pages. Implications for IS scholarship, practice and policy are discussed.     

An algorithmic theory of learning: Robust concepts and random projection

We study the phenomenon of cognitive learning from an algorithmic standpoint. How does the brain effectively learn concepts from a small number of examples despite the fact that each example contains a huge amount of information? We provide a novel algorithmic analysis via a model of robust concept learning (closely related to “margin classifiers”), and show that a relatively small number of examples are sufficient to learn rich concept classes. The new algorithms have several advantages—they are faster, conceptually simpler, and resistant to low levels of noise. For example, a robust half-space can be learned in linear time using only a constant number of training examples, regardless of the number of attributes. A general (algorithmic) consequence of the model, that “more robust concepts are easier to learn”, is supported by a multitude of psychological studies.     

A cost-function approach to rival penalized competitive learning (RPCL).

Rival penalized competitive learning (RPCL) has been shown to be a useful tool for clustering on a set of sample data in which the number of clusters is unknown. However, the RPCL algorithm was proposed heuristically and is still in lack of a mathematical theory to describe its convergence behavior. In order to solve the convergence problem, we investigate it via a cost-function approach. By theoretical analysis, we prove that a general form of RPCL, called distance-sensitive RPCL (DSRPCL), is associated with the minimization of a cost function on the weight vectors of a competitive learning network. As a DSRPCL process decreases the cost to a local minimum, a number of weight vectors eventually fall into a hypersphere surrounding the sample data, while the other weight vectors diverge to infinity. Moreover, it is shown by the theoretical analysis and simulation experiments that if the cost reduces into the global minimum, a correct number of weight vectors is automatically selected and located around the centers of the actual clusters, respectively. Finally, we apply the DSRPCL algorithms to unsupervised color image segmentation and classification of the wine data.     

From blind signal extraction to blind instantaneous signal separation: criteria, algorithms, and stability

This paper reports a study on the problem of the blind simultaneous extraction of specific groups of independent components from a linear mixture. This paper first presents a general overview and unification of several information theoretic criteria for the extraction of a single independent component. Then, our contribution fills the theoretical gap that exists between extraction and separation by presenting tools that extend these criteria to allow the simultaneous blind extraction of subsets with an arbitrary number of independent components. In addition, we analyze a family of learning algorithms based on Stiefel manifolds and the natural gradient ascent, present the nonlinear optimal activations (score) functions, and provide new or extended local stability conditions. Finally, we illustrate the performance and features of the proposed approach by computer-simulation experiments.     

Fuzzy identification using fuzzy neural networks with stable learning algorithms

In general, fuzzy neural networks cannot match nonlinear systems exactly. Unmodeled dynamic leads parameters drift and even instability problem. According to system identification theory, robust modification terms must be included in order to guarantee Lyapunov stability. This paper suggests new learning laws for Mamdani and Takagi-Sugeno-Kang type fuzzy neural networks based on input-to-state stability approach. The new learning schemes employ a time-varying learning rate that is determined from input-output data and model structure. Stable learning algorithms for the premise and the consequence parts of fuzzy rules are proposed. The calculation of the learning rate does not need any prior information such as estimation of the modeling error bounds. This offer an advantage compared to other techniques using robust modification.