Combining linear discriminant functions with neural networks for supervised learning

A novel supervised learning method is proposed by combining linear discriminant functions with neural networks. The proposed method results in a tree-structured hybrid architecture. Due to constructive learning, the binary tree hierarchical architecture is automatically generated by a controlled growing process for a specific supervised learning task. Unlike the classic decision tree, the linear discriminant functions are merely employed in the intermediate level of the tree for heuristically partitioning a large and complicated task into several smaller and simpler subtasks in the proposed method. These subtasks are dealt with by component neural networks at the leaves of the tree accordingly. For constructive learning, growing and credit-assignment algorithms are developed to serve for the hybrid architecture. The proposed architecture provides an efficient way to apply existing neural networks (e.g. multi-layered perceptron) for solving a large scale problem. We have already applied the proposed method to a universal approximation problem and several benchmark classification problems in order to evaluate its performance. Simulation results have shown that the proposed method yields better results and faster training in comparison with the multilayered perceptron.     

Collaborative filtering with maximum entropy

As users navigate through online document collections on high-volume Web servers, they depend on good recommendations. We present a novel maximum-entropy algorithm for generating accurate recommendations and a data-clustering approach for speeding up model training. Recommender systems attempt to automate the process of "word of mouth" recommendations within a community. Typical application environments such as online shops and search engines have many dynamic aspects.     

Batch map extensions of the kernel-based maximum entropy learning rule

In this letter, two batch-map extensions are described for the kernel-based maximum entropy learning rule (kMER). In the first, the weights are iteratively set to weighted component-wise medians, while in the second the generalized median is used, enabling kMER to process symbolic data. Simulations are performed to illustrate the extensions.     

最大熵模型在邮件分类中的应用

邮件分类是指在给定的分类体系下,根据邮件的内容和属性,确定其类别标签的过程。将最大熵模型应用于邮件分类中,给出了邮件的预处理过程,介绍了邮件信头特征,分析比较了特征数量和迭代次数、邮件特征字段对分类结果的影响,以及对层次分类和平面分类的效果进行了比较。实验表明,特征数量和迭代次数分别取2 000和250时为宜;充分利用邮件各字段信息,取得的总体分类效果最好,但对合法邮件,利用邮件头及邮件标题却取得了最好结果,并在层次分类中验证了这点,层次分类效果要优于平面分类。最后进行了总结和展望。     

Visual object recognition with supervised learning

A component-based approach to visual object recognition rooted in supervised learning allows for a vision system that is more robust against changes in an object's pose or illumination. Learning figures prominently in the study of visual systems from the viewpoints of visual neuroscience and computer vision. Whereas visual neuroscience concentrates on mechanisms that let the cortex adapt its circuitry and learn a new task, computer vision aims at devising effectively trainable systems. Vision systems that learn and adapt are one of the most important trends in computer vision research. They might offer the only solution to developing robust, reusable vision systems.     

Nonparametric linear tests with multiple events

A generalization of the nonparametric linear rank statistics is presented to handle the two-group comparison with multiple events. For a sample divided into two groups, in which each subject may experience at least two distinct failures, the logrank tests are extended to test the null hypothesis that the vector of the marginal survival distributions of the first group equals that of the second group. Two cases are distinguished depending on whether the null hypothesis does or does not imply the equality of the joint survival functions. In both cases, under the null hypothesis, the asymptotic joint distribution of the vector of the marginal statistics is shown to be Gaussian with covariance matrix consistently estimated using martingale properties. These theoretical results are illustrated by a simulation study and an application on the German Breast Cancer data. An extension to multiple hypotheses testing in multivariate proportional hazards models is also developed.     

Nonparametric interpolation of the Markov sequence

Consideration was given to the problem of interpolation (smoothing) of the nonobservable component of the composite Markov process within the framework of the conditional Markov scheme. In the case of the dynamic observation models such as autoregression, equations were derived for the a posteriori interpolation density of the probability of the state of the nonobservable component. The aim of the present paper was to construct a smoothing algorithm for an unknown family of the distributions of the nonobservable component of the partially observable random Markov sequence. The result was obtained for the strictly stationary random Markov processes with mixing and for the conditional densities in the observation model from the exponential family of distributions. Computer-aided modeling within the framework of the Kalman scheme demonstrated that the sampled root-mean-square error of the nonparametric smoothing algorithm constructed for an unknown state equation was situated between the errors of the optimal linear filtration and the optimal linear interpolation.     

Continuous affect recognition with weakly supervised learning

Multimedia Tools and Applications - Recognizing a person’s affective state from audio-visual signals is an essential capability for intelligent interaction. Insufficient training data and the...     

约束条件下BN参数最大熵模型扩展学习算法

在很多智能系统的参数建模时,用户往往面对建模样本稀少的困境。针对在小数据集条件下贝叶斯网络（BN）参数建模的问题,提出了一种约束数据最大熵BN参数学习算法（CDME）。首先利用小数据集估算BN参数,随后把定性的专家经验转换为不等式约束,并利用Bootstrap算法生成满足约束的一组参数候选集,再根据信息最大熵进行加权计算出BN参数。实验结果表明,当数据量充分时,CDME参数学习算法与经典的MLE算法的学习精度近似,表明了算法的正确性;在小数据集条件下,利用CDME算法,可以对BN进行参数建模,学习精度优于MLE算法和QMAP算法。CDME算法在实际故障诊断样本数据相对稀缺的条件下,获取了诊断BN模型参数,在此基础上完成的诊断推理结果也印证了算法的有效性,为小数据集条件下的参数建模提供了一条新途径。     

Credit rating modelling by kernel-based approaches with supervised and semi-supervised learning

This paper presents the modelling possibilities of kernel-based approaches to a complex real-world problem, i.e. corporate and municipal credit rating classification. Based on a model design that includes data pre-processing, the labelling of individual parameter vectors using expert knowledge, the design of various support vector machines with supervised learning as well as kernel-based approaches with semi-supervised learning, this modelling is undertaken in order to classify objects into rating classes. The results show that the rating classes assigned to bond issuers can be classified with high classification accuracy using a limited subset of input variables. This holds true for kernel-based approaches with both supervised and semi-supervised learning.     

A new feedback neural network with supervised learning

A model is introduced for continuous-time dynamic feedback neural networks with supervised learning ability. Modifications are introduced to conventional models to guarantee precisely that a given desired vector, and its negative, are indeed stored in the network as asymptotically stable equilibrium points. The modifications entail that the output signal of a neuron is multiplied by the square of its associated weight to supply the signal to an input of another neuron. A simulation of the complete dynamics is then presented for a prototype one neuron with self-feedback and supervised learning; the simulation illustrates the (supervised) learning capability of the network.     

A neural fuzzy system with fuzzy supervised learning

A neural fuzzy system learning with fuzzy training data (fuzzy if-then rules) is proposed in this paper. This system is able to process and learn numerical information as well as linguistic information. At first, we propose a five-layered neural network for the connectionist realization of a fuzzy inference system. The connectionist structure can house fuzzy logic rules and membership functions for fuzzy inference. We use alpha-level sets of fuzzy numbers to represent linguistic information. The inputs, outputs, and weights of the proposed network can be fuzzy numbers of any shape. Furthermore, they can be hybrid of fuzzy numbers and numerical numbers through the use of fuzzy singletons. Based on interval arithmetics, a fuzzy supervised learning algorithm is developed for the proposed system. It extends the normal supervised learning techniques to the learning problems where only linguistic teaching signals are available. The fuzzy supervised learning scheme can train the proposed system with desired fuzzy input-output pairs which are fuzzy numbers instead of the normal numerical values. With fuzzy supervised learning, the proposed system can be used for rule base concentration to reduce the number of rules in a fuzzy rule base. Simulation results are presented to illustrate the performance and applicability of the proposed system.     

Modelling time-varying higher moments with maximum entropy density

Since the introduction of the Autoregressive Conditional Heteroscedasticity (ARCH) model of Engle [R. Engle, Autoregressive conditional heteroscedasticity with estimates of the variance of United Kingdom inflation, Econometrica 50 (1982) 987–1007], the literature of modelling the conditional second moment has become increasingly popular in the last two decades. Many extensions and alternate models of the original ARCH have been proposed in the literature aiming to capture the dynamics of volatility more accurately. Interestingly, the Quasi Maximum Likelihood Estimator (QMLE) with normal density is typically used to estimate the parameters in these models. As such, the higher moments of the underlying distribution are assumed to be the same as those of the normal distribution. However, various studies reveal that the higher moments, such as skewness and kurtosis of the distribution of financial returns are not likely to be the same as the normal distribution, and in some cases, they are not even constant over time. These have significant implications in risk management, especially in the calculation of Value-at-Risk (VaR) which focuses on the negative quantile of the return distribution. Failed to accurately capture the shape of the negative quantile would produce inaccurate measure of risk, and subsequently lead to misleading decision in risk management. This paper proposes a solution to model the distribution of financial returns more accurately by introducing a general framework to model the distribution of financial returns using maximum entropy density (MED). The main advantage of MED is that it provides a general framework to estimate the distribution function directly based on a given set of data, and it provides a convenient framework to model higher order moments up to any arbitrary finite order k. However this flexibility comes with a high cost in computational time as k increases, therefore this paper proposes an alternative model that would reduce computation time substantially. Moreover, the sensitivity of the parameters in the MED with respect to the dynamic changes of moments is derived analytically. This result is important as it relates the dynamic structure of the moments to the parameters in the MED. The usefulness of this approach will be demonstrated using 5 min intra-daily returns of the Euro/USD exchange rate.     

Environment adaptation for robust speaker verification by cascading maximum likelihood linear regression and reinforced learning

In speaker verification over public telephone networks, utterances can be obtained from different types of handsets. Different handsets may introduce different degrees of distortion to the speech signals. This paper attempts to combine a handset selector with (1) handset-specific transformations, (2) reinforced learning, and (3) stochastic feature transformation to reduce the effect caused by the acoustic distortion. Specifically, during training, the clean speaker models and background models are firstly transformed by MLLR-based handset-specific transformations using a small amount of distorted speech data. Then reinforced learning is applied to adapt the transformed models to handset-dependent speaker models and handset-dependent background models using stochastically transformed speaker patterns. During a verification session, a GMM-based handset classifier is used to identify the most likely handset used by the claimant; then the corresponding handset-dependent speaker and background model pairs are used for verification. Experimental results based on 150 speakers of the HTIMIT corpus show that environment adaptation based on the combination of MLLR, reinforced learning and feature transformation outperforms CMS, Hnorm, Tnorm, and speaker model synthesis.     

Cross-domain person re-identification by hybrid supervised and unsupervised learning

Although the single-domain person re-identification (Re-ID) method has achieved great accuracy, the dependence on the label in the same image domain severely limits the scalability of this method. Therefore, cross-domain Re-ID has received more and more attention. In this paper, a novel cross-domain Re-ID method combining supervised and unsupervised learning is proposed, which includes two models: a triple-condition generative adversarial network (TC-GAN) and a dual-task feature extraction network (DFE-Net). We first use TC-GAN to generate labeled images with the target style, and then we combine supervised and unsupervised learning to optimize DFE-Net. Specifically, we use labeled generated data for supervised learning. In addition, we mine effective information in the target data from two perspectives for unsupervised learning. To effectively combine the two types of learning, we design a dynamic weighting function to dynamically adjust the weights of these two approaches. To verify the validity of TC-GAN, DFE-Net, and the dynamic weight function, we conduct multiple experiments on Market-1501 and DukeMTMC-reID. The experimental results show that the dynamic weight function can improve the performance of the models, and our method is better than many state-of-the-art methods.

     

Adaptive sparseness for supervised learning

The goal of supervised learning is to infer a functional mapping based on a set of training examples. To achieve good generalization, it is necessary to control the "complexity" of the learned function. In Bayesian approaches, this is done by adopting a prior for the parameters of the function being learned. We propose a Bayesian approach to supervised learning, which leads to sparse solutions; that is, in which irrelevant parameters are automatically set exactly to zero. Other ways to obtain sparse classifiers (such as Laplacian priors, support vector machines) involve (hyper)parameters which control the degree of sparseness of the resulting classifiers; these parameters have to be somehow adjusted/estimated from the training data. In contrast, our approach does not involve any (hyper)parameters to be adjusted or estimated. This is achieved by a hierarchical-Bayes interpretation of the Laplacian prior, which is then modified by the adoption of a Jeffreys' noninformative hyperprior. Implementation is carried out by an expectation-maximization (EM) algorithm. Experiments with several benchmark data sets show that the proposed approach yields state-of-the-art performance. In particular, our method outperforms SVMs and performs competitively with the best alternative techniques, although it involves no tuning or adjustment of sparseness-controlling hyperparameters.     

Spectral algorithms for supervised learning

We discuss how a large class of regularization methods, collectively known as spectral regularization and originally designed for solving ill-posed inverse problems, gives rise to regularized learning algorithms. All of these algorithms are consistent kernel methods that can be easily implemented. The intuition behind their derivation is that the same principle allowing for the numerical stabilization of a matrix inversion problem is crucial to avoid overfitting. The various methods have a common derivation but different computational and theoretical properties. We describe examples of such algorithms, analyze their classification performance on several data sets and discuss their applicability to real-world problems.     

A mesh optimization algorithm to decrease the maximum interpolation error of linear triangular finite elements

We present a mesh optimization algorithm for adaptively improving the finite element interpolation of a function of interest. The algorithm minimizes an objective function by swapping edges and moving nodes. Numerical experiments are performed on model problems. The results illustrate that the mesh optimization algorithm can reduce the W ^1,∞ semi-norm of the interpolation error. For these examples, the L ², L ^∞, and H ¹ norms decreased also.     

Tomographic analysis of sign-altering functions by maximum entropy method

Generalization of the maximum entropy method (MEM) for the reconstruction of sign-altering functions from two-dimensional tomographic measurement data is developed. Three-dimensional algorithms for parallel beam geometry are considered. Results of numerical simulations for composite model are presented.     

Using localizing learning to improve supervised learning algorithms

Slow learning of neural-network function approximators can frequently be attributed to interference, which occurs when learning in one area of the input space causes unlearning in another area. To mitigate the effect of unlearning, this paper develops an algorithm that adjusts the weights of an arbitrary, nonlinearly parameterized network such that the potential for future interference during learning is reduced. This is accomplished by the reduction of a biobjective cost function that combines the approximation error and a term that measures interference. An analysis of the algorithm's convergence properties shows that learning with this algorithm reduces future unlearning. The algorithm can be used either during online learning or can be used to condition a network to have immunity from interference during a future learning stage. A simple example demonstrates how interference manifests itself in a network and how less interference can lead to more efficient learning. Simulations demonstrate how this new learning algorithm speeds up the training in various situations due to the extra cost function term.