Adaptive multiple minor directions extraction in parallel using a PCA neural network

A principal component analysis (PCA) neural network is developed for online extraction of the multiple minor directions of an input signal. The neural network can extract the multiple minor directions in parallel by computing the principal directions of the transformed input signal so that the stability-speed problem of directly computing the minor directions can be avoided to a certain extent. On the other hand, the learning algorithms for updating the net weights use constant learning rates. This overcomes the shortcoming of the learning rates approaching zero. In addition, the proposed algorithms are globally convergent so that it is very simple to choose the initial values of the learning parameters. This paper presents the convergence analysis of the proposed algorithms by studying the corresponding deterministic discrete time (DDT) equations. Rigorous mathematical proof is given to prove the global convergence. The theoretical results are further confirmed via simulations.     

A globally convergent learning algorithm for PCA neural networks

Principal component analysis (PCA) by neural networks is one of the most frequently used feature extracting methods. To process huge data sets, many learning algorithms based on neural networks for PCA have been proposed. However, traditional algorithms are not globally convergent. In this paper, a new PCA learning algorithm based on cascade recursive least square (CRLS) neural network is proposed. This algorithm can guarantee the network weight vector converges to an eigenvector associated with the largest eigenvalue of the input covariance matrix globally. A rigorous mathematical proof is given. Simulation results show the effectiveness of the algorithm.     

基于Kalman滤波的神经网络学习算法及其应用

针对传统神经网络学习算法速度慢、容易陷入局部最优解的缺点,将卡尔曼滤波应用于人工神经网络的训练算法中。同时,在卡尔曼滤波计算中,将奇异值分解应用于卡尔曼滤波的递推公式中,提高了协方差阵计算的数值稳定性。最后,本文通过将神经网络的卡尔曼滤波算法应用于电力系统短期负荷预测中,验证了该方法不仅具有理论意义,同时也有实用价值。     

Basis functions and parameter optimisation in high-order iterative learning control

In this paper a new parameter-optimal high-order Iterative Learning Control (ILC) algorithms is proposed to extend the work of Owens and Feng [Parameter optimisation in iterative learning control. International Journal of Control 14(11), 1059-1069]. If the original plant is positive, this new algorithm will result in convergent learning where the convergence is monotonic to zero tracking error. If the original plant is not positive, it can be shown that by adding a suitable set of basis functions into the algorithm, the tracking error will again converge monotonically to zero. This provides a considerable improvement to earlier work on parameter-optimal ILC as it opens up the possibility of globally convergent algorithms for any linear plant G. The number of parameters needed to ensure convergence could, however, become large. The paper shows that the use of low-order parameterisations is capable of achieving much of the benefit achieved in the ‘ideal’ case.     

一种面向分布式深度学习系统的资源及批尺寸协同配置方法

如何在受限时间内满足深度学习模型的训练精度需求并最小化资源成本是分布式深度学习系统面临的一大挑战.资源和批尺寸超参数配置是优化模型训练精度及资源成本的主要方法.既有工作分别从计算效率和训练精度的角度,对资源及批尺寸超参数进行独立配置.然而,两类配置对于模型训练精度及资源成本的影响具有复杂的依赖关系,既有独立配置方法难以...     

A successive projection method for binary pattern recognition with multilayer feedforward neural networks

Error back-propagation (BP) is one of the most popular ideas used in learning algorithms for multilayer neural networks. In BP algorithms, there are two types of learning schemes, online learning and batch learning. The online BP has been applied to various problems in practice, because of its simplicity of implementation. However, efficient implementation of the online BP usually requires an ad hoc rule for determining the learning rate of the algorithm. In this paper, we propose a new learning algorithm called SPM, which is derived from the successive projection method for solving a system of nonlinear inequalities. Although SPM can be regarded as a modification of online BP, the former algorithm determines the learning rate (step-size) adoptively based on the output for each input pattern. SPM may also be considered a modification of the globally guided back-propagation (GGBP) proposed by Tang and Koehler. Although no theoretical proof of the convergence for SPM is given, some simulation results on pattern classification problems indicate that SPM is more effective and robust than the standard online BP and GGBP     

Convergent decomposition techniques for training RBF neural networks

In this article we define globally convergent decomposition algorithms for supervised training of generalized radial basis function neural networks. First, we consider training algorithms based on the two-block decomposition of the network parameters into the vector of weights and the vector of centers. Then we define a decomposition algorithm in which the selection of the center locations is split into sequential minimizations with respect to each center, and we give a suitable criterion for choosing the centers that must be updated at each step. We prove the global convergence of the proposed algorithms and report the computational results obtained for a set of test problems.     

Reinforcement Learning for Linear Continuous-time Systems: an Incremental Learning Approach

In this paper, we introduce a novel reinforcement learning (RL) scheme for linear continuous-time dynamical systems. Different from traditional batch learning algorithms, an incremental learning approach is developed, which provides a more efficient way to tackle the on-line learning problem in real-world applications. We provide concrete convergence and robust analysis on this incremental-learning algorithm. An extension to solving robust optimal control problems is also given. Two simulation examples are also given to illustrate the effectiveness of our theoretical result.      

On the global stabilization of locally convergent algorithms

There are a number of algorithms in the literature which both theoretically and empirically are known to be only locally convergent. These include such well known algorithms as secant, Newton, quasi-Newton and primal-dual algorithms. Locally, these algorithms tend to be highly efficient. Consequently, it is very desirable to find ways of extending, or modifying, these algorithms, so that they become globally convergent while retaining their attractive local properties. This paper describes a set of techniques which have recently emerged for stabilizing such algorithms and illustrates their application by means of a number of examples.     

Local learning regularization networks for localized regression

Local learning algorithms use a neighborhood of training data close to a given testing query point in order to learn the local parameters and create on-the-fly a local model specifically designed for this query point. The local approach delivers breakthrough performance in many application domains. This paper considers local learning versions of regularization networks (RN) and investigates several options for improving their online prediction performance, both in accuracy and speed. First, we exploit the interplay between locally optimized and globally optimized hyper-parameters (regularization parameter and kernel width) each new predictor needs to optimize online. There is a substantial reduction of the operation cost in the case we use two globally optimized hyper-parameters that are common to all local models. We also demonstrate that this global optimization of the two hyper-parameters produces more accurate models than the other cases that locally optimize online either the regularization parameter, or the kernel width, or both. Then by comparing Eigenvalue decomposition (EVD) with Cholesky decomposition specifically for the local learning training and testing phases, we also reveal that the Cholesky-based implementations are faster that their EVD counterparts for all the training cases. While EVD is suitable for validating cost-effectively several regularization parameters, Cholesky should be preferred when validating several neighborhood sizes (the number of k-nearest neighbors) as well as when the local network operates online. Then, we exploit parallelism in a multi-core system for these local computations demonstrating that the execution times are further reduced. Finally, although the use of pre-computed stored local models instead of the online learning local models is even faster, this option deteriorates the performance. Apparently, there is a substantial gain in waiting for a testing point to arrive before building a local model, and hence the online local learning RNs are more accurate than their pre-computed stored local models. To support all these findings, we also present extensive experimental results and comparisons on several benchmark datasets.

     

Evaluation of incremental learning algorithms for HMM in the recognition of alphanumeric characters

We present an evaluation of incremental learning algorithms for the estimation of hidden Markov model (HMM) parameters. The main goal is to investigate incremental learning algorithms that can provide as good performances as traditional batch learning techniques, but incorporating the advantages of incremental learning for designing complex pattern recognition systems. Experiments on handwritten characters have shown that a proposed variant of the ensemble training algorithm, employing ensembles of HMMs, can lead to very promising performances. Furthermore, the use of a validation dataset demonstrated that it is possible to reach better performances than the ones presented by batch learning.     

A normalized adaptive training of recurrent neural networks with augmented error gradient.

For training algorithms of recurrent neural networks (RNN), convergent speed and training error are always two contradictory performances. In this letter, we propose a normalized adaptive recurrent learning (NARL) to obtain a tradeoff between transient and steady-state response. An augmented term is added to error gradient to exactly model the derivative of cost function with respect to hidden layer weight. The influence of the induced gain of activation function on training stability is also taken into consideration. Moreover, adaptive learning rate is employed to improve the robustness of the gradient training. Finally, computer simulations of a model prediction problem are synthesized to give comparisons between NARL and conventional normalized real-time recurrent learning (N-RTRL).     

基于微粒群算法与模拟退火算法的协同进化方法

提出了一种基于模拟退火与微粒群算法的协同进化方法,利用了微粒群算法的易实现性、局部快速收敛性以及模拟退火算法的全局收敛性.通过两种算法的协同搜索,可以有效克服微粒群算法的早熟收敛.仿真结果表明,本文的协同进化方法不仅具有较好的全局收敛性能,而且具有较快的收敛速度.文章从理论上证明了该方法以概率1收敛于全局最优解.     

A combinational incremental ensemble of classifiers as a technique for predicting students’ performance in distance education

The ability to predict a student’s performance could be useful in a great number of different ways associated with university-level distance learning. Students’ marks in a few written assignments can constitute the training set for a supervised machine learning algorithm. Along with the explosive increase of data and information, incremental learning ability has become more and more important for machine learning approaches. The online algorithms try to forget irrelevant information instead of synthesizing all available information (as opposed to classic batch learning algorithms). Nowadays, combining classifiers is proposed as a new direction for the improvement of the classification accuracy. However, most ensemble algorithms operate in batch mode. Therefore a better proposal is an online ensemble of classifiers that combines an incremental version of Naive Bayes, the 1-NN and the WINNOW algorithms using the voting methodology. Among other significant conclusions it was found that the proposed algorithm is the most appropriate to be used for the construction of a software support tool.     

Deterministic nonmonotone strategies for effective training of multilayer perceptrons

We present deterministic nonmonotone learning strategies for multilayer perceptrons (MLPs), i.e., deterministic training algorithms in which error function values are allowed to increase at some epochs. To this end, we argue that the current error function value must satisfy a nonmonotone criterion with respect to the maximum error function value of the M previous epochs, and we propose a subprocedure to dynamically compute M. The nonmonotone strategy can be incorporated in any batch training algorithm and provides fast, stable, and reliable learning. Experimental results in different classes of problems show that this approach improves the convergence speed and success percentage of first-order training algorithms and alleviates the need for fine-tuning problem-depended heuristic parameters.     

Nonlocal estimation of manifold structure

We claim and present arguments to the effect that a large class of manifold learning algorithms that are essentially local and can be framed as kernel learning algorithms will suffer from the curse of dimensionality, at the dimension of the true underlying manifold. This observation invites an exploration of nonlocal manifold learning algorithms that attempt to discover shared structure in the tangent planes at different positions. A training criterion for such an algorithm is proposed, and experiments estimating a tangent plane prediction function are presented, showing its advantages with respect to local manifold learning algorithms: it is able to generalize very far from training data (on learning handwritten character image rotations), where local nonparametric methods fail.     

Developing learning algorithms via optimized discretization of continuous dynamical systems

Most of the existing numerical optimization methods are based upon a discretization of some ordinary differential equations. In order to solve some convex and smooth optimization problems coming from machine learning, in this paper, we develop efficient batch and online algorithms based on a new principle, i.e., the optimized discretization of continuous dynamical systems (ODCDSs). First, a batch learning projected gradient dynamical system with Lyapunov's stability and monotonic property is introduced, and its dynamical behavior guarantees the accuracy of discretization-based optimizer and applicability of line search strategy. Furthermore, under fair assumptions, a new online learning algorithm achieving regret O(√T) or O(logT) is obtained. By using the line search strategy, the proposed batch learning ODCDS exhibits insensitivity to the step sizes and faster decrease. With only a small number of line search steps, the proposed stochastic algorithm shows sufficient stability and approximate optimality. Experimental results demonstrate the correctness of our theoretical analysis and efficiency of our algorithms.     

An Edge-based Stochastic Proximal Gradient Algorithm for Decentralized Composite Optimization

This paper investigates decentralized composite optimization problems involving a common non-smooth regularization term over an undirected and connected network. In the same situation, there exist lots of gradient-based proximal distributed methods, but most of them are only sublinearly convergent. The proof of linear convergence for this series of algorithms is extremely difficult. To set up the problem, we presume all networked agents use the same non-smooth regularization term, which is the circumstance for most machine learning to implement based on centralized optimization. For this scenario, most existing proximal-gradient algorithms trend to ignore the cost of gradient evaluations, which results in degraded performance. To tackle this problem, we further set the local cost function to the average of a moderate amount of local cost subfunctions and develop an edge-based stochastic proximal gradient algorithm (SPG-Edge) by employing local unbiased stochastic averaging gradient method. When the non-smooth term does not exist, the proposed algorithm could be extended to some notable primal-dual domain algorithms, such as EXTRA and DIGing. Finally, we provide a simplified proof of linear convergence and conduct numerical experiments to illustrate the validity of theoretical results.

     

Ensembling local learners through multimodal perturbation.

Ensemble learning algorithms train multiple component learners and then combine their predictions. In order to generate a strong ensemble, the component learners should be with high accuracy as well as high diversity. A popularly used scheme in generating accurate but diverse component learners is to perturb the training data with resampling methods, such as the bootstrap sampling used in bagging. However, such a scheme is not very effective on local learners such as nearest-neighbor classifiers because a slight change in training data can hardly result in local learners with big differences. In this paper, a new ensemble algorithm named Filtered Attribute Subspace based Bagging with Injected Randomness (FASBIR) is proposed for building ensembles of local learners, which utilizes multimodal perturbation to help generate accurate but diverse component learners. In detail, FASBIR employs the perturbation on the training data with bootstrap sampling, the perturbation on the input attributes with attribute filtering and attribute subspace selection, and the perturbation on the learning parameters with randomly configured distance metrics. A large empirical study shows that FASBIR is effective in building ensembles of nearest-neighbor classifiers, whose performance is better than that of many other ensemble algorithms.     

基于em算法且能以概率1全局收敛的混合学习算法

文章指出了随机神经网络em学习算法仍然存在着收敛于局部极小值之缺陷.针对三层随机感知机,文章将em学习算法与Solis和Wets的随机优化算法结合起来,提出了三层随机感知机的混合型新学习算法HRem.文章从理论的角度证明了混合型新学习算法HRem能以概率1全局收敛于随机感知机的基于Kullback-Leibler差异度量的最小值.这一理论结果对em学习算法的深入研究有重要意义.