Stable dynamic backpropagation learning in recurrent neuralnetworks

To avoid unstable phenomenon during the learning process, two new learning schemes, called the multiplier and constrained learning rate algorithms, are proposed in this paper to provide stable adaptive updating processes for both the synaptic and somatic parameters of the network. Based on the explicit stability conditions, in the multiplier method these conditions are introduced into the iterative error index, and the new updating formulations contain a set of inequality constraints. In the constrained learning rate algorithm, the learning rate is updated at each iterative instant by an equation derived using the stability conditions. With these stable dynamic backpropagation algorithms, any analog target pattern may be implemented by a steady output vector which is a nonlinear vector function of the stable equilibrium point. The applicability of the approaches presented is illustrated through both analog and binary pattern storage examples.     

Globally asymptotical stability of discrete-time analog neuralnetworks

Some globally asymptotical stability criteria for the equilibrium states of a general class of discrete-time dynamic neural networks with continuous states are presented using a diagonal Lyapunov function approach. The neural networks are assumed to have the asymmetrical weight matrices throughout the paper. The resulting criteria are described by the diagonal stability of some matrices associated with the network parameters. Some novel stability conditions represented by either the existence of the positive diagonal solutions of the Lyapunov equations or some inequalities are given. Using the equivalence between the diagonal stability and the Schur stability for a nonnegative matrix, some simplified global stability conditions are also presented. Finally, some examples are provided for demonstrating the effectiveness of the global stability conditions presented.     

Output feedback control of nonlinear systems using RBF neuralnetworks

An adaptive output feedback control scheme for the output tracking of a class of continuous-time nonlinear plants is presented. An RBF neural network is used to adaptively compensate for the plant nonlinearities. The network weights are adapted using a Lyapunov-based design. The method uses parameter projection, control saturation, and a high-gain observer to achieve semi-global uniform ultimate boundedness. The effectiveness of the proposed method is demonstrated through simulations. The simulations also show that by using adaptive control in conjunction with robust control, it is possible to tolerate larger approximation errors resulting from the use of lower order networks.     

Absolute stability conditions for discrete-time recurrent neuralnetworks

An analysis of the absolute stability for a general class of discrete-time recurrent neural networks (RNN's) is presented. A discrete-time model of RNN's is represented by a set of nonlinear difference equations. Some sufficient conditions for the absolute stability are derived using Ostrowski's theorem and the similarity transformation approach. For a given RNN model, these conditions are determined by the synaptic weight matrix of the network. The results reported in this paper need fewer constraints on the weight matrix and the model than in previously published studies.     

Neural-network front ends in unsupervised learning

Proposed is an idea of partial supervision realized in the form of a neural-network front end to the schemes of unsupervised learning (clustering). This neural network leads to an anisotropic nature of the induced feature space. The anisotropic property of the space provides us with some of its local deformation necessary to properly represent labeled data and enhance efficiency of the mechanisms of clustering to be exploited afterwards. The training of the network is completed based upon available labeled patterns-a referential form of the labeling gives rise to reinforcement learning. It is shown that the discussed approach is universal and can be utilized in conjunction with any clustering method. Experimental studies are concentrated on three main categories of unsupervised learning including FUZZY ISODATA, Kohonen self-organizing maps, and hierarchical clustering.     

Memristor crossbar-based unsupervised image learning

This letter presents a new memristor crossbar array system and demonstrates its applications in image learning. The controlled pulse and image overlay technique are introduced for the programming of memristor crossbars and promising a better performance for noise reduction. The time-slot technique is helpful for improving the processing speed of image. Simulink and numerical simulations have been employed to demonstrate the useful applications of the proposed circuit structure in image learning.     

Quantum speed-up for unsupervised learning

We show how the quantum paradigm can be used to speed up unsupervised learning algorithms. More precisely, we explain how it is possible to accelerate learning algorithms by quantizing some of their subroutines. Quantization refers to the process that partially or totally converts a classical algorithm to its quantum counterpart in order to improve performance. In particular, we give quantized versions of clustering via minimum spanning tree, divisive clustering and k-medians that are faster than their classical analogues. We also describe a distributed version of k-medians that allows the participants to save on the global communication cost of the protocol compared to the classical version. Finally, we design quantum algorithms for the construction of a neighbourhood graph, outlier detection as well as smart initialization of the cluster centres.     

具分布参数的随机Hopfield神经网络的指数稳定

基于随机Fubini定理,将随机偏微分方程描述的Hopfield神经网络系统转化为用相应的随机常微分方程来描述.利用关于空间变量平均的Lyapunov函数与Ito^公式,通过对所构造的Lyapunov函数在Ito^微分规则下对相应系统求导的方法,获得了系统指数稳定的代数判据及其Lyapunov指数估计.实现了运用Lyapunov直接法对分布参数系统稳定性的研究.     

Dimensionality reduction in unsupervised learning of conditionalGaussian networks

This paper introduces a novel enhancement for unsupervised learning of conditional Gaussian networks that benefits from feature selection. Our proposal is based on the assumption that, in the absence of labels reflecting the cluster membership of each case of the database, those features that exhibit low correlation with the rest of the features can be considered irrelevant for the learning process. Thus, we suggest performing this process using only the relevant features. Then, every irrelevant feature is added to the learned model to obtain an explanatory model for the original database which is our primary goal. A simple and, thus, efficient measure to assess the relevance of the features for the learning process is presented. Additionally, the form of this measure allows us to calculate a relevance threshold to automatically identify the relevant features. The experimental results reported for synthetic and real-world databases show the ability of our proposal to distinguish between relevant and irrelevant features and to accelerate learning, while still obtaining good explanatory models for the original database     

CDUL: Class directed unsupervised learning

A novel neural network called Class Directed Unsupervised Learning (CDUL) is introduced. The architecture, based on a Kohonen self-organising network, uses additional input nodes to feed class knowledge to the network during training, in order to optimise the final positioning of Kohonen nodes in feature space. The structure and training of CDUL networks is detailed, showing that (a) networks cannot suffer from the problem of single Kohonen nodes being trained by vectors of more than one class, (b) the number of Kohonen nodes necessary to represent the classes is found during training, and (c) the number of training set passes CDUL requires is low in comparison to similar networks. CDUL is subsequently applied to the classification of chemical excipients from Near Infrared (NIR) reflectance spectra, and its performance compared with three other unsupervised paradigms. The results thereby obtained demonstrate a superior performance which remains relatively constant through a wide range of network parameters.     

Bot detection using unsupervised machine learning

Microsystem Technologies - This research focuses on bot detection through implementation of techniques such as traffic analysis, unsupervised machine learning, and similarity analysis between...     

一种面向识别的无监督特征学习算法

特征抽取是图像识别的关键环节,准确的特征表达能够产生更准确的分类效果。采用软阈值编码器和正交匹配追踪(OMP)算法正交化视觉词典的方法,以提高单级计算结构的识别率,并进一步构造两级计算结构,获取图像更准确的特征,以提高图像的识别率。实验表明,采用软阈值编码器和OMP算法能提高单级计算结构提取特征的能力,提高大样本数据集中图像的识别率。两级计算结构能够提高自选数据集中图像的识别率。采用OMP算法能提高VOC2012数据中图像的识别率。在自选数据集上,两级计算结构优于单级计算结构,与NIN结构相比表现出优势,与卷积神经网络CNN相当,说明两级计算结构在自选数据集上有很好的适应性。     

Text summarization using unsupervised deep learning

We present methods of extractive query-oriented single-document summarization using a deep auto-encoder (AE) to compute a feature space from the term-frequency (tf) input. Our experiments explore both local and global vocabularies. We investigate the effect of adding small random noise to local tf as the input representation of AE, and propose an ensemble of such noisy AEs which we call the Ensemble Noisy Auto-Encoder (ENAE). ENAE is a stochastic version of an AE that adds noise to the input text and selects the top sentences from an ensemble of noisy runs. In each individual experiment of the ensemble, a different randomly generated noise is added to the input representation. This architecture changes the application of the AE from a deterministic feed-forward network to a stochastic runtime model. Experiments show that the AE using local vocabularies clearly provide a more discriminative feature space and improves the recall on average 11.2%. The ENAE can make further improvements, particularly in selecting informative sentences. To cover a wide range of topics and structures, we perform experiments on two different publicly available email corpora that are specifically designed for text summarization. We used ROUGE as a fully automatic metric in text summarization and we presented the average ROUGE-2 recall for all experiments.     

On equilibria, stability, and instability of Hopfield neuralnetworks

Existence and uniqueness of equilibrium, as well as its stability and instability, of a continuous-time Hopfield neural network are studied. A set of new and simple sufficient conditions are derived.     

Preintegration lateral inhibition enhances unsupervised learning

A large and influential class of neural network architectures uses postintegration lateral inhibition as a mechanism for competition. We argue that these algorithms are computationally deficient in that they fail to generate, or learn, appropriate perceptual representations under certain circumstances. An alternative neural network architecture is presented here in which nodes compete for the right to receive inputs rather than for the right to generate outputs. This form of competition, implemented through preintegration lateral inhibition, does provide appropriate coding properties and can be used to learn such representations efficiently. Furthermore, this architecture is consistent with both neuroanatomical and neurophysiological data. We thus argue that preintegration lateral inhibition has computational advantages over conventional neural network architectures while remaining equally biologically plausible.     

Fuzzy identification of systems with unsupervised learning

The paper describes a mathematical tool to build a fuzzy model whose membership functions and consequent parameters rely on the estimates of a data set. The proposed method proved to be capable of approximating any real continuous function, also a strongly nonlinear one, on a compact set to arbitrary accuracy. Without resorting to domain experts, the algorithm constructs a model-free, complete function approximation system. Applications to the modeling of several functions among which classical nonlinear ones such as the Rosenbrock and the sine (x, y) functions are reported. The proposed algorithm can find applications in the development of fuzzy logic controllers (FLC).     

Recursive unsupervised learning of finite mixture models

There are two open problems when finite mixture densities are used to model multivariate data: the selection of the number of components and the initialization. In this paper, we propose an online (recursive) algorithm that estimates the parameters of the mixture and that simultaneously selects the number of components. The new algorithm starts with a large number of randomly initialized components. A prior is used as a bias for maximally structured models. A stochastic approximation recursive learning algorithm is proposed to search for the maximum a posteriori (MAP) solution and to discard the irrelevant components.     

Optimal, unsupervised learning in invariant object recognition

A means for establishing transformation-invariant representations of objects is proposed and analyzed, in which different views are associated on the basis of the temporal order of the presentation of these views, as well as their spatial similarity. Assuming knowledge of the distribution of presentation times, an optimal linear learning rule is derived. Simulations of a competitive network trained on a character recognition task are then used t highlight the success of this learning rule in relation to simple Hebbian learning and to show that the theory can give accurate quantitative predictions for the optimal parameters for such networks.     

非监督子空间学习中关联度量多维尺度分析

多维尺度分析已经在维度约减和数据挖掘领域得到了广泛应用。MDS的主要缺点是其定义在训练数据上,对于新的测试样本无法直接获得映射结果。另外,MDS基于欧氏距离度量,不适合获取相似数据中的非线性流形结构。将MDS扩展到关联度量空间,称为关联度量多维尺度分析(CMDS)。与传统MDS在训练数据中完成映射,进而缩小空间范围相比,CMDS 能够直接获得测试样本映射结果。此外,CMDS基于关联度量,能够有效学习相似数据中的非线性流形结构。理论分析表明,CMDS可以利用核方法扩展到新特征空间,解决非线性问题。实验结果表明,CMDS及其核形式KG-CMDS性能优于常用传统降维方法。