A new recurrent neural-network architecture for visual patternrecognition

We propose a new type of recurrent neural-network architecture, in which each output unit is connected to itself and is also fully connected to other output units and all hidden units. The proposed recurrent neural network differs from Jordan's and Elman's recurrent neural networks with respect to function and architecture, because it has been originally extended from being a mere multilayer feedforward neural network, to improve discrimination and generalization powers. We also prove the convergence properties of the learning algorithm in the proposed recurrent neural network, and analyze the performance of the proposed recurrent neural network by performing recognition experiments with the totally unconstrained handwritten numeric database of Concordia University, Montreal, Canada. Experimental results have confirmed that the proposed recurrent neural network improves discrimination and generalization powers in the recognition of visual patterns     

Optimal randomized RANSAC

A randomized model verification strategy for RANSAC is presented. The proposed method finds, like RANSAC, a solution that is optimal with user-specified probability. The solution is found in time that is (i) close to the shortest possible and (ii) superior to any deterministic verification strategy. A provably fastest model verification strategy is designed for the (theoretical) situation when the contamination of data by outliers is known. In this case, the algorithm is the fastest possible (on average) of all randomized \\RANSAC algorithms guaranteeing a confidence in the solution. The derivation of the optimality property is based on Wald's theory of sequential decision making, in particular a modified sequential probability ratio test (SPRT). Next, the R-RANSAC with SPRT algorithm is introduced. The algorithm removes the requirement for a priori knowledge of the fraction of outliers and estimates the quantity online. We show experimentally that on standard test data the method has performance close to the theoretically optimal and is 2 to 10 times faster than standard RANSAC and is up to 4 times faster than previously published methods.     

A theoretical study of linear and nonlinear equalization innonlinear magnetic storage channels

We present methods to systematically design a feedforward neural-network detector from the knowledge of the channel characteristics. Its performance is compared with the conventional linear equalizer in a magnetic recording channel suffering from signal-dependent noise and nonlinear intersymbol interference. The superiority of the nonlinear schemes are clearly observed in all cases studied, especially in the presence of severe nonlinearity and noise. We also show that the decision boundaries formed by a theoretically derived neural-network classifier are geometrically close to those of a neural network trained by the backpropagation algorithm. The approach in this work is suitable for quantifying the gain in using a neural-network method as opposed to linear methods in the classification of noisy patterns.     

Performance evaluation of multilayer perceptrons in signaldetection and classification

Multilayer perceptrons trained with the backpropagation algorithm are tested in detection and classification tasks and are compared to optimal algorithms resulting from likelihood ratio tests. The focus is on the problem of one of M orthogonal signals in a Gaussian noise environment, since both the Bayesian detector and classifier are known for this problem and can provide a measure for the performance evaluation of the neural networks. Two basic situations are considered: detection and classification. For the detection part, it was observed that for the signal-known-exactly case (M=1), the performance of the neural detector converges to the performance of the ideal Bayesian decision processor, while for a higher degree of uncertainty (i.e. for a larger M), the performance of the multilayer perceptron is inferior to that of the optimal detector. For the classification case, the probability of error of the neural network is comparable to the minimum Bayesian error, which can be numerically calculated. Adding noise during the training stage of the network does not affect the performance of the neural detector; however, there is an indication that the presence of noise in the learning process of the neural classifier results in a degraded classification performance.     

一种径向基函数神经网络预测在超临界温度控制系统中的应用研究

超临界温度控制系统具有较大的惯性、时滞和非线性,且动态特性随运行工况而改变,难以建立其精确的数学模型,本文采用GGAP算法的RBF神经网络构成神经网络预测控制器,将在线学习和预测控制相结合,以某超临界电厂主汽温度为研究对象,MATLAB仿真实验表明,该方法能对超临界温度控制系统实现有效的控制,动态性能较传统的PID控制有较大的提高。     

Predicting sun spots using a layered perceptron neural network

Interest in neural networks has expanded rapidly in recent years. Selecting the best structure for a given task, however, remains a critical issue in neural-network design. Although the performance of a network clearly depends on its structure, the procedure for selecting the optimal structure has not been thoroughly investigated, it is well known that the number of hidden units must be sufficient to discriminate each observation correctly. A large number of hidden units requires extensive computational time for training and often times prediction results may not be as accurate as expected. This study attempts to apply the principal component analysis (PCA) to determine the structure of a multilayered neural network for time series forecasting problems. The main focus is to determine the number of hidden units for a multilayered feedforward network. One empirical experiment with sunspot data is used to demonstrate the usefulness of the proposed approach.     

CARVE-a constructive algorithm for real-valued examples

A constructive neural-network algorithm is presented. For any consistent classification task on real-valued training vectors, the algorithm constructs a feedforward network with a single hidden layer of threshold units which implements the task. The algorithm, which we call CARVE, extends the "sequential learning" algorithm of Marchand et al. (1990) from Boolean inputs to the real-valued input case, and uses convex hull methods for the determination of the network weights. The algorithm is an efficient training scheme for producing near-minimal network solutions for arbitrary classification tasks. The algorithm is applied to a number of benchmark problems including German and Sejnowski's sonar data, the Monks problems and Fisher's iris data. A significant application of the constructive algorithm is in providing an initial network topology and initial weights for other neural-network training schemes, and this is demonstrated by application to backpropagation.     

Sequential monte carlo methods To train neural network models

We discuss a novel strategy for training neural networks using sequential Monte Carlo algorithms and propose a new hybrid gradient descent sampling importance resampling algorithm (HySIR). In terms of computational time and accuracy, the hybrid SIR is a clear improvement over conventional sequential Monte Carlo techniques. The new algorithm may be viewed as a global optimization strategy that allows us to learn the probability distributions of the network weights and outputs in a sequential framework. It is well suited to applications involving on-line, nonlinear, and nongaussian signal processing. We show how the new algorithm outperforms extended Kalman filter training on several problems. In particular, we address the problem of pricing option contracts, traded in financial markets. In this context, we are able to estimate the one-step-ahead probability density functions of the options prices.     

Genetics-based machine learning for the assessment of certainneuromuscular disorders

Clinical electromyography (EMG) provides useful information for the diagnosis of neuromuscular disorders. The utility of artificial neural networks (ANN's) in classifying EMG data trained with backpropagation or Rohonen's self-organizing feature maps algorithm has recently been demonstrated. The objective of this study is to investigate how genetics-based machine learning (GBML) can be applied for diagnosing certain neuromuscular disorders based on EMG data. The effect of GBML control parameters on diagnostic performance is also examined. A hybrid diagnostic system is introduced that combines both neural network and GBML models. Such a hybrid system provides the end-user with a robust and reliable system, as its diagnostic performance relies on more than one learning principle. GBML models demonstrated similar performance to neural-network models, but with less computation. The diagnostic performance of neural network and GBML models is enhanced by the hybrid system.     

A generalized Shiryayev sequential probability ratio test forchange detection and isolation

The authors derive an online multiple hypothesis Shiryayev sequential probability ratio test (SSPRT) by adopting a dynamic programming approach. It is shown that for a certain criterion of optimality, this generalized Shiryayev SPRT detects and isolates a change in hypothesis in the conditionally independent measurement sequence in minimum time, unlike the Wald SPRT, which assumes the entire measurement sequence to correspond to a single hypothesis. The measurement cost, the cost of a false alarm, and the cost of a miss-alarm are considered in our dynamic programming analysis. The algorithm is shown to be optimal in the infinite time case. Finally, the performance of the algorithm is evaluated by using a few examples. In particular, they implement the algorithm in a fault detection and identification scheme for advanced vehicle control systems     

A learning automata-based algorithm for determination of the number of hidden units for three-layer neural networks

There is no method to determine the optimal topology for multi-layer neural networks for a given problem. Usually the designer selects a topology for the network and then trains it. Since determination of the optimal topology of neural networks belongs to class of NP-hard problems, most of the existing algorithms for determination of the topology are approximate. These algorithms could be classified into four main groups: pruning algorithms, constructive algorithms, hybrid algorithms and evolutionary algorithms. These algorithms can produce near optimal solutions. Most of these algorithms use hill-climbing method and may be stuck at local minima. In this article, we first introduce a learning automaton and study its behaviour and then present an algorithm based on the proposed learning automaton, called survival algorithm, for determination of the number of hidden units of three layers neural networks. The survival algorithm uses learning automata as a global search method to increase the probability of obtaining the optimal topology. The algorithm considers the problem of optimization of the topology of neural networks as object partitioning rather than searching or parameter optimization as in existing algorithms. In survival algorithm, the training begins with a large network, and then by adding and deleting hidden units, a near optimal topology will be obtained. The algorithm has been tested on a number of problems and shown through simulations that networks generated are near optimal.     

模糊神经网络的二阶段变半径随机搜索算法

研究基于正态隶属函数的模糊神经网络的学习算法,将模糊神经网络对一组样本的逼近误差表示为两组相互独立,可分批学习的可调参数的非负函数这和,其中一组可调参数可通过令相应的非负函数为零直接求得,而与另一组可调参数相对应的非负函数就是用于这组参数学习的性能指标,经对性能指标性质的分析给出一种模糊神经网络的学习算法－二阶段变半径随机搜索法,实例表明,灾种方法简便易行,可使模糊神经网络达到 较高的逼近精度。     

基于自适应交叉与协方差学习的改进平衡优化器算法

针对平衡优化器算法存在的收敛精度低和易陷入局部停滞的问题,提出一种基于自适应交叉与协方差学习的改进平衡优化器算法。首先,构建外部存档来保留历史优势个体,增加种群多样性,以提高算法的全局寻优能力。其次,引入自适应交叉概率来平衡算法的全局探索能力和局部开发能力,以提高算法的寻优精度和鲁棒性。最后,采用协方差学习策略,充分利用浓度向量之间的关系来增强种群间信息交流,以避免算法陷入局部停滞。通过对CEC2019测试函数进行仿真实验,并将改进算法与反向传播（back propagation,BP）神经网络相结合用于预测新疆玛纳斯河的径流情况,实验结果表明,改进算法在收敛精度和鲁棒性方面有显著提升,且大幅提高了BP神经网络的径流预测效果。     

An application of sequential neural-network approximation for sitting and sizing passive harmonic filters

This paper presents a method which is combined by sequential neural-network approximation and orthogonal arrays (SNAOA) for reducing harmonic distortion with passive harmonic filters and determining the optimal locations for harmonic filters among existent capacitor busses in the power network. An orthogonal array is first conducted to obtain the initial solution set. The set is then treated as the initial training sample. Next, a back-propagation sequential neural network is trained to simulate the feasible domain for seeking the optimal filter design. A restart strategy is also incorporated into the SNAOA so that the searching process may have a better opportunity to reach a near global optimum solution. In order to determine a set of weights of objective function to represent the relative importance of each term, the simplest and most efficient form of triangular membership functions has been considered. To illustrate the performance of the SNAOA, a practical harmonic mitigation problem in a 36-bus radial distribution system is studied. The results show that the SNAOA performs better than the original scheme and satisfies the harmonic limitations with respect to the objective of minimizing total harmonic distortion of voltages and the cost of commercially available discrete sizes for sitting and sizing passive harmonic filters.     

高斯基神经网络的非线性PID控制方法

针对二阶非线性系统,提出了一种用高斯基函数作为神经元激励函数的PID（Proportion-Integral-Derivative）控制方法。该方法用高斯基函数模拟PID参数随误差变化的曲线,用神经网络算法在线调整各模拟曲线的系数,从而构造出具有非线性特征的PID控制策略,实现了基于高斯基神经网络的非线性PID智能控制方法。计算机仿真结果表明,该方法具有良好的非线性控制效果,因此在工业领域具有广泛的应用前景。     

基函数可递归的泛函神经元网络学习算法

将泛函神经元结构做了一个变形,给出了一种基函数可递归的泛函神经元网络学习算法,该算法借助于矩阵伪逆递归求解方法,完成对泛函神经元网络基函数的自适应调整,最终实现泛函网络结构和参数共同的最优求解。数值仿真实验结果表明,该算法具有自适应性、鲁棒性和较高的收敛精度,将在实时在线辨识中有着广泛的应用。     

多输入Sigmoid激励函数神经网络权值与结构确定法

结合伪逆直接计算得到神经元之间最优权值的方法,提出了一种双阶段自动搜索与确定最优网络结构的算法,克服了原有BP神经网络模型及其学习算法的固有缺陷。以函数逼近为例,计算机数值实验结果显示了算法有效且耗时短,证实了由该算法得到的网络对于多输入函数具有较优良的逼近(学习与校验)性能。     

神经网络结构搜索方法综述

如今,深度学习广泛地应用于生活、工作中的各个方面,给我们带来了极大的便利.在此背景下,需要设计针对不同任务的神经网络结构,满足不同的需求.但是,人工设计神经网络结构需要专业的知识,进行大量的实验.因此,神经网络结构搜索算法的研究显得极为重要.神经网络结构搜索(NAS)是自动深度学习(AutoDL)过程中的一个基本步骤,对深度学习的发展与应用有着重要的影响.早期,一些神经网络结构搜索算法虽然搜索到了性能优越的神经网络结构,但是需要大量的计算资源且搜索效率低下.因此,研究人员探索了多种设计神经网络结构的算法,也提出了许多减少计算资源、提高搜索效率的方法.本文首先简要介绍了神经网络结构的搜索空间,其次对神经网络结构搜索算法进行了全面的分类汇总、分析,主要包括随机搜索算法、进化算法、强化学习、基于梯度下降的方法、基于顺序模型的优化算法,再其次探索并总结了提高神经网络结构搜索效率的方法,最后探讨了目前神经网络结构搜索工作中存在的问题以及未来的研究方向.     

Integrating temporal difference methods and self-organizing neural networks for reinforcement learning with delayed evaluative feedback

This paper presents a neural architecture for learning category nodes encoding mappings across multimodal patterns involving sensory inputs, actions, and rewards. By integrating adaptive resonance theory (ART) and temporal difference (TD) methods, the proposed neural model, called TD fusion architecture for learning, cognition, and navigation (TD-FALCON), enables an autonomous agent to adapt and function in a dynamic environment with immediate as well as delayed evaluative feedback (reinforcement) signals. TD-FALCON learns the value functions of the state-action space estimated through on-policy and off-policy TD learning methods, specifically state-action-reward-state-action (SARSA) and Q-learning. The learned value functions are then used to determine the optimal actions based on an action selection policy. We have developed TD-FALCON systems using various TD learning strategies and compared their performance in terms of task completion, learning speed, as well as time and space efficiency. Experiments based on a minefield navigation task have shown that TD-FALCON systems are able to learn effectively with both immediate and delayed reinforcement and achieve a stable performance in a pace much faster than those of standard gradient-descent-based reinforcement learning systems.     

Associative learning in random environments using neural networks

Associative learning is investigated using neural networks and concepts based on learning automata. The behavior of a single decision-maker containing a neural network is studied in a random environment using reinforcement learning. The objective is to determine the optimal action corresponding to a particular state. Since decisions have to be made throughout the context space based on a countable number of experiments, generalization is inevitable. Many different approaches can be followed to generate the desired discriminant function. Three different methods which use neural networks are discussed and compared. In the most general method, the output of the network determines the probability with which one of the actions is to be chosen. The weights of the network are updated on the basis of the actions and the response of the environment. The extension of similar concepts to decentralized decision-making in a context space is also introduced. Simulation results are included. Modifications in the implementations of the most general method to make it practically viable are also presented. All the methods suggested are feasible and the choice of a specific method depends on the accuracy desired as well as on the available computational power.