A survey of artificial neural network training tools

Artificial neural networks (ANN) are currently an additional tool which the engineer can use for a variety of purposes. Classification and regression are the most common tasks; however, control, modeling, prediction and forecasting are common tasks as well. For over three decades, the field of ANN has been the center of intense research. As a result, one of the outcomes has been the development of a large set of software tools used to train these kinds of networks, making the selection of an adequate tool difficult for a new user. This paper aims to help the ANN user choose the most appropriate tool for its application by providing a large survey of the solutions available, as well as listing and explaining their characteristics and terms of use. The paper limits itself to focusing on the tools which were developed for ANN and the relevant characteristics of these tools, such as the operating systems, hardware requirements, license types, architectures and algorithms available.     

A constructive algorithm for training cooperative neural network ensembles

Presents a constructive algorithm for training cooperative neural-network ensembles (CNNEs). CNNE combines ensemble architecture design with cooperative training for individual neural networks (NNs) in ensembles. Unlike most previous studies on training ensembles, CNNE puts emphasis on both accuracy and diversity among individual NNs in an ensemble. In order to maintain accuracy among individual NNs, the number of hidden nodes in individual NNs are also determined by a constructive approach. Incremental training based on negative correlation is used in CNNE to train individual NNs for different numbers of training epochs. The use of negative correlation learning and different training epochs for training individual NNs reflect CNNEs emphasis on diversity among individual NNs in an ensemble. CNNE has been tested extensively on a number of benchmark problems in machine learning and neural networks, including Australian credit card assessment, breast cancer, diabetes, glass, heart disease, letter recognition, soybean, and Mackey-Glass time series prediction problems. The experimental results show that CNNE can produce NN ensembles with good generalization ability.     

A hybrid method for hydroelectric generation scheduling using an artificial neural network

This paper presents a decomposition method for finding an optimal operating policy of interconnected hydroelectric power plants using an artificial neural network. The coupling constraints on reservoir storage at the end of the planning horizon are relaxed using coordinating multipliers that result in interval wise decomposition of the overall problem. Resulting subproblems are solved sequentially, which reduces the complexity of the problem. Each subproblem is solved using a two-phase neural network approach. An efficient heuristic algorithm is developed to find the feasible solution. A case study considering scheduling of the Bhakra-Beas reservoir system is also presented in this paper. The new method demonstrates the potential of achieving an improved performance.     

构造性层状神经网络集成方法

在分析构造性神经网络集成和层状神经网络集成方法的基础上,提出了一种构造性层状神经网络集成方法。该方法自动确定神经网络集成中成员神经网络的数目,以及成员神经网络的结构等。集成在保证成员神经网络精度的同时,又保证了成员网络之间的差异度。用户只需要简单定义一些参数,就可以构造出性能较好的神经网络集成。     

Neighborhood based Levenberg-Marquardt algorithm for neural network training

Although the Levenberg-Marquardt (LM) algorithm has been extensively applied as a neural-network training method, it suffers from being very expensive, both in memory and number of operations required, when the network to be trained has a significant number of adaptive weights. In this paper, the behavior of a recently proposed variation of this algorithm is studied. This new method is based on the application of the concept of neural neighborhoods to the LM algorithm. It is shown that, by performing an LM step on a single neighborhood at each training iteration, not only significant savings in memory occupation and computing effort are obtained, but also, the overall performance of the LM method can be increased.     

前馈神经网络的混沌BP 混合学习算法

简要分析由Logistic映射产生的混沌数以及不同混沌序列之间的概率统计特性，为混沌全局性搜索提供了依据．将一种快速BP算法与混沌优化相结合，提出了混沌BP混合算法，由于混沌Logistic映射的遍历性、随机性，使得混合算法收敛速度快，且具有全局性，采用混合算法对XOR问题和非线性函数进行仿真，结果表明该算法明显优于标准BP算法和快速BP算法。     

A data clustering algorithm for stratified data partitioning in artificial neural network

The statistical properties of training, validation and test data play an important role in assuring optimal performance in artificial neural networks (ANNs). Researchers have proposed optimized data partitioning (ODP) and stratified data partitioning (SDP) methods to partition of input data into training, validation and test datasets. ODP methods based on genetic algorithm (GA) are computationally expensive as the random search space can be in the power of twenty or more for an average sized dataset. For SDP methods, clustering algorithms such as self organizing map (SOM) and fuzzy clustering (FC) are used to form strata. It is assumed that data points in any individual stratum are in close statistical agreement. Reported clustering algorithms are designed to form natural clusters. In the case of large multivariate datasets, some of these natural clusters can be big enough such that the furthest data vectors are statistically far away from the mean. Further, these algorithms are computationally expensive as well. We propose a custom design clustering algorithm (CDCA) to overcome these shortcomings. Comparisons are made using three benchmark case studies, one each from classification, function approximation and prediction domains. The proposed CDCA data partitioning method is evaluated in comparison with SOM, FC and GA based data partitioning methods. It is found that the CDCA data partitioning method not only perform well but also reduces the average CPU time.     

A novel hybrid learning algorithm for full Bayesian approach of artificial neural networks

The Bayesian neural networks are useful tools to estimate the functional structure in the nonlinear systems. However, they suffer from some complicated problems such as controlling the model complexity, the training time, the efficient parameter estimation, the random walk, and the stuck in the local optima in the high-dimensional parameter cases. In this paper, to alleviate these mentioned problems, a novel hybrid Bayesian learning procedure is proposed. This approach is based on the full Bayesian learning, and integrates Markov chain Monte Carlo procedures with genetic algorithms and the fuzzy membership functions. In the application sections, to examine the performance of proposed approach, nonlinear time series and regression analysis are handled separately, and it is compared with the traditional training techniques in terms of their estimation and prediction abilities.     

带蜂群策略的粒子群算法训练人工神经网络研究

提出一种带蜂群策略的粒子群优化算法,并将算法应用于神经网络训练。将蜂群优化算法中引领蜂和观察蜂的收益评价与贪婪选择策略以及侦察蜂的探索新解策略引入到粒子群优化算法中。粒子在飞行时,按维度对粒子速度和位置进行更新,根据对收益的评价,只接收能够提高解适应值的位置,从而加快了收敛速度;如果粒子多次迭代均无法改进解,则在解空间中随机搜索新的位置,增强算法跳出局部极值的能力。在求解异或问题、奇偶校验和编码解码问题的神经网络上进行了仿真,结果表明,该算法优于BP算法、粒子群优化算法和蜂群优化算法。     

Supervised adaptive clustering: A hybrid neural network clustering algorithm

A neural network architecture is introduced which implements a supervised clustering algorithm for the classification of feature vectors. The network is selforganising, and is able to adapt to the shape of the underlying pattern distribution as well as detect novel input vectors during training. It is also capable of determining the relative importance of the feature components for classification. The architecture is a hybrid of supervised and unsupervised networks, and combines the strengths of three wellknown architectures: learning vector quantisation, backpro-pagation and adaptive resonance theory. Network performance is compared to that of learning vector quantisation, back-propagation and cascade-correlation. It is found that performance is generally as good as or better than the performance of these other architectures, while training time is considerably shorter. However, the main advantage of the hybrid architecture is its ability to gain insight into the feature pattern space.Nomenclature O _j The output value of thejth unit - I _i Theith component of the input pattern - W _ij The weight of the cluster connection between theith input and thejth unit - B _ij The weight of the shape connection between theith input and thejth unit - N The dimension of the input patterns - v _j The vigilance parameter of thejth unit - v _init The initial vigilance parameter value - v _rate The change in the vigilance parameter value - X _i Theith direction in anN-dimensional coordinate system - T _k The classification tag of thekth unit - C The classification tag of the current input vector - (p) The learning rate at thepth epoch for the cluster weights - p The current epoch - P The total number of epochs - E _k The error associated with thekth unit - The constant learning rate for the shape weights - a _j The age in epochs of thejth unit     

COOT optimization algorithm on training artificial neural networks

Knowledge and Information Systems - In recent years, significant advancements have been made in artificial neural network models and they have been applied to a variety of real-world problems....     

A novel artificial neural network for sorting

An artificial neural network (ANN) is employed for sorting a sequence of real elements in monotonic (descending or ascending) order. Although inspired by harmony theory (HT), whereby the same construction as for the HT ANN is followed, the proposed ANN differs in the mode of operation, namely the obliteration of the consensus (harmony) function, the circumvention of simulated annealing as a means of settling to a solution, the simplification of the activation updating of the nodes of the upper layer, the clamping of the nodes of the lower layer, the gradual shrinking of the ANN and the use of an automatic termination criterion. The creation of the sorted sequence is progressive, whereby at most as many network updates are required as there are elements in the sequence. Ties between elements are resolved by simultaneous activation of the corresponding nodes. Finally, the min and max problems are solved in a single network update.     

A consensus-function artificial neural network for map-coloring

A harmony theory artificial neural network solution to the map coloring problem is presented. Map coloring aims at assigning a unique color to each area of a given map so that no two adjacent areas receive identical colors. The harmony theory implementation is able to determine whether the map coloring problem can be solved with a predefined number of colors as well as which is the smallest number of colors that can solve the map coloring problem. The present implementation directly encodes the given problem into the artificial neural network so that a solution is represented simply by node activation. Additionally, the consensus function of harmony theory produces a quick and definite solution to the colorability problem, obviating the need for manual validation of the result.     

基于混合神经网络的智能问答算法

为让用户通过输入自然语言就可以跟机器进行交互,实现文本的智能问答,提出基于混合神经网络的智能问答算法。将LSTM (long short-term memory)和CNN (convolutional neural network)相结合。利用LSTM计算问题和答案的语义特征,针对语义特征的选择进行改进。采用CNN对LSTM得到的语义特征进行筛选;通过计算问题和答案特征之间的相似度得到该模型的目标函数,给出问题对应的正确答案。仿真结果验证了该算法的可行性及有效性。     

递归pi-sigma神经网络的带惩罚项的梯度算法分析

传统的梯度算法存在收敛速度过慢的问题,针对这个问题,提出一种将惩罚项加到传统误差函数的梯度算法以训练递归pi-sigma神经网络,算法不仅提高了神经网络的泛化能力,而且克服了因网络初始权值选取过小而导致的收敛速度过慢的问题,相比不带惩罚项的梯度算法提高了收敛速度。从理论上分析了带惩罚项的梯度算法的收敛性,并通过实验验证了算法的有效性。     

An improved algorithm for neural network classification ofimbalanced training sets

The backpropagation algorithm converges very slowly for two-class problems in which most of the exemplars belong to one dominant class. An analysis shows that this occurs because the computed net error gradient vector is dominated by the bigger class so much that the net error for the exemplars in the smaller class increases significantly in the initial iteration. The subsequent rate of convergence of the net error is very low. A modified technique for calculating a direction in weight-space which decreases the error for each class is presented. Using this algorithm, the rate of learning for two-class classification problems is accelerated by an order of magnitude.     

An algorithm for characterizing pre-fuzzified linguistic nuance using artificial neural network

The attractiveness of artificial neural networks (ANNs) in solving many complex real world and computational demanding problems was used in characterizing linguistic nuance for harnessing malicious intent or decoding a communication trend. A set of adjectival watch lists was created a priori to serve as target convergence outputs to the ANN’s graphical user interface designed by the researchers. A set of pre-fuzzified or pre-processed speech conversation or written text was used as inputs to the neural network and represents a sub set of actual words used in the investigated two-way communication. The watch lists represents an editable set of words that represents malicious intent or key elements of conversation intent in bidirectional conversation or communication. The watch list database was generated a priori by identifying adjectives and specific nouns as used in the communication under investigation and then normalized. The pre-processed speech and text have been obtained from Recognizers utilizing the hidden Markov models and its hybrids for its processing. The algorithm showed robustness in sorting out pre-normalized and fuzzified speech that ordinarily contained certain elements of interest as conveyed by the investigated conversations. Analysis of a patient-to-healthcare provider’s bidirectional communication during malaria diagnosis and used for testing the developed algorithm showed significant accuracy when compared with the results of clinical analysis or consultation for the corresponding diagnosis.     

Multiple-page mapping artificial neural network algorithm used for constant tension control

Constant tension control is widely required in industrial applications such as paper machines, coating machines, rewinding and unwinding machines. In a metal film coating machine, which is a multi-input multi-output system, speed and tension have cross coupling and thus desired speed and tension responses are difficult to achieve by applying conventional analogue proportional-plus-integral (PI) control. This paper introduces a multiple-page mapping artificial neural network with back-propagation training algorithm. This method can successfully decouple the speed and tension control loops and both loops can operate quasi-independently. It overcomes the disadvantages of traditional PI control systems. To handle the variation of the rewinding roll diameter, multiple pages of neural networks are applied. Some simulation results show the effectiveness of this control algorithm.     

A new pose accuracy compensation method for parallel manipulators based on hybrid artificial neural network

Neural Computing and Applications - The pose accuracy of parallel manipulators is one of the most important performance indices in advanced industrial applications. The modeling and estimation of...     

Spectral identification by artificial neural network and genetic algorithm

With the increasing utilization of multi-spectral imaging sensors, automatic identification of spectral signatures would be an invaluable facility. Conventional, linear approaches have been shown to be limited: although reasonably stable in the presence of small amounts of noise, their performance degrades severelyat high noise levelsor when several spectra are combined. Nonlinear approaches are therefore attractive.

This paper describes a detailed investigation into the suitability of two nonlinear paradigms, namely artificial neural networks and genetic algorithms, for spectral identification. The ability of each method to distinguish between several different spectra is assessed, as is their stability to noise and their capacity to correctly identify combinations of spectra. These results are compared with a linear technique, cross-correlation.

Both non-linear methods are shown to be very effective for spectral identification, the results being far superior to those of cross-correlation. However, it will be demonstrated that the genetic algorithm approach requires a greater understanding of the physical processes involved. A practical scheme, utilizing both neural network and genetic algorithm approaches, is suggested.