Piecewise Linear Projection Based on Self-Organizing Map

A piecewise linear projection algorithm, based on kohonen's Self-Organizing Map, is presented. Using this new algorithm, neural network is able to adapt its neural weights to accommodate with input space, while obtaining reduced 2-dimensional subspaces at each neural node. After completion of learning process, first project input data into their corresponding 2-D subspaces, then project all data in the 2-D subspaces into a reference 2-D subspace defined by a reference neural node. By piecewise linear projection, we can more easily deal with large data sets than other projection algorithms like Sammon's nonlinear mapping (NLM). There is no need to re-compute all the input data to interpolate new input data to the 2-D output space.     

Application of error minimized extreme learning machine for simultaneous learning of a function and its derivatives

In this paper a new learning algorithm is proposed for the problem of simultaneous learning of a function and its derivatives as an extension of the study of error minimized extreme learning machine for single hidden layer feedforward neural networks. Our formulation leads to solving a system of linear equations and its solution is obtained by Moore-Penrose generalized pseudo-inverse. In this approach the number of hidden nodes is automatically determined by repeatedly adding new hidden nodes to the network either one by one or group by group and updating the output weights incrementally in an efficient manner until the network output error is less than the given expected learning accuracy. For the verification of the efficiency of the proposed method a number of interesting examples are considered and the results obtained with the proposed method are compared with that of other two popular methods. It is observed that the proposed method is fast and produces similar or better generalization performance on the test data.     

Fuzzy perceptron neural networks for classifiers with numericaldata and linguistic rules as inputs

This paper presents a novel learning algorithm of fuzzy perceptron neural networks (FPNNs) for classifiers that utilize expert knowledge represented by fuzzy IF-THEN rules as well as numerical data as inputs. The conventional linear perceptron network is extended to a second-order one, which is much more flexible for defining a discriminant function. In order to handle fuzzy numbers in neural networks, level sets of fuzzy input vectors are incorporated into perceptron neural learning. At different levels of the input fuzzy numbers, updating the weight vector depends on the minimum of the output of the fuzzy perceptron neural network and the corresponding nonfuzzy target output that indicates the correct class of the fuzzy input vector. This minimum is computed efficiently by employing the modified vertex method. Moreover, the fuzzy pocket algorithm is introduced into our fuzzy perceptron learning scheme to solve the nonseparable problems. Simulation results demonstrate the effectiveness of the proposed FPNN model     

Median radial basis function neural network

Radial basis functions (RBFs) consist of a two-layer neural network, where each hidden unit implements a kernel function. Each kernel is associated with an activation region from the input space and its output is fed to an output unit. In order to find the parameters of a neural network which embeds this structure we take into consideration two different statistical approaches. The first approach uses classical estimation in the learning stage and it is based on the learning vector quantization algorithm and its second-order statistics extension. After the presentation of this approach, we introduce the median radial basis function (MRBF) algorithm based on robust estimation of the hidden unit parameters. The proposed algorithm employs the marginal median for kernel location estimation and the median of the absolute deviations for the scale parameter estimation. A histogram-based fast implementation is provided for the MRBF algorithm. The theoretical performance of the two training algorithms is comparatively evaluated when estimating the network weights. The network is applied in pattern classification problems and in optical flow segmentation.     

A fast multilayer neural-network training algorithm based on thelayer-by-layer optimizing procedures

A faster new learning algorithm to adjust the weights of the multilayer feedforward neural network is proposed. In this new algorithm, the weight matrix (W(2)) of the output layer and the output vector (Y) of the previous layer are treated as two variable sets. An optimal solution pair (W(2)*,Y(P)*) is found to minimize the sum-square-error of the patterns input. Y(P)* is then used as the desired output of the previous layer. The optimal weight matrix and layer output vector of the hidden layers in the network is found with the same method as that used for the output layer. In addition, the dynamic forgetting factors method makes the proposed new algorithm even more powerful in dynamic system identification. Computer simulation shows that the new algorithm outmatches other learning algorithms both in converging speed and in computation time required.     

Intelligent system based on local linear wavelet neural network and recursive least square approach for breast cancer classification

A new learning technique for local linear wavelet neural network (LLWNN) is presented in this paper. The difference of the network with conventional wavelet neural network (WNN) is that the connection weights between the hidden layer and output layer of conventional WNN are replaced by a local linear model. A hybrid training algorithm of Error Back propagation and Recursive Least Square (RLS) is introduced for training the parameters of LLWNN. The variance and centers of LLWNN are updated using back propagation and weights are updated using Recursive Least Square (RLS). Results on extracted breast cancer data from University of Wisconsin Hospital Madison show that the proposed approach is very robust, effective and gives better classification.     

Application of ELM–Hammerstein model to the identification of solid oxide fuel cells

In this paper, a new method is proposed to identify solid oxide fuel cell using extreme learning machine–Hammerstein model (ELM–Hammerstein). The ELM–Hammerstein model consists of a static ELM neural network followed by a linear dynamic subsystem. First, the structure of ELM–Hammerstein model is determined by Lipschitz quotient criterion from input–output data. Then, a generalized ELM algorithm is proposed to estimate the parameters of ELM–Hammerstein model, including the parameters of linear dynamic part and the output weights of ELM. The proposed method can obtain accurate identification results and its computation is more efficient. Simulation results demonstrate its effectiveness.

     

An Improved Neural Network with Random Weights Using Backtracking Search Algorithm

This paper proposes a hybrid algorithm by combining backtracking search algorithm (BSA) and a neural network with random weights (NNRWs), called BSA-NNRWs-N. BSA is utilized to optimize the hidden layer parameters of the single layer feed-forward network (SLFN) and NNRWs is used to derive the output layer weights. In addition, to avoid over-fitting on the validation set, a new cost function is proposed to replace the root mean square error (RMSE). In the new cost function, a constraint is added by considering RMSE on both training and validation sets. Experiments on classification and regression data sets show promising performance of the proposed BSA-NNRWs-N.     

Fuzzy Associative Conjuncted Maps Network

The fuzzy associative conjuncted maps (FASCOM) is a fuzzy neural network that associates data of nonlinearly related inputs and outputs. In the network, each input or output dimension is represented by a feature map that is partitioned into fuzzy or crisp sets. These fuzzy sets are then conjuncted to form antecedents and consequences, which are subsequently associated to form if–then rules. The associative memory is encoded through an offline batch mode learning process consisting of three consecutive phases. The initial unsupervised membership function initialization phase takes inspiration from the organization of sensory maps in our brains by allocating membership functions based on uniform information density. Next, supervised Hebbian learning encodes synaptic weights between input and output nodes. Finally, a supervised error reduction phase fine-tunes the network, which allows for the discovery of the varying levels of influence of each input dimension across an output feature space in the encoded memory. In the series of experiments, we show that each phase in the learning process contributes significantly to the final accuracy of prediction. Further experiments using both toy problems and real-world data demonstrate significant superiority in terms of accuracy of nonlinear estimation when benchmarked against other prominent architectures and exhibit the network's suitability to perform analysis and prediction on real-world applications, such as traffic density prediction as shown in this paper.      

Bayesian dual neural networks for recommendation

Most traditional collaborative filtering (CF) methods only use the user-item rating matrix to make recommendations, which usually suffer from cold-start and sparsity problems. To address these problems, on the one hand, some CF methods are proposed to incorporate auxiliary information such as user/item profiles; on the other hand, deep neural networks, which have powerful ability in learning effective representations, have achieved great success in recommender systems. However, these neural network based recommendation methods rarely consider the uncertainty of weights in the network and only obtain point estimates of the weights. Therefore, they maybe lack of calibrated probabilistic predictions and make overly confident decisions. To this end, we propose a new Bayesian dual neural network framework, named BDNet, to incorporate auxiliary information for recommendation. Specifically, we design two neural networks, one is to learn a common low dimensional space for users and items from the rating matrix, and another one is to project the attributes of users and items into another shared latent space. After that, the outputs of these two neural networks are combined to produce the final prediction. Furthermore, we introduce the uncertainty to all weights which are represented by probability distributions in our neural networks to make calibrated probabilistic predictions. Extensive experiments on real-world data sets are conducted to demonstrate the superiority of our model over various kinds of competitors.     

Fast Blind Equalization Using Complex-Valued MLP

The blind equalizers based on complex valued feedforward neural networks, for linear and nonlinear communication channels, yield better performance as compared to linear equalizers. The learning algorithms are, generally, based on stochastic gradient descent, as they are simple to implement. However, these algorithms show a slow convergence rate. In the blind equalization problem, the unavailability of the desired output signal and the presence of nonlinear activation functions make the application of recursive least squares algorithm difficult. In this letter, a new scheme using recursive least squares algorithm is proposed for blind equalization. The learning of weights of the output layer is obtained by using a modified version of constant modulus algorithm cost function. For the learning of weights of hidden layer neuron space adaptation approach is used. The proposed scheme results in faster convergence of the equalizer.     

基于Kalman滤波的储备池多元时间序列在线预报器

针对多元非线性时间序列, 结合回声状态网络和Kalman滤波提出一种新的在线自适应预报方法. 该方法将Kalman滤波应用于回声状态网络储备池高维状态空间中, 直接对网络的输出权值进行在线更新, 省去了传统递归网络扩展Kalman滤波中Jacobian矩阵的计算, 在提高预测精度的同时令算法的适用范围得到扩展. 在回声状态网络稳定时给出所提算法的收敛性证明. 仿真实例验证了所提方法的有效性.     

基于可拓聚类的极限学习机神经网络

针对极限学习机(ELM)算法随机选择输入层权值的问题,借鉴第2类型可拓神经网络(ENN-2)聚类的思想,提出了一种基于可拓聚类的ELM(EC-ELM)神经网络。该神经网络是以隐含层神经元的径向基中心向量作为输入层权值,采用可拓聚类算法动态调整隐含层节点数目和径向基中心,并根据所确定的输入层权值,利用Moore-Penrose广义逆快速完成输出层权值的求解。同时,对标准的Friedman#1回归数据集和Wine分类数据集进行测试,结果表明,EC-ELM提供了一种简便的神经网络结构和参数学习方法,并且比基于可拓理论的径向基函数(ERBF)、ELM神经网络具有更高的建模精度和更快的学习速度,为复杂过程的建模提供了新思路。     

An optimized RBF neural network algorithm based on partial least squares and genetic algorithm for classification of small sample

Radial basis function (RBF) neural network can use linear learning algorithm to complete the work formerly handled by nonlinear learning algorithm, and maintain the high precision of the nonlinear algorithm. However, the results of RBF would be slightly unsatisfactory when dealing with small sample which has higher feature dimension and fewer numbers. Higher feature dimension will influence the design of neural network, and fewer numbers of samples will cause network training incomplete or over-fitted, both of which restrict the recognition precision of the neural network. RBF neural network has some drawbacks, for example, it is hard to determine the numbers, center and width of the hidden layer’s neurons, which constrain the success of training. To solve the above problems, partial least squares (PLS) and genetic algorithm(GA)are introduced into RBF neural network, and better recognition precision will be obtained, because PLS is good at dealing with the small sample data, it can reduce feature dimension and make low-dimensional data more interpretative. In addition, GA can optimize the network architecture, the weights between hidden layer and output layer of the RBF neural network can ease non-complete network training, the way of hybrid coding and simultaneous evolving is adopted, and then an accurate algorithm is established. By these two consecutive optimizations, the RBF neural network classification algorithm based on PLS and GA (PLS-GA-RBF) is proposed, in order to solve some recognition problems caused by small sample. Four experiments and comparisons with other four algorithms are carried out to verify the superiority of the proposed algorithm, and the results indicate a good picture of the PLS-GA-RBF algorithm, the operating efficiency and recognition accuracy are improved substantially. The new small sample classification algorithm is worthy of further promotion.     

NARX model based nonlinear dynamic system identification using low complexity neural networks and robust H∞ filter

This paper proposes NARX (nonlinear autoregressive model with exogenous input) model structures with functional expansion of input patterns by using low complexity ANN (artificial neural network) for nonlinear system identification. Chebyshev polynomials, Legendre polynomials, trigonometric expansions using sine and cosine functions as well as wavelet basis functions are used for the functional expansion of input patterns. The past input and output samples are modeled as a nonlinear NARX process and robust H_∞ filter is proposed as the learning algorithm for the neural network to identify the unknown plants. H_∞ filtering approach is based on the state space modeling of model parameters and evaluation of Jacobian matrices. This approach is the robustification of Kalman filter which exhibits robust characteristics and fast convergence properties. Comparison results for different nonlinear dynamic plants with forgetting factor recursive least square (FFRLS) and extended Kalman filter (EKF) algorithms demonstrate the effectiveness of the proposed approach.     

Indirect iterative learning control for a discrete visual servo without a camera-robot model.

This paper presents a discrete learning controller for vision-guided robot trajectory imitation with no prior knowledge of the camera-robot model. A teacher demonstrates a desired movement in front of a camera, and then, the robot is tasked to replay it by repetitive tracking. The imitation procedure is considered as a discrete tracking control problem in the image plane, with an unknown and time-varying image Jacobian matrix. Instead of updating the control signal directly, as is usually done in iterative learning control (ILC), a series of neural networks are used to approximate the unknown Jacobian matrix around every sample point in the demonstrated trajectory, and the time-varying weights of local neural networks are identified through repetitive tracking, i.e., indirect ILC. This makes repetitive segmented training possible, and a segmented training strategy is presented to retain the training trajectories solely within the effective region for neural network approximation. However, a singularity problem may occur if an unmodified neural-network-based Jacobian estimation is used to calculate the robot end-effector velocity. A new weight modification algorithm is proposed which ensures invertibility of the estimation, thus circumventing the problem. Stability is further discussed, and the relationship between the approximation capability of the neural network and the tracking accuracy is obtained. Simulations and experiments are carried out to illustrate the validity of the proposed controller for trajectory imitation of robot manipulators with unknown time-varying Jacobian matrices.     

Neural network control of nonlinear dynamic systems using hybrid algorithm

In this paper, a hybrid method is proposed to control a nonlinear dynamic system using feedforward neural network. This learning procedure uses different learning algorithm separately. The weights connecting the input and hidden layers are firstly adjusted by a self organized learning procedure, whereas the weights between hidden and output layers are trained by supervised learning algorithm, such as a gradient descent method. A comparison with backpropagation (BP) shows that the new algorithm can considerably reduce network training time.     

Minimal Structure of Self-Organizing HCMAC Neural Network Classifier

The authors previously proposed a self-organizing Hierarchical Cerebellar Model Articulation Controller (HCMAC) neural network containing a hierarchical GCMAC neural network and a self-organizing input space module to solve high-dimensional pattern classification problems. This novel neural network exhibits fast learning, a low memory requirement, automatic memory parameter determination and highly accurate high-dimensional pattern classification. However, the original architecture needs to be hierarchically expanded using a full binary tree topology to solve pattern classification problems according to the dimension of the input vectors. This approach creates many redundant GCMAC nodes when the dimension of the input vectors in the pattern classification problem does not exactly match that in the self-organizing HCMAC neural network. These redundant GCMAC nodes waste memory units and degrade the learning performance of a self-organizing HCMAC neural network. Therefore, this study presents a minimal structure of self-organizing HCMAC (MHCMAC) neural network with the same dimension of input vectors as the pattern classification problem. Additionally, this study compares the learning performance of this novel learning structure with those of the BP neural network,support vector machine (SVM), and original self-organizing HCMAC neural network in terms of ten benchmark pattern classification data sets from the UCI machine learning repository. In particular, the experimental results reveal that the self-organizing MHCMAC neural network handles high-dimensional pattern classification problems better than the BP, SVM or the original self-organizing HCMAC neural network. Moreover, the proposed self-organizing MHCMAC neural network significantly reduces the memory requirement of the original self-organizing HCMAC neural network, and has a high training speed and higher pattern classification accuracy than the original self-organizing HCMAC neural network in most testing benchmark data sets. The experimental results also show that the MHCMAC neural network learns continuous function well and is suitable for Web page classification.     

Design of highly effective multilayer feedforward neural network by using genetic algorithm

This paper presents a highly effective and precise neural network method for choosing the activation functions (AFs) and tuning the learning parameters (LPs) of a multilayer feedforward neural network by using a genetic algorithm (GA). The performance of the neural network mainly depends on the learning algorithms and the network structure. The backpropagation learning algorithm is used for tuning the network connection weights, and the LPs are obtained by the GA to provide both fast and reliable learning. Also, the AFs of each neuron in the network are automatically chosen by a GA. The present study consists of 10 different functions to accomplish a better convergence of the desired input–output mapping. Test studies are performed to solve a set of two-dimensional regression problems for the proposed genetic-based neural network (GNN) and conventional neural network having sigmoid AFs and constant learning parameters. The proposed GNN has also been tested by applying it to three real problems in the fields of environment, medicine, and economics. Obtained results prove that the proposed GNN is more effective and reliable when compared with the classical neural network structure.     

基于函数正交基展开的过程神经网络学习算法

过程神经网络的输入和连接权均可为时变函数，过程神经元增加了一个对于时间的聚合算子，使网络同时具有时空二维信息处理能力．该文在考虑过程神经网络对时间聚合运算的复杂性的基础上，提出了一种基于函数正交基展开的学习算法．在网络输入函数空间中选择一组适当的函数正交基，将输入函数和网络权函数都表示为该组正交基的展开形式，利用基函数的正交性．简化过程神经元对时间的聚合运算．应用表明，算法简化了过程神经网络的计算复杂度，提高了网络学习效率和对实际问题求解的适应性．以旋转机械故障诊断问题和油藏开发过程采收率的模拟为例验证了算法的有效性．