A hybrid neural network classifier combining ordered fuzzy ARTMAP and the dynamic decay adjustment algorithm

This paper presents a novel conflict-resolving neural network classifier that combines the ordering algorithm, fuzzy ARTMAP (FAM), and the dynamic decay adjustment (DDA) algorithm, into a unified framework. The hybrid classifier, known as Ordered FAMDDA, applies the DDA algorithm to overcome the limitations of FAM and ordered FAM in achieving a good generalization/performance. Prior to network learning, the ordering algorithm is first used to identify a fixed order of training patterns. The main aim is to reduce and/or avoid the formation of overlapping prototypes of different classes in FAM during learning. However, the effectiveness of the ordering algorithm in resolving overlapping prototypes of different classes is compromised when dealing with complex datasets. Ordered FAMDDA not only is able to determine a fixed order of training patterns for yielding good generalization, but also is able to reduce/resolve overlapping regions of different classes in the feature space for minimizing misclassification during the network learning phase. To illustrate the effectiveness of Ordered FAMDDA, a total of ten benchmark datasets are experimented. The results are analyzed and compared with those from FAM and Ordered FAM. The outcomes demonstrate that Ordered FAMDDA, in general, outperforms FAM and Ordered FAM in tackling pattern classification problems.     

A fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks

In this paper, we present a fast and accurate online self-organizing scheme for parsimonious fuzzy neural networks (FAOS-PFNN), where a novel structure learning algorithm incorporating a pruning strategy into new growth criteria is developed. The proposed growing procedure without pruning not only speeds up the online learning process but also facilitates a more parsimonious fuzzy neural network while achieving comparable performance and accuracy by virtue of the growing and pruning strategy. The FAOS-PFNN starts with no hidden neurons and parsimoniously generates new hidden units according to the proposed growth criteria as learning proceeds. In the parameter learning phase, all the free parameters of hidden units, regardless of whether they are newly created or originally existing, are updated by the extended Kalman filter (EKF) method. The effectiveness and superiority of the FAOS-PFNN paradigm is compared with other popular approaches like resource allocation network (RAN), RAN via the extended Kalman filter (RANEKF), minimal resource allocation network (MRAN), adaptive-network-based fuzzy inference system (ANFIS), orthogonal least squares (OLS), RBF-AFS, dynamic fuzzy neural networks (DFNN), generalized DFNN (GDFNN), generalized GAP-RBF (GGAP-RBF), online sequential extreme learning machine (OS-ELM) and self-organizing fuzzy neural network (SOFNN) on various benchmark problems in the areas of function approximation, nonlinear dynamic system identification, chaotic time-series prediction and real-world regression problems. Simulation results demonstrate that the proposed FAOS-PFNN algorithm can achieve faster learning speed and more compact network structure with comparably high accuracy of approximation and generalization.     

A review of online learning in supervised neural networks

Learning in neural networks can broadly be divided into two categories, viz., off-line (or batch) learning and online (or incremental) learning. In this paper, a review of a variety of supervised neural networks with online learning capabilities is presented. Specifically, we focus on articles published in main indexed journals in the past 10 years (2003–2013). We examine a number of key neural network architectures, which include feedforward neural networks, recurrent neural networks, fuzzy neural networks, and other related networks. How the online learning methodologies are incorporated into these networks is exemplified, and how they are applied to solving problems in different domains is highlighted. A summary of the review that covers different network architectures and their applications is presented.     

An iterative pruning algorithm for feedforward neural networks

The problem of determining the proper size of an artificial neural network is recognized to be crucial, especially for its practical implications in such important issues as learning and generalization. One popular approach for tackling this problem is commonly known as pruning and it consists of training a larger than necessary network and then removing unnecessary weights/nodes. In this paper, a new pruning method is developed, based on the idea of iteratively eliminating units and adjusting the remaining weights in such a way that the network performance does not worsen over the entire training set. The pruning problem is formulated in terms of solving a system of linear equations, and a very efficient conjugate gradient algorithm is used for solving it, in the least-squares sense. The algorithm also provides a simple criterion for choosing the units to be removed, which has proved to work well in practice. The results obtained over various test problems demonstrate the effectiveness of the proposed approach.     

Regularized online sequential learning algorithm for single-hidden layer feedforward neural networks

Online learning algorithms have been preferred in many applications due to their ability to learn by the sequentially arriving data. One of the effective algorithms recently proposed for training single hidden-layer feedforward neural networks (SLFNs) is online sequential extreme learning machine (OS-ELM), which can learn data one-by-one or chunk-by-chunk at fixed or varying sizes. It is based on the ideas of extreme learning machine (ELM), in which the input weights and hidden layer biases are randomly chosen and then the output weights are determined by the pseudo-inverse operation. The learning speed of this algorithm is extremely high. However, it is not good to yield generalization models for noisy data and is difficult to initialize parameters in order to avoid singular and ill-posed problems. In this paper, we propose an improvement of OS-ELM based on the bi-objective optimization approach. It tries to minimize the empirical error and obtain small norm of network weight vector. Singular and ill-posed problems can be overcome by using the Tikhonov regularization. This approach is also able to learn data one-by-one or chunk-by-chunk. Experimental results show the better generalization performance of the proposed approach on benchmark datasets.     

Hybrid system of ART and RBF neural networks for online clustering

An online clustering task is considered for machine state monitoring purpose. In the previous authors’ researches a classical ART-2 network was tested for online classification of operational states in the context of a wind turbine monitoring. Some drawbacks, however, were found when a data stream size had been increased. This case is investigated in this paper. Classical ART-2 network can cluster data incorrectly when data points are compared by using Euclidean distance. Furthermore, ART-2 network can lose accuracy when data stream is processed for long time. The way of improving the ART-2 network is considered and two main steps of that are taken. At first, the stereographic projection is implemented. At the second step, a new type of hybrid neural system which consists of ART-2 and RBF networks with data processed by using the stereographic projection is introduced. Tests contained elementary scenarios for low-dimensional cases as well as higher dimensional real data from wind turbine monitoring. All the tests implied that an efficient system for online clustering had been found.     

An adaptive Bayesian pruning for neural networks in a non-stationary environment

Pruning a neural network to a reasonable smaller size, and if possible to give a better generalization, has long been investigated. Conventionally the common technique of pruning is based on considering error sensitivity measure, and the nature of the problem being solved is usually stationary. In this article, we present an adaptive pruning algorithm for use in a nonstationary environment. The idea relies on the use of the extended Kalman filter (EKF) training method. Since EKF is a recursive Bayesian algorithm, we define a weight-importance measure in term of the sensitivity of a posteriori probability. Making use of this new measure and the adaptive nature of EKF, we devise an adaptive pruning algorithm called adaptive Bayesian pruning. Simulation results indicate that in a noisy nonstationary environment, the proposed pruning algorithm is able to remove network redundancy adaptively and yet preserve the same generalization ability.     

Drill flank wear estimation using supervised vector quantization neural networks

Drill wear detection and prognosis is one of the most important considerations in reducing the cost of rework and scrap and to optimize tool utilization in hole making industry. This study presents the development and implementation of two supervised vector quantization neural networks for estimating the flank-land wear size of a twist drill. The two algorithms are; the learning vector quantization (LVQ) and the fuzzy learning vector quantization (FLVQ). The input features to the neural networks were extracted from the vibration signals using power spectral analysis and continuous wavelet transform techniques. Training and testing were performed under a variety of speeds and feeds in the dry drilling of steel plates. It was found that the FLVQ is more efficient in assessing the flank wear size than the LVQ. The experimental procedure for acquiring vibration data and extracting features in the time-frequency domain using the wavelet transform is detailed. Experimental results demonstrated that the proposed neural network algorithms were effective in estimating the size of the drill flank wear.     

Detecting online auction shilling frauds using supervised learning

Online auction sites are a target for fraud due to their anonymity, number of potential targets and low likelihood of identification. Researchers have developed methods for identifying fraud. However, these methods must be individually tailored for each type of fraud, since each differs in the characteristics important for their identification. Using supervised learning methods, it is possible to produce classifiers for specific types of fraud by providing a dataset where instances with behaviours of interest are assigned to a separate class. However this requires multiple labelled datasets: one for each fraud type of interest. It is difficult to use real-world datasets for this purpose since they are difficult to label, often limited in size, and contain zero or multiple suspicious behaviours that may or may not be under investigation.The aims of this work are to: (1) demonstrate the approach of using supervised learning together with a validated synthetic data generator to create fraud detection models that are experimentally more accurate than existing methods and that is effective over real data, and (2) to evaluate a set of features for use in general fraud detection is shown to further improve the performance of the created detection models.The approach is as follows: the data generator is an agent-based simulation modelled on users in commercial online auction data. The simulation is extended using fraud agents which model a known type of online auction fraud called competitive shilling. These agents are added to the simulation to produce the synthetic datasets. Features extracted from this data are used as training data for supervised learning. Using this approach, we optimise an existing fraud detection algorithm, and produce classifiers capable of detecting shilling fraud.Experimental results with synthetic data show the new models have significant improvements in detection accuracy. Results with commercial data show the models identify users with suspicious behaviour.     

Improved gradient-based neural networks for online solution of Lyapunov matrix equation

By adding different activation functions, a type of gradient-based neural networks is developed and presented for the online solution of Lyapunov matrix equation. Theoretical analysis shows that any monotonically-increasing odd activation function could be used for the construction of neural networks, and the improved neural models have the global convergence performance. For the convenience of hardware realization, the schematic circuit is given for the improved neural solvers. Computer simulation results further substantiate that the improved neural networks could solve the Lyapunov matrix equation with accuracy and effectiveness. Moreover, when using the power-sigmoid activation functions, the improved neural networks have superior convergence when compared to linear models.     

一种基于灵敏度分析的神经网络剪枝方法

根据灵敏度矩阵提出了一种简单的灵敏度定义,该定义反映了单个输入节点对整个网络性能的影响。进而,基于该灵敏度定义提出了神经网络输入层剪枝算法。最后,通过UCI机器学习数据库中的两个模式分类例子验证方法的有效性。     

An online learning algorithm for adaptable topologies of neural networks

Many real scenarios in machine learning are of dynamic nature. Learning in these types of environments represents an important challenge for learning systems. In this context, the model used for learning should work in real time and have the ability to act and react by itself, adjusting its controlling parameters, even its structures, depending on the requirements of the process. In a previous work, the authors presented an online learning algorithm for two-layer feedforward neural networks that includes a factor that weights the errors committed in each of the samples. This method is effective in dynamic environments as well as in stationary contexts. As regards this method’s incremental feature, we raise the possibility that the network topology is adapted according to the learning needs. In this paper, we demonstrate and justify the suitability of the online learning algorithm to work with adaptive structures without significantly degrading its performance. The theoretical basis for the method is given and its performance is illustrated by means of its application to different system identification problems. The results confirm that the proposed method is able to incorporate units to its hidden layer, during the learning process, without high performance degradation.     

L-p Fuzzy ARTMAP neural network architecture

In this paper, an L-p based Fuzzy ARTMAP neural network is presented. The category choice of this network is based on the L-p norm. Geometrical properties of this architecture are presented. Comparisons between this category choice and the category choice of the Fuzzy ARTMAP are illustrated. And simulation results on the databases taken from the UCI repository are performed. It will be shown that using the L-p norm is geometrically more attractive. It will operate directly on the input patterns without the need for doing any preprocessing. It should be noted that the Fuzzy ARTMAP architecture requires two preprocessing steps: normalization and complement coding. Simulation results on different databases show the good generalization performance of the L-p Fuzzy ARTMAP compared to the performance of Fuzzy ARTMAP.     

Convergent on-line algorithms for supervised learning in neural networks

We define online algorithms for neural network training, based on the construction of multiple copies of the network, which are trained by employing different data blocks. It is shown that suitable training algorithms can be defined, in a way that the disagreement between the different copies of the network is asymptotically reduced, and convergence toward stationary points of the global error function can be guaranteed. Relevant features of the proposed approach are that the learning rate must be not necessarily forced to zero and that real-time learning is permitted.     

Combining linear discriminant functions with neural networks for supervised learning

A novel supervised learning method is proposed by combining linear discriminant functions with neural networks. The proposed method results in a tree-structured hybrid architecture. Due to constructive learning, the binary tree hierarchical architecture is automatically generated by a controlled growing process for a specific supervised learning task. Unlike the classic decision tree, the linear discriminant functions are merely employed in the intermediate level of the tree for heuristically partitioning a large and complicated task into several smaller and simpler subtasks in the proposed method. These subtasks are dealt with by component neural networks at the leaves of the tree accordingly. For constructive learning, growing and credit-assignment algorithms are developed to serve for the hybrid architecture. The proposed architecture provides an efficient way to apply existing neural networks (e.g. multi-layered perceptron) for solving a large scale problem. We have already applied the proposed method to a universal approximation problem and several benchmark classification problems in order to evaluate its performance. Simulation results have shown that the proposed method yields better results and faster training in comparison with the multilayered perceptron.     

Looking for trouble: A multilevel analysis of disagreeable contacts in online social networks

Identifying characteristics of troublemakers in online social networks, those contacts who violate norms via disagreeable or unsociable behaviour, is vital for supporting preventative strategies for undesirable, psychologically damaging online interactions. To date characterising troublemakers has relied on self-reports focused on the network holder, largely overlooking the role of network friends. In the present study, information was obtained on 5113 network contacts from 52 UK-based Facebook users (age range 13–45; 75% female) using digitally derived data and in-depth network surveys. Participants rated their contacts in terms of online disagreement, relational closeness and interaction patterns. Characteristics of online troublemakers were explored using binary logistic multilevel analysis. Instances of online disagreement were most apparent in the networks of emerging adults (19–21 years). Contacts were more likely to be identified as online troublemakers if they were well connected within the network. Rates of offline and Facebook exchanges interacted such that contacts known well offline but with low rates of Facebook communication were more likely to be identified as troublemakers. This may indicate that users were harbouring known troublemakers in a bid to preserve offline relationships and reputational status. Implications are discussed in terms of an individual's susceptibility to undesirable encounters online.     

Evolving fuzzy neural networks for supervised/unsupervised onlineknowledge-based learning

This paper introduces evolving fuzzy neural networks (EFuNNs) as a means for the implementation of the evolving connectionist systems (ECOS) paradigm that is aimed at building online, adaptive intelligent systems that have both their structure and functionality evolving in time. EFuNNs evolve their structure and parameter values through incremental, hybrid supervised/unsupervised, online learning. They can accommodate new input data, including new features, new classes, etc., through local element tuning. New connections and new neurons are created during the operation of the system. EFuNNs can learn spatial-temporal sequences in an adaptive way through one pass learning and automatically adapt their parameter values as they operate. Fuzzy or crisp rules can be inserted and extracted at any time of the EFuNN operation. The characteristics of EFuNNs are illustrated on several case study data sets for time series prediction and spoken word classification. Their performance is compared with traditional connectionist methods and systems. The applicability of EFuNNs as general purpose online learning machines, what concerns systems that learn from large databases, life-long learning systems, and online adaptive systems in different areas of engineering are discussed.     

An application of pruning in the design of neural networks for real time flood forecasting

We propose the application of pruning in the design of neural networks for hydrological prediction. The basic idea of pruning algorithms, which have not been used in water resources problems yet, is to start from a network which is larger than necessary, and then remove the parameters that are less influential one at a time, designing a much more parameter-parsimonious model. We compare pruned and complete predictors on two quite different Italian catchments. Remarkably, pruned models may provide better generalization than fully connected ones, thus improving the quality of the forecast. Besides the performance issues, pruning is useful to provide evidence of inputs relevance, removing measuring stations identified as redundant (30–40% in our case studies) from the input set. This is a desirable property in the system exercise since data may not be available in extreme situations such as floods; the smaller the set of measuring stations the model depends on, the lower the probability of system downtimes due to missing data. Furthermore, the Authority in charge of the forecast system may decide for real-time operations just to link the gauges of the pruned predictor, thus saving costs considerably, a critical issue in developing countries.

Exploring album structure for face recognition in online social networks

In this paper, we propose an album-oriented face-recognition model that exploits the album structure for face recognition in online social networks. Albums, usually associated with pictures of a small group of people at a certain event or occasion, provide vital information that can be used to effectively reduce the possible list of candidate labels. We show how this intuition can be formalized into a model that expresses a prior on how albums tend to have many pictures of a small number of people. We also show how it can be extended to include other information available in a social network. Using two real-world datasets independently drawn from Facebook, we show that this model is broadly applicable and can significantly improve recognition rates.     

Controlling opinion propagation in online networks

Motivated by applications such as the spread of ideologies and political views, we study opinion dynamics in online networks under voter models. It is well known that the binary version of these models, where the state (or opinion) of each agent is 0 or 1, always leads to consensus. We consider an extension, in which some nodes are “stubborn”, i.e., do not change their states based on other nodes. In such a system, the asymptotic average opinion could be between 0 and 1. The goal of this paper is to study the ease with which bias (i.e., the tendency of the opinion to become close to 0) can be controlled (so that the average opinion exceeds a specified threshold). We formalize a new parameter, called the Minimum Opinion Control Factor (MOCF), to capture this, and study it through analysis and simulations on real online and synthetic networks. Finally, we experimentally demonstrate the usefulness of combining the voter model with an independent cascade model in controlling bias and we explain these findings in terms of network structure.