A concept learning network based on correlation and backpropagation

A new concept learning neural network is presented. This network builds correlation learning into a rule learning neural network where the certainty factor model of traditional expert systems is taken as the network activation function. The main argument for this approach is that correlation learning can help when the neural network fails to converge to the target concept due to insufficient or noisy training data. Both theoretical analysis and empirical evaluation are provided to validate the system.     

A new backpropagation learning algorithm for layered neural networks with nondifferentiable units

We propose a digital version of the backpropagation algorithm (DBP) for three-layered neural networks with nondifferentiable binary units. This approach feeds teacher signals to both the middle and output layers, whereas with a simple perceptron, they are given only to the output layer. The additional teacher signals enable the DBP to update the coupling weights not only between the middle and output layers but also between the input and middle layers. A neural network based on DBP learning is fast and easy to implement in hardware. Simulation results for several linearly nonseparable problems such as XOR demonstrate that the DBP performs favorably when compared to the conventional approaches. Furthermore, in large-scale networks, simulation results indicate that the DBP provides high performance.     

Supervised contrastive learning over prototype-label embeddings for network intrusion detection

Contrastive learning makes it possible to establish similarities between samples by comparing their distances in an intermediate representation space (embedding space) and using loss functions designed to attract/repel similar/dissimilar samples. The distance comparison is based exclusively on the sample features. We propose a novel contrastive learning scheme by including the labels in the same embedding space as the features and performing the distance comparison between features and labels in this shared embedding space. Following this idea, the sample features should be close to its ground-truth (positive) label and away from the other labels (negative labels). This scheme allows to implement a supervised classification based on contrastive learning. Each embedded label will assume the role of a class prototype in embedding space, with sample features that share the label gathering around it. The aim is to separate the label prototypes while minimizing the distance between each prototype and its same-class samples. A novel set of loss functions is proposed with this objective. Loss minimization will drive the allocation of sample features and labels in embedding space. Loss functions and their associated training and prediction architectures are analyzed in detail, along with different strategies for label separation. The proposed scheme drastically reduces the number of pair-wise comparisons, thus improving model performance. In order to further reduce the number of pair-wise comparisons, this initial scheme is extended by replacing the set of negative labels by its best single representative: either the negative label nearest to the sample features or the centroid of the cluster of negative labels. This idea creates a new subset of models which are analyzed in detail.The outputs of the proposed models are the distances (in embedding space) between each sample and the label prototypes. These distances can be used to perform classification (minimum distance label), features dimensionality reduction (using the distances and the embeddings instead of the original features) and data visualization (with 2 or 3D embeddings).Although the proposed models are generic, their application and performance evaluation is done here for network intrusion detection, characterized by noisy and unbalanced labels and a challenging classification of the various types of attacks. Empirical results of the model applied to intrusion detection are presented in detail for two well-known intrusion detection datasets, and a thorough set of classification and clustering performance evaluation metrics are included.     

Improving hypergraph convolution network collaborative filtering with feature crossing and contrastive learning

Organizing user-item interaction data into a graph has brought many benefits to recommendation methods. Compared with the user-item bipartite graph structure, a hypergraph structure provides a natural way to directly model high-order correlations among users or items. Hypergraph Convolution Network (HGCN) has the capability of aggregating and propagating latent features of nodes in the hypergraph nonlinearly. Recently, recommendation models based on simplified HGCN have shown good performance. However, such models lose the powerful expression ability of feature crossing and suffer from limited labeled data. To tackle these two problems, a framework called HGCN-CC is proposed to improve HGCN with feature Crossing and Contrastive learning. Specifically, HGCN is combined with a feature cross network in a parallel manner to balance between feature crossing and over smoothing. By such a design, HGCN-CC not only utilizes simplified propagation operation in HGCN to capture high-order correlations among users or items, but also enjoys the powerful expressing ability of high-order feature interactions. Furthermore, HGCN-CC resorts to contrastive learning to help learn good representations. Under the HGCN-CC framework, two models called item-based HGCN-CC (I-HGCN-CC) and user-based HGCN-CC (U-HGCN-CC) are constructed to emphasize different aspects of data. Results of extensive experiments on four benchmark datasets demonstrate that proposed models have superiority in modelling hypergraph structure data for recommendations.

     

Fast and improved backpropagation learning of multi-layer artificial neural network using adaptive activation function

In the conventional backpropagation (BP) learning algorithm used for the training of the connecting weights of the artificial neural network (ANN), a fixed slope−based sigmoidal activation function is used. This limitation leads to slower training of the network because only the weights of different layers are adjusted using the conventional BP algorithm. To accelerate the rate of convergence during the training phase of the ANN, in addition to updates of weights, the slope of the sigmoid function associated with artificial neuron can also be adjusted by using a newly developed learning rule. To achieve this objective, in this paper, new BP learning rules for slope adjustment of the activation function associated with the neurons have been derived. The combined rules both for connecting weights and slopes of sigmoid functions are then applied to the ANN structure to achieve faster training. In addition, two benchmark problems: classification and nonlinear system identification are solved using the trained ANN. The results of simulation-based experiments demonstrate that, in general, the proposed new BP learning rules for slope and weight adjustments of ANN provide superior convergence performance during the training phase as well as improved performance in terms of root mean square error and mean absolute deviation for classification and nonlinear system identification problems.     

Self-association and Hebbian learning in linear neural networks

Studies Hebbian learning in linear neural networks with emphasis on the self-association information principle. This criterion, in one-layer networks, leads to the space of the principal components and can be generalized to arbitrary architectures. The self-association paradigm appears to be very promising because it accounts for the fundamental features of Hebbian synaptic learning and generalizes the various techniques proposed for adaptive principal component networks. The authors also include a set of simulations that compare various neural architectures and algorithms.     

Hebbian learning of recurrent connections: a geometrical perspective

We show how a Hopfield network with modifiable recurrent connections undergoing slow Hebbian learning can extract the underlying geometry of an input space. First, we use a slow and fast analysis to derive an averaged system whose dynamics derives from an energy function and therefore always converges to equilibrium points. The equilibria reflect the correlation structure of the inputs, a global object extracted through local recurrent interactions only. Second, we use numerical methods to illustrate how learning extracts the hidden geometrical structure of the inputs. Indeed, multidimensional scaling methods make it possible to project the final connectivity matrix onto a Euclidean distance matrix in a high-dimensional space, with the neurons labeled by spatial position within this space. The resulting network structure turns out to be roughly convolutional. The residual of the projection defines the nonconvolutional part of the connectivity, which is minimized in the process. Finally, we show how restricting the dimension of the space where the neurons live gives rise to patterns similar to cortical maps. We motivate this using an energy efficiency argument based on wire length minimization. Finally, we show how this approach leads to the emergence of ocular dominance or orientation columns in primary visual cortex via the self-organization of recurrent rather than feedforward connections. In addition, we establish that the nonconvolutional (or long-range) connectivity is patchy and is co-aligned in the case of orientation learning.     

Hebbian and error-correction learning for complex-valued neurons

In this paper, we observe some important aspects of Hebbian and error-correction learning rules for complex-valued neurons. These learning rules, which were previously considered for the multi-valued neuron (MVN) whose inputs and output are located on the unit circle, are generalized for a complex-valued neuron whose inputs and output are arbitrary complex numbers. The Hebbian learning rule is also considered for the MVN with a periodic activation function. It is experimentally shown that Hebbian weights, even if they still cannot implement an input/output mapping to be learned, are better starting weights for the error-correction learning, which converges faster starting from the Hebbian weights rather than from the random ones.     

Fuzzy assisted learning in backpropagation neural networks

This paper reports on studies to overcome difficulties associated with setting the learning rates of backpropagation neural networks by using fuzzy logic. Building on previous research, a fuzzy control system is designed which is capable of dynamically adjusting the individual learning rates of both hidden and output neurons, and the momentum term within a back-propagation network. Results show that the fuzzy controller not only eliminates the effort of configuring a global learning rate, but also increases the rate of convergence in comparison with a conventional backpropagation network. Comparative studies are presented for a number of different network configurations. The paper also presents a brief overview of fuzzy logic and backpropagation learning, highlighting how the two paradigms can enhance each other.     

Efficient mapping of backpropagation algorithm onto a network ofworkstations

In this paper, we present an efficient technique for mapping a backpropagation (BP) learning algorithm for multilayered neural networks onto a network of workstations (NOW's). We adopt a vertical partitioning scheme, where each layer in the neural network is divided into p disjoint partitions, and map each partition onto an independent workstation in a network of p workstations. We present a fully distributed version of the BP algorithm and also its speedup analysis. We compare the performance of our algorithm with a recent work involving the vertical partitioning approach for mapping the BP algorithm onto a distributed memory multiprocessor. Our results on SUN 3/50 NOW's show that we are able to achieve better speedups by using only two communication sets and also by avoiding some redundancy in the weights computation for one training cycle of the algorithm.     

Stable dynamic backpropagation learning in recurrent neuralnetworks

To avoid unstable phenomenon during the learning process, two new learning schemes, called the multiplier and constrained learning rate algorithms, are proposed in this paper to provide stable adaptive updating processes for both the synaptic and somatic parameters of the network. Based on the explicit stability conditions, in the multiplier method these conditions are introduced into the iterative error index, and the new updating formulations contain a set of inequality constraints. In the constrained learning rate algorithm, the learning rate is updated at each iterative instant by an equation derived using the stability conditions. With these stable dynamic backpropagation algorithms, any analog target pattern may be implemented by a steady output vector which is a nonlinear vector function of the stable equilibrium point. The applicability of the approaches presented is illustrated through both analog and binary pattern storage examples.     

Analysis of learning rate and momentum term in backpropagation neural network algorithm trained to predict pavement performance

Pavement performance modeling is a critical component of any pavement management system (PMS) decision-making process. A characteristic feature of pavement performance models is that they are formulated and estimated statistically from field data. The statistical modeling can only consider no more than a few of the parameters, in a simplified manner, and in some cases various transformations of the original data. Lately, artificial neural networks (ANNs) were applied to pavement performance modeling. The ANNs offer a number of advantages over the traditional statistical methods, caused by their generalization, massive parallelism and ability to offer real time solutions. Unfortunately, in pavement performance modeling, only simulated data were used in ANNs environment. In this paper, real pavement condition and traffic data and specific architecture are used to investigate the effect of learning rate and momentum term on back-propagation algorithm neural network trained to predict flexible pavement performance. On the basis of the analysis it is concluded that an extremely low learning rate around 0.001–0.005 combination and momentum term between 0.5–0.9 do not give satisfactory results for the specific data set and the architecture used. It is also established that the learning rate and momentum term, and validation data can be used to identify when over-learning is taking place in a training set.     

Charting the behavioural state of a person using a backpropagation neural network

This paper describes the application of a backpropagation artificial neural network (ANN) for charting the behavioural state of previously unseen persons. In a simulated theft scenario participants stole or did not steal some money and were interviewed about the location of the money. A video of each interview was presented to an automatic system, which collected vectors containing nonverbal behaviour data. Each vector represented a participant’s nonverbal behaviour related to “deception” or “truth” for a short period of time. These vectors were used for training and testing a backpropagation ANN which was subsequently used for charting the behavioural state of previously unseen participants. Although behaviour related to “deception” or “truth” is charted the same strategy can be used to chart different psychological states over time and can be tuned to particular situations, environments and applications. We thank those who kindly volunteered to participate in the study.     

A nonlinear extension of the Generalized Hebbian learning

In this letter, we introduce a nonlinear hierarchic PCA type neural network with a simple architecture. The learning algorithm is a kind of nonlinear extension of the well-known Sanger's Generalized Hebbian Algorithm (GHA). It is derived from a nonlinear optimization criterion. Experiments with sinusoidal data show that the neurons become sensitive to different sinusoids. Standard linear PCA algorithms don't have such a separation property.     

An application of Hebbian learning in the design process decision-making

The design of a product involves a process in which several different aspects are combined in order to obtain a final, suitable and optimum product. Designers must interact with different stakeholder groups, make decisions and complete the design process. In order to achieve this, different evaluation techniques are used. Depending on the chosen technique and on the field and environment in which each member of the design team was trained, each one of the members will consider one or several aspects of the design project but from a point of view or perspective in line with his/her particular professional background. As a result, all decisions which will affect the design process of the product are focused on these aspects and individual viewpoints. In this paper, an evaluation technique is proposed which allows one to take suitable decisions, taking into account all the factors and perspectives which affect the design process in the best way, searching for a balance among them in relation to the aims and interests of a specific design project. The development of this evaluation technique was inspired by the way in which neurons interact with one another in the brain and it has been based on the Hebbian learning rule for neural networks. Lastly, a real application of the proposed technique is presented to demonstrate its applicability in evaluating industrial designs.     

Dynamic learning rate optimization of the backpropagation algorithm

It has been observed by many authors that the backpropagation (BP) error surfaces usually consist of a large amount of flat regions as well as extremely steep regions. As such, the BP algorithm with a fixed learning rate will have low efficiency. This paper considers dynamic learning rate optimization of the BP algorithm using derivative information. An efficient method of deriving the first and second derivatives of the objective function with respect to the learning rate is explored, which does not involve explicit calculation of second-order derivatives in weight space, but rather uses the information gathered from the forward and backward propagation, Several learning rate optimization approaches are subsequently established based on linear expansion of the actual outputs and line searches with acceptable descent value and Newton-like methods, respectively. Simultaneous determination of the optimal learning rate and momentum is also introduced by showing the equivalence between the momentum version BP and the conjugate gradient method. Since these approaches are constructed by simple manipulations of the obtained derivatives, the computational and storage burden scale with the network size exactly like the standard BP algorithm, and the convergence of the BP algorithm is accelerated with in a remarkable reduction (typically by factor 10 to 50, depending upon network architectures and applications) in the running time for the overall learning process. Numerous computer simulation results are provided to support the present approaches.     

Performance of a backpropagation trained feedforward network on an MIMD architecture

Training of feedforward networks on sequential machines is a computationally expensive process. This has motivated the implementation of parallel versions of the backpropagation training algorithm on different parallel platforms in order to decrease the processing time required for training. In this paper, we are investigating the implementation of backpropagation on the Alex AVX-2 coarse-grained MIMD machine. A master–slave parallel implementation is carried out for the encoder–decoder benchmark problem. A communication model for the broadcasting/gathering is used to study the effect of using different topologies. Then the performance of the backpropagation algorithms is analyzed for different network sizes and numbers of processors when the nodes are arranged as a pipeline array and in a mesh topology. © 1998 John Wiley & Sons, Ltd.     

Nonlinear complex-valued extensions of Hebbian learning: an essay

The Hebbian paradigm is perhaps the best-known unsupervised learning theory in connectionism. It has inspired wide research activity in the artificial neural network field because it embodies some interesting properties such as locality and the capability of being applicable to the basic weight-and-sum structure of neuron models. The plain Hebbian principle, however, also presents some inherent theoretical limitations that make it impractical in most cases. Therefore, modifications of the basic Hebbian learning paradigm have been proposed over the past 20 years in order to design profitable signal and data processing algorithms. Such modifications led to the principal component analysis type class of learning rules along with their nonlinear extensions. The aim of this review is primarily to present part of the existing fragmented material in the field of principal component learning within a unified view and contextually to motivate and present extensions of previous works on Hebbian learning to complex-weighted linear neural networks. This work benefits from previous studies on linear signal decomposition by artificial neural networks, nonquadratic component optimization and reconstruction error definition, neural parameters adaptation by constrained optimization of learning criteria of complex-valued arguments, and orthonormality expression via the insertion of topological elements in the networks or by modifying the network learning criterion. In particular, the learning principles considered here and their analysis concern complex-valued principal/minor component/subspace linear/nonlinear rules for complex-weighted neural structures, both feedforward and laterally connected.     

A highly efficient implementation of a backpropagation learning algorithm using matrix ISA

BackPropagation (BP) is the most famous learning algorithm for Artificial Neural Networks (ANN). BP has received intensive research efforts to exploit its parallelism in order to reduce the training time for complex problems. A modified version of BP based on matrix–matrix multiplication was proposed for parallel processing. In this paper, we present the implementation of Matrix BackPropagation (MBP) using scalar, vector, and matrix Instruction Set Architectures (ISAs). Besides this, we show that the performance of the MBP is improved by switching from scalar ISA to vector ISA. It is further improved by switching from vector ISA to matrix ISA. On a practical application, speech recognition, the speedup of training a neural network using unrolling scalar ISA over scalar ISA is 1.83. On eight parallel lanes, the speedups of using vector, unrolling vector, and matrix ISAs are respectively 10.33, 11.88, and 15.36, where the maximum theoretical speedup is 16. The results obtained show that the use of matrix ISA gives a performance close to optimal, because of reusing the loaded data, decreasing the loop overhead, and overlapping the memory operations with arithmetic operations.     

Fuzzy grey cognitive maps and nonlinear Hebbian learning in process control

Fuzzy Grey Cognitive Maps (FGCM) is an innovative Grey System theory-based FCM extension. Grey systems have become a very effective theory for solving problems within environments with high uncertainty, under discrete small and incomplete data sets. In this study, the method of FGCMs and a proposed Hebbian-based learning algorithm for FGCMs were applied to a known reference chemical process problem, concerning a control process in chemical industry with two tanks, three valves, one heating element and two thermometers for each tank. The proposed mathematical formulation of FGCMs and the implementation of the NHL algorithm were analyzed and then successfully applied keeping the main constraints of the problem. A number of numerical experiments were conducted to validate the approach and verify the effectiveness. Also, the produced results were analyzed and compared with the results previously reported in the literature from the implementation of the FCMs and Nonlinear Hebbian learning algorithm. The advantages of FGCMs over conventional FCMs are their capabilities (i) to produce a length and greyness estimation at the outputs; the output greyness can be considered as an additional indicator of the quality of a decision, and (ii) to succeed desired behavior for the process system for every set of initial states, with and without Hebbian learning.