Qualitative behavior rules for the cold rolling process extracted from trained ANN via the FCANN method

Nowadays, artificial neural networks (ANN) are being widely used in the representation of different systems and physics processes. In this paper, a neural representation of the cold rolling process will be considered. In general, once trained, the networks are capable of dealing with operational conditions not seen during the training process, keeping acceptable errors in their responses. However, humans cannot assimilate the knowledge kept by those networks, since such knowledge is implicit and difficult to be extracted. For this reason, the neural networks are considered a “black-box”.In this work, the FCANN method based on formal concept analysis (FCA) is being used in order to extract and represent knowledge from previously trained ANN. The new FCANN approach permits to obtain a non-redundant canonical base with minimum implications, which qualitatively describes the process. The approach can be used to understand the relationship among the process parameters through implication rules in different operational conditions on the load-curve of the cold rolling process. Metrics for evaluation of the rules extraction process are also proposed, which permit a better analysis of the results obtained.     

Effective neuronal learning with ineffective Hebbian learning rules

In this article we revisit the classical neuroscience paradigm of Hebbian learning. We find that it is difficult to achieve effective associative memory storage by Hebbian synaptic learning, since it requires network-level information at the synaptic level or sparse coding level. Effective learning can yet be achieved even with nonsparse patterns by a neuronal process that maintains a zero sum of the incoming synaptic efficacies. This weight correction improves the memory capacity of associative networks from an essentially bounded one to a memory capacity that scales linearly with network size. It also enables the effective storage of patterns with multiple levels of activity within a single network. Such neuronal weight correction can be successfully carried out by activity-dependent homeostasis of the neuron's synaptic efficacies, which was recently observed in cortical tissue. Thus, our findings suggest that associative learning by Hebbian synaptic learning should be accompanied by continuous remodeling of neuronally driven regulatory processes in the brain.     

Extracting M-of-N rules from trained neural networks

An effective algorithm for extracting M-of-N rules from trained feedforward neural networks is proposed. First, we train a network where each input of the data can only have one of the two possible values, -1 or one. Next, we apply the hyperbolic tangent function to each connection from the input layer to the hidden layer of the network. By applying this squashing function, the activation values at the hidden units are effectively computed as the hyperbolic tangent (or the sigmoid) of the weighted inputs, where the weights have magnitudes that are equal one. By restricting the inputs and the weights to binary values either -1 or one, the extraction of M-of-N rules from the networks becomes trivial. We demonstrate the effectiveness of the proposed algorithm on several widely tested datasets. For datasets consisting of thousands of patterns with many attributes, the rules extracted by the algorithm are simple and accurate.     

SLAVE: a genetic learning system based on an iterative approach

SLAVE is an inductive learning algorithm that uses concepts based on fuzzy logic theory. This theory has been shown to be a useful representational tool for improving the understanding of the knowledge obtained from a human point of view. Furthermore, SLAVE uses an iterative approach for learning based on the use of a genetic algorithm (GA) as a search algorithm. We propose a modification of the initial iterative approach used in SLAVE. The main idea is to include more information in the process of learning one individual rule. This information is included in the iterative approach through a different proposal of calculus of the positive and negative example to a rule. Furthermore, we propose the use of a new fitness function and additional genetic operators that reduce the time needed for learning and improve the understanding of the rules obtained     

Exploring comprehensible classification rules from trained neural networks integrated with a time-varying binary particle swarm optimizer

Purpose

Extracting comprehensible classification rules is the most emphasized concept in data mining researches. In order to obtain accurate and comprehensible classification rules from databases, a new approach is proposed by combining advantages of artificial neural networks (ANN) and swarm intelligence.

Method

Artificial neural networks (ANNs) are a group of very powerful tools applied to prediction, classification and clustering in different domains. The main disadvantage of this general purpose tool is the difficulties in its interpretability and comprehensibility. In order to eliminate these disadvantages, a novel approach is developed to uncover and decode the information hidden in the black-box structure of ANNs. Therefore, in this paper a study on knowledge extraction from trained ANNs for classification problems is carried out. The proposed approach makes use of particle swarm optimization (PSO) algorithm to transform the behaviors of trained ANNs into accurate and comprehensible classification rules. Particle swarm optimization with time varying inertia weight and acceleration coefficients is designed to explore the best attribute-value combination via optimizing ANN output function.

Results

The weights hidden in trained ANNs turned into comprehensible classification rule set with higher testing accuracy rates compared to traditional rule based classifiers.     

An analog VLSI design for a neuron with a choice of learning rules

An analog implementation of a neuron using standard VLSI components is described. The node is capable of both delta-rule and simple error-correcting learning. Decomposition into functional blocks allows the parts of the design to be easily separated and understood. The connectivity problem is eased by serially encoding inputs so that all nodes in a layer are connected to a single line carrying activations from the previous layer. Performance implications of the architecture are considered. The design was simulated with the Spice transistor level simulator. Schemas for interconnection of large numbers of nodes and simulations of the circuitry required are presented. Results show that effective learning is achieved by both algorithms. Implementation of multiple learning rules in a single neuron is demonstrated as an effective way of increasing flexibility in neural network hardware implementations.     

Inhomogeneities in heteroassociative memories with linear learning rules

We investigate how various inhomogeneities present in synapses and neurons affect the performance of feedforward associative memories with linear learning, a high-level network model of hippocampal circuitry and plasticity. The inhomogeneities incorporated into the model are differential input attenuation, stochastic synaptic transmission, and memories learned with varying intensity. For a class of local learning rules, we determine the memory capacity of the model by extending previous analysis. We find that the signal-to-noise ratio (SNR), a measure of fidelity of recall, depends on the coefficients of variation (CVs) of the attenuation factors, the transmission variables, and the intensity of the memories, as well as the parameters of the learning rule, pattern sparsity and the number of memories stored. To predict the effects of attenuation due to extended dendritic trees, we use distributions of attenuations appropriate to unbranched and branched dendritic trees. Biological parameters for stochastic transmission are used to determine the CV of the transmission factors. The reduction in SNR due to differential attenuation is surprisingly low compared to the reduction due to stochastic transmission. Training a network by storing memories at different intensities is equivalent to using a learning rule incorporating weight decay. In this type of network, new memories can be stored continuously at the expense of older ones being forgotten (a palimpsest). We show that there is an optimal rate of weight decay that maximizes the capacity of the network, which is a factor of e lower than its nonpalimpsest equivalent.     

Recognition by variance: learning rules for spatiotemporal patterns

Recognizing specific spatiotemporal patterns of activity, which take place at timescales much larger than the synaptic transmission and membrane time constants, is a demand from the nervous system exemplified, for instance, by auditory processing. We consider the total synaptic input that a single readout neuron receives on presentation of spatiotemporal spiking input patterns. Relying on the monotonic relation between the mean and the variance of a neuron's input current and its spiking output, we derive learning rules that increase the variance of the input current evoked by learned patterns relative to that obtained from random background patterns. We demonstrate that the model can successfully recognize a large number of patterns and exhibits a slow deterioration in performance with increasing number of learned patterns. In addition, robustness to time warping of the input patterns is revealed to be an emergent property of the model. Using a leaky integrate-and-fire realization of the readout neuron, we demonstrate that the above results also apply when considering spiking output.     

Inferring new design rules by machine learning: a case study oftopological optimization

This paper presents a machine learning approach to the topological optimization of computer networks. Traditionally formulated as an integer program, this problem is well known to be a very difficult one, only solvable by means of heuristic methods. This paper addresses the specific problem of inferring new design rules that can reduce the cost of the network, or reduce the message delay below some acceptable threshold. More specifically, it extends a recent approach using a rule-based system in order to prevent the risk of combinatorial explosion and to reduce the search space of feasible network topologies. This extension essentially implements an efficient inductive learning algorithm leading to the refinement of existing rules and to the discovery of new rules from examples, defined as network topologies satisfying a given reliability constraint. The contribution of this paper is the integration of learning capabilities into topological optimization of computer networks. Computational results confirm the efficiency of the discovered rules     

Discovery of unapparent association rules based on extracted probability

Association rule mining is an important task in data mining. However, not all of the generated rules are interesting, and some unapparent rules may be ignored. We have introduced an “extracted probability” measure in this article. Using this measure, 3 models are presented to modify the confidence of rules. An efficient method based on the support-confidence framework is then developed to generate rules of interest. The adult dataset from the UCI machine learning repository and a database of occupational accidents are analyzed in this article. The analysis reveals that the proposed methods can effectively generate interesting rules from a variety of association rules.     

Some learning techniques in hierarchical censored production rules (HCPRs) system

This paper discusses learning techniques based upon the hierarchical censored production rules (HCPRs) system of knowledge representation. These HCPRs are written in the form: “A IF B UNLESS C GENERALITY G SPECIFICITY S,” where symbol A represents the conclusion, B is the set of preconditions, C is the set of exception conditions, G is the general information, while S represents the specific information. Learning can be classified into two major categories: the first includes the restructuring or modification of existing knowledge, and the second covers the creation of new knowledge depending upon externally supplied information and already acquired knowledge. In this system, schemes which modify various belief factors and information relegated to various operators (like IF, UNLESS, etc.) of an HCPR fall in the first category, while schemes which create a new HCPR in the system by using externally supplied information and already acquired knowledge fall in the second category. Using the growth algorithm, a new HCPR is added in the system by maintaining consistency as well as minimizing redundancy. The set of all related HCPRs connected to the SPECIFICITY or GENERALITY operators are shown to possess a tree structure, and hence it is given the name HCPRs tree. The fission algorithm restructures an HCPRs tree, thereby enabling the system to reorganize its knowledge base; a new HCPR may be created during this process. This is followed by the fusion algorithm that enables the merging of two related HCPRs trees in the HCPRs system. © 1998 John Wiley & Sons, Inc.     

Input-dependent neural network trained by real-coded genetic algorithm and its industrial applications

This paper presents an input-dependent neural network (IDNN) with variable parameters. The parameters of the neurons in the hidden nodes adapt to changes of the input environment, so that different test input sets separately distributed in a large domain can be tackled after training. Effectively, there are different individual neural networks for different sets of inputs. The proposed network exhibits a better learning and generalization ability than the traditional one. An improved real-coded genetic algorithm (RCGA) Ling and Leung (Soft Comput 11(1):7–31, 2007) is proposed to train the network parameters. Industrial applications on short-term load forecasting and hand-written graffiti recognition will be presented to verify and illustrate the improvement.     

A personalized e-course composition based on a genetic algorithm with forcing legality in an adaptive learning system

This paper proposes a personalized e-course composition based on a genetic algorithm with forcing legality (called GA^?) in adaptive learning systems, which efficiently and accurately finds appropriate e-learning materials in the database for individual learners. The forcing legality operation not only reduces the search space size and increases search efficiency but also is more explicit in finding the best e-course composition in a legal solution space. In serial experiments, the forcing legality operation is applied in Chu et al.'s the particle swarm optimization (called PSO^?) and Dheeban et al.'s the improved particle swarm optimization (called RPSO^?) to show the forcing legality can speed up the computational time and reduce the computational complexity of algorithm. Furthermore, GA^? regardless of the number of students or the number of materials in the database, to compose a personalized e-course within a limited time is much more efficient and accurate than PSO^? and RPSO^?. For the experiment increasing the number of students to 1200, the average improvement ratios of errors (learning concept error, materials difficulty error, learning time error), fitness value, stability, and execution time are above 96%, 79%, 90%, and 10%, respectively. For the experiment increasing the number of materials to 500 and the execution time set to the shortest execution time of RPSO^?, the average improvement ratios of errors (learning concept error, materials difficulty error, learning time error), fitness value, and stability are above 97%, 51%, and 80%, respectively. Therefore, GA^? is able to enhance the quality of personalized e-course compositions in adaptive learning environments.     

Applying a new localized generalization error model to design neural networks trained with extreme learning machine

High accuracy and low overhead are two key features of a well-designed classifier for different classification scenarios. In this paper, we propose an improved classifier using a single-hidden layer feedforward neural network (SLFN) trained with extreme learning machine. The novel classifier first utilizes principal component analysis to reduce the feature dimension and then selects the optimal architecture of the SLFN based on a new localized generalization error model in the principal component space. Experimental and statistical results on the NSL-KDD data set demonstrate that the proposed classifier can achieve a significant performance improvement compared with previous classifiers.     

Evolving lane merge traffic behaviour simulations via a macroscopic objective function and a machine learning system trained through bootstrapped human judgement

Multi-agent based traffic simulation models have become increasingly important in simulating, studying and analysing traffic behaviours due to their ability to model more sophisticated behaviours of traffic by codifying simple rules into agents. However, such models require selection of appropriate rules and tuning of parameters for the selected rules. This process demands extensive resources if to be done manually. Further, high complexity models (in terms of number of rules and parameters) require a large computational cost to run, imposing scalability problems. In this work, four simple rules are introduced by reformulating existing concepts in the literature in order to simulate the self-organising behaviour of traffic where two lanes form into one and when two types of vehicles (cars and trucks) are present. The optimal rule and parameter combinations are explored via an evolutionary framework to overcome the resource demanding nature of the process. Two forms of objective functions - 1) a macroscopic objective function which focuses on macroscopic properties of traffic 2) a machine learning system trained based on human judgement concerning microscopic interaction of traffic - are studied in order to evolve low complexity and high fidelity traffic simulations. The differences in the rule sets evolved by the two objective functions are discussed highlighting the importance of selecting an appropriate objective function based on the simulation requirements and available resources. Finally, the change of the rule distribution as a function of generation in the evolutionary process is investigated in order to understand the complexity change in the simulations as a function of rule count as simulations are evolving towards high fidelity. This provides an abstract understanding of the relationship between complexity and fidelity in multi-agent based simulations concerning the particular problem of simulation of lane merge traffic.     

Function approximation based on fuzzy rules extracted frompartitioned numerical data

We present an efficient method for extracting fuzzy rules directly from numerical input-output data for function approximation problems. First, we convert a given function approximation problem into a pattern classification problem. This is done by dividing the universe of discourse of the output variable into multiple intervals, each regarded as a class, and then by assigning a class to each of the training data according to the desired value of the output variable. Next, we partition the data of each class in the input space to achieve a higher accuracy in approximation of class regions. Partition terminates according to a given criterion to prevent excessive partition. For class region approximation, we discuss two different types of representations using hyperboxes and ellipsoidal regions, respectively. Based on a selected representation, we then extract fuzzy rules from the approximated class regions. For a given input datum, we convert, or in other words, defuzzify, the resulting vector of the class membership degrees into a single real value. This value represents the final result approximated by the method. We test the presented method on a synthetic nonlinear function approximation problem and a real-world problem in an application to a water purification plant. We also compare the presented method with a method based on neural networks.     

Recurrent sparse support vector regression machines trained by active learning in the time-domain

A method for the sparse solution of recurrent support vector regression machines is presented. The proposed method achieves a high accuracy versus complexity and allows the user to adjust the complexity of the resulting model. The sparse representation is guaranteed by limiting the number of training data points for the support vector regression method. Each training data point is selected based on the accuracy of the fully recurrent model using the active learning principle applied to the successive time-domain data. The user can adjust the training time by selecting how often the hyper-parameters of the algorithm should be optimised. The advantages of the proposed method are illustrated on several examples, and the experiments clearly show that it is possible to reduce the number of support vectors and to significantly improve the accuracy versus complexity of recurrent support vector regression machines.     

Evolutionary learning of hierarchical decision rules

This paper describes an approach based on evolutionary algorithms, hierarchical decision rules (HIDER), for learning rules in continuous and discrete domains. The algorithm produces a hierarchical set of rules, that is, the rules are sequentially obtained and must therefore be tried until one is found whose conditions are satisfied. Thus, the number of rules may be reduced because the rules could be inside of one another. The evolutionary algorithm uses both real and binary coding for the individuals of the population. We tested our system on real data from the UCI repository, and the results of a ten-fold cross-validation are compared to C4.5s, C4.5Rules, See5s, and See5Rules. The experiments show that HIDER works well in practice.     

Evolution of a vision system by genetic algorithm

This paper proposes a framework for a genetic algorithm applied to determine and construct an organ, especially the neural network of a virtual creature. The vision system of the creature is a result of genetic evolution, and we are trying to realize this on the computer. We examine how the visual organ of the animal is evolved under a special environment (e.g., the specialized visual organ of an animal to catch a moving insect), and how many variations of neural networks exist. We also think it is possible to generalize the method to an automatic generation of various kinds of visual recognition system by adding various kinds of evolution any directions. This work was presented, in part, at the Second International Symposium on Artificial Life and Robotics, Oita, Japan, February 18–20, 1997     

A genetic fuzzy expert system for automatic question classification in a competitive learning environment

Intelligent tutoring systems are efficient tools to automatically adapt the learning process to the student’s progress and needs. One of the possible adaptations is to apply an adaptive question sequencing system, which matches the difficulty of the questions to the student’s knowledge level. In this context, it is important to correctly classify the questions to be presented to students according to their difficulty level. Many systems have been developed for estimating the difficulty of questions. However the variety in the application environments makes difficult to apply the existing solutions directly to other applications. Therefore, a specific solution has been designed in order to determine the difficulty level of open questions in an automatic and objective way. This solution can be applied to activities with special temporal and running features, as the contests developed through QUESTOURnament, which is a tool integrated into the e-learning platform Moodle. The proposed solution is a fuzzy expert system that uses a genetic algorithm in order to characterize each difficulty level. From the output of the algorithm, it defines the fuzzy rules that are used to classify the questions. Data registered from a competitive activity in a Telecommunications Engineering course have been used in order to validate the system against a group of experts. Results show that the system performs successfully. Therefore, it can be concluded that the system is able to do the questions classification labour in a competitive learning environment.