Using single-layered neural networks for the extraction of conjunctive rules and hierarchical classifications

Machine Learning is an area concerned with the automation of the process of knowledge acquisition. Neural networks generally represent their knowledge at the lower level, while knowledge based systems use higher level knowledge representations. The method we propose here, provides a technique which automatically allows us to extract production rules from the lower level representation used by a single-layered neural networks trained by Hebb's rule. Even though a single-layered neural network can not model complex, nonlinear domains, their strength in dealing with noise has enabled us to produce correct rules in a noisy domain.     

Identification of nonlinear dynamical systems using multilayered neural networks

A novel multilayer discrete-time neural net paradigm is presented for the identification of multi-input multi-output (MIMO) nonlinear dynamical systems. The major novelty of this approach is a rigorous proof of identification error convergence that reveals a requirement for a new identifier structure and nonstandard weight tuning algorithms. The NN identifier includes modified delta rule weight tuning and exhibits a learning-while-functioning feature instead of learning-then-functioning, so that the identification is on-line with no explicit off-line learning phase needed. The structure of the neural net (NN) identifier is derived using a passivity aproach. Linearity in the parameters is not required and certainty equivalence is not used. The notion of persistency of excitation (PE) and passivity properties of the multilayer NN are defined and used in the convergence analysis of both the identification error and the weight estimates.     

An acquisition of operator's rules for collision avoidance usingfuzzy neural networks

The procedure for acquiring control rules to improve the performance of control systems has received considerable attention previously. This paper deals with a collision avoidance problem in which the controlled object is a ship with inertia which must avoid collision with a moving object. It has proven to be difficult to obtain collision avoidance rules, i.e., steering rules and speed control rules, which coincide with the operator's knowledge. This paper shows that rules of this type can be acquired directly from observational data using fuzzy neural networks (FNNs). This paper also shows that the FNN can obtain portions of the fuzzy rules for the inferences of the static and dynamic degrees of danger and the decision table based on the degrees of danger to avoid the moving obstacle     

Extracting rules from trained neural networks

Presents an algorithm for extracting rules from trained neural networks. The algorithm is a decompositional approach which can be applied to any neural network whose output function is monotone such as a sigmoid function. Therefore, the algorithm can be applied to multilayer neural networks, recurrent neural networks and so on. It does not depend on training algorithms, and its computational complexity is polynomial. The basic idea is that the units of neural networks are approximated by Boolean functions. But the computational complexity of the approximation is exponential, and so a polynomial algorithm is presented. The author has applied the algorithm to several problems to extract understandable and accurate rules. The paper shows the results for the votes data, mushroom data, and others. The algorithm is extended to the continuous domain, where extracted rules are continuous Boolean functions. Roughly speaking, the representation by continuous Boolean functions means the representation using conjunction, disjunction, direct proportion, and reverse proportion. This paper shows the results for iris data.     

Knowledge acquisition using hypertext

This paper addresses the issue of supporting knowledge acquisition using hypertext. We propose a way of tightly integrating hypertext and structured object representation, using Artificial Intelligence (AI) frames for the basic representation of hypertext nodes. Epistemologically, a dual view of the resulting space is of interest. One view is that of hypertext which emphasizes nodes containg g text, including formal knowledge representation. The other view focuses on objects with certain relationships, which define a semantic network. Both in hypertext and in semantic networks the relations between chunks of knowledge are explicitly represented by links. However, in today's hypertext systems a node typically contains just informal text and references to other nodes. Our approach additionally facilitates the explicit representation of structure “inside” hypertext nodes using partitions. We show the usefulness of such a tight integration for knowledge acquisition, providing several features useful for supporting it based on a level of basic hypertext functionality. In particular, we sketch a method for doing knowledge acquisition in such an environment. Hypertext is used as a mediating “semiformal” representation, which allows experts to directly represent knowledge without the immediate support of knowledge engineers. These help then to make this knowledge operational, supported by the system's facility to provide templates as well as their links to the semiformal representation. As an example of our results of using this method of knowledge acquisition, we illustrate the strategic knowledge in our application domain. More generally, our approach supports important aspects of (software) engineering knowledge-based systems and their maintenance. Also their user interface can be improved this way.     

Extracting M-of-N rules from trained neural networks

An effective algorithm for extracting M-of-N rules from trained feedforward neural networks is proposed. First, we train a network where each input of the data can only have one of the two possible values, -1 or one. Next, we apply the hyperbolic tangent function to each connection from the input layer to the hidden layer of the network. By applying this squashing function, the activation values at the hidden units are effectively computed as the hyperbolic tangent (or the sigmoid) of the weighted inputs, where the weights have magnitudes that are equal one. By restricting the inputs and the weights to binary values either -1 or one, the extraction of M-of-N rules from the networks becomes trivial. We demonstrate the effectiveness of the proposed algorithm on several widely tested datasets. For datasets consisting of thousands of patterns with many attributes, the rules extracted by the algorithm are simple and accurate.     

Knowledge acquisition using a neural network for a weather forecasting knowledge-based system

Neural network technology is experiencing rapid growth and is receiving considerable attention from almost every field of science and engineering. The attraction is due to the successful application of neural network techniques to several real world problems. Neural networks have not yet found widespread application in weather forecasting. The reason for this has been the difficulty in obtaining suitable weather forecasting data sets. In this paper we describe our experience in applying neural network techniques for acquiring the necessary knowledge to predict the weather conditions of Melbourne City and its suburbs in Australia during a 24 hour period beginning at 9 am local time. The accuracy of forecasts produced by a given forecasting procedure typically varies with factors such as geographical location, season, categories of weather, quality of input data, lead time and validity time. Two types of weather data sets assembled from the archives of the Australian Commonwealth Bureau of Meteorology are used for training the neural network. The results of the experiments are competitive and are discussed.     

On the emergence of rules in neural networks

A simple associationist neural network learns to factor abstract rules (i.e., grammars) from sequences of arbitrary input symbols by inventing abstract representations that accommodate unseen symbol sets as well as unseen but similar grammars. The neural network is shown to have the ability to transfer grammatical knowledge to both new symbol vocabularies and new grammars. Analysis of the state-space shows that the network learns generalized abstract structures of the input and is not simply memorizing the input strings. These representations are context sensitive, hierarchical, and based on the state variable of the finite-state machines that the neural network has learned. Generalization to new symbol sets or grammars arises from the spatial nature of the internal representations used by the network, allowing new symbol sets to be encoded close to symbol sets that have already been learned in the hidden unit space of the network. The results are counter to the arguments that learning algorithms based on weight adaptation after each exemplar presentation (such as the long term potentiation found in the mammalian nervous system) cannot in principle extract symbolic knowledge from positive examples as prescribed by prevailing human linguistic theory and evolutionary psychology.     

Perception control with improved expectation learning through multilayered neural networks

In this paper, we investigate the viability of multilayered neural network (NN)-based extension of a conventional "perception" control concept. The perception process selects and completes the information from the system to be controlled before passing it to the controlling agent so that control is not lost when sensory information from the system is incomplete. The perception process produces an expectation of the next set of information to be received from the system. The expectation is used to replace missing parts of the information received and it also influences the next perception. In the existing work, each of the expectation elements is linearly acquired such that the expectation tells only the dominant information in the recent past, i.e., this approach has no capability to sense the trend and the dynamics in the information. This handicap could become a serious problem when the perception process is applied to real physical systems. Here, we introduce an extension of the perception control process by using a radial basis function (RBF) feedforward NN to learn the trend and the dynamics in the information and produce the expectation of the next observation. Through some simulation comparisons, we show that the proposed RBFNN-based method is better than the existing one.     

An improved synthesis method for multilayered neural networks usingqualitative knowledge

An improved synthesis method for the multilayered neural network (NN) as function approximator is proposed. The method offers a translation mechanism that maps the qualitative knowledge into a multilayered NN structure. Qualitative knowledge is expressed in the form of representative points, which can be linguistically described as, `when x is around x_i, then y_i is around y'. Synthesis equations for the translation mechanism are provided. After the direct synthesis of the initial NN, the NN is tuned by backpropagation (BP), using the training data. The direct synthesis decreases the burden on BP and contributes to improved learning efficiency, accuracy, and stability. It is demonstrated that the translation mechanism is also useful for incremental modeling, i.e., increasing the number of neurons, or representative points, based on the results of BP     

Knowledge discovery by inductive neural networks

A new neural network model for inducing symbolic knowledge from empirical data is presented. This model capitalizes on the fact that the certainty factor-based activation function can improve the network generalization performance from a limited amount of training data. The formal properties of the procedure for extracting symbolic knowledge from such a trained neural network are investigated. In the domain of molecular genetics, a case study demonstrated that the described learning system effectively discovered the prior domain knowledge with some degree of refinement. Also, in cross-validation experiments, the system outperformed C4.5, a commonly used rule learning system     

Fuzzy neural networks for learning fuzzy if-then rules

This study is dedicated to developing a fuzzy neural network with linguistic teaching signals. The proposed network, which can be applied either as a fuzzy expert system or a fuzzy controller, is able to process and learn the numerical information as well as the linguistic information. The network consists of two parts: (1) initial weights generation and (2) error back-propagation (EBP)-type learning algorithm. In the first part, a genetic algorithm (GA) generates the initial weights for a fuzzy neural network in order to prevent the network getting stuck to the local minimum. The second part employs the EBP-type learning algorithm for fine-tuning. In addition, the unimportant weights are eliminated during the training process. The simulated results do not only indicate that the proposed network can accurately learn the relations of fuzzy inputs and fuzzy outputs, but also show that the initial weights from the GA can coverage better and weight elimination really can reduce the training error. Moreover, real-world problem results show that the proposed network is able to learn the fuzzy IF-THEN rules captured from the retailing experts regarding the promotion effect on the sales.     

Extraction of rules from artificial neural networks for nonlinearregression

Neural networks (NNs) have been successfully applied to solve a variety of application problems including classification and function approximation. They are especially useful as function approximators because they do not require prior knowledge of the input data distribution and they have been shown to be universal approximators. In many applications, it is desirable to extract knowledge that can explain how Me problems are solved by the networks. Most existing approaches have focused on extracting symbolic rules for classification. Few methods have been devised to extract rules from trained NNs for regression. This article presents an approach for extracting rules from trained NNs for regression. Each rule in the extracted rule set corresponds to a subregion of the input space and a linear function involving the relevant input attributes of the data approximates the network output for all data samples in this subregion. Extensive experimental results on 32 benchmark data sets demonstrate the effectiveness of the proposed approach in generating accurate regression rules.     

Two regularizers for recursive least squared algorithms infeedforward multilayered neural networks

Recursive least squares (RLS)-based algorithms are a class of fast online training algorithms for feedforward multilayered neural networks (FMNNs). Though the standard RLS algorithm has an implicit weight decay term in its energy function, the weight decay effect decreases linearly as the number of learning epochs increases, thus rendering a diminishing weight decay effect as training progresses. In this paper, we derive two modified RLS algorithms to tackle this problem. In the first algorithm, namely, the true weight decay RLS (TWDRLS) algorithm, we consider a modified energy function whereby the weight decay effect remains constant, irrespective of the number of learning epochs. The second version, the input perturbation RLS (IPRLS) algorithm, is derived by requiring robustness in its prediction performance to input perturbations. Simulation results show that both algorithms improve the generalization capability of the trained network.     

Knowledge transfer in SVM and neural networks

The paper considers general machine learning models, where knowledge transfer is positioned as the main method to improve their convergence properties. Previous research was focused on mechanisms of knowledge transfer in the context of SVM framework; the paper shows that this mechanism is applicable to neural network framework as well. The paper describes several general approaches for knowledge transfer in both SVM and ANN frameworks and illustrates algorithmic implementations and performance of one of these approaches for several synthetic examples.     

Control rules of aeration in a submerged biofilm wastewater treatment process using fuzzy neural networks

The present work is part of a global development of reliable real-time control and supervision tools applied to wastewater pollution removal processes. In these processes, oxygen is a key substrate in animal cell metabolism and its consumption is thus a parameter of great interest for the monitoring. In this paper, an integrated neural-fuzzy process controller was developed to control aeration in an Aerated Submerged Biofilm Wastewater Treatment Process (ASBWTP). In order to improve the fuzzy neural network performance, the self-learning ability embedded in the fuzzy neural network model was emphasized for improving the rule extraction performance. The fuzzy neural network proves to be very effective in modeling the aeration performs better than artificial neural networks (ANN).For comparing between operation with and without the fuzzy neural controller, an aeration unit in an Aerated Submerged Biofilm Wastewater Treatment Process (ASBWTP) was picked up to support the derivation of a solid fuzzy control rule base. It is shown that, using the fuzzy neural controller, in terms of the cost effectiveness, it enables us to save almost 33% of the operation cost during the time period when the controller can be applied. Thus, the fuzzy neural network proved to be a robust and effective DO control tool, easy to integrate in a global monitoring system for cost managing.     

Job-shop scheduling using neural networks

Job-shop scheduling cannot easily be analytically accomplished, so, it is done by computer simulation using heuristic priority rules. The SLACK rule for calculating the margins of jobs to their due-dates is effective in meeting the due-dates. However, the calculated margins are not precise because the actual margin is shortened due to conflicts with other jobs. The authors propose a method for estimating the margins by using a neural network. It is found that the method is effective for improving the average lateness to due-dates but not the maximum lateness. This paper proposes a method for adding a second neural network for judging the reliability of the estimated margins composed to the first one and for switching to the margins calculated by the SLACK rule when the reliability is low. The proposed method is verified by scheduling simulations to be effective in decreasing the maximum lateness to due-dates as much as the average lateness.     

Aeromagnetic compensation using neural networks

Airborne magnetic surveys in geophysical exploration can be subject to interference effects from the aircraft. Principal sources are the permanent magnetism of various parts of the aircraft, induction effects created by the earth's magnetic field and eddy-current fields produced by the aircraft's manoeuvres. Neural networks can model these effects as functions of roll, pitch, heading and their time derivatives, together with vertical acceleration, charging currents to the generator, etc., without assuming an explicit physical model. Separation of interference effects from background regional and diurnal fields can also be achieved in a satisfactory way.     

Dynamic modelling using neural networks

A neural network structure is presented that uses feedback of unmeasured system states to represent dynamic systems more efficiently than conventional feedforward and recurrent networks, leading to better predictions, reduced training requirement and more reliable extrapolation. The structure identifies the actual system states based on imperfect knowledge of the initial state, which is available in many practical systems, and is therefore applicable only to such systems. It also enables a natural integration of any available partial state-space model directly into the prediction scheme, to achieve further performance improvement. Simulation examples of three varied dynamic systems illustrate the various options and advantages offered by the state-feedback neural structure. Although the advantages of the proposed structure, compared with the conventional feedforward and recurrent networks, should hold for most practical dynamic systems, artificial systems can readily be created and real systems can surely be found for which one or more of these advantages would vanish or even get reversed. Caution is therefore recommended against interpreting the suggested advantages as strict theorems valid in all situations.