Modeling layered artificial neural networks using a visual programming paradigm

The understanding of soft computing methodology often requires grasping abstract concepts or imagining complex interactions of large models over long computing cycles. However, this can be difficult for students with a weak background in mathematics, especially in the early stages of soft computing education. This article introduces the idea of applying a visual programming paradigm as a tool for an educational introduction to soft computing methods. IntelligentPad, proposed by Y. Tanaka, was used as the visual programming paradigm. IntelligentPad gives a visual appearance to objects or classes, and allows users to operate and link different objects together using a mouse. This article reports on using IntelligentPad to teach the basic mechanisms of artificial neural networks. The proposed method was applied to 3rd-year college students to verify its validity as a teaching method.     

A comparison of genetic programming and artificial neural networks in metamodeling of discrete-event simulation models

Genetic programming (GP) and artificial neural networks (ANNs) can be used in the development of surrogate models of complex systems. The purpose of this paper is to provide a comparative analysis of GP and ANNs for metamodeling of discrete-event simulation (DES) models. Three stochastic industrial systems are empirically studied: an automated material handling system (AMHS) in semiconductor manufacturing, an (s,S) inventory model and a serial production line. The results of the study show that GP provides greater accuracy in validation tests, demonstrating a better generalization capability than ANN. However, GP when compared to ANN requires more computation in metamodel development. Even given this increased computational requirement, the results presented indicate that GP is very competitive in metamodeling of DES models.     

VLSI technologies for artificial neural networks

VLSI systems, basic integrated circuits, and silicon technologies are discussed. Novel circuit and design principles that provide a foundation for the implementation of a wide variety of neural network models in silicon are described. The key issues for a successful integration of neural systems are identified. The realization of algorithms in silicon is examined. Special-purpose hardware for carrying out the activation and transfer function and for the connection elements is discussed. A brief overview of the current silicon technologies is provided     

Intelligent-memory architecture for artificial neural networks

Execution of artificial neural networks, especially for online pattern recognition, mainly depends on time-efficient execution of weighted sums. A new architecture achieves this goal, with a computation time superior to the time complexity of sequential von Neumann machines. This architecture uses additional logic to extend the functionality of conventional RAM. The authors discuss an implementation of this architecture that uses reconfigurable logic     

ULSI architectures for artificial neural networks

No clear consensus exists about how to exploit the potential for massively parallel implementations of artificial neural networks. Three hardware implementations to demonstrate the key issues surrounding their use are: model specific integrated circuits for neural associative memories, self-organizing feature maps, and local cluster neural networks     

A low-complexity fuzzy activation function for artificial neural networks

A novel fuzzy-based activation function for artificial neural networks is proposed. This approach provides easy hardware implementation and straightforward interpretability in the basis of IF-THEN rules. Backpropagation learning with the new activation function also has low computational complexity. Several application examples ( XOR gate, chaotic time-series prediction, channel equalization, and independent component analysis) support the potential of the proposed scheme.     

A new evolutionary system for evolving artificial neural networks

This paper presents a new evolutionary system, i.e., EPNet, for evolving artificial neural networks (ANNs). The evolutionary algorithm used in EPNet is based on Fogel's evolutionary programming (EP). Unlike most previous studies on evolving ANN's, this paper puts its emphasis on evolving ANN's behaviors. Five mutation operators proposed in EPNet reflect such an emphasis on evolving behaviors. Close behavioral links between parents and their offspring are maintained by various mutations, such as partial training and node splitting. EPNet evolves ANN's architectures and connection weights (including biases) simultaneously in order to reduce the noise in fitness evaluation. The parsimony of evolved ANN's is encouraged by preferring node/connection deletion to addition. EPNet has been tested on a number of benchmark problems in machine learning and ANNs, such as the parity problem, the medical diagnosis problems, the Australian credit card assessment problem, and the Mackey-Glass time series prediction problem. The experimental results show that EPNet can produce very compact ANNs with good generalization ability in comparison with other algorithms.     

A new winners-take-all architecture in artificial neural networks

MAXNET is a common competitive architecture to select the maximum or minimum from a set of data. However, there are two major problems with the MAXNET. The first problem is its slow convergence rate if all the data have nearly the same value. The second one is that it fails when either nonunique extreme values exist or each initial value is smaller than or equal to the sum of initial inhibitions from other nodes. In this paper, a novel neural network model called SELECTRON is proposed to select the maxima or minima from a set of data. This model is able to select all the maxima or minima via competition among the processing units even when MAXNET fails. We then prove that SELECTRON converges to the correct state in every situation. In addition, the convergence rates of SELECTRON for three special data distributions are derived. Finally, simulation results indicate that SELECTRON converges much faster than MAXNET.     

Knowledge-based artificial neural networks

Hybrid learning methods use theoretical knowledge of a domain and a set of classified examples to develop a method for accurately classifying examples not seen during training. The challenge of hybrid learning systems is to use the information provided by one source of information to offset information missing from the other source. By so doing, a hybrid learning system should learn more effectively than systems that use only one of the information sources. KBANN (Knowledge-Based Artificial Neural Networks) is a hybrid learning system built on top of connectionist learning techniques. It maps problem-specific “domain theories”, represented in propositional logic, into neural networks and then refines this reformulated knowledge using backpropagation. KBANN is evaluated by extensive empirical tests on two problems from molecular biology. Among other results, these tests show that the networks created by KBANN generalize better than a wide variety of learning systems, as well as several techniques proposed by biologists.     

Rule-based artificial neural networks

Neural networks that are integrated with rule-based systems having a knowledge base offer more capabilities than networks not integrated with such systems.     

A new approach to artificial neural networks

A novel approach to artificial neural networks is presented. The philosophy of this approach is based on two aspects: the design of task-specific networks, and a new neuron model with multiple synapses. The synapses' connective strengths are modified through selective and cumulative processes conducted by axo-axonic connections from a feedforward circuit. This new concept was applied to the position control of a planar two-link manipulator exhibiting excellent results on learning capability and generalization when compared with a conventional feedforward network. In the present paper, the example shows only a network developed from a neuronal reflexive circuit with some useful artifices, nevertheless without the intention of covering all possibilities devised.     

A review of evolutionary artificial neural networks

Research on potential interactions between connectionist learning systems, i.e., artificial neural networks (ANNs), and evolutionary search procedures, like genetic algorithms (GAs), has attracted a lot of attention recently. Evolutionary ANNs (EANNs) can be considered as the combination of ANNs and evolutionary search procedures. This article first distinguishes among three kinds of evolution in EANNs, i.e., the evolution of connection weights, of architectures, and of learning rules. Then it reviews each kind of evolution in detail and analyzes critical issues related to different evolutions. the review shows that although a lot of work has been done on the evolution of connection weights and architectures, few attempts have been made to understand the evolution of learning rules. Interactions among different evolutions are seldom mentioned in current research. However, the evolution of learning rules and its interactions with other kinds of evolution, play a vital role in EANNs. Finally, this article briefly describes a general framework for EANNs, which not only includes the aforementioned three kinds of evolution, but also considers interactions among them. © 1993 John Wiley & Sons, Inc.     

A microkernel architecture for constraint programming

This paper presents a microkernel architecture for constraint programming organized around a small number of core functionalities and minimal interfaces. The architecture contrasts with the monolithic nature of many implementations. With this design, variables, domains and constraints all remain external to the microkernel which isolates the propagation logic and event protocols from the modeling constructions. The Objective-CP search blends the control primitives of the host language with search combinators in a completely transparent and fully compositional way, delivering a natural search procedure in which one can use native constructions and tools such as debuggers. Empirical results indicate that the software engineering benefits are not incompatible with runtime efficiency.     

Topology constraint free fuzzy gated neural networks for patternrecognition

A novel topology constraint free neural network architecture using a generalized fuzzy gated neuron model is presented for a pattern recognition task. The main feature is that the network does not require weight adaptation at its input and the weights are initialized directly from the training pattern set. The elimination of the need for iterative weight adaptation schemes facilitates quick network set up times which make the fuzzy gated neural networks very attractive. The performance of the proposed network is found to be functionally equivalent to spatio-temporal feature maps under a mild technical condition. The classification performance of the fuzzy gated neural network is demonstrated on a 12-class synthetic three dimensional (3-D) object data set, real-world eight-class texture data set, and real-world 12 class 3-D object data set. The performance results are compared with the classification accuracies obtained from a spatio-temporal feature map, an adaptive subspace self-organizing map, multilayer feedforward neural networks, radial basis function neural networks, and linear discriminant analysis. Despite the network's ability to accurately classify seen data and adequately generalize validation data, its performance is found to be sensitive to noise perturbations due to fine fragmentation of the feature space. This paper also provides partial solutions to the above robustness issue by proposing certain improvements to various modules of the proposed fuzzy gated neural network.     

A review of adaptive online learning for artificial neural networks

In real applications learning algorithms have to address several issues such as, huge amount of data, samples which arrive continuously and underlying data generation processes that evolve over time. Classical learning is not always appropriate to work in these environments since independent and indentically distributed data are assumed. Taking into account the requirements of the learning process, systems should be able to modify both their structures and their parameters. In this survey, our aim is to review the developed methodologies for adaptive learning with artificial neural networks, analyzing the strategies that have been traditionally applied over the years. We focus on sequential learning, the handling of the concept drift problem and the determination of the network structure. Despite the research in this field, there are currently no standard methods to deal with these environments and diverse issues remain an open problem.     

Evolving transfer functions for artificial neural networks

The paper describes a methodology for constructing transfer functions for the hidden layer of a back-propagation network, which is based on evolutionary programming. The method allows the construction of almost any mathematical form. It is tested using four benchmark classification problems from the well-known machine intelligence problems repository maintained by the University of California, Irvine. It was found that functions other than the commonly used sigmoidal function could perform well when used as hidden layer transfer functions. Three of the four problems showed improved test results when these evolved functions were used.     

Lagrange programming neural networks for time-of-arrival-based source localization

Finding the location of a mobile source from a number of separated sensors is an important problem in global positioning systems and wireless sensor networks. This problem can be achieved by making use of the time-of-arrival (TOA) measurements. However, solving this problem is not a trivial task because the TOA measurements have nonlinear relationships with the source location. This paper adopts an analog neural network technique, namely Lagrange programming neural network, to locate a mobile source. We also investigate the stability of the proposed neural model. Simulation results demonstrate that the mean-square error performance of our devised location estimator approaches the Cramér–Rao lower bound in the presence of uncorrelated Gaussian measurement noise.     

An educational tool for artificial neural networks

Artificial neural networks are some kind of data processing systems, which try to simulate features of the human brain and its learning process. So, they are widely used by researchers to solve different problems in optimization, classification, pattern recognition, associative memory and control. In this paper, an educational tool, which can be used to work on different kinds of neural network models and learn fundamentals of the artificial neural network, is described. At this point, the whole tool environment provides an advanced system to ensure mentioned functions. The developed system supports using MLP, LVQ and SOM models and related learning algorithms. It employs some visual, interactive tools, which enable users to compose their own neural networks and work on the developed networks easily. By using these tools, users can also understand and learn working mechanism of a typical artificial neural network, using features of different models and related learning algorithms.     

Stereopsis by constraint learning feed-forward neural networks

A neural network (NN) approach to the problem of steropsis is presented. The correspondence problem (finding the correct matches between pixels of the epipolar lines of the stereo pair from among all the possible matches) is posed as a noniterative many-to-one mapping. Two multilayer feedforward NNs are utilized to learn and code this nonlinear and complex mapping using the backpropagation learning rule and a training set. The first NN is a conventional fully connected net while the second is a sparsely connected NN with a fixed number of hidden layer nodes. All the applicable constraints are learned and internally coded by the NNs enabling them to be more flexible and more accurate than previous methods. The approach is successfully tested on several random-dot stereograms. It is shown that the nets can generalize their learned mappings to cases outside their training sets and to noisy images. Advantages over the Marr-Poggio algorithm are discussed, and it is shown that the NNs performances are superior.     

A hybrid recommender system using artificial neural networks

In the context of recommendation systems, metadata information from reviews written for businesses has rarely been considered in traditional systems developed using content-based and collaborative filtering approaches. Collaborative filtering and content-based filtering are popular memory-based methods for recommending new products to the users but suffer from some limitations and fail to provide effective recommendations in many situations. In this paper, we present a deep learning neural network framework that utilizes reviews in addition to content-based features to generate model based predictions for the business-user combinations. We show that a set of content and collaborative features allows for the development of a neural network model with the goal of minimizing logloss and rating misclassification error using stochastic gradient descent optimization algorithm. We empirically show that the hybrid approach is a very promising solution when compared to standalone memory-based collaborative filtering method.