30 years of adaptive neural networks: perceptron, Madaline, andbackpropagation

Fundamental developments in feedforward artificial neural networks from the past thirty years are reviewed. The history, origination, operating characteristics, and basic theory of several supervised neural-network training algorithms (including the perceptron rule, the least-mean-square algorithm, three Madaline rules, and the backpropagation technique) are described. The concept underlying these iterative adaptation algorithms is the minimal disturbance principle, which suggests that during training it is advisable to inject new information into a network in a manner that disturbs stored information to the smallest extent possible. The two principal kinds of online rules that have developed for altering the weights of a network are examined for both single-threshold elements and multielement networks. They are error-correction rules, which alter the weights of a network to correct error in the output response to the present input pattern, and gradient rules, which alter the weights of a network during each pattern presentation by gradient descent with the objective of reducing mean-square error (averaged over all training patterns)     

Signal cancellation phenomena in adaptive antennas: Causes and cures

Conventional adaptive beamformers utilizing some form of automatic minimization of mean square error exhibit signal cancellation phenomena when adapting rapidly. These effects result from adaptive interaction between signal and interference, when signal and interference are received simultaneously. Similar phenomena have been observed and analyzed in relatively simple adaptive noise cancelling systems. A study of these phenomena in the simpler systems is used to provide insight into similar behavior in adaptive antennas. A method for alleviating signal cancellation has been devised by Duvall, whereby the signal components are removed from the adaptive process, then reinserted to form the final system output. Widrow has devised a different solution to the problem: to move the receiving array spatially (or electronically) to modulate emanations received off the look direction, without distorting useful signals incident from the look direction. This approach is called "spatial dither" and introduces the additional possibility of modulating "smart" jammer signals, thereby limiting their effectiveness.     

The “rubber-mask” technique-II. Pattern storage and recognition

This paper briefly summarizes much of the work in pattern recognition to date, and relates the rubber mask technique to previous work. A scheme for incorporating flexible-mask methods into a proposed pattern recognition and memory system is presented. A discussion based on some facts and on some conjecture of the human eye/brain system and how it recognizes patterns, possibly by flexible matching, is also presented.     

Neural dynamic optimization for control systems.III. Applications

For pt.II. see ibid., p. 490-501. The paper presents neural dynamic optimization (NDO) as a method of optimal feedback control for nonlinear multi-input-multi-output (MIMO) systems. The main feature of NDO is that it enables neural networks to approximate the optimal feedback solution whose existence dynamic programming (DP) justifies, thereby reducing the complexities of computation and storage problems of the classical methods such as DP. This paper demonstrates NDO with several applications including control of autonomous vehicles and of a robot-arm, while the two other companion papers of this topic describes the background for the development of NDO and present the theory of the method, respectively.     

Neural dynamic optimization for control systems. I. Background

The paper presents neural dynamic optimization (NDO) as a method of optimal feedback control for nonlinear multi-input-multi-output (MIMO) systems. The main feature of NDO is that it enables neural networks to approximate the optimal feedback solution whose existence dynamic programming (DP) justifies, thereby reducing the complexities of computation and storage problems of the classical methods such as DP. This paper mainly describes the background and motivations for the development of NDO, while the two other subsequent papers of this topic present the theory of NDO and demonstrate the method with several applications including control of autonomous vehicles and of a robot arm, respectively.     

Sensitivity of feedforward neural networks to weight errors

An analysis is made of the sensitivity of feedforward layered networks of Adaline elements (threshold logic units) to weight errors. An approximation is derived which expresses the probability of error for an output neuron of a large network (a network with many neurons per layer) as a function of the percentage change in the weights. As would be expected, the probability of error increases with the number of layers in the network and with the percentage change in the weights. The probability of error is essentially independent of the number of weights per neuron and of the number of neurons per layer, as long as these numbers are large (on the order of 100 or more).     

On the Nature and Elimination of Stimulus Artifact in Nerve Signals Evoked and Recorded Using Surface Electrodes

The electrical stimulus pulse and the surface electrodes commonly used to study compound action potentials of peripheral nerves give rise to an artifact consisting of an initial spike and a longer lasting tail which often interferes with the recorded signal. The artifact has four sources: 1) the voltage gradient between the recording electordes caused by stimulus current flowing through the limb, 2) the common-mode voltage of the limb caused by current escaping through the ground electrode, 3) the capacitive coupling between the stimulating and recording leads, and 4) the high-pass filtering characteristics of the recording amplifier. This paper models these sources and presents several methodological rules for minimizing their effects. Also presented are three computer-based methods for subtracting the residual artifact from contaminated records using estimates of the artifact obtained from: 1) subthreshold stimulation, 2) a second recording site remote from the nerve, or 3) stimulation during the refractory period of the nerve.