VLSI architectures for neural networks

An introduction to neural networks and neural information processing is provided. Neurocomputers are discussed, focusing on how their design exploits the architectural properties of VLSI circuits. General-purpose and special-purpose neurocomputer developments throughout the world are examined. As illustration, and to put European developments in perspective, some of the important projects in the United States and Japan are described. European research is then discussed in greater detail     

VLSI technologies for artificial neural networks

VLSI systems, basic integrated circuits, and silicon technologies are discussed. Novel circuit and design principles that provide a foundation for the implementation of a wide variety of neural network models in silicon are described. The key issues for a successful integration of neural systems are identified. The realization of algorithms in silicon is examined. Special-purpose hardware for carrying out the activation and transfer function and for the connection elements is discussed. A brief overview of the current silicon technologies is provided     

Pulse stream VLSI neural networks

EPSILON, a large, working, VLSI device, demonstrates pulse stream methods in the wider context of analog neural networks. EPSILON uses dynamic weight storage techniques, but a nonvolatile alternative is desirable. To that end, we have developed an amorphous silicon memory, which we present in experiments incorporating the device in a modest pulse stream neural chip. We have also developed a target-based training algorithm, which we demonstrate in a prototype learning device using a realistic problem. Finally, we explore system-level problems in experiments with a second version of EPSILON in a small, autonomous robot     

Pulse arithmetic in VLSI neural networks

This review explores the use of biological signaling methods to build silicon networks. Recent designs using a technique called `pulse stream', which employs fully analog, dynamic weight storage, are described. The pulse-stream concept is explained, and a comparison is made with conventional analog neural networks. An analog synapse based on the pulse-stream approach is presented. Chip details and simulation results are given     

VLSI implementation of pulsating neural networks

We describe a generic approach for realizing networks of pulsating neurons based on charge pumping of interface states situated in the channel of MOS transistors. Two basic building blocks will be described: the pulse activated charge pumping (PSCP) synapse, and the charge sensitive oscillator (CSO). The PSCP synapse which operates as either a short or a long term memory device which produces a charge packet proportional to the number of pulses applied to its input, will be described in detail together with experimental results demonstrating its capability. The CSO circuit which is a charge controlled oscillator will be described together with simulations of its output frequency dependence on its input voltage, and the relation between the temporal dependence of output waveform on its input charge.     

Analog VLSI systems for image acquisition and fast early vision processing

This article describes a project to design and build prototype analog early vision systems that are remarkably low-power, small, and fast. Three chips are described in detail. A continuous-time CMOS imager and processor chip uses a fully parallel 2-D resistive grid to find an object's position and orientation at 5000 frames/second, using only 30 milliwatts of power. A CMOS/CCD imager and processor chip does high-speed image smoothing and segmentation in a clocked, fully parallel 2-D array. And a chip that merges imperfect depth and slope data to produce an accurate depth map is under development in switched-capacitor CMOS technology.     

Blind signal processing by complex domain adaptive spline neural networks

In this paper, neural networks based on an adaptive nonlinear function suitable for both blind complex time domain signal separation and blind frequency domain signal deconvolution, are presented. This activation function, whose shape is modified during learning, is based on a couple of spline functions, one for the real and one for the imaginary part of the input. The shape control points are adaptively changed using gradient-based techniques. B-splines are used, because they allow to impose only simple constraints on the control parameters in order to ensure a monotonously increasing characteristic. This new adaptive function is then applied to the outputs of a one-layer neural network in order to separate complex signals from mixtures by maximizing the entropy of the function outputs. We derive a simple form of the adaptation algorithm and present some experimental results that demonstrate the effectiveness of the proposed method.     

Analog implementation of pulse-coupled neural networks

This paper presents a compact architecture for analog CMOS hardware implementation of voltage-mode pulse-coupled neural networks (PCNN). The hardware implementation methods shows inherent fault tolerance specialties and high speed, which is usually more than an order of magnitude over the software counterpart. A computational style described in this article mimics a biological neural network using pulse-stream signaling and analog summation and multiplication, pulse-stream encoding technique uses pulse streams to carry information and control analog circuitry, while storing further analog information on the time axis. The main feature of the proposed neuron circuit is that the structure is compact, yet exhibiting all the basic properties of natural biological neurons. Functional and structural forms of neural and synaptic functions are presented along with simulation results. Finally, the proposed design is applied to image processing to demonstrate successful restoration of images and their features.     

A VLSI neural processor for image data compression usingself-organization networks

An adaptive electronic neural network processor has been developed for high-speed image compression based on a frequency-sensitive self-organization algorithm. The performance of this self-organization network and that of a conventional algorithm for vector quantization are compared. The proposed method is quite efficient and can achieve near-optimal results. The neural network processor includes a pipelined codebook generator and a paralleled vector quantizer, which obtains a time complexity O(1) for each quantization vector. A mixed-signal design technique with analog circuitry to perform neural computation and digital circuitry to process multiple-bit address information are used. A prototype chip for a 25-D adaptive vector quantizer of 64 code words was designed, fabricated, and tested. It occupies a silicon area of 4.6 mmx6.8 mm in a 2.0 mum scalable CMOS technology and provides a computing capability as high as 3.2 billion connections/s. The experimental results for the chip and the winner-take-all circuit test structure are presented.     

Pulse-stream VLSI neural networks mixing analog and digitaltechniques

The pulse-stream technique, which represents neural states as sequences of pulses, is reviewed. Several general issues are raised, and generic methods appraised, for pulsed encoding, arithmetic, and intercommunication schemes. Two contrasting synapse designs are presented and compared. The first is based on a fully analog computational form in which the only digital component is the signaling mechanism itself-asynchronous, pulse-rate encoded digital voltage pulses. In this circuit, multiplication occurs in the voltage/current domain. The second design uses more conventional digital memory for weight storage, with synapse circuits based on pulse stretching. Integrated circuits implementing up to 15000 analog, fully programmable synaptic connections are described. A demonstrator project is described in which a small robot localization network is implemented using asynchronous, analog, pulse-stream devices.     

Image processing by neural networks

Neural networks and related techniques are powerful tools that prove their efficiency in real-world applications, where problems are badly defined or difficult to formalize. Some applications, especially those involving images, require a huge number of operations and an enormous reduction of the dataflow from input to output data. For sorting objects by vision or image analysis, for example, inputs are images at video rate; outputs are correcting values, names of objects, or locations of specific objects in a picture. For several years, our group has been involved in image-processing applications, mainly with connectionist algorithms such as neural networks, that enable efficient use of parallel hardware. We designed and tested our first-generation processor, L-Neuro 1.0, on shape recognition with neural networks and image compression applications. However, unlike L-Neuro 1.0 and chips such as Intel's NI 1000 and IBMs ZISC, our newest processor, L-Neuro 2.3, is a multiDSP. It exhibits full general-purpose vector processing capabilities, like Siemens' MA16, Adaptive Solutions' CNAPS, Mitsubishi's Neuro 4, and AT&T's HIP. This fully programmable vector processor uses an array of 12 digital signal processors to perform up to 2 billion arithmetic operations per second and achieve a peak transfer throughout of 2 Gbytes/s for image-processing applications     

Analog VLSI system for active drag reduction

Drawing inspiration from the structure of shark skin, the authors are building a system to reduce drag along a surface. The entire question of active control of shark skin is speculative. Biologists hypothesize that sharks actively move their denticles. Indirect evidence of this is twofold. The denticles connect to muscles underneath the shark's skin. The total number of mechanoreceptive pressure sensors (pit organs) and their placement on a shark's body positively correlate with the speed of the species. For good active control, the shark may need many sensors to relay the current condition over its body. Although questions remain about sharks using active control, we concluded from this biological example that it may be beneficial to use controlled microscopic structures to reduce drag     

Analog memristive memory with applications in audio signal processing

Since the development of the HP memristor, much attention has been paid to studies of memris- tive devices and applications, particularly memristor-based nonvolatile semiconductor memory. Owing to its unique properties, theoretically, one could restart a memristor-based computer immediately without the need for reloading the data. Further, current memories are mainly binary and can store only ones and zeros, whereas memristors have multilevel states, which means a single memristor unit can replace many binary transistors and realize higher-density memory. It is believed that memristors can also implement analog storage besides binary and multilevel information memory. In this paper, an implementation scheme for analog memristive memory is considered. A charge-controlled memristor model is derived and the corresponding SPICE model is constructed. Special write and read operations are demonstrated through numerical analysis and circuit simulations. In addition, an audio analog record/play system using a memristor crossbar array is designed. This system can provide great storage capacity （long recording time） and high audio quality with a simple small circuit structure. A series of computer simulations and analyses verify the effectiveness of the proposed scheme.     

Subband neural networks prediction for on-line audio signal recovery

In this paper, a subbands multirate architecture is presented for audio signal recovery. Audio signal recovery is a common problem in digital music signal restoration field, because of corrupted samples that must be replaced. The subband approach allows for the reconstruction of a long audio data sequence from forward-backward predicted samples. In order to improve prediction performances, neural networks with spline flexible activation function are used as narrow subband nonlinear forward-backward predictors. Previous neural-networks approaches involved a long training process. Due to the small networks needed for each subband and to the spline adaptive activation functions that speed-up the convergence time and improve the generalization performances, the proposed signal recovery scheme works in online (or in continuous learning) mode as a simple nonlinear adaptive filter. Experimental results show the mean square reconstruction error and maximum error obtained with increasing gap length, from 200 to 5000 samples for different musical genres. A subjective performances analysis is also reported. The method gives good results for the reconstruction of over 100 ms of audio signal with low audible effects in overall quality and outperforms the previous approaches.     

Block-based neural networks for personalized ECG signal classification.

This paper presents evolvable block-based neural networks (BbNNs) for personalized ECG heartbeat pattern classification. A BbNN consists of a 2-D array of modular component NNs with flexible structures and internal configurations that can be implemented using reconfigurable digital hardware such as field-programmable gate arrays (FPGAs). Signal flow between the blocks determines the internal configuration of a block as well as the overall structure of the BbNN. Network structure and the weights are optimized using local gradient-based search and evolutionary operators with the rates changing adaptively according to their effectiveness in the previous evolution period. Such adaptive operator rate update scheme ensures higher fitness on average compared to predetermined fixed operator rates. The Hermite transform coefficients and the time interval between two neighboring R-peaks of ECG signals are used as inputs to the BbNN. A BbNN optimized with the proposed evolutionary algorithm (EA) makes a personalized heartbeat pattern classifier that copes with changing operating environments caused by individual difference and time-varying characteristics of ECG signals. Simulation results using the Massachusetts Institute of Technology/Beth Israel Hospital (MIT-BIH) arrhythmia database demonstrate high average detection accuracies of ventricular ectopic beats (98.1%) and supraventricular ectopic beats (96.6%) patterns for heartbeat monitoring, being a significant improvement over previously reported electrocardiogram (ECG) classification results.     

Maximum entropy signal reconstruction with neural networks

The implementation of the maximum entropy reconstruction algorithms by means of neural networks is discussed. It is shown that the solutions of the maximum entropy problem correspond to the steady states of the appropriate Hopfield net. The choice of network parameters is discussed, and existence of the maximum entropy solution is proved.     

Nonlinear channel equalization for QAM signal constellation usingartificial neural networks

Application of artificial neural networks (ANN's) to adaptive channel equalization in a digital communication system with 4-QAM signal constellation is reported in this paper. A novel computationally efficient single layer functional link ANN (FLANN) is proposed for this purpose. This network has a simple structure in which the nonlinearity is introduced by functional expansion of the input pattern by trigonometric polynomials. Because of input pattern enhancement, the FLANN is capable of forming arbitrarily nonlinear decision boundaries and can perform complex pattern classification tasks. Considering channel equalization as a nonlinear classification problem, the FLANN has been utilized for nonlinear channel equalization. The performance of the FLANN is compared with two other ANN structures [a multilayer perceptron (MLP) and a polynomial perceptron network (PPN)] along with a conventional linear LMS-based equalizer for different linear and nonlinear channel models. The effect of eigenvalue ratio (EVR) of input correlation matrix on the equalizer performance has been studied. The comparison of computational complexity involved for the three ANN structures is also provided.     

Wavelet based fault detection in analog VLSI circuits using neural networks

This paper deals with a new method of testing analog VLSI circuits, using wavelet transform for analog circuit response analysis and artificial neural networks (ANN) for fault detection. Pseudo-random patterns generated by Linear Feedback Shift Register (LFSR) are used as input test patterns. The wavelet coefficients obtained for the fault-free and faulty cases of the circuits under test (CUT) are used to train the neural network. Two different architectures, back propagation and probabilistic neural networks are trained with the test data. To minimize the neural network architecture, normalization and principal component analysis are done on the input data before it is applied to the neural network. The proposed method is validated with two IEEE benchmark circuits, namely, the operational amplifier and state variable filter.     

Analog VLSI model of binaural hearing

The stereausis model of biological auditory processing is proposed as a representation that encodes both binaural and spectral information in a unified framework. A working analog VLSI chip that implements this model of early auditory processing in the brain is described. The chip is a 100000-transistor integrated circuit that computes the stereausis representation in real time, using continuous-time analog processing. The chip receives two audio inputs, representing sound entering the two ears, computes the stereausis representation, and generates output signals that can directly drive a color CRT display. Outputs from the chips for a variety of artificial and speech stimuli are shown.     

Information processing in negative feedback neural networks

Information theory suggests that extraction of the principal sub-space from data is useful when the input to a neural network is corrupted with additive noise. A number of neural network algorithms exist which can find this principal sub-space, many of which also extract the principal components of the input. However, when there is noise on both input and output of a network, simply extracting the principal sub-space (or components) is not sufficient to optimize information capacity. An approximate solution to maximizing information capacity would be to extract the principal sub-space of components with variances above a certain threshold, and then ensure that these are uncorrelated and that they have equal variance at the output. A neural network is described which uses negative feedback connections to achieve this uncorrelated, equal-variance solution.