Deviation-tolerant floating gate structures as a way to design an on-chip learning neural networks

 Hardware implementation of artificial neural networks (ANN) based on MOS transistors with floating gate (Neuron MOS or νMOS) is discussed. Choosing analog approach as a weight storage rather than digital improves learning accuracy, minimizes chip area and power dissipation. However, since weight value can be represented by any voltage in the range of supplied voltage (e.g. from 0 to 3.3 V), minimum difference of two values is very small, especially in the case of using neuron with large sum of weights. This implies that ANN using analog hardware approach is weak against V _dd deviation. The purpose of this paper is to investigate main parts of analog ANN circuits (synapse and neuron) that can compensate all kinds of deviation and to develop their design methodologies.     

Adding learning to cellular genetic algorithms for trainingrecurrent neural networks

This paper proposes a hybrid optimization algorithm which combines the efforts of local search (individual learning) and cellular genetic algorithms (GA) for training recurrent neural nets (RNN). Each RNN weight is encoded as a floating point number, and a concatenation of numbers forms a chromosome. Reproduction takes place locally in a square grid, each grid point representing a chromosome. Lamarckian and Baldwinian (1896) mechanisms for combining cellular GA and learning are compared. Different hill-climbing algorithms are incorporated into the cellular GA. These include the real-time recurrent learning (RTRL) and its simplified versions, and the delta rule. RTRL has been successively simplified by freezing some of the weights to form simplified versions. The delta rule, the simplest form of learning, has been implemented by considering the RNN as feedforward networks. The hybrid algorithms are used to train the RNN to solve a long-term dependency problem. The results show that Baldwinian learning is inefficient in assisting the cellular GA. It is conjectured that the more difficult it is for genetic operations to produce the genotypic changes that match the phenotypic changes due to learning, the poorer is the convergence of Baldwinian learning. Most of the combinations using the Lamarckian mechanism show an improvement in reducing the number of generations for an optimum network; however, only a few can reduce the actual time taken. Embedding the delta rule in the cellular GA is the fastest method. Learning should not be too extensive.     

The Widrow-Hoff algorithm for McCulloch-Pitts type neurons

We analyze the convergence properties of the Widrow-Hoff delta rule applied to McCulloch-Pitts type neurons. We give sufficiency conditions under which the learning parameters converge and conditions under which the learning parameters diverge. In particular, we analyze how the learning rate affects the convergence of the learning parameters.     

Analog weight adaptation hardware

The demands on offsets in analog weight adaptation circuitry are very high for onchip learning feed-forward neural networks using a back-propagation type of learning rule. Exceeding of the specifications for weight adaptation offsets prevents the weights from converging to their optimum, which leads to a significantly degraded learning behavior. This letter presents a circuit, including a tuning system, that minimizes weight adaptation offsets and that can be used to implement analog on-chip back-propagation learning feed-forward neural networks.     

Comparison of floating gate neural network memory cells in standardVLSI CMOS technology

Several floating gate MOSFET structures, for potential use as analog memory elements in neural networks, have been fabricated in a standard 2 mum double-polysilicon CMOS process. Their physical and programming characteristics are compared with each other and with similar structures reported in the literature. None of the circuits under consideration require special fabrication techniques. The criteria used to determine the structure most suitable for neural network memory applications include the symmetry of charging and discharging characteristics, programming voltage magnitudes, the area required, and the effectiveness of geometric field enhancement techniques. This work provides a layout for an analog neural network memory based on previously unexplored criteria and results. The authors have found that the best designs (a) use the poly1 to poly2 oxide for injection; (b) need not utilize ;field enhancement' techniques; (c) use poly1 to diffusion oxide for a coupling capacitor; and (d) size capacitor ratios to provide a wide range of possible programming voltages.     

NEnv: Neural Environment Maps for Global Illumination

Environment maps are commonly used to represent and compute far-field illumination in virtual scenes. However, they are expensive to evaluate and sample from, limiting their applicability to real-time rendering. Previous methods have focused on compression through spherical-domain approximations, or on learning priors for natural, day-light illumination. These hinder both accuracy and generality, and do not provide the probability information required for importance-sampling Monte Carlo integration. We propose NEnv, a deep-learning fully-differentiable method, capable of compressing and learning to sample from a single environment map. NEnv is composed of two different neural networks: A normalizing flow, able to map samples from uniform distributions to the probability density of the illumination, also providing their corresponding probabilities; and an implicit neural representation which compresses the environment map into an efficient differentiable function. The computation time of environment samples with NEnv is two orders of magnitude less than with traditional methods. NEnv makes no assumptions regarding the content (i.e. natural illumination), thus achieving higher generality than previous learning-based approaches. We share our implementation and a diverse dataset of trained neural environment maps, which can be easily integrated into existing rendering engines.     

Adaptive weighted outer-product learning associative memory

Associative-memory neural networks with adaptive weighted outer-product learning are proposed in this paper. For the correct recall of a fundamental memory (FM), a corresponding learning weight is attached and a parameter called signal-to-noise-ratio-gain (SNRG) is devised. The sufficient conditions for the learning weights and the SNRG's are derived. It is found both empirically and theoretically that the SNRG's have their own threshold values for correct recalls of the corresponding FM's. Based on the gradient-descent approach, several algorithms are constructed to adaptively find the optimal learning weights with reference to global- or local-error measure.     

RSPOP: rough set-based pseudo outer-product fuzzy rule identification algorithm

System modeling with neuro-fuzzy systems involves two contradictory requirements: interpretability verses accuracy. The pseudo outer-product (POP) rule identification algorithm used in the family of pseudo outer-product-based fuzzy neural networks (POPFNN) suffered from an exponential increase in the number of identified fuzzy rules and computational complexity arising from high-dimensional data. This decreases the interpretability of the POPFNN in linguistic fuzzy modeling. This article proposes a novel rough set-based pseudo outer-product (RSPOP) algorithm that integrates the sound concept of knowledge reduction from rough set theory with the POP algorithm. The proposed algorithm not only performs feature selection through the reduction of attributes but also extends the reduction to rules without redundant attributes. As many possible reducts exist in a given rule set, an objective measure is developed for POPFNN to correctly identify the reducts that improve the inferred consequence. Experimental results are presented using published data sets and real-world application involving highway traffic flow prediction to evaluate the effectiveness of using the proposed algorithm to identify fuzzy rules in the POPFNN using compositional rule of inference and singleton fuzzifier (POPFNN-CRI(S)) architecture. Results showed that the proposed rough set-based pseudo outer-product algorithm reduces computational complexity, improves the interpretability of neuro-fuzzy systems by identifying significantly fewer fuzzy rules, and improves the accuracy of the POPFNN.     

Top-gate ZnO thin-film transistors with a polymer dielectric designed for ultraviolet optical gating

We report on the fabrication of ultraviolet (UV)-sensing top-gate ZnO thin-film transistors (TFTs) with a poly-4-vinylphenol (PVP) polymer gate dielectric on glass substrate. Our top-gate ZnO-TFT showed a field-effect mobility of 0.05 cm²/V s, maximum saturation current of 0.11 μA at a gate bias of 10 V and an on/off ratio of 10³ in the dark. Under UV illumination with a wavelength of 364 nm the ZnO-TFT exhibited 4.7 μA for a drain current (at the same gate bias of 10 V), which is 50 times higher than without UV. Such photo-transistor action appeared more pronounced under a depletion regime of 0 V gate bias and the photo-to-dark current ratio was more than about 10⁴. By adopting this high UV-sensitivity, our inverter device with the top-gate ZnO-TFT and a load resistance well demonstrated its optical gating behavior.     

FPGA implementation of a pulse density neural network with learning ability using simultaneous perturbation

Hardware realization is very important when considering wider applications of neural networks (NNs). In particular, hardware NNs with a learning ability are intriguing. In these networks, the learning scheme is of much interest, with the backpropagation method being widely used. A gradient type of learning rule is not easy to realize in an electronic system, since calculation of the gradients for all weights in the network is very difficult. More suitable is the simultaneous perturbation method, since the learning rule requires only forward operations of the network to modify weights unlike the backpropagation method. In addition, pulse density NN systems have some promising properties, as they are robust to noisy situations and can handle analog quantities based on the digital circuits. We describe a field-programmable gate array realization of a pulse density NN using the simultaneous perturbation method as the learning scheme. We confirm the viability of the design and the operation of the actual NN system through some examples.     

Weight perturbation: an optimal architecture and learning techniquefor analog VLSI feedforward and recurrent multilayer networks

Previous work on analog VLSI implementation of multilayer perceptrons with on-chip learning has mainly targeted the implementation of algorithms such as back-propagation. Although back-propagation is efficient, its implementation in analog VLSI requires excessive computational hardware. It is shown that using gradient descent with direct approximation of the gradient instead of back-propagation is more economical for parallel analog implementations. It is shown that this technique (which is called ;weight perturbation') is suitable for multilayer recurrent networks as well. A discrete level analog implementation showing the training of an XOR network as an example is presented.     

Design and VLSI implementation of a high-performance face detection engine

This paper proposes a novel hardware structure and field-programmable gate array (FPGA) implementation method for real-time detection of multiple human faces with robustness against illumination variations. These are designed to greatly improve face detection in various environments with using MCT techniques and the AdaBoost learning algorithm which is robust against variable illumination. We have designed, implemented, and verified the hardware architecture of the face detection engine for high-performance face detection and real-time processing. The face detection chip is developed by verifying and implementing it using a FPGA and an application-specific integrated circuit (ASIC). To verify and implement the chip, we used a Virtex5 LX330 FPGA board and a 0.18 μm 1-poly and 6-metal CMOS logic process. Performance results of the implementation and verification showed it is possible to detect at least 32 faces of a wide variety of sizes at a maximum speed of 147 frames per second.     

An iterative learning control theory for a class of nonlinear dynamic systems

An iterative learning control scheme is presented for a class of nonlinear dynamic systems which includes holonomic systems as its subset. The control scheme is composed of two types of control methodology: a linear feedback mechanism and a feedforward learning strategy. At each iteration, the linear feedback provides stability of the system and keeps its state errors within uniform bounds. The iterative learning rule, on the other hand, tracks the entire span of a reference input over a sequence of iterations. The proposed learning control scheme takes into account the dominant system dynamics in its update algorithm in the form of scaled feedback errors. In contrast to many other learning control techniques, the proposed learning algorithm neither uses derivative terms of feedback errors nor assumes external input perturbations as a prerequisite. The convergence proof of the proposed learning scheme is given under minor conditions on the system parameters.     

基于FPGA技术的浮点运算器的设计与实现

日趋进步和完善的FPGA(现场可编程门阵列)技术推动了当前数字电路的设计。浮点运算器是计算机的一个组成部件,结构比较复杂,利用FPGA技术设计浮点运算器可以缩短产品的开发周期。介绍了基于FPGA技术的浮点运算器的设计与实现。描述了采用VHDL(VHSIC硬件描述语言)和原理图方式设计完成浮点运算器的方法和步骤。利用FPGA技术,能方便灵活地设计出浮点运算器。     

基于最小二乘支持向量机的一类非自治系统自适应控制

提出了一种基于最小二乘支持向量机(LS-SVM)的一类不确定非自治系统自适应控制器设计方法.该方法基于最小二乘支持向量机来估计对象的部分未知非线性项,并给出了最小二乘支持向量机权向量和偏移值的在线学习规则.利用李亚普诺夫理论严格证明了整个闭环系统的跟踪误差、控制器参数以及最小二乘支持向量机权参数和偏移值的一致最终有界.此控制方法可以保证对象在线稳定地跟踪任何光滑的目标轨迹,仿真结果表明了此控制方法的可行性和有效性.     

Identification of nonlinear dynamical systems using multilayered neural networks

A novel multilayer discrete-time neural net paradigm is presented for the identification of multi-input multi-output (MIMO) nonlinear dynamical systems. The major novelty of this approach is a rigorous proof of identification error convergence that reveals a requirement for a new identifier structure and nonstandard weight tuning algorithms. The NN identifier includes modified delta rule weight tuning and exhibits a learning-while-functioning feature instead of learning-then-functioning, so that the identification is on-line with no explicit off-line learning phase needed. The structure of the neural net (NN) identifier is derived using a passivity aproach. Linearity in the parameters is not required and certainty equivalence is not used. The notion of persistency of excitation (PE) and passivity properties of the multilayer NN are defined and used in the convergence analysis of both the identification error and the weight estimates.     

Back-propagation learning and nonidealities in analog neuralnetwork hardware

Experimental results from adaptive learning using an optically controlled neural network are presented. The authors have used example problems in nonlinear system identification and signal prediction, two areas of potential neural network application, to study the capabilities of analog neural hardware. These experiments investigated the effects of a variety of nonidealities typical of analog hardware systems. They show that network using large arrays of nonuniform components can perform analog communications with a much higher degree of accuracy than might be expected given the degree of variation in the network's elements. The effects of other common nonidealities, such as noise, weight quantization, and dynamic range limitations, were also investigated.     

Obtaining Fault Tolerant Multilayer Perceptrons Using an Explicit Regularization

When the learning algorithm is applied to a MLP structure, different solutions for the weight values can be obtained if the parameters of the applied rule or the initial conditions are changed. Those solutions can present similar performance with respect to learning, but they differ in other aspects, in particular, fault tolerance against weight perturbations. In this paper, a backpropagation algorithm that maximizes fault tolerance is proposed. The algorithm presented explicitly adds a new term to the backpropagation learning rule related to the mean square error degradation in the presence of weight deviations in order to minimize this degradation. The results obtained demonstrate the efficiency of the learning rule proposed here in comparison with other algorithm.     

基于状态聚类的非参数化近似广义策略迭代增强学习算法

为解决当前近似策略迭代增强学习算法普遍存在计算量大、基函数不能完全自动构建的问题,提出一种基于状态聚类的非参数化近似广义策略迭代增强学习算法(NPAGPI-SC).该算法利用二级随机采样过程采集样本,利用trial-and-error过程和以样本完全覆盖为目标的估计方法计算逼近器初始参数,利用delta规则和最近邻思想在学习过程中自适应地调整逼近器,利用贪心策略选择应执行的动作.一级倒立摆平衡控制的仿真实验结果验证了所提出算法的有效性和鲁棒性.     

An analytical comparison of a neural network and a model-basedadaptive controller

A neural network inverse dynamics controller with adjustable weights is compared with a computed-torque type adaptive controller. Lyapunov stability techniques, usually applied to adaptive systems, are used to derive a globally asymptotically stable adaptation law for a single-layer neural network controller that bears similarities to the well-known delta rule for neural networks. This alternative learning rule allows the learning rates of each connection weight to be individually adjusted to give faster convergence. The role of persistently exciting inputs in ensuring parameter convergence, often mentioned in the context of adaptive systems, is emphasized in relation to the convergence of neural network weights. A coupled, compound pendulum system is used to develop inverse dynamics controllers based on adaptive and neural network techniques. Adaptation performance is compared for a model-based adaptive controller and a simple neural network utilizing both delta-rule learning and the alternative adaptation law.