Nonlinear vector prediction using feed-forward neural networks

The performance of a classical linear vector predictor is limited by its ability to exploit only the linear correlation between the blocks. However, a nonlinear predictor exploits the higher order correlations among the neighboring blocks, and can predict edge blocks with increased accuracy. We have investigated several neural network architectures that can be used to implement a nonlinear vector predictor, including the multilayer perceptron (MLP), the functional link (FL) network, and the radial basis function (RBF) network. Our experimental results show that a neural network predictor can predict the blocks containing edges with a higher accuracy than a linear predictor.     

Nonlinear adaptive prediction of nonstationary signals

We describe a computationally efficient scheme for the nonlinear adaptive prediction of nonstationary signals whose generation is governed by a nonlinear dynamical mechanism. The complete predictor consists of two subsections. One performs a nonlinear mapping from the input space to an intermediate space with the aim of linearizing the input signal, and the other performs a linear mapping from the new space to the output space. The nonlinear subsection consists of a pipelined recurrent neural network (PRNN), and the linear section consists of a conventional tapped-delay-line (TDL) filter. The nonlinear adaptive predictor described is of general application. The dynamic behavior of the predictor is demonstrated for the case of a speech signal; for this application, it is shown that the nonlinear adaptive predictor outperforms the traditional linear adaptive scheme in a significant way     

State-dependent vector hybrid linear and nonlinear ARMA modeling: Theory

A new model is proposed to represent a general vector nonstationary and nonlinear process by setting up a state-dependent vector hybrid linear and nonlinear autoregressive moving average (SVH-ARMA) model. The linear part of the process is represented by a vector ARMA model, the nonlinear part is represented by a vector nonlinear ARMA model employing a multilayer feedforward neural network, and the nonstationary characteristics are captured with a hidden Markov chain. Based on a unifiedQ-likelihood function, an expectation-maximization algorithm for model identification is derived, and the model parameters are estimated by applying a state-dependent training and nonlinear optimization technique iteratively, which finally yields maximum likelihood estimation of model parameters. This model can track the nonstationary varying of a vector linear and/or nonlinear process adaptively and represent a vector linear and/or nonlinear system with low order. Moreover, it is able to characterize and track the long-range, second-order correlation features of many time series and thus can be used for reliable multiple step ahead prediction. Some impressive applications of the SVH-ARMA model are being presented in the companion paper by Zheng et al., pp. 575–597, this issue.     

A modular neural network vector predictor for predictive imagecoding

In this paper, we present a modular neural network vector predictor that improves the predictive component of a predictive vector quantization (PVQ) scheme. The proposed vector prediction technique consists of five dedicated predictors (experts), where each expert predictor is optimized for a particular class of input vectors. An input vector is classified into one of five classes, based on its directional variances. One expert predictor is optimized for stationary blocks, and each of the other four expert predictors are optimized to predict horizontal, vertical, 45 degrees , and 135 degrees diagonally oriented edge-blocks, respectively. An integrating unit is then used to select or combine the outputs of the experts in order to form the final output of the modular network. Therefore, no side information is transmitted to the receiver about the selected predictor or the integration of the predictors. Experimental results show that the proposed scheme gives an improvement of 1.7 dB over a single multilayer perceptron (MLP) predictor. Furthermore, if the information about the predictor selection is sent to the receiver, the improvement could be up to 3 dB over a single MLP predictor. The perceptual quality of the predicted images is also significantly improved.     

Nonlinear prediction in image coding with DPCM

In contrast to the traditional linear differential pulse code modulation (DPCM) design for the encoding of images, a new, nonlinear, neural network-based, DPCM technique has been devised. The predictor is designed by supervised training, based on a typical sequence of pixel values in an image. A function link neural network architecture has been used to design the predictor for one dimensional (1-D) DPCM. Computer simulation experiments in still image coding have shown that the resulting encoders work very well. At a transmission rate of 1 bit/pixel, for the image LENA, the 1-D neural network DPCM provides a 4.2 dB improvement in SNR over the standard linear DPCM system.<>     

一种多分辨率图像混合编码方案

本文提出一种基于小波变换与神经网络的多分辨率图像混合编码方案，利用小波分解对图像的多分辨率表示来消除图像空间域和频率域的相关性，由于小波图像相邻行之间的复杂关系难以用线性表示式来描述，使用多层神经网络（ＭＬＮＮ）来确定这种未知关系。实验证明，神经网络非线性预测器性能优于线性预测器，对非线性预测后的差值图像用自组织特征映射（ＳＯＦＭ）码书进行矢量量化（ＶＱ）编码，编码图像主观质量好，压缩比高，算法简     

Predictive residual vector quantization [image coding]

This paper presents a new vector quantization technique called predictive residual vector quantization (PRVQ). It combines the concepts of predictive vector quantization (PVQ) and residual vector quantization (RVQ) to implement a high performance VQ scheme with low search complexity. The proposed PRVQ consists of a vector predictor, designed by a multilayer perceptron, and an RVQ that is designed by a multilayer competitive neural network. A major task in our proposed PRVQ design is the joint optimization of the vector predictor and the RVQ codebooks. In order to achieve this, a new design based on the neural network learning algorithm is introduced. This technique is basically a nonlinear constrained optimization where each constituent component of the PRVQ scheme is optimized by minimizing an appropriate stage error function with a constraint on the overall error. This technique makes use of a Lagrangian formulation and iteratively solves a Lagrangian error function to obtain a locally optimal solution. This approach is then compared to a jointly designed and a closed-loop design approach. In the jointly designed approach, the predictor and quantizers are jointly optimized by minimizing only the overall error. In the closed-loop design, however, a predictor is first implemented; then the stage quantizers are optimized for this predictor in a stage-by-stage fashion. Simulation results show that the proposed PRVQ scheme outperforms the equivalent RVQ (operating at the same bit rate) and the unconstrained VQ by 2 and 1.7 dB, respectively. Furthermore, the proposed PRVQ outperforms the PVQ in the rate-distortion sense with significantly lower codebook search complexity.     

Nonlinear adaptive prediction of speech with a pipelined recurrentneural network

New learning algorithms for an adaptive nonlinear forward predictor that is based on a pipelined recurrent neural network (PRNN) are presented. A computationally efficient gradient descent (GD) learning algorithm, together with a novel extended recursive least squares (ERLS) learning algorithm, are proposed. Simulation studies based on three speech signals that have been made public and are available on the World Wide Web (WWW) are used to test the nonlinear predictor. The gradient descent algorithm is shown to yield poor performance in terms of prediction error gain, whereas consistently improved results are achieved with the ERLS algorithm. The merit of the nonlinear predictor structure is confirmed by yielding approximately 2 dB higher prediction gain than a linear structure predictor that employs the conventional recursive least squares (RLS) algorithm     

基于神经网络的功率放大器预失真建模

提出一种神经网络结合分离信号对功率放大器预失真建模的方法。将输入/输出信号的线性与非线性部分分开处理,利用神经网络良好的逼近能力,采用LM算法,拟合出功率放大器特性曲线,进而建立预失真模型,使非线性功率放大器的输入/输出曲线整体呈线性化。在保证输出幅度限制和输出功率最大化的前提下,与未作信号分离的神经网络建模方法、多项式建模方法以及Saleh函数模型方法相比较,发现信号分离神经网络建模方法能得到较小的归一化均方误差和误差矢量幅度。仿真结果表明,采用信号分离神经网络对功率放大器及其预失真建模,整体线性化误差较小、精度高、效果更佳。     

Parallel,self-organizing,hierarchical neural networks with forward-backward training

A forward-backward training algorithm for parallel, self-organizing hierarchical neural networks (PSHNNs) is described. Using linear algebra, it is shown that the forward-backward training of ann-stage PSHNN until convergence is equivalent to the pseudo-inverse solution for a single, total network designed in the least-squares sense with the total input vector consisting of the actual input vector and its additional nonlinear transformations. These results are also valid when a single long input vector is partitioned into smaller length vectors. A number of advantages achieved are: small modules for easy and fast learning, parallel implementation of small modules during testing, faster convergence rate, better numerical error-reduction, and suitability for learning input nonlinear transformations by other neural networks. The backpropagation (BP) algorithm is proposed for learning input nonlinearitics. Better performance in terms of deeper minimum of the error function and faster convergence rate is achieved when a single BP network is replaced by a PSHNN of equal complexity in which each stage is a BP network of smaller complexity than the single BP network.     

Neural net nonlinear prediction for speech data

A new, nonlinear, neural network based predictor has been devised fro the encoding of speech data. It may be used in the design of a differential pulse code modulation (DPCM) coder for speech. A hybrid neural network architecture has been employed which combines the perceptron and backpropagation paradigms, thus called the PB-hybrid (PBH). Only two neurons are needed in the backpropagation section, keeping the required overhead modest. This predictor is designed by supervised training, based on a typical sequence of digitised values of samples in a speech frame. Simulation experiments have been carried out using 15 ms frames of 16 kHz speech data. The results obtained for the prediction gain show a 3 dB advantage of the PBH network over the linear predictor.<>     

Extraction of independent signal sources using a deflationaryexploratory projection pursuit network with lateral inhibition

A nonlinear self-organising neural network is proposed, which employs hierarchic linear negative feedback, and this network is applied to the blind separation of independent source signals from their mixtures. Blind separation of sources has become an important area of research, with significant contributions being made from both the statistical signal processing and artificial neural network research communities. A nonlinear extension of a negative feedback network is developed and it is shown that hierarchic linear feedback provides a deflation of the network residuals, which are employed in the Hebbian learning of the network. As each of the output neuron weights converge to a separating vector, then the weighted feedback will remove the contribution of the extracted source from the remaining residual mixture. It is shown that the data driven self-organisation of the proposed network using only Hebbian and anti-Hebbian learning will extract the underlying signals from the received mixture. The results of a simulation are reported, which demonstrates the ability of the network in restoring images after degradation with noise and interfering images     

神经网络动态逆在歼击机安全着陆中的控制

给出了基于神经网络动态逆的自适应跟踪控制方法,用以解决飞机着陆过程中的复杂非线性和出现舵机故障的情况.应用神经网络直接对非线性系统故障模型求逆,使得所设计的逆系统能够包含故障信息,克服了传统的控制设计中将过程模型线性化,从而将不可忽视的非线性关系用线性关系代替或忽略的弊端.对由于建模误差、不确定性因素等引起的非线性系统逆误差,通过自组织模糊小脑模型关节控制器(SOFCMAC)神经网络在线进行修正.并在此基础上对3个通道分别设计了参考模型和线性控制器,以实现对伪线性系统进行跟踪控制.通过将这种方法用于某型歼击机在着陆过程中发生平尾卡死故障控制的过程仿真,验证了该方法的可行性.     

Linear and nonlinear ARMA model parameter estimation using anartificial neural network

Addresses parametric system identification of linear and nonlinear dynamic systems by analysis of the input and output signals. Specifically, the authors investigate the relationship between estimation of the system using a feedforward neural network model and estimation of the system by use of linear and nonlinear autoregressive moving-average (ARMA) models. By utilizing a neural network model incorporating a polynomial activation function, the authors show the equivalence of the artificial neural network to the linear and nonlinear ARMA models. They compare the parameterization of the estimated system using the neural network and ARMA approaches by utilizing data generated by means of computer simulations. Specifically, the authors show that the parameters of a simulated ARMA system can be obtained from the neural network analysis of the simulated data or by conventional least squares ARMA analysis. The feasibility of applying neural networks with polynomial activation functions to the analysis of experimental data is explored by application to measurements of heart rate (HR) and instantaneous lung volume (ILV) fluctuations     

Statistical analysis of neural network modeling and identificationof nonlinear systems with memory

The paper presents a statistical analysis of neural network modeling and identification of nonlinear systems with memory. The nonlinear system model is comprised of a discrete-time linear filter H followed by a zero-memory nonlinear function g(.). The system is corrupted by input and output independent Gaussian noise. The neural network is used to identify and model the unknown linear filter H and the unknown nonlinearity g(.). The network architecture is composed of a linear adaptive filter and a two-layer nonlinear neural network (with an arbitrary number of neurons). The network is trained using the backpropagation algorithm. The paper studies the MSE surface and the stationary points of the adaptive system. Recursions are derived for the mean transient behavior of the adaptive filter coefficients and neural network weights for slow learning. It is shown that the Wiener solution for the adaptive filter is a scaled version of the unknown filter H. Computer simulations show good agreement between theory and Monte Carlo estimations     

基于子块集成神经网络法的PSD背景光补偿

提出一种基于神经网络高精度线性化子块网络集成插值实现光电位置敏感器件（PSD）背景光非线性补偿方法。利用神经网络具有逼近任意非线性函数的特点,通过训练,使神经网络建立在不同背景光下PSD输出与其标准值之间的非线性映射关系,实现PSD全程跟踪补偿。实验结果表明,该方法能有效地消除背景光的影响,在神经网络的输出端得到期望的线性输出。     

解无约束极大极小问题的非对称神经网络算法

本文构造了一种新的非对称神经网络模型，用于求解极大极小无约束优化问题，并证明了非对称线性神经网络和非线性神经网络是整体Ｌｙａｐｕｎｏｖ稳定的，且收敛于对应的Ｌａｇｒａｎｇｅ方程的稳定点，计算机模拟的结果表明此方法是可行的，且具有良好的整体收敛性。     

Support vector machine techniques for nonlinear equalization

The emerging machine learning technique called support vector machines is proposed as a method for performing nonlinear equalization in communication systems. The support vector machine has the advantage that a smaller number of parameters for the model can be identified in a manner that does not require the extent of prior information or heuristic assumptions that some previous techniques require. Furthermore, the optimization method of a support vector machine is quadratic programming, which is a well-studied and understood mathematical programming technique. Support vector machine simulations are carried out on nonlinear problems previously studied by other researchers using neural networks. This allows initial comparison against other techniques to determine the feasibility of using the proposed method for nonlinear detection. Results show that support vector machines perform as well as neural networks on the nonlinear problems investigated. A method is then proposed to introduce decision feedback processing to support vector machines to address the fact that intersymbol interference (ISI) data generates input vectors having temporal correlation, whereas a standard support vector machine assumes independent input vectors. Presenting the problem from the viewpoint of the pattern space illustrates the utility of a bank of support vector machines. This approach yields a nonlinear processing method that is somewhat different than the nonlinear decision feedback method whereby the linear feedback filter of the decision feedback equalizer is replaced by a Volterra filter. A simulation using a linear system shows that the proposed method performs equally to a conventional decision feedback equalizer for this problem     

A Neutral-Type Delayed Projection Neural Network for Solving Nonlinear Variational Inequalities

A neutral-type delayed projection neural network is proposed to deal with nonlinear variational inequalities. Compared with the existing delayed neural networks for linear variational inequalities, the proposed approach apparently has the larger application domain. By the theory of functional differential equation, a delay-dependent sufficient stability condition is derived. This stability condition is easily checked, and can guarantee that the proposed neural network is convergent to the solution of nonlinear variational inequality problem exponentially, which improves the existing stability criteria for the neutral-type delayed neural network. Moreover, many related problems, such as the projection equation and optimization problems, can also be dealt with by the proposed method. Finally, simulation examples are given to illustrate the satisfactory performance of the proposed method.      

Functional link neural network cascaded with Chebyshev orthogonal polynomial for nonlinear channel equalization

Nonlinear intersymbol interference (ISI) leads to significant error rate in nonlinear communication and digital storage channel. In this paper, therefore, a novel computationally efficient functional link neural network cascaded with Chebyshev orthogonal polynomial is proposed to combat nonlinear ISI. The equalizer has a simple structure in which the nonlinearity is introduced by functional expansion of the input pattern by trigonometric polynomial and Chebyshev orthogonal polynomial. Due to the input pattern and nonlinear approximation enhancement, the proposed structure can approximate arbitrarily nonlinear decision boundaries. It has been utilized for nonlinear channel equalization. The performance of the proposed adaptive nonlinear equalizer is compared with functional link neural network (FLNN) equalizer, multilayer perceptron (MLP) network and radial basis function (RBF) along with conventional normalized least-mean-square algorithms (NLMS) for different linear and nonlinear channel models. The comparison of convergence rate, bit error rate (BER) and steady state error performance, and computational complexity involved for neural network equalizers is provided.