Stochastic analysis of the LMS algorithm with a saturationnonlinearity following the adaptive filter output

This paper presents a statistical analysis of the least mean square (LMS) algorithm with a zero-memory scaled error function nonlinearity following the adaptive filter output. This structure models saturation effects in active noise and active vibration control systems when the acoustic transducers are driven by large amplitude signals. The problem is first defined as a nonlinear signal estimation problem and the mean-square error (MSE) performance surface is studied. Analytical expressions are obtained for the optimum weight vector and the minimum achievable MSE as functions of the saturation. These results are useful for adaptive algorithm design and evaluation. The LMS algorithm behavior with saturation is analyzed for Gaussian inputs and slow adaptation. Deterministic nonlinear recursions are obtained for the time-varying mean weight and MSE behavior. Simplified results are derived for white inputs and small step sizes. Monte Carlo simulations display excellent agreement with the theoretical predictions, even for relatively large step sizes. The new analytical results accurately predict the effect of saturation on the LMS adaptive filter behavior     

Statistical analysis of neural network modeling and identificationof nonlinear systems with memory

The paper presents a statistical analysis of neural network modeling and identification of nonlinear systems with memory. The nonlinear system model is comprised of a discrete-time linear filter H followed by a zero-memory nonlinear function g(.). The system is corrupted by input and output independent Gaussian noise. The neural network is used to identify and model the unknown linear filter H and the unknown nonlinearity g(.). The network architecture is composed of a linear adaptive filter and a two-layer nonlinear neural network (with an arbitrary number of neurons). The network is trained using the backpropagation algorithm. The paper studies the MSE surface and the stationary points of the adaptive system. Recursions are derived for the mean transient behavior of the adaptive filter coefficients and neural network weights for slow learning. It is shown that the Wiener solution for the adaptive filter is a scaled version of the unknown filter H. Computer simulations show good agreement between theory and Monte Carlo estimations     

Analysis of stochastic gradient tracking of time-varying polynomialWiener systems

This paper presents analytical and Monte Carlo results for a stochastic gradient adaptive scheme that tracks a time-varying polynomial Wiener (1958) system [i.e., a linear time-invariant (LTI) filter with memory followed by a time-varying memoryless polynomial nonlinearity]. The adaptive scheme consists of two phases: (1) estimation of the LTI memory using the LMS algorithm and (2) tracking the time-varying polynomial-type nonlinearity using a second coupled gradient search for the polynomial coefficients. The time-varying polynomial nonlinearity causes a time-varying scaling for the optimum Wiener filter for Phase 1. These time variations are removed for Phase 2 using a novel coupling scheme to Phase 1. The analysis for Gaussian data includes recursions for the mean behavior of the LMS algorithm for estimating and tracking the optimum Wiener filter for Phase 1 for several different time-varying polynomial nonlinearities and recursions for the mean behavior of the stochastic gradient algorithm for Phase 2. The polynomial coefficients are shown to be accurately tracked. Monte Carlo simulations confirm the theoretical predictions and support the underlying statistical assumptions     

Stochastic analysis of gradient adaptive identification ofnonlinear systems with memory for Gaussian data and noisy input andoutput measurements

This paper investigates the statistical behavior of two gradient search adaptive algorithms for identifying an unknown nonlinear system comprised of a discrete-time linear system H followed by a zero-memory nonlinearity g(·). The input and output of the unknown system are corrupted by additive independent noises. Gaussian models are used for all inputs. Two competing adaptation schemes are analyzed. The first is a sequential adaptation scheme where the LMS algorithm is first used to estimate the linear portion of the unknown system. The LMS algorithm is able to identify the linear portion of the unknown system to within a scale factor. The weights are then frozen at the end of the first adaptation phase. Recursions are derived for the mean and fluctuation behavior of the LMS algorithm, which are in excellent agreement with Monte Carlo simulations. When the nonlinearity is modeled by a scaled error function, the second part of the sequential gradient identification scheme is shown to correctly learn the scale factor and the error function scale factor. Mean recursions for the scale factors show good agreement with Monte Carlo simulations. For slow learning, the stationary points of the gradient algorithm closely agree with the stationary points of the theoretical recursions. The second adaptive scheme simultaneously learns both the linear and nonlinear portions of the unknown channel. The mean recursions for the linear and nonlinear portions show good agreement with Monte Carlo simulations for slow learning. The stationary points of the gradient algorithm also agree with the stationary points of the theoretical recursions     

Analysis of stochastic gradient identification ofWiener-Hammerstein systems for nonlinearities with Hermite polynomialexpansions

This paper investigates the statistical behavior of a sequential adaptive gradient search algorithm for identifying an unknown Wiener-Hammerstein (1958) system (WHS) with Gaussian inputs. The WHS nonlinearity is assumed to be expandable in a series of orthogonal Hermite polynomials. The sequential procedure uses (1) a gradient search for the unknown coefficients of the Hermite polynomials, (2) an LMS adaptive filter to partially identify the input and output linear filters of the WHS, and (3) the higher order terms in the Hermite expansion to identify each of the linear filters. The third step requires the iterative solution of a set of coupled nonlinear equations in the linear filter coefficients. An alternative scheme is presented if the two filters are known a priori to be exponentially shaped. The mean behavior of the various gradient recursions are analyzed using small step-size approximations (slow learning) and yield very good agreement with Monte Carlo simulations. Several examples demonstrate that the scheme provides good estimates of the WHS parameters for the cases studied     

Robust least mean square adaptive FIR filter algorithm

The authors propose a new robust adaptive FIR filter algorithm for system identification applications based on a statistical approach named the M estimation. The proposed robust least mean square algorithm differs from the conventional one by the insertion of a suitably chosen nonlinear transformation of the prediction residuals. The effect of nonlinearity is to assign less weight to a small portion of large residuals so that the impulsive noise in the desired filter response will not greatly influence the final parameter estimates. The convergence of the parameter estimates is established theoretically using the ordinary differential equation approach. The feasibility of the approach is demonstrated with simulations     

Stochastic gradient adaptation under general error criteria

Examines a family of adaptive filter algorithms of the form W_k+1=W_k+μf(d_k-W_k^t X_k)X_k in which f(·) is a memoryless odd-symmetric nonlinearity acting upon the error. Such algorithms are a generalization of the least-mean-square (LMS) adaptive filtering algorithm for even-symmetric error criteria. For this algorithm family, the authors derive general expressions for the mean and mean-square convergence of the filter coefficients For both arbitrary stochastic input data and Gaussian input data. They then provide methods for optimizing the nonlinearity to minimize the algorithm misadjustment for a given convergence rate. Using the calculus of variations, it is shown that the optimum nonlinearity to minimize misadjustment near convergence under slow adaptation conditions is independent of the statistics of the input data and can be expressed as -p'(x)/p(x), where p(x) is the probability density function of the uncorrelated plant noise. For faster adaptation under the white Gaussian input and noise assumptions, the nonlinearity is shown to be x/{1+μλx²/σ_k ²}, where λ is the input signal power and σ_k² is the conditional error power. Thus, the optimum stochastic gradient error criterion for Gaussian noise is not mean-square. It is shown that the equations governing the convergence of the nonlinear algorithm are exactly those which describe the behavior of the optimum scalar data nonlinear adaptive algorithm for white Gaussian input. Simulations verify the results for a host of noise interferences and indicate the improvement using non-mean-square error criteria     

Modeling of time-interleaved ADCs with nonlinear hybrid filter banks

In this paper, we model time-interleaved analog-to-digital converters (TIADCs) with nonlinear hybrid filter banks (NHFBs), which greatly unifies and simplifies the analysis of TIADCs. The input/output relation of such a nonlinear hybrid filter bank can be used to describe combined offset, gain, aperture delay, input behavior, and nonlinearity mismatches and is therefore an extendable starting point for profound analyses of TIADC behaviors. We show the connection of offset and gain mismatches to nonlinearity mismatches and reveal the two error sources of timing mismatches.     

Nonlinear loudspeaker compensation for hands free acoustic echocancellation

Loudspeaker nonlinearity at high volumes limits the achievable echo cancellation performance in linear acoustic echo cancellers. A new nonlinear adaptive filter for improving the echo cancellation performance at high volumes for hands free telephones is proposed. Experimental measurements show that an echo cancellation improvement of >8 dB can be obtained at high volumes as compared to a linear adaptive filter     

Neural network modeling and identification of nonlinear channelswith memory: algorithms, applications, and analytic models

This paper proposes a neural network (NN) approach for modeling nonlinear channels with memory. Two main examples are given: (1) modeling digital satellite channels and (2) modeling solid-state power amplifiers (SSPAs). NN models provide good generalization performance (in terms of output signal-to-error ratio). NN modeling of digital satellite channels allows the characterization of each channel component. Neural net models represent the SSPA as a system composed of a linear complex filter followed by a nonlinear memoryless neural net followed by a linear complex filter. If the new algorithms are to be used in real systems, it is important that the algorithm designer understands their learning behavior and performance capabilities. Some simplified neural net models are analyzed in support of the simulation results. The analysis provides some theoretical basis for the usefulness of NNs for modeling satellite channels and amplifiers. The analysis of the simplified adaptive models explains the simulation results qualitatively but not quantitatively. The analysis proceeds in several steps and involves several novel ideas to avoid solving the more difficult general nonlinear problem     

Analytical model for the first and second moments of an adaptiveinterpolated FIR filter using the constrained filtered-X LMS algorithm

The authors present an analytical model for the mean weight behaviour and weight covariance matrix of an adaptive interpolated FIR filter using the LMS algorithm to adapt the filter weights. The particular structure of this adaptive filter determines that special analytical considerations must be used. First, the introduction of an interpolating block cascaded with the adaptive sparse filter requires that the input signal correlations must be considered. It is well known that such correlations are disregarded by the independence theory, which is the basis for the analysis of the LMS algorithm adapting FIR structures. Secondly a constrained analysis is used to deal mathematically with the sparse nature of the adaptive section. Experimental results demonstrate the effectiveness of the proposed analytical models as compared with the results obtained by classical analysis     

用于低维混沌时间序列预测的一种非线性自适应预测滤波器

在二阶Volterra滤波器基础上,提出了一种用于低维混沌时间自适应预测的非线性自适应预测器。基于最小均方误差准则导出了一种NLMS类型的自适应算法来实时调整这种非线性滤波预测器的系数,仿真实验结果表明：这种线性化的非线性自适应滤波预测器能够有效地预测低维混时间序列,且它的模块化特征更易于VLSI电路实现,具有广泛的工程应用价值。     

Convergence analysis of adaptive filtering algorithms with singulardata covariance matrix

The paper provides a rigorous analysis of the behavior of adaptive filtering algorithms when the covariance matrix of the filter input is singular. The analysis is done in the context of adaptive plant identification. The considered algorithms are LMS, RLS, sign (SA), and signed regressor (SRA) algorithms. Both the signal and weight behavior of the algorithms are considered. The signal behavior is evaluated in terms of the moments of the excess output error of the filter. The weight behavior is evaluated in terms of the moments of the filter weight misalignment vector. It is found that the RLS and SRA diverge when the input covariance matrix is singular. The steady-state signal behavior of the LMS and SA can be made arbitrarily fine by using sufficiently small step sizes of the algorithms. Indeed, the long-term average of the mean square excess error of the LMS is proportional to the algorithm step size. The long-term average of the mean absolute excess error of the SA is proportional to the square root of the algorithm step size. On the other hand, the steady-state weight behavior of both the LMS and SA have biases that depend on the weight initialization. The analytical results of the paper are supported by simulations     

New nonlinear algorithms for narrowband interference suppression inCDMA spread-spectrum systems

The narrow-band interference suppression capability of spread-spectrum systems can be further enhanced by employing interference suppression filters. This paper proposes a number of new nonlinear algorithms for narrow-band interference suppression in code division multiple access spread-spectrum systems. Our research consists of two parts. (1) We propose a multiuser decision-directed Kalman (MDK) filter, which has the same performance as the nonlinear approximate conditional mean (ACM) filter but a much simpler algorithm. (2) We use the nonlinear function in the ACM and the MDK filters to develop nonlinear adaptive least mean square filters with significantly improved performance. Simulation results indicate that our nonlinear algorithms outperform conventional linear ones     

Adaptive-FRESH Filters for Compensation of Cycle-Frequency Errors

This paper deals with the problem of uncertainties in the periodicities of linear almost-periodically time-variant (LAPTV) filters. These filters are usually implemented as a set of branches, each consisting of a frequency shifter followed by a linear time-invariant (LTI) filter. This implementation is also known as FRESH filters. This paper is motivated by the fact that, when there exist errors in the frequency shifts, the optimum set of LTI filters is obtained by canceling the outputs of the corresponding branches. The purpose of this paper is to analyze the nonstationary behavior of adaptive filters in order to mitigate this problem. Our results show that an adaptive filter can offset the errors in the frequency shifts. The reason is that the coefficients of the adaptive filter are updated so that the filter actually performs as a linear periodically time-variant filter for each branch. This allows to track the errors in the frequency shifts when the rate of convergence of the adaptive algorithm is suitably selected. An analytical study of the convergence is presented, which allows to compute the optimal rate of convergence and the mean squared-error attained by the adaptive filter.     

Nonlinear techniques for interference suppression inspread-spectrum systems

The binary nature of direct-sequence signals is exploited to obtain nonlinear filters that outperform the linear filters hitherto used for this purpose. The case of a Gaussian interferer with known autoregressive parameters is considered. Using simulations, it is shown that an approximate conditional mean (ACM) filter of the Masreliez type performs significantly better than the optimum linear (Kalman-Bucy) filter. For the case of interferers with unknown parameters, the nature of the nonlinearity in the ACM filter is used to obtain an adaptive filtering algorithm that is identical to the linear transversal filter except that the previous prediction errors are transformed nonlinearly before being incorporated into the linear prediction. Two versions of this filter are considered: one in which the filter coefficients are updated using the Widrow LMS algorithm, and another in which the coefficients are updated using an approximate gradient algorithm. Simulations indicate that the nonlinear filter with LMS updates performs substantially better than the linear filter for both narrowband Gaussian and single-tone interferers, whereas the gradient algorithm gives slightly better performance for Gaussian interferers but is rather ineffective in suppressing a sinusoidal interferer     

A tree-structured piecewise linear adaptive filter

The authors propose and analyze a novel architecture for nonlinear adaptive filters. These nonlinear filters are piecewise linear filters obtained by arranging linear filters and thresholds in a tree structure. A training algorithm is used to adaptively update the filter coefficients and thresholds at the nodes of the tree, and to prune the tree. The resulting tree-structured piecewise linear adaptive filter inherits the robust estimation and fast adaptation of linear adaptive filters, along with the approximation and model-fitting properties of tree-structured regression models. A rigorous analysis of the training algorithm for the tree-structured filter is performed. Some techniques are developed for analyzing hierarchically organized stochastic gradient algorithms with fixed gains and nonstationary dependent data. Simulation results show the significant advantages of the tree-structured piecewise linear filter over linear and polynomial filters for adaptive echo cancellation     

A stable adaptive Hammerstein filter employing partial orthogonalization of the input signals

This paper presents an algorithm that adapts the parameters of a Hammerstein system model. Hammerstein systems are nonlinear systems that contain a static nonlinearity cascaded with a linear system. In this paper, the static nonlinearity is modeled using a polynomial system, and the linear filter that follows the nonlinearity is an infinite-impulse response (IIR) system. The adaptation of the nonlinear components is improved by orthogonalizing the inputs to the coefficients of the polynomial system. The step sizes associated with the recursive components are constrained in such a way as to guarantee bounded-input bounded-output (BIBO) stability of the overall system. This paper also presents experimental results that show that the algorithm performs well in a variety of operating environments, exhibiting stability and global convergence of the algorithm.     

Adaptive image restoration of sigma filter using local statisticsand human visual characteristics

The sigma filter is a nonlinear scheme for modifying an average (mean) filter to improve its edge-preserving characteristic. However, this filter is susceptible to impulsive noise, such as BSC (binary symmetric channel) noise. In this letter, a simple adaptive algorithm for a sigma filter is presented using local image statistics and human visual characteristics to compensate for its drawbacks. Experimental results for an image degraded by BSC noise show that the proposed algorithm has much better performance than nonadaptive ones both on SNR gain and on subjective image quality     

Stochastic analysis of an error power ratio scheme applied to the affine combination of two LMS adaptive filters

The affine combination of two adaptive filters that simultaneously adapt on the same inputs has been actively investigated. In these structures, the filter outputs are linearly combined to yield a performance that is better than that of either filter. Various decision rules can be used to determine the time-varying parameter for combining the filter outputs. A recently proposed scheme based on the ratio of error powers of the two filters has been shown by simulation to achieve nearly optimum performance. The purpose of this paper is to present a first analysis of the statistical behavior of this error power scheme for white Gaussian inputs. Expressions are derived for the mean behavior of the combination parameter and for the adaptive weight mean-square deviation. Monte Carlo simulations show good to excellent agreement with the theoretical predictions.