Stochastic gradient adaptation under general error criteria

Examines a family of adaptive filter algorithms of the form W_k+1=W_k+μf(d_k-W_k^t X_k)X_k in which f(·) is a memoryless odd-symmetric nonlinearity acting upon the error. Such algorithms are a generalization of the least-mean-square (LMS) adaptive filtering algorithm for even-symmetric error criteria. For this algorithm family, the authors derive general expressions for the mean and mean-square convergence of the filter coefficients For both arbitrary stochastic input data and Gaussian input data. They then provide methods for optimizing the nonlinearity to minimize the algorithm misadjustment for a given convergence rate. Using the calculus of variations, it is shown that the optimum nonlinearity to minimize misadjustment near convergence under slow adaptation conditions is independent of the statistics of the input data and can be expressed as -p'(x)/p(x), where p(x) is the probability density function of the uncorrelated plant noise. For faster adaptation under the white Gaussian input and noise assumptions, the nonlinearity is shown to be x/{1+μλx²/σ_k ²}, where λ is the input signal power and σ_k² is the conditional error power. Thus, the optimum stochastic gradient error criterion for Gaussian noise is not mean-square. It is shown that the equations governing the convergence of the nonlinear algorithm are exactly those which describe the behavior of the optimum scalar data nonlinear adaptive algorithm for white Gaussian input. Simulations verify the results for a host of noise interferences and indicate the improvement using non-mean-square error criteria     

On the convergence behavior of the LMS and the normalized LMSalgorithms

It is shown that the normalized least mean square (NLMS) algorithm is a potentially faster converging algorithm compared to the LMS algorithm where the design of the adaptive filter is based on the usually quite limited knowledge of its input signal statistics. A very simple model for the input signal vectors that greatly simplifies analysis of the convergence behavior of the LMS and NLMS algorithms is proposed. Using this model, answers can be obtained to questions for which no answers are currently available using other (perhaps more realistic) models. Examples are given to illustrate that even quantitatively, the answers obtained can be good approximations. It is emphasized that the convergence of the NLMS algorithm can be speeded up significantly by employing a time-varying step size. The optimal step-size sequence can be specified a priori for the case of a white input signal with arbitrary distribution     

Convergence and performance analysis of the normalized LMSalgorithm with uncorrelated Gaussian data

It is demonstrated that the normalized least mean square (NLMS) algorithm can be viewed as a modification of the widely used LMS algorithm. The NLMS is shown to have an important advantage over the LMS, which is that its convergence is independent of environmental changes. In addition, the authors present a comprehensive study of the first and second-order behavior in the NLMS algorithm. They show that the NLMS algorithm exhibits significant improvement over the LMS algorithm in convergence rate, while its steady-state performance is considerably worse     

Convergence behavior of affine projection algorithms

A class of equivalent algorithms that accelerate the convergence of the normalized LMS (NLMS) algorithm, especially for colored inputs, has previously been discovered independently. The affine projection algorithm (APA) is the earliest and most popular algorithm in this class that inherits its name. The usual APA algorithms update weight estimates on the basis of multiple, unit delayed, input signal vectors. We analyze the convergence behavior of the generalized APA class of algorithms (allowing for arbitrary delay between input vectors) using a simple model for the input signal vectors. Conditions for convergence of the APA class are derived. It is shown that the convergence rate is exponential and that it improves as the number of input signal vectors used for adaptation is increased. However, the rate of improvement in performance (time-to-steady-state) diminishes as the number of input signal vectors increases. For a given convergence rate, APA algorithms are shown to exhibit less misadjustment (steady-state error) than NLMS. Simulation results are provided to corroborate the analytical results     

A general class of nonlinear normalized adaptive filteringalgorithms

The normalized least mean square (NLMS) algorithm is an important variant of the classical LMS algorithm for adaptive linear filtering. It possesses many advantages over the LMS algorithm, including having a faster convergence and providing for an automatic time-varying choice of the LMS stepsize parameter that affects the stability, steady-state mean square error (MSE), and convergence speed of the algorithm. An auxiliary fixed step-size that is often introduced in the NLMS algorithm has the advantage that its stability region (step-size range for algorithm stability) is independent of the signal statistics. In this paper, we generalize the NLMS algorithm by deriving a class of nonlinear normalized LMS-type (NLMS-type) algorithms that are applicable to a wide variety of nonlinear filter structures. We obtain a general nonlinear NLMS-type algorithm by choosing an optimal time-varying step-size that minimizes the next-step MSE at each iteration of the general nonlinear LMS-type algorithm. As in the linear case, we introduce a dimensionless auxiliary step-size whose stability range is independent of the signal statistics. The stability region could therefore be determined empirically for any given nonlinear filter type. We present computer simulations of these algorithms for two specific nonlinear filter structures: Volterra filters and the previously proposed class of Myriad filters. These simulations indicate that the NLMS-type algorithms, in general, converge faster than their LMS-type counterparts     

Real-Time Implementation of Parallel Architecture Based Noise Minimization from Speech Signals on FPGA

In recent years, the real time hardware implementation of LMS based adaptive noise cancellation on FPGA is becoming popular. There are several works reported in this area in the literature. However, NLMS based implementation of adaptive noise cancellation on FPGA using Xilinx System Generator (XSG) is not reported. This paper explores the various forms of parallel architecture based on NLMS algorithm and its hardware implementation on FPGA using XSG for noise minimization from speech signals. In total, the direct form, binary tree direct form and transposed form of parallel architecture of normalized least mean square (NLMS), delayed normalized least mean square and retimed delayed normalized least mean square algorithms are implemented on FPGA using hardware co-simulation model. The performance parameters (SNR and MSE) of these algorithms are analyzed for the adaptive noise cancellation system and the comparison is made with parallel architectures of least mean square (LMS), delayed least mean square, and retimed delayed least mean square algorithms respectively. The hardware utilization of all the said algorithms are also analyzed and compared. The result shows that NLMS based implementations outperform than that of LMS for all forms of parallel architecture for the given parameters with negligence increase in device utility. The binary tree direct form of retimed delayed NLMS achieves the maximum SNR improvement (39.83 dB) in comparison to other NLMS variants at the cost of optimum resource utilization.     

The behavior of LMS and NLMS algorithms in the presence ofspherically invariant processes

The behavior of least-mean-square (LMS) and normalized least-mean-square (NLMS) algorithms with spherically invariant random processes (SIRPs) as excitations is shown. Many random processes fall into this category, and SIRPs closely resemble speech signals. The most pertinent properties of these random processes are summarized. The LMS algorithm is introduced, and the first- and second-order moments of the weight-error vector between the Wiener solution and the estimated solution are shown. The behavior of the NLMS algorithm is obtained, and the first- and second-order moments of the weight-error vector are calculated. The results are verified by comparison with known results when a white noise process and a colored Gaussian process are used as input sequences. Some simulation results for a K₀-process are then shown     

一种新的变步长NLMS自适应算法

文章根据最优步长理论提出一个向量因子来控制步长的变化，其长度及形式与权向量相同，并且由输入数据相关性大的部分决定两个向量中每次应更新的部分，从而使权向量异步更新达到最佳。计算机仿真结果表明，在输入数据相关性很强时，该算法与传统的变步长LMS及NLMS算法相比不仅具有更决的收敛速度，更小的稳态误差及更平稳的收敛过程，同时计算量也有一定的减少。     

Efficient subspace-based algorithm for adaptive bearing estimation and tracking

In some practical applications of array processing, the directions of the incident signals should be estimated adaptively, and/or the time-varying directions should be tracked promptly. In this paper, an adaptive bearing estimation and tracking (ABEST) algorithm is investigated for estimating and tracking the uncorrelated and correlated narrow-band signals impinging on a uniform linear array (ULA) based on the subspace-based method without eigendecomposition (SUMWE), where a linear operator is obtained from the array data to form a basis for the space by exploiting the array geometry and its shift invariance property. Specifically, the space is estimated using the least-mean-square (LMS) or normalized LMS (NLMS) algorithm, and the directions are updated using the approximate Newton method. The transient analyses of the LMS and NLMS algorithms are studied, where the "weight" (i.e., the linear operator) is in the form of a matrix and there is a correlation between the "additive noise" and "input data" that involve the instantaneous correlations of the received array data in the updating equation, and the step-size stability conditions are derived explicitly. In addition, the analytical expressions for the mean-square error (MSE) and mean-square deviation (MSD) learning curves of the LMS algorithm are clarified. The effectiveness of the ABEST algorithm is verified, and the theoretical analyses are corroborated through numerical examples. Simulation results show that the ABEST algorithm is computationally simple and has good adaptation and tracking abilities.     

Stochastic analysis of the LMS algorithm with a saturationnonlinearity following the adaptive filter output

This paper presents a statistical analysis of the least mean square (LMS) algorithm with a zero-memory scaled error function nonlinearity following the adaptive filter output. This structure models saturation effects in active noise and active vibration control systems when the acoustic transducers are driven by large amplitude signals. The problem is first defined as a nonlinear signal estimation problem and the mean-square error (MSE) performance surface is studied. Analytical expressions are obtained for the optimum weight vector and the minimum achievable MSE as functions of the saturation. These results are useful for adaptive algorithm design and evaluation. The LMS algorithm behavior with saturation is analyzed for Gaussian inputs and slow adaptation. Deterministic nonlinear recursions are obtained for the time-varying mean weight and MSE behavior. Simplified results are derived for white inputs and small step sizes. Monte Carlo simulations display excellent agreement with the theoretical predictions, even for relatively large step sizes. The new analytical results accurately predict the effect of saturation on the LMS adaptive filter behavior     

A Modular Analog NLMS Structure for Adaptive Filtering

This paper proposes a modular Analog Adaptive filter (AAF) algorithm, in which the coefficient adaptation is carried out by using a time varying step size analog normalized LMS (NLMS) algorithm, which is implemented as an external analog structure. The proposed time varying step size is estimated by using the first element of the crosscorrelation vector between the output error and reference signal, and the first element of the crosscorrelation vector between the output error and the adaptive filter output signal, respectively. Proposed algorithm reduces distortion when additive noise power increases or DC offsets are present, without significatively decreasing the convergence rate nor increasing the complexity of the conventional NLMS algorithms. Simulation results show that proposed algorithm improves the performance of AAF when DC offsets are present. The proposed VLSI structure for the time varying step size normalized NLMS algorithm has, potentially, a very small size and faster convergence rates than its digital counterparts. It is suitable for general purpose applications or oriented filtering solution such as echo cancellation and equalization in cellular telephony in which high performance, low power consumption, fast convergence rates and small size adaptive digital filters (ADF) are required. The convergence performance of analog adaptive filters using integrators like first order low pass filter is analyzed.     

一种约束稳定性最小均方噪声对消算法

研究了最小均方误差(LMS)算法、归一化的最小均方(NLMS)算法及变步长NLMS算法在自适应噪声干扰抵消器中的应用,针对目前这些算法在噪声对消器应用中的缺点,将约束稳定性最小均方(CS-LMS)算法应用到噪声处理中,并进一步结合变步长的思想提出来一种新的变步长CS-LMS算法。通过MATLAB进行仿真分析,结果证实提出的算法与其他算法相比,能很好地滤除掉噪声从而得到期望信号,明显的降低了稳态误差,并拥有好的收敛速度。     

基于DFP的二阶Volterra模型及其对Rössler混沌序列的预测

为克服最小二乘法或归一化最小二乘法在二阶Volterra建模时参数选择不当引起的问题,在最小二乘法基础上,应用一种基于后验误差假设的可变收敛因子技术,构建了一种基于Davidon-Fletcher-Powell算法的二阶Volterra模型（DFPSOVF）.给出参数估计中自相关逆矩阵估计的递归更新公式,并对其正定性、有界性和τ（n）的作用进行了研究.将DFPSOVF模型应用于Rössler混沌序列的单步预测,仿真结果表明其能够保证算法的稳定性和收敛性,不存在最小二乘法和归一化最小二乘法的发散问题.     

Use of adaptive RLS, LMS, and NLMS algorithms for nonlinearity modeling in a modified laser interferometer

Laser heterodyne interferometer is one kind of nano-metrology systems which has been widely used in industry for high-accuracy displacement measurements. The accuracy of the nano-metrology systems based on the laser heterodyne interferometers can be effectively limited by the periodic nonlinearity. In this paper, we present the nonlinearity modeling of the nano-metrology interferometric system using some adaptive filters. The adaptive algorithms consist of the least mean squares (LMS), normalized least mean squares (NLMS), and recursive least squares (RLS). It is shown that the RLS algorithm can obtain optimal modeling parameters of nonlinearity.     

Analysis of LMS and NLMS algorithms with delayed coefficient updateunder the presence of spherically invariant processes

Certain conditions require a delay in the coefficient update of the least mean square (LMS) and normalized least mean square (NLMS) algorithms. This paper presents an in-depth analysis of these modificated versions for the important case of spherically invariant random processes (SIRPs), which are known as an excellent model for speech signals. Some derived bounds and the predicted dynamic behavior of the algorithms are found to correspond very well to simulation results and a real time implementation on a fixed-point signal processor. A modification of the algorithm is proposed to assure the well known properties of the LMS and NLMS algorithms     

Convergence of the delayed normalized LMS algorithm with decreasingstep size

In several practical applications of the LMS algorithm, including certain VLSI implementations, the coefficient adaptation can be performed only after some fixed delay. The resulting algorithm is known as the delayed LMS (DLMS) algorithm in the literature. Previous published analyses of this algorithm are based on mean and moment convergence under the independence assumption between successive input vectors. These analyses are interesting and give valuable insights into the convergence properties but, from a practical viewpoint, they do not guarantee the correct performance of the particular realization with which the user must live. We consider a normalized version of this algorithm with a decreasing step size μ(n) and prove the almost sure convergence of the nonhomogeneous algorithm, assuming a mixing input condition and the satisfaction of a certain law of large numbers     

Stochastic analysis of gradient adaptive identification ofnonlinear systems with memory for Gaussian data and noisy input andoutput measurements

This paper investigates the statistical behavior of two gradient search adaptive algorithms for identifying an unknown nonlinear system comprised of a discrete-time linear system H followed by a zero-memory nonlinearity g(·). The input and output of the unknown system are corrupted by additive independent noises. Gaussian models are used for all inputs. Two competing adaptation schemes are analyzed. The first is a sequential adaptation scheme where the LMS algorithm is first used to estimate the linear portion of the unknown system. The LMS algorithm is able to identify the linear portion of the unknown system to within a scale factor. The weights are then frozen at the end of the first adaptation phase. Recursions are derived for the mean and fluctuation behavior of the LMS algorithm, which are in excellent agreement with Monte Carlo simulations. When the nonlinearity is modeled by a scaled error function, the second part of the sequential gradient identification scheme is shown to correctly learn the scale factor and the error function scale factor. Mean recursions for the scale factors show good agreement with Monte Carlo simulations. For slow learning, the stationary points of the gradient algorithm closely agree with the stationary points of the theoretical recursions. The second adaptive scheme simultaneously learns both the linear and nonlinear portions of the unknown channel. The mean recursions for the linear and nonlinear portions show good agreement with Monte Carlo simulations for slow learning. The stationary points of the gradient algorithm also agree with the stationary points of the theoretical recursions     

Set-membership binormalized data-reusing LMS algorithms

This paper presents and analyzes novel data selective normalized adaptive filtering algorithms with two data reuses. The algorithms [the set-membership binormalized LMS (SM-BN-DRLMS) algorithms] are derived using the concept of set-membership filtering (SMF). These algorithms can be regarded as generalizations of the previously proposed set-membership NLMS (SM-NLMS) algorithm. They include two constraint sets in order to construct a space of feasible solutions for the coefficient updates. The algorithms include data-dependent step sizes that provide fast convergence and low-excess mean-squared error (MSE). Convergence analyzes in the mean squared sense are presented, and closed-form expressions are given for both white and colored input signals. Simulation results show good performance of the algorithms in terms of convergence speed, final misadjustment, and reduced computational complexity.     

采用归一化最小均方误差准则的LM-BP算法

传统神经网络通常以最小均方误差(LMS)或最小二乘（RLS）为收敛准则,而在自适应均衡等一些应用中,使用归一化最小均方误差（NLMS）准则可以使神经网络性能更加优越。本文在NLMS准则基础上,提出了一种以Levenberg-Marquardt（LM）训练的神经网络收敛算法。通过将神经网络的误差函数归一化,然后采用LM算法作为训练算法,实现了神经网络的快速收敛。理论分析和实验仿真表明,与采用最速下降法的NLMS准则和采用LM算法的LMS准则相比,本文算法收敛速度快,归一化均方误差更小,应用于神经网络水印系统中实现了水印信息的盲提取,能更好的抵抗噪声、低通滤波和重量化等攻击,性能平均提高了4%。      

A set of algorithms linking NLMS and block RLS algorithms

This paper describes a set of block processing algorithms which contains as extremal cases the normalized least mean squares (NLMS) and the block recursive least squares (BRLS) algorithms. All these algorithms use small block lengths, thus allowing easy implementation and small input-output delay. It is shown that these algorithms require a lower number of arithmetic operations than the classical least mean squares (LMS) algorithm, while converging much faster. A precise evaluation of the arithmetic complexity is provided, and the adaptive behavior of the algorithm is analyzed. Simulations illustrate that the tracking characteristics of the new algorithm are also improved compared to those of the NLMS algorithm. The conclusions of the theoretical analysis are checked by simulations, illustrating that, even in the case where noise is added to the reference signal, the proposed algorithm allows altogether a faster convergence and a lower residual error than the NLMS algorithm. Finally, a sample-by-sample version of this algorithm is outlined, which is the link between the NLMS and recursive least squares (RLS) algorithms