A revisit to block and recursive least squares for parameter estimation

In this paper, the classical least squares (LS) and recursive least squares (RLS) for parameter estimation have been re-examined in the light of the present day computing capabilities. It has been demonstrated that for linear time-invariant systems, the performance of blockwise least squares (BLS) is always superior to that of RLS. In the context of parameter estimation for dynamic systems, the current computational capability of personal computers are more than adequate for BLS. However, for time-varying systems with abrupt parameter changes, standard blockwise LS may no longer be suitable due to its inefficiency in discarding “old” data. To deal with this limitation, a novel sliding window blockwise least squares approach with automatically adjustable window length triggered by a change detection scheme is proposed. Two types of sliding windows, rectangular and exponential, have been investigated. The performance of the proposed algorithm has been illustrated by comparing with the standard RLS and an exponentially weighted RLS (EWRLS) using two examples. The simulation results have conclusively shown that: (1) BLS has better performance than RLS; (2) the proposed variable-length sliding window blockwise least squares (VLSWBLS) algorithm can outperform RLS with forgetting factors; (3) the scheme has both good tracking ability for abrupt parameter changes and can ensure the high accuracy of parameter estimate at the steady-state; and (4) the computational burden of VLSWBLS is completely manageable with the current computer technology. Even though the idea presented here is straightforward, it has significant implications to virtually all areas of application where RLS schemes are used.     

一种用于数据校正的最小二乘与准最小二乘组合方法

过程系统的控制与优化要求可靠的过程数据。通过测量得到的过程数据含有随机误差和过失误差,采用数据校正技术可有效地减小过程测量数据的误差,从而提高过程控制与优化的准确性。针对传统基于最小二乘的数据校正方法:和基于准最小二乘的鲁棒数据校正方法:,分析了它们的优缺点,并提出了一种最小二乘与准最小二乘组合方法:。该方法:先采用准最小二乘估计器检测过失误差并剔除,然后再采用最小二乘估计器进行数据校正,可以综合前两种方法:各自的优点,使得数据校正结果:更加准确。将提出最小二乘与准最小二乘组合方法:应用于线性与非线性系统的数据校正中,通过校正结果:的比较说明此方法:的具有较好的过失误差检测能力和较准确的数据校正结果:。最后将此方法:应用于实际过程系统空气分离流程的数据校正中,结果:说明了此方法:的有效性。     

A continuous-time framework for least squares parameter estimation

This paper proposes a continuous-time framework for the least-squares parameter estimation method through evolution equations. Nonlinear systems in the standard state space representation that are linear in the unknown, constant parameters are investigated. Two estimators are studied. The first one consists of a linear evolution equation while the second one consists of an impulsive linear evolution equation. The paper discusses some theoretical aspects related to the proposed estimators: uniqueness of a solution and an attractive equilibrium point which solves for the unknown parameters. A deterministic framework for the estimation under noisy measurements is proposed using a Sobolev space with negative index to model the noise. The noise can be of large magnitude. Concrete signals issued from an electronic device are used to discuss numerical aspects.     

Bounds for an estimate of the optimal backward error for linear least squares problems

Algorithms for calculating sequences of upper and lower bounds for an estimate of the optimal backward error for linear least squares problems are developed, based on the Gauss quadrature theory. Numerical results show that the bounds converge quickly and are therefore useful in practice.     

The Koul-Susarla-Van Ryzin and weighted least squares estimates for censored linear regression model: A comparative study

The Koul-Susarla-Van Ryzin (KSV) and weighted least squares (WLS) methods are simple to use techniques for handling linear regression models with censored data. They do not require any iterations and standard computer routines can be employed for model fitting. Emphasis has been given to the consistency and asymptotic normality for both estimators, but the finite sample performance of the WLS estimator has not been thoroughly investigated. The finite sample performance of these two estimators is compared using an extensive simulation study as well as an analysis of the Stanford heart transplant data. The results demonstrate that the WLS approach performs much better than the KSV method and is reliable for use with censored data.     

Accuracy analysis of bias-eliminating least squares estimates for errors-in-variables systems

The bias-eliminating least squares (BELS) method is one of the consistent estimators for identifying dynamic errors-in-variables systems. In this paper, we investigate the accuracy properties of the BELS estimates. An explicit expression for the normalized asymptotic covariance matrix of the estimated parameters is derived and supported by some numerical examples.     

Decomposition-based recursive least squares identification methods for multivariate pseudo-linear systems using the multi-innovation

This paper studies the parameter estimation algorithms of multivariate pseudo-linear autoregressive systems. A decomposition-based recursive generalised least squares algorithm is deduced for estimating the system parameters by decomposing the multivariate pseudo-linear autoregressive system into two subsystems. In order to further improve the parameter accuracy, a decomposition based multi-innovation recursive generalised least squares algorithm is developed by means of the multi-innovation theory. The simulation results confirm that these two algorithms are effective.     

Comparison results on the preconditioned GAOR method for generalized least squares problems

Recently, Zhou et al. [Preconditioned GAOR methods for solving weighted linear least squares problems, J. Comput. Appl. Math. 224 (2009), pp. 242–249] have proposed the preconditioned generalized accelerated over relaxation (GAOR) methods for solving generalized least squares problems and studied their convergence rates. In this paper, we propose a new type of preconditioners and study the convergence rates of the new preconditioned GAOR methods for solving generalized least squares problems. Comparison results show that the convergence rates of the new preconditioned GAOR methods are better than those of the preconditioned GAOR methods presented by Zhou et al. whenever these methods are convergent. Lastly, numerical experiments are provided in order to confirm the theoretical results studied in this paper.     

Matrix strategies for computing the least trimmed squares estimation of the general linear and SUR models

An algorithm for computing the exact least trimmed squares (LTS) estimator of the standard regression model has recently been proposed. The LTS algorithm is adapted to the general linear and seemingly unrelated regressions models with possible singular dispersion matrices. It searches through a regression tree to find the optimal estimates and has combinatorial complexity. The model is formulated as a generalized linear least squares problem. Efficient matrix techniques are employed to update the generalized residual sum of squares of a subset model. Specifically, the new algorithm utilizes previous computations to update a generalized QR decomposition by a single row. The sparse structure of the model is exploited. Theoretical measures of computational complexity are provided. Experimental results confirm the ability of the new algorithms to identify outlying observations.     

Hierarchical Newton and least squares iterative estimation algorithm for dynamic systems by transfer functions based on the impulse responses

This paper develops a parameter estimation algorithm for linear continuous-time systems based on the hierarchical principle and the parameter decomposition strategy. Although the linear continuous-time system is a linear system, its output response is a highly nonlinear function with respect to the system parameters. In order to propose a direct estimation algorithm, a criterion function is constructed between the response output and the observation output by means of the discrete sampled data. Then a scheme by combining the Newton iteration and the least squares iteration is builded to minimise the criterion function and derive the parameter estimation algorithm. In light of the different features between the system parameters and the output function, two sub-algorithms are derived by using the parameter decomposition. In order to remove the associate terms between the two sub-algorithms, a Newton and least squares iterative algorithm is deduced to identify system parameters. Compared with the Newton iterative estimation algorithm without the parameter decomposition, the complexity of the hierarchical Newton and least squares iterative estimation algorithm is reduced because the dimension of the Hessian matrix is lessened after the parameter decomposition. The experimental results show that the proposed algorithm has good performance.     

Iterative algorithm for minimal norm least squares solution to general linear matrix equations

This paper is concerned with minimal norm least squares solution to general linear matrix equations including the well-known Lyapunov matrix equation and Sylvester matrix equation as special cases. Two iterative algorithms are proposed to solve this problem. The first method is based on the gradient search principle for solving optimization problem and the second one can be regarded as its dual form. For both algorithms, necessary and sufficient conditions guaranteeing the convergence of the algorithms are presented. The optimal step sizes such that the convergence rates of the algorithms are maximized are established in terms of the singular values of some coefficient matrix. It is believed that the proposed methods can perform important functions in many analysis and design problems in systems theory.     

Octree-based, GPU implementation of a continuous cellular automaton for the simulation of complex, evolving surfaces

Presently, dynamic surface-based models are required to contain increasingly larger numbers of points and to propagate them over longer time periods. For large numbers of surface points, the octree data structure can be used as a balance between low memory occupation and relatively rapid access to the stored data. For evolution rules that depend on neighborhood states, extended simulation periods can be obtained by using simplified atomistic propagation models, such as the Cellular Automata (CA). This method, however, has an intrinsic parallel updating nature and the corresponding simulations are highly inefficient when performed on classical Central Processing Units (CPUs), which are designed for the sequential execution of tasks. In this paper, a series of guidelines is presented for the efficient adaptation of octree-based, CA simulations of complex, evolving surfaces into massively parallel computing hardware. A Graphics Processing Unit (GPU) is used as a cost-efficient example of the parallel architectures. For the actual simulations, we consider the surface propagation during anisotropic wet chemical etching of silicon as a computationally challenging process with a wide-spread use in microengineering applications. A continuous CA model that is intrinsically parallel in nature is used for the time evolution. Our study strongly indicates that parallel computations of dynamically evolving surfaces simulated using CA methods are significantly benefited by the incorporation of octrees as support data structures, substantially decreasing the overall computational time and memory usage.     

Combining multi-core and GPU computing for solving combinatorial optimization problems

In this paper, we revisit the design and implementation of Branch-and-Bound (B&B) algorithms for solving large combinatorial optimization problems on GPU-enhanced multi-core machines. B&B is a tree-based optimization method that uses four operators (selection, branching, bounding and pruning) to build and explore a highly irregular tree representing the solution space. In our previous works, we have proposed a GPU-accelerated approach in which only a single CPU core is used and only the bounding operator is performed on the GPU device. Here, we extend the approach (LL-GB&B) in order to minimize the CPU–GPU communication latency and thread divergence. Such an objective is achieved through a GPU-based fine-grained parallelization of the branching and pruning operators in addition to the bounding one. The second contribution consists in investigating the combination of a GPU with multi-core processing. Two scenarios have been explored leading to two approaches: a concurrent (RLL-GB&B) and a cooperative one (PLL-GB&B). In the first one, the exploration process is performed concurrently by the GPU and the CPU cores. In the cooperative approach, the CPU cores prepare and off-load to GPU pools of tree nodes using data streaming while the GPU performs the exploration. The different approaches have been extensively experimented on the Flowshop scheduling problem. Compared to a single CPU-based execution, LL-GB&B allows accelerations up to (×$\times$160) for large problem instances. Moreover, when combining multi-core and GPU, we figure out that using RLL-GB&B is not beneficial while PLL-GB&B enables an improvement up to 36% compared to LL-GB&B.     

基于GEP的最小二乘支持向量机模型参数选择

针对最小二乘支持向量机的多参数寻优问题,提出了一种基于基因表达式编程的最小二乘支持向量机参数优选方法.该算法将最小二乘支持向量机参数(C,σ)样本作为GEP的基因,按其变异算子随着进化代数和染色体所含基因数目动态变化的机制执行,其收敛速度和精确度大大提高.并与基于粒子群算法和遗传算法参数优选方法比较,通过标准测试函数验证了该算法的拟合误差最低.最后用其建立氧化铝生产蒸发过程参数预测模型,应用工业生产数据进行验证,实验结果表明该方法有效且获得了满意的效果.     

On optimization of finite-difference time-domain (FDTD) computation on heterogeneous and GPU clusters

A model for the computational cost of the finite-difference time-domain (FDTD) method irrespective of implementation details or the application domain is given. The model is used to formalize the problem of optimal distribution of computational load to an arbitrary set of resources across a heterogeneous cluster. We show that the problem can be formulated as a minimax optimization problem and derive analytic lower bounds for the computational cost. The work provides insight into optimal design of FDTD parallel software. Our formulation of the load distribution problem takes simultaneously into account the computational and communication costs. We demonstrate that significant performance gains, as much as 75%, can be achieved by proper load distribution.     

Parallel and distributed computing models on a graphics processing unit to accelerate simulation of membrane systems

Membrane systems are parallel distributed computing models that are used in a wide variety of areas. Use of a sequential machine to simulate membrane systems loses the advantage of parallelism in Membrane Computing. In this paper, an innovative classification algorithm based on a weighted network is introduced. Two new algorithms have been proposed for simulating membrane systems models on a Graphics Processing Unit (GPU). Communication and synchronization between threads and thread blocks in a GPU are time-consuming processes. In previous studies, dependent objects were assigned to different threads. This increases the need for communication between threads, and as a result, performance decreases. In previous studies, dependent membranes have also been assigned to different thread blocks, requiring inter-block communications and decreasing performance. The speedup of the proposed algorithm on a GPU that classifies dependent objects using a sequential approach, for example with 512 objects per membrane, was 82×, while for the previous approach (Algorithm 1), it was 8.2×. For a membrane system with high dependency among membranes, the speedup of the second proposed algorithm (Algorithm 3) was 12×, while for the previous approach (Algorithm 1) and the first proposed algorithm (Algorithm 2) that assign each membrane to one thread block, it was 1.8×.     

A hybrid model of partial least squares and neural network for traffic incident detection

Development of a universal freeway incident detection algorithm is a task that remains unfulfilled and many promising approaches have been recently explored. The partial least squares (PLS) method and artificial neural network (NN) were found in previous studies to yield superior incident detection performance. In this article, a hybrid model which combines PLS and NN is developed to detect automatically traffic incident. A real traffic data set collected from motorways A12 in the Netherlands is presented to illustrate such an approach. Data cleansing has been introduced to preprocess traffic data sets to improve the data quality in order to increase the veracity and reliability of incident model. The detection performance is evaluated by the common criteria including detection rate, false alarm rate, mean time to detection, classification rate and the area under the curve (AUC) of the receiver operating characteristic. Computational results indicate that the hybrid approach is capable of increasing detection performance comparing to PLS, and simplifying the NN structure for incident detection. The hybrid model is a promising alternative to the usual PLS or NN for incident detection.     

Regularized least squares support vector regression for the simultaneous learning of a function and its derivatives

In this paper, we propose a regularized least squares approach based support vector machine for simultaneously approximating a function and its derivatives. The proposed algorithm is simple and fast as no quadratic programming solver needs to be employed. Effectively, only the solution of a structured system of linear equations is needed.     

基于最小二乘和最小投影距离的车牌倾斜校正

在车牌图像的采集过程中,经常会有车牌倾斜的现象发生,这种倾斜给后续的字符分割和字符识别造成了很多不利影响。为此,文中提出了一种基于最小二乘和最小投影距离的车牌倾斜校正方法。该方法将车牌倾斜分成水平倾斜和垂直倾斜两部分:对于水平倾斜,首先对二值化后的车牌去边框和铆钉,再对车牌利用最小二乘拟合直线求取倾斜角;而对于垂直倾斜,则引入分块查找法来降低查找最小投影距离的执行次数,从而提高算法的执行效率。实验结果表明:该算法简单实用,能够准确地对车牌进行校正。