Solving the least squares problem using a parallel linear algebra library

One of the important issues in the construction of a parallel Linear Algebra library is the choice of the process structure. The structure that is presented in this article allows for simple functional specifications of the processes and for their compositionality. Each functional specification describes the meaning of the parallel composition of a number of instances of a single process. The communication behaviours of the instances of a process do not occur in its specification. In such a specification, matrices and vectors occur as ordinary mathematical variables. The representations of the matrices and vectors are distributed across the process instances. All library processes conform to the same communication conventions. The library processes can be composed sequentially, without requiring global synchronisation between process calls. As an example, the parallel solution of the least squares problem is discussed.     

Solving linear differential equations as a minimum norm least squares problem with error-bounds

Linear ordinary/partial differential equations (DEs) with linear boundary conditions (BCs) are posed as an error minimization problem. This problem has a linear objective function and a system of linear algebraic (constraint) equations and inequalities derived using both the forward and the backward Taylor series expansion. The DEs along with the BCs are approximated as linear equations/inequalities in terms of the dependent variables and their derivatives so that the total error due to discretization and truncation is minimized. The total error along with the rounding errors render the equations and inequalities inconsistent to an extent or, equivalently, near-consistent, in general. The degree of consistency will be reasonably high provided the errors are not dominant. When this happens and when the equations/inequalities are compatible with the DEs, the minimum value of the total discretization and truncation errors is taken as zero. This is because of the fact that these errors could be negative as well as positive with equal probability due to the use of both the backward and forward series. The inequalities are written as equations since the minimum value of the error (implying error-bound and written/expressed in terms of a nonnegative quantity) in each equation will be zero. The minimum norm least-squares solution (that always exists) of the resulting over-determined system will provide the required solution whenever the system has a reasonably high degree of consistency. A lower error-bound and an upper error-bound of the solution are also included to logically justify the quality/validity of the solution.     

Partial condition number for the equality constrained linear least squares problem

In this paper, the normwise condition number of a linear function of the equality constrained linear least squares solution called the partial condition number is considered. Its expression and closed formulae are first presented when the data space and the solution space are measured by the weighted Frobenius norm and the Euclidean norm, respectively. Then, we investigate the corresponding structured partial condition number when the problem is structured. To estimate these condition numbers with high reliability, the probabilistic spectral norm estimator and the small-sample statistical condition estimation method are applied and two algorithms are devised. The obtained results are illustrated by numerical examples.     

USSOR methods for solving the rank deficient linear least squares problem

In order to find the least squares solution of minimal norm to linear system \(Ax=b\) with \(A \in \mathcal{C}^{m \times n}\) being a matrix of rank \(r< n \le m\), \(b \in \mathcal{C}^{m}\), Zheng and Wang (Appl Math Comput 169:1305–1323, 2005) proposed a class of symmetric successive overrelaxation (SSOR) methods, which is based on augmenting system to a block \(4 \times 4\) consistent system. In this paper, we construct the unsymmetric successive overrelaxation (USSOR) method. The semiconvergence of the USSOR method is discussed. Numerical experiments illustrate that the number of iterations and CPU time for the USSOR method with the appropriate parameters is respectively less and faster than the SSOR method with optimal parameters.     

Hermitian tridiagonal solution with the least norm to quaternionic least squares problem

Quaternionic least squares (QLS) is an efficient method for solving approximate problems in quaternionic quantum theory. In view of the extensive applications of Hermitian tridiagonal matrices in physics, in this paper we list some properties of basis matrices and subvectors related to tridiagonal matrices, and give an iterative algorithm for finding Hermitian tridiagonal solution with the least norm to the quaternionic least squares problem by making the best use of structure of real representation matrices, we also propose a preconditioning strategy for the Algorithm LSQR-Q in Wang, Wei and Feng (2008) [14] and our algorithm. Numerical experiments are provided to verify the effectiveness of our method.     

Solving nonlinear systems with least significant bit accuracy

We give an algorithm for constructing an inclusion of the solution of a system of nonlinear equations. In contrast to existing methods, the algorithm does not require properties which are difficult to verify such as the non-singularity of a matrix. In fact this latter property is demonstrated by the algorithm itself. The highly accurate computational results are obtained in terms of a residue of first or higher order of the system.     

Simplified neural networks for solving linear least squares andtotal least squares problems in real time

In this paper a new class of simplified low-cost analog artificial neural networks with on chip adaptive learning algorithms are proposed for solving linear systems of algebraic equations in real time. The proposed learning algorithms for linear least squares (LS), total least squares (TLS) and data least squares (DLS) problems can be considered as modifications and extensions of well known algorithms: the row-action projection-Kaczmarz algorithm and/or the LMS (Adaline) Widrow-Hoff algorithms. The algorithms can be applied to any problem which can be formulated as a linear regression problem. The correctness and high performance of the proposed neural networks are illustrated by extensive computer simulation results.     

A unifying theorem for linear and total linear least squares

It is shown how both linear least-squares and total linear least-squares estimation schemes are special cases of a rank one modification of the data matrix or the sample covariance matrix. For a problem with n unknowns, there exist n linear least-squares solutions while the total linear least-squares solution is (generically) unique. When the signal-to-noise ratio is sufficiently high, the total least-squares solution is a nonnegative combination of the least-squares solutions     

Weighted least squares stationary approximations to linear systems

The problem of replacing the time-varying linear systemdot{X} = A(t)Xby a stationary onedot{Y} = BYis investigated. The matrixBis selected so thatX(t) = Y(t)in the interval [0, T]. Several quadratic criteria are proposed to aid in determining suitable candidate systems. One criterion for choosingBis initial condition dependent, and another bounds the "worst case" homogeneous system performance. Both of these criteria produce weighted least squares fits toA(t).     

High accuracy asymptotics for a least squares spline error

The error of a least squares spline of an arbitrary degree p and all its derivatives is considered. Highly accurate asymptotic approximation is constructed for the dependence of the error coefficient on p.     

Solving non-negative matrix factorization by alternating least squares with a modified strategy

Non-negative matrix factorization (NMF) is a method to obtain a representation of data using non-negativity constraints. A popular approach is alternating non-negative least squares (ANLS). As is well known, if the sequence generated by ANLS has at least one limit point, then the limit point is a stationary point of NMF. However, no evdience has shown that the sequence generated by ANLS has at least one limit point. In order to overcome this shortcoming, we propose a modified strategy for ANLS in this paper. The modified strategy can ensure the sequence generated by ANLS has at least one limit point, and this limit point is a stationary point of NMF. The results of numerical experiments are reported to show the effectiveness of the proposed algorithm.     

Solving linear least squares by orthogonal factorization and pseudoinverse computation via the modified Huang algorithm in the ABS class

We consider three algorithms for solving linear least squares problems based upon the modified Huang algorithm (MHA) in the ABS class for linear systems recently introduced by Abaffy, Broyden and Spedicato. The first algorithm uses an explicit QR factorization of the coefficient matrixA computed by applying MHA to the matrixA ^T . The second and the third algorithm is based upon two representations of the Moore-Penrose pseudoinverse constructed with the use of MHA. The three algorithms are tested on a large set of problems and compared with the NAG code using QR factorization with Householder rotations. The comparison shows that the algorithms based on MHA are generally more accurate.     

On the solution of the linear least squares problems and pseudo-inverses

It is known that the computed least squares solutionx ofAx=b, in the presence of the round-off error, satisfies the perturbed equation(A+E)(x+h)=b+f. The practical considerations of computing the solution are discussed and it is found that rank(A+E)=rank (A). A general analysis of the condition of the linear least squares problems and pseudo-inverses is then presented using this assumption. Norms of relevant round-off error perturbations are estimated for two known methods of solution. Comparison between different algorithms is given by numerical examples.     

Fitting generalized linear models and their nonlinear extensions with least squares calculations

A class of maximum likelihood algorithms called NRL algorithms that can be implemented with a sequence of least squares calculations is developed. When applied to generalized linear models and their nonlinear extensions, this class includes several algorithms that have been previously proposed. Properties of the algorithms are examined both in the initial iterations and also near the maximum likelihood estimate; different types of algorithm often perform best in these two situations. A strategy for switching between two such NRL algorithms is presented.     

The regularized least squares algorithm and the problem of learning halfspaces

We provide sample complexity of the problem of learning halfspaces with monotonic noise, using the regularized least squares algorithm in the reproducing kernel Hilbert spaces (RKHS) framework.     

Algorithm Q-LSQR for the least squares problem in quaternionic quantum theory

In this paper, an iterative algorithm for the standard quaternionic least squares problem is proposed without using the real (complex) representation. Our algorithm is implemented in the quaternion field and by means of direct quaternion arithmetic and is a natural generalization of the LSQR algorithm for the real least squares problem.     

On the accuracy of least squares methods in the presence of corner singularities

This paper treats elliptic problems with corner singularities. Finite element approximations based on variational principles of the least squares type tend to display poor convergence properties in such contexts. Moreover, mesh refinement or the use of special singular elements do not appreciably improve matters. Here we show that if the least squares formulation is done in appropriately weighted spaces, then optimal convergence results in unweighted spaces like L².