Function Approximation With XCS: Hyperellipsoidal Conditions, Recursive Least Squares, and Compaction

An important strength of learning classifier systems (LCSs) lies in the combination of genetic optimization techniques with gradient-based approximation techniques. The chosen approximation technique develops locally optimal approximations, such as accurate classification estimates, Q-value predictions, or linear function approximations. The genetic optimization technique is designed to distribute these local approximations efficiently over the problem space. Together, the two components develop a distributed, locally optimized problem solution in the form of a population of expert rules, often called classifiers. In function approximation problems, the XCSF classifier system develops a problem solution in the form of overlapping, piecewise linear approximations. This paper shows that XCSF performance on function approximation problems additively benefits from: 1) improved representations; 2) improved genetic operators; and 3) improved approximation techniques. Additionally, this paper introduces a novel closest classifier matching mechanism for the efficient compaction of XCS's final problem solution. The resulting compaction mechanism can boil the population size down by 90% on average, while decreasing prediction accuracy only marginally. Performance evaluations show that the additional mechanisms enable XCSF to reliably, accurately, and compactly approximate even seven dimensional functions. Performance comparisons with other, heuristic function approximation techniques show that XCSF yields competitive or even superior noise-robust performance.     

Learning the stress function pattern of ordered weighted average aggregation using DBSCAN clustering

Ordered weighted average (OWA) operator provides a parameterized class of mean type operators between the minimum and the maximum. It is an important tool that can reflect the strategy of a decision maker for decision-making problems. In this study, the idea of obtaining the stress function from OWA weights has been put forward to generalize and characterize OWA weights. The main idea in this paper is mainly constructed on the basis that, generally, stress functions can be constructed using a mixture of constant and linear components. So, we can consider the stress function as a piecewise linear function. For obtaining stress functions as piecewise linear functions, we present a clustering-based approach for OWA weight generalization. This generalization is made using the DBSCAN algorithm as the learning method of a stress function associated with known OWA weights. In the learning process, the whole data set is divided into clusters, and then linear functions are obtained via a least squares estimator.     

A Learning Algorithm of CMAC Based on RLS

Conventionally, least mean square rule which can be named CMAC-LMS is used to update the weights of CMAC. The convergence ability of CMAC-LMS is very sensitive to the learning rate. Applying recursive least squares (RLS) algorithm to update the weights of CMAC, we bring forward an algorithm named CMAC-RLS. And the convergence ability of this algorithm is proved and analyzed. Finally, the application of CMAC-RLS to control nonlinear plant is investigated. The simulation results show the good convergence performance of CMAC-RLS. The results also reveal that the proposed CMAC-PID controller can reject disturbance effectively, and control nonlinear time-varying plant adaptively.     

基于蚁群算法优化回声状态网络的研究

针对输出权值采用最小二乘法的回声状态网络(ESN),在随机选取输入权值和隐层神经元阈值时,存在收敛速度慢、预测精度不稳定等问题,提出了基于蚁群算法优化回声状态网络(ACO-ESN)的算法。该算法将优化回声状态网络的初始输入权值、隐层神经元阈值问题转化为蚁群算法中蚂蚁寻找最佳路径的问题,输出权值采用最小二乘法计算,通过蚁群算法的更新、变异、遗传等操作训练回声状态网络,选择出使回声状态网络预测误差最小的输入权值和阈值,从而提高其预测性能。将ACO-ESN与ELM、I-ELM、OS-ELM、B-ELM等神经网络的仿真结果进行对比,结果验证经过蚁群算法优化的回声状态网络加快了其收敛速度,改善了其预测性能,并增强了隐层神经元的敏感度。     

A novel image encryption algorithm based on piecewise linear chaotic maps and least squares approximation

In this paper, we propose a novel image encryption algorithm based on chaotic maps and least squares approximations. The proposed algorithm consists of two main phases, which are applied sequentially in several rounds, namely a shuffling phase and a masking phase. Both phases are based on 1–dimensional piecewise linear chaotic maps and act on the rows/columns of the input plain image. Least squares approximations are used to strengthen the security of the proposed algorithm by providing strong mixing between the rows/columns of the image. Simulation results show that the proposed image encryption algorithm is robust against common statistical and security attacks. We present thorough comparison of the proposed algorithm with some existing image encryption algorithms.

     

基于- 范数约束的LSSVR 多核学习算法

针对核函数选择对最小二乘支持向量机回归模型泛化性的影响, 提出一种新的基于????- 范数约束的最小二乘支持向量机多核学习算法. 该算法提供了两种求解方法, 均通过两重循环进行求解, 外循环用于更新核函数的权值, 内循环用于求解最小二乘支持向量机的拉格朗日乘数, 充分利用该多核学习算法, 有效提高了最小二乘支持向量机的泛化能力, 而且对惩罚参数的选择具有较强的鲁棒性. 基于单变量和多变量函数的仿真实验表明了所提出算法的有效性.

     

Least squares approximate policy iteration for learning bid prices in choice-based revenue management

We consider the revenue management problem of capacity control under customer choice behavior. An exact solution of the underlying stochastic dynamic program is difficult because of the multi-dimensional state space and, thus, approximate dynamic programming (ADP) techniques are widely used. The key idea of ADP is to encode the multi-dimensional state space by a small number of basis functions, often leading to a parametric approximation of the dynamic program’s value function. In general, two classes of ADP techniques for learning value function approximations exist: mathematical programming and simulation. So far, the literature on capacity control largely focuses on the first class.In this paper, we develop a least squares approximate policy iteration (API) approach which belongs to the second class. Thereby, we suggest value function approximations that are linear in the parameters, and we estimate the parameters via linear least squares regression. Exploiting both exact and heuristic knowledge from the value function, we enforce structural constraints on the parameters to facilitate learning a good policy. We perform an extensive simulation study to investigate the performance of our approach. The results show that it is able to obtain competitive revenues compared to and often outperforms state-of-the-art capacity control methods in reasonable computational time. Depending on the scarcity of capacity and the point in time, revenue improvements of around 1% or more can be observed. Furthermore, the proposed approach contributes to simulation-based ADP, bringing forth research on numerically estimating piecewise linear value function approximations and their application in revenue management environments.     

OWA Operators in Regression Problems

We consider an application of fuzzy logic connectives to statistical regression. We replace the standard least squares, least absolute deviation, and maximum likelihood criteria with an ordered weighted averaging (OWA) function of the residuals. Depending on the choice of the weights, we obtain the standard regression problems, high-breakdown robust methods (least median, least trimmed squares, and trimmed likelihood methods), as well as new formulations. We present various approaches to numerical solution of such regression problems. OWA-based regression is particularly useful in the presence of outliers, and we illustrate the performance of the new methods on several instances of linear regression problems with multiple outliers.      

On the smooth convergence of subdivision and degree elevation for Bézier curves

Bézier subdivision and degree elevation algorithms generate piecewise linear approximations of Bézier curves that converge to the original Bézier curve. Discrete derivatives of arbitrary order can be associated with these piecewise linear functions via divided differences. Here we establish the convergence of these discrete derivatives to the corresponding continuous derivatives of the initial Bézier curve. Thus, we show that the control polygons generated by subdivision and degree elevation provide not only an approximation to a Bézier curve, but also approximations of its derivatives of arbitrary order.     

A unified, integral construction for coordinates over closed curves

We propose a simple generalization of Shephard's interpolation to piecewise smooth, convex closed curves that yields a family of boundary interpolants with linear precision. Two instances of this family reduce to previously known interpolants: one based on a generalization of Wachspress coordinates to smooth curves and the other an integral version of mean value coordinates for smooth curves. A third instance of this family yields a previously unknown generalization of discrete harmonic coordinates to smooth curves. For closed, piecewise linear curves, we prove that our interpolant reproduces a general family of barycentric coordinates considered by Floater, Hormann and Kós that includes Wachspress coordinates, mean value coordinates and discrete harmonic coordinates.     

自步上下文感知的相关滤波跟踪算法

针对目标跟踪中的目标尺度变换、遮挡、快速运动等问题,提出自步上下文感知的相关滤波跟踪算法。首先在正则化最小二乘分类器中引入目标的全局上下文信息,使得这些上下文信息能够被滤波器所学到,并对目标产生高响应,对上下文信息接近零响应;然后引入自步学习,给每一帧的目标和上下文信息赋予权重,挑选出可靠的目标和上下文信息,更新滤波模板;最后学习得到稳健和高效的外观模型。实验表明本文算法在距离精度（DP）提高了2.81%,成功率（SR）提高了13.9%,具有较好的跟踪效果。     

A multi-start algorithm to design a multi-class classifier for a multi-criteria ABC inventory classification problem

In this paper we deal with the problem of designing a classifier able to learn the classification of existing units in inventory and then use it to classify new units according to their attributes in a multi-criteria ABC inventory classification environment. To solve this problem we design a multi-start constructive algorithm to train a discrete artificial neural network using a randomized greedy strategy to add neurons to the network hidden layer. The process of weights’ searching for the neurons to be added is based on solving linear programming formulations. The computational experiments show that the proposed algorithm is much more efficient when the dual formulations are used to find the weights of the network neurons and that the obtained classifier has good levels of generalization accuracy. In addition, the proposed algorithm can be straight applied to other multi-class classification problems with more than three classes.     

A local linearized least squares algorithm for training feedforwardneural networks

In training the weights of a feedforward neural network, it is well known that the global extended Kalman filter (GEKF) algorithm has much better performance than the popular gradient descent with error backpropagation in terms of convergence and quality of solution. However, the GEKF is very computationally intensive, which has led to the development of efficient algorithms such as the multiple extended Kalman algorithm (MEKA) and the decoupled extended Kalman filter algorithm (DEKF), that are based on dimensional reduction and/or partitioning of the global problem. In this paper we present a new training algorithm, called local linearized least squares (LLLS), that is based on viewing the local system identification subproblems at the neuron level as recursive linearized least squares problems. The objective function of the least squares problems for each neuron is the sum of the squares of the linearized backpropagated error signals. The new algorithm is shown to give better convergence results for three benchmark problems in comparison to MEKA, and in comparison to DEKF for highly coupled applications. The performance of the LLLS algorithm approaches that of the GEKF algorithm in the experiments.     

A Kernel-Based Two-Class Classifier for Imbalanced Data Sets

Many kernel classifier construction algorithms adopt classification accuracy as performance metrics in model evaluation. Moreover, equal weighting is often applied to each data sample in parameter estimation. These modeling practices often become problematic if the data sets are imbalanced. We present a kernel classifier construction algorithm using orthogonal forward selection (OFS) in order to optimize the model generalization for imbalanced two-class data sets. This kernel classifier identification algorithm is based on a new regularized orthogonal weighted least squares (ROWLS) estimator and the model selection criterion of maximal leave-one-out area under curve (LOO-AUC) of the receiver operating characteristics (ROCs). It is shown that, owing to the orthogonalization procedure, the LOO-AUC can be calculated via an analytic formula based on the new regularized orthogonal weighted least squares parameter estimator, without actually splitting the estimation data set. The proposed algorithm can achieve minimal computational expense via a set of forward recursive updating formula in searching model terms with maximal incremental LOO-AUC value. Numerical examples are used to demonstrate the efficacy of the algorithm     

KNN-based least squares twin support vector machine for pattern classification

The least squares twin support vector machine (LSTSVM) generates two non-parallel hyperplanes by directly solving a pair of linear equations as opposed to solving two quadratic programming problems (QPPs) in the conventional twin support vector machine (TSVM), which makes learning speed of LSTSVM faster than that of the TSVM. However, LSTSVM fails to discover underlying similarity information within samples which may be important for classification performance. To address the above problem, we apply the similarity information of samples into LSTSVM to build a novel non-parallel plane classifier, called K-nearest neighbor based least squares twin support vector machine (KNN-LSTSVM). The proposed method not only retains the superior advantage of LSTSVM which is simple and fast algorithm but also incorporates the inter-class and intra-class graphs into the model to improve classification accuracy and generalization ability. The experimental results on several synthetic as well as benchmark datasets demonstrate the efficiency of our proposed method. Finally, we further went on to investigate the effectiveness of our classifier for human action recognition application.     

Classification of BMD and ADHD patients using their EEG signals

Bipolar Mood Disorder (BMD) and Attention Deficit Hyperactivity Disorder (ADHD) patients mostly share clinical signs and symptoms in children; therefore, accurate distinction of these two mental disorders is a challenging issue among the psychiatric society. In this study, 43 subjects are participated including 21 patients with ADHD and 22 subjects with BMD. Their electroencephalogram (EEG) signals are recorded by 22 electrodes in two eyes-open and eyes-closed resting conditions. After a preprocessing step, several features such as band power, fractal dimension, AR model coefficients and wavelet coefficients are extracted from recorded signals. This paper is aimed to achieve a high classification rate between ADHD and BMD patients using a suitable classifier to their EEG features. In this way, we consider a piece wise linear classifier which is designed based on XCSF. Experimental results of XCSF-LDA showed a significant improvement (86.44% accuracy) compare to that of standard XCSF (78.55%). To have a fair comparison, the other state-of-art classifiers such as LDA, Direct LDA, boosted JD-LDA (BJDLDA), and XCSF are assessed with the same feature set that finally the proposed method provided a better results in comparison with the other rival classifiers. To show the robustness of our method, additive white noise with different amplitude is added to the raw signals but the results achieved by the proposed classifier empirically confirmed a higher robustness against noise compare to the other classifiers. Consequently, the proposed classifier can be considered as an effective method to classify EEG features of BMD and ADHD patients.     

A comparison of two MLEM2 rule induction algorithms extended to probabilistic approximations

A probabilistic approximation is a generalization of the standard idea of lower and upper approximations, defined for equivalence relations. Recently probabilistic approximations were additionally generalized to an arbitrary binary relation so that probabilistic approximations may be applied for incomplete data. We discuss two ways to induce rules from incomplete data using probabilistic approximations, by applying true MLEM2 algorithm and an emulated MLEM2 algorithm. In this paper we report novel research on a comparison of both approaches: new results of experiments on incomplete data with three interpretations of missing attribute values. Our results show that both approaches do not differ much.     

Fast Blind Equalization Using Complex-Valued MLP

The blind equalizers based on complex valued feedforward neural networks, for linear and nonlinear communication channels, yield better performance as compared to linear equalizers. The learning algorithms are, generally, based on stochastic gradient descent, as they are simple to implement. However, these algorithms show a slow convergence rate. In the blind equalization problem, the unavailability of the desired output signal and the presence of nonlinear activation functions make the application of recursive least squares algorithm difficult. In this letter, a new scheme using recursive least squares algorithm is proposed for blind equalization. The learning of weights of the output layer is obtained by using a modified version of constant modulus algorithm cost function. For the learning of weights of hidden layer neuron space adaptation approach is used. The proposed scheme results in faster convergence of the equalizer.     

Model selection for the LS-SVM. Application to handwriting recognition

The support vector machine (SVM) is a powerful classifier which has been used successfully in many pattern recognition problems. It has also been shown to perform well in the handwriting recognition field. The least squares SVM (LS-SVM), like the SVM, is based on the margin-maximization principle performing structural risk minimization. However, it is easier to train than the SVM, as it requires only the solution to a convex linear problem, and not a quadratic problem as in the SVM. In this paper, we propose to conduct model selection for the LS-SVM using an empirical error criterion. Experiments on handwritten character recognition show the usefulness of this classifier and demonstrate that model selection improves the generalization performance of the LS-SVM.     

A constructive algorithm to synthesize arbitrarily connected feedforward neural networks

In this work we present a constructive algorithm capable of producing arbitrarily connected feedforward neural network architectures for classification problems. Architecture and synaptic weights of the neural network should be defined by the learning procedure. The main purpose is to obtain a parsimonious neural network, in the form of a hybrid and dedicate linear/nonlinear classification model, which can guide to high levels of performance in terms of generalization. Though not being a global optimization algorithm, nor a population-based metaheuristics, the constructive approach has mechanisms to avoid premature convergence, by mixing growing and pruning processes, and also by implementing a relaxation strategy for the learning error. The synaptic weights of the neural networks produced by the constructive mechanism are adjusted by a quasi-Newton method, and the decision to grow or prune the current network is based on a mutual information criterion. A set of benchmark experiments, including artificial and real datasets, indicates that the new proposal presents a favorable performance when compared with alternative approaches in the literature, such as traditional MLP, mixture of heterogeneous experts, cascade correlation networks and an evolutionary programming system, in terms of both classification accuracy and parsimony of the obtained classifier.