Algorithms for accelerated convergence of adaptive PCA

We derive and discuss adaptive algorithms for principal component analysis (PCA) that are shown to converge faster than the traditional PCA algorithms due to Oja and Karhunen (1985), Sanger (1989), and Xu (1993). It is well known that traditional PCA algorithms that are derived by using gradient descent on an objective function are slow to converge. Furthermore, the convergence of these algorithms depends on appropriate choices of the gain sequences. Since online applications demand faster convergence and an automatic selection of gains, we present new adaptive algorithms to solve these problems. We first present an unconstrained objective function, which can be minimized to obtain the principal components. We derive adaptive algorithms from this objective function by using: (1) gradient descent; (2) steepest descent; (3) conjugate direction; and (4) Newton-Raphson methods. Although gradient descent produces Xu's LMSER algorithm, the steepest descent, conjugate direction, and Newton-Raphson methods produce new adaptive algorithms for PCA. We also provide a discussion on the landscape of the objective function, and present a global convergence proof of the adaptive gradient descent PCA algorithm using stochastic approximation theory. Extensive experiments with stationary and nonstationary multidimensional Gaussian sequences show faster convergence of the new algorithms over the traditional gradient descent methods. We also compare the steepest descent adaptive algorithm with state-of-the-art methods on stationary and nonstationary sequences.     

AdaBoost算法研究进展与展望

AdaBoost是最优秀的Boosting算法之一, 有着坚实的理论基础, 在实践中得到了很好的推广和应用. 算法能够将比随机猜测略好的弱分类器提升为分类精度高的强分类器, 为学习算法的设计提供了新的思想和新的方法. 本文首先介绍Boosting猜想提出以及被证实的过程, 在此基础上, 引出AdaBoost算法的起源与最初设计思想;接着, 介绍AdaBoost算法训练误差与泛化误差分析方法, 解释了算法能够提高学习精度的原因;然后, 分析了AdaBoost算法的不同理论分析模型, 以及从这些模型衍生出的变种算法;之后, 介绍AdaBoost算法从二分类到多分类的推广. 同时, 介绍了AdaBoost及其变种算法在实际问题中的应用情况. 本文围绕AdaBoost及其变种算法来介绍在集成学习中有着重要地位的Boosting理论, 探讨Boosting理论研究的发展过程以及未来的研究方向, 为相关研究人员提供一些有用的线索. 最后,对今后研究进行了展望, 对于推导更紧致的泛化误差界、多分类问题中的弱分类器条件、更适合多分类问题的损失函数、 更精确的迭代停止条件、提高算法抗噪声能力以及从子分类器的多样性角度优化AdaBoost算法等问题值得进一步深入与完善.     

基于梯度优化的多维尺度节点定位算法

研究了分布式多维尺度分析技术在无线传感器网络节点定位中的应用,重点分析了其定位精度和收敛性能.根据传统的梯度优化算法,引入了最速下降算法作为目标函数的无约束优化方法.该算法采用最速下降法对节点的局部目标函数进行迭代优化.实验结果表明该优化算法比基于SMACOF算法的高斯核加权的dwMDS(G)算法在定位精度上有明显的提高, 并且算法简单,容易实现,是一种实用有效的无线传感器网络节点定位方法.     

Boosting Methods for Regression

In this paper we examine ensemble methods for regression that leverage or boost base regressors by iteratively calling them on modified samples. The most successful leveraging algorithm for classification is AdaBoost, an algorithm that requires only modest assumptions on the base learning method for its strong theoretical guarantees. We present several gradient descent leveraging algorithms for regression and prove AdaBoost-style bounds on their sample errors using intuitive assumptions on the base learners. We bound the complexity of the regression functions produced in order to derive PAC-style bounds on their generalization errors. Experiments validate our theoretical results.     

基于相关性的AdaBoost人脸检测算法

为提高传统AdaBoost算法的集成性能,降低算法复杂度,提出2种基于分类器相关性的AdaBoost算法。在弱分类器的训练过程中,加入Q统计量进行判定。每个弱分类器的权重更新不仅与当前分类器有关,而且需要考虑到前面的若干分类器,以有效降低弱分类器间的相似性,剔除相似特征。仿真结果表明,该算法具有更好的检测率,同时可降低误检率,改进分类器的整体性能。     

多标签代价敏感分类集成学习算法

尽管多标签分类问题可以转换成一般多分类问题解决,但多标签代价敏感分类问题却很难转换成多类代价敏感分类问题.通过对多分类代价敏感学习算法扩展为多标签代价敏感学习算法时遇到的一些问题进行分析,提出了一种多标签代价敏感分类集成学习算法.算法的平均错分代价为误检标签代价和漏检标签代价之和,算法的流程类似于自适应提升（Adaptive boosting,AdaBoost）算法,其可以自动学习多个弱分类器来组合成强分类器,强分类器的平均错分代价将随着弱分类器增加而逐渐降低.详细分析了多标签代价敏感分类集成学习算法和多类代价敏感AdaBoost算法的区别,包括输出标签的依据和错分代价的含义.不同于通常的多类代价敏感分类问题,多标签代价敏感分类问题的错分代价要受到一定的限制,详细分析并给出了具体的限制条件.简化该算法得到了一种多标签AdaBoost算法和一种多类代价敏感AdaBoost算法.理论分析和实验结果均表明提出的多标签代价敏感分类集成学习算法是有效的,该算法能实现平均错分代价的最小化.特别地,对于不同类错分代价相差较大的多分类问题,该算法的效果明显好于已有的多类代价敏感AdaBoost算法.     

Improvement of Boosting Algorithm by Modifying the Weighting Rule

AdaBoost is a method for improving the classification accuracy of a given learning algorithm by combining hypotheses created by the learning alogorithms. One of the drawbacks of AdaBoost is that it worsens its performance when training examples include noisy examples or exceptional examples, which are called hard examples. The phenomenon causes that AdaBoost assigns too high weights to hard examples. In this research, we introduce the thresholds into the weighting rule of AdaBoost in order to prevent weights from being assigned too high value. During learning process, we compare the upper bound of the classification error of our method with that of AdaBoost, and we set the thresholds such that the upper bound of our method can be superior to that of AdaBoost. Our method shows better performance than AdaBoost.     

一种限制输出模型规模的集成进化分类算法

AdaBoost算法是一种典型的集成学习框架,通过线性组合若干个弱分类器来构造成强学习器,其分类精度远高于单个弱分类器,具有很好的泛化误差和训练误差。然而AdaBoost 算法不能精简输出模型的弱分类器,因而不具备良好的可解释性。本文将遗传算法引入AdaBoost算法模型,提出了一种限制输出模型规模的集成进化分类算法(Ensemble evolve classification algorithm for controlling the size of final model,ECSM)。通过基因操作和评价函数能够在AdaBoost迭代框架下强制保留物种样本的多样性,并留下更好的分类器。实验结果表明,本文提出的算法与经典的AdaBoost算法相比,在基本保持分类精度的前提下,大大减少了分类器数量。     

基于函数变换的求解SAT问题的新算法

提出了一种基于函数变换的求解SAT问题的新算法,这个新算法利用SAT问题自身的特点将判定问题转化为连续函数的求极值问题。随机选取一组初始值,利用最速下降法求解变换后的连续函数在每个初始值邻域内所能达到的局部极值,如果这个局部极值为0,则该SAT问题就是可满足的。实验结果表明：与现有的求解SAT问题的算法相比,基于函数变换的求解算法在求解速度、成功率和求解问题的规模等方面都有明显的提高。     

Linear Programming Boosting via Column Generation

We examine linear program (LP) approaches to boosting and demonstrate their efficient solution using LPBoost, a column generation based simplex method. We formulate the problem as if all possible weak hypotheses had already been generated. The labels produced by the weak hypotheses become the new feature space of the problem. The boosting task becomes to construct a learning function in the label space that minimizes misclassification error and maximizes the soft margin. We prove that for classification, minimizing the 1-norm soft margin error function directly optimizes a generalization error bound. The equivalent linear program can be efficiently solved using column generation techniques developed for large-scale optimization problems. The resulting LPBoost algorithm can be used to solve any LP boosting formulation by iteratively optimizing the dual misclassification costs in a restricted LP and dynamically generating weak hypotheses to make new LP columns. We provide algorithms for soft margin classification, confidence-rated, and regression boosting problems. Unlike gradient boosting algorithms, which may converge in the limit only, LPBoost converges in a finite number of iterations to a global solution satisfying mathematically well-defined optimality conditions. The optimal solutions of LPBoost are very sparse in contrast with gradient based methods. Computationally, LPBoost is competitive in quality and computational cost to AdaBoost.     

Pattern recognition techniques for automatic detection of suspicious-looking anomalies in mammograms

We have employed two pattern recognition methods used commonly for face recognition in order to analyse digital mammograms. The methods are based on novel classification schemes, the AdaBoost and the support vector machines (SVM). A number of tests have been carried out to evaluate the accuracy of these two algorithms under different circumstances. Results for the AdaBoost classifier method are promising, especially for classifying mass-type lesions. In the best case the algorithm achieved accuracy of 76% for all lesion types and 90% for masses only. The SVM based algorithm did not perform as well. In order to achieve a higher accuracy for this method, we should choose image features that are better suited for analysing digital mammograms than the currently used ones.     

Improved Boosting Algorithms Using Confidence-rated Predictions

We describe several improvements to Freund and Schapire's AdaBoost boosting algorithm, particularly in a setting in which hypotheses may assign confidences to each of their predictions. We give a simplified analysis of AdaBoost in this setting, and we show how this analysis can be used to find improved parameter settings as well as a refined criterion for training weak hypotheses. We give a specific method for assigning confidences to the predictions of decision trees, a method closely related to one used by Quinlan. This method also suggests a technique for growing decision trees which turns out to be identical to one proposed by Kearns and Mansour. We focus next on how to apply the new boosting algorithms to multiclass classification problems, particularly to the multi-label case in which each example may belong to more than one class. We give two boosting methods for this problem, plus a third method based on output coding. One of these leads to a new method for handling the single-label case which is simpler but as effective as techniques suggested by Freund and Schapire. Finally, we give some experimental results comparing a few of the algorithms discussed in this paper.     

基于固定增量单样本感知器的AdaBoost算法

针对传统AdaBoost算法在分类过程中时间复杂度和算法学习复杂度较高的问题,提出一种改进的算法AdaBoostFISP。以固定增量单样本感知器为弱分类器,在感知器的权值更新上采用固定增量代替变量增量,从而减少运算时间、降低学习复杂度。实验结果证明了该算法在预测准确性、学习复杂度和时间复杂度等方面的优势。     

基于改进AdaBoost算法的复杂网络链路预测

链路预测是复杂网络的重要研究方向,当前的链路预测算法因可利用的网络信息有限,导致预测算法的精确度受限。为了提高预测算法的性能,采用改进的AdaBoost算法进行链路预测。首先根据复杂网络样本建立邻接矩阵,完成样本的矩阵化处理;然后采用AdaBoost算法进行分类训练,通过权重投票获取预测结果;最后,考虑到复杂网络弱分类器预测正负误差分布的不均衡问题,设置权重调整因子η及其调整范围[η1,η2],并根据η值动态调整AdaBoost算法的多个弱分类器分类结果的权重,从而获得准确的链路预测结果。实验结果证明,相比其他常用网络链路预测算法及传统AdaBoost算法,改进的AdaBoost算法的预测准确率优势明显,且在节点数量较多时,其预测时间性能和其他算法的差距较小。     

直线拟合算法

不管是平面直线拟合,还是空间直线拟合,直线拟合的应用范围都很广泛。文章对两种不同维度的直线拟合算法进行了综合介绍。其中空间直线拟合根据最佳平方逼近原理和最速下降法以及所给离散点的均值求得,并通过试验验证了此算法运算结果的正确性。该算法因为同时考虑了x、y、z不同方向的误差,所以准确度较高;同时因为采用了最速下降法,所以精确度可以任取,运算速度较快。     

Reformulated radial basis neural networks trained by gradientdescent

This paper presents an axiomatic approach for constructing radial basis function (RBF) neural networks. This approach results in a broad variety of admissible RBF models, including those employing Gaussian RBFs. The form of the RBFs is determined by a generator function. New RBF models can be developed according to the proposed approach by selecting generator functions other than exponential ones, which lead to Gaussian RBFs. This paper also proposes a supervised learning algorithm based on gradient descent for training reformulated RBF neural networks constructed using the proposed approach. A sensitivity analysis of the proposed algorithm relates the properties of RBFs with the convergence of gradient descent learning. Experiments involving a variety of reformulated RBF networks generated by linear and exponential generator functions indicate that gradient descent learning is simple, easily implementable, and produces RBF networks that perform considerably better than conventional RBF models trained by existing algorithms     

Random classification noise defeats all convex potential boosters

A broad class of boosting algorithms can be interpreted as performing coordinate-wise gradient descent to minimize some potential function of the margins of a data set. This class includes AdaBoost, LogitBoost, and other widely used and well-studied boosters. In this paper we show that for a broad class of convex potential functions, any such boosting algorithm is highly susceptible to random classification noise. We do this by showing that for any such booster and any nonzero random classification noise rate η, there is a simple data set of examples which is efficiently learnable by such a booster if there is no noise, but which cannot be learned to accuracy better than 1/2 if there is random classification noise at rate η. This holds even if the booster regularizes using early stopping or a bound on the L ₁ norm of the voting weights. This negative result is in contrast with known branching program based boosters which do not fall into the convex potential function framework and which can provably learn to high accuracy in the presence of random classification noise.     

通用集成学习算法的构造

集成学习算法的构造属于机器学习领域的重要研究内容,尽管弱学习定理指出了弱学习算法与强学习算法是等价的,但如何构造好的集成学习算法仍然是一个未得到很好解决的问题.Freund和Schapire提出的AdaBoost算法和Schapire和Singer提出的连续AdaBoost算法部分解决了该问题.提出了一种学习错误定义,以这种学习错误最小化为目标,提出了一种通用的集成学习算法,算法可以解决目前绝大多数分类需求的学习问题,如多分类、代价敏感分类、不平衡分类、多标签分类、模糊分类等问题,算法还对AdaBoost系列算法进行了统一和推广.从保证组合预测函数的泛化能力出发,提出了算法中的简单预测函数可统一基于样本的单个特征来构造.理论分析和实验结论均表明,提出的系列算法的学习错误可以任意小,同时又不用担心出现过学习现象.     

一种改进的AdaBoost算法在人脸检测系统中的应用

针对复杂背景条件下人脸检测的检测率低、速度慢的问题,提出了一种改进的AdaBoost算法,与遗传算法相结合,产生了一种识别率高、泛化能力好的强分类器,文中称之为GA-AdaBoost算法。该算法首先训练多个支持向量机作为弱分类器,然后用AdaBoost算法将多个弱分类器组合成一个强分类器,在组合的同时采用遗传算法对各弱分类器的权值进行全局寻优。最后,通过试验与传统AdaBoost进行对比,表明了该算法具有识别率高和速度快的优越性。     

关于AdaBoost有效性的分析

在机器学习领域,弱学习定理指明只要能够寻找到比随机猜测略好的弱学习算法,则可以通过一定方式,构造出任意误差精度的强学习算法.基于该理论下最常用的方法有AdaBoost和Bagging.AdaBoost和Bagging的误差分析还不统一;AdaBoost使用的训练误差并不是真正的训练误差,而是基于样本权值的一种误差,是否合理需要解释;确保AdaBoost有效的条件也需要有直观的解释以便使用.在调整Bagging错误率并采取加权投票法后,对AdaBoost和Bagging的算法流程和误差分析进行了统一,在基于大数定理对弱学习定理进行解释与证明基础之上,对AdaBoost的有效性进行了分析.指出AdaBoost采取的样本权值调整策略其目的是确保正确分类样本分布的均匀性,其使用的训练误差与真正的训练误差概率是相等的,并指出了为确保AdaBoost的有效性在训练弱学习算法时需要遵循的原则,不仅对AdaBoost的有效性进行了解释,还为构造新集成学习算法提供了方法.还仿照AdaBoost对Bagging的训练集选取策略提出了一些建议.