A new method of mining data streams using harmony search

Incremental learning has been used extensively for data stream classification. Most attention on the data stream classification paid on non-evolutionary methods. In this paper, we introduce new incremental learning algorithms based on harmony search. We first propose a new classification algorithm for the classification of batch data called harmony-based classifier and then give its incremental version for classification of data streams called incremental harmony-based classifier. Finally, we improve it to reduce its computational overhead in absence of drifts and increase its robustness in presence of noise. This improved version is called improved incremental harmony-based classifier. The proposed methods are evaluated on some real world and synthetic data sets. Experimental results show that the proposed batch classifier outperforms some batch classifiers and also the proposed incremental methods can effectively address the issues usually encountered in the data stream environments. Improved incremental harmony-based classifier has significantly better speed and accuracy on capturing concept drifts than the non-incremental harmony based method and its accuracy is comparable to non-evolutionary algorithms. The experimental results also show the robustness of improved incremental harmony-based classifier.     

保持局部邻域关系的增量Hessian LLE算法

Hessian LLE算法是一种经典的流形学习算法,但该方法是以批处理的方式进行的,当新的数据点加入时,必须重新运行整个算法,计算所有数据点低维嵌入,原来的运算结果被全部丢弃。鉴于此,提出了一种保持局部邻域关系的增量Hessian LLE(LIHLLE)算法,该方法通过保证流形新增样本点在原空间和嵌入空间局部邻域的线性关系不变,用其已有邻域点的低维坐标线性表示新增样本点,来得到新增点的低维嵌入,实现增量学习。在Swiss roll withhole和frey_rawface数据集上的实验表明,该方法简便、有效可行。     

基于局部平滑性的通用增量流形学习算法

目前大多数流形学习算法无法获取高维输入空间到低维嵌入空间的映射,无法处理新增数据,因此无增量学习能力。而已有的增量流形学习算法大多是通过扩展某一特定的流形学习算法使其具备增量学习能力,不具有通用性。针对这一问题,提出了一种通用的增量流形学习(GIML)算法。该方法充分考虑流形的局部平滑性这一本质特征,利用局部主成分分析法来提取数据集的局部平滑结构,并寻找包含新增样本点的局部平滑结构到对应训练数据的低维嵌入坐标的最佳变换。最后GIML算法利用该变换计算新增样本点的低维嵌入坐标。在人工数据集和实际图像数据集上进行了系统而广泛的比较实验,实验结果表明GIML算法是一种高效通用的增量流形学习方法,且相比当前主要的增量算法,能更精确地获取增量数据的低维嵌入坐标。     

An incremental approach to genetic-algorithms-based classification.

Incremental learning has been widely addressed in the machine learning literature to cope with learning tasks where the learning environment is ever changing or training samples become available over time. However, most research work explores incremental learning with statistical algorithms or neural networks, rather than evolutionary algorithms. The work in this paper employs genetic algorithms (GAs) as basic learning algorithms for incremental learning within one or more classifier agents in a multiagent environment. Four new approaches with different initialization schemes are proposed. They keep the old solutions and use an "integration" operation to integrate them with new elements to accommodate new attributes, while biased mutation and crossover operations are adopted to further evolve a reinforced solution. The simulation results on benchmark classification data sets show that the proposed approaches can deal with the arrival of new input attributes and integrate them with the original input space. It is also shown that the proposed approaches can be successfully used for incremental learning and improve classification rates as compared to the retraining GA. Possible applications for continuous incremental training and feature selection are also discussed.     

Incremental manifold learning by spectral embedding methods

Recent years have witnessed great success of manifold learning methods in understanding the structure of multidimensional patterns. However, most of these methods operate in a batch mode and cannot be effectively applied when data are collected sequentially. In this paper, we propose a general incremental learning framework, capable of dealing with one or more new samples each time, for the so-called spectral embedding methods. In the proposed framework, the incremental dimensionality reduction problem reduces to an incremental eigen-problem of matrices. Furthermore, we present, using this framework as a tool, an incremental version of Hessian eigenmaps, the IHLLE method. Finally, we show several experimental results on both synthetic and real world datasets, demonstrating the efficiency and accuracy of the proposed algorithm.     

An incremental ensemble of classifiers

Along with the increase of data and information, incremental learning ability turns out to be more and more important for machine learning approaches. The online algorithms try not to remember irrelevant information instead of synthesizing all available information (as opposed to classic batch learning algorithms). Today, combining classifiers is proposed as a new road for the improvement of the classification accuracy. However, most ensemble algorithms operate in batch mode. For this reason, we propose an incremental ensemble that combines five classifiers that can operate incrementally: the Naive Bayes, the Averaged One-Dependence Estimators (AODE), the 3-Nearest Neighbors, the Non-Nested Generalised Exemplars (NNGE) and the Kstar algorithms using the voting methodology. We performed a large-scale comparison of the proposed ensemble with other state-of-the-art algorithms on several datasets and the proposed method produce better accuracy in most cases.     

一种基于信念修正思想的SVR 增量学习算法

针对实际应用中数据的批量到达,以及系统的存储压力和学习效率低等问题,提出一种基于信念修正思想的SVR增量学习算法。首先从历史样本信息中提取信念集,根据信念集和新增数据的特点选择相应的信念集建立支持向量回归模型并进行预测;然后对信念集进行修正,调整当前认知状态,使该算法对在线和批处理增量学习都有很好的适应性。在标准数据集上的测试验证了算法的良好性能;在某机场噪声实测数据上的对比实验也表明,该算法的性能明显优于传统学习算法和一般增量学习算法。     

A combinational incremental ensemble of classifiers as a technique for predicting students’ performance in distance education

The ability to predict a student’s performance could be useful in a great number of different ways associated with university-level distance learning. Students’ marks in a few written assignments can constitute the training set for a supervised machine learning algorithm. Along with the explosive increase of data and information, incremental learning ability has become more and more important for machine learning approaches. The online algorithms try to forget irrelevant information instead of synthesizing all available information (as opposed to classic batch learning algorithms). Nowadays, combining classifiers is proposed as a new direction for the improvement of the classification accuracy. However, most ensemble algorithms operate in batch mode. Therefore a better proposal is an online ensemble of classifiers that combines an incremental version of Naive Bayes, the 1-NN and the WINNOW algorithms using the voting methodology. Among other significant conclusions it was found that the proposed algorithm is the most appropriate to be used for the construction of a software support tool.     

Multi-view Incremental Discriminant Analysis

In recent years, the use of multi-view data has attracted much attention resulting in many multi-view batch learning algorithms. However, these algorithms prove expensive in terms of training time and memory when used on the incremental data. In this paper, we propose Multi-view Incremental Discriminant Analysis (MvIDA), which updates the trained model to incorporate new data samples. MvIDA requires only the old model and newly added data to update the model. Depending on the nature of the increments, MvIDA is presented as two cases, sequential MvIDA and chunk MvIDA. We have compared the proposed method against the batch Multi-view Discriminant Analysis (MvDA) for its discriminability, order independence, the effect of the number of views, training time, and memory requirements. We have also compared our method with single-view Incremental Linear Discriminant Analysis (ILDA) for accuracy and training time. The experiments are conducted on four datasets with a wide range of dimensions per view. The results show that through order independence and faster construction of the optimal discriminant subspace, MvIDA addresses the issues faced by the batch multi-view algorithms in the incremental setting.     

Info-fuzzy algorithms for mining dynamic data streams

Most data-mining algorithms assume static behavior of the incoming data. In the real world, the situation is different and most continuously collected data streams are generated by dynamic processes, which may change over time, in some cases even drastically. The change in the underlying concept, also known as concept drift, causes the data-mining model generated from past examples to become less accurate and relevant for classifying the current data. Most online learning algorithms deal with concept drift by generating a new model every time a concept drift is detected. On one hand, this solution ensures accurate and relevant models at all times, thus implying an increase in the classification accuracy. On the other hand, this approach suffers from a major drawback, which is the high computational cost of generating new models. The problem is getting worse when a concept drift is detected more frequently and, hence, a compromise in terms of computational effort and accuracy is needed. This work describes a series of incremental algorithms that are shown empirically to produce more accurate classification models than the batch algorithms in the presence of a concept drift while being computationally cheaper than existing incremental methods. The proposed incremental algorithms are based on an advanced decision-tree learning methodology called “Info-Fuzzy Network” (IFN), which is capable to induce compact and accurate classification models. The algorithms are evaluated on real-world streams of traffic and intrusion-detection data.     

Incremental Learning of Linear Model Trees

A linear model tree is a decision tree with a linear functional model in each leaf. Previous model tree induction algorithms have been batch techniques that operate on the entire training set. However there are many situations when an incremental learner is advantageous. In this article a new batch model tree learner is described with two alternative splitting rules and a stopping rule. An incremental algorithm is then developed that has many similarities with the batch version but is able to process examples one at a time. An online pruning rule is also developed. The incremental training time for an example is shown to only depend on the height of the tree induced so far, and not on the number of previous examples. The algorithms are evaluated empirically on a number of standard datasets, a simple test function and three dynamic domains ranging from a simple pendulum to a complex 13 dimensional flight simulator. The new batch algorithm is compared with the most recent batch model tree algorithms and is seen to perform favourably overall. The new incremental model tree learner compares well with an alternative online function approximator. In addition it can sometimes perform almost as well as the batch model tree algorithms, highlighting the effectiveness of the incremental implementation. Editor: Johannes Fürnkranz     

An online incremental learning support vector machine for large-scale data

Support Vector Machines (SVMs) have gained outstanding generalization in many fields. However, standard SVM and most of modified SVMs are in essence batch learning, which make them unable to handle incremental learning or online learning well. Also, such SVMs are not able to handle large-scale data effectively because they are costly in terms of memory and computing consumption. In some situations, plenty of Support Vectors (SVs) are produced, which generally means a long testing time. In this paper, we propose an online incremental learning SVM for large data sets. The proposed method mainly consists of two components: the learning prototypes (LPs) and the learning Support Vectors (LSVs). LPs learn the prototypes and continuously adjust prototypes to the data concept. LSVs are to get a new SVM by combining learned prototypes with trained SVs. The proposed method has been compared with other popular SVM algorithms and experimental results demonstrate that the proposed algorithm is effective for incremental learning problems and large-scale problems.     

Evaluation of incremental learning algorithms for HMM in the recognition of alphanumeric characters

We present an evaluation of incremental learning algorithms for the estimation of hidden Markov model (HMM) parameters. The main goal is to investigate incremental learning algorithms that can provide as good performances as traditional batch learning techniques, but incorporating the advantages of incremental learning for designing complex pattern recognition systems. Experiments on handwritten characters have shown that a proposed variant of the ensemble training algorithm, employing ensembles of HMMs, can lead to very promising performances. Furthermore, the use of a validation dataset demonstrated that it is possible to reach better performances than the ones presented by batch learning.     

Increasing the accuracy of incremental naive bayes classifier using instance based learning

Along with the increase of data and information, incremental learning ability turns out to be more and more important for machine learning approaches. The online algorithms try not to remember irrelevant information instead of synthesizing all available information (as opposed to classic batch learning algorithms). In this study, we attempted to increase the prediction accuracy of an incremental version of Naive Bayes model by integrating instance based learning. We performed a large-scale comparison of the proposed method with other state-of-the-art algorithms on several datasets and the proposed method produce better accuracy in most cases.     

Incremental complete LDA for face recognition

The complete linear discriminant analysis (CLDA) algorithm has been proven to be an effective tool for face recognition. The CLDA method can make full use of the discriminant information of the training samples. However, the original implementation of CLDA may not suitable for incremental learning problem. In this paper, we first propose a new implementation of CLDA, which is theoretically equivalent to the original implementation of CLDA but is more efficient than the original one. Then, based on our proposed novel implementation of CLDA, we propose the incremental CLDA method which can accurately update the discriminant vectors of CLDA when new samples are inserted into the training set. Experiments on ORL, AR and PIE face databases show the efficiency of our proposed CLDA algorithms over the original implementation of CLDA.     

基于逆迭代的增量LLE算法

Locally Linear Embedding（LLE）算法是一种很好的流形学习算法,但是它只能以批处理的方式进行,只要有新的样本加入,就必须重作该算法的全部内容。而原来的运算结果被全部丢弃。提出了一种基于逆迭代的增量LLE算法,实现了流形的增量学习。在Swiss roll和S-curve数据库上的实验表明,该算法与LLE算法所计算出的投影值误差小于0.001%,运行的耗时少,具有很好的应用价值。     

Reinforcement Learning for Linear Continuous-time Systems: an Incremental Learning Approach

In this paper, we introduce a novel reinforcement learning (RL) scheme for linear continuous-time dynamical systems. Different from traditional batch learning algorithms, an incremental learning approach is developed, which provides a more efficient way to tackle the on-line learning problem in real-world applications. We provide concrete convergence and robust analysis on this incremental-learning algorithm. An extension to solving robust optimal control problems is also given. Two simulation examples are also given to illustrate the effectiveness of our theoretical result.      

FLEXFIS: A Robust Incremental Learning Approach for Evolving Takagi–Sugeno Fuzzy Models

In this paper, we introduce a new algorithm for incremental learning of a specific form of Takagi–Sugeno fuzzy systems proposed by Wang and Mendel in 1992. The new data-driven online learning approach includes not only the adaptation of linear parameters appearing in the rule consequents, but also the incremental learning of premise parameters appearing in the membership functions (fuzzy sets), together with a rule learning strategy in sample mode. A modified version of vector quantization is exploited for rule evolution and an incremental learning of the rules' premise parts. The modifications include an automatic generation of new clusters based on the nature, distribution, and quality of new data and an alternative strategy for selecting the winning cluster (rule) in each incremental learning step. Antecedent and consequent learning are connected in a stable manner, meaning that a convergence toward the optimal parameter set in the least-squares sense can be achieved. An evaluation and a comparison to conventional batch methods based on static and dynamic process models are presented for high-dimensional data recorded at engine test benches and at rolling mills. For the latter, the obtained data-driven fuzzy models are even compared with an analytical physical model. Furthermore, a comparison with other evolving fuzzy systems approaches is carried out based on nonlinear dynamic system identification tasks and a three-input nonlinear function approximation example.      

INCREMENTAL CONCEPT FORMATION ALGORITHMS BASED ON GALOIS (CONCEPT) LATTICES

The Galois (or concept) lattice produced from a binary relation has proved useful for many applications. Building the Galois lattice can be considered a conceptual clustering method because it results in a concept hierarchy. This article presents incremental algorithms for updating the Galois lattice and corresponding graph, resulting in an incremental concept formation method. Different strategies are considered based on a characterization of the modifications implied by such an update. Results of empirical tests are given in order to compare the performance of the incremental algorithms to three other batch algorithms. Surprisingly, when the total time for incremental generation is used, the simplest and less efficient variant of the incremental algorithms outperforms the batch algorithms in most cases. When only the incremental update time is used, the incremental algorithm outperforms all the batch algorithms. Empirical evidence shows that, on the average, the incremental update is done in time proportional to the number of instances previously treated. Although the worst case is exponential, when there is a fixed upper bound on the number of features related to an instance, which is usually the case in practical applications, the worst-case analysis of the algorithm also shows linear growth with respect to the number of instances.