Lazy Learning of Bayesian Rules

The naive Bayesian classifier provides a simple and effective approach to classifier learning, but its attribute independence assumption is often violated in the real world. A number of approaches have sought to alleviate this problem. A Bayesian tree learning algorithm builds a decision tree, and generates a local naive Bayesian classifier at each leaf. The tests leading to a leaf can alleviate attribute inter-dependencies for the local naive Bayesian classifier. However, Bayesian tree learning still suffers from the small disjunct problem of tree learning. While inferred Bayesian trees demonstrate low average prediction error rates, there is reason to believe that error rates will be higher for those leaves with few training examples. This paper proposes the application of lazy learning techniques to Bayesian tree induction and presents the resulting lazy Bayesian rule learning algorithm, called LBR. This algorithm can be justified by a variant of Bayes theorem which supports a weaker conditional attribute independence assumption than is required by naive Bayes. For each test example, it builds a most appropriate rule with a local naive Bayesian classifier as its consequent. It is demonstrated that the computational requirements of LBR are reasonable in a wide cross-section of natural domains. Experiments with these domains show that, on average, this new algorithm obtains lower error rates significantly more often than the reverse in comparison to a naive Bayesian classifier, C4.5, a Bayesian tree learning algorithm, a constructive Bayesian classifier that eliminates attributes and constructs new attributes using Cartesian products of existing nominal attributes, and a lazy decision tree learning algorithm. It also outperforms, although the result is not statistically significant, a selective naive Bayesian classifier.     

区间值属性决策树学习算法*

该文提出了一种区间值属性决策树的学习算法.区间值属性的值域不同于离散情况下的无序集和连续情况下的全序集,而是一种半序集.作为ID3算法在区间值意义下的推广,算法通过一种分割信息熵的极小化来选取扩展属性.通过非平稳点分析,减少了分割信息熵的计算次数,使算法的效率得到了提高.     

Induction of multiple fuzzy decision trees based on rough set technique

The integration of fuzzy sets and rough sets can lead to a hybrid soft-computing technique which has been applied successfully to many fields such as machine learning, pattern recognition and image processing. The key to this soft-computing technique is how to set up and make use of the fuzzy attribute reduct in fuzzy rough set theory. Given a fuzzy information system, we may find many fuzzy attribute reducts and each of them can have different contributions to decision-making. If only one of the fuzzy attribute reducts, which may be the most important one, is selected to induce decision rules, some useful information hidden in the other reducts for the decision-making will be losing unavoidably. To sufficiently make use of the information provided by every individual fuzzy attribute reduct in a fuzzy information system, this paper presents a novel induction of multiple fuzzy decision trees based on rough set technique. The induction consists of three stages. First several fuzzy attribute reducts are found by a similarity based approach, and then a fuzzy decision tree for each fuzzy attribute reduct is generated according to the fuzzy ID3 algorithm. The fuzzy integral is finally considered as a fusion tool to integrate the generated decision trees, which combines together all outputs of the multiple fuzzy decision trees and forms the final decision result. An illustration is given to show the proposed fusion scheme. A numerical experiment on real data indicates that the proposed multiple tree induction is superior to the single tree induction based on the individual reduct or on the entire feature set for learning problems with many attributes.     

一种新的基于属性—值对的决策树归纳算法

决策树归纳算法ID3是实例学习中具有代表性的学习方法。文中针对ID3易偏向于值数较多属性的缺陷,提出一种新的基于属性－值对的决策树归纳算法AVPI,它所产生的决策树大小及测试速度均优于ID3。该算法应用于色彩匹配系统,取得了较好效果。     

决策表分解及其最小属性约简研究

现有的很多属性约简算法都是由构造决策表的差别矩阵出发,将矩阵中非空元素的合取范式转化为极小析取范式。为提高对大规模数据的决策表进行约简的效率,文中指出基于U/{a}划分的最小约简算法存在的缺陷,给出以划分粒度为启发式信息,利用单个条件属性把论域划分成多个等价类,将计算整个全域上的属性约简问题转化为计算在相应划分的子区域上属性约简问题,提出了一种基于决策表分解的最小属性约简算法。理论分析和实例表明该约简算法是有效的。     

Cascade Generalization

Using multiple classifiers for increasing learning accuracy is an active research area. In this paper we present two related methods for merging classifiers. The first method, Cascade Generalization, couples classifiers loosely. It belongs to the family of stacking algorithms. The basic idea of Cascade Generalization is to use sequentially the set of classifiers, at each step performing an extension of the original data by the insertion of new attributes. The new attributes are derived from the probability class distribution given by a base classifier. This constructive step extends the representational language for the high level classifiers, relaxing their bias. The second method exploits tight coupling of classifiers, by applying Cascade Generalization locally. At each iteration of a divide and conquer algorithm, a reconstruction of the instance space occurs by the addition of new attributes. Each new attribute represents the probability that an example belongs to a class given by a base classifier. We have implemented three Local Generalization Algorithms. The first merges a linear discriminant with a decision tree, the second merges a naive Bayes with a decision tree, and the third merges a linear discriminant and a naive Bayes with a decision tree. All the algorithms show an increase of performance, when compared with the corresponding single models. Cascade also outperforms other methods for combining classifiers, like Stacked Generalization, and competes well against Boosting at statistically significant confidence levels.     

Set-valued ordered information systems

Set-valued ordered information systems can be classified into two categories: disjunctive and conjunctive systems. Through introducing two new dominance relations to set-valued information systems, we first introduce the conjunctive/disjunctive set-valued ordered information systems, and develop an approach to queuing problems for objects in presence of multiple attributes and criteria. Then, we present a dominance-based rough set approach for these two types of set-valued ordered information systems, which is mainly based on substitution of the indiscernibility relation by a dominance relation. Through the lower/upper approximation of a decision, some certain/possible decision rules from a so-called set-valued ordered decision table can be extracted. Finally, we present attribute reduction (also called criteria reduction in ordered information systems) approaches to these two types of ordered information systems and ordered decision tables, which can be used to simplify a set-valued ordered information system and find decision rules directly from a set-valued ordered decision table. These criteria reduction approaches can eliminate those criteria that are not essential from the viewpoint of the ordering of objects or decision rules.     

一种基于改进区分矩阵的属性约简算法

现有的很多约简算法都是由构造决策表的区分矩阵出发,将矩阵中非空元素的合取范式转化为极小析取范式。但是,基于Skowron提出的区分矩阵约简算法对不相容决策表会产生错误的结果。为此,提出一种改进的区分矩阵的定义,以及基于此区分矩阵的属性约简算法,该算法对相容或不相容决策表都是适用的,特别对不相容决策表会得到更加稀疏的区分矩阵,可大大节省计算时间和存储空间,该算法是一种简单、有效、普遍适用的求解属性约简方法。     

Classification of Multivariate Time Series and Structured Data Using Constructive Induction

We present a method of constructive induction aimed at learning tasks involving multivariate time series data. Using metafeatures, the scope of attribute-value learning is expanded to domains with instances that have some kind of recurring substructure, such as strokes in handwriting recognition, or local maxima in time series data. The types of substructures are defined by the user, but are extracted automatically and are used to construct attributes.Metafeatures are applied to two real domains: sign language recognition and ECG classification. Using metafeatures we are able to generate classifiers that are either comprehensible or accurate, producing results that are comparable to hand-crafted preprocessing and comparable to human experts.     

Id+: Enhancing medical knowledge acquisition with machine learning

Learning from patient records may aid medical knowledge acquisition and decision making. Decision tree induction, based on ID3, is a well-known approach of learning from examples. In this article we introduce a new data representation formalism that extends the original ID3 algorithm. We propose a new algorithm, ID+, which adopts this representation scheme. ID+ provides the capability of modeling dependencies between attributes or attribute values and of handling multiple values per attribute. We demonstrate our work via a series of medical knowledge acquisition experiments that are based on a ''real-world'' application of acute abdominal pain in children. In the context of these experiments, we compare ID+ with C4.5, NewId, and a Naive Bayesian classifier. Results demonstrate that the rules acquired via ID+ improve decision tree clinical comprehensibility and complement explanations supported by the Naive Bayesian classifier, while in terms of classification, accuracy decrease is marginal.     

Strategic induction of decision trees

An algorithm for decision-tree induction is presented in which attribute selection is based on the evidence-gathering strategies used by doctors in sequential diagnosis. Since the attribute selected by the algorithm at a given node is often the best attribute according to the Quinlan's information gain criterion, the decision tree it induces is often identical to the ID3 tree when the number of attributes is small. In problem-solving applications of the induced decision tree, an advantage of the approach is that the relevance of a selected attribute or test can be explained in strategic terms. An implementation of the algorithm in an environment providing integrated support for incremental learning, problem solving and explanation is presented.     

Evolving Fuzzy Min–Max Neural Network Based Decision Trees for Data Stream Classification

Learning from data streams is a challenging task which demands a learning algorithm with several high quality features. In addition to space complexity and speed requirements needed for processing the huge volume of data which arrives at high speed, the learning algorithm must have a good balance between stability and plasticity. This paper presents a new approach to induce incremental decision trees on streaming data. In this approach, the internal nodes contain trainable split tests. In contrast with traditional decision trees in which a single attribute is selected as the split test, each internal node of the proposed approach contains a trainable function based on multiple attributes, which not only provides the flexibility needed in the stream context, but also improves stability. Based on this approach, we propose evolving fuzzy min–max decision tree (EFMMDT) learning algorithm in which each internal node of the decision tree contains an evolving fuzzy min–max neural network. EFMMDT splits the instance space non-linearly based on multiple attributes which results in much smaller and shallower decision trees. The extensive experiments reveal that the proposed algorithm achieves much better precision in comparison with the state-of-the-art decision tree learning algorithms on the benchmark data streams, especially in the presence of concept drift.     

基于邻域粒化的混合信息系统动态规则提取

现有的混合信息系统知识发现模型涵盖的数据类型大多为符号型、数值型条件属性及符号型决策属性,且大多数模型的关注点是属性约简或特征选择,针对规则提取的研究相对较少。针对涵盖更多数据类型的混合信息系统构建一个动态规则提取模型。首先修正了现有的属性值距离的计算公式,对错层型属性值的距离给出了一种定义形式,从而定义了一个新的混合距离。其次提出了针对数值型决策属性诱导决策类的3种方法。其后构造了广义邻域粗糙集模型,提出了动态粒度下的上下近似及规则提取算法,构建了基于邻域粒化的动态规则提取模型。该模型可用于具有以下特点的信息系统的规则提取： （1）条件属性集可包括单层符号型、错层符号型、数值型、区间型、集值型、未知型等; （2）决策属性集可包括符号型、数值型。利用UCI数据库中的数据集进行了对比实验,分类精度表明了规则提取算法的有效性。     

An incremental decision tree algorithm based on rough sets and its application in intrusion detection

As we know, learning in real world is interactive, incremental and dynamical in multiple dimensions, where new data could be appeared at anytime from anywhere and of any type. Therefore, incremental learning is of more and more importance in real world data mining scenarios. Decision trees, due to their characteristics, have been widely used for incremental learning. In this paper, we propose a novel incremental decision tree algorithm based on rough set theory. To improve the computation efficiency of our algorithm, when a new instance arrives, according to the given decision tree adaptation strategies, the algorithm will only modify some existing leaf node in the currently active decision tree or add a new leaf node to the tree, which can avoid the high time complexity of the traditional incremental methods for rebuilding decision trees too many times. Moreover, the rough set based attribute reduction method is used to filter out the redundant attributes from the original set of attributes. And we adopt the two basic notions of rough sets: significance of attributes and dependency of attributes, as the heuristic information for the selection of splitting attributes. Finally, we apply the proposed algorithm to intrusion detection. The experimental results demonstrate that our algorithm can provide competitive solutions to incremental learning.     

Multivariate Decision Trees

Unlike a univariate decision tree, a multivariate decision tree is not restricted to splits of the instance space that are orthogonal to the features' axes. This article addresses several issues for constructing multivariate decision trees: representing a multivariate test, including symbolic and numeric features, learning the coefficients of a multivariate test, selecting the features to include in a test, and pruning of multivariate decision trees. We present several new methods for forming multivariate decision trees and compare them with several well-known methods. We compare the different methods across a variety of learning tasks, in order to assess each method's ability to find concise, accurate decision trees. The results demonstrate that some multivariate methods are in general more effective than others (in the context of our experimental assumptions). In addition, the experiments confirm that allowing multivariate tests generally improves the accuracy of the resulting decision tree over a univariate tree.     

基于辨识矩阵的不完备多粒度约简

面向具有缺失属性值的不完备数据,文中从辨识矩阵的角度构造不完备信息系统和不完备决策系统的多粒度约简结构.首先,讨论基于悲观和乐观多粒度近似的不完备信息系统的约简性质,构造不完备信息系统和不完备决策系统的3种多粒度辨识矩阵.然后,理论性证明通过对构造的辨识矩阵进行析取、合取逻辑运算,可精确得到不完备信息系统和不完备决策系统的所有多粒度近似约简.最后通过实例验证文中多粒度约简方法的有效性和实用性.     

基于相关性的类偏好敏感决策树算法

针对决策者在面对几个分类结果时会有选择其中某一个结果的倾向性这一事实,提出了一种基于相关性的类偏好敏感决策树分类算法（CPSDT）。该算法引入了类偏好度、偏好代价矩阵等概念。为弥补在传统决策树构造过程中,选择分裂属性时未考虑非类属性之间相关性的不足,该算法在进行学习之前先采用基于相关性的特征预筛选排除属性冗余并重新构造了基于相关性的属性选择因子。经实验证明,该算法能够有效减小决策树规模,且能够在实现对偏好类的高精度预测的同时保证决策树拥有较好的整体精度。