基于一致性度量属性约简的决策树构建优化研究

针对样本中有无关的、冗余的属性会降低决策树算法的分类精度,本文提出基于一致性度量属性约简后构建决策树的方法。对UCI机器学习数据库中5个两类分类样本离散化后,分别基于粗糙集和一致性度量的属性约简来构建C45和CART决策树,实验表明基于一致性度量属性约简构建的决策树有较高的精度和可行性。     

On fuzzy-rough attribute selection: Criteria of Max-Dependency,Max-Relevance,Min-Redundancy,and Max-Significance

Attribute selection is one of the important problems encountered in pattern recognition, machine learning, data mining, and bioinformatics. It refers to the problem of selecting those input attributes or features that are most effective to predict the sample categories. In this regard, rough set theory has been shown to be successful for selecting relevant and nonredundant attributes from a given data set. However, the classical rough sets are unable to handle real valued noisy features. This problem can be addressed by the fuzzy-rough sets, which are the generalization of classical rough sets. A feature selection method is presented here based on fuzzy-rough sets by maximizing both relevance and significance of the selected features. This paper also presents different feature evaluation criteria such as dependency, relevance, redundancy, and significance for attribute selection task using fuzzy-rough sets. The performance of different rough set models is compared with that of some existing feature evaluation indices based on the predictive accuracy of nearest neighbor rule, support vector machine, and decision tree. The effectiveness of the fuzzy-rough set based attribute selection method, along with a comparison with existing feature evaluation indices and different rough set models, is demonstrated on a set of benchmark and microarray gene expression data sets.     

Genetic Programming with a Genetic Algorithm for Feature Construction and Selection

The use of machine learning techniques to automatically analyse data for information is becoming increasingly widespread. In this paper we primarily examine the use of Genetic Programming and a Genetic Algorithm to pre-process data before it is classified using the C4.5 decision tree learning algorithm. Genetic Programming is used to construct new features from those available in the data, a potentially significant process for data mining since it gives consideration to hidden relationships between features. A Genetic Algorithm is used to determine which such features are the most predictive. Using ten well-known datasets we show that our approach, in comparison to C4.5 alone, provides marked improvement in a number of cases. We then examine its use with other well-known machine learning techniques.     

Induction of multiple fuzzy decision trees based on rough set technique

The integration of fuzzy sets and rough sets can lead to a hybrid soft-computing technique which has been applied successfully to many fields such as machine learning, pattern recognition and image processing. The key to this soft-computing technique is how to set up and make use of the fuzzy attribute reduct in fuzzy rough set theory. Given a fuzzy information system, we may find many fuzzy attribute reducts and each of them can have different contributions to decision-making. If only one of the fuzzy attribute reducts, which may be the most important one, is selected to induce decision rules, some useful information hidden in the other reducts for the decision-making will be losing unavoidably. To sufficiently make use of the information provided by every individual fuzzy attribute reduct in a fuzzy information system, this paper presents a novel induction of multiple fuzzy decision trees based on rough set technique. The induction consists of three stages. First several fuzzy attribute reducts are found by a similarity based approach, and then a fuzzy decision tree for each fuzzy attribute reduct is generated according to the fuzzy ID3 algorithm. The fuzzy integral is finally considered as a fusion tool to integrate the generated decision trees, which combines together all outputs of the multiple fuzzy decision trees and forms the final decision result. An illustration is given to show the proposed fusion scheme. A numerical experiment on real data indicates that the proposed multiple tree induction is superior to the single tree induction based on the individual reduct or on the entire feature set for learning problems with many attributes.     

一种基于粗糙集的网页分类方法

Internet的迅速发展带来了一个新的问题，如何有效，迅速地从浩瀚的Web网页中找到所需要的信息，机器学习的发展给这个问题的解决提供了一个新的方向，本文将粗糙集理论应用于网页分类，提出了一种基于粗糙集的决策表约简的增量式学习算法，并利用该算法实现了一个Web网页的分类器，实验结果表明该分类器具有良好的性能。     

基于粗糙集与属性值聚类的决策树改进算法

采用粗糙集理论和属性值聚类相结合的方法,从决策树最优化的三个原则对其进行优化。首先,采用粗糙集理论的约简功能求出相对核,并利用信息熵作为启发信息求出相对约简,以此来保证生成决策树的路径最短和减少决策树的节点数。其次,在选择特征属性时,在信息熵增益最大的前提下,根据属性值间的相异性距离来对属性值聚类使其能够接近单峰分布。通过对UCI数据实验分析,结果表明很大程度上减少了决策树的节点数和决策树的深度。     

粗糙集理论及其应用综述

粗糙集理论是一种较新的软计算方法,可以有效地分析和处理不完备信息,该理论近日益受到国际学术届的重视,已经在模式识别,机器学习,决策支持,过程控制,预测建模等许多科学与工程领域得到成功的应用,本文介绍了粗糙集理论的基本概念,对其在各领域的应用情况进行了综述。     

Semantics-preserving dimensionality reduction: rough and fuzzy-rough-based approaches

Semantics-preserving dimensionality reduction refers to the problem of selecting those input features that are most predictive of a given outcome; a problem encountered in many areas such as machine learning, pattern recognition, and signal processing. This has found successful application in tasks that involve data sets containing huge numbers of features (in the order of tens of thousands), which would be impossible to process further. Recent examples include text processing and Web content classification. One of the many successful applications of rough set theory has been to this feature selection area. This paper reviews those techniques that preserve the underlying semantics of the data, using crisp and fuzzy rough set-based methodologies. Several approaches to feature selection based on rough set theory are experimentally compared. Additionally, a new area in feature selection, feature grouping, is highlighted and a rough set-based feature grouping technique is detailed.     

Mining extremely small data sets with application to software reuse

A serious problem encountered by machine learning and data mining techniques in software engineering is the lack of sufficient data. For example, there are only 24 examples in the current largest data set on software reuse. In this paper, a recently proposed machine learning algorithm is modified for mining extremely small data sets. This algorithm works in a twice‐learning style. In detail, a random forest is trained from the original data set at first. Then, virtual examples are generated from the random forest and used to train a single decision tree. In contrast to the numerous discrepancies between the empirical data and expert opinions reported by previous research, our mining practice shows that the empirical data are actually consistent with expert opinions. Copyright © 2008 John Wiley & Sons, Ltd.     

Estimating performance gains for voted decision trees

Decision tree induction is a prominent learning method, typically yielding quick results with competitive predictive performance. However, it is not unusual to find other automated learning methods that exceed the predictive performance of a decision tree on the same application. To achieve near-optimal classification results, resampling techniques can be employed to generate multiple decision-tree solutions. These decision trees are individually applied and their answers voted. The potential for exceptionally strong performance is counterbalanced by the substantial increase in computing time to induce many decision trees. We describe estimators of predictive performance for voted decision trees induced from bootstrap (bagged) or adaptive (boosted) resampling. The estimates are found by examining the performance of a single tree and its pruned subtrees over a single, training set and a large test set. Using publicly available collections of data, we show that these estimates are usually quite accurate, with occasional weaker estimates. The great advantage of these estimates is that they reveal the predictive potential of voted decision trees prior to applying expensive computational procedures.     

Machine Learning for Intelligent Processing of Printed Documents

A paper document processing system is an information system component which transforms information on printed or handwritten documents into a computer-revisable form. In intelligent systems for paper document processing this information capture process is based on knowledge of the specific layout and logical structures of the documents. This article proposes the application of machine learning techniques to acquire the specific knowledge required by an intelligent document processing system, named WISDOM++, that manages printed documents, such as letters and journals. Knowledge is represented by means of decision trees and first-order rules automatically generated from a set of training documents. In particular, an incremental decision tree learning system is applied for the acquisition of decision trees used for the classification of segmented blocks, while a first-order learning system is applied for the induction of rules used for the layout-based classification and understanding of documents. Issues concerning the incremental induction of decision trees and the handling of both numeric and symbolic data in first-order rule learning are discussed, and the validity of the proposed solutions is empirically evaluated by processing a set of real printed documents.     

Acquisition of hierarchy-structured probabilistic decision tables and rules from data

Abstract: The paper is concerned with the creation of predictive models from data within the framework of the variable precision rough set model. It is focused on two aspects of the model derivation: computation of uncertain, in general, rules from information contained in probabilistic decision tables and forming hierarchies of decision tables with the objective of reduction or elimination of decision boundaries in the resulting classifiers. A new technique of creation of a linearly structured hierarchy of decision tables is introduced and compared to tree‐structured hierarchy. It is argued that the linearly structured hierarchy has significant advantages over tree‐structured hierarchy.     

An incremental decision tree algorithm based on rough sets and its application in intrusion detection

As we know, learning in real world is interactive, incremental and dynamical in multiple dimensions, where new data could be appeared at anytime from anywhere and of any type. Therefore, incremental learning is of more and more importance in real world data mining scenarios. Decision trees, due to their characteristics, have been widely used for incremental learning. In this paper, we propose a novel incremental decision tree algorithm based on rough set theory. To improve the computation efficiency of our algorithm, when a new instance arrives, according to the given decision tree adaptation strategies, the algorithm will only modify some existing leaf node in the currently active decision tree or add a new leaf node to the tree, which can avoid the high time complexity of the traditional incremental methods for rebuilding decision trees too many times. Moreover, the rough set based attribute reduction method is used to filter out the redundant attributes from the original set of attributes. And we adopt the two basic notions of rough sets: significance of attributes and dependency of attributes, as the heuristic information for the selection of splitting attributes. Finally, we apply the proposed algorithm to intrusion detection. The experimental results demonstrate that our algorithm can provide competitive solutions to incremental learning.     

A weighted rough set based method developed for class imbalance learning

In this paper, we introduce weights into Pawlak rough set model to balance the class distribution of a data set and develop a weighted rough set based method to deal with the class imbalance problem. In order to develop the weighted rough set based method, we design first a weighted attribute reduction algorithm by introducing and extending Guiasu weighted entropy to measure the significance of an attribute, then a weighted rule extraction algorithm by introducing a weighted heuristic strategy into LEM2 algorithm, and finally a weighted decision algorithm by introducing several weighted factors to evaluate extracted rules. Furthermore, in order to estimate the performance of the developed method, we compare the weighted rough set based method with several popular methods used for class imbalance learning by conducting experiments with twenty UCI data sets. Comparative studies indicate that in terms of AUC and minority class accuracy, the weighted rough set based method is better than the re-sampling and filtering based methods, and is comparable to the decision tree and SVM based methods. It is therefore concluded that the weighted rough set based method is effective for class imbalance learning.     

基于核属性的知识获取算法

Rough集理论是近年来发展起来的一种有效地处理不精确、不确定、含糊信息的数学理论方法,在机器学习、数据挖掘、智能数据分析、控制算法获取等领域取得了很大的成功.决策表是Rough Set理论的处理对象,用RoughSet对决策表进行规则提取通常有代数观和信息观两种主要理论和方法.使用哪一种方法提取的规则集更好是很多研究者的目标.本文针对Rough Set理论的核心内容之一的知识获取进行了研究,提出了一种基于属性重要性排序的知识获取算法,并且证明了在不相容系统中使用信息观方法比使用代数观的方法更好,能够提取更合理的规则集.     

决策树算法及其核心技术

决策树是归纳学习和数据挖掘的重要方法,通常用来形成分类器和预测模型。概述了决策树分类算法,指出了决策树算法的核心技术:测试属性的选择和树枝修剪技术。通过对当前数据挖掘中具有代表性的优秀分类算法进行分析和比较,总结出了各种算法的特性,为使用者选择算法或研究者改进算法提供了依据。最后,通过一个实例说明决策树分类在实际生产中的应用。     

基于改进的代价敏感决策树的网络贷款分类

在网络贷款用户数据集中，贷款成功和贷款失败的用户数量存在着严重的不平衡，传统的机器学习算法在解决该类问题时注重整体分类正确率，导致贷款成功用户的预测精度较低。针对此问题，在代价敏感决策树敏感函数的计算中加入类分布，以减弱正负样本数量对误分类代价的影响，构建改进的代价敏感决策树；以该决策树作为基分类器并以分类准确度作为衡量标准选择表现较好的基分类器，将它们与最后阶段生成的分类器集成得到最终的分类器。实验结果表明，与已有的常用于解决此类问题的算法（如MetaCost算法、代价敏感决策树、AdaCost算法等）相比，改进的代价敏感决策树对网络贷款用户分类可以降低总体的误分类错误率，具有更强的泛化能力。     

A new weighted rough set framework based classification for Egyptian NeoNatal Jaundice

Prediction of diseases would help physicians to make informal decision regarding the type of treatment. Jaundice is the most common condition that requires medical attention in newborn babies. Although most newborns develop some degree of jaundice, a high level bilirubin puts a newborn at risk of bilirubin encephalopathy and kernicterus, which are rare but still occur in Egypt. This paper presents a new weighted rough set framework for early intervention and prevention of neurological dysfunction and kernicterus that are catastrophic sequels of neonatal jaundice. The obtained results illustrate that the weighted rough set can provide significantly more accurate and reliable predictive accuracy than well known algorithms such as weighted SVM and decision tree considering the fact that physicians do not have any estimation about probability of jaundice appearance.     

Three-way decision reduction in neighborhood systems

Rough set reduction has been used as an important preprocessing tool for pattern recognition, machine learning and data mining. As the classical Pawlak rough sets can just be used to evaluate categorical features, a neighborhood rough set model is introduced to deal with numerical data sets. Three-way decision theory proposed by Yao comes from Pawlak rough sets and probability rough sets for trading off different types of classification error in order to obtain a minimum cost ternary classifier. In this paper, we discuss reduction questions based on three-way decisions and neighborhood rough sets. First, the three-way decision reducts of positive region preservation, boundary region preservation and negative region preservation are introduced into the neighborhood rough set model. Second, three condition entropy measures are constructed based on three-way decision regions by considering variants of neighborhood classes. The monotonic principles of entropy measures are proved, from which we can obtain the heuristic reduction algorithms in neighborhood systems. Finally, the experimental results show that the three-way decision reduction approaches are effective feature selection techniques for addressing numerical data sets.     

一种粗糙集-决策树结合的入侵检测方法

针对入侵检测系统收集数据海量、高维、检测模型复杂和检测准确率低等问题,采用粗糙集属性约简的优势寻找与判断入侵与否相关的属性,利用决策树分类算法生成模型并对网络连接进行入侵预测分类检测,从而提出了一种粗糙集属性约简和决策树预测分类相结合的网络入侵检测方法.实验结果表明,该方法在入侵检测准确率上有很大的提高,对DoS攻击、Probe攻击和R2L攻击的检测效果均有所提高,同时大大降低了检测的误报率.