基于扩展树状知识库的海量数据清洗算法

针对传统知识库表示的局限性，通过分解和重组领域知识，建立扩展树状结构的知识库，其中叶结点对应具体知识实例，称为原子知识，非叶结点只对应知识概念。同时提出相关的数据清洗算法，根据用户的选择，自动提取原子知识进行分析，消除重复，按照处理权重建立原子知识序列，然后逐一对数据进行清洗。实验表明，该算法能有效优化用户的请求，减少对海量数据的遍历次数，海量数据的清洗效率明显提高。

Mass data cleaning algorithm based on extended tree-like knowledge base

By analyzing the limitation of traditional structures of knowledge base,an extended tree-like knowledge base is built by decomposing and recomposing the domain knowledge.The leaf node of the tree is linked with the knowledge instance called atomic knowledge and the non-leaf node is linked with the concept of knowledge.Based on the knowledge base,a data cleaning algorithm is proposed.It extracts atomic knowledge of the selected nodes firstly,then analyzes their relations,deletes the same objects,builds an atomic knowledge sequence based on weights,lastly cleans data according to the sequence.The experiment shows that the count of scaning mass data can be reduced rapidly by adopting the algorithm to optimize the users＇requests and the data cleaning efficiency can be improved clearly.