基于潜在语义分析的BBS文档Bayes鉴别器

电子公告栏(BBS)的滥用是一种以信息污染为特色的社会问题，对BBS文档进行鉴别已成为信息安全重要内容之一，该文融合了数据挖掘技术、数理统计技术和自然语言理解技术，提出了基于潜在语义分析与Bayes分类的BBS文档鉴别方法：利用自然语言处理技术从训练文档中抽取典型短语集；通过潜在语义分析进行典型短语同义归约，应用关联规则采掘技术提高典型短语间的独立性；用Bayes分类器对BBS文档进行鉴别。该文还对影响系统的关键参数进行了大量的讨论和测试，实验表明该方法对于BBS文档的鉴别是可行而有效的。

Bayes Discriminator for BBS Documents Based on Latent Semantic Analysis

With the rapid development of Internet, the abuse and misuse of BBS become a social problem of information pollution and call on the demand to the discrimination techniques for BBS document. Borrowing the techniques from data mining, probability-statistics and Natural Language Understanding, this paper proposes a new discrimination method for BBS document, called Bayes Discrimination based on Latent Semantic Analysis(BDLSA). The main steps of the new method includes following steps: (1)Makes typical phrase set by extracting the typical sentences from training documents in preprocessing stage with natural language understanding techniques.(2)Applies synonymy reduction on typical phrases by Latent Semantic Analysis.(3)Discovers the association rules between typical phrases to increase the independency of phrases so that the traditional Bayes discriminator works efficiently.(4)Discriminates BBS document by Bayes classifier. The algorithms to construct typical phrase set and to reduce synonymy are proposed and implemented. The experiment is based on real document form Web, with training data of 583 documents and test-data of 308 documents, the correctness is up to 75%. This shows the effetiveness and validation of the new method.