| 基于MapReduce的术语权重计算方法研究 |
| |
| 引用本文： | 王锴,施水才,王涛,吕学强.基于MapReduce的术语权重计算方法研究[J].电信科学,2011,27(11):62-65. |
| |
| 作者姓名： | 王锴 施水才 王涛 吕学强 |
| |
| 作者单位： | 1. 北京信息科技大学中文信息处理研究中心 北京100101
2. 北京信息科技大学中文信息处理研究中心 北京100101;北京拓尔思信息技术股份有限公司 北京100101 |
| |
| 基金项目： | 国家自然科学基金资助项目，北京市自然科学基金资助项目，北京市教委科技发展计划资助项目 |
| |
| 摘 要： | 术语识别在本体构建、词典构建等领域应用广泛,而术语权重计算是术语识别中的关键步骤。本文通过改进TF-IDF公式,将组成术语词条的长度作为权重因素之一,同时考虑术语在文档集中的领域相关性。整个过程基于MapReduce编程模型实现,在Hadoop云平台中以分布式方式计算候选领域术语的权重。实验结果表明,该方法不仅简化了术语权重计算的实施步骤,也提高了算法执行效率。
|
| 关 键 词： | 术语权重 TF-IDF MapReduce 分布式 |
Research on Term Weighting Based on MapReduce |
| |
| Wang Kai,Shi Shuicai,Wang Tao,Lv Xueqiang.Research on Term Weighting Based on MapReduce[J].Telecommunications Science,2011,27(11):62-65. |
| |
| Authors: | Wang Kai Shi Shuicai Wang Tao Lv Xueqiang |
| |
| Affiliation: | 1,2(1.Beijing Information Science and Technology University,Chinese Information Processing Research Center, Beijing 100101,China;2.Beijing TRS Information Technology Co.,Ltd.,Beijing 100101,China) |
| |
| Abstract: | Term recognition is widely used in the ontology construction,dictionary construction and other fields.And term weighting is a key step in the term recognition.In this paper,several improvements have been made to TF-IDF algorithm,e.g.,the length of terms is considered in weighting,also with terms’ correlations to documentation set.The candidate term weight is calculated in a distributed manner based on MapReduce on Hadoop.Experimental results show that the method proposed not only simplifies the steps of term weighting,but also improves the efficiency of the algorithm. |
| |
| Keywords: | term weight TF-IDF MapReduce distributed |
| 本文献已被 CNKI 万方数据 等数据库收录！ |
|
