|
|||||
|
|
| 多语种网络文本快速新词抽取 | |
| 引用本文： | 刘冰洋,刘 倩,张 瑾,刘欣然,程学旗.多语种网络文本快速新词抽取[J].中文信息学报,2014,28(2):78-84. |
| 作者姓名： | 刘冰洋 刘 倩 张 瑾 刘欣然 程学旗 |
| 作者单位： | 1. 中国科学院计算技术研究所网络数据科学与工程研究中心,北京 100190; 2. 中国科学院大学,北京 100190; 3. 国家计算机网络应急技术处理协调中心,北京 100029 |
| 基金项目： | 国家自然科学基金(609933005,60903139)、国家242专项(2011A001, 2012G129)、973项目(2013CB329601) |
| 摘 要： | 从网络文本中提取新词是网络信息处理中的一个重要问题,在信息检索、文本挖掘、词典编纂、中文分词等领域中都有重要应用。本文提出了一种与语言无关的快速新词提取算法,首先针对后缀树的数据结构将多语言文本进行统一编码,然后使用改进的统计方法在双后缀树上以线性时间统计重复串与邻接类别,并计算字符串的整体度,同时通过剪枝大幅度减少计算量,在中、英文语料上较好地实现了新词的抽取及排序。 |
| 关 键 词： | 新词 邻接类别 字符串整体度 后缀树 多语言 |
Fast New Words Extraction from Multi-lingual Web Texts |
|
| LIU Bingyang,LIU Qian,ZHANG Jin,LIU Xinran,CHENG Xueqi.Fast New Words Extraction from Multi-lingual Web Texts[J].Journal of Chinese Information Processing,2014,28(2):78-84. | |
| Authors: | LIU Bingyang LIU Qian ZHANG Jin LIU Xinran CHENG Xueqi |
| Affiliation: | 1. Research Center of Web Data & Engineering, Institute of Computing Technology, Beijing 100190, China; 2. University of Chinese Academy of Sciences, Beijing 100190, China; 3. National Computer Network Emergency Response Technical Team Coordination Center, Beijing 100029, China |
| Abstract: | Extracting new words from web texts is one key problem in the area of information processing with direct application in information retrieval, public opinion, dictionary compilation, Chinese word segmentation and other fields. A language-independent method is implemented to fast extract new words from web texts:Encoding multi-lingual texts into a uniform binary stream, extracting repeat strings, calculating the adjacency variety and string integrity measurement. Two suffix trees in 4-bit based structureare used to calculate these statistics in linear time. This method outputs new words and their order on both Chinese and English web texts. |
| Keywords: | new words adjacency variety string integrity measurement suffix tree multi-lingual |
| 本文献已被 CNKI 等数据库收录！ | |
| 点击此处可从《中文信息学报》浏览原始摘要信息 | |
| 点击此处可从《中文信息学报》下载全文 | |