Skip-Gram模型融合词向量投影的微博新词发现

随着微博等社交网络的普及，新词源源不断涌现，分词系统经常将新词错误切分为单字.新词发现已经成为中文自然语言处理领域的研究热点.现有新词识别方法依赖大规模语料统计数据，对低频新词识别能力差.本文提出一种扩展Skip-gram模型和词向量投影方法，将两者结合后能缓解自然语言处理中常见的数据稀疏问题，有效识别低频新词，进而提高分词系统的准确率和召回率.

Microblog New Word Recognition Combining Skip-Gram Model and Word Vector Projection

With the popularity of microblog and other social networks, a steady stream of new words emerge, Chinese word segmentation systems often cut the new words into Chinese characters. The new word discovery has become a hot topic in the field of Chinese natural language processing. Existing new word recognition methods rely on the statistical data of large-scale corpus, the ability of new low-frequency word recognition is poor. This paper presents an extension of skip-gram model and word vector projection method, after the combination of the this two methods can ease the data sparseness problem effectively in natural language processing, to identify new low-frequency words, and to improve the precision and recall rate of Chinese word segmentation system.