一种改进的GloVe词向量表示学习方法

GloVe模型是一种广泛使用的词向量表示学习的模型。许多研究发现,学习得到的词向量维数越大,性能越好;但维数越大,模型学习耗时越长。事实上,GloVe模型中,耗时主要表现在两方面,一是统计词对共现矩阵,二是训练学习词向量表示。该文在利用GloVe模型统计语料中词对共现时,基于对称或非对称窗口得到两个共现矩阵,然后分别学习得到较低维度的词向量表示,再拼接得到较高维度的词向量表示。从计算的复杂度来看,该文方法并不会产生多的计算量,但显然统计共现矩阵和训练学习可通过并行方式实现,能够显著提高计算效率。在使用大规模语料的实验中,以对称和非对称窗口分别统计得到共现矩阵,分别学习得到300维词向量表示,再使用拼接方式得到600维词向量表示。与GloVe模型对称和非对称的600维的词向量相比,在中文和英文的词语推断任务上,显著地提高了预测的准确率,在词语聚类任务上,有较好的聚类效果,验证了该文方法的有效性。

An Improved Learning Method for GloVe Word Vector Representation

GloVe model is a popular word vector model, which is revealed a better performance with the increase of word vector dimensions. To avoid intolerable time cost in training high-dimension word vector, we propose an improved method of GloVe which is easily implemented in a parallel manner. We first construct a co-occurrence matrix with symmetrical windows and a co-occurrence matrix with asymmetrical windows on a corpus respectively, and apply the original GloVe model for two low-dimension word vectors. Then we concatenate the word vectors as the final high-dimension vectors. Tested in large-scale corpus, the 600-dimension word vector trained by the proposed method achieves better performance in the Chinese and English word analogy task and word clustering task, compared with the 600-dimension word vector trained directly by GloVe model.