基于词汇聚类方法的现代汉语分期与分期体系构建

当前对现代汉语史的研究多借用政治-社会史的分期方式将现代汉语分为新文化运动至1949年、1950—1966年、1967—1976年和1977—至今四个时期,并在这一基础上开展了许多研究。语言尤其是书面语虽然与社会政治生活有密切联系,但语言系统有其自身的演化规律。从语言数据出发对语言进行分期是更加合适的选择。该文将语言的分期问题视作历时语料的分期问题,进而成为历时文本的聚类问题。该文工作基于历时报刊语料库遴选出的时间敏感程度较好的词汇。使用机器学习领域中广泛使用的K均值和期望最大算法进行聚类,以该部分词汇频率为特征对70年跨度(1945—2015)的历时报刊语料进行聚类,并在不同的聚类数量下绘制了具有层次性的词汇使用分期树。据此构建了过去70年现代汉语的词汇层次分期模型,揭示了改革开放的开始作为词汇使用变迁最重要分水岭的地位。

Lexicon Clustering Based Modern Chinese Staging

State-of-the-art research tend to divide modern Chinese into 4 periods according to the political history: new culture movement to 1949, 1950—1966, 1967—1976, and 1977 till now. Though written language is deeply influenced by the social and political movements, language evolve by its own pattern, and language staging should be based on language data.
In this paper, we regards the langage staging as a text classification problem. Based on the time sensitive words and its frequency as features, K-means and EM algorithm are applied to cluster the corpus of 70 years of “People’s Daily”. Hierarchical staging scheme is formed and shown in divding tree, revealing the beginning of Reform and Open Policy as divide crest of written language use in the past century.