基于规则与统计相结合的中文文本自动查错模型与算法

中文文本自动校对是自然语言处理领域具有挑战性的研究课题。本文提出了一种规则与统计相结合的中文文本自动查错模型与算法。根据正确文本分词后单字词的出现规律以及“非多字词错误”的概念,提出一组错误发现规则,并与针对分词后单字散串建立的字二元、三元统计模型和词性二元、三元统计模型相结合,建立了文本自动查错模型与实现算法。通过对30篇含有578个错误测试点的文本进行实验,所提算法的查错召回率为86.85%、准确率为69.43% ,误报率为30.57%。

A Hybrid Model of Combining Rule-based and Statistics-based Approaches for Automatic Detecting Errors in Chinese Text

Chinese text automatic proofreading is an important research subject in NLP.A hybrid model based on the combination of rules and statistics are proposed in this article.According to the distribution of Chinese single-character after word segmentation in Chinese text and the conception of "non-multi-character word error",we proposed a group of rules to find errors in texts,to construct the automatic error-detection model and to implement its algorithm by combining the scattered single-character Bigram models,part-of-speech Bigram and Trigram models.Our experiment for the 30 texts that contain 578 error test points shows that the recall rate is 86.85% and accuracy rate is 69.43%,distorting rate is 30.57%.