基于N-Gram算法的数据清洗技术

针对数据库中存在的大量相似重复数据,对相似重复记录的属性结构以及产生原因进行了分析,采用N-Gram算法对数据记录进行计算,得到代表每条记录属性的键值,即N-Gram值.依据该键值将数据库中的数据记录进行排序处理,建立有序的数据库,并对其中的数据记录进行相似度计算.运用排列合并的清洗思想对识别出来的相似重复数据记录进行清洗,实验结果表明,N-Gram算法有效提高了相似重复数据记录的查全率和查准率.

Data cleaning technology based on N-Gram algorithm

Aiming at the plentiful approximately duplicate data in the database, the attribute structure of approximately duplicate records and the causing reason were analyzed. The data records were calculated with the N-Gram algorithm to get the key values, namely N-Gram values, which represented the attribute of every record. According to the key values, the data records in the database were ordered so as to form a well-organized database. In addition, the similarity of data records in the database was calculated. The identified approximately duplicate records were cleaned by applying the arranged combination cleaning idea. The experimental results show that the N-Gram algorithm effectively increases the recall ratio and precision ratio of approximately duplicate data records.