基于数据分组匹配的相似重复记录检测

针对数据集成中相似重复记录的识别问题，提出一种数据特征属性优选分组的算法。通过计算特征属性的方差来确定某维属性的权值，基于数据分组思想选择权值大的属性，将数据集分割成不相交的小数据集，并在各小数据集中用模糊匹配算法进行相似重复记录的识别。理论分析和实验结果表明，该方法识别效率和检测精度较高。

Detection of Approximately Duplicated Records Based on Data Grouping Matching

Approximately duplicated records in multi-source data integration is one of the key factors affecting the data quality. A data grouping algorithm based on properties optimization of records is proposed in order to improve identification accuracy and detection efficiency. The method firstly calculates the variance of a property to determine the weight of the property, then chooses the property of larger weight to split the data sets into small data sets according to the thoughts on data grouping and duplicated records are identified based on the algorithm of fuzzy matching. Through theory analysis and experiments, it shows that identification accuracy and detection efficiency of the method are higher and it can effectively solve the problems of identification in approximately duplicate records of the data integration.