基于长度过滤和动态容错的SNM改进算法

数据仓库中相似重复记录的清洗对于数据质量影响很大，传统的基本邻近排序算法（Sorted-Neighborhood Method, SNM）时间效率和准确率均不高。针对SNM算法的缺陷，提出了一种基于长度过滤和动态容错的SNM改进算法。根据两条记录的长度比例和属性缺失情况，首先排除一部分不可能构成相似重复记录的数据，减少比较次数，提高检测效率；进一步提出了动态容错法，校准字段相似度评判结果，解决了因属性缺失而误判的问题，提高了准确率。针对实际数据集的实验分析表明，在相同的运算环境下，优化算法在准确率和时间效率上有明显优势。

An improved SNM algorithm based on length filtering and dynamic fault-tolerance

In data warehouse systems, cleaning similar and duplicated records can effectively impact data quality. Traditional SNM (Sorted-Neighborhood Method) has performance issues with time efficiency and accuracy rate. In order to improve its performance, an enhance SNM algorithm based on length filtering and dynamic fault-tolerance (LF-SNM) is proposed. Firstly, it improves the detection efficiency by excluding the records which are impossible to be duplicated according to the length proportion and attribute absence of two records. Then, it calibrates field similarity results using dynamic fault-tolerance method. It ensures accuracy even though some attributes are absent. Experimental results indicate that the LF-SNM performs obviously better than traditional SNM method on actual datasets under the same experimental conditions.