基于蚁群特征选择的相似重复记录分类检测

为实现相似重复记录的检测,提出一种基于蚁群算法特征选择的分类检测方法。将相似重复记录检测看成二分类问题,定义了字符串型、枚举型和日期型3种典型属性类型的相似特征和归一化算法,以两记录的相似特征向量作为分类器的输入进行检测;建立了以召回率、准确率和特征规模综合最优的特征选择多目标优化模型,并根据问题特点将多目标模型转化为单目标模型,应用蚁群算法设计了模型求解算法。最后,用欧氏距离分类法和支持向量机2种分类器验证了该方法的有效性。

Classification Detection of Approximately Duplicate Records Based on Feature Selection Using Ant Colony Algorithm

In order to realize detection of approximately duplicate records, a classification method based on feature selection using ant colony algorithm was proposed. The detection of approximately duplicate records can be regarded as a classification problem of two classes. The similar feature and normalization algorithm of three typical attribute types, such as character string, enumeration and date, were defined. Taken the similar feature vectors of two records as the classifier's input, the detection of approximately duplicated records was realized. A multi-object optimization model based on comprehensive optimization of recall rate, precision and feature set's size was set up. According to the characteristics of the problem, the multi-object optimization model was transfered into a single object optimization model. A solving algorithm using ant colony algorithm was designed. Finally, the method was validated by Euclidean distance classifier and support vector machine classifier.