基于动态可配置规则的数据清洗方法

针对传统数据清洗方法通过硬编码方法来实现业务逻辑而导致系统的可重用性、可扩展性与灵活性较差等问题，提出了一种基于动态可配置规则的数据清洗方法——DRDCM。该方法支持多种类型规则间的复杂逻辑运算，并支持多种脏数据修复行为，集数据检测、数据修复与数据转换于一体，具有跨领域、可重用、可配置、可扩展等特点。首先，对DRDCM方法中的数据检测和数据修复的概念、实现步骤以及实现算法进行描述；其次，阐述了DRDCM方法中支持的多种规则类型以及规则配置；最后，对DRDCM方法进行实现，并通过实际项目数据集验证了该实现系统在脏数据修复中，丢弃修复行为具有很高的准确率，尤其是对需遵守法定编码规则的属性（例如身份证号码）处理时其准确率可达100%。实验结果表明，DRDCM实现系统可以将动态可配置规则无缝集成于多个数据源和多种不同应用领域且该系统的性能并不会随着规则条数增加而极速降低，这也进一步验证了DRDCM方法在真实环境中的切实可行性。

Data cleaning method based on dynamic configurable rules

Traditional data cleaning approaches usually implement cleaning rules specified by business requirements through hard-coding mechanism, which leads to well-known issues in terms of reusability, scalability and flexibility. In order to address these issues, a new Dynamic Rule-based Data Cleaning Method (DRDCM) was proposed, which supports the complex logic operation between various types of rules and three kinds of dirty data repair behavior. It integrates data detection, error correction and data transformation in one system and contributes several unique characteristics, including domain-independence, reusability and configurability. Besides, the formal concepts and terms regarding data detection and correction were defined, while necessary procedures and algorithms were also introduced. Specially, the supported multiple rule types and rule configurations in DRDCM were presented in detail. At last, the DRDCM approach was implemented. Experimental results show that the implemented system provides a high accuracy on the discarded behavior of dirty data repair with real-life data sets. Especially for the attribute required to comply with the statutory coding rules (such as ID card number), whose accuracy can reach 100%. Moreover, these results also indicate that this reference implementation of DRDCM can successfully support multiple data sources in cross-domain scenarios, and its performance does not sharply decrease with the increase of the number of rules. These results further validate that the proposed DRDCM is practical in real-world scenarios.