CrowdOLA: Online Aggregation on Duplicate Data Powered by Crowdsourcing |
| |
| Authors: | An-Zhen Zhang Jian-Zhong Li Hong Gao Yu-Biao Chen Heng-Zhao Ma Mohamed Jaward Bah |
| |
| Affiliation: | 1.School of Computer Science and Technology,Harbin Institute of Technology,Harbin,China |
| |
| Abstract: | Recently there is an increasing need for interactive human-driven analysis on large volumes of data. Online aggregation (OLA), which provides a quick sketch of massive data before a long wait of the final accurate query result, has drawn significant research attention. However, the direct processing of OLA on duplicate data will lead to incorrect query answers, since sampling from duplicate records leads to an over representation of the duplicate data in the sample. This violates the prerequisite of uniform distributions in most statistical theories. In this paper, we propose CrowdOLA, a novel framework for integrating online aggregation processing with deduplication. Instead of cleaning the whole dataset, CrowdOLA retrieves block-level samples continuously from the dataset, and employs a crowd-based entity resolution approach to detect duplicates in the sample in a pay-as-you-go fashion. After cleaning the sample, an unbiased estimator is provided to address the error bias that is introduced by the duplication. We evaluate CrowdOLA on both real-world and synthetic workloads. Experimental results show that CrowdOLA provides a good balance between efficiency and accuracy. |
| |
| Keywords: | |
| 本文献已被 SpringerLink 等数据库收录！ |
|
