基于Spark的交互式数据预处理系统

高质量的决策依赖于高质量的数据，数据预处理是数据挖掘至关重要的环节.传统的数据预处理系统并不能很好的适用于大数据环境，企业现阶段主要使用Hadoop/Hive对海量数据进行预处理，但普遍存在耗时长、效率低、无交互等问题.提出了一种基于Spark的交互式数据预处理系统，系统提供一套通用的数据预处理组件，并支持组件的扩展，数据以电子表格的形式展现，系统记录用户的处理过程并支持撤销重做.本文从数据模型、数据预处理操作、交互式执行引擎以及交互式前端四个方面描述了系统架构.最后使用医疗脑卒中的真实数据对系统进行验证，实验结果表明，系统能够在大数据场景下满足交互式处理需求.

Interactive Data Preprocessing System Based on Spark

The high quality decision-making depends on high quality data, hence data preprocessing is an essential phase for data analytics applications. In the big data area, traditional data preprocessing systems cannot be directly applied. To handle the large-scale data, enterprises adopt Hadoop/Hive as a popular solution at the present stage. However, it brings many defects, such as poor performance, the lack of interaction and so on. To fill this gap, this paper proposes and implements an interactive data preprocessing system based on Spark. This system provides a series of common preprocessing logics as basic components and supports flexible user-defined extensions. To get an interactive interface, the system presents data to users in the form of spreadsheets, while it can automatically records users operations to provide undo and redo support. In this paper, we introduce the architecture of this system with four aspects:data model, data preprocessing operations, interactive execution engine and interactive GUI. In the end, we conduct experiments with real stroke data and the result shows that the system can meet interactive demands in most big data scenarios.