| 两步解码式空间矢量数据并行转换算法 |
| |
| 引用本文： | 孙乐乐,金宝轩. 两步解码式空间矢量数据并行转换算法[J]. 浙江大学学报(工学版), 2020, 54(9): 1768-1776. DOI: 10.3785/j.issn.1008-973X.2020.09.013 |
| |
| 作者姓名： | 孙乐乐 金宝轩 |
| |
| 作者单位： | 1. 云南师范大学 旅游与地理科学学院，云南 昆明 6505002. 云南省自然资源厅，云南 昆明 650224 |
| |
| 基金项目： | 国家自然科学基金资助项目（41661086） |
| |
| 摘 要： | 传统单机转换工具与基于范围分区方案的并行转换算法存在扩展性差、数据倾斜的问题，为此提出两步解码式空间矢量数据（SVD）并行转换算法. 通过归纳地理空间数据库（GDB）中空间矢量数据的存储编码模式，构建优化后的几何解码函数作为基础工具. 初次解码：仅解析空间元数据，根据几何复杂度平衡解析任务，提高解析与数据量的均衡度；二次解码：借助几何并行解析机制提取、解析压缩几何字节，提高转换效率. 该算法基于Spark实现，将其与ArcGIS单机转换工具、基于范围分区方案的并行查询转换算法进行对比可知，所提算法具有显著的效率、性能扩展优势，转换效率提升了2.5~117倍，大幅降低了几何复杂度不均导致的数据倾斜情况.
|
| 关 键 词： | 地理信息系统 空间矢量数据（SVD） 数据并行转换 数据倾斜 |
Spatial vector data parallel conversion algorithm based on two-step decoding |
| |
| Le-le SUN,Bao-xuan JIN. Spatial vector data parallel conversion algorithm based on two-step decoding[J]. Journal of Zhejiang University(Engineering Science), 2020, 54(9): 1768-1776. DOI: 10.3785/j.issn.1008-973X.2020.09.013 |
| |
| Authors: | Le-le SUN Bao-xuan JIN |
| |
| Abstract: | In view of the poor scalability and data skew in traditional single-machine conversion tools and RangePartitioner-based parallel methods, A spatial vector data (SVD) parallel conversion was proposed based on two-step decoding. An optimized geometry-parsing algorithm was introduced as a basic decoding tool with the storage schema of SVD in geospatial database (GDB). Only the spatial metadata was parsed in the first-step decoding, and the task was balanced according to the set geometry complexity to improve the balance between parsing and data. In the later-step decoding, the compressed geometry bytes were extracted and parsed with the geometric parallel parsing mechanism, to improve the conversion efficiency. This algorithm was implemented on Apache Spark, which was compared with ArcGIS conversion tool and the RangePartitioner-based parallel query transform algorithm. The experimental results verify that the proposed algorithm has significant advantages in efficiency and performance expansion; the conversion efficiency is promoted by 2.5?117 times; and the data skew caused by uneven geometric complexity is greatly reduced. |
| |
| Keywords: | geographic information system spatial vector data (SVD) data parallel conversion data skew |
| 本文献已被 CNKI 等数据库收录！ |
| 点击此处可从《浙江大学学报(工学版)》浏览原始摘要信息 |
|
点击此处可从《浙江大学学报(工学版)》下载全文 |
