pRPL + pGTIOL: The marriage of a parallel processing library and a parallel I/O library for big raster data |
| |
| Affiliation: | 1. Informatics and Knowledge Management Graduate Program, Universidade Nove de Julho, São Paulo, Brazil;2. Department of Computer Science, Institute of Mathematics and Statistics, Universidade de São Paulo, São Paulo, Brazil;1. Department of Information Engineering, Electronics and Telecommunications, Sapienza University of Rome, Via Eudossiana 18, 00184 Rome, Italy;2. CIMA Research Foundation, Via A. Magliotto 2, 1700 Savona, Italy;3. Serco SpA., Via Sciadonna 24/26, 00044 Frascati, Rome, Italy;4. ESA-ESRIN, Via Galileo Galilei, Casella Postale 00044 Frascati, Rome, Italy;5. European Centre for Medium-range Weather Forecasts (ECMWF), Reading, UK;1. Rubenstein School of the Environment and Natural Resources, University of Vermont, Burlington, VT 05405, USA;2. Department of Geology, University of Vermont, Burlington, VT 05405, USA;3. Department of Geosciences, Skidmore College, Saratoga Springs, NY 12866, USA;4. Department of Earth Science and Engineering, Imperial College London, South Kensington Campus, London SW7 2AZ, UK.;5. Lawerence Livermore National Laboratory, Livermore, CA 94550, USA |
| |
| Abstract: | Data I/O has become a major bottleneck of computational performance of geospatial analysis and modeling. In this study, a parallel GeoTIFF I/O library (pGTIOL) was developed. Through the storage mapping and data arrangement techniques, pGTIOL can operate on files in either strip or tile storage mode, read/write any sub-domain of data within the raster dataset. pGTIOL enables asynchronized I/O, which means a process can read/write its own sub-domains of data when necessary without synchronizing with other processes. pGTIOL was integrated into the parallel raster processing library (pRPL). Several pGTIOL-based data I/O functions and options were added to pRPL, while the existing functions of pRPL stay intact. Experiments showed that the integration of pRPL and pGTIOL achieved higher performance than the original pRPL that uses GDAL as the I/O interface. Therefore, pRPL + pGTIOL enables transparent parallelism for high-performance raster processing with the capability of true parallel I/O of massive raster datasets. |
| |
| Keywords: | Raster Parallel computing Parallel I/O Programming library |
| 本文献已被 ScienceDirect 等数据库收录！ |
|
