曙光星云分布式文件系统:海量小文件存取

随着互联网应用的发展和云计算的兴起,在线图片、音频、视频以及微博等服务逐渐广泛发展,这些应用展示了与传统应用截然不同的数据访问和存储模式.数据中心内每秒钟都有大量较小文件的生成、分析和返回,这些应用对高并发海量文件的高吞吐、低延迟读写提出了新的挑战.提出基于分布式表存储的全新的分布式文件系统HVFS来管理数以十亿计的文件,并同时支持高吞吐和低延迟的文件访问.HVFS通过改进分布式可扩展哈希来管理元数据、日志结构的格式和列存储来利用时空局部性.本文描述了HVFS的设计和实现并进行了中等规模的实验.实验显示HVFS的表存储结构能够线性的扩展,并在82个结点上提供超过240,000次/秒、100,000次/秒的数据(<1KB)写和读;基于FUSE的实现在32个节点上提供超过180,000个/秒的文件创建速度.

Dawning Nebula Distributed File System HVFS: for Large Scale Small File Access

The emerging popular Internet applications such as online photo,audio,video and micro blogging services exhibit very different data access and storage requirements from traditional applications.Large number of small data are generated,analyzed,and returned every second in data center.These applications requires challenging performance for highly concurrent both high throughput and low latency access to tiny files.In this paper,we propose HVFS,a novel distributed file system built over distributed tabular storage,to manage billions of small files and support highly concurrent accesses.HVFS uses extendible hash to index metadata,log-structured storage format and columnar storage to exploit temporal and spatial locality.We present the design and implementation of HVFS.Our evaluation results demonstrate that the core tabular storage of HVFS can serve more than 100,000/240,000 aggregated data read/write requests per second on 82 nodes(<1KB),and FUSE implementation can serve more than 180,000 aggregated file creations per second on 32 nodes.