基于Nutch的分布式文件系统研究

成功的搜索引擎都有稳定的分布式文件系统,采用分布式计算的方式,使得网页数据、索引文件都分散存储在网络中,检索从分散的数据中通过分布式计算获取。通过分析分布式搜索引擎,研究了Google的分布式文件系统GoogleFS以及Map Reduce算法,针对Nutch的特性,提出了FoundClear算法和一种NDFS的解决方案,最后与GoogleFS系统作了比较。

Research of Distributed File System on Nutch

Distributed file systems were designed by successful search engines providers,and adopt distributing computing,which let Web pages' data and indexing files spread in the network,searching comes from distributing computing.This article analyzes distributing search engines,the Google's distributed file system and MapReduce arithmetic are researched,put forward another arithmetic named FoundClear and a settled project of distributed file system on Nutch,and lastly comparison with GoogleFS system.