中文网页分布式并行索引的设计与实现

Internet搜索引擎主要由搜索器、索引器和用户界面三部分组成。通过分析索引数据库在搜索引擎时效性及有效性方面的重要作用,提出一种多进程并行分词建立索引的方法。以中文网页数据库为基本语料库,采用正向最大匹配法进行中文分词,并用一种高效的倒排索引方式存储索引表。这种方法能够加快索引建立与更新的速度,并且在空间效率上也有较大的提高。

The design and implementation of distributed andparallel index of Chinese website

Internet search engine consists of three parts:searcher,index database and user interface.This paper first analyzes the importance of the indexer in real-time and effectiveness of the search engine.Then a new index approach is presented which carries out Chinese word segmentations by multiple processing concurrently.The maximum matching method is used to create index data of Chinese web pages and the inverted index table is carried to store index data.This method accelerates the speed of the indexing and up...