一种基于状态转换图的Ajax爬行算法

传统Web爬虫无法解决爬行Ajax应用所面临的JavaScript执行、状态识别与切换、重复状态检测等问题。为此，首先定义Ajax应用的状态转换图，并设计了一种基于状态转换图的Ajax爬行算法，通过该算法可以获取Ajax应用状态信息和后台Deep Web资源。为了提高Ajax爬行的准确性、减少待爬行的状态数目，使用Ajax指纹识别、DOM结构过滤等方法改进上述算法。实验结果表明了算法的有效性和性能。

Ajax crawling algorithm based on state transition graph

Traditional Web crawler could not meet the challenges of crawling Ajax application, such as JavaScript execution, state identification and navigation, duplicate states elimination etc.By exploring such challenges,this paper introduced state transition graph, based on which an algorithm was proposed to retrieve Ajax states and the background Deep Web. In order to uplift the accuracy,reduce the unnecessary states,improved the algorithm by Ajax fingerprinting and DOM filtering. The experimental results indicate the effectivity and efficiency of this algorithm.