基于突发事件热度的站点地图构建算法

为确保及时准确地获取监测网站的突发事件网络舆情数据,提出了一种基于突发事件热度的站点地图构建算法。该算法利用突发事件主题词典和改进Shark search算法采集样本网页,在此基础上对目标网站的超链接结构进行数据挖掘,完整构建出含有网站各版块突发事件热度的站点地图。以该站点地图为指导的网页采集器能够及时调整更新频率,准确采集所需网页,较好地适应监测网站的动态变化。实验表明,在站点地图的指导下,突发事件相关网页的采集有效性和效率均有明显提高。

Method of generalized sitemaps with emergency hot degree

In order to get the data about the emergency Web public opinion from the target sites in time, this paper proposed a method to generalize the sitemap based on the emergency hot degree. Utilizing emergency topic dictionary and advanced Shark search algorithm, the method collected enough sample Web pages to produce a sitemap containing the emergency hot degree of every board that was related to emergency in the target Web site. Under the guidance of the sitemaps generalized by this method, the Web clawer was intelligent enough to adapt well to the dynamical changes of the target sites, collected the needed Web page precisely and adjusted its update frequency when necessary. Experiments show that the Web clawer produces an outstanding performance both in the effectiveness and the efficiency with the help of the sitemap.