优化的Web日志会话识别方法

关于Web使用挖掘中的会话识别问题,提出了一种改进的会话识别方法.统计每个页面的访问时间,并结合页面内容及站点结构生成页面访问时间阀值,再用页面访问时间阁值划分会话,然后对产生的会话集使用会话重组,即合并与断开的方法,对生成的会话集进行进一步地优化.通过实验结果表明,该方法所得到的会话集更具有真实性.     

一种基于MFSP-DG的个性化推荐算法

提出一种基于最大频繁序列模式有向图的页面个性化推荐技术,由于考虑了用户会话的页面访问顺序,比一些不考虑页面访问顺序的推荐技术有更高的准确率。有向图结构压缩存储了所有最大频繁序列模式,推荐引擎依据截取的用户最近访问页面子序列,与有向图的部分路径进行匹配并进行横向推荐和纵向推荐,无需在整个模式库中搜索相同或相似的模式,从而加快了模式匹配的速度,更好地满足了页面推荐的特性和实时要求。实验证明,方法是有效的。     

网络日志预处理中优化的会话识别算法

研究网络日志预处理中会话识别问题,会话识别是网络日志数据预处理中最蘑要的一个环节.为使用户准确快速地找到需要的资源,传统网络日志预处理方法采用固定阈值会话识别算法,不能适合网络日志的动态性和不能很好消除网络日志中的冗余信息,导致后继网络日志数据挖掘效率和挖掘精度低.为更好消除网络日志冗余信息,提高后继数据挖掘的效率和精度,提出一种改进的网络日志预处理会话识别算法.新算法可根据页面内容、站点结构确定页面重要程度,对阈值进行动态调整,克服传统固定阈值缺陷,根据用户对页面内容的兴趣度删除不感兴趣页面,消除冗余信息,并对该算法进行了仿真.结果表明,相对于传统网络预测的会话识别算法,新算法能更准确地确定页面访问时间阈值,消除了网络日志冗余信息,提高了网络日志预处理效率和数据挖掘精度.     

基于分类方法的Web站点实时个性化推荐

提出一种新的基于分类方法的实时个性化推荐方法，该文首先根据用户访问事务文法生成序列访问事务集，用于得到每个用户访问的序列特性并且便于分类器进行分类，然后利用该事务集训练一个多类分类器，作者通过推荐引擎得到每个用户的当前访问序列和用户当前请求页面，然后把该序列送入分类器进行分类，以得到用户的下面一些可能访问的页面，这些推荐页面的地址被附加到用户当前请求的页面的底部由推荐引擎返回以进行推荐，在这种方法中，用户不需要注册信息，推荐不打扰用户，可以为用户提供实时个性化的服务，实验表明这种方法是成功的。     

Web日志会话的个性化识别方法的研究

会话识别是Web日志挖掘中的重要步骤。针对目前的各种会话识别方法,提出了一种改进的基于页面内容、下载时间等多个参数综合得到的针对每个用户的个性化识别方法。该方法通过使用访问时间间隔,判断是否在极大、极小两个阈值范围内来识别会话。根据页面内容、站点结构确定页面重要程度,通过页面的信息容量确定用户正常的阅读时间,通过Web日志中页面下载时间来确定起始阅读时间,对以上因素进行综合后对该阈值进行调整。实验结果表明,相对于目前的对所有用户页面使用单一先验阈值进行会话识别的方法及使用针对用户页面的阈值动态调整方法,提出的方法能更准确地个性化确定出页面访问时间阈值,更为合理有效。     

Web日志挖掘中数据预处理技术的研究

数据预处理在Web日志挖掘过程中起着至关重要的作用,直接影响日志挖掘的质量和结果。详细分析了数据预处理的过程,提出一种改进的数据清洗方法,以提高日志挖掘中数据预处理的效率,并针对Web日志数据预处理中会话识别这一重要环节,提出一种改进的会话识别方法。在用户识别后,根据页面内容、站点结构确定页面重要程度,对阈值进行调整。然后,根据用户对页面内容的兴趣度来删除会话中的链接页面和不感兴趣的页面。实验结果表明,提出的方法能更准确地确定页面访问时间阈值,得到更为合理有效的会话集合。     

Web日志挖掘中数据预处理技术的研究

数据预处理在Web日志挖掘过程中起着至关重要的作用,直接影响日志挖掘的质量和结果.详细分析了数据预处理的过程,提出一种改进的数据清洗方法,以提高日志挖掘中数据预处理的效率,并针对Web日志数据预处理中会话识别这一重要环节,提出一种改进的会话识别方法.在用户识别后,根据页面内容、站点结构确定页面重要程度,对阈值进行调整.然后,根据用户对页面内容的兴趣度来删除会话中的链接页面和不感兴趣的页面.实验结果表明,提出的方法能更准确地确定页面访问时间阈值,得到更为合理有效的会话集合.     

Web日志预处理中优化的会话识别方法

针对Web日志数据预处理中会话识别这一重要环节,提出一种优化的会话识别算法。在用户识别后,通过过滤框架页面大幅度减少实验产生的有效页面数,为每个页面设置访问时间阈值,并根据页面重要程度对该阈值进行调整,页面的重要性由页面内容及站点结构确定。实验数据显示,与对所有页面使用单一的先验阈值进行会话识别的方法相比较,该方法得到了真实性更强的会话集。     

基于URL结构和访问时间的Web页面访问相似性度量

Web页面相似性是Web页面聚类和Web会话聚类的基础，其准确性直接影响聚类的质量。本文在分析基于URL结构的静态Web页面相似性度量的基础上，提出了将URL结构和页面访问时间结合起来度量Web页面对之间的相似性并给出了度量标准。实验证明，采用我们提出的度量标准得到的结果准确性更高，更接近于用户浏览兴趣。     

一种改进的Web日志会话识别方法

会话识别是Web日志挖掘中的数据预处理中的一个重要步骤。文中提出了一种改进的会话识别方法。首先,在用户识别后,进行框架页面的过滤,从而大大地减少了实验产生的有效页面,然后为页面设置访问时间阙值,并根据页面内容及站点结构确定的页面重要程度对该阈值进行调整。通过实验证明,相对于传统的对所有页面使用单一的先验阈值进行会话识别的方法,该方法所得到的会话集更具有真实性。     

EFFICIENT METHODOLOGIES TO HANDLE HANGING PAGES USING VIRTUAL NODE

In this article we first explain the knowledge extraction (KE) process from the World Wide Web (WWW) using search engines. Then we explore the PageRank algorithm of Google search engine (a well-known link-based search engine) with its hidden Markov analysis. We also explore one of the problems of link-based ranking algorithms called hanging pages or dangling pages (pages without any forward links). The presence of these pages affects the ranking of Web pages. Some of the hanging pages may contain important information that cannot be neglected by the search engine during ranking. We propose methodologies to handle the hanging pages and compare the methodologies. We also introduce the TrustRank algorithm (an algorithm to handle the spamming problems in link-based search engines) and include it in our proposed methods so that our methods can combat Web spam. We implemented the PageRank algorithm and TrustRank algorithm and modified those algorithms to implement our proposed methodologies.     

主题爬虫研究进展

随着互联网技术的飞速发展,网页数量急剧增加,搜索引擎的地位已经不可取代,成为人们使用Internet的入口。网络蜘蛛作为搜索引擎的信息来源是搜索引擎必不可少的组成部分。介绍网络蜘蛛设计中的关键技术。另外,随着用户个性化需求越来越强以及网页数量的急剧增加导致通用搜索引擎无法满足特定用户的需求,专业搜索引擎得到快速的发展。同时对于主题爬虫的研究也有很大的突破和进展。主题爬虫有别于通用爬虫,通用爬虫注重爬取的完整性,而主题爬虫强调网页与特定主题的相关性。同时对主题爬虫的研究现状进行介绍和总结。     

搜索引擎的设计研究

搜索引擎是Internet信息服务的主体,搜索引擎的设计是各网站建设的重要部分。介绍了搜索引擎的分类和各类搜索引擎的工作过程。在此基础上,指出了蜘蛛程序是由网页下载和网页内容分析及信息提取两部分组成,并结合用C Builder作为开发工具给出了这两部分的源代码示例。最后介绍了蜘蛛程序设计要注意的问题。     

What users see - Structures in search engine results pages

This paper investigates the composition of search engine results pages. We define what elements the most popular web search engines use on their results pages (e.g., organic results, advertisements, shortcuts) and to which degree they are used for popular vs. rare queries. Therefore, we send 500 queries of both types to the major search engines Google, Yahoo, Live.com and Ask. We count how often the different elements are used by the individual engines. In total, our study is based on 42,758 elements. Findings include that search engines use quite different approaches to results pages composition and therefore, the user gets to see quite different results sets depending on the search engine and search query used. Organic results still play the major role in the results pages, but different shortcuts are of some importance, too. Regarding the frequency of certain host within the results sets, we find that all search engines show Wikipedia results quite often, while other hosts shown depend on the search engine used. Both Google and Yahoo prefer results from their own offerings (such as YouTube or Yahoo Answers). Since we used the .com interfaces of the search engines, results may not be valid for other country-specific interfaces.     

基于移动爬虫的专用Web信息收集系统的设计

搜索引擎已经成为网上导航的重要工具。为了能够提供强大的搜索能力,搜索引擎对网上可访问文档维持着详尽的索引。创建和维护索引的任务由网络爬虫完成,网络爬虫代表搜索引擎递归地遍历和下载Web页面。Web页面在下载之后,被搜索引擎分析、建索引,然后提供检索服务。文章介绍了一种更加有效的建立Web索引的方法,该方法是基于移动爬虫(MobileCrawler)的。在此提出的爬虫首先被传送到数据所在的站点,在那里任何不需要的数据在传回搜索引擎之前在当地被过滤。这个方法尤其适用于实施所谓的“智能”爬行算法,这些算法根据已访问过的Web页面的内容来决定一条有效的爬行路径。移动爬虫是移动计算和专业搜索引擎两大技术趋势的结合,能够从技术上很好地解决现在通用搜索引擎所面临的问题。     

Using Local Popularity of Web Resources for Geo-Ranking of Search Engine Results

Search engines retrieve and rank Web pages which are not only relevant to a query but also important or popular for the users. This popularity has been studied by analysis of the links between Web resources. Link-based page ranking models such as PageRank and HITS assign a global weight to each page regardless of its location. This popularity measurement has shown successful on general search engines. However unlike general search engines, location-based search engines should retrieve and rank higher the pages which are more popular locally. The best results for a location-based query are those which are not only relevant to the topic but also popular with or cited by local users. Current ranking models are often less effective for these queries since they are unable to estimate the local popularity. We offer a model for calculating the local popularity of Web resources using back link locations. Our model automatically assigns correct locations to the links and content and uses them to calculate new geo-rank scores for each page. The experiments show more accurate geo-ranking of search engine results when this model is used for processing location-based queries.     

Measuring index quality using random walks on the Web

Recent research has studied how to measure the size of a search engine, in terms of the number of pages indexed. In this paper, we consider a different measure for search engines, namely the quality of the pages in a search engine index. We provide a simple, effective algorithm for approximating the quality of an index by performing a random walk on the Web, and we use this methodology to compare the index quality of several major search engines.     

Mining the Web's link structure

The Web is a hypertext body of approximately 300 million pages that continues to grow at roughly a million pages per day. Page variation is more prodigious than the data's raw scale: taken as a whole, the set of Web pages lacks a unifying structure and shows far more authoring style and content variation than that seen in traditional text document collections. This level of complexity makes an “off-the-shelf” database management and information retrieval solution impossible. To date, index based search engines for the Web have been the primary tool by which users search for information. Such engines can build giant indices that let you quickly retrieve the set of all Web pages containing a given word or string. Experienced users can make effective use of such engines for tasks that can be solved by searching for tightly constrained key words and phrases. These search engines are, however, unsuited for a wide range of equally important tasks. In particular, a topic of any breadth will typically contain several thousand or million relevant Web pages. How then, from this sea of pages, should a search engine select the correct ones-those of most value to the user? Clever is a search engine that analyzes hyperlinks to uncover two types of pages: authorities, which provide the best source of information on a given topic; and hubs, which provide collections of links to authorities. We outline the thinking that went into Clever's design, report briefly on a study that compared Clever's performance to that of Yahoo and AltaVista, and examine how our system is being extended and updated     

Accelerating dynamic Web content generation

As a middle-tier, server-side caching engine, the dynamic content accelerator reduces dynamic page-generation processing delays by caching fragments of dynamically generated Web pages. This fragment-level solution, combined with intelligent cache management strategies, can significantly reduce the processing load on the Web application server, letting it handle higher user loads and thus significantly outperforming existing middle-tier caching solutions.     

基于Lucene.Net校园网搜索引擎的设计与实现

在庞大的网络信息资源利用中,搜索引擎成为人们发现资源的有力工具。但是如果用常用的搜索引擎去搜索专门的门户网站,或者特定范围的网页,则显得力不从心。比如搜索一个学校内的网页,这些搜索引擎就很难进行全面高效的搜索。基于此,利用适应.Net环境的Lucene.Net作为基础,设计了自己的校园网搜索引擎,在特定范围的搜索表现出了自己独特的优势。