Web数据挖掘技术

随着Intemet／Web技术的快速普及和迅猛发展,各种信息可以以非常低的成本在网络上获得。如何在这些信息中找到用户真正需要的内容,成为数据组织和Web相关领域专家学者关注的焦点。Web数据挖掘旨在发现隐藏在Web数据中潜在的有用知识、提供决策支持,已经成为数据挖掘领域中新兴的研究热点。该文主要从Web内容挖掘、Web结构挖掘和Web使用挖掘三个方面阐述Web数据挖掘的基本知识。     

空间关联规则的增量维护

为了得到有趣且有效的空间关联规则通常需要多次执行挖掘操作,可以使用增量维护算法来提高挖掘效率。然而,能够直接使用空间数据的关联规则增量更新算法尚属空白。为解决这一问题,对挖掘阈值改变和空间数据集更新后通过筛选或增量挖掘等方法实现规则维护的策略进行了分析,并提出适用于支持度阈值减小和空间图层增加这两类情况的增量挖掘算法——ISA。ISA算法不依赖于空间事务表的构建与更新,可以直接使用空间图层作为输入数据。在基于实际数据的实验中,采用ISA算法所得结果与类Apriori算法一致,耗时则相对缩短20.0%至71.0%;此外,对1372772条规则进行了基于筛选的更新,耗时低于0.1s。实验结果表明,所提出的空间关联规则增量维护策略和算法是可行、正确且高效的。     

A Data Cube Model for Prediction-Based Web Prefetching

Reducing the web latency is one of the primary concerns of Internet research. Web caching and web prefetching are two effective techniques to latency reduction. A primary method for intelligent prefetching is to rank potential web documents based on prediction models that are trained on the past web server and proxy server log data, and to prefetch the highly ranked objects. For this method to work well, the prediction model must be updated constantly, and different queries must be answered efficiently. In this paper we present a data-cube model to represent Web access sessions for data mining for supporting the prediction model construction. The cube model organizes session data into three dimensions. With the data cube in place, we apply efficient data mining algorithms for clustering and correlation analysis. As a result of the analysis, the web page clusters can then be used to guide the prefetching system. In this paper, we propose an integrated web-caching and web-prefetching model, where the issues of prefetching aggressiveness, replacement policy and increased network traffic are addressed together in an integrated framework. The core of our integrated solution is a prediction model based on statistical correlation between web objects. This model can be frequently updated by querying the data cube of web server logs. This integrated data cube and prediction based prefetching framework represents a first such effort in our knowledge.     

Prediction of user navigation patterns by mining the temporal web usage evolution

Advances in the data mining technologies have enabled the intelligent Web abilities in various applications by utilizing the hidden user behavior patterns discovered from the Web logs. Intelligent methods for discovering and predicting user’s patterns is important in supporting intelligent Web applications like personalized services. Although numerous studies have been done on Web usage mining, few of them consider the temporal evolution characteristic in discovering web user’s patterns. In this paper, we propose a novel data mining algorithm named Temporal N-Gram (TN-Gram) for constructing prediction models of Web user navigation by considering the temporality property in Web usage evolution. Moreover, three kinds of new measures are proposed for evaluating the temporal evolution of navigation patterns under different time periods. Through experimental evaluation on both of real-life and simulated datasets, the proposed TN-Gram model is shown to outperform other approaches like N-gram modeling in terms of prediction precision, in particular when the web user’s navigating behavior changes significantly with temporal evolution.     

A web personalizing technique using adaptive data structures: The case of bursts in web visits

The explosive growth in the size and use of the World Wide Web continuously creates new great challenges and needs. The need for predicting the users’ preferences in order to expedite and improve the browsing though a site can be achieved through personalizing of the Websites. Recommendation and personalization algorithms aim at suggesting WebPages to users based on their current visit and past users’ navigational patterns. The problem that we address is the case where few WebPages become very popular for short periods of time and are accessed very frequently in a limited temporal space. Our aim is to deal with these bursts of visits and suggest these highly accessed pages to the future users that have common interests. Hence, in this paper, we propose a new web personalization technique, based on advanced data structures.The data structures that are used are the Splay tree (1) and Binary heaps (2). We describe the architecture of the technique, analyze the time and space complexity and prove its performance. In addition, we compare both theoretically and experimentally the proposed technique to another approach to verify its efficiency. Our solution achieves O(P²) space complexity and runs in k log P time, where k is the number of pages and P the number of categories of WebPages.     

Efficient algorithms for incremental Web log mining with dynamic thresholds

With the fast increase in Web activities, Web data mining has recently become an important research topic and is receiving a significant amount of interest from both academic and industrial environments. While existing methods are efficient for the mining of frequent path traversal patterns from the access information contained in a log file, these approaches are likely to over evaluate associations. Explicitly, most previous studies of mining path traversal patterns are based on the model of a uniform support threshold, where a single support threshold is used to determine frequent traversal patterns without taking into consideration such important factors as the length of a pattern, the positions of Web pages, and the importance of a particular pattern, etc. As a result, a low support threshold will lead to lots of uninteresting patterns derived whereas a high support threshold may cause some interesting patterns with lower supports to be ignored. In view of this, this paper broadens the horizon of frequent path traversal pattern mining by introducing a flexible model of mining Web traversal patterns with dynamic thresholds. Specifically, we study and apply the Markov chain model to provide the determination of support threshold of Web documents; and further, by properly employing some effective techniques devised for joining reference sequences, the proposed algorithm dynamic threshold miner (DTM) not only possesses the capability of mining with dynamic thresholds, but also significantly improves the execution efficiency as well as contributes to the incremental mining of Web traversal patterns. Performance of algorithm DTM and the extension of existing methods is comparatively analyzed with synthetic and real Web logs. It is shown that the option of algorithm DTM is very advantageous in reducing the number of unnecessary rules produced and leads to prominent performance improvement.     

Web数据挖掘技术

随着Internet/Web技术的快速普及和迅猛发展,各种信息可以以非常低的成本在网络上获得,如何在这些信息中找到用户真正需要的内容,成为数据组织和Web相关领域专家学者关注的焦点。Web数据挖掘旨在发现隐藏在Web数据中潜在的有用知识、提供决策支持,已经成为数据挖掘领域中新兴的研究热点。该文主要从Web内容挖掘、Web结构挖掘和Web使用挖掘三个方面阐述Web数据挖掘的基本知识。     

基于三元互关联后继树的Web日志挖掘

针对二元的互关联后继树模型进行改进,构造三元互关联后继树,并结合Web日志的特点,构造Web事务集的互关联后继树和增量更新模型,设计基于三元互关联后继树的频繁路径挖掘算法,挖掘Web日志中的频繁路径。通过实验证明了基于三元互关联后继树和改进后的模型的出色插入查询性能。     

Flexible online association rule mining based on multidimensional pattern relations

Most incremental mining and online mining algorithms concentrate on finding association rules or patterns consistent with entire current sets of data. Users cannot easily obtain results from only interesting portion of data. This may prevent the usage of mining from online decision support for multidimensional data. To provide ad-hoc, query-driven, and online mining support, we first propose a relation called the multidimensional pattern relation to structurally and systematically store context and mining information for later analysis. Each tuple in the relation comes from an inserted dataset in the database. We then develop an online mining approach called three-phase online association rule mining (TOARM) based on this proposed multidimensional pattern relation to support online generation of association rules under multidimensional considerations. The TOARM approach consists of three phases during which final sets of patterns satisfying various mining requests are found. It first selects and integrates related mining information in the multidimensional pattern relation, and then if necessary, re-processes itemsets without sufficient information against the underlying datasets. Some implementation considerations for the algorithm are also stated in detail. Experiments on homogeneous and heterogeneous datasets were made and the results show the effectiveness of the proposed approach.     

一种Web用户行为聚类算法

提出了一种新的路径相似度系数计算方法,并使之与雅可比相似系数结合,用于计算用户访问行为的相似度,在此基础之上又提出了一种分析web用户行为的聚类算法(FCC)。通过挖掘Web日志,找出具有相似行为的web用户,由于FCC聚类算法过滤了小于指定阚值的相似度系数,大大缩小了数据规模,很好地解决了其他聚类算法(如层次聚类)在高堆空间聚类时的“堆数灾难”问题,最后的实验结果很好。     

WEB日志挖掘及其实现

Web日志中积累了大量的有用信息,从Web日志中发现有用的信息是非常必要的。该文研究了Web日志挖掘的机理,提出了通过访问路径挖掘来分析用户浏览模式的方法,并实现了一种有效的访问路径模式挖掘算法。     

一种Web使用模式挖掘模型的设计

Web使用模式挖掘是对用户浏览Web后在服务器日志上所留信息的数据挖掘.介绍了挖掘中常用技术及流程,并提出一种Web使用模式挖掘体系结构,介绍了系统的工作原理,对系统设计中的数据清洗和会话识别等关键技术作了详细讨论.     

网页驻留时间约束的用户访问序列挖掘

对Web用户的访问序列进行分析,可以发现用户的爱好、兴趣、习惯等因素,为Web网站的升级修正提供必要的信息支持,提出一种通过对用户访问序列进行分析的数据挖掘方法,该方法采用网页驻留时间为参数来约减会话序列中的网页数,压缩频繁访问序列的规模。实验结果表明,该算法可以降低挖掘成本,为Web用户的商业数据挖掘提供有益的借鉴。     

一种Web挖掘的框架

随着Web信息量的增长,Web用户也迅速增长,如何在海量信息中找出用户需要的信息变得更加重要。基于Web服务器日志,分析在线用户的浏览行为,挖掘Web数据并找出用户的遍历模式已经成为一个新的研究领域。针对Web站点的结构,给出了一个Web挖掘的完整框架,允许在分析复杂的遍历模式时加入约束条件,然后对框架中算法的执行效率和执行准确性进行比较和分析,同时展望了Web挖掘的未来研究方向。     

Mining constrained frequent itemsets from distributed uncertain data

Nowadays, high volumes of massive data can be generated from various sources (e.g., sensor data from environmental surveillance). Many existing distributed frequent itemset mining algorithms do not allow users to express the itemsets to be mined according to their intention via the use of constraints. Consequently, these unconstrained mining algorithms can yield numerous itemsets that are not interesting to users. Moreover, due to inherited measurement inaccuracies and/or network latencies, the data are often riddled with uncertainty. These call for both constrained mining and uncertain data mining. In this journal article, we propose a data-intensive computer system for tree-based mining of frequent itemsets that satisfy user-defined constraints from a distributed environment such as a wireless sensor network of uncertain data.     

基于WEB挖掘的企业营运能力绩效评价模型的研究

企业的Web日志中蕴藏着丰富的信息.首先从企业绩效的角度出发,提出以企业营运能力的绩效评价为目标的Web被访信息的空间存储模型,模型中存储了访问时间序列、访问轨迹和评价指标等信息;然后将绩效评价体系的理论、基于Web的信息技术、数据挖掘技术集成,利用关联规则挖掘算法实现对因特网内的大范围Web日志的内容分析和主题挖掘,建立了基于Web挖掘的企业绩效方法.该模型为企业进行绩效评价提供了一种新思路.     

数据挖掘在Web智能化中应用研究

分析了Web信息的特点和目前开发利用的局限,提出在Web上采用数据挖掘技术即Web挖掘,促进web智能化的观点。全面阐述了Web挖掘在Web智能化中的几个重要应用。指出Web挖掘是Web技术中一个重要的研究领域,是发现蕴藏在web上知识、区分权威链接、理解用户访问模式和网页语义结构的关键,它使充分利用Web大量的真正有价值的信息成为可能,为智能化Web奠定了基础。     

分布式Web用户兴趣迁移模式挖掘研究

提出了一个结合Web文本挖掘的分布式Web使用挖掘模型DWLMST,以及基于该模型的局部浏览兴趣迁移模式更新算法LITP和全局浏览兴趣迁移模式更新算法GITP。利用页面聚类来表示用户兴趣。通过将用户事务中的页面替代为相应的聚类号来得到用户浏览兴趣序列。从用户浏览兴趣序列中分析得到用户浏览兴趣迁移模式。算法较好地解决了Web访问信息的异地存储、实时增长等因素给模式分析过程带来的困难,同时也提高了用户浏览兴趣表示的准确性。     

Web挖掘在电子商务中的应用

如何从海量的Web数据中发现有用的知识是一个迫切需要研究的课题,因此,Web挖掘应运而生,成为一个全新的研究领域。Web挖掘就是从Web文档和Web活动中抽取潜在的有用模式和隐藏信息。随着电子商务的发展,Web挖掘进入了一个新的应用领域,介绍了Web挖掘技术在电子商务中的具体应用,运用Web挖掘技术对Web数据进行挖掘,了解客户的行为,从而调整站点结构、市场策略等,使电子商务活动具有针对性。     

Web日志挖掘中数据预处理技术的研究

在Web数据挖掘研究领域中,Web日志挖掘是一个极其重要的应用方面,而数据预处理技术在Web日志挖掘中又起到非常重要的作用.介绍Web日志文件的记录格式和Web日志挖掘预处理的一般过程,针对实际应用中遇到的问题提出一种解决方法,最后给出算法代码.