基于频繁词义序列的检索结果聚类算法研究

目前大多搜索引擎结果聚类算法针对用户查询生成的网页摘要进行聚类,由于网页摘要较短且质量良莠不齐,聚类效果难以保证。提出了一种基于频繁词义序列的检索结果聚类算法,利用WordNet结合句法和语义特征对搜索结果构建聚类及标签。不像传统的基于向量空间模型的聚类算法,考虑了词语在文档中的序列模式。算法首先对文本进行预处理,生成压缩文档以降低文本数据维度,构建广义后缀树,挖掘出最大频繁项集,然后获取频繁词义序列。从文档中获取的有序频繁项集可以更好地反映文档的主题,把相同主题的搜索结果聚类在一起,与用户查询相关度高的优先排序。实验表明,该算法可以获得与查询相关的高质量聚类及基于语义的聚类标签,具有更高的聚类准确度和更高的运行效率,并且可扩展性良好。     

基于网页聚类的搜索结果优化算法研究

针对目前搜索引擎搜索结果中普遍存在大量重复网页的现象,提出了一种基于聚类算法DBSCAN的搜索结果优化算法.该算法选取源搜索结果中排名靠前的部分网页,对这部分网页根据网页相似度进行DBSCAN聚类,最大限度剔除冗余网页,实现搜索结果的优化.实验结果表明本算法可以提高搜索结果的全面性和准确性,提升用户使用搜索引擎的满意度.     

一种考虑用户行为的改进N-PageRank算法

针对Google PageRank算法中存在的“平均分配原则”及网络链接结构所造成的“旧网页问题”、“主题漂移问题”,提出一种改进的网页排序算法N-PageRank.该算法通过对搜索日志进行数据挖掘,捕捉用户与搜索引擎之间的交互过程,发现隐藏在用户搜索行为背后的用户兴趣和搜索规律,利用用户行为反馈模型,分析网络日志里用户的各项行为特点,改善了排序结果的准确率,保证了搜索引擎的返回结果正是用户所希望看到的网页.实验证明该算法有效地降低了网页排序时的客观因素的影响,充分考虑了用户对于网页质量的评价,所得到的排序结果更加能够满足用户的需求.     

搜索引擎中的聚类浏览技术

搜索引擎大多以文档列表的形式将搜索结果显示给用户,随着Web文档数量的剧增,使得用户查找相关信息变得越来越困难,一种解决方法是对搜索结果进行聚类提高其可浏览性。搜索引擎的聚类浏览技术能使用户在更高的主题层次上查看搜索结果,方便地找到感兴趣的信息。本文介绍了搜索引擎的聚类浏览技术对聚类算法的基本要求及其分类方法,研究分析了主要聚类算法及其改进方法的特点,讨论了对聚类质量的评价,最后指出了聚类浏览技术的发展趋势。     

基于归类的链接分析技术

在目前主流搜索引擎中,链接分析是最常用的计算网页价值度的工具,但是对于用户输入比较宽泛的查询主题,链接分析算法很难得到一个令所有用户都满意的结果。论文试图从另外一个角度来改进链接分析算法,即在传统的链接分析基础上,增加Web聚类算法的有关思想,并对这两种算法进行了改进和组合,提出了一种基于归类的链接分析技术,并用实验结果证明了该算法的性能。     

针对小文本的Web数据挖掘技术及其应用

现有搜索引擎技术返回给用户的信息太多太杂,为此提出一种针对小文本的基于近似网页聚类算法的Web文本数据挖掘技术,该技术根据用户的兴趣程度形成词汇库,利用模糊聚类方法获得分词词典组,采用MD5算法去除重复页面,采用近似网页聚类算法对剩余页面聚类,并用马尔可夫Web序列挖掘算法对聚类结果排序,从而提供用户感兴趣的网页簇序列,使用户可以迅速找到感兴趣的页面。实验证明该算法在保证查全率和查准率的基础上大大提高了搜索效率。由于是针对小文本的数据挖掘,所研究的算法时间和空间复杂度都不高,因此有望成为一种实用、有效的信息检索技术。     

搜索引擎结果聚类算法研究

随着Web文档数量的剧增,搜索引擎也暴露了许多问题,用户不得不在搜索引擎返回的大量文档摘要列表中查找。而对搜索引擎结果聚类能使用户在更高的主题层次上来查看搜索引擎返回的结果。该文提出了搜索引擎结果聚类的几个重要指标并给出了一个新的基于PAT—tree的搜索引擎结果聚类算法。     

企业搜索引擎个性化表示与结果排序算法研究

针对企业搜索引擎提出一种基于本地文档库的个性化表示与结果排序算法,以帮助用户找到真正感兴趣的结果.首先,采用聚类分析对用户浏览的历史文档聚类;其次,采用模糊推理技术对所形成的分类进行分析,发现用户对各分类的喜好程度;再次,按用户对各分类喜好程度的不同,为各分类分配抽样文档数;最后,采用多种抽样技术,从各分类中抽取典型文档.来自不同分类的典型文档构成了表示用户个性的本地文档库.结果排序算法通过计算通用企业搜索引擎的搜索结果与本地文档库中各文档的相似性,对结果集重新排序,从而体现出用户个性.实验结果表明,与传统的基于关键词的个性化表示与结果排序算法相比,基于本地文档集的个性化表示与结果排序算法可以给出更能反映用户个性的查询结果,且可以对用户偏好的变化作出更迅速的反映。     

一个基于搜索结果的个性化推荐系统

搜索引擎返回的结果太多且不能根据用户的兴趣提供检索结果是当前较受关注的问题。把用户兴趣模型和STC聚类算法相结合,提出了改进的STC算法,并提出个性化推荐的策略和兴趣描述更新的方法,实现了一个基于搜索结果的个性化推荐系统(SRPRS)。SRPRS基于改进的STC算法自动组织搜索结果,帮助用户利用主题的方式发现所需的资源。通过实验,分析了SRPRS系统的聚类特性和时间特性。针对搜索引擎的列表显示结果,SRPRS系统在快速查找用户感兴趣的文档上有较好的性能。     

基于主题相关概念和网页分块的主题爬虫研究

针对传统主题爬虫的不足, 提出一种基于主题相关概念和网页分块的主题爬虫。先通过主题分类树获取主题相关概念集合, 然后结合主题描述文档构建主题向量来描述主题; 下载网页后引入网页分块来穿越“灰色隧道”; 采用文本内容和链接结构相结合的策略计算候选链接优先级, 并在HITS算法的基础上提出了R-HITS算法计算链接结构对候选链接优先级的贡献。实验结果表明, 利用该方法实现的主题爬虫查准率达66%、信息量总和达53%, 在垂直搜索引擎和舆情分析应用方面有更好的搜索效果。     

Clustering of document collection – A weighting approach

Clustering algorithms are used to assess the interaction among documents by organizing documents into clusters such that document within a cluster are more similar to each other than are documents belonging to different clusters. Document clustering has been traditionally investigated as a means of improving the performance of search engines by pre-clustering the entire corpus, and a post-retrieval document browsing technique as well. It has long been studied as a post-retrieval document visualization technique. The purpose of present paper to show that assignment weight to documents improves clustering solution.     

A transduction-based approach to fuzzy clustering, relevance ranking and cluster label generation on web search results

This paper details a modular, self-contained web search results clustering system that enhances search results by (i) performing clustering on lists of web documents returned by queries to search engines, and (ii) ranking the results and labeling the resulting clusters, by using a calculated relevance value as a degree of membership to clusters. In addition, we demonstrate an external evaluation method based on precision for comparing fuzzy clustering techniques, as well as internal measures suitable for working on non-training data. The built-in label generator uses the membership degrees and relevance values to weight the most relevant results more heavily. The membership degrees of documents to fuzzy clusters also facilitate effective detection and removal of overly similar clusters. To achieve this, our transduction-based clustering algorithm (TCA) and its fuzzy counterpart (FTCA) employ a transduction-based relevance model (TRM) to consider local relationships between each web document. Results from testing on five different real-world and synthetic datasets results show favorable results compared to established label-based clustering algorithms Suffix Tree Clustering (STC) and Lingo.     

Making scanned Arabic documents machine accessible using an ensemble of SVM classifiers

International Journal on Document Analysis and Recognition (IJDAR) - Raster-image PDF files originating from scanning or photographing paper documents are inaccessible to both text search engines...     

A statistical method for estimating the usefulness of text databases

Searching desired data on the Internet is one of the most common ways the Internet is used. No single search engine is capable of searching all data on the Internet. The approach that provides an interface for invoking multiple search engines for each user query has the potential to satisfy more users. When the number of search engines under the interface is large, invoking all search engines for each query is often not cost effective because it creates unnecessary network traffic by sending the query to a large number of useless search engines and searching these useless search engines wastes local resources. The problem can be overcome if the usefulness of every search engine with respect to each query can be predicted. We present a statistical method to estimate the usefulness of a search engine for any given query. For a given query, the usefulness of a search engine in this paper is defined to be a combination of the number of documents in the search engine that are sufficiently similar to the query and the average similarity of these documents. Experimental results indicate that our estimation method is much more accurate than existing methods.     

Web document clustering using a hybrid neural network

The list of documents returned by Internet search engines in response to a query these days can be quite overwhelming. There is an increasing need for organising this information and presenting it in a more compact and efficient manner. This paper describes a method developed for the automatic clustering of World Wide Web documents, according to their relevance to the user’s information needs, by using a hybrid neural network. The objective is to reduce the time and effort the user has to spend to find the information sought after. Clustering documents by features representative of their contents—in this case, key words and phrases—increases the effectiveness and efficiency of the search process. It is shown that a two-dimensional visual presentation of information on retrieved documents, instead of the traditional linear listing, can create a more user-friendly interface between a search engine and the user.     

Performing Binary-Categorization on Multiple-Record Web Documents Using Information Retrieval Models and Application Ontologies

To retrieve Web documents of interest, most of the Web users rely on Web search engines. All existing search engines provide query facility for users to search for the desired documents using search-engine keywords. However, when a search engine retrieves a long list of Web documents, the user might need to browse through each retrieved document in order to determine which document is of interest. We observe that there are two kinds of problems involved in the retrieval of Web documents: (1) an inappropriate selection of keywords specified by the user; and (2) poor precision in the retrieved Web documents. In solving these problems, we propose an automatic binary-categorization method that is applicable for recognizing multiple-record Web documents of interest, which appear often in advertisement Web pages. Our categorization method uses application ontologies and is based on two information retrieval models, the Vector Space Model (VSM) and the Clustering Model (CM). We analyze and cull Web documents to just those applicable to a particular application ontology. The culling analysis (i) uses CM to find a virtual centroid for the records in a Web document, (ii) computes a vector in a multi-dimensional space for this centroid, and (iii) compares the vector with the predefined ontology vector of the same multi-dimensional space using VSM, which we consider the magnitudes of the vectors, as well as the angle between them. Our experimental results show that we have achieved an average of 90% recall and 97% precision in recognizing Web documents belonged to the same category (i.e., domain of interest). Thus our categorization discards very few documents it should have kept and keeps very few it should have discarded.     

基于搜索结果的个性化推荐系统研究

搜索引擎返回的信息太多且不能根据用户的兴趣提供检索结果,使得用户使用搜索引擎难以用简便的方式找到感兴趣的文档。个性化推荐是一种旨在减轻用户在信息检索方面负担的有效方法。文中把内容过滤技术和文档聚类技术相结合,实现了一个基于搜索结果的个性化推荐系统,以聚类的方法自动组织搜索结果,主动推荐用户感兴趣的文档。通过建立用户概率兴趣模型,对搜索结果STC聚类的基础上进行内容过滤。实验表明,概率模型比矢量空间模型更好地表达了用户的兴趣和变化。     

基于模糊概念图的文档聚类及其在Web中的应用

随着World Wide Web上数据量的日益庞大,现有的搜索引擎已经不能满足用户日益增长的需求.利用数据挖掘技术,提高搜索效率,实现了查询的用户化.首先提出了模糊概念图的模型来描述词语间的关系,然后在聚类过程中引入概念知识,提出了基于模糊概念图的文档聚类算法,通过分析用户的浏览行为发现兴趣模式.在上述技术的基础上,给出了一种用户化的智能搜索系统的实现策略,通过分析概念间的关系和用户的兴趣模式,评价超链/文档和查询的相关程度,从而帮助用户得到更准确的信息.     

基于移动爬虫的专用Web信息收集系统的设计

搜索引擎已经成为网上导航的重要工具。为了能够提供强大的搜索能力,搜索引擎对网上可访问文档维持着详尽的索引。创建和维护索引的任务由网络爬虫完成,网络爬虫代表搜索引擎递归地遍历和下载Web页面。Web页面在下载之后,被搜索引擎分析、建索引,然后提供检索服务。文章介绍了一种更加有效的建立Web索引的方法,该方法是基于移动爬虫(MobileCrawler)的。在此提出的爬虫首先被传送到数据所在的站点,在那里任何不需要的数据在传回搜索引擎之前在当地被过滤。这个方法尤其适用于实施所谓的“智能”爬行算法,这些算法根据已访问过的Web页面的内容来决定一条有效的爬行路径。移动爬虫是移动计算和专业搜索引擎两大技术趋势的结合,能够从技术上很好地解决现在通用搜索引擎所面临的问题。