A Multi-Threaded Semantic Focused Crawler

The Web comprises of voluminous rich learning content.The volume of ever growing learning resources however leads to the problem of information overload.A large number of irrelevant search results generated from search engines based on keyword matching techniques further augment the problem.A learner in such a scenario needs semantically matched learning resources as the search results.Keeping in view the volume of content and significance of semantic knowledge,our paper proposes a multi-threaded semantic focused crawler(SFC) specially designed and implemented to crawl on the WWW for educational learning content.The proposed SFC utilizes domain ontology to expand a topic term and a set of seed URLs to initiate the crawl.The results obtained by multiple iterations of the crawl on various topics are shown and compared with the results obtained by executing an open source crawler on the similar dataset.The results are evaluated using Semantic Similarity,a vector space model based metric,and the harvest ratio.     

融合本体和改进禁忌搜索策略的气象灾害主题爬虫方法

针对传统主题爬虫方法容易陷入局部最优和主题描述不足的问题,提出一种融合本体和改进禁忌搜索策略（On-ITS）的主题爬虫方法。首先利用本体语义相似度计算主题语义向量,基于超级文本标记语言（HTML）网页文本特征位置加权构建网页文本特征向量,然后采用向量空间模型计算网页的主题相关度。在此基础上,计算锚文本主题相关度以及链接指向网页的PR值,综合分析链接优先度。另外,为了避免爬虫陷入局部最优,设计了基于ITS的主题爬虫,优化爬行队列。以暴雨灾害和台风灾害为主题,在相同的实验环境下,基于On-ITS的主题爬虫方法比对比算法的爬准率最多高58%,最少高8%,其他评价指标也很好。基于On-ITS的主题爬虫方法能有效提高获取领域信息的准确性,抓取更多与主题相关的网页。     

Exploiting interclass rules for focused crawling

Crawling the Web quickly and entirely is an expensive, unrealistic goal because of the required hardware and network resources. We started with a focused-crawling approach designed by Soumen Chakrabarti, Martin van den Berg, and Byron Dom, and we implemented the underlying philosophy of their approach to derive our baseline crawler. This crawler employs a canonical topic taxonomy to train a naive-Bayesian classifier, which then helps determine the relevancy of crawled pages. The crawler also relies on the assumption of topical locality to decide which URLs to visit next. Building on this crawler, we developed a rule-based crawler, which uses simple rules derived from interclass (topic) linkage patterns to decide its next move. This rule-based crawler also enhances the baseline crawler by supporting tunneling. A focused crawler gathers relevant Web pages on a particular topic. This rule-based Web-crawling approach uses linkage statistics among topics to improve a baseline focused crawler's harvest rate and coverage.     

扩展UDDI以支持语义信息的研究

介绍了一种扩展UDDI以支持语义信息的方法,即在注册Web服务时添加语义信息,并支持基于语义的查询。首先在UDDI系统中加入一个领域本体库,再为该UDDI中的每个注册服务添加语义信息,并将服务和本体库的对应关系存入到UDDI的数据库中。在服务申请者查询Web服务时,由用户提供语义查询模板,根据用户描述的本体语义信息得到候选服务列表,再根据用户对服务质量的要求计算候选服务的匹配度,将候选服务依照其匹配度的大小顺序返回给用户。     

一种自适应语义Web检索方法

基于内容的语义Web检索只考虑内容本身,没有考虑用户的不同,不能准确反映用户需求。为此,提出一个自适应语义Web检索框架,对于Web中文文档,借助HowNet知识库给出一种本体学习方法,通过提取用户客观、显式和隐式信息建立用户信息库,并设计用户初始查询本体和个性化查询本体构建算法,从而实现用户的自适应检索。实验结果表明,该方法具有较高的检索效率。     

基于主题图的本体信息检索模型研究

针对本体在定义领域概念时具有规范性、明确性和可共享性等特点,结合主题图对文档资源组织方式具有语义可导航性,提出了一种基于主题图的本体信息检索模型,并给出了模型的形式化定义。选择旅游领域作为研究对象,定义了旅游本体和旅游文档资源主题图,分析了在信息检索模型中利用本体来规范用户自然语言查询输入,识别用户检索意图和扩展查询语义方面的作用,并展示了主题图在语义导航和用户相关度排序方面的价值。最后通过实验表明基于主题图的本体信息检索模型较传统的检索系统有较好的性能。     

基于动态主题库的主题爬虫

通过对基于不同策略过滤URL的主题爬虫的研究,提出了一种基于动态主题库的主题爬虫.它能够在运行期间实时地更新主题库,提高了对URL过滤的准确度.实验表明,所提的主题爬虫能够在相对较少的时间中,检索尽量少的网络空间,抓取到较多与主题相关的网页.     

Empirical evaluation of the link and content-based focused Treasure-Crawler

Indexing the Web is becoming a laborious task for search engines as the Web exponentially grows in size and distribution. Presently, the most effective known approach to overcome this problem is the use of focused crawlers. A focused crawler employs a significant and unique algorithm in order to detect the pages on the Web that relate to its topic of interest. For this purpose we proposed a custom method that uses specific HTML elements of a page to predict the topical focus of all the pages that have an unvisited link within the current page. These recognized on-topic pages have to be sorted later based on their relevance to the main topic of the crawler for further actual downloads. In the Treasure-Crawler, we use a hierarchical structure called T-Graph which is an exemplary guide to assign appropriate priority score to each unvisited link. These URLs will later be downloaded based on this priority. This paper embodies the implementation, test results and performance evaluation of the Treasure-Crawler system. The Treasure-Crawler is evaluated in terms of specific information retrieval criteria such as recall and precision, both with values close to 50%. Gaining such outcome asserts the significance of the proposed approach.     

面向主题爬取的多粒度URLs优先级计算方法

垂直检索系统中主题爬虫的性能对整个系统至关重要。在设计主题爬虫时需要解决两个问题一是计算当前页面与给定主题的相关度, 二是计算待爬取URLs的访问优先级。对第一个问题,给出利用页面的主题文本块和相关链接块的相关度计算方法; 对第二个问题, 给出基于主题上下文和四种不同的粒度(即站点级、页面级、块级和链接级)的优先级计算方法。在此基础上, 提出基于上述方法的主题爬取算法。实验证明, 新算法在不增加时间复杂度的前提下, 在查准率和信息量总和方面明显优于其他三种经典的爬取算法。     

领域相关的Web网站抓取方法

本文提出了一种抓取领域相关的Web站点的方法，可以在较小的代价下准确地收集用户所关心领域内的网站。这种方法主要改进了传统的聚焦爬虫（Focused Crawler）技术，首先利用Meta-Search技术来改进传统Crawler的通过链接分析来抓取网页的方法，而后利用启发式搜索大大降低了搜索代价，通过引入一种评价领域相关性的打分方法，迭到了较好的准确率。本文详细地描述了上述算法并通过详细的实验验证了算法的效率和效果。     

Algorithmic Computation and Approximation of Semantic Similarity

Automatic extraction of semantic information from text and links in Web pages is key to improving the quality of search results. However, the assessment of automatic semantic measures is limited by the coverage of user studies, which do not scale with the size, heterogeneity, and growth of the Web. Here we propose to leverage human-generated metadata—namely topical directories—to measure semantic relationships among massive numbers of pairs of Web pages or topics. The Open Directory Project classifies millions of URLs in a topical ontology, providing a rich source from which semantic relationships between Web pages can be derived. While semantic similarity measures based on taxonomies (trees) are well studied, the design of well-founded similarity measures for objects stored in the nodes of arbitrary ontologies (graphs) is an open problem. This paper defines an information-theoretic measure of semantic similarity that exploits both the hierarchical and non-hierarchical structure of an ontology. An experimental study shows that this measure improves significantly on the traditional taxonomy-based approach. This novel measure allows us to address the general question of how text and link analyses can be combined to derive measures of relevance that are in good agreement with semantic similarity. Surprisingly, the traditional use of text similarity turns out to be ineffective for relevance ranking.     

An efficient approach to suggesting topically related web queries using hidden topic model

Keyword-based Web search is a widely used approach for locating information on the Web. However, Web users usually suffer from the difficulties of organizing and formulating appropriate input queries due to the lack of sufficient domain knowledge, which greatly affects the search performance. An effective tool to meet the information needs of a search engine user is to suggest Web queries that are topically related to their initial inquiry. Accurately computing query-to-query similarity scores is a key to improve the quality of these suggestions. Because of the short lengths of queries, traditional pseudo-relevance or implicit-relevance based approaches expand the expression of the queries for the similarity computation. They explicitly use a search engine as a complementary source and directly extract additional features (such as terms or URLs) from the top-listed or clicked search results. In this paper, we propose a novel approach by utilizing the hidden topic as an expandable feature. This has two steps. In the offline model-learning step, a hidden topic model is trained, and for each candidate query, its posterior distribution over the hidden topic space is determined to re-express the query instead of the lexical expression. In the online query suggestion step, after inferring the topic distribution for an input query in a similar way, we then calculate the similarity between candidate queries and the input query in terms of their corresponding topic distributions; and produce a suggestion list of candidate queries based on the similarity scores. Our experimental results on two real data sets show that the hidden topic based suggestion is much more efficient than the traditional term or URL based approach, and is effective in finding topically related queries for suggestion.     

A topic-specific crawling strategy based on semantics similarity

With the Internet growing exponentially, search engines are encountering unprecedented challenges. A focused search engine selectively seeks out web pages that are relevant to user topics. Determining the best strategy to utilize a focused search is a crucial and popular research topic. At present, the rank values of unvisited web pages are computed by considering the hyperlinks (as in the PageRank algorithm), a Vector Space Model and a combination of them, and not by considering the semantic relations between the user topic and unvisited web pages. In this paper, we propose a concept context graph to store the knowledge context based on the user's history of clicked web pages and to guide a focused crawler for the next crawling. The concept context graph provides a novel semantic ranking to guide the web crawler in order to retrieve highly relevant web pages on the user's topic. By computing the concept distance and concept similarity among the concepts of the concept context graph and by matching unvisited web pages with the concept context graph, we compute the rank values of the unvisited web pages to pick out the relevant hyperlinks. Additionally, we constitute the focused crawling system, and we retrieve the precision, recall, average harvest rate, and F-measure of our proposed approach, using Breadth First, Cosine Similarity, the Link Context Graph and the Relevancy Context Graph. The results show that our proposed method outperforms other methods.     

基于语义主题相似度的Web教育资源查询方法

针对传统Web教育主体难以获得高可用教育资源的问题,提出了一种面向语义主题相似度的Web教育资源查询方法。该方法建立了本体概念语义网络（Ontology Concept Semantic Network,OCSN）,在此基础上,设计了基于语义主题相似度匹配的概念检索方法：在检索前主动将教育资源根据其语义和主题组织到本体概念语义网络中,然后建立一个基于语义特性的Web教育资源发现的垂直搜索引擎,并通过构造满足条件的相似度函数,将对应的语义距离映射为相似度,有效地提高了查询效率。实验结果表明此方法能够提高Web教育资源的查准率和查全率。     

Search personalization through query and page topical analysis

Thousands of users issue keyword queries to the Web search engines to find information on a number of topics. Since the users may have diverse backgrounds and may have different expectations for a given query, some search engines try to personalize their results to better match the overall interests of an individual user. This task involves two great challenges. First the search engines need to be able to effectively identify the user interests and build a profile for every individual user. Second, once such a profile is available, the search engines need to rank the results in a way that matches the interests of a given user. In this article, we present our work towards a personalized Web search engine and we discuss how we addressed each of these challenges. Since users are typically not willing to provide information on their personal preferences, for the first challenge, we attempt to determine such preferences by examining the click history of each user. In particular, we leverage a topical ontology for estimating a user’s topic preferences based on her past searches, i.e. previously issued queries and pages visited for those queries. We then explore the semantic similarity between the user’s current query and the query-matching pages, in order to identify the user’s current topic preference. For the second challenge, we have developed a ranking function that uses the learned past and current topic preferences in order to rank the search results to better match the preferences of a given user. Our experimental evaluation on the Google query-stream of human subjects over a period of 1 month shows that user preferences can be learned accurately through the use of our topical ontology and that our ranking function which takes into account the learned user preferences yields significant improvements in the quality of the search results.     

主题分布在Deep Web数据库选择中的应用

随着越来越多的信息隐藏在Deep Web中,针对用户查询找出最相关的Web数据库成为亟待解决的问题。提出了一种基于Web数据库主题分布的方法用于Deep Web数据集成中的Web数据库选择。获取主题覆盖度形式的Web数据库内容描述,而后利用选定的Web数据库获取查询主题,最终由查询主题和主题分布矩阵来选择Web数据库。在真实Web数据库上的实验结果表明,该方法既取得了较高的查询召回率,也可有效降低数据库内容描述建立的代价。     

基于语义的主题爬行策略

为使主题爬行能够充分利用资源的语义信息,提出基于语义的主题爬行策略.该策略利用领域本体刻画爬行主题,将本体语义映射到关键词表.通过定义断言集一致性扩展和域值关联推理任务,推演关键词间语义关系.在定义网页主题概念的基础上,结合本体推理方案提出主题概念的语义叠加效应模型.最后,利用主题概念的语义包含关系判定URLs抓取顺序.实验结果表明,该语义主题爬行策略在抓取收获率和爬行效率上优于现有同类方法,该方案有效、可行.     

Effective Tag Recommendation System Based on Topic Ontology Using Wikipedia and WordNet

In this paper, we proposed a novel approach based on topic ontology for tag recommendation. The proposed approach intelligently generates tag suggestions to blogs. In this approach, we construct topic ontology through enriching the set of categories in existing small ontology called as Open Directory Project. To construct topic ontology, a set of topics and their associated semantic relationships is identified automatically from the corpus‐based external knowledge resources such as Wikipedia and WordNet. The construction relies on two folds such as concept acquisition and semantic relation extraction. In the first fold, a topic‐mapping algorithm is developed to acquire the concepts from the semantic of Wikipedia. A semantic similarity‐clustering algorithm is used to compute the semantic similarity measure to group the set of similar concepts. The second is the semantic relation extraction algorithm, which derives associated semantic relations between the set of extracted topics from the lexical patterns between synsets in WordNet. A suitable software prototype is created to implement the topic ontology construction process. A Jena API framework is used to organize the set of extracted semantic concepts and their corresponding relationship in the form of knowledgeable representation of Web ontology language. Thus, Protégé tool provides the platform to visualize the automatically constructed topic ontology successfully. Using the constructed topic ontology, we can generate and suggest the most suitable tags for the new resource to users. The applicability of topic ontology with a spreading activation algorithm supports efficient recommendation in practice that can recommend the most popular tags for a specific resource. The spreading activation algorithm can assign the interest scores to the existing extracted blog content and tags. The weight of the tags is computed based on the activation score determined from the similarity between the topics in constructed topic ontology and content of the existing blogs. High‐quality tags that has the highest activation score is recommended to the users. Finally, we conducted experimental evaluation of our tag recommendation approach using a large set of real‐world data sets. Our experimental results explore and compare the capabilities of our proposed topic ontology with the spreading activation tag recommendation approach with respect to the existing AutoTag mechanism. And also discuss about the improvement in precision and recall of recommended tags on the data sets of Delicious and BibSonomy. The experiment shows that tag recommendation using topic ontology results in the folksonomy enrichment. Thus, we report the results of an experiment mean to improve the performance of the tag recommendation approach and its quality.     

基于XQuery扩展的异构语义Web服务发现

语义Web服务发现作为分布式计算的前提和关键,备受研究者关注。多种语义Web服务描述语言的存在给异构语义Web服务的发现带来了挑战。本文提出了一种支持异构语义Web服务描述语言的发现框架i XQuery。该框架从两个方面扩展XQuery,使其支持异构语义Web服务的结构化查询与逻辑推理、模糊匹配的联合查询。一方面i XQuery利用XQuery的外部函数机制,建立了多种类型的相似度比较函数;另一方面i XQuery制定了一个统一的语义Web服务抽象描述本体,并建立了语义Web服务语言SAWSDL与OWLS与该本体之间的映射,并利用XQuery的用户自定义函数机制,建立了一系列用于抽取异构语义Web服务中信息的抽象描述操作子。最后,介绍了基于i XQuery框架的异构语义Web服务匹配器H-i Matcher。     

基于自然语言理解的SPARQL本体查询

为了用户能够方便地获取本体知识,提出了基于自然语言理解的SPARQL本体查询。利用Stanford Parser分析用户的自然语言查询,根据语法构建查询三元组,与关键词的方法相比,有效地减少了组合的个数。结合用户词典,能较准确地把查询三元组的词汇映射到本体实体。分值计算时除了考虑词语的形式相似和语义相似外,还考虑了概念的模糊性,尽量返回具体的概念。利用本体推理获取隐藏在本体中的信息,对查询进行过滤和限制,提高了准确率。用户通过图形交互界面和系统进行交互,选择需要的结果,最后返回树形查询结果,并能看到相关的信息。实验结果表明,该方法达到了预期的效果。