基于Lucene.Net校园网搜索引擎的设计与实现

在庞大的网络信息资源利用中，搜索引擎成为人们发现资源的有力工具。但是如果用常用的搜索引擎去搜索专门的门户网站，或者特定范围的网页，则显得力不从心。比如搜索一个学校内的网页，这些搜索引擎就很难进行全面高效的搜索。基于此，利用适应．Net环境的Lucene．Net作为基础，设计了自己的校园网搜索引擎，在特定范围的搜索表现出了自己独特的优势。     

基于网页内容和链接价值的相关度方法的实现

专业搜索引擎提供特定主题的信息检索服务,是新一代搜索引擎的发展方向之一,而网页主题相关度分析是专业搜索引擎的核心技术,它指导着robot进行有价值的搜索,专门搜索与主题相关的页面;提出_种综合的网页主题相关度分析方法,方法同时对网页内容价值和链接价值进行了考察,从而保证了robot搜索的网页与主题有着较高的相关度;在网页内容价值评价时,对传统的方法进行了改进,新的方法能高好的实现.该方法也用于服装行业的搜索引擎,效果明显.     

搜索引擎 网上淘金

在美国,搜索引擎(SEARCH EN-GINE)通常指的是基于互联网的搜索引擎,它能够搜索互联网上几千万到几亿个网页,并且每一个网页上的每一个词都可以按收录,即全文检索,典型的如ALTAVISTA,INKTOMI,INFOSEEK,GOOGLE等。而在中国,搜索引擎的概念发生了变化,通常指基于网站目录的搜索服务或特定网站的搜索服务,前者如搜狐、新浪等公司开发的网站搜索服务,后者如《北京青年报》网站上提供的全文检索服务。 在过去的几年中,国内对基于互联网的搜索引擎曾进行了一些开发尝试,如北大的天网,清华的网络指南针等,但是由于技术门槛相当高,开发这种搜索引擎的技术者即使在硅谷也凤毛麟角,加之它对硬件的投入要求很高,因此商业化的互联网中文搜索引擎一直没有出现。 回首搜索引擎在中国的这些年,可以说中国没有过自己的商业化中文全文网页搜索技术,而不久之前,这一空白被从硅谷回来的创业团队百度公司从技术上突破,使中文搜索技术站在了与英文搜索平起平坐的地位。     

网页标题分析对主题爬虫的改进

随着网络信息资源的爆发式增长,现有的搜索引擎已经无法满足迅速获取准确信息的需要,为搜索引擎引入搜索内容更为精确的主题爬虫显得十分迫切.然而目前的主题爬虫所采用的两种基本抓取网页的方式效率比较低下.提出了一种通过网页标题分析对主题爬虫的改进方案,比较了引人标题分析前后的结果,论证了设计的可行性与可操作性,优化了主题爬虫对同类型特定信息的抓取.     

一种基于常用搜索引擎的智能信息检索系统

针对用户利用常用搜索引擎查询信息时，搜索引擎返回海量杂乱、无序的网页，用户难以从中快速、准确地获得真正关心的信息的现状，从Internet用户的兴趣度出发，设计了一种基于近似网页聚类算法的智能搜索系统。该系统在用户利用常用搜索引擎系统进行信息检索时，消除搜索引擎返回的重复页，对剩余页面进行聚类，返回给用户聚类后的网页簇，这样用户就可以选择浏览自己感兴趣的页面，从而大大提高了信息检索的查准率；实验证明该系统在保证查全率和查准率的基础上大大提高了搜索效率。     

定题搜索引擎Robot的设计与算法

定题搜索引擎将信息检索限定在特定主题领域，提供特定主题的信息检索服务，是新一代搜索引擎的发展方向之一。该文介绍了一个定题搜索robot系统NetBat 2．02版，它可以实现在web上爬行下载主题相关网页。定题搜索的关键技术是主题相关信息的搜索及网页相关度分析。该文分析了传统定题搜索算法的优缺点，提出了基于反向链接结合超链文本分析的定题搜索算法。文章还对基于内容的网页相关度分析算法进行了详细的论述。对比搜索实验表明系统有着较好的性能，能准确地爬行到主题相关网页。     

基于遗传算法的定题信息搜索策略

定题检索将信息检索限定在特定主题领域,提供主题领域内信息的检索服务。它是新一代搜索引擎的发展方向之一。定题检索的关键技术是主题相关信息的搜索。本文提出了基于遗传算法的定题信息搜索策略,提高链接于内容相似度不高的网页之后的页面被搜索的机会,扩大了相关网页的搜索范围。同时,借助超链Metadata的提示信息预测链接页面的主题相关度,加快了搜索速度。对比搜索试验证明了算法具有较好的性能。     

接触Lucene

互联网中的资源非常丰富,而且每天都以指数级的速度增长,但是如何有效的寻找信息却是一件困难的事情。搜索引擎的出现帮我们解决了这个问题。它就像整个网络资源的接口,网络用户可以通过它找到自己想要去的网页。但是,信息越来越多样化,而通用的搜索引擎往往讲究高覆盖率,查询时往往会产生很多不相干的网页。也就是说它的查准率比较低。这种泛化的搜索已经不能满足特定的用户需要更精确信息查询的需求。     

基于Nutch农业搜索引擎的研究与设计

针对目前通用搜索引擎对专门领域及特定主题信息覆盖率较低,在开源的Nutch搜索引擎架构的基础上,通过Hash索引在多语种农业叙词表AGROVOC上进行农业词典的构建,利用已有的空间向量算法进行农业相关度计算,并结合PageRank的改进算法对结果综合排序,搭建了一个面向互联网上农业相关信息资源的搜索引擎.相对于通用搜索引擎来说减少了搜索结果的信息量,提高了搜索速度,同时提高了专业信息搜索的准确率.     

基于近似网页聚类的智能搜索系统

从Internet用户的兴趣度出发,设计了一种基于近似网页聚类的智能搜索系统。该系统在用户利用常用搜索引擎系统进行信息检索时,消除搜索引擎返回的重复页,对剩余页面进行聚类,返回给用户聚类后的网页簇,这样用户就可以选择浏览自己感兴趣的页面,从而大大提高了信息检索的查准率;实验证明该系统在保证查全率和查准率的基础上大大提高了搜索效率。     

Mining the Web's link structure

The Web is a hypertext body of approximately 300 million pages that continues to grow at roughly a million pages per day. Page variation is more prodigious than the data's raw scale: taken as a whole, the set of Web pages lacks a unifying structure and shows far more authoring style and content variation than that seen in traditional text document collections. This level of complexity makes an “off-the-shelf” database management and information retrieval solution impossible. To date, index based search engines for the Web have been the primary tool by which users search for information. Such engines can build giant indices that let you quickly retrieve the set of all Web pages containing a given word or string. Experienced users can make effective use of such engines for tasks that can be solved by searching for tightly constrained key words and phrases. These search engines are, however, unsuited for a wide range of equally important tasks. In particular, a topic of any breadth will typically contain several thousand or million relevant Web pages. How then, from this sea of pages, should a search engine select the correct ones-those of most value to the user? Clever is a search engine that analyzes hyperlinks to uncover two types of pages: authorities, which provide the best source of information on a given topic; and hubs, which provide collections of links to authorities. We outline the thinking that went into Clever's design, report briefly on a study that compared Clever's performance to that of Yahoo and AltaVista, and examine how our system is being extended and updated     

EFFICIENT METHODOLOGIES TO HANDLE HANGING PAGES USING VIRTUAL NODE

In this article we first explain the knowledge extraction (KE) process from the World Wide Web (WWW) using search engines. Then we explore the PageRank algorithm of Google search engine (a well-known link-based search engine) with its hidden Markov analysis. We also explore one of the problems of link-based ranking algorithms called hanging pages or dangling pages (pages without any forward links). The presence of these pages affects the ranking of Web pages. Some of the hanging pages may contain important information that cannot be neglected by the search engine during ranking. We propose methodologies to handle the hanging pages and compare the methodologies. We also introduce the TrustRank algorithm (an algorithm to handle the spamming problems in link-based search engines) and include it in our proposed methods so that our methods can combat Web spam. We implemented the PageRank algorithm and TrustRank algorithm and modified those algorithms to implement our proposed methodologies.     

Tuning up the search engine

Advances in Internet search engine technology may not help you blast Klingons into outer space, but they should help you find them more quickly on the Web. The whole arena for Internet searching has become rather interesting. Search engines appear poised to make some serious breakthroughs in relevancy ranging and personalization that promise to increase the accuracy and reliability of search. On the ether hand, data suggests that users are becoming increasingly disenchanted with search engines that don't actually search the Web, but rather search records of the Web sites their robots have visited. Some online merchants (Victoria's Secret, for example) don't even enable keyword searches on their sites. The Web's increasingly dynamic nature complicates searching. New pages created on the fly using personalization information, and even static content, with dynamically inserted sidebars, navigation bars, advertising and commentary, can present a rapidly changing picture for any robot to discover. And as indexes grow larger, search system performance becomes a significant problem     

搜索引擎的设计研究

搜索引擎是Internet信息服务的主体,搜索引擎的设计是各网站建设的重要部分。介绍了搜索引擎的分类和各类搜索引擎的工作过程。在此基础上,指出了蜘蛛程序是由网页下载和网页内容分析及信息提取两部分组成,并结合用C Builder作为开发工具给出了这两部分的源代码示例。最后介绍了蜘蛛程序设计要注意的问题。     

What users see - Structures in search engine results pages

This paper investigates the composition of search engine results pages. We define what elements the most popular web search engines use on their results pages (e.g., organic results, advertisements, shortcuts) and to which degree they are used for popular vs. rare queries. Therefore, we send 500 queries of both types to the major search engines Google, Yahoo, Live.com and Ask. We count how often the different elements are used by the individual engines. In total, our study is based on 42,758 elements. Findings include that search engines use quite different approaches to results pages composition and therefore, the user gets to see quite different results sets depending on the search engine and search query used. Organic results still play the major role in the results pages, but different shortcuts are of some importance, too. Regarding the frequency of certain host within the results sets, we find that all search engines show Wikipedia results quite often, while other hosts shown depend on the search engine used. Both Google and Yahoo prefer results from their own offerings (such as YouTube or Yahoo Answers). Since we used the .com interfaces of the search engines, results may not be valid for other country-specific interfaces.     

Weather without the weatherman [on the World Wide Web]

If you have any interest in meteorology, you can learn a lot by trawling the World Wide Web. As with other subjects, it presents a wealth of in-depth data; a modem is the key to the previously inaccessible ivory tower of knowledge. You can get weather data and images that you can't get from TV or newspapers, and you can do it when you want rather than waiting for a newscast or newspaper edition. You can also find other travel-related information, like traffic flow and marine conditions online. In short, if someone has information to share, it is likely to be on the Web, and if someone wants information, they can probably find it on the Web. It just takes a little searching     

基于目的分析的作弊页面分类

随着互联网的飞速发展,因网络作弊而产生的垃圾页面越来越多,严重影响了搜索引擎的检索效率和用户体验。反作弊已经成为搜索引擎所面临的最重要挑战之一。但目前的反作弊研究大都是基于页面内容或链接特征的,没有一个通用可行的识别方法。本文主要基于作弊目的的分析,给出作弊页面另一种体系的分类,为基于目的的作弊页面识别起到良好的导向作用。     

基于移动爬虫的专用Web信息收集系统的设计

搜索引擎已经成为网上导航的重要工具。为了能够提供强大的搜索能力,搜索引擎对网上可访问文档维持着详尽的索引。创建和维护索引的任务由网络爬虫完成,网络爬虫代表搜索引擎递归地遍历和下载Web页面。Web页面在下载之后,被搜索引擎分析、建索引,然后提供检索服务。文章介绍了一种更加有效的建立Web索引的方法,该方法是基于移动爬虫(MobileCrawler)的。在此提出的爬虫首先被传送到数据所在的站点,在那里任何不需要的数据在传回搜索引擎之前在当地被过滤。这个方法尤其适用于实施所谓的“智能”爬行算法,这些算法根据已访问过的Web页面的内容来决定一条有效的爬行路径。移动爬虫是移动计算和专业搜索引擎两大技术趋势的结合,能够从技术上很好地解决现在通用搜索引擎所面临的问题。     

A statistical method for estimating the usefulness of text databases

Searching desired data on the Internet is one of the most common ways the Internet is used. No single search engine is capable of searching all data on the Internet. The approach that provides an interface for invoking multiple search engines for each user query has the potential to satisfy more users. When the number of search engines under the interface is large, invoking all search engines for each query is often not cost effective because it creates unnecessary network traffic by sending the query to a large number of useless search engines and searching these useless search engines wastes local resources. The problem can be overcome if the usefulness of every search engine with respect to each query can be predicted. We present a statistical method to estimate the usefulness of a search engine for any given query. For a given query, the usefulness of a search engine in this paper is defined to be a combination of the number of documents in the search engine that are sufficiently similar to the query and the average similarity of these documents. Experimental results indicate that our estimation method is much more accurate than existing methods.