Exploiting concept clusters for content-based information retrieval

Current approaches to index weighting for information retrieval from texts are based on statistical analysis of the texts' contents. A key shortcoming of these indexing schemes, which consider only the terms in a document, is that they cannot extract semantically exact indexes that represent the semantic content of a document. To address this issue, we proposed a new indexing formalism that considers not only the terms in a document, but also the concepts. In the proposed method, concepts are extracted by exploiting clusters of terms that are semantically related, referred to as concept clusters. Through experiments on the TREC-2 collection of Wall Street Journal documents, we show that the proposed method outperforms an indexing method based on term frequency (TF), especially in regard to the highest-ranked documents. Moreover, the index term dimension was 53.3% lower for the proposed method than for the TF-based method, which is expected to significantly reduce the document search time in a real environment.     

Image and video indexing using vector quantization

Visual (image and video) database systems require efficient indexing to enable fast access to the images in a database. In addition, the large memory capacity and channel bandwidth requirements for the storage and transmission of visual data necessitate the use of compression techniques. We note that image/video indexing and compression are typically pursued independently. This reduces the storage efficiency and may degrade the system performance. In this paper, we present novel algorithms based on vector quantization (VQ) for indexing of compressed images and video. To start with, the images are compressed using VQ. In the first technique, for each codeword in the codebook, a histogram is generated and stored along with the codeword. We note that the superposition of the histograms of the codewords, which are used to represent an image, is a close approximation of the histogram of the image. This histogram is used as an index to store and retrieve the image. In the second technique, the histogram of the labels of an image is used as an index to access the image. We also propose an algorithm for indexing compressed video sequences. Here, each frame is encoded in the intraframe mode using VQ. The labels are used for the segmentation of a video sequence into shots, and for indexing the representative frame of each shot. The proposed techniques not only provide fast access to stored visual data, but also combine compression and indexing. The average retrieval rates are 95% and 94% at compression ratios of 16:1 and 64:1, respectively. The corresponding cut detection rates are 97% and 90%, respectively.     

The Indexing and Retrieval of Document Images: A Survey

The economic feasibility of maintaining large data bases of document images has created a tremendous demand for robust ways to access and manipulate the information these images contain. In an attempt to move toward a paperless office, large quantities of printed documents are often scanned and archived as images, without adequate index information. One way to provide traditional data-base indexing and retrieval capabilities is to fully convert the document to an electronic representation which can be indexed automatically. Unfortunately, there are many factors which prohibit complete conversion including high cost, low document quality, and the fact that many nontext components cannot be adequately represented in a converted form. In such cases, it can be advantageous to maintain a copy of and use the document in image form. In this paper, we provide a survey of methods developed by researchers to access and manipulate document images without the need for complete and accurate conversion. We briefly discuss traditional text indexing techniques on imperfect data and the retrieval of partially converted documents. This is followed by a more comprehensive review of techniques for the direct characterization, manipulation, and retrieval, of images of documents containing text, graphics, and scene images.     

Iteration-free clustering algorithm for nonstationary image database

Image database systems must effectively and efficiently handle and retrieve images from a large collection of images. A serious problem faced by these systems is the requirement to deal with the nonstationary database. In an image database system, image features are typically organized into an indexing structure, and updating the indexing structure involves many computations. In this paper, this difficult problem is converted into a constrained optimization problem, and the iteration-free clustering (IFC) algorithm based on the Lagrangian function, is presented for adapting the existing indexing structure for a nonstationary database. Experimental results concerning recall and precision indicate that the proposed method provides a binary tree that is almost optimal. Simulation results further demonstrate that the proposed algorithm can maintain 94% precision in seven-dimensional feature space, even when the number of new-coming images is one-half the number of images in the original database. Finally, our IFC algorithm outperforms other methods usually applied to image databases.     

基于Lucene的PDF文档的全文检索的实现

在Lucene的全文检索中,直接对PDF文档进行全文检索几乎是不可能的。在实际应用中又需要对大量的PDF文档进行检索,通过Xpdf工具先对PDF文档转换为TXT文本,然后对TXT文本建立索引,在进行检索时通过文件名实现和原始PDF文档的一一对应,最终实现PDF文档的全文检索功能,同时还能实现对PDF文档所检索的包含关键词的内容进行高亮显示,实现全文检索的功能,通过实际项目应用,检索效果能够达到很好的效果。     

全文搜索引擎快速搭建的设计与实现

搜索引擎的诞生,给信息搜集带来了极大的方便与好处。一套完备、成熟的搜索引擎的开发需要耗费大量资源,本文围绕如何快速搭建一个简易的搜索引擎展开。基于各开源组织独立研发并对外提供的搜索引擎组件与框架,本文在JBuilder开发平台上调用各组件对外提供的Java API,快速地搭建起由数据抓取、建立索引及执行搜索3大部分组成的简易的全文搜索引擎,实现网页文档类数据的抓取与保存、文本提取、索引文档及索引库的建立、基本关键词的检索等功能,并描述搜索引擎实现及运行的一般过程。     

A new full-text indexing model with low space overhead for chinese text retrieval

Text retrieval systems require an index to allow efficient retrieval of documents at the cost of some storage overhead. This paper proposes a novel full-text indexing model for Chinese text retrieval based on the concept of adjacency matrix of directed graph. Using this indexing model, on one hand, retrieval systems need to keep only the indexing data, instead of the indexing data and the original text data as the traditional retrieval systems always do. On the other hand, occurrences of index term are identified by labels of the so-called s-strings where the index term appears, rather than by its positions as in traditional indexing models. Consequently, system space cost as a whole can be reduced drastically while retrieval efficiency is maintained satisfactory. Experiments over several real-world Chinese text collections are carried out to demonstrate the effectiveness and efficiency of this model. In addition to Chinese, The proposed indexing model is also effective and efficient for text retrieval of other Oriental languages, such as Japanese and Korean. It is especially useful for digital library application areas where storage resource is very limited (e.g., e-books and CD-based text retrieval systems).     

BitCube: A Three-Dimensional Bitmap Indexing for XML Documents

XML is a new standard for exchanging and representing information on the Internet. Documents can be hierarchically represented by XML-elements. In this paper, we propose that an XML document collection be represented and indexed using a bitmap indexing technique. We define the similarity and popularity operations suitable for bitmap indexes. We also define statistical measurements in the BitCube: center, and radius. Based on these measurements, we describe a new bitmap indexing based technique to cluster XML documents. The techniques for clustering are motivated by the fact that the bitmap indexes are expected to be very sparse.Furthermore, a 2-dimensional bitmap index is extended to a 3-dimensional bitmap index, called the BitCube. Sophisticated querying of XML document collections can be performed using primitive operations such as slice, project, and dice. Experiments show that the BitCube can be created efficiently and the primitive operations can be performed more efficiently with the BitCube than with other alternatives.     

A New Approach to Object-Related Image Retrieval

Image retrieval has been commonly attempted using non-semantic approaches. It is clear though, that semantic retrieval is more desirable because it facilitates the user's task. In this paper, we present a new approach to semantic access of a database of images by asking for the presence of certain objects; this is known as object-related image retrieval.This approach is built within a classical computer vision framework (i.e. localization, segmentation and identification). Our approach first searches for the main areas of attention (most salient areas of an image) and then applies appearance-based methods to classify (index) all images by ‘symbolic’ names. These names are referred to objects, which finally allows the use of semantics driven by these object names, e.g. retrieve ‘all those images that have a bull and Melissa's face'.The use of a totally automatic system would cause some errors of indexing (and so retrieval). To solve this we use a human-in-the-loop strategy where a human expert is placed after the two outputs of the system to confirm their ‘correctness’. An experimental result using a database of 3000 images is presented.     

Federating diverse collections of scientific literature

The Digital Library Initiative (DLI) project at the University of Illinois at Urbana-Champaign is developing the information infrastructure to effectively search technical documents on the Internet. The authors are constructing a large testbed of scientific literature, evaluating its effectiveness under significant use, and researching enhanced search technology. They are building repositories (organized collections) of indexed multiple-source collections and federating (merging and mapping) them by searching the material via multiple views of a single virtual collection. Developing widely usable Web technology is also a key goal. Improving Web search beyond full-text retrieval will require using document structure in the short term and document semantics in the long term. Their testbed efforts concentrate on journal articles from the scientific literature, with structure specified by the Standard Generalized Markup Language (SGML). Research efforts extract semantics from documents using the scalable technology of concept spaces based on context frequency. They then merge these efforts with traditional library indexing to provide a single Internet interface to indexes of multiple repositories     

Universal indexes for highly repetitive document collections

Indexing highly repetitive collections has become a relevant problem with the emergence of large repositories of versioned documents, among other applications. These collections may reach huge sizes, but are formed mostly of documents that are near-copies of others. Traditional techniques for indexing these collections fail to properly exploit their regularities in order to reduce space.We introduce new techniques for compressing inverted indexes that exploit this near-copy regularity. They are based on run-length, Lempel–Ziv, or grammar compression of the differential inverted lists, instead of the usual practice of gap-encoding them. We show that, in this highly repetitive setting, our compression methods significantly reduce the space obtained with classical techniques, at the price of moderate slowdowns. Moreover, our best methods are universal, that is, they do not need to know the versioning structure of the collection, nor that a clear versioning structure even exists.We also introduce compressed self-indexes in the comparison. These are designed for general strings (not only natural language texts) and represent the text collection plus the index structure (not an inverted index) in integrated form. We show that these techniques can compress much further, using a small fraction of the space required by our new inverted indexes. Yet, they are orders of magnitude slower.     

Energy- and Latency-Efficient Processing of Full-Text Searches on a Wireless Broadcast Stream

In wireless mobile computing environments, broadcasting is an effective and scalable technique to disseminate information to a massive number of clients, wherein the energy usage and latency are considered major concerns. This paper presents an indexing scheme for the energy- and latency-efficient processing of full-text searches over the wireless broadcast data stream. Although a lot of access methods and index structures have been proposed in the past for full-text searches, all of them are targeted for data in disk storage, not wireless broadcast channels. For full-text searches on a wireless broadcast stream, we firstly introduce a naive, inverted list-style indexing method, where inverted lists are placed in front of the data on the wireless channel. In order to reduce the latency overhead, we propose a two-level indexing method which adds another level of index structure to the basic inverted list-style index. In addition, we propose a replication strategy of the index list and index tree to further improve the latency performance. We analyze the performance of the proposed indexing scheme with respect to the latency and energy usage measures, and show the optimality of index replication. The correctness of the analysis is demonstrated through simulation experiments, and the effectiveness of the proposed scheme is shown by implementing a real wireless information delivery system.     

Font adaptive word indexing of modern printed documents

We propose an approach for the word-level indexing of modern printed documents which are difficult to recognize using current OCR engines. By means of word-level indexing, it is possible to retrieve the position of words in a document, enabling queries involving proximity of terms. Web search engines implement this kind of indexing, allowing users to retrieve Web pages on the basis of their textual content. Nowadays, digital libraries hold collections of digitized documents that can be retrieved either by browsing the document images or relying on appropriate metadata assembled by domain experts. Word indexing tools would therefore increase the access to these collections. The proposed system is designed to index homogeneous document collections by automatically adapting to different languages and font styles without relying on OCR engines for character recognition. The approach is based on three main ideas: the use of self organizing maps (SOM) to perform unsupervised character clustering, the definition of one suitable vector-based word representation whose size depends on the word aspect-ratio, and the run-time alignment of the query word with indexed words to deal with broken and touching characters. The most appropriate applications are for processing modern printed documents (17th to 19th centuries) where current OCR engines are less accurate. Our experimental analysis addresses six data sets containing documents ranging from books of the 17th century to contemporary journals.     

FIRST: Fractal Indexing and Retrieval SysTem for Image Databases

We present an image indexing method and a system to perform content-based retrieval in heterogeneous image databases (IDB). The method is based upon the fractal framework of the iterated function systems (IFS) widely used for image compression. The image index is represented through a vector of numeric features, corresponding to contractive functions (CF) of the IFS framework. The construction of the index vector requires a preliminary processing of the images to select an appropriate set of indexing features (i.e. contractive functions). The latter will be successively used to fill in the vector components, computed as frequencies by which the selected contractive functions appear inside the images. In order to manipulate the index vectors efficiently we use discrete Fourier transform (DFT) to reduce their cardinalities and use a spatial access method (SAM), like R*-tree, to improve search performances. The sound theoretical framework underlying the method enabled us to formally prove some properties of the index. However, for a complete validation of the indexing method, also in terms of effectiveness and efficacy, we performed several experiments on a large collection of images from different domains, which revealed good system performances with a low percentage of false alarms and false dismissals.     

Query-driven indexing for scalable peer-to-peer text retrieval

In this paper, we present a query-driven indexing/retrieval strategy for efficient full text retrieval from large document collections distributed within a structured P2P network. Our indexing strategy is based on two important properties: (1) the generated distributed index stores posting lists for carefully chosen indexing term combinations that are frequently present in user queries, and (2) the posting lists containing too many document references are truncated to a bounded number of their top-ranked elements. These two properties guarantee acceptable latency and bandwidth requirements, essentially because the number of indexing term combinations remains scalable and the posting lists transmitted during retrieval never exceed a constant size. A novel index update mechanism efficiently handles adding of new documents to the document collection. Thus, the generated distributed index corresponds to a constantly evolving query-driven indexing structure that efficiently follows current information needs of the users and changes in the document collection.We show that the size of the index and the generated indexing/retrieval traffic remains manageable even for Web-size document collections at the price of a marginal loss in precision for rare queries. Our theoretical analysis and experimental results provide convincing evidence about the feasibility of the query-driven indexing strategy for large scale P2P text retrieval.     

Oracle Text全文检索技术在文档资料管理中的应用

本文利用Oracle Text全文检索技术,根据数据库业务逻辑构建了关键词表,通过为关键词表建立索引的方式进行检索,提高了检索效率;以ViusalC++6为开发平台,采用C/S结构技术研发了多类型文档资料管理系统,实现了办公文档资料的高效管理.     

Oracle数据库中Clob大字段的查询优化技术研究

在数据量达到几十万的数据库中查询Clob字段,其响应的时间是影响数据库应用的关键技术。为了提高Oracle数据库中Clob大字段的查询速度以满足用户的需求,介绍了数据库参数配置、数据表结构配置、全文索引等常用的优化技术对查询进行优化,接着介绍了笔者在开发一个应用系统中采用的建立一个优化JOB、分区建立表、分区建立表全文索引和Oracle的并行执行等优化技术来进行调整和优化。使查询Clob字段速度得到了明显的提高,最终满足了用户的需要。     

索引时间戳：一种高效的文本索引系统设计方法

特征文件索引、时间戳排序技术是数据库技术研究方面的两个重要课题，前者通常用于支持文本数据的索引和检索操作，后者为实现数据库并发控制的两个基本方法之一。本文主要讨论面向文本数据库管理系统（ＦＩＭＳ）基于索引时间戳概念的文本对象索引模型的形式化描述、检索相关性计算及特征文件系统逻辑设计等问题。