Combining intra-image and inter-class semantics for consumer image retrieval

Unconstrained consumer photos pose great challenge for content-based image retrieval. Unlike professional images or domain-specific images, consumer photos vary significantly. More often than not, the objects in the photos are ill-posed, occluded, and cluttered with poor lighting, focus and exposure. In this paper, we propose a cascading framework for combining intra-image and inter-class similarities in image retrieval, motivated from probabilistic Bayesian principles. Support vector machines are employed to learn local view-based semantics based on just-in-time fusion of color and texture features. A new detection-driven block-based segmentation algorithm is designed to extract semantic features from images. The detection-based indexes also serve as input for support vector learning of image classifiers to generate class-relative indexes. During image retrieval, both intra-image and inter-class similarities are combined to rank images. Experiments using query-by-example on 2400 genuine heterogeneous consumer photos with 16 semantic queries show that the combined matching approach is better than matching with single index. It also outperformed the method of combining color and texture features by 55% in average precision.     

A formal framework for image indexing with triples: Toward a concept-based image retrieval

This paper provides a formal specification for concept-based image retrieval using triples. To effectively manage a vast amount of images, we may need an image retrieval system capable of indexing and searching images based on the characteristics of their content. However, such a content-based image retrieval technique alone may not satisfy user queries if retrieved images turn out to be relevant only when they are conceptually related with the queries. In this paper, we develop an image retrieval mechanism to extract semantics of images based on triples. The semantics can be captured by deriving concepts from its constituent objects and spatial relationships between them. The concepts are basically composite objects formed from the aggregation of the constituents. In our mechanism, all the spatial relationships between objects including the concepts are uniformly represented by triples, which are used for indexing images as well as capturing their semantics. We also develop a query evaluation for supporting the concept-based image retrieval. ©1999 John Wiley & Sons, Inc.     

Building Visual Vocabulary for Image Indexation and Query Formulation

With advances in digital imaging, the amount of digital images will increase tremendously. To locate relevant images in a large collection of images presents a challenging and genuine problem for content-based retrieval research. This paper presents a novel framework called visual keywords for image indexation and query formulation. Visual keywords are flexible and intuitive visual prototypes specified perceptually from sample domain images. A visual content is described and indexed by flexible spatial aggregation of the soft presence of visual keywords. A new query method based on visual constraints is also proposed to allow direct and explicit content specification. Last but not least, we have developed a digital album prototype to demonstrate query and retrieval on both home photos and stock photos based on visual keywords.     

SemQuery: semantic clustering and querying on heterogeneous features for visual data

The effectiveness of content-based image retrieval can be enhanced using heterogeneous features embedded in the images. However, since the features in texture, color, and shape are generated using different computation methods and thus may require different similarity measurements, the integration of the retrievals on heterogeneous features is a nontrivial task. We present a semantics-based clustering and indexing approach, termed SemQuery, to support visual queries on heterogeneous features of images. Using this approach, the database images are classified based on their heterogeneous features. Each semantic image cluster contains a set of subclusters that are represented by the heterogeneous features that the images contain. An image is included in a semantic cluster if it falls within the scope of all the heterogeneous clusters of the semantic cluster. We also design a neural network model to merge the results of basic queries on individual features. A query processing strategy is then presented to support visual queries on heterogeneous features. An experimental analysis is conducted and presented to demonstrate the effectiveness and efficiency of the proposed approach.     

基于内容的图象检索中的语义处理方法

基于内容的图象检索系统,其目标是最大限度地减小图象简单视觉特征与用户检索丰富语义之间的“语义鸿沟”,因此图象语义处理则成为基于内容的图象检索进一步发展的关键。为了使人们对基于内容的图象检索中的语义处理方法有个概略了解,首先从图象语义模型和图象语义提取方法这两个方面对利用语义进行图象检索的研究状况进行了总结,并将图象语义模型概括为图象语义知识、图象语义层次模型和语义抽取模型等3个主要组成部分;然后将图象语义提取方法分为用户交互、将查询请求作为语义模板、对象及其空间关系、场景和行为语义及情感语义等类别,同时对其中有代表性的方法进行了详细的分析,还指出了其局限性;最后从对象建模和识别、语义抽取规则和用户检索模型3个方面,阐明了实现图象语义处理所面临的问题,并提出了一些初步的解决思路。     

Association and content-based retrieval

In spite of important efforts in content-based indexing and retrieval during these last years, seeking relevant and accurate images remains a very difficult query. In the state-of-the-art approaches, the retrieval task may be efficient for some queries in which the semantic content of the query can be easily translated into visual features. For example, finding images of fires is simple because fires are characterized by specific colors (yellow and red). However, it is not efficient in other application fields in which the semantic content of the query is not easily translated into visual features. For example, finding images of birds during migrations is not easy because the system has to understand the query semantic. In the query, the basic visual features may be useful (a bird is characterized by a texture and a color), but they are not sufficient. What is missing is the generalization capability. Birds during migrations belong to the same repository of birds, so they share common associations among basic features (e.g., textures and colors) that the user cannot specify explicitly. We present an approach that discovers hidden associations among features during image indexing. These associations discriminate image repositories. The best associations are selected on the basis of measures of confidence. To reduce the combinatory explosion of associations, because images of the database contain very large numbers of colors and textures, we consider a visual dictionary that group together similar colors and textures.     

Content-based image retrieval by hierarchical linear subspace method

We describe a hierarchical linear subspace method to query large on-line image databases using image similarity as the basis of the queries. The method is based on the generic multimedia indexing (GEMINI) approach which is used in the IBM query through the image content search system. Our approach is demonstrated on image indexing, in which the subspaces correspond to different resolutions of the images. During content-based image retrieval, the search starts in the subspace with the lowest resolution of the images. In this subspace, the set of all possible similar images is determined. In the next subspace, additional metric information corresponding to a higher resolution is used to reduce this set. This procedure is repeated until the similar images can be determined. For evaluation we used three image databases and two different subspace sequences.     

Integrated spatial and feature image query

We present a new system for querying for images by regions and their spatial and feature attributes. The system enables the user to find the images that contain arrangements of regions similar to those diagrammed in a query image. By indexing the attributes of regions, such as sizes, locations and visual features, a wide variety of complex joint spatial and feature queries are efficiently computed. In order to demonstrate the utility of the system, we develop a process for the extracting color regions from photographic images. We demonstrate that integrated spatial and feature querying using color regions improves image search capabilities over non-spatial content-based image retrieval methods.     

查询子空间在图像检索中的应用

近几年来,为了解决图像检索系统中由底层视觉特征和高层语义之间的差异所造成的检索困难,从信息捡索中引入了相关反馈技术。在过去几年中,它在该研究领域取得了一定的成功。文章提出了一种利用反馈信息建立“查询子空间”的检索模型,它将用户的语义查询进行基于视觉特征的分类,构造多个“查询子空间”,这些子空间拥有自身的查询模型和检索模型,最后的检索结果根据这多个“查询子空间”的检索结果得到。该模型具有较强的灵活性、扩展性,有效地利用了用户的反馈信息,动态地建立了底层视觉特征和高层语义之间的映射,能适应不同用户的查询。同时,将负反馈信息合理地融入到该模型中,提高了系统的检索效率。实验结果表明采用该检索模型的系统相比现有的检索系统性能有了较大提高。     

Mental image search by boolean composition of region categories

Existing content-based image retrieval paradigms almost never address the problem of starting the search, when the user has no starting example image but rather a mental image. We propose a new image retrieval system to allow the user to perform mental image search by formulating boolean composition of region categories. The query interface is a region photometric thesaurus which can be viewed as a visual summary of salient regions available in the database. It is generated from the unsupervised clustering of regions with similar visual content into categories. In this thesaurus, the user simply selects the types of regions which should and should not be present in the mental image (boolean composition). The natural use of inverted tables on the region category labels enables powerful boolean search and very fast retrieval in large image databases. The process of query and search of images relates to that of documents with Google. The indexing scheme is fully unsupervised and the query mode requires minimal user interaction (no example image to provide, no sketch to draw). We demonstrate the feasibility of such a framework to reach the user mental target image with two applications: a photo-agency scenario on Corel Photostock and a TV news scenario. Perspectives will be proposed for this simple and innovative framework, which should motivate further development in various research areas.

Deep Root Memory Optimized Indexing Methodology for Image Search Engines

Digitization has created an abundance of new information sources by altering how pictures are captured. Accessing large image databases from a web portal requires an opted indexing structure instead of reducing the contents of different kinds of databases for quick processing. This approach paves a path toward the increase of efficient image retrieval techniques and numerous research in image indexing involving large image datasets. Image retrieval usually encounters difficulties like a) merging the diverse representations of images and their Indexing, b) the low-level visual characters and semantic characters associated with an image are indirectly proportional, and c) noisy and less accurate extraction of image information (semantic and predicted attributes). This work clearly focuses and takes the base of reverse engineering and de-normalizing concept by evaluating how data can be stored effectively. Thus, retrieval becomes straightforward and rapid. This research also deals with deep root indexing with a multi-dimensional approach about how images can be indexed and provides improved results in terms of good performance in query processing and the reduction of maintenance and storage cost. We focus on the schema design on a non-clustered index solution, especially cover queries. This schema provides a filter predication to make an index with a particular content of rows and an index table called filtered indexing. Finally, we include non-key columns in addition to the key columns. Experiments on two image data sets ‘with and without’ filtered indexing show low query cost. We compare efficiency as regards accuracy in mean average precision to measure the accuracy of retrieval with the developed coherent semantic indexing. The results show that retrieval by using deep root indexing is simple and fast.     

Colour Image Indexing Using SOM for Region-of-Interest Retrieval

In this paper, we present a novel approach to image indexing by incorporating a neural network model, Kohonen’s Self-Organising Map (SOM), for content-based image retrieval. The motivation stems from the idea of finding images by regarding users’ specifications or requirements imposed on the query, which has been ignored in most existing image retrieval systems. An important and unique aspect of our interactive scheme is to allow the user to select a Region-Of-Interest (ROI) from the sample image, and subsequent query concentrates on matching the regional colour features to find images containing similar regions as indicated by the user. The SOM algorithm is capable of adaptively partitioning each image into several homogeneous regions for representing and indexing the image. This is achieved by unsupervised clustering and classification of pixel-level features, called Local Neighbourhood Histograms (LNH), without a priori knowledge about the data distribution in the feature space. The indexes generated from the resultant prototypes of SOM learning demonstrate fairly good performance over an experimental image database, and therefore suggest the effectiveness and significant potential of our proposed indexing and retrieval strategy for application to content-based image retrieval. Receiveed: 4 June 1998?,Received in revised form: 7 January 1999?Accepted: 7 January 1999     

Adaptive image segmentation for region-based object retrieval using generalized Hough transform

Finding an object inside a target image by querying multimedia data is desirable, but remains a challenge. The effectiveness of region-based representation for content-based image retrieval is extensively studied in the literature. One common weakness of region-based approaches is that perform detection using low level visual features within the region and the homogeneous image regions have little correspondence to the semantic objects. Thus, the retrieval results are often far from satisfactory. In addition, the performance is significantly affected by consistency in the segmented regions of the target object from the query and database images. Instead of solving these problems independently, this paper proposes region-based object retrieval using the generalized Hough transform (GHT) and adaptive image segmentation. The proposed approach has two phases. First, a learning phase identifies and stores stable parameters for segmenting each database image. In the retrieval phase, the adaptive image segmentation process is also performed to segment a query image into regions for retrieving visual objects inside database images through the GHT with a modified voting scheme to locate the target visual object under a certain affine transformation. The learned parameters make the segmentation results of query and database images more stable and consistent. Computer simulation results show that the proposed method gives good performance in terms of retrieval accuracy, robustness, and execution speed.     

Image Indexing and Retrieval Using Object-based Point Feature Maps

Multimedia data such as audios, images, and videos are semantically richer than standard alphanumeric data. Because of the nature of images as combinations of objects, content-based image retrieval should allow users to query by image objects with finer granularity than a whole image. In this paper, we address a web-based object-based image retrieval (OBIR) system . Its prototype implementation particularly explores image indexing and retrieval using object-based point feature maps. An important contribution of this work is its ability to allow a user to easily incorporate both low- and high-level semantics into an image query. This is accomplished through the inclusion of the spatial distribution of point-based image object features, the spatial distribution of the image objects themselves, and image object class identifiers. We introduce a generic image model, give our ideas on how to represent the low- and high-level semantics of an image object, discuss our notion of image object similarity, and define four types of image queries supported by the OBIR system. We also propose an application of our approach to neurological surgery training.     

A survey of content-based image retrieval with high-level semantics

In order to improve the retrieval accuracy of content-based image retrieval systems, research focus has been shifted from designing sophisticated low-level feature extraction algorithms to reducing the ‘semantic gap’ between the visual features and the richness of human semantics. This paper attempts to provide a comprehensive survey of the recent technical achievements in high-level semantic-based image retrieval. Major recent publications are included in this survey covering different aspects of the research in this area, including low-level image feature extraction, similarity measurement, and deriving high-level semantic features. We identify five major categories of the state-of-the-art techniques in narrowing down the ‘semantic gap’: (1) using object ontology to define high-level concepts; (2) using machine learning methods to associate low-level features with query concepts; (3) using relevance feedback to learn users’ intention; (4) generating semantic template to support high-level image retrieval; (5) fusing the evidences from HTML text and the visual content of images for WWW image retrieval. In addition, some other related issues such as image test bed and retrieval performance evaluation are also discussed. Finally, based on existing technology and the demand from real-world applications, a few promising future research directions are suggested.     

A multi-agent platform for content-based image retrieval

Efficient and possibly intelligent image retrieval is an important task, often required in many fields of human activity. While traditional database indexing techniques exhibit a remarkable performance in textual information retrieval current research in content-based image retrieval is focused on developing novel techniques that are biologically motivated and efficient. It is well known that humans have a remarkable ability to process visual information and to handle the volume and complexity of such information quite efficiently. In this paper, we present a content-based image retrieval platform that is based on a multi-agent architecture. Each agent is responsible for assessing the similarity of the query image to each candidate image contained in a collection based on a specific primitive feature and a corresponding similarity criterion. The outputs of various agents are integrated using one of several voting schemes supported by the system. The system’s performance has been evaluated using various collections of images, as well as images obtained in specific application domains such as medical imaging. The initial evaluation has yielded very promising results.

Web-based image indexing and retrieval in JPEG compressed domain

As the majority of content-based image retrieval systems operate on full images in pixel domain, decompression is a prerequisite for the retrieval of compressed images. To provide a possible on-line indexing and retrieval technique for those jpg image files, we propose a novel pseudo-pixel extraction algorithm to bridge the gap between the existing image indexing technology, developed in the pixel domain, and the fact that an increasing number of images stored on the Web are already compressed by JPEG at the source. Further, we describe our Web-based image retrieval system, WEBimager, by using the proposed algorithm to provide a prototype visual information system toward automatic management, indexing, and retrieval of compressed images available on the Internet. This provides users with efficient tools to search the Web for compressed images and establish a database or a collection of special images to their interests. Experiments using texture- and colour-based indexing techniques support the idea that the proposed algorithm achieves significantly better results in terms of computing cost than their full decompression or partial decompression counterparts. This technology will help control the explosion of media-rich content by offering users a powerful automated image indexing and retrieval tool for compressed images on the Web.J. Jiang: Contacting author     

Indexing chromatic and achromatic patterns for content-based colour image retrieval

In this paper, we present a method to represent achromatic and chromatic image signals independently for content-based image indexing and retrieval for image database applications. Starting from an opponent colour representation, human colour vision theories and modern digital signal processing technologies are applied to develop a compact and computationally efficient visual appearance model for coloured image patterns. We use the model to compute the statistics of achromatic and chromatic spatial patterns of colour images for indexing and content-based retrieval. Two types of colour images databases, one colour texture database and another photography colour image database are used to evaluate the performance of the developed method in content-based image indexing and retrieval. Experimental results are presented to show that the new method is superior or competitive to state-of-the-art content-based image indexing and retrieval techniques.