Content-based image retrieval with relevance feedback using random walks

In this paper, we propose a novel approach to content-based image retrieval with relevance feedback, which is based on the random walker algorithm introduced in the context of interactive image segmentation. The idea is to treat the relevant and non-relevant images labeled by the user at every feedback round as “seed” nodes for the random walker problem. The ranking score for each unlabeled image is computed as the probability that a random walker starting from that image will reach a relevant seed before encountering a non-relevant one. Our method is easy to implement, parameter-free and scales well to large datasets. Extensive experiments on different real datasets with several image similarity measures show the superiority of our method over different recent approaches.     

Exploring statistical correlations for image retrieval

Bridging the cognitive gap in image retrieval has been an active research direction in recent years, of which a key challenge is to get enough training data to learn the mapping functions from low-level feature spaces to high-level semantics. In this paper, image regions are classified into two types: key regions representing the main semantic contents and environmental regions representing the contexts. We attempt to leverage the correlations between types of regions to improve the performance of image retrieval. A Context Expansion approach is explored to take advantages of such correlations by expanding the key regions of the queries using highly correlated environmental regions according to an image thesaurus. The thesaurus serves as both a mapping function between image low-level features and concepts and a store of the statistical correlations between different concepts. It is constructed through a data-driven approach which uses Web data (images, their surrounding textual annotations) as training data source to learn the region concepts and to explore the statistical correlations. Experimental results on a database of 10,000 general-purpose images show the effectiveness of our proposed approach in both improving search precision (i.e. filter irrelevant images) and recall (i.e. retrieval relevant images whose context may be varied). Several major factors which have impact on the performance of our approach are also studied.     

An adaptive technique for content-based image retrieval

We discuss an adaptive approach towards Content-Based Image Retrieval. It is based on the Ostensive Model of developing information needs—a special kind of relevance feedback model that learns from implicit user feedback and adds a temporal notion to relevance. The ostensive approach supports content-assisted browsing through visualising the interaction by adding user-selected images to a browsing path, which ends with a set of system recommendations. The suggestions are based on an adaptive query learning scheme, in which the query is learnt from previously selected images. Our approach is an adaptation of the original Ostensive Model based on textual features only, to include content-based features to characterise images. In the proposed scheme textual and colour features are combined using the Dempster-Shafer theory of evidence combination. Results from a user-centred, work-task oriented evaluation show that the ostensive interface is preferred over a traditional interface with manual query facilities. This is due to its ability to adapt to the user's need, its intuitiveness and the fluid way in which it operates. Studying and comparing the nature of the underlying information need, it emerges that our approach elicits changes in the user's need based on the interaction, and is successful in adapting the retrieval to match the changes. In addition, a preliminary study of the retrieval performance of the ostensive relevance feedback scheme shows that it can outperform a standard relevance feedback strategy in terms of image recall in category search.     

Content-based image retrieval using fuzzy perceptual feedback

In this paper, a new framework called fuzzy relevance feedback in interactive content-based image retrieval (CBIR) systems is introduced. Conventional binary labeling scheme in relevance feedback requires a crisp decision to be made on the relevance of the retrieved images. However, it is inflexible as user interpretation of visual content varies with respect to different information needs and perceptual subjectivity. In addition, users tend to learn from the retrieval results to further refine their information requests. It is, therefore, inadequate to describe the user’s fuzzy perception of image similarity with crisp logic. In view of this, we propose a fuzzy relevance feedback approach which enables the user to make a fuzzy judgement. It integrates the user’s fuzzy interpretation of visual content into the notion of relevance feedback. An efficient learning approach is proposed using a fuzzy radial basis function (FRBF) network. The network is constructed based on the user’s feedbacks. The underlying network parameters are optimized by adopting a gradient-descent training strategy due to its computational efficiency. Experimental results using a database of 10,000 images demonstrate the effectiveness of the proposed method.

Speed up interactive image retrieval

In multimedia retrieval, a query is typically interactively refined towards the “optimal” answers by exploiting user feedback. However, in existing work, in each iteration, the refined query is re-evaluated. This is not only inefficient but fails to exploit the answers that may be common between iterations. Furthermore, it may also take too many iterations to get the “optimal” answers. In this paper, we introduce a new approach called OptRFS (optimizing relevance feedback search by query prediction) for iterative relevance feedback search. OptRFS aims to take users to view the “optimal” results as fast as possible. It optimizes relevance feedback search by both shortening the searching time during each iteration and reducing the number of iterations. OptRFS predicts the potential candidates for the next iteration and maintains this small set for efficient sequential scan. By doing so, repeated candidate accesses (i.e., random accesses) can be saved, hence reducing the searching time for the next iteration. In addition, efficient scan on the overlap before the next search starts also tightens the search space with smaller pruning radius. As a step forward, OptRFS also predicts the “optimal” query, which corresponds to “optimal” answers, based on the early executed iterations’ queries. By doing so, some intermediate iterations can be saved, hence reducing the total number of iterations. By taking the correlations among the early executed iterations into consideration, OptRFS investigates linear regression, exponential smoothing and linear exponential smoothing to predict the next refined query so as to decide the overlap of candidates between two consecutive iterations. Considering the special features of relevance feedback, OptRFS further introduces adaptive linear exponential smoothing to self-adjust the parameters for more accurate prediction. We implemented OptRFS and our experimental study on real life data sets show that it can reduce the total cost of relevance feedback search significantly. Some interesting features of relevance feedback search are also discovered and discussed.     

HIRBIR: A hierarchical approach to region-based image retrieval

This paper proposes a hierarchical approach to region-based image retrieval (HIRBIR) based on wavelet transform whose decomposition property is similar to human visual processing. First, automated image segmentation is performed fast in the low-low (LL) frequency subband of the wavelet domain that shows the desirable low image resolution. In the proposed system, boundaries between segmented regions are deleted to improve the robustness of region-based image retrieval against segmentation-related uncertainty. Second, a region feature vector is hierarchically represented by information in all wavelet subbands, and each feature component of a feature vector is a unified color–texture feature. Such a feature vector captures well the distinctive features (e.g., semantic texture) inside one region. Finally, employing a hierarchical feature vector, the weighted distance function for region matching is tuned meaningfully and easily, and a progressive stepwise indexing mechanism with relevance feedback is performed naturally and effectively in our system. Through experimental results and comparison with other methods, the proposed HIRBIR shows a good tradeoff between retrieval effectiveness and efficiency as well as easy implementation for region-based image retrieval.     

Content-based image retrieval using growing hierarchical self-organizing quadtree map

In this paper, a growing hierarchical self-organizing quadtree map (GHSOQM) is proposed and used for a content-based image retrieval (CBIR) system. The incorporation of GHSOQM in a CBIR system organizes images in a hierarchical structure. The retrieval time by GHSOQM is less than that by using direct image comparison using a flat structure. Furthermore, the ability of incremental learning enables GHSOQM to be a prospective neural-network-based approach for CBIR systems. We also propose feature matrices, image distance and relevance feedback for region-based images in the GHSOQM-based CBIR system. Experimental results strongly demonstrate the effectiveness of the proposed system.     

Collaborative image retrieval via regularized metric learning

In content-based image retrieval (CBIR), relevant images are identified based on their similarities to query images. Most CBIR algorithms are hindered by the semantic gap between the low-level image features used for computing image similarity and the high-level semantic concepts conveyed in images. One way to reduce the semantic gap is to utilize the log data of users' feedback that has been collected by CBIR systems in history, which is also called “collaborative image retrieval.” In this paper, we present a novel metric learning approach, named “regularized metric learning,” for collaborative image retrieval, which learns a distance metric by exploring the correlation between low-level image features and the log data of users' relevance judgments. Compared to the previous research, a regularization mechanism is used in our algorithm to effectively prevent overfitting. Meanwhile, we formulate the proposed learning algorithm into a semidefinite programming problem, which can be solved very efficiently by existing software packages and is scalable to the size of log data. An extensive set of experiments has been conducted to show that the new algorithm can substantially improve the retrieval accuracy of a baseline CBIR system using Euclidean distance metric, even with a modest amount of log data. The experiment also indicates that the new algorithm is more effective and more efficient than two alternative algorithms, which exploit log data for image retrieval.     

Integrated probability function and its application to content-based image retrieval by relevance feedback

In the last few years, we have seen an upsurge of interest in content-based image retrieval (CBIR)—the selection of images from a collection via features extracted from images themselves. Often, a single image attribute may not have enough discriminative information for successful retrieval. On the other hand when multiple features are used, it is hard to determine the suitable weighing factors for various features for optimal retrieval. In this paper, we present a relevance feedback framework with Integrated Probability Function (IPF) which combines multiple features for optimal retrieval. The IPF is based on a new posterior probability estimator and a novel weight updating approach. We perform experiments on 1400 monochromatic trademark images have been performed. The proposed IPF is shown to be more effective and efficient to retrieve deformed trademark images than the commonly used integrated dissimilarity function. The new posterior probability estimator is shown to be generally better than the existing one. The proposed novel weight updating approach by relevance feedback is shown to be better than both the existing scoring approach and the existing ratio approach. In experiments, 95% of the targets are ranked at the top five positions. By two iterations of relevance feedback, retrieval performance can be improved from 75% to over 95%. The IPF and its relevance feedback framework proposed in this paper can be effectively and efficiently used in content-based image retrieval.     

Web-based image indexing and retrieval in JPEG compressed domain

As the majority of content-based image retrieval systems operate on full images in pixel domain, decompression is a prerequisite for the retrieval of compressed images. To provide a possible on-line indexing and retrieval technique for those jpg image files, we propose a novel pseudo-pixel extraction algorithm to bridge the gap between the existing image indexing technology, developed in the pixel domain, and the fact that an increasing number of images stored on the Web are already compressed by JPEG at the source. Further, we describe our Web-based image retrieval system, WEBimager, by using the proposed algorithm to provide a prototype visual information system toward automatic management, indexing, and retrieval of compressed images available on the Internet. This provides users with efficient tools to search the Web for compressed images and establish a database or a collection of special images to their interests. Experiments using texture- and colour-based indexing techniques support the idea that the proposed algorithm achieves significantly better results in terms of computing cost than their full decompression or partial decompression counterparts. This technology will help control the explosion of media-rich content by offering users a powerful automated image indexing and retrieval tool for compressed images on the Web.J. Jiang: Contacting author     

Color indexing for efficient image retrieval

Content based image retrieval is an active area of research. Many approaches have been proposed to retrieve images based on matching of some features derived from the image content. Color is an important feature of image content. The problem with many traditional matching-based retrieval methods is that the search time for retrieving similar images for a given query image increases linearly with the size of the image database. We present an efficient color indexing scheme for similarity-based retrieval which has a search time that increases logarithmically with the database size.In our approach, the color features are extracted automatically using a color clustering algorithm. Then the cluster centroids are used as representatives of the images in 3-dimensional color space and are indexed using a spatial indexing method that usesR-tree. The worst case search time complexity of this approach isOn _q log(N^* n_avg)), whereN is the number of images in the database, andn _q andn _avg are the number of colors in the query image and the average number of colors per image in the database respectively. We present the experimental results for the proposed approach on two databases consisting of 337 Trademark images and 200 Flag images.     

Independent query refinement and feature re-weighting using positive and negative examples for content-based image retrieval

Query refinement and feature re-weighting are the two core techniques underlying the relevance feedback of content-based image retrieval. Most existing relevance feedback mechanisms generally model the user’s query target with a single query point and weight each extracted feature with a single importance factor. A designed estimation procedure then estimates the best query point and all importance factors by optimizing a formulated criterion which measures the goodness of the estimation. This formulated criterion simultaneously encapsulates all positive and negative examples supplied from the user’s feedback. Under such formulation, the positive and negative examples may contribute contradictorily to the criterion and sometimes may introduce higher difficulty in attaining a good estimation. In this paper, we propose a different statistical formulation to estimate independently two pairs of query points and feature weights from the positive examples and negative examples, respectively. These two pairs then define the likelihood ratio, a criterion term used to rank the relevance of all database images. This approach simplifies the criterion formulation and also avoids the mutual impeditive influence between positive examples and negative examples. The experimental results demonstrate that the proposed approach outperforms some other related approaches.

Image retrieval model based on weighted visual features determined by relevance feedback

An accurate and rapid method is required to retrieve the overwhelming majority of digital images. To date, image retrieval methods include content-based retrieval and keyword-based retrieval, the former utilizing visual features such as color and brightness, and the latter utilizing keywords that describe the image. However, the effectiveness of these methods in providing the exact images the user wants has been under scrutiny. Hence, many researchers have been working on relevance feedback, a process in which responses from the user are given as feedback during the retrieval session in order to define a user’s need and provide an improved result. Methods that employ relevance feedback, however, do have drawbacks because several pieces of feedback are necessary to produce an appropriate result, and the feedback information cannot be reused. In this paper, a novel retrieval model is proposed, which annotates an image with keywords and modifies the confidence level of the keywords in response to the user’s feedback. In the proposed model, not only the images that have been given feedback, but also other images with visual features similar to the features used to distinguish the positive images are subjected to confidence modification. This allows for modification of a large number of images with relatively little feedback, ultimately leading to faster and more accurate retrieval results. An experiment was performed to verify the effectiveness of the proposed model, and the result demonstrated a rapid increase in recall and precision using the same amount of feedback.     

基于内容的图像检索中相关反馈算法综述

1 引言从70年代开始,人们就进行图像检索技术的研究,那时候主要工作是对图像进行人工标注,然后应用文本检索技术实现对标注图像的检索。然而图像内容是复杂多样的,不同人对图像的理解不完全一致,一般难用准确的词汇描述图像内容;另外,对大量图像做标注也是一项费时费力的工作,这些因素导致基于文本标注的图像检索方法在实际应用中很难达到预期效果。进入90年代,全球网络通讯技术和光盘等大容量信息存储技术迅速发展,图像、视频和语音等多媒体信息的数据量急剧增加,如何从这些海量的多媒体信息中寻找感兴趣的内容已成为目前急需解决的、有挑战性的难题。近几年来,研究人员企图用基于内容的信息检索技术解决这个难题。基本路线是自动或半自动提取多媒体对象的底层特征,对图像而言可以提取颜色直方图、主色调、形状、纹理等特征,然后基于这些特征定义图像之间的相似度,从而实现基于内容或特征的检索。