Relevance Feedback and Learning in Content-Based Image Search

A major bottleneck in content-based image retrieval (CBIR) systems or search engines is the large gap between low-level image features used to index images and high-level semantic contents of images. One solution to this bottleneck is to apply relevance feedback to refine the query or similarity measures in image search process. In this paper, we first address the key issues involved in relevance feedback of CBIR systems and present a brief overview of a set of commonly used relevance feedback algorithms. Almost all of the previously proposed methods fall well into such framework. We present a framework of relevance feedback and semantic learning in CBIR. In this framework, low-level features and keyword annotations are integrated in image retrieval and in feedback processes to improve the retrieval performance. We have also extended framework to a content-based web image search engine in which hosting web pages are used to collect relevant annotations for images and users' feedback logs are used to refine annotations. A prototype system has developed to evaluate our proposed schemes, and our experimental results indicated that our approach outperforms traditional CBIR system and relevance feedback approaches.     

Query-by-Gesture: An Alternative Content-Based Image Retrieval Query Scheme

Current research on content-based image retrieval (CBIR) is centered on designing efficient query schemes in order to provide a user with effective mechanisms for image database search. Among representative CBIR query schemes, query-by-sketch has been one of the attractive query tools that are highly adaptive to user's subjectivity. However, query-by-sketch has a few limitations. That is, most sketch tools demand expertise in image processing or computer vision of the user to provide good enough sketches that can be used as query. Furthermore, sketching the exact shape of an object using a mouse can be a burden on the user. To overcome some of the limitations associated with query-by-sketch, we propose a new query method for CBIR, query-by-gesture, that does not require sketches, thereby minimizing user interaction. In our system, the user does not need to use a mouse to make a sketch. Instead, the user draws the shape of the object that heshe intends to search in front of a camera by hand. In addition, our query-by-gesture technique uses relevance feedback to interactively improve retrieval performance and allow progressive refinement of query results according to the user's specification. The efficacy of our proposed method is validated using images from the Corel-Photo CD.     

An image retrieval system with automatic query modification

Most interactive "query-by-example" based image retrieval systems utilize relevance feedback from the user for bridging the gap between the user's implied concept and the low-level image representation in the database. However, traditional relevance feedback usage in the context of content-based image retrieval (CBIR) may not be very efficient due to a significant overhead in database search and image download time in client-server environments. In this paper, we propose a CBIR system that efficiently addresses the inherent subjectivity in user perception during a retrieval session by employing a novel idea of intra-query modification and learning. The proposed system generates an object-level view of the query image using a new color segmentation technique. Color, shape and spatial features of individual segments are used for image representation and retrieval. The proposed system automatically generates a set of modifications by manipulating the features of the query segment(s). An initial estimate of user perception is learned from the user feedback provided on the set of modified images. This largely improves the precision in the first database search itself and alleviates the overheads of database search and image download. Precision-to-recall ratio is improved in further iterations through a new relevance feedback technique that utilizes both positive as well as negative examples. Extensive experiments have been conducted to demonstrate the feasibility and advantages of the proposed system.     

一种结合多示例学习的图像检索方法①

提出一种基于多示例学习（Multiple—instance learning）的图像检索方法,将多示例学习应用于图像检索中,以有效的处理图像的歧义性。该方法首先将图像作为多示例包,其次采用自适应k—means图像分割算法将图像自动分成多个示例,然后根据用户选择的实例图像生成正包和反包,再采用EM—DD（expectation maximization diversedensity）算法进行多示例学习,实现图像检索和相关反馈,最终使用户得到比较满意的结果。     

Mental image search by boolean composition of region categories

Existing content-based image retrieval paradigms almost never address the problem of starting the search, when the user has no starting example image but rather a mental image. We propose a new image retrieval system to allow the user to perform mental image search by formulating boolean composition of region categories. The query interface is a region photometric thesaurus which can be viewed as a visual summary of salient regions available in the database. It is generated from the unsupervised clustering of regions with similar visual content into categories. In this thesaurus, the user simply selects the types of regions which should and should not be present in the mental image (boolean composition). The natural use of inverted tables on the region category labels enables powerful boolean search and very fast retrieval in large image databases. The process of query and search of images relates to that of documents with Google. The indexing scheme is fully unsupervised and the query mode requires minimal user interaction (no example image to provide, no sketch to draw). We demonstrate the feasibility of such a framework to reach the user mental target image with two applications: a photo-agency scenario on Corel Photostock and a TV news scenario. Perspectives will be proposed for this simple and innovative framework, which should motivate further development in various research areas.

Active learning paradigms for CBIR systems based on optimum-path forest classification

This paper discusses methods for content-based image retrieval (CBIR) systems based on relevance feedback according to two active learning paradigms, named greedy and planned. In greedy methods, the system aims to return the most relevant images for a query at each iteration. In planned methods, the most informative images are returned during a few iterations and the most relevant ones are only presented afterward. In the past, we proposed a greedy approach based on optimum-path forest classification (OPF) and demonstrated its gain in effectiveness with respect to a planned method based on support-vector machines and another greedy approach based on multi-point query. In this work, we introduce a planned approach based on the OPF classifier and demonstrate its gain in effectiveness over all methods above using more image databases. In our tests, the most informative images are better obtained from images that are classified as relevant, which differs from the original definition. The results also indicate that both OPF-based methods require less user involvement (efficiency) to satisfy the user's expectation (effectiveness), and provide interactive response times.     

A new relevance feedback technique for iconic image retrieval based on spatial relationships

Due to the popularity of Internet and the growing demand of image access, the volume of image databases is exploding. Hence, we need a more efficient and effective image searching technology. Relevance feedback technique has been popularly used with content-based image retrieval (CBIR) to improve the precision performance, however, it has never been used with the retrieval systems based on spatial relationships. Hence, we propose a new relevance feedback framework to deal with spatial relationships represented by a specific data structure, called the 2D B_e-string. The notions of relevance estimation and query reformulation are embodied in our method to exploit the relevance knowledge. The irrelevance information is collected in an irrelevant set to rule out undesired pictures and to expedite the convergence speed of relevance feedback. Our system not only handles picture-based relevance feedback, but also deals with region-based feedback mechanism, such that the efficacy and effectiveness of our retrieval system are both satisfactory.     

RETIN: A Content-Based Image Indexing and Retrieval System

This paper presents RETIN, a new system for automatic image indexing and interactive content-based image retrieval. The most original aspect of our work rests on the distance computation and its adjustment by relevance feedback. First, during an offline stage, the indexes are computed from attribute vectors associated with image pixels. The feature spaces are partitioned through an unsupervised classification, and then, thanks to these partitions, statistical distributions are processed for each image. During the online use of the system, the user makes an iconic request, i.e. he brings an example of the type of image he is looking for. The query may be global or partial, since the user can reduce his request to a region of interest. The comparison between the query distribution and that of every image in the collection is carried out by using a weighted dissimilarity function which manages the use of several attributes. The results of the search are then refined by means of relevance feedback, which tunes the weights of the dissimilarity measure via user interaction. Experiments are then performed on large databases and statistical quality assessment shows the good properties of RETIN for digital image retrieval. The evaluation also shows that relevance feedback brings flexibility and robustness to the search.     

Content-based image retrieval based on a fuzzy approach

A typical content-based image retrieval (CBIR) system would need to handle the vagueness in the user queries as well as the inherent uncertainty in image representation, similarity measure, and relevance feedback. We discuss how fuzzy set theory can be effectively used for this purpose and describe an image retrieval system called FIRST (fuzzy image retrieval system) which incorporates many of these ideas. FIRST can handle exemplar-based, graphical-sketch-based, as well as linguistic queries involving region labels, attributes, and spatial relations. FIRST uses fuzzy attributed relational graphs (FARGs) to represent images, where each node in the graph represents an image region and each edge represents a relation between two regions. The given query is converted to a FARG, and a low-complexity fuzzy graph matching algorithm is used to compare the query graph with the FARGs in the database. The use of an indexing scheme based on a leader clustering algorithm avoids an exhaustive search of the FARG database. We quantify the retrieval performance of the system in terms of several standard measures.     

Content-based image retrieval by using tree-structured features and multi-layer self-organizing map

A new approach for content-based image retrieval (CBIR) is described. In this study, a tree-structured image representation together with a multi-layer self-organizing map (MLSOM) is proposed for efficient image retrieval. In the proposed tree-structured image representation, a root node contains the global features, while child nodes contain the local region-based features. This approach hierarchically integrates more information of image contents to achieve better retrieval accuracy compared with global and region features individually. MLSOM in the proposed method provides effective compression and organization of tree-structured image data. This enables the retrieval system to operate at a much faster rate than that of directly comparing query images with all images in databases. The proposed method also adopts a relevance feedback scheme to improve the retrieval accuracy by a respectable level. Our obtained results indicate that the proposed image retrieval system is robust against different types of image alterations. Comparative results corroborate that the proposed CBIR system is promising in terms of accuracy, speed and robustness.     

Rich Get Richer——图像检索中的一种自适应的相关反馈方法

早期的基于内容的图像检索系统以图像处理技术为核心,研究重点集中在视觉特征的选择和提取方面,而没有充分利用人们在视觉方面的主观性和人类所广泛使用的高层次概念和低层次视觉特征之间的相关性。为解决上述问题,近年来相关反馈在基于内容的图像检索中受到重视。提出了一种新的相关反馈方法,使得高层次语义特征能够逐步嵌入到基于 低层次特征的图像检索中,该方法不仅能够记忆以前的交互信息,而且能够记忆相应的交互信息给系统带来的影响,实验结果表明该方法准确率高、响应速度快。     

Learning from User Behavior in Image Retrieval: Application of Market Basket Analysis

This article describes an approach to learn feature weights for content-based image retrieval (CBIR) from user interaction log files. These usage log files are analyzed for images marked together by a user in the same query step. The problem is somewhat similar to one of the traditional data mining problems, the market basket analysis problem, where items bought together in a supermarket are analyzed. This paper outlines similarities and differences between the two fields and explains how to use the interaction data for deriving a better feature weighting. Experiments with existing log files are done and a significant improvement in performance is reached with a feature weighting calculated from the information contained in the log files. Even with several steps of relevance feedback the results remain much better than without the learning, which means that not only information from feedback is taken into account earlier, but a better quality of retrieval is reached in all steps.     

A survey of browsing models for content based image retrieval

The problem of content based image retrieval (CBIR) has traditionally been investigated within a framework that emphasises the explicit formulation of a query: users initiate an automated search for relevant images by submitting an image or draw a sketch that exemplifies their information need. Often, relevance feedback is incorporated as a post-retrieval step for optimising the way evidence from different visual features is combined. While this sustained methodological focus has helped CBIR to mature, it has also brought out its limitations more clearly: There is often little support for exploratory search and scaling to very large collections is problematic. Moreover, the assumption that users are always able to formulate an appropriate query is questionable. An effective, albeit much less studied, method of accessing image collections based on visual content is that of browsing. The aim of this survey paper is to provide a structured overview of the different models that have been explored over the last one to two decades, to highlight the particular challenges of the browsing approach and to focus attention on a few interesting issues that warrant more intense research.

Neighborhood preserving regression for image retrieval

In content-based image retrieval (CBIR), relevance feedback has been proven to be a powerful tool for bridging the gap between low level visual features and high level semantic concepts. Traditionally, relevance feedback driven CBIR is often considered as a supervised learning problem where the user provided feedbacks are used to learn a distance metric or classification function. However, CBIR is intrinsically a semi-supervised learning problem in which the testing samples (images in the database) are present during the learning process. Moreover, when there are no sufficient feedbacks, these methods may suffer from the overfitting problem. In this paper, we propose a novel neighborhood preserving regression algorithm which makes efficient use of both labeled and unlabeled images. By using the unlabeled images, the geometrical structure of the image space can be incorporated into the learning system through a regularizer. Specifically, from all the functions which minimize the empirical loss on the labeled images, we select the one which best preserves the local neighborhood structure of the image space. In this way, our method can obtain a regression function which respects both semantic and geometrical structures of the image database. We present experimental evidence suggesting that our algorithm is able to use unlabeled data effectively for image retrieval.     

基于多示例学习的图像检索方法

由于多示例学习能够有效处理图像的歧义性,因此被应用于基于内容的图像检索(CBIR)。本文提出一种基于多示例学习的CBIR方法,该方法将图像作为多示例包,基于高斯混合模型和改进的EM算法全自动分割图像,并提取颜色、纹理、形状和不变矩等区域信息作为示例向量生成测试图像包。根据用户选择的实例图像生成正包和反包,使用多种多示例学习算法进行学习,实现图像检索和相关反馈,得到了较好的效果。     

一种基于内容的图像检索界面

基于内容和对象的图像压缩和检索是下一代的图像处理技术,具有较广阔的应用前景。目前该领域的研究主要从设计方便、快捷的用户查询界面和发展图像数据库检索技术两方面展开。为此,该文提出了一种基于内容的图像检索用户界面的设计方法来满足用户复杂的检索要求。在图像的检索过程中,通过用户组合图标的方法来描述检索要求,同时将图像的颜色和空间信息相结合进行图像查询,并借助用户的反馈信息实现系统的自学习功能,最终逐步提高系统图像检索的速度和准确性。     

A unified log-based relevance feedback scheme for image retrieval

Relevance feedback has emerged as a powerful tool to boost the retrieval performance in content-based image retrieval (CBIR). In the past, most research efforts in this field have focused on designing effective algorithms for traditional relevance feedback. Given that a CBIR system can collect and store users' relevance feedback information in a history log, an image retrieval system should be able to take advantage of the log data of users' feedback to enhance its retrieval performance. In this paper, we propose a unified framework for log-based relevance feedback that integrates the log of feedback data into the traditional relevance feedback schemes to learn effectively the correlation between low-level image features and high-level concepts. Given the error-prone nature of log data, we present a novel learning technique, named soft label support vector machine, to tackle the noisy data problem. Extensive experiments are designed and conducted to evaluate the proposed algorithms based on the COREL image data set. The promising experimental results validate the effectiveness of our log-based relevance feedback scheme empirically.     

FAST: Toward more effective and efficient image retrieval

This paper focuses on developing a Fast And Semantics-Tailored (FAST) image retrieval methodology. Specifically, the contributions of FAST methodology to the CBIR literature include: (1) development of a new indexing method based on fuzzy logic to incorporate color, texture, and shape information into a region-based approach to improving the retrieval effectiveness and robustness; (2) development of a new hierarchical indexing structure and the corresponding hierarchical, elimination-based A* retrieval (HEAR) algorithm to significantly improve the retrieval efficiency without sacrificing the retrieval effectiveness; it is shown that HEAR is guaranteed to deliver a logarithm search in the average case; (3) employment of user relevance feedback to tailor the effective retrieval to each user's individualized query preference through the novel indexing tree pruning (ITP) and adaptive region weight updating (ARWU) algorithms. Theoretical analysis and experimental evaluations show that FAST methodology holds great promise in delivering fast and semantics-tailored image retrieval in CBIR.     

Fast Query Point Movement Techniques for Large CBIR Systems

Target search in content-based image retrieval (CBIR) systems refers to finding a specific (target) image such as a particular registered logo or a specific historical photograph. Existing techniques, designed around query refinement based on relevance feedback, suffer from slow convergence, and do not guarantee to find intended targets. To address these limitations, we propose several efficient query point movement methods. We prove that our approach is able to reach any given target image with fewer iterations in the worst and average cases. We propose a new index structure and query processing technique to improve retrieval effectiveness and efficiency. We also consider strategies to minimize the effects of users' inaccurate relevance feedback. Extensive experiments in simulated and realistic environments show that our approach significantly reduces the number of required iterations and improves overall retrieval performance. The experimental results also confirm that our approach can always retrieve intended targets even with poor selection of initial query points.