Independent query refinement and feature re-weighting using positive and negative examples for content-based image retrieval

Query refinement and feature re-weighting are the two core techniques underlying the relevance feedback of content-based image retrieval. Most existing relevance feedback mechanisms generally model the user’s query target with a single query point and weight each extracted feature with a single importance factor. A designed estimation procedure then estimates the best query point and all importance factors by optimizing a formulated criterion which measures the goodness of the estimation. This formulated criterion simultaneously encapsulates all positive and negative examples supplied from the user’s feedback. Under such formulation, the positive and negative examples may contribute contradictorily to the criterion and sometimes may introduce higher difficulty in attaining a good estimation. In this paper, we propose a different statistical formulation to estimate independently two pairs of query points and feature weights from the positive examples and negative examples, respectively. These two pairs then define the likelihood ratio, a criterion term used to rank the relevance of all database images. This approach simplifies the criterion formulation and also avoids the mutual impeditive influence between positive examples and negative examples. The experimental results demonstrate that the proposed approach outperforms some other related approaches.

Content-based image retrieval with relevance feedback using random walks

In this paper, we propose a novel approach to content-based image retrieval with relevance feedback, which is based on the random walker algorithm introduced in the context of interactive image segmentation. The idea is to treat the relevant and non-relevant images labeled by the user at every feedback round as “seed” nodes for the random walker problem. The ranking score for each unlabeled image is computed as the probability that a random walker starting from that image will reach a relevant seed before encountering a non-relevant one. Our method is easy to implement, parameter-free and scales well to large datasets. Extensive experiments on different real datasets with several image similarity measures show the superiority of our method over different recent approaches.     

Image retrieval model based on weighted visual features determined by relevance feedback

An accurate and rapid method is required to retrieve the overwhelming majority of digital images. To date, image retrieval methods include content-based retrieval and keyword-based retrieval, the former utilizing visual features such as color and brightness, and the latter utilizing keywords that describe the image. However, the effectiveness of these methods in providing the exact images the user wants has been under scrutiny. Hence, many researchers have been working on relevance feedback, a process in which responses from the user are given as feedback during the retrieval session in order to define a user’s need and provide an improved result. Methods that employ relevance feedback, however, do have drawbacks because several pieces of feedback are necessary to produce an appropriate result, and the feedback information cannot be reused. In this paper, a novel retrieval model is proposed, which annotates an image with keywords and modifies the confidence level of the keywords in response to the user’s feedback. In the proposed model, not only the images that have been given feedback, but also other images with visual features similar to the features used to distinguish the positive images are subjected to confidence modification. This allows for modification of a large number of images with relatively little feedback, ultimately leading to faster and more accurate retrieval results. An experiment was performed to verify the effectiveness of the proposed model, and the result demonstrated a rapid increase in recall and precision using the same amount of feedback.     

Content-based image retrieval using fuzzy perceptual feedback

In this paper, a new framework called fuzzy relevance feedback in interactive content-based image retrieval (CBIR) systems is introduced. Conventional binary labeling scheme in relevance feedback requires a crisp decision to be made on the relevance of the retrieved images. However, it is inflexible as user interpretation of visual content varies with respect to different information needs and perceptual subjectivity. In addition, users tend to learn from the retrieval results to further refine their information requests. It is, therefore, inadequate to describe the user’s fuzzy perception of image similarity with crisp logic. In view of this, we propose a fuzzy relevance feedback approach which enables the user to make a fuzzy judgement. It integrates the user’s fuzzy interpretation of visual content into the notion of relevance feedback. An efficient learning approach is proposed using a fuzzy radial basis function (FRBF) network. The network is constructed based on the user’s feedbacks. The underlying network parameters are optimized by adopting a gradient-descent training strategy due to its computational efficiency. Experimental results using a database of 10,000 images demonstrate the effectiveness of the proposed method.

Speed up interactive image retrieval

In multimedia retrieval, a query is typically interactively refined towards the “optimal” answers by exploiting user feedback. However, in existing work, in each iteration, the refined query is re-evaluated. This is not only inefficient but fails to exploit the answers that may be common between iterations. Furthermore, it may also take too many iterations to get the “optimal” answers. In this paper, we introduce a new approach called OptRFS (optimizing relevance feedback search by query prediction) for iterative relevance feedback search. OptRFS aims to take users to view the “optimal” results as fast as possible. It optimizes relevance feedback search by both shortening the searching time during each iteration and reducing the number of iterations. OptRFS predicts the potential candidates for the next iteration and maintains this small set for efficient sequential scan. By doing so, repeated candidate accesses (i.e., random accesses) can be saved, hence reducing the searching time for the next iteration. In addition, efficient scan on the overlap before the next search starts also tightens the search space with smaller pruning radius. As a step forward, OptRFS also predicts the “optimal” query, which corresponds to “optimal” answers, based on the early executed iterations’ queries. By doing so, some intermediate iterations can be saved, hence reducing the total number of iterations. By taking the correlations among the early executed iterations into consideration, OptRFS investigates linear regression, exponential smoothing and linear exponential smoothing to predict the next refined query so as to decide the overlap of candidates between two consecutive iterations. Considering the special features of relevance feedback, OptRFS further introduces adaptive linear exponential smoothing to self-adjust the parameters for more accurate prediction. We implemented OptRFS and our experimental study on real life data sets show that it can reduce the total cost of relevance feedback search significantly. Some interesting features of relevance feedback search are also discovered and discussed.     

Relevance feedback for real-world human action retrieval

Content-based video retrieval is an increasingly popular research field, in large part due to the quickly growing catalogue of multimedia data to be found online. Even though a large portion of this data concerns humans, however, retrieval of human actions has received relatively little attention. Presented in this paper is a video retrieval system that can be used to perform a content-based query on a large database of videos very efficiently. Furthermore, it is shown that by using ABRS-SVM, a technique for incorporating Relevance feedback (RF) on the search results, it is possible to quickly achieve useful results even when dealing with very complex human action queries, such as in Hollywood movies.     

Exploring statistical correlations for image retrieval

Bridging the cognitive gap in image retrieval has been an active research direction in recent years, of which a key challenge is to get enough training data to learn the mapping functions from low-level feature spaces to high-level semantics. In this paper, image regions are classified into two types: key regions representing the main semantic contents and environmental regions representing the contexts. We attempt to leverage the correlations between types of regions to improve the performance of image retrieval. A Context Expansion approach is explored to take advantages of such correlations by expanding the key regions of the queries using highly correlated environmental regions according to an image thesaurus. The thesaurus serves as both a mapping function between image low-level features and concepts and a store of the statistical correlations between different concepts. It is constructed through a data-driven approach which uses Web data (images, their surrounding textual annotations) as training data source to learn the region concepts and to explore the statistical correlations. Experimental results on a database of 10,000 general-purpose images show the effectiveness of our proposed approach in both improving search precision (i.e. filter irrelevant images) and recall (i.e. retrieval relevant images whose context may be varied). Several major factors which have impact on the performance of our approach are also studied.     

Content-based image retrieval using growing hierarchical self-organizing quadtree map

In this paper, a growing hierarchical self-organizing quadtree map (GHSOQM) is proposed and used for a content-based image retrieval (CBIR) system. The incorporation of GHSOQM in a CBIR system organizes images in a hierarchical structure. The retrieval time by GHSOQM is less than that by using direct image comparison using a flat structure. Furthermore, the ability of incremental learning enables GHSOQM to be a prospective neural-network-based approach for CBIR systems. We also propose feature matrices, image distance and relevance feedback for region-based images in the GHSOQM-based CBIR system. Experimental results strongly demonstrate the effectiveness of the proposed system.     

An adaptive technique for content-based image retrieval

We discuss an adaptive approach towards Content-Based Image Retrieval. It is based on the Ostensive Model of developing information needs—a special kind of relevance feedback model that learns from implicit user feedback and adds a temporal notion to relevance. The ostensive approach supports content-assisted browsing through visualising the interaction by adding user-selected images to a browsing path, which ends with a set of system recommendations. The suggestions are based on an adaptive query learning scheme, in which the query is learnt from previously selected images. Our approach is an adaptation of the original Ostensive Model based on textual features only, to include content-based features to characterise images. In the proposed scheme textual and colour features are combined using the Dempster-Shafer theory of evidence combination. Results from a user-centred, work-task oriented evaluation show that the ostensive interface is preferred over a traditional interface with manual query facilities. This is due to its ability to adapt to the user's need, its intuitiveness and the fluid way in which it operates. Studying and comparing the nature of the underlying information need, it emerges that our approach elicits changes in the user's need based on the interaction, and is successful in adapting the retrieval to match the changes. In addition, a preliminary study of the retrieval performance of the ostensive relevance feedback scheme shows that it can outperform a standard relevance feedback strategy in terms of image recall in category search.     

A powerful relevance feedback mechanism for content-based 3D model retrieval

The technique of relevance feedback has been introduced to content-based 3D model retrieval, however, two essential issues which affect the retrieval performance have not been addressed. In this paper, a novel relevance feedback mechanism is presented, which effectively makes use of strengths of different feature vectors and perfectly solves the problem of small sample and asymmetry. During the retrieval process, the proposed method takes the user’s feedback details as the relevant information of query model, and then dynamically updates two important parameters of each feature vector, narrowing the gap between high-level semantic knowledge and low-level object representation. The experiments, based on the publicly available 3D model database Princeton Shape Benchmark (PSB), show that the proposed approach not only precisely captures the user’s semantic knowledge, but also significantly improves the retrieval performance of 3D model retrieval. Compared with three state-of-the-art query refinement schemes for 3D model retrieval, it provides superior retrieval effectiveness only with a few rounds of relevance feedback based on several standard measures.

A personalized query expansion approach for engineering document retrieval

Engineers create engineering documents with their own terminologies, and want to search existing engineering documents quickly and accurately during a product development process. Keyword-based search methods have been widely used due to their ease of use, but their search accuracy has been often problematic because of the semantic ambiguity of terminologies in engineering documents and queries. The semantic ambiguity can be alleviated by using a domain ontology. Also, if queries are expanded to incorporate the engineer’s personalized information needs, the accuracy of the search result would be improved. Therefore, we propose a framework to search engineering documents with less semantic ambiguity and more focus on each engineer’s personalized information needs. The framework includes four processes: (1) developing a domain ontology, (2) indexing engineering documents, (3) learning user profiles, and (4) performing personalized query expansion and retrieval. A domain ontology is developed based on product structure information and engineering documents. Using the domain ontology, terminologies in documents are disambiguated and indexed. Also, a user profile is generated from the domain ontology. By user profile learning, user’s interests are captured from the relevant documents. During a personalized query expansion process, the learned user profile is used to reflect user’s interests. Simultaneously, user’s searching intent, which is implicitly inferred from the user’s task context, is also considered. To retrieve relevant documents, an expanded query in which both user’s interests and intents are reflected is then matched against the document collection. The experimental results show that the proposed approach can substantially outperform both the keyword-based approach and the existing query expansion method in retrieving engineering documents. Reflecting a user’s information needs precisely has been identified to be the most important factor underlying this notable improvement.     

Conceptual document modelling and retrieval

This paper presents the main current theoretical issues in Information Retrieval. The principles of conceptual modelling, as they have emerged in the database area, are presented and their application to document modelling in order to enhance document retrieval is discussed. Finally, the main features of the MULTOS project are presented and critically reviewed confronting them with the requirements which have been identified during the general discussion on document conceptual modelling for information retrieval.     

Supporting web query expansion efficiently using multi-granularity indexing and query processing

The problem of word mismatch in information retrieval (IR) occurs because users often use different words to describe concepts in their queries than authors use to describe the same concepts in their documents. Query expansion is used to deal with the mismatch between author and user vocabularies. To support query expansion, indices on words related by lexical semantics and syntactical co-occurrence need to be maintained. Two issues become paramount in supporting query expansion: the size of index tables and the query processing overhead. In this paper, we propose to use the notion of multi-granularity for more efficient indexing and query processing while the same degrees of precision and recall are maintained. We also describes extensions of this technique to handle: (1) query relaxation to handle words with multiple senses and with other semantic relationships; (2) progressive processing of queries with top N results and (3) progressive processing of queries with specification of the importance of each keyword.     

Interactive pattern analysis for relevance feedback in multimedia information retrieval

Relevance feedback is a mechanism to interactively learn a users query concept online. It has been extensively used to improve the performance of multimedia information retrieval. In this paper, we present a novel interactive pattern analysis method that reduces relevance feedback to a two-class classification problem and classifies multimedia objects as relevant or irrelevant. To perform interactive pattern analysis, we propose two online pattern classification methods, called interactive random forests (IRF) and adaptive random forests (ARF), that adapt a composite classifier known as random forests for relevance feedback. IRF improves the efficiency of regular random forests (RRF) with a novel two-level resampling technique called biased random sample reduction, while ARF boosts the performance of RRF with two adaptive learning techniques called dynamic feature extraction and adaptive sample selection. During interactive multimedia retrieval, both ARF and IRF run two to three times faster than RRF while achieving comparable precision and recall against the latter. Extensive experiments on a COREL image set (with 31,438 images) demonstrate that our methods (i.e., IRF and RRF) achieve at least a improvement on average precision and recall over the state-of-the-art approaches.     

Collaborative image retrieval via regularized metric learning

In content-based image retrieval (CBIR), relevant images are identified based on their similarities to query images. Most CBIR algorithms are hindered by the semantic gap between the low-level image features used for computing image similarity and the high-level semantic concepts conveyed in images. One way to reduce the semantic gap is to utilize the log data of users' feedback that has been collected by CBIR systems in history, which is also called “collaborative image retrieval.” In this paper, we present a novel metric learning approach, named “regularized metric learning,” for collaborative image retrieval, which learns a distance metric by exploring the correlation between low-level image features and the log data of users' relevance judgments. Compared to the previous research, a regularization mechanism is used in our algorithm to effectively prevent overfitting. Meanwhile, we formulate the proposed learning algorithm into a semidefinite programming problem, which can be solved very efficiently by existing software packages and is scalable to the size of log data. An extensive set of experiments has been conducted to show that the new algorithm can substantially improve the retrieval accuracy of a baseline CBIR system using Euclidean distance metric, even with a modest amount of log data. The experiment also indicates that the new algorithm is more effective and more efficient than two alternative algorithms, which exploit log data for image retrieval.     

Image retrieval using nonlinear manifold embedding

The huge number of images on the Web gives rise to the content-based image retrieval (CBIR) as the text-based search techniques cannot cater to the needs of precisely retrieving Web images. However, CBIR comes with a fundamental flaw: the semantic gap between high-level semantic concepts and low-level visual features. Consequently, relevance feedback is introduced into CBIR to learn the subjective needs of users. However, in practical applications the limited number of user feedbacks is usually overwhelmed by the large number of dimensionalities of the visual feature space. To address this issue, a novel semi-supervised learning method for dimensionality reduction, namely kernel maximum margin projection (KMMP) is proposed in this paper based on our previous work of maximum margin projection (MMP). Unlike traditional dimensionality reduction algorithms such as principal component analysis (PCA) and linear discriminant analysis (LDA), which only see the global Euclidean structure, KMMP is designed for discovering the local manifold structure. After projecting the images into a lower dimensional subspace, KMMP significantly improves the performance of image retrieval. The experimental results on Corel image database demonstrate the effectiveness of our proposed nonlinear algorithm.