A semantic model for general purpose content-based image retrieval systems

The ‘semantic gap’ is the main challenge in content-based image retrieval. To overcome this challenge, we propose a semantic-based model to retrieve images efficiently with considering user-interested concepts. In this model, an interactive image segmentation algorithm is carried out on the query image to extract the user-interested regions. To recognize the image objects from regions, a neural network classifier is used in this model. In order to have a general-purpose system, no priori assumptions should be made regarding the nature of images in extracting features. So a large number of features should be extracted from all aspect of the image. The high dimensional feature space, not only increases the complexity and required memory, but also may reduce the efficiency and accuracy. Hence, the ant colony optimization algorithm is employed to eliminate irrelevant and redundant features. To find the most similar images to the query image, the similarity between images is measured based on their semantic objects which are defined according to a predefined ontology.     

增量学习语义属性的图像内容检索系统增强

大规模图像内容检索是实现图像语义信息获取的重要手段, 其首要需解决图像低层特征与用户高层语义间的语义鸿沟问题。针对该问题, 引入图像语义属性, 并结合增量分类学习方法(online core vector machine, OCVM), 提出了一种增量构建大规模图像内容检索系统的新方法。该方法借助检索反馈学习机制可以提升图像语义属性的辨别准确性, 能在扩张图像库规模的同时, 提升图像内容检索的可靠性。实验结果表明了上述方法的有效性, 其检索性能可逐步地达到离线构建方法的最佳性能, 但具有更好的可扩展性和自提升能力。     

Sparse semantic metric learning for image retrieval

Typical content-based image retrieval solutions usually cannot achieve satisfactory performance due to the semantic gap challenge. With the popularity of social media applications, large amounts of social images associated with user tagging information are available, which can be leveraged to boost image retrieval. In this paper, we propose a sparse semantic metric learning (SSML) algorithm by discovering knowledge from these social media resources, and apply the learned metric to search relevant images for users. Different from the traditional metric learning approaches that use similar or dissimilar constraints over a homogeneous visual space, the proposed method exploits heterogeneous information from two views of images and formulates the learning problem with the following principles. The semantic structure in the text space is expected to be preserved for the transformed space. To prevent overfitting the noisy, incomplete, or subjective tagging information of images, we expect that the mapping space by the learned metric does not deviate from the original visual space. In addition, the metric is straightforward constrained to be row-wise sparse with the ?_2,1-norm to suppress certain noisy or redundant visual feature dimensions. We present an iterative algorithm with proved convergence to solve the optimization problem. With the learned metric for image retrieval, we conduct extensive experiments on a real-world dataset and validate the effectiveness of our approach compared with other related work.     

An intelligent content-based image retrieval methodology using transfer learning for digital IP protection

Trademarks are used by companies to help customers identify products or services using images or logos in addition to slogans, words, names, sounds, smells, color, and motions. Trademark logos are widely distributed through advertising and published through online media websites and social networks such as Facebook, Pinterest, and Flicker. The intellectual property (IP) rights of the trademark owners have strong legal protection when registered with international intellectual property platforms such as the US Patent and Trademark Office and the World Intellectual Property Office. Using a registered trademark without prior consent of the owner may result in intellectual property infringement with severe legal consequences under civil or criminal law. Companies invest large capital resources in protecting their trademark from being copied or misused in ways that confuse the customers or steal market share. This research focuses on trademark (TM) logo image retrieval systems used in the cyber marketplaces to identify similar TM logo images online automatically and intelligently. The methodology developed for TM logo similarity measurement is based on content-based image retrieval. Content retrieval reduces the gap between high-level semantic interpretation of human vision and the low-level features processed by the machine. The proposed transfer learning methodology uses embedded learning with triplet loss to fine-tune a pre-trained convolutional neural network model. The Logo-2K+ large-scale logo dataset is re-organized and divided into the top 70% as the training set and the remaining 30% as the testing set. The results show that the novel transfer learning approach is developed and demonstrated in this research for the intelligent automatic detection of similar TM logo images with high accuracy. The verification experiments (trained with 7625 logos and tested with 3221 logos) demonstrates that the Recall@10 of the test set can reach 95% using the advanced convolutional neural network model (VGG19) adjusted with the novel transfer learning methodology.     

A multi-agent platform for content-based image retrieval

Efficient and possibly intelligent image retrieval is an important task, often required in many fields of human activity. While traditional database indexing techniques exhibit a remarkable performance in textual information retrieval current research in content-based image retrieval is focused on developing novel techniques that are biologically motivated and efficient. It is well known that humans have a remarkable ability to process visual information and to handle the volume and complexity of such information quite efficiently. In this paper, we present a content-based image retrieval platform that is based on a multi-agent architecture. Each agent is responsible for assessing the similarity of the query image to each candidate image contained in a collection based on a specific primitive feature and a corresponding similarity criterion. The outputs of various agents are integrated using one of several voting schemes supported by the system. The system’s performance has been evaluated using various collections of images, as well as images obtained in specific application domains such as medical imaging. The initial evaluation has yielded very promising results.

A Center-Surround Histogram for content-based image retrieval

In this paper, a new type of histogram which incorporates only the visual information surrounding the edges of the image is introduced. The edge extraction operation is performed with the use of a center-surround operator of the Human Visual System. The proposed Center-Surround Histogram (CSH) has two main advantages over the classic histogram. First, it reduces the amount of visual information that needs to be processed and second, it incorporates a degree of spatial information when used in content based image retrieval applications. The method is compared with other contemporary image retrieval methods, including that of another edge color histogram, on two different databases. The comparison shows that the use of CSH exhibits better results in shorter execution times.     

Localized content-based image retrieval

We define localized content-based image retrieval as a CBIR task where the user is only interested in a portion of the image, and the rest of the image is irrelevant. In this paper we present a localized CBIR system, Accio, that uses labeled images in conjunction with a multiple-instance learning algorithm to first identify the desired object and weight the features accordingly, and then to rank images in the database using a similarity measure that is based upon only the relevant portions of the image. A challenge for localized CBIR is how to represent the image to capture the content. We present and compare two novel image representations, which extend traditional segmentation-based and salient point-based techniques respectively, to capture content in a localized CBIR setting.     

A genetic programming framework for content-based image retrieval

The effectiveness of content-based image retrieval (CBIR) systems can be improved by combining image features or by weighting image similarities, as computed from multiple feature vectors. However, feature combination do not make sense always and the combined similarity function can be more complex than weight-based functions to better satisfy the users’ expectations. We address this problem by presenting a Genetic Programming framework to the design of combined similarity functions. Our method allows nonlinear combination of image similarities and is validated through several experiments, where the images are retrieved based on the shape of their objects. Experimental results demonstrate that the GP framework is suitable for the design of effective combinations functions.     

A short-term learning approach based on similarity refinement in content-based image retrieval

Multimedia Tools and Applications - This paper presents a new relevance feedback approach based on similarity refinement. In the proposed approach weight correction of feature’s components is...     

A generic content-based image retrieval framework for mobile devices

Multimedia mobile devices have created new possibilities for developing and accessing a variety of multimedia items such as images, audio and video clips. Personal multimedia items are, nowadays, being consumed at an enormous rate. Therefore, the management of these media items has become a pressing problem. In this paper, a client-server content-based image retrieval framework for mobile platforms is developed, which provides the capability of content-based query and browsing from mobile devices. The proposed framework provides an adaptive user interface and a generic structure, which supports a wide range of mobile devices. In this framework, a client requests the server for retrieval of particular images with a particular content. The server performs a content-based retrieval of images from a selected database and streams the retrieved results back to the client in an efficient way. The query results are transmitted over a wireless network and a certain number of similar images are rendered on the mobile device screen using thumbnail sizes. The proposed framework serves as a basis of content-based image retrieval on mobile devices. It addresses several important challenges such as hardware and software limitations as well as efficient use of the available network bandwidth.     

A framework for linguistic relevance feedback in content-based image retrieval using fuzzy logic

We describe a new approach for exploiting relevance feedback in content-based image retrieval (CBIR). In our approach to relevance feedback we try to capture more of the users’ relevance judgments by allowing the use of natural language like comments on the retrieved images. Using methods from fuzzy logic and computational intelligence we are able to reflect these comments into new targets for searching the image database. Such enhanced information is utilized to develop a system that can provide more effective and efficient retrieval.     

Series feature aggregation for content-based image retrieval

Feature aggregation is a critical technique in content-based image retrieval (CBIR) systems that employs multiple visual features to characterize image content. Most previous feature aggregation schemes apply parallel topology, e.g., the linear combination scheme, which suffer from two problems. First, the function of individual visual feature is limited since the ranks of the retrieved images are determined only by the combined similarity. Second, the irrelevant images seriously affect the retrieval performance of feature aggregation scheme since all images in a collection will be ranked. To address these problems, we propose a new feature aggregation scheme, series feature aggregation (SFA). SFA selects relevant images using visual features one by one in series from the images highly ranked by the previous visual feature. The irrelevant images will be effectively filtered out by individual visual features in each stage, and the remaining images are collectively described by all visual features. Experiments, conducted with IAPR TC-12 benchmark image collection (ImageCLEF2006) that contains over 20,000 photographic images and defined queries, have shown that the proposed SFA can outperform conventional parallel feature aggregation schemes.     

An adaptive technique for content-based image retrieval

We discuss an adaptive approach towards Content-Based Image Retrieval. It is based on the Ostensive Model of developing information needs—a special kind of relevance feedback model that learns from implicit user feedback and adds a temporal notion to relevance. The ostensive approach supports content-assisted browsing through visualising the interaction by adding user-selected images to a browsing path, which ends with a set of system recommendations. The suggestions are based on an adaptive query learning scheme, in which the query is learnt from previously selected images. Our approach is an adaptation of the original Ostensive Model based on textual features only, to include content-based features to characterise images. In the proposed scheme textual and colour features are combined using the Dempster-Shafer theory of evidence combination. Results from a user-centred, work-task oriented evaluation show that the ostensive interface is preferred over a traditional interface with manual query facilities. This is due to its ability to adapt to the user's need, its intuitiveness and the fluid way in which it operates. Studying and comparing the nature of the underlying information need, it emerges that our approach elicits changes in the user's need based on the interaction, and is successful in adapting the retrieval to match the changes. In addition, a preliminary study of the retrieval performance of the ostensive relevance feedback scheme shows that it can outperform a standard relevance feedback strategy in terms of image recall in category search.     

Cluster-driven refinement for content-based digital image retrieval

Increasing application demands are pushing databases toward providing effective and efficient support for content-based retrieval over multimedia objects. In addition to adequate retrieval techniques, it is also important to enable some form of adaptation to users' specific needs. This paper introduces a new refinement method for retrieval based on the learning of the users' specific preferences. The proposed system indexes objects based on shape and groups them into a set of clusters, with each cluster represented by a prototype. Clustering constructs a taxonomy of objects by forming groups of closely-related objects. The proposed approach to learn the users' preferences is to refine corresponding clusters from objects provided by the users in the foreground, and to simultaneously adapt the database index in the background. Queries can be performed based solely on shape, or on a combination of shape with other features such as color. Our experimental results show that the system successfully adapts queries into databases with only a small amount of feedback from the users. The quality of the returned results is superior to that of a color-based query, and continues to improve with further use.     

Probability based document clustering and image clustering using content-based image retrieval

Clustering of related or similar objects has long been regarded as a potentially useful contribution of helping users to navigate an information space such as a document collection. Many clustering algorithms and techniques have been developed and implemented but as the sizes of document collections have grown these techniques have not been scaled to large collections because of their computational overhead. To solve this problem, the proposed system concentrates on an interactive text clustering methodology, probability based topic oriented and semi-supervised document clustering. Recently, as web and various documents contain both text and large number of images, the proposed system concentrates on content-based image retrieval (CBIR) for image clustering to give additional effect to the document clustering approach. It suggests two kinds of indexing keys, major colour sets (MCS) and distribution block signature (DBS) to prune away the irrelevant images to given query image. Major colour sets are related with colour information while distribution block signatures are related with spatial information. After successively applying these filters to a large database, only small amount of high potential candidates that are somewhat similar to that of query image are identified. Then, the system uses quad modelling method (QM) to set the initial weight of two-dimensional cells in query image according to each major colour and retrieve more similar images through similarity association function associated with the weights. The proposed system evaluates the system efficiency by implementing and testing the clustering results with Dbscan and K-means clustering algorithms. Experiment shows that the proposed document clustering algorithm performs with an average efficiency of 94.4% for various document categories.     

Bayesian relevance feedback for content-based image retrieval

Despite the efforts to reduce the so-called semantic gap between the user's perception of image similarity and the feature-based representation of images, the interaction with the user remains fundamental to improve performances of content-based image retrieval systems. To this end, relevance feedback mechanisms are adopted to refine image-based queries by asking users to mark the set of images retrieved in a neighbourhood of the query as being relevant or not. In this paper, the Bayesian decision theory is used to estimate the boundary between relevant and non-relevant images. Then, a new query is computed whose neighbourhood is likely to fall in a region of the feature space containing relevant images. The performances of the proposed query shifting method have been compared with those of other relevance feedback mechanisms described in the literature. Reported results show the superiority of the proposed method.