A similarity learning approach to content-based image retrieval: application to digital mammography

In this paper, we describe an approach to content-based retrieval of medical images from a database, and provide a preliminary demonstration of our approach as applied to retrieval of digital mammograms. Content-based image retrieval (CBIR) refers to the retrieval of images from a database using information derived from the images themselves, rather than solely from accompanying text indices. In the medical-imaging context, the ultimate aim of CBIR is to provide radiologists with a diagnostic aid in the form of a display of relevant past cases, along with proven pathology and other suitable information. CBIR may also be useful as a training tool for medical students and residents. The goal of information retrieval is to recall from a database information that is relevant to the user's query. The most challenging aspect of CBIR is the definition of relevance (similarity), which is used to guide the retrieval machine. In this paper, we pursue a new approach, in which similarity is learned from training examples provided by human observers. Specifically, we explore the use of neural networks and support vector machines to predict the user's notion of similarity. Within this framework we propose using a hierarchal learning approach, which consists of a cascade of a binary classifier and a regression module to optimize retrieval effectiveness and efficiency. We also explore how to incorporate online human interaction to achieve relevance feedback in this learning framework. Our experiments are based on a database consisting of 76 mammograms, all of which contain clustered microcalcifications (MCs). Our goal is to retrieve mammogram images containing similar MC clusters to that in a query. The performance of the retrieval system is evaluated using precision-recall curves computed using a cross-validation procedure. Our experimental results demonstrate that: 1) the learning framework can accurately predict the perceptual similarity reported by human observers, thereby serving as a basis for CBIR; 2) the learning-based framework can significantly outperform a simple distance-based similarity metric; 3) the use of the hierarchical two-stage network can improve retrieval performance; and 4) relevance feedback can be effectively incorporated into this learning framework to achieve improvement in retrieval precision based on online interaction with users; and 5) the retrieved images by the network can have predicting value for the disease condition of the query.     

一种基于区域的动态分块图象检索方法

本文提出一种新颖的基于内容和图像检索方法，基于运动子块分割并根据视觉特性对不同区域做不同的加权，比较各子块相似度，分析相似度矩阵，并检索查询物体。通过将图象分割细化，充分利用了原图的颜色位置信息，通过实验，实现了对特定物体进行检索。该物体检索方法可进一步发展，为特定的后续处理奠定基础，如在人脸识别等功能中发挥重要作用。     

Content-based microscopic image retrieval system for multi-image queries

In this paper, we describe the design and development of a multitiered content-based image retrieval (CBIR) system for microscopic images utilizing a reference database that contains images of more than one disease. The proposed CBIR system uses a multitiered approach to classify and retrieve microscopic images involving their specific subtypes, which are mostly difficult to discriminate and classify. This system enables both multi-image query and slide-level image retrieval in order to protect the semantic consistency among the retrieved images. New weighting terms, inspired from information retrieval theory, are defined for multiple-image query and retrieval. The performance of the system was tested on a dataset including 1666 imaged high power fields extracted from 57 follicular lymphoma (FL) tissue slides with three subtypes and 44 neuroblastoma (NB) tissue slides with four subtypes. Each slide is semantically annotated according to their subtypes by expert pathologists. By using leave-one-slide out testing scheme, the multi-image query algorithm with the proposed weighting strategy achieves about 93% and 86% of average classification accuracy at the first rank retrieval, outperforming the image-level retrieval accuracy by about 38 and 26 percentage points, for FL and NB diseases, respectively.     

基于内容的图像检索技术研究

在对海量的图像数据进行检索时,传统的基于数值/字符的信息检索技术并不能满足要求.因此,基于内容的图像检索技术(CBIR:Content-Based Image Retrieval)的研究应运而生,并引起了广泛关注.本文主要讨论CBIR研究中的一些关键问题:图像的内容特征及其提取、特征之间的相似度计算、查询条件的表达、检索性能的评价、压缩域的图像检索技术等等,并指出了一些可值得深入研究的方向.     

Image retrieval using both color and texture features

In order to improve the retrieval performance of images, this paper proposes an efficient approach for extracting and retrieving color images. The block diagram of our proposed approach to content-based image retrieval (CBIR) is given firstly, and then we introduce three image feature extracting arithmetic including color histogram, edge histogram and edge direction histogram, the histogram Euclidean distance, cosine distance and histogram intersection are used to measure the image level similarity. On the basis of using color and texture features separately, a new method for image retrieval using combined features is proposed. With the test for an image database including 766 general-purpose images and comparison and analysis of performance evaluation for features and similarity measures, our proposed retrieval approach demonstrates a promising performance. Experiment shows that combined features are superior to every single one of the three features in retrieval.     

Similarity-based online feature selection in content-based image retrieval.

Content-based image retrieval (CBIR) has been more and more important in the last decade, and the gap between high-level semantic concepts and low-level visual features hinders further performance improvement. The problem of online feature selection is critical to really bridge this gap. In this paper, we investigate online feature selection in the relevance feedback learning process to improve the retrieval performance of the region-based image retrieval system. Our contributions are mainly in three areas. 1) A novel feature selection criterion is proposed, which is based on the psychological similarity between the positive and negative training sets. 2) An effective online feature selection algorithm is implemented in a boosting manner to select the most representative features for the current query concept and combine classifiers constructed over the selected features to retrieve images. 3) To apply the proposed feature selection method in region-based image retrieval systems, we propose a novel region-based representation to describe images in a uniform feature space with real-valued fuzzy features. Our system is suitable for online relevance feedback learning in CBIR by meeting the three requirements: learning with small size training set, the intrinsic asymmetry property of training samples, and the fast response requirement. Extensive experiments, including comparisons with many state-of-the-arts, show the effectiveness of our algorithm in improving the retrieval performance and saving the processing time.     

A new way for multidimensional medical data management: volume of interest (VOI)-based retrieval of medical images with visual and functional features.

The advances in digital medical imaging and storage in integrated databases are resulting in growing demands for efficient image retrieval and management. Content-based image retrieval (CBIR) refers to the retrieval of images from a database, using the visual features derived from the information in the image, and has become an attractive approach to managing large medical image archives. In conventional CBIR systems for medical images, images are often segmented into regions which are used to derive two-dimensional visual features for region-based queries. Although such approach has the advantage of including only relevant regions in the formulation of a query, medical images that are inherently multidimensional can potentially benefit from the multidimensional feature extraction which could open up new opportunities in visual feature extraction and retrieval. In this study, we present a volume of interest (VOI) based content-based retrieval of four-dimensional (three spatial and one temporal) dynamic PET images. By segmenting the images into VOIs consisting of functionally similar voxels (e.g., a tumor structure), multidimensional visual and functional features were extracted and used as region-based query features. A prototype VOI-based functional image retrieval system (VOI-FIRS) has been designed to demonstrate the proposed multidimensional feature extraction and retrieval. Experimental results show that the proposed system allows for the retrieval of related images that constitute similar visual and functional VOI features, and can find potential applications in medical data management, such as to aid in education, diagnosis, and statistical analysis.     

Efficient Cloud Image Retrieval System Using Weighted-Inverted Index and Database Filtering Algorithms

With the advance of multimedia technology and communications, images and videos become the major streaming information through the Internet. How to fast retrieve desired similar images precisely from the Internet scale image/video databases is the most important retrieval control target. In this paper, a cloud based content-based image retrieval (CBIR) scheme is presented. Database-categorizing based on weighted-inverted index (DCWⅡ) and database filtering algorithm (DFA) is used to speed up the features matching process. In the DCWⅡ, the weights are assigned to discrete cosine transform (DCT) coefficients histograms and the database is categorized by weighted features. In addition, the DFA filters out the irrelevant image in the database to reduce unnecessary computation loading for features matching. Experiments show that the proposed CBIR scheme outperforms previous work in the precision-recall performance and maintains mean average precision (mAP) about 0.678 in the large-scale database comprising one million images. Our scheme also can reduce about 50% to 85% retrieval time by pre-filtering the database, which helps to improve the efficiency of retrieval systems.     

Indexing and encoding based image feature representation with bin overlapped similarity measure for CBIR applications

In Content Based Image Retrieval (CBIR) system, the exhaustive search for a given query image to find the relevant images in the database are non-scalable. In this paper, we propose indexing, coding technique and similarity measure to address the above mentioned problem. We consider the color histogram of the image and its bin values are analyzed to understand the color information in the image. The histogram dimension is reduced by removing trivial bins and only those bins that represent color information significantly are considered. Based on the dimensions of the histogram, it is clustered and indexed. The Golomb–Rice (GR) coding is used to encode the indexed histograms. The Bin Overlapped Similarity Measure (BOSM) is proposed to compute the distance values between query and database image histograms. The performance of proposed approach is evaluated on benchmark datasets and found that the performance of the proposed approach is encouraging.     

Learning semantic and visual similarity for endomicroscopy video retrieval

Content-based image retrieval (CBIR) is a valuable computer vision technique which is increasingly being applied in the medical community for diagnosis support. However, traditional CBIR systems only deliver visual outputs, i.e., images having a similar appearance to the query, which is not directly interpretable by the physicians. Our objective is to provide a system for endomicroscopy video retrieval which delivers both visual and semantic outputs that are consistent with each other. In a previous study, we developed an adapted bag-of-visual-words method for endomicroscopy retrieval, called "Dense-Sift," that computes a visual signature for each video. In this paper, we present a novel approach to complement visual similarity learning with semantic knowledge extraction, in the field of in vivo endomicroscopy. We first leverage a semantic ground truth based on eight binary concepts, in order to transform these visual signatures into semantic signatures that reflect how much the presence of each semantic concept is expressed by the visual words describing the videos. Using cross-validation, we demonstrate that, in terms of semantic detection, our intuitive Fisher-based method transforming visual-word histograms into semantic estimations outperforms support vector machine (SVM) methods with statistical significance. In a second step, we propose to improve retrieval relevance by learning an adjusted similarity distance from a perceived similarity ground truth. As a result, our distance learning method allows to statistically improve the correlation with the perceived similarity. We also demonstrate that, in terms of perceived similarity, the recall performance of the semantic signatures is close to that of visual signatures and significantly better than those of several state-of-the-art CBIR methods. The semantic signatures are thus able to communicate high-level medical knowledge while being consistent with the low-level visual signatures and much shorter than them. In our resulting retrieval system, we decide to use visual signatures for perceived similarity learning and retrieval, and semantic signatures for the output of an additional information, expressed in the endoscopist own language, which provides a relevant semantic translation of the visual retrieval outputs.     

A CBIR system based on saliency driven local image features and multi orientation texture features

In Content-based Image Retrieval (CBIR), the user provides the query image in which only a selective portion of the image carries the foremost vital information known as the object region of the image. However, the human visual system also focuses on a particular salient region of an image to instinctively understand its semantic meaning. Therefore, the human visual attention technique can be well imposed in the CBIR scheme. Inspired by these facts, we initially utilized the signature saliency map-based approach to decompose the image into its respective main object region (ObR) and non-object region (NObR). ObR possesses most of the vital image information, so block-level normalized singular value decomposition (SVD) has been used to extract salient features of the ObR. In most natural images, NObR plays a significant role in understanding the actual semantic meaning of the image. Accordingly, multi-directional texture features have been extracted from NObR using Gabor filter on different wavelengths. Since the importance of ObR and NObR features are not equal, a new homogeneity-based similarity matching approach has been devised to enhance retrieval accuracy. Finally, we have demonstrated retrieval performances using both the combined and distinct ObR and NObR features on seven standard coral, texture, object, and heterogeneous datasets. The experimental outcomes show that the proposed CBIR system has a promising retrieval efficiency and outperforms various existing systems substantially.     

基于Zernike矩与SIFT特征的商标检索算法

为降低商标检索算法的误检率,提出一种结合Zernike矩(ZM)和尺度不变特征变换(SIFT)的商标检索算法,该算法由离线数据库构建和在线检索组成。分别从查询图像中提取ZM和SIFT特征;根据查询图像的特征集与数据库中存储的图像的特征集之间的ZM特征进行相似度度量,形成候选商标集;最后,利用SIFT特征对查询图像与候选图像精准检测,对相似距离进行排序,将结果返回给用户。实验结果表明：与当前流行的商标检索算法相比,该算法具备更好的检索性能,在缩放、平移、模糊、透视、斜切、扭曲等变换干扰下,仍呈现出更理想的Precision- Recall曲线以及F值。     

一种新的图像检索方法

常规基于内容图像检索的方法是提取图像的颜色、纹理等物理特征,运用相似性度量准则从图像库中查询相似的图像。为了提高图像检索的正确率,这里提出改进的方法。具体方法是:提取图像的物理特征,并将特征作为支持向量机(SVM)的输入向量,对图像进行分类,然后利用分类结果,对检索图像进行相似性匹配,从同类图像中找出相似的图像。实验结果显示,该方法的检索结果优于常规方法。     

A framework for medical image retrieval using machine learning and statistical similarity matching techniques with relevance feedback.

A content-based image retrieval (CBIR) framework for diverse collection of medical images of different imaging modalities, anatomic regions with different orientations and biological systems is proposed. Organization of images in such a database (DB) is well defined with predefined semantic categories; hence, it can be useful for category-specific searching. The proposed framework consists of machine learning methods for image prefiltering, similarity matching using statistical distance measures, and a relevance feedback (RF) scheme. To narrow down the semantic gap and increase the retrieval efficiency, we investigate both supervised and unsupervised learning techniques to associate low-level global image features (e.g., color, texture, and edge) in the projected PCA-based eigenspace with their high-level semantic and visual categories. Specially, we explore the use of a probabilistic multiclass support vector machine (SVM) and fuzzy c-mean (FCM) clustering for categorization and prefiltering of images to reduce the search space. A category-specific statistical similarity matching is proposed in a finer level on the prefiltered images. To incorporate a better perception subjectivity, an RF mechanism is also added to update the query parameters dynamically and adjust the proposed matching functions. Experiments are based on a ground-truth DB consisting of 5000 diverse medical images of 20 predefined categories. Analysis of results based on cross-validation (CV) accuracy and precision-recall for image categorization and retrieval is reported. It demonstrates the improvement, effectiveness, and efficiency achieved by the proposed framework.     

一种改进的颜色匹配对图像检索算法

以子块直方图彩色图像检索算法为基础， 分析了进一步利用图像空间相似信息的颜色匹配对检索算法的性能。在子块直方图的构成、直方图距离值的归类等方面提出了行之有效的改进方法；给出了子块大小、相似度阈值等参数选择的优化原则，使查准率、查全率等检索性能指标得到了较大的提高，得出了几个有用的结论并形成了实验系统。     

Semantic content-based image retrieval: A comprehensive study

The complexity of multimedia contents is significantly increasing in the current digital world. This yields an exigent demand for developing highly effective retrieval systems to satisfy human needs. Recently, extensive research efforts have been presented and conducted in the field of content-based image retrieval (CBIR). The majority of these efforts have been concentrated on reducing the semantic gap that exists between low-level image features represented by digital machines and the profusion of high-level human perception used to perceive images. Based on the growing research in the recent years, this paper provides a comprehensive review on the state-of-the-art in the field of CBIR. Additionally, this study presents a detailed overview of the CBIR framework and improvements achieved; including image preprocessing, feature extraction and indexing, system learning, benchmarking datasets, similarity matching, relevance feedback, performance evaluation, and visualization. Finally, promising research trends, challenges, and our insights are provided to inspire further research efforts.     

Development of Content-Based Trademark Retrieval System on the World Wide Web

In this paper, we describe a new trademark retrieval system based upon the content or the shape of trademark. The system has an on-line graphical user interface for the World Wide Web (WWW) that allows user to provide a query in forms of a sketch or a visual image to search for similar trademarks from database. User interfaces for the WWW were implemented by utilizing HTML and Java applets. The query can occur in arbitrary size and orientation. A shape representation scheme invariant to scale and rotation was developed to measure the similarity between two trademarks using the magnitude of Zernike moments as a feature set. Performance evaluation has been carried out with a database of 3,000 trademarks. It takes only about 0.6 second for the retrieval on a 200 MHz Pentium PC. The average recall of the original one among top 30 candidates queried by noisy or deformed images was 100%.     

A shape-based image retrieval method using salient edges

Content-based image retrieval is emerging as an important research area with applications in digital libraries and multimedia databases. In this paper, we present a novel five-stage image retrieval method based on salient edges. In the first stage, the Canny operator is performed to detect edge points. Then, the Water-Filling algorithm is employed to extract edge curves. In the third stage, salient edges are selected and the shape features in terms of the salient edges are yielded. In the fourth stage, a similarity measure, namely the integrated salient edge matching, that integrates properties of all the salient edges, is introduced, and used to compare the similarity of the query image with the images in the database. Finally, the best matches are returned in similarity order. The presented approach is easy to implement and can be efficiently applied to retrieve images with clear edges. Preliminary experimental results on a database containing 6500 images are very promising.     

基于“点到面”策略的图像检索

图像检索是计算机视觉领域的一个重要分支。其主要目的是从图像数据库中找出与查询图像相似的语义图像。传统的图像检索方法是在查询图像和数据库图像之间进行“点到点”检索。但是,单个查询图像包含的类别提示较少,即类别信息较弱,使得检索结果并不理想。为了解决这个问题,本文提出了一种基于“点到面”的类别检索策略来扩展一个图像（点）到一个图像类别（面）,这意味着从单个查询图像到整个图像类别的语义扩展。该方法挖掘了查询图像的类别信息。在两个常用的数据集上对所提出方法的性能进行了评估。实验表明,该方法可以显著提高图像检索的性能。