Semantic image retrieval in magnetic resonance brain volumes

Practitioners in the area of neurology often need to retrieve multimodal magnetic resonance (MR) images of the brain to study disease progression and to correlate observations across multiple subjects. In this paper, a novel technique for retrieving 2-D MR images (slices) in 3-D brain volumes is proposed. Given a 2-D MR query slice, the technique identifies the 3-D volume among multiple subjects in the database, associates the query slice with a specific region of the brain, and retrieves the matching slice within this region in the identified volumes. The proposed technique is capable of retrieving an image in multimodal and noisy scenarios. In this study, support vector machines (SVM) are used for identifying 3-D MR volume and for performing semantic classification of the human brain into various semantic regions. In order to achieve reliable image retrieval performance in the presence of misalignments, an image registration-based retrieval framework is developed. The proposed retrieval technique is tested on various modalities. The test results reveal superior robustness performance with respect to accuracy, speed, and multimodality.     

基于“点到面”策略的图像检索

图像检索是计算机视觉领域的一个重要分支。其主要目的是从图像数据库中找出与查询图像相似的语义图像。传统的图像检索方法是在查询图像和数据库图像之间进行“点到点”检索。但是,单个查询图像包含的类别提示较少,即类别信息较弱,使得检索结果并不理想。为了解决这个问题,本文提出了一种基于“点到面”的类别检索策略来扩展一个图像（点）到一个图像类别（面）,这意味着从单个查询图像到整个图像类别的语义扩展。该方法挖掘了查询图像的类别信息。在两个常用的数据集上对所提出方法的性能进行了评估。实验表明,该方法可以显著提高图像检索的性能。      

Indexing and encoding based image feature representation with bin overlapped similarity measure for CBIR applications

In Content Based Image Retrieval (CBIR) system, the exhaustive search for a given query image to find the relevant images in the database are non-scalable. In this paper, we propose indexing, coding technique and similarity measure to address the above mentioned problem. We consider the color histogram of the image and its bin values are analyzed to understand the color information in the image. The histogram dimension is reduced by removing trivial bins and only those bins that represent color information significantly are considered. Based on the dimensions of the histogram, it is clustered and indexed. The Golomb–Rice (GR) coding is used to encode the indexed histograms. The Bin Overlapped Similarity Measure (BOSM) is proposed to compute the distance values between query and database image histograms. The performance of proposed approach is evaluated on benchmark datasets and found that the performance of the proposed approach is encouraging.     

Spatial template extraction for image retrieval by region matching

This paper presents a template and its relation extraction and estimation (TREE) algorithm for indexing images from picture libraries with more semantics-sensitive meanings. This algorithm can learn the commonality of visual concepts from multiple images to give a middle-level understanding about image contents. In this approach, each image is represented by a set of templates and their spatial relations as keys to capture the essence of this image. Each template is characterized by a set of dominant regions, which reflect different appearances of an object at different conditions and can be obtained by the template extraction and analysis (TEA) algorithm through region matching. The spatial template relation extraction and measurement (STREAM) algorithm is then proposed for obtaining the spatial relations between these templates. Due to the nature of a template, which can represent object's appearances at different conditions, the proposed approach owns better capabilities and flexibilities to capture image contents than traditional region-based methods. In addition, through maintaining the spatial layout of images, the semantic meanings of the query images can be extracted and lead to significant improvements in the accuracy of image retrieval. Since no time-consuming optimization process is involved, the proposed method learns the visual concepts extremely fast. Experimental results are provided to prove the superiority of the proposed method.     

Content-based microscopic image retrieval system for multi-image queries

In this paper, we describe the design and development of a multitiered content-based image retrieval (CBIR) system for microscopic images utilizing a reference database that contains images of more than one disease. The proposed CBIR system uses a multitiered approach to classify and retrieve microscopic images involving their specific subtypes, which are mostly difficult to discriminate and classify. This system enables both multi-image query and slide-level image retrieval in order to protect the semantic consistency among the retrieved images. New weighting terms, inspired from information retrieval theory, are defined for multiple-image query and retrieval. The performance of the system was tested on a dataset including 1666 imaged high power fields extracted from 57 follicular lymphoma (FL) tissue slides with three subtypes and 44 neuroblastoma (NB) tissue slides with four subtypes. Each slide is semantically annotated according to their subtypes by expert pathologists. By using leave-one-slide out testing scheme, the multi-image query algorithm with the proposed weighting strategy achieves about 93% and 86% of average classification accuracy at the first rank retrieval, outperforming the image-level retrieval accuracy by about 38 and 26 percentage points, for FL and NB diseases, respectively.     

Effective image retrieval based on hidden concept discovery in image database.

This paper addresses content-based image retrieval in general, and in particular, focuses on developing a hidden semantic concept discovery methodology to address effective semantics-intensive image retrieval. In our approach, each image in the database is segmented into regions associated with homogenous color, texture, and shape features. By exploiting regional statistical information in each image and employing a vector quantization method, a uniform and sparse region-based representation is achieved. With this representation, a probabilistic model based on statistical-hidden-class assumptions of the image database is obtained, to which the expectation-maximization technique is applied to analyze semantic concepts hidden in the database. An elaborated retrieval algorithm is designed to support the probabilistic model. The semantic similarity is measured through integrating the posterior probabilities of the transformed query image, as well as a constructed negative example, to the discovered semantic concepts. The proposed approach has a solid statistical foundation; the experimental evaluations on a database of 10000 general-purposed images demonstrate its promise and effectiveness.     

基于嵌入式零树小波编码直方图图像检索

图像和视频应用的快速增长，使得根据图像和视频内容进行查询的技术变得越来越重要，人们提出了许多基于像素域或压缩域的图像检索技术，因为多媒体数据库通常具有相当大的数据量，所以基于像素域图像检索技术的计算复杂度相当大，因此，许多文献提出更快的基于压缩域的图像检索技术，本文提出一种改进的基于嵌入式零树小波编码直方图的图像检索技术，特征提取综合考虑图像的颜色，纹理，频率和空间信息，所有的特征可以在压缩过程中自动得到，图像检索的过程就是匹配待检索图像和来自数据库的侯选图像的索引，实验证明这种方法具有好的检索性能。     

A new way for multidimensional medical data management: volume of interest (VOI)-based retrieval of medical images with visual and functional features.

The advances in digital medical imaging and storage in integrated databases are resulting in growing demands for efficient image retrieval and management. Content-based image retrieval (CBIR) refers to the retrieval of images from a database, using the visual features derived from the information in the image, and has become an attractive approach to managing large medical image archives. In conventional CBIR systems for medical images, images are often segmented into regions which are used to derive two-dimensional visual features for region-based queries. Although such approach has the advantage of including only relevant regions in the formulation of a query, medical images that are inherently multidimensional can potentially benefit from the multidimensional feature extraction which could open up new opportunities in visual feature extraction and retrieval. In this study, we present a volume of interest (VOI) based content-based retrieval of four-dimensional (three spatial and one temporal) dynamic PET images. By segmenting the images into VOIs consisting of functionally similar voxels (e.g., a tumor structure), multidimensional visual and functional features were extracted and used as region-based query features. A prototype VOI-based functional image retrieval system (VOI-FIRS) has been designed to demonstrate the proposed multidimensional feature extraction and retrieval. Experimental results show that the proposed system allows for the retrieval of related images that constitute similar visual and functional VOI features, and can find potential applications in medical data management, such as to aid in education, diagnosis, and statistical analysis.     

Three-dimensional surface reconstruction using optical flow formedical imaging

The recovery of a three-dimensional (3-D) model from a sequence of two-dimensional (2-D) images is very useful in medical image analysis. Image sequences obtained from the relative motion between the object and the camera or the scanner contain more 3-D information than a single image. Methods to visualize the computed tomograms can be divided into two approaches: the surface rendering approach and the volume rendering approach. In this paper, a new surface rendering method using optical flow is proposed. Optical flow is the apparent motion in the image plane produced by the projection of real 3-D motion onto the 2-D image. The 3-D motion of an object can be recovered from the optical-flow field using additional constraints. By extracting the surface information from 3-D motion, it is possible to obtain an accurate 3-D model of the object. Both synthetic and real image sequences have been used to illustrate the feasibility of the proposed method. The experimental results suggest that the proposed method is suitable for the reconstruction of 3-D models from ultrasound medical images as well as other computed tomograms     

CLUE: cluster-based retrieval of images by unsupervised learning.

In a typical content-based image retrieval (CBIR) system, target images (images in the database) are sorted by feature similarities with respect to the query. Similarities among target images are usually ignored. This paper introduces a new technique, cluster-based retrieval of images by unsupervised learning (CLUE), for improving user interaction with image retrieval systems by fully exploiting the similarity information. CLUE retrieves image clusters by applying a graph-theoretic clustering algorithm to a collection of images in the vicinity of the query. Clustering in CLUE is dynamic. In particular, clusters formed depend on which images are retrieved in response to the query. CLUE can be combined with any real-valued symmetric similarity measure (metric or nonmetric). Thus, it may be embedded in many current CBIR systems, including relevance feedback systems. The performance of an experimental image retrieval system using CLUE is evaluated on a database of around 60,000 images from COREL. Empirical results demonstrate improved performance compared with a CBIR system using the same image similarity measure. In addition, results on images returned by Google's Image Search reveal the potential of applying CLUE to real-world image data and integrating CLUE as a part of the interface for keyword-based image retrieval systems.     

Color Texture Segmentation Based on the Modal Energy of Deformable Surfaces

This paper presents a new approach for the segmentation of color textured images, which is based on a novel energy function. The proposed energy function, which expresses the local smoothness of an image area, is derived by exploiting an intermediate step of modal analysis that is utilized in order to describe and analyze the deformations of a 3-D deformable surface model. The external forces that attract the 3-D deformable surface model combine the intensity of the image pixels with the spatial information of local image regions. The proposed image segmentation algorithm has two steps. First, a color quantization scheme, which is based on the node displacements of the deformable surface model, is utilized in order to decrease the number of colors in the image. Then, the proposed energy function is used as a criterion for a region growing algorithm. The final segmentation of the image is derived by a region merge approach. The proposed method was applied to the Berkeley segmentation database. The obtained results show good segmentation robustness, when compared to other state of the art image segmentation algorithms.     

Query Expansion by Text and Image Features in Image Retrieval

We present a two-pass image retrieval system in which retrieval techniques for text and image documents are combined in a novel approach. In the first pass, the text-based initial query is matched against the text captions of the images in the database to obtain the initial retrieved set. In the second pass, text and image features obtained from this initial retrieved set are used to expand the initial query. Additional images from the database are then retrieved based on the expanded query. The image features that we have used are color histograms, DC coefficients from the discrete cosine transform, and two texture features: multiresolution simultaneous autoregressive model and local binary pattern. These are low-level statistical image features that can be easily computed. Extensive experiments have been performed on 1019 color pictures of mixed variety with captions, relevance judgments and queries supplied by a national archives agency. Objective precision-recall results have been obtained with various combinations of text and image features. The results show that the image features do not perform well when used on their own. However, when image features are used in query expansion, they increase the average precision more significantly than text annotations. Moreover, these findings are valid at all precision levels and are not sensitive to the image feature acquisition parameters.     

图像集闭包建模的协同表示人脸识别算法

针对样本集距离分类算法忽略样本集内部变化的不足,利用图像多重描述提供的互补信息,提出图像集闭包建模的协同表示人脸识别算法。首先,扩展具有多重描述能力的图像集,图像的中等强度像素携带鉴别信息利用原始图像生成中等像素图像,镜像图像可增添图像细节信息利用原始图像产生镜像图像,将此两种源域图像与原始图像联合构成扩展的图像集。然后,以无参建模构建扩展的图像集为字典闭包,同类异源域的测试图像构成图像集且构建为测试闭包,借鉴协同表示思想利用字典学习迭代求解闭包系数。最后,采用残差判别函数进行模式分类。本文方法不仅构建具有多重描述能力的图像集,而且充分利用样本集内部关联性从而获得较好的分类结果。本文分别在ORL、GT（Georgia Tech Face Database）、CMU PIE人脸数据库上进行实验。      

Camera constraint-free view-based 3-D object retrieval

Recently, extensive research efforts have been dedicated to view-based methods for 3-D object retrieval due to the highly discriminative property of multiviews for 3-D object representation. However, most of state-of-the-art approaches highly depend on their own camera array settings for capturing views of 3-D objects. In order to move toward a general framework for 3-D object retrieval without the limitation of camera array restriction, a camera constraint-free view-based (CCFV) 3-D object retrieval algorithm is proposed in this paper. In this framework, each object is represented by a free set of views, which means that these views can be captured from any direction without camera constraint. For each query object, we first cluster all query views to generate the view clusters, which are then used to build the query models. For a more accurate 3-D object comparison, a positive matching model and a negative matching model are individually trained using positive and negative matched samples, respectively. The CCFV model is generated on the basis of the query Gaussian models by combining the positive matching model and the negative matching model. The CCFV removes the constraint of static camera array settings for view capturing and can be applied to any view-based 3-D object database. We conduct experiments on the National Taiwan University 3-D model database and the ETH 3-D object database. Experimental results show that the proposed scheme can achieve better performance than state-of-the-art methods.     

Perceptual Image Indexing and Retrieval

We describe a perceptual approach to generating features for use in indexing and retrieving images from an image database. Salient regions that immediately attract the eye are large color regions that usually dominate an image. Features derived from these will allow search for images that are similar perceptually. We compute color features and Gabor color texture features on regions obtained from a multiscale representation of the image, generated by a multiband smoothing algorithm based on human psychophysical measurements of color appearance. The combined feature vector is then used for indexing all salient regions of an image. For retrieval, those images are selected that contain more similar regions to the query image by using a multipass retrieval and ranking mechanism. Matches are found using the L₂ metric. The results demonstrate that the proposed method performs very well.     

面向微光/红外融合彩色夜视的场景解析方法

彩色夜视技术可以将微光/红外双谱图像融合成一幅适于人眼观察的彩色图像,而恰当的场景解析方法能够对彩色夜视图像的内容做出自动化分析,进一步减轻人眼的观测负担。针对彩色夜视场景丰富多变、对算法灵活性要求高的特点,提出了一种可在线扩展的场景解析方法。该方法基于非参数模型,预测景物类别时不需要训练过程,只需要使用数据库中具有语义标记的样本图像, 通过将待解析图像与样本图像进行全局及局部匹配来实现语义标签的传递。而且,数据库可以根据应用场景的不同随时进行动态扩充。实验结果表明:该方法在包含城市、乡野等多种场景的夜视图像上,以及由统计色彩映射、TNO、NRL等多种融合方法得到的、具有不同色彩表征的彩色夜视图像上都具有令人满意的准确率。     

Three Degree Binary Graph and Shortest Edge Clustering for re-ranking in multi-feature image retrieval

Graph methods have been widely employed in re-ranking for image retrieval. Although we can effectively find visually similar images through these methods, the ranking lists given by those approaches may contain some candidates which appear to be irrelevant to a query. Most of these candidates fall into two categories: (1) the irrelevant outliers located near to the query images in a graph; and (2) the images from another cluster which close to the query. Therefore, eliminating these two types of images from the ordered retrieval sets is expected to further boost the retrieval precision. In this paper, we build a Three Degree Binary Graph (TDBG) to eliminate the outliers and utilize a set-based greedy algorithm to reduce the influence of adjacent manifolds. Moreover, a multi-feature fusion method is proposed to enhance the retrieval performance further. Experimental results obtained on three public datasets demonstrate the superiority of the proposed approach.     

Multi-spectral and hyperspectral image fusion using 3-D wavelet transform

Image fusion is performed between one band of multi-spectral image and two bands of hyperspectral image to produce fused image with the same spatial resolution as source multi-spectral image and the same spectral resolution as source hyperspeetral image. According to the characteristics and 3-Dimensional （3-D） feature analysis of multi-spectral and hyperspectral image data volume, the new fusion approach using 3-D wavelet based method is proposed. This approach is composed of four major procedures： Spatial and spectral resampling, 3-D wavelet transform, wavelet coefficient integration and 3-D inverse wavelet transform. Especially, a novel method, Ratio Image Based Spectral Resampling （RIBSR）method, is proposed to accomplish data resampling in spectral domain by utilizing the property of ratio image. And a new fusion rule, Average and Substitution （A＆S） rule, is employed as the fusion rule to accomplish wavelet coefficient integration. Experimental results illustrate that the fusion approach using 3-D wavelet transform can utilize both spatial and spectral characteristics of source images more adequately and produce fused image with higher quality and fewer artifacts than fusion approach using 2-D wavelet transform. It is also revealed that RIBSR method is capable of interpolating the missing data more effectively and correctly, and A＆S rule can integrate coefficients of source images in 3-D wavelet domain to preserve both spatial and spectral features of source images more properly.     

Robust Gradient-Based 3-D/2-D Registration of CT and MR to X-Ray Images

One of the most important technical challenges in image-guided intervention is to obtain a precise transformation between the intrainterventional patient's anatomy and corresponding preinterventional 3-D image on which the intervention was planned. This goal can be achieved by acquiring intrainterventional 2-D images and matching them to the preinterventional 3-D image via 3-D/2-D image registration. A novel 3-D/2-D registration method is proposed in this paper. The method is based on robustly matching 3-D preinterventional image gradients and coarsely reconstructed 3-D gradients from the intrainterventional 2-D images. To improve the robustness of finding the correspondences between the two sets of gradients, hypothetical correspondences are searched for along normals to anatomical structures in 3-D images, while the final correspondences are established in an iterative process, combining the robust random sample consensus algorithm (RANSAC) and a special gradient matching criterion function. The proposed method was evaluated using the publicly available standardized evaluation methodology for 3-D/2-D registration, consisting of 3-D rotational X-ray, computed tomography, magnetic resonance (MR), and 2-D X-ray images of two spine segments, and standardized evaluation criteria. In this way, the proposed method could be objectively compared to the intensity, gradient, and reconstruction-based registration methods. The obtained results indicate that the proposed method performs favorably both in terms of registration accuracy and robustness. The method is especially superior when just a few X-ray images and when MR preinterventional images are used for registration, which are important advantages for many clinical applications.