Collaborative image retrieval via regularized metric learning

In content-based image retrieval (CBIR), relevant images are identified based on their similarities to query images. Most CBIR algorithms are hindered by the semantic gap between the low-level image features used for computing image similarity and the high-level semantic concepts conveyed in images. One way to reduce the semantic gap is to utilize the log data of users' feedback that has been collected by CBIR systems in history, which is also called “collaborative image retrieval.” In this paper, we present a novel metric learning approach, named “regularized metric learning,” for collaborative image retrieval, which learns a distance metric by exploring the correlation between low-level image features and the log data of users' relevance judgments. Compared to the previous research, a regularization mechanism is used in our algorithm to effectively prevent overfitting. Meanwhile, we formulate the proposed learning algorithm into a semidefinite programming problem, which can be solved very efficiently by existing software packages and is scalable to the size of log data. An extensive set of experiments has been conducted to show that the new algorithm can substantially improve the retrieval accuracy of a baseline CBIR system using Euclidean distance metric, even with a modest amount of log data. The experiment also indicates that the new algorithm is more effective and more efficient than two alternative algorithms, which exploit log data for image retrieval.     

Neighborhood preserving regression for image retrieval

In content-based image retrieval (CBIR), relevance feedback has been proven to be a powerful tool for bridging the gap between low level visual features and high level semantic concepts. Traditionally, relevance feedback driven CBIR is often considered as a supervised learning problem where the user provided feedbacks are used to learn a distance metric or classification function. However, CBIR is intrinsically a semi-supervised learning problem in which the testing samples (images in the database) are present during the learning process. Moreover, when there are no sufficient feedbacks, these methods may suffer from the overfitting problem. In this paper, we propose a novel neighborhood preserving regression algorithm which makes efficient use of both labeled and unlabeled images. By using the unlabeled images, the geometrical structure of the image space can be incorporated into the learning system through a regularizer. Specifically, from all the functions which minimize the empirical loss on the labeled images, we select the one which best preserves the local neighborhood structure of the image space. In this way, our method can obtain a regression function which respects both semantic and geometrical structures of the image database. We present experimental evidence suggesting that our algorithm is able to use unlabeled data effectively for image retrieval.     

Automatic localization and annotation of facial features using machine learning techniques

Content-based image retrieval (CBIR) systems traditionally find images within a database that are similar to query image using low level features, such as colour histograms. However, this requires a user to provide an image to the system. It is easier for a user to query the CBIR system using search terms which requires the image content to be described by semantic labels. However, finding a relationship between the image features and semantic labels is a challenging problem to solve. This paper aims to discover semantic labels for facial features for use in a face image retrieval system. Face image retrieval traditionally uses global face-image information to determine similarity between images. However little has been done in the field of face image retrieval to use local face-features and semantic labelling. Our work aims to develop a clustering method for the discovery of semantic labels of face-features. We also present a machine learning based face-feature localization mechanism which we show has promise in providing accurate localization.     

基于深度学习的图像检索系统

基于内容的图像检索系统关键的技术是有效图像特征的获取和相似度匹配策略.在过去,基于内容的图像检索系统主要使用低级的可视化特征,无法得到满意的检索结果,所以尽管在基于内容的图像检索上花费了很大的努力,但是基于内容的图像检索依旧是计算机视觉领域中的一个挑战.在基于内容的图像检索系统中,存在的最大的问题是“语义鸿沟”,即机器从低级的可视化特征得到的相似性和人从高级的语义特征得到的相似性之间的不同.传统的基于内容的图像检索系统,只是在低级的可视化特征上学习图像的特征,无法有效的解决“语义鸿沟”.近些年,深度学习技术的快速发展给我们提供了希望.深度学习源于人工神经网络的研究,深度学习通过组合低级的特征形成更加抽象的高层表示属性类别或者特征,以发现数据的分布规律,这是其他算法无法实现的.受深度学习在计算机视觉、语音识别、自然语言处理、图像与视频分析、多媒体等诸多领域取得巨大成功的启发,本文将深度学习技术用于基于内容的图像检索,以解决基于内容的图像检索系统中的“语义鸿沟”问题.     

A review on automatic image annotation techniques

Nowadays, more and more images are available. However, to find a required image for an ordinary user is a challenging task. Large amount of researches on image retrieval have been carried out in the past two decades. Traditionally, research in this area focuses on content based image retrieval. However, recent research shows that there is a semantic gap between content based image retrieval and image semantics understandable by humans. As a result, research in this area has shifted to bridge the semantic gap between low level image features and high level semantics. The typical method of bridging the semantic gap is through the automatic image annotation (AIA) which extracts semantic features using machine learning techniques. In this paper, we focus on this latest development in image retrieval and provide a comprehensive survey on automatic image annotation. We analyse key aspects of the various AIA methods, including both feature extraction and semantic learning methods. Major methods are discussed and illustrated in details. We report our findings and provide future research directions in the AIA area in the conclusions     

Relevance Feedback and Learning in Content-Based Image Search

A major bottleneck in content-based image retrieval (CBIR) systems or search engines is the large gap between low-level image features used to index images and high-level semantic contents of images. One solution to this bottleneck is to apply relevance feedback to refine the query or similarity measures in image search process. In this paper, we first address the key issues involved in relevance feedback of CBIR systems and present a brief overview of a set of commonly used relevance feedback algorithms. Almost all of the previously proposed methods fall well into such framework. We present a framework of relevance feedback and semantic learning in CBIR. In this framework, low-level features and keyword annotations are integrated in image retrieval and in feedback processes to improve the retrieval performance. We have also extended framework to a content-based web image search engine in which hosting web pages are used to collect relevant annotations for images and users' feedback logs are used to refine annotations. A prototype system has developed to evaluate our proposed schemes, and our experimental results indicated that our approach outperforms traditional CBIR system and relevance feedback approaches.     

基于深度语义模型的乳腺X线图像检索

图像特征是基于内容的图像检索（Content-based image retrieval,CBIR）的关键,大部分使用的手工特征难以有效地表示乳腺肿块的特征,底层特征与高层语义之间存在语义鸿沟。为了提高CBIR的检索性能,本文采用深度学习来提取图像的高层语义特征。由于乳腺X线图像的深度卷积特征在空间和特征维度上存在一定的冗余和噪声,本文在词汇树和倒排文件的基础上,对深度特征的空间和语义进行优化,构建了两种不同的深度语义树。为了充分发挥深度卷积特征的识别能力,根据乳腺图像深度特征的局部特性对树节点的权重进行细化,提出了两种节点加权方法,得到了更好的检索结果。本文从乳腺X线图像数据库（Digital database for screening mammography, DDSM）中提取了2 200个感兴趣区域（Region of interest,ROIs）作为数据集,实验结果表明,该方法能够有效提高感兴趣肿块区域的检索精度和分类准确率,并且具有良好的可扩展性。     

EGO: A personalized multimedia management and retrieval tool

The problems of content‐based image retrieval (CBIR) systems can be attributed to the semantic gap between the low‐level data representation and the high‐level concepts the user associates with images, on the one hand, and the time‐varying and often vague nature of the underlying information need, on the other. These problems can be addressed by improving the interaction between the user and the system. In this article, we sketch the development of CBIR interfaces and introduce our view on how to solve some of the problems these interfaces present. To address the semantic gap and long‐term multifaceted information needs, we propose a “retrieval in context” system, EGO. EGO is a tool for the management of image collections, supporting the user through personalization and adaptation. We will describe how it learns from the user's personal organization, allowing it to recommend relevant images to the user. The recommendation algorithm is described, which is based on relevance feedback techniques. Additionally, we provide results of a performance analysis of the recommendation system and of a preliminary user study. © 2006 Wiley Periodicals, Inc. Int J Int Syst 21: 725–745, 2006.     

A multiple instance learning based framework for semantic image segmentation

Most image segmentation algorithms extract regions satisfying visual uniformity criteria. Unfortunately, because of the semantic gap between low-level features and high-level semantics, such regions usually do not correspond to meaningful parts. This has motivated researchers to develop methods that, by introducing high-level knowledge into the segmentation process, can break through the performance ceiling imposed by the semantic gap. The main disadvantage of those methods is their lack of flexibility due to the assumption that such knowledge is provided in advance. In content-based image retrieval (CBIR), relevance feedback (RF) learning has been successfully applied as a technique aimed at reducing the semantic gap. Inspired by this, we present a RF-based CBIR framework that uses multiple instance learning to perform a semantically-guided context adaptation of segmentation parameters. A partial instantiation of this framework that uses mean shift-based segmentation is presented. Experiments show the effectiveness and flexibility of the proposed framework on real images.     

An effective technique for the content based image retrieval to reduce the semantic gap based on an optimal classifier technique

Content Based Image Retrieval (CBIR) systems use Relevance Feedback (RF) in order to improve the retrieval accuracy. Research focus has been shifted from designing sophisticated low-level feature extraction algorithms to reducing the “semantic gap” between the visual features and the richness of human semantics. In this paper, a novel system is proposed to enhance the gain of long-term relevance feedback. In the proposed system, the general CBIR involves two steps—ABC based training and image retrieval. First, the images other than the query image are pre-processed using median filter and gray scale transformation for removal of noise and resizing. Secondly, the features such as Color, Texture and shape of the image are extracted using Gabor Filter, Gray Level Co-occurrence Matrix and Hu-Moment shape feature techniques and also extract the static features like mean and standard deviation. The extracted features are clustered using k-means algorithm and each cluster are trained using ANN based ABC technique. A method using artificial bee colony (ABC) based artificial neural network (ANN) to update the weights assigned to features by accumulating the knowledge obtained from the user over iterations. Eventually, the comparative analysis performed using the commonly used methods namely precision and recall were clearly shown that the proposed system is suitable for the better CBIR and it can reduce the semantic gap than the conventional systems.     

基于内容的图象检索技术

随着数字图象的日益增多,基于内容的图象检索已成为图象使用者和管理者迫切需要解决的问题,近年来,各国研究者纷纷加入该领域的研究.为了使人们对该领域现状有个概略了解,以推动该领域研究进一步开展,首先概括介绍了基于内容图象检索的产生、发展及其关键技术;然后介绍了特征提取(包括低层特征和语义特征)及其相似性计算、相关反馈等的原理及算法;最后指出了基于内容的图象检索技术与计算机视觉技术的区别所在,并对目前存在的问题和应着重的研究内容以及发展方向进行了分析.     

基于内容的图象检索中的语义处理方法

基于内容的图象检索系统,其目标是最大限度地减小图象简单视觉特征与用户检索丰富语义之间的“语义鸿沟”,因此图象语义处理则成为基于内容的图象检索进一步发展的关键。为了使人们对基于内容的图象检索中的语义处理方法有个概略了解,首先从图象语义模型和图象语义提取方法这两个方面对利用语义进行图象检索的研究状况进行了总结,并将图象语义模型概括为图象语义知识、图象语义层次模型和语义抽取模型等3个主要组成部分;然后将图象语义提取方法分为用户交互、将查询请求作为语义模板、对象及其空间关系、场景和行为语义及情感语义等类别,同时对其中有代表性的方法进行了详细的分析,还指出了其局限性;最后从对象建模和识别、语义抽取规则和用户检索模型3个方面,阐明了实现图象语义处理所面临的问题,并提出了一些初步的解决思路。     

多媒体技术研究:2014——深度学习与媒体计算

目的 海量数据的快速增长给多媒体计算带来了深刻挑战。与传统以手工构造为核心的媒体计算模式不同,数据驱动下的深度学习(特征学习)方法成为当前媒体计算主流。方法 重点分析了深度学习在检索排序与标注、多模态检索与语义理解、视频分析与理解等媒体计算方面的最新进展和所面临的挑战,并对未来的发展趋势进行展望。结果 在检索排序与标注方面, 基于深度学习的神经编码等方法取得了很好的效果;在多模态检索与语义理解方面,深度学习被用于弥补不同模态间的“异构鸿沟“以及底层特征与高层语义间的”语义鸿沟“,基于深度学习的组合语义学习成为研究热点;在视频分析与理解方面, 深度神经网络被用于学习视频的有效表示方式及动作识别,并取得了很好的效果。然而,深度学习是一种数据驱动的方法,易受数据噪声影响, 对于在线增量学习方面还不成熟,如何将深度学习与众包计算相结合是一个值得期待的问题。结论 该综述在深入分析现有方法的基础上,对深度学习框架下为解决异构鸿沟和语义鸿沟给出新的思路。     

Content-based image retrieval and semantic automatic image annotation based on the weighted average of triangular histograms using support vector machine

In recent years, the rapid growth of multimedia content makes content-based image retrieval (CBIR) a challenging research problem. The content-based attributes of the image are associated with the position of objects and regions within the image. The addition of image content-based attributes to image retrieval enhances its performance. In the last few years, the bag-of-visual-words (BoVW) based image representation model gained attention and significantly improved the efficiency and effectiveness of CBIR. In BoVW-based image representation model, an image is represented as an order-less histogram of visual words by ignoring the spatial attributes. In this paper, we present a novel image representation based on the weighted average of triangular histograms (WATH) of visual words. The proposed approach adds the image spatial contents to the inverted index of the BoVW model, reduces overfitting problem on larger sizes of the dictionary and semantic gap issues between high-level image semantic and low-level image features. The qualitative and quantitative analysis conducted on three image benchmarks demonstrates the effectiveness of the proposed approach based on WATH.     

An image retrieval system with automatic query modification

Most interactive "query-by-example" based image retrieval systems utilize relevance feedback from the user for bridging the gap between the user's implied concept and the low-level image representation in the database. However, traditional relevance feedback usage in the context of content-based image retrieval (CBIR) may not be very efficient due to a significant overhead in database search and image download time in client-server environments. In this paper, we propose a CBIR system that efficiently addresses the inherent subjectivity in user perception during a retrieval session by employing a novel idea of intra-query modification and learning. The proposed system generates an object-level view of the query image using a new color segmentation technique. Color, shape and spatial features of individual segments are used for image representation and retrieval. The proposed system automatically generates a set of modifications by manipulating the features of the query segment(s). An initial estimate of user perception is learned from the user feedback provided on the set of modified images. This largely improves the precision in the first database search itself and alleviates the overheads of database search and image download. Precision-to-recall ratio is improved in further iterations through a new relevance feedback technique that utilizes both positive as well as negative examples. Extensive experiments have been conducted to demonstrate the feasibility and advantages of the proposed system.     

Medical image semantic segmentation based on deep learning

The image semantic segmentation has been extensively studying. The modern methods rely on the deep convolutional neural networks, which can be trained to address this problem. A few years ago networks require the huge dataset to be trained. However, the recent advances in deep learning allow training networks on the small datasets, which is a critical issue for medical images, since the hospitals and research organizations usually do not provide the huge amount of data. In this paper, we address medical image semantic segmentation problem by applying the modern CNN model. Moreover, the recent achievements in deep learning allow processing the whole image per time by applying concepts of the fully convolutional neural network. Our qualitative and quantitate experiment results demonstrated that modern CNN can successfully tackle the medical image semantic segmentation problem.