Topic modelling of clickthrough data in image search

In this paper we explore the benefits of latent variable modelling of clickthrough data in the domain of image retrieval. Clicks in image search logs are regarded as implicit relevance judgements that express both user intent and important relations between selected documents. We posit that clickthrough data contains hidden topics and can be used to infer a lower dimensional latent space that can be subsequently employed to improve various aspects of the retrieval system. We use a subset of a clickthrough corpus from the image search portal of a news agency to evaluate several popular latent variable models in terms of their ability to model topics underlying queries. We demonstrate that latent variable modelling reveals underlying structure in clickthrough data and our results show that computing document similarities in the latent space improves retrieval effectiveness compared to computing similarities in the original query space. These results are compared with baselines using visual and textual features. We show performance substantially better than the visual baseline, which indicates that content-based image retrieval systems that do not exploit query logs could improve recall and precision by taking this historical data into account.     

Neural ranking for automatic image annotation

Automatic image annotation aims to predict labels for images according to their semantic contents and has become a research focus in computer vision, as it helps people to edit, retrieve and understand large image collections. In the last decades, researchers have proposed many approaches to solve this task and achieved remarkable performance on several standard image datasets. In this paper, we propose a novel learning to rank approach to address image auto-annotation problem. Unlike typical learning to rank algorithms for image auto-annotation which directly rank annotations for image, our approach consists of two phases. In the first phase, neural ranking models are trained to rank image’s semantic neighbors. Then nearest-neighbor based models propagate annotations from these semantic neighbors to the image. Thus our approach integrates learning to rank algorithms and nearest-neighbor based models, including TagProp and 2PKNN, and inherits their advantages. Experimental results show that our method achieves better or comparable performance compared with the state-of-the-art methods on four challenging benchmarks including Corel5K, ESP Games, IAPR TC-12 and NUS-WIDE.

     

Methods for automatic and assisted image annotation

Personal memories composed of digital pictures are very popular at the moment. To retrieve these media items annotation is required. During the last years, several approaches have been proposed in order to overcome the image annotation problem. This paper presents our proposals to address this problem. Automatic and semi-automatic learning methods for semantic concepts are presented. The automatic method is based on semantic concepts estimated using visual content, context metadata and audio information. The semi-automatic method is based on results provided by a computer game. The paper describes our proposals and presents their evaluations.     

Incorporating multiple SVMs for automatic image annotation

In this paper, a novel automatic image annotation system is proposed, which integrates two sets of support vector machines (SVMs), namely the multiple instance learning (MIL)-based and global-feature-based SVMs, for annotation. The MIL-based bag features are obtained by applying MIL on the image blocks, where the enhanced diversity density (DD) algorithm and a faster searching algorithm are applied to improve the efficiency and accuracy. They are further input to a set of SVMs for finding the optimum hyperplanes to annotate training images. Similarly, global color and texture features, including color histogram and modified edge histogram, are fed into another set of SVMs for categorizing training images. Consequently, two sets of image features are constructed for each test image and are, respectively, sent to the two sets of SVMs, whose outputs are incorporated by an automatic weight estimation method to obtain the final annotation results. Our proposed annotation approach demonstrates a promising performance for an image database of 12 000 general-purpose images from COREL, as compared with some current peer systems in the literature.     

基于日志的协同图像自动标注

反馈日志隐含的图像语义信息有助于图像自动标注,但日志数据中存在的噪声、片面性等问题制约了其作用,故提出基于日志的协同图像自动标注算法。根据日志获取的特点,采用增量关联规则挖掘处理日志信息去除其噪声,利用协同滤波思想扩展图像标注词数量,利用WordNet得到标注词间关系,并结合图像底层特征利用混合概率模型实现图像自动标注。在Corel5K和互联网数据集上的实验表明：该算法降低了日志噪声及片面性所带来的影响,提高了图像自动标注效率和质量。     

A two-level model for automatic image annotation

Image automatic annotation is a significant and challenging problem in pattern recognition and computer vision. Current image annotation models almost used all the training images to estimate joint generation probabilities between images and keywords, which would inevitably bring a lot of irrelevant images. To solve the above problem, we propose a hierarchical image annotation model which combines advantages of discriminative model and generative model. In first annotation layer, discriminative model is used to assign topic annotations to unlabeled images, and then relevant image set corresponding to each unlabeled image is obtained. In second annotation layer, we propose a keywords-oriented method to establish links between images and keywords, and then our iterative algorithm is used to expand relevant image sets. Candidate labels will be given higher weights by using our method based on visual keywords. Finally, generative model is used to assign detailed annotations to unlabeled images on expanded relevant image sets. Experiments conducted on Corel 5K datasets verify the effectiveness of our hierarchical image annotation model.     

图像自动标注技术研究进展

现有图像自动标注技术算法可以大致划分为基于语义的标注算法、基于矩阵分解的标注算法、基于概率的标注算法以及基于图学习的标注算法等四大类。介绍了各类别中具有代表性的图像自动标注算法,分析了这些算法的问题模型及其功能特点,并归纳了图像自动标注算法中主要的优化求解方法及算法评价中常用的图像数据集和性能评价指标。最后,指出了图像自动标注技术目前存在的主要问题,并且提出了这些问题的解决思路。分析结果表明,对于图像自动标注技术的研究,可充分利用现有算法的优势互补,或借助多学科交叉的优势,寻找更有效的算法。     

A hybrid automatic image annotation approach

Multimedia Tools and Applications - Automated image annotation (AIA) is an important issue in computer vision and pattern recognition, and plays an extremely important role in retrieving...     

The effectiveness of image features based on fractal image coding for image annotation

Image annotation is a process of assigning metadata to digital images in the form of captions or keywords, and has been regarded as image management and one of the most crucial processes of image retrieval. And many automatic methods have been proposed. However, these methods still have some problems respectively. Fractals are fragmented geometries and can be considered separate parts; each part is similar to the contracted overall shape. Fractal features provide geometric information of an image that is irrelevant to the shape and size of an object in the image; therefore, fractal features are more robust than color and texture features. Therefore, this study proposed a fractal-driven image annotation (FIA) schema that extracts fractal features through fractal image coding and integrates color and texture as new visual features to conduct image-based annotation. Experimental results indicate that the effect of thresholds on annotating accuracy is insignificant. This finding supports the application of FIA on complex practical environments, reduces the time for identifying the optimal thresholds, and improves the practicality of using FIA in real environments.     

混合生成式和判别式模型的图像自动标注

目的 由于图像检索中存在着低层特征和高层语义之间的“语义鸿沟”,图像自动标注成为当前的关键性问题.为缩减语义鸿沟,提出了一种混合生成式和判别式模型的图像自动标注方法.方法 在生成式学习阶段,采用连续的概率潜在语义分析模型对图像进行建模,可得到相应的模型参数和每幅图像的主题分布.将这个主题分布作为每幅图像的中间表示向量,那么图像自动标注的问题就转化为一个基于多标记学习的分类问题.在判别式学习阶段,使用构造集群分类器链的方法对图像的中间表示向量进行学习,在建立分类器链的同时也集成了标注关键词之间的上下文信息,因而能够取得更高的标注精度和更好的检索效果.结果 在两个基准数据集上进行的实验表明,本文方法在Corel5k数据集上的平均精度、平均召回率分别达到0.28和0.32,在IAPR-TC12数据集上则达到0.29和0.18,其性能优于大多数当前先进的图像自动标注方法.此外,从精度—召回率曲线上看,本文方法也优于几种典型的具有代表性的标注方法.结论 提出了一种基于混合学习策略的图像自动标注方法,集成了生成式模型和判别式模型各自的优点,并在图像语义检索的任务中表现出良好的有效性和鲁棒性.本文方法和技术不仅能应用于图像检索和识别的领域,经过适当的改进之后也能在跨媒体检索和数据挖掘领域发挥重要作用.     

A MapReduce-based distributed SVM algorithm for automatic image annotation

Machine learning techniques have facilitated image retrieval by automatically classifying and annotating images with keywords. Among them Support Vector Machines (SVMs) have been used extensively due to their generalization properties. However, SVM training is notably a computationally intensive process especially when the training dataset is large. This paper presents MRSMO, a MapReduce based distributed SVM algorithm for automatic image annotation. The performance of the MRSMO algorithm is evaluated in an experimental environment. By partitioning the training dataset into smaller subsets and optimizing the partitioned subsets across a cluster of computers, the MRSMO algorithm reduces the training time significantly while maintaining a high level of accuracy in both binary and multiclass classifications.     

A review on automatic image annotation techniques

Nowadays, more and more images are available. However, to find a required image for an ordinary user is a challenging task. Large amount of researches on image retrieval have been carried out in the past two decades. Traditionally, research in this area focuses on content based image retrieval. However, recent research shows that there is a semantic gap between content based image retrieval and image semantics understandable by humans. As a result, research in this area has shifted to bridge the semantic gap between low level image features and high level semantics. The typical method of bridging the semantic gap is through the automatic image annotation (AIA) which extracts semantic features using machine learning techniques. In this paper, we focus on this latest development in image retrieval and provide a comprehensive survey on automatic image annotation. We analyse key aspects of the various AIA methods, including both feature extraction and semantic learning methods. Major methods are discussed and illustrated in details. We report our findings and provide future research directions in the AIA area in the conclusions     

Combining global, regional and contextual features for automatic image annotation

This paper presents a novel approach to automatic image annotation which combines global, regional, and contextual features by an extended cross-media relevance model. Unlike typical image annotation methods which use either global or regional features exclusively, as well as neglect the textual context information among the annotated words, the proposed approach incorporates the three kinds of information which are helpful to describe image semantics to annotate images by estimating their joint probability. Specifically, we describe the global features as a distribution vector of visual topics and model the textual context as a multinomial distribution. The global features provide the global distribution of visual topics over an image, while the textual context relaxes the assumption of mutual independence among annotated words which is commonly adopted in most existing methods. Both the global features and textual context are learned by a probability latent semantic analysis approach from the training data. The experiments over 5k Corel images have shown that combining these three kinds of information is beneficial in image annotation.     

A multi-view-group non-negative matrix factorization approach for automatic image annotation

Multimedia Tools and Applications - In automatic image annotation (AIA) different features describe images from different aspects or views. Part of information embedded in some views is common for...     

基于图文混排的传统服饰图像以文标图算法

针对高效解读和智能处理海量图文资料是一项极具挑战并具有实用价值工作,而自动标注精度又面临依赖训练样本的难题,提出了一种基于数字图文混排书籍以文标图方法,由混排版式识别预处理、领域图像语义标签构建和大标签空间以文标图算法3部分组成.首先,通过提出的混排版式识别离算法,提取数字图文混排版式中图像、标题及描述文本等内容.然后...     

基于决策融合的图像自动标注方法

结合多媒体描述接口（MPEG-7）和MM（Mixture Model）混合模型,实现了基于决策融合的图像自动标注。在图像标注过程中,分别利用颜色描述子和纹理描述子为每个主题下的图像建立MM混合模型,实现低层视觉特征到高层语义空间的映射,利用局部决策融合方式融合在颜色和纹理MM混合模型下的标注结果,实现图像自动标注。通过在corel图像数据集上的实验,表明提出的局部决策融合方式能更充分利用图像的颜色和纹理信息,提高了图像标注性能。     

基于外部数据库的图像自动标注改善模型

针对图像标注中所使用数据集存在的数据不平衡问题,提出一种新的基于外部数据库的自动平衡模型。该模型先依据原始数据库中词频分布来找出低频点,再根据自动平衡模式,对每个低频词,从外部数据库中增加相应的图片;然后对图片进行特征提取,对Corel 5k数据集中的47065个视觉词汇和从外部数据库中追加的图片中提取出来的996个视觉词汇进行聚类;最后利用基于外部数据库的图像自动标注改善模型对图像进行标注。此方法克服了图像标注中数据库存在的不平衡问题,使得至少被正确标注一次的词的数量、精确率和召回率等均明显提高。