一种基于稀疏典型性相关分析的图像检索方法?

图像语义检索的一个关键问题就是要找到图像底层特征与语义之间的关联,由于文本是表达语义的一种有效手段,因此提出通过研究文本与图像两种模态之间关系来构建反映两者间潜在语义关联的有效模型的思路。基于该模型,可使用自然语言形式(文本语句)来表达检索意图,最终检索到相关图像。该模型基于稀疏典型性相关分析(sparse canonical correlation analysis,简称sparse CCA),按照如下步骤训练得到：首先利用隐语义分析方法构造文本语义空间,然后以视觉词袋(bag of visual words)来表达文本所对应的图像,最后通过Sparse CCA算法找到一个语义相关空间,以实现文本语义与图像视觉单词间的映射。使用稀疏的相关性分析方法可以提高模型可解释性和保证检索结果稳定性。实验结果验证了Sparse CCA方法的有效性,同时也证实了所提出的图像语义检索方法的可行性。     

相关空间嵌入算法及其在图像检索中的应用

图像语义检索的一个有效解决途径是找到图像底层特征与文本语义之间的关联.文中在核方法和图拉普拉斯矩阵的基础上,提出一种相关空间嵌入算法,并利用文本隐性语义索引和图像特征的视觉单词,构造出文本语义空间与图像特征空间这两个异构空间的相关关系,从而找出文本语义与视觉单词间潜在关联,实现图像的语义检索.文中算法把保持数据流形结构的一致性作为一种先验约束,将文本语义空间和图像特征空间中的数据点嵌入到同一个相关空间中.因此,与典型相关分析算法相比,这种相关嵌入映射不仅可揭示不同数据空间之间存在的相关关系,还可在相关空间中保留原始数据分布结构,从而提高算法的可靠性.实验验证文中算法的有效性,为图像语义检索提供一种可行方法.     

基于潜语义主题加强的跨媒体检索算法

针对不同模态数据对相同语义主题表达存在差异性,以及传统跨媒体检索算法忽略了不同模态数据能以合作的方式探索数据的内在语义信息等问题,提出了一种新的基于潜语义主题加强的跨媒体检索（LSTR）算法。首先,利用隐狄利克雷分布（LDA）模型构造文本语义空间,然后以词袋（BoW）模型来表达文本对应的图像;其次,使用多分类逻辑回归对图像和文本分类,用得到的基于多分类的后验概率表示文本和图像的潜语义主题;最后,利用文本潜语义主题去正则化图像的潜语义主题,使图像的潜语义主题得到加强,同时使它们之间的语义关联最大化。在Wikipedia数据集上,文本检索图像和图像检索文本的平均查准率为57.0%,比典型相关性分析（CCA）、SM（Semantic Matching）、SCM（Semantic Correlation Matching）算法的平均查准率分别提高了35.1%、34.8%、32.1%。实验结果表明LSTR算法能有效地提高跨媒体检索的平均查准率。     

基于语义短语的空间金字塔词袋模型图像分类方法

针对空间金字塔词袋模型缺少对局部特征之间语义分布关系的表达,提出了一种基于语义短语的空间金字塔词袋模型图像表示方法.首先,将局部特征映射为具有一定语义信息的视觉单词,通过统计局部特征邻域范围内其他相关特征点的语义分布情况来构造语义短语.其次,将语义短语采用稀疏编码进行量化生成语义词典,图像则表示成基于语义词典的空间金字塔式稀疏统计直方图向量.最后,将图像表示向量代入分类器中进行训练和测试.实验结果表明,本文方法能够较大幅度地提高图像分类的准确率.     

基于虚拟属性学习的文本-图像行人检索方法

文本-图像行人检索旨在从行人数据库中查找符合特定文本描述的行人图像.近年来受到学术界和工业界的广泛关注.该任务同时面临两个挑战:细粒度检索以及图像与文本之间的异构鸿沟.部分方法提出使用有监督属性学习提取属性相关特征,在细粒度上关联图像和文本.然而属性标签难以获取,导致这类方法在实践中表现不佳.如何在没有属性标注的情况下提取属性相关特征,建立细粒度的跨模态语义关联成为亟待解决的关键问题.为解决这个问题,融合预训练技术提出基于虚拟属性学习的文本-图像行人检索方法,通过无监督属性学习建立细粒度的跨模态语义关联.第一,基于行人属性的不变性和跨模态语义一致性提出语义引导的属性解耦方法,所提方法利用行人的身份标签作为监督信号引导模型解耦属性相关特征.第二,基于属性之间的关联构建语义图提出基于语义推理的特征学习模块,所提模块通过图模型在属性之间交换信息增强特征的跨模态识别能力.在公开的文本-图像行人检索数据集CUHK-PEDES和跨模态检索数据集Flickr30k上与现有方法进行实验对比,实验结果表明了所提方法的有效性.     

跨媒体主题分析及应用研究

研究食品安全领域跨媒体数据的主题分析技术,融合多种媒体形式数据的语义,准确表达跨媒体文档的主题。由于食品安全事件相关多媒体数据的大量涌现,单一媒体的主题分析技术不能全面反映整个数据集的主题分布,存在语义缺失、主题空间不统一,语义融合困难等问题。提出一种跨媒体主题分析方法,首先以概率生成方法分别对文本和图像数据进行语义分析,然后利用跨媒体数据间的语义相关性进行视觉主题学习,建立视觉主题模型,进而实现视觉数据和文本主题之间的映射。仿真结果表明,跨媒体主题分析方法能够有效获取与图像语义相关的文本主题,且主题跟踪的准确度优于文本主题跟踪方法,能够为食品安全事件的监测提供依据。     

基于CCA子空间和GMM的自动图像标注

为了避免图像分割,并提高图像标注精度,提出一种基于典型相关分析(CCA)和高斯混合模型(GMM)的自动图像标注方法.利用CCA对图像的全局颜色特征与全局局部二值模式(LBP)纹理特征进行特征融合.使用融合后的语义特征,对每一个关键词建立GMM模型来估计单词类密度,从而在特征子空间中得到每个单词的概率分布.采用贝叶斯分类器确定每个标注词和测试图像的联合概率,运用词间语义关系优化标注结果.实验结果表明,使用该方法后的图像标注性能有了较大程度的改善.     

基于潜在语义与图结构的微博语义检索

微博文本短小、特征稀疏、与用户查询之间存在语义鸿沟的特点会降低语义检索效率。针对该问题,结合文本特征和知识库语义,构建基于潜在语义与图结构的语义检索模型。通过Tversky算法计算基于Hashtag的特征相关度;利用隐含狄利克雷分布算法对Wikipedia语料库训练主题模型,基于JSD距离计算映射到该模型的文本主题相关度;抽取DBpedia中实体及其网络关系连接图,使用SimRank算法计算图中实体间的相关度。综合以上3个结果得到最终相关度。通过短文本和长文本检索对Twitter子集进行实验,结果表明,与基于开放关联数据和图论的方法相比,该模型在评估指标MAP,P@30,R-Prec上分别提高了2.98%,6.40%,5.16%,具有较好的检索性能。     

建模连续视觉特征的图像语义标注方法

针对图像检索中存在的"语义鸿沟"问题,提出一种对连续视觉特征直接建模的图像自动标注方法.首先对概率潜语义分析(PLSA)模型进行改进,使之能处理连续量,并推导对应的期望最大化算法来确定模型参数;然后根据不同模态数据各自的特点,提出一个对不同模态数据分别处理的图像语义标注模型,该模型使用连续PLSA建模视觉特征,使用标准PLSA建模文本关键词,并通过不对称的学习方法学习2种模态之间的关联,从而能较好地对未知图像进行标注.通过在一个包含5000幅图像的标准Corel数据集中进行实验,并与几种典型的图像标注方法进行比较的结果表明,文中方法具有更高的精度和更好的效果.     

基于文本和内容的图像检索算法

为了提高图像检索的效率,提出一种基于文本和内容的图像检索算法。该算法采用稠密的尺度不变特征转换（ DSIFT）构造视觉单词的方式来描述图像内容,依据基于概率潜在语义分析（ PLSA）模型的图像自动标注方法获取的视觉语义对查询图像进行初步检索,在此结果集上对筛选出的语义相关图像按内容相似度排序输出。在数据集Corel1000上的实验结果表明,该算法能够实现有效的图像检索,检索效率优于单一的基于内容的图像检索算法。     

一种基于视觉单词的图像检索方法

基于内容的图像检索技术最主要的问题是图像的低层特征和高层语义之间存在着"语义鸿沟"。受文本内容分析的启发,有研究学者借鉴传统词典中用文本单词组合解释术语的思路,将图像视为视觉单词的组合,利用一系列视觉单词的组合来描述图像的语义内容。为此,利用SIFT进行图像的视觉单词特征提取,然后构建视觉单词库,最后实现了一个基于视觉单词的图像检索系统。实验结果表明,该方法在一定程度上提高了图像检索的查准率。     

A Multi-View Embedding Space for Modeling Internet Images,Tags, and Their Semantics

This paper investigates the problem of modeling Internet images and associated text or tags for tasks such as image-to-image search, tag-to-image search, and image-to-tag search (image annotation). We start with canonical correlation analysis (CCA), a popular and successful approach for mapping visual and textual features to the same latent space, and incorporate a third view capturing high-level image semantics, represented either by a single category or multiple non-mutually-exclusive concepts. We present two ways to train the three-view embedding: supervised, with the third view coming from ground-truth labels or search keywords; and unsupervised, with semantic themes automatically obtained by clustering the tags. To ensure high accuracy for retrieval tasks while keeping the learning process scalable, we combine multiple strong visual features and use explicit nonlinear kernel mappings to efficiently approximate kernel CCA. To perform retrieval, we use a specially designed similarity function in the embedded space, which substantially outperforms the Euclidean distance. The resulting system produces compelling qualitative results and outperforms a number of two-view baselines on retrieval tasks on three large-scale Internet image datasets.     

基于目标区域的语义图像检索

提出了一种基于高层语义的图像检索方法,该方法首先将图像分割成区域,提取每个区域的颜色、形状、位置特征,然后使用这些特征对图像对象进行聚类,得到每幅图像的语义特征向量;采用模糊C均值算法对图像进行聚类,在图像检索时,查询图像和聚类中心比较,然后在距离最小的类中进行检索。实验表明,提出的方法可以明显提高检索效率,缩小低层特征和高层语义之间的“语义鸿沟”。     

A survey of content-based image retrieval with high-level semantics

In order to improve the retrieval accuracy of content-based image retrieval systems, research focus has been shifted from designing sophisticated low-level feature extraction algorithms to reducing the ‘semantic gap’ between the visual features and the richness of human semantics. This paper attempts to provide a comprehensive survey of the recent technical achievements in high-level semantic-based image retrieval. Major recent publications are included in this survey covering different aspects of the research in this area, including low-level image feature extraction, similarity measurement, and deriving high-level semantic features. We identify five major categories of the state-of-the-art techniques in narrowing down the ‘semantic gap’: (1) using object ontology to define high-level concepts; (2) using machine learning methods to associate low-level features with query concepts; (3) using relevance feedback to learn users’ intention; (4) generating semantic template to support high-level image retrieval; (5) fusing the evidences from HTML text and the visual content of images for WWW image retrieval. In addition, some other related issues such as image test bed and retrieval performance evaluation are also discussed. Finally, based on existing technology and the demand from real-world applications, a few promising future research directions are suggested.     

Featured correspondence topic model for semantic search on social image collections

Nowadays, due to the rapid growth of digital technologies, huge volumes of image data are created and shared on social media sites. User-provided tags attached to each social image are widely recognized as a bridge to fill the semantic gap between low-level image features and high-level concepts. Hence, a combination of images along with their corresponding tags is useful for intelligent retrieval systems, those are designed to gain high-level understanding from images and facilitate semantic search. However, user-provided tags in practice are usually incomplete and noisy, which may degrade the retrieval performance. To tackle this problem, we present a novel retrieval framework that automatically associates the visual content with textual tags and enables effective image search. To this end, we first propose a probabilistic topic model learned on social images to discover latent topics from the co-occurrence of tags and image features. Moreover, our topic model is built by exploiting the expert knowledge about the correlation between tags with visual contents and the relationship among image features that is formulated in terms of spatial location and color distribution. The discovered topics then help to predict missing tags of an unseen image as well as the ones partially labeled in the database. These predicted tags can greatly facilitate the reliable measure of semantic similarity between the query and database images. Therefore, we further present a scoring scheme to estimate the similarity by fusing textual tags and visual representation. Extensive experiments conducted on three benchmark datasets show that our topic model provides the accurate annotation against the noise and incompleteness of tags. Using our generalized scoring scheme, which is particularly advantageous to many types of queries, the proposed approach also outperforms state-of-the-art approaches in terms of retrieval accuracy.     

Multimodal Retrieval using Mutual Information based Textual Query Reformulation

Multimodal Retrieval is a well-established approach for image retrieval. Usually, images are accompanied by text caption along with associated documents describing the image. Textual query expansion as a form of enhancing image retrieval is a relatively less explored area. In this paper, we first study the effect of expanding textual query on both image and its associated text retrieval. Our study reveals that judicious expansion of textual query through keyphrase extraction can lead to better results, either in terms of text-retrieval or both image and text-retrieval. To establish this, we use two well-known keyphrase extraction techniques based on tf-idf and KEA. While query expansion results in increased retrieval efficiency, it is imperative that the expansion be semantically justified. So, we propose a graph-based keyphrase extraction model that captures the relatedness between words in terms of both mutual information and relevance feedback. Most of the existing works have stressed on bridging the semantic gap by using textual and visual features, either in combination or individually. The way these text and image features are combined determines the efficacy of any retrieval. For this purpose, we adopt Fisher-LDA to adjudge the appropriate weights for each modality. This provides us with an intelligent decision-making process favoring the feature set to be infused into the final query. Our proposed algorithm is shown to supersede the previously mentioned keyphrase extraction algorithms for query expansion significantly. A rigorous set of experiments performed on ImageCLEF-2011 Wikipedia Retrieval task dataset validates our claim that capturing the semantic relation between words through Mutual Information followed by expansion of a textual query using relevance feedback can simultaneously enhance both text and image retrieval.     

Complementary relevance feedback-based content-based image retrieval

We propose a complementary relevance feedback-based content-based image retrieval (CBIR) system. This system exploits the synergism between short-term and long-term learning techniques to improve the retrieval performance. Specifically, we construct an adaptive semantic repository in long-term learning to store retrieval patterns of historical query sessions. We then extract high-level semantic features from the semantic repository and seamlessly integrate low-level visual features and high-level semantic features in short-term learning to effectively represent the query in a single retrieval session. The high-level semantic features are dynamically updated based on users’ query concept and therefore represent the image’s semantic concept more accurately. Our extensive experimental results demonstrate that the proposed system outperforms its seven state-of-the-art peer systems in terms of retrieval precision and storage space on a large scale imagery database.     

查询子空间在图像检索中的应用

近几年来,为了解决图像检索系统中由底层视觉特征和高层语义之间的差异所造成的检索困难,从信息捡索中引入了相关反馈技术。在过去几年中,它在该研究领域取得了一定的成功。文章提出了一种利用反馈信息建立“查询子空间”的检索模型,它将用户的语义查询进行基于视觉特征的分类,构造多个“查询子空间”,这些子空间拥有自身的查询模型和检索模型,最后的检索结果根据这多个“查询子空间”的检索结果得到。该模型具有较强的灵活性、扩展性,有效地利用了用户的反馈信息,动态地建立了底层视觉特征和高层语义之间的映射,能适应不同用户的查询。同时,将负反馈信息合理地融入到该模型中,提高了系统的检索效率。实验结果表明采用该检索模型的系统相比现有的检索系统性能有了较大提高。