Featured correspondence topic model for semantic search on social image collections

Nowadays, due to the rapid growth of digital technologies, huge volumes of image data are created and shared on social media sites. User-provided tags attached to each social image are widely recognized as a bridge to fill the semantic gap between low-level image features and high-level concepts. Hence, a combination of images along with their corresponding tags is useful for intelligent retrieval systems, those are designed to gain high-level understanding from images and facilitate semantic search. However, user-provided tags in practice are usually incomplete and noisy, which may degrade the retrieval performance. To tackle this problem, we present a novel retrieval framework that automatically associates the visual content with textual tags and enables effective image search. To this end, we first propose a probabilistic topic model learned on social images to discover latent topics from the co-occurrence of tags and image features. Moreover, our topic model is built by exploiting the expert knowledge about the correlation between tags with visual contents and the relationship among image features that is formulated in terms of spatial location and color distribution. The discovered topics then help to predict missing tags of an unseen image as well as the ones partially labeled in the database. These predicted tags can greatly facilitate the reliable measure of semantic similarity between the query and database images. Therefore, we further present a scoring scheme to estimate the similarity by fusing textual tags and visual representation. Extensive experiments conducted on three benchmark datasets show that our topic model provides the accurate annotation against the noise and incompleteness of tags. Using our generalized scoring scheme, which is particularly advantageous to many types of queries, the proposed approach also outperforms state-of-the-art approaches in terms of retrieval accuracy.     

The effects of multiple query evidences on social image retrieval

System performance assessment and comparison are fundamental for large-scale image search engine development. This article documents a set of comprehensive empirical studies to explore the effects of multiple query evidences on large-scale social image search. The search performance based on the social tags, different kinds of visual features and their combinations are systematically studied and analyzed. To quantify the visual query complexity, a novel quantitative metric is proposed and applied to assess the influences of different visual queries based on their complexity levels. Besides, we also study the effects of automatic text query expansion with social tags using a pseudo relevance feedback method on the retrieval performance. Our analysis of experimental results shows a few key research findings: (1) social tag-based retrieval methods can achieve much better results than content-based retrieval methods; (2) a combination of textual and visual features can significantly and consistently improve the search performance; (3) the complexity of image queries has a strong correlation with retrieval results’ quality—more complex queries lead to poorer search effectiveness; and (4) query expansion based on social tags frequently causes search topic drift and consequently leads to performance degradation.     

多任务分段紧凑特征的车辆检索方法

目的 随着公共安全领域中大规模图像监控及视频数据的增长以及智能交通的发展,车辆检索有着极其重要的应用价值。针对已有车辆检索中自动化和智能化水平低、难以获取精确的检索结果等问题,提出一种多任务分段紧凑特征的车辆检索方法,有效利用车辆基本信息的多样性和关联性实现实时检索。方法 首先,利用相关任务之间的联系提高检索精度和细化图像特征,因此构造了一种多任务深度卷积网络分段学习车辆不同属性的哈希码,将图像语义和图像表示相结合,并采用最小化图像编码使学习到的车辆的不同属性特征更具有鲁棒性;然后,选用特征金字塔网络提取车辆图像的实例特征并利用局部敏感哈希再排序方法对提取到的特征进行检索;最后,针对无法获取查询车辆目标图像的特殊情况,采用跨模态辅助检索方法进行检索。结果 提出的检索方法在3个公开数据集上均优于目前主流的检索方法,其中在CompCars数据集上检索精度达到0.966,在VehicleID数据集上检索精度提升至0.862。结论 本文提出的多任务分段紧凑特征的车辆检索方法既能得到最小化图像编码及图像实例特征,还可在无法获取目标检索图像信息时进行跨模态检索,通过实验对比验证了方法的有效性。     

一种基于混合特征核的图像检索方法

为了更准确地描述图像的视觉特征,提高图像检索的查准率与查全率,提出了一种基于混合特征核的图像检索方法.该方法提取图像的颜色、纹理、SIFT特征,引入高斯核函数,建立图像的混合特征核模型,在高维的核空间进行基于核的图像聚类.实验表明,该混合模型与传统多特征融合方法以及单一特征核方法相比,能够更好地表示图像的视觉特征,提高检索的查准率和查全率.     

基于视觉和情感特征模型的图像概率检索

基于内容的图像检索在很多领域都有广泛的应用.传统方法通常提取图片的底层视觉特征,如颜色、纹理和形状等,进行在视觉特征空间下的相似查询.然而,这些视觉特征无法表达图片需要传递的情感和概念信息.提出一种基于视觉和主观特征的统一图像概率检索方法.具体来说,图片通过三种类型特征(视觉特征、风格特征和情感特征等)来表达.通过线性加权得到图片间的统一相似距离,其中权重参数通过多元回归得到.不同于常规图像检索方法,只采用视觉相拟度作为查询的相似尺度,该方法允许用户选择三种特征作为查询元素,而且,通过引入概率模型实现对检索结果进行一定置信度保证的进一步细化.实验表明该检索方法及其索引的有效性.     

基于视觉显著计算的图像语义检索方法

网络标签已经开始广泛地用于图像内容的标注和分享,由于图像本身的差异和人们对图像的不同理解,对图像语义检索提出了新的挑战。该文首先引入视觉显著模型,突出图像的显著信息;然后提取视觉显著特征,建立图像内容的相似关系;最后基于随机漫步模型平衡图像内容及网络标签间的关系。实验表明该文提出的方法能够有效地实现图像的语义理解并用于图像检索。     

Topic modelling of clickthrough data in image search

In this paper we explore the benefits of latent variable modelling of clickthrough data in the domain of image retrieval. Clicks in image search logs are regarded as implicit relevance judgements that express both user intent and important relations between selected documents. We posit that clickthrough data contains hidden topics and can be used to infer a lower dimensional latent space that can be subsequently employed to improve various aspects of the retrieval system. We use a subset of a clickthrough corpus from the image search portal of a news agency to evaluate several popular latent variable models in terms of their ability to model topics underlying queries. We demonstrate that latent variable modelling reveals underlying structure in clickthrough data and our results show that computing document similarities in the latent space improves retrieval effectiveness compared to computing similarities in the original query space. These results are compared with baselines using visual and textual features. We show performance substantially better than the visual baseline, which indicates that content-based image retrieval systems that do not exploit query logs could improve recall and precision by taking this historical data into account.     

Kernelized locality-sensitive hashing

Fast retrieval methods are critical for many large-scale and data-driven vision applications. Recent work has explored ways to embed high-dimensional features or complex distance functions into a low-dimensional Hamming space where items can be efficiently searched. However, existing methods do not apply for high-dimensional kernelized data when the underlying feature embedding for the kernel is unknown. We show how to generalize locality-sensitive hashing to accommodate arbitrary kernel functions, making it possible to preserve the algorithm's sublinear time similarity search guarantees for a wide class of useful similarity functions. Since a number of successful image-based kernels have unknown or incomputable embeddings, this is especially valuable for image retrieval tasks. We validate our technique on several data sets, and show that it enables accurate and fast performance for several vision problems, including example-based object classification, local feature matching, and content-based retrieval.     

Mobile multi-view object image search

High user interaction capability of mobile devices can help improve the accuracy of mobile visual search systems. At query time, it is possible to capture multiple views of an object from different viewing angles and at different scales with the mobile device camera to obtain richer information about the object compared to a single view and hence return more accurate results. Motivated by this, we propose a new multi-view visual query model on multi-view object image databases for mobile visual search. Multi-view images of objects acquired by the mobile clients are processed and local features are sent to a server, which combines the query image representations with early/late fusion methods and returns the query results. We performed a comprehensive analysis of early and late fusion approaches using various similarity functions, on an existing single view and a new multi-view object image database. The experimental results show that multi-view search provides significantly better retrieval accuracy compared to traditional single view search.     

基于协作式标注图像数据的垃圾标签检测方法

由于用户标签的不准确和语义模糊使得协作式标注图像检索正确率低,而现有垃圾标签过滤方法往往关注标签本身,忽略了协作式标签与图像的关联性。本文在分析协作式标注图像视觉内容与标签的关联性的基础上,提出一种基于协作式标注图像视觉内容的垃圾标签检测方法。该方法分析同一标签下图像视觉内容,设计不同的核函数用于颜色和SIFT(Scale invariant feature transform)特征子集,同时将2种低维特征映射到高维多模特征空间形成混合核函数,对同一标签下的图像进行基于混合核的最大最小距离聚类,少数群体的标签说明与图像内容关联性小则为用户标注错误的标签,从而检测垃圾标签。实验结果表明,该方法能够提高协作式图像垃圾标签检测的正确性。     

Complementary relevance feedback-based content-based image retrieval

We propose a complementary relevance feedback-based content-based image retrieval (CBIR) system. This system exploits the synergism between short-term and long-term learning techniques to improve the retrieval performance. Specifically, we construct an adaptive semantic repository in long-term learning to store retrieval patterns of historical query sessions. We then extract high-level semantic features from the semantic repository and seamlessly integrate low-level visual features and high-level semantic features in short-term learning to effectively represent the query in a single retrieval session. The high-level semantic features are dynamically updated based on users’ query concept and therefore represent the image’s semantic concept more accurately. Our extensive experimental results demonstrate that the proposed system outperforms its seven state-of-the-art peer systems in terms of retrieval precision and storage space on a large scale imagery database.     

Image retrieval using multiple evidence ranking

The World Wide Web is the largest publicly available image repository and a natural source of attention. An immediate consequence is that searching for images on the Web has become a current and important task. To search for images of interest, the most direct approach is keyword-based searching. However, since images on the Web are poorly labeled, direct application of standard keyword-based image searching techniques frequently yields poor results. We propose a comprehensive solution to this problem. In our approach, multiple sources of evidence related to the images are considered. To allow combining these distinct sources of evidence, we introduce an image retrieval model based on Bayesian belief networks. To evaluate our approach, we perform experiments on a reference collection composed of 54000 Web images. Our results indicate that retrieval using an image surrounding text passages is as effective as standard retrieval based on HTML tags. This is an interesting result because current image search engines in the Web usually do not take text passages into consideration. Most important, according to our results, the combination of information derived from text passages with information derived from HTML tags leads to improved retrieval, with relative gains in average precision figures of roughly 50 percent, when compared to the results obtained by the use of each source of evidence in isolation.     

Tagging image by merging multiple features in a integrated manner

Image tagging is a task that automatically assigns the query image with semantic keywords called tags, which significantly facilitates image search and organization. Since tags and image visual content are represented in different feature space, how to merge the multiple features by their correlation to tag the query image is an important problem. However, most of existing approaches merge the features by using a relatively simple mechanism rather than fully exploiting the correlations between different features. In this paper, we propose a new approach to fusing different features and their correlation simultaneously for image tagging. Specifically, we employ a Feature Correlation Graph to capture the correlations between different features in an integrated manner, which take features as nodes and their correlations as edges. Then, a revised probabilistic model based on Markov Random Field is used to describe the graph for evaluating tag??s relevance to query image. Based on that, we design an image tagging algorithm for large scale web image dataset. We evaluate our approach using two large real-life corpuses collected from Flickr, and the experimental results indicate the superiority of our proposed approach over state-of-the-art techniques.     

Combining global and local matching of multiple features for precise item image retrieval

With the fast-growing of online shopping services, there are millions even billions of commercial item images available on the Internet. How to effectively leverage visual search method to find the items of users’ interests is an important yet challenging task. Besides global appearances (e.g., color, shape or pattern), users may often pay more attention to the local styles of certain products, thus an ideal visual item search engine should support detailed and precise search of similar images, which is beyond the capabilities of current search systems. In this paper, we propose a novel system named iSearch and global/local matching of local features are combined to do precise retrieval of item images in an interactive manner. We extract multiple local features including scale-invariant feature transform (SIFT), regional color moments and object contour fragments to sufficiently represent the visual appearances of items; while global and local matching of large-scale image dataset are allowed. To do this, an effective contour fragments encoding and indexing method is developed. Meanwhile, to improve the matching robustness of local features, we encode the spatial context with grid representations and a simple but effective verification approach using triangle relations constraints is proposed for spatial consistency filtering. The experimental evaluations show the promising results of our approach and system.     

一种基于多层语义特征的图像理解方法

视觉场景理解包括检测和识别物体、推理被检测物体之间的视觉关系以及使用语句描述图像区域.为了实现对场景图像更全面、更准确的理解,将物体检测、视觉关系检测和图像描述视为场景理解中3种不同语义层次的视觉任务,提出一种基于多层语义特征的图像理解模型,并将这3种不同语义层进行相互连接以共同解决场景理解任务.该模型通过一个信息传递图将物体、关系短语和图像描述的语义特征同时进行迭代和更新,更新后的语义特征被用于分类物体和视觉关系、生成场景图和描述,并引入融合注意力机制以提升描述的准确性.在视觉基因组和COCO数据集上的实验结果表明,所提出的方法在场景图生成和图像描述任务上拥有比现有方法更好的性能.