A hybrid probabilistic framework for content-based image retrieval with feature weighting

In this paper, we propose a probabilistic framework for efficient retrieval and indexing of image collections. This framework uncovers the hierarchical structure underlying the collection from image features based on a hybrid model that combines both generative and discriminative learning. We adopt the generalized Dirichlet mixture and maximum likelihood for the generative learning in order to estimate accurately the statistical model of the data. Then, the resulting model is refined by a new discriminative likelihood that enhances the power of relevant features. Consequently, this new model is suitable for modeling high-dimensional data described by both semantic and low-level (visual) features. The semantic features are defined according to a known ontology while visual features represent the visual appearance such as color, shape, and texture. For validation purposes, we propose a new visual feature which has nice invariance properties to image transformations. Experiments on the Microsoft's collection (MSRCID) show clearly the merits of our approach in both retrieval and indexing.     

一种基于流行排序的区域图像检索方法*

提出了一种结合多示例学习和流行排序的图像检索方法,将图像检索作为多示例学习框架下的流行排序,通过给出适合图像在包空间的有效度量方式,将流行排序的方法和多示例学习有效结合起来,从而获得更准确的检索结果。实验结果表明,运用流行排序的区域图像检索方法是可行的,同时,检索结果与传统的区域图像检索方法相比,检索率得到了明显的提高。     

Combining visual dictionary, kernel-based similarity and learning strategy for image category retrieval

This paper presents a search engine architecture, RETIN, aiming at retrieving complex categories in large image databases. For indexing, a scheme based on a two-step quantization process is presented to compute visual codebooks. The similarity between images is represented in a kernel framework. Such a similarity is combined with online learning strategies motivated by recent machine-learning developments such as active learning. Additionally, an offline supervised learning is embedded in the kernel framework, offering a real opportunity to learn semantic categories. Experiments with real scenario carried out from the Corel Photo database demonstrate the efficiency and the relevance of the RETIN strategy and its outstanding performances in comparison to up-to-date strategies.     

Watermarking with retrieval systems

We are interested in the problem of associating messages with multimedia content for the purpose of identifying them. This problem can be addressed by a watermarking system that embeds the associated messages into the multimedia content (also called Work). A drawback of watermarking is that the content will be distorted during embedding. On the other hand, if we assume that the database is available, the problem can be addressed by a retrieval system. Although no undesirable distortion is introduced when a retrieval system is used, the overhead of searching in large databases is fundamentally difficult (also known as the dimensionality curse). In this paper we present a novel framework that strikes a trade-off between watermarking and retrieval systems. Our framework avoids the dimensionality curse by introducing small distortions (watermark) into the multimedia content. From another perspective, the framework improves the watermarking performance, marked by significant reduction in distortion, by introducing searching ability in the message detection stage. To prove the concept, we give an algorithm based on the proposed notion of active clustering.     

Fisher Linear Discriminant Analysis for text-image combination in multimedia information retrieval

With multimedia information retrieval, combining different modalities – text, image, audio or video provides additional information and generally improves the overall system performance. For this purpose, the linear combination method is presented as simple, flexible and effective. However, it requires to choose the weight assigned to each modality. This issue is still an open problem and is addressed in this paper.     

面向用户的多媒体检索中的多模态界面框架设计

本文提出并设计了一种面向用户的多媒体信息检索中的多模态界面框架。该框架将知识指导、语义概念学习、自然语言处理及用户特性分析等技术于一体，从而为设计通用多媒体信息检索系统奠定了基础。     

A framework for developing a daily activity and multimodal travel planner

There is a need to further explore ways to use Advanced Traveler Information Systems (ATIS) to encourage transit and ridesharing. One mechanism is to provide convenient travel itinerary information, not just for one trip, but for a day's travel. The formulation should consider time constraints, activity needs, real transit service parameters and the actual street system. Basic algorithmic development is needed to spur the private sector interest in such a product by demonstrating its utility. The activity travel planner will aid its user in planning his daily itinerary by arranging the sequence of stops, suggesting and possibly selecting stop locations, providing transit route and schedule information, and suggesting travel routes. This paper develops a framework for solving the Travel Itinerary Planning Problem (TIPP) which is a variant of the Traveling Salesman Problem (TSP). Implementation of the solution algorithm would be used to develop and test a prototype for an activity and travel planner.     

Quantum-inspired multimodal fusion for video sentiment analysis

We tackle the crucial challenge of fusing different modalities of features for multimodal sentiment analysis. Mainly based on neural networks, existing approaches largely model multimodal interactions in an implicit and hard-to-understand manner. We address this limitation with inspirations from quantum theory, which contains principled methods for modeling complicated interactions and correlations. In our quantum-inspired framework, the word interaction within a single modality and the interaction across modalities are formulated with superposition and entanglement respectively at different stages. The complex-valued neural network implementation of the framework achieves comparable results to state-of-the-art systems on two benchmarking video sentiment analysis datasets. In the meantime, we produce the unimodal and bimodal sentiment directly from the model to interpret the entangled decision.     

A genetic programming framework for content-based image retrieval

The effectiveness of content-based image retrieval (CBIR) systems can be improved by combining image features or by weighting image similarities, as computed from multiple feature vectors. However, feature combination do not make sense always and the combined similarity function can be more complex than weight-based functions to better satisfy the users’ expectations. We address this problem by presenting a Genetic Programming framework to the design of combined similarity functions. Our method allows nonlinear combination of image similarities and is validated through several experiments, where the images are retrieved based on the shape of their objects. Experimental results demonstrate that the GP framework is suitable for the design of effective combinations functions.     

基于Log-Gabor小波和二维半监督判别分析的人脸图像检索*

针对光照、表情变化给人脸识别造成的影响以及大型人脸图像库的训练样本中只有部分标记的问题,结合多通道Log-Gabor小波和半监督流形学习算法,提出一种新的人脸图像检索方法。该方法首先使用Log-Ga-bor小波对人脸图像进行滤波获得特征矩阵,进一步利用提出的二维半监督流形学习算法进行维数约简,得到低维判别特征。由于该方法直接作用于Log-Gabor特征矩阵,克服了小样本带来的奇异问题;另外,通过充分利用标记和未标记信息,还保留了数据的局部流形结构,增强了特征匹配的相似性。在CMU PIE和AR人脸数据库上的实验结果表明,该方法有效且优于其他方法。     

A geometric framework for data fusion in information retrieval

Data fusion in information retrieval has been investigated by many researchers and a number of data fusion methods have been proposed. However, problems such as why data fusion can increase effectiveness and favorable conditions for the use of data fusion methods are poorly resolved at best. In this paper, we formally describe data fusion under a geometric framework, in which each component result returned from an information retrieval system for a given query is represented as a point in a multi-dimensional space. The Euclidean distance is the measure by which the effectiveness and similarity of search results are judged. This allows us to explain all component results and fused results using geometrical principles. In such a framework, score-based data fusion becomes a deterministic problem. Several interesting features of the centroid-based data fusion method and the linear combination method are discussed. Nevertheless, in retrieval evaluation, ranking-based measures are the most popular. Therefore, this paper investigates the relation and correlation between the Euclidean distance and several typical ranking-based measures. We indeed find that a very strong correlation exists between these. It means that the theorems and observations obtained using the Euclidean distance remain valid when ranking-based measures are used. The proposed framework enables us to have a better understanding of score-based data fusion and use score-based data fusion methods more precisely and effectively in various ways.     

A unified multimedia and semantic perspective for data retrieval in the semantic web

In recent years, the emerging diffusion of peer-to-peer networks is going beyond the single-domain paradigm like, for instance, the mono-thematic file sharing one (e.g. Napster for music). Peers are more and more heterogeneous data sources which need to share data with commercial, educational, and/or collaboration purposes, just to mention a few. Moreover, in current information processing applications data cannot be meaningfully searched by precise database queries that would return exact matches (e.g. when dealing with multimedia, proteomic, statistical data).     

Deep unsupervised network for multimodal perception,representation and classification

In this paper, we tackle the problem of multimodal learning for autonomous robots. Autonomous robots interacting with humans in an evolving environment need the ability to acquire knowledge from their multiple perceptual channels in an unsupervised way. Most of the approaches in the literature exploit engineered methods to process each perceptual modality. In contrast, robots should be able to acquire their own features from the raw sensors, leveraging the information elicited by interaction with their environment: learning from their sensorimotor experience would result in a more efficient strategy in a life-long perspective. To this end, we propose an architecture based on deep networks, which is used by the humanoid robot iCub to learn a task from multiple perceptual modalities (proprioception, vision, audition). By structuring high-dimensional, multimodal information into a set of distinct sub-manifolds in a fully unsupervised way, it performs a substantial dimensionality reduction by providing both a symbolic representation of data and a fine discrimination between two similar stimuli. Moreover, the proposed network is able to exploit multimodal correlations to improve the representation of each modality alone.     

基于流行排序的多示例图像检索方法

为了提高图像检索的性能,提出了一种基于流行排序的多示例图像检索方法,将分割后的图像表示为多示例的形式,通过给出适合图像在包空间的度量方式,有效结合流行排序和多示例学习的方法来进行图像检索.实验结果表明,采用所提出的方法的检索结果与传统的检索方法相比,检索率得到了明显的提高,检索结果更符合人的视觉习惯.     

DocMIR: An automatic document-based indexing system for meeting retrieval

This paper describes the DocMIR system which captures, analyzes and indexes automatically meetings, conferences, lectures, etc. by taking advantage of the documents projected (e.g. slideshows, budget tables, figures, etc.) during the events. For instance, the system can automatically apply the above-mentioned procedures to a lecture and automatically index the event according to the presented slides and their contents. For indexing, the system requires neither specific software installed on the presenter’s computer nor any conscious intervention of the speaker throughout the presentation. The only material required by the system is the electronic presentation file of the speaker. Even if not provided, the system would temporally segment the presentation and offer a simple storyboard-like browsing interface. The system runs on several capture boxes connected to cameras and microphones that records events, synchronously. Once the recording is over, indexing is automatically performed by analyzing the content of the captured video containing projected documents and detects the scene changes, identifies the documents, computes their duration and extracts their textual content. Each of the captured images is identified from a repository containing all original electronic documents, captured audio–visual data and metadata created during post-production. The identification is based on documents’ signatures, which hierarchically structure features from both layout structure and color distributions of the document images. Video segments are finally enriched with textual content of the identified original documents, which further facilitate the query and retrieval without using OCR. The signature-based indexing method proposed in this article is robust and works with low-resolution images and can be applied to several other applications including real-time document recognition, multimedia IR and augmented reality systems.

Relevance feedback for category search in music retrieval based on semantic concept learning

Traditional content-based music retrieval systems retrieve a specific music object which is similar to what a user has requested. However, the need exists for the development of category search for the retrieval of a specific category of music objects which share a common semantic concept. The concept of category search in content-based music retrieval is subjective and dynamic. Therefore, this paper investigates a relevance feedback mechanism for category search of polyphonic symbolic music based on semantic concept learning. For the consideration of both global and local properties of music objects, a segment-based music object modeling approach is presented. Furthermore, in order to discover the user semantic concept in terms of discriminative features of discriminative segments, a concept learning mechanism based on data mining techniques is proposed to find the discriminative characteristics between relevant and irrelevant objects. Moreover, three strategies, the Most-Positive, the Most-Informative, and the Hybrid, to return music objects concerning user relevance judgments are investigated. Finally, comparative experiments are conducted to evaluate the effectiveness of the proposed relevance feedback mechanism. Experimental results show that, for a database of 215 polyphonic music objects, 60% average precision can be achieved through the use of the proposed relevance feedback mechanism.