Semantic content-based image retrieval: A comprehensive study

The complexity of multimedia contents is significantly increasing in the current digital world. This yields an exigent demand for developing highly effective retrieval systems to satisfy human needs. Recently, extensive research efforts have been presented and conducted in the field of content-based image retrieval (CBIR). The majority of these efforts have been concentrated on reducing the semantic gap that exists between low-level image features represented by digital machines and the profusion of high-level human perception used to perceive images. Based on the growing research in the recent years, this paper provides a comprehensive review on the state-of-the-art in the field of CBIR. Additionally, this study presents a detailed overview of the CBIR framework and improvements achieved; including image preprocessing, feature extraction and indexing, system learning, benchmarking datasets, similarity matching, relevance feedback, performance evaluation, and visualization. Finally, promising research trends, challenges, and our insights are provided to inspire further research efforts.     

基于音频子带能量动态范围的快速音频检索

提出了利用音频子带能量动态范围特征实现两阶段快速音频检索的方法。在预处理阶段根据音频库的子带能量动态范围（DRSBE）特征首先建立1个索引库,检索时分为2步：第一步先计算输入参考音频片段的DRSBE特征,然后根据数据库中建立的索引找到候选音频;第二步计算参考音频和候选音频之间的相似度,输出最后结果。实验结果表明,基于DRSBE特征的快速音频检索方法对于同源音频检索的速度和精度都非常高,在高质量的广播音频检索中达到了实用要求。     

Models for motion-based video indexing and retrieval

With the rapid proliferation of multimedia applications that require video data management, it is becoming more desirable to provide proper video data indexing techniques capable of representing the rich semantics in video data. In real-time applications, the need for efficient query processing is another reason for the use of such techniques. We present models that use the object motion information in order to characterize the events to allow subsequent retrieval. Algorithms for different spatiotemporal search cases in terms of spatial and temporal translation and scale invariance have been developed using various signal and image processing techniques. We have developed a prototype video search engine, PICTURESQUE (pictorial information and content transformation unified retrieval engine for spatiotemporal queries) to verify the proposed methods. Development of such technology will enable true multimedia search engines that will enable indexing and searching of the digital video data based on its true content.     

Multimedia search and retrieval using multimodal annotation propagation and indexing techniques

In this paper, a complete solution for search and retrieval of rich multimedia content over modern databases is presented. The framework proposed in this paper combines the advantages of multimodal search with those of annotation propagation into a unified system. Moreover, an effective technique, which is appropriate for large-scale indexing, is adopted, extended and integrated to the proposed framework so as to achieve optimized search and retrieval of rich media content even from large-scale databases.     

Audio Feature Extraction and Analysis for Scene Segmentation and Classification

Understanding of the scene content of a video sequence is very important for content-based indexing and retrieval of multimedia databases. Research in this area in the past several years has focused on the use of speech recognition and image analysis techniques. As a complimentary effort to the prior work, we have focused on using the associated audio information (mainly the nonspeech portion) for video scene analysis. As an example, we consider the problem of discriminating five types of TV programs, namely commercials, basketball games, football games, news reports, and weather forecasts. A set of low-level audio features are proposed for characterizing semantic contents of short audio clips. The linear separability of different classes under the proposed feature space is examined using a clustering analysis. The effective features are identified by evaluating the intracluster and intercluster scattering matrices of the feature space. Using these features, a neural net classifier was successful in separating the above five types of TV programs. By evaluating the changes between the feature vectors of adjacent clips, we also can identify scene breaks in an audio sequence quite accurately. These results demonstrate the capability of the proposed audio features for characterizing the semantic content of an audio sequence.     

Digital watermarking algorithm in SWT domain based on robust local feature

Aiming at the challenging work to design a robust digital audio watermarking algorithm against desynchronization attacks,a new second generation digital audio watermarking in stationary wavelet transform (SWT) domain based on robust local audio feature was proposed.First,the first-order smooth gradient response of the low-pass sub-band coefficient was calculated using Gaussian filter.Then,the short-term energy was utilized to adaptively determine local feature audio segments for embedding.Finally,the watermark information was embedded into local feature audio segments with spread transform dither modulation.The experimental results show that the proposed approach has not only good transparency,but also has strong robustness against common audio processing such as MP3 compression and good robustness against the desynchronization attacks such as pitch-scale modification et al.A SWT domain audio feature point extraction method based on smooth gradient is proposed,which effectively solves the drawbacks of poor stability and uneven distribution of audio feature points,and improves the resistance of digital audio watermarks to amplitude-scale modification,pitch-scale modification,random cropping,and jittering attacks.     

面向跨模态检索的音频数据库内容匹配方法研究

跨模态检索旨在通过以某一模态的数据为查询词，使人们能够得到与之相关的其他不同模态数据的检索结果的新型检索方法，这已成为多媒体和信息检索领域中一个有趣的研究问题。但是，目前大多数的研究成果集中于文本到图像、文本到视频以及歌词到音频等跨模态相关任务上，而关于如何为特定的视频通过跨模态检索得到合适的音乐这一跨模态的相关研究却很有限。此外，大多现有的关于视频和音频跨模态的研究依赖于元数据（例如关键字，标签或描述）。本文介绍了一种基于音频和视频这两种模态数据内容的跨模态检索的方法，该方法以新型的双流处理网络为框架，并通过神经网络学习两模态数据在公共子空间的特征表达，以计算音频和视频数据之间的相似度。本文所提出的方法的创新点主要在以下三个方面:1）在原有的提取各模态特征的模型基础上引入注意力机制，以此得到了视频和音频的特征选择模型，并筛选出相应的特征表达。2）使用了样本挖掘机制，剔除了无效样本，使得数据的训练更加高效。3）从计算模态间相似性和保持模态内结构不变两方面出发，设计了相应的损失函数进行模型的训练。且所提出的模型在VEGAS数据集和自建数据集上都取得了较高的准确度。      

Efficient Content-Based Image Retrieval Methods Using Color and Texture

In this paper, we propose efficient content-based image retrieval methods using the automatic extraction of the low-level visual features as image content. Two new feature extraction methods are presented. The first one is an advanced color feature extraction derived from the modification of Stricker's method. The second one is a texture feature extraction using some DCT coefficients which represent some dominant directions and gray level variations of the image. In the experiment with an image database of 200 natural images, the proposed methods show higher performance than other methods. They can be combined into an efficient hierarchical retrieval method.     

Embedded lattices tree: An efficient indexing scheme for content based retrieval on image databases

One of the challenges in the development of a content-based multimedia indexing and retrieval application is to achieve an efficient indexing scheme. To retrieve a particular image from a large scale image database, users can be frustrated by the long query times. Conventional indexing structures cannot usually cope with the presence of a large amount of feature vectors in high-dimensional space. This paper addresses such problems and presents a novel indexing technique, the embedded lattices tree, which is designed to bring an effective solution especially for realizing the trade off between the retrieval speed up and precision.The embedded lattices tree is based on a lattice vector quantization algorithm that divides the feature vectors progressively into smaller partitions using a finer scaling factor. The efficiency of the similarity queries is significantly improved by using the hierarchy and the good algebraic and geometric properties of the lattice. Furthermore, the dimensionality reduction that we perform on the feature vectors, translating from an upper level to a lower one of the embedded tree, reduces the complexity of measuring similarity between feature vectors. In addition, it enhances the performance on nearest neighbor queries especially for high dimensions. Our experimental results show that the retrieval speed is significantly improved and the indexing structure shows no sign of degradations when the database size is increased.     

An Indexing and Retrieval Mechanism for Complex Similarity Queries in Image Databases

A content-based image retrieval mechanism to support complex similarity queries is presented. The image content is defined by three kinds of features: quantifiable features describing the visual information, nonquantifiable features describing the semantic information, and keywords describing more abstract semantic information. In correspondence with these feature sets, we construct three types of indexes: visual indexes, semantic indexes, and keyword indexes. Index structures are elaborated to provide effective and efficient retrieval of images based on their contents. The underlying index structure used for all indexes is the HG-tree. In addition to the HG-tree, the signature file and hashing technique are also employed to index keywords and semantic features. The proposed indexing scheme combines and extends the HG-tree, the signature file, and the hashing scheme to support complex similarity queries. We also propose a new evaluation strategy to process the complex similarity queries. Experiments have been carried out on large image collections to demonstrate the effectiveness of the proposed retrieval mechanism.     

The use of watermarks in the protection of digital multimediaproducts

The watermarking of digital images, audio, video, and multimedia products in general has been proposed for resolving copyright ownership and verifying originality of content. This paper studies the contribution of watermarking for developing protection schemes. A general watermarking framework (GWF) is studied and the fundamental demands are listed. The watermarking algorithms, namely watermark generation, embedding, and detection, are analyzed and necessary conditions for a reliable and efficient protection are stated. Although the GWF satisfies the majority of requirements for copyright protection and content verification, there are unsolved problems inside a pure watermarking framework. Particular solutions, based on product registration and related network services, are suggested to overcome such problems     

Distributed multimedia systems

A distributed multimedia system (DMS) is an integrated communication, computing, and information system that enables the processing, management, delivery, and presentation of synchronized multimedia information with quality-of-service guarantees. Multimedia information may include discrete media data, such as text, data, and images, and continuous media data, such as video and audio. Such a system enhances human communications by exploiting both visual and aural senses and provides the ultimate flexibility in work and entertainment, allowing one to collaborate with remote participants, view movies on demand, access on-line digital libraries from the desktop, and so forth. In this paper, we present a technical survey of a DMS. We give an overview of distributed multimedia systems, examine the fundamental concept of digital media, identify the applications, and survey the important enabling technologies