音频检索技术研究

结合音频检索发展现状,描述了当前相关研究的进展,介绍了现在最常用到的音频检索方法,讨论了与音频检索相关的关键技术：音频特征提取、音频分割和分类。基于内容的音乐检索研究是一种涉及音乐理论、信号处理、模式识别等相关领域的综合学科研究,其在音乐数据库管理、Internet音乐检索以及生活娱乐等方面都具有非常重要的意义。分析并总结出音乐内容及其检索的概念,给出音乐检索的系统结构,综述了基于内容的音乐检索方法,最后指出了音频检索发展的前景。     

Querying distributed multimedia databases and data sources for sensor data fusion

Sensor data fusion imposes a number of novel requirements on query languages and query processing techniques. A spatial/temporal query language called /spl Sigma/QL has been proposed to support the retrieval and fusion of multimedia information from multiple sources and databases. In this paper we investigate fusion techniques, multimedia data transformations and /spl Sigma/QL query processing techniques for sensor data fusion. Fusion techniques including fusion by the merge operation, the detection of moving objects, and the incorporation of belief values, have been developed. An experimental prototype has been implemented and tested to demonstrate the feasibility of these techniques.     

Multimedia ontology matching by using visual and textual modalities

Ontologies have been intensively applied for improving multimedia search and retrieval by providing explicit meaning to visual content. Several multimedia ontologies have been recently proposed as knowledge models suitable for narrowing the well known semantic gap and for enabling the semantic interpretation of images. Since these ontologies have been created in different application contexts, establishing links between them, a task known as ontology matching, promises to fully unlock their potential in support of multimedia search and retrieval. This paper proposes and compares empirically two extensional ontology matching techniques applied to an important semantic image retrieval issue: automatically associating common-sense knowledge to multimedia concepts. First, we extend a previously introduced textual concept matching approach to use both textual and visual representation of images. In addition, a novel matching technique based on a multi-modal graph is proposed. We argue that the textual and visual modalities have to be seen as complementary rather than as exclusive sources of extensional information in order to improve the efficiency of the application of an ontology matching approach in the multimedia domain. An experimental evaluation is included in the paper.     

Extracting semantic knowledge from web context for multimedia IR: a taxonomy,survey and challenges

Since its invention, the Web has evolved into the largest multimedia repository that has ever existed. This evolution is a direct result of the explosion of user-generated content, explained by the wide adoption of social network platforms. The vast amount of multimedia content requires effective management and retrieval techniques. Nevertheless, Web multimedia retrieval is a complex task because users commonly express their information needs in semantic terms, but expect multimedia content in return. This dissociation between semantics and content of multimedia is known as the semantic gap. To solve this, researchers are looking beyond content-based or text-based approaches, integrating novel data sources. New data sources can consist of any type of data extracted from the context of multimedia documents, defined as the data that is not part of the raw content of a multimedia file. The Web is an extraordinary source of context data, which can be found in explicit or implicit relation to multimedia objects, such as surrounding text, tags, hyperlinks, and even in relevance-feedback. Recent advances in Web multimedia retrieval have shown that context data has great potential to bridge the semantic gap. In this article, we present the first comprehensive survey of context-based approaches for multimedia information retrieval on the Web. We introduce a data-driven taxonomy, which we then use in our literature review of the most emblematic and important approaches that use context-based data. In addition, we identify important challenges and opportunities, which had not been previously addressed in this area.

     

Content-based indexing of multimedia databases

Content-based retrieval of multimedia database calls for content-based indexing techniques. Different from conventional databases, where data items are represented by a set of attributes of elementary data types, multimedia objects in multimedia databases are represented by a collection of features; similarity of object contents depends on context and frame of reference; and features of objects are characterized by multimodal feature measures. These lead to great challenges for content-based indexing. On the other hand, there are special requirements on content-based indexing: to support visual browsing, similarity retrieval, and fuzzy retrieval, nodes of the index should represent certain meaningful categories. That is to say that certain semantics must be added when performing indexing. ContIndex, the context-based indexing technique presented in this paper, is proposed to meet these challenges and special requirements. The indexing tree is formally defined by adapting a classification-tree concept. Horizontal links among nodes in the same level enhance the flexibility of the index. A special neural-network model, called Learning based on Experiences and Perspectives (FEP), has been developed to create node categories by fusing multimodal feature measures. It brings into the index the capability of self-organizing nodes with respect to certain context and frames of reference. An icon image is generated for each intermediate node to facilitate visual browsing. Algorithms have been developed to support multimedia object archival and retrieval using Contlndex     

基于压缩域的图像检索技术

图像检索技术是多媒体应用中的关键技术，现有的基于内容图像检索技术大都是基于非压缩域的，对于目前普遍存在的压缩格式图像，采用这种技术必须先解压再检索，不但计算量大，而且需占用较多的中介存储空间，所以严重影响了检索系统的实时性和灵活性，同时各种压缩标准(如JPEG，MPEG，JPEG2000等)的推出与普及也促使人们寻求可以直接在压缩域操作的检索技术，该文对现有的压缩域图像检索技术的发展进行综述，并讨论了未来可能的研究方向。     

Supporting ranked Boolean similarity queries in MARS

To address the emerging needs of applications that require access to and retrieval of multimedia objects, we are developing the Multimedia Analysis and Retrieval System (MARS). In this paper, we concentrate on the retrieval subsystem of MARS and its support for content-based queries over image databases. Content-based retrieval techniques have been extensively studied for textual documents in the area of automatic information retrieval. This paper describes how these techniques can be adapted for ranked retrieval over image databases. Specifically, we discuss the ranking and retrieval algorithms developed in MARS based on the Boolean retrieval model and describe the results of our experiments that demonstrate the effectiveness of the developed model for image retrieval     

Collaborative Multimedia Presentations in Mobile Environments

Distributed multimedia documents systems, distributed video servers are examples of multimedia presentations involving collaboration among multiple information sources. In such applications, objects have to be retrieved from their sources and presented to users according to specified temporal relationships. Objects retrieval in these collaborative applications is influenced by their presentation times, durations, and network throughput available to their sources. Replication of objects amongst the set of collaborating systems gives a choice for object retrieval. Client going through a multimedia presentation can be in a mobile environment. Here, object retrievals from collaborating servers are carried out by base stations to which the client is attached. Mobile client then downloads objects from the base station.In this paper, we present a graph-search based algorithm for computing and negotiating throughput requirements of collaborating multimedia presentations with replicated objects in a mobile environment. This algorithm maximizes the number of cached objects (that have already been played) for handling operations such as reverse presentation.     

The Sentient Map

The sentient map is a new paradigm for visual information retrieval. It enables the user to view data as maps, so that gestures, more specifically c-gestures, can be used for the interaction between the user and the multimedia information system. Different c-gestures are then dynamically transformed into spatial/temporal queries, or σ-queries, for multimedia information sources and databases. An e-learning environment involving many academic institutions serves as a test bed to evaluate this approach.     

移动设备上基于内容的多媒体检索技术研究

随着互联网上多媒体资源的急剧增加以及移动设备功能的不断完善,如何在海量多媒体资源中快速检索出所需要的资料显得至关重要。传统的基于关键词描述的多媒体检索技术因其描述能力有限等原因已不能满足用户对于多媒体语义内容的检索要求,而且这些技术直接应用到性能相当有限的移动设备上也是不可行的。移动设备用户对精确而快速地检索多媒体资源的需求越来越强烈。本文详细介绍了在移动设备上进行基于内容的多媒体检索技术的研究成果。相比现有技术,这些研究成果无论准确度还是速度都有很大提升。     

Localized Co-Occurrence Model for Fast Approximate Search in 3D Structure Databases

Similarity search for 3D structure data sets is fundamental to many database applications such as molecular biology, image registration, and computer-aided design. Identifying the common 3D subtructures between two objects is an important research problem. However, it is well known that computing structural similarity is very expensive due to the high exponential time complexity of structure similarity measures. As the structure databases keep growing rapidly, real-time search from large-structure databases becomes problematic. In this paper, we present a novel statistical model, that is, the multiresolution Localized Co-Occurrence Model (LCM), to approximately measure the similarity between the two point-based 3D structures in linear time complexity for fast retrieval. LCM could capture both distribution characteristics and spatial structure of 3D data by localizing the point co-occurrence relationship within a predefined neighborhood system. As a step further, a novel structure query processing method called the incremental and Bounded search (iBound) is also proposed to speed up the search process. iBound avoids a large amount of expensive computation at higher resolution LCMs. By superposing two LCMs, their largest common substructure can also be found quickly. Finally, our experiment results prove the effectiveness and efficiency of our methods.     

CORE: a content-based retrieval engine for multimedia information systems

Rapid advances in multimedia technology necessitate the development of a generic multimedia information system with a powerful retrieval engine for prototyping multimedia applications. We develop a content-based retrieval engine (CORE) that makes use of novel indexing techniques for multimedia object retrieval. We formalize the concepts related to multimedia information systems such as multimedia objects and content-based retrieval. We bring out the requirements and challenges of a multimedia information system. The architecture of CORE is described in detail along with the associated retrieval mechanisms and indexing techniques. Various modules developed for efficient retrieval are presented with some APIs. The efficacy of CORE is demonstrated in the development of two multimedia systems, a computer-aided facial image inference and retrieval (CAFIIR) system and a system for trademark archival and retrieval (STAR), which have been developed at the Institute of Systems Science (ISS). We expect that CORE will be useful for effective prototyping of other such multimedia applications.Mainly supported by National Science & Technology Board of SingaporePartly working in Real World Computing Partnership, Novel Function Institute of Systems Science Laboratory since April 1994.     

Fast retrieval of similar configurations

Configuration similarity is a special form of content-based image retrieval that considers relative object locations. It can be used as a standalone method, or to complement retrieval based on visual or semantic features. The corresponding queries ask for sets of objects that satisfy some spatio-temporal constraints, e.g., "find all triplets of objects (v/sub 1/, v/sub 2/, v/sub 3/), such that v/sub 1/ is northeast of v/sub 2/, which is inside v/sub 3/." Exhaustive processing (i.e., retrieval of the best solutions) of configuration similarity queries, in general, has exponential complexity and fast search for sub-optimal solutions is the only way to deal with the vast amounts of multimedia information in several real-time applications. In this paper we first discuss the utilization of nonsystematic search heuristics, based on genetic algorithms, simulated annealing and hill climbing approaches. An extensive experimentation with real and synthetic datasets reveals that hill climbing techniques are the best for the current problem; therefore, as a subsequent step we study the search space, and develop improved variations of hill climbing that take advantage of the special structure of the problem to enhance speed. The proposed heuristic methods significantly outperform systematic search when there is only limited time for query processing.     

Indexing and Retrieval of Audio: A Survey

With more and more audio being captured and stored, there is a growing need for automatic audio indexing and retrieval techniques that can retrieve relevant audio pieces quickly on demand. This paper provides a comprehensive survey of audio indexing and retrieval techniques. We first describe main audio characteristics and features and discuss techniques for classifying audio into speech and music based on these features. Indexing and retrieval of speech and music is then described separately. Finally, significance of audio in multimedia indexing and retrieval is discussed.     

Exploring context and content links in social media: a latent space method

Social media networks contain both content and context-specific information. Most existing methods work with either of the two for the purpose of multimedia mining and retrieval. In reality, both content and context information are rich sources of information for mining, and the full power of mining and processing algorithms can be realized only with the use of a combination of the two. This paper proposes a new algorithm which mines both context and content links in social media networks to discover the underlying latent semantic space. This mapping of the multimedia objects into latent feature vectors enables the use of any off-the-shelf multimedia retrieval algorithms. Compared to the state-of-the-art latent methods in multimedia analysis, this algorithm effectively solves the problem of sparse context links by mining the geometric structure underlying the content links between multimedia objects. Specifically for multimedia annotation, we show that an effective algorithm can be developed to directly construct annotation models by simultaneously leveraging both context and content information based on latent structure between correlated semantic concepts. We conduct experiments on the Flickr data set, which contains user tags linked with images. We illustrate the advantages of our approach over the state-of-the-art multimedia retrieval techniques.     

Quality-based I/O resource managementfor multimedia document database servers

Slow access to disk-based multimedia data is a major limiting factor in the performance of modern multimedia Web servers connected over broadband networks. The I/O bottleneck becomes even more pronounced for currently evolving systems handling multimedia data, such as audio and video. Retrieval of complex multimedia documents needs to be handled at two levels: I/O bandwidth management for multiple multimedia streams, and interstream and intrastream synchronization for multimedia objects constituting these documents. In this paper, based on the diverse characteristics of multimedia data, we propose efficient techniques for synchronous retrieval and delivery of such data from the storage system to the main memory of the server. We propose methods to quantify user perceived quality via quality-of-presentation (QoP) parameters. We combine QoP and Object Composition Petri Net (OCPN) multimedia data modeling to develop techniques for efficient synchronous retrieval of multimedia data. Since I/O bandwidth is a precious resource, the proposed techniques have low overhead, which is , where m is the number of logical I/O channels and n is the total number of frames of multimedia data in a scheduling period. We simulate the relative performance of these techniques under diverse I/O conditions and determine the tradeoffs between the system resources, such as memory, bandwidth, and the improvement in QoP for multimedia applications.Published online: 9 February 2005 Correspondence to: M. Farrukh Khan     

Multimedia displays for conceptual discovery: information seeking with strand maps

This article explores the use of a multimedia search interface for digital libraries based on strand maps developed by the American Association for the Advancement of Science. As semantic-spatial displays, strand maps provide a visual organization of relevant conceptual information that may promote the use of science content during digital library use. A study was conducted to compare users' cognitive processes during information seeking tasks when using a multimedia strand maps interface, versus the textual search interface currently implemented in the Digital Library for Earth System Education. Quantitative and qualitative data from think-aloud protocols revealed that students were more likely to engage with science content (e.g., analyzing the relevance of science concepts with regard to task needs) during search when using the strand maps interface compared to those using textual searching. In contrast, students using a textual search interface engaged more frequently with surface-level information (e.g., the type of a resource regardless of its science content) during search and retrieval. As a multimedia search interface for digital libraries, strand maps appear to be promising tools to promote conceptual discovery and learning through content-based processes that promote learner engagement with relevant science knowledge.     

Information access in multimedia databases based on feature models

With the increasing popularity of the WWW, the main challenge in computer science has become content-based retrieval of multimedia objects. Access to multimedia objects in databases has long been limited to the information provided in manually assigned keywords. Now, with the integration of feature-detection algorithms in database systems software, content-based retrieval can be fully integrated with query processing. We describe our experimentation platform under development, making database technology available to multimedia. Our approach is based on the new notion of feature databases. Its architecture fully integrates traditional query processing and content-based retrieval techniques. Arjen P. de Vries, Ph.D.: He received his Ph.D. in Computer Science from the University of Twente in 1999, on the integration of content management in database systems. He is especially interested in the new requirements on the design of database systems to support content-based retrieval in multimedia digital libraries. He has continued to work on multimedia database systems as a postdoc at the CWI in Amsterdam as well as University of Twente. Menzo Windhouwer: He received his MSc in Computer Science and Management from the University of Amsterdam in 1997. Currently he is working in the CWI Database Research Group on his Ph.D., which is concerned with multimedia indexing and retrieval using feature grammars. Peter M.G. Apers, Ph.D.: He is a full professor in the area of databases at the University of Twente, the Netherlands. He obtained his MSc and Ph.D. at the Free University, Amsterdam, and has been a visiting researcher at the University of California, Santa Cruz and Stanford University. His research interests are query optimization in parallel and distributed database systems to support new application domains, such as multimedia applications and WWW. He has served on the program committees of major database conferences: VLDB, SIGMOD, ICDE, EDBT. In 1996 he was the chairman of the EDBT PC. In 2001 he will, for the second time, be the chairman of the European PC of the VLDB. Currently he is coordinating Editor-in-Chief of the VLDB Journal, editor of Data & Knowledge Engineering, and editor of Distributed and Parallel Databases. Martin Kersten, Ph.D.: He received his PhD in Computer Science from the Vrije Universiteit in 1985 on research in database security, whereafter he moved to CWI to establish the Database Research Group. Since 1994 he is professor at the University of Amsterdam. Currently he is heading a department involving 60 researchers in areas covering BDMS architectures, datamining, multimedia information systems, and quantum computing. In 1995 he co-founded Data Distilleries, specialized in data mining technology, and became a non-executive board member of the software company Consultdata Nederland. He has published ca. 130 scientific papers and is member of the editorial board of VLDB journal and Parallel and Distributed Systems. He acts as a reviewer for ESPRIT projects and is a trustee of the VLDB Endowment board.     

Techniques and data structures for efficient multimedia retrieval based on similarity

As more and more information is captured and stored in digital form, many techniques and systems have been developed for indexing and retrieval of text documents, audio, images, and video. The retrieval is normally based on similarities between extracted feature vectors of the query and stored items. Feature vectors are usually multidimensional. When the number of stored objects and/or the number of dimensions of the feature vectors are large, it will be too slow to linearly search all stored feature vectors to find those that satisfy the query criteria. Techniques and data structures are thus required to organize feature vectors and manage the search process so that objects relevant to the query can be located quickly. This paper provides a survey of these techniques and data structures.