Query Processing Techniques for Multimedia Presentations

A multimedia presentation is a synchronized, and possibly interactive, delivery of multimedia data to users. We expect that, in the future, multimedia presentations will be stored into and queried from multimedia databases. In an earlier work, we have designed a graphical query language, called GVISUAL, that allows users to query multimedia presentations based on content information. In this paper, we discuss GVISUAL query processing techniques for multimedia presentations. More specifically, we discuss the translation of GVISUAL queries into an operator-based language, called O-Algebra, with three new operators, and efficient implementations of the new O-Algebra operators using a coding system called nodecodes.     

Canonical processes for creating personalized semantically rich multimedia presentations

Authoring of multimedia content can be considered as composing media assets such as images, videos, text, and audio in time, space, and interaction into a coherent multimedia presentation. Personalization of such content means that it reflects the users’ or user groups’ profile information and context information. Enriching the multimedia content with semantically rich metadata allows for a better search and retrieval of the content. To actually create personalized semantically-rich multimedia content, a manual authoring of the many different documents for all the different users’ and user groups’ needs is not feasible. Rather a (semi-)automatic authoring of the content seems reasonable. We have analyzed in detail today’s approaches and systems for authoring, personalizing, and semantically enriching multimedia presentations. Based on this analysis, we derived a general creation chain for the (semi-)automatic generation of such content. In this paper, we introduce this creation chain. We present our software engineering support for the chain, the component framework SemanticMM4U. The canonical processes supported by the creation chain and SemanticMM4U framework are described in detail. We also provide an explicit mapping of SemanticMM4U framework components to the processes and argue for the benefits of defining canonical processes for creating personalized semantically rich multimedia presentations.     

Case studies in intelligent multimedia presentation design systems in terms of the standard reference model: the IMMPS project, the PreGen system, and the StrMP system

This paper contributes to a Standard Reference Model (RM) for intelligent multimedia presentation systems. In the past few years, we have developed several intelligent presentation systems at the Multimedia Information NEtwork (MINE) Lab at Tamkang University. We describe these systems in terms of architecture and function of components. The first system, Intelligent MultiMedia Presentation System (IMMPS), allows a presentation designer to construct knowledge rules reflecting the background of individual audiences. Therefore, the generated presentation is able to learn from the audiences, which results in different presentation for each individual. The second system (PreGen) uses inference rules for the automatic generation of multimedia presentations. An ICON programming technique is also used in the PreGen project. Another system for designing structured multimedia presentations using data flow/control flow diagrams is also discussed. In order to run multimedia presentations generated by different systems, we are developing a multimedia abstract machine, which is a time Petri net based software simulator for running multimedia assembly instructions.     

View management in multimedia databases

Abstract. Though there has been extensive work on multimedia databases in the last few years, there is no prevailing notion of a multimedia view, nor there are techniques to create, manage, and maintain such views. Visualizing the results of a dynamic multimedia query or materializing a dynamic multimedia view corresponds to assembling and delivering an interactive multimedia presentation in accordance with the visualization specifications. In this paper, we suggest that a non-interactive multimedia presentation is a set of virtual objects with associated spatial and temporal presentation constraints. A virtual object is either an object, or the result of a query. As queries may have different answers at different points in time, scheduling the presentation of such objects is nontrivial. We then develop a probabilistic model of interactive multimedia presentations, extending the non-interactive model described earlier. We also develop a probabilistic model of interactive visualization where the probabilities reflect the user profiles, or the likelihood of certain user interactions. Based on this probabilistic model, we develop three utility-theoretic based types of prefetching algorithms that anticipate how users will interact with the presentation. These prefetching algorithms allow efficient visualization of the query results in accordance with the underlying specification. We have built a prototype system that incorporates these algorithms. We report on the results of experiments conducted on top of this implementation. Received June 10, 1998 / Accepted November 10, 1999     

Authoring of scalable multimedia documents

With the diversity of usage conditions affecting the consumption of multimedia content, the adaptation of dynamic and interactive multimedia presentations is essential. The challenge consists in allowing a multimedia presentation to adapt in all its dimensions: spatially, temporally and interactively, without lessening its attractiveness and still giving the author the control over adapted versions. Additionally, the authoring of adaptable content should not increase the complexity of authoring. To address this challenge, we propose to transpose the concept of scalability to the world of multimedia documents by introducing the so-called Scalable MSTI model. In this paper, we show the properties of this model and how, from an authoring point of view, scalable multimedia documents can be created to address a wide range of usage conditions.     

A Constraint-Driven Approach to Automate the Organization and Playout of Presentations in Multimedia Databases

We introduce a constraint-driven methodology for the automated assembly, organization and playout of presentations from multimedia databases. We use inclusion and exclusion constraints for extracting a semantically coherent set of multimedia segments. Presentationorganization constraints are utilized for organizing the multimedia segments into apresentation, which in turn helps decide the playout order of the extractedmultimedia segments. The playout order of the segments is represented in a presentationgraph. If the specified set of organization constraints are not sufficient toconstruct a unique presentation graph, we propose two techniques so that a unique graph isconstructible. We also propose two playout algorithms, one for the generation, start andtermination of playout agents, the other for dynamic control of playout management on organized presentations.The characteristics of these algorithms are expressed in terms of presentation playout parameters.     

Integrating Computer Animation and Multimedia

Multimedia provides an immensely powerful tool for the dissemination of both information and entertainment. Current multimedia presentations consist of synchronised excepts of media (such as sound, video & text) which are coordinated by an author to ensure a clear narrative is presented to the audience. However, each of the segments of the presentation consist of previously recorded footage, only the timing and synchronisation are dynamically constructed. The next logical advance for such systems is therefore to include the capability of generating material ‘on-the-fly’ in response to the actions of the audience. This paper describes a mechanism for using computer animation to generate this interactive material. Unlike previous animation techniques the approach presented here is suitable for use in constructing a storyline which the author can control, but the user can influence. In order to allow such techniques to be used we also present a multimedia authoring & playback system which incorporates interactive animation with existing media.     

Dynamic playout scheduling algorithms for continuous multimedia streams

In this paper, we investigate a playout scheduling framework for supporting the continuous and synchronized presentations of multimedia streams in a distributed multimedia presentation system. We assume a situation in which the server and network transmissions provide sufficient support for the delivery of media objects. In this context, major issues regarding the enforcement of the smooth presentation of multimedia streams at client sites must be addressed to deal with rate variance of stream presentations and delay variance of networks. We develop various playout-scheduling algorithms that are adaptable to quality-of-service parameters. The proposed algorithms permit the local adjustment of unsynchronized presentations by gradually accelerating or retarding presentation components, rather than abruptly skipping or pausing the presentation materials. A comprehensive experimental analysis of the proposed algorithms demonstrates that our algorithms can effectively avoid playout gaps (or hiccups) in the presentations. This scheduling framework can be readily used to support customized multimedia presentations.     

HyOctane: a HyTime engine for an MMIS

We are interested in the development of distributed multimedia information systems that use the HyTime international standard as the data model and interchange format. We have developed and implemented a prototype system in which interactive multimedia presentations can be stored and retrieved. Sample document instances are externally encoded in HyTime and stored in the database using the HyTime data model. The architecture and operation of our system are presented. Issues related to using a HyTime engine for general multimedia presentation and interchange are discussed.     

Retrieval in multimedia presentations

In this paper we discuss issues concerning the consistent retrieval of parts of multimedia presentations from multimedia repositories. We introduce a class of multimedia presentations made of independent and synchronized media and discuss retrieval requirements of presentation fragments. Then we discuss a retrieval model capable of reconstructing the fragments of a presentation from the atomic components returned by the execution of queries to multimedia presentation repositories. The retrieval model is based on an automaton that formally describes the presentation states entered by the events that trigger media playback. Retrieving a consistent fragment corresponds to building a new presentation with all the media related to the retrieved ones, with their original structural and synchronization relationships.     

Querying multimedia presentations based on content

Considers the problem of querying multimedia presentations based on content information. Multimedia presentations are modeled as presentation graphs, which are directed acyclic graphs that visually specify the presentations. We present a graph data model for the specification of multimedia presentations and discuss query languages as effective tools to query and manipulate multimedia presentation graphs with respect to content information. To query the information flow throughout a multimedia presentation, as well as in each individual multimedia stream, we use revised versions of temporal operators Next, Connected and Until, together with path formulas. These constructs allow us to specify and query paths along a presentation graph. We present an icon-based graphical query language, GVISUAL, that provides iconic representations for these constructs and a user-friendly graphical interface for query specification. We also present an OQL-like language, GOQL (Graph OQL), with similar constructs, that allows textual and more traditional specifications of graph queries. Finally, we introduce GCalculus (Graph Calculus), a calculus-based language that establishes the formal grounds for the use of temporal operators in path formulas and for querying presentation graphs with respect to content information. We also discuss GCalculus/S (GCalculus with Sets) which avoids highly complex query expressions by eliminating the universal path quantifier, the negation operator and the universal quantifier. GCalculus/S represents the formal basis for GVISUAL, i.e. GVISUAL uses the constructs of GCalculus/S directly     

PressBase : a presentation synchronization database for distributed multimedia systems

Multimedia presentations are the basic objects of multimedia databases. Since a multimedia presentation is not an instant display of a query result, the control knowledge (or synchronization requirements) has to be incorporated into the database and necessary precautions have to be taken for a lengthy presentation. Active databases provide a mechanism for incorporation of control knowledge by using event-condition-action (ECA) rules. In this paper, we describe how multimedia synchronization can be handled within a database using ECA rules. We present a prototype presentation synchronization database, named as PressBase, for distributed multimedia systems. We have adopted one of the synchronization models, SynchRuler, and then incorporated into a relational database system.     

Scheduling multimedia presentations in educational digital libraries

An educational digital library is a specialized digital library containing instructional materials, such as class lectures, seminar presentations, and various training materials. These materials consist of a combination of audio, video, and image data. In such an environment, basic parts of multimedia data are usually stored in databases and sophisticated multimedia presentations may be assembled to generate various presentations. In this paper, we investigate a theory of the scheduling strategies for supporting the synchronized presentations of multimedia streams which is applicable to educational digital libraries. This scheduling theory includes the specification and representation of synchronization on media streams, the realization of appropriate synchronization granularity, and the scheduling principles for the presentations of multimedia streams. This investigation formulates criteria for specifying and scheduling the skipping/pausing of media streams with asynchronous presentations when various delays occur. Adaptability to various quality-of-service requirements is supported in the scheduling strategies. Various synchronization mechanisms at both client and server sides are proposed to implement the scheduling theory. Experimental analysis is conducted using instructional materials.     

On Similarity Measures for Multimedia Database Applications

A multimedia database query consists of a set of fuzzy and boolean (or crisp) predicates, constants, variables, and conjunction, disjunction, and negation operators. The fuzzy predicates are evaluated based on different media criteria, such as color, shape, layout, keyword. Since media-based evaluation yields similarity values, results to such a query is defined as an ordered set. Since many multimedia applications require partial matches, query results also include tuples which do not satisfy all predicates. Hence, any fuzzy semantics which extends the boolean semantics of conjunction in a straight forward manner may not be desirable for multimedia databases. In this paper, we focus on the problem of ‘given a multimedia query which consists of multiple fuzzy and crisp predicates, how to provide the user with a meaningful overall ranking.’ More specifically, we study the problem of merging similarity values in queries with multiple fuzzy predicates. We describe the essential multimedia retrieval semantics, compare these with the known approaches, and propose a semantics which captures the retrieval requirements in multimedia databases. Received 13 August 1999 / Revised 13 May 2000 / Accepted in revised form 26 July 2000     

Multimedia presentation organization and playout management using intelligent agents

This paper presents a model for organization, computation and management of automated multimedia presentations based on active multimedia segments retrieved from a multimedia information system as well as the Web. Existing multimedia presentation system is extended in such a way that its content selection component spans from local data resources to the entire Web by use of intelligent agents which associates related multimedia segments. We describe a multimedia presentation authoring environment in which a new heuristic method is introduced based on the multimedia resources selected and organization operators to apply. This method is used for the construction of a presentation graph representing an organized presentation. Once the presentation graph is constructed, we show how to obtain an event-point representation of the presentation graph. Based on event-point representation, three methods are given for playing out the constructed presentations at the presentation terminal. Presentation environments of the end-users are modeled as (i) without any constraint, (ii) with a single constraint, and (iii) with multiple constraints (one constraint for each type of multimedia segments in the organized presentation). In accordance with these limitations (i.e., without violating any of the end-user specified constraints), three methods play out any organized presentation.     

QoS Management in Educational Digital Library Environments

Advances in multimedia computing technologies offer new approaches to the support of computer-assisted education and training within many application domains. Novel interactive presentation tools can be built to enhance traditional teaching methods with more active learning. Since a variety of user expectations are possible in such an environment, research must address the incorporation of these factors into presentation tools. During an interactive learning/training process, presentation tools must be able to handle various types of delays. A flexibly adjustable quality of service (QoS) should thus be supported. In this paper, we investigate a framework and systematic strategies for supporting the continuous and synchronized retrieval and presentation of multimedia data streams in a client/server distributed multimedia environment for educational digital libraries. Specifically, we establish a practical framework for specifying multimedia objects, tasks, schedules, and synchronization constraints between media streams. We identify the QoS parameters critical to the support of multimedia presentations for learning and training activities. Based on the proposed framework and QoS specifications, we develop presentation scheduling and buffer management strategies which can enforce the specified QoS requirements in an educational digital library environment.     

An Algebra for PowerPoint Sources

There are now millions of PowerPoint documents available within corporate intranets and/or over the Internet. In this paper, we develop a formal model of PowerPoint databases. We propose a relational style algebra called pptA (PowerPoint Algebra) to query PowerPoint databases. The algebra contains some new operators (such as the APPLY operator that changes properties of objects, slides and presentations) as well as interesting twists on relational operators (e.g. join and cartesian product allow different entities being joined together to share attributes whose values may be merged). We prove a set of equivalence results within this algebra. We have implemented a version of pptA—the paper provides a cost model and experimental results on the conditions under which these equivalences are useful.     

True multimedia presentations: A case study

This paper describes work undertaken as part of a three-stranded project. The main deliverable will be a suite of software tools, classes and components, as well as teaching resources and methodologies to facilitate the production of intelligent multimedia tutoring systems.

The three strands of this project explore differing aspects of multimedia, and while they stand as separate pieces of work in their own right, they also form the building blocks of the final deliverable, which is an intelligent tutoring system. Intermediate outcomes include a multimedia shell and the Manley Group Demonstrator.

This paper focuses on the design and development of the “true multimedia” demonstrator that has now reached the stage of being a fairly mature prototype. It is used commercially by a communications company to illustrate the varying levels of sophistication which can be achieved in presentation systems and provides an exemplar demonstrating the varying degrees of functionality possible using current technology. The main requirement in the design and implementation has been usability both for end users and content developers.

Real-time control of the multimedia devices is achieved using sophisticated software which is tightly integrated with dedicated control hardware. The software can be used to produce canned presentations or highly interactive systems.

The system is installed in a dedicated presentation suite, thus allowing the true nature of multimedia to be demonstrated. It is scaleable in that it can be used to produce stand-alone portable solutions, highly interactive touch-screen controlled applications or complete multi-device high end presentations. This is achieved using the appropriate combination of software tools and hardware for a specific application.

Macromedia Director is used to produce visual and creative material for the system, while Macromedia Authorware can be used to provide interactivity. © 1997 Elsevier Science B.V.     

多媒体试题模板化生成工具的设计与开发

为了改善目前多媒体试题呈现方式单一、交互性不足、复用率低以及试题编辑软件操作复杂等现状,设计实现一个模板化的多媒体试题生成工具。通过对小学英语实际教学中使用的试题样本进行分析,确定本系统支持的多媒体试题类型,并为每种题型设计多种交互式的多媒体试题模板,结合使用XML技术进行试题描述,开发实现了本试题生成工具。     

多媒体系统中基于Petri网的同步模型

本文主要介绍了多媒体系统中的媒体同步技术和基于Ｐｅｔｒｉ网的多种同步模型,并讨论了交互式同步模型ＩＳＰＮ。该模型支持用户交互的多媒体同步关系描述,是一种灵活的动态规范。