Quality-based I/O resource managementfor multimedia document database servers

Slow access to disk-based multimedia data is a major limiting factor in the performance of modern multimedia Web servers connected over broadband networks. The I/O bottleneck becomes even more pronounced for currently evolving systems handling multimedia data, such as audio and video. Retrieval of complex multimedia documents needs to be handled at two levels: I/O bandwidth management for multiple multimedia streams, and interstream and intrastream synchronization for multimedia objects constituting these documents. In this paper, based on the diverse characteristics of multimedia data, we propose efficient techniques for synchronous retrieval and delivery of such data from the storage system to the main memory of the server. We propose methods to quantify user perceived quality via quality-of-presentation (QoP) parameters. We combine QoP and Object Composition Petri Net (OCPN) multimedia data modeling to develop techniques for efficient synchronous retrieval of multimedia data. Since I/O bandwidth is a precious resource, the proposed techniques have low overhead, which is , where m is the number of logical I/O channels and n is the total number of frames of multimedia data in a scheduling period. We simulate the relative performance of these techniques under diverse I/O conditions and determine the tradeoffs between the system resources, such as memory, bandwidth, and the improvement in QoP for multimedia applications.Published online: 9 February 2005 Correspondence to: M. Farrukh Khan     

A novel model of adaptation decision-taking engine in multimedia adaptation

In heterogeneous environments, universal multimedia access (UMA) is proposed to provide multimedia content services. Multimedia adaptation is one of technologies to perform UMA, in which adaptation decision-taking engine (ADTE) is a key component. Though there are many models of ADTE existing, it needs to be reconsidered for personalized content services. In this paper, a novel model of ADTE is proposed based on decision tree termed adaptation decision tree (ADT) in which adaptation decision is viewed as sequence decision: modality decision and format decision. Correspondingly, user preferences are divided into two types: user modality preferences and user format preferences. By utilizing user preferences, the ADT model is built up. Before making decision, an optimal multimedia variation set (OMVS) with respect to user modality preferences is constructed and any element here is with the shortest distance to user format preferences for every modality. Therefore, adaptation decision can be executed by letting the element in OMVS travel along the ADT one by one. Finally, the first element that reaches the leaf with the logical value true is the decision result, or the one with the smallest value in distance is the decision variation if no elements get to proper leaf. Quantitative analysis and experimental simulation prove that the model is effective and efficient to cope with adaptation decision in multimedia adaptation especially in dynamic user preferences and resource-limited cases.     

A Multimedia Presentation Toolkit for the World Wide Web

This paper describes the design and implementation of MPRES, a Multimedia Presentation Toolkit for the WWW. The WWW has seen phenomenal growth over the last couple of years. It has become a vast repository of multimedia information that is accessible to virtually anyone having a browser. MPRES is a multimedia presentation system that allows a user to compose and render a presentation consisting of objects referenced by their URLs (Uniform Resource Locators). It uses the concept of dynamic documents to render on a WWW browser, a sequence of multimedia scenarios, having objects of types such as audio, image, plaintext, HTML (Hypertext Markup Language) document and animation. MPRES Author, the authoring subsystem, allows the user to interactively test and compose such a presentation, using the Netscape Navigator to collect multimedia resources from the WWW. A presentation database stores the presentations and provides a convenient frontend for accessing them. © 1997 John Wiley & Sons,Ltd.     

ZYX-a multimedia document model for reuse and adaptationof multimedia content

Advanced multimedia applications require adequate support for the modeling of multimedia content by multimedia document models. More and more this support calls for not only the adequate modeling of the temporal and spatial course of a multimedia presentation and its interactions, but also for the partial reuse of multimedia documents and adaptation to a given user context. However, our thorough investigation of existing standards for multimedia document models such as HTML, MHEG, SMIL, and HyTime leads to us the conclusion that these standard models do not provide sufficient modeling support for reuse and adaptation. Therefore, we propose a new approach for the modeling of adaptable and reusable multimedia content, the Z_YX model. The model offers primitives that provide-beyond the more or less common primitives for temporal, spatial, and interaction modeling-a variform support for reuse of structure and layout of document fragments and for the adaptation of the content and its presentation to the user context. We present the model in detail and illustrate the application and effectiveness of these concepts by samples taken from our Cardio-OP application in the domain of cardiac surgery. With the Z_YX model, we developed a comprehensive means for advanced multimedia content creation: support for template-driven authoring of multimedia content and support for flexible, dynamic composition of multimedia documents customized to the user's local context and needs. The approach significantly impacts and supports the authoring process in terms of methodology and economic aspects     

Universal Access in the Information Society: Methods, Tools, and Interaction Technologies

Accessibility and high quality of interaction with products, applications, and services by anyone, anywhere, and at any time are fundamental requirements for universal access in the emerging Information Society. This paper discusses these requirements, and their relation to the concept of automated adaptation of user interfaces. An example application is presented, showing how adaptation can be used to accommodate the requirements of different user categories and contexts of use. This application is then used as a vehicle for discussing a new engineering paradigm appropriate for the development of adaptation-based user interfaces. Finally, the paper investigates issues concerning the interaction technologies required for universal access. Published online: 23 May 2001     

Toward better retrieval and presentation by exploring cross-media correlations

This paper addresses explicit correlation and implicit correlation between various media streams in a composite multimedia document, the so-called navigated hypermedia document in our language learning system, in order to facilitate document retrieval and synchronized presentation. For replaying a recorded lecture in a form as close as possible to the original classroom experience, we devised a capturing mechanism to explicitly record all the lecturing media streams and relations between them, including instructors voice, slide change of the HTML lectures, and various guiding actions (e.g., tele-pointers, pen strokes, document scrolling, keyword highlighting, and text annotations) on HTML-based slides. In addition, for more effective learning, we study three different aspects - temporal, spatial, and content relation - of the implicit correlations that are inherently hidden between the media involved. The implicit relations are discovered by three designed processes: the speech-text alignment process for temporally synchronized speech-text presentation, the automatic scrolling process for the viewing windows spatial synchronization, and the content dependency checking process to ensure consistency of the content processed and the relations involved. The experimental results show that exploring cross-media correlations is helpful for system development in document presentation and retrieving. Users are allowed to replay a vivid and learning-effective multimedia lecture and to access the desired part of the document very easily via cross-media indexing. Hence the results have been applied to the development of online multimedia language learning systems aimed at improving students English and Chinese language capabilities.Published online: 14 December 2004     

Distributed Context Monitoring for the Adaptation of Continuous Services

This paper describes a middleware designed for distributed context acquisition and reconciliation intended to support the adaptation of continuous Internet services, like e.g., multimedia streaming. These services persist in time, and are characterized by multiple transmissions of data by the service provider, as a result of a single request from the user. Adapting these services to the current context requires the continuous monitoring of context data, and a real-time adjustment of the adaptation parameters upon the detection of a relevant context change. The proposed solution is based on asynchronous context change notifications, and specific techniques have been designed to minimize the number of unnecessary updates and the re-evaluation of policies. The paper also provides experimental results obtained by developing an adaptive video streaming system and running it on top of the proposed middleware.     

A holistic approach for personalization,relevance feedback & recommendation in enriched multimedia content

As the prerequisites of production houses, broadcasters, advertising agencies and online publishing companies for enriched multimedia content increase rapidly, the need of innovative methods for the effective creation of enriched multimedia content is undeniable. Stemming from this need, in this paper we focus on the design, development and evaluation of a framework consisting of personalization, relevance feedback and recommendation mechanisms, as a principal method for the creation of enriched multimedia content targeted to each user’s needs, preferences and interests. As the multimedia content proliferates along with its consumption by the users, more effective ways of presenting it to the viewers are demanded in order to facilitate them with the multimedia content search and selection and improve their Quality of Experience (QoE). The main contribution of the paper is the introduction of a holistic framework that offers personalized enriched multimedia content, by extending the recommendation process to the set of enrichments that accompany the video except from the video itself and by collecting explicit and implicit relevance feedback from the interactions of the user with both the video and its enrichments. We evaluate the proposed framework following a two-step approach. Firstly, we perform extended experiments by applying reasonably simulated user interactions, in order to calibrate its parameters that refer to multiple aspects of the enriched multimedia content, aiming at high performance in terms of QoE. Here, most importantly, we have shown that appropriately designing the enrichments and considering users’ interactions with them allows for achieving a better quality in inferring users’ profiles in many realistic cases. Secondly, we integrated our proposed recommender framework within the MECANEX streaming platform in order to perform user studies about its usability within a realistic environment of use.     

Merging Results for Distributed Content Based Image Retrieval

Searching information through the Internet often requires users to separately contact several digital libraries, use each library interface to author the query, analyze retrieval results and merge them with results returned by other libraries. Such a solution could be simplified by using a centralized server that acts as a gateway between the user and several distributed repositories: The centralized server receives the user query, forwards the user query to federated repositories—possibly translating the query in the specific format required by each repository—and fuses retrieved documents for presentation to the user. To accomplish these tasks efficiently, the centralized server should perform some major operations such as: resource selection, query transformation and data fusion. In this paper we report on some aspects of MIND, a system for managing distributed, heterogeneous multimedia libraries (MIND, 2001, http://www.mind-project.org). In particular, this paper focusses on the issue of fusing results returned by different image repositories. The proposed approach is based on normalization of matching scores assigned to retrieved images by individual libraries. Experimental results on a prototype system show the potential of the proposed approach with respect to traditional solutions.     

Multimedia Document Models: Sealed Fate or Setting Out for New Shores?

Existing multimedia document models like HTML, MHEG, SMIL, and HyTime lack appropriate modeling primitives to fit the needs of next generation multimedia applications which bring up requirements like reusability of multimedia content in different presentations and contexts, and adaptation to user preferences. In this paper, we motivate and present new requirements stemming from advanced multimedia applications and the resulting consequences for multimedia document models. Along these requirements, we discuss the document model standards HTML, HyTime, MHEG, SMIL, and Z_YX, a new model that has been developed with special focus on reusability and adaptation. The analysis and comparison of the models show the limitations of existing models, point the way to the need for new flexible multimedia document models, and throw light on the many implications on authoring systems, multimedia content management, and presentation.     

Design and implementation of a video browsing system for the Internet

Recent advances in multimedia processing technologies, internet‐working technologies, and the World Wide Web phenomenon have resulted in a vast creation and use of digital videos. Due to this reason, an efficient technique to locate and retrieve a desired video from a remote video archive is needed. A trial‐and‐error approach popularly used in current Web search engines is not applicable for searching for a desired video segment since the technique incurs intolerable delays. In this paper, we present the design and implementation of a video browsing system that lets the user view a summary of a selected video and search within the video while being downloaded so that the user can determine the relevance of the video as early as possible. The system is inexpensive and scalable, making it suitable for large‐scale distributed systems such as the Internet. The browsing system consists of two major software components: a video server, and a video browser and player called VideoCenter implemented using Microsoft DirectShow multimedia development kit. The implementation of VideoCenter enables us to assess the ease of use of DirectShow as well as its drawbacks in developing multimedia applications. Copyright © 2001 John Wiley & Sons, Ltd.     

Classification algorithms for interactive multimedia services: a review

Interactive multimedia services, which integrate and unify techniques from a variety of disciplines, have been an active research topic for many years. However, two major challenges need to be overcome to provide a better service: The first is that interactive multimedia systems have to provide the contents a user needs at the right time no matter where the user is located and what device the user is using; the second is that the performance of such systems needs to be improved. Apparently, classification and clustering (also called unsupervised classification) algorithms play an indispensable role in these respects. Thus, this paper contains a review of the classification algorithms for interactive multimedia systems. Also discussed in this paper are several important issues, open questions, and trends.     

多媒体文件在线展示通用框架设计及实现

一般视音频网站在上传多媒体文件后要转换文件格式,不但消耗服务器软硬件资源,而且不同程度地降低视音频质量。针对这些问题,提出一种不用转换视音频格式即可在线展示视音频以及其他类型多媒体文件的通用框架,并给出了实现方法。该框架适用于视频、音频、图像、文档等多媒体文件的在线展示。具有实时性、高质量、通用性、自适应性、可扩展性等特点。     

View management in multimedia databases

Abstract. Though there has been extensive work on multimedia databases in the last few years, there is no prevailing notion of a multimedia view, nor there are techniques to create, manage, and maintain such views. Visualizing the results of a dynamic multimedia query or materializing a dynamic multimedia view corresponds to assembling and delivering an interactive multimedia presentation in accordance with the visualization specifications. In this paper, we suggest that a non-interactive multimedia presentation is a set of virtual objects with associated spatial and temporal presentation constraints. A virtual object is either an object, or the result of a query. As queries may have different answers at different points in time, scheduling the presentation of such objects is nontrivial. We then develop a probabilistic model of interactive multimedia presentations, extending the non-interactive model described earlier. We also develop a probabilistic model of interactive visualization where the probabilities reflect the user profiles, or the likelihood of certain user interactions. Based on this probabilistic model, we develop three utility-theoretic based types of prefetching algorithms that anticipate how users will interact with the presentation. These prefetching algorithms allow efficient visualization of the query results in accordance with the underlying specification. We have built a prototype system that incorporates these algorithms. We report on the results of experiments conducted on top of this implementation. Received June 10, 1998 / Accepted November 10, 1999     

NinSuna: a fully integrated platform for format-independent multimedia content adaptation and delivery using Semantic Web technologies

The current multimedia landscape is characterized by a significant heterogeneity in terms of coding and delivery formats, usage environments, and user preferences. The main contribution of this paper is a discussion of the design and functioning of a fully integrated platform for multimedia adaptation and delivery, called NinSuna. This platform is able to efficiently deal with the aforementioned heterogeneity in the present-day multimedia ecosystem, thanks to the use of format-agnostic adaptation engines (i.e., engines independent of the underlying coding format) and format-agnostic packaging engines (i.e., engines independent of the underlying delivery format). Moreover, NinSuna also provides a seamless integration between metadata standards and adaptation processes. Both our format-independent adaptation and packaging techniques rely on a model for multimedia bitstreams, describing the structural, semantic, and scalability properties of these multimedia streams. News sequences were used as a test case for our platform, enabling the user to select news fragments matching his/her specific interests and usage environment characteristics.     

Seamless video access for mobile devices by content-aware utility-based adaptation

Today’s Internet multimedia services are characterized by heterogeneous networks, a wide range of terminals, diverse user preferences, and varying natural environment conditions. Heterogeneity of terminals, networks, and user preferences impose nontrivial challenges to the Internet multimedia services for providing seamless multimedia access particularly for mobile devices (e.g., laptops, tablet PCs, PDAs, mobile phones, etc.). Thus, it is essential that advanced multimedia technologies are developed to deal with these challenges. One of these technologies is video adaptation, which has gained significant importance with its main objective of enabling seamless access to video contents available over the Internet. Adaptation decision taking, which can be considered as the “brain” of video adaptation, assists video adaptation to achieve this objective. Scalable Video Coding (SVC) offers flexibility for video adaptation through providing a comprehensive set of scalability parameters (i.e., temporal, spatial, and quality) for producing scalable video streams. Deciding the best combination of scalability parameters to adapt a scalable video stream while satisfying a set of constraints (e.g., device specifics, network bandwidth, etc.) poses challenges for the existing adaptation services to enable seamless video access. To ease such challenges, an adaptation decision taking technique employing a utility-based approach to decide on the most adequate scalability parameters for adaptation operations is developed. A Utility Function (UF), which models the relationships among the scalability parameters and weights specifying the relative importance of these parameters considering video content characteristics (i.e., motion activity and structural feature), is proposed to assist the developed technique. In order to perform the developed adaptation decision taking technique, a video adaptation framework is also proposed in this paper. The adaptation experiments performed using the proposed framework prove the effectiveness of the framework to provide an important step towards enabling seamless video access for mobile devices to enhance viewing experience of users.     

Event-driven grammars: relating abstract and concrete levels of visual languages

In this work we introduce event-driven grammars, a kind of graph grammars that are especially suited for visual modelling environments generated by meta-modelling. Rules in these grammars may be triggered by user actions (such as creating, editing or connecting elements) and in their turn may trigger other user-interface events. Their combination with triple graph transformation systems allows constructing and checking the consistency of the abstract syntax graph while the user is building the concrete syntax model, as well as managing the layout of the concrete syntax representation. As an example of these concepts, we show the definition of a modelling environment for UML sequence diagrams. A discussion is also presented of methodological aspects for the generation of environments for visual languages with multiple views, its connection with triple graph grammars, the formalization of the latter in the double pushout approach and its extension with an inheritance concept. This is a revised and extended version of a paper presented at the ICGT’04 conference, see [21].     

Metadata-driven optimal transcoding in a multimedia proxy

An adaptive multimedia proxy is presented which provides (1) caching, (2) filtering, and (3) media gateway functionalities. The proxy can perform media adaptation on its own, either relying on layered coding or using transcoding mainly in the decompressed domain. A cost model is presented which incorporates user requirements, terminal capabilities, and video variations in one formula. Based on this model, the proxy acts as a general broker of different user requirements and of different video variations. This is a first step towards What You Need is What You Get (WYNIWYG) video services, which deliver videos to users in exactly the quality they need and are willing to pay for. The MPEG-7 and MPEG-21 standards enable this in an interoperable way. A detailed evaluation based on a series of simulation runs is provided.