Zodiac: A history-based interactive video authoring system

Easy-to-use audio/video authoring tools play a crucial role in moving multimedia software from research curiosity to mainstream applications. However, research in multimedia authoring systems has rarely been documented in the literature. This paper describes the design and implementation of an interactive video authoring system called Zodiac, which employs an innovative edit history abstraction to support several unique editing features not found in existing commercial and research video editing systems. Zodiac provides users a conceptually clean and semantically powerful branching history model of edit operations to organize the authoring process, and to navigate among versions of authored documents. In addition, by analyzing the edit history, Zodiac is able to reliably detect a composed video stream's shot and scene boundaries, which facilitates interactive video browsing. Zodiac also features a video object annotation capability that allows users to associate annotations to moving objects in a video sequence. The annotations themselves could be text, image, audio, or video. Zodiac is built on top of MMFS, a file system specifically designed for interactive multimedia development environments, and implements an internal buffer manager that supports transparent lossless compression/decompression. Shot/scene detection, video object annotation, and buffer management all exploit the edit history information for performance optimization.     

MING-I: a distributed interactive multimedia document development mechanism

The next generation of interactive multimedia documents can contain both static media, e.g., text, graph, image, and continuous media, e.g., audio and video, and can provide user interactions in distributed environments. However, the temporal information of multimedia documents cannot be described using traditional document structures, e.g., Open Document Architecture (ODA) and Standard Generalized Mark-up Language (SGML); the continuous transmission of media units also raises some new synchronization problems, which have not been met before, for processing user interactions. Thus, developing a distributed interactive multimedia document system should resolve the issues of document model, presentation control architecture, and control scheme. In this paper, we (i) propose a new multimedia document model that contains the logical structure, the layout structure, and the temporal structure to formally describe multimedia documents, and (ii) point out main interaction-based synchronization problems, and propose a control architecture and a token-based control scheme to solve these interaction-based synchronization problems. Based on the proposed document model, control architecture, and control scheme, a distributed interactive multimedia document development mechanism, which is called MING-I, is developed on SUN workstations.     

Modeling, authoring and formatting hypermedia documents in the HyperProp system

This paper discusses multimedia and hypermedia modeling, authoring and formatting tools, presenting the proposals of the HyperProp system and comparing them to related work. It also highlights several research challenges that still need to be addressed. Moreover, it stresses the importance of document logical structuring and considers the use of compositions in order to represent context relations, synchronization relations, derivation relations and task relations in hypermedia systems. It discusses temporal and spatial synchronization among multimedia objects and briefly presents the HyperProp graphical authoring and formatting tools. Integration between the proposed system and the WWW is also addressed.     

Checking the temporal integrity of interactive multimedia documents

Abstract. When authoring multimedia scenarios, and in particular scenarios with user interaction, where the sequence and time of occurrence of interactions is not predefined, it is difficult to guarantee the consistency of the resulting scenarios. As a consequence, the execution of the scenario may result in unexpected behavior or inconsistent use of media. The present paper proposes a methodology for checking the temporal integrity of interactive multimedia document (IMD) scenarios at authoring time at various levels. The IMD flow is mainly defined by the events occurring during the IMD session. Integrity checking consists of a set of discrete steps, during which we transform the scenario into temporal constraint networks representing the constraints linking the different possible events in the scenario. Temporal constraint verification techniques are applied to verify the integrity of the scenario, deriving a minimal network, showing possible temporal relationships between events given a set of constraints. Received June 9, 1998 / Accepted November 10, 1999     

Content-based navigation of music using melodic pitch contours

We describe a system which supports dynamic user interaction with multimedia information using content-based hypermedia navigation techniques, specialising in a technique for navigation of musical content. The model combines the principles of open hypermedia, whereby hypermedia link information is maintained by a link service, with content-based retrieval techniques in which a database is queried based on a feature of the multimedia content; our approach could be described as ‘content-based retrieval of hypermedia links’. The experimental system focuses on temporal media and consists of a set of component-based navigational hypermedia tools. We propose the use of melodic pitch contours in this context and we present techniques for storing and querying contours, together with experimental results. Techniques for integrating the contour database with open hypermedia systems are also discussed.     

Eventor: an authoring system for interactive multimedia applications

Programming, testing, and maintaining interactive multimedia applications (IMAs) are still difficult and expensive, while substantial progress has been made to reduce the burden on authors. As IMAs get larger and more complex the difficulties will increase. To overcome the complexity of such IMAs, we argue that authoring systems should provide such facilities as (1) a traditional and intuitivedivide-and-conquer paradigm for solving large and complex problems in various fields, (2)formal specification of the behaviors of IMAs for checking the syntactic correctness of visual expressions or semantic anomalies, and (3)automatic aids like validation of temporal constraints and verification of visual expressions. In this paper, we investigate the properties of IMAs for recognizing the inherent interactivity and concurrency. We propose a specification method based on Milner'sCalculus of Communicating Systems (CCS), which is a well-known formal mechanism for specifying the concurrency in various distributed applications. We also design and implement an authoring system calledEventor (Event Editor), which is based on CCS and composed of three tools: a Temporal Synchronizer, a Spatial Synchronizer, and a User Interaction Builder. They focus on describing the temporal and spatial synchronizations and user interactions while they rely on existing tools in Intel's Digital Video Interactive (DVI) for supporting other functionalities. By editing a simple computer aided instruction (CAI) application, we illustrate that our specification mechanism is well-suited for handling the interactivity of multimedia applications, and Eventor is a simple, efficient, and powerful enough tool to handle practical applications. Especially the incremental refinement and the formal specification based on the CCS allow Eventor to be extended with formal verifications to cope with large and complex applications.     

A data model and a query languagefor multimedia documents databases

A multimedia application involves information that may be in a form of video, images, audio, text and graphics, need to be stored, retrieved and manipulated in large databases. In this paper, we propose an object-oriented database schema that supports multimedia documents and their temporal, spatial and logical structures. We present a document example and show how the schema can adress all the structures described. We also present a multimedia query specification language that can be used to describe a multimedia content portion to be retrieved from the database. The language provides means by which the user can specify the information on the media as well as the temoral and spatial relationships among these media.     

Checking consistency in multimedia synchronization constraints

Constraint-based design is often used to correctly author a multimedia scenario due to its flexibility and efficiency. However, such a system must provide a mechanism with which users can easily manipulate the underlying structures to meet the application requirements. This paper proposes a novel method for analyzing multimedia synchronization constraints based on the constraint graph and classification, which is essential in developing efficient system support tools for constraint-based authoring systems. We specify temporal and spatial relations between multimedia objects, and use a directed graph to represent the constraints among the objects in a multimedia scenario. Moreover, we develop a method for analyzing temporal and spatial synchronization constraints based on graph theory, solving the problems of completeness checking, consistency checking, constraints relaxation and automatic spatio-temporal layout generation in a unified theoretical framework. We also discuss the effects of user interactive authoring. Compared with other methods, the proposed approach is simpler, more efficient, and easier to implement.     

View management in multimedia databases

Abstract. Though there has been extensive work on multimedia databases in the last few years, there is no prevailing notion of a multimedia view, nor there are techniques to create, manage, and maintain such views. Visualizing the results of a dynamic multimedia query or materializing a dynamic multimedia view corresponds to assembling and delivering an interactive multimedia presentation in accordance with the visualization specifications. In this paper, we suggest that a non-interactive multimedia presentation is a set of virtual objects with associated spatial and temporal presentation constraints. A virtual object is either an object, or the result of a query. As queries may have different answers at different points in time, scheduling the presentation of such objects is nontrivial. We then develop a probabilistic model of interactive multimedia presentations, extending the non-interactive model described earlier. We also develop a probabilistic model of interactive visualization where the probabilities reflect the user profiles, or the likelihood of certain user interactions. Based on this probabilistic model, we develop three utility-theoretic based types of prefetching algorithms that anticipate how users will interact with the presentation. These prefetching algorithms allow efficient visualization of the query results in accordance with the underlying specification. We have built a prototype system that incorporates these algorithms. We report on the results of experiments conducted on top of this implementation. Received June 10, 1998 / Accepted November 10, 1999     

An Evaluation of the Potential of Task Analysis in the Evolution of Interactive Work Systems

Developed forms of task analysis allow designers to focus on both utility and usability issues in the development of interactive work systems. The models they generate represent aspects of the human, computer and domain elements of an interactive work system. Many interactive work systems are embedded in an organisational context. Pressure for changes are present in this context and provide impetus to stakeholders to change work tasks and the supporting tools. Interactive work systems also provide evolutionary pressures of their own, changing the very task they were designed to support. One approach to coping with change has been to evolve interactive work systems. Currently none of these techniques place focus on the performance of tasks as central, and consideration of usability is minimal. However, an evolutionary design approach forces an evolutionary experience upon users, and we cannot be sure whether this approach enhances the user’s experience or degrades their performance. Given the strength of task analysis it is likely that it will be applied within evolutionary contexts. Yet, little work has been undertaken to examine whether its role will, or could be different. We ask how we can move task analysis towards being used in a principled manner in the evolution of interactive work systems. This paper examines a number of features of the approach called task knowledge structures that may be useful in evolving interactive work systems. We look at tasks and their representativeness, roles, goals, objects (their attributes, relationships, typicality and centrality) and actions. We present a developing framework for examining other task analysis approaches for their utility in supporting interactive work systems evolution. Finally, we discuss future work within the area of applying task analysis in the evolution of interactive work systems.     

An algebra for creating and querying multimedia presentations

Over the last few years, there has been a tremendous increase in the number of interactive multimedia presentations prepared by different individuals and organizations. In this paper, we present an algebra for creating and querying interactive multimedia presentation databases. This algebra operates on trees whose branches reflect different possible playouts of a set of presentations. The algebra not only extends all the classical relational operators to such databases, but also introduces a variety of novel operators for combining multiple presentations. As our algebra supports merging parts or all of existing presentations, this algebra can also be used as an authoring tool for creating multimedia presentations. We prove a host of equivalence results for queries in this algebra, which may be used to build query optimizers for interactive presentation databases.     

Transformation and exchange of multimedia objects in distributed multimedia systems

One of the challenges in the design of a distributed multimedia system is devising suitable specification models for various schemas in different levels of the system. Another important research issue is the integration and synchronization of heterogeneous multimedia objects. In this paper, we present our models for multimedia schemas and transformation algorithms. They transform high-level multimedia objects into schemas that can be used to support the presentation and communication of the multimedia objects. A key module in the system is the Object Exchange Manager (OEM). In this paper, we present the design and implementation of the OEM module, and discuss in detail the interaction between the OEM and other modules in a distributed multimedia system.     

Representations in Human–Computer Systems Development

When system developers design a computer system (or other information artefact), they must inevitably make judgements as to how to abstract the domain and how to represent this abstraction in their designs. Over the years human–computer interaction, or more generally information systems design, has had a history of developing competing methods and models for both the process and products of its development. Various paradigms have been suggested, often trying to keep pace with the changing nature of the design problem; from batch processing to interactive systems to work situations and most recently to designing for household environments. It appears timely, then, to review the nature of the design problem that faces the developers of human–computer systems and to consider some of the impact that different representations and different conceptualisations may have on their activities. Green (1998) has suggested that a single model of developing human–computer systems is not desirable, instead arguing for a number of limited theories each of which provides a useful perspective. The aim of this paper is to place competing methods side by side in order to see their strengths and weaknesses more clearly. The central tenet of the paper is that different views of both the human–computer system design process and the different abstractions, or models, that are produced during the design process have varying degrees of utility for designers. It is unlikely that any single method or modelling approach will be optimal in all circumstances. Designers need to be aware of the range of views that exist and of the impact that taking a particular approach may have on the design solution.     

Control of perceived quality of service in multimedia retrieval services: prediction-based mechanism vs. compensation buffers

In multimedia systems end-to-end delay jitter has a great impact on the continuity of information playback. Therefore, it is necessary to introduce appropriate mechanisms to compensate for delay variations, so that the intramedia and intermedia temporal relationships can be preserved. In this paper, two methods for compensation of the network delay jitter in a distributed multimedia retrieval service are compared: the first is based on prediction of the network delay jitter suffered by each information unit and retrieval time modification at the source site; the second is based on a compensation buffer at the destination site. Comparison is made by assuming a master/slave relationship between the monomedia streams composing the multimedia data flow.     

Scheduling and data layout policies for a near-line multimedia storage architecture

Recent advances in computer technologies have made it feasible to provide multimedia services, such as news distribution and entertainment, via high-bandwidth networks. The storage and retrieval of large multimedia objects (e.g., video) becomes a major design issue of the multimedia information system. While most other works on multimedia storage servers assume an on-line disk storage system, we consider a two-tier storage architecture with a robotic tape library as the vast near-line storage and an on-line disk system as the front-line storage. Magnetic tapes are cheaper, more robust, and have a larger capacity; hence, they are more cost effective for large scale storage systems (e.g., video-on-demand (VOD) systems may store tens of thousands of videos). We study in detail the design issues of the tape subsystem and propose some novel tape-scheduling algorithms which give faster response and require less disk buffer space. We also study the disk-striping policy and the data layout on the tape cartridge in order to fully utilize the throughput of the robotic tape system and to minimize the on-line disk storage space.     

Dynamic playout scheduling algorithms for continuous multimedia streams

In this paper, we investigate a playout scheduling framework for supporting the continuous and synchronized presentations of multimedia streams in a distributed multimedia presentation system. We assume a situation in which the server and network transmissions provide sufficient support for the delivery of media objects. In this context, major issues regarding the enforcement of the smooth presentation of multimedia streams at client sites must be addressed to deal with rate variance of stream presentations and delay variance of networks. We develop various playout-scheduling algorithms that are adaptable to quality-of-service parameters. The proposed algorithms permit the local adjustment of unsynchronized presentations by gradually accelerating or retarding presentation components, rather than abruptly skipping or pausing the presentation materials. A comprehensive experimental analysis of the proposed algorithms demonstrates that our algorithms can effectively avoid playout gaps (or hiccups) in the presentations. This scheduling framework can be readily used to support customized multimedia presentations.     

Analysis,management and indexing of distributed multimedia performances based on audience feedback

Interactive performance systems allow multiple, networked users to take part in a performance either as players or audience members and influence the performance's development in real time. Among the most prominent research issues concerning the development of these systems are the provision of effective interaction capabilities and adequate means of expression to the audience. This article describes a set of methods for the detection, analysis, synchronization and rendering of audience behavior in real-time during such an event. Furthermore, it describes ways in which audience feedback is utilized to dynamically index the contents of a performance so as to embed this event in large-scale multimedia services. All these methods have been applied in the creation of MISSION, a multi-user game involving both players and audience. This article presents and analyzes the design of the system and the results of user trials with it.     

Display Design of Process Systems Based on Functional Modelling

The prevalent way to present information in industrial computer displays is by using piping and instrumentation diagrams. Such interfaces have sometimes resulted in difficulties for operators because they are not sufficient to fulfil their needs. A systematic way that supports interface design therefore has to be considered. In the new design framework, two questions must be answered. Firstly, a modelling method is required to describe a process system. Such a modelling method can define the information content that must be displayed in interfaces. Secondly, how to communicate this information to operators efficiently must be considered. This will provide a basis for determining the visual forms that the information should take. This study discusses interface design of human–machine systems from these two points of view. Based on other scholars’ work, a comprehensive set of functional primitives is summarised as a basis to build a functional model of process systems. A library of geometrical presentations for these primitives is then developed. To support effective interface design, the concept of ‘functional macro’ is introduced and a way to map functional model to interface display is illustrated by applying several principles. To make our ideas clear, a central heating system is taken as an example and its functional model is constructed. Based on the functional model, the information to be displayed is determined. Several functional macros are then found in the model and their corresponding displays are constructed. Finally, by using the library of geometrical presentations for functional primitives and functional macros, the display hierarchy of the central heating system is developed. Reusability of functional primitives makes it possible to use the methodology to support interface design of different process systems.     

User Environments and Individuals: Experience with Adaptive Interface Agents

The traditional style of working with computers generally revolves around the computer being used as a tool, with individual users directly initiating operations and waiting for the results of them. A more recent paradigm of human-computer interaction, based on the indirect management of computing resources, is agent-based interaction. The idea of delegation plays a key part in this approach to computer-based work, which allows individuals to relinquish the routine, mechanistic parts of their everyday tasks, having them performed automatically instead. Adaptive interfaces combine elements of both these approaches, where the goal is to have the interface adapt to its users rather than the reverse. This paper addresses some of the issues arising from a practical software development process which aimed to support individuals using this style of interaction. This paper documents the development of a set of classes which implement an architecture for adaptive interfaces. These classes are intended to be used as part of larger user interface systems which are to exhibit adaptive behaviour. One approach to the implementation of an adaptive interface is to use a set of software “agents”– simple processes which effectively run “in the background”– to decompose the task of implementing the interface. These agents form part of a larger adaptive interface architecture, which in turn forms a component of the adaptive system.