A model-based authorware for the construction of distributed multimedia systems

Distributed multimedia systems typically involve a sophisticated user interaction. Further, objects are allocated on physically distributed computing systems, and multimedia data must be transferred across heterogeneous networks in a timely manner. These systems often have complex requirements on a user interaction, quality of service and temporal order among media streams. The design and implementation of these requirements are inherently complex and present an extraordinary design and programming challenge. Generally, these complex requirements cannot be adequately captured using a single model or a design notation. The challenge amounts to (i) identification of multiple, often orthogonal models, each capturing a specific aspect of the requirements, and (ii) provision of an authorware that supports the composition of these models. In this paper, we propose to capture the multimedia requirements in three different models: configuration, user control and presentation, and demonstrate how the composition of these models can be supported by an authorware using the Java and CORBA technologies. The concepts are illustrated using a real-life example based on a virtual city tour application that features distributed controls, collaborative work and multimedia presentations. Various distributed multimedia applications like video phone, video conferencing and distributed presentation have been successfully constructed using the proposed multiple models and authorware. The results are encouraging and the approach can shorten the development of multimedia applications considerably.     

A Novel Approach Towards Large Scale Cross-Media Retrieval

With the rapid development of Internet and multimedia technology,cross-media retrieval is concerned to retrieve all the related media objects with multi-modality by submitting a query media object.Unfortunately,the complexity and the heterogeneity of multi-modality have posed the following two major challenges for cross-media retrieval:1) how to construct a unified and compact model for media objects with multi-modality,2) how to improve the performance of retrieval for large scale cross-media database.In this paper,we propose a novel method which is dedicate to solving these issues to achieve effective and accurate cross-media retrieval.Firstly,a multi-modality semantic relationship graph(MSRG) is constructed using the semantic correlation amongst the media objects with multi-modality.Secondly,all the media objects in MSRG are mapped onto an isomorphic semantic space.Further,an efficient indexing MK-tree based on heterogeneous data distribution is proposed to manage the media objects within the semantic space and improve the performance of cross-media retrieval.Extensive experiments on real large scale cross-media datasets indicate that our proposal dramatically improves the accuracy and efficiency of cross-media retrieval,outperforming the existing methods significantly.     

CORE: a content-based retrieval engine for multimedia information systems

Rapid advances in multimedia technology necessitate the development of a generic multimedia information system with a powerful retrieval engine for prototyping multimedia applications. We develop a content-based retrieval engine (CORE) that makes use of novel indexing techniques for multimedia object retrieval. We formalize the concepts related to multimedia information systems such as multimedia objects and content-based retrieval. We bring out the requirements and challenges of a multimedia information system. The architecture of CORE is described in detail along with the associated retrieval mechanisms and indexing techniques. Various modules developed for efficient retrieval are presented with some APIs. The efficacy of CORE is demonstrated in the development of two multimedia systems, a computer-aided facial image inference and retrieval (CAFIIR) system and a system for trademark archival and retrieval (STAR), which have been developed at the Institute of Systems Science (ISS). We expect that CORE will be useful for effective prototyping of other such multimedia applications.Mainly supported by National Science & Technology Board of SingaporePartly working in Real World Computing Partnership, Novel Function Institute of Systems Science Laboratory since April 1994.     

Collaborative Multimedia Presentations in Mobile Environments

Distributed multimedia documents systems, distributed video servers are examples of multimedia presentations involving collaboration among multiple information sources. In such applications, objects have to be retrieved from their sources and presented to users according to specified temporal relationships. Objects retrieval in these collaborative applications is influenced by their presentation times, durations, and network throughput available to their sources. Replication of objects amongst the set of collaborating systems gives a choice for object retrieval. Client going through a multimedia presentation can be in a mobile environment. Here, object retrievals from collaborating servers are carried out by base stations to which the client is attached. Mobile client then downloads objects from the base station.In this paper, we present a graph-search based algorithm for computing and negotiating throughput requirements of collaborating multimedia presentations with replicated objects in a mobile environment. This algorithm maximizes the number of cached objects (that have already been played) for handling operations such as reverse presentation.     

Temporal Semantics for Concurrent M M

Concurrent is a programming language based on the notion of concurrent, communicating objects, where each object directly executes a specification given in temporal logic, and communicates with other objects using asynchronous broadcast message-passing. Thus, Concurrent represents a combination of the direct execution of temporal specifications, together with a novel model of concurrent computation. In contrast to the notions of predicates as processes and stream parallelism seen in concurrent logic languages, Concurrent represents a more coarse-grained approach, where an object consists of a set of logical rules and communication is achieved by the evaluation of certain types of predicate. Representing concurrent systems as groups of such objects provides a powerful tool for modelling complex reactive systems. In order to reason about the behaviour of Concurrent systems, we requir a suitable semantics. Being based upon executable temporal logic, objects in isolation have an intuitive semantics. However, the addition of both operational constraints upon the object's execution and global constraints provided by the asynchronous model of concurrency and communication, complicates the overall semantics of networks of objects. It is this, more complex, semantics that we address here, where temporal semantics for varieties of Concurrent are provided.     

Content-based management of video in a multimedia authoring environment

This paper describes architecture and implementation of a multimedia authoring environment which features content-based management of video.The goal of our work is to provide multimedia application developers with a useful video database, where they gather video materials from various sources, store them in a organized format, and use them in multimedia applications. The principal design concepts are that (1) it should have a flexible means to store video materials in a organized format so that video clips can be retrieved easily, and that (2) it should have a view to each application so that various applications can share the video materials.To meet these requirements we designed a multimedia authoring system based on a video object model. The video object model features (1) a method to attach annotations not only to scenes but also to things shot in the video, and (2) a method to define video clips independently from physical video data.Our experiences in using the system to develop a multimedia CAI application are also described to show the advantages of our system.     

A model for secure multimedia document database system in a distributed environment

The Internet provides a universal platform for large-scale distribution of information and supports inter-organizational services, system integration, and collaboration. Use of multimedia documents for dissemination and sharing of massive amounts of information is becoming a common practice for Internet-based applications and enterprises. With the rapid proliferation of multimedia data management technologies over the Internet, there is growing concern about security and privacy of information. Composing multimedia documents in a distributed heterogeneous environment involves integrating media objects from multiple security domains that may employ different access control policies for media objects. In this paper, we present a security model for distributed document management system that allows creation, storage, indexing, and presentation of secure multimedia documents. The model is based on a time augmented Petri-net and provides a flexible, multilevel access control mechanism that allows clearance-based access to different levels of information in a document. In addition, the model provides detailed multimedia synchronization requirements including deterministic and non-deterministic temporal relations and incomplete timing information among media objects.     

Towards a theory of multimedia metacomputing

Multimedia metacomputing is a new approach to the management and processing of multimedia data in web-based information systems. It offers high flexibility and openness while shielding the applications from any system internals. Starting with the vision of a completely open and globally distributed multimedia information system, we consider abstraction concepts required, especially transformation independence, and an appropriate semantic model.Thus, the major focus of this paper is on the abstract data and processing model called VirtualMedia,¹ which provides a transformation independence framework for multimedia processing. In particular, we describe how transformation requests are represented and processed, exploiting semantic equivalence relations on filter graphs and redundant materialization, finally yielding instantiatable plans for materializing the requested media object(s) at the client.     

Spatio-temporal composition and indexing for large multimedia applications

Multimedia applications usually involve a large number of multimedia objects (texts, images, sounds, etc.). An important issue in this context is the specification of spatial and temporal relationships among these objects. In this paper we define such a model, based on a set of spatial and temporal relationships between objects participating in multimedia applications. Our work exploits existing approaches for spatial and temporal relationships. We extend these relationships in order to cover the specific requirements of multimedia applications and we integrate the results in a uniform framework for spatio-temporal composition representation. Another issue is the efficient handling of queries related to the spatio-temporal relationships among the objects during the authoring process. Such queries may be very costly and appropriate indexing schemes are needed so as to handle them efficiently. We propose efficient such schemes, based on multidimensional (spatial) data structures, for large multimedia applications that involve thousands of objects. Evaluation models of the proposed schemes are also presented, as well as hints for the selection of the most appropriate one, according to the multimedia author's requirements.     

Sequence optimization for media objects with due date constraints in multimedia presentations from digital libraries

This study investigates sequence optimization of media objects in a multimedia presentation that is dynamically composed from digital libraries. Each media object can be associated with a due date. The aim is to schedule the media objects in a delay-prone network environment such that the overall presentation lag and the due date penalties of the media objects of presentations can be minimized. We formulate the sequencing problem with buffer constraints in the media player into a flowshop scheduling problem and present a reduction strategy with a branch and bound algorithm to derive optimal sequences. The algorithm can be applied in applications with up to a dozen objects to be scheduled. In addition, we propose a modified NEH-based heuristic algorithm which can provide approximate solutions with an average error rate of less than 4%. The computation-efficient heuristic, when deployed in applications with heavily loaded servers, can obtain near-optimal sequences for problems with more than a dozen objects. The proposed algorithms are embedded into a prototype system for providing digital library services.     

An ODA-like Multimedia Document System

A multimedia document is composed of different media objects. ISO's Open Document Architecture (ODA) proposes a standard multimedia document model. However, the current ODA profile only includes static media, e.g. text, geometric graphics and images. Because the future multimedia documents not only include static media but also continuous media, e.g. video and audio, continuous media document parts should be added to have a complete multimedia document model. In this paper, we propose a multimedia document model, which is derived from ODA's concept. The proposed model is based on the object-oriented approach. Objects in the proposed document model are divided into two types: data objects and pseudo objects. Data objects are data structures of a document; pseudo objects are used to manage data objects. Based on the proposed model, a multimedia document authoring and presenting system (MMDS) is also developed on SUN SPARC workstations using the Solaris 2.X operating system     

A Domain-Specific Metamodel for Multimedia Processing Systems

In this paper, we introduce 4MPS, a metamodel for multimedia processing systems. The goal of 4MPS is to offer a generic system metamodel that can be instantiated to describe any multimedia processing design. The metamodel combines the advantages of the object-oriented paradigm and metamodeling techniques with system engineering principles and graphical models of computation. 4MPS is based on the classification of multimedia processing objects into two main categories: Processing objects that operate on data and controls, and Data objects that passively hold media content. Processing objects encapsulate a method or algorithm. They also include support for synchronous data processing and asynchronous event-driven Controls as well as a configuration mechanism and an explicit life cycle state model. Data input to and output from Processing objects is done through Ports. Data objects offer a homogeneous interface to media data, and support for metaobject-like facilities such as reflection and serialization. The metamodel can be expressed in the language of graphical models of computation such as the Dataflow Networks and presents a comprehensive conceptual framework for media signal processing applications. 4MPS has its practical validation in several existing environments, including the author's CLAM framework.     

多媒体数据库系统中的同步表示管理

支持多媒体应用的扩展数据库系统必须具有新的机制以确保多种媒体数据流的同步表示。为了使用户同时具有多媒体数据流表示的灵活性和高效率,必须将媒体数据流分割成媒体对象,并且定义和保持其间的时间限制。为此要在现有数据库系统上采用能高效管理多媒体数据时间传性的新管理工具。文中讨论了多媒体数据库系统中同步表示管理。当考虑延迟的影响时,提出了多种媒体数据流同步表示的原则。基于这些原则,可以设计出各种同步机制。     

On the Design of Distributed Object Placement and Load Balancing Strategies in Large-Scale Networked Multimedia Storage Systems

In a large-scale multimedia storage system (LMSS) where client requests for different multimedia objects may have different demands, the placement and replication of the objects is an important factor, as it may result in an imbalance in server loading across the system. Since replication management and load balancing are all the more crucial issues in multimedia systems, in the literature, these problems are handled by centralized servers. Each object storage server (OSS) responds to the requests that come from the centralized servers independently and has no communication with other OSSs in the system. In this paper, we design a novel distributed load balancing strategy for LMSS, in which OSSs can cooperate to achieve higher performance. Such OSS modeled as an M/G/m system can replicate the objects to and balance the requests among other servers to achieve a near-optimal average waiting time (AWT) of the requests in the system. We validate the performance of the system via rigorous simulations with respect to several influencing factors and prove that our proposed strategy is scalable, flexible, and efficient for real-life applications.     

Multimedia integrated annotation based on common space learning

Multimedia automatic annotation, which assigns text labels to multimedia objects, has been widely studied. However, existing methods usually focus on modeling two types of media data or pairwise correlation. In fact, heterogeneous media are complementary to each other and optimizing them simultaneously can further improve accuracy. In this paper, a novel common space learning (CSL) algorithm for multimedia integrated annotation is presented, by which heterogeneous media data can be projected into a unified space and multimedia annotation is transformed to the nearest neighbor search in the space. Optimizing these heterogeneous media simultaneously makes the heterogeneous media complementary to each other and aligned in the common space. We solve the proposed CSL as an optimization problem mainly considering the following issues. First, different types of media objects with the similar labels should be closer in the common space. Second, the media similarity of the original space and the common space should be consistent. We attempt to solve the optimization problem in a sparse and semi-supervised learning framework, thus more unlabeled data can be integrated into the learning process, which can boost the performance of space learning. In addition, we proposed an iterative optimization algorithm to solve the problem. Since the projected samples in the common space share the same representation, the labels for new media object are assigned by a simple nearest neighbor voting mechanism. To the best of our knowledge, our method has made the first attempt to multimedia integrated annotation. Experiments on data sets with up to four media types (image, sound, video and 3D model) show the effectiveness of our proposed approach, as compared with the state-of-the-art methods.

     

人造复杂系统的多媒体认知软件中数据模型的研究

文章根据人造复杂系统的特性和人的认知规律,提出了一个基于对象的多媒体认知软件系统数据模型。数据模型由媒体存取层、多媒体对象层和多媒体表现层和多媒体应用层构成。这种层次模型将媒体异构环境存取、多媒体对象处理和语义合成表现以及各种时序关系等问题在不同层次分别进行处理,使软件系统具有很好的灵活性和可复用性。     

Multimedia object modeling and storage allocation strategies

The improvements in disk speeds have not kept up with improvements in processor and memory speeds. Many techniques have been proposed and utilized to maximize the bandwidths of storage devices. These techniques have proven useful for conventional data, but when applied to multimedia data, they tend to be insufficient or inefficient due to the diversified data types, bandwidth requirements, file sizes and structures of complex objects of multimedia data. In this paper, we discuss the design of an efficient multimedia object allocation strategy that strives to achieve the expected retrieval rates and I/O computational requirements of objects; and also effectively balances the loads on the storage devices. We define a multimedia object model, describe the multimedia object and storage device characteristics, the classification of the multimedia objects according to their I/O requirements, and the fragmentation strategies. We use a bipartite graph model for mapping of fragments to storage devices. A cost function based on a disk utilization per allocated space, the amount of free space, and the bandwidth of a storage device is used to determine the optimal allocation for an object's data.     

MPEG-4形式化同步描述信息的研究与设计

媒体对象内部和多个媒体对象之间的同步是多媒体研究的一个重要研究方向。简述了MPEG-4的相关概念以及体系结构,融合并扩展了诸多传统方法,提出加权有向图的形式化同步描述方法在MPEG-4体系结构中的同步层中应用,这种方法更加适应MPEG-4面向对象的内容编码和传输。通过测试结果可以看出,加权有向图的形式化同步描述方法可以有效地实现同步。