Spatio-temporal composition and indexing for large multimedia applications

Multimedia applications usually involve a large number of multimedia objects (texts, images, sounds, etc.). An important issue in this context is the specification of spatial and temporal relationships among these objects. In this paper we define such a model, based on a set of spatial and temporal relationships between objects participating in multimedia applications. Our work exploits existing approaches for spatial and temporal relationships. We extend these relationships in order to cover the specific requirements of multimedia applications and we integrate the results in a uniform framework for spatio-temporal composition representation. Another issue is the efficient handling of queries related to the spatio-temporal relationships among the objects during the authoring process. Such queries may be very costly and appropriate indexing schemes are needed so as to handle them efficiently. We propose efficient such schemes, based on multidimensional (spatial) data structures, for large multimedia applications that involve thousands of objects. Evaluation models of the proposed schemes are also presented, as well as hints for the selection of the most appropriate one, according to the multimedia author's requirements.     

一个多媒体数据同步模型的建立和实现

文章对多媒体数据流间存在的时序问题和时序关系进行介绍，对现有研究情况进行分析.结合一个多媒体数据模型，基于PETRI网，提出了一个同步模型，以表达媒体数据之间存在的同步关系.根据这两个模型对多媒体数据建立了相应的数据结构，并采用该数据结构介绍了对多媒体数据存取播放的实现算法.     

Interval-based conceptual models for time-dependent multimedia data

Multimedia data often have time dependencies that must be satisfied at presentation time. To support a general-purpose multimedia information system, these timing relationships must be managed to provide utility to both the data presentation system and the multimedia author. New conceptual models for capturing these timing relationships, and managing them as part of a database are proposed. Specifically, n-ary and reverse temporal relations are introduced and defined along with their temporal constraints. These new relations are a generalization of earlier temporal models and establish the basis for conceptual database structures and temporal access control algorithms to facilitate forward, reverse, and partial-interval evaluation during multimedia object playout. The proposed relations are defined to ensure a property of monotonically increasing playout deadlines to facilitate both real-time deadline-driven playout scheduling or optimistic interval-based process playout. A translation of the conceptual models to a structure suitable for a relational database is presented     

A model-based authorware for the construction of distributed multimedia systems

Distributed multimedia systems typically involve a sophisticated user interaction. Further, objects are allocated on physically distributed computing systems, and multimedia data must be transferred across heterogeneous networks in a timely manner. These systems often have complex requirements on a user interaction, quality of service and temporal order among media streams. The design and implementation of these requirements are inherently complex and present an extraordinary design and programming challenge. Generally, these complex requirements cannot be adequately captured using a single model or a design notation. The challenge amounts to (i) identification of multiple, often orthogonal models, each capturing a specific aspect of the requirements, and (ii) provision of an authorware that supports the composition of these models. In this paper, we propose to capture the multimedia requirements in three different models: configuration, user control and presentation, and demonstrate how the composition of these models can be supported by an authorware using the Java and CORBA technologies. The concepts are illustrated using a real-life example based on a virtual city tour application that features distributed controls, collaborative work and multimedia presentations. Various distributed multimedia applications like video phone, video conferencing and distributed presentation have been successfully constructed using the proposed multiple models and authorware. The results are encouraging and the approach can shorten the development of multimedia applications considerably.     

A framework for synchronous delivery of time-dependent multimedia data

Multimedia data often have time dependencies that must be satisfied at presentation time. To support a general purpose multimedia information system, these timing relationships must be managed to provide utility to both the data presentation system and the multimedia author. Timing management encompasses specification, data representation, temporal access control, playout scheduling, and run-time intermedia synchronization. In this paper we describe the components of our framework for supporting time-dependent multimedia data encompassing these areas and how they are assembled into a unified system.     

A spatio-temporal semantic model for multimedia database systemsand multimedia information systems

As more information sources become available in multimedia systems, the development of abstract semantic models for video, audio, text, and image data is becoming very important. An abstract semantic model has two requirements: it should be rich enough to provide a friendly interface of multimedia presentation synchronization schedules to the users and it should be a good programming data structure for implementation in order to control multimedia playback. An abstract semantic model based on an augmented transition network (ATN) is presented. The inputs for ATNs are modeled by multimedia input strings. Multimedia input strings provide an efficient means for iconic indexing of the temporal/spatial relations of media streams and semantic objects. An ATN and its subnetworks are used to represent the appearing sequence of media streams and semantic objects. The arc label is a substring of a multimedia input string. In this design, a presentation is driven by a multimedia input string. Each subnetwork has its own multimedia input string. Database queries relative to text, image, and video can be answered via substring matching at subnetworks. Multimedia browsing allows users the flexibility to select any part of the presentation they prefer to see. This means that the ATN and its subnetworks can be included in multimedia database systems which are controlled by a database management system (DBMS). User interactions and loops are also provided in an ATN. Therefore, ATNs provide three major capabilities: multimedia presentations, temporal/spatial multimedia database searching, and multimedia browsing     

Dynamic Meta Modeling with Time: Specifying the Semantics of Multimedia Sequence Diagrams

UML offers different diagram types to model behavior and dynamics of software systems. In some domains like embedded real-time systems or multimedia systems, it is necessary to include specifications of time since the correctness of these applications depends on the fulfillment of temporal requirements in addition to functional requirements. UML thus already incorporates language features to model time and temporal constraints. Such model elements must have an equivalent in the semantic domain. We have proposed Dynamic Meta Modeling (DMM) as a means for the specification of the formal operational semantics of UML models by applying graph transformation to the meta modeling of dynamic behavior. Within this paper, we extend this approach to also account for time by building on timed graph transformations. We apply these concepts to the domain of multimedia application modeling in which we adopt UML sequence diagrams. The DMM rules with time then specify an interpreter that can be used to analyze or test a model of multimedia sequence diagrams.     

Adaptive Multimedia Presentation Strategies

Multimedia presentations comprise various media objects such as text, audio, image, and video that are delivered to users according to certain temporal relationships. In stored multimedia presentations, these temporal relationships are explicitly formulated by the author(s) and stored along with the presentations. However, it is difficult to ensure that these temporal relationships are always strictly preserved in real-time, distributed multimedia presentations. This is due to the fact that various components of the run-time environment such as operating system and network may offer only best effort services, i.e., they may not be able to provide any real-time guarantees. In this paper, we survey the different approaches that can be used for adapting multimedia presentations to handle instances where temporal relationships cannot be preserved in a strict manner. We classify these approaches into three categories and discuss when these categories of adaptations can be used.     

Nondeterministic Temporal Relations in Multimedia Data

Synchronization is an important issue in multimedia systems which integrate a variety of temporally related media objects.One part of synchronization is the representation of temporal information.Wish the emerging interactive multimedia,deterministic temporal models are replaced by nondeterministic ones with more expressiveness.This paper classifies temporal models by their expressiveness,and evealuates relevant nondeterministic temporal relations in multimedia data.Additionally,an interval-based nondeterministic model based on a complete temporal operator set is proposed providing high-leval abstractions and a high degree of expressiveness for interactive multimedia systems.     

演员模型：一种多媒体数据表达模型

本文提出一种多媒体数据表达模型：演员模型，该模型建立在多媒体数据库概念模型基础之上，基于库所—时间Ｐｅｔｒｉ网，描述多媒体对象之间的时态关系，同时引入与变迁相关联的场景表达式，描述多媒体对象之间的空间关系，从而模拟多媒体数据的时空编组，文中形式化地定义了演员模型的抽象模型和实现模型，提出多媒体对象的表达算法以及多媒体数据库的查询模型．     

分布存储媒体传输的同步算法

分布多媒体数据库（ＤＭＤＢ）和分布多媒体信息系统（ＤＭＩＳ）的一个重要需求是多媒体同步传输．文中先分析了在ＤＭＤＢ和ＤＩＭＳ中多媒体同步传输的特点和要求，然后提出了一个能满足这些要求的同步传输方案．在这个方案中，多媒体同步关系是用动态同步Ｐｅｔｒｉ网（ＤＳＰＮ）来显式描述的；通过文中给出的同步传输算式和同步发送算法，可以在ＤＳＰＮ模型的基础上产生同步传输调度方案；同时接收端能根据实际通信状况和多媒体的同步要求，对来自多个服务器的多媒体对象进行动态同步控制．     

Predicting celebrity attendees at public events using stock photo metadata

The large collections of news images available from stock photo agencies provide interesting insights into how different celebrities are related to each other, in terms of the events they attend together and also in terms of how often they are photographed together. In this paper, we leverage such collections to predict which celebrities will attend future events. The main motivation for this is in the event-based indexing of online collections of multimedia content, an area that has attracted much attention in recent years. Based on the metadata associated with a corpus of stock photos, we propose a language model for predicting celebrities attending future events. A temporal hierarchical version of the language model exploits fresh data while still making use of all historical data. We extract a social network from co-appearance of public figures in the events depicted in the photographs, and combine this latent social information with the language model to further improve prediction accuracy. The experimental results show that combining textual, network and temporal information gives the best prediction performance. Our analysis also shows that the prediction models, when trained by the most recent data, are most accurate for political and sports events.     

Multiquality Data Replication in Multimedia Databases

In contrast to other database applications, multimedia data can have a wide range of quality parameters, such as spatial and temporal resolution and compression format. Users can request data with specific quality requirements due to the needs of their application or the limitations of their resources. The database can support multiple qualities by converting data from the original (high) quality to another (lower) quality to support a user's query or precompute and store multiple quality replicas of data items. On-the-fly conversion of multimedia data (such as video transcoding) is very CPU intensive and can limit the level of concurrent access supported by the database. Storing all possible replicas, on the other hand, requires unacceptable increases in storage requirements. In this paper, we address the problem of multiple-quality replica selection subject to an overall storage constraint. We establish that the problem is NP-hard and provide heuristic solutions under two different system models: hard-quality and soft-quality. Under the soft-quality model, users are willing to negotiate their quality needs, as opposed to the hard-quality system wherein users can only accept the exact quality requested. Extensive simulations show that our algorithm performs significantly better than other heuristics. Our algorithms are flexible in that they can be extended to deal with changes in query pattern     

A model for specification and synchronization of data for distributed multimedia applications

As network technology provides the capability to handle multimedia traffic and the demand of multimedia services increases, protocols are required for effective communication of multimedia data in a distributed environment. Synchronization is one of the key issues in a multimedia system. Most of the current approaches do not support an integrated solution to the problem of synchronization. In this paper we propose a mechanism for synchronization of multimedia data in distributed environment where the accuracy of the protocol can be tailored to the application. The system model supports live and video-on-demand service. We present a scheme where the specification of the temporal requirements provided by the application can be directly mapped to obtain the information necessary to enforce the synchronization required. We present two examples of specifying the temporal requirements and process of obtaining the information and present performance results of our simulation studies.     

Multimedia network protocols: where are we?

Protocols for multimedia communication are needed to integrate into a single network services intended to satisfy the different requirements of multiple types of traffic. An essential prerequisite for designing these protocols is that the services to be offered by the network must be selected and specified in detail. We present the service models proposed, or being developed, by the Internet community, by the ATM community, and by the Tenet Group. We compare their common characteristics, which reveal the characteristics of the first integrated services networks are likely to offer. The services referred to in this paper are those at the network and transport layers, which support the services to be offered to the system's end users.     

Joint Source/Channel Coding to Maximize Perceptual Video Quality on Packet Networks

The QoS-guarantee in real-time communication for multimedia applications is significantly important. An architectural framework for multimedia networks based on substreams or flows are effectively exploited for combining source and channel coding for multimedia data. In this paper, firstly, we designed a source coding scheme based on wavelet transform to exploit the human visual systems's spatial and temporal characteristics of video data. Secondly, we designed a classification scheme to partition video data into multiple substreams which have their own quality of service (QoS) requirements. Finally, we designed a management (reservation and scheduling) scheme for substreams to support better perceptual video quality such as the bound of end-to-end jitter. We have shown that our joint source/channel coding scheme is better than two other schemes by simulation and a real video experiment.     

A generalization of spatial and temporal fusion methods for remotely sensed surface parameters

Remotely sensed surface parameters, such as vegetation index, leaf area index, surface temperature, and evapotranspiration, show diverse spatial scales and temporal dynamics. Generally the spatial and temporal resolutions of remote-sensing data should match the characteristics of surface parameters under observation. These requirements sometimes cannot be provided by a single sensor due to the trade-off between spatial and temporal resolutions. Many spatial and temporal fusion (STF) methods have been proposed to derive the required data. However, the methodology suffers from disorderly development. To better inform future research, this study generalizes the existing methods from around 100 studies as spatial or temporal categories based on their physical assumptions related to spatial scales and temporal dynamics. To be specific, the assumptions are related to the scale invariance of the temporal information and temporal constancy of the spatial information. The spatial information can be contexture or spatial details. Experiments are conducted using Landsat data acquired on 13 dates in two study areas and simulated Moderate Resolution Imaging Spectroradiometer (MODIS) data. The results are presented to demonstrate the typical methods from each category. This study concludes the following. (1) Contexture methods depend heavily on how components maps (contexture) are defined. They are not recommended except when components maps can be estimated properly from observed images. (2) The spatial and temporal adaptive reflectance fusion model (STARFM) and enhanced STARFM (ESTARFM) methods belong to the temporal and spatial categories, respectively. Thus, STARFM and ESTARFM should be better applied to temporal variance – dominated and spatial variance – -dominated areas, respectively. (3) Non-linear methods, such as the sparse representation-based spatio-temporal reflectance fusion model, can successfully address land-cover changes in addition to phonological changes, thereby providing a promising option for STF problems in the future.     

A Language for Content-Based Video Retrieval

We present an effective technique for automatic extraction, representation, and classification of digital video, and a visual language for formulation of queries to access the semantic information contained in digital video. We have devised an algorithm that extracts motion information from a video sequence. This algorithm provides a low-cost extension to the motion compensation component of the MPEG compression algorithm. In this paper, we present a visual language called VEVA for querying multimedia information in general, and video semantic information in particular. Unlike many other proposals that concentrate on browsing the data, VEVA offers a complete set of capabilities for specifying relationships between the image components and formulating queries that search for objects, their motions and their other associated characteristics. VEVA has been shown to be very expressive in this context mainly due to the fact that many types of multimedia information are inherently visual in nature.