Immersive authoring of Tangible Augmented Reality content: A user study

Immersive authoring refers to the style of programming or developing content from within the targetexecutable environment. Immersive authoring is important for fields such as augmented reality (AR) in which interaction usability and user perception of the target content must be checked first hand, in situ. In addition, the interaction efficiency and usability of the authoring tools itself is equally important forease of authoring. In this paper, we propose design principles and describe an implementation of animmersive authoring system for AR. More importantly, we present a formal user study demonstrating its benefits and weaknesses. In particular, our results demonstrate that, compared to using the traditional 2D desktop development method, immersive authoring gained significant efficiency in specifying spatial arrangements and behavior tasks, a major component of AR content authoring. However, it was not so successful for abstract tasks such as logical programming. Based on this result, we suggest that a comprehensive AR authoring tool should include such immersive authoring functionality to help, particularly non-technical media artists, create effective contents based on the characteristics of the underlying media and interaction style.     

Crowdsourcing authoring of sensory effects on videos

Human perception is inherently multi-sensorial involving five traditional senses: sight, hearing, touch, taste, and smell. In contrast to traditional multimedia, based on audio and visual stimuli, mulsemedia seek to stimulate all the human senses. One way to produce multi-sensorial content is authoring videos with sensory effects. These effects are represented as metadata attached to the video content, which are processed and rendered through physical devices into the user’s environment. However, creating sensory effects metadata is not a trivial activity because authors have to identify carefully different details in a scene such as the exact point where each effect starts, finishes, and also its presentation features such as intensity, direction, etc. It is a subjective task that requires accurate human perception and time. In this article, we aim at finding out whether a crowdsourcing approach is suitable for authoring coherent sensory effects associated with video content. Our belief is that the combination of a collective common sense to indicate time intervals of sensory effects with an expert fine-tuning is a viable way to generate sensory effects from the point of view of users. To carry out the experiment, we selected three videos from a public mulsemedia dataset, sent them to the crowd through a cascading microtask approach. The results showed that the crowd can indicate intervals in which users agree that there should be insertions of sensory effects, revealing a way of sharing authoring between the author and the crowd.

     

DanVideo: an MPEG-7 authoring and retrieval system for dance videos

A well-annotated dance media is an essential part of a nation’s identity, transcending cultural and language barriers. Many dance video archives suffer from problems concerning authoring and access, because of the complex spatio-temporal relationships that exist between the dancers in terms of movements of their body parts and the emotions expressed by them in a dance. This paper presents a system named DanVideo for semi-automatic authoring and access to dance archives. DanVideo provides methods of annotation and authoring and retrieval tools for choreographers, dancers, and students. We demonstrate how dance media can be semantically annotated and how this information can be used for the retrieval of the dance video semantics. In particular, DanVideo offers an MPEG-7 based semi-automatic authoring tool that takes dance video annotations generated by dance experts and produces MPEG-7 metadata. DanVideo also has a search engine that takes users’ queries and retrieves dance semantics from metadata arranged using tree-embedding technique and based on spatial, temporal and spatio-temporal features of dancers. The search engine also leverages a domain-specific ontology to process knowledge-based queries. We have assessed the dance-video queries and semantic annotations in terms of precision, recall, and fidelity.     

Towards an easy to use authoring tool for interactive non-linear video

With growing bandwidths in the Internet and seemingly unlimited storage capacities on web servers, media became more and more important in the daily use of the World Wide Web. While about ten years ago only text and images with small file sizes (and as a result small resolutions) could be used, it is possible to watch high quality multimedia presentations nowadays. But those rarely exist because of tedious to learn authoring tools. A specialization to one main medium, in our case video, allows creating efficient authoring tools using well known paradigms. This work introduces an authoring tool called SIVA Producer. An iterative process for improving the usability of the authoring tool is described. Furthermore, a distinction of the terms “interactive video”, “annotated video”, “non-linear video” and “hypervideo” is given.     

Boosting web video categorization with contextual information from social web

Web video categorization is a fundamental task for web video search. In this paper, we explore web video categorization from a new perspective, by integrating the model-based and data-driven approaches to boost the performance. The boosting comes from two aspects: one is the performance improvement for text classifiers through query expansion from related videos and user videos. The model-based classifiers are built based on the text features extracted from title and tags. Related videos and user videos act as external resources for compensating the shortcoming of the limited and noisy text features. Query expansion is adopted to reinforce the classification performance of text features through related videos and user videos. The other improvement is derived from the integration of model-based classification and data-driven majority voting from related videos and user videos. From the data-driven viewpoint, related videos and user videos are treated as sources for majority voting from the perspective of video relevance and user interest, respectively. Semantic meaning from text, video relevance from related videos, and user interest induced from user videos, are combined to robustly determine the video category. Their combination from semantics, relevance and interest further improves the performance of web video categorization. Experiments on YouTube videos demonstrate the significant improvement of the proposed approach compared to the traditional text based classifiers.     

Drawing with constraints

The success of constraint-based approaches to drawing has been limited by difficulty in creating constraints, solving them, and presenting them to users. In this paper, we discuss techniques used in theBriar drawing program to address all of these issues. Briar's approach separates the problem of initially establishing constraints from the problem of maintaining them during subsequent editing. We describe how non-constraint-based drawing tools can be augmented to specify constraints in addition to positions. These constraints are then maintained as the user drags the model, which allows the user to explore configurations consistent with the constraints. Visual methods are provided for displaying and editing the constraints.     

An ODA-like Multimedia Document System

A multimedia document is composed of different media objects. ISO's Open Document Architecture (ODA) proposes a standard multimedia document model. However, the current ODA profile only includes static media, e.g. text, geometric graphics and images. Because the future multimedia documents not only include static media but also continuous media, e.g. video and audio, continuous media document parts should be added to have a complete multimedia document model. In this paper, we propose a multimedia document model, which is derived from ODA's concept. The proposed model is based on the object-oriented approach. Objects in the proposed document model are divided into two types: data objects and pseudo objects. Data objects are data structures of a document; pseudo objects are used to manage data objects. Based on the proposed model, a multimedia document authoring and presenting system (MMDS) is also developed on SUN SPARC workstations using the Solaris 2.X operating system     

Zodiac: A history-based interactive video authoring system

Easy-to-use audio/video authoring tools play a crucial role in moving multimedia software from research curiosity to mainstream applications. However, research in multimedia authoring systems has rarely been documented in the literature. This paper describes the design and implementation of an interactive video authoring system called Zodiac, which employs an innovative edit history abstraction to support several unique editing features not found in existing commercial and research video editing systems. Zodiac provides users a conceptually clean and semantically powerful branching history model of edit operations to organize the authoring process, and to navigate among versions of authored documents. In addition, by analyzing the edit history, Zodiac is able to reliably detect a composed video stream's shot and scene boundaries, which facilitates interactive video browsing. Zodiac also features a video object annotation capability that allows users to associate annotations to moving objects in a video sequence. The annotations themselves could be text, image, audio, or video. Zodiac is built on top of MMFS, a file system specifically designed for interactive multimedia development environments, and implements an internal buffer manager that supports transparent lossless compression/decompression. Shot/scene detection, video object annotation, and buffer management all exploit the edit history information for performance optimization.     

Automatically Determining Semantics for World Wide Web Multimedia Information Retrieval

Search engines are useful because they allow the user to find information of interest from the World Wide Web (WWW). However, most of the popular search engines today are textual; they do not allow the user to find images from the web. For effective retrieval, determining the semantics of the images is essential. In this paper, we describe the problems in determining the semantics of images on the WWW and the approach of AMORE, a WWW search engine that we have developed. AMORE's techniques can be extended to other media like audio and video. We explain how we assign keywords to the images based on HTML pages and the method to determine similar images based on the assigned text. We also discuss some statistics showing the effectiveness of our technique. Finally, we present the visual interface of AMORE with the help of several retrieval scenarios.     

Creating a clickable TV program by sketching and tracking freeform triggers

Viewers watching Interactive TV can click on video objects on the screen, connecting them to an advertisement or more detailed content. But this requires an interactive authoring process to attach triggers to clickable objects, which is time-consuming because these objects can appear intermittently, move, and change shape. We facilitate authoring by combining a sketching interface with shot-based object tracking and trigger verification. Experiments involving professional video authors and non-specialist viewers suggest that this approach is an improvement on previous methods.     

Adaptive Multimedia Presentation Strategies

Multimedia presentations comprise various media objects such as text, audio, image, and video that are delivered to users according to certain temporal relationships. In stored multimedia presentations, these temporal relationships are explicitly formulated by the author(s) and stored along with the presentations. However, it is difficult to ensure that these temporal relationships are always strictly preserved in real-time, distributed multimedia presentations. This is due to the fact that various components of the run-time environment such as operating system and network may offer only best effort services, i.e., they may not be able to provide any real-time guarantees. In this paper, we survey the different approaches that can be used for adapting multimedia presentations to handle instances where temporal relationships cannot be preserved in a strict manner. We classify these approaches into three categories and discuss when these categories of adaptations can be used.     

Canonical processes for creating personalized semantically rich multimedia presentations

Authoring of multimedia content can be considered as composing media assets such as images, videos, text, and audio in time, space, and interaction into a coherent multimedia presentation. Personalization of such content means that it reflects the users’ or user groups’ profile information and context information. Enriching the multimedia content with semantically rich metadata allows for a better search and retrieval of the content. To actually create personalized semantically-rich multimedia content, a manual authoring of the many different documents for all the different users’ and user groups’ needs is not feasible. Rather a (semi-)automatic authoring of the content seems reasonable. We have analyzed in detail today’s approaches and systems for authoring, personalizing, and semantically enriching multimedia presentations. Based on this analysis, we derived a general creation chain for the (semi-)automatic generation of such content. In this paper, we introduce this creation chain. We present our software engineering support for the chain, the component framework SemanticMM4U. The canonical processes supported by the creation chain and SemanticMM4U framework are described in detail. We also provide an explicit mapping of SemanticMM4U framework components to the processes and argue for the benefits of defining canonical processes for creating personalized semantically rich multimedia presentations.     

A system for parallel media processing

We describe a parallel computer system for processing media: audio,video, and graphics, among others. The system supports medium to coarse grain parallelism, using a dataflow model of execution, on a range of machine architectures scaling from a single von Neumann or general purpose processor (GPP) up to networks of several hundred heterogeneous processors. A distributed resource manager, extending or subsuming the functionality of a traditional operating system, is an integral and necessary part of the system. While we are building a system for processing a variety of media, in this paper we concentrate on video because it provides an extreme case in terms of both data rates and available parallelism.     

DSOL: a declarative approach to self-adaptive service orchestrations

Service oriented computing (SOC) has brought a simplification in the way distributed applications can be built. Mainstream approaches, however, failed to support dynamic, self-managed compositions that would empower even non-technical users to build their own orchestrations. Indeed, because of the changeable world in which they are embedded, service compositions must be able to adapt to changes that may happen at run-time. Unfortunately, mainstream SOC languages, like BPEL and BPMN, make it quite hard to develop such kind of self-adapting orchestrations. We claim that this is mostly due to the imperative programming paradigm they are based on. To overcome this limitation we propose a radically different, strongly declarative approach to model service orchestration, which is easier to use and results in more flexible and self-adapting orchestrations. An ad-hoc engine, leveraging well-known planning techniques, interprets such models to support dynamic service orchestration at run-time.     

Translation and Run-Time Validation of Loop Transformations

This paper presents new approaches to the validation of loop optimizations that compilers use to obtain the highest performance from modern architectures. Rather than verify the compiler, the approach of translation validationperforms a validation check after every run of the compiler, producing a formal proof that the produced target code is a correct implementation of the source code. As part of an active and ongoing research project on translation validation, we have previously described approaches for validating optimizations that preserve the loop structure of the code and have presented a simulation-based general technique for validating such optimizations. In this paper, for more aggressive optimizations that alter the loop structure of the code—such as distribution, fusion, tiling, and interchange—we present a set of permutation ruleswhich establish that the transformed code satisfies all the implied data dependencies necessary for the validity of the considered transformation. We describe the extensions to our tool voc-64 which are required to validate these structure-modifying optimizations. This paper also discusses preliminary work on run-time validation of speculative loop optimizations. This involves using run-time tests to ensure the correctness of loop optimizations whose correctness cannot be guaranteed at compile time. Unlike compiler validation, run-time validation must not only determine when an optimization has generated incorrect code, but also recover from the optimization without aborting the program or producing an incorrect result. This technique has been applied to several loop optimizations, including loop interchange and loop tiling, and appears to be quite promising. This research was supported in part by NSF grant CCR-0098299, ONR grant N00014-99-1-0131, and the John von Neumann Minerva Center for Verification of Reactive Systems.     

Translation and Run-Time Validation of Optimized Code

The paper presents approaches to the validation of optimizing compilers. The emphasis is on aggressive and architecture-targeted optimizations which try to obtain the highest performance from modern architectures, in particular EPIC-like micro-processors. Rather than verify the compiler, the approach of translation validation performs a validation check after every run of the compiler, producing a formal proof that the produced target code is a correct implementation of the source code.First we survey the standard approach to validation of optimizations which preserve the loop structure of the code (though they may move code in and out of loops and radically modify individual statements), present a simulation-based general technique for validating such optimizations, and describe a tool, VOC-64, which implements these technique. For more aggressive optimizations which, typically, alter the loop structure of the code, such as loop distribution and fusion, loop tiling, and loop interchanges, we present a set of permutation rules which establish that the transformed code satisfies all the implied data dependencies necessary for the validity of the considered transformation. We describe the necessary extensions to the VOC-64 in order to validate these structure-modifying optimizations.Finally, the paper discusses preliminary work on run-time validation of speculative loop optimizations, that involves using run-time tests to ensure the correctness of loop optimizations which neither the compiler nor compiler-validation techniques can guarantee the correctness of. Unlike compiler validation, run-time validation has not only the task of determining when an optimization has generated incorrect code, but also has the task of recovering from the optimization without aborting the program or producing an incorrect result. This technique has been applied to several loop optimizations, including loop interchange, loop tiling, and software pipelining and appears to be quite promising.     

Comparison of transform coding techniques for two-dimensional arbitrarily shaped images

Envisioned advanced multimedia video services include arbitrarily shaped (AS) image segments as well as regular rectangular images. Image segments of the TV weather report produced by the chromo-key technique [1] and image segments produced by video analysis and image segmentation [2–4] are typical examples of AS image segments. This paper explores efficient intraframe transform coding techniques for general two-dimensional (2D) AS image segments, treating the traditional rectangular images as a special case. In particular, we focus on the transform coding of the partially defined image blocks along the boundary of the AS image segments. We recognize two different approaches — thebrute force transform coding approach and theshape-adaptive transform coding approach. The former fills the uncovered area with the optimal redundant data such that the resulting transform spectrum is compact. A simple but efficient mirror image extension technique is proposed. Once augmented into full image blocks, these boundary blocks can be processed by traditional block-based transform techniques like the popular discrete cosine transform (DCT). In the second approach, we change either the transform basis or the coefficient calculation process adaptively based on the shape of the AS image segment. We propose an efficientshape-projected problem formulation to reduce the dimension of the problem. Existing coding algorithms, such as the orthogonal transform by Gilge [5] and the iterative coding by Kaup and Aach [6], can be interpreted intuitively. We also propose a new adaptive transform based on the same principle as that used in deriving the DCT from the optimal Karhunen-Loeve transform (KLT). We analyze the tradeoff relationship between compression performance, computational complexity, and codec complexity for different coding schemes. Simulation results show that complicated algorithms (e.g., iterative, adaptive) can improve the quality by 5–10 dB at some computational or hardware cost. Alternatively, the simple mirror image extension technique improves the quality by 3–4 dB without any overheads. The contributions of this paper lie in efficient problem formulations, new transform coding techniques, and numerical tradeoff analyses.     

Supporting interactive presentation for distributed multimedia applications

In this paper we present IMP, a platform to support the authoring and run-time management of interactive multimedia applications in distributed environments. The multi-level platform model encourages slot-in extensibility to cater for the evolving environment. We illustrate with our prototype platform built above the Cambridge multimedia environment. The authoring model emphasises reusability, tailorability and use of environment features in a uniform way. We have shown the validity of our approach by constructing a variety of application types.