Olfaction-enhanced multimedia: perspectives and challenges

Olfaction—or smell—is one of the last challenges which multimedia and multimodal applications have to conquer. Enhancing such applications with olfactory stimuli has the potential to create a more complex—and richer—user multimedia experience, by heightening the sense of reality and diversifying user interaction modalities. Nonetheless, olfaction-enhanced multimedia still remains a challenging research area. More recently, however, there have been initial signs of olfactory-enhanced applications in multimedia, with olfaction being used towards a variety of goals, including notification alerts, enhancing the sense of reality in immersive applications, and branding, to name but a few. However, as the goal of a multimedia application is to inform and/or entertain users, achieving quality olfaction-enhanced multimedia applications from the users’ perspective is vital to the success and continuity of these applications. Accordingly, in this paper we have focused on investigating the user perceived experience of olfaction-enhanced multimedia applications, with the aim of discovering the quality evaluation factors that are important from a user’s perspective of these applications, and consequently ensure the continued advancement and success of olfaction-enhanced multimedia applications.     

Striped-texture image segmentation with application to multimedia security

In today’s rapid growth of volume of multimedia data, security is important yet challenging problem in multimedia applications. Image, which covers the highest percentage of the multimedia data, it is very important for multimedia security. Image segmentation is utilized as a fundamental preprocessing of various multimedia applications such as surveillance for security by breaking a given image into multiple salient regions. In this paper, we present a new image segmentation approach based on frequency-domain filtering for images with stripe texture, and generalize it to lattice fence images. Our method significantly reduces the impact of stripes on segmentation performance. The approach proposed in this paper consists of three phases. Given the images, we weaken the effect of stripe texture by filtering in the frequency domain automatically. Then, structure-preserving image smoothing is employed to remove texture details and extract the main image structures. Last, we use an effective threshold method to produce segmentation results. Our method achieves very promising results for the test image dataset and could benefit a number of new multimedia applications such as public security.

     

Intelligent multimedia data: data + indices + inference

In this paper, we present a novel approach for multimedia data indexing and retrieval that is machine independent and highly flexible for sharing multimedia data across applications. Traditional multimedia data indexing and retrieval problems have been attacked using the central data server as the main focus, and most of the indexing and query-processing for retrieval are highly application dependent. This precludes the use of created indices and query processing mechanisms for multimedia data which, in general, have a wide variety of uses across applications. The approach proposed in this paper addresses three issues: 1. multimedia data indexing; 2. inference or query processing; and 3. combining indices and inference or query mechanism with the data to facilitate machine independence in retrieval and query processing. We emphasize the third issue, as typically multimedia data are huge in size and requires intra-data indexing. We describe how the proposed approach addresses various problems faced by the application developers in indexing and retrieval of multimedia data. Finally, we present two applications developed based on the proposed approach: video indexing; and video content authorization for presentation.     

Canonical processes for creating personalized semantically rich multimedia presentations

Authoring of multimedia content can be considered as composing media assets such as images, videos, text, and audio in time, space, and interaction into a coherent multimedia presentation. Personalization of such content means that it reflects the users’ or user groups’ profile information and context information. Enriching the multimedia content with semantically rich metadata allows for a better search and retrieval of the content. To actually create personalized semantically-rich multimedia content, a manual authoring of the many different documents for all the different users’ and user groups’ needs is not feasible. Rather a (semi-)automatic authoring of the content seems reasonable. We have analyzed in detail today’s approaches and systems for authoring, personalizing, and semantically enriching multimedia presentations. Based on this analysis, we derived a general creation chain for the (semi-)automatic generation of such content. In this paper, we introduce this creation chain. We present our software engineering support for the chain, the component framework SemanticMM4U. The canonical processes supported by the creation chain and SemanticMM4U framework are described in detail. We also provide an explicit mapping of SemanticMM4U framework components to the processes and argue for the benefits of defining canonical processes for creating personalized semantically rich multimedia presentations.     

ZYX-a multimedia document model for reuse and adaptationof multimedia content

Advanced multimedia applications require adequate support for the modeling of multimedia content by multimedia document models. More and more this support calls for not only the adequate modeling of the temporal and spatial course of a multimedia presentation and its interactions, but also for the partial reuse of multimedia documents and adaptation to a given user context. However, our thorough investigation of existing standards for multimedia document models such as HTML, MHEG, SMIL, and HyTime leads to us the conclusion that these standard models do not provide sufficient modeling support for reuse and adaptation. Therefore, we propose a new approach for the modeling of adaptable and reusable multimedia content, the Z_YX model. The model offers primitives that provide-beyond the more or less common primitives for temporal, spatial, and interaction modeling-a variform support for reuse of structure and layout of document fragments and for the adaptation of the content and its presentation to the user context. We present the model in detail and illustrate the application and effectiveness of these concepts by samples taken from our Cardio-OP application in the domain of cardiac surgery. With the Z_YX model, we developed a comprehensive means for advanced multimedia content creation: support for template-driven authoring of multimedia content and support for flexible, dynamic composition of multimedia documents customized to the user's local context and needs. The approach significantly impacts and supports the authoring process in terms of methodology and economic aspects     

Grammar-based articulation for multimedia document design

This paper describes an approach to the problem of articulating multimedia information based on parsing and syntax-directed translation that uses Relational Grammars. This translation is followed by a constraint-solving mechanism to create the final layout. Grammatical rules provide the mechanism for mapping from a representation of the content and context of a presentation to forms that specify the media objects to be realized. These realization forms include sets of spatial and temporal constraints between elements of the presentation. Individual grammars encapsulate the “look and feel” of a presentation and can be used as generators of such a style. By making the grammars sensitive to the requirements of the output medium, parsing can introduce flexibility into the information realization process.     

CGMP: cloud-assisted green multimedia processing

With continued advancements of mobile computing and communications, emerging novel multimedia services and applications have attracted lots of attention and been developed for mobile users, such as mobile social network, mobile cloud medical treatment, mobile cloud game. However, because of limited resources on mobile terminals, it is of great challenge to improve the energy efficiency of multimedia services. In this paper, we propose a cloud-assisted green multimedia processing architecture (CGMP) based on mobile cloud computing. Specifically, the tasks of multimedia processing with energy-extensive consumption can be offloaded to the cloud, and the face recognition algorithm with improved principal component analysis and nearest neighbor classifier is realized on CGMP based cloud platform. Experimental results show that the proposed scheme can effectively save the energy consumption of the equipment.     

TrendsSummary: a platform for retrieving and summarizing trendy multimedia contents

With the flood and popularity of various multimedia contents on the Internet, searching for appropriate contents and representing them effectively has become an essential part for user satisfaction. So far, many contents recommendation systems have been proposed for this purpose. A popular approach is to select hot or popular contents for recommendation using some popularity metric. Recently, various social network services (SNSs) such as Facebook and Twitter have become a widespread social phenomenon owing to the smartphone boom. Considering the popularity and user participation, SNS can be a good source for finding social interests or trends. In this study, we propose a platform called TrendsSummary for retrieving trendy multimedia contents and summarizing them. To identify trendy multimedia contents, we select candidate keywords from raw data collected from Twitter using a syntactic feature-based filtering method. Then, we merge various keyword variants based on several heuristics. Next, we select trend keywords and their related keywords from the merged candidate keywords based on term frequency and expand them semantically by referencing portal sites such as Wikipedia and Google. Based on the expanded trend keywords, we collect four types of relevant multimedia contents—TV programs, videos, news articles, and images—from various websites. The most appropriate media type for the trend keywords is determined based on a naïve Bayes classifier. After classification, appropriate contents are selected from among the contents of the selected media type. Finally, both trend keywords and their related multimedia contents are displayed for effective browsing. We implemented a prototype system and experimentally demonstrated that our scheme provides satisfactory results.     

CyVOD: a novel trinity multimedia social network scheme

How to comprehensively explore, improve and deploy multimedia social networks (MSNs) has become a hot topic in the era of emerging pervasive mobile multimedia. More and more MSNs offer a great number of convenient tools, services, and applications for multimedia contents including video and audio that users share willingly and on demand. However, concerns with digital rights management (DRM)-oriented multimedia security, as well as the efficiency of multimedia usage and sharing are meanwhile intensified due to easier distribution and reproduction of multimedia content in a wide range of social networks. The paper proposes a comprehensive framework for multimedia social network, and realized a cross-platform MSN prototype system, named as CyVOD, to support two kinds of DRM modes. The proposed framework effectively protects copyrighted multimedia contents against piracy, and supports a more efficient recommendation system for its better handling of the tradeoff between multimedia security and ease of use.     

User-author centered multimedia building blocks

The advances of multimedia models and tools popularized the access and production of multimedia contents: in this new scenario, there is no longer a clear distinction between authors and end-users of a production. These user-authors often work in a collaborative way. As end-users, they collectively participate in interactive environments, consuming multimedia artifacts. In their authors’ role, instead of starting from scratch, they often reuse others’ productions, which can be decomposed, fusioned and transformed to meet their goals. Since the need for sharing and adapting productions is felt by many communities, there has been a proliferation of standards and mechanisms to exchange complex digital objects, for distinct application domains. However, these initiatives have created another level of complexity, since people have to define which share/ reuse solution they want to adopt, and may even have to resort to programming tasks. They also lack effective strategies to combine these reused artifacts. This paper presents a solution to this demand, based on a user-author centered multimedia building block model—the digital content component (DCC). DCCs upgrade the notion of digital objects to digital components, as they homogenously wrap any kind of digital content (e.g., multimedia artifacts, software) inside a single component abstraction. The model is fully supported by a software infrastructure, which exploits the model’s semantic power to automate low level technical activities, thereby freeing user-authors to concentrate on creative tasks. Model and infrastructure improve recent research initiatives to standardize the means of sharing and reuse domain specific digital contents. The paper’s contributions are illustrated using examples implemented in a DCC-based authoring tool, in real life situations.     

An open framework for a multimedia medical document system (a multimedia document system framework)

A novel and generalized open framework for multimedia information manipulation is proposed. The document system consists of functional modules for capturing, storing and presenting multimedia data. These modules work independently and communicate through aHyperDocument Interfacelayer. The modular approach has the added advantage of easy-to-maintain and the convenience for future enhancements. Hypermedia and temporal –spatial relationships between document media nodes are embedded into the data structure to support flexible presentation of multimedia information. Windows and a specially designedDocument Tree Dialogprovide an environment for efficient and user-friendly information retrieval. Other system issues such as security and customization are also addressed in the design. Based on this approach, a medical multimedia document system to handle cardiac catheterization record (CCR) was designed as a demonstration for the research idea. The proposed design can be applied and extended to cover a variety of multimedia applications.     

Spatio-temporal composition and indexing for large multimedia applications

Multimedia applications usually involve a large number of multimedia objects (texts, images, sounds, etc.). An important issue in this context is the specification of spatial and temporal relationships among these objects. In this paper we define such a model, based on a set of spatial and temporal relationships between objects participating in multimedia applications. Our work exploits existing approaches for spatial and temporal relationships. We extend these relationships in order to cover the specific requirements of multimedia applications and we integrate the results in a uniform framework for spatio-temporal composition representation. Another issue is the efficient handling of queries related to the spatio-temporal relationships among the objects during the authoring process. Such queries may be very costly and appropriate indexing schemes are needed so as to handle them efficiently. We propose efficient such schemes, based on multidimensional (spatial) data structures, for large multimedia applications that involve thousands of objects. Evaluation models of the proposed schemes are also presented, as well as hints for the selection of the most appropriate one, according to the multimedia author's requirements.     

Retrieval scheduling for collaborative multimedia presentations

The single-system approach is no longer sufficient to handle the load on popular Internet servers, especially for those offering extensive multimedia content. Such services have to be replicated to enhance their availability, performance, and reliability. In a highly replicated and available environment, server selection is an important issue. In this paper, we propose an application-layer broker (ALB) for this purpose. ALB employs a content-based, client-centric approach to negotiate with the servers and to identify the best server for the requested objects. ALB aims to maximize client buffer utilization in order to efficiently handle dynamic user interactions such as skip, reverse presentation, go back in time. We also present details of a collaborative multimedia presentation platform that we have developed based on ALB.     

A lightweight framework for authoring XML multimedia content on the web

This paper addresses the issue of authoring XML multimedia content on the web. It focuses on methods that apply to different kinds of contents, including structured documents, factual data, and multimedia objects. It argues in favor of a template-based approach that enhances the ability for multiple applications to use the produced content. This approach is illustrated by AXEL, an innovative multipurpose client-side authoring framework (previously described in Sire et al. (2010)), intended for web users with limited skills. The versatility of the tool is illustrated through a series of use cases that demonstrate the flexibility of the approach for creating various kinds of web content.     

Interactive authoring of multimedia documents in a constraint-based authoring system

As multimedia applications spread widely, it is crucial for programming and design support systems to handle “time” in multimedia documents effectively and flexibly. This paper presents a set of interactive system support tools for designing and maintaining the temporal behavior of multimedia documents. The tool set provides mechanisms for anomaly detection, temporal query processing, and interactive scheduling. It is based on a fast incremental constraint solver we have developed, which can be adapted by any constraint-based system. The incremental constraint solver provides immediate feedback to the user, supporting a highly interactive design process. Combined with existing optimal layout generation mechanisms proposed in the literature, our tools effectively utilize the flexibility provided by constraint-based systems.     

A multiresolution representation scheme for multimedia databases

We present a compression scheme that is useful for interactive video applications such as browsing a multimedia database. The focus of our approach the development of a compression scheme (and a corresponding retrieval scheme) that is optimal for any data rate. To browse a multimedia database, such a compression scheme is essential. We use a multiresolution setting, but eliminate the need for wavelets. This results in much better compression. We show experimental results and explain in detail how to extend our approach to multidimensional data.     

Multi-constraints self-adaptation for reconfigurable multimedia embedded systems

The application of multimedia in embedded systems (ES), such as Virtual reality and 3-D imaging, represents the current trend in ES development. Coupling multimedia with ES has raised new multimedia-related challenges that have been added to the common ES constraints. These challenges deal with the real-time, quality, performance and efficient processing requirements of multimedia applications. The integration of self-adaptation in ES development has been, for many years, a paramount solution to cope with these issues. Although there has been extensive research on the topic of ES self-adaptation, the related works still lack global approaches that better deal with multimedia-related constraints. Coordinating different adaptation mechanisms, monitoring multiple system constraints and supporting multi-application contexts are still underexplored. The aim of the present work is to fill in these gaps by providing a global adaptation approach that offers better adaptation decisions with fair resource sharing among competing multimedia applications. With the above challenges in mind, we propose a multi-constraints combined adaptation approach that targets multimedia ES. It addresses four critical system constraints: maximizing the overall system‘s Quality of Application (QoA) under the real-time constraint, the remaining system energy and the available network bandwidth. It coordinates the adaptation at both application and architecture levels. To test and validate the proposed technique, a videophone system is designed on a Xilinx FPGA development board. It executes two complex multimedia applications. The validation results show the aptitude of the proposed system to successfully reconfigure itself at run-time in response to its constraints.