Annotation based personalized adaptation and presentation of videos for mobile applications

Personalized multimedia content which suits user preferences and the usage environment, and as a result improves the user experience, gains more importance. In this paper, we describe an architecture for personalized video adaptation and presentation for mobile applications which is guided by automatically generated annotations. By including this annotation information, more intelligent adaptation techniques can be realized which primarily reduce the quality of unimportant regions in case a bit rate reduction is necessary. Furthermore, a presentation layer is added to enable advanced multimedia viewers to adequately present the interesting parts of a video in case the user wants to zoom in. This architecture is the result of collaborative research done in the EU FP6 IST INTERMEDIA project.     

Usage derived recommendations for a video digital library

We describe a minimalist methodology to develop usage-based recommender systems for multimedia digital libraries. A prototype recommender system based on this strategy was implemented for the Open Video Project, a digital library of videos that are freely available for download. Sequential patterns of video retrievals are extracted from the project's web download logs and analyzed to generate a network of video relationships. A spreading activation algorithm locates video recommendations by searching for associative paths connecting query-related videos. We evaluate the performance of the resulting system relative to an item-based collaborative filtering technique operating on user profiles extracted from the same log data.     

Intelligent multimedia data: data + indices + inference

In this paper, we present a novel approach for multimedia data indexing and retrieval that is machine independent and highly flexible for sharing multimedia data across applications. Traditional multimedia data indexing and retrieval problems have been attacked using the central data server as the main focus, and most of the indexing and query-processing for retrieval are highly application dependent. This precludes the use of created indices and query processing mechanisms for multimedia data which, in general, have a wide variety of uses across applications. The approach proposed in this paper addresses three issues: 1. multimedia data indexing; 2. inference or query processing; and 3. combining indices and inference or query mechanism with the data to facilitate machine independence in retrieval and query processing. We emphasize the third issue, as typically multimedia data are huge in size and requires intra-data indexing. We describe how the proposed approach addresses various problems faced by the application developers in indexing and retrieval of multimedia data. Finally, we present two applications developed based on the proposed approach: video indexing; and video content authorization for presentation.     

Extracting semantics from audio-visual content: the final frontier in multimedia retrieval

Multimedia understanding is a fast emerging interdisciplinary research area. There is tremendous potential for effective use of multimedia content through intelligent analysis. Diverse application areas are increasingly relying on multimedia understanding systems. Advances in multimedia understanding are related directly to advances in signal processing, computer vision, pattern recognition, multimedia databases, and smart sensors. We review the state-of-the-art techniques in multimedia retrieval. In particular, we discuss how multimedia retrieval can be viewed as a pattern recognition problem. We discuss how reliance on powerful pattern recognition and machine learning techniques is increasing in the field of multimedia retrieval. We review the state-of-the-art multimedia understanding systems with particular emphasis on a system for semantic video indexing centered around multijects and multinets. We discuss how semantic retrieval is centered around concepts and context and the various mechanisms for modeling concepts and context.     

A probabilistic framework for semantic video indexing, filtering,and retrieval

Semantic filtering and retrieval of multimedia content is crucial for efficient use of the multimedia data repositories. Video query by semantic keywords is one of the most difficult problems in multimedia data retrieval. The difficulty lies in the mapping between low-level video representation and high-level semantics. We therefore formulate the multimedia content access problem as a multimedia pattern recognition problem. We propose a probabilistic framework for semantic video indexing, which call support filtering and retrieval and facilitate efficient content-based access. To map low-level features to high-level semantics we propose probabilistic multimedia objects (multijects). Examples of multijects in movies include explosion, mountain, beach, outdoor, music etc. Semantic concepts in videos interact and to model this interaction explicitly, we propose a network of multijects (multinet). Using probabilistic models for six site multijects, rocks, sky, snow, water-body forestry/greenery and outdoor and using a Bayesian belief network as the multinet we demonstrate the application of this framework to semantic indexing. We demonstrate how detection performance can be significantly improved using the multinet to take interconceptual relationships into account. We also show how the multinet can fuse heterogeneous features to support detection based on inference and reasoning     

A multimedia system to produce and deliver video fragments on demand on parliamentary websites

Parliamentary websites have become one of the most important windows for citizens and media to follow the activities of their legislatures and to hold parliaments to account. Therefore, most parliamentary institutions aim to provide new multimedia solutions capable of displaying video fragments on demand on plenary activities. This paper presents a multimedia system for parliamentary institutions to produce video fragments on demand through a website with linked information and public feedback that helps to explain the content shown in these fragments. A prototype implementation has been developed for the Canary Islands Parliament (Spain) and shows how traditional parliamentary streaming systems can be enhanced by the use of semantics and computer vision for video analytics. The semantic web technologies used make search capabilities on parliamentary websites available to users to retrieve video fragments on demand with accurate and timely information. In addition, video analytic techniques enable the automation of identifying representative keyframes to be annotated by parliamentary experts. As a result, parliaments are able to enhance citizens’ access to information and ensure that these institutions are more open and accountable on their websites and; at the same time, the labor-intensive tasks of parliamentary experts are considerably reduced.     

Future multimedia user interfaces

No matter how powerful or elegant the technologies underlying multimedia computing, it is the user interface that ultimately determines how these systems will be used. We provide an overview of two emerging areas of user-interface research that will play major roles in future multimedia systems: virtual environments and ubiquitous computing. We discuss what they are and why they are important, and describe their relationship to current multimedia systems. While mouse and window-based user interfaces provide access to 2D graphics, video, and sound, virtual environments not only involve interactive 3D graphics, but further address all our senses through the use of spatial audio, haptics, and other modalities. While it is still a research goal to provide high-throughput video on demand, ubiquitous computing will require that multimedia network infrastructure also support extremely low-latency interaction among large numbers of mobile wireless users. To arrive at an understanding of the needs, capabilities, and potential of these new paradigms, we introduce their terminology and technology, and explain the basic system architectures being explored, leading up to a discussion of key current research issues and future directions.     

Development and evaluation of multimedia whiteboard system for improving mathematical problem solving

This study developed a web-based multimedia whiteboard system to help students learning with mathematical problem solving. The purpose is to promote a new online mathematical learning model that students not only use electronic whiteboard to write down their mathematical problem solving solutions but also use voice recording tool to give oral explanations about their thinking behind the solutions. To cultivate students’ critical thinking capability and encourage collaborative peer learning, the new learning model also requests students to criticize others’ solutions and reply to others’ arguments. With the multimedia supporting tools, students can communicate easily with each other about what they think and how they solve mathematical problems. We have conducted an experiment with sixth grade primary school students for evaluation. After the experiment, a questionnaire about students’ attitude toward the multimedia whiteboard system for math learning was then held. The results show that students were satisfied with the use of the multimedia whiteboard system for helping them with learning fractional division. Most students were interested in studying mathematics with the multimedia whiteboard system and thought this tool is particularly useful for doing collaborative learning. After analyzing the recorded solving processes and discussions content of students, we found that the performance of female students was superior to male students in communications and mathematical problem solving. Additionally, students with higher final exam grades had better mathematical abilities for doing critiques, arguments and communications.     

A data model and a query languagefor multimedia documents databases

A multimedia application involves information that may be in a form of video, images, audio, text and graphics, need to be stored, retrieved and manipulated in large databases. In this paper, we propose an object-oriented database schema that supports multimedia documents and their temporal, spatial and logical structures. We present a document example and show how the schema can adress all the structures described. We also present a multimedia query specification language that can be used to describe a multimedia content portion to be retrieved from the database. The language provides means by which the user can specify the information on the media as well as the temoral and spatial relationships among these media.     

Vaudeville: A High Performance,Voice Activated Teleconferencing Application

We present Vaudeville, a voice-activated, hands-free, ATM-based video conferencing application. This system is scalable; although video bandwidth is normally a limiting factor in the number of conferences participants, the bandwidth attributed to the video is not a function of conference size. This is achieved through an automatic, distributed floor control mechanism that gives the appearance of an open floor. Audio and video are encoded in hardware using a platform-independent, ATM hardware multimedia interface. Vaudeville features digitally transmitted NTSC video, voice-activated audio transmission, audio bridging of two audio streams, and voice-activated video switching. Multiple simultaneous multiparty conferences are supported. Users can move freely among conferences without knowledge of the underlying network structure. We describe how Vaudeville was built using a component-based distributed programming environment. We also describe the algorithms used to control the audio and video of the application.     

Collaborative knowledge building with shared video representations

Online video has become established as a fundamental part of the fabric of the web; widely used by people for information sharing, learning and entertainment. We report results from a design study that explored how people interact to create shared multi-path video representations in a social video environment. The participants created multiple versions of a video by providing alternative and interchangeable scenes that formed different paths through the video content. This multi-path video approach was designed to circumvent limitations of traditionally linear video for use as a shared representation in collaborative knowledge building activities. The article describes how people created video resources in collaborative activities in two different settings. We discuss different modes of working that were observed and outline the specific challenges of using the video medium as shared representation. Finally we demonstrate how an analysis of collaborative dimensions of the shared multi-path video representation can be applied to discuss the design space and to raise the discourse about the usefulness of these representations in knowledge building environments.     

Trial Realization of Human-Centered Multimedia Navigation for Video Retrieval

A trial realization of human-centered navigation for video retrieval is presented in this article. This system consists of the following functions: (a) multimodal analysis for collaborative use of multimedia data, (b) preference extraction for the system to adapt to users' individual demands, and (c) adaptive visualization for users to be guided to their desired contents. By using these functions, users can find their desired video contents more quickly and accurately than with the conventional retrieval schemes since our system can provide new pathways to the desired contents. Experimental results verify the effectiveness of the proposed system.     

Optimum adaptation trajectories for streamed multimedia

Most adaptive delivery mechanisms for streaming multimedia content do not explicitly consider user-perceived quality when making adaptations. We propose that an optimal adaptation trajectory through the set of possible encodings exists and that it indicates how to adapt encoding quality in response to changes in network conditions to maximize user-perceived quality. Such an optimum adaptation trajectory can be used with any transmission adaptation policy. We describe the subjective tests we carried out to find such trajectories for a number of different MPEG-4 video clips and indicate how this knowledge could be used in the operation of a practical system.     

The conceptual framing,design and evaluation of device ecologies for collaborative activities

A variety of computing technologies, in addition to the personal computer, are now commonly used in many settings. As networking infrastructures mature, it is increasingly feasible and affordable to consider closer integration and use of these heterogeneous devices in tandem. However, little is known about how best to design or evaluate such ‘device ecologies’; in particular, how best to combine devices to achieve a desired type of collaborative user experience. A central concern is how users switch their attention between devices, to utilize the various elements to best effect. We describe here the development of an ecology of devices for groups of students to use when engaged in collaborative inquiry-learning activities. This included a multi-touch tabletop, laptops, projections, video streams and telephone. In situ studies of students and tutors using it in three different settings showed how individuals and groups switched their foci between the multiple devices. We present our findings, using a novel method for analysing users’ transitions between foci, identifying patterns and emergent characteristics. We then discuss the importance of designing for transitions that enable groups to appropriately utilise an ecology of devices, using the concepts of seams, bridges, niches and focal character.     

基于内容的多媒体数据库系统引擎CDB

CDB(Content-based DataBase)是一种基于内容的多媒体数据库引擎，可以嵌入到通用的对象一关系数据库中，使数据库系统综合支持对多媒体数据的常规和基于内容的壹询．本文首先阐述CDB的体系结构，它把信息检索和数据检索结合到数据库中，支持多媒体数据库的基于内容的建立、操纵和维护；然后给出其层次型内容模型，描述多媒体内容的时空结构特征以及信息线索；最后描述用于CDB的基于内容信息检索技术及其设计和实现的用户壹询和操纵接口，包括示例壹询、主观颜色壹询、视频概要和浏览、扩展SQL内容壹询等．     

Computational mereology: A study of part-of relations for multimedia indexing

How to index or retrieve multimedia objects is by no means obvious, because the computer can retrieve right multimedia material only if it reasons about its contents. We show that it is possible to write formal specifications of this reasoning process using set theory and mereology. We discuss the theoretical consequences of trying to use mereology and set theory for multimedia indexing and retrieval. We re-examine the roles of mereology and set theory in knowledge representation. We conclude that both commonsense set theories and mereologies should play the role of constraining databases of arbitrary multimedia objects, e.g. video clips. But although both should be viewed as database constraints, we argue that part-of hierarchies should be used to encode relatively permanent background knowledge, elsewhere names thereferential level, while member-of hierarchies should describe arbitrary multimedia records. We also propose a language and a set of axioms, SetNM, for natural mereologies with sets. A multimedia indexing system can then be viewed as a particular SetNM theory.     

Learning Personal Preference From Viewer's Operations for Browsing and Its Application to Baseball Video Retrieval and Summarization

Personalization is one of the most important mechanisms to make multimedia systems easy to use. In video applications, its embodiment is to tailor video contents for a particular viewer. For this purpose, we are now developing a system of retrieving and browsing video segments, called video portal with personalization (VIPP). VIPP is characterized by 1) supporting the viewer's access to video contents and making a summarized video clip by taking his/her preference into account and 2) acquiring the viewer's profile from his/her operations automatically. In this paper, we propose a method for learning to personalize from the viewer's operations such as retrieval and browsing, as well as describe how the personalized retrieval and summarization of videos can be realized. From the experiments, we clarify the effect of personalization on retrieval and summarization of baseball videos on VIPP.