Interactive multi-user video retrieval systems

In this paper we present two interactive multi-user systems for video search and browsing. The first is composed by web applications which allows multiuser interaction in a distributed environment; such applications are based on the Rich Internet Application paradigm, designed to obtain the levels of responsiveness and interactivity typical of a desktop application. The second system implements a multi-user collaborative application within a single location, exploiting multi-touch devices. Both systems use the same backend, based on a service oriented architecture (SOA) that provides services for automatic and manual annotation, and an ontology-based video search and browsing engine. Ontology-based browsing let users to inspect the content of video collections; user queries are expanded through ontology reasoning. User-centered field trials of the systems, conducted to assess the user experience and satisfaction, have shown that the approach followed to design these interfaces is extremely appreciated by professional archivists and people working on multimedia.     

How do interactive tabletop systems influence collaboration?

This paper examines some aspects of the usefulness of interactive tabletop systems, if and how these impact collaboration. We chose creative problem solving such as brainstorming as an application framework to test several collaborative media: the use of pen-and-paper tools, the “around-the-table” form factor, the digital tabletop interface, the attractiveness of interaction styles. Eighty subjects in total (20 groups of four members) participated in the experiments. The evaluation criteria were task performance, collaboration patterns (especially equity of contributions), and users’ subjective experience. The “around-the-table” form factor, which is hypothesized to promote social comparison, increased performance and improved collaboration through an increase of equity. Moreover, the attractiveness of the tabletop device improved subjective experience and increased motivation to engage in the task. However, designing attractiveness seems a highly challenging issue, since overly attractive interfaces may distract users from the task.     

The design,development, and deployment of a tabletop interface for collaborative exploration of genomic data

In this paper, we reflect on the design, development, and deployment of G-nome Surfer; a multi-touch tabletop user interface for collaborative exploration of genomic data. G-nome Surfer lowers the threshold for using advanced bioinformatics tools, reduces the mental workload associated with manipulating genomic information, and fosters effective collaboration. We describe our two-year-long effort from design strategy to iterations of design, development, and evaluation. This paper presents four main contributions: (1) a set of design requirements for supporting collaborative exploration in data-intensive domains, (2) the design, implementation, and validation of a multi-touch tabletop interface for collaborative exploration, (3) a methodology for evaluating the strengths and limitations of tabletop interaction for collaborative exploration, and (4) empirical evidence for the feasibility and value of integrating tabletop interaction in college-level education.     

Multi-Modal Human Interactions with an Intelligent Interface Utilizing Images, Sounds, and Force Feedback

One goal of research in the area of human–machine interaction is to improve the ways a human user interacts with a computer through a multimedia interface. This interaction comprises of not only text, graphical animation, stereo sounds, and live video images, but also force and haptic feedback, which can provide more “real” feeling to the user. The force feedback joystick, a human interface device is an input–output device. It not only tracks user's physical manipulation input, but also provides realistic physical sensations of force coordinated with system output. As part of our research, we have developed a multimedia computer game that assimilates images, sounds, and force feedback. We focused on the issues of how to combine these media to allow the user feel the compliance, damping, and vibration effects through the force feedback joystick. We conducted series of human subject experiments that incorporated different combinations of media, including the comparative study of the different performances of 60 human users, aiming to answer the question: What are the effects of force feedback (and associated time delays) when used in combination with visual and auditory information as part of a multi-modal interface? It is hoped that these results can be utilized in the design of enhanced multimedia systems that incorporate force feedback.     

A framework for search interfaces

Searching textual databases can be confusing for users. Popular search systems for the World Wide Web and stand alone systems typically provide a simple interface: users type in keywords and receive a relevance ranked list of 10 results. This is appealing in its simplicity, but users are often frustrated because search results are confusing or aspects of the search are out of their control. If we are to improve user performance, reduce mistaken assumptions, and increase successful searches, we need more predictable design. To coordinate design practice, we suggest a four-phase framework that would satisfy first time, intermittent, and frequent users accessing a variety of textual and multimedia libraries     

Exploring how children use their hands to think: an embodied interactional analysis

In order to better understand how to design hands-on child-computer interaction, we explore how different styles of interaction facilitate children's thinking while they use their hands to manipulate objects. We present an exploratory study of children solving a spatial puzzle task. We investigate how the affordances of physical, graphical and tangible interfaces may facilitate the development of thinking skills including mental visualisation, problem space exploration and collaboration. We utilise the theory of complementary actions taken from embodied cognition to develop a video coding methodology that allows us to classify behavioural activity and make inferences about thinking skills development. Our findings indicated that the combination of direct hands-on input style with audio-visual feedback facilitated by the tangible user interface enabled a dynamic task completion strategy, which supports the development of mental skills with a slight time cost. The mouse and graphical user interface supported a trial and error approach, which may limit skills development. The physical cardboard puzzle enabled effective task completion but provided less support for social interaction and problem space exploration. We conclude with design recommendations.     

Co-located collaborative visual analytics around a tabletop display

Co-located collaboration can be extremely valuable during complex visual analytics tasks. We present an exploratory study of a system designed to support collaborative visual analysis tasks on a digital tabletop display. Fifteen participant pairs employed Cambiera, a visual analytics system, to solve a problem involving 240 digital documents. Our analysis, supported by observations, system logs, questionnaires, and interview data, explores how pairs approached the problem around the table. We contribute a unique, rich understanding of how users worked together around the table and identify eight types of collaboration styles that can be used to identify how closely people work together while problem solving. We show how the closeness of teams’ collaboration and communication influenced how they performed on the task overall. We further discuss the role of the tabletop for visual analytics tasks and derive design implications for future co-located collaborative tabletop problem solving systems.     

Design and implementation of a video browsing system for the Internet

Recent advances in multimedia processing technologies, internet‐working technologies, and the World Wide Web phenomenon have resulted in a vast creation and use of digital videos. Due to this reason, an efficient technique to locate and retrieve a desired video from a remote video archive is needed. A trial‐and‐error approach popularly used in current Web search engines is not applicable for searching for a desired video segment since the technique incurs intolerable delays. In this paper, we present the design and implementation of a video browsing system that lets the user view a summary of a selected video and search within the video while being downloaded so that the user can determine the relevance of the video as early as possible. The system is inexpensive and scalable, making it suitable for large‐scale distributed systems such as the Internet. The browsing system consists of two major software components: a video server, and a video browser and player called VideoCenter implemented using Microsoft DirectShow multimedia development kit. The implementation of VideoCenter enables us to assess the ease of use of DirectShow as well as its drawbacks in developing multimedia applications. Copyright © 2001 John Wiley & Sons, Ltd.     

Semiotic analysis of multi-touch interface design: The MuTable case study

Although multi-touch applications and user interfaces have become increasingly common in the last few years, there is no agreed-upon multi-touch user interface language yet. In order to gain a deeper understanding of the design of multi-touch user interfaces, this paper presents semiotic analysis of multi-touch applications as an interesting approach to gain deeper understanding of the way users use and understand multi-touch interfaces. In a case study example, user tests of a multi-touch tabletop application platform called MuTable are analysed with the Communicability Evaluation Method to evaluate to what extent users understand the intended messages (e.g., cues about interaction and functionality) the MuTable platform communicates. The semiotic analysis of this case study shows that although multi-touch interfaces can facilitate user exploration, the lack of well-known standards in multi-touch interface design and in the use of gestures makes the user interface difficult to use and interpret. This conclusion points to the importance of the elusive balance between letting users explore multi-touch systems on their own on one hand, and guiding users, explaining how to use and interpret the user interface, on the other.     

A holistic approach for personalization,relevance feedback & recommendation in enriched multimedia content

As the prerequisites of production houses, broadcasters, advertising agencies and online publishing companies for enriched multimedia content increase rapidly, the need of innovative methods for the effective creation of enriched multimedia content is undeniable. Stemming from this need, in this paper we focus on the design, development and evaluation of a framework consisting of personalization, relevance feedback and recommendation mechanisms, as a principal method for the creation of enriched multimedia content targeted to each user’s needs, preferences and interests. As the multimedia content proliferates along with its consumption by the users, more effective ways of presenting it to the viewers are demanded in order to facilitate them with the multimedia content search and selection and improve their Quality of Experience (QoE). The main contribution of the paper is the introduction of a holistic framework that offers personalized enriched multimedia content, by extending the recommendation process to the set of enrichments that accompany the video except from the video itself and by collecting explicit and implicit relevance feedback from the interactions of the user with both the video and its enrichments. We evaluate the proposed framework following a two-step approach. Firstly, we perform extended experiments by applying reasonably simulated user interactions, in order to calibrate its parameters that refer to multiple aspects of the enriched multimedia content, aiming at high performance in terms of QoE. Here, most importantly, we have shown that appropriately designing the enrichments and considering users’ interactions with them allows for achieving a better quality in inferring users’ profiles in many realistic cases. Secondly, we integrated our proposed recommender framework within the MECANEX streaming platform in order to perform user studies about its usability within a realistic environment of use.     

Future multimedia user interfaces

No matter how powerful or elegant the technologies underlying multimedia computing, it is the user interface that ultimately determines how these systems will be used. We provide an overview of two emerging areas of user-interface research that will play major roles in future multimedia systems: virtual environments and ubiquitous computing. We discuss what they are and why they are important, and describe their relationship to current multimedia systems. While mouse and window-based user interfaces provide access to 2D graphics, video, and sound, virtual environments not only involve interactive 3D graphics, but further address all our senses through the use of spatial audio, haptics, and other modalities. While it is still a research goal to provide high-throughput video on demand, ubiquitous computing will require that multimedia network infrastructure also support extremely low-latency interaction among large numbers of mobile wireless users. To arrive at an understanding of the needs, capabilities, and potential of these new paradigms, we introduce their terminology and technology, and explain the basic system architectures being explored, leading up to a discussion of key current research issues and future directions.     

Through-Walls Communication for Medical Emergency Services

The authors present a model for bringing the coordination power of workflow management systems to outdoor wearable augmented reality (AR) systems. They portray how mobile equipment may be integrated with adaptive, context-aware work environments. A scenario of a medical emergency task is described to illustrate the functionality of this form of collaboration system. Appropriate information stickers are introduced to support data collection in medical emergency scenarios in a sophisticated form through a hands-free user interface for medical personnel. They propose the use of new user interface technology, including multimedia, AR information stickers, and the allocation of patient medical records to identified locations of the human body. A key feature is the access to relevant information for users in the mobile environment as well as for those in the advanced control room. An additional advantage is the automatic recording of on-site data, which helps to build the medical record of a patient without interfering with the work of the emergency team.     

JASMINE: A Java Tool for Multimedia Collaboration on the Internet

Although collaboration tools have existed for a long time [8], Internet-based multimedia collaboration has recently received a lot of attention mainly due to easy accessibility of the Internet by ordinary users. The Java platform and programming language has also introduced yet another level of easy access: platform-independent computing. As a result, it is very attractive to use Java to design multimedia collaboration systems for the Internet. Today there are many systems, which use Java for multimedia collaboration. However, most of these systems require the shared Java application to be re-written according to the collaboration system's Application Programming Interface (API)—a task which is sometimes difficult or even impossible. In this paper, we describe a practical approach for transparent collaboration with Java. Our approach is transparent in that the Java application can be shared as is with no modifications. The main idea behind our system is that user events occurring through the interactions with the application can be caught, distributed, and reconstructed, hence allowing Java applications to be shared transparently. Our architecture allows us to make the huge installed base of Java applications collaborative, without any modification to their original code. We also prove the feasibility of our architecture by implementation of the JASMINE¹ prototype.     

A case study of intended versus actual experience of adaptivity in a tangible storytelling system

This article presents a case study of an adaptive, tangible storytelling system called “The Reading Glove”. The research addresses a gap in the field of adaptivity for ubiquitous systems by taking a critical look at the notion of “adaptivity” and how users experience it. The Reading Glove is an interactive storytelling system featuring a wearable, glove-based interface and a set of narratively rich objects. A tabletop display provides adaptive recommendations which highlight objects to select next, functioning as an expert storytelling system. The recommendation engine can be run in three different configurations to examine the effects of different adaptive methods. The study of the design process as well as the user experience of the Reading Glove allows us to develop a deeper understanding of the experience of adaptivity that is useful for designers of intelligent systems, particularly those with ubiquitous and tangible forms of interaction.     

System design and user evaluation of Co-Star: An immersive stereoscopic system for cable harness design

The system design and results of a user evaluation of Co-Star an immersive design system for cable harness design is described. The system used a stereoscopic head mounted graphical display, user motion tracking and hand-gesture controlled interface to enable cable harnesses to be designed using direct 3D user interaction with a product model. In order to determine how such a system interface would be used by a designer and to obtain user feedback on its main features a practical user evaluation was undertaken involving ten participants each completing three cable harness design tasks with the system. All user interactions with the system were recorded in a time stamped log file during each of the tasks, which were also followed by questionnaire (5 point scale) and interview sessions with each participant. The recorded interaction data for the third task was analysed using functional decomposition techniques and used to construct a single activity profile for the task based on the mean results obtained from the participant group. The goal was to identify in general terms the relative distribution of user activity between specific purposes during practical system operation, and it was found that in this task Navigation accounted for 41%, Design 27%, System Operation 23% and looking at Task Instructions 9% of all user activity. The scored questionnaire data collected immediately after the completion of each task was used to rank the major features of the system according to user opinion. This was further enhanced by also collecting real interview comments from the user group about these same features. The combination of both quantitative performance analysis and subjective user opinion data obtained during a practical design exercise has enabled an in depth evaluation of the system, leading to a much greater understanding of many of the key user and interface requirements that should be considered during the development of immersive interfaces and systems for practical engineering applications.     

Modeling users' task performance on the mobile device: PC convergence system

This study aims to establish a model-based approach for user interface design that simultaneously considers the system's information hierarchy, users' task procedure knowledge, and system interfaces. The approach is based on a framework that contains multiple interaction models to express both system elements and users' knowledge. The framework evaluates system interface through the interaction between user's knowledge on interface, task procedure and information structure perceived by the user in the system. The interface is evaluated by its contribution to the users' task performance and system navigation.These three factors were defined as design factors that affect users' task performance. Through the crosscheck process of models, the relation between information, interface, and task procedure is calculated into combined difficulty index (CDI) that expresses the difficulty of a system interface that users would experience during the use of system. A user test was conducted for the validation of the CDI. The difficulties of the interface of a mobile healthcare system were predicted with the CDI, and the predictions were compared with the experimental results, where the users' performance showed consistence with the prediction.     

数字图书馆中基于Java的用户界面组件库研究与开发

It is one of the most important task in digital library to develop a universal and customized user interface for search and retrieval in heterogeneous,distributed environment.In this paper,we briefly introduce working principles and workflow of SiteSearch system developed by OCLC.Based on analyzing the functions of this system,we study and design composing elements and implementation mechanism of the customized interface.By analyzing and illustrating the examples,we develop Java based component library for dynamically generating the customized interface in Chinese information platform of SiteSearch system.Finally,we make some remarks on the difference of develop-ment of this system with other similar systems.     

Navigation methods of special needs users in multimedia systems

In today’s information society, computer users frequently need to seek for information on home pages as well as to select among software functions. A well-designed interface is essential in order to find everything necessary and meet the requirements of both the average user and users with special needs. Proper placement of objects on the screen is important to decrease perception time. One of the well-known researchers of web ergonomics, Jakob Nielsen (2006) established in an eye-tracking experiment that users scan displayed homepages in an F shape. In the present project the task was to find similar shapes in a number of playful visual search games. Several multimedia tasks were developed for this investigation. Our experiments included normal users and users with intellectual disabilities. We tested whether the characteristic searching routes and navigation methods differed between normal users and those with intellectual disabilities. The results of this investigation can inform the design and position of graphical user interface elements.