Mixed reality storytelling environments

The focus of this paper is the collaborative storytelling environments created as part of a three-year multi-disciplinary research project, KidStory. The project team, from the UK, Sweden and the USA, developed a series of story creation and telling tools and virtual environments for children aged 5–7 years. This paper concerns work with a Nottingham primary school to design and develop collaborative storytelling tools, including tangible interfaces devices and reactive spaces, with the aim of integrating these within the school context. The final set-up allowed children to dynamically produce story content, to create basic narrative structures and to retell their stories in a collaborative and adaptable physical space. Acknowledgements.This work has been conducted as part of the EU Esprit 29310 KidStory project. The authors acknowledge support from all partners (SICS/Maryland/KTH). Thanks also to the teachers and children at the primary school in Nottingham for their collaboration in this work. To download KidPad see http://www.kidpad.org.     

TaKo: Providing transparent collaboration on single-user applications

The conversion of legacy single-user applications into collaborative multi-user tools is a recurrent topic in groupware scenarios. Many recent literature works have tried to achieve transparent collaboration, which consists of enabling collaborative features without modifying the original application source code.In this paper, we define the available whitebox and blackbox models of transparent collaboration. Each of them differs on which degree the developer must have of the target application inner knowledge, in order to convert it into a multi-user tool. Moreover, we propose and define a novel blackbox model and its implementation (TaKo). Our proposal achieves complete transparency by intercepting user interface libraries and input events. This is the first blackbox solution constructed on top of interception technologies (Aspect Oriented Programming) and, unlike previous approaches, it provides support to both AWT and Swing applications. Our solution solves five important problems: simultaneous work, management of replicated resources (random number generators), collaborative services binding, detailed group awareness information, late joining, and unanticipated sharing support. We are also researching a prospective work on wide-area collaboration scenarios by using a peer-to-peer event substrate. Finally, this work provides validation of TaKo with several Swing-based and AWT-based tools, demonstrating that it is generic and imposes very low overhead.     

Leveraging Java applets: toward collaboration transparency in Java

Widespread use of the Internet is giving rise to the need for collaborative applications that link users at remote sites. Many toolkits support the development of collaboration-aware applications, those developed specifically for cooperative work by multiple users. Another approach is collaboration transparency, the collaborative use of applications originally developed for a single user. When the runtime environment supports collaboration transparency, an application programmer need not write new code to make an application collaborative. Thus, collaboration transparency leverages the existing base of single user applications by extending them to collaborative use. We review options for providing collaboration transparency. We also discuss how collaboration transparency can be incorporated into Sun Microsystems' Java, the most widely used vehicle for developing interactive World Wide Web applications     

IMMIView: a multi-user solution for design review in real-time

IMMIView is an interactive system that relies on multiple modalities and multi-user interaction to support collaborative design review. It was designed to offer natural interaction in visualization setups such as large-scale displays, head mounted displays or TabletPC computers. To support architectural design, our system provides content creation and manipulation, 3D scene navigation and annotations. Users can interact with the system using laser pointers, speech commands, body gestures and mobile devices. In this paper, we describe how we design a system to answer architectural user requirements. In particular, our system takes advantage of multiple modalities to provide a natural interaction for design review. We also propose a new graphical user interface adapted to architectural user tasks, such as navigation or annotations. The interface relies on a novel stroke-based interaction supported by simple laser pointers as input devices for large-scale displays. Furthermore, input devices such as speech and body tracking allow IMMIView to support multiple users. Moreover, they allow each user to select different modalities according to their preference and modality adequacy for the user task. We present a multi-modal fusion system developed to support multi-modal commands on a collaborative, co-located, environment, i.e. with two or more users interacting at the same time, on the same system. The multi-modal fusion system listens to inputs from all the IMMIView modules in order to model user actions and issue commands. The multiple modalities are fused based on a simple rule-based sub-module developed in IMMIView and presented in this paper. User evaluation performed over IMMIView is presented. The results show that users feel comfortable with the system and suggest that users prefer the multi-modal approach to more conventional interactions, such as mouse and menus, for the architectural tasks presented.     

The ECLIPSE user interface

This paper describes the user interface facilities of the ECLIPSE integrated project support environment. This interface is based on a consistent metaphor called the ‘control panel’ metaphor and includes standard help and message-handling systems. The paper describes these as well as some of the interface standards which have been developed. The interface has been implemented on top of the ‘applications interface’, which provides a portable, hardware-independent interface for software tools.     

User interface design for military AR applications

Designing a user interface for military situation awareness presents challenges for managing information in a useful and usable manner. We present an integrated set of functions for the presentation of and interaction with information for a mobile augmented reality application for military applications. Our research has concentrated on four areas. We filter information based on relevance to the user (in turn based on location), evaluate methods for presenting information that represents entities occluded from the user??s view, enable interaction through a top-down map view metaphor akin to current techniques used in the military, and facilitate collaboration with other mobile users and/or a command center. In addition, we refined the user interface architecture to conform to requirements from subject matter experts. We discuss the lessons learned in our work and directions for future research.     

协作笔式用户界面开发工具研究

笔式用户界面是一种重要的Post-WIMP(window icon menu pointer)界面,它给用户提供了自然的交互方式.然而,当前的笔式用户界面工具箱大多是面向单用户任务的,不能很好地支持协作应用场景.通过对笔式交互特征和协作环境功能需求的分析,设计并实现了一个工具箱CoPen Toolkit,用于支持协作笔式用户界面的开发.它提供了灵活的架构和可扩展的组件,支持笔迹描述、事件处理和网络协作等功能.基于CoPen Toolkit,构造了多个原型系统,实践表明,它能够很好地支持协作笔式用户界面的开发.     

An Integrated Approach to Designing and Evaluating Collaborative Applications and Infrastructures

Collaborative systems include both general infrastructures and specific applications for supporting collaboration. Because of the relative newness and complexity of these systems, it has been unclear what approach should be used to design and evaluate them. Based on the lessons learned from our work and that of others on collaborative systems, we have derived an integrated approach to researching collaborative applications and infrastructures. The approach can be described as a sequence of steps: We decompose the functionality of collaboration systems into smaller functions that can be researched more-or-less independently. For each of these functions, we adopt general (system-independent) principles regarding the design and implementation of the function, identify collaboration scenarios at multiple levels of abstraction, identify requirements based on the scenarios, adopt an interaction model to meet the requirements, realize the interaction model as a concrete user interface, develop a logical architecture of the system, identify a physical architecture for placing the logical components in a distributed system, develop infrastructure abstractions, use the abstractions to implement applications, and perform lab studies, field experiments, and simulations to evaluate the infrastructure and applications. As in other models with multiple phases, feedback from subsequent phases is used to modify the results from the previous phases. In this paper, we describe, illustrate and motivate this research plan.     

Evaluation of a Geospatial Annotation Tool for Unmanned Vehicle Specialist Interface

Map-based interfaces have been developed to support collaborative control of unmanned vehicles (i.e., robots). Annotation on the map (or geospatial annotation) has been proposed as an effective way to support team collaboration; however, there is a lack of research focused on the design of geospatial annotation tools to promote usability and task performance. The utility of location reference in geospatial annotations for communication and information sharing is the focus of this article. Two annotation tool designs were developed. The annotation contents were directly anchored on the map in the first design, whereas in the second design annotations were summarized in a separate panel on the interface. Evaluation participants followed instructions from a simulated team leader and assigned unmanned vehicles to different tasks for two simulated scenarios that include searching for victims and collecting hazardous materials samples. The results demonstrate the potential of using geospatial annotations to enrich communication and support map-based unmanned vehicle control. Participants appreciated the direct location reference feature of the first design and had generally shorter response time, but felt that the second design provided better usability and lower task workload. These results suggest that the user experience depends on the manner of obtaining information from the annotation tools, and the integration of the tool with user's task flow and other interface components, such as the map display. The presented results can be used as a basis for designing geospatial annotation tools for team collaboration that better fit user needs and preferences.     

Introduction to building projection-based tiled display systems

This tutorial introduces the concepts and technologies needed to build projector-based display systems. Tiled displays offer scalability, high resolution, and large formats for various applications. Tiled displays are an emerging technology for constructing semi-immersive visualization environments capable of presenting high-resolution images from scientific simulation. The largest impact may well arise from using large-format tiled displays as one of possibly multiple displays in building information or active spaces that surround the user with diverse ways of interacting with data and multimedia information flows. These environments may prove the ultimate successor to the desktop metaphor for information technology work. Several fundamental technological problems must be addressed to make tiled displays practical. These include: the choice of screen materials and support structures; choice of projectors, projector supports, and optional fine positioners; techniques for integrating image tiles into a seamless whole; interface devices for interaction with applications; display generators and interfaces; and the display software environment     

Designing peer-to-peer distributed user interfaces: Case studies on building distributed applications

Building a distributed user interface (DUI) application should ideally not require any additional effort beyond that necessary to build a non-distributed interface. In practice, however, DUI development is fraught with several technical challenges such as synchronization, resource management, and data transfer. In this paper, we present three case studies on building distributed user interface applications: a distributed media player for multiple displays and controls, a collaborative search system integrating a tabletop and mobile devices, and a multiplayer Tetris game for multi-surface use. While there exist several possible network architectures for such applications, our particular approach focuses on peer-to-peer (P2P) architectures. This focus leads to a number of challenges and opportunities. Drawing from these studies, we derive general challenges for P2P DUI development in terms of design, architecture, and implementation. We conclude with some general guidelines for practical DUI application development using peer-to-peer architectures.     

Using Distributable User Interfaces in CSCL In-Situ Classrooms

Distributable user interfaces enable users to distribute user interface interaction objects (i.e. panels, buttons, input fields, checkboxes, etc.) across different displays using a set of distribution primitives to manipulate them in real time. This work presents how this kind of user interfaces facilitates the computer supported collaborative learning in modern classrooms. These classrooms provide teachers and students with display ecosystems consisting of stationary displays, such as smart projectors and smart TVs as well as mobile displays owned by teachers and students, such as smartphones, tablets, and laptops. The distribution of user interface interaction objects enables teachers to modify the user interface interaction objects that are available to students in real time to control and promote the collaboration and participation among them during learning activities. We propose the development of this type of applications using an extension of the CAMELEON reference framework that supports the definition of UI distribution models. The Essay exercise is presented as a case of study where teachers control the collaboration among students by distributing user interface interaction objects.     

Multiple decoupled interaction: An interaction design approach for groupware interaction in co-located virtual environments

Interactive visualizations such as virtual environments and their associated input and interface techniques have traditionally focused on localized single-user interactions and have lacked co-present active collaboration mechanisms where two or more co-located users can share and actively cooperate and interact with the visual simulation. VR facilities such as CAVEs or PowerWalls, among many others, seem to promise such collaboration but due to the special requirements in terms of 3D input and output devices and the physical configuration and layout, they are generally designed to support an active controlling participant—the immersed user—and a passive viewing only audience. In this paper we explore the integration of different technologies, such as small handheld devices and wireless networks with VR/VEs in order to develop a technical and conceptual interaction approach that allows creation of a more ad hoc, interaction rich, multimodal and multi-device environment, where multiple users can access certain interactive capabilities of VE and support co-located collaboration.     

Potential of multimodal and multiuser interaction with virtual holography

Virtual holography is a disruptive technology that can inspire innovations in variety of fields through blending the physical world with sensory-rich virtual world. The technology enables users to naturally and intuitively manipulate objects and navigate in 3D space. The zSpace virtual holography platform is described. The platform provides a 3D display and head-tracking technology that transforms PCs into virtual-holographic computing facility using a stereoscopic user interface with an interactive stylus. The major components of the platform and its potential benefits, along with some of the current applications are briefly described.The use of the zSpace platform to explore and manipulate a number of space system models is outlined. The models considered include Titan Saturn system, a joint NASA/ESA mission; Mars Science Lab concept; James Webb Telescope; and a Lunar Lander concept. In each of the applications considered, the platform is enhanced by using multimodal interactions, and providing support for multiuser collaboration. The multimodal interactions, which enable more engaging, enhanced accessibility, are achieved by fusing information from a set of stylus input, 3D gestures, and neural input. Simultaneous and collaborative multiuser interactions are described, which support both local and distributed teams, using variety of displays.Emerging and future enhancements of the zSpace platform are outlined, along with novel future applications.     

Integrated and distributed computational support for building performance evaluation

This paper reports on the progress of the SEMPER prototype 2 (S2) project. This is an ongoing effort toward an Internet-based environment for distributed collaborative performance-based building design and evaluation. A user can access the S2 system regardless of hardware, operating system or location on a network. Geographically distributed users can generate and edit building models via a platform-independent user interface. These building models can then be made subject to concurrent analysis by multiple simulation applications running on remote servers. Persistent storage is provided for project data and evaluation results. Designers using the system have access to multiple libraries with semantic building information.     

Architectural patterns for applications with external user interface elements

Coupling mobile devices and other remote interaction technology with software systems surrounding the user enables for building interactive environments under explicit user control. The realization of explicit interaction in ubiquitous or pervasive computing environments introduces a physical distribution of input devices, and technology embedded into the environment of the user. To fulfill the requirements of emerging trends in mobile interaction, common approaches for system design need adaptations and extensions. This paper presents the adaptation and extension of the Model-View-Controller approach to design applications of remote, complementary, duplicated and detached user interface elements.     

Augmented reality agents for user interface adaptation

Most augmented reality (AR) applications are primarily concerned with letting a user browse a 3D virtual world registered with the real world. More advanced AR interfaces let the user interact with the mixed environment, but the virtual part is typically rather finite and deterministic. In contrast, autonomous behavior is often desirable in ubiquitous computing (Ubicomp), which requires the computers embedded into the environment to adapt to context and situation without explicit user intervention. We present an AR framework that is enhanced by typical Ubicomp features by dynamically and proactively exploiting previously unknown applications and hardware devices, and adapting the appearance of the user interface to persistently stored and accumulated user preferences. Our framework explores proactive computing, multi‐user interface adaptation, and user interface migration. We employ mobile and autonomous agents embodied by real and virtual objects as an interface and interaction metaphor, where agent bodies are able to opportunistically migrate between multiple AR applications and computing platforms to best match the needs of the current application context. We present two pilot applications to illustrate design concepts. Copyright © 2007 John Wiley & Sons, Ltd.     

A painting interface for interactive surface deformations

A long-standing challenge in geometric modeling is providing a natural, intuitive interface for making local deformations to 3D surfaces. Previous approaches have provided either interactive manipulation or physical simulation to control surface deformations. In this paper, we investigate combining these two approaches with a painting interface that gives the user direct, local control over a physical simulation. The “paint” a user applies to the model defines its instantaneous surface velocity. By interactively simulating this velocity, the user can effect surface deformations. We have found that this painting metaphor gives the user direct, local control over surface deformations for several applications: creating new models, removing noise from existing models, and adding geometric texture to an existing surface at multiple scales.     

Integration of Adaptable and Adaptive Approaches for Interface Personalization Through Collaborative Menu

This article examines different user-system collaboration models in the adaptation of a menu interface. Four collaboration models were implemented on a prototype of mobile phone menu: (a) basic collaboration with no system support (for user adaptation) and no user control (over system adaptation), (b) system support only, (c) user control only, and (d) system support plus user control. The prototype mobile phone menu includes a hotlist (a quickly accessible collection of menu items) as well as a hierarchical menu. The hotlist is collaborative, because it combines adaptable and adaptive approaches by allowing both the user and the system to manage the items in it. A controlled experiment compared different types of collaborative menus in order to investigate the effects of system support and user control. Twenty participants performed menu selection tasks in the experiment, and both performance and subjective measures were taken. The results showed that, in a certain condition, the system support and the user control improved the user performance when applied independently, but their effects were not additive. Although the effects disappeared when the selection frequency distribution changed, the system support was preferred by most of the users. The advantages and disadvantages of the collaborative menus and implications for the adaptation of menus are discussed.