On tangible user interfaces,humans and spatiality

Like the prehistoric twig and stone, tangible user interfaces (TUIs) are objects manipulated by humans. Tangible user interface success will depend on how well they exploit spatiality, the intuitive spatial skills humans have with the objects they use. In this paper, we carefully examine the relationship between humans and physical objects, and related previous research. From this examination, we distill a set of observations and turn these into heuristics for incorporation of spatiality into TUI application design, a cornerstone for their success. Following this line of thought, we identify spatial TUIs, the subset of TUIs that mediate interaction with shape, space and structure. We then examine several existing spatial TUIs using our heuristics.     

System design and user evaluation of Co-Star: An immersive stereoscopic system for cable harness design

The system design and results of a user evaluation of Co-Star an immersive design system for cable harness design is described. The system used a stereoscopic head mounted graphical display, user motion tracking and hand-gesture controlled interface to enable cable harnesses to be designed using direct 3D user interaction with a product model. In order to determine how such a system interface would be used by a designer and to obtain user feedback on its main features a practical user evaluation was undertaken involving ten participants each completing three cable harness design tasks with the system. All user interactions with the system were recorded in a time stamped log file during each of the tasks, which were also followed by questionnaire (5 point scale) and interview sessions with each participant. The recorded interaction data for the third task was analysed using functional decomposition techniques and used to construct a single activity profile for the task based on the mean results obtained from the participant group. The goal was to identify in general terms the relative distribution of user activity between specific purposes during practical system operation, and it was found that in this task Navigation accounted for 41%, Design 27%, System Operation 23% and looking at Task Instructions 9% of all user activity. The scored questionnaire data collected immediately after the completion of each task was used to rank the major features of the system according to user opinion. This was further enhanced by also collecting real interview comments from the user group about these same features. The combination of both quantitative performance analysis and subjective user opinion data obtained during a practical design exercise has enabled an in depth evaluation of the system, leading to a much greater understanding of many of the key user and interface requirements that should be considered during the development of immersive interfaces and systems for practical engineering applications.     

Ontology-Based User Modeling in an Augmented Audio Reality System for Museums

Ubiquitous computing is a challenging area that allows us to further our understanding and techniques of context-aware and adaptive systems. Among the challenges is the general problem of capturing the larger context in interaction from the perspective of user modeling and human–computer interaction (HCI). The imperative to address this issue is great considering the emergence of ubiquitous and mobile computing environments. This paper provides an account of our addressing the specific problem of supporting functionality as well as the experience design issues related to museum visits through user modeling in combination with an audio augmented reality and tangible user interface system. This paper details our deployment and evaluation of ec(h)o – an augmented audio reality system for museums. We explore the possibility of supporting a context-aware adaptive system by linking environment, interaction objects and users at an abstract semantic level instead of at the content level. From the user modeling perspective ec(h)o is a knowledge-based recommender system. In this paper we present our findings from user testing and how our approach works well with an audio and tangible user interface within a ubiquitous computing system. We conclude by showing where further research is needed.     

On the interaction between plan recognition and intelligent interfaces

Plan recognition is an active research area in automatic reasoning, as well as a promising approach to engineering interfaces that can exploit models of user's plans and goals. Much research in the field has focused on the development of plan recognition algorithms to support particular user/system interactions, such as found in naturally occurring dialogues. However, two questions have typically remained unexamined: 1) exactly what kind of interface tasks can knowledge of a user's plans be used to support across communication modalities, and 2) how can such tasks in turn constrain development of plan recognition algorithms? In this paper we present a concrete exploration of these issues. In particular, we provide an assessment of plan recognition, with respect to the use of plan recognition in enhancing user interfaces. We clarify how use of a user model containing plans makes interfaces more intelligent and interactive (by providing an intelligent assistant that supports such tasks as advice generation, task completion, context-sensitive responses, error detection and recovery). We then show how interface tasks in turn provide constraints that must be satisfied in order for any plan recognizer to construct and represent a plan in ways that efficiently support these tasks. Finally, we survey how interfaces are fundamentally limited by current plan recognition approaches, and use these limitations to identify and motivate current research. Our research is developed in the context of CHECS, a plan-based design interface.     

Robot-enabled tangible virtual assembly with coordinated midair object placement

The assembly in Virtual Reality (VR) enables users to fit virtual parts into existing 3D models immersively. However, users cannot physically feel the haptic feedback when connecting the parts with the virtual model. This work presents a robot-enabled tangible interface that dynamically moves a physical structure with a robotic arm to provide physical feedback for holding a handheld proxy in VR. This enables the system to provide force feedback during virtual assembly. The cooperation between the physical support and the handheld proxy produces realistic physical force feedback, providing a tangible experience for various virtual parts in virtual assembly scenarios. We developed a prototype system that allowed the operator to place a virtual part onto other models in VR by placing the proxy onto the matched structure attached to a robotic arm. We conducted a user evaluation to explore user performance and system usability in a virtual assembly task. The results indicated that the robot-enabled tangible support increased the task completion time but significantly improved the system usability and sense of presence with a more realistic haptic experience.     

Comparing Parameter Manipulation with Mouse, Pen, and Slider User Interfaces

Visual fixation on one's tool(s) takes much attention away from one's primary task. Following the belief that the best tools 'disappear' and become invisible to the user, we present a study comparing visual fixations (eye gaze within locations on a graphical display) and performance for mouse, pen, and physical slider user interfaces. Participants conducted a controlled, yet representative, color matching task that required user interaction representative of many data exploration tasks such as parameter exploration of medical or fuel cell data. We demonstrate that users may spend up to 95% fewer visual fixations on physical sliders versus standard mouse and pen tools without any loss in performance for a generalized visual performance task.     

Using Distributable User Interfaces in CSCL In-Situ Classrooms

Distributable user interfaces enable users to distribute user interface interaction objects (i.e. panels, buttons, input fields, checkboxes, etc.) across different displays using a set of distribution primitives to manipulate them in real time. This work presents how this kind of user interfaces facilitates the computer supported collaborative learning in modern classrooms. These classrooms provide teachers and students with display ecosystems consisting of stationary displays, such as smart projectors and smart TVs as well as mobile displays owned by teachers and students, such as smartphones, tablets, and laptops. The distribution of user interface interaction objects enables teachers to modify the user interface interaction objects that are available to students in real time to control and promote the collaboration and participation among them during learning activities. We propose the development of this type of applications using an extension of the CAMELEON reference framework that supports the definition of UI distribution models. The Essay exercise is presented as a case of study where teachers control the collaboration among students by distributing user interface interaction objects.     

The Impact of Tangible User Interfaces on Designers' Spatial Cognition

ABSTRACT

Most studies on tangible user interfaces for the tabletop design systems are being undertaken from a technology viewpoint. Although there have been studies that focus on the development of new interactive environments employing tangible user interfaces for designers, there is a lack of evaluation with respect to designers' spatial cognition. In this research we study the effects of tangible user interfaces on designers' spatial cognition to provide empirical evidence for the anecdotal views of the effect of tangible user interfaces. To highlight the expected changes in spatial cognition while using tangible user interfaces, we compared designers using a tangible user interface on a tabletop system with 3D blocks to designers using a graphical user interface on a desktop computer with a mouse and keyboard. The ways in which designers use the two different interfaces for 3D design were examined using a protocol analysis method. The result reveals that designers using 3D blocks perceived more spatial relationships among multiple objects and spaces and discovered new visuo-spatial features when revisiting their design configurations. The designers using the tangible interfaces spent more time in relocating objects to different locations to test the moves, and interacted with the external representation through large body movements implying an immersion in the design model. These two physical actions assist in designers' spatial cognition by reducing cognitive load in mental visual reasoning. Further, designers using the tangible interfaces spent more time in restructuring the design problem by introducing new functional issues as design requirements and produced more discontinuities to the design processes, which provides opportunity for reflection and modification of the design. Therefore this research shows that tangible user interfaces changes designers' spatial cognition, and the changes of the spatial cognition are associated with creative design processes.     

Modular robotics as a tool for education and entertainment

We developed I-BLOCKS, a modular electronic building block system and here we show how this system has proven useful, especially as an educational tool that allows hands-on learning in an easy manner. Through user studies we find limitations of the first I-BLOCKS system, and we show how the system can be improved by introducing a graphical user interface for authoring the contents of the individual I-BLOCK. This is done by developing a new cubic block shape with new physical and electrical connectors, and by including new embedded electronics. We developed and evaluated the I-BLOCKS as a manipulative technology through studies in both schools and hospitals, and in diverse cultures such as in Denmark, Finland, Italy and Tanzania.     

Understanding novelty: how task structure and tool familiarity moderate performance

Consumers and corporate end users are regularly faced with the challenge of adapting to new technology tools and their user interfaces. Understanding the role of user experience in using these new interfaces is an important aspect of assessing and planning for how new innovations are received. Through a controlled laboratory experiment in the information retrieval domain, we find that the effect of task (web search) familiarity on performance depends on the task structure (open-ended versus closed-ended) and the degree of tool familiarity (a traditional textual list versus a novel graphical interface for presenting web search results). Our results show that task experience has a positive effect on performance when the task structure is closed-ended and the interface is familiar, and a negative effect when the task structure is open-ended and the interface is unfamiliar. This study not only adds to the literature on user experience in the context of IT use, but also has practical implications for the testing and adoption of new technologies, indicating that novices may be better, or at least an important group, for testing truly novel tools and their interfaces.     

A visual tool for user-interface development

VITUID is a visual tool for user interface development. It aims to help user-interface designers, who may be non-programmers, to specify and create interactive, graphical user interfaces through graphical interaction. VITUID lets designers specify the user command set and the human-computer dialogs by building a tree and specifying the behavior of each dialog via a dialog-control word, adopting the object-oriented approach. The main strategy of VITUID is to separate a user interface into an application-dependent part and an application-independent part and then let the designer specify only the application-dependent part.     

Real-time landscape model interaction using a tangible geospatial modeling environment

Emerging technologies that combine the flexibility of digital landscape representation with easy-to-interpret 3D physical models open new possibilities for user interaction with geospatial data. A prototype tangible geospatial modeling environment lets users interact with landscape analysis and simulations using a tangible physical model. We introduce a concept that builds upon previous independent tangible user interface (TUI) and terrain analysis research and aims at more intuitive collaborative interaction with digital landscape data.     

文字形态与图符信息在网络媒体界面设计的融合应用

文字形态是文字个性及文字图符化信息展示的主要内容,图符信息又构成了网络媒体界面设计信息传播的核心部分,如何将两者娴熟地应用其中,从而推动文字形态在此领域更好的应用。文章通过从“文字形态”到“图符信息”设计应用的深度剖析,从中归纳出网络媒体界面设计根据表现主题进行创意设计的关键、应用方法和表现方式,并通过实例映证随着科技的不断进步,网络媒体界面设计未来的发展空间与方向,得出将文字形态与图符信息做到内容和形式的和谐统一,并配合新技术的互动表现方式将是网络媒体界面设计发展的主旋律。     

MagicMeeting: A Collaborative Tangible Augmented Reality System

We describe an augmented reality (AR) system that allows multiple participants to interact with 2D and 3D data using tangible user interfaces. The system features face-to-face communication, collaborative viewing and manipulation of 3D models, and seamless access to 2D desktop applications within the shared 3D space. All virtual content, including 3D models and 2D desktop windows, is attached to tracked physical objects in order to leverage the efficiencies of natural two-handed manipulation. The presence of 2D desktop space within 3D facilitates data exchange between the two realms, enables control of 3D information by 2D applications, and generally increases productivity by providing access to familiar tools. We present a general concept for a collaborative tangible AR system, including a comprehensive set of interaction techniques, a distributed hardware setup, and a component-based software architecture that can be flexibly configured using XML. We show the validity of our concept with an implementation of an application scenario from the automotive industry.     

基于PCTBTA任务模型的用户界面开发方法

为了适应普适计算环境中用户、设备、使用环境和开发平台的多样性,基于模型的方法被应用于用户界面开发过程中,试图在抽象层次上描述界面,通过模型转换,使其适用于不同的平台.然而,由于目前基于模型的用户界面开发方法（model-based user interface development,简称MBUID）中所采用任务模型的局限性,致使生成的界面难以满足动态环境下用户的可用性需求.提出一种基于任务模型的用户界面开发框架,旨在建模和生成有效、高效、令用户满意的用户界面.在可用性方面,为了准确描述普适计算环境中用户任务,提出一种基于感知控制理论的任务分析方法（perceptual-control-theory-based task analysis,简称PCTBTA）,将使用上下文信息引入到任务分析过程中,并且在较高的抽象层次上反映交互的内容,给可用性设计提供任务空间;在技术方面,为PCTBTA任务模型向界面模型的转换提供技术支持.最后,通过实例说明所提出方法的可行性,并通过与其他方法在可用性和性能方面的比较,表明该方法的有效性.     

Designing for pen and speech input in an object-action framework: the case of email

This study presents a user interface that was intentionally designed to support multimodal interaction by compensating for the weaknesses of speech compared with pen input and vice versa. The test application was email using a web pad with pen and speech input. In the case of pen input, information was represented as visual objects, which were easily accessible. Graphical metaphors were used to enable faster and easier manipulation of data. Speech input was facilitated by displaying the system speech vocabulary to the user. All commands and accessible fields with text labels could be spoken in by name. Commands and objects that the user could access via speech input were shown on a dynamic basis in a window. Multimodal interaction was further enhanced by creating a flexible object-action order such that the user could utter or select a command with a pen followed by the object which was to be enacted upon, or the other way round (e.g., New Message or Message New). The flexible action-object interaction design combined with voice and pen input led to eight possible action-object-modality combinations. The complexity of the multimodal interface was further reduced by making generic commands such as New applicable across corresponding objects. Use of generic commands led to a simplification of menu structures by reducing the number of instances in which actions appeared. In this manner, more content information could be made visible and consistently accessible via pen and speech input. Results of a controlled experiment indicated that the shortest task completion times for the eight possible input conditions were when speech-only was used to refer to an object followed by the action to be performed. Speech-only input with action-object order was also relatively fast. In the case of pen input-only, the shortest task completion times were found when an object was selected first followed by the action to be performed. In multimodal trials in which both pen and speech were used, no significant effect was found for object-action order, suggesting benefits of providing users with a flexible action-object interaction style in multimodal or speech-only systems.     

TangiWheel: A Widget for Manipulating Collections on Tabletop Displays Supporting Hybrid Input Modality

In this paper we present TangiWheel,a collection manipulation widget for tabletop displays.Our implementation is flexible,allowing either multi-touch or interaction,or even a hybrid scheme to better suit user choice and convenience.Different TangiWheel aspects and features are compared with other existing widgets for collection manipulation.The study reveals that TangiWheel is the first proposal to support a hybrid input modality with large resemblance levels between touch and tangible interaction styles.Several experiments were conducted to evaluate the techniques used in each input scheme for a better understanding of tangible surface interfaces in complex tasks performed by a single user (e.g.,involving a typical master-slave exploration pattern).The results show that tangibles perform significantly better than fingers,despite dealing with a greater number of interactions,in situations that require a large number of acquisitions and basic manipulation tasks such as establishing location and orientation.However,when users have to perform multiple exploration and selection operations that do not require previous basic manipulation tasks,for instance when collections are fixed in the interface layout,touch input is significantly better in terms of required time and number of actions.Finally,when a more elastic collection layout or more complex additional insertion or displacement operations are needed,the hybrid and tangible approaches clearly outperform finger-based interactions.     

SwingStates: adding state machines to Java and the Swing toolkit

This article describes SwingStates, a Java toolkit designed to facilitate the development of graphical user interfaces and bring advanced interaction techniques to the Java platform. SwingStates is based on the use of finite‐state machines specified directly in Java to describe the behavior of interactive systems. State machines can be used to redefine the behavior of existing Swing widgets or, in combination with a new canvas widget that features a rich graphical model, to create brand new widgets. SwingStates also supports arbitrary input devices to implement novel interaction techniques based, for example, on bi‐manual or pressure‐sensitive input. We have used SwingStates in several Master's‐level classes over the past two years and have developed a benchmark approach to evaluate the toolkit in this context. The results demonstrate that SwingStates can be used by non‐expert developers with little training to successfully implement advanced interaction techniques. Copyright © 2007 John Wiley & Sons, Ltd.     

Object-oriented specification of user interfaces

This paper presents an object-oriented approach for the specification of graphical user interfaces. Specification starts with the analysis of the end user's operations. The user interface is then designed on the basis of this analysis. Operation analysis is followed by structure and component specification which presents the dialogue structure of the application and the contents of each dialogue. Visualization produces the final screen layouts, and task specification documents the usage of the user interface for the purpose of creating user's guides. The method presented in this paper makes it easier for a designer to take the end user's needs into account. Still, it does not automatically guarantee good quality user interfaces. The top-down nature of the method allows the designer to concentrate on the most important aspects of the user interface and split the design procedure into manageable pieces. Also, the visibility of the process allows the designer to communicate with other people while specifying the user interface. This paper connects the method with the object-oriented specification of entire applications. It briefly explains the connections with object-oriented analysis and design, and demonstrates how to implement the specified user interface in an object oriented fashion. The approach presented in this paper is being applied in the development of a large network management system with about two million lines of C++ code running in the XII environment. Still, the method does not require the specification being implemented with any specific windowing system. The only requirement is that the user interface is based on graphical elements, such as dialogues, push-buttons and text fields.