Dynamically reconfigurable vision-based user interfaces

We describe a system that supports practical, vision-based user interfaces, addressing the issues of a usable interaction paradigm, support for application developers, and support for application deployment in real-world environments. Interfaces are defined as configurations of predefined interactive widgets that can be moved from one surface to another. Complex interfaces can be dynamically reconfigured, changing both form and location on the fly, because the functional definition of the interface is decoupled from the specification of its location in the environment. We illustrate the power of such an architecture in the context of projected interactive displays.Published online: 13 July 2004     

Neuro-cognitively inspired haptic user interfaces

Haptic systems and devices are a recent addition to multimodal systems. These devices have widespread applications such as surgical simulations, medical and procedural training, scientific visualizations, assistive and rehabilitative devices for individuals who have physical or neurological impediments and assistive devices for individuals who are blind. While the potential of haptics in natural human machine interaction is undisputable, the realization of such means is still a long way ahead. There are considerable research challenges to development of natural haptic interfaces. The study of human tactile abilities is a recent endeavor and many of the available systems still do not incorporate the domain knowledge of psychophysics, biomechanics and neurological elements of haptic perception. Development of smart and effective haptic interfaces and devices requires extensive studies that link perceptual phenomena with measurable parameters and incorporation of such domain knowledge in the engineering of haptic interfaces. This paper presents design, development and usability testing of a neuro-cognitively inspired haptic user interface for individuals who are blind. The proposed system design is inspired by neuro-cognitive basis of haptic perception and incorporates the computational aspects and requirements of multimodal information processing system. Usability testing of the system suggests that a biologically inspired haptic user interfaces may form a powerful paradigm for haptic user interface design.

Design and evaluation of an adaptive icon toolbar

As information systems become increasingly important in many different domains, the potential to adapt them to individual users and their needs also becomes more important. Adaptive user interfaces offer many possible ways to adjust displays and improve procedures for a user's individual patterns of work. This paper describes an attempt to design an adaptive user interface in a computer environment familiar to many users. According to one classification of adaptive user interfaces, the adaptive bar described in this paper would be classified as a user-controlled self-adaptation system.At the user's convenience, the adaptive bar offers suggestions for adding or removing command icons, based on the frequency and probability of specific commands. It also implements these changes once the user has agreed to them. Beyond the adaptive bar, the general behavior of the whole user interface does not change, thereby allowing the user to maintain a clear general model of the system. This paper describes the decision-making algorithm implemented in the bar. It also describes the bar's self-adaptive behavior of displaying the frequency of each icon's use through the icon's size. Finally, we present some encouraging preliminary results of evaluations by users.     

Embarking on Spoken-Language NL Interface Design

Natural language (NL) user interfaces are growing in popularity. Unfortunately, the complexity of NL interaction makes these interfaces difficult to design. For NL interfaces to become successful, universal tools are needed to help support the NL design process. What work practice should these tools explicitly support? Interviews with NL designers and our own experiments have identified a specific work practice that designers should consider as they begin to incorporate NL into user interface designs. The work practice study highlights the value of using Wizard of Oz prototyping in NL design. We describe a tool that we have built, called SUEDE, to explicitly support the first stage of NL design for spoken-language user interfaces. Our tools and tools like it will help make NL in human-computer interaction (HCI) more commonplace.     

A multi-formalism approach for model-based dynamic distribution of user interfaces of critical interactive systems

Evolution in the context of use requires evolutions in the user interfaces even when they are currently used by operators. User Centered Development promotes reactive answers to this kind of evolutions either by software evolutions through iterative development approaches or at runtime by providing additional information to the operators such as contextual help for instance. This paper proposes a model-based approach to support proactive management of context of use evolutions. By proactive management we mean mechanisms in place to plan and implement evolutions and adaptations of the entire user interface (including behaviour) in a generic way. The approach proposed handles both concentration and distribution of user interfaces requiring both fusion of information into a single UI or fission of information into several ones. This generic model-based approach is exemplified on a safety critical system from space domain. It presents how the new user interfaces can be generated at runtime to provide a new user interface gathering in a single place all the information required to perform the task. These user interfaces have to be generated at runtime as new procedures (i.e. sequences of operations to be executed in a semi-autonomous way) can be defined by operators at any time in order to react to adverse events and to keep the space system in operation. Such contextual, activity-related user interfaces complement the original user interfaces designed for operating the command and control system. The resulting user interface thus corresponds to a distribution of user interfaces in a focus+context way improving usability by increasing both efficiency and effectiveness.     

Augmented reality agents for user interface adaptation

Most augmented reality (AR) applications are primarily concerned with letting a user browse a 3D virtual world registered with the real world. More advanced AR interfaces let the user interact with the mixed environment, but the virtual part is typically rather finite and deterministic. In contrast, autonomous behavior is often desirable in ubiquitous computing (Ubicomp), which requires the computers embedded into the environment to adapt to context and situation without explicit user intervention. We present an AR framework that is enhanced by typical Ubicomp features by dynamically and proactively exploiting previously unknown applications and hardware devices, and adapting the appearance of the user interface to persistently stored and accumulated user preferences. Our framework explores proactive computing, multi‐user interface adaptation, and user interface migration. We employ mobile and autonomous agents embodied by real and virtual objects as an interface and interaction metaphor, where agent bodies are able to opportunistically migrate between multiple AR applications and computing platforms to best match the needs of the current application context. We present two pilot applications to illustrate design concepts. Copyright © 2007 John Wiley & Sons, Ltd.     

A formal model for plastic human computer interfaces

The considerable and significant progress achieved in the design and development of new interaction devices between man and machine has enabled the emergence of various powerful and efficient input and/or output devices. Each of these new devices brings specific interaction modes.With the emergence of these devices, new interaction techniques and modes arise and new interaction capabilities are offered. New user interfaces need to be designed or former ones need to evolve. The design of so called plastic user interfaces contributes to handling such evolutions. The key requirement for the design of such a user interface is that the new obtained user interface shall be adapted to the application and have, at least, the same behavior as the previous (adapted) one. This paper proposes to address the problem of user interface evolution due to the introduction of new interaction devices and/or new interaction modes. More, precisely, we are interested by the study of the design process of a user interface resulting from the evolution of a former user interface due to the introduction of new devices and/or new interaction capabilities. We consider that interface behaviors are described by labelled transition systems and comparison between user interfaces is handled by an extended definition of the bi-simulation relationship to compare user interface behaviors when interaction modes are replaced by new ones.     

Distributed Augmented Reality for Collaborative Design Applications

This paper presents a system for constructing collaborative design applications based on distributed augmented reality. Augmented reality interfaces are a natural method for presenting computer-based design by merging graphics with a view of the real world. Distribution enables users at remote sites to collaborate on design tasks. The users interactively control their local view, try out design options, and communicate design proposals. They share virtual graphical objects that substitute for real objects which are not yet physically created or are not yet placed into the real design environment. We describe the underlying augmented reality system and in particular how it has been extended in order to support multi-user collaboration. The construction of distributed augmented reality applications is made easier by a separation of interface, interaction and distribution issues. An interior design application is used as an example to demonstrate the advantages of our approach.     

Exploring how children use their hands to think: an embodied interactional analysis

In order to better understand how to design hands-on child-computer interaction, we explore how different styles of interaction facilitate children's thinking while they use their hands to manipulate objects. We present an exploratory study of children solving a spatial puzzle task. We investigate how the affordances of physical, graphical and tangible interfaces may facilitate the development of thinking skills including mental visualisation, problem space exploration and collaboration. We utilise the theory of complementary actions taken from embodied cognition to develop a video coding methodology that allows us to classify behavioural activity and make inferences about thinking skills development. Our findings indicated that the combination of direct hands-on input style with audio-visual feedback facilitated by the tangible user interface enabled a dynamic task completion strategy, which supports the development of mental skills with a slight time cost. The mouse and graphical user interface supported a trial and error approach, which may limit skills development. The physical cardboard puzzle enabled effective task completion but provided less support for social interaction and problem space exploration. We conclude with design recommendations.     

Two-handed tangible interaction techniques for composing augmented blocks

Modeling tools typically have their own interaction methods for combining virtual objects. For realistic composition in 3D space, many researchers from the fields of virtual and augmented reality have been trying to develop intuitive interactive techniques using novel interfaces. However, many modeling applications require a long learning time for novice users because of unmanageable interfaces. In this paper, we propose two-handed tangible augmented reality interaction techniques that provide an easy-to-learn and natural combination method using simple augmented blocks. We have designed a novel interface called the cubical user interface, which has two tangible cubes that are tracked by marker tracking. Using the interface, we suggest two types of interactions based on familiar metaphors from real object assembly. The first, the screw-driving method, recognizes the user??s rotation gestures and allows them to screw virtual objects together. The second, the block-assembly method, adds objects based on their direction and position relative to predefined structures. We evaluate the proposed methods in detail with a user experiment that compares the different methods.     

Smart objects to support the discrimination training of children with autism

Teachers spend considerable amount of time keeping students with autism “on task” giving away prompts and rewards and maintaining a detailed record of students’ progress during the object discrimination training. We hypothesize that tangible computing, in particular smart objects, could help teachers cope with the problems faced during the object discrimination training of students with autism. In this paper, we describe design principles for smart objects to support the object discrimination training and present several example prototypes. First, we present the design and implementation of “Things that think” (T3), a smart device that converts traditional objects into smart objects that promote interactivity with a playful and engaging interaction, and are capable of the automatic recording of students’ progress. Then, we present four T3 smart objects assembled in a board. The results of a 7-week deployment study of the use of such smart objects in three classrooms of students with autism (n = 25, 7 teachers and 18 students with autism) demonstrate T3 smart objects reduce the workload of teachers, ease the record-keeping and increase its reliability, and reduce students’ behavioral problems while improving their cognitive efficacy. We close discussing directions for future work.     

支持用户界面自动生成的界面模型

为实现软件用户界面的工程设计,论文提出了一种支持用户界面自动生成的界面模型FMP。它利用功能模型(F)和对象模型(M)捕捉与界面相关的功能和数据信息,用交互模型(M)表达这些信息和控制关系。然后依据交互模型的描述,利用表示模型(P)建立内部对象和外部显示元素的对应关系,规划界面布局,从而指导实现用户界面的自动生成。     

To model objects in information systems and user interfaces: implications for object-oriented methods

Abstract. A general way of contrasting the object-oriented to the function-oriented approach, called 'object-order' and 'function-order', is proposed in this paper. In addition, when modelling information systems, there is a third principle of ordering, 'subject-order'. Object and function-order are used to define the two approaches to modelling information systems and in human-computer interaction, in ways that are compatible to the corresponding distinction in programming. The conditions for selecting either of the two approaches are, however, not discussed here.
By means of an object-ordered method, e.g. Jackson System Development (JSD), an object-ordered model is produced, and the user interface that can be derived from the model is object-ordered too. However, other guidelines in JSD support subject-order. A function-ordered interface can be derived from the subject-ordered model. A computer system designed by means of an object-organized method may therefore obtain an interface that is partially object-ordered and partially function-ordered. Other object-oriented methods give guidelines for designing function-oriented interfaces without relating these guidelines to the model.
The system designer may benefit from having three principles of ordering when modelling a system. However, a user may be confused by an interface that sometimes gives preference to objects, sometimes to functions.     

Supporting Cooperation through Customisation: The Tviews Approach

User interfaces for groupware systems rarely reflectthe different requirements for support of theirend-users. Here we present an approach to designingmulti-user interfaces for cooperative systems whichbuilds on previous work from the HCI community in thearea of end-user customisation. Using this approach wehave developed an approach and a system prototype basedon tailorable views, or Tviews, which allows end-usersengaged in group working to configure theircooperative system interfaces to support theirdifferent tasks, preferences and levels of expertise.Tviews are user interface components which can bedragged and dropped over representations ofapplication objects to customise presentation,interaction and event updating properties, and canthemselves be tailored using high-level, incrementalcustomisation techniques. We discuss the implicationsof this work for CSCW system development by referenceto studies of work carried out by the CSCW communitywhich point to a need for more flexible and tailorablesystem interfaces.     

On tangible user interfaces,humans and spatiality

Like the prehistoric twig and stone, tangible user interfaces (TUIs) are objects manipulated by humans. Tangible user interface success will depend on how well they exploit spatiality, the intuitive spatial skills humans have with the objects they use. In this paper, we carefully examine the relationship between humans and physical objects, and related previous research. From this examination, we distill a set of observations and turn these into heuristics for incorporation of spatiality into TUI application design, a cornerstone for their success. Following this line of thought, we identify spatial TUIs, the subset of TUIs that mediate interaction with shape, space and structure. We then examine several existing spatial TUIs using our heuristics.     

Sisl: Several Interfaces, Single Logic

Modern interactive services such as information and e-commerce services are becoming increasingly more flexible in the types of user interfaces they support. These interfaces incorporate automatic speech recognition and natural language understanding and include graphical user interfaces on the desktop and web-based interfaces using applets and HTML forms. To what extent can the user interface software be decoupled from the service logic software (the code that defines the essential function of a service)? Decoupling of user interface from service logic directly impacts the flexibility of services, or how easy they are to modify and extend. To explore these issues, we have developed Sisl, an architecture and domain-specific language for designing and implementing interactive services with multiple user interfaces. A key principle underlying Sisl is that all user interfaces to a service share the same service logic. Sisl provides a clean separation between the service logic and the software for a variety of interfaces, including Java applets, HTML pages, speech-based natural language dialogue, and telephone-based voice access. Sisl uses an event-based model of services that allows service providers to support interchangeable user interfaces (or add new ones) to a single consistent source of service logic and data. As part of a collaboration between research and development, Sisl is being used to prototype a new generation of call processing services for a Lucent Technologies switching product.     

Experiments with Face-To-Face Collaborative AR Interfaces

We describe a design approach, Tangible Augmented Reality, for developing face-to-face collaborative Augmented Reality (AR) interfaces. Tangible Augmented Reality combines Augmented Reality techniques with Tangible User Interface elements to create interfaces in which users can interact with spatial data as easily as real objects. Tangible AR interfaces remove the separation between the real and virtual worlds, and so enhance natural face-to-face communication. We present several examples of Tangible AR interfaces and results from a user study that compares communication in a collaborative AR interface to more traditional approaches. We find that in a collaborative AR interface people use behaviours that are more similar to unmediated face-to-face collaboration than in a projection screen interface.     

协作笔式用户界面开发工具研究

笔式用户界面是一种重要的Post-WIMP(window icon menu pointer)界面,它给用户提供了自然的交互方式.然而,当前的笔式用户界面工具箱大多是面向单用户任务的,不能很好地支持协作应用场景.通过对笔式交互特征和协作环境功能需求的分析,设计并实现了一个工具箱CoPen Toolkit,用于支持协作笔式用户界面的开发.它提供了灵活的架构和可扩展的组件,支持笔迹描述、事件处理和网络协作等功能.基于CoPen Toolkit,构造了多个原型系统,实践表明,它能够很好地支持协作笔式用户界面的开发.     

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARFs user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.