Usability engineering for augmented reality: employing user-based studies to inform design

A major challenge, and thus opportunity, in the field of human-computer interaction and specifically usability engineering is designing effective user interfaces for emerging technologies that have no established design guidelines or interaction metaphors or introduce completely new ways for users to perceive and interact with technology and the world around them. Clearly, augmented reality is one such emerging technology. We propose a usability engineering approach that employs user-based studies to inform design, by iteratively inserting a series of user-based studies into a traditional usability engineering lifecycle to better inform initial user interface designs. We present an exemplar user-based study conducted to gain insight into how users perceive text in outdoor augmented reality settings and to derive implications for design in outdoor augmented reality. We also describe lessons learned from our experiences conducting user-based studies as part of the design process.     

Ontology-Based User Modeling in an Augmented Audio Reality System for Museums

Ubiquitous computing is a challenging area that allows us to further our understanding and techniques of context-aware and adaptive systems. Among the challenges is the general problem of capturing the larger context in interaction from the perspective of user modeling and human–computer interaction (HCI). The imperative to address this issue is great considering the emergence of ubiquitous and mobile computing environments. This paper provides an account of our addressing the specific problem of supporting functionality as well as the experience design issues related to museum visits through user modeling in combination with an audio augmented reality and tangible user interface system. This paper details our deployment and evaluation of ec(h)o – an augmented audio reality system for museums. We explore the possibility of supporting a context-aware adaptive system by linking environment, interaction objects and users at an abstract semantic level instead of at the content level. From the user modeling perspective ec(h)o is a knowledge-based recommender system. In this paper we present our findings from user testing and how our approach works well with an audio and tangible user interface within a ubiquitous computing system. We conclude by showing where further research is needed.     

A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARFs user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.     

Glove Based User Interaction Techniques for Augmented Reality in an Outdoor Environment

This paper presents a set of pinch glove-based user interface tools for an outdoor wearable augmented reality computer system. The main form of user interaction is the use of hand and head gestures. We have developed a set of augmented reality information presentation techniques. To support direct manipulation, the following three selection techniques have been implemented: two-handed framing, line of sight and laser beam. A new glove-based text entry mechanism has been developed to support symbolic manipulation. A scenario for a military logistics task is described to illustrate the functionality of this form of interaction.     

Two-handed tangible interaction techniques for composing augmented blocks

Modeling tools typically have their own interaction methods for combining virtual objects. For realistic composition in 3D space, many researchers from the fields of virtual and augmented reality have been trying to develop intuitive interactive techniques using novel interfaces. However, many modeling applications require a long learning time for novice users because of unmanageable interfaces. In this paper, we propose two-handed tangible augmented reality interaction techniques that provide an easy-to-learn and natural combination method using simple augmented blocks. We have designed a novel interface called the cubical user interface, which has two tangible cubes that are tracked by marker tracking. Using the interface, we suggest two types of interactions based on familiar metaphors from real object assembly. The first, the screw-driving method, recognizes the user??s rotation gestures and allows them to screw virtual objects together. The second, the block-assembly method, adds objects based on their direction and position relative to predefined structures. We evaluate the proposed methods in detail with a user experiment that compares the different methods.     

Direct hand touchable interactions in augmented reality environments for natural and intuitive user experiences

Several studies have been carried out on augmented reality (AR)-based environments that deal with user interfaces for manipulating and interacting with virtual objects aimed at improving immersive feeling and natural interaction. Most of these studies have utilized AR paddles or AR cubes for interactions. However, these interactions overly constrain the users in their ability to directly manipulate AR objects and are limited in providing natural feeling in the user interface. This paper presents a novel approach to natural and intuitive interactions through a direct hand touchable interface in various AR-based user experiences. It combines markerless augmented reality with a depth camera to effectively detect multiple hand touches in an AR space. Furthermore, to simplify hand touch recognition, the point cloud generated by Kinect is analyzed and filtered out. The proposed approach can easily trigger AR interactions, and allows users to experience more intuitive and natural sensations and provides much control efficiency in diverse AR environments. Furthermore, it can easily solve the occlusion problem of the hand and arm region inherent in conventional AR approaches through the analysis of the extracted point cloud. We present the effectiveness and advantages of the proposed approach by demonstrating several implementation results such as interactive AR car design and touchable AR pamphlet. We also present an analysis of a usability study to compare the proposed approach with other well-known AR interactions.     

BRIDGING THE GAPS: HYBRID TRACKING FOR ADAPTIVE MOBILE AUGMENTED REALITY

Tracking accuracy in a location-aware mobile system can change dynamically as a function of the user's location and other variables specific to the tracking technologies used. This is especially problematic for mobile augmented reality systems, which ideally require extremely precise position tracking for the user's head, but which may not always be able to achieve that level of accuracy. While it is possible to ignore variable positional accuracy in an augmented reality user interface, this can make for a confusing system; for example, when accuracy is low, virtual objects that are nominally registered with real ones may be too far off to be of use.

To address this problem, we describe an experimental mobile augmented reality system that: (1) employs multiple position-tracking technologies, including ones that apply heuristics based on environmental knowledge; (2) coordinates these concurrently monitored tracking systems; and (3) automatically adapts the user interface to varying degrees of confidence in tracking accuracy. We share our experiences with managing these multiple tracking technologies, employing various techniques to facilitate smooth and reasonable “hand-offs” between the cooperating systems. We present these results in the context of an intelligent navigational guidance system that helps users to orient themselves in an unfamiliar environment, using path planning to guide them toward destinations they choose, and sometimes towards ones the system infers as equally relevant.     

Through-Walls Communication for Medical Emergency Services

The authors present a model for bringing the coordination power of workflow management systems to outdoor wearable augmented reality (AR) systems. They portray how mobile equipment may be integrated with adaptive, context-aware work environments. A scenario of a medical emergency task is described to illustrate the functionality of this form of collaboration system. Appropriate information stickers are introduced to support data collection in medical emergency scenarios in a sophisticated form through a hands-free user interface for medical personnel. They propose the use of new user interface technology, including multimedia, AR information stickers, and the allocation of patient medical records to identified locations of the human body. A key feature is the access to relevant information for users in the mobile environment as well as for those in the advanced control room. An additional advantage is the automatic recording of on-site data, which helps to build the medical record of a patient without interfering with the work of the emergency team.     

WalkMap: Developing an augmented reality map application for wearable computers

We have designed, implemented, and evaluated a map application for wearable computer users. Our application, called WalkMap, is targeted at a walking user in an urban environment, offering the user both navigational aids as well as contextual information. WalkMap uses augmented reality techniques to display a map on the surrounding area on the user's head-worn display. WalkMap is constructed by the means of software development, user interface design and evaluations, and existing knowledge on how humans use maps and navigate. The key design driver in our approach is intuitivity of use. In this paper, we present the design and implementation process of our application, considering human-map interfaces, technical implementation, and human-computer interfaces. We identify some of the key issues in these areas, and present the way they have been solved. We also present some usability evaluation results.     

Augmented reality agents for user interface adaptation

Most augmented reality (AR) applications are primarily concerned with letting a user browse a 3D virtual world registered with the real world. More advanced AR interfaces let the user interact with the mixed environment, but the virtual part is typically rather finite and deterministic. In contrast, autonomous behavior is often desirable in ubiquitous computing (Ubicomp), which requires the computers embedded into the environment to adapt to context and situation without explicit user intervention. We present an AR framework that is enhanced by typical Ubicomp features by dynamically and proactively exploiting previously unknown applications and hardware devices, and adapting the appearance of the user interface to persistently stored and accumulated user preferences. Our framework explores proactive computing, multi‐user interface adaptation, and user interface migration. We employ mobile and autonomous agents embodied by real and virtual objects as an interface and interaction metaphor, where agent bodies are able to opportunistically migrate between multiple AR applications and computing platforms to best match the needs of the current application context. We present two pilot applications to illustrate design concepts. Copyright © 2007 John Wiley & Sons, Ltd.     

User interface design for military AR applications

Designing a user interface for military situation awareness presents challenges for managing information in a useful and usable manner. We present an integrated set of functions for the presentation of and interaction with information for a mobile augmented reality application for military applications. Our research has concentrated on four areas. We filter information based on relevance to the user (in turn based on location), evaluate methods for presenting information that represents entities occluded from the user??s view, enable interaction through a top-down map view metaphor akin to current techniques used in the military, and facilitate collaboration with other mobile users and/or a command center. In addition, we refined the user interface architecture to conform to requirements from subject matter experts. We discuss the lessons learned in our work and directions for future research.     

Building and Employing Cross-Reality

Considers present and future practical applications of cross-reality. From tools to build new 3D virtual worlds to the products of those tools, cross-reality is becoming a staple of our everyday reality. Practical applications of cross-reality include the ability to virtually visit a factory to manage and maintain resources from the comfort of your laptop or desktop PC as well as sentient visors that augment reality with additional information so that users can make more informed choices. Tools and projects considered are:Project Wonderland for multiuser mixed reality;ClearWorlds: mixed- reality presence through virtual clearboards; VICI (Visualization of Immersive and Contextual Information) for ubiquitous augmented reality based on a tangible user interface; Mirror World Chocolate Factory; and sentient visors for browsing the world.     

智能手机的移动增强现实技术研究

随着增强现实技术的快速发展以及智能手机技术的革新,移动增强现实(M-AR)技术得以脱离体积庞大的PC机设备的限制,使其向户外和无线化发展成为可能,并由此推动了增强现实技术在智能手机上的应用。针对智能手机的移动增强现实技术在应用发展中所要解决的问题,进行了分析和探讨,较为系统地阐述了该领域最新的发展状况,并展望了该技术未来的发展前景。     

Modelling and design of an augmented reality differential drive mobility aid in an enabled environment

In this paper the authors discuss the modelling and design of an augmented reality platform for disabled wheeled mobility studies. This design consists of a virtual environment with two degrees of freedom motion platform and integrated ground contact force feedback. Differential drive mobility users continue to be in touch with reality on their own mobility aid while interacting with virtual objects. The major domain related to the differential drive mobility of the disabled members of society which coincides with the use of manual wheelchairs, electric wheelchairs and mobility scooters. In order to account for environmental and dynamic effects, the wheeled mobility user needs to map the intended trajectory into the virtual world. Motion and inertia force feedback produced on the augmented simulator give the users a haptic sensory stimulus input regarding spatial movement and ground contact forces. The main objective is to model and design an augmented reality platform with real world kinematic and dynamic properties to place a wheeled mobility user closer to real world encounters. The use of the designed augmented reality platform will be beneficial to disabled wheeled mobility users in need of occupational therapist training and evaluation.     

A-BI+: A framework for Augmented Business Intelligence

Augmented reality allows users to superimpose digital information (typically, of operational type) upon real-world objects. The synergy of analytical frameworks and augmented reality opens the door to a new wave of situated analytics, in which users within a physical environment are provided with immersive analyses of local contextual data. In this paper, we propose an approach named A-BI⁺ (Augmented Business Intelligence) that, based on the sensed augmented context (provided by wearable and smart devices), proposes a set of relevant analytical queries to the user. This is done by relying on a mapping between the objects that can be recognized by the devices and the elements of the enterprise multidimensional cubes, and also by taking into account the queries preferred by users during previous interactions that occurred in similar contexts. A set of experimental tests evaluates the proposed approach in terms of efficiency, effectiveness, and user satisfaction.     

Exploring the use of handheld AR for outdoor navigation

Recently, mobile-phone based outdoor augmented reality (AR) systems have become readily available. One of the most popular applications are AR browsers that show virtual points of interest (POIs) overlaid on top of the phone's camera view. These virtual cues can be used to guide people to the POIs. However, the usefulness of AR systems for guiding users to POI has not yet been evaluated, especially when compared to map interfaces. In this paper we present results of a user study comparing navigation with information typically provided by currently available handheld AR browsers, to navigation with a digital map, and a combined map and AR condition. We found no overall difference in task completion time, but found evidence that AR browsers are less useful for navigation in some environment conditions. We also found that navigation performance differed significantly with gender for the Map and AR+Map interfaces, but is very similar across gender for the AR interface. Users preferred the combined AR+Map condition, and felt that there were significant problems with using the AR view alone for navigation.     

The MagicBook - moving seamlessly between reality and virtuality

The MagicBook project is an early attempt to explore how we can use a physical object to smoothly transport users between reality and virtuality. Young children often fantasize about flying into the pages of a fairy tale and becoming part of the story. The MagicBook project makes this fantasy a reality using a normal book as the main interface object. People can turn the pages of the book, look at the pictures, and read the text without any additional technology. However, if a person looks at the pages through an augmented reality display, they see 3D virtual models appearing out of the pages. The models appear attached to the real page so users can see the augmented reality scene from any perspective by moving themselves or the book. The virtual content can be any size and is animated, so the augmented reality view is an enhanced version of a traditional 3D pop-up book.     

Scape: supporting stereoscopic collaboration in augmented and projective environments

Our collaborative augmented reality system for multiple users simultaneously creates outside-in workbench views and inside-out life-size walkthrough views, bridging the virtual and augmented reality paradigms. Here we introduce a collaborative infrastructure - stereoscopic collaboration in augmented and projective environments(Scape). Scape seamlessly merges the paradigm of virtual reality with that of augmented reality in a single system and provides multimodality interface devices and unique widget interfaces to facilitate collaborative work in 3D environments. A custom-designed cross-platform application-programming interface (API) allows high and medium-level controls over the Scape workspace to enable augmented interaction and collaboration.     

Augmented reality using personal projection and retroreflection

The support of realistic and flexible training simulations for military, law enforcement, emergency response, and other domains has been an important motivator for the development of augmented reality technology. An important vision for achieving this goal has been the creation of a versatile “stage” for physical, emotional, and cognitive training that combines virtual characters and environments with real world elements, such as furniture and props. This paper presents REFLCT, a mixed reality projection framework that couples a near-axis personal projector design with tracking and novel retroreflective props and surfaces. REFLCT provides multiple users with personalized, perspective-correct imagery that is uniquely composited for each user directly into and onto a surrounding environment, without any optics positioned in front of the user’s eyes or face. These characteristics facilitate team training experiences which allow users to easily interact with their teammates while wearing their standard issue gear. REFLCT can present virtual humans who can make deictic gestures and establish eye contact without the geometric ambiguity of a typical projection display. It can also display perspective-correct scenes that require a realistic approach for detecting and communicating potential threats between multiple users in disparate locations. In addition to training applications, this display system appears to be well matched with other user interface and application domains, such as asymmetric collaborative workspaces and personal information guides.     

Design and implementations of Ninf: towards a global computing infrastructure

The world-wide computing infrastructure on the growing computer network technology is a leading technology to make a variety of information services accessible through the Internet for every user from the high-performance computing users through many of personal computing users. The important feature of such services is location transparency; information can be obtained irrespective of time or location in virtually shared manner. In this article, we overview Ninf, an ongoing global network-wide computing infrastructure project which allows users to access computational resources including hardware, software and scientific data distributed across a wide area network. Preliminary performance result on measuring software and network overhead is shown, and that promises the future reality of world-wide network computing.