A rapid prototyping software infrastructure for user interfaces in ubiquitous augmented reality

Recent user interface concepts, such as multimedia, multimodal, wearable, ubiquitous, tangible, or augmented-reality-based (AR) interfaces, each cover different approaches that are all needed to support complex human–computer interaction. Increasingly, an overarching approach towards building what we call ubiquitous augmented reality (UAR) user interfaces that include all of the just mentioned concepts will be required. To this end, we present a user interface architecture that can form a sound basis for combining several of these concepts into complex systems. We explain in this paper the fundamentals of DWARFs user interface framework (DWARF standing for distributed wearable augmented reality framework) and an implementation of this architecture. Finally, we present several examples that show how the framework can form the basis of prototypical applications.     

Multimodal augmented reality tangible gaming

This paper presents tangible augmented reality gaming environment that can be used to enhance entertainment using a multimodal tracking interface. Players can interact using different combinations between a pinch glove, a Wiimote, a six-degrees-of-freedom tracker, through tangible ways as well as through I/O controls. Two tabletop augmented reality games have been designed and implemented including a racing game and a pile game. The goal of the augmented reality racing game is to start the car and move around the track without colliding with either the wall or the objects that exist in the gaming arena. Initial evaluation results showed that multimodal-based interaction games can be beneficial in gaming. Based on these results, an augmented reality pile game was implemented with goal of completing a circuit of pipes (from a starting point to an end point on a grid). Initial evaluation showed that tangible interaction is preferred to keyboard interaction and that tangible games are much more enjoyable.     

Distributed Augmented Reality for Collaborative Design Applications

This paper presents a system for constructing collaborative design applications based on distributed augmented reality. Augmented reality interfaces are a natural method for presenting computer-based design by merging graphics with a view of the real world. Distribution enables users at remote sites to collaborate on design tasks. The users interactively control their local view, try out design options, and communicate design proposals. They share virtual graphical objects that substitute for real objects which are not yet physically created or are not yet placed into the real design environment. We describe the underlying augmented reality system and in particular how it has been extended in order to support multi-user collaboration. The construction of distributed augmented reality applications is made easier by a separation of interface, interaction and distribution issues. An interior design application is used as an example to demonstrate the advantages of our approach.     

Ontology-Based User Modeling in an Augmented Audio Reality System for Museums

Ubiquitous computing is a challenging area that allows us to further our understanding and techniques of context-aware and adaptive systems. Among the challenges is the general problem of capturing the larger context in interaction from the perspective of user modeling and human–computer interaction (HCI). The imperative to address this issue is great considering the emergence of ubiquitous and mobile computing environments. This paper provides an account of our addressing the specific problem of supporting functionality as well as the experience design issues related to museum visits through user modeling in combination with an audio augmented reality and tangible user interface system. This paper details our deployment and evaluation of ec(h)o – an augmented audio reality system for museums. We explore the possibility of supporting a context-aware adaptive system by linking environment, interaction objects and users at an abstract semantic level instead of at the content level. From the user modeling perspective ec(h)o is a knowledge-based recommender system. In this paper we present our findings from user testing and how our approach works well with an audio and tangible user interface within a ubiquitous computing system. We conclude by showing where further research is needed.     

Two-handed tangible interaction techniques for composing augmented blocks

Modeling tools typically have their own interaction methods for combining virtual objects. For realistic composition in 3D space, many researchers from the fields of virtual and augmented reality have been trying to develop intuitive interactive techniques using novel interfaces. However, many modeling applications require a long learning time for novice users because of unmanageable interfaces. In this paper, we propose two-handed tangible augmented reality interaction techniques that provide an easy-to-learn and natural combination method using simple augmented blocks. We have designed a novel interface called the cubical user interface, which has two tangible cubes that are tracked by marker tracking. Using the interface, we suggest two types of interactions based on familiar metaphors from real object assembly. The first, the screw-driving method, recognizes the user??s rotation gestures and allows them to screw virtual objects together. The second, the block-assembly method, adds objects based on their direction and position relative to predefined structures. We evaluate the proposed methods in detail with a user experiment that compares the different methods.     

Scape: supporting stereoscopic collaboration in augmented and projective environments

Our collaborative augmented reality system for multiple users simultaneously creates outside-in workbench views and inside-out life-size walkthrough views, bridging the virtual and augmented reality paradigms. Here we introduce a collaborative infrastructure - stereoscopic collaboration in augmented and projective environments(Scape). Scape seamlessly merges the paradigm of virtual reality with that of augmented reality in a single system and provides multimodality interface devices and unique widget interfaces to facilitate collaborative work in 3D environments. A custom-designed cross-platform application-programming interface (API) allows high and medium-level controls over the Scape workspace to enable augmented interaction and collaboration.     

Physical query interface for tangible augmented tagging and interaction

A new physical query interface is proposed for tangible augmented tagging and interaction using two types of context-aware AR interfaces, visual and invisible interfaces. In visual interfaces, augmented reality markers are utilized for supporting intuitive interactions and manipulations with digital contents. RFID is used as an invisible interface for supporting line-of-sight, direct and query interactions. By combining the advantages of visual and invisible interfaces, more natural interaction with digital contents can be provided, which can remove the difficulty of using typical AR paddles, cubes and gadgets that are widely used in AR interactions. Semantic ontology is adopted in order to effectively support physical querying, filtering and grouping through the combination of multiple RFID tags. The proposed approach has been applied to a variety of multi-media interactions, such as digital product review and digital catalog or book interaction. We show the effectiveness and the advantages of the proposed approach by demonstrating several implementation results and by a usability study.     

INTERVALES: INTERactive Virtual and Augmented framework for industriaL Environment and Scenarios

One of Industry 4.0’s greatest challenges for companies is the digitization of their processes and the integration of new related technologies such as virtual reality (VR) and augmented reality (AR), which can be used for training purposes, design, or assistance during industrial operations. Moreover, recent results and industrial proofs of concept show that these technologies demonstrate critical advantages in the industry. Nevertheless, the authoring and editing process of virtual and augmented content remains time-consuming, especially in complex industrial scenarios. While the use of interactive virtual environments through virtual and augmented reality presents new possibilities for many domains, a wider adoption of VR/AR is possible only if the authoring process is simplified, allowing for more rapid development and configuration without the need for advanced IT skills. To meet this goal, this study presents a new framework: INTERVALES. First, framework architecture is proposed, along with its different modules; this study then shows that the framework can be updated by not only IT workers, but also other job experts. The UML data model is presented to format and simplify the authoring processes for both VR and AR. This model takes into account virtual and augmented environments, the possible interactions, and ease operations orchestration. Finally, this paper presents the implementation of an industrial use case composed of collaborative robotic (cobotic) and manual assembly workstations in VR and AR based on INTERVALES data.     

Configuring and controlling ubiquitous computing infrastructure with semantic connections: a tangible and an AR approach

In the transition from a device-oriented paradigm toward a more task-oriented paradigm with increased interoperability, people are struggling with inappropriate user interfaces, competing standards, technical incompatibilities, and other difficulties. The current handles for users to explore, make, and break connections between devices seem to disappear in overly complex menu structures displayed on small screens. This paper tackles the problem of establishing connections between devices in a smart home environment, by introducing an interaction model that we call semantic connections. Two prototypes are demonstrated that introduce both a tangible and an augmented reality approach toward exploring, making, and breaking connections. In the augmented reality approach, connections between real-world objects are visualized by displaying visible lines and icons from a mobile device containing a pico projector. In the tangible approach, objects in the environment are tagged and can be scanned and interacted with, to explore connection possibilities, and manipulate the connections. We discuss the technical implementation of a pilot study setup used to evaluate both our interaction approaches. We conclude the paper with the results of a user study that shows how the interaction approaches influence the mental models users construct after interacting with our setup.     

Building and Employing Cross-Reality

Considers present and future practical applications of cross-reality. From tools to build new 3D virtual worlds to the products of those tools, cross-reality is becoming a staple of our everyday reality. Practical applications of cross-reality include the ability to virtually visit a factory to manage and maintain resources from the comfort of your laptop or desktop PC as well as sentient visors that augment reality with additional information so that users can make more informed choices. Tools and projects considered are:Project Wonderland for multiuser mixed reality;ClearWorlds: mixed- reality presence through virtual clearboards; VICI (Visualization of Immersive and Contextual Information) for ubiquitous augmented reality based on a tangible user interface; Mirror World Chocolate Factory; and sentient visors for browsing the world.     

MagicMeeting: A Collaborative Tangible Augmented Reality System

We describe an augmented reality (AR) system that allows multiple participants to interact with 2D and 3D data using tangible user interfaces. The system features face-to-face communication, collaborative viewing and manipulation of 3D models, and seamless access to 2D desktop applications within the shared 3D space. All virtual content, including 3D models and 2D desktop windows, is attached to tracked physical objects in order to leverage the efficiencies of natural two-handed manipulation. The presence of 2D desktop space within 3D facilitates data exchange between the two realms, enables control of 3D information by 2D applications, and generally increases productivity by providing access to familiar tools. We present a general concept for a collaborative tangible AR system, including a comprehensive set of interaction techniques, a distributed hardware setup, and a component-based software architecture that can be flexibly configured using XML. We show the validity of our concept with an implementation of an application scenario from the automotive industry.     

Multi-interfaces approach to situated knowledge management for complex instruments: first step toward industrial deployment

This paper presents an approach to managing knowledge specific to a particular location for complex instruments. The goal is to improve the knowledge communication between experts and end-users of scientific instruments. We propose a computational framework that integrates augmented reality and augmented virtuality as interface for manipulating knowledge. The augmented virtuality-based interface can be produced and distributed without extra costs. It allows knowledge dissemination at a larger scale. The prominent feature of our model is that the knowledge representation is independent from those interfaces. A preliminary version of our framework has been implemented and deployed in customers’ environments.     

Usability engineering for augmented reality: employing user-based studies to inform design

A major challenge, and thus opportunity, in the field of human-computer interaction and specifically usability engineering is designing effective user interfaces for emerging technologies that have no established design guidelines or interaction metaphors or introduce completely new ways for users to perceive and interact with technology and the world around them. Clearly, augmented reality is one such emerging technology. We propose a usability engineering approach that employs user-based studies to inform design, by iteratively inserting a series of user-based studies into a traditional usability engineering lifecycle to better inform initial user interface designs. We present an exemplar user-based study conducted to gain insight into how users perceive text in outdoor augmented reality settings and to derive implications for design in outdoor augmented reality. We also describe lessons learned from our experiences conducting user-based studies as part of the design process.     

TUIs vs. GUIs: comparing the learning potential with preschoolers

In an effort to better understand the learning potential of a tangible interface, we conducted a comparison study between a tangible and a traditional graphical user interface for teaching preschoolers (In Portugal, children enter preschool at the age of three and they attend it till entering school, normally at the age of six) about good oral hygiene. The study was carried with two groups of children aged 4 to 5 years. Questionnaires to parents, children’s drawings, and interviews were used for data collection and analysis and revealed important indicators about children’s change of attitude, involvement, and preferences for the interfaces. The questionnaires showed a remarkable change of attitude toward tooth brushing in the children that interacted with the tangible interface; particularly children’s motivation increased significantly. Children’s drawings were used to assess their degree of involvement with the interfaces. The drawings from the children that interacted with the tangible interface were very complete and detailed suggesting that the children felt actively involved with the experience. The results suggest that the tangible interface was capable of promoting a stronger and long-lasting involvement having a greater potential to engage children, therefore potentially promoting learning. Evaluation through drawing seems to be a promising method to work with preliterate children; however, it is advisable to use it together with other methods.     

Usability Principles for Augmented Reality Applications in a Smartphone Environment

Through the rapid spread of smartphones, users have access to many types of applications similar to those on desktop computer systems. Smartphone applications using augmented reality (AR) technology make use of users' location information. As AR applications will require new evaluation methods, improved usability and user convenience should be developed. The purpose of the current study is to develop usability principles for the development and evaluation of smartphone applications using AR technology. We develop usability principles for smartphone AR applications by analyzing existing research about heuristic evaluation methods, design principles for AR systems, guidelines for handheld mobile device interfaces, and usability principles for the tangible user interface. We conducted a heuristic evaluation for three popularly used smartphone AR applications to identify usability problems. We suggested new design guidelines to solve the identified problems. Then, we developed an improved AR application prototype of an Android-based smartphone, which later was conducted a usability testing to validate the effects of usability principles.     

Registration of technical drawings and calibrated images for industrial augmented reality

Despite tremendous progress in 3D modelling technology, most sites in traditional industries do not have a computer model of their facilities at their disposal. In these industries, 2D technical drawings are typically the most commonly used documents. In many cases, a database of fully calibrated and oriented photogrammetric images of parts of the plant is also available. These images are often used for metric measurement and 3D as-built modelling. For planning revamps and maintenance, it is necessary to use industrial drawings as well as images and 3D models represented in a common “world” coordinate system. This paper proposes a method for full integration of technical drawings, calibrated images and as-built 3D models. A new algorithm is developed in order to use only a few correspondences between points on a technical drawing and multiple images to estimate a metric planar transformation between the drawing and the world coordinate system. The paper describes the mathematical relationship between this transformation and the set of homographies needed for merging the technical drawing with all the calibrated images. The method is implemented and fully integrated into an industrial software we developed for 3D as-built reconstruction. We present examples of a real application, in which the method is successfully applied to create an augmented reality representation of a waste water plant. Accepted: 13 August 2001     

HCI and business practices in a collaborative method for augmented reality systems

ContextEvery interactive system is composed of a functional core and a user interface. However, the software engineering (SE) and human–computer interaction (HCI) communities do not share the same methods, models or tools. This usually induces a large work overhead when specialists from the two domains try to connect their applicative studies, especially when developing augmented reality systems that feature complex interaction cores.ObjectiveWe present in this paper the essential activities and concepts of a development method integrating the SE and HCI development practices, from the specifications down to the design, as well as their application on a case study.MethodThe efficiency of the method was tested in a qualitative study involving four pairs of SE and HCI experts in the design of an application for which an augmented reality interaction would provide better user performance than a classic interactive system. The effectivity of the method was evaluated in a qualitative study comparing the quality of three implementations of the same application fragment (based on the same analysis model), using software engineering metrics.ResultsThe first evaluation confirmed the ease of use of our method and the relevance of our tools for guiding the design process, but raised concerns on the handling of conflicting collaborative activities. The second evaluation gave indications that the structure of the analysis model facilitates the implementation of quality software (in terms of coupling, stability and complexity).ConclusionIt is concluded that our method enables design teams with different backgrounds in application development to collaborate for integrating augmented reality applications with information systems. Areas of improvement are also described.     

Magic Lenses for augmented virtual environments

The authors developed a Magic Lens framework for Scape, an augmented virtual environment (AVE). They generalize the functional characteristics of Magic Lenses in terms of 3D visualization in AVEs and present two tangible Magic Lens-enabled devices with complementary interface capabilities. The authors also demonstrate their Magic Lens devices through testbed applications relevant to urban planning and medical training. This article explores an interface technique that manages viewing complexity and offers a set of versatile visualization capabilities in 3D augmented virtual environments (AVE).     

Communication Behaviors in Colocated Collaborative AR Interfaces

The authors present an analysis of communication behavior in face-to-face collaboration using a multi-user augmented reality (AR) interface. 2 experiments were conducted. In the 1st experiment, collaboration with AR technology was compared with more traditional unmediated and screen-based collaboration. In the 2nd experiment, the authors compared collaboration with 3 different AR displays. Several measures were used to analyze communication behavior, and the authors found that users exhibited many of the same behaviors in a collaborative AR interface as in face-to-face unmediated collaboration. However, user communication behavior changed with the type of AR display used. The authors describe implications of these results for the design of collaborative AR interfaces and directions for future research.     

空间增强现实中的人机交互技术综述

空间增强现实是增强现实的三大形式之一,它利用投影等方式在物理空间中增强显示虚拟内容.在空间增强现实系统中,用户可以在物理空间与虚拟内容交互,具有良好的环境感知、空间感知与协同感知,交互任务执行过程中认知负荷低.首先以空间增强现实显示方式为例,通过其系统架构介绍空间增强现实的技术原理;然后将空间增强现实中的人机交互技术分...