3D modeling with the Tinmith mobile outdoor augmented reality system

At the Wearable Computer Lab at the University of South Australia, we have been performing research into outdoor augmented reality (AR) systems for the last seven years. During this time the technology has vastly improved, resulting in more accurate systems that have better quality output. While tracking and registration are important issues in the area of AR, it's also important that we have suitable user interfaces that let people effectively view information and control the computer to get the desired output. Therefore, the Tinmith project explores the problem of interacting with a mobile AR system outdoors and the types of possible applications.     

Information filtering for mobile augmented reality

The use of augmented reality (AR) techniques can revolutionize the way people interact with unfamiliar environments. By tracking the user's position and orientation, complicated spatial information can be registered against the real world. My colleagues and I are researching the problem of developing mobile AR systems to be worn by individual users operating in large, complicated environments such as cities. However, an urban environment is extremely complicated. It is populated by large numbers of buildings, each of which can have numerous facts stored about it. Therefore, it is easy for a user to experience information overload. This problem is illustrated. To minimize problems of information overload, we have begun to develop algorithms for information filtering. These tools automatically restrict the amount of information displayed     

Accurate overlaying for mobile augmented reality

Mobile augmented reality requires accurate alignment of virtual information with objects visible in the real world. We describe a system for mobile communications to be developed to meet these strict alignment criteria using a combination of computer vision, inertial tracking and low-latency rendering techniques. A prototype low-power and low-latency renderer using an off-the-shelf 3D card is discussed.     

Tangible authoring of 3D virtual scenes in dynamic augmented reality environment

This paper proposes tangible interfaces and interactions for authoring 3D virtual and immersive scenes easily and intuitively in tangible augmented reality (AR) environment. It provides tangible interfaces for manipulating virtual objects in a natural and intuitive manner and supports adaptive and accurate vision-based tracking in AR environments. In particular, RFID is used to directly integrate physical objects with virtual objects and to systematically support the tangible query of the relation between physical objects and virtual ones, which can provide more intuitive tangibility and a new way of virtual object manipulation. Moreover, the proposed approach offers an easy and intuitive switching mechanism between tangible environment and virtual environment. This paper also proposes a context-adaptive marker tracking method which can remove an inconsistent problem while embedding virtual objects into physical ones in tangible AR environments. The context-adaptive tracking method not only adjusts the locations of invisible markers by interpolating the locations of existing reference markers and those of previous ones, but also removes a jumping effect of movable virtual objects when their references are changed from one marker to another. Several case studies for generating tangible virtual scenes and comparison with previous work are given to show the effectiveness and novelty of the proposed approach.     

An experimental study on the role of 3D sound in augmented reality environment

Investigation of augmented reality (AR) environments has become a popular research topic for engineers, computer and cognitive scientists. Although application oriented studies focused on audio AR environments have been published, little work has been done to vigorously study and evaluate the important research questions of the effectiveness of three-dimensional (3D) sound in the AR context, and to what extent the addition of 3D sound would contribute to the AR experience.

Thus, we have developed two AR environments and performed vigorous experiments with human subjects to study the effects of 3D sound in the AR context. The study concerns two scenarios. In the first scenario, one participant must use vision only and vision with 3D sound to judge the relative depth of augmented virtual objects. In the second scenario, two participants must cooperate to perform a joint task in a game-based AR environment.

Hence, the goals of this study are (1) to access the impact of 3D sound on depth perception in a single-camera AR environment, (2) to study the impact of 3D sound on task performance and the feeling of ‘human presence and collaboration’, (3) to better understand the role of 3D sound in human–computer and human–human interactions, (4) to investigate if gender can affect the impact of 3D sound in AR environments. The outcomes of this research can have a useful impact on the development of audio AR systems, which provide more immersive, realistic and entertaining experiences by introducing 3D sound. Our results suggest that 3D sound in AR environment significantly improves the accuracy of depth judgment and improves task performance. Our results also suggest that 3D sound contributes significantly to the feeling of human presence and collaboration and helps the subjects to ‘identify spatial objects’.     

A generalized registration method for augmented reality systems

In augmented reality systems, registration is one of the most difficult problems currently limiting their applications. In this paper, we propose a generalized registration method using projective reconstruction technique in computer vision. This registration method is composed of embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In this stage, any arbitrary two unrelated images or any 3×4 projective matrices with rank 3 can be used to calculate the 3D pseudo-projective coordinates of the four specified points. In the tracking process, these 3D pseudo-projective coordinates are used to track the four specified points to compute the registration matrix for augmentation. The proposed method is simple, as only four points need to be specified at the embedding stage, and the virtual object can then be easily augmented onto a real scene from a video sequence. One advantage is that the virtual objects can still be superimposed on the specified regions even when the regions are occluded in the video sequence. Another advantage of the proposed method is that the registration errors can be adjusted in real-time to ensure that they are less than certain thresholds that have been specified at the initial embedding stage. Several experiments have been conducted to validate the performance of the proposed generalized method.     

Gesture-based interaction via finger tracking for mobile augmented reality

The goal of this research is to explore new interaction metaphors for augmented reality on mobile phones, i.e. applications where users look at the live image of the device’s video camera and 3D virtual objects enrich the scene that they see. Common interaction concepts for such applications are often limited to pure 2D pointing and clicking on the device’s touch screen. Such an interaction with virtual objects is not only restrictive but also difficult, for example, due to the small form factor. In this article, we investigate the potential of finger tracking for gesture-based interaction. We present two experiments evaluating canonical operations such as translation, rotation, and scaling of virtual objects with respect to performance (time and accuracy) and engagement (subjective user feedback). Our results indicate a high entertainment value, but low accuracy if objects are manipulated in midair, suggesting great possibilities for leisure applications but limited usage for serious tasks.     

A distributed device diagnostics system utilizing augmented reality and 3D audio

Augmented reality (AR), combining virtual environments with the perception of the real world, can be used to provide instructions for routine maintenance and error diagnostics of technical devices. The Rockwell Science Center (RSC) is developing a system that utilizes AR techniques to provide “X-ray vision” into real objects. The system can overlay 3D rendered objects, animations, and text annotations onto the video image of a known object. An automated speech recognition system allows the user to query the status of device components. The response is given as an animated rendition of a CAD model and/or as auditory cues using 3D audio. This diagnostics system also allows the user to leave spoken annotations attached to device modules as ASCII text. The position of the user/camera relative to the device is tracked by a computer-vision-based tracking system using fiducial markers. The system is implemented on a distributed network of PCs, utilizing standard commercial off-the-shelf components (COTS).     

Tracking in unprepared environments for augmented reality systems

Many augmented reality applications require accurate tracking. Existing tracking techniques require prepared environments to ensure accurate results. This paper motivates the need to pursue augmented reality tracking techniques that work in unprepared environments, where users are not allowed to modify the real environment, such as in outdoor applications. Accurate tracking in such situations is difficult, requiring hybrid approaches. This paper summarizes two 3DOF results: a real-time system with a compass — inertial hybrid, and a non-real-time system fusing optical and inertial inputs. We then describe the preliminary results of 5- and 6-DOF tracking methods run in simulation. Future work and limitations are described.     

Registration using natural features for augmented reality systems

Registration is one of the most difficult problems in augmented reality (AR) systems. In this paper, a simple registration method using natural features based on the projective reconstruction technique is proposed. This method consists of two steps: embedding and rendering. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In rendering, the Kanade-Lucas-Tomasi (KLT) feature tracker is used to track the natural feature correspondences in the live video. The natural features that have been tracked are used to estimate the corresponding projective matrix in the image sequence. Next, the projective reconstruction technique is used to transfer the four specified points to compute the registration matrix for augmentation. This paper also proposes a robust method for estimating the projective matrix, where the natural features that have been tracked are normalized (translation and scaling) and used as the input data. The estimated projective matrix will be used as an initial estimate for a nonlinear optimization method that minimizes the actual residual errors based on the Levenberg-Marquardt (LM) minimization method, thus making the results more robust and stable. The proposed registration method has three major advantages: 1) It is simple, as no predefined fiducials or markers are used for registration for either indoor and outdoor AR applications. 2) It is robust, because it remains effective as long as at least six natural features are tracked during the entire augmentation, and the existence of the corresponding projective matrices in the live video is guaranteed. Meanwhile, the robust method to estimate the projective matrix can obtain stable results even when there are some outliers during the tracking process. 3) Virtual objects can still be superimposed on the specified areas, even if some parts of the areas are occluded during the entire process. Some indoor and outdoor experiments have been conducted to validate the performance of this proposed method.     

Experimental research into human cognitive processing in an augmented reality environment for embedded training systems

Research into human factors issues associated with the use of augmented reality (AR) technology is very limited. Consequently, there is a need for formal human factors design guidelines to underpin the integration of AR into systems. The Defence Evaluation and Research Agency (DERA) Centre for Human Sciences (CHS) is evaluating the potential of AR for providing real-time training feedback in future advanced embedded training systems for the military. In order to understand the important human factors issues of augmented reality, DERA funded the Advanced VR Research Centre (AVRRC) at Loughborough University to investigate the cognitive ergonomics of this technology. An important aspect of this research is concerned with identifying any human information processing issues that may arise when information is presented via AR and overlaid upon one or more primary display surfaces such as a visual display unit (VDU). Two main issues are addressed in this research. First, the impact of AR on human information processing and second, subjective workload experienced when displaying information via the AR medium. The experiments reported in this paper assess issues of reaccommodation and reaction times to alarms on different display formats. They demonstrate also that AR performs as well as standard display formats.