Evaluating the effectiveness of augmented reality displays for a manual assembly task

The focus of this research was to examine how effectively augmented reality displays, generated with a wearable computer, could be used for aiding an operator performing a manual assembly task. Fifteen subjects were asked to assemble a computer motherboard using four types of instructional media: paper manual, computer-aided, opaque augmented reality display, and see-through augmented reality display. The time of assembly and assembly errors were measured for each type of instructional media, and a questionnaire focusing on usability was administered to each subject at the end of each condition. The results of the experiment indicated that the augmented reality conditions were more effective instructional aids for the assembly task than either the paper instruction manual or the computer-aided instruction. The see-through augmented reality display resulted in the fastest assembly times, followed by the opaque augmented reality display, the computer-aided instruction, and the paper instructions respectively. In addition, subjects made fewer errors using the augmented reality conditions compared to the computer-aided and paper instructional media. However, while the two augmented reality conditions were a more effective instructional media when time for assembly was the response measure, there were still some important usability issues associated with the augmented reality technology that were not present in the non-augmented reality conditions.     

Tracking for augmented reality on wearable computers

Wearable computers afford a degree of mobility that makes tracking for augmented reality difficult. This paper presents a novel object-centric tracking architecture for presenting augmented reality media in spatial relationships to objects, regardless of the objects' positions or motions in the world. The advance this system provides is the ability to sense and integrate new features into its tracking database, thereby extending the tracking region automatically. A lazy evaluation of the structure from motion problem uses images obtained from a single calibrated moving camera and applies recursive filtering to identify and estimate the 3D positions of new features. We evaluate the performance of two filters; a classic Extended Kalman Filter (EKF) and a filter based on a Recursive-Average of Covariances (RAC). Some implementation issues and results are discussed in conclusion.     

An experimental study on augmented reality assisted manual assembly with occluded components

Augmented Reality (AR) technology has increasingly been applied to facilitate manufacturing tasks such as training, maintenance, and safety management, in which manual assembly frequently occurs. Previously, studies confirmed the values of AR-based assembly systems, but few explored support of spatially restricted assembly where components are visually occluded from operator. In this regard, this research aims to develop new AR functions that assist manual assembly in such situations. The focus is on validating the effectiveness of various assistive information in AR, including assembly interface, operator’s hand movement, and operator held components. Subjective and objective measures are used to evaluate assembly experiments of different degrees of difficulty. Analyses of the experimental data reveal whether or not and how effectively each information offers guidance in occluded condition. In particular, a time reduction in more difficult assembly is realized by showing operator hand movement in AR. The hand model initially offers an operator a visual clue for quickly and roughly locating the assembly interface in a large unseen area, prior to precisely localizing in a smaller region guided by tactile sensing. However, the effectiveness of incorporating the held component is not evident, as positional deviation between real and virtual objects may reduce human’s hand-eye coordination. These findings not only provide preliminary design guidelines of AR assembly functions for occluded components but also demonstrates a novel yet practical application of AR technology in smart manufacturing.     

Optically based direct manipulation for augmented reality

Augmented reality (AR) constitutes a very powerful three-dimensional user interface for many “hands-on” application scenarios. To fully exploit the AR paradigm, the computer must not only augment the real world, but also accept feedback from it. In this paper, we present an optical approach for collecting such feedback by analyzing video sequences to track users and the objects they work with. Our system can be set up in any room after quickly placing a few known optical targets in the scene. We present two demonstration scenarios to illustrate the overall concept and potential of our approach and then discuss the research issues involved.     

The effects of augmented reality on improving spatial problem solving for object assembly

The capability of Augmented Reality (AR) technology to track and visualize relations of objects in space has led to diverse industry applications to support complex engineering tasks. Object assembly is one of them. For an AR aid to support Ready-to-Assemble (RTA) furniture particularly, the challenge is to effectively design the visual features and mode of interaction, so that the first-time users can quickly conceive spatial relations of its parts. However, AR developers and engineers do not have sufficient guidelines to achieve such performance-driven goals. The scientific evidence and account of how one could cognitively benefit in object assembly can be useful to guide them. This experimental research developed an Augmented Reality (AR) application on the Microsoft HoloLens™ headset, and tested it on the first-time users of RTA furniture. The controlled experiments and behavioral analyses of fourteen participants in working out the two RTA furniture with different assembly complexity showed that, the application was effective to improve spatial problem-solving abilities. Especially, the positive effects of the AR-based supports stood out against the assembly of higher complexity. The findings have implications for the information design of advanced AR applications to support assembly tasks with high demands of spatial knowledge. In addition, the extensive analyses of the participants’ performance, behaviors, cognitive workload, and subjective responses helped identify usability issues with the Microsoft HoloLens™.     

Accurate overlaying for mobile augmented reality

Mobile augmented reality requires accurate alignment of virtual information with objects visible in the real world. We describe a system for mobile communications to be developed to meet these strict alignment criteria using a combination of computer vision, inertial tracking and low-latency rendering techniques. A prototype low-power and low-latency renderer using an off-the-shelf 3D card is discussed.     

A generalized registration method for augmented reality systems

In augmented reality systems, registration is one of the most difficult problems currently limiting their applications. In this paper, we propose a generalized registration method using projective reconstruction technique in computer vision. This registration method is composed of embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In this stage, any arbitrary two unrelated images or any 3×4 projective matrices with rank 3 can be used to calculate the 3D pseudo-projective coordinates of the four specified points. In the tracking process, these 3D pseudo-projective coordinates are used to track the four specified points to compute the registration matrix for augmentation. The proposed method is simple, as only four points need to be specified at the embedding stage, and the virtual object can then be easily augmented onto a real scene from a video sequence. One advantage is that the virtual objects can still be superimposed on the specified regions even when the regions are occluded in the video sequence. Another advantage of the proposed method is that the registration errors can be adjusted in real-time to ensure that they are less than certain thresholds that have been specified at the initial embedding stage. Several experiments have been conducted to validate the performance of the proposed generalized method.     

The visual impact of augmented reality during an assembly task

Many studies have shown the benefits of augmented reality (AR) to improve manufacturing and control processes in industry. However, the presentation of AR content through optical see-through head-mounted displays may induce unnatural viewing conditions, which consequences on the user’s visual system have not been investigated yet. This study aimed at assessing whether using AR instructions to guide a manual task, i.e., conditions where the user is forced to repeatedly look at and accommodate in different planes to extract information from both real and virtual environments, could potentially impact the visual system of operators. A before/after design study was carried out, asking 26 participants to perform Lego assemblies for 30 min with either paper or AR instructions. The effects of using AR compared to paper instructions were evaluated both on binocular vision, with classical optometric measurements, and on visual fatigue, with the Virtual Reality Symptoms Questionnaire. No clinically significant differences in optometric measurements (far/near visual acuity, stereoacuity, phoria, convergence, fusional amplitude, accommodation amplitude, accommodative convergence) have been found between AR and paper instructions, and only negligible fatigue symptoms have been seen specifically for AR. Results from both objective and subjective measurements suggest that there is no impact of AR on the oculomotor system and that, in this specific case of use, AR can be safely used for production operators.     

Human acceptance evaluation of AR-assisted assembly scenarios

The contribution examines the acceptance of Augmented Reality (AR) in assembly scenarios by a model-based approach for acceptance evaluation. After a critical literature research and analysis, a proprietary model for acceptance measurement is developed, which includes and synthesizes previous models and simplifies them considerably for the purpose of industrial assembly. Consequently, a structural model is derived which is based on the basic concepts of the Unified Theory of Acceptance and Use of Technology (UTAUT) and Technology Acceptance Model (TAM). This is built up as a path model to explain user acceptance. Following, a laboratory experiment is set up to collect data. At the final assembly workstation of the FHWS c-Factory the study participants assemble a toy truck once without and once with AR support. The c-Factory is a concept factory for smart production in an IoT environment. Afterwards, relevant data are collected by means of a survey. The evaluation shows that the acceptance of AR is given and the model is resilient. The results show also, that AR is accepted by the participants supporting their work.     

A topometric system for wide area augmented reality

We describe a system designed to facilitate efficient communication of information relating to the physical world using augmented reality (AR). We bring together a range of technologies to create a system capable of operating in real-time, over wide areas and for both indoor and outdoor operations. The central concept is to integrate localised mapping and tracking based on real-time visual SLAM with global positioning from both GPS and indoor ultra-wide band (UWB) technology. The former allows accurate and repeatable creation and visualisation of AR annotations within local metric maps, whilst the latter provides a coarse global representation of the topology of the maps. We call this a ‘Topometric System’. The key elements are: robust and efficient vision based tracking and mapping using a Kalman filter framework; rapid and reliable vision based relocalisation of users within local maps; user interaction mechanisms for effective annotation insertion; and an integrated framework for managing and fusing mapping and positioning data. We present the results of experiments conducted over a wide area, with indoor and outdoor operations, which demonstrates successful creation and visualisation of large numbers of AR annotations over a range of different locations.     

An experimental study on the role of 3D sound in augmented reality environment

Investigation of augmented reality (AR) environments has become a popular research topic for engineers, computer and cognitive scientists. Although application oriented studies focused on audio AR environments have been published, little work has been done to vigorously study and evaluate the important research questions of the effectiveness of three-dimensional (3D) sound in the AR context, and to what extent the addition of 3D sound would contribute to the AR experience.

Thus, we have developed two AR environments and performed vigorous experiments with human subjects to study the effects of 3D sound in the AR context. The study concerns two scenarios. In the first scenario, one participant must use vision only and vision with 3D sound to judge the relative depth of augmented virtual objects. In the second scenario, two participants must cooperate to perform a joint task in a game-based AR environment.

Hence, the goals of this study are (1) to access the impact of 3D sound on depth perception in a single-camera AR environment, (2) to study the impact of 3D sound on task performance and the feeling of ‘human presence and collaboration’, (3) to better understand the role of 3D sound in human–computer and human–human interactions, (4) to investigate if gender can affect the impact of 3D sound in AR environments. The outcomes of this research can have a useful impact on the development of audio AR systems, which provide more immersive, realistic and entertaining experiences by introducing 3D sound. Our results suggest that 3D sound in AR environment significantly improves the accuracy of depth judgment and improves task performance. Our results also suggest that 3D sound contributes significantly to the feeling of human presence and collaboration and helps the subjects to ‘identify spatial objects’.     

Smart augmented reality instructional system for mechanical assembly towards worker-centered intelligent manufacturing

Quality and efficiency are crucial indicators of any manufacturing company. Many companies are suffering from a shortage of experienced workers across the production line to perform complex assembly tasks. To reduce time and error in an assembly task, a worker-centered system consisting of multi-modal Augmented Reality (AR) instructions with the support of a deep learning network for tool detection is introduced. The integrated AR is designed to provide on-site instructions including various visual renderings with a fine-tuned Region-based Convolutional Neural Network, which is trained on a synthetic tool dataset. The dataset is generated using CAD models of tools and displayed onto a 2D scene without using real tool images. By experimenting the system to a mechanical assembly of a CNC carving machine, the result of a designed experiment shows that the system helps reduce the time and errors of the given assembly tasks by 33.2 % and 32.4 %, respectively. With the integrated system, an efficient, customizable smart AR instruction system capable of sensing, characterizing requirements, and enhancing worker’s performance has been built and demonstrated.     

Exploring augmented reality for worker assistance versus training

This paper aims at advancing the fundamental understanding of the affordances of Augmented Reality (AR) as a workplace-based learning and training technology in supporting manual or semi-automated manufacturing tasks that involve both complex manipulation and reasoning. Between-subject laboratory experiments involving 20 participants are conducted on a real-life electro-mechanical assembly task to investigate the impacts of various modes of information delivery through AR compared to traditional training methods on task efficiency, number of errors, learning, independence, and cognitive load. The AR application is developed in Unity and deployed on HoloLens 2 headsets. Interviews with experts from industry and academia are also conducted to create new insights into the affordances of AR as a training versus assistive tool for manufacturing workers, as well as the need for intelligent mechanisms that enable adaptive and personalized interactions between workers and AR. The findings indicate that despite comparable performance between the AR and control groups in terms of task completion time, learning curve, and independence from instructions, AR dramatically decreases the number of errors compared to traditional instruction, which is sustained after the AR support is removed. Several insights drawn from the experiments and expert interviews are discussed to inform the design of future AR technologies for both training and assisting incumbent and future manufacturing workers on complex manipulation and reasoning tasks.     

Shared space: An augmented reality approach for computer supported collaborative work

Virtual Reality (VR) appears a natural medium for three-dimensional computer supported collaborative work (CSCW). However the current trend in CSCW is to adapt the computer interface to work with the user's traditional tools, rather than separating the user from the real world as does immersive VR. One solution is through Augmented Reality, the overlaying of virtual objects on the real world. In this paper we describe the Shared Space concept—the application of Augmented Reality for three-dimensional CSCW. This combines the advantages of Virtual Reality with current CSCW approaches. We describe a collaborative experiment based on this concept and present preliminary results which show that this approach may be better for some applications.     

Marker-less registration based on template tracking for augmented reality

Accurate 3D registration is a key issue in the Augmented Reality (AR) applications, particularly where are no markers placed manually. In this paper, an efficient markerless registration algorithm is presented for both outdoor and indoor AR system. This algorithm first calculates the correspondences among frames using fixed region tracking, and then estimates the motion parameters on projective transformation following the homography of the tracked region. To achieve the illumination insensitive tracking, the illumination parameters are solved jointly with motion parameters in each step. Based on the perspective motion parameters of the tracked region, the 3D registration, the camera’s pose and position, can be calculated with calibrated intrinsic parameters. A marker-less AR system is described using this algorithm, and the system architecture and working flow are also proposed. Experimental results with comparison quantitatively demonstrate the correctness of the theoretical analysis and the robustness of the registration algorithm.

Educational magic toys developed with augmented reality technology for early childhood education

Shaping children's experience, enhancing their imagination and affecting their behaviors, toys have great importance. Recently, toys have gained a digital characteristic and many children have tended to use them. For this reason, educational magic toys (EMT) were developed with augmented reality technology in this study. It is called as EMT because virtual objects such as story animations, 3D objects and flash animations appear on the toys. EMT has included puzzles, flash cards and match cards to teach animals, fruits, vegetables, vehicles, objects, professions, colors, numbers and shapes for average 5–6 age children in Early Childhood Education. The aim of this study is to reveal teachers' and children's opinions on EMT, to determine children's behavioral patterns and their cognitive attainment, and the relationship between them while playing EMT. Mix method was used and the sample consisted of 30 teachers and 33 children aged 5–6 in early childhood education. As data collection tools, a survey, an observation and interview form were used. This study revealed that teachers and children liked EMT activity. In addition, children interactively played with these toys but not had high cognitive attainment. From this point, we can say that these toys can be effectively used in early childhood education. However, collaborative and interactive learning with these toys should be provided. Moreover, this study will provide an important contribution, present a new educational AR application, and fill the gap in the educational technology field.     

Autostereoscopic augmented reality visualization for depth perception in endoscopic surgery

Augmented reality (AR) has received increasing attention in minimally invasive surgery (MIS) applications. The goal of applying AR techniques to MIS is to enhance a surgeon's perception of the spatial relationship by overlaying invisible structures (e.g. tumor or vessels) onto the in vivo endoscopic video acquired during the surgery. One of primary issues of AR visualization is to provide correct depth perception for visible and invisible structures. In this paper, we present a video-based AR system consisting of functional modules for real-time 3D surface capture, reconstruction, and registration with pre-operative segmented CT model. The real-time 3D registration allows precise overlay of invisible structures onto 2D video for AR visualization. The AR overlay result is displayed on a multi-view autostereoscopic lenticular LCD. To study and compare the efficacy of AR visualization techniques, we investigated five different AR visualization modes. Both simulated and in vivo experiments were carried out and autostereoscopic AR visualization results were given. Evaluation and comparison for depth perception between five AR visualization modes are presented. Finally, we conclude the characteristics of these visualization modes. The novelty of our work lies in successful implementation of an end-to-end 3D autostereoscopic AR system from real-time reconstruction and registration with our multi-channel 3D endoscope, and systematic evaluation and comparison of five different visualization modes for depth perception.     

Robust detection and tracking of annotations for outdoor augmented reality browsing

A common goal of outdoor augmented reality (AR) is the presentation of annotations that are registered to anchor points in the real world. We present an enhanced approach for registering and tracking such anchor points, which is suitable for current generation mobile phones and can also successfully deal with the wide variety of viewing conditions encountered in real life outdoor use. The approach is based on on-the-fly generation of panoramic images by sweeping the camera over the scene. The panoramas are then used for stable orientation tracking, while the user is performing only rotational movements. This basic approach is improved by several new techniques for the re-detection and tracking of anchor points. For the re-detection, specifically after temporal variations, we first compute a panoramic image with extended dynamic range, which can better represent varying illumination conditions. The panorama is then searched for known anchor points, while orientation tracking continues uninterrupted. We then use information from an internal orientation sensor to prime an active search scheme for the anchor points, which improves matching results. Finally, global consistency is enhanced by statistical estimation of a global rotation that minimizes the overall position error of anchor points when transforming them from the source panorama in which they were created, to the current view represented by a new panorama. Once the anchor points are redetected, we track the user's movement using a novel 3-degree-of-freedom orientation tracking approach that combines vision tracking with the absolute orientation from inertial and magnetic sensors. We tested our system using an AR campus guide as an example application and provide detailed results for our approach using an off-the-shelf smartphone. Results show that the re-detection rate is improved by a factor of 2 compared to previous work and reaches almost 90% for a wide variety of test cases while still keeping the ability to run at interactive frame rates.     

移动增强现实框架系统

在已有的移动增强现实( MAR)系统中存在开发困难,虚拟现实( AR)效果观察缺乏通用性的问题,提出了一种MAR系统的通用框架,该框架包括三个部分：为智能移动设备用户所使用的通用MAR观察器、为MAR应用提供网络服务的MAR服务器,以及为开发人员提供了MAR应用定制器。该框架结合高通的Vuforia SDK针对平面图像实现了MAR的应用定制和通用AR效果浏览。框架为开发人员提供了无代码开发功能,为移动用户提供了遍览一类AR应用的通用观察器。实验结果表明该MAR框架系统降低了MAR应用的开发门槛与时间费用,大大提高了开发效率。