Real world teleconferencing

We've been exploring how augmented reality (AR) technology can create fundamentally new forms of remote collaboration for mobile devices. AR involves the overlay of virtual graphics and audio on reality. Typically, the user views the world through a handheld or head-mounted display (HMD) that's either see-through or overlays graphics on video of the surrounding environment. Unlike other computer interfaces that draw users away from the real world and onto the screen, AR interfaces enhance the real world experience. For example, with this technology doctors could see virtual ultrasound information superimposed on a patient's body.     

Calibration of an HMPD-Based Augmented Reality System

In augmented reality (AR) applications, accurately registering a virtual object with its real counterpart is a challenging problem. The size, depth, and geometry, as well as the physical attributes of the virtual object, have to be rendered precisely relative to a physical reference. This paper presents a systematic calibration process to address the registration challenge in a custom-designed AR system, which is based upon recent head-mounted projective display (HMPD) technology. Following a concise review of the HMPD concept and our system configuration, we first present a computational model of the HMPD viewing system and requirements for the system calibration. Then, we describe, in detail, the calibration procedures to obtain estimates of unknown transformations, summarize the application of the estimates in a customized graphics rendering toolkit, and discuss the evaluation experiments and observations. Finally, the implementation of a testbed to demonstrate a successful registration is briefly described, and experimental results are presented     

基于Java的增强现实视频子系统的设计和实现

负责虚实显示的视频子系统是增强现实系统一个重要部分。视频子系统的性能是增强现实达到期望效果的关键因素。文章介绍了一个基子软件常用设计模式并用;高级Java图形图像技术实现的增强现实系统的视频子系统。通过应用合适的Java多线程技术并优化显示逻辑的算法流程,从而实现了高效的虚实显示,并给出了实验效果和性能统计结果。     

Sensory-motor enhancement in a virtual therapeutic environment

The sensory-motor skills of persons with neuromuscular disabilities have been shown to be enhanced by intensive and repetitive therapeutic interventions. This paper describes a form of low immersion virtual reality and a prototype, open source system that allow a user with significant physical disability to actively interact with computer-generated objects whose behaviors promote a game-like interaction. Unlike fully immersive and haptic virtual reality, this approach frees the user from head-mounted displays and gloves. It extracts the user’s real-time silhouette from the output of a remote video camera and uses that two-dimensional outline to interact with graphical objects on screen. In contrast to video games that have been modified with specialized interfaces, this virtual interaction system promotes the repetitive use of goal directed movements of the arms and body, which are essential to promote cortical reorganization, as well as discourage unwanted changes in muscle tissue that result in contracture. A prototype system demonstrates the potential of low immersion technology to motivate users and encourage participation in therapy. It also offers the potential of accommodating the sensory-motor skills of individuals with very significant impairment. The behaviors of the computer-generated graphics can be altered to allow use by those with very limited range of motion and/or motor control. These behaviors can be adjusted to provide a continuing challenge as the user’s skills improve. This prototype is described in terms of functional capabilities that include a silhouette extraction from a video image, and generation of graphical objects that interact with the silhouette. The work is extended with a discussion of a more sophisticated region of interest detection algorithm that can select specific parts of the body.     

Building and using a scalable display wall system

Princeton's scalable display wall project explores building and using a large-format display with commodity components. The prototype system has been operational since March 1998. Our goal is to construct a collaborative space that fully exploits a large-format display system with immersive sound and natural user interfaces. Our prototype system is built with low-cost commodity components: a cluster of PCs, PC graphics accelerators, consumer video and sound equipment, and portable presentation projectors. This approach has the advantages of low cost and of tracking technology well, as high-volume commodity components typically have better price-performance ratios and improve at faster rates than special-purpose hardware. We report our early experiences in building and using the display wall system. In particular, we describe our approach to research challenges in several specific research areas, including seamless tiling, parallel rendering, parallel data visualization, parallel MPEG decoding, layered multiresolution video input, multichannel immersive sound, user interfaces, application tools, and content creation     

Motion sickness, console video games, and head-mounted displays

OBJECTIVE: We evaluated the nauseogenic properties of commercial console video games (i.e., games that are sold to the public) when presented through a head-mounted display. BACKGROUND: Anecdotal reports suggest that motion sickness may occur among players of contemporary commercial console video games. METHODS: Participants played standard console video games using an Xbox game system. We varied the participants' posture (standing vs. sitting) and the game (two Xbox games). Participants played for up to 50 min and were asked to discontinue if they experienced any symptoms of motion sickness. RESULTS: Sickness occurred in all conditions, but it was more common during standing. During seated play there were significant differences in head motion between sick and well participants before the onset of motion sickness. CONCLUSION: The results indicate that commercial console video game systems can induce motion sickness when presented via a head-mounted display and support the hypothesis that motion sickness is preceded by instability in the control of seated posture. APPLICATION: Potential applications of this research include changes in the design of console video games and recommendations for how such systems should be used.     

Visual inspection with augmented reality head-mounted display: An Australian usability case study

Augmented reality (AR) is an Industry 4.0 technology. For more than a decade, advancements in AR technology and their applications have been expected to revolutionise the manufacturing industry and deliver quality and productivity gains. However, due to factors such as equipment costs, skills shortages and technological limitations of AR devices, operational deployment beyond prototypes has been constrained. Real-world, usability studies can explore barriers to implementation and improve system design. This paper details a mixed method usability case study of an AR head-mounted display (HMD) to perform a short, simple visual inspection task. Twenty-two participants from South Australian manufacturing businesses inspected a pump and pipe skid while working at height. Overall, workload demands for the task were considered acceptable and just below the “low” workload threshold (NASA Task Load Index, mean = 29.3) and the system usability was rated “average” (system usability scale, mean = 68.5). The results suggest the task did not place too high a burden on users and was an appropriate initial exposure to AR HMDs, but further refinement to the interface would be desirable before implementation to minimise frustration and promote learning. Users were enthusiastic and open-minded about the AR HMD although results indicate that even with recent advancements in AR HMD technology, interactions between the task, technology and environment continue to cause human and technical challenges—some of which are relatively straightforward to address but others are dependent on larger-scale efforts.     

Binocular mobile augmented reality based on stereo camera tracking

Mobile augmented reality (AR) applications have become feasible with the evolution of mobile hardware. For example, the advent of the smartphone allowed implementing real-time mobile AR, which triggered the release of various applications. Recently, rapid development of display technology, especially for stereoscopic displays, has encouraged researches to implement more immersive and realistic AR. In this paper, we present a framework of binocular augmented reality based on stereo camera tracking. Our framework was implemented on a smartphone and supports autostereoscopic display and video see-through display in which a smartphone can be docked. We modified edge-based 3-D object tracking in order to estimate poses of left and right cameras jointly; this guarantees consistent registration across left and right views. Then, virtual contents were overlaid onto camera images using estimated poses, and the augmented stereo images were distorted to be shown through a video see-through display. The feasibility of the proposed framework is shown by experiments and demonstrations.     

Depth perception in shuffleboard: Depth cues effect on depth perception in virtual and augmented reality system

The virtual reality (VR) and augmented reality (AR) applications have been widely used in a variety of fields; one of the key requirements in a VR or AR system is to understand how users perceive depth in the virtual environment and AR. Three different graphics depth cues are designed in shuffleboard to explore what kind of graphics depth cues are beneficial for depth perception. We also conduct a depth‐matching experiment to compare performance in VR and AR systems using an optical see‐through head‐mounted display (HMD). The result shows that the absolute error increases as the distance becomes farther. Analysis from the inverse of distance shows that box depth cues have a significant effect on depth perception, while the points depth cues and line depth cues have no significant effect. The error in diopter in AR experiment is lower than that in VR experiment. Participants in the AR experiment under medium illuminance condition have less error than those under low and high illuminance conditions. Men have less error than women in certain display conditions, but the advantage disappears when there is a strong depth cue. Besides, there is no significant effect of completion time on depth perception.     

基于虚拟化的系统安全增强及显卡透传研究

针对个人终端操作系统安全问题,提出一种基于系统虚拟化技术的操作系统安全增强模型,研究提高该模型下KVM虚拟机显示性能的显卡透传技术。实验结果表明,显卡透传技术能够克服虚拟机客户操作系统的显示性能缺陷,使得客户机操作系统能够像真实操作系统一样满足图形显示与处理的应用,适用于终端安全领域。     

Markerless cooperative augmented reality-based smart manufacturing double-check system: Case of safe PCBA inspection following automatic optical inspection

Augmented reality (AR) is a key technology anchored towards realizing Industry 4.0 smart manufacturing aims. In manufacturing inspection, AR has previously been employed to support operators in assessing thickness of manufactured parts, ship building, aircraft subassemblies and printed circuit board assemblies (PCBAs) with or without external markers in standalone head-mounted (HMD) or handheld systems. These AR systems often use optical see-through (OST) or video see-through (VST) technologies. Currently, cyber physical integration of processes in an industrial system synergizes production through increased efficiency and improved quality while facilitating customization. An operator manually double-checking/inspecting a product that has previously been automatically inspected can better rely on this existing defect location information once it is contextually and spatially overlaid within their field of view to intuitively and efficiently execute the task. Additional interactive information provided can quickly reorient a user, provide necessary contextual reference, and monitor progress of the task while alerting them of their safety. In this study, adopting an automatic optical inspection (AOI) wirelessly aided HMD-OST AR-based manual inspection system requiring no external markers for contextual registration is a promising direction towards smart and safe PCBA manufacturing in line with Industry 4.0. Animated rectangular bracket of high color contrast is employed to localize the spawned AOI defect point, an arrow overlaid with distance text provides user guidance, a contextual resizable image facilitates user double-checking and a progress bar tracks the inspection progress while monitoring tracking state. We evaluate the developed system's robustness, effectiveness of display technology used, suitability of contextual against static display modes, and display technology influence on various inspection attributes in a user study. Results demonstrate the system to be robust, OST-HMD outperforms handheld VST devices, contextual display mode to be significantly preferred to static mode, and display technology employed has no significant influence on the inspection attributes. Finally, registration precision results demonstrate usability of the system while superposition of distance to orientation information raises the inspection rate of PCBAs.     

An augmented reality interface to contextual information

In this paper, we report on a prototype augmented reality (AR) platform for accessing abstract information in real-world pervasive computing environments. Using this platform, objects, people, and the environment serve as contextual channels to more information. The user??s interest with respect to the environment is inferred from eye movement patterns, speech, and other implicit feedback signals, and these data are used for information filtering. The results of proactive context-sensitive information retrieval are augmented onto the view of a handheld or head-mounted display or uttered as synthetic speech. The augmented information becomes part of the user??s context, and if the user shows interest in the AR content, the system detects this and provides progressively more information. In this paper, we describe the first use of the platform to develop a pilot application, Virtual Laboratory Guide, and early evaluation results of this application.     

Impact of information display on worker performance for wood frame wall assembly using AR HMD under different task conditions

Augmented Reality (AR) head-mounted displays (HMDs) provide users with an immersive virtual experience in the real world. The portability of this technology affords various information display options for construction workers that are not possible otherwise. However, the impact of these different information presentation options on human performance should be carefully evaluated before such technology is deployed in the jobsite. In this paper, we describe a research effort examining how different information displays presented via AR HMD influence task performance when assembling three sized wooden wall frame assembly tasks. We asked 18 construction engineering students with framing experience to finish three wood frame assembly tasks (large, medium, and small) using one of the three information displays (AR 3D conformal, AR 2D tag-along, and paper blueprints). The task performance was measured by time of completion and framing errors, which were analyzed and compared among each factor.     

Applications of Augmented Vision Head-Mounted Systems in Vision Rehabilitation

Vision loss typically affects either the wide peripheral vision (important for mobility), or central vision (important for seeing details). Traditional optical visual aids usually recover the lost visual function, but at a high cost for the remaining visual function. We have developed a novel concept of vision-multiplexing using augmented vision head-mounted display systems to address vision loss. Two applications are discussed in this paper. In the first, minified edge images from a head-mounted video camera are presented on a see-through display providing visual field expansion for people with peripheral vision loss, while still enabling the full resolution of the residual central vision to be maintained. The concept has been applied in daytime and nighttime devices. A series of studies suggested that the system could help with visual search, obstacle avoidance, and nighttime mobility. Subjects were positive in their ratings of device cosmetics and ergonomics. The second application is for people with central vision loss. Using an on-axis aligned camera and display system, central visibility is enhanced with 1:1 scale edge images, while still enabling the wide field of the unimpaired peripheral vision to be maintained. The registration error of the system was found to be low in laboratory testing.     

具有图形化界面的车载DVD影音系统设计

介绍了一种新型的具有图形化界面的车载DVD影音系统,给出了DVD影音系统的主机模块和显示屏模块的设计方案,并介绍了基于YGV629的DVD视频画面和图形化界面的叠加原理,最后给出了系统的软件实现方法。该设计方案实现的车载影音系统操作界面友好、灵活性强,只需修改相应的图形库,即可应用于需要视频图形叠加的其他领域。     

The MATRIS project: real-time markerless camera tracking for Augmented Reality and broadcast applications

The term “Augmented Reality” covers a wide range of applications, from the overlay of virtual graphics on a real scene using a head-mounted display for use in areas such as industrial maintenance, through to the insertion of real-time virtual graphics in TV programmes. Fundamental to all these applications is the need to be able to accurately track the motion of the camera, so that the graphics may be rendered so as to appear rigidly locked to the real world. To overcome the limitations of existing tracking systems, the MATRIS project has developed a real-time system for measuring the movement of a camera, which uses image analysis to track naturally occurring features in the scene, and data from an inertial sensor. No additional sensors, special markers, or camera mounts are required. This paper gives an overview of the system, provides the context for the other articles in this journal and presents some results.     

Gesture-based interactive augmented reality content authoring system using HMD

This paper proposes an augmented reality content authoring system that enables ordinary users who do not have programming capabilities to easily apply interactive features to virtual objects on a marker via gestures. The purpose of this system is to simplify augmented reality (AR) technology usage for ordinary users, especially parents and preschool children who are unfamiliar with AR technology. The system provides an immersive AR environment with a head-mounted display and recognizes users’ gestures via an RGB-D camera. Users can freely create the AR content that they will be using without any special programming ability simply by connecting virtual objects stored in a database to the system. Following recognition of the marker via the system’s RGB-D camera worn by the user, he/she can apply various interactive features to the marker-based AR content using simple gestures. Interactive features applied to AR content can enlarge, shrink, rotate, and transfer virtual objects with hand gestures. In addition to this gesture-interactive feature, the proposed system also allows for tangible interaction using markers. The AR content that the user edits is stored in a database, and is retrieved whenever the markers are recognized. The results of comparative experiments conducted indicate that the proposed system is easier to use and has a higher interaction satisfaction level than AR environments such as fixed-monitor and touch-based interaction on mobile screens.     

Three views of virtual reality: an overview

Virtual reality (VR) presents a synthetically generated environment to the user through visual, auditory, and other stimuli. A VR systems's head-mounted display, stereo graphics, and direct manipulation in three dimensions are outlined. Early work in VR and major technological hurdles in the areas of tracking, display, image generation, and software support are reviewed. Research directions and potential applications of VR are discussed