A multi‐plane optical see‐through head mounted display design for augmented reality applications

Augmented reality (AR) display technology greatly enhances users' perception of and interaction with the real world by superimposing a computer‐generated virtual scene on the real physical world. The main problem of the state‐of‐the‐art 3D AR head‐mounted displays (HMDs) is the accommodation‐vergence conflict because the 2D images displayed by flat panel devices are at a fixed distance from the eyes. In this paper, we present a design for an optical see‐through HMD utilizing multi‐plane display technology for AR applications. This approach manages to provide correct depth information and solves the accommodation‐vergence conflict problem. In our system, a projector projects slices of a 3D scene onto a stack of polymer‐stabilized liquid crystal scattering shutters in time sequence to reconstruct the 3D scene. The polymer‐stabilized liquid crystal shutters have sub‐millisecond switching time that enables sufficient number of shutters to achieve high depth resolution. A proof‐of‐concept two‐plane optical see‐through HMD prototype is demonstrated. Our design can be made lightweight, compact, with high resolution, and large depth range from near the eye to infinity and thus holds great potential for fatigue‐free AR HMDs.     

Use of physiological signals to predict cybersickness

Cybersickness is a common and unpleasant side effect of virtual reality immersion. We measured physiological changes that were experienced by seated subjects who interacted with a virtual environment (VE) first while viewing a display monitor and second while using a head-mounted display (HMD). Comparing results for these two conditions let us identify physiological consequences of HMD use. In both viewing conditions, subjects rated the severity of their symptoms verbally and completed a post-immersion cybersickness assessment questionnaire. In the HMD viewing condition but not in the display monitor condition, verbal reports of cybersickness severity increased significantly relative to baseline. Half of the subjects chose to exit the VE after six minutes of HMD use and reported feeling some nausea at that time. We found that changes in stomach activity, blinking, and breathing can be used to estimate post-immersion symptom scores, with R² values reaching as high as 0.75. These results suggest that HMD use by seated subjects is strongly correlated with the development of cybersickness. Finally, a linear discriminant analysis shows that physiological measures alone can be used to classify subject data as belonging to the HMD or monitor viewing condition with an accuracy of 78%.     

Geometric calibration of head-mounted displays and its effects on distance estimation

Head-mounted displays (HMDs) allow users to observe virtual environments (VEs) from an egocentric perspective. However, several experiments have provided evidence that egocentric distances are perceived as compressed in VEs relative to the real world. Recent experiments suggest that the virtual view frustum set for rendering the VE has an essential impact on the user's estimation of distances. In this article we analyze if distance estimation can be improved by calibrating the view frustum for a given HMD and user. Unfortunately, in an immersive virtual reality (VR) environment, a full per user calibration is not trivial and manual per user adjustment often leads to mini- or magnification of the scene. Therefore, we propose a novel per user calibration approach with optical see-through displays commonly used in augmented reality (AR). This calibration takes advantage of a geometric scheme based on 2D point - 3D line correspondences, which can be used intuitively by inexperienced users and requires less than a minute to complete. The required user interaction is based on taking aim at a distant target marker with a close marker, which ensures non-planar measurements covering a large area of the interaction space while also reducing the number of required measurements to five. We found the tendency that a calibrated view frustum reduced the average distance underestimation of users in an immersive VR environment, but even the correctly calibrated view frustum could not entirely compensate for the distance underestimation effects.     

Exploiting perceptual limitations and illusions to support walking through virtual environments in confined physical spaces

Head-mounted displays (HMDs) allow users to immerse in a virtual environment (VE) in which the user’s viewpoint can be changed according to the tracked movements in real space. Because the size of the virtual world often differs from the size of the tracked lab space, a straightforward implementation of omni-directional and unlimited walking is not generally possible. In this article we review and discuss a set of techniques that use known perceptual limitations and illusions to support seemingly natural walking through a large virtual environment in a confined lab space. The concept behind these techniques is called redirected walking. With redirected walking, users are guided unnoticeably on a physical path that differs from the path the user perceives in the virtual world by manipulating the transformations from real to virtual movements. For example, virtually rotating the view in the HMD to one side with every step causes the user to unknowingly compensate by walking a circular arc in the opposite direction, while having the illusion of walking on a straight trajectory. We describe a number of perceptual illusions that exploit perceptual limitations of motion detectors to manipulate the user’s perception of the speed and direction of his motion. We describe how gains of locomotor speed, rotation, and curvature can gradually alter the physical trajectory without the users observing any discrepancy, and discuss studies that investigated perceptual thresholds for these manipulations. We discuss the potential of self-motion illusions to shift or widen the applicable ranges for gain manipulations and to compensate for over- or underestimations of speed or travel distance in VEs. Finally, we identify a number of key issues for future research on this topic.     

Influence of virtual objects' shadows and lighting coherence on distance perception in optical see‐through augmented reality

This paper focuses on how virtual objects' shadows as well as differences in alignment between virtual and real lighting influence distance perception in optical see‐through (OST) augmented reality (AR). Four hypotheses are proposed: (H1) Participants underestimate distances in OST AR; (H2) Virtual objects' shadows improve distance judgment accuracy in OST AR; (H3) Shadows with different realism levels have different influence on distance perception in OST AR; (H4) Different levels of lighting misalignment between real and virtual lights have different influence on distance perception in OST AR scenes. Two experiments were designed with an OST head mounted display (HMD), the Microsoft HoloLens. Participants had to match the position of a virtual object displayed in the OST‐HMD with a real target. Distance judgment accuracy was recorded under the different shadows and lighting conditions. The results validate hypotheses H2 and H4 but surprisingly showed no impact of the shape of virtual shadows on distance judgment accuracy thus rejecting hypothesis H3. Regarding hypothesis H1, we detected a trend toward underestimation; given the high variance of the data, more experiments are needed to confirm this result. Moreover, the study also reveals that perceived distance errors and completion time of trials increase along with targets' distance.     

Oculomotor changes within virtual environments

This paper discusses the oculomotor changes which might be expected to occur during immersion in a virtual environment whilst wearing a Head mounted display (HMD). To do so, it first examines the stimulus presented to the eyes, and then considers how this stimulus could affect the visual system. Theoretical analysis and empirical results from the use of three different HMDs point towards the same conclusion, that in this context a mismatch between the instrument inter-ocular distance (IOD) and the user's inter-pupillary distance is of little concern, unlike the mismatch between the instrument IOD and the inter-screen distance.     

Subjective and objective evaluation of visual fatigue caused by continuous and discontinuous use of HMDs

During continuous use of displays, a short rest can relax users' eyes and relieve visual fatigue. As one of the most important devices of virtual reality, head‐mounted displays (HMDs) can create an immersive 3D virtual world. When users have a short rest during the using of HMDs, they will experience a transition from virtual world to real world. In order to investigate how this change affects users' eye condition, we designed a 2 × 2 experiment to explore the effects of short rest during continuous using of HMDs and compared the results with those of 2D displays. The Visual Fatigue Scale, critical flicker frequency, visual acuity, pupillary diameter, and accommodation response of 80 participants were measured to assess the subject's performance. The experimental results indicated that a short rest during the using of 2D displays could significantly reduce users' visual fatigue. However, the experimental results of using HMDs showed that short rest during continuous using of HMD induced more severe symptoms of subjectively visual discomfort, but reduced the objectively visual fatigue.     

Spatialized Vibrotactile Feedback Improves Goal-Directed Movements in Cluttered Virtual Environments

Spatial awareness in virtual reality (VR) is a dominant research topic. It plays an essential role in the assessment of human operators’ behavior in simulated tasks, notably for the evaluation of the feasibility of manual maintenance tasks in cluttered industrial settings. In such contexts, it is decisive to evaluate the spatial and temporal correspondence between the operator’s movement kinematics and that of his/her virtual avatar in the virtual environment (VE). Often, in a cluttered VE, direct kinesthetic (force) feedback is limited or absent. We tested whether vibrotactile (cutaneous) feedback would increase visuo-proprioceptive consistency, spatial awareness, and thus the validity of VR studies, by augmenting the perception of the operator’s contact(s) with virtual objects. We present experimental results obtained using a head-mounted display (HMD) during a goal-directed task in a cluttered VE. Results suggest the contribution of spatialized vibrotactile feedback to visuo-proprioceptive consistency.     

Lessons learned: Evaluating visualizations for occluded objects in handheld augmented reality

Handheld devices like smartphones and tablets have emerged as one of the most promising platforms for Augmented Reality (AR). The increased usage of these portable handheld devices has enabled handheld AR applications to reach the end-users; hence, it is timely and important to seriously consider the user experience of such applications. AR visualizations for occluded objects enable an observer to look through objects. AR visualizations have been predominantly evaluated using Head-Worn Displays (HWDs), handheld devices have rarely been used. However, unless we gain a better understanding of the perceptual and cognitive effects of handheld AR systems, effective interfaces for handheld devices cannot be designed. Similarly, human perception of AR systems in outdoor environments, which provide a higher degree of variation than indoor environments, has only been insufficiently explored.In this paper, we present insights acquired from five experiments we performed using handheld devices in outdoor locations. We provide design recommendations for handheld AR systems equipped with visualizations for occluded objects. Our key conclusions are the following: (1) Use of visualizations for occluded objects improves the depth perception of occluded objects akin to non-occluded objects. (2) To support different scenarios, handheld AR systems should provide multiple visualizations for occluded objects to complement each other. (3) Visual clutter in AR visualizations reduces the visibility of occluded objects and deteriorates depth judgment; depth judgment can be improved by providing clear visibility of the occluded objects. (4) Similar to virtual reality interfaces, both egocentric and exocentric distances are underestimated in handheld AR. (5) Depth perception will improve if handheld AR systems can dynamically adapt their geometric field of view (GFOV) to match the display field of view (DFOV). (6) Large handheld displays are hard to carry and use; however, they enable users to better grasp the depth of multiple graphical objects that are presented simultaneously.     

Vection and cybersickness generated by head-and-display motion in the Oculus Rift

Cybersickness is often experienced when viewing virtual environments through head-mounted displays (HMDs). This study examined whether vection (i.e., illusory self-motion) and mismatches between perceived and physical head motions contribute to such adverse experiences. Observers made oscillatory yaw head rotations while viewing stereoscopic optic flow through an Oculus Rift HMD. Vection and cybersickness were measured under 3 conditions of visual compensation for physical head movements: “compensated”, “uncompensated”, and “inversely compensated”. When a nearer aperture was simulated by the HMD, vection was found to be strongest in the “compensated” condition and weakest in the “inversely compensated” condition. However, vection was similar for all 3 conditions during full-field exposures. Cybersickness was most severe for the “inversely compensated” condition, but was not different for the other two conditions. We conclude that mismatches between perceived and physical head-movements can contribute strongly to cybersickness. The relationship between vection and cybersickness is weaker and appears complex.     

Effect of VR technology matureness on VR sickness

In this paper relationship of perceived virtual reality (VR) sickness phenomenon with different generations of virtual reality head mounted displays (VR HMD) is presented. Action content type omnidirectional video clip was watched by means of four HMDs of different levels of technological matureness, with a 2D monitor used as a reference point. In addition to subjective estimation of VR sickness effects by means of the SSQ questionnaire, psychophysiology of the participants was monitored. Participant’s electrodermal activity, heart rate, skin temperature and respiration rate were measured. Results of the study indicate differences between HMDs in both SSQ score and changes of physiology. Skin conductance was found to be significantly correlated with VR sickness. Mobile HMD did not induce significantly higher levels of VR sickness. Disorientation SSQ was proven to be a useful tool for assessing the VR sickness effects.

     

光学透视式头盔显示器标定的退化影响

为了成功地应用增强现实，需要有观察仪器的准确投影信息，这就要求对所使用的光学透视式头盔显示器进行标定。针对实验室的光学透视式增强现实系统的组成特点，选定了虚拟摄像机（头盔显示器与人眼的结合）的标定算法。根据对投影矩阵的退化点结构的影响，改进了选取标定的对应点个数和用户头部的运动轨迹。实验结果表明，标定效果明显改善了，成功率也有明显提高。     

Effect of geometric field of view on stereoscopic spatial judgments

Within a stereoscopic display the field of view (FOV) was held constant at 13.86 degrees while the geometric field of view (GFOV) was varied across four levels: 0 degrees (parallel), 13.86 degrees (veridical), 50 degrees and 100 degrees. Participants performed a distance-matching task where they adjusted the distance of a standard track from the centre of the display to match the distance of a target track from the same point. The results indicated that while the least error occurred in the veridical GFOV condition, small variations of GFOV away from the veridical have little effect. Large differences between FOV and GFOV (36 degrees and 86 degrees) increased errors markedly. A trend toward better performance in the veridical GFOV condition relative to the parallel GFOV condition suggests that the use of linear perspective information in a stereoscopic display may facilitate more accurate spatial perception. Actual or potential applications of this work include stereoscopic display design in aviation and non-aviation settings.     

A study on differences in human perception between a real and an AR scene viewed in an OST‐HMD

With the recent growth in the development of augmented reality (AR) technologies, it is becoming important to study human perception of AR scenes. In order to detect whether users will suffer more from visual and operator fatigue when watching virtual objects through optical see‐through head‐mounted displays (OST‐HMDs), compared with watching real objects in the real world, we propose a comparative experiment including a virtual magic cube task and a real magic cube task. The scores of the subjective questionnaires (SQ) and the values of the critical flicker frequency (CFF) were obtained from 18 participants. In our study, we use several electrooculogram (EOG) and heart rate variability (HRV) measures as objective indicators of visual and operator fatigue. Statistical analyses were performed to deal with the subjective and objective indicators in the two tasks. Our results suggest that participants were very likely to suffer more from visual and operator fatigue when watching virtual objects presented by the OST‐HMD. In addition, the present study provides hints that HRV and EOG measures could be used to explore how visual and operator fatigue are induced by AR content. Finally, three novel HRV measures are proposed to be used as potential indicators of operator fatigue.     

Intelligent Cruise-Control Navigation: A new navigation/travel method for use in virtual environments

An important feature of virtual reality is the facility for the user to move around a virtual environment in a natural and easily controlled manner. Navigation, also called locomotion, travel or motion, involves changing the perspective of the user in the virtual environment (VE). It allows the user to move in the VE as well as reorient themselves to look at the world differently. Natural locomotion methods are able to contribute to a sense of presence and reality. The illusion of presence can be lost through unnatural experiences during travel in the VE. This can be caused by poor interactive metaphors or by experiences which do not agree with the user's everyday understanding of the real world. This paper focuses on the navigation method in the VE, one of the major interfaces for the interactivity between human and VE in virtual reality circumstances and worlds. It proposes a new navigation method. Intelligent Cruise-Control Navigation (ICCN), which provides a natural and user-centred approach to navigation in the VE and can improve the user's sense of reality and presence. ICCN is composed of three major phases: Constant Velocity Navigation, Collision Detection and Avoidance, and Path Adjustment. The ICCN can reduce the user's fatigue and improve the user's presence in the VE. The small experimental study reported in this paper suggests that the ICCN will be a natural, straightforward, and useful navigation interface in VE.     

Head up versus head down: the costs of imprecision, unreliability, and visual clutter on cue effectiveness for display signaling

We conducted 2 experiments to investigate the clutter-scan trade-off between the cost of increasing clutter by overlaying complex information onto the forward field of view using a helmet-mounted display (HMD) and the cost of scanning when presenting this information on a handheld display. In the first experiment, this trade-off was examined in terms of the spatial accuracy of target cuing data in a relatively sparse display; in the second, the spatial accuracy of the cue was varied more radically in an information-rich display. Participants were asked to detect and identify targets hidden in the far domain while performing a monitoring task in the near domain using either an HMD or a handheld display. The results revealed that on a sparse display, the reduced scanning from the HMD presentation of cuing out-weighed the costs of clutter for cued targets, regardless of cue precision, but no benefit was found for uncued targets. When the HMD displayed task-irrelevant information, however, target detection was hindered by the extraneous clutter in the forward field of view relative to the handheld display condition, and this cost of clutter increased as the amount of data that needed to be inspected increased. Potential applications of this research include the development of design considerations for head-up displays for aviation and military applications.     

Interaction and visual performance in stereoscopic displays: A review

In this paper, authors systematically selected and reviewed articles related to stereoscopic displays and their advances, with a special focus on perception, interaction, and corresponding challenges. The aim was to understand interaction‐related problems, provide possible explanations, and identify factors that limit their applications. Despite promising advancements, there are still issues that researchers in the field fail to explain precisely. The two major problems in stereoscopic viewing are, compared with the real world, objects are perceived to be smaller than they actually are and there are discomfort and visual syndromes. Furthermore, there is general agreement that humans underestimate their egocentric distance in a virtual environment (VE). Our analysis revealed that in the real world, distance estimation is about 94% accurate, but in VE, it is only about 80% accurate. This problem could reduce the efficacy of different sensory motor‐based applications where interaction is important. Experts from human factors, computing, psychology, and others have studied contributing factors such as types of perception/response method, quality of graphics, associated stereoscopic conditions, experience in virtual reality (VR), and distance signals. This paper discusses the factors requiring further investigation if the VR interaction is to be seamlessly realized. In addition, engineering research directions aiming at improving current interaction performances are recommended.     

Effect of head-mounted displays on posture

OBJECTIVE: The aim of the present study was to determine if a wearable system based on a head-mounted display (HMD) causes users to alter their head position and adopt postures that place greater stress on the musculoskeletal system. BACKGROUND: HMDs are common output devices used with wearable computers. HMDs provide the wearer with visual information by projecting computer-generated virtual images in front of the eyes. Deviations of neck posture from a neutral upright position increase the stresses on the musculoskeletal system of the head and neck. METHOD: Seven paramedics simulated the treatment of a patient under a normal condition and when using an HMD wearable computer system. During the simulations a posture analysis was performed using the Rapid Upper Limb Assessment method. RESULTS: The postures adopted when wearing an HMD, as compared with a normal condition, scored significantly higher for the neck (z = 2.463, p < .05) and for overall body posture (left side of the body: z = 2.447, p < .05; right side of the body: z = 2.895, p < .05). CONCLUSION: Wearing an HMD can force the wearers to modify their neck posture. As such, the musculoskeletal system of the head and neck may be placed under increased levels of stress. APPLICATION: Potential users should be made aware that HMDs could dictate modifications in neck posture, which may have detrimental effects and may compound the weight effect of the HMD.     

Differential effects of head-mounted displays on visual performance

Head-mounted displays (HMDs) virtually augment the visual world to aid visual task completion. Three types of HMDs were compared [look around (LA); optical see-through with organic light emitting diodes and virtual retinal display] to determine whether LA, leaving the observer functionally monocular, is inferior. Response times and error rates were determined for a combined visual search and Go-NoGo task. The costs of switching between displays were assessed separately. Finally, HMD effects on basic visual functions were quantified. Effects of HMDs on visual search and Go-NoGo task were small, but for LA display-switching costs for the Go-NoGo-task the effects were pronounced. Basic visual functions were most affected for LA (reduced visual acuity and visual field sensitivity, inaccurate vergence movements and absent stereo-vision). LA involved comparatively high switching costs for the Go-NoGo task, which might indicate reduced processing of external control cues. Reduced basic visual functions are a likely cause of this effect.     

A VE framework to study visual perception and action

In the real world, vision operates in harmony with self-motion yielding the observer to unambiguous perception of the three-dimensional (3D) space. In laboratory conditions, because of technical difficulties, researchers studying 3D perception have often preferred to use the substitute of a stationary observer, somehow neglecting aspects of the action-perception cycle. Recent results in visual psychophysics have proved that self-motion and visual processes interact, leading the moving observer to interpret a 3D virtual scene differently from a stationary observer. In this paper we describe a virtual environment (VE) framework which presents very interesting characteristics for designing experiments in visual perception during action. These characteristics arise in a number of ways from the design of a unique motion capture device. First, its accuracy and the minimal latency in position measurement; second, its ease of use and the adaptability to different display interfaces. Such a VE framework enables the experimenter to recreate stimulation conditions characterised by a degree of sensory coherence typical of the real world. Moreover, because of its accuracy and flexibility, the same device can be used as a measurement tool to perform elementary but essential calibration procedures. The VE framework has been used to conduct two studies which compare the perception of 3D variables of the environment in moving and in stationary observers under monocular vision. The first study concerns the perception of absolute distance, i.e. the distance separating an object and the observer. The second study refers to the perception of the orientation of a surface both in the absence and presence of conflicts between static and dynamic visual cues. In the two cases, the VE framework has enabled the design of optimal experimental conditions, permitting light to be shed on the role of action in 3D visual perception.