Use of physiological signals to predict cybersickness

Cybersickness is a common and unpleasant side effect of virtual reality immersion. We measured physiological changes that were experienced by seated subjects who interacted with a virtual environment (VE) first while viewing a display monitor and second while using a head-mounted display (HMD). Comparing results for these two conditions let us identify physiological consequences of HMD use. In both viewing conditions, subjects rated the severity of their symptoms verbally and completed a post-immersion cybersickness assessment questionnaire. In the HMD viewing condition but not in the display monitor condition, verbal reports of cybersickness severity increased significantly relative to baseline. Half of the subjects chose to exit the VE after six minutes of HMD use and reported feeling some nausea at that time. We found that changes in stomach activity, blinking, and breathing can be used to estimate post-immersion symptom scores, with R² values reaching as high as 0.75. These results suggest that HMD use by seated subjects is strongly correlated with the development of cybersickness. Finally, a linear discriminant analysis shows that physiological measures alone can be used to classify subject data as belonging to the HMD or monitor viewing condition with an accuracy of 78%.     

Natural perspective projections for head-mounted displays

The display units integrated in today's head-mounted displays (HMDs) provide only a limited field of view (FOV) to the virtual world. In order to present an undistorted view to the virtual environment (VE), the perspective projection used to render the VE has to be adjusted to the limitations caused by the HMD characteristics. In particular, the geometric field of view (GFOV), which defines the virtual aperture angle used for rendering of the 3D scene, is set up according to the display field of view (DFOV). A discrepancy between these two fields of view distorts the geometry of the VE in a way that either minifies or magnifies the imagery displayed to the user. It has been shown that this distortion has the potential to affect a user's perception of the virtual space, sense of presence, and performance on visual search tasks. In this paper, we analyze the user's perception of a VE displayed in a HMD, which is rendered with different GFOVs. We introduce a psychophysical calibration method to determine the HMD's actual field of view, which may vary from the nominal values specified by the manufacturer. Furthermore, we conducted two experiments to identify perspective projections for HMDs, which are identified as natural by subjects--even if these perspectives deviate from the perspectives that are inherently defined by the DFOV. In the first experiment, subjects had to adjust the GFOV for a rendered virtual laboratory such that their perception of the virtual replica matched the perception of the real laboratory, which they saw before the virtual one. In the second experiment, we displayed the same virtual laboratory, but restricted the viewing condition in the real world to simulate the limited viewing condition in a HMD environment. We found that subjects evaluate a GFOV as natural when it is larger than the actual DFOV of the HMD--in some cases up to 50 percent--even when subjects viewed the real space with a limited field of view.     

Rating non-technical skills: developing a behavioural marker system for use in anaesthesia

Studies of performance in medicine are often based on observation. Videotape provides a valuable tool for recording events from both real environments and simulators. When analysing observational data it is important that robust tools are used, particularly when investigating non-technical (cognitive and social) skills. This paper describes the method used to identify the key non-technical skills required in anaesthesia and to develop a behavioural marker system for their measurement. A prototype taxonomy was designed on the basis of a literature review; an examination of existing marker systems; cognitive task analysis interviews; an iterative development process involving workshops; and cross-checking in theatre. The resulting anaesthetists non-technical skills (ANTS) system comprises four skill categories (task management, team working, situation awareness, and decision making) that divide into 15 elements, each with example behaviours. Preliminary evaluation using ratings of videotaped scenarios indicated that the skills were observable and could be rated with reasonable agreement.Georgina Fletcher is now with the National Patient Safety Agency, London.     

Effects of physical exertion on task performance in modern manufacturing: a taxonomy,a review,and a model

Modern manufacturing systems have the characteristic of demanding human cognitive task performance in an environment where sudden physical demands are also possible. There is little research addressing the effects of physical work on cognitive task performance and it is not well organized for application to manufacturing systems design. This paper uses an initial three-dimensional taxonomy to review the literature. Some clear findings emerge, but many studies produced contradictory results. While part of this contradiction was due to experimental technique, some was also due to the lack of a detailed structure to guide future research. Such a model is thus proposed, using the concept of limited resources for both the cognitive and physical aspects of tasks. In addition, specific predictions of research needs for modern manufacturing systems arise.     

Vection and cybersickness generated by head-and-display motion in the Oculus Rift

Cybersickness is often experienced when viewing virtual environments through head-mounted displays (HMDs). This study examined whether vection (i.e., illusory self-motion) and mismatches between perceived and physical head motions contribute to such adverse experiences. Observers made oscillatory yaw head rotations while viewing stereoscopic optic flow through an Oculus Rift HMD. Vection and cybersickness were measured under 3 conditions of visual compensation for physical head movements: “compensated”, “uncompensated”, and “inversely compensated”. When a nearer aperture was simulated by the HMD, vection was found to be strongest in the “compensated” condition and weakest in the “inversely compensated” condition. However, vection was similar for all 3 conditions during full-field exposures. Cybersickness was most severe for the “inversely compensated” condition, but was not different for the other two conditions. We conclude that mismatches between perceived and physical head-movements can contribute strongly to cybersickness. The relationship between vection and cybersickness is weaker and appears complex.     

Instant Volume Visualization using Maximum Intensity Difference Accumulation

It has long been recognized that transfer function setup for Direct Volume Rendering (DVR) is crucial to its usability. However, the task of finding an appropriate transfer function is complex and time-consuming even for experts. Thus, in many practical applications simpler techniques which do not rely on complex transfer functions are employed. One common example is Maximum Intensity Projection (MIP) which depicts the maximum value along each viewing ray. In this paper, we introduce Maximum Intensity Difference Accumulation (MIDA), a new approach which combines the advantages of DVR and MIP. Like MIP, MIDA exploits common data characteristics and hence does not require complex transfer functions to generate good visualization results. It does, however, feature occlusion and shape cues similar to DVR. Furthermore, we show that MIDA – in addition to being a useful technique in its own right – can be used to smoothly transition between DVR and MIP in an intuitive manner. MIDA can be easily implemented using volume raycasting and achieves real-time performance on current graphics hardware.     

The shaping of information by visual metaphors

The nature of an information visualization can be considered to lie in the visual metaphors it uses to structure information. The process of understanding a visualization therefore involves an interaction between these external visual metaphors and the user's internal knowledge representations. To investigate this claim, we conducted an experiment to test the effects of visual metaphor and verbal metaphor on the understanding of tree visualizations. Participants answered simple data comprehension questions while viewing either a treemap or a node-link diagram. Questions were worded to reflect a verbal metaphor that was either compatible or incompatible with the visualization a participant was using. The results suggest that the visual metaphor indeed affects how a user derives information from a visualization. Additionally, we found that the degree to which a user is affected by the metaphor is strongly correlated with the user's ability to answer task questions correctly. These findings are a first step towards illuminating how visual metaphors shape user understanding, and have significant implications for the evaluation, application, and theory of visualization.     

Selective rendering for efficient ray traced stereoscopic images

Depth-related visual effects are a key feature of many virtual environments. In stereo-based systems, the depth effect can be produced by delivering frames of disparate image pairs, while in monocular environments, the viewer has to extract this depth information from a single image by examining details such as perspective and shadows. This paper investigates via a number of psychophysical experiments, whether we can reduce computational effort and still achieve perceptually high-quality rendering for stereo imagery. We examined selectively rendering the image pairs by exploiting the fusing capability and depth perception underlying human stereo vision. In ray-tracing-based global illumination systems, a higher image resolution introduces more computation to the rendering process since many more rays need to be traced. We first investigated whether we could utilise the human binocular fusing ability and significantly reduce the resolution of one of the image pairs and yet retain a high perceptual quality under stereo viewing condition. Secondly, we evaluated subjects’ performance on a specific visual task that required accurate depth perception. We found that subjects required far fewer rendered depth cues in the stereo viewing environment to perform the task well. Avoiding rendering these detailed cues saved significant computational time. In fact it was possible to achieve a better task performance in the stereo viewing condition at a combined rendering time for the image pairs less than that required for the single monocular image. The outcome of this study suggests that we can produce more efficient stereo images for depth-related visual tasks by selective rendering and exploiting inherent features of human stereo vision.     

Cognitive task analysis and interface design in a technical troubleshooting domain

A model of the interface design process is proposed that uses two interdependent levels of cognitive analysis: (a) study of the criterion task through the analysis of expert/novice differences, and (b) evaluation of the working user interface design through the application of practical interface analysis methodology (the GOMS model). This dual analysis is reviewed in the context of HYDRIVE, an intelligent tutoring system that has been designed to facilitate the development of aircraft hydraulics systems troubleshooting skills. Initial cognitive task analyses identified critical troubleshooting skills and procedures. However, even with an initial cognitive task analysis, the GOMS analysis resulted in significant and beneficial design changes.     

Anxiety management training as a strategy for enhancing computer user performance

This study assessed the effects of self-managed relaxation and cognitive coping skills training on the anxiety and performance of 80 apprehensive computer trainees. The influence of a computer's user friendliness upon these measures was also examined. The college student subjects were taught to use spreadsheet software and were then assigned an independent computer task to complete. Experimental sessions included a control condition as well as manipulations in which subjects were trained to use relaxation coping skills, cognitive coping skills, or a combination of the two skills. Half of the subjects worked with “friendly” and half with “less-friendly” software. Training in 3elaxation or cognitive coping skills reduced error rates and task times, but not computer anxiety. Combined relaxation and cognitive coping skills training had similar effects, as did the use of friendly software. The findings indicate that teaching anxiety management skills may be most appropriate when software is relatively difficult to use.     

Depth of focus and visual recognition of imagery presented on simultaneously viewed displays: implications for head-mounted displays

OBJECTIVE: We sought to determine the optimal focal distance for a semitransparent monocular head-mounted display (HMD) integrated with a flight simulator display and to investigate whether observers experienced visual discomfort or impaired target recognition when using an HMD set at the optimal distance. BACKGROUND: When an observer wears a monocular HMD and views a simulator display, focal distances of both displays must be within the observers' depth of focus to prevent blurred imagery. Because focal distance can vary by as much as 0.5 m in U.S. Air Force multifaceted simulator displays, we determined whether a monocular HMD could be integrated with a simulator display without blurred imagery or discomfort. METHOD: Depth of focus and visual recognition were measured with a staircase procedure, and visual discomfort was measured with a questionnaire. RESULTS: Depth of focus was 0.64 diopters in one condition tested, but it was affected by luminance level and display resolution. It was recommended that HMD focal distance equal the optical midpoint of the range of viewing distances encountered in the simulator. Moreover, wearing an HMD produced a decline in recognition performance for targets presented on the simulator display despite both displays being within observers' depth of focus and producing no visual discomfort. CONCLUSION: Monocular HMDs can be integrated with multifaceted simulator displays without blurred imagery or visual discomfort, provided that the correct focal distance is adopted. APPLICATION: For situations involving simultaneously viewed visual displays.     

Integrating specification of human-computer interface with Jackson system development

Procedures for integrating task analysis and design of the human-computer interface (HCl) with a structured system development method, Jackson system development (JSD), are described. Process structure diagrams (PSDs), the major modelling notation of JSD, described tasks, and task specifications were evaluated for cognitive complexity. Task complexity analysis helped allocation and design of human tasks and to produce computer task support specifications. The cognitive dimension of task analysis highlighted the need for task support actions, especially information display support for working memory. Dialogue specification for a direct manipulation interface design was based on the JSD entity/event model from which the interaction metaphor, screen design, and permissible manipulations were derived. Further PSDs were constructed to specify interface object management processes. Preliminary evaluation of the method was carried out using JSD analysts. Results showed the method proved easy to learn even for non-HCl specialists.     

Visual inspection with augmented reality head-mounted display: An Australian usability case study

Augmented reality (AR) is an Industry 4.0 technology. For more than a decade, advancements in AR technology and their applications have been expected to revolutionise the manufacturing industry and deliver quality and productivity gains. However, due to factors such as equipment costs, skills shortages and technological limitations of AR devices, operational deployment beyond prototypes has been constrained. Real-world, usability studies can explore barriers to implementation and improve system design. This paper details a mixed method usability case study of an AR head-mounted display (HMD) to perform a short, simple visual inspection task. Twenty-two participants from South Australian manufacturing businesses inspected a pump and pipe skid while working at height. Overall, workload demands for the task were considered acceptable and just below the “low” workload threshold (NASA Task Load Index, mean = 29.3) and the system usability was rated “average” (system usability scale, mean = 68.5). The results suggest the task did not place too high a burden on users and was an appropriate initial exposure to AR HMDs, but further refinement to the interface would be desirable before implementation to minimise frustration and promote learning. Users were enthusiastic and open-minded about the AR HMD although results indicate that even with recent advancements in AR HMD technology, interactions between the task, technology and environment continue to cause human and technical challenges—some of which are relatively straightforward to address but others are dependent on larger-scale efforts.     

The effects of 0.2 Hz varying latency with 20–100 ms varying amplitude on simulator sickness in a helmet mounted display

The relationship between the occurrence of simulator sickness (SS) and varying latency in a helmet-mounted display (HMD) was explored in this study. Previous work has always considered latency to be a constant. The aim of this research was to determine if a latency that varied over time would impact the experience of SS for HMD users. An object location task was used while viewing real, live video scenes via HMD. A planned comparisons approach was utilized with four experimental conditions, 2 of them having constant latency (0 ms added to system baseline, 200 ms added to system baseline) and 2 of them having sinusoidally varying latency (100 ms amplitude at 0.2 Hz frequency, and 20–100 ms varying amplitude at 0.2 Hz frequency). These conditions allowed for the assessment of the effects of constant latency vs. varying latency on the experience of SS. The results indicated that a varying latency is associated with greater experience of SS among HMD users than constant latency. Results also indicated, as has other recent research, that added constant latency on its own does not appear to be associated with the experience of higher levels of SS in an HMD.     

Luminance effects influencing perception of 3‐D TV imagery

Abstract— This study investigates whether screen luminance or ambient illumination has a significant effect on the perception of 3‐D TV imagery for static images and dynamic films. Two types of stimuli were shown on a multi‐view stereoscopic display: the static image, which included computer‐generated and photographic images, and dynamic film, which contained real‐life and animation images. In each treatment with a different level of screen luminance, subjects completed psychophysical and physiological measurements and subjective comfort evaluations. The results showed that when subjects viewed 3‐D static images, the ambient illumination affected psychophysical visual fatigue and screen luminance had a significant effect on subjective comfort evaluation and visual discrimination performance. However, when subjects viewed 3‐D dynamic films, screen luminance was the major factor causing psychophysical visual fatigue, and ambient illumination significantly affected subjective comfort evaluation. The outcomes contribute to knowledge concerning the suitable viewing conditions for the 3‐D viewing experience. Future work will explore the intolerance threshold of the lowest display luminance or the effect of decomposition of the screen on other physiological measurements.     

Interactive gaming reduces experimental pain with or without a head mounted display

While virtual reality environments have been shown to reduce pain, the precise mechanism that produces the pain attenuating effect has not been established. It has been suggested that it may be the ability to command attentional resources with the use of head mounted displays (HMDs) or the interactivity of the environment. Two experiments compared participants’ pain ratings to high and low levels of electrical stimulation while engaging in interactive gaming with an HMD. In the first, gaming with the HMD was compared to a positive emotion induction condition; and in the second experiment the HMD was compared to a condition in which the game was projected onto a wall. Interactive gaming significantly reduced numerical ratings of painful stimuli when compared to the baseline and affect condition. However, when the two gaming conditions were directly compared, they equally reduced participants’ pain ratings. These data are consistent with past research showing that interactive gaming can attenuate experimentally induced pain and its effects are comparable whether presented in a head mounted display or projected on a wall.     

Level of automation effects on performance, situation awareness and workload in a dynamic control task

Various levels of automation (LOA) designating the degree of human operator and computer control were explored within the context of a dynamic control task as a means of improving overall human/machine performance. Automated systems have traditionally been explored as binary function allocations; either the human or the machine is assigned to a given task. More recently, intermediary levels of automation have been discussed as a means of maintaining operator involvement in system performance, leading to improvements in situation awareness and reductions in out-of-the-loop performance problems. A LOA taxonomy applicable to a wide range of psychomotor and cognitive tasks is presented here. The taxonomy comprises various schemes of generic control system function allocations. The functions allocated to a human operator and/or computer included monitoring displays, generating processing options, selecting an 'optimal' option and implementing that option. The impact of the LOA taxonomy was assessed within a dynamic and complex cognitive control task by measuring its effect on human/system performance, situation awareness and workload. Thirty subjects performed simulation trials involving various levels of automation. Several automation failures occurred and out-of-the-loop performance decrements were assessed. Results suggest that, in terms of performance, human operators benefit most from automation of the implementation portion of the task, but only under normal operating conditions; in contrast, removal of the operator from task implementation is detrimental to performance recovery if the automated system fails. Joint human/system option generation significantly degraded performance in comparison to human or automated option generation alone. Lower operator workload and higher situation awareness were observed under automation of the decision making portion of the task (i.e. selection of options), although human/system performance was only slightly improved. The implications of these findings for the design of automated systems are discussed.     

Cervicobrachial muscle response to cognitive load in a dual-task scenario

People working in an office environment often have to deal with significant cognitive workload due to the coordination of multiple, simultaneous tasks. The objective of this research was to examine the impact of cognitive load in office-type tasks on physical-stress response, using a dual-task paradigm involving a primary cognitive task and secondary typing task. The central hypothesis of this research was that altering the demands of the cognitive task would lead to a difference in physical stress-level and performance. Cognitive load was manipulated by presenting participants with three different types of cognitive tasks described in taxonomy, including skill-, rule-, and knowledge-based tasks. Dependent variables examined in the study included: (1) electromyographic activity of the upper trapezius (pars descendens) and cervical erector spinae muscles, (2) performance in a secondary typing task, and (3) subjective measures of stress and cognitive workload. The results of this study revealed that the primary task causing the highest level of perceived workload also produced 61% higher muscle activity in the right trapezius, and 6 and 11% higher activity in the left and right cervical erector spinae, respectively, in comparison to muscle activity associated with the cognitive task causing the lowest perceived workload. With respect to performance, a 23% decrease was observed in typing productivity when the rule-based task was completed simultaneously vs. typing in the absence of any additional cognitive task (the baseline condition). This information may be used to better organize work activities in office environments to increase performance and reduce stress.     

Cervicobrachial muscle response to cognitive load in a dual-task scenario

People working in an office environment often have to deal with significant cognitive workload due to the coordination of multiple, simultaneous tasks. The objective of this research was to examine the impact of cognitive load in office-type tasks on physical-stress response, using a dual-task paradigm involving a primary cognitive task and secondary typing task. The central hypothesis of this research was that altering the demands of the cognitive task would lead to a difference in physical stress-level and performance. Cognitive load was manipulated by presenting participants with three different types of cognitive tasks described in Rasmussen's () taxonomy, including skill-, rule-, and knowledge-based tasks. Dependent variables examined in the study included: (1) electromyographic activity of the upper trapezius (pars descendens) and cervical erector spinae muscles, (2) performance in a secondary typing task, and (3) subjective measures of stress and cognitive workload. The results of this study revealed that the primary task causing the highest level of perceived workload also produced 61% higher muscle activity in the right trapezius, and 6 and 11% higher activity in the left and right cervical erector spinae, respectively, in comparison to muscle activity associated with the cognitive task causing the lowest perceived workload. With respect to performance, a 23% decrease was observed in typing productivity when the rule-based task was completed simultaneously vs. typing in the absence of any additional cognitive task (the baseline condition). This information may be used to better organize work activities in office environments to increase performance and reduce stress.     

Perceived orientation in physical and virtual environments: changes in perceived orientation as a function of idiothetic information available

Two experiments examined perceived spatial orientation in a small environment as a function of experiencing that environment under three conditions: real-world, desktop-display (DD), and head-mounted display (HMD). Across the three conditions, participants acquired two targets located on a perimeter surrounding them, and attempted to remember the relative locations of the targets. Subsequently, participants were tested on how accurately and consistently they could point in the remembered direction of a previously seen target. Results showed that participants were significantly more consistent in the real-world and HMD conditions than in the DD condition. Further, it is shown that the advantages observed in the HMD and real-world conditions were not simply due to nonspatial response strategies. These results suggest that the additional idiothetic information afforded in the real-world and HMD conditions is useful for orientation purposes in our presented task domain. Our results are relevant to interface design issues concerning tasks that require spatial search, navigation, and visualization.