Effects of 2D wedge design as a wayfinding facilitator in a 3D virtual environment

The purpose of this study is to investigate users' wayfinding task performance and subjective preference in a three‐dimensional (3D) virtual environment. The research variables are interaction mode (touch sensitive vs body movement modes), viewing perspective [first‐person perspective (1PP) versus third‐person perspective (3PP)], and gender (man versus woman). Three difficult levels of tasks were conducted, i.e., A, A‐P, and A‐A types. The experiment is 2 × 2 × 2 between‐subject design. Participants needed to complete three wayfinding tasks and fill out subjective preference questionnaires regarding task fluency, satisfaction, system usability scale, and NASA Task Load Index (NASA‐TLX). Fifty‐six participants (28 men and 28 women) were invited using convenient sampling method. The results are as follows: (1) Participants adopting touch‐sensitive mode performed better than body movement mode. (2) Participants using touch‐sensitive mode could have better task fluency and higher SUS score, and with less effort and frustration. (3) Participants adopting 3PP had better wayfinding task performance than first‐person perspective. They also had higher system usability scale score and spent less effort for having wider viewing perspective. (4) The gender variable only showed significant effect in A‐P type of wayfinding task. Men performed better than women. It is concluded that adopting touch‐sensitive interaction and 3PP designs can better facilitate users' wayfinding tasks.     

Effects of control order,augmented feedback,input device and practice on tracking performance and perceived workload

Virtual interfaces to advanced human-machine systems will present operators with a variety of perceptual-motor challenges. To inform the virtual interface design processes, the present experiments examined the effects of track order, level of knowledge of performance, type of control device, and the extent of practice on tracking performance and associated mental workload. Tracking was assessed by root mean square error. Subjective workload was measured using both the NASA Task Load Index (TLX) and the Subjective Workload Assessment Technique (SWAT). Results indicated non-linear effects, where tracking error and subjective workload both increased non-proportionally with track order. Trackball use resulted in more accurate performance and was judged to be of lower subjective workload than input using a mouse. Augmented knowledge of performance had little effect on either performance or workload. There were a number of interactions affecting performance that were replicated in perceived workload. Over acquisition trials, second-order tracking exhibited continuous improvement. This capability was retained even after a 30-day rest interval. Decrease in workload followed performance improvement in both initial acquisition and subsequent retention phases. The two subjective workload scales were essentially equivalent in terms of their sensitivity to task manipulations. These results support the direct association between workload and performance and confirms the use of workload in helping to evaluate the influence of diverse task-related demands. The implications for the design of virtual interfaces to real-world systems are examined in the light of these findings.     

Evaluation of hydraulic excavator Human–Machine Interface concepts using NASA TLX

This study evaluated newly proposed Human–Machine Interface (HMI) design concepts for improving the ergonomics of hydraulic excavators. The design concepts were based on an augmented interaction technique which involved the use of heads-up display (HUD) and coordinated control as HMI elements. Two alternative HMI designs were elaborated in order to separately evaluate the ergonomic impacts of the head-up display and the coordinated control by comparing them to the standard HMI design. The effectiveness of these three HMI designs in terms of the reduction of the operators' mental and physical workload were assessed by conducting experiments utilizing human subjects, ages 23–35 years. The National Aeronautics and Space Administration's Task Load Index (NASA TLX) method was used for collecting subjective workload scores based on a weighted average of ratings of six factors: Mental Demand, Physical Demand, Temporal Demand, Own Performance, Effort, and Frustration Level. The results showed that the type of HMI design affects different aspects of the operator's workload. Indeed, it showed how the proposed augmented interaction is an effective solution for reducing the ergonomic gaps in terms of mental workload, and to a lesser extent the physical workload, subjected by the standard HMI design.     

Empirical studies to investigate the usability of text- and image-based CAPTCHAs

This article reports the findings of two empirical studies investigating the usability of image-based and text-based CAPTCHAs that authenticate users accessing the web. The first investigated four image-based CAPTCHAs, Asirra, ESP-PIX, SQ-PIX and IMAGINATION, with twenty participants interacting with five of each type on a desktop computer. The participants then completed the computer system usability questionnaire (CSUQ), the NASA Task Load Index (NASA-TLX), and a final post-test subjective questionnaire ranking the CAPTCHAs based on preference. The results found statistically significant differences among the CAPTCHAs for all dependent variables except task completion time, with participants preferring Asirra and ESP-PIX. The second compared the usability of these two with the two most commonly used text-based CAPTCHAs, ReCAPTCHA and Google's CAPTCHA. After receiving training on Asirra and ESP-PIX, twenty-four participants completed 10 trials with each type on a desktop computer, subsequently completing the CSUQ and the NASA-TLX followed by a retrospective think-aloud session. The results of this second study indicated that Google's CAPTCHA and ReCAPTCHA took the least amount of time to complete. The workload associated with Asirra and ReCAPTCHA was lower than that for ESP-PIX and Google's CAPTCHA. Perceived overall usability tended to be highest for Asirra and ReCAPTCHA. Participant comments indicated that Google's CAPTCHA was too visually distorted and ESP-PIX had commonalities between the images that weren't immediately obvious, factors perhaps contributing to their lower perceived usability scores. Overall, ReCAPTCHA is recommended for use on a desktop computer because of its high perceived usability, low completion time, and low workload.     

Effects of stress and effort on self-rated reports in experimental study of design activities

This paper investigates the relationships between subjective rating measure and physiological measure of mental stress and mental effort during design activities. Mental stress and mental effort were captured by skin conductance and EEG beta2 power (20–30 Hz) whereas self-rated mental stress and self-rated mental effort were obtained using the NASA Task Load Index. A strong association between self-rated effort and EEG beta2 power was found in several design tasks. The analysis shows that self-rating is by itself a mental activity which may be affected by psychological stress and may be influenced by the amount of cognitive effort allocated. Researchers who rely on subjective rating should take into account the stress and effort of respondents during the rating activities to ensure the validity of the self-report measures. The study also demonstrated that design tasks induced mental stress that continued to stay above the baseline even after the tasks were completed.     

Monitoring of mental workload levels during an everyday life office-work scenario

Personal and ubiquitous healthcare applications offer new opportunities to prevent long-term health damage due to increased mental workload by continuously monitoring physiological signs related to prolonged high workload and providing just-in-time feedback. In order to achieve a quantification of mental load, different load levels that occur during a workday have to be discriminated. In this work, we present how mental workload levels in everyday life scenarios can be discriminated with data from a mobile ECG logger by incorporating individual calibration measures. We present an experiment design to induce three different levels of mental workload in calibration sessions and to monitor mental workload levels in everyday life scenarios of seven healthy male subjects. Besides the recording of ECG data, we collect subjective ratings of the perceived workload with the NASA Task Load Index (TLX), whereas objective measures are assessed by collecting salivary cortisol. According to the subjective ratings, we show that all participants perceived the induced load levels as intended from the experiment design. The heart rate variability (HRV) features under investigation can be classified into two distinct groups. Features in the first group, representing markers associated with parasympathetic nervous system activity, show a decrease in their values with increased workload. Features in the second group, representing markers associated with sympathetic nervous system activity or predominance, show an increase in their values with increased workload. We employ multiple regression analysis to model the relationship between relevant HRV features and the subjective ratings of NASA-TLX in order to predict the mental workload levels during office-work. The resulting predictions were correct for six out of the seven subjects. In addition, we compare the performance of three classification methods to identify the mental workload level during office-work. The best results were obtained with linear discriminant analysis (LDA) that yielded a correct classification for six out of the seven subjects. The k-nearest neighbor algorithm (k-NN) and the support vector machine (SVM) resulted in a correct classification of the mental workload level during office-work for five out of the seven subjects.     

Effects of Airborne Data Link Communication on Demands, Workload and Situation Awareness

An airborne air-to-ground data link communication interface was evaluated in a multi-sector-planning scenario using an Airbus A 340 full flight simulator. In a close-to-reality experimental setting, eight professional crews performed a flight mission in a mixed voice/data link environment. Experimental factors were the medium (voice vs. data link), workload (low vs. high) and the role in the cockpit (pilot flying vs. pilot non-flying). Data link communication and the usability of the newly developed communication interface were rated positively by the pilots, but there is a clear preference for using a data link only in the phase of cruise. Cognitive demands were determined for selected sections of en-route flight. Demands are affected mainly by increased communication needs. In the pilots’ view, although a data link has no effect on safety or the possibilities of intervention, it causes more problems. The subjective workload, as measured with the NASA Task Load Index, increased moderately under data link conditions. A data link has no general effect on pilots’ situation awareness although flight plan negotiations with a data link cause a distraction of attention from monitoring tasks. The use of a data link has an impact on air-to-ground as well as intra-crew communication. Under data link conditions the pilot non-flying plays a more active role in the cockpit. Before introducing data link communication, several aspects of crew resource management have to be reconsidered. Correspondence and offprint requests to: T. Müller, Technical University of Berlin, Institute of Psychology and Ergonomics, Department of Human–Machine Systems, Jebensstrasse 1, 10623 Berlin, Germany.     

A covert attention P300-based brain-computer interface: Geospell

The Farwell and Donchin P300 speller interface is one of the most widely used brain-computer interface (BCI) paradigms for writing text. Recent studies have shown that the recognition accuracy of the P300 speller decreases significantly when eye movement is impaired. This report introduces the GeoSpell interface (Geometric Speller), which implements a stimulation framework for a P300-based BCI that has been optimised for operation in covert visual attention. We compared the Geospell with the P300 speller interface under overt attention conditions with regard to effectiveness, efficiency and user satisfaction. Ten healthy subjects participated in the study. The performance of the GeoSpell interface in covert attention was comparable with that of the P300 speller in overt attention. As expected, the effectiveness of the spelling decreased with the new interface in covert attention. The NASA task load index (TLX) for workload assessment did not differ significantly between the two modalities. PRACTITIONER SUMMARY: This study introduces and evaluates a gaze-independent, P300-based brain-computer interface, the efficacy and user satisfaction of which were comparable with those off the classical P300 speller. Despite a decrease in effectiveness due to the use of covert attention, the performance of the GeoSpell far exceeded the threshold of accuracy with regard to effective spelling.     

Order effects and the weighting process in workload assessment

Operators controlling a power grid were subjects in a mental workload experiment in which they performed their normal tasks under simulated quiet and busy conditions. Workload was assessed using the NASA Task-Load Index (TLX). It was found that there were significant differences between the two conditions, but there were no order effects. The value of the weighting procedure was assessed by correlating unweighted and weighted workload ratings. Although the correlations were high, they were not perfect, indicating some value for weighting the dimensions of the TLX. Large individual differences were observed in perceived workload.     

Controversy in human factors constructs and the explosive use of the NASA-TLX: a measurement perspective

Situation awareness and workload are popular constructs in human factors science. It has been hotly debated whether these constructs are scientifically credible, or whether they should merely be seen as folk models. Reflecting on the works of psychophysicist Stanley Smith Stevens and of measurement theorist David Hand, we suggest a resolution to this debate, namely that human factors constructs are situated towards the operational end of a representational–operational continuum. From an operational perspective, human factors constructs do not reflect an empirical reality, but they aim to predict. For operationalism to be successful, however, it is important to have suitable measurement procedures available. To explore how human factors constructs are measured, we focused on (mental) workload and its measurement by questionnaires and applied a culturomic analysis to investigate secular trends in word use. The results reveal an explosive use of the NASA Task Load Index (TLX). Other questionnaires, such as the Cooper Harper rating scale and the Subjective Workload Assessment Technique, show a modest increase, whereas many others appear short lived. We found no indication that the TLX is improved by iterative self-correction towards optimal validity, and we argue that usage of the NASA-TLX has become dominant through a Matthew effect. Recommendations for improving the quality of human factors research are provided.     

Investigating the validity of subjective workload rating (NASA TLX) and subjective situation awareness rating (SART) for cognitively complex human–machine work

Subjective workload and situation awareness measures, such as the NASA task load index (TLX) and the situational awareness rating technique (SART), are frequently used in human–system evaluation. However, the interpretation of these ratings is debated. In this study, empirical evidence for the measures' theoretical assumptions was investigated by comparing operators' ratings collected immediately after performing a scenario and ratings collected after operators' acquisition through a video review of the scenario, knowledge of actual system states. Eighteen licensed control room operators participated in the simulator study, running 12 relatively challenging scenarios. It was found that the interpretation of TLX items involving introspection remained stable after operators acquired factual scenario knowledge, while the interpretation of items involving the perception of external events, such as situation awareness and performance, depended on the operators' scenario knowledge. The result shows that operators’ ratings could discriminate between mental effort, performance, frustration, and situation awareness. No clear evidence for the SART index as a measure of situation awareness was found. Instead, a subjective situation awareness measure developed for this study was distinct from workload and related to operator performance, showing that this type of measure warrants future investigation of its validity. The study findings help in developing measurement procedures and interpreting subjective measures. Finally, the study reveals that informing operators about the scenario can provide useful subjective ratings of situation awareness and performance. Future research should include procedures for how to inform participants adequately and efficiently in subjective assessments.     

Visual inspection with augmented reality head-mounted display: An Australian usability case study

Augmented reality (AR) is an Industry 4.0 technology. For more than a decade, advancements in AR technology and their applications have been expected to revolutionise the manufacturing industry and deliver quality and productivity gains. However, due to factors such as equipment costs, skills shortages and technological limitations of AR devices, operational deployment beyond prototypes has been constrained. Real-world, usability studies can explore barriers to implementation and improve system design. This paper details a mixed method usability case study of an AR head-mounted display (HMD) to perform a short, simple visual inspection task. Twenty-two participants from South Australian manufacturing businesses inspected a pump and pipe skid while working at height. Overall, workload demands for the task were considered acceptable and just below the “low” workload threshold (NASA Task Load Index, mean = 29.3) and the system usability was rated “average” (system usability scale, mean = 68.5). The results suggest the task did not place too high a burden on users and was an appropriate initial exposure to AR HMDs, but further refinement to the interface would be desirable before implementation to minimise frustration and promote learning. Users were enthusiastic and open-minded about the AR HMD although results indicate that even with recent advancements in AR HMD technology, interactions between the task, technology and environment continue to cause human and technical challenges—some of which are relatively straightforward to address but others are dependent on larger-scale efforts.     

Evaluation of P300-Based Brain-Computer Interface in Real-World Contexts

Despite recent advances in brain-computer interface (BCI) development, system usability still remains a large oversight. The goal of this study was to investigate the usability of a P300-based BCI system, P300 Speller, by assessing how background noise and interface color contrast affect user performance and BCI usage preference. Fifteen able-bodied participants underwent a 2 (low and high interface color contrast)?×?3 (low, medium, and high background noise level) within-subjects design experiment, in which participants were asked to type six 10-character phrases in the P300 Speller paradigm. The overall accuracy in the study was 80.2%. Participants showed higher accuracy, higher information transfer rate, bigger amplitude, and smaller latency in the high interface color contrast condition than in the low contrast condition. Participants had better performance in the noisy condition than in the quiet condition, but the background noise effects were not statistically significant in the present study. These results should give some insight to the real-world applicability of the current P300 Speller as a nonmuscular communication system, especially for individuals with severe neuromuscular disabilities.     

Predicting dual-task performance with the Multiple Resources Questionnaire (MRQ)

OBJECTIVE: The objective was to assess the validity of the Multiple Resources Questionnaire (MRQ) in predicting dual-task interference. BACKGROUND: Subjective workload measures such as the Subjective Workload Assessment Technique (SWAT) and NASA Task Load Index are sensitive to single-task parameters and dual-task loads but have not attempted to measure workload in particular mental processes. An alternative is the MRQ. METHOD: In Experiment 1, participants completed simple laboratory tasks and the MRQ after each. Interference between tasks was then correlated to three different task similarity metrics: profile similarity, based on r(2) between ratings; overlap similarity, based on summed minima; and overall demand, based on summed ratings. Experiment 2 used similar methods but more complex computer-based games. RESULTS: In Experiment 1 the MRQ moderately predicted interference (r = +.37), with no significant difference between metrics. In Experiment 2 the metric effect was significant, with overlap similarity excelling in predicting interference (r = +.83). Mean ratings showed high diagnosticity in identifying specific mental processing bottlenecks. CONCLUSION: The MRQ shows considerable promise as a cognitive-process-sensitive workload measure. APPLICATION: Potential applications of the MRQ include the identification of dual-processing bottlenecks as well as process overloads in single tasks, preparatory to redesign in areas such as air traffic management, advanced flight displays, and medical imaging.     

Multimodal feedback: an assessment of performance and mental workload

Multimodal interfaces offer great potential to humanize interactions with computers by employing a multitude of perceptual channels. This paper reports on a novel multimodal interface using auditory, haptic and visual feedback in a direct manipulation task to establish new recommendations for multimodal feedback, in particular uni-, bi- and trimodal feedback. A close examination of combinations of uni-, bi- and trimodal feedback is necessary to determine which enhances performance without increasing workload. Thirty-two participants were asked to complete a task consisting of a series of 'drag-and-drops' while the type of feedback was manipulated. Each participant was exposed to three unimodal feedback conditions, three bimodal feedback conditions and one trimodal feedback condition that used auditory, visual and haptic feedback alone, and in combination. Performance under the different conditions was assessed with measures of trial completion time, target highlight time and a self-reported workload assessment captured by the NASA Task Load Index (NASA-TLX). The findings suggest that certain types of bimodal feedback can enhance performance while lowering self-perceived mental demand.     

Human performance assessment of a single air traffic controller conducting multiple remote tower operations

The innovative concept of multiple remote tower operations (MRTO) can maximize cost savings by applying video panorama‐based remote tower working positions, which can facilitate fewer air traffic controllers (ATCO) to provide the air traffic services (ATS) function for more airports. Five subject‐matter experts, qualified remote tower ATCOs, participated in this research work by applying the human error template (HET) and comparing workload between physical tower operations and MRTO using NASA‐TLX (Task Load Index). The results demonstrate that augmented visualization provided sufficient technical support for a single ATCO to perform tasks originally designed to be performed by four ATCOs, however, the demands of the associated multiple tasks induced significant workload. There were significant differences in ATCOs’ mental demand, temporal demand, effort, and frustration between MRTO and physical tower operations. This innovative technology may induce human–computer interaction (HCI) issues that impact ATCO's perceived workload. This creates a need for further research on how to manage ATCO's workload in a multiple remote tower environment. This research work provided scientific evidence that MRTO can achieve the objectives of Single European Sky Air Traffic Management Research program. The findings can be applied to both ATCO training design and remote tower system design.     

Multimodal feedback: an assessment of performance and mental workload

Multimodal interfaces offer great potential to humanize interactions with computers by employing a multitude of perceptual channels. This paper reports on a novel multimodal interface using auditory, haptic and visual feedback in a direct manipulation task to establish new recommendations for multimodal feedback, in particular uni-, bi- and trimodal feedback. A close examination of combinations of uni-, bi- and trimodal feedback is necessary to determine which enhances performance without increasing workload. Thirty-two participants were asked to complete a task consisting of a series of ‘drag-and-drops’ while the type of feedback was manipulated. Each participant was exposed to three unimodal feedback conditions, three bimodal feedback conditions and one trimodal feedback condition that used auditory, visual and haptic feedback alone, and in combination. Performance under the different conditions was assessed with measures of trial completion time, target highlight time and a self-reported workload assessment captured by the NASA Task Load Index (NASA-TLX). The findings suggest that certain types of bimodal feedback can enhance performance while lowering self-perceived mental demand.     

Enheduanna – A Manifesto of Falling: first demonstration of a live brain-computer cinema performance with multi-brain BCI interaction for one performer and two audience members

The new commercial-grade Electroencephalography (EEG)-based Brain-Computer Interfaces (BCIs) have led to a phenomenal development of applications across health, entertainment and the arts, while an increasing interest in multi-brain interaction has emerged. In the arts, there is already a number of works that involve the interaction of more than one participants with the use of EEG-based BCIs. However, the field of live brain-computer cinema and mixed-media performances is rather new, compared to installations and music performances that involve multi-brain BCIs. In this context, we present the particular challenges involved. We discuss Enheduanna – A Manifesto of Falling, the first demonstration of a live brain-computer cinema performance that enables the real-time brain-activity interaction of one performer and two audience members; and we take a cognitive perspective on the implementation of a new passive multi-brain EEG-based BCI system to realise our creative concept. This article also presents the preliminary results and future work.     

Tester interactivity makes a difference in search-based software testing: A controlled experiment

Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required.The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants.Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system.Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment.Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results.In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance.Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result.     

Brain–Computer Interfaces for Multimodal Interaction: A Survey and Principles

For decades, brain–computer interfaces (BCIs) have been used for restoring the communication and mobility of disabled people through applications such as spellers, web browsers, and wheelchair controls. In parallel to advances in computational intelligence and the production of consumer BCI products, BCIs have recently started to be considered as alternative modalities in human–computer interaction (HCI). One of the popular topics in HCI is multimodal interaction (MMI), which deals with combining multiple modalities in order to provide powerful, flexible, adaptable, and natural interfaces. This article discusses the situation of BCI as a modality within MMI research. State-of-the-art, real-time multimodal BCI applications are surveyed in order to demonstrate how BCI can be helpful as a modality in MMI. It is shown that multimodal use of BCIs can improve error handling, task performance, and user experience and that they can broaden the user spectrum. The techniques for employing BCI in MMI are described, and the experimental and technical challenges with some guidelines to overcome these are shown. Issues in input fusion, output fission, integration architectures, and data collection are covered.