Exploring Relationships Between Eye Tracking and Traditional Usability Testing Data

This study explored the relationships between eye tracking and traditional usability testing data in the context of analyzing the usability of Algebra Nation?, an online system for learning mathematics used by hundreds of thousands of students. Thirty-five undergraduate students (20 females) completed seven usability tasks in the Algebra Nation? online learning environment. The participants were asked to log in, select an instructor for the instructional video, post a question on the collaborative wall, search for an explanation of a mathematics concept on the wall, find information relating to Karma Points (an incentive for engagement and learning), and watch two instructional videos of varied content difficulty. Participants’ eye movements (fixations and saccades) were simultaneously recorded by an eye tracker. Usability testing software was used to capture all participants’ interactions with the system, task completion time, and task difficulty ratings. Upon finishing the usability tasks, participants completed the System Usability Scale. Important relationships were identified between the eye movement metrics and traditional usability testing metrics such as task difficulty rating and completion time. Eye tracking data were investigated quantitatively using aggregated fixation maps, and qualitative examination was performed on video replay of participants’ fixation behavior. Augmenting the traditional usability testing methods, eye movement analysis provided additional insights regarding revisions to the interface elements associated with these usability tasks.     

Modeling the Influences of Cyclic Top-Down and Bottom-Up Processes for Reinforcement Learning in Eye Movements

Understanding and reproducing complex human oculomotor behaviors using computational models is a challenging task. In this paper, two studies are presented, which focus on the development and evaluation of a computational model to show the influences of cyclic top-down and bottom-up processes on eye movements. To explain these processes, reinforcement learning was used to control eye movements. The first study showed that, in a picture-viewing task, different policies obtained from different picture-viewing conditions produced different types of eye movement patterns. In another visual search task, the second study illustrated that feedback information from each saccadic eye movement could be used to update the model's eye movement policy, generating different patterns in the following saccade. These two studies demonstrate the value of an integrated reinforcement learning model in explaining both top-down and bottom-up processes of eye movements within one computational model.     

Showing a model's eye movements in examples does not improve learning of problem-solving tasks

Eye movement modeling examples (EMME) are demonstrations of a computer-based task by a human model (e.g., a teacher), with the model's eye movements superimposed on the task to guide learners' attention. EMME have been shown to enhance learning of perceptual classification tasks; however, it is an open question whether EMME would also improve learning of procedural problem-solving tasks. We investigated this question in two experiments. In Experiment 1 (72 university students, M_age = 19.94), the effectiveness of EMME for learning simple geometry problems was addressed, in which the eye movements cued the underlying principle for calculating an angle. The only significant difference between the EMME and a no eye movement control condition was that participants in the EMME condition required less time for solving the transfer test problems. In Experiment 2 (68 university students, M_age = 21.12), we investigated the effectiveness of EMME for more complex geometry problems. Again, we found no significant effects on performance except for time spent on transfer test problems, although it was now in the opposite direction: participants who had studied EMME took longer to solve those items. These findings suggest that EMME may not be more effective than regular video examples for teaching procedural problem-solving skills.     

Error detection through mouse movement in an online adaptive learning environment

While response time and accuracy indicate overall performance, their value in uncovering cognitive processes, underlying learning, is limited. A promising online measure, designed to track decision-making, is computer mouse tracking, where mouse attraction towards different locations may reflect the consideration of alternative response options. Using a speedy arithmetic multiple-choice game in an online adaptive learning environment, we examined whether mouse movements could reflect arithmetic difficulties when error rates are low. Results showed that mouse movements towards alternative responses in correctly answered questions mapped onto the frequency of errors made in this online learning system. This mapping was stronger for the younger children, as well as for easy arithmetic problems. On an individual level, users showed more mouse movement towards their previously made response errors than towards other alternative options. This opens the possibility of adapting feedback and instruction on an individual basis through mouse tracking.     

The Role of Visual Feedback and Age When Grasping,Transferring and Passing Objects in Virtual Environments

ABSTRACT

Visual feedback about one’s own movements can be important for effective performance in natural and computer-generated environments. Previous work suggested that, for young adults, performance in virtual environments was influenced by the presence of a crude representation of the hand in a task-specific fashion. The current study was performed to determine whether this pattern of results holds across the lifespan. Specifically, we were interested in determining whether a representation of the hand is useful for movement performance in children, middle adults, and older adults when they reach to grasp an object, transfer it between their two hands, or receive it from a partner. Surprisingly, visual feedback condition had very little effect on performance in the current experiment, with the exception that participants in all three age groups altered how wide they opened their hand when grasping depending on visual condition. Patterns of reach and grasp performance changed across task depending on age, with older adults displaying different patterns than children and middle-aged adults. These results suggest that feedforward planning and the use of feedback are modulated by both age and task.     

Predictive capability of cognitive ability and cognitive style for spaceflight emergency operation performance

This study explores the effects of cognitive ability (information seeking, inference, spatial recognition, attention span, and attention allocation) and cognitive style (active-reflective, sensing-intuitive, visual-verbal, and sequential-global) on task performance of simulated spaceflight emergency operations that require judgment and operation on a Chinese spaceflight instrument board and the possible interaction effect with training experience. The performance criteria included task completion time and number of human errors. It was found that inference ability, spatial recognition ability, and attention span had significant effects on task completion time, while attention allocation ability had significant effect on the number of error. The participants with a sequential cognitive style made significantly fewer errors than those with a global cognitive style. Training experience significantly decreased task completion time. The participants with sequential cognitive style learnt faster than those with global cognitive style in the spaceflight instrument operations. With increasing training experience, the predictive capability of cognitive ability on performance decreased, whereas the predictive capability of the sequential-global cognitive style on performance increased.     

Haptic User Interfaces for the Visually Impaired: Implications for Haptically Enhanced Science Learning Systems

The overall quality of haptic user interfaces designed to support visually impaired students' science learning through sensorial feedback was systematically studied to investigate task performance and user behavior. Fourteen 6th- to 11th-grade students with visual impairments recruited from a state-funded blind school were asked to perform three main tasks (i.e., menu selection, structure exploration, and force recognition) using haptic user interfaces and a haptic device. This study used several dependent measures that are categorized into three types of variables: (a) task performance including success rate, workload, and task completion time; (b) user behavior defined as cursor movements proportionately represented from the user's cursor positional data; and (c) user preference. Results showed that interface type has significant effects on task performance, user behavior, and user preference, with varying degree of impact to participants with severe visual impairments performing the tasks. The results of this study as well as a set of refined design guidelines and principles should provide insights to the future research of haptic user interfaces that can be used when developing haptically enhanced science learning systems for the visually impaired.     

Effects of prior knowledge and joint attention on learning from eye movement modelling examples

Eye movement modelling examples (EMMEs) are instructional videos of a model's demonstration and explanation of a task that also show where the model is looking. EMMEs are expected to synchronize students' visual attention with the model's, leading to better learning than regular video modelling examples (MEs). However, synchronization is seldom directly tested. Moreover, recent research suggests that EMMEs might be more effective than ME for low prior knowledge learners. We therefore used a 2 × 2 between-subjects design to investigate if the effectiveness of EMMEs (EMMEs/ME) is moderated by prior knowledge (high/low, manipulated by pretraining), applying eye tracking to assess synchronization. Contrary to expectations, EMMEs did not lead to higher learning outcomes than ME, and no interaction with prior knowledge was found. Structural equation modelling shows the mechanism through which EMMEs affect learning: Seeing the model's eye movements helped learners to look faster at referenced information, which was associated with higher learning outcomes.     

Improving interface quality: an investigation of human-computer interaction task learning

Abstract

User learning is of critical importance in evaluating interface usability (and in turn interface quality). The focus of this research in on interface leamability, where a stochastic model represents the learning process required for successful completion of human-computer interaction tasks. The parameter used to quantify learning is a learning rale. Of interest here is the validation of learning rate as a measure of interface quality. Learning rate was validated against two traditional measures of interface quality: task completion time, and error frequency. SuperCard(tm), a Macintosh project utility, provided an empirical learning environment in which 32 participants learned 16 fundamental SuperCard tasks. Results of correlation analyses suggested the usefulness of learning rate as an indicator of interfacequality. Our learning rate analysis identified four tasks presenting learning difficulties. (Analysis of task completion times identified two of these four tasks, and error frequency analysis identified one). Learning rate data captured all of the information available from the two traditional interface quality measures and identified two tasks disregarded by them. Incorporating learning rates in the interface evaluation process precludes time-intensive video tape analysis typically required by more traditional interface quality measures.     

Effect of tactile feedback on movement speed and precision during work-related tasks using a computer mouse

Effects of tactile feedback on movement accuracy and speed were studied. Younger and older participants performed three tasks (1, select and drag word; 2, menu navigation; 3, select and drag cell) using commercial software and a mouse with or without tactile feedback. Task time and error number were recorded. Tasks were divided according to presence or absence of tactile feedback, and participants were divided into subgroups (high, average, low) based on Task 1 performance. Overall, older participants took longer (p < .0001) and made more errors (p < .001) than younger participants. There was an effect of feedback by task in younger participants for all six outcomes (p < 0.02). At the task level, with feedback, younger participants reduced performance time (13%) and errors (24%) on Task 1. Low- and average-performance younger participants benefited most from feedback for Task 1. Older low-performance participants also benefited from feedback for Task 1. For Task 3, older participants tended to take more time and make more errors with feedback. Tactile feedback may enhance performance when feedback is event related. Older people may not integrate sensation as well as younger individuals to enhance performance. Potential applications of this research include the development of tactile feedback interfaces to facilitate computer use.     

Learning differences and eye fixation patterns in virtual and physical science laboratories

This project analyzed high school students' performance and eye movement while learning in a simulation-based laboratory (SBL) and a microcomputer-based laboratory (MBL). Although the SBL and the MBL both used computers to collect, graph, and analyze data, the MBL involved manual manipulation of concrete materials, whereas the SBL displayed everything on a monitor. Fifty senior high school students at three urban public high schools in Taipei were randomly assigned to the MBL and SBL settings. The participants conducted the Boyle's Law experiment with an accompanying worksheet and completed pre- and post-conceptual tests. FaceLAB and ASL MobileEye were used to record each participant's eye movements in the SBL and MBL settings, respectively. The results showed that lower achievers improved significantly from the pre-to post-conceptual tests. The SBL group tended to carry out more experiments. Moreover, the MBL group's performance on the worksheet was moderately correlated with their post-test. However, this correlation was not found for the SBL group. Furthermore, at the beginning of the laboratories, the SBL group had a higher percentage of fixations with longer fixation duration, which implies more attention to and deeper cognitive processing of the equipment and running experiments, while the MBL group focused on the worksheet. This study concludes that, for e-learning like SBLs, students tend to start off doing an experiment, and then think about the questions on the worksheets, whereas for physical laboratories like MBLs, they tend to think before doing.     

Effects on performance of adding simple complementary auditory feedback to a visual-spatial task

The current study examined the effect of simple complementary auditory feedback (CAF) on performance and performance strategies in a puzzle completion task. Fifty-three undergraduate students were randomly assigned to one of two training groups: a CAF group, which received simple auditory feedback during the task; and a Control group, which did not receive the auditory feedback. The difference between the higher and lower achievers (in terms of time needed to complete the puzzle) was larger in the CAF group compared to the Control group. As for performance strategies, participants in the CAF group engaged less in strategies which involved pre-planning (for example, separating out edge pieces) and focused more on trying to maximize short-term achievements (for example, through trial and error). The findings suggest that CAF which entails low-level feedback is not recommended for weaker performers, for whom it promotes less-effective strategies focused on short-term goals.     

Assessing mental workload in virtual reality based EOT crane operations: A multi-measure approach

BackgroundEye-movement metrics and subjective workload measures are extensively used to determine mental workload of participants. The aim of this study was to assess Electric overhead travelling (EOT) crane operators’ mental workload variability based on eye movement metrics such as fixation frequency, fixation duration, saccade duration, saccade amplitude, and fixation/saccade ratio during EOT crane operations in virtual reality (VR) based EOT crane simulator.MethodsA 2^k (k = 3) factorial experiment with factors namely, hazardous scenario, activity level, and trial was designed and conducted to demonstrate the proposed assessment approach. Throughout the experiment, we recorded the eye movements of 12 EOT crane operators of a steel industry of authors’ country. Post experiment, the National Aeronautics and Space Administration task load index (NASA-TLX) was adopted as a subjective workload measure and run time of task completion was recorded. Eye-movement metrics, subjective workload measure, run time were tested with multivariate analysis of variance (MANOVA), and three way repeated measure analysis of variance (ANOVA).ResultsAt the level of α = 0.05, the experimental factors significantly influence the means of eye movement metrics, subjective ratings and run time. There was also significant influence among their interactions. A positive correlation was also found for eye movements metrics with NASA-TLX and run time.ConclusionsEye movement metrics help in understanding the mental workload of participants unobtrusively and continuously. Analysis of subjective workload measure and run time along with eye-gaze analysis provide a deeper understanding on the pattern of mental workload.     

Measuring and defining the experience of immersion in games

Despite the word's common usage by gamers and reviewers alike, it is still not clear what immersion means. This paper explores immersion further by investigating whether immersion can be defined quantitatively, describing three experiments in total. The first experiment investigated participants’ abilities to switch from an immersive to a non-immersive task. The second experiment investigated whether there were changes in participants’ eye movements during an immersive task. The third experiment investigated the effect of an externally imposed pace of interaction on immersion and affective measures (state anxiety, positive affect, negative affect). Overall the findings suggest that immersion can be measured subjectively (through questionnaires) as well as objectively (task completion time, eye movements). Furthermore, immersion is not only viewed as a positive experience: negative emotions and uneasiness (i.e. anxiety) also run high.     

Visual Comparison of Eye Movement Patterns

In eye tracking research, finding eye movement patterns and similar strategies between participants’ eye movements is important to understand task solving strategies and obstacles. In this application paper, we present a graph comparison method using radial graphs that show Areas of Interest (AOIs) and their transitions. An analyst investigates a single graph based on dwell times, directed transitions, and temporal AOI sequences. Two graphs can be compared directly and temporal changes may be analyzed. A list and matrix approach facilitate the analyst to contrast more than two graphs guided by visually encoded graph similarities. We evaluated our approach in case studies with three eye tracking and visualization experts. They identified temporal transition patterns of eye movements across participants, groups of participants, and outliers.     

一种脑肢融合的神经康复训练在线评价与调整方法

在神经康复训练中, 保持患者积极主动参与、提供适配其运动能力的训练难度, 对于取得良好的康复效果至关重要. 针对患者在长期康复训练过程中容易懈怠甚至出现惰性效应、运动能力有波动等挑战, 系统提出了一种脑肢融合的神经康复训练在线评价与调整方法. 首先, 从脑、肢体以及训练任务三个层面, 基于脑电(Electroencephalography, EEG)信号、肢体运动数据和任务评分, 建立了对患者神经参与程度、运动控制能力和任务完成情况的量化评价方法. 进而, 在任务操作难度、辅助或干扰力场以及视觉辅助等方面, 设计了康复训练任务内和任务间的在线调整方法. 通过一个针对手功能康复的灵巧操作任务, 实现了基于所提出的脑肢融合在线评价与调整方法的闭环神经康复训练. 开展实验, 招募16名受试者参加, 对比分析开环训练和闭环训练两种情况下的实验结果, 验证了所提出方法的可行性和有效性. 该工作可推广应用到脑功能障碍患者的运动康复训练, 进一步提高康复效果.     

Prediction of human eye movements in facial discrimination tasks

Under natural viewing conditions, human observers selectively allocate their attention to subsets of the visual input. Since overt allocation of attention appears as eye movements, the mechanism of selective attention can be uncovered through computational studies of eyemovement predictions. Since top-down attentional control in a task is expected to modulate eye movements significantly, the models that take a bottom-up approach based on low-level local properties are not expected to suffice for prediction. In this study, we introduce two representative models, apply them to a facial discrimination task with morphed face images, and evaluate their performance by comparing them with the human eye-movement data. The result shows that they are not good at predicting eye movements in this task.     

Using eye-movement modelling examples to improve critical reading of multiple webpages on a conflicting topic

Interventions to promote students' source evaluations have used various methods designed for the classroom context. In the present study, we tested an approach that is easily adaptable to online courses, based on eye movement modelling examples (EMME), that is, short videos displaying an expert student's eye gaze while s/he reads multiple pages on the Internet to learn about a conflicting topic. Using an eye-tracking methodology in a pre–post design, we analysed how an intervention using EMME changed students' attention to source information, and how this processing affected their learning. EMME increased participants' attention to the search engine results page, author information and decreased attention of texts from untrustworthy pages. In addition, EMME increased the number of participants who cited at least one document source at post-test. We discuss the potential benefits and limitations of EMME in teaching complex literacy strategies, and the importance of measuring processing data in educational research studies.     

Social Feedback: Quantitative Division of Labor in Social Interactions

Abstract

This experiment investigated the problem of quantitative division of labor in group performance in a new way by measuring variations in the extent to which members of a two-person group could control the range of error in feedback control of social tracking. Social tracking is defined as cross-linking of the motor and sensory systems of two individuals in a common feedback loop so that the movements of one constitute a stimulus source of sensory, feedback to the second and vice versa. In this study, a parallel-linked mode of social manual-visual tracking with matched systems feedback was measured under two conditions of division of control of the total range of error in tracking an environmental stimulus. A real-time computer system was used to measure the dynamic systems error and synchronization of movements in the social tracking, and to control the measured systems error for immediate feedback control. The computer system also was used to partition the division of feedback control between the members of two-person groups, so that in one condition both individuals of the group had 100 percent control over the error range of the feedback display, while in another condition each individual had only 50 percent control over the error range. Results on 12 two-person groups showed that the division of labor in feedback control produced marked differences in performance level, with the 100 percent control showing the highest accuracy by some 30 percent. Significant learning occurred under both conditions. Variations in tracking speed caused marked differences in systems error under both conditions. The degree of synchronization of movements between individuals within particular team pairs varied for the two conditions of feedback control and was modified with practice. The methods and results demonstrate that social performance and learning are determined by dynamic systems feedback factors involved in synchronizing movements between persons for guidance and for control of environmental stimuli.