Multi-modal approach to evaluate adaptive visual stimuli of remote operation system using gaze behavior

A multi-modal approach is proposed to evaluate the usability of Adaptive Visual Stimuli for User Interface (AVS4UI) of remote operation systems. This study focuses on the evaluation of AVS4UI for forklift work because the operation complexity includes driving and cargo handling, which typically requires multiple salient attention. Presenting this amount of information simultaneously on a User Interface (UI) tends to cause confusions to operators and reduces operation efficiency. AVS4UI can therefore be one of the promising solutions where the optimal visual stimuli are autonomously presented for different work conditions. However, evaluation of AVS4UI is challenging because operators may be disoriented by adaptive information and worked without safety considerations. Therefore, novel gaze metrics are proposed to evaluate responses of forklift operators to AVS4UI so that undesired behavior can be evaluated. The proposed metrics implicitly represent gaze pattern in terms of transition and distribution between UI elements, operation safety, and familiarity with the adaptive system. The ideal AVS4UI is expected to minimize the proposed gaze metrics and outperform the non-adaptive UI. More importantly, the results of these metrics are consistent with those of perceived workload defined by NASA-Task Load Index. We also propose a correlation model using stepwise linear regression that provides reasonable estimation of perceived workload. Such novel metrics and correlation model enable objective and online evaluation to minimize biases of subjective response. Therefore, online work support system can be developed to support workers.     

Can eye movements be effectively measured to assess product design?: Gender differences should be considered

This study investigates the relationship between eye movements and user emotional response to product features using eighteen product images as stimuli. The stimuli are composed of unpleasant, neutral and pleasant images. Regarding the emotional design, gender difference could be a major factor that should be taken into consideration. A task-free eye-tracking experiment was conducted in which forty participants (20 males and 20 females, mean age = 35.6, SD = 6.38, range 21–48 years) looked at randomly displayed product images. The eye movements of participants were measured while viewing product images. After viewing stimuli, participants rated immediately their emotional response to the product images on a seven-point scale. The results revealed that stimuli category and gender difference, caused different variations in fixation count, fixation duration, pupil diameter, and hit ratio. For stimuli category, participants have larger fixation count, fixation duration, pupil diameter, and hit ratio in response to pleasant and neutral product images than unpleasant product images. For gender difference, the results indicated that females provided larger scores in fixation count and fixation duration, but a smaller score in hit ratio in all stimuli than that provided by males. However, the measurement of pupil diameter yielded different results. The results showed that females provided larger pupil diameter in pleasant and neutral product images but a smaller pupil diameter in unpleasant product images than that provided by males, which is consistent with the result of subjective ratings.Relevance to industryThese findings could enable industrial counselors, product designers, and academics to categorize emotional response that can be subsequently incorporated into final product design. An understanding of gender-based processing differences of product features is important to designers since it enables them to communicate with these different market segments and to produce effective products for each segment.     

Eye tracking as an MT evaluation technique

Eye tracking has been used successfully as a technique for measuring cognitive load in reading, psycholinguistics, writing, language acquisition etc. for some time now. Its application as a technique for measuring the reading ease of MT output has not yet, to our knowledge, been tested. We report here on a preliminary study testing the use and validity of an eye tracking methodology as a means of semi-automatically evaluating machine translation output. 50 French machine translated sentences, 25 rated as excellent and 25 rated as poor in an earlier human evaluation, were selected. Ten native speakers of French were instructed to read the MT sentences for comprehensibility. Their eye gaze data were recorded non-invasively using a Tobii 1750 eye tracker. The average gaze time and fixation count were found to be higher for the “bad” sentences, while average fixation duration and pupil dilations were not found to be substantially different for output rated as good and output rated as bad. Comparisons between HTER scores and eye gaze data were also found to correlate well with gaze time and fixation count, but not with pupil dilation and fixation duration. We conclude that the eye tracking data, in particular gaze time and fixation count, correlate reasonably well with human evaluation of MT output but fixation duration and pupil dilation may be less reliable indicators of reading difficulty for MT output. We also conclude that eye tracking has promise as a semi-automatic MT evaluation technique, which does not require bi-lingual knowledge, and which can potentially tap into the end users’ experience of machine translation output.     

基于瞳孔-角膜反射技术的视线跟踪方法

视线跟踪是基于多通道的人机交互技术的重要研究内容．而基于瞳孔-角膜反射技术的视线方向是目前应用最广泛的视线跟踪技术之一。瞳孔-角膜反射技术的主要目的是提取人眼图像中瞳孔-角膜反射向量作为视线方向计算模型所需的视觉信息,通过搭建红外光源设备提取瞳孔-角膜反射向量构建基于瞳孔-角膜反射技术的视线跟踪系统,为面向人机交互的视线跟踪研究提供可行的低成本解决方案。     

眼动跟踪研究进展与展望

眼动跟踪是指自动检测瞳孔中心位置或者识别三维视线方向及注视点的过程, 被广泛应用于人机交互、智能驾驶、人因工程等. 由于不同场景下的光照变化、个体眼球生理构造差异、遮挡、头部姿态多样等原因, 眼动跟踪的研究目前仍然是一个具有挑战性的热点问题. 针对眼动跟踪领域,首先概述眼动跟踪研究内容, 然后分别论述近年来瞳孔中心检测及视线估计领域的国内外研究进展, 综述目前眼动跟踪主要数据集、评价指标及研究成果, 接着介绍眼动跟踪在人机交互、智能驾驶等领域的应用, 最后对眼动跟踪领域的未来发展趋势进行展望.     

视线追踪系统眼部特征检测及视线方向计算方法研究

基于单视觉主动红外光源系统,提出了一种视线检测方法．在眼部特征检测阶段,采用投影法定位人脸;根据人脸对称性和五官分布的先验知识,确定瞳孔潜在区域;最后进行人眼特征的精确分割．在视线方向建模阶段,首先在头部静止的情况下采用非线性多项式建立从平面视线参数到视线落点的映射模型;然后采用广义回归神经网络对不同头部位置造成的视线偏差进行补偿,使非线性映射函数扩展到任何头部位置．实验结果及在交互式图形界面系统中的应用验证了该方法的有效性．     

The impact of engineering information format on task performance: Gaze scanning pattern analysis

The emergence of new visualization technologies such as Virtual Reality (VR) and Augmented Reality (AR) had been widely implemented in the Architecture, Engineering, and Construction (AEC) industry. Although cumulative evidence pointed out a positive impact of these visualization technologies on construction task performance, there is still an obvious disagreement on the benefits or implications of these new visualization technologies, due to the lack of understanding of the mechanisms in which the visualization affects cognitive processes related to information processing. To obtain more evidence, this paper presents a human-subject experiment (n = 90) to investigate the impact of information format on the performance of an industrial pipeline maintenance task. The investigation centers around how different engineering information formats affect the attention patterns as a potential explanation for the changes in performance. A between-group experiment design was used where the participants were randomly assigned to one of the three groups (2D group, 3D group, and VR group) depending on what type of information was given to review the pipe operation instruction. After the review session, the participants were asked to perform the operation task in the virtual environment based on their memory. The results showed that the 3D and VR groups outperformed the 2D group in task performance. The analysis of eye-tracking data further indicated that the information format significantly changed the gaze scanning pattern when participants were reviewing the operational instructions. We also found that the task performance was correlated with eye-tracking features including gaze movement and pupil dilation. Our findings provided more evidence about the mechanisms in which new visualization technologies affect the attention patterns, helped resolve the current disagreement within the literature. In addition, a prediction model was proposed to use eye-tracking features to predict construction task performance.     

视线追踪系统中眼睛跟踪方法研究

为解决视线追踪系统中红外图像瞳孔跟踪鲁棒性差的问题, 提出一种基于伪彩色图的粒子滤波瞳孔跟踪算法. 利用亮暗瞳现象, 提出三通道伪彩色图(Triple-channel pseudo-color map, TCPCM)的概念, 并将其引入瞳孔跟踪过程. TCPCM充分利用了各通道信息, 瞳孔区域的色彩明显与人脸其他部位不同, 提高了跟踪的稳定性与精确性. 采用了直方图相似性度量与几何相似性度量相结合的二次更新的瞳孔感知模型, 提高了粒子权重的可信性. 针对实时性要求, 引入快速特征提取步骤, 减少特征提取的时间, 提高特征提取的可靠性. 实验结果表明, 该算法在瞳孔目标检测效果、跟踪稳定性和运行时间方面都有良好的性能.     

Distinguishing and quantifying the visual aesthetics of a product: An integrated approach of eye-tracking and EEG

Although some methods of measuring visual aesthetics have been established, such as subjective reporting, feature calculating, and physiological assessing, designers still lack an integrated and quantified method in measuring the visual aesthetics of their products. This study aims to integrate eye-tracking metrics and EEG measurements to distinguish and quantify the visual aesthetics of a product. Thirty-two 3D prototypes of LED desk lamp with multiple views were designed to simulate an aesthetic appreciation flow. Eye-tracking and EEG signals were simultaneously recorded when participants were freely browsing each lamp. The evaluation of subjective visual aesthetics was conducted after each browsing. The results demonstrated that fixation time ratio and dwell time ratio significantly differed among the three clusters of visual aesthetic lamps. Meanwhile, average fixation duration only significantly differed between low and high aesthetic lamps and pupil size had no significant variation. Moreover, low aesthetic lamps evoked significantly weakened relative alpha power and enhanced relative gamma power. Thus, the eye-tracking metrics and the EEG measurements can distinguish the visual aesthetics of lamps. Regarding the results of quantification, the integrated multimodal physiological signals achieved an improved and reasonable accuracy. It seems beneficial to integrate multimodal physiological signals involved in different flows of visual aesthetic appreciation in quantifying the visual aesthetics of a product.Relevance to industryAs a premise of attracting consumers’ attention, visual aesthetics has been identified as a crucial role in product design and marketing. Thus, thorough research on the variations of multimodal physiological signals involved in information retrieval and processing in appreciation flow can provide a distinction between product visual aesthetics. The quantification method can be utilized by designers in measuring the visual aesthetics of their products.     

Eye activity correlates of workload during a visuospatial memory task

Changes in six measures of eye activity were assessed as a function of task workload in a target identification memory task. Eleven participants completed four 2-hr blocks of a mock anti-air-warfare task, in which they were required to examine and remember target classifications (friend/enemy) for subsequent prosecution (fire upon/allow to pass), while targets moved steadily toward two centrally located ship icons. Target density served as the task workload variable; between one and nine targets were simultaneously present on the display. For each participant, moving estimates of blink frequency and duration, fixation frequency and dwell time, saccadic extent, and mean pupil diameter, integrated over periods of 10 to 20 s, demonstrated systematic changes as a function of target density. Nonlinear regression analyses found blink frequency, fixation frequency, and pupil diameter to be the most predictive variables relating eye activity to target density. Participant-specific artificial neural network models, developed through training on two or three sessions and subsequently tested on a different session from the same participant, correlated well with actual target density levels (mean R = 0.66). Results indicate that moving mean estimation and artificial neural network techniques enable information from multiple eye measures to be combined to produce reliable near-real-time indicators of workload in some visuospatial tasks. Potential applications include the monitoring of visual activity of system opetators for indications of visual workload and scanning efficiency.     

阅读眼动追踪语料库的构建与应用研究综述

阅读文字时眼球的运动反映了人类的认知过程。阅读眼动数据是认知心理学、应用语言学、计算机科学等领域中重要的基础数据,而我国在阅读眼动研究的基础数据方面较欠缺。针对这一现状,首先介绍了阅读眼动追踪语料库产生的背景以及国内外的相关文献;然后从影响阅读眼动的低水平视觉因素和高水平视觉因素角度介绍了阅读眼动追踪语料库的内容及所使用的各项眼动指标,如单一注视时间、首次注视时间、凝视时间、总注视时间、回视出次数、回视入次数等,并分析了使用语料库研究法进行阅读眼动研究相比传统阅读眼动研究具有的3个优势;最后从语料库眼动指标变量、语料规模、语料内容、语料语种、被试规模、被试特征、采集设备等方面介绍了国外已经建成的较有影响力的若干阅读眼动追踪语料库,以供阅读眼动研究者参考。在眼动追踪语料库应用研究方面,对认知心理学、应用语言学和计算机科学等相关领域已开展的主要研究进行述评,重点介绍了在计算机科学的眼动可计算模型、自然语言处理、模式识别3个领域中基于阅读眼动追踪语料库开展的典型研究。在中文阅读眼动追踪语料库的构建与应用研究方面,介绍了我国相关研究的开展现状,分析了我国在眼动基础数据方面欠缺的原因,并从国家、科研机构、科研工作者3个层面提出了解决此问题的对策和建议。     

The influence of training level on manual flight in connection to performance,scan pattern,and task load

This work focuses on the analysis of pilots’ performance during manual flight operations in different stages of training and their influence on gaze strategy. The secure and safe operation of air traffic is highly dependent on the individual performances of the pilots. Before becoming a pilot, he/she has to acquire a broad set of skills by training to pass all the necessary qualification and licensing standards. A basic skill for every pilot is manual control operations, which is a closed-loop control process with several cross-coupled variables. Even with increased automation in the cockpit, the manual control operations are essential for every pilot as a last resort in the event of automation failure. A key element in the analysis of manual flight operations is the development over time in relation to performance and visual perception. An experiment with 28 participants (including 11 certified pilots) was conducted in a Boeing 737 simulator. For defined flight phases, the dynamic time warping method was applied to evaluate the performance for selected criteria, and eye-tracking methodology was utilized to analyze the gaze-pattern development. The manipulation of workload and individual experience influences the performance and the gaze pattern at the same time. Findings suggest that the increase of workload has an increased influence on pilots depending on the flight phase. Gaze patterns from experienced pilots provide insights into the training requirements of both novices and experts. The connection between workload, performance and gaze pattern is complex and needs to be analyzed under as many differing conditions. The results imply the necessity to evaluate manual flight operations with respect to more flight phases and a detailed selection of performance indications.

     

Assessing mental workload in virtual reality based EOT crane operations: A multi-measure approach

BackgroundEye-movement metrics and subjective workload measures are extensively used to determine mental workload of participants. The aim of this study was to assess Electric overhead travelling (EOT) crane operators’ mental workload variability based on eye movement metrics such as fixation frequency, fixation duration, saccade duration, saccade amplitude, and fixation/saccade ratio during EOT crane operations in virtual reality (VR) based EOT crane simulator.MethodsA 2^k (k = 3) factorial experiment with factors namely, hazardous scenario, activity level, and trial was designed and conducted to demonstrate the proposed assessment approach. Throughout the experiment, we recorded the eye movements of 12 EOT crane operators of a steel industry of authors’ country. Post experiment, the National Aeronautics and Space Administration task load index (NASA-TLX) was adopted as a subjective workload measure and run time of task completion was recorded. Eye-movement metrics, subjective workload measure, run time were tested with multivariate analysis of variance (MANOVA), and three way repeated measure analysis of variance (ANOVA).ResultsAt the level of α = 0.05, the experimental factors significantly influence the means of eye movement metrics, subjective ratings and run time. There was also significant influence among their interactions. A positive correlation was also found for eye movements metrics with NASA-TLX and run time.ConclusionsEye movement metrics help in understanding the mental workload of participants unobtrusively and continuously. Analysis of subjective workload measure and run time along with eye-gaze analysis provide a deeper understanding on the pattern of mental workload.     

Using eye movement parameters for evaluating human–machine interface frameworks under normal control operation and fault detection situations

A human–machine interface framework provides general guidelines for what information should be put on an interface display screen. The framework is thus a first step towards the design of an effective and efficient interface. This paper reports on an experimental study of two proposed frameworks: the ecological interface design framework and the function–behaviour–state framework. In order to provide an unbiased comparative evaluation for both interfaces, the same application problem is used. The interfaces, based on each of the two frameworks, are implemented with as similar look-and-feel forms as possible in the presentation of information contents. Only the normal control operation and fault detection situations are considered at this stage of the study. In addition, in this study three categories of measures are used, namely: the performance measure; the physiological measure (the eye movement measure: the eye fixation and the pupil diameter change, in particular); and the subjective (or the user-rated) measure. The major results obtained from the study includes the following: (1) the information called the abstract function in the ecological interface design framework may not positively correlate to the performance improvement yet may increase the mental workload, (2) the function–behaviour–state framework seems more agreeable with the operator's mental model, and (3) operators may perform equally well with a function–behaviour–state interface but with a reduced mental workload. It is also found that the eye fixation measure is highly consistent with the performance measure and the subjective measure. The pupil diameter measure is found not to be significantly sensitive to the mental workload information; however, it appears sensitive to the mental workload information among individual participants and shows a consistent result with the other measures used.     

What's skill got to do with it? Vehicle automation and driver mental workload

Previous research has found that vehicle automation systems can reduce driver mental workload, with implications for attentional resources that can be detrimental to performance. The present paper considers how the development of automaticity within the driving task may influence performance in underload situations. Driver skill and vehicle automation were manipulated in a driving simulator, with four levels of each variable. Mental workload was assessed using a secondary task measure and eye movements were recorded to infer attentional capacity. The effects of automation on driver mental workload were quite robust across skill levels, but the most intriguing findings were from the eye movement data. It was found that, with little exception, attentional capacity and mental workload were directly related at all levels of driver skill, consistent with earlier studies. The results are discussed with reference to applied theories of cognition and the design of automation.     

基于眼部自商图—梯度图共生矩阵的疲劳驾驶检测

目的 疲劳驾驶是引发车辆交通事故的主要原因之一,针对现有方法在驾驶员面部遮挡情况下对眼睛状态识别效果不佳的问题,提出了一种基于自商图—梯度图共生矩阵的驾驶员眼部疲劳检测方法。方法 利用以残差网络（residual network,ResNet）为前置网络的SSD（single shot multibox detector）人脸检测器来获取视频中的有效人脸区域,并通过人脸关键点检测算法分割出眼睛局部区域图像;建立驾驶员眼部的自商图与梯度图共生矩阵模型,分析共生矩阵的数字统计特征,选取效果较好的特征用以判定人眼的开闭状态;结合眼睛闭合时间百分比（percentage of eyelid closure,PERCLOS）与最长闭眼持续时间（maximum closing duration,MCD）两个疲劳指标来判别驾驶员的疲劳状态。结果 在六自由度汽车性能虚拟仿真实验平台上模拟汽车驾驶,采集并分析驾驶员面部视频,本文方法能够有效识别驾驶员面部遮挡时眼睛的开闭状态,准确率高达99.12%,面部未遮挡时的识别精度为98.73%,算法处理视频的速度约为32帧/s。对比方法1采用方向梯度直方图特征与支持向量机分类器相结合的人脸检测算法,并以眼睛纵横比判定开闭眼状态,在面部遮挡时识别较弱;以卷积神经网络（convolutional neural network,CNN）判别眼睛状态的对比方法2虽然在面部遮挡情况下的准确率高达98.02%,但眨眼检测准确率效果不佳。结论 基于自商图—梯度图共生矩阵的疲劳检测方法能够有效识别面部遮挡时眼睛的开闭情况和驾驶员的疲劳状态,具有较快的检测速度与较高的准确率。     

Effects of apparent image velocity and complexity on the dynamic visual field using a high-speed train driving simulator

Train driving is a highly visual task. The visual capabilities of the train driver affects driving safety and driving performance. Understanding the effects of train speed and background image complexity on the visual behavior of the high-speed train driver is essential for optimizing performance and safety. This study investigated the role of the apparent image velocity and complexity on the dynamic visual field of drivers. Participants in a repeated-measures experiment drove a train at nine different speeds in a state-of-the-art high-speed train simulator. Eye movement analysis indicated that the effect of image velocity on the dynamic visual field of high-speed train driver was significant while image complexity had no effect on it. The fixation range was increasingly concentrated on the middle of the track as the speed increased, meanwhile there was a logarithmic decline in fixation range for areas surrounding the track. The extent of the visual search field decreased gradually, both vertically and horizontally, as the speed of train increased, and the rate of decrease was more rapid in the vertical direction. A model is proposed that predicts the extent of this tunnel vision phenomenon as a function of the train speed.Relevance to industryThis finding can be used as a basis for the design of high-speed railway system and as a foundation for improving the operational procedures of high-speed train driver for safety.     

Occlusion as a measure for visual workload: an overview of TNO occlusion research in car driving

A process-oriented approach by systematically studying driver performance, distraction, and workload is the way to go for assessing safety effects of new telematics applications in vehicles. These systems may strive for drivers' attention and possibly lead to distraction from the primary task. Visual occlusion techniques appear to be an effective means of studying drivers' visual information processing performance. Studies to illustrate this include identifying the minimum visual information drivers need for driving (asking for visual information) and the evaluation of driver support systems such as heading control and adaptive cruise control with respect to visual workload. In other applications the occlusion technique is used to exclude 'visual array' information from the optic flow field. The temporary exclusion of part of the visual field of view was applied in evaluating the effects of different types of driver's side rearview mirrors.     

On-line driver workload estimation. Effects of road situation and age on secondary task measures

In order to develop a driver-car interface that adapts the presentation of messages generated by in-vehicle information systems to driver workload, two experiments investigated potential determinants of driver visual and mental workload as indicated by performance on two secondary tasks. Experiment 1 suggested that road situation is a major determinant of visual and mental workload of the driver and that the processing resources of older drivers are somewhat more limited than those of younger and middle-aged drivers. Familiarity with the area of driving (when guided) and time of day (associated with traffic density) showed no secondary task effects. Experiment 2 showed that the categorization of road situations, proposed in Experiment 1, could underlie adaptation of visually loading messages to the workload incurred by driving. This was not found with respect to mentally loading messages.