Automatic gaze-based user-independent detection of mind wandering during computerized reading

Mind wandering is a ubiquitous phenomenon where attention involuntarily shifts from task-related thoughts to internal task-unrelated thoughts. Mind wandering can have negative effects on performance; hence, intelligent interfaces that detect mind wandering can improve performance by intervening and restoring attention to the current task. We investigated the use of eye gaze and contextual cues to automatically detect mind wandering during reading with a computer interface. Participants were pseudorandomly probed to report mind wandering while an eye tracker recorded their gaze during the reading task. Supervised machine learning techniques detected positive responses to mind wandering probes from eye gaze and context features in a user-independent fashion. Mind wandering was detected with an accuracy of 72 % (expected accuracy by chance was 60 %) when probed at the end of a page and an accuracy of 67 % (chance was 59 %) when probed in the midst of reading a page. Global gaze features (gaze patterns independent of content, such as fixation durations) were more effective than content-specific local gaze features. An analysis of the features revealed diagnostic patterns of eye gaze behavior during mind wandering: (1) certain types of fixations were longer; (2) reading times were longer than expected; (3) more words were skipped; and (4) there was a larger variability in pupil diameter. Finally, the automatically detected mind wandering rate correlated negatively with measures of learning and transfer even after controlling for prior knowledge, thereby providing evidence of predictive validity. Possible improvements to the detector and applications that utilize the detector are discussed.     

The effects of graphical encodings on reading performance of the data chart displayed on the periphery of attention

The accuracy and efficiency of estimating the true data value in visualizations is critical to the user interface design for large displays. As a variable, the attention engagement has not been the focus of existing related studies. In the current study, we intended to measure the performance users estimate proportions from a visualization under the low-attention condition. On this purpose, the dual-task paradigm was employed to simulate the scenario that data charts were read simultaneously with searching and memorizing information on the display. We evaluated eight data charts with three graphical encodings (shape, orientation and area) for the proportion value. The experimental results show that the vertical stacked chart is relatively more suitable for quantitative reading under the situation of temporary distraction. This is reflected by the high accuracy of estimation of this chart with relative less time consumption. When data charts are difficult to read, the users tend to pay less attention to information processing and make more inaccurate inferences of this primary task. Graphical encoding and the task time have a significant effect on task performances overall, while the reading accuracy seems not to vary with the difference of the primary task. The present study can be a supplement to the understanding of graphical perception and provide implications for the design of data visualization in display-human interfaces.     

基于最大熵模型的中文阅读理解问题回答技术研究

该文基于山西大学自主开发的中文阅读理解语料库CRCC v1.1版,根据问句和候选答案句的对应关系,构建了词层面以及句法层面共计35个特征,基于最大熵模型对中文阅读理解问题回答进行了建模,在35个特征全部加入最大熵模型的情况下,测试集上得到了75.46%的HumSent准确率。考虑到特征取值之间的相关性对权重估计的影响,笔者先对35个特征观测值矩阵进行主成分降维,选择适当的主成分个数重构特征,然后再使用最大熵模型进行建模,在测试集上的HumSent准确率达到80.18%. 实验结果表明,在阅读理解问答系统中,采用特征的主成分降维方法,能有效融合全部特征信息,回避了最大熵模型中特征筛选的过程,并且提高了阅读理解系统的准确率。     

Development of a system usability assessment procedure using oculo-motors for input operation

This paper investigates the relationship between oculo-motors, such as eye-movement and pupillary change, and the conventional subjective “usability” index, using time-domain and frequency-domain approaches, with the objective to determine the possibility of evaluating interaction through oculo-motors. An evaluation experiment was conducted by operating a target on a computer display using mouse, keyboard and key pad as input devices. The results show that there is a significant correlation between pupil size and SU-score, which is an established subjective evaluation index for system usability. Cross spectrum densities (CSD) between horizontal and vertical eye-movements and coherence as standardized CSD also significantly correlate with the results of the SU-scores and error rates. To determine the frequency range of CSD and coherence for usability assessment, frequency components used as factors were extracted using factor analysis. According to the correlation coefficients between these and the performance of factor scores for predicting the conventional metrics, factor scores of CSD are better indices for assessing usability than factor scores of coherence. These two results suggest that pupil size and index of eye-movement as oculo-motor indices based on time-domain and frequency-domain approaches can provide information about a system’s overall usability regarding the input operation task.     

采用上下文专注机制的特定目标观点抽取

针对现有的目标和观点抽取模型未能充分考虑两者的联系的问题,提出一种基于上下文专注机制的特定目标观点抽取模型。将抽取出的目标特征向量与每个位置的上下文词向量拼接构成最终的句子表示,加强目标与句子之间的交互,实现目标融合;采用上下文专注机制把注意力更多地放在目标词的周围,削弱远距离词的语义特征。提出的模型采用双向长短时记忆（bi-directional long short-term memory,BiLSTM）网络将句子编码,并提取特征。与现有模型相比,所提模型的精确率、召回率和F1值都有一定程度的提升,证明了所提算法的有效性。同时,预训练的BERT模型也被应用到当前任务中,使模型效果获得了进一步的提升。     

多跳式文本阅读理解方法综述

多跳阅读理解成为近年来自然语言理解领域的研究热点,与简单阅读理解相比,它更加复杂,需要面对如下挑战：(1)结合多处内容线索,如多文档阅读等;(2)具有可解释性,如给出推理路径等。为应对这些挑战,出现了各类不同的工作。因此该文综述了多跳式文本阅读理解这一复杂阅读理解任务,首先给出了多跳文本阅读理解任务的定义;由于推理是多跳阅读理解模型的基础能力,根据推理方式的不同,多跳阅读理解模型可以分为三类：基于结构化推理的多跳阅读理解模型、基于线索抽取的多跳阅读理解模型、基于问题拆分的多跳阅读理解模型,该文接下来比较分析了各类模型在常见多跳阅读理解模型任务数据集上的实验结果,发现这三类模型之间各有优劣。最后探讨了未来的研究方向。     

Eye movement as a mediator of the relationships among time pressure,feedback, and learning performance

The goal of this study is to examine the effects of time pressure and feedback on learning performance, as mediated by eye movement. Time pressure is one of main causes of human error in the workplace. Providing participants with feedback about their performance before task completion has been shown to reduce human error in diverse domains. Since both time pressure and feedback induce motivation, which is closely related to attention, we measured participants' eye movements to trace their attention and information acquisition coupled with a visual display. Time-to-deadline (long and short) and the presence of feedback were the independent factors used while measuring participants’ performance and eye movements as they learned new information about the subject of project management and answered multiple-choice questions via self-paced online learning systems. Using structural equation modeling, we found a mediating effect of eye movement on the relationships among time-to-deadline, feedback, and learning performance. Insufficient time-to-deadline accelerated the number of fixations on the screen, which resulted in longer task completion times and increased correct rates for participants learning about project management. The models in this study suggest the possibility of predicting performance from eye movement under time-to-deadline and feedback conditions. The structural equation model in the study can be applied to online and remote learning systems, in which time management is one of the main challenges for individual learners.     

Prediction of human eye movements in facial discrimination tasks

Under natural viewing conditions, human observers selectively allocate their attention to subsets of the visual input. Since overt allocation of attention appears as eye movements, the mechanism of selective attention can be uncovered through computational studies of eyemovement predictions. Since top-down attentional control in a task is expected to modulate eye movements significantly, the models that take a bottom-up approach based on low-level local properties are not expected to suffice for prediction. In this study, we introduce two representative models, apply them to a facial discrimination task with morphed face images, and evaluate their performance by comparing them with the human eye-movement data. The result shows that they are not good at predicting eye movements in this task.     

Qualitative changes in recall memory during day and night shifts

The effect of time of day on recall of words and drawings was studied in 29 French shiftworkers. Tests were carried out at 18:00, 22:00, 02:00 and 06:00. The task involved memorizing a list of 18 drawings for one half of the group, and 18 words (designating the corresponding figures) for the other half. At each time of day, the lists were presented both with and without an interpolated task. In the interpolated task condition, the first 9 items were followed by an interpolated imagery task and the last 9 by a spelling task, or vice versa. The results indicated an overall effect of time of day on the recall test, especially when memorizing drawings. At 18:00 in both the control and the interference condition, subjects recalled more words and drawings from the beginning of the list. Conversely, at 02:00, subjects recalled more words and drawings from the end of the list. The time spent in the interpolated comparison task varied with the time of day and was longer when memorizing drawings. The same time of day pattern of performance was found for spelling times. The implications upon issues concerning shiftwork and job demands are also discussed.     

融合BERT语义表示的高考语文阅读理解问答研究

高考语文阅读理解问答相对普通阅读理解问答难度更大,同时高考问答任务中的训练数据较少,目前的深度学习方法不能取得良好的答题效果。针对这些问题,该文提出融合BERT语义表示的高考阅读理解答案候选句抽取方法。首先,采用改进的MMR算法对段落进行筛选;其次,运用微调之后的BERT模型对句子进行语义表示;再次,通过SoftMax分类器对答案候选句进行抽取,最后利用PageRank排序算法对输出结果进行二次排序。该方法在北京近十年高考语文阅读理解问答题上的召回率和准确率分别达到了61.2%和50.1%,验证了该方法的有效性。     

Eye movements and visual discomfort when viewing stereoscopic 3D content

The visual brain fuses the left and right images projected onto the two eyes from a stereoscopic 3D (S3D) display, perceives parallax, and rebuilds a sense of depth. In this process, the eyes adjust vergence and accommodation to adapt to the depths and parallax of the points they gazed at. Conflicts between accommodation and vergence when viewing S3D content potentially lead to visual discomfort. A variety of approaches have been taken towards understanding the perceptual bases of discomfort felt when viewing S3D, including extreme disparities or disparity gradients, negative disparities, dichoptic presentations, and so on. However less effort has been applied towards understanding the role of eye movements as they relate to visual discomfort when viewing S3D. To study eye movements in the context of S3D viewing discomfort, a Shifted-S3D-Image-Database (SSID) is constructed using 11 original nature scene S3D images and their 6 shifted versions. We conducted eye-tracking experiments on humans viewing S3D images in SSID while simultaneously collecting their judgments of experienced visual discomfort. From the collected eye-tracking data, regions of interest (ROIs) were extracted by kernel density estimation using the fixation data, and an empirical formula fitted between the disparities of salient objects marked by the ROIs and the mean opinion scores (MOS). Finally, eye-tracking data was used to analyze the eye movement characteristics related to S3D image quality. Fifteen eye movement features were extracted, and a visual discomfort predication model learned using a support vector regressor (SVR). By analyzing the correlations between features and MOS, we conclude that angular disparity features have a strong correlation with human judgments of discomfort.     

Coupled Action Recognition and Pose Estimation from Multiple Views

Action recognition and pose estimation are two closely related topics in understanding human body movements; information from one task can be leveraged to assist the other, yet the two are often treated separately. We present here a framework for coupled action recognition and pose estimation by formulating pose estimation as an optimization over a set of action-specific manifolds. The framework allows for integration of a 2D appearance-based action recognition system as a prior for 3D pose estimation and for refinement of the action labels using relational pose features based on the extracted 3D poses. Our experiments show that our pose estimation system is able to estimate body poses with high degrees of freedom using very few particles and can achieve state-of-the-art results on the HumanEva-II benchmark. We also thoroughly investigate the impact of pose estimation and action recognition accuracy on each other on the challenging TUM kitchen dataset. We demonstrate not only the feasibility of using extracted 3D poses for action recognition, but also improved performance in comparison to action recognition using low-level appearance features.     

基于视频描述和阅读理解的视频问答研究

针对大多数视频问答(VideoQA)模型将视频和问题嵌入到同一空间进行答案推理所面临的多模态交互困难、视频语义特征保留能力差等问题,提出了一种视频描述机制来获得视频语义特征的文本表示,从而避免了多模态的交互.提出方法将视频特征通过描述机制得到相应的视频描述文本,并将描述文本特征与问题特征进行阅读理解式的交互与分析,最后推理出问题的答案.在MSVD-QA以及MSRVTT-QA数据集上的测试结果显示,提出问答模型的回答准确率较现有模型均有不同程度的提升,说明所提方法能更好地完成视频问答任务.     

Study on directional eye movements in non‐frontal face images for eye‐controlled interaction

Directional eye movements based eye‐controlled interaction focuses on interpreting the horizontal, vertical, and diagonal eye movements or their combinations as inputs to design user interfaces for people who suffer with severe mobility disabilities. In this paper, we take into consideration the inherent eye jitter and evaluate the accuracy of dynamic tracking of horizontal, vertical, diagonal, and rectangular eye movements prior to using them. We observe that the rectangular eye gesture composed of short horizontal and vertical eye movements has the best tracking accuracy in the presence of jitter. Finally, we present methods for identifying horizontal and vertical eye movements based on the trajectory of eye pupil centers from non‐frontal face images. We find that the methods are robust and effective within ±20°deflective azimuths of non‐frontal faces. This effectiveness is demonstrated by using the rectangular eye gesture as an interface to perform a painting task.     

基于多任务学习的生成式阅读理解

生成式阅读理解是机器阅读理解领域一项新颖且极具挑战性的研究。与主流的抽取式阅读理解相比,生成式阅读理解模型不再局限于从段落中抽取答案,而是能结合问题和段落生成自然和完整的表述作为答案。然而,现有的生成式阅读理解模型缺乏对答案在段落中的边界信息以及对问题类型信息的理解。为解决上述问题,该文提出一种基于多任务学习的生成式阅读理解模型。该模型在训练阶段将答案生成任务作为主任务,答案抽取和问题分类任务作为辅助任务进行多任务学习,同时学习和优化模型编码层参数;在测试阶段加载模型编码层进行解码生成答案。实验结果表明,答案抽取模型和问题分类模型能够有效提升生成式阅读理解模型的性能。     

基于BiLSTM模型的定义抽取方法

定义抽取是从非结构化文本中自动识别定义句的任务,定义抽取问题可建模为句子中术语及相应定义的序列标注问题,并利用标注结果完成抽取任务。针对传统的定义抽取方法在抽取定义特征过程中费时且容易造成错误传播的不足,提出一个基于双向长短时记忆(BiLSTM)的序列标注神经网络模型,对输入文本进行自动化定义抽取。通过将原始数据输入到BiLSTM神经网络中,完成输入句的特征表示,并采用基于LSTM的解码器进行解码得到标注结果。在Wikipedia英文数据集上的实验结果表明,该方法的精确率、召回率和F1值分别为94.21%、90.10%和92.11%,有效提升了基准模型效果。     

基于不同序列特征值预测氧化还原酶辅酶类型的研究

如何有效提取蛋白质序列特征值,一直是生物信息学研究的重要任务.本文研究8种序列特征值提取方法,并考察它们在不同分类器中的表现,以用于预测氧化还原酶辅酶依赖类型.其中,氨基酸组成法效果最差,平均预测精度仅及64.96%;而将两性伪氨基酸组成和新氨基酸组成分布两种方法合并后,以支持向量机作为分类器时,其识别效果最佳,可达92.93%.此外,不同特征值的提取方法与分类器之间似乎有着一定的匹配关系,只有找到其间的最佳匹配,才能获得最佳的识别效果.     

Human eye sclera detection and tracking using a modified time-adaptive self-organizing map

This paper investigates the use of time-adaptive self-organizing map (TASOM)-based active contour models (ACMs) for detecting the boundaries of the human eye sclera and tracking its movements in a sequence of images. The task begins with extracting the head boundary based on a skin-color model. Then the eye strip is located with an acceptable accuracy using a morphological method. Eye features such as the iris center or eye corners are detected through the iris edge information. TASOM-based ACM is used to extract the inner boundary of the eye. Finally, by tracking the changes in the neighborhood characteristics of the eye-boundary estimating neurons, the eyes are tracked effectively. The original TASOM algorithm is found to have some weaknesses in this application. These include formation of undesired twists in the neuron chain and holes in the boundary, lengthy chain of neurons, and low speed of the algorithm. These weaknesses are overcome by introducing a new method for finding the winning neuron, a new definition for unused neurons, and a new method of feature selection and application to the network. Experimental results show a very good performance for the proposed method in general and a better performance than that of the gradient vector field (GVF) snake-based method.     

Predicting separation errors of air traffic controllers through integrated sequence analysis of multimodal behaviour indicators

Predicting separation errors in the daily tasks of air traffic controllers (ATCOs) is essential for the timely implementation of mitigation strategies before performance declines and the prevention of loss of separation and aircraft collisions. However, three challenges impede accurate separation errors forecasting: 1) compounding relationships between many human factors and control processes require sufficient operation process data to capture how separation errors occur and propagate within controller-in-the-loop processes; 2) previous human factor measurement approaches are disruptive to controllers’ daily operations because they use invasive sensors, such as electroencephalography (EEG) and electrocardiography (ECG), 3) errors accumulated in using the tasks and human behaviors for estimating system dynamics challenge accurate separation error predictions with sufficient leading time for proactive control actions. This study proposed a separation error prediction framework with a long leading time (>50 s) to address the above challenges, including 1) a multi-factorial model that characterizes the inter-relationships between task complexity, behavioral activity, cognitive load, and operational performance; 2) a multimodal data analytics approach to non-intrusively extract the task features (i.e., traffic density) from high-fidelity simulation systems and visual behavioral features (i.e., head pose, eyelid movements, and facial expressions) from ATCOs’ facial videos; 3) an encoder-decoder Long Short-Term Memory (LSTM) network to predict long-time-ahead separation errors by integrating multimodal features for reducing accumulated errors. A user study with six experienced ATCOs tested the proposed framework using the Phoenix Terminal Radar Approach Control (TRACON) simulator. The authors evaluated the model performance through two types of metrics: 1) point-level metrics, including precision, recall, and F1-score, and 2) sequence-level metrics, including alignment accuracy and sequence similarity. The results showed that 1) the model using the task and visual behavioral features significantly improved the prediction performance compared to the model using one single feature (eyelid movements), with an improvement of up to 26.95% in alignment accuracy for 10s-ahead prediction; 2) the model that combined task and visual behavioral features had a higher or comparable performance to models with different hybrid features, achieving an alignment accuracy of 82.38% for 50s-ahead error prediction; and (3) the proposed method outperformed three baseline models – Convolutional Neural Network (CNN), Gated Recurrent Unit (GRU), and classic LSTM – by 8.21%, 3.47%, and 3.14% in alignment accuracy, respectively, for predicting 50s-ahead separation errors. These results suggest that the proposed model can effectively predict separation errors in air traffic control.     

基于Word2vec和句法规则的自动问答系统问句相似度研究

自动问答系统问句相似度计算的准确率直接影响系统返回答案的准确率,对此提出一种基于Word2vec和句法规则的问句相似度计算方法。构造Text-CNN问句分类模型将问句进行分类,再构造Word2vec词向量模型将问句中词与词的空间向量相似度转换成语义相似度,并加入句法规则的分析。随机从搜狗公开问答数据集中抽取200条数据进行测试,结果表明,该方法与TF-IDF方法相比,自动问答系统返回答案的准确率和召回率分别提高了0.259和0.154。