Fixational eye movements, natural image statistics, and fine spatial vision

Perception and motor control are often regarded as two separate branches of neuroscience. Like most species, however, humans are not passively exposed to the incoming flow of sensory data, but actively seek useful information. By shaping input signals in ways that simplify perceptual tasks, behavior might play an important role in establishing efficient sensory representations in the brain. Under natural viewing conditions, the main source of motion of the stimulus on the retina is not the scene but our own behavior. The retinal image is never still, even during visual fixation, when small eye movements combine with movements of the head and body to continually perturb the location of gaze. This article examines the impact of the fixational motion of the retinal image on the statistics of visual input and the neural encoding of visual information. Building upon recent theoretical and experimental results, it is argued that an unstable fixation constitutes an efficient strategy for acquiring information from natural scenes. According to this theory, the fluctuations of luminance caused by the incessant motion of the eye equalize the power present at different spatial frequencies in the spatiotemporal stimulus on the retina. This phenomenon yields compact neural representations, emphasizes fine spatial detail, and might enable a temporal multiplexing of visual information from the retina to the cortex. This theory posits motor contributions to early visual representations and suggests that perception and behavior are more intimately tied than commonly thought.     

Fixational instability and natural image statistics: implications for early visual representations

Under natural viewing conditions, small movements of the eye, head and body prevent the maintenance of a steady direction of gaze. It is known that stimuli tend to fade when they are stabilized on the retina for several seconds. However, it is unclear whether the physiological motion of the retinal image serves a visual purpose during the brief periods of natural visual fixation. This study examines the impact of fixational instability on the statistics of the visual input to the retina and on the structure of neural activity in the early visual system. We show that fixational instability introduces a component in the retinal input signals that, in the presence of natural images, lacks spatial correlations. This component strongly influences neural activity in a model of the LGN. It decorrelates cell responses even if the contrast sensitivity functions of simulated cells are not perfectly tuned to counter-balance the power-law spectrum of natural images. A decorrelation of neural activity at the early stages of the visual system has been proposed to be beneficial for discarding statistical redundancies in the input signals. The results of this study suggest that fixational instability might contribute to the establishment of efficient representations of natural stimuli.     

Local statistics of retinal optic flow for self-motion through natural sceneries

Image analysis in the visual system is well adapted to the statistics of natural scenes. Investigations of natural image statistics have so far mainly focused on static features. The present study is dedicated to the measurement and the analysis of the statistics of optic flow generated on the retina during locomotion through natural environments. Natural locomotion includes bouncing and swaying of the head and eye movement reflexes that stabilize gaze onto interesting objects in the scene while walking. We investigate the dependencies of the local statistics of optic flow on the depth structure of the natural environment and on the ego-motion parameters. To measure these dependencies we estimate the mutual information between correlated data sets. We analyze the results with respect to the variation of the dependencies over the visual field, since the visual motions in the optic flow vary depending on visual field position. We find that retinal flow direction and retinal speed show only minor statistical interdependencies. Retinal speed is statistically tightly connected to the depth structure of the scene. Retinal flow direction is statistically mostly driven by the relation between the direction of gaze and the direction of ego-motion. These dependencies differ at different visual field positions such that certain areas of the visual field provide more information about ego-motion and other areas provide more information about depth. The statistical properties of natural optic flow may be used to tune the performance of artificial vision systems based on human imitating behavior, and may be useful for analyzing properties of natural vision systems.     

目标颜色对视线交互的影响研究

随着眼动跟踪技术的日益成熟,面向终端用户的视线输入产品问世,视线交互(Gaze-based Interaction)的实用性越来越高。然而,由于眼睛并不是与生俱来的控制器官,用户界面中无论动态或静态的各种视觉反馈,在视线交互过程中都可能干扰用户的眼动,从而影响视线输入(视点坐标)。因此,通过两个视线点击(Eye Pointing)实验,从视点的空间分布特征和视线交互的人机工效两个方面,系统地评估了目标颜色因素对视线交互的影响。结果表明,目标颜色这类静态视觉反馈虽然不影响用户凝视目标时视点坐标的稳定性,但的确会对用户的眼动扫视过程造成显著影响,从而影响视线点击任务的人机工效。特别是在视线移动距离较长的情况下,这种影响更为明显。     

Gaze behaviour, believability, likability and the iCat

The iCat is a user-interface robot with the ability to express a range of emotions through its facial features. This article summarizes our research to see whether we can increase the believability and likability of the iCat for its human partners through the application of gaze behaviour. Gaze behaviour serves several functions during social interaction such as mediating conversation flow, communicating emotional information and avoiding distraction by restricting visual input. There are several types of eye and head movements that are necessary for realizing these functions. We designed and evaluated a gaze behaviour system for the iCat robot that implements realistic models of the major types of eye and head movements found in living beings: vergence, vestibulo ocular reflexive, smooth pursuit movements and gaze shifts. We discuss how these models are integrated into the software environment of the iCat and can be used to create complex interaction scenarios. We report about some user tests and draw conclusions for future evaluation scenarios.     

What Are You Looking at?

In this paper we present a novel mechanism to obtain enhanced gaze estimation for subjects looking at a scene or an image. The system makes use of prior knowledge about the scene (e.g. an image on a computer screen), to define a probability map of the scene the subject is gazing at, in order to find the most probable location. The proposed system helps in correcting the fixations which are erroneously estimated by the gaze estimation device by employing a saliency framework to adjust the resulting gaze point vector. The system is tested on three scenarios: using eye tracking data, enhancing a low accuracy webcam based eye tracker, and using a head pose tracker. The correlation between the subjects in the commercial eye tracking data is improved by an average of 13.91%. The correlation on the low accuracy eye gaze tracker is improved by 59.85%, and for the head pose tracker we obtain an improvement of 10.23%. These results show the potential of the system as a way to enhance and self-calibrate different visual gaze estimation systems.     

An integrated head pose and eye gaze tracking approach to non-intrusive visual attention measurement for wide FOV simulators

Eye gaze tracking is very useful for quantitatively measuring visual attention in virtual environments. However, most eye trackers have a limited tracking range, e.g., ±35° in the horizontal direction. This paper proposed a method to combine head pose tracking and eye gaze tracking together to achieve a large range of tracking in virtual driving simulation environments. Multiple parallel multilayer perceptrons were used to reconstruct the relationship between head images and head poses. Head images were represented with the coefficients extracted from Principal Component Analysis. Eye gaze tracking provides precise results on the front view, while head pose tracking is more suitable for tracking areas of interest than for tracking points of interest on the side view.     

Computational modeling of human performance in multiple monitor environments with ACT-R cognitive architecture

Much research on modeling human performance associated with visual perception is formulated by schematic models based on neural mechanisms or cognitive architectures. But, these two modeling paradigms are limited in the domains of multiple monitor environments. Although the schematic model based on neural mechanisms can represent human visual systems in multiple monitor environments by providing a detailed account of eye and head movements, these models cannot easily be applied in complex cognitive interactions. On the other hand, the cognitive architectures can model the interaction of multiple aspects of cognition, but these architectures have not focused on modeling the visual orienting behavior of eye and head movements. Thus, in this study, a specific cognitive architecture, which is ACT-R, is extended by an existing schematic model of human visual systems based on neural mechanisms in order to model human performance in multiple monitor environments more accurately. And, this study proposes a method of modeling human performance using the extended ACT-R. The proposed method is validated by an experiment, confirming that the proposed method is able to predict human performance more accurately in multiple monitor environments.Relevance to industryPredicting human performance with a computational model can be used as an alternative method to implementing iterative user testing for developing a system interface. The computational model in this study can predict human performance in multiple monitor environments, so that the model can be applied early on in the design phase, to evaluate the system interface in multiple monitor environments.     

Gaze stability for liveness detection

Spoofing attacks on biometric systems are one of the major impediments to their use for secure unattended applications. This paper explores features for face liveness detection based on tracking the gaze of the user. In the proposed approach, a visual stimulus is placed on the display screen, at apparently random locations, which the user is required to follow while their gaze is measured. This visual stimulus appears in such a way that it repeatedly directs the gaze of the user to specific positions on the screen. Features extracted from sets of collinear and colocated points are used to estimate the liveness of the user. Data are collected from genuine users tracking the stimulus with natural head/eye movements and impostors holding a photograph, looking through a 2D mask or replaying the video of a genuine user. The choice of stimulus and features are based on the assumption that natural head/eye coordination for directing gaze results in a greater accuracy and thus can be used to effectively differentiate between genuine and spoofing attempts. Tests are performed to assess the effectiveness of the system with these features in isolation as well as in combination with each other using score fusion techniques. The results from the experiments indicate the effectiveness of the proposed gaze-based features in detecting such presentation attacks.     

考场环境下考生视线估计方法

考生异常行为的监测容易使监考人员产生视觉疲劳。借鉴监考人员发现异常的过程,提出一种可用于考场异常行为分析的视线估计模型。为了减少图像中视线的信息损失,采用注视向量表示视线的大小和方向。该模型分为生成器、视线合成模块、鉴别器,先将考生头部图像输入生成器生成注视向量,再将头部位置和注视位置输入到合成模块得到真实注视向量。将头部图像与上述所得的两种向量输入鉴别器中,其生成对抗模式达到最优时,可得到生成真实值的生成器模型。实验结果表明,在多个考场环境中,该方法的性能优于所对比的几种方法。其中与Lian等人方法相比AUC（Area Under Curve）提高了2.6%,Ang（Angular error）和Dist（Euclidean distance）分别有效降低了20.3%和8.0%。     

A high speed gaze control system based on the Vestibulo-Ocular Reflex

Stabilizing the visual system is a crucial issue for any sighted mobile creature, whether it will be natural or artificial. The more immune the gaze of an animal or a robot is to various kinds of disturbances (e.g., those created by body or head movements when walking or flying), the less troublesome it will be for the visual system to carry out its many information processing tasks. The gaze control system that we describe in this paper takes a lesson from the Vestibulo-Ocular Reflex (VOR), which is known to contribute to stabilizing the human gaze and keeping the retinal image steady. The gaze control system owes its originality and its high performances to the combination of two sensory modalities, as follows:

• a visual sensor called Optical Sensor for the Control of Autonomous Robots (OSCAR) which delivers a retinal angular position signal. A new, miniature (10 g), piezo-based version of this visual sensor is presented here;

• an inertial sensor which delivers an angular head velocity signal.

We built a miniature (30 g), one degree of freedom oculomotor mechanism equipped with a micro-rate gyro and the new version of the OSCAR visual sensor. The gaze controller involves a feedback control system based on the retinal position error measurement and a feedforward control system based on the angular head velocity measurement. The feedforward control system triggers a high-speed “Vestibulo-ocular reflex” that efficiently and rapidly compensates for any rotational disturbances of the head. We show that a fast rotational step perturbation (3° in 40 ms) applied to the head is almost completely (90%) rejected within a very short time (70 ms). Sinusoidal head perturbations are also rapidly compensated for, thus keeping the gaze stabilized on its target (an edge) within a 10 times smaller angular range than the perturbing head rotations, which were applied here at frequencies of up to 6 Hz in an amplitude range of up to 6°. This high standard of performance in terms of head rotational disturbance rejection is comparable to that afforded by the human vestibulo-oculomotor system.     

Utilizing eye movements: Overcoming inaccuracy while tracking the focus of attention during reading

Even though eye movements during reading have been studied intensively for decades, applications that track the reading of longer passages of text in real time are rare. The problems encountered in developing such an application (a reading aid, iDict), and the solutions to the problems are described. Some of the issues are general and concern the broad family of Attention Aware Systems. Others are specific to the modality of interest: eye gaze. One of the most difficult problems when using eye tracking to identify the focus of visual attention is the inaccuracy of the eye trackers used to measure the point of gaze. The inaccuracy inevitably affects the design decisions of any application exploiting the point of gaze for localizing the point of visual attention. The problem is demonstrated with examples from our experiments. The principles of the drift correction algorithms that automatically correct the vertical inaccuracy are presented and the performance of the algorithms is evaluated.     

Salamander locomotion-induced head movement and retinal motion sensitivity in a correlation-based motion detector model

We report on a computational model of retinal motion sensitivity based on correlation-based motion detectors. We simulate object motion detection in the presence of retinal slip caused by the salamander's head movements during locomotion. Our study offers new insights into object motion sensitive ganglion cells in the salamander retina. A sigmoidal transformation of the spatially and temporally filtered retinal image substantially improves the sensitivity of the system in detecting a small target moving in place against a static natural background in the presence of comparatively large, fast simulated eye movements, but is detrimental to the direction-selectivity of the motion detector. The sigmoid has insignificant effects on detector performance in simulations of slow, high contrast laboratory stimuli. These results suggest that the sigmoid reduces the system's noise sensitivity.     

Bi-directional navigation intent communication using spatial augmented reality and eye-tracking glasses for improved safety in human–robot interaction

Safety, legibility and efficiency are essential for autonomous mobile robots that interact with humans. A key factor in this respect is bi-directional communication of navigation intent, which we focus on in this article with a particular view on industrial logistic applications. In the direction robot-to-human, we study how a robot can communicate its navigation intent using Spatial Augmented Reality (SAR) such that humans can intuitively understand the robot’s intention and feel safe in the vicinity of robots. We conducted experiments with an autonomous forklift that projects various patterns on the shared floor space to convey its navigation intentions. We analyzed trajectories and eye gaze patterns of humans while interacting with an autonomous forklift and carried out stimulated recall interviews (SRI) in order to identify desirable features for projection of robot intentions. In the direction human-to-robot, we argue that robots in human co-habited environments need human-aware task and motion planning to support safety and efficiency, ideally responding to people’s motion intentions as soon as they can be inferred from human cues. Eye gaze can convey information about intentions beyond what can be inferred from the trajectory and head pose of a person. Hence, we propose eye-tracking glasses as safety equipment in industrial environments shared by humans and robots. In this work, we investigate the possibility of human-to-robot implicit intention transference solely from eye gaze data and evaluate how the observed eye gaze patterns of the participants relate to their navigation decisions. We again analyzed trajectories and eye gaze patterns of humans while interacting with an autonomous forklift for clues that could reveal direction intent. Our analysis shows that people primarily gazed on that side of the robot they ultimately decided to pass by. We discuss implications of these results and relate to a control approach that uses human gaze for early obstacle avoidance.     

Active estimation of distance in a robotic system that replicates human eye movement

In a moving agent, the different apparent motion of objects located at various distances provides an important source of depth information. While motion parallax is evident for large translations of the agent, a small parallax also occurs in most head/eye systems during rotations of the cameras. A similar parallax is also present in the human eye, so that a redirection of gaze shifts the projection of an object on the retina by an amount that depends not only on the amplitude of the rotation, but also on the distance of the object with respect to the observer. This study examines the accuracy of distance estimation on the basis of the parallax produced by camera rotations. Sequences of human eye movements were used to control the motion of a pan/tilt system specifically designed to reproduce the oculomotor parallax present in the human eye. We show that the oculomotor strategies by which humans scan visual scenes produce parallaxes that provide accurate estimation of distance. This information simplifies challenging visual tasks such as image segmentation and figure/ground segregation.     

Gaze-based prediction of pen-based virtual interaction tasks

In typical human–computer interaction, users convey their intentions through traditional input devices (e.g. keyboards, mice, joysticks) coupled with standard graphical user interface elements. Recently, pen-based interaction has emerged as a more intuitive alternative to these traditional means. However, existing pen-based systems are limited by the fact that they rely heavily on auxiliary mode switching mechanisms during interaction (e.g. hard or soft modifier keys, buttons, menus). In this paper, we describe how eye gaze movements that naturally occur during pen-based interaction can be used to reduce dependency on explicit mode selection mechanisms in pen-based systems. In particular, we show that a range of virtual manipulation commands, that would otherwise require auxiliary mode switching elements, can be issued with an 88% success rate with the aid of users׳ natural eye gaze behavior during pen-only interaction.     

Improving Head Movement Tolerance of Cross-Ratio Based Eye Trackers

When first introduced, the cross-ratio (CR) based remote eye tracking method offered many attractive features for natural human gaze-based interaction, such as simple camera setup, no user calibration, and invariance to head motion. However, due to many simplification assumptions, current CR-based methods are still sensitive to head movements. In this paper, we revisit the CR-based method and introduce two new extensions to improve the robustness of the method to head motion. The first method dynamically compensates for scale changes in the corneal reflection pattern, and the second method estimates true coplanar eye features so that the cross-ratio can be applied. We present real-time implementations of both systems, and compare the performance of these new methods using simulations and user experiments. Our results show a significant improvement in robustness to head motion and, for the user experiments in particular, an average reduction of up to 40 % in gaze estimation error was observed.     

A field evaluation of driver eye and head movement strategies toward environmental targets and distractors

A new systems approach for evaluating field performance of drivers is presented in a practical experimental demonstration. The emphasis reflected in this paper is on driver head and eye movement behaviour toward environmental targets and distractors. A unique video technology set-up was designed to record eye and head movements. Simultaneous video-tapings of driver eye/head movements and the environmental scene were time synchronised. Two intersections (one busy and one quiet) were carefully selected and analysed for their static and dynamic visual targets and distractors. A subject made 40 repetitive left-turns alternately at busy and quiet intersections in a naturally moving traffic environment. Both head movement patterns and eye movement frequencies differed significantly between the two intersections. Results suggest that the eye and head movements are highly dependent upon the type of turn configuration, type and frequency of targets, type and frequency of distractors, and traffic control configurations.     

视线追踪系统眼部特征检测及视线方向计算方法研究

基于单视觉主动红外光源系统,提出了一种视线检测方法．在眼部特征检测阶段,采用投影法定位人脸;根据人脸对称性和五官分布的先验知识,确定瞳孔潜在区域;最后进行人眼特征的精确分割．在视线方向建模阶段,首先在头部静止的情况下采用非线性多项式建立从平面视线参数到视线落点的映射模型;然后采用广义回归神经网络对不同头部位置造成的视线偏差进行补偿,使非线性映射函数扩展到任何头部位置．实验结果及在交互式图形界面系统中的应用验证了该方法的有效性．