A Computational Model of “Active Vision” for Visual Search in Human–Computer Interaction

Human visual search plays an important role in many human–computer interaction (HCI) tasks. Better models of visual search are needed not just to predict overall performance outcomes, such as whether people will be able to find the information needed to complete an HCI task, but to understand the many human processes that interact in visual search, which will in turn inform the detailed design of better user interfaces. This article describes a detailed instantiation, in the form of a computational cognitive model, of a comprehensive theory of human visual processing known as “active vision” (Findlay & Gilchrist, 2003). The computational model is built using the Executive Process-Interactive Control cognitive architecture. Eye-tracking data from three experiments inform the development and validation of the model. The modeling asks—and at least partially answers—the four questions of active vision: (a) What can be perceived in a fixation? (b) When do the eyes move? (c) Where do the eyes move? (d) What information is integrated between eye movements? Answers include: (a) Items nearer the point of gaze are more likely to be perceived, and the visual features of objects are sometimes misidentified. (b) The eyes move after the fixated visual stimulus has been processed (i.e., has entered working memory). (c) The eyes tend to go to nearby objects. (d) Only the coarse spatial information of what has been fixated is likely maintained between fixations. The model developed to answer these questions has both scientific and practical value in that the model gives HCI researchers and practitioners a better understanding of how people visually interact with computers, and provides a theoretical foundation for predictive analysis tools that can predict aspects of that interaction.     

On the dynamic version of the minimum hand jerk criterion

This paper is concerned with the problem of trajectory formation of humanlike reaching movements. First, we review conventional criteria of optimality adopted in robotics and computational neuroscience for the prediction of reaching movements and formulate a dynamic version of the minimum hand jerk criteria. We call it a minimum driving force change criterion and check its performance for the free‐space movements. Next, we test the performance of the new criterion for the movements where the human hand is geometrically constrained by the external environment, and for the movements with flexible objects. The main feature of these movements is that the hand velocity profiles are not always bell shaped. Our simulations and initial experimental results show that the minimum driving force change criterion can roughly capture this feature and, therefore, can be a reasonable candidate for modeling of humanlike reaching movements. © 2005 Wiley Periodicals, Inc.     

A Portable Bio-Inspired Architecture for Efficient Robotic Vergence Control

In stereoscopic vision, the ability of perceiving the three-dimensional structure of the surrounding environment is subordinated to a precise and effective motor control for the binocular coordination of the eyes/cameras. If, on the one side, the binocular coordination of camera movements is a complicating factor, on the other side, a proper vergence control, acting on the binocular disparity, facilitates the binocular fusion and the subsequent stereoscopic perception process. In real-world situations, an effective vergence control requires further features other than real time capabilities: real robot systems are indeed characterized by mechanical and geometrical imprecision that affect the binocular vision, and the illumination conditions are changeable and unpredictable. Moreover, in order to allow an effective visual exploration of the peripersonal space, it is necessary to cope with different gaze directions and provide a large working space. The proposed control strategy resorts to a neuromimetic approach that provides a distributed representation of disparity information. The vergence posture is obtained by an open-loop and a closed-loop control, which directly interacts with saccadic control. Before saccade, the open-loop component is computed in correspondence of the saccade target region, to obtain a vergence correction to be applied simultaneously with the saccade. At fixation, the closed-loop component drives the binocular disparity to zero in a foveal region. The obtained vergence servos are able to actively drive both the horizontal and the vertical alignment of the optical axes on the object of interest, thus ensuring a correct vergence posture. Experimental tests were purposely designed to measure the performance of the control in the peripersonal space, and were performed on three different robot platforms. The results demonstrated that the proposed approach yields real-time and effective vergence camera movements on a visual stimulus in a wide working range, regardless of the illumination in the environment and the geometry of the system.     

A formal framework to represent spatial knowledge

Visual representations are an essential element in human–computer interaction and can be conceived as a collection of graphical objects arranged in a two-dimensional space. It is quite natural to model visual representations through the qualitative relationships holding between their objects, and therefore, qualitative spatial relations are a fundamental way of representing spatial knowledge. To this aim, in this paper we present a framework of qualitative spatial relations providing a general, domain-independent approach to specify visual representations.     

A general framework for trajectory data warehousing and visual OLAP

In this paper we present a formal framework for modelling a trajectory data warehouse (TDW), namely a data warehouse aimed at storing aggregate information on trajectories of moving objects, which also offers visual OLAP operations for data analysis. The data warehouse model includes both temporal and spatial dimensions, and it is flexible and general enough to deal with objects that are either completely free or constrained in their movements (e.g., they move along a road network). In particular, the spatial dimension and the associated concept hierarchy reflect the structure of the environment in which the objects travel. Moreover, we cope with some issues related to the efficient computation of aggregate measures, as needed for implementing roll-up operations. The TDW and its visual interface allow one to investigate the behaviour of objects inside a given area as well as the movements of objects between areas in the same neighbourhood. A user can easily navigate the aggregate measures obtained from OLAP queries at different granularities, and get overall views in time and in space of the measures, as well as a focused view on specific measures, spatial areas, or temporal intervals. We discuss two application scenarios of our TDW, namely road traffic and vessel movement analysis, for which we built prototype systems. They mainly differ in the kind of information available for the moving objects under observation and their movement constraints.     

Modeling and driving a reduced human mannequin through motioncaptured data: a neural network approach

One of the major problems which arises in the field of virtual design is the realization of virtual mannequins able to move in a human like way. This work focuses on the analysis of the human sitting working posture, which is described by a 30-DOF mannequin, modeling the upper part of the body (pelvis, trunk, arms, and head). Trajectories formation in point to point reaching movements represents the main topic. Our approach is based on the acquisition of real human kinematics data, collected by means of an automatic motion analyzer. Starting from the kinematics database of one subject, sit in front of a desk, a neural network was trained in order to generate the movements of the virtual mannequin. The work is divided into four parts: mannequin modeling, 3D human data collection, data preprocessing according to the biomechanical model, and design and training of a multilayer perceptron neural network     

Communicating emotions and mental states to robots in a real time parallel framework using Laban movement analysis

This paper presents a parallel real time framework for emotions and mental states extraction and recognition from video fragments of human movements. In the experimental setup human hands are tracked by evaluation of moving skin-colored objects. The tracking analysis demonstrates that acceleration and frequency characteristics of the traced objects are relevant for classification of the emotional expressiveness of human movements. The outcomes of the emotional and mental states recognition are cross-validated with the analysis of two independent certified movement analysts (CMA’s) who use the Laban movement analysis (LMA) method. We argue that LMA based computer analysis can serve as a common language for expressing and interpreting emotional movements between robots and humans, and in that way it resembles the common coding principle between action and perception by humans and primates that is embodied by the mirror neuron system. The solution is part of a larger project on interaction between a human and a humanoid robot with the aim of training social behavioral skills to autistic children with robots acting in a natural environment.     

A method for the analysis of the interaction between users and objects in 3D navigational space

Along with the improvement of eye-tracking technology, more and more distinct field of researches have introduced movements of the eye in relation to the head to understand user behavior. Most of current researches focus on the perception process of single 2-dimensional images by fixed eye-tracking devices or the head-mount devices. A method of applying eye-tracking on the analysis of the interaction between users and objects in 3D navigational space is proposed in this article. It aims to understand the visual stimulation of 3D objects and the user’s spatial navigational reactions while receiving the stimulation, and proposes the concept of 3D object attention heat map. It also proposes to construct a computational visual attention model for different geometric featured 3D objects by applying the method of feature curves. The VR results of this study also provide future assistance in the incoming immersive world. This study sets to promote eye-tracking from the mainstream of 2D field to 3D spaces and points to a deeper understanding between human and artificial product or natural objects. It would also serve an important role in the field of human-computer interaction, product usability, aids devices for cognition degenerative individuals, and even the field of visual recognition of daily human behavior.     

Pose estimation through cue integration: a neuroscience-inspired approach

The aim of this paper is to improve the skills of robotic systems in their interaction with nearby objects. The basic idea is to enhance visual estimation of objects in the world through the merging of different visual estimators of the same stimuli. A neuroscience-inspired model of stereoptic and perspective orientation estimators, merged according to different criteria, is implemented on a robotic setup and tested in different conditions. Experimental results suggest that the integration of multiple monocular and binocular cues can make robot sensory systems more reliable and versatile. The same results, compared with simulations and data from human studies, show that the model is able to reproduce some well-recognized neuropsychological effects.     

A globally optimal estimator for the delta-lognormal modeling of fast reaching movements

Fast reaching movements are an important component of our daily interaction with the world and are consequently under investigation in many fields of science and engineering. Today, useful models are available for such studies, with tools for solving the inverse dynamics problem involved by these analyses. These tools generally provide a set of model parameters that allows an accurate and locally optimal reconstruction of the original movements. Although the solutions that they generate may provide a data curve fitting that is sufficient for some pattern recognition applications, the best possible solution is often necessary in others, particularly those involving neuroscience and biomedical signal processing. To generate these solutions, we present a globally optimal parameter extractor for the delta-lognormal modeling of reaching movements based on the branch-and-bound strategy. This algorithm is used to test the impact of white noise on the delta-lognormal modeling of reaching movements and to benchmark the state-of-the-art locally optimal algorithm. Our study shows that, even with globally optimal solutions, parameter averaging is important for obtaining reliable figures. It concludes that physiologically derived rules are necessary, in addition to global optimality, to achieve meaningful ?Λ extractions which can be used to investigate the control patterns of these movement primitives.     

军事工程实体信息及数据描述方法研究

针对军事工程真三维连续特性及其实体表达的不确定性,给出了军事工程实体信息及数据描述方法,目的是为视景仿真、毁伤建模与仿真等应用领域提供服务。在表达军事工程地理实体认知的基础上,提出了实体信息描述要求;以概念建模为顶层设计,从地学、几何、拓扑、材质等层面进行抽象,给出了军事工程真三维空间集成建模方法;抽象点、线、面、体等实体,给出了军事工程实体模型集成表达方法,并以UML类模型表达了军事工程实体的数据模型。该方法应用于军事工程实体建模,能够较好地表示军事工程三维空间对象,并能有效支持围绕军事工程实体的各种拓扑操作和军事应用。     

Trajectory databases: Data models,uncertainty and complete query languages

Moving objects produce trajectories. We describe a data model for trajectories and trajectory samples and an efficient way of modeling uncertainty via beads for trajectory samples. We study transformations of the ambient space for which important physical properties of trajectories, such as speed, are invariant. We also determine which transformations preserve beads. We give conceptually easy first-order complete query languages and computationally complete query languages for trajectory databases, which allow to talk directly about speed and uncertainty in terms of beads. The queries expressible in these languages are invariant under speed- and bead-preserving transformations.     

Content-trajectory approach for searching video databases

In the past few years, modeling and querying video databases have been a subject of extensive research to develop tools for effective search of videos. In this paper, we present a hierarchal approach to model videos at three levels, object level (OL), frame level (FL), and shot level (SL). The model captures the visual features of individual objects at OL, visual-spatio-temporal (VST) relationships between objects at FL, and time-varying visual features and time-varying VST relationships at SL. We call the combination of the time-varying visual features and the time-varying VST relationships a Content trajectory which is used to represent and index a shot. A novel query interface that allows users to describe the time-varying contents of complex video shots such as those of skiers, soccer players, etc., by sketch and feature specification is presented. Our experimental results prove the effectiveness of modeling and querying shots using the content trajectory approach.     

Multiperson visual focus of attention from head pose and meeting contextual cues

This paper introduces a novel contextual model for the recognition of people's visual focus of attention (VFOA) in meetings from audio-visual perceptual cues. More specifically, instead of independently recognizing the VFOA of each meeting participant from his own head pose, we propose to jointly recognize the participants' visual attention in order to introduce context-dependent interaction models that relate to group activity and the social dynamics of communication. Meeting contextual information is represented by the location of people, conversational events identifying floor holding patterns, and a presentation activity variable. By modeling the interactions between the different contexts and their combined and sometimes contradictory impact on the gazing behavior, our model allows us to handle VFOA recognition in difficult task-based meetings involving artifacts, presentations, and moving people. We validated our model through rigorous evaluation on a publicly available and challenging data set of 12 real meetings (5 hours of data). The results demonstrated that the integration of the presentation and conversation dynamical context using our model can lead to significant performance improvements.     

Efficient pose estimation using view-based object representations

We present an efficient method for estimating the pose of a three-dimensional object. Its implementation is embedded in a computer vision system which is motivated by and based on cognitive principles concerning the visual perception of three-dimensional objects. Viewpoint-invariant object recognition has been subject to controversial discussions for a long time. An important point of discussion is the nature of internal object representations. Behavioral studies with primates, which are summarized in this article, support the model of view-based object representations. We designed our computer vision system according to these findings and demonstrate that very precise estimations of the poses of real-world objects are possible even if only a small number of sample views of an object is available. The system can be used for a variety of applications.     

Design, construction, and application of a generic visual languagegeneration environment

The implementation of visual programming languages (VPLs) and their supporting environments is time-consuming and tedious. To ease the task, researchers have developed some high-level tools to reduce the development effort. None of these tools, however, can be easily used to create a complete visual language in a seamless way as the lex/yacc tools do for textual language constructions. This paper presents the design, construction and application of a generic visual language generation environment, called VisPro. The VisPro design model improves the conventional model-view-controller framework in that its functional modules are decoupled to allow independent development and integration. The VisPro environment consists of a set of visual programming tools. Using VisPro, the process of VPL construction can be divided into two steps: lexicon definition and grammar specification. The former step defines visual objects and a visual editor, and the latter step provides language grammars with graph rewriting rules. The compiler for the VPL is automatically created according to the grammar specification. A target VPL is generated as a programming environment which contains the compiler and the visual editor. The paper demonstrates how VisPro is used by building a simple visual language and a more complex visual modeling language for distributed programming     

Gesture modeling for architectural design

This paper presents an intuitive method to simulate architectural space and objects by using gesture modeling. This method applies hand movements to retrieve data and generate objects under circumstances allowing larger tolerance. Communication between gesture (actor) and computer (listener) is conducted with a set of basic components like gestures, regulations, and interfaces. The components are used as theoretical models to assist gesture-aided 3-D modeling. Gesture is analyzed by both surface and deep structures in syntactic structure. At the outset, persons of different backgrounds were requested to model a virtual building. Several basic gesture types were concluded based on experiment categorization. The test result was used to build a theoretical model and conduct subsequent simulation. A 3-D digitizer associated with software is applied as a communication interface between user and computer. Gestures are simulated by moving a stylus held by one hand.     

Solid modeling for architectural design using octpaths

A modeling system for Architectural Design which is based on a very general data structure is presented. Octpaths break the modeling space into solid and not solid material thus describing an entire volume of 3-D objects. Boolean operators are used to combine solid primitives. Very efficient algorithms allow the design of complex objects in a limited raster-oriented computing environment.     

Computational modeling of human performance in multiple monitor environments with ACT-R cognitive architecture

Much research on modeling human performance associated with visual perception is formulated by schematic models based on neural mechanisms or cognitive architectures. But, these two modeling paradigms are limited in the domains of multiple monitor environments. Although the schematic model based on neural mechanisms can represent human visual systems in multiple monitor environments by providing a detailed account of eye and head movements, these models cannot easily be applied in complex cognitive interactions. On the other hand, the cognitive architectures can model the interaction of multiple aspects of cognition, but these architectures have not focused on modeling the visual orienting behavior of eye and head movements. Thus, in this study, a specific cognitive architecture, which is ACT-R, is extended by an existing schematic model of human visual systems based on neural mechanisms in order to model human performance in multiple monitor environments more accurately. And, this study proposes a method of modeling human performance using the extended ACT-R. The proposed method is validated by an experiment, confirming that the proposed method is able to predict human performance more accurately in multiple monitor environments.Relevance to industryPredicting human performance with a computational model can be used as an alternative method to implementing iterative user testing for developing a system interface. The computational model in this study can predict human performance in multiple monitor environments, so that the model can be applied early on in the design phase, to evaluate the system interface in multiple monitor environments.     

A fast biologically inspired algorithm for recurrent motion estimation

We have previously developed a neurodynamical model of motion segregation in cortical visual area V1 and MT of the dorsal stream. The model explains how motion ambiguities caused by the motion aperture problem can be solved for coherently moving objects of arbitrary size by means of cortical mechanisms. The major bottleneck in the development of a reliable biologically inspired technical system with real-time motion analysis capabilities based on this neural model is the amount of memory necessary for the representation of neural activation in velocity space. We propose a sparse coding framework for neural motion activity patterns and suggest a means by which initial activities are detected efficiently. We realize neural mechanisms such as shunting inhibition and feedback modulation in the sparse framework to implement an efficient algorithmic version of our neural model of cortical motion segregation. We demonstrate that the algorithm behaves similarly to the original neural model and is able to extract image motion from real world image sequences. Our investigation transfers a neuroscience model of cortical motion computation to achieve technologically demanding constraints such as real-time performance and hardware implementation. In addition, the proposed biologically inspired algorithm provides a tool for modeling investigations to achieve acceptable simulation time