Activity recognition using eye-gaze movements and traditional interactions

The need for intelligent HCI has been reinforced by the increasing numbers of human-centered applications in our daily life. However, in order to respond adequately, intelligent applications must first interpret users’ actions. Identifying the context in which users’ interactions occur is an important step toward automatic interpretation of behavior. In order to address a part of this context-sensing problem, we propose a generic and application-independent framework for activity recognition of users interacting with a computer interface. Our approach uses Layered Hidden Markov Models (LHMM) and is based on eye-gaze movements along with keyboard and mouse interactions. The main contribution of the proposed framework is the ability to relate users’ interactions to a task model in variant applications and for different monitoring purposes. Experimental results from two user studies show that our activity recognition technique is able to achieve good predictive accuracy with a relatively small amount of training data.     

A swarm intelligence approach to the quadratic minimum spanning tree problem

The quadratic minimum spanning tree problem (Q-MST) is an extension of the minimum spanning tree problem (MST). In Q-MST, in addition to edge costs, costs are also associated with ordered pairs of distinct edges and one has to find a spanning tree that minimizes the sumtotal of the costs of individual edges present in the spanning tree and the costs of the ordered pairs containing only edges present in the spanning tree. Though MST can be solved in polynomial time, Q-MST is NP-Hard. In this paper we present an artificial bee colony (ABC) algorithm to solve Q-MST. The ABC algorithm is a new swarm intelligence approach inspired by intelligent foraging behavior of honey bees. Computational results show the effectiveness of our approach.     

Task-based annotation and retrieval for image information management

Continuing advances in digital image capture and storage are resulting in a proliferation of imagery and associated problems of information overload in image domains. In this work we present a framework that supports image management using an interactive approach that captures and reuses task-based contextual information. Our framework models the relationship between images and domain tasks they support by monitoring the interactive manipulation and annotation of task-relevant imagery. During image analysis, interactions are captured and a task context is dynamically constructed so that human expertise, proficiency and knowledge can be leveraged to support other users in carrying out similar domain tasks using case-based reasoning techniques. In this article we present our framework for capturing task context and describe how we have implemented the framework as two image retrieval applications in the geo-spatial and medical domains. We present an evaluation that tests the efficiency of our algorithms for retrieving image context information and the effectiveness of the framework for carrying out goal-directed image tasks.     

Connected-component labeling based on hypercubes for memory constrained scenarios

The extraction and labeling of connected components in images play an important role in a wide range of fields, such as computer vision, remote sensing, medicine, biometrics, document analysis, robotics, among others. The automatic identification of relevant image regions allows for the development of intelligent systems to address complex problems for segmentation, classification and interpretation purposes. In this work, we present novel algorithms for labeling connected components that do not require any data structure on the labeling process. The algorithms are derived from other based upon independent spanning trees over the hypercube graph. Initially, the image coordinates are mapped to a binary Gray code axis, such that all pixels that are neighbors in the image are neighbors on the hypercube and each node that belongs to the hypercube represents a pixel in the image. We then use the algorithm proposed by Silva et al. (2013) to generate the log N independent spanning trees over the image. The proposed methods for connected-component labeling are applied to a number of images to demonstrate its effectiveness.     

Bayesian modeling of uncertainty in low-level vision

The need for error modeling, multisensor fusion, and robust algorithms is becoming increasingly recognized in computer vision. Bayesian modeling is a powerful, practical, and general framework for meeting these requirements. This article develops a Bayesian model for describing and manipulating the dense fields, such as depth maps, associated with low-level computer vision. Our model consists of three components: a prior model, a sensor model, and a posterior model. The prior model captures a priori information about the structure of the field. We construct this model using the smoothness constraints from regularization to define a Markov Random Field. The sensor model describes the behavior and noise characteristics of our measurement system. We develop a number of sensor models for both sparse and dense measurements. The posterior model combines the information from the prior and sensor models using Bayes' rule. We show how to compute optimal estimates from the posterior model and also how to compute the uncertainty (variance) in these estimates. To demonstrate the utility of our Bayesian framework, we present three examples of its application to real vision problems. The first application is the on-line extraction of depth from motion. Using a two-dimensional generalization of the Kalman filter, we develop an incremental algorithm that provides a dense on-line estimate of depth whose accuracy improves over time. In the second application, we use a Bayesian model to determine observer motion from sparse depth (range) measurements. In the third application, we use the Bayesian interpretation of regularization to choose the optimal smoothing parameter for interpolation. The uncertainty modeling techniques that we develop, and the utility of these techniques in various applications, support our claim that Bayesian modeling is a powerful and practical framework for low-level vision.     

Improving the Behavior of Intelligent Tutoring Agents with Data Mining

This article presents a novel framework for adapting the behavior of intelligent agents. The framework consists of an extended sequential pattern mining algorithm that, in combination with association rule discovery techniques, is used to extract temporal patterns and relationships from the behavior of human agents executing a procedural task. The proposed framework has been integrated within the CanadarmTutor, an intelligent tutoring agent aimed at helping students solve procedural problems that involve moving a robotic arm in a complex virtual environment. We present the results of an evaluation that demonstrates the benefits of this integration to agents acting in ill-defined domains.     

Agent Technologies for Sensor Networks

Wireless sensor networks are increasingly seen as a solution to the problem of performing continuous wide-area monitoring in many environmental, security, and military scenarios. The distributed nature of such networks and the autonomous behavior expected of them present many novel challenges. In this article, the authors argue that a new synthesis of electronic engineering and agent technology is required to address these challenges, and they describe three examples where this synthesis has succeeded. In more detail, they describe how these novel approaches address the need for communication and computationally efficient decentralized algorithms to coordinate the behavior of physically distributed sensors, how they enable the real-world deployment of sensor agent platforms in the field, and finally, how they facilitate the development of intelligent agents that can autonomously acquire data from these networks and perform information processing tasks such as fusion, inference, and prediction.     

Identifying shoplifting behaviors and inferring behavior intention based on human action detection and sequence analysis

Identification of abnormal behaviors affecting public safety (e.g., shoplifting, robbery, and stealing) is essential for preventing human casualties and property damage. Many studies have attempted to automatically identify abnormal behaviors by detecting relevant human actions by developing intelligent video surveillance systems. However, these studies have focused on catching predefined actions associated explicitly with the target abnormal behavior, which can lead to errors in judgment when such actions are undetected or inaccurately detected. To better identify abnormal behaviors, it is essential to understand a series of performed actions to capture behaviors’ pre- and post-indications (e.g., repeatably looking around and spotting CCTVs) and infer the intentions underlying such behaviors. Thus, in the present study, we propose a framework to identify abnormal behaviors through deep-learning-based detection of non-semantic-level human action components segmented with a window size of several seconds (e.g., walking, standing, and watching) and performing sequence analyses of the detected action components to infer behavior intentions. Then, we tested the applicability of the framework to the specific scenario of shoplifting, one of the most common crimes. Analysis of actual incident data confirmed that shoplifting intentions could be effectively gauged based on distinct action sequence features, and the intention inference results are continuously updated with the accumulated series of detected actions during the course of the input video stream. The results of this study can help enhance the ability of intelligent surveillance systems by providing a new means for monitoring abnormal behaviors and deeply understanding the underlying intentions.     

SACST: smart anthropomorphic contact surface technology

In this article, we present the ongoing research work of smart anthropomorphic contact surface technology (SACST) and applications in robotics and human-augmented systems. Integrating MEMS technology with contact theory and intelligent-control techniques, SACST can alter the behavior of contact surfaces based on robotics theories. To make SACST products more reliable and efficient, certain critical components are indispensable. For example, MEMS valves should be able to work under more resistant force and with less power consumption; the contact sensor should have higher and flexible resolution; and the actuation mechanism of the bladders should become more integrated, more flexible, and easier to control. Next-generation SACST technology will also integrate intelligent operation systems and control with actuators and sensors.     

新型嵌入式网络智能视频监控系统构架方案

运用高性能的ARM处理器设计出了一种新型嵌入式网络智能视频监控系统构架方案,此监控系统方案具有预处理的功能,其检测精度较高,该系统能够满足实时监控和智能检测要求,本文将具体对该系统的硬件系统和软件系统做出具体介绍。     

Wi-Fi based non-invasive detection of indoor wandering using LSTM model

Wandering is a significant indicator in the clinical diagnosis of dementia and other related diseases for elders. Reliable monitoring of long-term continuous movement in indoor setting for detection of wandering movement is challenging because most elders are prone to forget to carry or wear sensors that collect motion information daily due to their declining memory. Wi-Fi as an emerging sensing modality has been widely used to monitor human indoor movement in a noninvasive manner. In order to continuously monitor individuals’ indoor motion and reliably identify wandering movement in a non-invasive manner, in this work, we develop a LSTMbased deep classification method that is able to differentiate the wandering-causedWi-Fi signal change from the others. Specifically, we first use the off-the-shelf Wi-Fi devices to capture a resident’s indoor motion information, enabling to collect a group ofWi-Fi signal streams, which will be split into variablesize segments. Second, the deep network LSTM is adopted to develop wandering detection method that is able to classify every variable-size segment of Wi-Fi signals into categories according to the well-known wandering spatiotemporal patterns. Last, experimental evaluation conducted on a group of realworld Wi-Fi signal streams shows that our proposed LSTMbased detection method is workable and effective to identify indoor wandering behavior, obtaining an average value of 0.9286, 0.9618, 0.9634 and 0.9619 for accuracy, precision, recall and F-1 score, respectively.     

Visualizing the history of living spaces

The technology available to building designers now makes it possible to monitor buildings on a very large scale. Video cameras and motion sensors are commonplace in practically every office space, and are slowly making their way into living spaces. The application of such technologies, in particular video cameras, while improving security, also violates privacy. On the other hand, motion sensors, while being privacy-conscious, typically do not provide enough information for a human operator to maintain the same degree of awareness about the space that can be achieved by using video cameras. We propose a novel approach in which we use a large number of simple motion sensors and a small set of video cameras to monitor a large office space. In our system we deployed 215 motion sensors and six video cameras to monitor the 3,000-square-meter office space occupied by 80 people for a period of about one year. The main problem in operating such systems is finding a way to present this highly multidimensional data, which includes both spatial and temporal components, to a human operator to allow browsing and searching recorded data in an efficient and intuitive way. In this paper we present our experiences and the solutions that we have developed in the course of our work on the system. We consider this work to be the first step in helping designers and managers of building systems gain access to information about occupants' behavior in the context of an entire building in a way that is only minimally intrusive to the occupants' privacy.     

A TRAINING SIMULATION SYSTEM WITH REALISTIC AUTONOMOUS SHIP CONTROL

In this article we present a computational approach to developing effective training systems for virtual simulation environments. In particular, we focus on a Naval simulation system, used for training of conning officers. The currently existing training solutions require multiple expert personnel to control each vessel in a training scenario, or are cumbersome to use by a single instructor. The inability of current technology to provide an automated mechanism for competitive realistic boat behaviors thus compromises the goal of flexible, anytime, anywhere training. In this article we propose an approach that reduces the time and effort required for training of conning officers, by integrating novel approaches to autonomous control within a simulation environment. Our solution is to develop intelligent, autonomous controllers that drive the behavior of each boat. To increase the system's efficiency we provide a mechanism for creating such controllers, from the demonstration of a navigation expert, using a simple programming interface. In addition, our approach deals with two significant and related challenges: the realism of behavior exhibited by the automated boats and their real-time response to changes in the environment. In this article, we describe the control architecture we developed that enables the real-time response of boats and the repertoire of realistic behaviors we designed for this application. We also present our approach for facilitating the automatic authoring of training scenarios and we demonstrate the capabilities of our system with experimental results.     

The metacognitive loop I: Enhancing reinforcement learning with metacognitive monitoring and control for improved perturbation tolerance

Maintaining adequate performance in dynamic and uncertain settings has been a perennial stumbling block for intelligent systems. Nevertheless, any system intended for real-world deployment must be able to accommodate unexpected change—that is, it must be perturbation tolerant. We have found that metacognitive monitoring and control—the ability of a system to self-monitor its own decision-making processes and ongoing performance, and to make targeted changes to its beliefs and action-determining components—can play an important role in helping intelligent systems cope with the perturbations that are the inevitable result of real-world deployment. In this article we present the results of several experiments demonstrating the efficacy of metacognition in improving the perturbation tolerance of reinforcement learners, and discuss a general theory of metacognitive monitoring and control, in a form we call the metacognitive loop.     

MTConnect compliant monitoring for finishing assembly interfaces of large-scale components: A vertical tail section application

Monitoring is a significant issue for finishing the assembly interfaces of large-scale components before final assembly. Acquisition and supervision of the pivotal data is essential to ensure the security and reliability for machining the large and complicated components with high-value. This process is generally cumbersome and time-consuming because there are various types of data coming from different components and sensors. The problem becomes more serious when considering the whole shop floor. Recently, MTConnect has been proven to be an effective method to realize standardized data collection and monitoring process. However, MTConnect is still under development and cannot cover the whole finishing process such as on-machining measuring (OMM) and fixturing. To address the issue, an MTConnect compliant method with extended data models is proposed in this paper to implement a standardized monitoring system. Firstly, a finishing system for the assembly interfaces is introduced, including the framework, workflow and key procedures and data. Then extended MTConnect data models are proposed to represent the finishing system including on-machine touch-trigger probe and sensor-based intelligent fixturing related information. Based on the extended MTConnect data models, a web-based monitoring system is developed for data collection and monitoring by combining an MTConnect agent and an OPC adapter. The proposed approach is validated by collecting and monitoring the key process data using an airplane vertical tail as an application. The advantages of using MTConnect would be more significant when extended to the entire factory and implemented in cloud manufacturing in the future.     

A temporal logic-based planning and execution monitoring framework for unmanned aircraft systems

Research with autonomous unmanned aircraft systems is reaching a new degree of sophistication where targeted missions require complex types of deliberative capability integrated in a practical manner in such systems. Due to these pragmatic constraints, integration is just as important as theoretical and applied work in developing the actual deliberative functionalities. In this article, we present a temporal logic-based task planning and execution monitoring framework and its integration into a fully deployed rotor-based unmanned aircraft system developed in our laboratory. We use a very challenging emergency services application involving body identification and supply delivery as a vehicle for showing the potential use of such a framework in real-world applications. TALplanner, a temporal logic-based task planner, is used to generate mission plans. Building further on the use of TAL (Temporal Action Logic), we show how knowledge gathered from the appropriate sensors during plan execution can be used to create state structures, incrementally building a partial logical model representing the actual development of the system and its environment over time. We then show how formulas in the same logic can be used to specify the desired behavior of the system and its environment and how violations of such formulas can be detected in a timely manner in an execution monitor subsystem. The pervasive use of logic throughout the higher level deliberative layers of the system architecture provides a solid shared declarative semantics that facilitates the transfer of knowledge between different modules.     

Making pervasive sensing possible: Effective travel mode sensing based on smartphones

Smartphones with embedded Global Positioning Systems (GPS) sensors and accelerometers provide outstanding opportunities to gather information about transportation modes. In comparison to traditional approaches of measuring travel behavior, such as self-reports and travel behavior surveys, a smartphone application that tracks movement increases spatiotemporal resolution and reduces the burden on individuals to manually recall and log travel behavior. Studies using smartphones to detect travel modes mainly use segmentation approaches, which divide movement data into single-mode segments. These approaches hinge on the accurate detection of transitional nodes, which are occasionally difficult to identify. In this study, we proposed a method to detect travel modes based on the chained random forest (RF) model, which automatically classifies smartphone data into different travel modes without using a prior search for transitional nodes. We evaluated the proposed method by collecting and analyzing 12 people's travel behavior spanning six days. The proposed method achieved 93.8% overall accuracy and performed well in both indoor and outdoor environments. This travel mode detection model offers potentials in conducting pervasive sensing, which will eventually benefit many areas of research that require large scale travel behavior monitoring.     

Robot guidance using ultrasonic arrays

The flexibility of robots can be considerably increased by having sensors that can detect the location and orientation of the components that are being manipulated by the robots. Vision systems have been widely applied to this problem, but there has been very little reported work on the use of ultrasonic sensors. This article reports the use of novel ultrasonic sensors that have been manufactured in-house to measure the range and bearing of a target and guide the robot in a pick and place operation. The configuration of the transducer is described as well as the signal processing method employed. Experimental results are given and future developments and applications of the system are described. © 1994 John Wiley & Sons, Inc.