Predictive modeling of human operators using parametric and neuro-fuzzy models by means of computer-based identification experiment

Driving a car and piloting an airplane are the most common examples for manual control of complicated processes. Human operators are known to be nonlinear, adaptive, time varying and intelligent controllers. In some cases, the human operator may or may not be well trained or an expert, showing different dynamics from operator to operator as in driving example. Therefore, it is very difficult to obtain mathematical models of human operators in a human-in-the-loop-manual control tasks. The goal of this research is to find a simple dynamic model for the prediction of the human operator actions in a manual control system. A computer-based experiment has been designed using the system identification theory to collect data from human operators. The autoregressive with exogenous inputs (ARX), as a parametric model and the adaptive-network-based fuzzy inference system (ANFIS), as an intelligent modeling approach that has the advantages of both neural networks and fuzzy logic, have been investigated and compared for simple and fast implementation to predict the response of human operators. ANFIS, having only 32 rules, provided much better prediction results than ARX model.     

Neural networks and genetic algorithms can support human supervisory control to reduce fossil fuel power plant emissions

Artificial neural networks and genetic algorithms are two intelligent approaches initially targeted to model human information processing and natural evolutionary process, with the aim of using the models in problem solving. During the last decade these two intelligent approaches have been widely applied to a variety of social, economic and engineering systems. In this paper, they have been shown as modelling tools to support human supervisory control to reduce fossil fuel power plant emissions, particularly NO_x emissions. Human supervisory control of fossil fuel power generation plants has been studied, and the need of an advisory system for operator support is emphasized. Plant modelling is an important block in such an advisory system and is the key issue of this study. In particular, three artificial neural network models and a genetic algorithm-based grey-box model have been built to model and predict the NO_x emissions in a coal-fired power plant. In non-linear dynamic system modelling, training data is always limited and cannot cover all system dynamics; therefore the generalization performance of the resultant model over unseen data is the focus of this study. These models will then be used in the advisory system to support human operators on aspects such as task analysis, condition monitoring and operation optimization, with the aim of improving thermal efficiency, reducing pollutant emissions and ensuring that the power system runs safely.     

Discriminative Learning for Dynamic State Prediction

We consider the problem of predicting a sequence of real-valued multivariate states that are correlated by some unknown dynamics, from a given measurement sequence. Although dynamic systems such as the State-Space Models are popular probabilistic models for the problem, their joint modeling of states and observations, as well as the traditional generative learning by maximizing a joint likelihood may not be optimal for the ultimate prediction goal. In this paper, we suggest two novel discriminative approaches to the dynamic state prediction: 1) learning generative state-space models with discriminative objectives and 2) developing an undirected conditional model. These approaches are motivated by the success of recent discriminative approaches to the structured output classification in discrete-state domains, namely, discriminative training of Hidden Markov Models and Conditional Random Fields (CRFs). Extending CRFs to real multivariate state domains generally entails imposing density integrability constraints on the CRF parameter space, which can make the parameter learning difficult. We introduce an efficient convex learning algorithm to handle this task. Experiments on several problem domains, including human motion and robot-arm state estimation, indicate that the proposed approaches yield high prediction accuracy comparable to or better than state-of-the-art methods.     

Towards models of tasks and task complexity in supervisory control applications

Dynamic task environments in supervisory control situations differ from those traditionally investigated in problem-solving research in that (1) several task goals exist in parallel, (2) task goals change dynamically as die behaviour of the technical process changes, and (3) information required to accomplish task goals changes across time. In the present work, it is suggested that such dynamic task environments can be described using two types of task goal networks, namely a control task goal (CTG) network and an information processing goal (IPG) network. CTG networks are generated by analysis of the operational states required to produce the commodity for which a technical system has been designed. For example, such analyses can be performed using approaches such as Mitchell's operator function model or canonical means-end analyses. IPG networks are generated by using the recenUy proposed functional information and knowledge acquisition (FIKA) modelling technique. Two examples from different domains illustrate how these task goal networks can be used to describe dynamic task environments. Finally, two different ways of using the task modelling approach are briefly discussed.     

Predictive human operator model to be utilized as a controller using linear,neuro-fuzzy and fuzzy-ARX modeling techniques

Modeling human operator's behavior as a controller in a closed-loop control system recently finds applications in areas such as training of inexperienced operators by expert operator's model or developing warning systems for drivers by observing the driver model parameter variations. In this research, first, an experimental setup has been developed for collecting data from human operators as they controlled a nonlinear system. Appropriate reference signals and scenarios were designed according to the system identification and human operator modeling theory, to collect data from subjects. Different modeling schemes, namely ARX models as linear approach, and adaptive-network-based fuzzy inference system (ANFIS) as intelligent modeling approach have been evaluated. A hybrid modeling method, fuzzy-ARX (F-ARX) model, has been developed and its performance was found to be better in terms of predicting human operator's control actions as well as replacing the operator as a stand-alone controller. It has been concluded that F-ARX models can be a good alternative for modeling the human operator.     

Using discrete event simulation to model excavator operator performance

The traditional design process of fluid power systems such as hydraulic excavators has placed much emphasis on technical performance rather than human components. This research aims to develop human‐performance models to assess operator performance and human interaction during excavation processes. Task analysis, time studies, and statistical distributions were developed into task‐network models and imbedded into four Micro Saint simulation models with regard to various expertise and control types. An empirical study was conducted using the simulation models. Results indicated that both expertise and control type had a significant impact on operator performance, resulting in both time and consistency differences at various points during excavation processes. Models also revealed implications of operator fatigue leading to stress for the operator. Recommendations suggest that designers consider the placement of controls and measures to reduce operator workload for better performance in future systems. © 2010 Wiley Periodicals, Inc.     

Designing automation for human use: empirical studies and quantitative models

An emerging knowledge base of human performance research can provide guidelines for designing automation that can be used effectively by human operators of complex systems. Which functions should be automated and to what extent in a given system? A model for types and levels of automation that provides a framework and an objective basis for making such choices is described. The human performance consequences of particular types and levels of automation constitute primary evaluative criteria for automation design when using the model. Four human performance areas are considered—mental workload, situation awareness, complacency and skill degradation. Secondary evaluative criteria include such factors as automation reliability, the risks of decision/action consequences and the ease of systems integration. In addition to this qualitative approach, quantitative models can inform design. Several computational and formal models of human interaction with automation that have been proposed by various researchers are reviewed. An important future research need is the integration of qualitative and quantitative approaches. Application of these models provides an objective basis for designing automation for effective human use.     

Parts-probability-based vehicle detection

Detecting vehicles is important in aerial surveillance. Traditional methods used classifiers to detect vehicles, but a single classifier was limited to detecting vehicles of only one intensity and orientation. Therefore, the task required the use of multiple classifiers of different intensities and orientations. To solve this problem, we first used a latent Dirichlet allocation （LDA） model that improved the previous approaches to vehicle detection. Previous text modeling approaches have been generative. They could be used to build probability models of vehicles in different intensities and from various orientations simultaneously using unlabeled data. Using a probability model, we can detect vehicles in a region with high probability. Next, we used a parts-probability model that improves the LDA model. The model effectively encodes spatial structure among visual words by adding spatial relationships among vehicle parts as priors of words. A parts probability model represents a vehicle hierarchically according to parts appearances and a vehicle＇s features within the parts to enforce spatial coherency. Then, we used our model to detect vehicles from a collection of images and demonstrate its performs more effectively.     

LSESpeak: A spoken language generator for Deaf people

This paper describes the development of LSESpeak, a spoken Spanish generator for Deaf people. This system integrates two main tools: a sign language into speech translation system and an SMS (Short Message Service) into speech translation system. The first tool is made up of three modules: an advanced visual interface (where a deaf person can specify a sequence of signs), a language translator (for generating the sequence of words in Spanish), and finally, an emotional text to speech (TTS) converter to generate spoken Spanish. The visual interface allows a sign sequence to be defined using several utilities. The emotional TTS converter is based on Hidden Semi-Markov Models (HSMMs) permitting voice gender, type of emotion, and emotional strength to be controlled. The second tool is made up of an SMS message editor, a language translator and the same emotional text to speech converter. Both translation tools use a phrase-based translation strategy where translation and target language models are trained from parallel corpora. In the experiments carried out to evaluate the translation performance, the sign language-speech translation system reported a 96.45 BLEU and the SMS-speech system a 44.36 BLEU in a specific domain: the renewal of the Identity Document and Driving License. In the evaluation of the emotional TTS, it is important to highlight the improvement in the naturalness thanks to the morpho-syntactic features, and the high flexibility provided by HSMMs when generating different emotional strengths.     

PM2.5 concentration prediction using hidden semi-Markov model-based times series data mining

In this paper, a novel framework and methodology based on hidden semi-Markov models (HSMMs) for high PM_2.5 concentration value prediction is presented. Due to lack of explicit time structure and its short-term memory of past history, a standard hidden Markov model (HMM) has limited power in modeling the temporal structures of the prediction problems. To overcome the limitations of HMMs in prediction, we develop the HSMMs by adding the temporal structures into the HMMs and use them to predict the concentration levels of PM_2.5. As a model-driven statistical learning method, HSMM assumes that both data and a mathematical model are available. In contrast to other data-driven statistical prediction models such as neural networks, a mathematical functional mapping between the parameters and the selected input variables can be established in HSMMs. In the proposed framework, states of HSMMs are used to represent the PM_2.5 concentration levels. The model parameters are estimated through modified forward–backward training algorithm. The re-estimation formulae for model parameters are derived. The trained HSMMs can be used to predict high PM_2.5 concentration levels. The validation of the proposed framework and methodology is carried out in real world applications: prediction of high PM_2.5 concentrations at O’Hare airport in Chicago. The results show that the HSMMs provide accurate predictions of high PM_2.5 concentration levels for the next 24 h.     

Structured learning with constrained conditional models

Making complex decisions in real world problems often involves assigning values to sets of interdependent variables where an expressive dependency structure among these can influence, or even dictate, what assignments are possible. Commonly used models typically ignore expressive dependencies since the traditional way of incorporating non-local dependencies is inefficient and hence leads to expensive training and inference. The contribution of this paper is two-fold. First, this paper presents Constrained Conditional Models (CCMs), a?framework that augments linear models with declarative constraints as a way to support decisions in an expressive output space while maintaining modularity and tractability of training. The paper develops, analyzes and compares novel algorithms for CCMs based on Hidden Markov Models and Structured Perceptron. The proposed CCM framework is also compared to task-tailored models, such as semi-CRFs. Second, we propose CoDL, a?constraint-driven learning algorithm, which makes use of constraints to guide semi-supervised learning. We provide theoretical justification for CoDL along with empirical results which show the advantage of using declarative constraints in the context of semi-supervised training of probabilistic models.     

Establishing information requirements for supervisory controllers in a flexible manufacturing system using GTA

In this article we consider the technological change that has occurred in complex manufacturing systems within the past two decades and the implications it has had on the role of human operators in manufacturing systems control. Our examination ranges from the traditional production line manned by skilled machinists to flexible manufacturing systems (FMS) under supervisory control. On the basis of this study, we raise the question as to whether new advanced manufacturing technology interfaces are supportive of human operators in their responsibilities to manufacturing systems. We address this problem by analyzing supervisory controller information requirements for intervening in complex process control tasks as part of FMS operation. This analysis was conducted using a cognitive engineering research methodology, which has not previously been applied, in the domain of manufacturing. The method of GTA was applied to supervisory control of an FMS and produced detailed information requirements, which facilitated the formulation of general design guidelines for FMS interface design. The guidelines are aimed at supporting human operator process strategy development and decision making. © 2000 John Wiley & Sons, Inc.     

基于条件随机域的上下文人类动作识别

提出一种新的基于条件随机域和隐马尔可夫模型（HMM）的人类动作识别方法——HMCRF。目前已有的动作识别方法均使用隐马尔可夫模型及其变型,这些模型一个最突出的不足就是要求观察值相互独立。条件模型很容易表示上下文相关性,且可使用动态规划做到有效且精确的推论,它的参数可以通过凸函数优化训练得到。把条件图形模型应用于动作识别之上,并通过大量的实验表明,所提出的方法在识别正确率方面明显优于一般线性结构的CRF和HMM。     

Integration of task networks and cognitive user models using coloured Petri nets and its application to job design for safety and productivity

Changes of task demands due to unforeseen events and technological changes can cause variations in job design such as modifications to job procedures and task allocation. Failure to adapt to job design variations can lead to human errors that may have severe consequences for system safety. Existing techniques for task modelling cannot adequately model how task networks can be adapted to changing work conditions and task demands. Therefore, there is a need to integrate task networks with cognitive user models that indicate how operators process information, make decisions, or cope with suspended tasks and errors. The work described here presents a tool for integrating task and cognitive models using coloured Petri nets. The cognitive user model comprises two modules of attention management (selective and divided attention), a module of memory management of suspended tasks and a module of work organization. Performance Shaping Factors (e.g., workload, fatigue and mental-tracking load) are calculated at any point in time to take into account the context of work (e.g., competing activities, errors and suspended tasks). Different types of human error can be modelled for rule-based behaviours required in proceduralized work environments. Simulation analysis and formal analysis techniques can be applied to process control tasks to verify job procedures, workload management strategies and task allocation schemes in response to technological changes and unfamiliar events.     

Hidden Markov models with arbitrary state dwell-time distributions

A hidden Markov model (HMM) with a special structure that captures the ‘semi’-property of hidden semi-Markov models (HSMMs) is considered. The proposed model allows arbitrary dwell-time distributions in the states of the Markov chain. For dwell-time distributions with finite support the HMM formulation is exact while for those that have infinite support, e.g. the Poisson, the distribution can be approximated with arbitrary accuracy. A benefit of using the HMM formulation is that it is easy to incorporate covariates, trend and seasonal variation particularly in the hidden component of the model. In addition, the formulae and methods for forecasting, state prediction, decoding and model checking that exist for ordinary HMMs are applicable to the proposed class of models. An HMM with explicitly modeled dwell-time distributions involving seasonality is used to model daily rainfall occurrence for sites in Bulgaria.     

GMV control of non-linear continuous-time systems including common delays and state-space models

A non-linear generalized minimum variance control law is proposed for the control of non-linear continuous-time multivariable systems with common delays on input and output channels. The quadratic cost index involves both error and control signal costing terms. The solution for the control law is obtained using a non-linear operator representation of the plant and a linear state-equation model for the disturbance and reference models. The reference and disturbance models are represented by linear subsystems. However, the plant model can be in a very general non-linear operator form, which could involve state-space, transfer operators or non-linear function look up tables. The structure of the system and criterion is chosen so that a simple controller structure and solution is obtained. The controller obtained is simple to implement, particularly in one form, which might be considered to be a state-space version of a non-linear Smith predictor. The results are related to those for discrete-time systems but the presence of the transport delay terms complicates the solution rather more in the continuous-time case.     

Detecting objects of variable shape structure with hidden state shape models

This paper proposes a method for detecting object classes that exhibit variable shape structure in heavily cluttered images. The term "variable shape structure" is used to characterize object classes in which some shape parts can be repeated an arbitrary number of times, some parts can be optional, and some parts can have several alternative appearances. Hidden State Shape Models (HSSMs), a generalization of Hidden Markov Models (HMMs), are introduced to model object classes of variable shape structure using a probabilistic framework. A polynomial inference algorithm automatically determines object location, orientation, scale and structure by finding the globally optimal registration of model states with the image features, even in the presence of clutter. Experiments with real images demonstrate that the proposed method can localize objects of variable shape structure with high accuracy. For the task of hand shape localization and structure identification, the proposed method is significantly more accurate than previously proposed methods based on chamfer-distance matching. Furthermore, by integrating simple temporal constraints, the proposed method gains speed-ups of more than an order of magnitude, and produces highly accurate results in experiments on non-rigid hand motion tracking.     

Bio-informatic activity modeling for human-artifacts symbiosis under resource boundedness

We investigate a biologically inspired design of an interface agent that is embedded inside human-artifact interactions rather than as an external observer, and has to work as an intelligent associate for a human user/operator in a time-critical situation like in an emergency. First, recent paradigmatic shifts of artifact design principles are discussed from an interdisciplinary viewpoint. Then, after the idea of Clancey’s activity modeling, we discuss the design principles of a situated interface agent. That is, different from the conventional supervisory agent’s task of seeking to optimize an isolated control task, such an agent has to be able to maintain its identity as an organism living within multiple contexts and looking inwards to consider the the nature of memory and perception, and looking outwards to consider the nature of social action with a human operator. Initially, our prior work using such a design principle is presented, and then decision-theoretic formulations of an interface agent’s activities are provided. This work was presented, in part, at the Third International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–21, 1998