A Survey and Tutorial of EEG-Based Brain Monitoring for Driver State Analysis

The driver’s cognitive and physiological states affect his/her ability to control the vehicle. Thus, these driver states are essential to the safety of automobiles. The design of advanced driver assistance systems (ADAS) or autonomous vehicles will depend on their ability to interact effectively with the driver. A deeper understanding of the driver state is, therefore, paramount. Electroencephalography (EEG) is proven to be one of the most effective methods for driver state monitoring and human error detection. This paper discusses EEG-based driver state detection systems and their corresponding analysis algorithms over the last three decades. First, the commonly used EEG system setup for driver state studies is introduced. Then, the EEG signal preprocessing, feature extraction, and classification algorithms for driver state detection are reviewed. Finally, EEG-based driver state monitoring research is reviewed in-depth, and its future development is discussed. It is concluded that the current EEG-based driver state monitoring algorithms are promising for safety applications. However, many improvements are still required in EEG artifact reduction, real-time processing, and between-subject classification accuracy.      

Toward a decision-theoretic framework for affect recognition and user assistance

There is an increasing interest in developing intelligent human–computer interaction systems that can fulfill two functions—recognizing user affective states and providing the user with timely and appropriate assistance. In this paper, we present a general unified decision-theoretic framework based on influence diagrams for simultaneously modeling user affect recognition and assistance. Affective state recognition is achieved through active probabilistic inference from the available multi modality sensory data. User assistance is automatically accomplished through a decision-making process that balances the benefits of keeping the user in productive affective states and the costs of performing user assistance. We discuss three theoretical issues within the framework, namely, user affect recognition, active sensory action selection, and user assistance. Validation of the proposed framework via a simulation study demonstrates its capability in efficient user affect recognition as well as timely and appropriate user assistance. Besides the theoretical contributions, we build a non-invasive real-time prototype system to recognize different user affective states (stress and fatigue) from four-modality user measurements, namely physical appearance features, physiological measures, user performance, and behavioral data. The affect recognition component of the prototype system is subsequently validated through a real-world study involving human subjects.     

Detecting emotional state of a child in a conversational computer game

The automatic recognition of user’s communicative style within a spoken dialog system framework, including the affective aspects, has received increased attention in the past few years. For dialog systems, it is important to know not only what was said but also how something was communicated, so that the system can engage the user in a richer and more natural interaction. This paper addresses the problem of automatically detecting “frustration”, “politeness”, and “neutral” attitudes from a child’s speech communication cues, elicited in spontaneous dialog interactions with computer characters. Several information sources such as acoustic, lexical, and contextual features, as well as, their combinations are used for this purpose. The study is based on a Wizard-of-Oz dialog corpus of 103 children, 7–14 years of age, playing a voice activated computer game. Three-way classification experiments, as well as, pairwise classification between polite vs. others and frustrated vs. others were performed. Experimental results show that lexical information has more discriminative power than acoustic and contextual cues for detection of politeness, whereas context and acoustic features perform best for frustration detection. Furthermore, the fusion of acoustic, lexical and contextual information provided significantly better classification results. Results also showed that classification performance varies with age and gender. Specifically, for the “politeness” detection task, higher classification accuracy was achieved for females and 10–11 years-olds, compared to males and other age groups, respectively.     

基于图像相似度与深度学习的变电站刀闸状态识别算法的对比研究

为解决以往变电站中基于各类传感器的刀闸状态检测方式成本高、稳定性差的问题，本文探索了两类基于图像识别的刀闸状态检测算法，相较于传统的基于图像相似度的刀闸状态识别算法，基于深度学习的目标检测算法对刀闸状态识别准确率更高，能够有效对变电站内刀闸状态进行检测。本文在对483张包含各类刀闸状态的图像进行标注后，使用Yolov5的预训练模型进行训练，训练后的模型在包含80张各类刀闸状态的测试集上进行测试，结果表明综合准确率为89.31%，综合召回率为98.32%。本文所提出的基于深度学习的刀闸识别算法能够对变电站刀闸状态进行有效识别，且识别准确率高、部署较为简单，对保障变电站安全稳定运行有着重要作用。     

基于VR设备中IMU的无手交互方法研究

随着头戴式显示设备的发展,在基于虚拟现实(VR,virtual reality)的教育培训中,存在用户与设备间进行交互的场景;针对用户与VR视频中对象的模拟靠近与躲避问题,提出了姿态感知与步态识别相结合的方法;通过姿态感知算法解算用户头部姿态,通过步态识别算法识别出人体的静止与行走状态,进而在行走状态时,将计算所得航向角和固定步长代入三角函数公式进行位置更新,在静止状态时,保持位置不变;实验证明,提出的姿态感知算法可以有效的计算出使用者头部的姿态,与商用惯性测量单元提供的姿态角相比具有1.1×10-2的平均姿态偏差;提出的步态识别算法可以有效地识别出人体的静止与行走状态;所提出的两者结合的交互方法,可以有效地实现虚拟的靠近与躲避.     

时间自动机可达性分析中的状态空间约减技术综述

时间自动机是检验实时系统建模的有效工具,其可达性分析可以检验系统是否可能达到某些特定的状态,其算法通常采用对符号状态的枚举来遍历其状态空间。因为引入了时钟变量,时间自动机的可达性分析算法会产生大量的中间状态,需要巨大的存储空间,往往超出了计算机能力的极限,导致分析和检验不能完成。这就是所谓的“状态空间爆炸”。研究人员设计了很多种优化技术来约减可迭性分析所需的存储空间,以解决或者缓解这个问题。本文首先介绍了时间自动机及其可达性分析的基本概念,然后分类讨论了现有的空间约减优化技术并对此做出总结,最后提出了一些未来的研究方向。     

Approaching the accuracy–cost conflict in embedded classification system design

Smart embedded systems often run sophisticated pattern recognition algorithms and are found in many areas like automotive, sports and medicine. The developer of such a system is often confronted with the accuracy–cost conflict as the resulting system should be as accurate as possible while being able to run on resource constraint hardware. This article introduces a method to support the solution of this design conflict with accuracy–cost reports. These reports compare classification systems regarding their classification rate (accuracy) and the mathematical operations and parameters of the working phase (cost). Our method is used to deduce the specific cost of various popular pattern recognition algorithms and to derive the overall cost of a classification system. We also show how our analysis can be used to estimate the computational cost for specific hardware architectures. A software toolbox to create accuracy–cost reports was implemented to facilitate the automatic classification system comparison with the presented methodology. The software is available for download and as supplementary material. We performed different experiments on synthetic and real-world data to underline the value of this analysis. Accurate and computationally cheap classification systems were easily identified. We were even able to find a better implementation candidate in an existing embedded classification problem. This work is the first step towards a comprehensive support tool for the design of embedded classification systems.     

基于公路监控视频的车辆检测和分类

在学习了已有的检测与分类算法以后,设计了一种将改进的高斯混合模型（GMM）与分类网络（GoogLeNet）融合的方案用于车辆的检测和分类.针对高斯混合模型存在模型初始化速度慢和计算复杂的问题,改进了初始化模型的算法提升初始化效率.运用五帧差法做车辆初提取,在提取到的车辆区域上运用高斯混合模型获得车辆图片,把五帧差法和高斯混合模型结合起来减小了建模的区域,提升了检测速度,提高了系统实时性.最后使用GoogLeNet对车辆分类.实验证明相较于现有的车辆检测分类方法,本文所提方法在检测速度和分类准确性上都有很大提升,满足了现实场景下对监控视频的车辆检测和分类的实时性要求.     

Motion estimation based on optical flow and an artificial neural network (ANN)

Motion estimation provides an attractive alternative to cumbersome interface devices for human-computer interaction (HCI). Worthy of note is that the visual recognition of hand gestures can help to achieve an easy and natural interaction between human and computer. The interfaces of HCI and other virtual reality systems depend on accurate, real-time hand and fingertip tracking for an association between real objects and the corresponding digital information. However, they are expensive, and complicated operations can make them troublesome. We are developing a real-time, view-based gesture recognition system. The optical flow is estimated and segmented into motion fragments. Using an artificial neural network (ANN), the system can compute and estimate the motions of gestures. Compared with traditional approaches, theoretical and experimental results show that this method has simpler hardware and algorithms, but is more effective. It can be used in moving object recognition systems for understanding human body languages.     

Integrated Person Tracking Using Stereo,Color, and Pattern Detection

We present an approach to real-time person tracking in crowded and/or unknown environments using integration of multiple visual modalities. We combine stereo, color, and face detection modules into a single robust system, and show an initial application in an interactive, face-responsive display. Dense, real-time stereo processing is used to isolate users from other objects and people in the background. Skin-hue classification identifies and tracks likely body parts within the silhouette of a user. Face pattern detection discriminates and localizes the face within the identified body parts. Faces and bodies of users are tracked over several temporal scales: short-term (user stays within the field of view), medium-term (user exits/reenters within minutes), and long term (user returns after hours or days). Short-term tracking is performed using simple region position and size correspondences, while medium and long-term tracking are based on statistics of user appearance. We discuss the failure modes of each individual module, describe our integration method, and report results with the complete system in trials with thousands of users.     

A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing

Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subject’s psychological state.     

数据驱动的智能车辆目标检测能力测试评价方法

本文针对智能车辆目标检测能力测评存在的指标体系不完整、量化程度和测评实时性低等问题,聚焦智能车辆目标检测能力中的目标分类和目标识别,在这两个测评项目上提出了一套量化的评价指标体系,并用TOPSIS方法进行综合的评价.然后在此指标体系的基础上搭建数据驱动的智能车辆目标检测能力测评平台,平台可满足对智能车辆目标检测能力测评的实时性要求.最后采用了若干组车辆检测算法对指标体系进行验证.     

基于实时视觉分析算法的智能图像传感器系统设计

设计了一种智能交通图像传感器系统以实现对监控场景的快速移动侦测和对象识别。该系统具有有线以太网和无线GPRS双重网络接入功能,硬件由基于Au1200嵌入式处理器的网络接口端和基于BlackFin 533 DSP处理器的图像分析端组成。软件系统包括运行于Au1200处理器上的基于嵌入式Linux架构的网络收发软件和运行于BlackFin 533 DSP上的视觉分析算法。本系统引入了基于区域分割的背景模型和基于特征的对象识别算法。实验结果表明该系统能够实时高效地进行自动移动检测和对象分类识别。     

Rapid Prototyping of Activity Recognition Applications

The CRN Toolbox enables fast implementation of activity and context recognition systems, featuring mechanisms for distributed processing and support for mobile and wearable devices. CRN Toolbox is a tool set specifically optimized for implementing multimodal, distributed activity and context recognition systems running on Posix operating systems. Like conventional rapid- prototyping tools, the CRN Toolbox contains a collection of ready-to-use algorithms (signal processing, pattern classification, and so on). Unlike classic event detection in homogeneous sensor networks-for example, DSWare (Data Service Middleware)-it supports complex activity detection from heterogeneous sensors. Its implementation is particularly optimized for mobile devices. This includes the ability to execute algorithms, whether in floating-point or fixed-point arithmetic, without recoding. Moreover, with its mature functionality, the CRN Toolbox isn't likely to suffer from limited user acceptance as the Context toolkit framework did.     

基于无线信号的人类行为检测和识别

物联网技术实现了物与物、人与物的全面互联,其中信息传感设备与人的交互需要对人体行为活动进行感知。目前广泛使用的有基于视觉和利用穿戴式传感器的识别方法,但这些方法在很多场景下应用有所限制。文章提出一种基于无线信号识别人类行为的方法,通过对通信中传输数据包状态的统计和分析,能够利用少量通信节点达到感知非携带设备的目标在室内检测区域行为活动的目的。对于不同的行为活动特征,采用序列最小优化算法、 K-最近邻算法等不同算法进行分类研究。相对于传统基于无线信号接收信号强度指标的免携带设备行为识别方法,文章提出的方法对不同运动速度等级的识别精度平均提高了 25.1%。     

Classification of primitive shapes using brain–computer interfaces

Brain–computer interfaces (BCIs) are recent developments in alternative technologies of user interaction. The purpose of this paper is to explore the potential of BCIs as user interfaces for CAD systems. The paper describes experiments and algorithms that use the BCI to distinguish between primitive shapes that are imagined by a user. Users wear an electroencephalogram (EEG) headset and imagine the shape of a cube, sphere, cylinder, pyramid or a cone. The EEG headset collects brain activity from 14 locations on the scalp. The data is analyzed with independent component analysis (ICA) and the Hilbert–Huang Transform (HHT). The features of interest are the marginal spectra of different frequency bands (theta, alpha, beta and gamma bands) calculated from the Hilbert spectrum of each independent component. The Mann–Whitney U-test is then applied to rank the EEG electrode channels by relevance in five pair-wise classifications. The features from the highest ranking independent components form the final feature vector which is then used to train a linear discriminant classifier. Results show that this classifier can discriminate between the five basic primitive objects with an average accuracy of about 44.6% (compared to naïve classification rate of 20%) over ten subjects (accuracy range of 36%–54%). The accuracy classification changes to 39.9% when both visual and verbal cues are used. The repeatability of the feature extraction and classification was checked by conducting the experiment on 10 different days with the same participants. This shows that the BCI holds promise in creating geometric shapes in CAD systems and could be used as a novel means of user interaction.     

Vision-based short range interaction between a personal service robot and a user

Interaction between a personal service robot and a human user is contingent on being aware of the posture and facial expression of users in the home environment. In this work, we propose algorithms to robustly and efficiently track the head, facial gestures, and the upper body movements of a user. The face processing module consists of 3D head pose estimation, modeling nonrigid facial deformations, and expression recognition. Thus, it can detect and track the face, and classify expressions under various poses, which is the key for human–robot interaction. For body pose tracking, we develop an efficient algorithm based on bottom-up techniques to search in a tree-structured 2D articulated body model, and identify multiple pose candidates to represent the state of current body configuration. We validate these face and body modules in varying experiments with different datasets, and the experimental results are reported. The implementation of both modules can run in real-time, which meets the requirement for real-world human–robot interaction task. These two modules have been ported onto a real robot platform by the Electronics and Telecommunications Research Institute.     

Bayesian filter based behavior recognition in workflows allowing for user feedback

In this paper, we propose a novel online framework for behavior understanding, in visual workflows, capable of achieving high recognition rates in real-time. To effect online recognition, we propose a methodology that employs a Bayesian filter supported by hidden Markov models. We also introduce a novel re-adjustment framework of behavior recognition and classification by incorporating the user’s feedback into the learning process through two proposed schemes: a plain non-linear one and a more sophisticated recursive one. The proposed approach aims at dynamically correcting erroneous classification results to enhance the behavior modeling and therefore the overall classification rates. The performance is thoroughly evaluated under real-life complex visual behavior understanding scenarios in an industrial plant. The obtained results are compared and discussed.     

SAGE: Semantic Annotation of Georeferenced Environments

The emergence of portable 3D mapping systems are revolutionizing the way we generate digital 3D models of environments. These systems are human-centric and require the user to hold or carry the device while continuously walking and mapping an environment. In this paper, we adapt this unique coexistence of man and machines to propose SAGE (Semantic Annotation of Georeferenced Environments). SAGE consists of a portable 3D mobile mapping system and a smartphone that enables the user to assign semantic content to georeferenced 3D point clouds while scanning a scene. The proposed system contains several components including touchless speech acquisition, background noise adaptation, real time audio and vibrotactile feedback, automatic speech recognition, distributed clock synchronization, 3D annotation localization, user interaction, and interactive visualization. The most crucial advantage of SAGE technology is that it can be used to infer dynamic activities within an environment. Such activities are difficult to be identified with existing post-processing semantic annotation techniques. The capability of SAGE leads to many promising applications such as intelligent scene classification, place recognition and navigational aid tasks. We conduct several experiments to demonstrate the effectiveness of the proposed system.     

基于静电信号的人体动作识别

提出一种通过检测人体行为动作产生的静电信号进行人体动作识别的方法.在分析人体荷电特性的基础上,设计静电信号检测系统采集被测人员的5种典型动作（行走、踏步、坐下、拿取物品、挥手）的静电感应信号.对采集的5种动作的静电信号进行特征参量提取和显著性差异分析,优化用于分类的特征参数.基于Weka平台使用3种分类算法（支持向量机、决策树C4.5和随机森林）分别对采集到的250组样本数据通过10折交叉验证进行了分类识别,结果显示随机森林算法的识别效果最好,正确率可达99.6%.研究表明本文提出的单人环境下基于人体静电信号的动作分类识别方法能够有效地对典型人体动作进行识别.