System for monitoring the wellness state of people in domestic environments employing emoticon-based HCI

Wellness state is affected by the habitability state of the domestic environment. Monitoring it can help to discover the causes of a low wellness levels aiding people in the improvement of their quality of life. In this paper, we propose a system to monitor the wellness state of people utilizing Likert’s scale to determine the state of the user through an emoticon-based human–computer interaction. The system is intended for domestic environments and measures the habitability conditions of the dwelling (such as temperature, humidity, luminosity and noise) employing sensors. An algorithm is designed in order to establish how to measure those conditions and to calculate the statistics that allows tracking their progress. The obtained information is presented to the user to compare his/her wellness state with the habitability conditions. Measures in a real domestic environment were performed in order to determine the configuration of our system. The energy efficiency of the algorithm provides an improvement between 99.36 and 99.62% in the energy consumption depending on the selected parameters.

     

THE EMPATHIC COMPANION: A CHARACTER-BASED INTERFACE THAT ADDRESSES USERS' AFFECTIVE STATES

ABSTRACT

In this paper, we report on our efforts in developing affective character-based interfaces, i.e., interfaces that recognize and measure affective information of the user and address user affect by employing embodied characters. In particular, we describe the Empathic Companion, an animated interface agent that accompanies the user in the setting of a virtual job interview. This interface application takes physiological data (skin conductance and electromyography) of a user in realtime, interprets them as emotions, and addresses the user's affective states in the form of empathic feedback. The Empathic Companion is conceived as an educational agent that supports job seekers preparing for a job interview.

We also present results from an exploratory study that aims to evaluate the impact of the Empathic Companion by measuring users' skin conductance and heart rate. While an overall positive effect of the Empathic Companion could not be shown, the outcome of the experiment suggests that empathic feedback has a positive effect on the interviewee's stress level while hearing the interviewer question.     

情感建模与情感识别

情感计算是关于、产生于和影响于情感方面的计算,其目的是赋予计算机识别、理解、表达和适应人情感的能力。情感计算通过各种传感器获取由人的情感所引起的表情及生理变化信号,利用“情感模型”对这些信号进行识别,从而理解人的情感并做出适当的响应。该文主要讨论了Picard教授在情感计算中情感识别部分的研究成果,着重分析了面部表情、语音、生理信号的情感模型与情感识别,这是情感计算研究的一个关键问题之一,也是建立和谐人机环境的基础之一。     

Dynamic Difficulty Adjustment in Computer Games Through Real-Time Anxiety-Based Affective Feedback

A number of studies in recent years have investigated the dynamic difficulty adjustment (DDA) mechanism in computer games to automatically tailor gaming experience to individual player's characteristics. Although most of these existing works focus on game adaptation based on player's performance, affective state experienced by the players could play a key role in gaming experience and may provide a useful indicator for a DDA mechanism. In this article, an affect-based DDA was designed and implemented for computer games. In this DDA mechanism, a player's physiological signals were analyzed to infer his or her probable anxiety level, which was chosen as the target affective state, and the game difficulty level was automatically adjusted in real time as a function of the player's affective state. Peripheral physiological signals were measured through wearable biofeedback sensors and several physiological indices were explored to determine their correlations with anxiety. An experimental study was conducted to evaluate the effects of the affect-based DDA on game play by comparing it with a performance-based DDA. This is the first time, that is known, that the impact of a real-time affect-based DDA has been demonstrated experimentally.     

A sensor-aided self coaching model for uncocking improvement in golf swing

This paper describes an autonomous kinematic analysis platform for wrist angle measurement that is capable of evaluating a user’s uncocking motion in his or her golf swing and providing instructional multimodal feedback to improve his or her skills. This uncocking motion, which is a characteristic movement of the wrist during the golf swing, is an important factor in achieving accurate ball hitting and long driving distances, but is difficult to measure. In order to efficiently compute the wrist angle for uncocking evaluation, we present a sensor-based intelligent Inertial Measurement Unit (IMU) agent that collects three-dimensional orientation data during the golf swing from two IMU sensors placed on the forearm and on the golf club. It accurately analyzes changes in wrist angle to detect uncocking throughout the sequence of golf swing motions. In this paper, we first introduce the design considerations based on the concept of the uncocking motion and explain the system architecture with the sensors used for quantitative measurement and qualitative feedback generation. Then, we illustrate the detailed algorithms for wrist angle computation, golf swing motion segmentation based on key pose detection, and uncocking evaluation. A multimodal feedback-based user interface for our system is also presented. Experimental results show that the proposed system has the ability to accurately calculate the wrist angle in real time and also that it can be applied to a practical self-coaching system to improve the uncocking motion.     

Multi-level assessment model for wellness service based on human mental stress level

In this paper, we measure human physiological changes from different body parts to quantify human mental stress level by using multimodal bio-sensors. By integrating these physiological responses, we generate bio-index and rule for the prediction of mental status, such as tension, normal, and relax. We also develop an inspection service middleware for analyzing health parameters such as electroencephalography (EEG), electrocardiography (ECG), oxygen saturation (SpO2), blood pressure (BP), and respiration rate (RR). In this service middleware, we use the multi-level assessment model for mental stress level that consists of three steps as follows; classification, reasoning, and decision making. The classification of datasets from bio-sensors is enabled by fuzzy logic and SVM algorithm. The reasoning uses the decision-tree model and random forest algorithm to classify the mental stress level from the health parameters. Finally, we propose a prediction model to make a decision for the wellness contents by using Expectation Maximization (EM).     

Tune in to your emotions: a robust personalized affective music player

The emotional power of music is exploited in a personalized affective music player (AMP) that selects music for mood enhancement. A biosignal approach is used to measure listeners’ personal emotional reactions to their own music as input for affective user models. Regression and kernel density estimation are applied to model the physiological changes the music elicits. Using these models, personalized music selections based on an affective goal state can be made. The AMP was validated in real-world trials over the course of several weeks. Results show that our models can cope with noisy situations and handle large inter-individual differences in the music domain. The AMP augments music listening where its techniques enable automated affect guidance. Our approach provides valuable insights for affective computing and user modeling, for which the AMP is a suitable carrier application.     

A survey of methods for data fusion and system adaptation using autonomic nervous system responses in physiological computing

Physiological computing represents a mode of human–computer interaction where the computer monitors, analyzes and responds to the user’s psychophysiological activity in real-time. Within the field, autonomic nervous system responses have been studied extensively since they can be measured quickly and unobtrusively. However, despite a vast body of literature available on the subject, there is still no universally accepted set of rules that would translate physiological data to psychological states. This paper surveys the work performed on data fusion and system adaptation using autonomic nervous system responses in psychophysiology and physiological computing during the last ten years. First, five prerequisites for data fusion are examined: psychological model selection, training set preparation, feature extraction, normalization and dimension reduction. Then, different methods for either classification or estimation of psychological states from the extracted features are presented and compared. Finally, implementations of system adaptation are reviewed: changing the system that the user is interacting with in response to cognitive or affective information inferred from autonomic nervous system responses. The paper is aimed primarily at psychologists and computer scientists who have already recorded autonomic nervous system responses and now need to create algorithms to determine the subject’s psychological state.     

Toward a decision-theoretic framework for affect recognition and user assistance

There is an increasing interest in developing intelligent human–computer interaction systems that can fulfill two functions—recognizing user affective states and providing the user with timely and appropriate assistance. In this paper, we present a general unified decision-theoretic framework based on influence diagrams for simultaneously modeling user affect recognition and assistance. Affective state recognition is achieved through active probabilistic inference from the available multi modality sensory data. User assistance is automatically accomplished through a decision-making process that balances the benefits of keeping the user in productive affective states and the costs of performing user assistance. We discuss three theoretical issues within the framework, namely, user affect recognition, active sensory action selection, and user assistance. Validation of the proposed framework via a simulation study demonstrates its capability in efficient user affect recognition as well as timely and appropriate user assistance. Besides the theoretical contributions, we build a non-invasive real-time prototype system to recognize different user affective states (stress and fatigue) from four-modality user measurements, namely physical appearance features, physiological measures, user performance, and behavioral data. The affect recognition component of the prototype system is subsequently validated through a real-world study involving human subjects.     

Stress Detection in Human–Computer Interaction: Fusion of Pupil Dilation and Facial Temperature Features

In order to differentiate the affective state of a computer user as it changes from relaxation to stress, features derived from pupil dilation and periorbital temperature can be utilized. Absolute signal values and measurements computed from these can be fused to increase the accuracy of affective classification. In this study, entropy in a sliding window was used to accommodate the time differences in the physiological rise and fall profiles of pupil and thermal data. Two methods, decision tree and Adaboost with Random Forest (ABRF), were used for classification tests. Detection accuracy of stressful states varied between 65% and 83.8%. Best results can be reported as 83.9% for sensitivity and 83.8% for specificity. ABRF classifier outperformed the decision tree model. This study emphasizes the importance of data fusion, particularly when physiological signals differ with respect to their rise and fall windows across time. Use of entropy within a predefined time window provides a useful set of features to combine with actual measurements. Furthermore, the collection of pupil and thermal data is feasible because surface sensors are eliminated.     

Modeling Multimodal Expression of User’s Affective Subjective Experience

With the growing importance of information technology in our everyday life, new types of applications are appearing that require the understanding of information in a broad sense. Information that includes affective and subjective content plays a major role not only in an individual’s cognitive processes but also in an individual’s interaction with others. We identify three key points to be considered when developing systems that capture affective information: embodiment (experiencing physical reality), dynamics (mapping experience and emotional state with its label) and adaptive interaction (conveying emotive response, responding to a recognized emotional state). We present two computational systems that implement those principles: MOUE (Model Of User Emotions) is an emotion recognition system that recognizes the user’s emotion from his/her facial expressions, and from it, adaptively builds semantic definitions of emotion concepts using the user’s feedback; MIKE (Multimedia Interactive Environment for Kansei communication) is an interactive adaptive system that, along with the user, co-evolves a language for communicating over subjective impressions.     

人机交互中的智能体情感计算研究

机器的情感是通过融入具有情感能力的智能体实现的,虽然目前在人机交互领域已经有大量研究成果,但有关智能体情感计算方面的研究尚处起步阶段,深入开展这项研究对推动人机交互领域的发展具有重要的科学和应用价值。本文通过检索Scopus数据库选择有代表性的文献,重点关注情感在智能体和用户之间的双向流动,分别从智能体对用户的情绪感知和对用户情绪调节的角度开展分析总结。首先梳理了用户情绪的识别方法,即通过用户的表情、语音、姿态、生理信号和文本信息等多通道信息分析用户的情绪状态,归纳了情绪识别中的一些机器学习方法。其次从用户体验角度分析具有情绪表现力的智能体对用户的影响,总结了智能体的情绪生成和表现技术,指出智能体除了通过表情之外,还可以通过注视、姿态、头部运动和手势等非言语动作来表现情绪。并且梳理了典型的智能体情绪架构,举例说明了强化学习在智能体情绪设计中的作用。同时为了验证模型的准确性,比较了已有的情感评估手段和评价指标。最后指出智能体情感计算急需解决的问题。通过对现有研究的总结,智能体情感计算研究是一个很有前景的研究方向,希望本文能够为深入开展相关研究提供借鉴。     

Some insights into the impact of affective information when delivering feedback to students

ABSTRACT

The relation between affect-driven feedback and engagement on a given task has been largely investigated. This relation can be used to make personalised instructional decisions and/or modify the affect content within the feedback. However, although it is generally assumed that providing encouraging feedback to students should help them adopt a state of flow, there are instances where those messages might result counterproductive. In this paper, we present a case study with 48 secondary school students using an Intelligent Tutoring System for arithmetical word problem solving. This system, which makes some common assumptions on how to relate affective state with performance, takes into account subjective (user's affective state) and objective information (previous problem performance) to decide the upcoming difficulty levels and the type of affective feedback to be delivered. Surprisingly, results revealed that feedback was more effective when no emotional content was included, and lead to the conclusion that purely instructional and concise help messages are more important than the emotional reinforcement contained therein. This finding shows that this is still an open issue. Different settings present different constraints generating related compounding factors that affect obtained results. This research confirms that new approaches are required to determine when, how and where affect-driven feedback is needed. Affect-driven feedback, engagement and their mutual relation have been largely investigated. Student's interactions combined with their emotional state can be used to make personalised instructional decisions and/or modify the affect content within the feedback, aiming to entice engagement on the task. However, although it is generally assumed that providing encouraging feedback to the students should help them adopt a state of flow, there are instances where those encouraging messages might result counterproductive. In this paper, we analyze these issues in terms of a case study with 48 secondary school students using an Intelligent Tutoring System for arithmetical word problem solving. This system, which makes some common assumptions on how to relate affective state with performance, takes into account subjective (user's affective state) and objective (previous problem performance) information to decide the difficulty level of the next exercise and the type of affective feedback to be delivered. Surprisingly, findings revealed that feedback was more effective when no emotional content was included in the messages, and lead to the conclusion that purely instructional and concise help messages are more important than the emotional reinforcement contained therein. This finding, which coincides with related work, shows that this is still an open issue. Different settings present different constraints and there are related compounding factors that affect obtained results, such as the message's contents and their target, how to measure the effect of the message on engagement through affective variables considering other issues involved, and to what extent engagement can be manipulated solely in terms of affective feedback. The contribution here is that this research confirms that new approaches are needed to determine when, how and where affect-driven feedback is needed. In particular, based on our previous experience in developing educational recommender systems, we suggest the combination of user-centred design methodologies with data mining methods to yield a more effective feedback.     

Cloud-oriented emotion feedback-based Exergames framework

Exergames platform can be more appealing to the users if they can interact with emotion. Therefore, in this paper, we propose an automatic emotion recognition system from speech to be embedded in the Exergames platform. While playing and doing exercise, the user expresses his or her feeling by uttering some phrases. The speech is recorded by an omnidirectional mic and transmitted to an emotion recognition server in a cloud environment, where the emotion (e.g. happy, sad or neutral) is recognized. For the recognition, we use MPEG-7 low-level audio features and a Gaussian mixture model based classifier. A tactile vibration is generated based on the emotion and feedback to the user for a real feeling. The user can thus have an instantaneous vibrational feeling based on his or her satisfaction. The recognized emotion can also be used as the user’s satisfaction of the framework on the fly without the need of a survey session. The experimental study and performance comparison show that the proposed framework has positive effects on the perception of physical activities.     

Teaching a pet-robot to understand user feedback through interactive virtual training tasks

In this paper, we present a human-robot teaching framework that uses “virtual” games as a means for adapting a robot to its user through natural interaction in a controlled environment. We present an experimental study in which participants instruct an AIBO pet robot while playing different games together on a computer generated playfield. By playing the games and receiving instruction and feedback from its user, the robot learns to understand the user’s typical way of giving multimodal positive and negative feedback. The games are designed in such a way that the robot can reliably predict positive or negative feedback based on the game state and explore its user’s reward behavior by making good or bad moves. We implemented a two-staged learning method combining Hidden Markov Models and a mathematical model of classical conditioning to learn how to discriminate between positive and negative feedback. The system combines multimodal speech and touch input for reliable recognition. After finishing the training, the system was able to recognize positive and negative reward with an average accuracy of 90.33%.     

A framework for the assessment of synthetic personalities according to user perception

Endowing artificial conversational agents with personality is a very promising way to obtain more believable user interactions with robots and computers. However, although many authors have studied how to create an agent׳s personality and how it affects performance and user satisfaction, less attention has been paid to assess whether the designed agent׳s personality corresponds to the users׳ perception, whether it is easily recognizable, and what is the effect that the user׳s own personality has in the discrimination of the agents׳ personality. In this paper we present an assessment framework to address these issues in an integrated way, which in our opinion offers enough flexibility to consider the diversity of application domains and evaluation approaches that can be found in the literature. The framework is based on numerical measures, which facilitate the interpretation of results and makes it possible to compare and rank different agents with respect to the user׳s perception of the rendered personality. In addition, we have developed a tool that implements the framework, which may be very useful for researchers in order to easily evaluate different agent personalities.     

Flow,social interaction anxiety and salivary cortisol responses in serious games: A quasi-experimental study

Serious games are supposed to instigate engagement and, in turn, improve learning. High engagement is frequently connected with a positive affective state and a high flow state. However, the alleged link between a learner’s affective state, his/her flow state and learning outcomes has not been investigated in detail in the context of serious games. Even less information is available on how serious games may influence markers of physiological arousal. To fill this gap, participants of this exploratory study (N = 171) played one of the six different serious game-based treatments, while we measured their affect, flow, cortisol secretion and learning achievement. The treatments were supposed to generate different levels of engagement and cortisol responses, because some of them were designed for a single user, while others were team-based, featuring so-called social-evaluative threat (ST) components. Our results revealed that flow was positively related to positive affect and negatively to negative affect. While flow and positive affect were related to learning gains, almost no relationship between either of these three variables and cortisol levels was found. Negative affect and cortisol were elevated in social interaction anxious males in team-based conditions. This study contributes to the limited body of research on the relationship between engagement and learning in serious games. We provide new perspectives on the relationships between flow, positive/negative affect and cortisol. Our findings highlight the fact that team-based serious games with ST components may have adverse effects on learners, particularly males, with high social interaction anxiety.     

一种基于 PageRank 的微博用户影响度评估算法

微博已成为主流的在线社交网络平台,用户的影响力已成为衡量用户价值的一个重要指标。本文基于PageRank算法,通过分析用户之间的兴趣相似度、相对发帖活跃度、相互反馈互动程度来计算一个用户对其所关注的用户的关注程度,提出一个能够评估用户在微博上实际影响度WeiboRank算法。实验数据分析表明,该算法得到的用户影响度值能较客观地反映用户在其所处的虚拟社交网络中的实际影响度。     

Extracting relevance and affect information from physiological text annotation

We present physiological text annotation, which refers to the practice of associating physiological responses to text content in order to infer characteristics of the user information needs and affective responses. Text annotation is a laborious task, and implicit feedback has been studied as a way to collect annotations without requiring any explicit action from the user. Previous work has explored behavioral signals, such as clicks or dwell time to automatically infer annotations, and physiological signals have mostly been explored for image or video content. We report on two experiments in which physiological text annotation is studied first to (1) indicate perceived relevance and then to (2) indicate affective responses of the users. The first experiment tackles the user’s perception of relevance of an information item, which is fundamental towards revealing the user’s information needs. The second experiment is then aimed at revealing the user’s affective responses towards a -relevant- text document. Results show that physiological user signals are associated with relevance and affect. In particular, electrodermal activity was found to be different when users read relevant content than when they read irrelevant content and was found to be lower when reading texts with negative emotional content than when reading texts with neutral content. Together, the experiments show that physiological text annotation can provide valuable implicit inputs for personalized systems. We discuss how our findings help design personalized systems that can annotate digital content using human physiology without the need for any explicit user interaction.     

A metagenetic algorithm for information filtering and collection from the World Wide Web

This paper describes the implementation of evolutionary techniques for information filtering and collection from the World Wide Web. We consider the problem of building intelligent agents to facilitate a person's search for information on the Web. An intelligent agent has been developed that uses a metagenetic algorithm in order to collect and recommend Web pages that will be interesting to the user. The user's feedback on the agent's recommendations drives the learning process to adapt the user's profile with his/her interests. The software agent utilizes the metagenetic algorithm to explore the search space of user interests. Experimental results are presented in order to demonstrate the suitability of the metagenetic algorithm's approach on the Web.