Younger and older users׳ recognition of virtual agent facial expressions

As technology advances, robots and virtual agents will be introduced into the home and healthcare settings to assist individuals, both young and old, with everyday living tasks. Understanding how users recognize an agent׳s social cues is therefore imperative, especially in social interactions. Facial expression, in particular, is one of the most common non-verbal cues used to display and communicate emotion in on-screen agents (Cassell et al., 2000). Age is important to consider because age-related differences in emotion recognition of human facial expression have been supported (Ruffman et al., 2008), with older adults showing a deficit for recognition of negative facial expressions. Previous work has shown that younger adults can effectively recognize facial emotions displayed by agents (Bartneck and Reichenbach, 2005, Courgeon et al., 2009, Courgeon et al., 2011, Breazeal, 2003); however, little research has compared in-depth younger and older adults’ ability to label a virtual agent׳s facial emotions, an import consideration because social agents will be required to interact with users of varying ages. If such age-related differences exist for recognition of virtual agent facial expressions, we aim to understand if those age-related differences are influenced by the intensity of the emotion, dynamic formation of emotion (i.e., a neutral expression developing into an expression of emotion through motion), or the type of virtual character differing by human-likeness. Study 1 investigated the relationship between age-related differences, the implication of dynamic formation of emotion, and the role of emotion intensity in emotion recognition of the facial expressions of a virtual agent (iCat). Study 2 examined age-related differences in recognition expressed by three types of virtual characters differing by human-likeness (non-humanoid iCat, synthetic human, and human). Study 2 also investigated the role of configural and featural processing as a possible explanation for age-related differences in emotion recognition. First, our findings show age-related differences in the recognition of emotions expressed by a virtual agent, with older adults showing lower recognition for the emotions of anger, disgust, fear, happiness, sadness, and neutral. These age-related difference might be explained by older adults having difficulty discriminating similarity in configural arrangement of facial features for certain emotions; for example, older adults often mislabeled the similar emotions of fear as surprise. Second, our results did not provide evidence for the dynamic formation improving emotion recognition; but, in general, the intensity of the emotion improved recognition. Lastly, we learned that emotion recognition, for older and younger adults, differed by character type, from best to worst: human, synthetic human, and then iCat. Our findings provide guidance for design, as well as the development of a framework of age-related differences in emotion recognition.     

Film mood induction and emotion classification using physiological signals for health and wellness promotion in older adults living alone

This paper introduces a wearable hardware/software system specifically tailored to detect seven emotions (neutral, tenderness, amusement, anger, disgust, fear, and sadness) aimed at promoting health and wellness in older adults living alone at home. The complete software and hardware architectures acquiring and processing electrodermal activity and photoplethysmography signals are introduced. The wearable emotion detection system is trained by eliciting the desired emotions on 39 older adults through a film mood induction procedure. Seventeen features are calculated on skin conductance response and heart rate variability data, grouped into five statistical, four temporal, and eight morphological features. Then, these features are used to run emotion classification considering support vector machines, decision trees, and quadratic discriminant analysis. In line with psychological findings, the results offer a global accuracy of 82% in negative emotion (anger, disgust, fear, and sadness) classification. For positive emotions (tenderness and amusement), also in conformity with previous psychological outcomes, amusement shows the highest ratio of hits (92%) but tenderness the lowest one (66%). These results demonstrate that our wearable emotion detection system can be used by ageing adults, especially for detecting negative emotions that usually damage health and wellness and lead to social isolation.     

Age differences in identifying words in synthetic speech

OBJECTIVES: We investigated whether context or different speech rates could improve older adult performance on identification of synthetically generated words. BACKGROUND: Synthetic speech systems can potentially improve the daily functioning of older adults. However, research must determine whether older adults can effectively implement current text-to-speech technologies, which few studies have examined. Older adults' sensory and cognitive declines may cause difficulties in identifying words in synthetic speech. METHODS: Ninety-six participants (young, middle-aged, and older adults) identified auditory monosyllabic words (half natural, half synthetic) presented in isolation or at the ends of sentences. Participants heard speech at either normal or slower rates. RESULTS: We found an interaction of age, context, and voice type and that slower speech rates worsened performance for all groups. Contrasts revealed that context reduced age differences, though only for natural speech. Hearing acuity was highly correlated with age and fully accounts for the interaction. CONCLUSIONS: Context improves performance for everyone in natural speech. However, whereas context improves performance for synthetic speech, it does not differentially reduce the age impairment for older adults. Slower speed generally impairs everyone's performance compared with the normal rate. APPLICATIONS: Systems using synthetic speech should avoid presenting words in isolation, and rich contextual support should be consistently adopted. Synthetic speech fidelity must be improved significantly before becoming truly useful for older adult populations.     

Linguistic-based emotion analysis and recognition for measuring consumer satisfaction: an application of affective computing

A growing body of research suggests that affective computing has many valuable applications in enterprise systems research and e-businesses. This paper explores affective computing techniques for a vital sub-area in enterprise systems—consumer satisfaction measurement. We propose a linguistic-based emotion analysis and recognition method for measuring consumer satisfaction. Using an annotated emotion corpus (Ren-CECps), we first present a general evaluation of customer satisfaction by comparing the linguistic characteristics of emotional expressions of positive and negative attitudes. The associations in four negative emotions are further investigated. After that, we build a fine-grained emotion recognition system based on machine learning algorithms for measuring customer satisfaction; it can detect and recognize multiple emotions using customers’ words or comments. The results indicate that blended emotion recognition is able to gain rich feedback data from customers, which can provide more appropriate follow-up for customer relationship management.     

How emotion is made and measured

How we design and evaluate for emotions depends crucially on what we take emotions to be. In affective computing, affect is often taken to be another kind of information—discrete units or states internal to an individual that can be transmitted in a loss-free manner from people to computational systems and back. While affective computing explicitly challenges the primacy of rationality in cognitivist accounts of human activity, at a deeper level it often relies on and reproduces the same information-processing model of cognition. Drawing on cultural, social, and interactional critiques of cognition which have arisen in human–computer interaction (HCI), as well as anthropological and historical accounts of emotion, we explore an alternative perspective on emotion as interaction: dynamic, culturally mediated, and socially constructed and experienced. We demonstrate how this model leads to new goals for affective systems—instead of sensing and transmitting emotion, systems should support human users in understanding, interpreting, and experiencing emotion in its full complexity and ambiguity. In developing from emotion as objective, externally measurable unit to emotion as experience, evaluation, too, alters focus from externally tracking the circulation of emotional information to co-interpreting emotions as they are made in interaction.     

Assessing design features of virtual keyboards for text entry

OBJECTIVE: The present research examined design of a virtual keyboard for text entry with a rotary controller, emphasizing users who differ in age and system experience. BACKGROUND: Existing research has minimally addressed usage frequency, age, and the effects of display shape and letter arrangement on movement and visual search components of text entry tasks. The present research was conducted to close these gaps. METHOD: Two experiments were completed to examine younger (18-28 years) and older (60-75 years) adults' movement and visual search capabilities using four keyboard shapes and three keyboard arrangements. In a third experiment examining combined effects on shape design, 32 younger (18-28 years) and 32 older (60-75 years) adults entered words on the two best shapes from the first experiments. RESULTS: For the movement task, movement time was lowest for shapes with higher shape-controller compatibility. For the visual search task, search time and accuracy were best on the alphabetic arrangement. In the combined task, shape did not significantly influence performance at different levels of practice. Transfers, however, suggested that the shape with salient visual features elicited a text entry strategy for older adults that may promote more consistent performance under occasional usage. CONCLUSION: The studies together demonstrate that keyboard shape is important for efficient performance. Shape-controller compatibility facilitated performance in both age groups. Salient features facilitate performance, especially for older adults. In nearly all cases alphabetic arrangement yielded the best performance. APPLICATION: Recommendations are provided for virtual keyboard design for different usage frequencies, contexts, and users.     

Effects of label format on knowledge acquisition and perceived readability by younger and older adults

This research examines consumers' information acquisition and preference for labels of a simulated over-the-counter (OTC) medication. Twelve otherwise identical OTC drug bottles were compared with different back labels varying in (a) print size, (b) amount of white space between text, and (c) label design (standard vs extended/pull-out). A no back label condition served as a control. Older (mean age=77.7 years) and younger (mean age=21 years) adults were given one of the 12 bottles and asked to perform one of two information acquisition tasks: (a) they examined the bottle for 3 minutes and then completed a questionnaire with the bottle absent, or (b) they answered the same questionnaire while the bottle was present. Afterwards, participants were given all of the bottles and asked to rank them according to perceived readability. The younger adults' information acquisition performance was significantly better than the older adults' for all label conditions except the control condition where both groups' low performance did not differ. Specifically, the older adults' performance was significantly better in the medium and large print conditions than in the small print conditions--with the latter conditions not differing from the control condition. Younger adults showed no performance differences among the different print-size conditions. No substantial effects on knowledge acquisition performance from the white space manipulations were found. However, the perceived readability ranks showed that both groups preferred larger print size and white space. The white space effect was smaller than for print size, particularly for older adults. The extended/pull-out label design was facilitative for older adults in that it allowed the use of larger print. The results suggest that older consumers may be unable to acquire information in the 'fine' print frequently found in various kinds of product literature.     

Aging and response selection in spatial choice tasks

Factors that make response selection more difficult, most notably incompatibility between displays and controls, degrade performance. The cost of incompatibility on performance is often much greater for older than for younger adults. To design products that accommodate decreased response selection capabilities of older adults, designers need to understand the specific ways in which response selection processes change with age. The purpose of the present paper is to review research on age-related changes in stimulus-response compatibility and response precuing effects, the two effects that are most directly linked to basic response selection processes. Several specific aspects of response selection that are particularly harmful for older adults' performance are identified. Potential applications of this research include initial guidelines for minimizing the effects of those aspects when designing for older adults.     

Reading in the dark: effects of age and contrast on reading speed and comprehension

OBJECTIVE: To explore the effects of contrast reduction on younger and older adults' reading behavior and to examine whether readers rely on word predictability to compensate for poor contrast. BACKGROUND: Poor contrast can degrade text and may influence reading behavior. Readers may compensate for visual degradation of text by taking advantage of word predictability. METHOD: In Experiment 1, 5 younger and 5 older adults read sentences presented with 10 levels of contrast. In Experiment 2, 40 younger and 40 older adults read high-, medium-, and low-contrast sentences that varied in target word predictability (high vs. low). RESULTS: Relative to those of younger adults, older adults' reading rates were more slowed by low contrast; comprehension was less influenced by contrast (i.e., not significantly). Older adults read high-predictability words faster and comprehended them better than low-predictability words, significantly so for high- and medium-contrast sentences. Younger adults comprehended high-predictability words significantly better than low-predictability words for high- and low-contrast sentences. CONCLUSION: Low contrast was more detrimental for older adults. High-predictability words benefited older adults by significantly reducing their reading times and benefited all readers by significantly increasing their comprehension. APPLICATION: The current findings demonstrate the benefit of adequate contrast and word predictability for optimal text design to facilitate both reading times and reading comprehension.     

Emotion in reinforcement learning agents and robots: a survey

This article provides the first survey of computational models of emotion in reinforcement learning (RL) agents. The survey focuses on agent/robot emotions, and mostly ignores human user emotions. Emotions are recognized as functional in decision-making by influencing motivation and action selection. Therefore, computational emotion models are usually grounded in the agent’s decision making architecture, of which RL is an important subclass. Studying emotions in RL-based agents is useful for three research fields. For machine learning (ML) researchers, emotion models may improve learning efficiency. For the interactive ML and human–robot interaction community, emotions can communicate state and enhance user investment. Lastly, it allows affective modelling researchers to investigate their emotion theories in a successful AI agent class. This survey provides background on emotion theory and RL. It systematically addresses (1) from what underlying dimensions (e.g. homeostasis, appraisal) emotions can be derived and how these can be modelled in RL-agents, (2) what types of emotions have been derived from these dimensions, and (3) how these emotions may either influence the learning efficiency of the agent or be useful as social signals. We also systematically compare evaluation criteria, and draw connections to important RL sub-domains like (intrinsic) motivation and model-based RL. In short, this survey provides both a practical overview for engineers wanting to implement emotions in their RL agents, and identifies challenges and directions for future emotion-RL research.     

情感计算与理解研究发展概述

情感在感知、决策、逻辑推理和社交等一系列智能活动中起到核心作用,是实现人机交互和机器智能的重要元素。近年来,随着多媒体数据爆发式增长及人工智能的快速发展,情感计算与理解引发了广泛关注。情感计算与理解旨在赋予计算机系统识别、理解、表达和适应人的情感的能力来建立和谐人机环境,并使计算机具有更高、更全面的智能。根据输入信号的不同,情感计算与理解包含不同的研究方向。本文全面回顾了多模态情感识别、孤独症情感识别、情感图像内容分析以及面部表情识别等不同情感计算与理解方向在过去几十年的研究进展并对未来的发展趋势进行展望。对于每个研究方向,首先介绍了研究背景、问题定义和研究意义;其次从不同角度分别介绍了国际和国内研究现状,包括情感数据标注、特征提取、学习算法、部分代表性方法的性能比较和分析以及代表性研究团队等;然后对国内外研究进行了系统比较,分析了国内研究的优势和不足;最后讨论了目前研究存在的问题及未来的发展趋势与展望,例如考虑个体情感表达差异问题和用户隐私问题等。     

Authentic facial expression analysis

There is a growing trend toward emotional intelligence in human–computer interaction paradigms. In order to react appropriately to a human, the computer would need to have some perception of the emotional state of the human. We assert that the most informative channel for machine perception of emotions is through facial expressions in video. One current difficulty in evaluating automatic emotion detection is that there are currently no international databases which are based on authentic emotions. The current facial expression databases contain facial expressions which are not naturally linked to the emotional state of the test subject. Our contributions in this work are twofold: first, we create the first authentic facial expression database where the test subjects are showing the natural facial expressions based upon their emotional state. Second, we evaluate the several promising machine learning algorithms for emotion detection which include techniques such as Bayesian networks, SVMs, and decision trees.     

Evidence for an elders' advantage in the naive product usability judgments of older and younger adults

OBJECTIVE: To determine whose naive judgments of consumer product usability are more accurate--those of younger or older adults. Accuracy is here defined as judgments compatible with results from performance-based usability tests. BACKGROUND: Older adults may be better able to predict usability problems than younger adults, making them particularly good participants in studies contributing to the user-centered design of products. This advantage, if present, may stem from older adults' motivation for more usable products or from their experience adapting their own environments to meet their changing physical, cognitive, and sensory needs. METHOD: Sixty older participants (ages 65-75 years) and 60 younger ones (ages 18-22 years) evaluated illustrations of consumer products on specific criteria (e.g., readability, learnability, or error rates). They either rated a single design for each product or ranked six alternative designs. They also explained their choices, indicated which features were most critical for usability, and selected usability-enhancing modifications. RESULTS: Although there was no reliable age difference in the amount of usability information provided in the open-ended explanations, older adults were more accurate at ranking alternative designs, selecting the most usability-critical features, and selecting usability-enhancing modifications (all ps < .05). CONCLUSION: The usability judgments of older adults are more accurate than those of younger adults when these judgments are solicited in a fixed-alternative, but not open-ended, format. APPLICATION: Because older adults are more discerning about potential product usability problems, they may be particularly valuable as research participants in early-stage design research (prior to the availability of working prototypes).     

Emotion Recognition with Capsule Neural Network

For human-machine communication to be as effective as human-to-human communication, research on speech emotion recognition is essential. Among the models and the classifiers used to recognize emotions, neural networks appear to be promising due to the network’s ability to learn and the diversity in configuration. Following the convolutional neural network, a capsule neural network (CapsNet) with inputs and outputs that are not scalar quantities but vectors allows the network to determine the part-whole relationships that are specific 6 for an object. This paper performs speech emotion recognition based on CapsNet. The corpora for speech emotion recognition have been augmented by adding white noise and changing voices. The feature parameters of the recognition system input are mel spectrum images along with the characteristics of the sound source, vocal tract and prosody. For the German emotional corpus EMO-DB, the average accuracy score for 4 emotions, neutral, boredom, anger and happiness, is 99.69%. For Vietnamese emotional corpus BKEmo, this score is 94.23% for 4 emotions, neutral, sadness, anger and happiness. The accuracy score is highest when combining all the above feature parameters, and this score increases significantly when combining mel spectrum images with the features directly related to the fundamental frequency.     

Individual difference of artificial emotion applied to a service robot

In order to enable personalized natural interaction in service robots, artificial emotion is needed which helps robots to appear as individuals. In the emotion modeling theory of emotional Markov chain model (eMCM) for spontaneous transfer and emotional hidden Markov model (eHMM) for stimulated transfer, there are three problems: 1) Emotion distinguishing problem: whether adjusting parameters of the model have any effects on individual emotions; 2) How much effect the change makes; 3) The problem of different initial emotional states leading to different resultant emotions from a given stimuli. To solve these problems, a research method of individual emotional difference is proposed based on metric multidimensional scaling theory. Using a dissimilarity matrix, a scalar product matrix is calculated. Subsequently, an individual attribute reconstructing matrix can be obtained by principal component factor analysis. This can display individual emotion difference with low dimension. In addition, some mathematical proofs are carried out to explain experimental results. Synthesizing the results and proofs, corresponding conclusions are obtained. This new method provides guidance for the adjustment of parameters of emotion models in artificial emotion theory.     

Mutual influences between message volume and emotion intensity on emerging infectious diseases: An investigation with microblog data

The dynamic interdependence between microblog volume and emotions expressed in message content has been largely underexplored. To understand the public’s reaction to emerging infectious diseases, we draw upon theories in psychology and social media, and propose that there is a cyclical relationship between message volume and the intensity of emotions (positive or negative) in which they influence each other positively over time. Furthermore, negative and positive emotions mutually suppress each other, yet are autocorrelated, respectively. Relying on more than 560,000 microblogs collected from Sina Weibo between February 2013 and June 2013 on the outbreak of avian influenza in China, we used vector autoregression to test the research model. We find a cyclical relationship between microblog volume and the intensity of negative emotions (fear in particular, a subcategory of negative emotions). Microblog volume positively contributes to positive emotions. While the intensity of negative emotions, and that of positive emotions is autocorrelated, respectively, surprisingly our results suggest that negative emotion intensity positively affects positive emotion intensity. We further rely on impulse response functions to illustrate how the impacts of a variable on another change over time, and generalized forecast error variance decomposition to understand the importance of each variable contributing to the others. Theoretical and practical implications of the study are discussed.     

Expressing reactive emotion based on multimodal emotion recognition for natural conversation in human–robot interaction*

Human–human interaction consists of various nonverbal behaviors that are often emotion-related. To establish rapport, it is essential that the listener respond to reactive emotion in a way that makes sense given the speaker's emotional state. However, human–robot interactions generally fail in this regard because most spoken dialogue systems play only a question-answer role. Aiming for natural conversation, we examine an emotion processing module that consists of a user emotion recognition function and a reactive emotion expression function for a spoken dialogue system to improve human–robot interaction. For the emotion recognition function, we propose a method that combines valence from prosody and sentiment from text by decision-level fusion, which considerably improves the performance. Moreover, this method reduces fatal recognition errors, thereby improving the user experience. For the reactive emotion expression function, the system's emotion is divided into emotion category and emotion level, which are predicted using the parameters estimated by the recognition function on the basis of distributions inferred from human–human dialogue data. As a result, the emotion processing module can recognize the user's emotion from his/her speech, and expresses a reactive emotion that matches. Evaluation with ten participants demonstrated that the system enhanced by this module is effective to conduct natural conversation.     

基于ConvLSTM网络的维度情感识别模型研究

学业情绪能够影响和调节学习者的注意、记忆、思维等认知活动,情绪自动识别是智慧学习环境中情感交互和教学决策的基础。目前情绪识别研究主要集中在离散情绪的识别,其在时间轴上是非连续的,无法精准刻画学生学业情绪演变过程,为解决这个问题,基于众包方法建立真实在线学习情境中的中学生学习维度情感数据集,设计基于连续维度情感预测的深度学习分析模型。实验中根据学生学习风格确定触发学生学业情绪的学习材料,并招募32位实验人员进行自主在线学习,实时采集被试面部图像,获取157个学生学业情绪视频;对每个视频进行情感Arousal和Valence二维化,建立包含2 178张学生面部表情的维度数据库;建立基于ConvLSTM网络的维度情感模型,并在面向中学生的维度情感数据库上进行实验,得到一致性相关系数（Concordance Correlation Coefficient,CCC）均值为0.581,同时在Aff-Wild公开数据集上进行实验,得到的一致相关系数均值为0.222。实验表明,提出的基于维度情感模型在Aff-Wild公开数据集维度情绪识别中CCC相关度系数指标提升了7.6%~43.0%。     

Learning to use a home medical device: mediating age-related differences with training

We examined the differential benefits of instructional materials for younger and older adults learning to use a home medical device. Participants received training on use of a blood glucose meter via either a user manual (a text guide with pictures) or an instructional video. Performance was measured immediately and then after a 2-week retention interval. Type of instruction was critical for determining older adults' performance. Older adults trained using the manual had poorer performance than did all other groups. After only 1 calibration, older adults who received video training performed as accurately as did the younger adults. Older adults' performance was more influenced by the retention interval; however, the benefit of the video training was maintained for the older adults across the retention interval. Confidence ratings paralleled subjective workload ratings. The data provide practical information to guide the development of training programs for systems that will be used by both younger and older adults; they also demonstrate the need for age-related usability testing even for training program design.     

Recommendations to support interaction with broadcast debates: a study on older adults’ interaction with <Emphasis Type="Italic">The Moral Maze</Emphasis>

Current methods to capture, analyse and present the audience participation of broadcast events are increasingly carried out using social media. Uptake of such technology tools has so far been poor amongst older adults, and it has the worrying effect of excluding the demographic from participation. Our work explores whether a common desire to interact with debates can be tapped with technology with a very low barrier to entry, to both support better engagement with broadcast debates and encourage greater use of social media. This paper describes experiments where older adults interact with a BBC radio debate programme: The Moral Maze. As a result, we obtained common interaction patterns which then are used to define recommendations for software-supported interaction with debates based on theories of argumentation. Our goal is to combine research on computational models of argument and user-driven research on human-centred computing in a project with the potential for high-profile impact in addressing older adults inclusion in the digital economy.