Automatic detection of learner’s affect from conversational cues

We explored the reliability of detecting a learner’s affect from conversational features extracted from interactions with AutoTutor, an intelligent tutoring system (ITS) that helps students learn by holding a conversation in natural language. Training data were collected in a learning session with AutoTutor, after which the affective states of the learner were rated by the learner, a peer, and two trained judges. Inter-rater reliability scores indicated that the classifications of the trained judges were more reliable than the novice judges. Seven data sets that temporally integrated the affective judgments with the dialogue features of each learner were constructed. The first four datasets corresponded to the judgments of the learner, a peer, and two trained judges, while the remaining three data sets combined judgments of two or more raters. Multiple regression analyses confirmed the hypothesis that dialogue features could significantly predict the affective states of boredom, confusion, flow, and frustration. Machine learning experiments indicated that standard classifiers were moderately successful in discriminating the affective states of boredom, confusion, flow, frustration, and neutral, yielding a peak accuracy of 42% with neutral (chance = 20%) and 54% without neutral (chance = 25%). Individual detections of boredom, confusion, flow, and frustration, when contrasted with neutral affect, had maximum accuracies of 69, 68, 71, and 78%, respectively (chance = 50%). The classifiers that operated on the emotion judgments of the trained judges and combined models outperformed those based on judgments of the novices (i.e., the self and peer). Follow-up classification analyses that assessed the degree to which machine-generated affect labels correlated with affect judgments provided by humans revealed that human-machine agreement was on par with novice judges (self and peer) but quantitatively lower than trained judges. We discuss the prospects of extending AutoTutor into an affect-sensing ITS.     

Multimodal semi-automated affect detection from conversational cues,gross body language,and facial features

We developed and evaluated a multimodal affect detector that combines conversational cues, gross body language, and facial features. The multimodal affect detector uses feature-level fusion to combine the sensory channels and linear discriminant analyses to discriminate between naturally occurring experiences of boredom, engagement/flow, confusion, frustration, delight, and neutral. Training and validation data for the affect detector were collected in a study where 28 learners completed a 32- min. tutorial session with AutoTutor, an intelligent tutoring system with conversational dialogue. Classification results supported a channel × judgment type interaction, where the face was the most diagnostic channel for spontaneous affect judgments (i.e., at any time in the tutorial session), while conversational cues were superior for fixed judgments (i.e., every 20 s in the session). The analyses also indicated that the accuracy of the multichannel model (face, dialogue, and posture) was statistically higher than the best single-channel model for the fixed but not spontaneous affect expressions. However, multichannel models reduced the discrepancy (i.e., variance in the precision of the different emotions) of the discriminant models for both judgment types. The results also indicated that the combination of channels yielded superadditive effects for some affective states, but additive, redundant, and inhibitory effects for others. We explore the structure of the multimodal linear discriminant models and discuss the implications of some of our major findings.     

Toward an Affect-Sensitive AutoTutor

Here, we consider the possibility of enabling AutoTutor, an intelligent tutoring system, to process learners' affective and cognitive states. AutoTutor is a fully automated computer tutor that simulates human tutors and converses with students in natural language.     

AutoTutor: Incorporating Back-Channel Feedback and Other Human-Like Conversational Behaviors into an Intelligent Tutoring System

This paper describes our recent attempts to incorporate human-like conversational behaviors into the dialog moves delivered by an animated pedagogical agent that simulates human tutors. We first present a brief overview of the modules comprising AutoTutor, an intelligent tutoring system. The second section describes a set of conversational behaviors that are being incorporated into AutoTutor. The behaviors of interest involve variations in intonation, head movements, arm and hand movements, facial expressions, eye blinking, gaze direction, and back-channel feedback. The final section presents a recent empirical study concerned with back-channel feedback events during human-to-human tutoring sessions. The back-channel feedback events emitted by tutors are mostly positive (63%), mostly verbal (77%), and immediately follow speech-act boundaries or noun-phrase boundaries (83%). Tutors also deliver back-channelevents at a very high rate when students are emitting dialog, about 13 events per minute. Conversely, 88% of students' back-channel feedback events are head nods, and they occur at unbounded locations (63%).     

Agent based affective tutoring systems: A pilot study

An important trend in the development of Intelligent tutoring systems (ITSs) has been that providing the student with a more personalized and friendly environment for learning. Many researchers now feel strongly that the ITSs would significantly improve performance if they could adapt to the affective state of the learner. This idea has spawned the developing field of affective tutoring systems (ATSs): ATSs are ITSs that are able to adapt to the affective state of students. However, ATSs are not widely employed in the tutoring system market. In this paper, a survey was conducted to investigate the critical factors affecting learner’s satisfaction in ATSs based on an ATS developed by us. The results revealed that learner’s attitude toward affective computing, agent tutor’s expressiveness, emotion recognition accuracy, number of emotions recognized by agent tutor, pedagogical action and easy of the use of the system have significant influence on learner’s satisfaction. The results indicate institutions how to further strengthen the ATSs’ implementation.     

The nature of adolescent learner interaction in a virtual high school setting

This study used survey data to measure the effect of learners' reported interactions with content, peers, and instructors on several course outcomes in two virtual high school courses that emphasized interactive learning. Surveys found that the large majority of students viewed all investigated types of interaction as educational and motivational. Students perceived learner–instructor and learner–content interactions to have significantly higher educational value (α < 0.01) than learner–learner interactions, and viewed learner–instructor interaction to be significantly more motivational (α < 0.01) than learner–content interaction. Furthermore, nine significant correlations were found involving the time students reported spending on human interaction and course outcomes. Seven of the significant correlations were related to the time students reported spending in human interaction and the more affective outcomes, such as course satisfaction and disposition towards the subject area. Outcomes also indicate that learner–learner interaction had higher correlations with course outcomes than learners' interactions with the content or their instructor. Students' perceived learning was not significantly correlated with any type of interaction, and only students' total reported time spent on learner–learner interaction and students' social learner–learner interaction were significantly correlated with their grade.     

Diagnosing and acting on student affect: the tutor’s perspective

In this paper we explore human tutors’ inferences in relation to learners’ affective states and the relationship between those inferences and the actions that tutors take as their consequence. At the core of the investigations presented in this paper lie fundamental questions associated with the role of affective considerations in computer-mediated educational interactions. Theory of linguistic politeness is used as the basis for determining the contextual factors relevant to human tutors’s actions, with special attention being dedicated to learner affective states. A study was designed to determine what affective states of the learners are relevant to tutoring mathematics and to identify the mechanisms used by tutors to predict such states. Logs of tutor-student dialogues were recorded along with contextual factors taken into consideration by tutors in relation to their specific tutorial dialogue moves. The logs were annotated in order to determine the types and range of student and tutor actions. Machine learning techniques were then applied to those actions to predict the values of three factors: student confidence, interest and effort. Whilst due to limited size and sparsity of data the results are not conclusive, they are very valuable as the basis for empirically derived hypotheses to be tested in further studies. The potential implications of the hypotheses, if they were confirmed by further studies, are discussed in relation to the impact of tutor’s ability to diagnose student affect on the nature of computer-mediated tutorial interactions.     

Managing programmed instruction and collaborative peer tutoring in the classroom: Applications in teaching Java™

To fulfill part of the course requirements, 34 undergraduate students in two courses completed an online programmed instruction tutor as the first technical training exercise in a Java™ programming course designed for information systems majors. The tutor taught a simple JApplet program to display a text string within a browser window on the Web. Students in the first course next participated in a collaborative peer tutoring session, based on the JApplet program, followed by a lecture on the program and by successfully running the program on the Web. For the second course, the peer tutoring session was scheduled after the lecture and after successfully running the program. Students in both courses completed tests of far transfer (“meaningful learning”) and software self-efficacy before using the tutor and on several subsequent occasions following that initial learning. Students in the second course also completed a 4-item scale to assess the development of classification and functionality knowledge regarding elements of the program. Students in both courses showed progressive improvement in all performance measures across the several assessment occasions. Students’ positive ratings of the effectiveness of both the tutor and the collaborative peer tutoring supported the value of these learning experiences in a technical knowledge domain. The results of this study, based on student performance observed within the context of the classroom, show the importance of providing a range of synergistic learning experiences that culminate in a level of skill and confidence that prepares and motivates all students for advanced instruction in Java. They also show how to manage the instructional techniques in the classroom to accomplish that educational outcome.     

Bi-modal emotion recognition from expressive face and body gestures

Psychological research findings suggest that humans rely on the combined visual channels of face and body more than any other channel when they make judgments about human communicative behavior. However, most of the existing systems attempting to analyze the human nonverbal behavior are mono-modal and focus only on the face. Research that aims to integrate gestures as an expression mean has only recently emerged. Accordingly, this paper presents an approach to automatic visual recognition of expressive face and upper-body gestures from video sequences suitable for use in a vision-based affective multi-modal framework. Face and body movements are captured simultaneously using two separate cameras. For each video sequence single expressive frames both from face and body are selected manually for analysis and recognition of emotions. Firstly, individual classifiers are trained from individual modalities. Secondly, we fuse facial expression and affective body gesture information at the feature and at the decision level. In the experiments performed, the emotion classification using the two modalities achieved a better recognition accuracy outperforming classification using the individual facial or bodily modality alone.     

Developing sound tutor training for online writing centers: creating productive peer reviewers

It is our experience that tutors trained for face-to-face writing centers are not adequately prepared for the challenges they encounter working with online writing centers. The purpose of our article is to provide an overview—especially for administrators, developers, and tutors new to electronic tutoring environments—of the issues and considerations unique to online tutoring that training programs need to address. In our discussion, we hope to engender enthusiasm for online tutoring by discussing three aspects of online tutoring: appreciating text-only environments, developing procedures for responding online, and creating appropriate roles for online tutors. We offer suggestions about how to address these three aspects in online tutor training, and we suggest that addressing these issues leads to an understanding of the online tutor as a productive peer reviewer.     

Toward Spoken Human–Computer Tutorial Dialogues

Oral discourse is the primary form of human–human communication, hence, computer interfaces that communicate via unstructured spoken dialogues will presumably provide a more efficient, meaningful, and naturalistic interaction experience. Within the context of learning environments, there are theoretical positions supporting a speech facilitation hypothesis that predicts that spoken tutorial dialogues will increase learning more than typed dialogues. We evaluated this hypothesis in an experiment where 24 participants learned computer literacy via a spoken and a typed conversation with AutoTutor, an intelligent tutoring system with conversational dialogues. The results indicated that (a) enhanced content coverage was achieved in the spoken condition; (b) learning gains for both modalities were on par and greater than a no-instruction control; (c) although speech recognition errors were unrelated to learning gains, they were linked to participants' evaluations of the tutor; (d) participants adjusted their conversational styles when speaking compared to typing; (e) semantic and statistical natural language understanding approaches to comprehending learners' responses were more resilient to speech recognition errors than syntactic and symbolic-based approaches; and (f) simulated speech recognition errors had differential impacts on the fidelity of different semantic algorithms. We discuss the impact of our findings on the speech facilitation hypothesis and on human–computer interfaces that support spoken dialogues.     

Designing automated adaptive support to improve student helping behaviors in a peer tutoring activity

Adaptive collaborative learning support systems analyze student collaboration as it occurs and provide targeted assistance to the collaborators. Too little is known about how to design adaptive support to have a positive effect on interaction and learning. We investigated this problem in a reciprocal peer tutoring scenario, where two students take turns tutoring each other, so that both may benefit from giving help. We used a social design process to generate three principles for adaptive collaboration assistance. Following these principles, we designed adaptive assistance for improving peer tutor help-giving, and deployed it in a classroom, comparing it to traditional fixed support. We found that the assistance improved the conceptual content of help and the use of interface features. We qualitatively examined how each design principle contributed to the effect, finding that peer tutors responded best to assistance that made them feel accountable for help they gave.     

Boosting metacomprehension accuracy in computer-supported learning: The role of judgment task and judgment scope

Although learners' judgments of their own learning are crucial for self-regulated study, judgment accuracy tends to be low. To increase accuracy, we had participants make combined judgments. In Experiment 1, 247 participants studied a ten-chapter expository text. In the simple judgments group, participants after each chapter rated the likelihood of answering correctly a knowledge question on that chapter (judgment of learning; JOL). In the combined judgments group, participants rated text difficulty before making a JOL. No accuracy differences emerged between groups, but a comparison of early-chapter and late-chapter judgment magnitudes showed that the judgment manipulation had induced cognitive processing differences. In Experiment 2, we therefore manipulated judgment scope. Rather than predicting answers correct for an entire chapter, another 256 participants rated after each chapter the likelihood of answering correctly a question on a specific concept from that chapter. Both judgment accuracy and knowledge test scores were higher in the combined judgments group. Moreover, while judgment accuracy dropped to an insignificant level between early and late chapters in the simple judgments group, accuracy remained constant with combined judgments. We discuss implications for research into metacomprehension processes in computer-supported learning and for adaptive learner support based on judgment prompts.     

Designing learner-controlled educational interactions based on learning/cognitive style and learner behaviour

Recently, research in individual differences and in particular, learning and cognitive style, has been used as a basis to consider learner preferences in a web-based educational context. Modelling style in a web-based learning environment demands that developers build a specific framework describing how to design a variety of options for learners with different approaches to learning. In this paper two representative examples of educational systems, Flexi-OLM and INSPIRE, that provide learners a variety of options designed according to specific style categorisations, are presented. Experimental results from two empirical studies performed on the systems to investigate learners' learning and cognitive style, and preferences during interaction, are described. It was found that learners do have a preference regarding their interaction, but no obvious link between style and approaches offered, was detected. Derived from an examination of this experimental data, we suggest that while style information can be used to inform the design of learning environments that accommodate learners' individual differences, it would be wise to recommend interactions based on learners' behaviour. Learning environments should allow learners or learners' interaction behaviour to select or trigger the appropriate approach for the particular learner in the specific context. Alternative approaches towards these directions are also discussed.     

基于情感识别的智能教学系统研究

针对传统的智能教学系统(ITS)在情感方面的缺失,提出了基于情感识别技术的ITS模型.该系统模型在传统的教学系统上新增情感识别模块,利用人脸表情识别以及文本识别等技术所构建,可以获取和识别学生的学习情感,并根据学习情感进行相应的情感激励策略,实现情感化的教学.     

基于人工情绪的智能情感网络教学系统研究

针时传统智能网络教学系统在情感教学方面的缺陷，基于人工情绪技术提出了一种Web环境下的智能情感网络教学系统结构．该系统由学习情绪模型、情绪教学模型、认知教学模型和学生模型等主要模块所构成，可以获取和识别每个学生的学习表情，并能够根据不同学生的学习情绪和学习效果，实现认知和情感相互协调的个性化教学．     

融合MAML和CBAM的安卓恶意应用家族分类模型

为满足对新兴安卓恶意应用家族的快速检测需求,提出一种融合MAML(model-agnostic meta-learning)和CBAM(convolutional block attention module)的安卓恶意应用家族分类模型MAML-CAS。将安卓恶意应用样本集中的DEX文件可视化为灰度图,并构建任务集;融合混合域注意力机制CBAM,设计两个具有同等结构的卷积神经网络,分别作为基学习器和元学习器,这两个学习器在自动提取任务集中样本特征的同时,可从通道和空间两个维度来增强关键特征表达;利用元学习方法 MAML对两个学习器进行训练,其中基学习器完成特定恶意家族分类任务的属性学习,元学习器则学习不同任务的共性;在两个学习器训练完成后,MAML-CAS将获得初始化参数,在面对新的安卓恶意应用家族分类任务时,不需要重新训练,只需要少量样本就可以快速迭代;利用训练完成的基学习器提取安卓恶意应用家族特征,并利用SVM进行恶意家族分类。实验结果表明,MAML-CAS模型对新兴小样本安卓恶意应用家族具有良好的检测效果,检测速度较快,并具有较好的稳定性。     

Cross-cultural differences in recognizing affect from body posture

Conveyance and recognition of human emotion and affective expression is influenced by many factors, including culture. Within the user modeling field, it has become increasingly necessary to understand the role affect can play in personalizing interactive interfaces using embodied animated agents. However, little research within the computer science field aims at understanding cultural differences within this vein. Therefore, we conducted a study to evaluate if differences exist in the way various cultures perceive emotion from body posture. We used static posture images of affectively expressive avatars to conduct recognition experiments with subjects from three cultures. After analyzing the subjects' judgments using multivariate analysis, we grounded the identified differences into a set of low-level posture features. We then used Mixture Discriminant Analysis (MDA) and an unsupervised expectation maximization (EM) model to build separate cultural models for affective posture recognition. Our results could prove useful to aide designers in creating more effective affective avatars.     

An expert system manipulating knowledge to help human learners into virtual environment

This research is situated within the context of the creation of human learning environments using virtual reality. We propose the integration of a generic and adaptable intelligent tutoring system (Pegase). The aim is to instruct the learner, and to assist the instructor. The multi-agent system emits a set of knowledge (actions carried out by the learner, knowledge of the field, etc.) used by an artificial intelligence to make pedagogical decisions. Our study focuses on the representation of knowledge about the environment, and on the adaptable pedagogical agent providing instructive assistance.     

Affective learning: improving engagement and enhancing learning with affect-aware feedback

This paper describes the design and ecologically valid evaluation of a learner model that lies at the heart of an intelligent learning environment called iTalk2Learn. A core objective of the learner model is to adapt formative feedback based on students’ affective states. Types of adaptation include what type of formative feedback should be provided and how it should be presented. Two Bayesian networks trained with data gathered in a series of Wizard-of-Oz studies are used for the adaptation process. This paper reports results from a quasi-experimental evaluation, in authentic classroom settings, which compared a version of iTalk2Learn that adapted feedback based on students’ affective states as they were talking aloud with the system (the affect condition) with one that provided feedback based only on the students’ performance (the non-affect condition). Our results suggest that affect-aware support contributes to reducing boredom and off-task behavior, and may have an effect on learning. We discuss the internal and ecological validity of the study, in light of pedagogical considerations that informed the design of the two conditions. Overall, the results of the study have implications both for the design of educational technology and for classroom approaches to teaching, because they highlight the important role that affect-aware modelling plays in the adaptive delivery of formative feedback to support learning.