Online Affect Detection and Robot Behavior Adaptation for Intervention of Children With Autism

Investigation into robot-assisted intervention for children with autism spectrum disorder (ASD) has gained momentum in recent years. Therapists involved in interventions must overcome the communication impairments generally exhibited by children with ASD by adeptly inferring the affective cues of the children to adjust the intervention accordingly. Similarly, a robot must also be able to understand the affective needs of these children—an ability that the current robot-assisted ASD intervention systems lack—to achieve effective interaction that addresses the role of affective states in human–robot interaction and intervention practice. In this paper, we present a physiology-based affect-inference mechanism for robot-assisted intervention where the robot can detect the affective states of a child with ASD as discerned by a therapist and adapt its behaviors accordingly. This paper is the first step toward developing “understanding” robots for use in future ASD intervention. Experimental results with six children with ASD from a proof-of-concept experiment (i.e., a robot-based basketball game) are presented. The robot learned the individual liking level of each child with regard to the game configuration and selected appropriate behaviors to present the task at his/her preferred liking level. Results show that the robot automatically predicted individual liking level in real time with 81.1% accuracy. This is the first time, to our knowledge, that the affective states of children with ASD have been detected via a physiology-based affect recognition technique in real time. This is also the first time that the impact of affect-sensitive closed-loop interaction between a robot and a child with ASD has been demonstrated experimentally.      

Participatory Design of a Hybrid Kinect Game to Promote Collaboration between Autistic Players and Their Peers

ABSTRACT

We report on the participatory design of a hybrid two-player Kinect game to encourage complex emotion recognition and collaboration between autistic people and their peers. From its inception, autistic college students have been involved in an iterative process to design, evaluate, and redesign the game. The emotion recognition game has two playing phases. In the first phase, the players independently assemble pieces in a digital puzzle. In the second phase, players communicate in-person to agree on the appropriate emotion for the context and construct the emotional face for the body they have assembled together. We also designed collaborative reward games that require the two players to cooperate and one that encourages players to look at each other. In order to assess the level of in-game cooperation, we added a face tracking component that automatically quantifies collaboration and can replace time-consuming hand-coded evaluations. We report on how this game was designed and built by a team of autistic students, their peer mentors, a psychologist, computer scientists, and a graphic artist. Preliminary observations show that modeling of emotion recognition and collaboration by peers with stronger social skills is emerging as a central aspect of the effectiveness of our game. The participatory process has led to several design changes including one that dramatically increased player collaboration. We share insights and lessons learned that can guide others working in participatory design.     

An inclusive design approach for developing video games for children with Autism Spectrum Disorder

The efficacy of therapeutic treatments for Autism Spectrum Disorder is mainly associated with the treatment's intensity in terms of weekly hours. This has led mental health professionals to explore the use of video games to complement traditional treatments. However, major weaknesses have been found due to poor game design, which has failed to fulfill therapeutic objectives or to properly engage children. These weaknesses are typically characterized by the poor integration of knowledge from mental health experts, children's interests and designers' expertise. Starting from this necessity, we propose an inclusive design approach to develop therapeutic games. The method presents strategies to integrate the expertise of clinicians, contributions of children and experience of designers through a set of elicitation and merging techniques. The goal of this method is to design games that are effective in terms of therapeutic objectives and that are enjoyable for children. To describe this method, we present its application in the design and development of a Kinect-based game for high-functioning children with ASD called “Pico's Adventures”. This game aims at promoting social initiation in young children with autism. Findings from a first exploratory study with 10 children with ASD showed the effectiveness of the game in eliciting social initiation behaviors. This provides a first validation of the method. An essential aspect of the game's success was the use of elements and mechanics that were appealing for the children. As a result, we have identified effective design concepts and paths for further research on games for children with ASD.     

Expressing and interpreting emotional movements in social games with robots

This paper provides a framework for recording, analyzing and modeling of 3 dimensional emotional movements for embodied game applications. To foster embodied interaction, we need interfaces that can develop a complex, meaningful understanding of intention—both kinesthetic and emotional—as it emerges through natural human movement. The movements are emulated on robots or other devices with sensory-motor features as a part of games that aim improving the social interaction skills of children. The design of an example game platform that is used for training of children with autism is described since the type of the emotional behaviors depends on the embodiment of the robot and the context of the game. The results show that quantitative movement parameters can be matched to emotional state of the embodied agent (human or robot) using the Laban movement analysis. Emotional movements that were emulated on robots using this principle were tested with children in the age group 7–9. The tests show reliable recognition on most of the behaviors.     

Outcomes for design and learning when teenagers with autism codesign a serious game: A pilot study

Digital educational games research tends to lack ecological validity by not adequately taking into account the views and perspectives of children and young people with autism spectrum disorders (ASD). This paper is a pilot study that explores and analyses an academic‐based educational game that was co‐designed with and for young people with ASD. The serious game aims to help the players learn Geography‐specific knowledge and integrates several strategic features so that users can collaborate together against the computer or compete against each other. The educational game was evaluated over 5 sessions by 3 peer teams from 2 different special educational institutions, involving a total of 6 students with ASD. The participants were positive about their enjoyment, motivation, and social engagement. The results showed that the players' level of competitiveness not only influenced the experience within the game but also the interaction within the peer teams. The game mechanisms did help the participants with ASD increase their knowledge in Geography content. The main conclusion is that there are considerable benefits of including children with ASD in the design process and future research should explore more fully on how their involvement can enhance curriculum‐based learning as well as social engagement within the classroom.     

Using emotion recognition technologies to teach children with autism spectrum disorder how to identify and express emotions

Autism spectrum disorder (ASD), which since 2013 is considered as an umbrella term for several disorders such as autistic syndrome, Asperger’s disorder and pervasive developmental disorder, is characterized, among other aspects, by deficits in social-emotion reciprocity. This deficit manifests itself as a reduced sharing of emotions and an increased difficulty in interpreting emotions other people are feeling, which in the end leads to more impairments in social communication. Since it is possible to help a person with ASD (especially children) to improve their ability to understand and detect emotions, we have developed a proposal which integrates emotion recognition technologies, often used in the field of HCI, to try to overcome this difficulty. In this paper, we present a novel software application developed as a serious game to teach children with autism spectrum disorder (ASD) to identify and express emotions. The system incorporates cutting-edge technology to support novel interaction mechanisms based on tangible user interfaces (TUIs) and emotion recognition from facial expressions. In this way, children interact with the system in a natural way by simply grasping objects with their hands and using their faces. The system has been assessed on the premises of an association with children with ASD. The outcomes of the evaluation are very positive and support the validity of the proposal.

     

基于视觉认知的自闭症儿童干预图卡角色造型设计研究

以自闭症儿童视觉认知特点为基础,探究自闭症儿童的视觉意象偏好,解析干预 图卡角色造型的设计要素,充分发挥干预图卡的效能。首先通过康复机构、教辅人员、特殊教 育网站等渠道整理目前市场上流通的自闭症儿童干预图卡,筛选出具有代表性的图卡角色造型, 并结合自闭症儿童视觉认知特点对角色造型的特征进行意象分析,整理视觉意象词汇;然后运 用语义差异法,将视觉意象词汇和图卡样本相结合制作评估量表,对获得数据进行统计整理, 通过结合因子分析法提取出视觉意象主因子;最终构建自闭症儿童视觉认知偏好与主意象因子 之间的映射关系。通过调研与实验,整理了具有代表性的干预图卡角色造型,结合自闭症儿童 的视觉偏好与意象词汇,有针对性地调整造型特征,能够增强干预图卡角色造型的趣味性,提 高干预训练的有效性,为教辅人员进行干预训练提供参考。     

Audio-augmented paper for therapy and educational intervention for children with autistic spectrum disorder

Autism affects children׳s learning and social development. Commonly used rehabilitative treatments are aimed at stimulating the social skills of children with autism. In this article, we present a prototype and a pilot study on an audio-augmented paper to support the therapy of children with autism spectrum disorder (ASD). The prototype supports audio recording with standard sheets of paper by using tangible tools that can be shared between the therapist and the child. The prototype is a tool for the therapist to engage the child in a storytelling activity. We use a progressive design method based on a dynamic process that merges concept generation, technology benchmarking and activity design into continuously enriching actions. The paper highlights the qualities and benefits of using tangible audio-augmented artefacts for therapy and educational intervention for children with ASD. The work describes three main qualities of our prototype: from building cooperation to attention control, flow control, and using the children׳s own voices to foster attention.     

Serious Game for Children with Autism to Learn Vocabulary: An Experimental Evaluation

The use of serious games (SGs) to provide intervention for various skills to children with autism spectrum disorder (ASD) has increased in the recent years. However, the potential of these games has not been utilized to provide learning of vocabulary to children with ASD. In designing games for children with ASD, there is a need to consider components specifically from the perspectives of ASD and the generic components that can support from the theoretical, content, and game design aspects. In the absence of any existing framework to design games for children with ASD, this article first introduces a framework named serious game design framework (SGDF), constructed to provide support throughout the design process. This framework was constructed based on the extensive review of the literature on ASD and existing SGDFs. The article then focuses on the design and development of a SG prototype based on SGDF. An experimental evaluation prototype was conducted to examine the effectiveness using the withdrawal design of single-subject research design (SSRD) in improving the receptive identification of vocabulary items among children with autism before and after using the prototype. The receptive identification of vocabulary items is measured in terms of the number of correct responses given and the number of attempts made to identify the correct response. The pre- and post-evaluations of the SG prototype show that learning of vocabulary items among children with ASD improved after using the game and they retained items at the end of weeks 1 and 2 following the withdrawal of intervention.     

Making Space for Voice: Technologies to Support Children’s Fantasy and Storytelling

Fantasy play and storytelling serve an important role in young children’s development. While computers are increasingly present in the world of young children, there is a lack of computational tools to support children’s voices in everyday storytelling, particularly in the context of fantasy play. We believe that there is a need for computational systems that engage in story-listening rather than story-telling. This paper introduces StoryMat, a system that supports and listens to children’s voices in their own storytelling play. StoryMat offers a child-driven, story-listening space by recording and recalling children’s narrating voices, and the movements they make with their stuffed animals on a colourful story-evoking quilt. Empirical research with children shows that StoryMat fosters developmentally advanced forms of storytelling of the kind that has been shown to provide a bridge to written literacy, and provides a space where children engage in fantasy storytelling collaboratively with or without a playmate. The paper addresses the importance of supporting young children’s fantasy play and suggests a new way for technology to play an integral part in that activity.     

Multitouch tablet applications and activities to enhance the social skills of children with autism spectrum disorders

In spite of great improvements in early diagnosis and interventions, most children diagnosed with autism spectrum disorders (ASD) are unlikely to live independently when they reach adulthood. We have been conducting research on novel computer-based interventions with the goal of promoting social skills. Working with 26 children with ASD, their teachers, and other stakeholders, we have iteratively developed a set of activities based on applications that run on multitouch tablets. Our observations suggest these activities increased pro-social behaviors such as collaboration and coordination, augmented appreciation for social activities, and provided children with novel forms of expression.     

Touch-Space: Mixed Reality Game Space Based on Ubiquitous, Tangible, and Social Computing

This paper presents a novel computer entertainment system which recaptures human touch and physical interaction with the real-world environment as essential elements of the game play, whilst also maintaining the exciting fantasy features of traditional computer entertainment. Our system called ‘Touch-Space’ is an embodied (ubiquitous, tangible, and social) computing based Mixed Reality (MR) game space which regains the physical and social aspects of traditional game play. In this novel game space, the real-world environment is an essential and intrinsic game element, and the human’s physical context influences the game play. It also provides the full spectrum of game interaction experience ranging from the real physical environment (human to human and human to physical world interaction), to augmented reality, to the virtual environment. It allows tangible interactions between players and virtual objects, and collaborations between players in different levels of reality. Thus, the system re-invigorates computer entertainment systems with social human-to-human and human-to-physical touch interactions. Correspondence to: Professor A. Cheok, National University of Singapore, 10 Kent Ridge Crescent, Singapore 119260. Email: adriancheok@nus.edu.sg     

Sports over a Distance

Sport is a domain full of movement-based interactions. These interactions typically have positive health effects as well as an impact on social bonding. We have investigated ways in which computer augmented devices can lead to new sport experiences and explored opportunities to combine physical activities with remote social bonding. Three prototypes have been implemented which showcase movement-based interaction in sports. “Breakout for Two” allows geographically distant users to play a physically exhausting ball game together. “FlyGuy” gives users a hang-glide experience controllable through body movement. “Push’N’Pull” uses isometric exercise equipment over a network to encourage users to complete a cooperative game whilst performing intense muscular actions. A comparison of these applications shows that such movement-based interaction in a networked environment allows players in different locations to achieve a work out and also to socialize. Based on these projects, we conclude with practical design implications for future Exertion Interfaces.     

Collaboration Support in Co-located Collaborative Systems for Users with Autism Spectrum Disorders: A Systematic Literature Review

ABSTRACT

Individuals with Autism Spectrum Disorders (ASD) often display difficulty in social interaction, verbal and non-verbal communication skills. They often have problems with recognizing and interpreting gestures and mental states of others, which restricts their capacity to understand implicit information essential to the social awareness and, consequently, the performance of collaborative activities in face-to-face situations (co-located collaboration). This study aims to identify, using a systematic literature review, how co-located collaborative systems designed for individuals with ASD provide the users with collaboration support mechanisms to encourage the performance of collaborative activities and interaction with their partners. Twenty-six studies from six electronic databases between the years 2006 and 2018 were selected and analyzed. Results evidenced the inclusion of different strategies to enforce or stimulate the collaboration among users, although there is a lack of resources for collaboration support as well as of studies designed specifically for individuals with severe ASD.     

Child–robot interaction across cultures: How does playing a game with a social robot compare to playing a game alone or with a friend?

The present study investigates how children from two different cultural backgrounds (Pakistani, Dutch) and two different age groups (8 and 12 year olds) experience interacting with a social robot (iCat) during collaborative game play. We propose a new method to evaluate children’s interaction with such a robot, by asking whether playing a game with a state-of-the-art social robot like the iCat is more similar to playing this game alone or with a friend. A combination of self-report scores, perception test results and behavioral analyses indicate that child–robot interaction in game playing situations is highly appreciated by children, although more by Pakistani and younger children than by Dutch and older children. Results also suggest that children enjoyed playing with the robot more than playing alone, but enjoyed playing with a friend even more. In a similar vein, we found that children were more expressive in their non-verbal behavior when playing with the robot than when they were playing alone, but less expressive than when playing with a friend. Our results not only stress the importance of using new benchmarks for evaluating child–robot interaction but also highlight the significance of cultural differences for the design of social robots.     

Virtual peers as partners in storytelling and literacy learning

Abstract   Literacy learning — learning how to read and write — begins long before children enter school. One of the key skills to reading and writing is the ability to represent thoughts symbolically and share them in language with an audience who may not necessarily share the same temporal and spatial context. Children learn and practice these important language skills everyday, telling stories with the peers and adults around them. In particular, storytelling in the context of peer collaboration provides a key environment for children to learn language skills important for literacy. In light of this, an embodied conversational agent, Sam , who tells stories collaboratively with children was designed. Sam looks like a peer for pre-school children, but tells stories in a developmentally advanced way, modelling narrative skills important for literacy. Results demonstrated that children who played with the virtual peer told stories that more closely resembled the virtual peer's linguistically advanced stories: using more quoted speech and temporal and spatial expressions. In addition, children listened to Sam's stories carefully, assisting her and suggesting improvements. The potential benefits of having technology play a social role in young children's literacy learning is discussed.     

Using multimedia to reveal the hidden code of everyday behaviour to children with autistic spectrum disorders (ASDs)

This paper describes a framework which was developed for carers (teachers and parents) to help them create personalised social stories for children with autistic spectrum disorders (ASDs). It explores the social challenges experienced by individuals with ASDs and outlines an intervention aimed at revealing the hidden code that underpins social interactions to them. It focuses on how technology can facilitate the implementation of this intervention by utilising multimedia-authoring tools. Findings from a case study, involving seven carers over a five-month period, are encouraging revealing that the participants quickly learned the basics of the intervention and either adapted or created social stories. They found the framework easy to use and sufficiently flexible to cater for children with social interaction, communication and behaviour problems.     

Impact of mainstream classroom setting on attention of children with autism spectrum disorder: an eye-tracking study

It has long been reported that children with autism spectrum disorder (ASD) exhibit attention difficulties while learning. They tend to focus on irrelevant information and can easily be distracted. As a result, they are often confined to a one-to-one teaching environment, with fewer distractions and social interactions than would be present in a mainstream educational setting. In recent years, inclusive mainstream schools have been growing in popularity due to government policies on equality rights. Therefore, it is crucial to investigate attentional patterns of children with ASD in mainstream schools. This study aims to explore the attentional behaviors of children with ASD in a virtual reality simulated classroom. We analyzed four eye-gaze behaviors and performance scores of 45 children: children with ASD (ASD n = 20) and typically developing children (TD n = 25) when performing attention tasks. The gaze behaviors included time to first fixate (TTFF), first fixation duration (FFD), average fixation duration (AFD) and the sum of fixation count (SFC) on fourteen areas of interest (AOIs) in the classroom. Our results showed that children with ASD exhibit similar gaze behaviors to TD children, but with significantly lower performance scores and SFC on the target AOI. These findings showed that classroom settings can influence attentional patterns and the academic performance of children with ASD. Further studies are needed on different modalities for supporting the attention of children with ASD in a mainstream setting.

     

A cognitive-behavioral model of problematic online gaming in adolescents aged 12–22 years

The aim of this study was to apply the cognitive behavioral model of problematic Internet use to the context of online game use to obtain a better understanding of problematic use of online games and its negative consequences. In total, 597 online game playing adolescents aged 12–22 years participated in this study. Results showed that the cognitive behavioral model of problematic Internet use can also be used in the context of online game use. More specifically, preference for online social interaction, mood regulation and deficient self-regulation appeared to play an important role in predicting negative outcomes from problematic online game use. Together, these cognitions and behaviors explained 79% of the variance of negative outcomes scores. These findings can be used to develop strategies that aim at reducing problematic online game behavior and its negative consequences.