Designing and Evaluating an Adaptive Spoken Dialogue System

Spoken dialogue system performance can vary widely for different users, as well for the same user during different dialogues. This paper presents the design and evaluation of an adaptive version of TOOT, a spoken dialogue system for retrieving online train schedules. Based on rules learned from a set of training dialogues, adaptive TOOT constructs a user model representing whether the user is having speech recognition problems as a particular dialogue progresses. Adaptive TOOT then automatically adapts its dialogue strategies based on this dynamically changing user model. An empirical evaluation of the system demonstrates the utility of the approach.     

Designing habitable dialogues for speech-based interaction with computers

Habitability refers to the match between the language people employ when using a computer system and the language that the system can accept. In this paper, the concept of “habitability” is explored in relation to the design of dialogues for speech-based systems. Two studies investigating the role of habitability in speech systems for banking applications are reported. The first study employed a speech-driven automated teller machine (ATM), using a visual display to indicate available vocabulary. Users made several distinct types of error with this system, indicating that habitability in speech systems cannot be achieved simply by displaying the input language. The second study employed a speech input/speech output home banking application, in which system constraints were indicated by either a spoken menu of words or a “query-style” prompt (e.g. “what service do you require?”). Between-subjects comparisons of these two conditions confirmed that the “menu-style” dialogue was rated as more habitable than the “query-style”. It also led to fewer errors, and was rated as easier to use, suggesting that habitability is a key issue in speech system usability. Comparison with the results of the first study suggests that for speech input, spoken menu prompts may be more habitable than similar menus shown on a visual display. The implications of these results to system design are discussed, and some initial dialogue design recommendations are presented.     

Towards incremental speech generation in conversational systems

This paper presents a model of incremental speech generation in practical conversational systems. The model allows a conversational system to incrementally interpret spoken input, while simultaneously planning, realising and self-monitoring the system response. If these processes are time consuming and result in a response delay, the system can automatically produce hesitations to retain the floor. While speaking, the system utilises hidden and overt self-corrections to accommodate revisions in the system. The model has been implemented in a general dialogue system framework. Using this framework, we have implemented a conversational game application. A Wizard-of-Oz experiment is presented, where the automatic speech recognizer is replaced by a Wizard who transcribes the spoken input. In this setting, the incremental model allows the system to start speaking while the user's utterance is being transcribed. In comparison to a non-incremental version of the same system, the incremental version has a shorter response time and is perceived as more efficient by the users.     

Towards building intelligent speech interfaces through the use of more flexible,robust and natural dialogue management solutions

In this paper a Bayesian Networks-based solution for dialogue modelling is presented. This solution is combined with carefully designed contextual information handling strategies. With the purpose of validating these solutions, and introducing a spoken dialogue system for controlling a Hi-Fi audio system as the selected prototype, a real-user evaluation has been conducted. Two different versions of the prototype are compared. Each version corresponds to a different implementation of the algorithm for the management of the actuation order, the algorithm for deciding the proper order to carry out the actions required by the user. The evaluation is carried out in terms of a battery of both subjective and objective metrics collected from speakers interacting with the Hi-Fi audio box through predefined scenarios. Defined metrics have been specifically adapted to measure: first, the usefulness and the actual relevance of the proposed solutions, and, secondly, their joint performance through their intelligent combination mainly measured as the level achieved with regard to the user satisfaction. A thorough and comprehensive study of the main differences between both approaches is presented. Two-way analysis of variance (ANOVA) tests are also included to measure the effects of both: the system used and the type of scenario factors, simultaneously. Finally, the effect of bringing this flexibility, robustness and naturalness into our home dialogue system is also analyzed through the results obtained. These results show that the intelligence of our speech interface has been well perceived, highlighting its excellent ease of use and its good acceptance by users, therefore validating the approached dialogue management solutions and demonstrating that a more natural, flexible and robust dialogue is possible thanks to them.     

Evaluating spoken dialogue agents with PARADISE: Two case studies

This paper presents PARADISE (PARAdigm for DIalogue System Evaluation), a general framework for evaluating and comparing the performance of spoken dialogue agents. The framework decouples task requirements from an agent's dialogue behaviours, supports comparisons among dialogue strategies, enables the calculation of performance over subdialogues and whole dialogues, specifies the relative contribution of various factors to performance, and makes it possible to compare agents performing different taks by normalizing for task complexity. After presenting PARADISE, we illustrate its application to two different spoken dialogue agents. We show how to derive a performance function for each agent and how to generalize results across agents. We then show that once such a performance function has been derived, it can be used both for making predictions about future versions of an agent, and as feedback to the agent so that the agent can learn to optimize its behaviour based on its experiences with users over time.     

Making it easier for older people to talk to smart homes: the effect of early help prompts

It is well known that help prompts shape how users talk to spoken dialogue systems. This study investigated the effect of help prompt placement on older users’ interaction with a smart home interface. In the dynamic help condition, help was only given in response to system errors; in the inherent help condition, it was also given at the start of each task. Fifteen older and sixteen younger users interacted with a smart home system using two different scenarios. Each scenario consisted of several tasks. The linguistic style users employed to communicate with the system (interaction style) was measured using the ratio of commands to the overall utterance length (keyword ratio) and the percentage of content words in the user’s utterance that could be understood by the system (shared vocabulary). While the timing of help prompts did not affect the interaction style of younger users, it was early task-specific help supported older users in adapting their interaction style to the system’s capabilities. Well-placed help prompts can significantly increase the usability of spoken dialogue systems for older people.     

口语对话系统中对话管理方法研究综述

口语对话系统是人机交互领域的核心技术,也是实现和谐人机交互的重要途径,具有重大的研究意义和应用价值,其中的各项理论和技术的研究进展一直备受关注.较为全面深入地总结了对话管理及口语对话系统的研究进展和现状.首先阐述了口语对话系统中的主要研究问题,包括系统各模块的研究内容与关键技术、系统的可移植性和鲁棒性设计等;然后从理论模型、研究进展及可用性等角度系统地剖析了现有的多种口语对话管理策略;最后展望了未来可能的研究方向和亟待解决的问题.     

Real user evaluation of a POMDP spoken dialogue system using automatic belief compression

This article describes an evaluation of a POMDP-based spoken dialogue system (SDS), using crowdsourced calls with real users. The evaluation compares a “Hidden Information State” POMDP system which uses a hand-crafted compression of the belief space, with the same system instead using an automatically computed belief space compression. Automatically computed compressions are a way of introducing automation into the design process of statistical SDSs and promise a principled way of reducing the size of the very large belief spaces which often make POMDP approaches intractable. This is the first empirical comparison of manual and automatic approaches on a problem of realistic scale (restaurant, pub and coffee shop domain) with real users. The evaluation took 2193 calls from 85 users. After filtering for minimal user participation the two systems were compared on more than 1000 calls.     

Evaluation of a hierarchical reinforcement learning spoken dialogue system

We describe an evaluation of spoken dialogue strategies designed using hierarchical reinforcement learning agents. The dialogue strategies were learnt in a simulated environment and tested in a laboratory setting with 32 users. These dialogues were used to evaluate three types of machine dialogue behaviour: hand-coded, fully-learnt and semi-learnt. These experiments also served to evaluate the realism of simulated dialogues using two proposed metrics contrasted with ‘Precision-Recall’. The learnt dialogue behaviours used the Semi-Markov Decision Process (SMDP) model, and we report the first evaluation of this model in a realistic conversational environment. Experimental results in the travel planning domain provide evidence to support the following claims: (a) hierarchical semi-learnt dialogue agents are a better alternative (with higher overall performance) than deterministic or fully-learnt behaviour; (b) spoken dialogue strategies learnt with highly coherent user behaviour and conservative recognition error rates (keyword error rate of 20%) can outperform a reasonable hand-coded strategy; and (c) hierarchical reinforcement learning dialogue agents are feasible and promising for the (semi) automatic design of optimized dialogue behaviours in larger-scale systems.     

Usefulness,localizability, humanness,and language-benefit: additional evaluation criteria for natural language dialogue systems

Human–computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue system evaluation. We present some of the different evaluation techniques used in natural language dialogue systems, including black box and glass box, comparative, quantitative, and qualitative evaluation. Four aspects of NLP dialogue system evaluation are often overlooked: “usefulness” in terms of a user’s qualitative needs, “localizability” to new genres and languages, “humanness” or “naturalness” compared to human–human dialogues, and “language benefit” compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers.     

Designing and evaluating a wizarded uncertainty-adaptive spoken dialogue tutoring system

We describe the design and evaluation of two different dynamic student uncertainty adaptations in wizarded versions of a spoken dialogue tutoring system. The two adaptive systems adapt to each student turn based on its uncertainty, after an unseen human “wizard” performs speech recognition and natural language understanding and annotates the turn for uncertainty. The design of our two uncertainty adaptations is based on a hypothesis in the literature that uncertainty is an “opportunity to learn”; both adaptations use additional substantive content to respond to uncertain turns, but the two adaptations vary in the complexity of these responses. The evaluation of our two uncertainty adaptations represents one of the first controlled experiments to investigate whether substantive dynamic responses to student affect can significantly improve performance in computer tutors. To our knowledge we are the first study to show that dynamically responding to uncertainty can significantly improve learning during computer tutoring. We also highlight our ongoing evaluation of our uncertainty-adaptive systems with respect to other important performance metrics, and we discuss how our corpus can be used by the wider computer speech and language community as a linguistic resource supporting further research on effective affect-adaptive spoken dialogue systems in general.     

A dialogue‐based (natural language) platform for human interaction with an intelligent mining machine

A mining environment is one of the most complex and unstructured in the manufacturing industry. In order to minimize the problems associated with these characteristics, recent strategies in mining operation are to automate the task performance and to design mining machines that are “intelligent.” These strategies, among other things, will require that the human operators and the machine interact and collaborate to perform tasks in a symbiotic manner. To achieve this, a prototype dialogue‐based interaction platform has been developed. The platform known as OASIP is a knowledge‐based system driven by the operator‐planned actions and behaviors known as intentions. OASIP is an adaptive system which exploits several sources of environmental knowledge from built‐in blackboard cells.     

What Is Good Help? Responses to Solicited and Unsolicited Assistance

Users rarely ask for help, and they tend to respond negatively to unsolicited help, even when they can benefit from it. It is not clear whether this is due to the way systems provide help or whether people in general dislike unsolicited help. To address this issue the authors studied responses to solicited and unsolicited help from a human adviser regarding the use of an unfamiliar e-mail system. Novice users (the advisees) received spoken advice from distant experienced users (the advisers) upon the advisee’s request (the “pull” condition”) or the adviser’s initiative (the “push” condition”). Unsolicited advice helped performance more than advice requested by the advisee, but only for unfamiliar tasks and especially for difficult tasks. Although advisees perceived unsolicited advice as helpful, they were not interested in receiving such advice in the future. This study demonstrates that although advice can help to improve performance, it may still not be welcome, even when provided by a person.     

Speech-based interaction in multitask conditions: impact of prompt modality

Speech-based interaction is often recognized as appropriate for hands-busy, eyes-busy multitask situations. The objective of this study was to explore prompt-guided speech-based interaction and the impact of prompt modality on overall performance in such situations. A dual-task paradigm was employed, with tracking as a primary task and speech-based data input as a secondary task. There were three tracking conditions: no tracking, basic, and difficult tracking. Two prompt modalities were used for the speech interaction: a dialogue with spoken prompts and a dialogue with visual prompts. Data entry duration was longer with the speech prompts than with the visual prompts, regardless of whether or not there was tracking or its level of difficulty. However, when tracking was difficult, data entry duration was similar for both spoken and visual prompts. Tracking performance was also affected by the prompt modality, with poorer performance obtained when the prompts were visual. The findings are discussed in terms of multiple resource theory and the possible implications for speech-based interactions in multitask situations. Actual or potential applications of this research include the design of speech-based dialogues for multitask situations such as driving and other hands-busy, eyes-busy situations.     

Evolution of the Information-Retrieval System for Blind and Visually-Impaired People

Blind and visually-impaired people face many problems in interacting with information retrieval systems. State-of-the-art spoken language technology offers potential to overcome many of them. In the mid-nineties our research group decided to develop an information retrieval system suitable for Slovene-speaking blind and visually-impaired people. A voice-driven text-to-speech dialogue system was developed for reading Slovenian texts obtained from the Electronic Information System of the Association of Slovenian Blind and Visually Impaired Persons Societies. The evolution of the system is presented. The early version of the system was designed to deal explicitly with the Electronic Information System where the available text corpora are stored in a plain text file format without any, or with just some, basic non-standard tagging. Further improvements to the system became possible with the decision to transfer the available corpora to the new web portal, exclusively dedicated to blind and visually-impaired users. The text files were reformatted into common HTML/XML pages, which comply with the basic recommendations set by the Web Access Initiative. In the latest version of the system all the modules of the early version are being integrated into the user interface, which has some basic web-browsing functionalities and a text-to-speech screen-reader function controlled by the mouse as well.     

What should your speech system say?

Even as telephone based spoken language dialogue systems (SLDS) are becoming commercially available, developers can benefit from guidelines designed to help remove dialogue problems as early as possible in the design life cycle. SLDS designers generally rely on a Wizard of Oz (WOZ) simulation technique to ensure that the system's dialogue facilitates user interaction as much as possible. In a WOZ simulation, users are made to believe they are interacting with a real system, when in fact they are interacting with a hidden researcher. These researchers record, transcribe, and analyze the dialogues, and then use the results to improve the dialogue in the SLDS being developed. Using current methods, dialogue designers must be both very careful and very lucky, or interaction problems will remain during the implementation and testing stages. We have found that a sound, comprehensive set of dialogue design guidelines is an effective tool to support systematic development and evaluation during early SLDS design. We believe guidelines could significantly reduce development time by reducing the need for lengthy WOZ experimentation, controlled user testing, and field trial cycles     

On the dynamic adaptation of language models based on dialogue information

We present an approach to adapt dynamically the language models (LMs) used by a speech recognizer that is part of a spoken dialogue system. We have developed a grammar generation strategy that automatically adapts the LMs using the semantic information that the user provides (represented as dialogue concepts), together with the information regarding the intentions of the speaker (inferred by the dialogue manager, and represented as dialogue goals). We carry out the adaptation as a linear interpolation between a background LM, and one or more of the LMs associated to the dialogue elements (concepts or goals) addressed by the user. The interpolation weights between those models are automatically estimated on each dialogue turn, using measures such as the posterior probabilities of concepts and goals, estimated as part of the inference procedure to determine the actions to be carried out. We propose two approaches to handle the LMs related to concepts and goals. Whereas in the first one we estimate a LM for each one of them, in the second one we apply several clustering strategies to group together those elements that share some common properties, and estimate a LM for each cluster. Our evaluation shows how the system can estimate a dynamic model adapted to each dialogue turn, which helps to significantly improve the performance of the speech recognition, which leads to an improvement in both the language understanding and the dialogue management tasks.     

Evaluating the Markov assumption in Markov Decision Processes for spoken dialogue management

The goal of dialogue management in a spoken dialogue system is to take actions based on observations and inferred beliefs. To ensure that the actions optimize the performance or robustness of the system, researchers have turned to reinforcement learning methods to learn policies for action selection. To derive an optimal policy from data, the dynamics of the system is often represented as a Markov Decision Process (MDP), which assumes that the state of the dialogue depends only on the previous state and action. In this article, we investigate whether constraining the state space by the Markov assumption, especially when the structure of the state space may be unknown, truly affords the highest reward. In simulation experiments conducted in the context of a dialogue system for interacting with a speech-enabled web browser, models under the Markov assumption did not perform as well as an alternative model which classifies the total reward with accumulating features. We discuss the implications of the study as well as its limitations.     

User Modeling in Spoken Dialogue Systems to Generate Flexible Guidance

We address the issue of appropriate user modeling to generate cooperative responses to users in spoken dialogue systems. Unlike previous studies that have focused on a user’s knowledge, we propose more generalized modeling. We specifically set up three dimensions for user models: the skill level in use of the system, the knowledge level about the target domain, and the degree of urgency. Moreover, the models are automatically derived by decision tree learning using actual dialogue data collected by the system. We obtained reasonable accuracy in classification for all dimensions. Dialogue strategies based on user modeling were implemented on the Kyoto City Bus Information System that was developed at our laboratory. Experimental evaluations revealed that the cooperative responses adapted to each subject type served as good guides for novices without increasing the duration dialogue lasted for skilled users.     

An Efficient Two-Phase Model for Computing Influential Nodes in Social Networks Using Social Actions

在社会网络的影响的测量在数据采矿社区收到了很多注意。影响最大化指发现尽量利用信息或产品采纳的有影响的用户的过程。在真实设置,在一个社会网络的一个用户的影响能被行动的集合建模(例如,份额,重新鸣叫,注释) 在其出版物以后由网络的另外的用户表现了。就我们的知识而言,在文学的所有建议模型同等地对待这些行动。然而,它是明显的一工具少些比一样的出版的份额影响的一份出版物相似。这建议每个行动有它影响的自己的水平(或重要性) 。在这份报纸,我们建议一个模型(叫的社会基于行动的影响最大化模型, SAIM ) 为在社会网络的影响最大化。在 SAIM,行动没在测量一个个人的影响力量同等地被考虑,并且它由二主要的步组成。在第一步,我们在社会网络计算每个个人的影响力量。这影响力量用 PageRank 从用户行动被计算。在这步的结束,我们得到每个节点被它的影响力量在标记的一个加权的社会网络。在 SAIM 的第二步,我们计算一个新概念说出 influence-BFS 树的使用的有影响的节点的一个最佳的集合。在大规模真实世界、合成的社会网络上进行的实验在计算揭示我们的模型 SAIM 的好表演,在可接受的时间规模,允许信息的最大的传播的有影响的节点的一个最小的集合。