Usefulness,localizability, humanness,and language-benefit: additional evaluation criteria for natural language dialogue systems

Human–computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue system evaluation. We present some of the different evaluation techniques used in natural language dialogue systems, including black box and glass box, comparative, quantitative, and qualitative evaluation. Four aspects of NLP dialogue system evaluation are often overlooked: “usefulness” in terms of a user’s qualitative needs, “localizability” to new genres and languages, “humanness” or “naturalness” compared to human–human dialogues, and “language benefit” compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers.     

主动联想型对话机器人的设计与实现

人机对话系统通常仅是用户命令的执行者,很少能主动发起对话,或依据对话以外的内容展开个性化的人机交互.本文提出了联想型对话机器人的概念和对应设计方案,即通过融合用户提问和用户个人特征的方式产生对话内容,这样的对话系统不仅可以主动发出对话邀请,也可以个性化地回答用户问题,提升人机对话的满意度.文章首先介绍了联想型对话机器人...     

An Assessment of Written/Interactive Dialogue for Information Retrieval Applications

For the foreseeable future, natural language access to databases and other information systems will require the exchange of written messages between the system and user. Such written/interactive dialogue is unique, having the qualities of both written text and spoken discourse yet, also, differing significantly from both. In the present study, we used "Wizard of Oz" techniques to elicit written/interactive dialogue for information retrieval purposes. Our objectives in doing this were (a) to assess the general nature and prevalence of linguistic and dialogue phenomena within the written/interactive register and (b) to determine the impact of user interface shortcuts, such as precanned messages and patterned output, on both the complexity of written/interactive dialogue and general measures of user satisfaction. Our findings indicate that written/interactive dialogue for information retrieval would be very feasible. In spite of slow system response times, subjective reactions from the users were positive, the size of the lexicon used in the dialogues was small, the dialogues decomposed readily into hierarchical structures, and the number and distribution of anaphors were also rather reasonable. Two independent variables were also manipulated: (a) the degree of prefamiliarization given to participants about the base of travel information they would be accessing (i.e., the primer variable) and (b) the degree of constraint on the Wizard's ability to formulate natural language responses to the user (i.e., the natural language output variable). Failure to provide either a primer or a realistic, human natural language output made the dialogues more complex in a number of different ways.     

Evaluation of a hierarchical reinforcement learning spoken dialogue system

We describe an evaluation of spoken dialogue strategies designed using hierarchical reinforcement learning agents. The dialogue strategies were learnt in a simulated environment and tested in a laboratory setting with 32 users. These dialogues were used to evaluate three types of machine dialogue behaviour: hand-coded, fully-learnt and semi-learnt. These experiments also served to evaluate the realism of simulated dialogues using two proposed metrics contrasted with ‘Precision-Recall’. The learnt dialogue behaviours used the Semi-Markov Decision Process (SMDP) model, and we report the first evaluation of this model in a realistic conversational environment. Experimental results in the travel planning domain provide evidence to support the following claims: (a) hierarchical semi-learnt dialogue agents are a better alternative (with higher overall performance) than deterministic or fully-learnt behaviour; (b) spoken dialogue strategies learnt with highly coherent user behaviour and conservative recognition error rates (keyword error rate of 20%) can outperform a reasonable hand-coded strategy; and (c) hierarchical reinforcement learning dialogue agents are feasible and promising for the (semi) automatic design of optimized dialogue behaviours in larger-scale systems.     

Testing the performance of spoken dialogue systems by means of an artificially simulated user

This paper proposes a new technique to test the performance of spoken dialogue systems by artificially simulating the behaviour of three types of user (very cooperative, cooperative and not very cooperative) interacting with a system by means of spoken dialogues. Experiments using the technique were carried out to test the performance of a previously developed dialogue system designed for the fast-food domain and working with two kinds of language model for automatic speech recognition: one based on 17 prompt-dependent language models, and the other based on one prompt-independent language model. The use of the simulated user enables the identification of problems relating to the speech recognition, spoken language understanding, and dialogue management components of the system. In particular, in these experiments problems were encountered with the recognition and understanding of postal codes and addresses and with the lengthy sequences of repetitive confirmation turns required to correct these errors. By employing a simulated user in a range of different experimental conditions sufficient data can be generated to support a systematic analysis of potential problems and to enable fine-grained tuning of the system.     

An Approach to User Interface Specification with Attribute Grammars

An approach to supporting user interfaces using an attribute grammar combined with an event model is described.The main emphasis is how to represent a multi-thread dialogue model in direct manipulation user interfaces.It is shown that control sequence within dialogues,communication with other dialogues, and some computations for applications can be specified with a syntactic and semantic notation.The attribute grammar specification can be implemented using an attribute grammar intepreter embedded in the run-time structure supporting communication,synchronization,and dialogue executon.     

Shaping user input: a strategy for natural language dialogue design

Traditional approaches to natural language dialogue interface design have adopted ordinary human-human conversation as the model for online human-computer interaction. The attempt to deal with all the subtleties of natural dialogues, such as topic focus, coherence, ellipsis, pronominal reference, etc. has resulted in prototype systems that are enormously complex and computationally expensive.

In a series of experiments^†, we explored ways of minimizing the processing burden of a dialogue system by channeling user input towards a more tractable, though still natural, form of Englishlanguage questions. Through linking a pair of terminals, we presented subjects with two different dialogue styles as a framework for online help in the domain of word-processing. The first dialogue style involved ordinary conversational format. The second style involved a simulation of an automated dialogue system, including apparent processing restrictions and ‘system process messages’ to inform the subject of the steps taken by the system during query analysis. In both cases human tutors played the role of the help system. After each dialogue session, subjects were interviewed to determine their assessments of the naturalness and usability of the dialogue interface.

We found that user input became more tractable to parsing and query analysis as the dialogue style became more formalized, yet the subjective assessment of naturalness and usability remained fairly constant. This suggests that techniques for channeling user input in a dialogue system may be effectively employed to reduce processing demands without compromising the benefits of a natural language interface. Theoretically, this data lends support to the hypothesis that unrestricted human-human conversation is not the most appropriate model for the design of human-computer dialogue interfaces.     

What should your speech system say?

Even as telephone based spoken language dialogue systems (SLDS) are becoming commercially available, developers can benefit from guidelines designed to help remove dialogue problems as early as possible in the design life cycle. SLDS designers generally rely on a Wizard of Oz (WOZ) simulation technique to ensure that the system's dialogue facilitates user interaction as much as possible. In a WOZ simulation, users are made to believe they are interacting with a real system, when in fact they are interacting with a hidden researcher. These researchers record, transcribe, and analyze the dialogues, and then use the results to improve the dialogue in the SLDS being developed. Using current methods, dialogue designers must be both very careful and very lucky, or interaction problems will remain during the implementation and testing stages. We have found that a sound, comprehensive set of dialogue design guidelines is an effective tool to support systematic development and evaluation during early SLDS design. We believe guidelines could significantly reduce development time by reducing the need for lengthy WOZ experimentation, controlled user testing, and field trial cycles     

A meta-rule approach to flexible plan recognition in dialogue

Although a number of researchers have demonstrated that reasoning on a model of the user's plans and goals is helpful in language understanding and response generation, current models of plan inference cannot handle naturally occurring dialogue. This paper argues that model building from less than ideal dialogues has a great deal in common with processing ill-formed input. It defines well-formedness constraints for information-seeking dialogues and contends that strategies for interpreting ill-formed input can be applied to the problem of modeling the user's plan during an ill-formed dialogue. It presents a meta-rule approach for hypothesizing the cause of dialogue ill-formedness, and describes meta-rules for relaxing the plan inference process and enabling the consideration of alternative hypotheses. The advantages of this approach are that it provides a unified framework for handling both well-formed and ill-formed dialogue, avoids unnatural interpretations when the dialogue is proceeding smoothly, and facilitates a nonmonotonic plan recognition system.Rhonda Eller is a Ph.D. candidate in Computer Science at the University of Delaware. She received her B.S. in Computer Science from Millersville University of Pennsylvania in 1987, and her M.S. degree in the same field from the University of Delaware. Her primary interests lie in the areas of natural language processing, plan recognition, and user modelling. This paper summarizes the current state of her thesis work on repair of an incorrectly inferred plan for a user.Sandra Carberry is an associate professor of computer science at the University of Delaware. Her research interests include discourse understanding, user modeling, planning and plan recognition, and intelligent natural language interfaces. This paper describes work that is part of an ongoing research project to develop a robust model of plan recognition.     

User-Tailored Planning of Mixed Initiative Information-Seeking Dialogues

Intelligent dialogue systems usually concentrate on user support at the level of the domain of discourse, following a plan-based approach. Whereas this is appropriate for collaborative planning tasks, the situation in interactive information retrieval systems is quite different: there is no inherent plan-goal hierarchy, and users are known to often opportunistically change their goals and strategies during and through interaction. We need to allow for mixed-initiative retrieval dialogues, where the system evaluates the user's individual dialogue behavior and performs situation-dependent interpretation of user goals, to determine when to take the initiative and to change the control of the dialogue, e.g., to propose (new) problem-solving strategies to the user. In this article, we present the dialogue planning component of a concept- oriented, logic-based retrieval system (MIRACLE). Users are guided through the global stages of the retrieval interaction but may depart, at any time, from this guidance and change the direction of the dialogue. When users submit ambiguous queries or enter unexpected dialogue control acts, abductive reasoning is used to generate interpretations of these user inputs in light of the dialogue history and other internal knowledge sources. Based on these interpretations, the system initiates a short dialogue offering the user suitable options and strategies for proceeding with the retrieval dialogue. Depending on the user's choice and constraints resulting from the history, the system adapts its strategy accordingly.     

开拓面向图象语义的人机对话

建立面对图的对话者形式的人工机通信的关锓在于产使双方的对话行为符合人对图的语义理论和语义运用。     

Multi-agent human–machine dialogue: issues in dialogue management and referring expression semantics

This paper presents an implemented model of dialogue management for simple dialogues involving multiple speakers. In our model, the user is one speaker, and the system ‘plays’ a number of other speakers. We present a number of principles governing dialogue management in such cases, which relate to turn-taking and the identification of the addressees of utterances. We also consider how to extend a syntactic and semantic treatment of first- and second-person personal pronouns, and of addressee terms, in order to deal with the multi-speaker scenario. We give some examples of our current system, and conclude by outlining some extensions of the system to include disagreements, interruptions, and private communication between subgroups of dialogue participants.     

Adaptation and user expertise modelling in AthosMail

This article describes the User Model component of AthosMail, a speech-based interactive e-mail application developed in the context of the EU project DUMAS. The focus is on the system’s adaptive capabilities and user expertise modelling, exemplified through the User Model parameters dealing with initiative and explicitness of the system responses. The purpose of the conducted research was to investigate how the users could interact with a system in a more natural way, and the two aspects that mainly influence the system’s interaction capabilities, and thus the naturalness of the dialogue as a whole, are considered to be the dialogue control and the amount of information provided to the user. The User Model produces recommendations of the system’s appropriate reaction depending on the user’s observed competence level, monitored and computed on the basis of the user’s interaction with the system. The article also discusses methods for the evaluation of adaptive user models and presents results from the AthosMail evaluation.The research was done while the author was affiliated with the University of Art and Design Helsinki as the scientific coordinator of the DUMAS project.     

Towards an Object-Oriented Interaction Model for Graphics User Interfaces

The increasing importance of the User Interface Management System THESEUS stimulated the development of a formalized graphics input model that is based on the THESEUS input concepts. This paper presents cooperative work by the Zentrum fur Gra-phische Datenverarbeitung (ZGDV), West Germany and IST/INESC, Portugal which aims to formalize, generalize and enhance the THESEUS input model and transfer it to a clean object-oriented approach. The main goal was to establish concepts for specifying and managing user interfaces tailored to the requirements of graphics dialogues; among these concepts are the diversity of input devices, many different kinds of interaction techniques, higher complexity in dialogue structures, dynamic control over dialogue specification, a strong coupling of input and output instead of ping-pong sequencing, powerful output capabilities, support of continuous input operations and a close interlocking between user interface component and application.     

面向用户偏好建模的个性化对话推荐算法

推荐系统的目标是从物品数据库中，选择出与用户兴趣偏好相匹配的子集，缓解用户面临的“信息过载”问题。因而近年来推荐系统越来越多地应用到电商、社交等领域，展现出巨大的商业潜力。传统推荐系统中，系统对用户的认知往往来源于历史交互记录，例如点击率或者购买记录，这是一种隐式用户反馈。对话推荐系统能够通过自然语言与用户进行多轮对话，逐步深入挖掘其兴趣偏好，从而向对方提供高质量的推荐结果。相比于传统推荐系统，对话推荐系统主要有两方面的不同。其一，对话推荐系统能够利用自然语言与用户进行语义上连贯的多轮对话，提升了人机交互中的用户体验；其二，系统能够询问特定的问题直接获取用户的显式反馈，从而更深入地理解用户兴趣偏好，提供更可靠的推荐结果。目前已经有不少工作在不同的问题设定下对该领域进行了探索，然而尽管如此，这些工作仍仅局限于关注当前正在进行的对话，忽视了过去交互记录中蕴涵的丰富信息，导致对用户偏好建模的不充分。为了解决这个问题，本文提出了一个面向用户偏好建模的个性化对话推荐算法框架，通过双线性模型注意力机制与自注意力层次化编码结构进行用户偏好建模，从而完成对候选物品的排序与推荐。本文设计的模型结构能够在充分利用用户历史对话信息的同时，权衡历史对话与当前对话两类数据的重要性。丰富的用户相关信息来源使得推荐结果在契合用户个性化偏好的同时，更具备多样性，从而缓解“信息茧房”等现象带来的不良影响。基于公开数据集的实验表明了本文方法在个性化对话推荐任务上的有效性。     

An inquiry dialogue system

The majority of existing work on agent dialogues considers negotiation, persuasion or deliberation dialogues; we focus on inquiry dialogues, which allow agents to collaborate in order to find new knowledge. We present a general framework for representing dialogues and give the details necessary to generate two subtypes of inquiry dialogue that we define: argument inquiry dialogues allow two agents to share knowledge to jointly construct arguments; warrant inquiry dialogues allow two agents to share knowledge to jointly construct dialectical trees (essentially a tree with an argument at each node in which a child node is a counter argument to its parent). Existing inquiry dialogue systems only model dialogues, meaning they provide a protocol which dictates what the possible legal next moves are but not which of these moves to make. Our system not only includes a dialogue-game style protocol for each subtype of inquiry dialogue that we present, but also a strategy that selects exactly one of the legal moves to make. We propose a benchmark against which we compare our dialogues, being the arguments that can be constructed from the union of the agents’ beliefs, and use this to define soundness and completeness properties that we show hold for all inquiry dialogues generated by our system.     

An Evidential Model for Tracking Initiative in Collaborative Dialogue Interactions

In this paper, we argue for the need to distinguish between task initiative and dialogue initiative, and present an evidential model for tracking shifts in both types of initiatives in collaborative dialogue interactions. Our model predicts the task and dialogue initiative holders for the next dialogue turn based on the current initiative holders and the effect that observed cues have on changing them. Our evaluation across various corpora shows that the use of cues consistently provides significant improvement in the system's prediction of task and dialogue initiative holders. Finally, we show how this initiative tracking model may be employed by a dialogue system to enable the system to tailor its responses to user utterances based on application domain, system's role in the domain, dialogue history, and user characteristics.     

A Dialogue Game Protocol for Agent Purchase Negotiations

We propose a dialogue game protocol for purchase negotiation dialogues which identifies appropriate speech acts, defines constraints on their utterances, and specifies the different sub-tasks agents need to perform in order to engage in dialogues according to this protocol. Our formalism combines a dialogue game similar to those in the philosophy of argumentation with a model of rational consumer purchase decision behaviour adopted from marketing theory. In addition to the dialogue game protocol, we present a portfolio of decision mechanisms for the participating agents engaged in the dialogue and use these to provide our formalism with an operational semantics. We show that these decision mechanisms are sufficient to generate automated purchase decision dialogues between autonomous software agents interacting according to our proposed dialogue game protocol.     

融合敏感词规则和字符级RCNN模型的用户意图识别

用户意图识别是基于用户对话用语识别用户的真实对话意图,是人机对话研究中的一项关键任务。针对现有用户意图识别方法的不足,提出融合敏感词规则和字符级RCNN模型的用户意图识别方法。构建敏感句子与敏感词词典,并通过规则及相似度匹配策略对特征明显的对话进行意图识别。针对类别特征不明显的对话提出深层语义分类模型,该模型以单字符串作为输入序列,利用RCNN模型构建意图分类框架,既可以避免分词结果不准确带来的错误传导问题,同时利用字符的分布向量表示方法还可以获取句子的深层语义信息。实验结果表明,该方法在两个数据集上都取得了较好的结果,明显优于传统的意图识别方法。