Usefulness,localizability, humanness,and language-benefit: additional evaluation criteria for natural language dialogue systems

Human–computer dialogue systems interact with human users using natural language. We used the ALICE/AIML chatbot architecture as a platform to develop a range of chatbots covering different languages, genres, text-types, and user-groups, to illustrate qualitative aspects of natural language dialogue system evaluation. We present some of the different evaluation techniques used in natural language dialogue systems, including black box and glass box, comparative, quantitative, and qualitative evaluation. Four aspects of NLP dialogue system evaluation are often overlooked: “usefulness” in terms of a user’s qualitative needs, “localizability” to new genres and languages, “humanness” or “naturalness” compared to human–human dialogues, and “language benefit” compared to alternative interfaces. We illustrated these aspects with respect to our work on machine-learnt chatbot dialogue systems; we believe these aspects are worthwhile in impressing potential new users and customers.     

面向虚实融合的人机交互

面向虚实融合的人机交互涉及计算机科学、认知心理学、人机工程学、多媒体技术和虚拟现实等领域，旨在提高人机交互的效率，同时响应人类认知与情感的需求，在办公教育、机器人和虚拟/增强现实设备中都有广泛应用。本文从人机交互涉及感知计算、人与机器人交互及协同、个性化人机对话和数据可视化等4个维度系统阐述面向虚实融合人机交互的发展现状。对国内外研究现状进行对比，展望未来的发展趋势。本文认为兼具可迁移与个性化的感知计算、具备用户行为深度理解的人机协同、用户自适应的对话系统等是本领域的重要研究方向。     

Testing the performance of spoken dialogue systems by means of an artificially simulated user

This paper proposes a new technique to test the performance of spoken dialogue systems by artificially simulating the behaviour of three types of user (very cooperative, cooperative and not very cooperative) interacting with a system by means of spoken dialogues. Experiments using the technique were carried out to test the performance of a previously developed dialogue system designed for the fast-food domain and working with two kinds of language model for automatic speech recognition: one based on 17 prompt-dependent language models, and the other based on one prompt-independent language model. The use of the simulated user enables the identification of problems relating to the speech recognition, spoken language understanding, and dialogue management components of the system. In particular, in these experiments problems were encountered with the recognition and understanding of postal codes and addresses and with the lengthy sequences of repetitive confirmation turns required to correct these errors. By employing a simulated user in a range of different experimental conditions sufficient data can be generated to support a systematic analysis of potential problems and to enable fine-grained tuning of the system.     

Model-checking user behaviour using interacting components

This article describes a framework to formally model and analyse human behaviour. This is shown by a simple case study of a chocolate vending machine, which represents many aspects of human behaviour. The case study is modelled and analysed using the Maude rewrite system. This work extends a previous work by Basuki which attempts to model interactions between human and machine and analyse the possibility of errors occurring in the interactions. By redesigning the interface, it can be shown that certain kinds of error can be avoided for some users. This article overcomes the limitation of Basuki’s approach by incorporating many aspects of user behaviour into a single user model, and introduces a more natural approach to model human–computer interaction.     

An Evidential Model for Tracking Initiative in Collaborative Dialogue Interactions

In this paper, we argue for the need to distinguish between task initiative and dialogue initiative, and present an evidential model for tracking shifts in both types of initiatives in collaborative dialogue interactions. Our model predicts the task and dialogue initiative holders for the next dialogue turn based on the current initiative holders and the effect that observed cues have on changing them. Our evaluation across various corpora shows that the use of cues consistently provides significant improvement in the system's prediction of task and dialogue initiative holders. Finally, we show how this initiative tracking model may be employed by a dialogue system to enable the system to tailor its responses to user utterances based on application domain, system's role in the domain, dialogue history, and user characteristics.     

Predicting user preferences of environment design: a perceptual mechanism of user interface customisation

It is a well-known fact that users vary in their preferences and needs. Therefore, it is very crucial to provide the customisation or personalisation for users in certain usage conditions that are more associated with their preferences. With the current limitation in adopting perceptual processing into user interface personalisation, we introduced the possibility of inferring interface design preferences from the user’s eye-movement behaviour. We firstly captured the user’s preferences of graphic design elements using an eye-tracker. Then we diagnosed these preferences towards the region of interests to build a prediction model for interface customisation. The prediction models from eye-movement behaviour showed a high potential for predicting users’ preferences of interface design based on the paralleled relation between their fixation and saccadic movement. This mechanism provides a novel way of user interface design customisation and opens the door for new research in the areas of human–computer interaction and decision-making.     

Affective command-based control system integrating brain signals in commands control systems

Speech interaction systems are currently highly demanded for quick hands-free interactions. Conventional speech interaction systems (SISs) are trained to the user’s voice whilst most modern systems learn from interaction experience overtime. However, because speech expresses a human computer natural interaction (HCNI) with the world, SIS design must lead to interface computer system that can receive spoken information and act appropriately upon that information. In spite of significant advancements in recent years SISs, there still remain a large number of problems which must be solved in order to successfully apply the SISs in practice and also comfortably accepted by the users. Among many other problems, problems of devising and efficient modeling are considered the primary and important step in the speech recognition deployment in hands-free applications. Meanwhile, the brain–computer interfaces (BCIs) allow users to control applications by brain activity. The work presented in this paper emphasizes an improved implementation of SIS by integrating BCI in order to associate the brain signals for a list of commands as identification criteria for each specific command for controlling the wheelchair with spoken commands.     

Designing screens for people to use easily

The use of interactive computer systems is expanding rapidly. In many applications users can communicate directly with computer-based systems through an interactive dialogue at a display terminal. However, this direct form of communication has created problems. Whereas early users of on-line systems were skilled computer professionals, today's users may have little or no training in computer technology. How then does this group communicate effectively with a computer systems?

The answer is that many users communicate only with considerable difficulty. The result is that the effectiveness of many systems is very much less than expected. The reason for this is simple: these systems are not very good at communicating with their users.

The most common communications bridge between a person and a computer system is a display screen. A well-designed screen format can enhance user productivity, eliminate or reduce user input errors, and promote end user satisfaction. A poorly designed screen has the opposite effect: it will decrease human processing speed, provoke user mistakes, and complicate computer operations.

Screen design requires the same care as designing the overall application. It employs the same techniques as planning and preparing good user documents. Screens, like documents, must be easy to understand and easy to use. To achieve these objectives, screen designers must make good choices regarding three ease-of-use parameters: format; screen content and layout; and style. This paper describes how standards may be established for each of these parameters in a proposed application or system. Futher, an interactive tutorial computer program has been developed (using Bricklin's DEMO program) which may be used to illustrate the application of these standards.     

Reinforcement learning for parameter estimation in statistical spoken dialogue systems

Reinforcement techniques have been successfully used to maximise the expected cumulative reward of statistical dialogue systems. Typically, reinforcement learning is used to estimate the parameters of a dialogue policy which selects the system's responses based on the inferred dialogue state. However, the inference of the dialogue state itself depends on a dialogue model which describes the expected behaviour of a user when interacting with the system. Ideally the parameters of this dialogue model should be also optimised to maximise the expected cumulative reward.This article presents two novel reinforcement algorithms for learning the parameters of a dialogue model. First, the Natural Belief Critic algorithm is designed to optimise the model parameters while the policy is kept fixed. This algorithm is suitable, for example, in systems using a handcrafted policy, perhaps prescribed by other design considerations. Second, the Natural Actor and Belief Critic algorithm jointly optimises both the model and the policy parameters. The algorithms are evaluated on a statistical dialogue system modelled as a Partially Observable Markov Decision Process in a tourist information domain. The evaluation is performed with a user simulator and with real users. The experiments indicate that model parameters estimated to maximise the expected reward function provide improved performance compared to the baseline handcrafted parameters.     

利用多模态机器注意改善视频会议用户感受

远程视频会议的应用已很多年了,但是,目前的视频会议系统对使用者的限制较多,往往需要使用者对系统保持关注方可正常使用。如何通过改善视频会议系统的人机交互模式来改善用户感受是视频会议系统进一步完善的重要途径。多模态机器注意技术来自于仿生学的研究成果,本文通过对基于多模态信息集成和同步来实现机器注意技术的讨论,将这种技术用于改进视频会议系统的人机交互模式,通过使视觉、听觉信号集成和同步的方法实现人工系统中视觉关注于感兴趣的对象——发言人的脸部及其发言。改进后的系统大大减少了发言人受到的约束,能够准确地自动跟踪发言人,而且在画面上更加突出视频会议画面中感兴趣的对象。     

Interacting meaningfully with machine learning systems: Three experiments

Although machine learning is becoming commonly used in today's software, there has been little research into how end users might interact with machine learning systems, beyond communicating simple “right/wrong” judgments. If the users themselves could work hand-in-hand with machine learning systems, the users’ understanding and trust of the system could improve and the accuracy of learning systems could be improved as well. We conducted three experiments to understand the potential for rich interactions between users and machine learning systems. The first experiment was a think-aloud study that investigated users’ willingness to interact with machine learning reasoning, and what kinds of feedback users might give to machine learning systems. We then investigated the viability of introducing such feedback into machine learning systems, specifically, how to incorporate some of these types of user feedback into machine learning systems, and what their impact was on the accuracy of the system. Taken together, the results of our experiments show that supporting rich interactions between users and machine learning systems is feasible for both user and machine. This shows the potential of rich human–computer collaboration via on-the-spot interactions as a promising direction for machine learning systems and users to collaboratively share intelligence.     

Do (and Say) as I Say: Linguistic Adaptation in Human–Computer Dialogs

There is strong research evidence showing that people naturally align to each other’s vocabulary, sentence structure, and acoustic features in dialog, yet little is known about how the alignment mechanism operates in the interaction between users and computer systems let alone how it may be exploited to improve the efficiency of the interaction. This article provides an account of lexical alignment in human–computer dialogs, based on empirical data collected in a simulated human–computer interaction scenario. The results indicate that alignment is present, resulting in the gradual reduction and stabilization of the vocabulary-in-use, and that it is also reciprocal. Further, the results suggest that when system and user errors occur, the development of alignment is temporarily disrupted and users tend to introduce novel words to the dialog. The results also indicate that alignment in human–computer interaction may have a strong strategic component and is used as a resource to compensate for less optimal (visually impoverished) interaction conditions. Moreover, lower alignment is associated with less successful interaction, as measured by user perceptions. The article distills the results of the study into design recommendations for human–computer dialog systems and uses them to outline a model of dialog management that supports and exploits alignment through mechanisms for in-use adaptation of the system’s grammar and lexicon.     

Adaptive dialogue management using intent clustering and fuzzy rules

Conversational systems have become an element of everyday life for billions of users who use speech‐based interfaces to services, engage with personal digital assistants on smartphones, social media chatbots, or smart speakers. One of the most complex tasks in the development of these systems is to design the dialogue model, the logic that provided a user input selects the next answer. The dialogue model must also consider mechanisms to adapt the response of the system and the interaction style according to different groups and user profiles. Rule‐based systems are difficult to adapt to phenomena that were not taken into consideration at design‐time. However, many of the systems that are commercially available are based on rules, and so are the most widespread tools for the development of chatbots and speech interfaces. In this article, we present a proposal to: (a) automatically generate the dialogue rules from a dialogue corpus through the use of evolving algorithms, (b) adapt the rules according to the detected user intention. We have evaluated our proposal with several conversational systems of different application domains, from which our approach provided an efficient way for adapting a set of dialogue rules considering user utterance clusters.     

PCC:一个对单用户建模的个性化对话系统

对话系统是自然语言处理（NLP）领域中一个重要的下游任务,在近几年得到了越来越多的关注,并取得了很大的发展。然而尽管对话领域已经取得了许多优秀的成果,现有的对话模型在拓展个性化方面依然有很大的局限性。为了使对话模型更符合人类的对话方式,拥有更好的个性化建模能力,该文提出一种新的对单个用户建模的个性化模型PCC（a Personalized Chatbot with Convolution mechanism）。在编码端,PCC通过文本卷积神经网络（TextCNN）处理用户历史回复帖子以得到用户兴趣信息;在解码端,使用相似度搜寻用户历史回答中与当前问题最为匹配的回复和用户ID一起指导生成。实验结果证明,该文模型在生成回复的准确性和多样性上均有较大提升,证明了历史回复信息在个性化建模方面的有效性。     

Computers that recognise and respond to user emotion: theoretical and practical implications

Prototypes of interactive computer systems have been built that can begin to detect and label aspects of human emotional expression, and that respond to users experiencing frustration and other negative emotions with emotionally supportive interactions, demonstrating components of human skills such as active listening, empathy, and sympathy. These working systems support the prediction that a computer can begin to undo some of the negative feelings it causes by helping a user manage his or her emotional state. This paper clarifies the philosophy of this new approach to human–computer interaction: deliberately recognising and responding to an individual user's emotions in ways, that help users meet their needs. We define user needs in a broader perspective than has been hitherto discussed in the HCI community, to include emotional and social needs, and examine technology's emerging capability to address and support such needs. We raise and discuss potential concerns and objections regarding this technology, and describe several opportunities for future work.     

Human knowledge acquisition from 3D interaction in virtual environments

Recently,virtual environment (VE) based design and planning,which is a kind of interaction intensive computing,has shown great potential.Most users of such systems are specialists or technicians and their 3D interactions with the system embed a great deal of knowledge and skills.Thus,how to integrate human knowledge and skills into VE is a great challenge to researchers in the human computer interaction field.This paper proposes a method for acquiring user knowledge from VE.The main ideas and work include,abstracting the interactive process and formalizing the interactive semantics,as well as providing a semantic model of an interactive information repository,from which user interactive processes can be retrieved and all kinds of application logics can be established.A virtual assembly planning system is introduced as an example of a practical application of this method.Experiments show that the related models can well capture user knowledge and retrieve the interaction process.     

Interactions between human–human multi-threaded dialogues and driving

In-car devices with speech user interfaces are proliferating. How can we build these interfaces such that they allow human–computer interactions with multiple devices to overlap in time, but without interfering with the driving task? We suggest that interface design can be inspired by the way people deal with this problem in human–human dialogues and propose discovering human dialogue behaviors of interest through experiments. In this paper, we discuss how to design an appropriate human–human dialogue scenario for such experiments. We also report on one human–human experiment, in terms of the dialogue behaviors found, and impact on the verbal tasks and on driving. We also offer design considerations based on the results of the study.     

User Modeling in Spoken Dialogue Systems to Generate Flexible Guidance

We address the issue of appropriate user modeling to generate cooperative responses to users in spoken dialogue systems. Unlike previous studies that have focused on a user’s knowledge, we propose more generalized modeling. We specifically set up three dimensions for user models: the skill level in use of the system, the knowledge level about the target domain, and the degree of urgency. Moreover, the models are automatically derived by decision tree learning using actual dialogue data collected by the system. We obtained reasonable accuracy in classification for all dimensions. Dialogue strategies based on user modeling were implemented on the Kyoto City Bus Information System that was developed at our laboratory. Experimental evaluations revealed that the cooperative responses adapted to each subject type served as good guides for novices without increasing the duration dialogue lasted for skilled users.