Bringing context-aware access to the web through spoken interaction

The web has become the largest repository of multimedia information and its convergence with telecommunications is now bringing the benefits of web technology to hand-held devices. To optimize data access using these devices and provide services which meet the user needs through intelligent information retrieval, the system must sense and interpret the user environment and the communication context. In addition, natural spoken conversation with handheld devices makes possible the use of these applications in environments in which the use of GUI interfaces is not effective, provides a more natural human-computer interaction, and facilitates access to the web for people with visual or motor disabilities, allowing their integration and the elimination of barriers to Internet access. In this paper, we present an architecture for the design of context-aware systems that use speech to access web services. Our contribution focuses specifically on the use of context information to improve the effectiveness of providing web services by using a spoken dialog system for the user-system interaction. We also describe an application of our proposal to develop a context-aware railway information system, and provide a detailed evaluation of the influence of the context information in the quality of the services that are supplied.     

A framework for improving error detection and correction in spoken dialog systems

Despite the recent improvements in performance and reliably of the different components of dialog systems, it is still crucial to devise strategies to avoid error propagation from one another. In this paper, we contribute a framework for improved error detection and correction in spoken conversational interfaces. The framework combines user behavior and error modeling to estimate the probability of the presence of errors in the user utterance. This estimation is forwarded to the dialog manager and used to compute whether it is necessary to correct possible errors. We have designed an strategy differentiating between the main misunderstanding and non-understanding scenarios, so that the dialog manager can provide an acceptable tailored response when entering the error correction state. As a proof of concept, we have applied our proposal to a customer support dialog system. Our results show the appropriateness of our technique to correctly detect and react to errors, enhancing the system performance and user satisfaction.     

Self-configuring agents for ambient assisted living applications

Ambient assisted living (AAL) is advocated as the technological solution that will enable the elderly population to maintain their independence for a longer period of time than would otherwise be the case. The inherently heterogeneous nature of AmI environments and special requirements of the elderly population pose new challenges for the design and implementation of AAL systems. Thus, the development of these systems demands a context-aware, open, scalable, and distributed software technology that incorporates both intelligent and autonomic reconfiguration techniques. In this contribution, we focus on the design and implementation challenges of an agent-based AAL system that incorporates self-configuring tasks, by means of applying autonomic computing to software agents’ internal architecture. We use an agent-based system for tracking elderly people’s activity using common commercially available electronic devices as case study. We have validated our approach focusing on response time (a main concern in AAL) using different tests and the results are satisfactory.     

Contextual music information retrieval and recommendation: State of the art and challenges

Increasing amount of online music content has opened new opportunities for implementing new effective information access services–commonly known as music recommender systems–that support music navigation, discovery, sharing, and formation of user communities. In the recent years a new research area of contextual (or situational) music recommendation and retrieval has emerged. The basic idea is to retrieve and suggest music depending on the user’s actual situation, for instance emotional state, or any other contextual conditions that might influence the user’s perception of music. Despite the high potential of such idea, the development of real-world applications that retrieve or recommend music depending on the user’s context is still in its early stages. This survey illustrates various tools and techniques that can be used for addressing the research challenges posed by context-aware music retrieval and recommendation. This survey covers a broad range of topics, starting from classical music information retrieval (MIR) and recommender system (RS) techniques, and then focusing on context-aware music applications as well as the newer trends of affective and social computing applied to the music domain.     

The RavenClaw dialog management framework: Architecture and systems

In this paper, we describe RavenClaw, a plan-based, task-independent dialog management framework. RavenClaw isolates the domain-specific aspects of the dialog control logic from domain-independent conversational skills, and in the process facilitates rapid development of mixed-initiative systems operating in complex, task-oriented domains. System developers can focus exclusively on describing the dialog task control logic, while a large number of domain-independent conversational skills such as error handling, timing and turn-taking are transparently supported and enforced by the RavenClaw dialog engine. To date, RavenClaw has been used to construct and deploy a large number of systems, spanning different domains and interaction styles, such as information access, guidance through procedures, command-and-control, medical diagnosis, etc. The framework has easily adapted to all of these domains, indicating a high degree of versatility and scalability.     

Cascaded model adaptation for dialog act segmentation and tagging

There are many speech and language processing problems which require cascaded classification tasks. While model adaptation has been shown to be useful in isolated speech and language processing tasks, it is not clear what constitutes system adaptation for such complex systems. This paper studies the following questions: In cases where a sequence of classification tasks is employed, how important is to adapt the earlier or latter systems? Is the performance improvement obtained in the earlier stages via adaptation carried on to later stages in cases where the later stages perform adaptation using similar data and/or methods? In this study, as part of a larger scale multiparty meeting understanding system, we analyze various methods for adapting dialog act segmentation and tagging models trained on conversational telephone speech (CTS) to meeting style conversations. We investigate the effect of using adapted and unadapted models for dialog act segmentation with those of tagging, showing the effect of model adaptation for cascaded classification tasks. Our results indicate that we can achieve significantly better dialog act segmentation and tagging by adapting the out-of-domain models, especially when the amount of in-domain data is limited. Experimental results show that it is more effective to adapt the models in the latter classification tasks, in our case dialog act tagging, when dealing with a sequence of cascaded classification tasks.     

A general-purpose natural language interface to an operating system

One of the most difficult tasks facing new computer users is the management of data and programs through an operating system (OS) interface. Both command languages and graphical interfaces can impede a user unfamiliar with the terminology and organizational techniques of modern file management systems. Various attempts have been made in recent years to develop systems that will enable users to interact with the computer in natural language. By simplifying the form of user input to an OS, such interfaces allow the user to concentrate on file management tasks rather than on the means of expressing OS-specific instructions. The Informational Network for a Natural Talking (INFANT) System was originally developed as a conversational system for communication at the level of a small child. It has since been adapted for use as a general-purpose English language interface for standard OS file management operations. The INFANT System reduces an input sentence to a hierarchy of standardized propositional forms, which are mapped through a propositional knowledge base and a compositional semantics process into a high-level representation of meaning. Its effectiveness as a working natural language interface has been verified in a pair of experiments performed in a college computer lab. © 1996 John Wiley & Sons, Inc.     

基于对话回合衰减的cache语言模型在线自适应研究

目前由于特定任务域语料的稀疏并且难以收集,这严重阻碍了对话系统的可移植性。如何利用在线收集的少量训练语料,实现语言模型的快速自适应,从而有效提高对话系统在新任务域的识别率是本文的目的所在。本文对传统cache模型修正后,提出了基于历史单元衰减的cache语言模型,以在线递增方式收集语料进行自适应,并与通用语言模型进行线性插值。在对话系统中,以对话回合为历史单元,也可称为基于对话回合衰减的cache语言模型。在两个完全不同任务域——颐和园导游与火车票订票任务域进行的实验表明,在自适应语料不到1千句时,与无自适应模型相比,有监督模式下的识别错误率分别降低了47.8%和74.0% ,无监督模式下的识别错误率分别降低了30.1%和51.1%。     

Do (and Say) as I Say: Linguistic Adaptation in Human–Computer Dialogs

There is strong research evidence showing that people naturally align to each other’s vocabulary, sentence structure, and acoustic features in dialog, yet little is known about how the alignment mechanism operates in the interaction between users and computer systems let alone how it may be exploited to improve the efficiency of the interaction. This article provides an account of lexical alignment in human–computer dialogs, based on empirical data collected in a simulated human–computer interaction scenario. The results indicate that alignment is present, resulting in the gradual reduction and stabilization of the vocabulary-in-use, and that it is also reciprocal. Further, the results suggest that when system and user errors occur, the development of alignment is temporarily disrupted and users tend to introduce novel words to the dialog. The results also indicate that alignment in human–computer interaction may have a strong strategic component and is used as a resource to compensate for less optimal (visually impoverished) interaction conditions. Moreover, lower alignment is associated with less successful interaction, as measured by user perceptions. The article distills the results of the study into design recommendations for human–computer dialog systems and uses them to outline a model of dialog management that supports and exploits alignment through mechanisms for in-use adaptation of the system’s grammar and lexicon.     

An evaluation of the chat and knowledge delivery components of a low-level dialog system:The AZ-ALICE experiment

An effective networked knowledge delivery platform is one of the Holy Grails of Web computing. Knowledge delivery approaches range from the heavy and narrow to the light and broad. This paper explores a lightweight and flexible dialog framework based on the ALICE system, and evaluates its performance in chat and knowledge delivery using both a conversational setting and a specific telecommunications knowledge domain. Metrics for evaluation are presented, and the evaluations of three experimental systems (a pure dialog system, a domain knowledge system, and a hybrid system combining dialog and domain knowledge) are presented and discussed. Our study of 257 subjects shows approximately a 20% user correction rate on system responses. Certain error classes (such as nonsense replies) were particular to the dialog system, while others (such as mistaking opinion questions for definition questions) were particular to the domain system. A third type of error, wordy and awkward responses, is a basic system property and spans all three experimental systems. We also show that the highest response satisfaction results are obtained when coupling domain-specific knowledge together with conversational dialog.     

Hybreed: A software framework for developing context-aware hybrid recommender systems

This article introduces Hybreed, a software framework for building complex context-aware applications, together with a set of components that are specifically targeted at developing hybrid, context-aware recommender systems. Hybreed is based on a concept for processing context that we call dynamic contextualization. The underlying notion of context is very generic, enabling application developers to exploit sensor-based physical factors as well as factors derived from user models or user interaction. This approach is well aligned with context definitions that emphasize the dynamic and activity-oriented nature of context. As an extension of the generic framework, we describe Hybreed RecViews, a set of components facilitating the development of context-aware and hybrid recommender systems. With Hybreed and RecViews, developers can rapidly develop context-aware applications that generate recommendations for both individual users and groups. The framework provides a range of recommendation algorithms and strategies for producing group recommendations as well as templates for combining different methods into hybrid recommenders. Hybreed also provides means for integrating existing user or product data from external sources such as social networks. It combines aspects known from context processing frameworks with features of state-of-the-art recommender system frameworks, aspects that have been addressed only separately in previous research. To our knowledge, Hybreed is the first framework to cover all these aspects in an integrated manner. To evaluate the framework and its conceptual foundation, we verified its capabilities in three different use cases. The evaluation also comprises a comparative assessment of Hybreed’s functional features, a comparison to existing frameworks, and a user study assessing its usability for developers. The results of this study indicate that Hybreed is intuitive to use and extend by developers.     

Simulating empathic behavior in a social assistive robot

When used as an interface in the context of Ambient Assisted Living (AAL), a social robot should not just provide a task-oriented support. It should also try to establish a social empathic relation with the user. To this aim, it is crucial to endow the robot with the capability of recognizing the user’s affective state and reason on it for triggering the most appropriate communicative behavior. In this paper we describe how such an affective reasoning has been implemented in the NAO robot for simulating empathic behaviors in the context of AAL. In particular, the robot is able to recognize the emotion of the user by analyzing communicative signals extracted from speech and facial expressions. The recognized emotion allows triggering the robot’s affective state and, consequently, the most appropriate empathic behavior. The robot’s empathic behaviors have been evaluated both by experts in communication and through a user study aimed at assessing the perception and interpretation of empathy by elderly users. Results are quite satisfactory and encourage us to further extend the social and affective capabilities of the robot.     

A middleware architecture for privacy protection

The issue of user privacy is constantly in spotlight since an ever increasing number of online services collects and processes personal information from users, in the context of personalized service provision. In fact, recent advances in mobile communications, location and sensing technologies and data processing are boosting the deployment of context-aware personalized services and the creation of smart environments; but at the same time, they pose a serious risk on individuals’ privacy rights. Although technology makes the collection of data easy, its protection against abuse is left to data protection legislation. However, the privacy requirements, other than being general and abstract terms to be regarded as legislature issues, should be brought down in technological reality and carefully accounted for in devising technical solutions. In order to limit the disclosure and avoid the misuse of personal data, this paper discusses an architectural proposal for a middleware system that will enforce protection of user privacy through technical means. The proposed architecture mediates between the users, the service providers and the law, constituting a middleware shield for individuals’ personal data.     

Conversational natural language understanding interfacing city event information

The article describes aspects of the development of a conversational natural language understanding (NLU) system done during the first year of the European research project CATCH-2004 (Converse in AThens Cologne and Helsinki) [http://www.catch2004.org]. The project is co-funded by the European Union in the scope of the IST programme (IST 1999-11103).

Its objectives focus on multi-modal, multi-lingual conversational natural language access to information systems. The paper emphasises on architecture, and telephony-based speech and NLU components as well as aspects of the implementation of a city event information (CEI) system in English, Finnish, German and Greek. The CEI system accesses two different databases in Athens and Helsinki using a common retrieval interface. Furthermore the paper singles out methodologies involved for acoustic and language model of the speech recognition component, parsing techniques and dialog modelling for the conversational natural language subsystem. For the implementation it outlines an incremental system refinement methodology necessary to adapt the system components to real-life data. It addresses the implementation of language specific characteristics and a common dialog design for all four languages, but also deals with aspects towards a multilingual conversational system. Finally, it presents prospects for further developments of the project.     

Context‐Centric Speech‐Based Human–Computer Interaction

This paper describes research that addresses the problem of dialog management from a strong, context‐centric approach. We further present a quantitative method of measuring the importance of contextual cues when dealing with speech‐based human–computer interactions. It is generally accepted that using context in conjunction with a human input, such as spoken speech, enhances a machine's understanding of the user's intent as a means to pinpoint an adequate reaction. For this work, however, we present a context‐centric approach in which the use of context is the primary basis for understanding and not merely an auxiliary process. We employ an embodied conversation agent that facilitates the seamless engagement of a speech‐based information‐deployment entity by its human end user. This dialog manager emphasizes the use of context to drive its mixed‐initiative discourse model. A typical, modern automatic speech recognizer (ASR) was incorporated to handle the speech‐to‐text translations. As is the nature of these ASR systems, the recognition rate is consistently less than perfect, thus emphasizing the need for contextual assistance. The dialog system was encapsulated into a speech‐based embodied conversation agent platform for prototyping and testing purposes. Experiments were performed to evaluate the robustness of its performance, namely through measures of naturalness and usefulness, with respect to the emphasized use of context. The contribution of this work is to provide empirical evidence of the importance of conversational context in speech‐based human–computer interaction using a field‐tested context‐centric dialog manager.     

The history and future of direct manipulation

Abstract

The earliest interactive computer systems were based on a conversational mode of interaction in which user and computer communicated through the exchange of linguistic utterances. Since the advent of ‘direct manipulation’ technology there has been a tendency to develop and promote an alternative mode of interaction, based on the user's manipulation of computer-displayed objects. This paper reviews recent developments in the implementation and understanding of direct manipulation interfaces. These point to various limitations in manipulative interaction which might be overcome through the selective reintroduction of conversational interaction. A new philosophy of graceful interaction is suggested to accommodate these developments in which directness is said to be a property of both action and language based systems. A number of practical guidelines are offered to reduce the incidence of clumsy manipulation.     

Adaptive dialogue management using intent clustering and fuzzy rules

Conversational systems have become an element of everyday life for billions of users who use speech‐based interfaces to services, engage with personal digital assistants on smartphones, social media chatbots, or smart speakers. One of the most complex tasks in the development of these systems is to design the dialogue model, the logic that provided a user input selects the next answer. The dialogue model must also consider mechanisms to adapt the response of the system and the interaction style according to different groups and user profiles. Rule‐based systems are difficult to adapt to phenomena that were not taken into consideration at design‐time. However, many of the systems that are commercially available are based on rules, and so are the most widespread tools for the development of chatbots and speech interfaces. In this article, we present a proposal to: (a) automatically generate the dialogue rules from a dialogue corpus through the use of evolving algorithms, (b) adapt the rules according to the detected user intention. We have evaluated our proposal with several conversational systems of different application domains, from which our approach provided an efficient way for adapting a set of dialogue rules considering user utterance clusters.     

Utilizing contextual ontological user profiles for personalized recommendations

As users may have different needs in different situations and contexts, it is increasingly important to consider user context data when filtering information. In the field of web personalization and recommender systems, most of the studies have focused on the process of modelling user profiles and the personalization process in order to provide personalized services to the user, but not on contextualized services. Rather limited attention has been paid to investigate how to discover, model, exploit and integrate context information in personalization systems in a generic way. In this paper, we aim at providing a novel model to build, exploit and integrate context information with a web personalization system. A context-aware personalization system (CAPS) is developed which is able to model and build contextual and personalized ontological user profiles based on the user’s interests and context information. These profiles are then exploited in order to infer and provide contextual recommendations to users. The methods and system developed are evaluated through a user study which shows that considering context information in web personalization systems can provide more effective personalization services and offer better recommendations to users.     

Adapting groupware systems to changes in the collaborator’s context of use

Owing to the dynamic nature of collaborative environments, the software intended to support collaborative work should adapt itself to the different situations that may occur. This requirement is related to the concept of “context of use”, which has been considered as an important aspect in the design of interactive systems. Nevertheless, two main problems about this concept have been identified by current research in context-aware computing: (1) most of the studies have mainly focused on the context of a single user, so the context of multiple users involved in a common endeavor remains little explored, and (2) adaptability in context-aware systems generally takes into account a reduced number of contextual variables (mainly the user’s location and platform). In this paper, we firstly re-conceptualize the notion of “context of use”, in order to consider the main characteristics of collaborative environments. Based on this new notion, we then design and implement a framework that allows application developers to specify the adaptability of groupware systems in terms of the state of activities, roles, collaborators’ location, available resources, and other typical variables of working groups. This framework has been generalized from scenarios that highlight dynamic situations presented in real collaborative settings. Finally, we validate our proposal by a set of applications that are able to adapt their user interface and functionality, when significant changes are produced in the environment, the working group, and/or the used devices.