Creating natural language interfaces to VR systems

Two research projects are described that explore the use of spoken natural language interfaces to virtual reality (VR) systems. Both projects combine off-the-shelf speech recognition and synthesis technology with in-house command interpreters that interface to the VR applications. Details about the interpreters and other technical aspects of the projects are provided, together with a discussion of some of the design decisions involved in the creation of speech interfaces. Questions and issues raised by the projects are presented as inspiration for future work. These issues include: requirements for object and information representation in VR models to support natural language interfaces; use of the visual context to establish the interaction context; difficulties with referencing events in the virtual world; and problems related to the usability of speech and natural language interfaces in general.     

A data management strategy for transportable natural language interfaces

By transportability of a natural language interface (NLI) we mean the ease with which it can be customized to a new domain and database by the database administrator as opposed to a linguistic expert. Next-generation database systems (Genesis, Exodus, Starburst) are expansible to support a variety of applications each of which may use a different knowledge representation strategy. an enhanced architecture for natural language interfaces is presented that takes advantage of expansible next-generation database systems to facilitate transportability. A two-pass strategy for transportability is required within the enhanced architecture. On the first pass the NLI builds a representation space to facilitate its own transportability. On the second pass a transportability solution (viewed as a learning task) occurs in that representation space. © 1995 John Wiley & Sons, Inc.     

Towards portable natural language interfaces based on case-based reasoning

Natural Language Interfaces allow non-technical people to access information stored in Knowledge Bases keeping them unaware of the particular structure of the model or the underlying formal query language. Early research in the field was devoted to improve the performance of a particular system for a given Knowledge Base. Since adapting the system to new domains usually entailed considerable effort, investigating how to bring Portability to NLI became a new challenge. In this article, we investigate how Case-Based Reasoning could serve to assist the expert in porting the system so as to improve its retrieval performance. Our method HITS is based on a novel grammar learning algorithm combined with language acquisition techniques that exploit structural analogies. The learner (system) is able to engage the teacher (expert) with clarification dialogues to validate conjectures (hypotheses and deductions) about the language. Our method presents the following advantages: (i) the customization is naturally defined in the case-based cycle, (ii) the types of questions the system can deal with are not delimited in advance, and (iii) the system ‘reasons’ about precedent cases to deal with unseen questions.     

Providing RIA user interfaces with accessibility properties

Rich Internet Applications (RIAs) technologies are challenging the way in which the Web is being developed. However, from the UI accessibility point of view, these technologies pose new challenges that the Web Accessibility Initiative of the W3C is trying to solve through the use of a standard specification for Accessible Rich Internet Applications (WAI-ARIA). Currently, the introduction of properties defined in WAI-ARIA is being done in an ad-hoc manner due to the lack of models, methodologies and tools to support the design of accessible RIA UIs. In this paper we propose a semantic approach to deal with this modeling issue by extending the RUX-Method, a model-based method to build RIA UIs. The approach includes the validation process of the accessibility issues at two different levels: the UI structure and the interactions behavior.     

Improving habitability of natural language interfaces for querying ontologies with feedback and clarification dialogues

Natural Language Interfaces (NLIs) are a viable, human-readable alternative to complex, formal query languages like SPARQL, which are typically used for accessing semantically structured data (e.g. RDF and OWL repositories). However, in order to cope with natural language ambiguities, NLIs typically support a more restricted language. A major challenge when designing such restricted languages is habitability–how easily, naturally and effectively users can use the language to express themselves within the constraints imposed by the system. In this paper, we investigate two methods for improving the habitability of a Natural Language Interface: feedback and clarification dialogues. We model feedback by showing the user how the system interprets the query, thus suggesting repair through query reformulation. Next, we investigate how clarification dialogues can be used to control the query interpretations generated by the system. To reduce the cognitive overhead, clarification dialogues are coupled with a learning mechanism. Both methods are shown to have a positive effect on the overall performance and habitability.     

Modelling human tutors' feedback to inform natural language interfaces for learning

Modelling human tutors' socio-linguistic abilities computationally is necessary to underpin human–computer educational interactions with best pedagogic practices and to enhance the naturalness of such interactions. In this paper we present a computational model of tutorial feedback selection based on the context of the immediate situation for which the feedback is selected as well as on politeness considerations shown to be of importance to increasing pedagogical efficacy of computer assisted learning (Wang et al., 2008). The model is based on empirical studies with human tutors and on extensive linguistic analysis of human tutorial dialogues collected with the specific aim to inform the implementation of a natural language tutorial dialogue interface. The evaluation of the model, involving the comparison of its output with the linguistic responses produced by a human tutor, demonstrates the model's plausibility and highlights future directions for improving natural language human–computer interactions for educational purposes.     

Improving natural language and speech interfaces by the use of metalinguistic phenomena

Current shortcomings of Natural Language and Speech recognition interfaces are discussed. The argument examined is that people's familiarity with Natural Language and Speech, generally regarded as the raison d'etre of such interfaces, is in fact a barrier to their effective use. It is demonstrated, however, that prudent use of human metalingusitic strategies can alleviate some of the shortcomings in such interfaces. Examples are provided of metalinguistic strategies (response matching, mutual modelling and message adaptation), identified in human-human dialogue, being investigated in the laboratory and adapted for effective use in the design of human-computer dialogues.     

Representing natural language with Prolog

The authors describe the use of Prolog to build a syntactic list structure and a syntactic-semantic structure and to incorporate the semantic structure into a background structure that conveys the meanings of the individual words in the sentence in the context of general work knowledge. This semantic structure is then placed in the context of the sentence being parsed. Rather than rely solely on case grammar to represent the functions of words in a sentence, the authors have extended this technique to include a frame structure (which gives semantic details of the verb and the relations of other words to it) for a sentence's verbs to build the syntactic-semantic structure. Using this frame structure conveys more information than using only the case-grammar approach. The authors then place the syntactic-semantic structure in the context of its background knowledge, using the principles of partitioned networks in which word meanings are placed in a hierarchical structure that represents the background knowledge.<>     

Intention structure and extended responses in a portable natural language interface

This paper describes discourse processing inKing Kong, a portable natural language interface.King Kong enables users to pose questions and issue commands to a back end system. The notion of a discourse is central toKing Kong, and underlies much of the intelligent assistance thatkong provides to its users.kong's approach to modeling discourse is based on the work of Grosz and Sidner (1986). We extend Grosz and Sidner's framework in several ways, principally to allow multiple independent discourse contexts to remain active at the same time. This paper also describesKing Kong's method of intention recognition, which is similar to that described in Kautz and Allen (1986) and Carberry (1988). We demonstrate that a relatively simple intention recognition component can be exploited by many other discourserelated mechanisms, for example to disambiguate input and resolve anaphora. In particular, this paper describes in detail the mechanism inKing Kong that uses information from the discourse model to form a range of cooperative extended responses to queries in an effort to aid the user in accomplishing her goals.Judith Schaffer Sider received her Bachelor of Arts degree in Computer Science and Linguistics and Cognitive Science from Brandeis University. Since 1987 she has been a member of the technical staff at the MITRE Corporation, where she works on King Kong, the natural language interface under development there. The joint research with John D. Burger described in this volume reflects some of her work in the areas of cooperative responding and plan recognition.John D. Burger is a Project Leader at the MITRE Corporation and an instructor at Boston University. He received a Bachelor of Science degree in Mathematics and Computer Science from Carnegie Melon University. His research interests lie in the fields of natural language processing and intelligent multimedia interfaces. The joint work with Judith Schaffer Sider described in this volume reflects his interest in making use of discourse models in practical intelligent interfaces.     

Evaluating the usability of natural language query languages and interfaces to Semantic Web knowledge bases

The need to make the contents of the Semantic Web accessible to end-users becomes increasingly pressing as the amount of information stored in ontology-based knowledge bases steadily increases. Natural language interfaces (NLIs) provide a familiar and convenient means of query access to Semantic Web data for casual end-users. While several studies have shown that NLIs can achieve high retrieval performance as well as domain independence, this paper focuses on usability and investigates if NLIs and natural language query languages are useful from an end-user's point of view. To that end, we introduce four interfaces each allowing a different query language and present a usability study benchmarking these interfaces. The results of the study reveal a clear preference for full natural language query sentences with a limited set of sentence beginnings over keywords or formal query languages. NLIs to ontology-based knowledge bases can, therefore, be considered to be useful for casual or occasional end-users. As such, the overarching contribution is one step towards the theoretical vision of the Semantic Web becoming reality.     

Providing standard-oriented data models and interfaces to eGovernment services: A semantic-driven approach

As eGovernment becomes a very active research area, a lot of solutions to solve citizens' needs are being deployed. These solutions, even of a high quality, suffer from some drawbacks. Most of them related to the lack of interoperability among different Public Administrations or the difficulties to locate or invoke the desired service. To deal with these issues, a semantic-based approach centered in citizens is proposed. This paper tackles the provision of a front-end solution to access services in Public Administrations. A complete software interface within a holistic model of service is deeply discussed.     

Communicating with operating systems in natural language

We aim to conduct a dialogue with a machine in a natural manner without having to learn an artificial language. With the goal of supplying a natural language interface to a variety of operating systems, we have developed a TURBO-PROLOG ®-based system called NLDOS that facilitates the communication with the well known operating system MS-DOS®. NLDOS is capable of understanding English requests, providing an automatic spelling correction facility, resolving ambiguity, and discovering and locating logical errors. © 1993 John Wiley & Sons, Inc.     

A collaborative behavior-based approach for handling ambiguity, uncertainty, and vagueness in robot natural language interfaces

Service robots are used by ordinary people in their houses and offices. For such users, a desirable way to communicate with robots is through natural language interfaces. So far, techniques for developing robot natural language interfaces are far from mature. A major challenge is handling ambiguity, uncertainty, and vagueness in parsing, object resolution, and vague natural language words. In this research, we develop a new approach called the collaborative behavior-based approach, in which behaviors of robots and behaviors of human users, as well as the changes of object states caused by the behaviors, are taken into consideration integratedly in processing natural language user instructions. In this paper, we analyze the special features of a human–robot interface that may affect language understanding, describe our approach that is designed based on the features, and present the implementation and some experimental results.     

Twenty guidelines for the design of Web-based interfaces with consistent language

The 20 guidelines presented in this paper for the design of consistent computer interfaces are based on the data derived from the use of a 94-item, nine-factor questionnaire, the Interface Consistency Testing Questionnaire (ICTQ), that was administered to 140 participants. The questionnaire data was collected from a total of 20 diversified tasks. The questionnaire on which the guidelines are based has a good construct validity, interrater reliability (0.75), and internal consistency as measured by Cronbach's α (0.81). The questionnaire proved to be adequately sensitive to discriminate between consistent and inconsistent interface designs.     

Interacting in natural language with artificial systems: The donau project

This paper is intended to propose a new methodological approach to the conception and development of natural language understanding systems. This new contribution is supported by the design, implementation, and experimentation of DONAU: a general purpose domain oriented natural language understanding system developed and presently running at the Milan Polytechnic Artificial Intelligence Project. The system is based on a two level modular architecture intended to overcome the lack of flexibility and generality often pointed out in many existing systems, and to facilitate the exchange of results and actual experiences between different projects. The horizontal level allows an independent and parallel development of the single segments of the system (syntactic analyser, semantic analyser, information extractor, legality controller). The vertical level ensures the possibility of changing (enlarging or redefining) the definition of the semantic domain on which each particular version of the system is oriented and specialized in a simple, incremental, and user-oriented way. In the paper the general architecture of the system and the mode of operation of each segment are illustrated in detail. Linguistic models, knowledge representation, and parsing algorithms are described and illustrated by means of selected examples. Performance evaluations of the system in the application version on data base inquiry are reported and discussed. Promising directions for future research are presented in the conclusions.     

Integrating natural language understanding with document structure analysis

Document understanding, the interpretation of a document from its image form, is a technology area which benefits greatly from the integration of natural language processing with image processing. We have developed a prototype of an Intelligent Document Understanding System (IDUS) which employs several technologies: image processing, optical character recognition, document structure analysis and text understanding in a cooperative fashion. This paper discusses those areas of research during development of IDUS where we have found the most benefit from the integration of natural language processing and image processing: document structure analysis, optical character recognition (OCR) correction, and text analysis. We also discuss two applications which are supported by IDUS: text retrieval and automatic generation of hypertext links     

Using a natural language interface with casual users

Although there is much controversy about the merits of natural language interfaces, little empirical research has been conducted on the use of natural language interfaces for database access, especially for casual users. In this work casual users were observed while interacting with a real-life database using a natural language interface, Intellect.Results show that natural language is an efficient and powerful means for expressing requests. This is especially true for users with a good knowledge of the database contents regardless of training or previous experience with computers. Users generally have a positive attitude towards natural language. The majority of errors users make are directly related to restrictions in the vocabulary. However, feedback helps users understand the language limitations and learn how to avoid or recover from errors. Natural language processing technology is developed enough to handle the limited domain of discourse associated with a database; it is simple enough to support casual users with a general knowledge of the database contents; and it is flexible enough to assist problem-solving behaviour.     

An experimental graphics system with natural language input

An experimental graphics system is described which allows interaction through natural language and touches on the screen. Artificial Intelligence techniques for processing language and representing knowledge are investigated as adequate tools with which to construct a habitable graphics system. The system is evaluated and conclusions presented.