Combining natural language with direct manipulation: the conceptual framework for a hybrid human-computer interface

Abstract

This paper describes the development of a conceptual model of an interaction system for future human-computer interaction. It is suggested that an integrated system is the most promising solution for diverse users and tasks. A ‘ hybrid interaction system’ is considered, which tries to efficiently direct manipulation, menu selection, and natural language. The mechanism of the natural language interface module is described in more detail. Finally, the allocation of functionalities for generalized tasks on the conceptual model is considered from the taxonomic approach.     

Enhancing information systems management with natural language processing techniques

Natural language and databases are core components of information systems. They are related to each other because they share the same purpose: the conceptualization aspects of the real world in order to deal with them in some way. Natural language processing (NLP) techniques may substantially enhance most phases of the information system lifecycle, starting with requirements analysis, specification and validation, and going up to conflict resolution, result processing and presentation. Furthermore, natural language based query languages and user interfaces facilitate the access to information for anyone and allow for new paradigms in the usage of computerized services. This paper investigates the use of NLP techniques in the design phase of information systems. Then, it reports on data base querying and information retrieval enhanced with NLP.     

A third modality of natural language?

In the late Eighties the Natural Language Processing community began appreciating the role of multimodality in interactive systems. Intelligent multimodal systems are systems that integrate natural language (generally so far keyboard-based input, shortly also voice) with other media such as gestures in input or graphics in output. The perspective of what can be called visible interactive communication is discussed and considered as a possible new modality of natural language, after the spoken and the written ones. This should not be confused with the type of hypermedia that are now being developed. There, basically, the interface space is finite, even if one dimension may be added. Here the infinite creativity of human language is potentially preserved as the fundamental communication instrument.This is a revised version of an invited talk that was delivered at the10th European Conference on Artificial Intelligence in Vienna, and published in the Proceedings (B. Neuman, ed.), John Wiley & Sons, 1992, pp. 853–862.     

Creating natural language interfaces to VR systems

Two research projects are described that explore the use of spoken natural language interfaces to virtual reality (VR) systems. Both projects combine off-the-shelf speech recognition and synthesis technology with in-house command interpreters that interface to the VR applications. Details about the interpreters and other technical aspects of the projects are provided, together with a discussion of some of the design decisions involved in the creation of speech interfaces. Questions and issues raised by the projects are presented as inspiration for future work. These issues include: requirements for object and information representation in VR models to support natural language interfaces; use of the visual context to establish the interaction context; difficulties with referencing events in the virtual world; and problems related to the usability of speech and natural language interfaces in general.     

Integrating specification of human-computer interface with Jackson system development

Procedures for integrating task analysis and design of the human-computer interface (HCl) with a structured system development method, Jackson system development (JSD), are described. Process structure diagrams (PSDs), the major modelling notation of JSD, described tasks, and task specifications were evaluated for cognitive complexity. Task complexity analysis helped allocation and design of human tasks and to produce computer task support specifications. The cognitive dimension of task analysis highlighted the need for task support actions, especially information display support for working memory. Dialogue specification for a direct manipulation interface design was based on the JSD entity/event model from which the interaction metaphor, screen design, and permissible manipulations were derived. Further PSDs were constructed to specify interface object management processes. Preliminary evaluation of the method was carried out using JSD analysts. Results showed the method proved easy to learn even for non-HCl specialists.     

Visible speech improves human language understanding: Implications for speech processing systems

Evidence from the study of human language understanding is presented suggesting that our ability to perceive visible speech can greatly influence our ability to understand and remember spoken language. A view of the speaker's face can greatly aid in the perception of ambiguous or noisy speech and can aid cognitive processing of speech leading to better understanding and recall. Some of these effects have been replicated using computer synthesized visual and auditory speech. Thus, it appears that when giving an interface a voice, it may be best to give it a face too.     

Chicon—a Chinese text manipulation language

Text processing is an important computer application. Due to its importance, a number of text manipulation programming languages have been devised (e.g. Icon). These programming languages are very useful for applications such as natural language processing, text analysis, text editing, document formatting, text generation, etc. However, they were mainly designed to handle English texts, and are ineffective for Chinese. This is because English and Chinese texts are represented very differently in a computer. An English character is mainly represented in 7-bit ASCII, and its Chinese counterpart commonly in 16-bit GB or BIG-5. This difference makes direct application of English-based text manipulation programming languages to Chinese erroneous, e.g. application of Icon to reverse a string of Chinese characters. In this paper, a new dialect of Icon, referred to as Chicon (i.e. Chinese Icon), is proposed. In the design of Chicon, new data types were introduced to differentiate pure English and English/Chinese mixed texts. In addition, existing Icon text manipulation functions were modified to account for Chinese texts. Experiments have shown that Chicon not only could overcome the problems of Chinese processing in Icon, but its execution speed was actually superior to Icon in handling Chinese. Furthermore, application of Chicon to a real sized problem, namely word segmentation, has proved that the language is practical. © 1998 John Wiley & Sons, Ltd.     

Natural language understanding within a cognitive semantics framework

The article argues that cognitive linguistic theory may prove an alternative to the Montague paradigm for designing natural language understanding systems. Within this framework it describes a system which models language understanding as a dialogical process between user and computer. The system operates with natural language texts as input and represent language meaning as entity-relationship diagrams.     

Natural language interface for information management on mobile devices

A natural language interface (NLI) enables the ease-of-use of information systems in performing sophisticated human - computer interaction. To address the challenges of mobile devices to user interaction in information management, we propose an NLI as a promising solution. In this paper, we review state-of-the-art NLI technologies and analyse user requirements for managing notable information on mobile devices. To minimize any technical difficulties arising from developing and improving the usability of NLI systems we develop general principles for NLI design, which fills in a gap in the literature. In order to satisfy user requirements for information management on mobile devices, we innovatively design NLI-enabled information management architecture. It is shown from two usage scenarios that the architecture could lead to reduced effort in user navigation and improved efficiency and effectiveness of managing information on mobile devices. We conclude the article with the implications of this study and suggestions for future direction.     

Formal syntax methods for natural language

This paper describes on-going research into the applications of some techniques normally used to formally specify and analyze the context-sensitive syntax of programming languages to the specification and analysis of the syntax of a natural language, namely English. The specific formal methods presently being investigated are two-level grammar (2LG) and the Vienna Definition Language (VDL). A preliminary subset of English has been established consisting of fifteen basic sentence patterns. 2LG and VDL specifications are given for one of these sentence types and the syntactic analysis of an English sentence using each of the two specifications is illustrated through an example.     

A review of methods for automatic understanding of natural language mathematical problems

This article addresses the problem of understanding mathematics described in natural language. Research in this area dates back to early 1960s. Several systems have so far been proposed to involve machines to solve mathematical problems of various domains like algebra, geometry, physics, mechanics, etc. This correspondence provides a state of the art technical review of these systems and approaches proposed by different research groups. A unified architecture that has been used in most of these approaches is identified and differences among the systems are highlighted. Significant achievements of each method are pointed out. Major strengths and weaknesses of the approaches are also discussed. Finally, present efforts and future trends in this research area are presented.     

基于数据库查询的自然语言接口研究

语音识别技术是近年来信息领域重要的科技发展技术之一.作为一门交叉学科,语音识别正逐步成为信息技术中人机接口的关键技术.探讨了为数据库查询提供自然语言接口的可能性,介绍了图书资料查询系统中语音识别的基本框架,并详细描述了采用微软Speech SDK技术实现图书资料查询的详细过程.给出了查询流程,基于状态转换图的词法分析和语法分析方法,将疑问句转换成SQL查询语句的方法,以及由查询结果生成答句的方法.     

A linguistic ontology of space for natural language processing

We present a detailed semantics for linguistic spatial expressions supportive of computational processing that draws substantially on the principles and tools of ontological engineering and formal ontology. We cover language concerned with space, actions in space and spatial relationships and develop an ontological organization that relates such expressions to general classes of fixed semantic import. The result is given as an extension of a linguistic ontology, the Generalized Upper Model, an organization which has been used for over a decade in natural language processing applications. We describe the general nature and features of this ontology and show how we have extended it for working particularly with space. Treaitng the semantics of natural language expressions concerning space in this way offers a substantial simplification of the general problem of relating natural spatial language to its contextualized interpretation. Example specifications based on natural language examples are presented, as well as an evaluation of the ontology's coverage, consistency, predictive power, and applicability.     

Movement-based interaction in camera spaces: a conceptual framework

In this paper we present three concepts that address movement-based interaction using camera tracking. Based on our work with several movement-based projects we present four selected applications, and use these applications to leverage our discussion, and to describe our three main concepts space, relations, and feedback. We see these as central for describing and analysing movement-based systems using camera tracking and we show how these three concepts can be used to analyse other camera tracking applications.     

Requirements for tools for ambiguity identification and measurement in natural language requirements specifications

This paper proposes a two-step approach to identifying ambiguities in natural language (NL) requirements specifications (RSs). In the first step, a tool would apply a set of ambiguity measures to a RS in order to identify potentially ambiguous sentences in the RS. In the second step, another tool would show what specifically is potentially ambiguous about each potentially ambiguous sentence. The final decision of ambiguity remains with the human users of the tools. The paper describes several requirements-identification experiments with several small NL RSs using four prototypes of the first tool based on linguistic instruments and resources of different complexity and a manual mock-up of the second tool.

The Role of a Natural Language Conversational Interface in Online Sales: A Case Study

This paper describes the evaluation of a natural language dialog-based navigation system (HappyAssistant) that helps users access e-commerce sites to find relevant information about products and services. The prototype system leverages technologies in natural language processing and human-computer interaction to create a faster and more intuitive way of interacting with websites, especially for less experienced users. The result of a comparative study shows that users prefer the natural language-enabled navigation two to one over the menu driven navigation. In addition, the study confirmed the efficiency of using natural language dialog in terms of the number of clicks and the amount of time required to obtain the relevant information. In the case study, as compared to the menu driven system, the average number of clicks used in the natural language system was reduced by 63.2% and the average time was reduced by 33.3%.     

Integrating natural language understanding with document structure analysis

Document understanding, the interpretation of a document from its image form, is a technology area which benefits greatly from the integration of natural language processing with image processing. We have developed a prototype of an Intelligent Document Understanding System (IDUS) which employs several technologies: image processing, optical character recognition, document structure analysis and text understanding in a cooperative fashion. This paper discusses those areas of research during development of IDUS where we have found the most benefit from the integration of natural language processing and image processing: document structure analysis, optical character recognition (OCR) correction, and text analysis. We also discuss two applications which are supported by IDUS: text retrieval and automatic generation of hypertext links     

Topic-aware pivot language approach for statisticalmachine translation

The pivot language approach for statistical machine translation （SMT） is a good method to break the resource bottleneck for certain language pairs. However, in the implementation of conventional approaches, pivot- side context information is far from fully utilized, resulting in erroneous estimations of translation probabilities. In this study, we propose two topic-aware pivot language approaches to use different levels of pivot-side context. The first method takes advantage of document-level context by assuming that the bridged phrase pairs should be similar in the document-level topic distributions. The second method focuses on the effect of local context. Central to this approach are that the phrase sense can be reflected by local context in the form of probabilistic topics, and that bridged phrase pairs should be compatible in the latent sense distributions. Then, we build an interpolated model bringing the above methods together to further enhance the system performance. Experimental results on French-Spanish and French-German translations using English as the pivot language demonstrate the effectiveness of topic-based context in pivot-based SMT.