Examining modality usage in a conversational multimodal application for mobile e-mail access

As Third Generation (3G) networks emerge they provide not only higher data transmission rates but also the ability to transmit both voice and low latency data within the same session. This paper describes the architecture and implementation of a multimodal application (voice and text) that uses natural language understanding combined with a WAP browser to access email messages on a cell phone. We present results from the use of the system by users as part of a laboratory trial that evaluated usage. The user trial also compared the multimodal system with a text-only system that is representative of current products in the market today. We discuss the observed modality issues and highlight implementation problems and usability concerns that were encountered in the trial. Findings indicate that speech was used the majority of the time by participants for both input and navigation even though most of the participants had little or no prior experience with speech systems (yet did have prior experience with text-only access to applications on their phones). To our knowledge this represents the first implementation and evaluation of its kind using this combination of technologies on an unmodified cell phone. Design implications resulting from the study findings and usability issues encountered are presented to inform the design of future conversational multimodal mobile applications.     

Exploiting convergence to improve natural language understanding

Convergence is the phenomenon in human dialogue whereby participants adopt characteristics of each other's speech. Communicants are unaware of this occurring. If it were possible to invoke such a phenomenon in a natural language interface it would provide a means of keeping user inputs within the range of lexical and syntactic coverage of the system, while keeping the dialogue ‘natural’ in the sense of requiring no more conscious effort in observing conventions of format than human-human dialogue.

A ‘Wizard of Oz’ study was conducted to test the feasibility of this technique. Subjects were required to type queries into what they thought was a natural language database querying system. On completion of input the system presented a paraphrase for confirmation by subjects before presenting the answer. The paraphrases were constructed using particular terms and syntactic structures. Subjects began to use these terms and structures spontaneously in subsequent queries.

Observation of convergence in human-computer dialogue suggests that the technique can be incorporated in user interfaces to improve communication. The implementation issues for natural language dialogue are discussed, and other applications of the technique in HCI are outlined.     

Evaluating the usability of natural language query languages and interfaces to Semantic Web knowledge bases

The need to make the contents of the Semantic Web accessible to end-users becomes increasingly pressing as the amount of information stored in ontology-based knowledge bases steadily increases. Natural language interfaces (NLIs) provide a familiar and convenient means of query access to Semantic Web data for casual end-users. While several studies have shown that NLIs can achieve high retrieval performance as well as domain independence, this paper focuses on usability and investigates if NLIs and natural language query languages are useful from an end-user's point of view. To that end, we introduce four interfaces each allowing a different query language and present a usability study benchmarking these interfaces. The results of the study reveal a clear preference for full natural language query sentences with a limited set of sentence beginnings over keywords or formal query languages. NLIs to ontology-based knowledge bases can, therefore, be considered to be useful for casual or occasional end-users. As such, the overarching contribution is one step towards the theoretical vision of the Semantic Web becoming reality.     

Automatic speech grammar generation during conceptual modelling of virtual environments

Speech interfaces are becoming more and more popular as a means to interact with virtual environments but the development and integration of these interfaces is usually still ad hoc, especially the speech grammar creation of the speech interface is a process commonly performed by hand. In this paper, we introduce an approach to automatically generate a speech grammar which is generated using semantic information. The semantic information is represented through ontologies and gathered from the conceptual modelling phase of the virtual environment application. The utterances of the user will be resolved using queries onto these ontologies such that the meaning of the utterance can be resolved. For validation purposes we augmented a city park designer with our approach. Informal tests validate our approach, because they reveal that users mainly use words represented in the semantic data, and therefore also words which are incorporated in the automatically generated speech grammar.

Conversational natural language understanding interfacing city event information

The article describes aspects of the development of a conversational natural language understanding (NLU) system done during the first year of the European research project CATCH-2004 (Converse in AThens Cologne and Helsinki) [http://www.catch2004.org]. The project is co-funded by the European Union in the scope of the IST programme (IST 1999-11103).

Its objectives focus on multi-modal, multi-lingual conversational natural language access to information systems. The paper emphasises on architecture, and telephony-based speech and NLU components as well as aspects of the implementation of a city event information (CEI) system in English, Finnish, German and Greek. The CEI system accesses two different databases in Athens and Helsinki using a common retrieval interface. Furthermore the paper singles out methodologies involved for acoustic and language model of the speech recognition component, parsing techniques and dialog modelling for the conversational natural language subsystem. For the implementation it outlines an incremental system refinement methodology necessary to adapt the system components to real-life data. It addresses the implementation of language specific characteristics and a common dialog design for all four languages, but also deals with aspects towards a multilingual conversational system. Finally, it presents prospects for further developments of the project.     

智能电话查询系统

介绍了一个用于交通信息查询的交互式对话系统-ShanghaiQuest。该系统集成了语音识别、文语转换和自然语言理解等技术，用户可以同这样一个自动的交通代理对话以获取所需的交通信息。     

A collaborative behavior-based approach for handling ambiguity, uncertainty, and vagueness in robot natural language interfaces

Service robots are used by ordinary people in their houses and offices. For such users, a desirable way to communicate with robots is through natural language interfaces. So far, techniques for developing robot natural language interfaces are far from mature. A major challenge is handling ambiguity, uncertainty, and vagueness in parsing, object resolution, and vague natural language words. In this research, we develop a new approach called the collaborative behavior-based approach, in which behaviors of robots and behaviors of human users, as well as the changes of object states caused by the behaviors, are taken into consideration integratedly in processing natural language user instructions. In this paper, we analyze the special features of a human–robot interface that may affect language understanding, describe our approach that is designed based on the features, and present the implementation and some experimental results.     

Collaborative conversational interfaces

This paper proposes a method of designing human-computer speech interfaces based on principles of human conversation. It argues that conversation is the primary mode of language use and that it is fundamentally collaborative. Speech interfaces should therefore be designed to recreate the collaborative nature of natural conversations. The paper presents five strategies for designingcollaborative conversational interfaces, and it describes the principles of human-language use that underly these strategies. The paper also argues that collaborative conversational interfaces have a crucial advantage over other kinds of interfaces in that they are readily adaptive to different levels of experience and styles of use. The paper gives examples of collaborative conversational interfaces that we have developed, and discusses the ways in which these interfaces have been made adaptive.     

Sisl: Several Interfaces, Single Logic

Modern interactive services such as information and e-commerce services are becoming increasingly more flexible in the types of user interfaces they support. These interfaces incorporate automatic speech recognition and natural language understanding and include graphical user interfaces on the desktop and web-based interfaces using applets and HTML forms. To what extent can the user interface software be decoupled from the service logic software (the code that defines the essential function of a service)? Decoupling of user interface from service logic directly impacts the flexibility of services, or how easy they are to modify and extend. To explore these issues, we have developed Sisl, an architecture and domain-specific language for designing and implementing interactive services with multiple user interfaces. A key principle underlying Sisl is that all user interfaces to a service share the same service logic. Sisl provides a clean separation between the service logic and the software for a variety of interfaces, including Java applets, HTML pages, speech-based natural language dialogue, and telephone-based voice access. Sisl uses an event-based model of services that allows service providers to support interchangeable user interfaces (or add new ones) to a single consistent source of service logic and data. As part of a collaboration between research and development, Sisl is being used to prototype a new generation of call processing services for a Lucent Technologies switching product.     

Enhancing information systems management with natural language processing techniques

Natural language and databases are core components of information systems. They are related to each other because they share the same purpose: the conceptualization aspects of the real world in order to deal with them in some way. Natural language processing (NLP) techniques may substantially enhance most phases of the information system lifecycle, starting with requirements analysis, specification and validation, and going up to conflict resolution, result processing and presentation. Furthermore, natural language based query languages and user interfaces facilitate the access to information for anyone and allow for new paradigms in the usage of computerized services. This paper investigates the use of NLP techniques in the design phase of information systems. Then, it reports on data base querying and information retrieval enhanced with NLP.     

On creating multimodal virtual humans—real time speech driven facial gesturing

Because of extensive use of different computer devices, human-computer interaction design nowadays moves towards creating user centric interfaces. It assumes incorporating different modalities that humans use in everyday communication. Virtual humans, who look and behave believably, fit perfectly in the concept of designing interfaces in more natural, effective, as well as social oriented way. In this paper we present a novel method for automatic speech driven facial gesturing for virtual humans capable of real time performance. Facial gestures included are various nods and head movements, blinks, eyebrow gestures and gaze. A mapping from speech to facial gestures is based on the prosodic information obtained from the speech signal. It is realized using a hybrid approach—Hidden Markov Models, rules and global statistics. Further, we test the method using an application prototype—a system for speech driven facial gesturing suitable for virtual presenters. Subjective evaluation of the system confirmed that the synthesized facial movements are consistent and time aligned with the underlying speech, and thus provide natural behavior of the whole face.     

Speak-Correct: A Computerized Interface for the Analysis of Mispronounced Errors

Any natural language may have dozens of accents. Even though the equivalent phonemic formation of the word, if it is properly called in different accents, humans do have audio signals that are distinct from one another. Among the most common issues with speech, the processing is discrepancies in pronunciation, accent, and enunciation. This research study examines the issues of detecting, fixing, and summarising accent defects of average Arabic individuals in English-speaking speech. The article then discusses the key approaches and structure that will be utilized to address both accent flaws and pronunciation issues. The proposed SpeakCorrect computerized interface employs a cutting-edge speech recognition system and analyses pronunciation errors with a speech decoder. As a result, some of the most essential types of changes in pronunciation that are significant for speech recognition are performed, and accent defects defining such differences are presented. Consequently, the suggested technique increases the Speaker’s accuracy. SpeakCorrect uses 100 h of phonetically prepared individuals to construct a pronunciation instruction repository. These prerecorded sets are used to train Hidden Markov Models (HMM) as well as weighted graph systems. Their speeches are quite clear and might be considered natural. The proposed interface is optimized for use with an integrated phonetic pronounced dataset, as well as for analyzing and identifying speech faults in Saudi and Egyptian dialects. The proposed interface detects, analyses, and assists English learners in correcting utterance faults, overcoming problems, and improving their pronunciations.     

Interactive navigation interface for Virtual Reality using the human body

The use of Virtual Reality (VR) and interactive real-time rendering in urban planning and building design is becoming more and more common. However, the integration of desktop-VR in the urban planning process suffers from complicated navigation interfaces. In particular, people unfamiliar to gaming environments and computers are less prone to interact with a VR visualization using keyboard and mouse as controlling devices. This paper addresses this issue by presenting an implementation of the XBOX 360 Kinect sensor system, which uses the human body to interact with the virtual environment. This type of interaction interface enables a more natural and user-friendly way of interacting with the virtual environment. The validation of the system was conducted with 60 participants using quantitative and qualitative methods. The result showed that participants perceived the interface as non-demanding and easy to use and the interface was perceived better in relation to mouse/keyboard interaction. The implemented interface supported users to switch between different architecture proposals of an urban plan and the switching positively affected learning, understanding and spatial reasoning of the participants. The study also shows that females perceived the system as less demanding than males. Furthermore, the users associated and related their body (human interaction interface) to VR, which could indicate that they used their body during spatial reasoning. This type of spatial reasoning has been argued to enhance the spatial-perception.     

Cross-Fertilization Between Human Computer Interaction and Natural Language Processing: Why and How

Many of the central notions and ultimate goals of both human-computer interaction (HCI) and natural language processing (NLP) are common to both disciplines. Both are concerned with communication as a core concept, and both attempt to maximize the naturalness of this communication for the end-user. A central challenge to both disciplines is the issue of the choice and adaptation of the appropriate form of communication for the specific user and context at hand. Despite these strong commonalities, we observe very little collaboration, cross-references or even mutual knowledge between the HCI and NLP communities. And, surprisingly enough, although their goals might be very similar, the methods and the evaluation frameworks used in both research and applicative work in both areas are distinct. We think that it is time to step back and re-assess the potential for collaboration between the two disciplines.In this paper, we argue that importing ideas and methods from each discipline into the other can be fruitful, and we review specific areas where this is the case. We argue that cross-fertilization between HCI and NLP is desirable in wider and in more fundamental ways than only for the design of natural language interfaces. The reflection presented in this paper is motivated by our own work over the last four years in a team comprised of both HCI and specialists in natural language generation (NLG), a subfield of NLP specifically concerned with the automatic generation of language.     

Japanese activities in virtual reality

Most virtual reality hardware in Japan has been imported from the US, and the large-scale applications have focused on games, where Sega Enterprises and Nintendo remain the market leaders. In addition to its successful home video and arcade machines businesses, Sega is moving aggressively into the amusement-park ride market. Outside these markets, Matsushita plans to release three commercial products in spring 1994. One is an immersive system for product showrooms to help Japanese home owners visualize new kitchens before ordering them. Another is a massage chair that uses VR images and sound to help relax users. The third is an exercise bicycle that lets the user cycle through a 3D virtual world while exercising. Many Japanese companies are interested in VR technology as a problem-solving tool. They are making progress in its use in equipment operation. They also realize that the ability to operate devices intuitively by ordinary gesture-like motions can be exploited in many practical fields. This ability could make human-machine interfaces easier, even if these interfaces fall short of outright VR. This article reviews a few recent industry and academic research projects. The current worldwide recession has flattened the overall rate of growth in Japanese research and development spending. Nevertheless, Japan reportedly has approximately 20 major VR research projects, mostly emphasizing communication. Japanese emphases on long-term planning and on targeting development toward specific products are their strengths in this emerging technology