Pervasive speech recognition

As mobile computing devices grow smaller and as in-car computing platforms become more common, we must augment traditional methods of human-computer interaction. Although speech interfaces have existed for years, the constrained system resources of pervasive devices, such as limited memory and processing capabilities, present new challenges. We provide an overview of embedded automatic speech recognition (ASR) on the pervasive device and discuss its ability to help us develop pervasive applications that meet today's marketplace needs. ASR recognizes spoken words and phrases. State-of-the-art ASR uses a phoneme-based approach for speech modeling: it gives each phoneme (or elementary speech sound) in the language under consideration a statistical representation expressing its acoustic properties.     

A Listening Keyboard for Users with Motor Impairments—A Usability Study

Computer users with motor impairments find it difficult and, in many cases, impossible to access PC functionality through the physical keyboard-and-mouse interface. Studies show that even able-bodied users experience similar difficulties when interacting with mobile devices; this is due to the reduced size/usability of the input interfaces. Advances in speech recognition have made it possible to design speech interfaces for alphanumeric data entry and indirect manipulation (cursor control). Although several related commercial applications exist, such systems do not provide a complete solution for arbitrary keyboard and mouse access, such as the access needed for, say, typing, compiling, and executing a C++ program.We carried out a usability study to support the development of a speech user interface for arbitrary keyboard access and mouse control. The study showed that speech interaction with an ideal listening keyboard is better for users with motor impairments than handstick, in terms of task completion time (37% better), typing rate (74% better), and error rates (63% better). We believe that these results apply to both permanent and task-induced motor impairments. In particular, a follow-up experiment showed that handstick approximates conventional modes of alphanumeric input available on mobile devices (e.g., PDAs, cellular phones, and personal organizers). These modes of input include miniaturized keyboards, stylus soft keyboards, cellular phone numberpads, and handwriting recognition software. This result suggests that a listening keyboard would be an effective mode for alphanumeric input on future mobile devices.This study contributed to the development of SUITEKeys—a speech user interface for arbitrary keyboard and mouse access available for MS platforms as freeware.     

A natural language model for managing TV-Anytime information in mobile environments

The TV-Anytime standard describes the structures of categories of digital TV program metadata, as well as user profile metadata for TV programs. We describe a natural language (NL) model for the users to interact with the TV-Anytime metadata and preview TV programs from their mobile devices. The language utilises completely the TV-Anytime metadata specifications (upper ontologies), as well as domain-specific ontologies. The interaction model does not use clarification dialogues, but it uses the user profiles as well as TV-Anytime metadata information and ontologies to rank the possible responses in case of ambiguities. We describe implementations of the model that run on a PDA and on a mobile phone, and manage the metadata on a remote TV-Anytime-compatible TV set. We present user evaluations of the approach. Finally, we propose a generalised implementation framework that can be used to easily provide NL interfaces for mobile devices for different applications and ontologies.     

Interface with angels: the future of VIR and AR interfaces

Although real guardian angels aren't easy to get hold of, some of the computer technology needed for such a personal assistant is already available. Other parts exist in the form of research prototypes, but some technological breakthroughs are necessary before we can realize their potential, let alone integrate into our daily routines. Future VR and AR interfaces won't necessarily try to provide a perfect imitation of reality but instead will adapt their display mechanisms to their users' individual requirements. The emergence of these interfaces won't rely on a single technology but will depend on the advances in many areas, including computer graphics, display technology, tracking and recognition devices, natural and intuitive interactions, 3D interaction techniques, mobile and ubiquitous computing, intelligent agents, and conversational user interfaces, to name a few. The guardian angel scenario exemplifies how future developments in AR and VR user interfaces might change the way we interact with computers. Although this example is just one of several plausible scenarios, it demonstrates that AR and VP, in combination with user-centered design of their post-WIMP interfaces, can provide increased access, convenience, usability, and efficiency     

Bringing context-aware access to the web through spoken interaction

The web has become the largest repository of multimedia information and its convergence with telecommunications is now bringing the benefits of web technology to hand-held devices. To optimize data access using these devices and provide services which meet the user needs through intelligent information retrieval, the system must sense and interpret the user environment and the communication context. In addition, natural spoken conversation with handheld devices makes possible the use of these applications in environments in which the use of GUI interfaces is not effective, provides a more natural human-computer interaction, and facilitates access to the web for people with visual or motor disabilities, allowing their integration and the elimination of barriers to Internet access. In this paper, we present an architecture for the design of context-aware systems that use speech to access web services. Our contribution focuses specifically on the use of context information to improve the effectiveness of providing web services by using a spoken dialog system for the user-system interaction. We also describe an application of our proposal to develop a context-aware railway information system, and provide a detailed evaluation of the influence of the context information in the quality of the services that are supplied.     

Mobile information access in the real world: A story of three wireless devices

The importance of the user perspective to the wireless information access experience cannot be understated: simply put, users will not indulge in devices that are perceived to be difficult to use and in technologies that do not offer quality infotainment – combined information and entertainment – content. In this paper, we investigate the impact that mobile devices have on the user wireless infotainment access experience in practice. To this end, we have undertaken an empirical study placed in a ‘real-world’ setting, in which participants undertook typical infotainment access tasks on three different wireless-enabled mobile devices: a laptop, a personal digital assistant and a head mounted display device. Results show that, with the exception of participants’ level of self-consciousness when using such devices in public environments, the user wireless information access experience is generally unaffected by device type. Location was shown, though, to be a significant factor when users engage in tasks such as listening to online music or navigation. Whilst the interaction between device and environment was found to influence entertainment-related tasks in our experiments, the informational ones were not affected. However, the interaction effects between device and user type was found to affect both types of tasks. Lastly, a user’s particular computing experience was shown to influence the perceived ease of wireless information access only in the case of online searching, irrespective of whether this is done for primarily informational purposes or entertainment ones.     

A soft computing based location-aware access control for smart buildings

The evolution of wireless communications and pervasive computing is transforming current physical spaces into real smart environments. These emerging scenarios are expected to be composed by a potentially huge amount of heterogeneous smart objects which can be remotely accessed by users via their mobile devices anytime, anywhere. In this paper, we propose a distributed location-aware access control mechanism and its application in the smart building context. Our approach is based on an access control engine embedded into smart objects, which are responsible to make authorization decisions by considering both user location data and access credentials. User location data are estimated using a novel indoor localization system based on magnetic field data sent by user through her personal phone. This localization system implements a combination of soft computing techniques over the data collected by smartphones. Therefore, our location-aware access control mechanism does not require any intermediate entity, providing the benefits of a decentralized approach for smart environments. From the results obtained, we can consider our proposal as a promising approach to tackle the challenging security requirements of typical pervasive environments.     

Two-handed gesture recognition and fusion with speech to command a robot

Assistance is currently a pivotal research area in robotics, with huge societal potential. Since assistant robots directly interact with people, finding natural and easy-to-use user interfaces is of fundamental importance. This paper describes a flexible multimodal interface based on speech and gesture modalities in order to control our mobile robot named Jido. The vision system uses a stereo head mounted on a pan-tilt unit and a bank of collaborative particle filters devoted to the upper human body extremities to track and recognize pointing/symbolic mono but also bi-manual gestures. Such framework constitutes our first contribution, as it is shown, to give proper handling of natural artifacts (self-occlusion, camera out of view field, hand deformation) when performing 3D gestures using one or the other hand even both. A speech recognition and understanding system based on the Julius engine is also developed and embedded in order to process deictic and anaphoric utterances. The second contribution deals with a probabilistic and multi-hypothesis interpreter framework to fuse results from speech and gesture components. Such interpreter is shown to improve the classification rates of multimodal commands compared to using either modality alone. Finally, we report on successful live experiments in human-centered settings. Results are reported in the context of an interactive manipulation task, where users specify local motion commands to Jido and perform safe object exchanges.     

Mobile access to real-time information—the case of autonomous stock brokering

When services providing real-time information are accessible from mobile devices, functionality is often restricted and no adaptation of the user interface to the mobile device is attempted. Mobile access to real-time information requires designs for multi-device access and automated facilities for the adaptation of user interfaces. We present TapBroker, a push update service that provides mobile and stationary access to information on autonomous agents trading stocks. TapBroker is developed for the Ubiquitous Interactor system and is accessible from Java Swing user interfaces and Web user interfaces on desktop computers, and from a Java Awt user interface on mobile phones. New user interfaces can easily be added without changes in the service logic.     

Sisl: Several Interfaces, Single Logic

Modern interactive services such as information and e-commerce services are becoming increasingly more flexible in the types of user interfaces they support. These interfaces incorporate automatic speech recognition and natural language understanding and include graphical user interfaces on the desktop and web-based interfaces using applets and HTML forms. To what extent can the user interface software be decoupled from the service logic software (the code that defines the essential function of a service)? Decoupling of user interface from service logic directly impacts the flexibility of services, or how easy they are to modify and extend. To explore these issues, we have developed Sisl, an architecture and domain-specific language for designing and implementing interactive services with multiple user interfaces. A key principle underlying Sisl is that all user interfaces to a service share the same service logic. Sisl provides a clean separation between the service logic and the software for a variety of interfaces, including Java applets, HTML pages, speech-based natural language dialogue, and telephone-based voice access. Sisl uses an event-based model of services that allows service providers to support interchangeable user interfaces (or add new ones) to a single consistent source of service logic and data. As part of a collaboration between research and development, Sisl is being used to prototype a new generation of call processing services for a Lucent Technologies switching product.     

Toward IP converged heterogeneous mobility: A network controlled approach

Envisioning a future where mobile terminals equipped with one or more network devices are able to roam across wireless or wired networks, in a diverse macro and micro wireless cells environment, requires the development of enhanced methods to control IP-based mobility. These methods should consider traditional terminal mobility (mainly due to user movement) as well as mobility across heterogeneous networks in the presence of semi-static users. For this to become reality, a cross layer interaction is required starting from a potentially large diversity of layer two access technologies up to the common IP layer, allowing the exchange of messages between terminals and network components. Furthermore, traditional host mobility driven concepts need to evolve, and include more stringent mobile operator requirements in context of fully driven network controlled mobility. This paper presents and evaluates a novel framework design, based on the IEEE 802.21 future standard, encompassing network driven as well as host driven mobility. This paper evaluates signalling aspects, algorithm design and performance issues.     

Semantic management of multiple contexts in a pervasive computing framework

Mobile devices can perceive greater details of user states with the increasing integration of mobile sensors into a pervasive computing framework, yet they consume large amounts of batteries and computational resources. This paper proposes a semantic management method which efficiently integrates multiple contexts into the mobile system by analyzing the semantic hierarchy and temporal relations. The proposed method semantically decides the recognition order of the contexts and identifies each context using a corresponding dynamic Bayesian network (DBN). To sort out the contexts, we designed a semantic network using a knowledge-driven approach, whereas DBNs are constructed with a data-driven approach. The proposed method was validated on a pervasive computing framework, which included multiple mobile sensors (such as motion sensors, data-gloves, and bio-signal sensors). Experimental results showed that the semantic management of multiple contexts dramatically reduced the recognition cost.     

Selecting proper wireless network interfaces for user experience enhancement with guaranteed probability

With the increasing capabilities of mobile phones, mobile users access data via wireless interfaces pervasively. Although WiFi has limited coverage and resulted in a bigger delay of data access, it is not uncommon that mobile users are willing to use WiFi to transmit data to decrease communication costs instead of 3G. Hence it is reasonable to use delay tolerance strategies to balance execution time, energy consumption, and communication cost. In this paper, we model mobile user experience as a combination of three random variables (energy consumption, execution time and communication cost). We present a wireless interface scheduling algorithm to select proper wireless interfaces for a set of data-dependent sporadic tasks to enhance user experience under the constraints of execution time, energy consumption, and communication cost with a guaranteed confidence probability in a delay-tolerant environment. The experimental results show that our approach can effectively enhance the user experience.     

无线传感器网络便携式用户接口的设计与实现

重点考察无线传感器网络的人机交互问题,设计并实现了一种基于手持移动设备的无线传感器网络用户接口,以方便人们对传感器网络节点的灵活访问与控制。该用户接口搭建于Pocket PC平台,通过蓝牙与汇聚节点进行双向通信,从而实现对整个无线传感器网络系统的监控。用户可以通过便携式接口现场采集与处理传感器网络节点的数据,同时也可以发送指令对汇聚节点和普通节点进行各种设置。演示实验结果验证了该接口系统的可行性与有效性。     

Motion does matter: an examination of speech-based text entry on the move

Desktop interaction solutions are often inappropriate for mobile devices due to small screen size and portability needs. Speech recognition can improve interactions by providing a relatively hands-free solution that can be used in various situations. While mobile systems are designed to be transportable, few have examined the effects of motion on mobile interactions. This paper investigates the effect of motion on automatic speech recognition (ASR) input for mobile devices. Speech recognition error rates (RER) have been examined with subjects walking or seated, while performing text input tasks and the effect of ASR enrollment conditions on RER. The obtained results suggest changes in user training of ASR systems for mobile and seated usage.     

普适计算中多通道交互建模研究

计算机处理速度和性能的迅猛提高并没有相应提高用户使用计算机交互的能力,其中一个重要原因就是缺少一个与之相适应的高效、自然的人机交互界面.普适计算的发展为人们提供了一个无处不在的获取服务的环境,而多通道用户界面允许用户使用自然的交互方式,如语音、手势、表情等在此环境中与系统进行协同工作,改变了传统交互手段的局限性.重点讨论了基于笔和语音的多通道交互,给出了普适环境中对多通道交互建模的框架,进一步研究了在图形信息应用中的多通道交互实例,论证了创建自然多通道交互方式的优势.     

The Case for VM-Based Cloudlets in Mobile Computing

Mobile computing continuously evolve through the sustained effort of many researchers. It seamlessly augments users' cognitive abilities via compute-intensive capabilities such as speech recognition, natural language processing, etc. By thus empowering mobile users, we could transform many areas of human activity. This article discusses the technical obstacles to these transformations and proposes a new architecture for overcoming them. In this architecture, a mobile user exploits virtual machine (VM) technology to rapidly instantiate customized service software on a nearby cloudlet and then uses that service over a wireless LAN; the mobile device typically functions as a thin client with respect to the service. A cloudlet is a trusted, resource-rich computer or cluster of computers that's well-connected to the Internet and available for use by nearby mobile devices. Our strategy of leveraging transiently customized proximate infrastructure as a mobile device moves with its user through the physical world is called cloudlet-based, resource-rich, mobile computing. Crisp interactive response, which is essential for seamless augmentation of human cognition, is easily achieved in this architecture because of the cloudlet's physical proximity and one-hop network latency. Using a cloudlet also simplifies the challenge of meeting the peak bandwidth demand of multiple users interactively generating and receiving media such as high-definition video and high-resolution images. Rapid customization of infrastructure for diverse applications emerges as a critical requirement, and our results from a proof-of-concept prototype suggest that VM technology can indeed help meet this requirement.     

Embarking on Spoken-Language NL Interface Design

Natural language (NL) user interfaces are growing in popularity. Unfortunately, the complexity of NL interaction makes these interfaces difficult to design. For NL interfaces to become successful, universal tools are needed to help support the NL design process. What work practice should these tools explicitly support? Interviews with NL designers and our own experiments have identified a specific work practice that designers should consider as they begin to incorporate NL into user interface designs. The work practice study highlights the value of using Wizard of Oz prototyping in NL design. We describe a tool that we have built, called SUEDE, to explicitly support the first stage of NL design for spoken-language user interfaces. Our tools and tools like it will help make NL in human-computer interaction (HCI) more commonplace.     

Comparison of speech input and manual control of in-car devices while on the move

Speech recognition has a number of potential advantages over traditional manual controls for the operation of in-car and other mobile devices. Two laboratory experiments aimed to test these proposed benefits, and to optimise the design of future speech interfaces. Participants carried out tasks with a phone or in-car enteratainment system, while engaged in a concurrent driving task. Speech input reduced the adverse effects of system operation on driving performance, but manual control led to faster transaction times and improved task accuracy. Explicit feedback of the recognition results was found to be necessary, with audio-only feedback leading to better task performance than combined audio-plus-visual. It is recommended that speech technology is incorporated into the user interface as a redundant alternative to manual operation. However, the importance of good human factors in the design of speech dialogues is emphasised.