The value of handhelds in smart environments

The severe resource restrictions of computer-augmented everyday artifacts imply substantial problems for the design of applications in smart environments. Some of these problems can be overcome by exploiting the resources, I/O interfaces, and computing capabilities of nearby mobile devices in an ad-hoc fashion. We identify the means by which smart objects can make use of handheld devices such as PDAs and mobile phones, and derive the following major roles of handhelds in smart environments: (1) mobile infrastructure access point; (2) user interface; (3) remote sensor; (4) mobile storage medium; (5) remote resource provider; and (6) weak user identifier. We present concrete applications that illustrate these roles, and describe how handhelds can serve as mobile mediators between computer-augmented everyday artifacts, their users, and background infrastructure services. The presented applications include a remote interaction scenario, a smart medicine cabinet, and an inventory monitoring application.     

Environment Analysis as a Basis for Designing Multimodal and Multidevice User Interfaces

In this article, we present a practical approach to analyzing mobile usage environments. We propose a framework for analyzing the restrictions that characteristics of different environments pose on the user's capabilities. These restrictions along with current user interfaces form the cost of interaction in a certain environment. Our framework aims to illustrate that cost and what causes it. The framework presents a way to map features of the environment to the effects they cause on the resources of the user and in some cases on the mobile device. This information can be used for guiding the design of adaptive and/or multimodal user interfaces or devices optimized for certain usage environments. An example of using the framework is presented along with some major findings and three examples of applying them in user interface design.     

以交互为中心的Post-WIMP界面模型

随着硬件设备和软件技术的发展,国内外开展了大量基于Post-WIMP界面的新型交互技术的研究.面对多种交互设备以及使用环境,在界面设计的过程中,根据上下文选择适当的交互组件或技术,并有效地进行组合与评估,成为构造Post-WIMP界面的关键问题.将界面设计与应用语义分离,使设计者能够灵活地置换各种交互技术.通过对Post-WIMP界面交互过程与界面设计层次的分析,建立起以交互为中心的分层Post-WIMP界面模型,将交互的各个层次进行分离.在模型的基础上,描述一个了Post-WIMP界面生成工具.借助该工具,设计者能够在设计过程中方便地引入新的交互技术,并能在最终的软件系统中加以灵活应用,从而可以快速、有效地进行界面原型实现和迭代评估.应用实例表明,Post-WIMP模型的建立以及生成工具的实现,有利于设计者确定设计方案及对方案的整体评估.     

Supporting serendipitous integration in mobile computing environments

In the richly networked world of the near future, mobile computing users will be confronted with an ever-expanding array of devices and services accessible in their environments. In such a world, we cannot expect to have available to us specific applications that allow us to accomplish every conceivable combination of devices that we may wish. Instead, we believe that many of our interactions with the network will be characterized by the use of “general purpose” tools that allow us to discover, use, and integrate multiple devices around us. This paper lays out the case for why we believe that so-called “serendipitous integration” is a necessary fact that we will face in mobile computing, and explores a number of design experiments into supporting end user configuration and control of networked environments through general purpose tools. We present an iterative design approach to creating such tools and their user interfaces, discuss our observations about the challenges of designing for such a world, and then explore a number of tools that take differing design approaches to overcoming these challenges. We conclude with a set of reflections on the user experience issues that we believe are inherent in dealing with ad hoc mobile computing in richly networked environments.     

Trusting mobile user devices and security modules

The market for devices like mobile phones, multifunctional watches, and personal digital assistants is growing rapidly. Most of these mobile user devices need security for their prospective electronic commerce applications. While new technology has simplified many business and personal transactions, it has also opened the door to high-tech crime. We investigate design options for mobile user devices that are used in legally significant applications     

Natural language interface for information management on mobile devices

A natural language interface (NLI) enables the ease-of-use of information systems in performing sophisticated human - computer interaction. To address the challenges of mobile devices to user interaction in information management, we propose an NLI as a promising solution. In this paper, we review state-of-the-art NLI technologies and analyse user requirements for managing notable information on mobile devices. To minimize any technical difficulties arising from developing and improving the usability of NLI systems we develop general principles for NLI design, which fills in a gap in the literature. In order to satisfy user requirements for information management on mobile devices, we innovatively design NLI-enabled information management architecture. It is shown from two usage scenarios that the architecture could lead to reduced effort in user navigation and improved efficiency and effectiveness of managing information on mobile devices. We conclude the article with the implications of this study and suggestions for future direction.     

Development of a virtual aquarium system interacting with a smart device

New applications of smart devices interacting with other computing devices are recently providing interesting and feasible solutions in ubiquitous computing environments. In this study, we propose an interactive virtual aquarium system that interacts with a smart device as a user interface. We developed a virtual aquarium graphic system and a remote interaction application of a smart device for building an interactive virtual aquarium system. We performed an experiment that demonstrates the feasibility and the effectiveness of the proposed system as an example of a new type of interactive application of a smart display, where a smart device serves as a remote user interface.     

User requirements of mobile technology: results from a content analysis of user reviews

Advanced mobile technology continues to shape professional environments. Smart cell phones, pocket computers and laptop computers reduce the need of users to remain close to a wired information system infrastructure and allow for task performance in many different contexts. Among the consequences are changes in technology requirements, such as the need to limit weight and size of the devices. In the current paper, we focus on the factors that users find important in mobile devices. Based on a content analysis of online user reviews that was followed by structural equation modeling, we found four factors to be significantly related with overall user evaluation, namely functionality, portability, performance, and usability. Besides the practical relevance for technology developers and managers, our research results contribute to the discussion about the extent to which previously established theories of technology adoption and use are applicable to mobile technology. We also discuss the methodological suitability of online user reviews for the assessment of user requirements, and the complementarity of automated and non-automated forms of content analysis.     

Mixed reality participants in smart meeting rooms and smart home environments

Human–computer interaction requires modeling of the user. A user profile typically contains preferences, interests, characteristics, and interaction behavior. However, in its multimodal interaction with a smart environment the user displays characteristics that show how the user, not necessarily consciously, verbally and nonverbally provides the smart environment with useful input and feedback. Especially in ambient intelligence environments we encounter situations where the environment supports interaction between the environment, smart objects (e.g., mobile robots, smart furniture) and human participants in the environment. Therefore it is useful for the profile to contain a physical representation of the user obtained by multi-modal capturing techniques. We discuss the modeling and simulation of interacting participants in a virtual meeting room, we discuss how remote meeting participants can take part in meeting activities and they have some observations on translating research results to smart home environments.     

Automatic adaptation of mobile applications to different user devices using modular mobile agents

Wearable, handheld, and embedded or standalone intelligent devices are becoming quite common and can support a diverse range of applications. In order to simplify development of applications which can adapt to a variety of mobile devices, we propose an adaptation framework which includes three techniques: follow‐me, context‐aware adaptation, and remote control scheme. For the first, we construct a personal agent capable of carrying its owner's applications. Second, we design a personal agent capable of carrying applications with an adaptable hierarchical structure. Then, applications can be adapted approximately to the context of evices by using an attribute‐based component decision algorithm. Finally, to achieve a remote control scheme, we distribute the computational load of applications on the resource‐restricted mobile devices. An application is divided into two parts that can be executed on a user device and a server separately. In short, this framework facilitates the development of widespread applications for ubiquitous computing environments. Furthermore, it enables the applications to follow their owners and automatically adapt to different devices. Copyright © 2005 John Wiley & Sons, Ltd.     

Exploring multi-user interactions with dynamic NFC-displays

Near Field Communication (NFC) is an emerging technology for touch-based mobile interactions with single- and multi-tagged objects. Although the latter may allow for simultaneous and collaborative interactions, most prototypes were not designed for multiple users and were only evaluated with single-user interactions. In this paper, we investigate the design, usability and user experience of multi-user interactions on dynamic NFC-displays. These interactive surfaces use a grid of NFC-tags for the direct manipulation of projected application user interfaces. In two user studies with three prototypes for multi-user interaction, we evaluated the performance of dynamic NFC-displays, interactions among users and the interplay between mobile devices and large displays.     

Beyond WIMP

The WIMP (windows, icons, menus, point-and-click devices) graphical user interface (GUI) is not an ideal interface. Expert users find pure WIMP GUIs frustratingly slow and thus use keyboard shortcuts. In addition, they don't scale well-GUI bloat accompanies feature bloat. The most serious limitation, however, is that WIMP GUIs were designed for keyboard-plus-mouse desktop computing environments-environments that takes advantage only of vision and primitive touch. Our goal should be to design user interfaces that match our human perceptual, cognitive, manipulative and social abilities. We want to interact as naturally with computers and intelligent devices as we communicate and collaborate with each other and as we manipulate our physical environments. Indeed, computers are increasingly being used to facilitate human communication, collaboration and social interaction. Therefore, we should increasingly focus on human-human interaction, not just human-computer interaction. This change in focus reflects not only changes in the mode of computer use but also the increasingly invisible nature of the computer in our environments. New environments must serve as forcing functions to give us far higher expectations for user interfaces than we have had previously. Furthermore, the potential of future interfaces for handicapped people is phenomenal. Thus, despite the technology challenges, the greatest challenge lies in our understanding of human capabilities and how to incorporate that understanding into new design tools, methodologies and user interfaces     

Human interaction with IoT-based smart environments

This paper describes concepts, design, implementation, and performance evaluation of a 3D-based user interface for accessing IoT-based Smart Environments (IoT-SE). The generic interaction model of the described work addresses some major challenges of Human-IoT-SE-Interaction such as cognitive overload associated with manual device selection in complex IoT-SE, loss of user control, missing system image or over-automation. To address these challenges we propose a 3D-based mobile interface for mixed-initiative interaction in IoT-SE. The 3D visualization and 3D UI, acting as the central feature of the system, create a logical link between physical devices and their virtual representation on the end user’s mobile devices. By so doing, the user can easily identify a device within the environment based on its position, orientation, and form, and access the identified devices through the 3D interface for direct manipulation within the scene. This overcomes the problem of manual device selection. In addition, the 3D visualization provides a system image for the IoT-SE, which supports users in understanding the ambience and things going on in it. Furthermore, the mobile interface allows users to control the amount and the way the IoT-SE automates the environment. For example, users can stop or postpone system triggered automatic actions, if they don’t like or want them. Users also can remove a rule forever. By so doing, users can delete smart behaviors of their IoT-SE. This helps to overcome the automation challenges. In this paper, we present the design, implementation and evaluation of the proposed interaction system. We chose smart meeting rooms as the context for prototyping and evaluating our interaction concepts. However, the presented concepts and methods are generic and could be adapted to similar environments such as smart homes. We conducted a subjective usability evaluation (ISO-Norm 9241/110) with 16 users. All in one the study results indicate that the proposed 3D-User Interface achieved a good high score according to the ISO-Norm scores.     

A natural language model for managing TV-Anytime information in mobile environments

The TV-Anytime standard describes the structures of categories of digital TV program metadata, as well as user profile metadata for TV programs. We describe a natural language (NL) model for the users to interact with the TV-Anytime metadata and preview TV programs from their mobile devices. The language utilises completely the TV-Anytime metadata specifications (upper ontologies), as well as domain-specific ontologies. The interaction model does not use clarification dialogues, but it uses the user profiles as well as TV-Anytime metadata information and ontologies to rank the possible responses in case of ambiguities. We describe implementations of the model that run on a PDA and on a mobile phone, and manage the metadata on a remote TV-Anytime-compatible TV set. We present user evaluations of the approach. Finally, we propose a generalised implementation framework that can be used to easily provide NL interfaces for mobile devices for different applications and ontologies.     

Social Tele-Embodiment: Understanding Presence

Humans live and interact within the real world but our current online world neglects this. This paper explores research into Personal Roving Presence (PRoP) devices that provide a physical mobile proxy, controllable over the Internet to provide tele-embodiment. Leveraging off of its physical presence in the remote space, PRoPs provide important human verbal and non-verbal communication cues. The ultimate goal is a computer mediated communication (CMC) tool for rich natural human interaction beyond currently available systems. This paper examines PRoP design choices, system architecture, social issues, and evaluations of several user studies.     

面向用户体验的手持移动设备软件界面设计

为进一步探索手持移动设备中的界面设计问题,提出面向用户体验的界面设计方法.通过对用户体验设计进行分析,提出界面提供者与界面接受者之间的映射模型;将界面分为显性要素和隐性要素,并建立界面隐性要素的显性化设计模型.针对全新的软件设计,通过知识在需求分析、开发设计和设计测试之间的传递与转化,构建了面向用户体验的界面设计过程模型.以"奥运伴我行"手机电视的软件开发为例,让用户参与设计全过程,采用设计人类学方法进行了用户研究和需求分析,通过迭代设计完成了原型设计、测试评估和开发设计,建立了系统原型并完成了可用性测试评估,验证了面向用户体验的界面设计理论体系.     

Remote navigation of a mobile robot in an RFID-augmented environment

In the current article, we address the problem of constructing radiofrequency identification (RFID)-augmented environments for mobile robots and the issues related to creating user interfaces for efficient remote navigation with a mobile robot in such environments. First, we describe an RFID-based positioning and obstacle identification solution for remotely controlled mobile robots in indoor environments. In the robot system, an architecture specifically developed by the authors for remotely controlled robotic systems was tested in practice. Second, using the developed system, three techniques for displaying information about the position and movements of a remote robot to the user were compared. The experimental visualization techniques displayed the position of the robot on an indoor floor plan augmented with (1) a video view from a camera attached to the robot, (2) display of nearby obstacles (identified using RFID technology) on the floor plan, and (3) both features. In the experiment, test subjects controlled the mobile robot through predetermined routes as quickly as possible avoiding collisions. The results suggest that the developed RFID-based environment and the remote control system can be used for efficient control of mobile robots. The results from the comparison of the visualization techniques showed that the technique without a camera view (2) was the fastest, and the number of steering motions made was smallest using this technique, but it also had the highest need for physical human interventions. The technique with both additional features (3) was subjectively preferred by the users. The similarities and differences between the current results and those found in the literature are discussed.     

Ubiquitous Access to Unified Messaging: A Study of Usability and the Use of Pervasive Computing

The advent of mobile devices and the wireless Internet is having a profound impact on the way people communicate, as well as on the user interaction paradigms used to access information that was traditionally accessible only through visual interfaces. Applications for mobile devices entail the integration of various data sources optimized for delivery to limited hardware resources and intermittently connected devices through wireless networks. Although telephone interfaces arise as one of the most prominent pervasive applications, they present interaction challenges such as the augmentation of speech recognition through natural language (NL) understanding and high-quality text-to-speech conversion. This article presents an experience in building an automated assistant that is natural to use and could become an alternative to a human assistant. The Mobile Assistant (MA) can read e-mail messages, book appointments, take phone messages, and provide access to personal-organizer information. Key components are a conversational interface, enterprise integration, and notifications tailored to user preferences. The focus of the research has been on supporting the pressing communication needs of mobile workers and overcoming technological hurdles such as achieving high accuracy speech recognition in noisy environments, NL understanding, and optimal message presentation on a variety of devices and modalities. The article outlines findings from the 2 broad field trials and lessons learned regarding the support of mobile workers with pervasive computing devices and emerging technologies.     

Exploring casual point-and-tilt interactions for mobile geo-blogging

People record and share their experiences through text, audio and video. Increasingly they do this blogging from mobile devices. We illustrate a novel, mobile, low interaction cost approach to support the creation of a rich record of journeys made and places encountered: by pointing and tilting a mobile, users indicate their interests in a location. We built three mobile prototypes to explore the approach—the first one combines gestures and visual map feedback; the second provides a simpler visual interface; the third supports eyes-free interaction, allowing the user to simply point-and-tilt, with no visual display required. We describe two field studies undertaken to understand the value of the interaction styles afforded, then continue with a further user study to assess the interaction speed and accuracy between these interaction methods. We present the results of these studies and raise issues relevant to their design and to the wider class of devices and services concerned with mobile spatial information access.     

A Context-Aware Interaction Model for the Analysis of Users’ QoE in Mobile Environments

This article describes a novel approach to model the quality of experience (QoE) of users in mobile environments. The context-aware and ratings interaction model (CARIM) addresses the open questions of how to quantify user experiences from the analysis of interaction in mobile scenarios, and how to compare different QoE records to each other. A set of parameters are used to dynamically describe the interaction between the user and the system, the context in which it is performed and the perceived quality of users. CARIM structures these parameters into a uniform representation, supporting the dynamic analysis of interaction to determine QoE of users and enabling the comparison between different interaction records. Its run-time nature allows applications to make context- and QoE-based decisions in real time to adapt themselves, and thus provide a better experience to users. As a result, CARIM provides unified criteria for the inference and analysis of QoE in mobile scenarios. Its design and implementation can be integrated (and easily extended if needed) into many different development environments. An experiment with real users comparing two different interaction designs and validating user behavior hypotheses proved the effectiveness of applying CARIM for the assessment of QoE in mobile applications.