A Translucent Sketchpad for the Virtual Table Exploring Motion-based Gesture Recognition

The Virtual Table presents stereoscopic graphics to a user in a workbench-like setting. For this device, a user interface and new interaction techniques have been developed based on transparent props -a tracked hand-held pen and a pad. These props, particularly the pad, are augmented with 3D graphics from the Virtual Table's display that can serve as a palette for tools and controls as well as a window-like see-through interface, a plane-shaped and through-the-plane tool, supporting a variety of new interaction techniques. This paper reports on an extension of this user-interface design space which uses gestural input to create and control solid geometries for CAD and conceptual design. The application of gestural interfaces is a common method for interacting with virtual environments on a habitual and natural basis. The motion-based gesture recognition presented here uses Fuzzy Logic to support a predictable, flexible, and efficient learning process. This new interaction paradigm greatly increases the Virtual Table's suitability for design tasks. Traditional CAD dialogue can be combined with intuitive rapid sketching of geometry on the pad. Additionally, the resulting events and objects can be associated with scene details below the translucent tablet.     

The design-by-adaptation approach to universal access: learning from videogame technology

This paper proposes an alternative approach to the design of universally accessible interfaces to that provided by formal design frameworks applied ab initio to the development of new software. This approach, design-by-adaptation, involves the transfer of interface technology and/or design principles from one application domain to another, in situations where the recipient domain is similar to the host domain in terms of modelled systems, tasks and users. Using the example of interaction in 3D virtual environments, the paper explores how principles underlying the design of videogame interfaces may be applied to a broad family of visualization and analysis software which handles geographical data (virtual geographic environments, or VGEs). One of the motivations behind the current study is that VGE technology lags some way behind videogame technology in the modelling of 3D environments, and has a less-developed track record in providing the variety of interaction methods needed to undertake varied tasks in 3D virtual worlds by users with varied levels of experience. The current analysis extracted a set of interaction principles from videogames which were used to devise a set of 3D task interfaces that have been implemented in a prototype VGE for formal evaluation.     

Interactive Mechanical Design Variation for Haptics and CAD

A fast design variation technique for mechanical systems is presented. It is used to interactively optimize mechanical characteristics while "self-assembling" or satisfying large systems of mechanical constraints. The high speed method is central to providing inverse dynamics force feedback in haptics and control applications. Performance advantages with the use of augmented coordinates for inverse dynamics of closed loop topologies are also noted. The interaction framework allows manipulation of complex assemblies while maintaining kinematically admissible configurations though linkage and joint limit constraints. Furthermore, design variables such as link length can be treated as free variables and optimized to meet design criteria such as assembly dexterity. Assemblies with flexible bodies fit naturally within this framework. Thus, the contribution of this paper is the advancement of techniques in augmented coordinates for the kinematic and force feedback interaction with virtual mechanical assembly design optimization at force control rates.     

Egocentric Object Manipulation in Virtual Environments: Empirical Evaluation of Interaction Techniques

The acceptance of virtual environment (VE) technology requires scrupulous optimization of the most basic interactions in order to maximize user performance and provide efficient and enjoyable virtual interfaces. Motivated by insufficient understanding of the human factors design implications of interaction techniques and tools for virtual interfaces, this paper presents results of a formal study that compared two basic interaction metaphors for egocentric direct manipulation in VEs, virtual hand and virtual pointer, in object selection and positioning experiments. The goals of the study were to explore immersive direct manipulation interfaces, compare performance characteristics of interaction techniques based on the metaphors of interest, understand their relative strengths and weaknesses, and derive design guidelines for practical development of VE applications.     

Virtual prototyping of automated manufacturing systems with Geometry-driven Petri nets

The design process of automated manufacturing systems typically involves physical prototypes to validate the interactions between hardware and software components. However, physical prototyping is expensive and time consuming, which often leads to insufficient opportunities for testing early during the development cycle. Our objective is to improve this situation by providing a method to develop realistic prototypes using virtual reality technology that can be applied during earlier development stages. Our approach combines a virtual reality engine capable of enacting the laws of rigid body physics with a new hybrid software modelling language to control the simulated hardware using virtual sensors and actuators as they would be present in a physical prototype. The new modelling language is called Geometry-driven Petri nets (GPN) and combines a class of timed, high-level Petri nets with data structures used in state-of-the-art VR environments. This article describes the new GPN approach, applies it to a case study of an automated manufacturing line, and compares it with related approaches.     

The ALIVE system: wireless, full-body interaction with autonomous agents

The cumbersome nature of wired interfaces often limits the range of application of virtual environments. In this paper, we discuss the design and implementation of a novel system, called ALIVE, which allows unencumbered full-body interaction between a human participant and a rich graphical world inhabited by autonomous agents. Based on results obtained with thousands of users, the paper argues that this kind of system can provide more complex and very different experiences than traditional virtual reality systems. The ALIVE system significantly broadens the range of potential applications of virtual reality systems; in particular, the paper discusses novel applications in the area of training and teaching, entertainment, and digital assistants or interface agents. We give an overview of the methods used in the implementation of the existing ALIVE systems.     

Seamless Integration of Databases in VR for Constructing Virtual Environments

An approach for authoring virtual environments within the virtual environments themselves is presented, integrating a database containing arbitrary components that are used to construct the 3D scene. The issues important to a seamless integration of the database, such as multimedia data storage and database linkage are discussed, with the focus on the human-computer interaction component. A concept for a 3D database interface is described for query, presentation and usage of query results in the virtual environment. Finally, an implementation of the concept using VRML and Java is presented.     

Design evaluation and modification of mechanical systems in virtual environments

Design is one of the most important stages in the manufacturing cycle and influences all the subsequent stages of product development. In the context of today’s iterative design methodology, the modification of any design is a process involving many evaluations and improvements to the solutions chosen in earlier stages. For this purpose, in the most recent decade, 3D computer simulations have become common tools used within industry. Whilst virtual reality (VR) technology is seen as the interaction technology of the future, much of the current research in this area is carried out to explore the potential benefits and added value brought by the integration of this into standard software technologies currently used at various stages in manufacturing life cycle. A lot of attention has been given to exploring the usability and benefits of interactive VR for assembly planning, knowledge elicitation and design and simulation. However, little research has focussed on the analytical aspects of the design process, for example in the use of VR as an interface for simulation software in finite element methods and multi-body systems. This paper introduces research focussing on applications of virtual environments (VEs) for interactive design evaluation and modification adapted and used with standard simulation software. The use of such interactive visualisation offers the engineer more realistic real-time representations of the design and advanced facilities to interact with the model during the design process. While design evaluation is based mainly on visualisation; design modification requires interactive changes of the model during the simulation, and the interfaces described here highlights such applications. In this work, two software prototypes have been developed using VR technology. First, a software tool called design evaluation in virtual environments is presented together with an application in civil engineering to illustrate the mode of operation and added value of the use of an interactive visualisation environment. Linking the simulation software with the VE provides real time bi-directional communication of graphical information which can be successfully achieved even within the limits of current computer technology. The tool includes a suite of software modules and a user interface to facilitate the link between the simulation results and the VE. The second tool facilitates the design modification in virtual environments system by providing real time dynamic simulation. Two dynamic approaches are investigated in order to study the real time simulation issues in the context of design modification and system performance: the classic approach based on rigid interconnected bodies, and a new and novel approach developed by the authors based on particles dynamics. Both implementations have been tested and compared on a mechanism application under the same computing conditions. The research applications presented demonstrate the practicality, flexibility and versatility of the visualisation in virtual environment in design evaluation and modification. However, the computer efficiency whilst carrying out real time dynamic simulation is limiting the range of applications to models of moderate size; however, this is an improvement on previous similar applications.     

应用虚拟现实技术的人机交互仿真系统开发

针对现有虚拟仿真系统在人机交互、实时响应、数据采集等方面存在的不足,综合运用多种虚拟现实软、硬件技术构建出一个可供人与虚拟操作对象进行实时人机交互的仿真系统。利用数据转换技术将设计对象导入该系统后,系统能通过实时采集人机交互过程中的相关操作数据,测量人机交互绩效,继而评价设计质量。实例应用证明了系统的可行性。     

Designing the User Interface for Multimodal Speech and Pen-Based Gesture Applications: State-of-the-Art Systems and Future Research Directions

The growing interest in multimodal interface design is inspired in large part by the goals of supporting more transparent, flexible, efficient, and powerfully expressive means of human-computer interaction than in the past. Multimodal interfaces are expected to support a wider range of diverse applications, be usable by a broader spectrum of the average population, and function more reliably under realistic and challenging usage conditions. In this article, we summarize the emerging architectural approaches for interpreting speech and pen-based gestural input in a robust manner-including early and late fusion approaches, and the new hybrid symbolic-statistical approach. We also describe a diverse collection of state-of-the-art multimodal systems that process users' spoken and gestural input. These applications range from map-based and virtual reality systems for engaging in simulations and training, to field medic systems for mobile use in noisy environments, to web-based transactions and standard text-editing applications that will reshape daily computing and have a significant commercial impact. To realize successful multimodal systems of the future, many key research challenges remain to be addressed. Among these challenges are the development of cognitive theories to guide multimodal system design, and the development of effective natural language processing, dialogue processing, and error-handling techniques. In addition, new multimodal systems will be needed that can function more robustly and adaptively, and with support for collaborative multiperson use. Before this new class of systems can proliferate, toolkits also will be needed to promote software development for both simulated and functioning systems.     

On multiple model control for multiple contact systems

Multiple contact interaction is common in many robotic applications. These include grasping, wheeled vehicles, and distributed manipulation. All of these applications are capable of experiencing stick/slip phenomena at the contact interfaces. In particular, when there are more kinematic constraints then there are degrees of freedom, some contact interfaces must slip. Moreover, this stick/slip behavior is difficult to predict a priori due to strong sensitivities with respect to friction modeling and normal forces. Motivated by a simple multi-point manipulation device, we discuss how these effects can be accounted for and mitigated using tools from hybrid systems theory and multiple model adaptive control both for analysis and control design. In the context of the multi-point manipulation example, we show how multiple model supervisor-based adaptive control can provide stabilizing controllers for such systems. Our results are validated with simulations that both illustrate the need for these techniques and show their effectiveness.     

State of the Art of Molecular Visualization in Immersive Virtual Environments

Visualization plays a crucial role in molecular and structural biology. It has been successfully applied to a variety of tasks, including structural analysis and interactive drug design. While some of the challenges in this area can be overcome with more advanced visualization and interaction techniques, others are challenging primarily due to the limitations of the hardware devices used to interact with the visualized content. Consequently, visualization researchers are increasingly trying to take advantage of new technologies to facilitate the work of domain scientists. Some typical problems associated with classic 2D interfaces, such as regular desktop computers, are a lack of natural spatial understanding and interaction, and a limited field of view. These problems could be solved by immersive virtual environments and corresponding hardware, such as virtual reality head-mounted displays. Thus, researchers are investigating the potential of immersive virtual environments in the field of molecular visualization. There is already a body of work ranging from educational approaches to protein visualization to applications for collaborative drug design. This review focuses on molecular visualization in immersive virtual environments as a whole, aiming to cover this area comprehensively. We divide the existing papers into different groups based on their application areas, and types of tasks performed. Furthermore, we also include a list of available software tools. We conclude the report with a discussion of potential future research on molecular visualization in immersive environments.     

A haptic-based approach to virtual training for aerospace industry

In the last years, the industrial world has been increasingly adopting computer-aided solutions for design for maintainability and maintenance training tasks with the goal to reduce development costs and to shorten time, and to improve product and service quality. Computer-based training systems created to simulate machine assembly maintenance are normally operated by means of ordinary human–computer interfaces (keyboard, mouse, etc.), but this usually results in systems that are far from the real procedures, and therefore not effective in terms of training. In this study, we show that a better solution may come from the combination of virtual reality techniques and haptic interaction. To this regard, we present the results of a research aimed at testing and evaluating the effectiveness of the haptic feedback for first-person maintenance tasks targeted to the aerospace industry. The proposed system implements an interaction environment in which each of the main maintenance activities can be simulated by the trainee exploiting a hand-based commercial haptic device, operated by means of specific haptic-rendering techniques to provide realistic feedbacks during manipulation. A usability study is included to help assessing the potential of this approach.     

Toward the design of transitional interfaces: an exploratory study on a semi-immersive hybrid user interface

A task that can be decomposed into subtasks with different technological demands may be a challenge, since it requires multiple interactive environments as well as transitions between them. Some of these transitions may involve changes in hardware devices and interface paradigms at the same time. Some previous works have proposed various setups for hybrid user interfaces, but none of them focused on the design of transition interactions. Our work emphasizes the importance of interaction continuity as a guideline in the design and evaluation of transitional interfaces within a hybrid user interface (HUI). Finally, an exploratory study demonstrates how this design aspect is perceived by users during transitions in an HUI composed by three interactive environments.     

Context-driven interaction in immersive virtual environments

There are many interaction tasks a user may wish to accomplish in an immersive virtual environment. A careful examination of these tasks reveals that they are often performed under different contexts. For each task and context, specialized interaction techniques can be developed. We present the context-driven interaction model: a design pattern that represents contextual information as a first-class, quantifiable component within a user interface and supports the development of context-sensitive applications by decoupling context recognition, context representation, and interaction technique development. As a primary contribution, this model provides an enumeration of important representations of contextual information gathered from across the literature and describes how these representations can effect the selection of an appropriate interaction technique. We also identify how several popular 3D interaction techniques adhere to this design pattern and describe how the pattern itself can lead to a more focused development of effective interfaces. We have constructed a formalized programming toolkit and runtime system that serves as a reference implementation of the context-driven model and a discussion is provided explaining how the toolkit can be used to implement a collection of representative 3D interaction interfaces.     

Virtual environments applications and applied ergonomics

The usability of virtual environments has attracted considerable efforts from ergonomists. Work has included studies of the side or after effects of participation in a virtual environment (VE) as well as the appropriateness of the Virtual Reality hardware and software interfaces and the understanding of factors which determine participant performance. Equally important for applied ergonomics is to understand how best to specify, build, implement and evaluate virtual environment solutions to everyday industrial, commercial, educational and medical problems. The potential value of ergonomics applied to virtual environments, and vice versa, are discussed. Two particular instances of VE development relevant to applied ergonomics are described - structured development and evaluation of industrial training and participatory redesign of workplaces. This paper is one of a number of contributions to a special issue on ergonomics in the study and use of virtual environments.     

Who benefits from virtuality?

For educational software to take advantage of contemporary views of learning, instructional designers need to employ design models that incorporate the variety of ideas that are based on constructivist frameworks for developing learning environments. These environments, if well designed, can support learner construction of knowledge, however, such frameworks are based upon arguments that learners should be placed in authentic environments that incorporate sophisticated representations of context through such constructs as “virtual worlds”. Within these environments the learner is supported by visual metaphors constructed to represent the information structure and how the “world” operates. This paper will discuss the framework employed in the development of several virtual solutions and the process by which they were constructed.     

Taking fun seriously: Using cognitive models to reason about interaction with computer games

This paper seeks to extend the application of techniques from affective psychology to show how cognitive models can be used to represent and reason about interaction with computer games. It is argued that this modelling activity provides insights into the motivational appeal that often distinguishes computer games from other forms of human computer interaction. The long-term aim behind this research is to use our improved understanding of interaction with computer games to inform the subsequent development of more general classes of interactive systems. Barnard's Interacting Cognitive Subsystems (ICS) is used throughout this paper. This decision is justified by the fact that ICS has already been applied to analyse the negative emotions surrounding clinical depression. This previous work provides a useful starting point for our investigation of the more positive emotions evoked during interaction with computer games. A further justification is that ICS has also been successfully used to represent and reason about the design of human computer interfaces.     

Linking GIS with real-time visualisation for exploration of landscape changes in rural community workshops

To allow rural communities to evaluate possible future landscape scenarios, we have created a portable environment for landscape simulation (envisioning system). The goal of this system is to give communities the opportunity to plan their desired futures. Our system is designed for workshop environments and allows workshop attendees to explore and to interact with representations of virtual landscapes. We are using virtual reality technology to visualise the landscape representations, a geographic information system to allow participants to change the current landscape configuration, and mobile computing devices to allow the attendees to navigate in the virtual landscape, and give feedback and opinions on the landscape changes. Here, we describe the technology that implements the interaction between geographical information systems and real-time rendering needed to achieve real-time visualisation of landscape changes. To achieve this functionality we have programmed two software clients (a renderer and an ESRI ArcMap extension) and a server that handles message flow. The landscape has been divided into management units that each supports one land use type. Using the GIS interface, users can change the land uses associated to the units and the renderer will update the landscape correspondingly in real time.     

A multiscale progressive model on virtual navigation

Navigation designs in virtual environments often draw on research findings on human navigation behaviors in the real world, in particular the landmark–route–survey spatial knowledge model. Geographers and cognitive psychologists have argued that this model is insufficient to capture the complexity of spatial cognition related to navigation. They have suggested that new theories are needed to understand the integration of various kinds of spatial knowledge and their relationship with spatial activities, such as route planning, route choosing and so on. In virtual environments, users can scale up and down the virtual space to obtain different spatial knowledge and interaction domains. Such flexibility offers an opportunity to deepen our understanding of the relationship between spatial knowledge and spatial action. This paper reports a study on how scaling in virtual environments can improve the integration of spatial knowledge and spatial action. This paper first proposes a multiscale progressive model that couples spatial knowledge and movement across scale in navigation in virtual environments. Then, the paper introduces the design of multiscale environments to support the coupling. Results of an experimental study show the benefits of the coupled spatial knowledge and movement for navigation involving subtasks at different scale levels. In addition to helping better understand the relationship between spatial knowledge and spatial action, this research also gives some insight into designs to support navigation in virtual environments as well as designs to support cross-scale spatial knowledge access in the real world.