Realistic haptic rendering of interacting deformable objects in virtual environments

A new computer haptics algorithm to be used in general interactive manipulations of deformable virtual objects is presented. In multimodal interactive simulations, haptic feedback computation often comes from contact forces. Subsequently, the fidelity of haptic rendering depends significantly on contact space modeling. Contact and friction laws between deformable models are often simplified in up to date methods. They do not allow a "realistic" rendering of the subtleties of contact space physical phenomena (such as slip and stick effects due to friction or mechanical coupling between contacts). In this paper, we use Signorini's contact law and Coulomb's friction law as a computer haptics basis. Real-time performance is made possible thanks to a linearization of the behavior in the contact space, formulated as the so-called Delassus operator, and iteratively solved by a Gauss-Seidel type algorithm. Dynamic deformation uses corotational global formulation to obtain the Delassus operator in which the mass and stiffness ratio are dissociated from the simulation time step. This last point is crucial to keep stable haptic feedback. This global approach has been packaged, implemented, and tested. Stable and realistic 6D haptic feedback is demonstrated through a clipping task experiment.     

A dual-representation strategy for the virtual assembly of thin deformable objects

The two main objectives of virtual assembly are: (1) to train assembly-operators through virtual assembly models, and (2) to simultaneously evaluate products for ease-of-assembly. The focus of this paper is on developing computational techniques for virtual assembly of thin deformable beam and plate-like objects. To meet the objectives of virtual assembly, the underlying computational technique must: (1) be carried out at a high frame-rate (>20 frames/second), (2) be accurate (<5% error in deformation and force estimation), (3) be conducive to collision detection, and (4) support rapid design evaluations. We argue in this paper that popular computational techniques such as 3-D finite element analysis, boundary element analysis and classic beam/plate/shell analysis fail to meet these requirements. We therefore propose a new class of dual representation techniques for virtual assembly of thin solids, where the geometry is retained in its full 3-D form, while the underlying physics is dimensionally reduced, delivering: (1) high computational efficiency and accuracy (over 20 frames per second with <1% deformation error), and (2) direct CAD model processing, i.e., the CAD model is not geometrically simplified, and 3-D finite element mesh is not generated. In particular, a small-size stiffness matrix with about 300 degrees of freedom per deformable object is generated directly from a coarse surface triangulation, and its LU-decomposition is then exploited during real-time simulation. The accuracy and efficiency of the proposed method is established through numerical experiments and a case study.     

A computational model of perceptual saliency for 3D objects in virtual environments

When giving directions to the location of an object, people typically use other attractive objects as reference, that is, reference objects. With the aim to select proper reference objects, useful for locating a target object within a virtual environment (VE), a computational model to identify perceptual saliency is presented. Based on the object’s features with the major stimulus for the human visual system, three basic features of a 3D object (i.e., color, size, and shape) are individually evaluated and then combined to get a degree of saliency for each 3D object in a virtual scenario. An experiment was conducted to evaluate the extent to which the proposed measure of saliency matches with the people’s subjective perception of saliency; the results showed a good performance of this computational model.     

A hierarchical parametric algorithm for deformable multimodal image registration

Image fusion is of utmost importance for many applications in image analysis. Particularly in medical imaging, images of different modalities are necessary because they provide complementary information that must be merged for an optimal use. The fusion of these images, which can be achieved through a registration process, makes it possible to superimpose all available information on the same frame. In many cases, a rigid transformation is sufficient to align correctly the images. However, there are cases where a non-rigid transformation is needed: geometrical distortions present in one image, non-rigid motion, etc. The purpose of this paper is to propose a generic method to account for these deformations in case of multimodal images. We have applied the algorithm in the particular context of 3D medical images and present results on simulated and real data.     

A virtual cockpit for a distributed interactive simulation

We developed the virtual cockpit as an inexpensive flight simulator, using off-the-shelf equipment. This system functions as a component of a distributed interactive simulation. Any flight simulator has three principal tasks: image display, image generation, and flight dynamics. We built the flight simulator, the virtual cockpit, using a head-mounted display (HMD) to display the out-the-window imagery and the cockpit interior. The virtual cockpit consists of a Silicon Graphics workstation and supporting hardware components, such as an HMD and position tracker. The software falls into three areas by function: flight dynamics and cockpit instruments; network interface (either SimNet or DIS PDUs); and display of the out-the-window view. The virtual cockpit developed uses a multiprocessor Silicon Graphics Iris (the 4D/440VGXT) connected to a Polhemus Laboratories fiber-optic head-mounted display, a Polhemus magnetic head tracker and a hands-on-throttle-and-stick (HOTAS) by Thrustmaster. The software uses the AT and T C++ translator and the Silicon Graphics C compiler running under Irix 4.0.5. a Unix operating system. We use Software Systems' MultiGen to create the geometric models and the terrain database     

Building interactive virtual environments for simulated training in medicine using VRML and Java/JavaScript

Medicine is a difficult thing to learn. Experimenting with real patients should not be the only option; simulation deserves a special attention here. Virtual Reality Modelling Language (VRML) as a tool for building virtual objects and scenes has a good record of educational applications in medicine, especially for static and animated visualisations of body parts and organs. However, to create computer simulations resembling situations in real environments the required level of interactivity and dynamics is difficult to achieve. In the present paper we describe some approaches and techniques which we used to push the limits of the current VRML technology further toward dynamic 3D representation of virtual environments (VEs). Our demonstration is based on the implementation of a virtual baby model, whose vital signs can be controlled from an external Java application. The main contributions of this work are: (a) outline and evaluation of the three-level VRML/Java implementation of the dynamic virtual environment, (b) proposal for a modified VRML Timesensor node, which greatly improves the overall control of system performance, and (c) architecture of the prototype distributed virtual environment for training in neonatal resuscitation comprising the interactive virtual newborn, active bedside monitor for vital signs and full 3D representation of the surgery room.     

A multimodal virtual reality interface for 3D interaction with VTK

The object-oriented visualization Toolkit (VTK) is widely used for scientific visualization. VTK is a visualization library that provides a large number of functions for presenting three-dimensional data. Interaction with the visualized data is controlled with two-dimensional input devices, such as mouse and keyboard. Support for real three-dimensional and multimodal input is non-existent. This paper describes VR-VTK: a multimodal interface to VTK on a virtual environment. Six degree of freedom input devices are used for spatial 3D interaction. They control the 3D widgets that are used to interact with the visualized data. Head tracking is used for camera control. Pedals are used for clutching. Speech input is used for application commands and system control. To address several problems specific for spatial 3D interaction, a number of additional features, such as more complex interaction methods and enhanced depth perception, are discussed. Furthermore, the need for multimodal input to support interaction with the visualization is shown. Two existing VTK applications are ported using VR-VTK to run in a desktop virtual reality system. Informal user experiences are presented. Arjan J. F. Kok is an assistant professor at the Department of Computer Science at the Open University of the Netherlands. He studied Computer Science at the Delft University of Technology, The Netherlands. He received his Ph.D. from the same university. He worked as a Scientist for TNO (Netherlands Organization for Applied Scientific Research) and as assistant professor at the Eindhoven University of Technology before he joined the Open University. His research interests are visualization, virtual reality, and computer graphics. Robert van Liere studied Computer Science at the Delft University of Technology, the Netherlands. He received his Ph.D. with the thesis “Studies in Interactive Scientific Visualization” at the University of Amsterdam. Since 1985, he has worked at CWI, the Center for Mathematics and Computer Science in Amsterdam in which he is the head of CWI’s visualization research group. Since 2004, he holds a part-time position as full professor at the Eindhoven University of Technology. His research interests are in interactive data visualization and virtual reality. He is a member of IEEE.     

Badumna: A decentralised network engine for virtual environments

Many of today’s virtual environment applications, such as massively multiplayer online games, involve a large and rapidly changing set of users. The server-based architecture of these applications makes its hard for them to support the dynamic user base and provide a scalable solution. Badumna, a scalable network engine for large-scale virtual environments, was developed to address these issues. The approach is based on a decentralised architecture using a structured peer-to-peer network. In order to ensure a highly scalable and efficient design, Badumna uses techniques that include neighbour introductions and the formation of dynamic bounded regions to reduce the load on the peer-to-peer network. The performance of Badumna is evaluated using extensive simulations. Results indicate that Badumna is able to scale to a large number of users without placing any additional overheads on the network that degrade performance.     

DIVE—A platform for multi-user virtual environments

The Distributed Interactive Virtual Environment (DIVE) is an experimental software platform for the development of multi-user virtual reality applications. DIVE uses active replication and reliable multicast protocols to distribute data between participants. The distribution model enables a large number of users and applications to participate and interact in the same virtual environment. The DIVE platform offers a wide range of interaction support and user-related abstractions to ease the task of application and user interface construction.     

Traffic modeling flows in a developing urban infrastructure with a software suite for creating interactive virtual environments

We present a design paradigm for a cloud-based service for interactive modeling of traffic flows in a developing urban infrastructure. The mathematical forecasting model for the load of a transportation network is constructed as a result of synthesizing a gravitational model that describes origin-destination trips and a multimodal traffic equilibrium problem with elastic demand. The search for traffic equilibrium reduces to solving a variational inequality. Three-dimensional visualization of the transportation network and adjacent infrastructure is implemented with the virtual environment’s declarative model.     

Peltier Haptic Interface (PHI) for improved sensation of touch in virtual environments

In this report we describe an advanced virtual reality glove that we are developing, called the Peltier Haptic Interface (PHI), which will provide improved sensation of touch in virtual environments. PHI will provide force/pressure feedback that can be varied independently on each finger, as well as temperature sensation that can be varied non-uniformly over the whole hand. The combination of these sensations will provide a more realistic sense of touch and significantly increase the realism of virtual environments. PHI will find extensive applications in biomedical simulations, teaching, industrial line training, and many other areas.     

A distributed platform for personalized advertising in digital interactive TV environments

Advertising plays an important role in modern free markets. Furthermore, advertising is moving towards the establishment of one-to-one marketing relationships. Thus, personalized advertisement is currently considered as a hot topic in product promotion as it can be proved beneficial for all the key players, such as the advertisers, the advertised companies, as well as the consumers. Interactive TV and WWW can provide the means for personalized advertising. But of course, special systems and platforms for personalization must be first developed. This paper proposes a prototype system which efficiently achieves the personalization of the advertisements in the environment of digital interactive TV. Thus, the environment for the exploitation of the proposed system are examined, the details in design and implementation are given, while extensive operation testing and evaluation are provided proving its high applicability in real business environments.     

A hyper elasticity method for interactive virtual design of hearing aids

We present a computational efficient method for isotropic hyper elasticity based on functional analysis. By selecting a class of shape functions, we arrive at a computational scheme which yields very sparse tensors. This enables fast computations of the hyper elastic energy potential and its derivatives. We achieve efficiency and performance through the use of shape functions that are linear in their parameters and through rotation into the eigenspace of the right Cauchy–Green strain tensor. This makes near real time evaluation of hyper elasticity of complex meshes on CPU relatively easy to implement. The approach does not rely on a specific shape function or material model but offers a general framework for isotropic hyper elasticity. The method is aimed at interactive and accurate non-linear hyper elastic modeling for a wide range of industrial virtual design applications, which we exemplify by insertion of hearing aid domes into the ear canal. We validate the method for tetrahedral meshes with linear shape functions with an Ogden material model by comparing simulations to deformations of real material. We illustrate the use of other shape functions and models using uniform cubic B-splines in combination with Riemannian elasticity.     

A multimodal approach for 3D face modeling and recognition using 3D deformable facial mask

We present a multimodal approach for face modeling and recognition. The algorithm uses three cameras to capture stereo images, two frontal and one profile, of the face. 2D facial features are extracted from one of the frontal images and a dense disparity map is computed from the two frontal images. Using the extracted 2D features and their corresponding disparities, we compute their 3D coordinates. We next align a low resolution 3D mesh model to the 3D features, re-project its vertices onto the frontal 2D image and adjust its profile silhouette vertices using the profile view image. We increase the resolution of the resulting 2D model at its center region to obtain a facial mask model covering distinctive features of the face. The 2D coordinates of the vertices, along with their disparities, result in a deformed 3D mask model specific to a given subject’s face. Our method integrates information from the extracted facial features from the 2D image modality with information from the 3D modality obtained from the stereo images. Application of the models in 3D face recognition, for 112 subjects, validates the algorithm with a 95% identification rate and 92% verification rate at 0.1% false acceptance rate.

The roles of sensory modalities in collaborative virtual environments (CVEs)

This study was conducted to assess the effects of sensorial modalities on user performance, perception, and behavior in collaborative virtual environments (CVEs). Participants played a CVE game, air hockey, together with a remote partner under different sensory modality conditions, depending on the type of sensory feedback provided: visual-only (V), visual–haptic (V + H), and visual–haptic–audio feedback (V + H + A). Three types of measurements were used as dependent variables: (1) task performance measured as playing time, (2) user perception including the sense of presence, the sense of togetherness, and perceived collaboration, and (3) behavior measurement including the amount of force applied and the mallet deviation. Results of the study indicated that the task performance, perception, and user behavior in CVEs can be affected due to supported sensory modalities. Therefore, the multiple sensory information types that are required to perform the task at hand should be provided to effectively support collaboration between people in CVEs. The outcomes of this research should have a broad impact on multimodal user interaction, including research on physiological, psychophysical, and psychological mechanisms underlying human perception on multisensory feedback in CVEs.     

A brush device with visual and haptic feedback for virtual painting of 3D virtual objects

We have previously developed a mixed reality (MR) painting system with which a user could take a physical object in the real world and apply virtual paint to it. However, this system could not provide the sensation of painting on virtual objects in MR space. Therefore, we subsequently proposed and developed mechanisms that simulated the effect of touch and movement when a brush device was used to paint on a virtual canvas. In this paper, we use visual and haptic feedback to provide the sensation of painting on virtual three-dimensional objects using a new brush device called the MAI Painting Brush++. We evaluate and confirm its effectiveness through several user studies.     

A model of (en)action to approach embodiment: a cornerstone for the design of virtual environments for learning

This paper presents a model of (en)action from a conceptual and theoretical point of view. This model is used to provide solid bases to overcome the complexity of designing virtual environments for learning (VEL). It provides a common grounding for trans-disciplinary collaborations where embodiment can be perceived as the cornerstone of the project. Where virtual environments are concerned, both computer scientists and educationalists have to deal with the learner/user’s body; therefore the model provides tools with which to approach both human actions and learning processes within a threefold model. It is mainly based on neuroscientific research, including enaction and the neurophysiology of action.     

A multi-resolution statistical deformable model (MISTO) for soft-tissue organ reconstruction

While active shape model (ASM) has been increasingly adopted in the medical domain, there are issues that need to be addressed for it to be applicable in practice. Among them, the small sample size problem and how to represent the variation of the clutter of surroundings are two of the challenges. In this paper, to overcome these problems, we propose a novel multi-resolution statistical deformable model and the associated techniques for the reconstruction of soft-tissue organs such as livers. To address the small sample size problem, we define a multi-resolution integrated model for soft-tissue organs called MISTO that is able to capture the most significant deformations from a small training set as well as to generate representative variation modes of the organ shapes. To deal with the complex surroundings of the model surface or landmark points in the underlying medical images during model deformation, we propose to apply multi-resolution appearance models which allows the surrounding visual context of the model surface points to be learnt and characterized automatically from the training samples. By combining the powerful shape models and the resulting context constraints, the object segmentation and reconstruction process can be carried out very robustly. Furthermore, to avoid the local minima during model optimization, we develop an adaptive deformation strategy such that the more stable parts of the surface are moved prior to the rest of the model surface. The experimental and validation results verify that our proposed approaches can be successfully and robustly applied to the reconstruction of the soft-tissue organs such as the human liver. The major contributions of our approaches are that we extend the traditional ASM to address open problems associated with reconstructing significantly deformable three-dimensional anatomies in cluttered surrounding, and we propose effective ways to formulate the perceptual knowledge of the anatomies and make use of it in the process of model construction and deformation for medical reconstruction.     

A domain-oriented end-user design environment for generating interactive 3D virtual chemistry experiments

Three-dimensional virtual worlds are potentially feasible for building virtual educational environments. However, educators face technical challenges to apply these technologies because creating virtual educational environments based on virtual worlds demands 3D modeling and programming skills. This paper proposes a method to lower the technical barriers through domain-oriented interfaces with languages and environments that are familiar to educators. A domain-oriented end-user design environment, iVirtualWorld, is designed and developed to implement the proposed method in a specific educational domain, namely introductory chemistry experiments. This web-based environment provides end-users with domain-oriented building blocks, which can be assembled to create 3D virtual chemistry experiments. A usability evaluation and a comparative case study are designed to evaluate the system among chemistry educators, who are the target audience. The usability evaluation contains a task requiring participants to create a 3D virtual chemistry experiment and a voluntary semi-structured interview. The case study compares a virtual experiment generated using iVirtualWorld with an experiment in a commercial virtual chemistry laboratory system. The results show that 1) the domain-oriented end-user design environment enables participants to generate the 3D virtual chemistry experiment within 30 min; 2) participants gain confidence on creating 3D virtual experiments by themselves using iVirtualWorld; 3) participants confirm the usefulness of applying the system in introductory chemistry education; and 4) iVirtualWorld is considered more intuitive and straightforward for students to focus on finishing the experiment without being distracted than the commercial virtual chemistry laboratory system. Areas that can benefit from the system most and areas where the system is less effective are identified by participants. The responses also reveal the limitations of the current system and suggest directions for future improvement.