Phantom-based multimodal interactions for medical education and training: the Munich knee joint simulator

Simulation environments based on virtual reality technologies can support medical education and training. In this paper, the novel approach of an "interactive phantom" is presented that allows a realistic display of haptic contact information typically generated when touching and moving human organs or segments. The key idea of the haptic interface is to attach passive phantom objects to a mechanical actuator. The phantoms look and feel as real anatomical objects. Additional visualization of internal anatomical and physiological information and sound generated during the interaction with the phantom yield a multimodal approach that can increase performance, didactic value, and immersion into the virtual environment. Compared to classical approaches, this multimodal display is convenient to use, provides realistic tactile properties, and can be partly adjusted to different, e.g., pathological properties. The interactive phantom is exemplified by a virtual human knee joint that can support orthopedic education, especially for the training of clinical knee joint evaluation. It is suggested that the technical principle can be transferred to many other fields of medical education and training such as obstetrics and dentistry.     

Haptics in virtual reality and multimedia

Imagine that you're chatting with a good friend who tells you about her new cashmere pullover, and that you're able to touch it and appreciate the soft sensation of the cashmere wool - remotely over the Web in your virtual chatroom! In this article, the authors present their work on haptic sensing of virtual textiles. This is work done in the context of the Haptex research project that aims at integrating the human sense of touch into multimodal user interfaces. Being able to support haptic perception in the user interface will be a great step toward next-generation immersive multimedia experiences.     

Adaptive and reliable video transmission over UMTS for enhanced performance

This paper proposes a mechanism for the congestion control for video transmission over universal mobile telecommunications system (UMTS). Our scheme is applied when the mobile user experiences real‐time multimedia content and adopts the theory of a widely accepted rate control method in wired networks, namely equation‐based rate control. In this approach, the transmission rate of the multimedia data is determined as a function of the packet loss rate, the round trip time and the packet size and the server explicitly adjusts its sending rate as a function of these parameters. Furthermore, we examine the performance of the UMTS for real‐time video transmission using real‐time protocols. Through a number of experiments, we measure performance parameters such as end‐to‐end delay, delay in radio access network, delay jitter and throughput in the wireless link. Copyright © 2006 John Wiley & Sons, Ltd.     

Near-to-Eye Display—An Accessory for Handheld Multimedia Devices: Subjective Studies

Fifty-eight people used a mobile phone/near-to-eye display combination with three different applications: movie viewing, game playing, and reading. After a 40-min immersion period participants completed several questionnaires related to the usage experience, comfort, workload, and individual features. We found many differences in user experiences between tasks as well as a number of associations between individual factors and experiences reported. Participants' opinions on the near-to-eye display/mobile phone combination were generally positive, and several benefits were reported: namely, a more engrossing experience and longer use time in the future. We also compared experiences from mobile phone/near-to-eye display setups to mobile phone and TV usage.      

Construction of a haptic-enabled broadcasting system based on the MPEG-V standard

With rapid developments in communications technology and digital multimedia, there has been increasing demand in recent years for realistic broadcasting technology beyond conventional audio-visual media. In response to this demand, this paper presents an example of the construction of a haptic-enabled broadcasting system based on the MPEG-V standard that was established recently. The construction processes of the proposed haptic-enabled broadcasting system include various types of media acquisition, haptic content creation by modeling and authoring, transmission, rendering, and interaction. This paper illustrates the data flow within the system, from the creation of haptic contents to the rendering of these contents to the end user, and explains a method of building the system with the MPEG-V standard. The constructed haptic-enabled broadcasting system allows users to have more immersive interaction with the synthesized haptic multimedia, which is closely synchronized with audio-visual data.     

Organic Haptics: Intersection of Materials Chemistry and Tactile Perception

The goal of the field of haptics is to create technologies that manipulate the sense of touch. In virtual and augmented reality, haptic devices are for touch what loudspeakers and RGB displays are for hearing and vision. Haptic systems that utilize micromotors or other miniaturized mechanical devices (e.g., for vibration and pneumatic actuation) produce interesting effects, but are quite far from reproducing the feeling of real materials. They are especially deficient in recapitulating surface properties: fine texture, friction, viscoelasticity, tack, and softness. The central argument of this progress report is that in order to reproduce the feel of everyday objects, molecular control must be established over the properties of materials; ultimately, such control will enable the design of materials which can change these properties in real time. Stimuli‐responsive organic materials, such as polymers and composites, are a class of materials which can change their oxidation state, conductivity, shape, and rheological properties, and thus might be useful in future haptic technologies. Moreover, the use of such materials in research on tactile perception could help elucidate the limits of human tactile sensitivity. The work described represents the beginnings of this new area of inquiry, in which the defining approach is the marriage of materials science and psychology.     

Augmented System for Immersive 3D Expansion and Interaction

In the field of augmented reality technologies, commercial optical see‐through‐type wearable displays have difficulty providing immersive visual experiences, because users perceive different depths between virtual views on display surfaces and see‐through views to the real world. Many cases of augmented reality applications have adopted eyeglasses‐type displays (EGDs) for visualizing simple 2D information, or video see‐through‐type displays for minimizing virtual‐ and real‐scene mismatch errors. In this paper, we introduce an innovative optical see‐through‐type wearable display hardware, called an EGD. In contrast to common head‐mounted displays, which are intended for a wide field of view, our EGD provides more comfortable visual feedback at close range. Users of an EGD device can accurately manipulate close‐range virtual objects and expand their view to distant real environments. To verify the feasibility of the EGD technology, subject‐based experiments and analysis are performed. The analysis results and EGD‐related application examples show that EGD is useful for visually expanding immersive 3D augmented environments consisting of multiple displays.     

Pseudo-haptics for rigid tool/soft surface interaction feedback in virtual environments

This paper proposes a novel pseudo-haptic soft surface stiffness simulation technique achieved by displaying the deformation of the soft surface and maneuvering an indenter avatar over a virtual soft surface by means of a touch-sensitive tablet. The visual feedback of the surface deformation and the alterations to the indenter avatar behavior produced by the proposed technique create the illusion of interaction with a hard inclusion embedded in the virtual soft surface. The proposed pseudo-haptics technique is validated with a series of experiments conducted by employing a tablet computer with an S-pen input and a tablet computer with a bare finger input. Tablet computers provide unique opportunities for presenting the pseudo-haptic (indenter avatar speed), haptic (contact reaction force from the device surface) and visual cues (surface information) at the same active point of interaction which facilitates information fusion. Hence, here, we evaluate the performance of tablet computers in identification of hard inclusions within virtual soft objects and compare it with the performance of a touchpad input device. A direct hand-soft surface interaction is used for benchmarking of this study. We found that compared with using a touchpad, both the sensitivity and the positive predictive value of the hard inclusion detection can be significantly improved by 33.3% and 13.9%, respectively, by employing tablet computers. Using tablet computers could produce results comparable to the direct hand-soft surface interaction in detecting hard inclusions in a soft object. The experimental results presented here confirm the potential of the proposed technique for conveying haptic information in rigid tool/soft surface interaction in virtual environments.     

A haptic-enabled multimodal interface for the planning of hip arthroplasty

Multimodal environments help fuse a diverse range of sensory modalities, which is particularly important when integrating the complex data involved in surgical preoperative planning. The authors apply a multimodal interface for preoperative planning of hip arthroplasty with a user interface that integrates immersive stereo displays and haptic modalities. This article overviews this multimodal application framework and discusses the benefits of incorporating the haptic modality in this area.     

Reality-based models for vibration feedback in virtual environments

Reality-based modeling of vibrations has been used to enhance the haptic display of virtual environments for impact events such as tapping, although the bandwidths of many haptic displays make it difficult to accurately replicate the measured vibrations. We propose modifying reality-based vibration parameters through a series of perceptual experiments with a haptic display. We created a vibration feedback model, a decaying sinusoidal waveform, by measuring the acceleration of the stylus of a three degree-of-freedom haptic display as a human user tapped it on several real materials. A series of perceptual experiments, where human users rated the realism of various parameter combinations, were performed to further enhance the realism of the vibration display for impact events. The results provided different parameters than those derived strictly from acceleration data. Additional experiments verified the effectiveness of these modified model parameters by showing that users could differentiate between materials in a virtual environment     

Dual Autostereoscopic Display Platform for Multi‐user Collaboration with Natural Interaction

In this letter, we propose a dual autostereoscopic display platform employing a natural interaction method, which will be useful for sharing visual data with users. To provide 3D visualization of a model to users who collaborate with each other, a beamsplitter is used with a pair of autostereoscopic displays, providing a visual illusion of a floating 3D image. To interact with the virtual object, we track the user's hands with a depth camera. The gesture recognition technique we use operates without any initialization process, such as specific poses or gestures, and supports several commands to control virtual objects by gesture recognition. Experiment results show that our system performs well in visualizing 3D models in real‐time and handling them under unconstrained conditions, such as complicated backgrounds or a user wearing short sleeves.     

智能终端发展关键技术探讨

智能终端产品(包括手机、平板和混合式的移动计算设备)在个人和商业应用场合大量使用,终端在高速计算能力方面更新迅猛,处理能力军备竞赛将陷入硬件瓶颈,同质现象成为现实,而人机交互技术、终端应用开发环境、终端安全云服务等将成为影响智能终端向纵深人群渗透和更商业化应用领域发展的决定要素。本文旨在探讨、分析和展示上述几个关键技术的新进展和演进趋势。     

Haptic Rendering of Visual Data for the Visually Impaired

We introduce a force-field haptic rendering method for converting visual data to haptic data. The method includes a novel framework to convert specialized 3D map models into force fields. We generate the final force fields by using structure from motion and implicit surface approximation algorithms. Visually impaired people then can learn to navigate with these maps, using off-the-shelf haptic     

肝脏三维重建中模型平滑和网格细化方法

针对力触觉再现系统对三维模型真实性和准确性的要求,在基于二维医学图像重构肝脏三维模型的过程中,提出了一种基于B样条拟合曲线的模型平滑方法和基于最邻近点搜索插值的模型网格细化方法。前者是以最初获得的轮廓点为控制顶点,拟合出平滑的B样条曲线并重新采样,以此实现对模型的平滑操作。后者是在每一次连线构造三角形面片过程中,取两点之间的几何中心作为插值点,以此实现三维网格模型的细化。实验表明,这两种算法能够有效提高肝脏三维模型的重建效果,增强力触觉再现系统的真实性。     

Haptic holography: a primitive computational plastic

We describe our work on haptic holography, a combination of computational modeling and multimodal spatial display, which allows a person to see, feel, and interact with three-dimensional freestanding holographic images of material surfaces. In this paper, we combine various holographic displays with a force-feedback device to render multimodal images with programmatically prescribed material properties and behavior. After a brief overview of related work which situates visual display within the manual workspace, we describe our holo-haptic approach and survey three implementations, Touch, Lathe, and Poke, each named for the primitive functional affordance it offers. In Touch, static holographic images of simple geometric scenes are reconstructed in front of the hologram plane, and coregistered with a force model of the same geometry. These images can be visually inspected and haptically explored using a handheld interface. In Lathe, a holo-haptic image can be reshaped by haptic interaction in a dynamic but constrained manner. Finally in Poke, using a new technique for updating interference-modeled holographic fringe patterns, we render a holo-haptic image that permits more flexible interactive reshaping of its reconstructed surface. We situate this work within the context of related research and describe the strengths, shortcomings, and implications of our approach.     

A high bandwidth interface for haptic human computer interaction

Current force feedback, haptic interface devices are generally limited to the display of low frequency, high amplitude spatial data. A typical device consists of a low impedance framework of one or more degrees-of-freedom (dof), allowing a user to explore a pre-defined workspace via an end effector such as a handle, thimble, probe or stylus. The movement of the device is then constrained using high gain positional feedback, thus reducing the apparent dof of the device and conveying the illusion of hard contact to the user. Such devices are, however, limited to a narrow bandwidth of frequencies, typically below 30 Hz, and are not well suited to the display of surface properties, such as object texture. This paper details a device to augment an existing force feedback haptic display with a vibrotactile display, thus providing a means of conveying low amplitude, high frequency spatial information of object surface properties.     

Interactive Dynamic Simulation Schemes for Articulated Bodies through Haptic Interface

This paper describes interactive dynamic simulation schemes for articulated bodies in virtual environments, where user interaction is allowed through a haptic interface. We incorporated these schemes into our dynamic simulator I‐GMS, which was developed in an object‐oriented framework for simulating motions of free bodies and complex linkages, such as those needed for robotic systems or human body simulation. User interaction is achieved by performing push and pull operations with the PHANToM haptic device, which runs as an integrated part of I‐GMS. We use both forward and inverse dynamics of articulated bodies for the haptic interaction by the push and pull operations, respectively. We demonstrate the userinteraction capability of I‐GMS through on‐line editing of trajectories for 6‐dof (degrees of freedom) articulated bodies.     

Acceptance‐Diffusion Strategies for Tablet‐PCs: Focused on Acceptance Factors of Non‐Users and Satisfaction Factors of Users

Among emerging devices propelling the growth of mobile devices, smartphones and tablet‐PCs are among the most recognizable. In this study, a research model is designed for exploring acceptance‐diffusion strategies for tablet‐PCs from the viewpoint of consumer perception, which is verified through a survey. The results of this study show that tablet‐PCs have great potential to be versatile, multifunctional devices, even though they are currently considered mostly as entertainment‐oriented rather than fulfilling the essential needs of everyday life. An analysis of the acceptance model for tablet‐PCs revealed that playability, cost level, functionality, and complexity significantly affect user acceptance. An analysis of the diffusion model, on the other hand, showed that playability and user interface have a significant influence on satisfaction, trust, and positive behavioral intention. We also found that cost level is not a major hindrance in the market diffusion of tablet‐PCs. The results of this study can be used to establish effective acceptance and diffusion strategies for tablet‐PCs and other emerging devices.     

QoS provisioning for multiple non‐real‐time services in cellular wireless networks

Future mobile services are expected to include various non‐voice oriented services. One important category of non‐voice oriented mobile services is non‐real‐time services. When a mobile user establishes a connection to access non‐real‐time service, the mobile user usually cares about whether the total time to complete its data transfer is within its time tolerance. In addition, different mobile users may have different bandwidth requirements and different tolerances in the total completion time. It is essential for wireless systems to provide various mobile users with different total completion times. In this paper, two quality‐of‐service (QoS) metrics, called stretch ratio and eligibility percentage, are employed at a connection level to present the degree of the length of the total completion time. We devise a measurement based call admission control scheme that provides multiple QoSs for various mobile users which have different requirements of stretch ratios, eligibility percentages, and bandwidths. Extensive simulation results show that the measurement based call admission control scheme not only provides various satisfactory QoSs for mobile users but also produces high throughput. Copyright © 2010 John Wiley & Sons, Ltd.     

Touch and Go — Designing Haptic Feedback for a Hand-Held Mobile Device

Increasingly, our mobile devices are acquiring the ability to be aware of their surroundings. These devices are also acquiring the ability to sense what is happening to them — how they are being held and moved. The coincidence of connectedness, awareness and richly multimodal input and output capabilities brings into the hand a device capable of supporting an entirely new class of haptic or touch-based interactions, where gestures can be captured and reactions to these gestures conveyed as haptic feedback directly into the hand. Thus, one can literally shake the hand of a friend, toss a file off one's PDA, or be led by the hand to a desired location in a strange city. In this paper I will propose that, for the mobile user negotiating these multiple frames of reference for their actions, a better understanding of the senses of touch, of the body's motion and its sense of its own motion, may be the key to providing a meaningful bridge between these interleaved and interdependent spaces.