Touch-enabled haptic modeling of deformable multi-resolution surfaces

Currently, interactive data exploration in virtual environments is mainly focused on vision-based and non-contact sensory channels such as visual/auditory displays. The lack of tactile sensation in virtual environments removes an important source of information to be delivered to the users. In this paper, we propose the touch-enabled haptic modeling of deformable multi-resolution surfaces in real time. The 6-DOF haptic manipulation is based on a dynamic model of Loop surfaces, where the dynamic parameters are computed easily without subdividing the control mesh recursively. A local deforming scheme is developed to approximate the solution of the dynamics equations, thus the order of the linear equations is reduced greatly. During each of the haptic interaction loop, the contact point is traced and reflected to the rendering of updated graphics and haptics. The sense of touch against the deforming surface is calculated according to the surface properties and the damping-spring force profile. Our haptic system supports the dynamic modeling of deformable Loop surfaces intuitively through the touch-enabled interactive manipulation.     

Contact and Deformation Modeling for Interactive Environments

Contact and deformation modeling for interactive environments has seen many applications, from surgical simulation and training, to virtual prototyping, to teleoperation, etc., where both visual feedback and haptic feedback are needed. High-quality feedback demands a high level of physical realism as well as a high update rate in rendering, which are often conflicting requirements. In this paper, we present a unique approach to modeling force and deformation between a rigid body and an elastic object under complex contacts, which achieves a good compromise of reasonable physical realism and real-time update rate (at least 1 kHz). We simulate contact forces based on a nonlinear physical model. We further introduce a novel approximation of material deformation suitable for interactive environments based on applying Bernoulli-Euler bending beam theory to the simulation of elastic shape deformation. Our approach is able to simulate the contact forces exerted upon the rigid body (that can be virtually held by a user via a haptic device) not only when it forms one or more than one contact with the elastic object, but also when it moves compliantly on the surface of the elastic object, taking friction into account. Our approach is also able to simulate the global and local shape deformation of the elastic object due to contact. All the simulations can be performed in a combined update rate of over 1 kHz, which we demonstrate in several examples.     

面向磁悬浮视触觉交互的多速率系统框架

非接触式磁悬浮视触觉交互克服了机械式交互的固有摩擦,具有广阔应用前景,但存在交互过程中虚拟工具穿透物体、图形渲染与触觉渲染速率不一致等问题。针对上述问题,提出面向磁悬浮视触觉交互的多速率系统框架,通过扩展三自由度（3-DOF）单射线触觉渲染方法,利用多射线对虚拟工具进行建模,避免工具穿透,实现六自由度（6-DOF）触觉渲染;通过多速率并行,实现速率不一致模块间相互协同;通过构建映射滤波算法,实现视觉定位数据到虚拟工具位姿的稳定映射。实验结果表明,该系统能有效避免交互过程中的穿透现象,并提供稳定、真实的视触觉反馈。     

虚拟体空间中的触觉雕刻

目前，在虚拟环境中大多数的信息获取是通过视觉、听觉等非接触感觉获得的。然而缺乏触觉反馈的信息减少了很大一部分的信息源。在看和听之外，能够触摸、感觉和操纵物体，在很大程度上提高了虚拟环境的真实性。该文研究了触觉绘制的基本模型，提出了采用虚平面作为中介实现体数据的实时触觉绘制。并在此基础上探讨了体的局部变形及结合触觉反馈模型，实现了具有触觉反馈的虚拟雕刻交互系统。该系统可应用于融化、燃烧、印记、构造和着色实时交互操作。     

Enhancing Realism of Wet Surfaces in Temporal Bone Surgical Simulation

We present techniques to improve visual realism in an interactive surgical simulation application: a mastoidectomy simulator that offers a training environment for medical residents as a complement to using a cadaver. As well as displaying the mastoid bone through volume rendering, the simulation allows users to experience haptic feedback and appropriate sound cues while controlling a virtual bone drill and suction/irrigation device. The techniques employed to improve realism consist of a fluid simulator and a shading model. The former allows for deformable boundaries based on volumetric bone data, while the latter gives a wet look to the rendered bone to emulate more closely the appearance of the bone in a surgical environment. The fluid rendering includes bleeding effects, meniscus rendering, and refraction. We incorporate a planar computational fluid dynamics simulation into our three-dimensional rendering to effect realistic blood diffusion. Maintaining real-time performance while drilling away bone in the simulation is critical for engagement with the system.     

引入力触觉的数字文物多模交互方法

目的 针对当前文物资源由传统的实体文物向虚拟展示和数字文物进行扩展的趋势,如何提供一种多模态的信息呈现方式就显得尤为重要。通过将力触觉技术引入3维文物展示领域,提出一种基于多模感知的3维文物交互式呈现的算法框架。在对文物的基本特征进行视、听、触觉多通道分析的基础上,依据用户与文物模型的接触状态对多通道信息进行计算和整合。方法 在力触觉计算渲染方面,基于嵌入深度构建弹簧系统模拟轮廓形状的接触过程,引入动摩擦和静摩擦因数来反映表面摩擦力这一材质特征,通过法线贴图来实现文物表面纹理的触觉处理;针对交互的环境由2维平面拓展至立体空间,结合力触觉设备将操作时的行为和状态映射为虚拟环境中的操作代理,借助操作代理构建"旋转"和"选择-移动-释放"两种基本的操作范式来实现用户意图;最后,物理引擎的引入将物体的基本运动规律集成至虚拟场景,提升场景交互的真实感.结果 使用Phantom Omni手控器搭建面向馆藏文物的多模感知实验系统,抽取志愿者对实验系统进行测评。实验结果表明：运用本文方法,用户可从视觉、听觉、触觉多个通道对数字文物的整体和细节信息进行感知,且交互的整体过程简单、自然、有效。结论 本文提出的基于多模感知的数字文物交互式呈现方法,可有效实现对各类数字遗产特别是3维文物的多模重现,在保证较高实时性的同时拥有良好的可用性和情感体验效果。     

Haptic rendering of dynamic volumetric data

With current methods for volume haptics in scientific visualization, features in time-varying data can freely move straight through the haptic probe without generating any haptic feedback the algorithms are simply not designed to handle variation with time but consider only the instantaneous configuration when the haptic feedback is calculated. This article introduces haptic rendering of dynamic volumetric data to provide a means for haptic exploration of dynamic behaviour in volumetric data. We show how haptic feedback can be produced that is consistent with volumetric data moving within the virtual environment and with data that, in itself, evolves over time. Haptic interaction with time-varying data is demonstrated by allowing palpation of a CT sequence of a beating human heart.     

Design and evaluation of haptic effects for use in a computer desktop for the physically disabled

The human–computer interface remains a mostly visual environment with little or no haptic interaction. While haptics is finding inroads in specialized areas such as surgery, gaming, and robotics, there has been little work to bring haptics to the computer desktop, which is largely dominated today by the GUI/mouse relationship. The mouse as an input device, however, poses many challenges for users with physical disabilities, and it is believed that a haptically enhanced interface could have significant impact assisting in target selection. This paper presents a study intended to evaluate haptic effects used with a force feedback mouse on a computer desktop and a prediction algorithm designed to focus those effects on the desired target. Results of the experiment were partially successful and indicated future directions for improvement. The paper introduces the proposed framework and presents experimental results from targeting tasks using differing haptic effects with a group of physically disabled users.     

Biological cell injection visual and haptic interface

This paper presents a new three-dimensional (3-D) biomicromanipulation system for biological objects such as embryos, cells or oocytes. As the cell is very small, kept in liquid and observed through a microscope, 2-D visual feedback makes accurate manipulation in the 3-D world difficult. To improve the manipulation work, we proposed an intelligent human–machine interface. The 3-D visual information is provided to the operator through a 3-D reconstruction method using vision-based tracking deformations of the cell embryo. In order to perform stable microinjection tasks, the operator needs force feedback and haptic assistance during penetration of the cell envelop — the chorion. Thus, realistic haptic rendering techniques have been implemented to validate stable insertion of a micropipette in a living cell. The proposed human–machine user's interface allows real-time realistic visual and haptic control strategies for constrained motion in image coordinates, virtual haptic rendering to constrain the path of insertion and removal in the 3-D scene or to avoid cell destruction by adequately controlling position, velocity and force parameters. Experiments showed that the virtualized reality interface acts as a tool for total guidance and assistance during microinjection tasks.     

Stable adaptive algorithm for Six Degrees-of-Freedom haptic rendering in a dynamic environment

Recently, physically-based simulations with haptics interaction attracted many researchers. In this paper, we propose an adaptive Six Degrees-of-Freedom (6-DOF) haptic rendering algorithm based on virtual coupling, which can automatically adjust virtual coupling parameters according to mass values of the simulated virtual tools. The algorithm can overcome the virtual tool displacement problem caused by the large mass values of the virtual tool and can provide stable force/torque display. The force/torque magnitude is saturated to the maximum force/torque values of the haptic device automatically. The implemented algorithm is tested on the simple and complex standard benchmarks. The experimental results confirm that the proposed adaptive 6-DOF haptic rendering algorithm displays good stability and accuracy for haptic rendering of dynamic virtual objects with mass values.     

基于图像特征的力反馈渲染改进算法

传统基于图像特征的力反馈渲染算法在图像预处理降噪阶段容易丢失虚拟对象表面粗糙度信息。为此,提出一种改进的力反馈渲染算法。在快速矢量滤波器内置噪声检测器之前,增加一级基于图像边缘特征矢量的噪声检测器,以提高噪声检测准确率、保护图像边缘细节特征。实验结果表明,改进算法能够准确描述虚拟对象表面细微的粗糙度信息,实现更逼真的触觉与视觉融合的力反馈。     

Development scheme of haptic-based system for interactive deformable simulation

This paper aims to develop a novel scheme for interactively deformable simulation with haptic feedback. All design modules are packaged and implemented, and the experiments are conducted to study the effects of interactive deformation. The study compares the experimental results of haptic feedback by different force propagation methods, according to the Hounsfield unit (HU) of volume data. Additionally, the estimation of optimal propagation depth is illustrated by using a mass-spring model. Finally, by using the proposed scheme, a haptic-based medical simulation system for brain surgery is investigated. The integration test results with haptic feedback scenarios show that the proposed development scheme can certainly comply with the design modules, and the deformable simulation and haptic force reach a good agreement.     

Deriving haptic design guidelines from human physiological, psychophysical, and neurological foundations

We survey the haptics literature and identify conditions under which haptic interaction displays can enhance human perception and performance. Integrating haptic interactions in multimodal systems requires understanding user's sensory, perceptual, and cognitive abilities and limitations. Haptic design guidelines can aid developers of multimodal interactive systems. Haptic interaction relates to all aspects of touch and body movement and the application of these senses to computer interaction. This involves not only sensation and perception, but also motor and cognitive aspects of active movement (that is, self-initiated movement) for which detailed motor plans are created, stored in memory, and compared to receptor feedback from the muscles, joints, and skin.     

视触觉融合的增强现实三维注册方法

增强现实技术是近年来人机交互领域的研究热点。在增强现实环境下加入触觉感知,可使用户在真实场景中看到并感知到虚拟对象。为了实现增强现实环境下与虚拟对象之间更加自然的交互,提出一种视触觉融合的三维注册方法。基于图像视觉技术获得三维注册矩阵;借助空间转换关系求解出触觉空间与图像空间的转换关系;结合两者与摄像头空间的关系实现视触觉融合的增强现实交互场景。为验证该方法的有效性,设计了一个基于视触觉增强现实的组装机器人项目。用户可触摸并移动真实环境中的机器人零件,还能在触摸时感受到反馈力,使交互更具真实感。     

Generating haptic texture using solid noise

Texture enhances haptic interaction by providing unique, distinguishable, and versatile surfaces. In computer haptics, texture can render environments more realistic and provide useful information. In this paper, an algorithm is proposed for virtual texture simulation by using solid noise, where only a few parameters need to be altered to generate a range of realistic and diverse textures by reproducing different frequencies similar to that of real vibrational signals in a virtual environment. The proposed method can capture the textural effect in a haptic simulation while retaining a simple overall geometry and stable update rate. This method also allows the user to change the texture at runtime and can be easily incorporated into any existing code and used in any traditional haptic device without affecting overall haptic-rendering performance. Moreover, the solid noise texture is independent of object geometry and can be applied to any shape without additional computations. We conducted a human-subject study to evaluate the recognition accuracy for each generated haptic texture as well as its realism and correspondence to real texture. The results indicated the high performance of the method and its ability to generate haptic textures with a very high recognition rate that were highly realistic.     

Haptic rendering: introductory concepts

Haptic rendering allows users to "feel" virtual objects in a simulated environment. We survey current haptic systems and discuss some basic haptic-rendering algorithms. In the past decade we've seen an enormous increase in interest in the science of haptics. Haptics broadly refers to touch interactions (physical contact) that occur for the purpose of perception or manipulation of objects. These interactions can be between a human hand and a real object; a robot end-effector and a real object; a human hand and a simulated object (via haptic interface devices); or a variety of combinations of human and machine interactions with real, remote, or virtual objects. Rendering refers to the process by which desired sensory stimuli are imposed on the user to convey information about a virtual haptic object.     

Improving realism of a surgery simulator: linear anisotropic elasticity,complex interactions and force extrapolation

In this article, we describe the latest developments of the minimally invasive hepatic surgery simulator prototype developed at INRIA. The goal of this simulator is to provide a realistic training test bed to perform laparoscopic procedures. Therefore, its main functionality is to simulate the action of virtual laparoscopic surgical instruments for deforming and cutting tridimensional anatomical models. Throughout this paper, we present the general features of this simulator including the implementation of several biomechanical models and the integration of two force‐feedback devices in the simulation platform. More precisely, we describe three new important developments that improve the overall realism of our simulator. First, we have developed biomechanical models, based on linear elasticity and finite element theory, that include the notion of anisotropic deformation. Indeed, we have generalized the linear elastic behaviour of anatomical models to ‘transversally isotropic’ materials, i.e. materials having a different behaviour in a given direction. We have also added to the volumetric model an external elastic membrane representing the ‘liver capsule’, a rather stiff skin surrounding the liver, which creates a kind of ‘surface anisotropy’. Second, we have developed new contact models between surgical instruments and soft tissue models. For instance, after detecting a contact with an instrument, we define specific boundary constraints on deformable models to represent various forms of interactions with a surgical tool, such as sliding, gripping, cutting or burning. In addition, we compute the reaction forces that should be felt by the user manipulating the force‐feedback devices. The last improvement is related to the problem of haptic rendering. Currently, we are able to achieve a simulation frequency of 25 Hz (visual real time) with anatomical models of complex geometry and behaviour. But to achieve a good haptic feedback requires a frequency update of applied forces typically above 300 Hz (haptic real time). Thus, we propose a force extrapolation algorithm in order to reach haptic real time. Copyright © 2002 John Wiley & Sons, Ltd.     

Experiments in haptic-based authentication of humans

With the rapid advancement of the technological revolution, computer technology such as faster processors, advanced graphic cards, and multi-media systems are becoming more affordable. Haptics technology is a force/tactile feedback technology growing in disciplines linked to human–computer interaction. Similar to the increasing complexity of silicon-based components, haptics technology is becoming more advanced. On the other hand, currently available commercial haptics interfaces are expensive, and their application is mostly dedicated to enormous research projects or systems. However, the trend of the market is forcing haptic developers to release products for use in conjunction with current keyboards and mice technologies. Haptics allows a user to touch, fell, manipulate, create, and/or alter simulated three-dimensional objects in a virtual environment. Most of the existing applications of haptics are dedicated to hone human physical skills such as sensitive hardware repair, medical procedures, handling hazardous substances, etc. These skills can be trained in a realistic virtual world, and describe human behavioural patterns in human–computer interaction environments. The measurement of such psychomotor patterns can be used to verify a person’s identity by assessing unique-to-the-individual behavioural attributes. This paper explores the unique behaviour exhibited by different users interacting with haptic systems. Through several haptic-based applications, users’ physical attributes output data from the haptic interface for use in the construction of a biometric system.

Slingshot 3D: A synchronous haptic-audio-video game

Recent advances in multimedia and human computer interaction technologies have paved the way for rich contents across multiple media such as haptics, audio, and video. This paper introduces a multimodal game named the Slingshot 3D Game: an interactive and synchronous haptic-audio-video shooter game over the Internet network. The game incorporates two types of haptic feedback: tactile feedback using a haptic jacket and kinesthetic feedback using the Novint Falcon haptic interface. Furthermore, the game utilizes a depth camera to track the player's (upper) body movements and detect collisions between the player's body and the shot projectile. To promote availability and cost, the game uses the Internet network as the communication medium between the players, by utilizing the Admux communication framework [10]. The game design and implementation are detailed in this paper. Both the player performance analysis and the user satisfaction analysis have shown that the incorporation of synchronous haptic-video multimedia has enhanced the perception of player presence and the overall quality of performance.