Real-time area-based haptic rendering and the augmented tactile display device for a palpation simulator

Although people usually contact a surface with some area rather than a point, most haptic devices allow a user to interact with a virtual object at one point at a time and likewise most haptic rendering algorithms deal with such situations only. In a palpation procedure, medical doctors push and rub the organ's surface, and are provided the sensation of distributed pressure and contact force (reflecting force) for discerning doubtable areas of the organ. In this paper, we suggest real-time area-based haptic rendering to describe distributed pressure and contact force simultaneously, and present a haptic interface system to generate surface properties in accordance with the haptic rendering algorithm. We represent the haptic model using the shape-retaining chain link (S-chain) framework for a fast and stable computation of the contact force and distributed pressure from a volumetric virtual object. In addition, we developed a compact pin-array-type tactile display unit and attached it to the PHANToM^TM haptic device to complement each other. For the evaluation, experiments were conducted with non-homogenous volumetric cubic objects consisting of approximately 500 000 volume elements. The experimental results show that compared to the point contact, the area contact provides the user with more precise perception of the shape and softness of the object's composition, and that our proposed system satisfies the real-time and realism constraints to be useful for a virtual reality application.     

Implicit Surface-Based Geometric Fusion

This paper introduces a general purpose algorithm for reliable integration of sets of surface measurements into a single 3D model. The new algorithm constructs a single continuous implicit surface representation which is the zero-set of a scalar field function. An explicit object model is obtained using any implicit surface polygonization algorithm. Object models are reconstructed from both multiple view conventional 2.5D range images and hand-held sensor range data. To our knowledge this is the first geometric fusion algorithm capable of reconstructing 3D object models from noisy hand-held sensor range data.This approach has several important advantages over existing techniques. The implicit surface representation allows reconstruction of unknown objects of arbitrary topology and geometry. A continuous implicit surface representation enables reliable reconstruction of complex geometry. Correct integration of overlapping surface measurements in the presence of noise is achieved using geometric constraints based on measurement uncertainty. The use of measurement uncertainty ensures that the algorithm is robust to significant levels of measurement noise. Previous implicit surface-based approaches use discrete representations resulting in unreliable reconstruction for regions of high curvature or thin surface sections. Direct representation of the implicit surface boundary ensures correct reconstruction of arbitrary topology object surfaces. Fusion of overlapping measurements is performed using operations in 3D space only. This avoids the local 2D projection required for many previous methods which results in limitations on the object surface geometry that is reliably reconstructed. All previous geometric fusion algorithms developed for conventional range sensor data are based on the 2.5D image structure preventing their use for hand-held sensor data. Performance evaluation of the new integration algorithm against existing techniques demonstrates improved reconstruction of complex geometry.     

Haptics-based dynamic implicit solid modeling

We systematically present a novel, interactive solid modeling framework, haptics-based dynamic implicit solid modeling, which is founded upon volumetric implicit functions and powerful physics-based modeling. In particular, we augment our modeling framework with a haptic mechanism in order to take advantage of additional realism associated with a 3D haptic interface. Our dynamic implicit solids are semialgebraic sets of volumetric implicit functions and are governed by the principles of dynamics, hence responding to sculpting forces in a natural and predictable manner. In order to directly manipulate existing volumetric data sets as well as point clouds, we develop a hierarchical fitting algorithm to reconstruct and represent discrete data sets using our continuous implicit functions, which permit users to further design and edit those existing 3D models in real-time using a large variety of haptic and geometric toolkits, and visualize their interactive deformation at arbitrary resolution. The additional geometric and physical constraints afford more sophisticated control of the dynamic implicit solids. The versatility of our dynamic implicit modeling enables the user to easily modify both the geometry and the topology of modeled objects, while the inherent physical properties can offer an intuitive haptic interface for direct manipulation with force feedback.     

Haptic interaction and volume modeling techniques for realistic dental simulation

We present haptic simulation and volume modeling techniques for a virtual dental training system. The system allows dental students to learn dental procedures and master their skills with realistic tactual feelings. It supports various dental procedures, such as dental probing, to diagnose carious lesions, drilling operation for cavity preparation, and filling the prepared cavities with amalgam. The system requires fast and stable haptic rendering and volume modeling techniques working on the virtual tooth. Collision detection and force computation are implemented on an offset surface in volumetric representation to simulate reasonable physical interactions between dental tools with a certain volume and the teeth model. To avoid discrete haptic feeling due to the gap between the fast haptic process (1 KHz) and much slower visual update frequency (30 Hz) during drilling and filling the cavities, we employed an intermediate implicit surface to be animated between the original and target surfaces. The volumetric teeth model is converted into a geometric model by an adaptive polygonization method to maintain sharp features in every visual frame. Volumetric material properties are represented by stiffness and color values to simulate the resistance and texture information depending on anatomical tissues. Finally, we made a dental workbench to register sensory modalities like visual, auditory and haptic sensation.     

Haptic rendering based on spatial run-length encoding

In this paper, an extendable volumetric representation based on run-lengths called spatial run-length encoding (S-RLE) is presented. The S-RLE representation is developed for a haptic shape modeling system that is based on simulated machining processes. In the system, shape modeling is simulated as virtual material removal processes similar to machining processes with volume-based haptic rendering. The object and the tools are represented by S-RLE. The data structure of S-RLE consists of two cross-referenced databases: one is a stack of lists in geometrical domain, recording the runs describing the space occupation of the object; the other is a table in physical domain, describing the physical properties of each element. The latter is extendable to include more diverse physical properties such as parts composed of heterogeneous materials. Algorithms for geometric operations and haptic rendering based on S-RLE are developed. The proposed S-RLE data structure has the features of efficient memory usage, quick collision detection, inherent representation for heterogeneous objects, and fast visual rendering.     

Geometric fusion for a hand-held 3D sensor

Abstract. This article presents a geometric fusion algorithm developed for the reconstruction of 3D surface models from hand-held sensor data. Hand-held systems allow full 3D movement of the sensor to capture the shape of complex objects. Techniques previously developed for reconstruction from conventional 2.5D range image data cannot be applied to hand-held sensor data. A geometric fusion algorithm is introduced to integrate the measured 3D points from a hand-held sensor into a single continuous surface. The new geometric fusion algorithm is based on the normal-volume representation of a triangle, which enables incremental transformation of an arbitrary mesh into an implicit volumetric field function. This system is demonstrated for reconstruction of surface models from both hand-held sensor data and conventional 2.5D range images. Received: 30 August 1999 / Accepted: 21 January 2000     

虚拟装配中基于导纳控制的力觉渲染技术

传统的力触觉渲染多采用阻抗控制,不能很好地满足虚拟装配的应用要求,相比之下导纳控制模式更适用这一领域.为此提出一种基于导纳控制的双线程力觉渲染构架,并给出相应的力觉渲染算法.首先建立用于导纳控制的动力学模型,并讨论了碰撞和约束这2个状态下的力觉渲染;为了使用力觉交互接口进行虚拟装配中的小间隙装配,提出物理约束与几何约束结合的力觉渲染方法;最后针对物理计算和力反馈循环2个线程刷新频率不匹配的问题,利用二次拉格朗日多项式进行数值插值,实现了力觉交互接口的平稳输出.通过力反馈设备与自主开发的虚拟装配原型系统VAPP的连接与应用,验证了所提出的算法满足虚拟装配系统中力觉交互的应用要求.     

6DoF haptic rendering using distance maps over implicit representations

This paper presents a haptic rendering scheme based on distance maps over implicit surfaces. Using the successful concept of support planes and mappings, a support plane mapping formulation is used so as to generate a convex representation and efficiently perform collision detection. The proposed scheme enables, under specific assumptions, the analytical reconstruction of the rigid 3D object’s surface, using the equations of the support planes and their respective distance map. As a direct consequence, the problem of calculating the force feedback can be analytically solved using only information about the 3D object’s spatial transformation and position of the haptic interaction point. Moreover, several haptic effects are derived by the proposed mesh-free haptic rendering formulation. Experimental evaluation and computational complexity analysis demonstrates that the proposed approach can reduce significantly the computational cost when compared to existing methods.     

Research issues in volume visualization

Volume visualization is a method of extracting meaningful information from volumetric data sets through the use of interactive graphics and imaging. It addresses the representation, manipulation, and rendering of volumetric data sets, providing mechanisms for peering into structures and understanding their complexity and dynamics. Typically, the data set is represented as a 3D regular grid of volume elements (voxels) and stored in a volume buffer (also called a cubic frame buffer), which is a large 3D array of voxels. However, data is often defined at scattered or irregular locations that require using alternative representations and rendering algorithms. There are eight major research issues in volume visualization: volume graphics, volume rendering, transform coding of volume data, scattered data, enriching volumes with knowledge, segmentation, real-time rendering and parallelism, and special purpose hardware     

高度场八叉树的体特征表达算法

体特征表达对用户理解和认知虚拟环境有着至关重要的作用。当前的体特征表达算法由于存储量大且不易于在GPU中加速等问题,渲染效率低下,难以满足场景可视化的实时性需求。针对这一问题,提出了一种高效的高度场八叉树体特征表达算法,不仅解决了传统高度场仅能表达2.5维模型,无法表达真三维模型的问题,而且为体特征表达提供了一种新的可行途径。算法使用八叉树结构生成三维模型的高度场表示,将传统的z向高度场扩展到x,y,z三个方向的高度场。首先,提出了三角面片预处理方法,保证模型精度和数据的完整性;其次,提出了基于投影变换的高度场表示判断及栅格化方法,将几何图元转换成二维空间的高度场数据;最后,提出了基于高度场八叉树的光线投射算法。实验结果表明,算法能极大地减少存储量,具有较高的光线投射效率,表达三维模型时取得较好效果。     

Immersive structural biology: a new approach to hybrid modeling of macromolecular assemblies

Advanced biophysical imaging techniques, such as cryo-electron microscopy or tomography, enable 3D volumetric reconstructions of large macromolecular complexes in a near-native environment. However, pure volumetric data is insufficient for a detailed understanding of the underlying protein–protein interactions. This obstacle can be overcome by assembling an atomic model of the whole protein complex from known atomic structures, which are available from either X-ray crystallography or homology modeling. Due to many factors such as noise, conformational variability, experimental artifacts, and inexact model structures, existing automatic docking procedures are known to report false positives for a significant number of cases. The present paper focuses on a new technique to combine an offline exhaustive search algorithm with interactive visualization, collision detection, and haptic rendering. The resulting software system is highly immersive and allows the user to efficiently solve even difficult multi-resolution docking problems. Stereoscopic viewing, combined with head tracking and force feedback, generates an ideal virtual environment for true interaction with and solution of hybrid biomolecular modeling problems.     

Real-Time Volume Deformations

Real-time free-form deformation tools are primarily based on surface or particle representations to allow for interactive modification and fast rendering of complex models. The efficient handling of volumetric representations, however, is still a challenge and has not yet been addressed sufficiently. Volumetric models, on the other hand, form an important class of representation in many applications. In this paper we present a novel approach to the real-time deformation of scalar volume data sets taking advantage of hardware supported 3D texture mapping. In a prototype implementation a modeling environment has been designed that allows for interactive manipulation of arbitrary parts of volumetric objects. In this way, any desired shape can be modeled and used subsequently in various applications. The underlying algorithms have wide applicability and can be exploited effectively for volume morphing and medical data processing.     

Haptic rendering of dynamic volumetric data

With current methods for volume haptics in scientific visualization, features in time-varying data can freely move straight through the haptic probe without generating any haptic feedback the algorithms are simply not designed to handle variation with time but consider only the instantaneous configuration when the haptic feedback is calculated. This article introduces haptic rendering of dynamic volumetric data to provide a means for haptic exploration of dynamic behaviour in volumetric data. We show how haptic feedback can be produced that is consistent with volumetric data moving within the virtual environment and with data that, in itself, evolves over time. Haptic interaction with time-varying data is demonstrated by allowing palpation of a CT sequence of a beating human heart.     

Spatially varying image based lighting using HDR-video

Illumination is one of the key components in the creation of realistic renderings of scenes containing virtual objects. In this paper, we present a set of novel algorithms and data structures for visualization, processing and rendering with real world lighting conditions captured using High Dynamic Range (HDR) video. The presented algorithms enable rapid construction of general and editable representations of the lighting environment, as well as extraction and fitting of sampled reflectance to parametric BRDF models. For efficient representation and rendering of the sampled lighting environment function, we consider an adaptive (2D/4D) data structure for storage of light field data on proxy geometry describing the scene. To demonstrate the usefulness of the algorithms, they are presented in the context of a fully integrated framework for spatially varying image based lighting. We show reconstructions of example scenes and resulting production quality renderings of virtual furniture with spatially varying real world illumination including occlusions.     

Improving virtual reality navigation tasks using a haptic vest and upper body tracking

Haptic feedback is an important component of immersive virtual reality (VR) applications that is often suggested to complement visual information through the sense of touch. This paper investigates the use of a haptic vest in navigation tasks. The haptic vest produces a repulsive vibrotactile feedback from nearby static virtual obstacles that augments the user spatial awareness. The tasks require the user to perform complex movements in a 3D cluttered virtual environment, like avoiding obstacles while walking backwards and pulling a virtual object. The experimental setup consists of a room-scale environment. Our approach is the first study where a haptic vest is tracked in real time using a motion capture device so that proximity-based haptic feedback can be conveyed according to the actual movement of the upper body of the user.User study experiments have been conducted with and without haptic feedback in virtual environments involving both normal and limited visibility conditions. A quantitative evaluation was carried out by measuring task completion time and error (collision) rate. Multiple haptic rendering techniques have also been tested. Results show that under limited visibility conditions proximity-based haptic feedback generated by a wearable haptic vest can significantly reduce the number of collisions with obstacles in the virtual environment.     

Parametric Surface Representation with Bump Image for Dense 3D Modeling Using an RBG-D Camera

When constructing a dense 3D model of an indoor static scene from a sequence of RGB-D images, the choice of the 3D representation (e.g. 3D mesh, cloud of points or implicit function) is of crucial importance. In the last few years, the volumetric truncated signed distance function (TSDF) and its extensions have become popular in the community and largely used for the task of dense 3D modelling using RGB-D sensors. However, as this representation is voxel based, it offers few possibilities for manipulating and/or editing the constructed 3D model, which limits its applicability. In particular, the amount of data required to maintain the volumetric TSDF rapidly becomes huge which limits possibilities for portability. Moreover, simplifications (such as mesh extraction and surface simplification) significantly reduce the accuracy of the 3D model (especially in the color space), and editing the 3D model is difficult. We propose a novel compact, flexible and accurate 3D surface representation based on parametric surface patches augmented by geometric and color texture images. Simple parametric shapes such as planes are roughly fitted to the input depth images, and the deviations of the 3D measurements to the fitted parametric surfaces are fused into a geometric texture image (called the Bump image). A confidence and color texture image are also built. Our 3D scene representation is accurate yet memory efficient. Moreover, updating or editing the 3D model becomes trivial since it is reduced to manipulating 2D images. Our experimental results demonstrate the advantages of our proposed 3D representation through a concrete indoor scene reconstruction application.     

Interactive visualization of three-dimensional vector fields with flexible appearance control

In this paper, we present an interactive texture-based algorithm for visualizing three-dimensional steady and unsteady vector fields. The goal of the algorithm is to provide a general volume rendering framework allowing the user to compute three-dimensional flow textures interactively and to modify the appearance of the visualization on the fly. To achieve our goal, we decouple the visualization pipeline into two disjoint stages. First, flow lines are generated from the 3D vector data. Various geometric properties of the flow paths are extracted and converted into a volumetric form using a hardware-assisted slice sweeping algorithm. In the second phase of the algorithm, the attributes stored in the volume are used as texture coordinates to look up an appearance texture to generate both informative and aesthetic representations of the vector field. Our algorithm allows the user to interactively navigate through different regions of interest in the underlying field and experiment with various appearance textures. With our algorithm, visualizations with enhanced structural perception using various visual cues can be rendered in real time. A myriad of existing geometry-based and texture-based visualization techniques can also be emulated.     

A B-spline based framework for volumetric object modeling

With the recent development of Iso-geometric Analysis (IGA) (Cottrell et al., 2009) and advanced manufacturing technologies employing heterogeneous materials, such as additive manufacturing (AM) of functionally graded material, there is a growing emerging need for a full volumetric representation of 3D objects, that prescribes the interior of the object in addition to its boundaries. In this paper, we propose a volumetric representation (V-rep) for geometric modeling that is based on trimmed B-spline trivariates and introduce its supporting volumetric modeling framework. The framework includes various volumetric model (V-model) construction methods from basic non-singular volumetric primitives to high level constructors, as well as Boolean operations’ support for V-models. A V-model is decomposed into and defined by a complex of volumetric cells (V-cells), each of which can also represent a variety of additional varying fields over it, and hence over the entire V-model. With these capabilities, the proposed framework is able of supporting volumetric IGA needs as well as represent and manage heterogeneous materials for AM. Further, this framework is also a seamless extension to existing boundary representations (B-reps) common in all contemporary geometric modeling systems, and allows a simple migration of existing B-rep data, tools and algorithms. Examples of volumetric models constructed using the proposed framework are presented.     

Sensation-preserving haptic rendering

Most human-computer interactive systems focus primarily on the graphical rendering of visual information and, to a lesser extent, on the display of auditory information. Haptic interfaces have the potential to increase the quality of human-computer interaction by accommodating the sense of touch. They provide an attractive augmentation to visual display and enhance the level of understanding of complex data sets. A haptic rendering system generates contact or restoring forces to prevent penetration into the virtual objects and create a sense of touch. The system computes contact forces by first detecting if a collision or penetration has occurred. Then, the system determines the (projected) contact points on the model surface. Finally, it computes restoring forces based on the amount of penetration. Researchers have recently investigated the problem of rendering the contact forces and torques between 3D virtual objects. This problem is known as six-degrees-of-freedom (6-DOF) haptic rendering, as the computed output includes both 3-DOF forces and 3-DOF torques. This article presents an overview of our work in this area. We suggest different approximation methods based on the principle of preserving the dominant perceptual factors in haptic exploration.     

Scalar-function-driven editing on point set surfaces

Three-dimensional acquisition is an increasingly popular means of creating surface models. As 3D digital photographic and scanning devices produce higher resolution images, acquired geometric data sets grow more complex in terms of the modeled objects' size, geometry, and topology. Point-based geometry is a form of 3D content acquisition popular in graphics and related visual computing areas. Point set surfaces are enjoying a renaissance in modeling and rendering, with many efforts focused on direct rendering techniques and effective modeling mechanisms. We've developed a scalar-field-driven editing paradigm and system for point set surfaces that let users manipulate and sculpt point clouds intuitively and efficiently. The paradigm is primarily based on the representation of versatile, embedded scalar fields associated with any region of the point set surface. After constructing the surface distance field from the point clouds, we incorporate the dynamic implicit volumetric model into the point-based geometry deformation. We've also integrated a haptics interface into our surface-modeling framework.