Whole-hand kinesthetic feedback and haptic perception in dextrous virtual manipulation

One of the key requirements for a Virtual Reality system is the multimodal, real-time interaction between the human operator and a computer simulated and animated environment. This paper investigates problems related particularly to the haptic interaction between the human operator and a virtual environment. The work presented here focuses on two issues: 1) the synthesis of whole-hand kinesthetic feedback, based on the application of forces (torques) on individual phalanges (joints) of the human hand, and 2) the experimental evaluation of this haptic feedback system, in terms of human haptic perception of virtual physical properties (such as the weight of a virtual manipulated object), using psychophysical methods. The proposed kinesthetic feedback methodology is based on the solution of a generalized force distribution problem for the human hand during virtual manipulation tasks. The solution is computationally efficient and has been experimentally implemented using an exoskeleton force-feedback glove. A series of experiments is reported concerning the perception of weight of manipulated virtual objects and the obtained results demonstrate the feasibility of the concept. Issues related to the use of sensory substitution techniques for the application of haptic feedback on the human hand are also discussed.     

Biological cell injection visual and haptic interface

This paper presents a new three-dimensional (3-D) biomicromanipulation system for biological objects such as embryos, cells or oocytes. As the cell is very small, kept in liquid and observed through a microscope, 2-D visual feedback makes accurate manipulation in the 3-D world difficult. To improve the manipulation work, we proposed an intelligent human–machine interface. The 3-D visual information is provided to the operator through a 3-D reconstruction method using vision-based tracking deformations of the cell embryo. In order to perform stable microinjection tasks, the operator needs force feedback and haptic assistance during penetration of the cell envelop — the chorion. Thus, realistic haptic rendering techniques have been implemented to validate stable insertion of a micropipette in a living cell. The proposed human–machine user's interface allows real-time realistic visual and haptic control strategies for constrained motion in image coordinates, virtual haptic rendering to constrain the path of insertion and removal in the 3-D scene or to avoid cell destruction by adequately controlling position, velocity and force parameters. Experiments showed that the virtualized reality interface acts as a tool for total guidance and assistance during microinjection tasks.     

Task-based and stable telenanomanipulation in a nanoscale virtual environment

In a haptic interface system with a nanoscale virtual environment (NVE) using an atomic force microscope, not only is stability important, but task-based performance (or fidelity) is crucial. In this paper, we introduce a nanoscale virtual coupling (NSVC) concept and explicitly derive the relationship between performance, stability, and scaling factors of velocity (or position) and force. An available scaling factor region is represented based on Llewellyn's absolute stability criteria and the physical limitation of the haptic device. For the stable haptic interface, the sampled time passivity controller is implemented in the NVE. Experiments have been performed for telenanomanipulation tasks, such as positioning, indenting, and nanolithography with guaranteed stability in the NVE. Note to Practitioners-This paper suggests methods and control schemes for the task-based and stable telenanomanipulation in the nanoscale virtual environment (NVE). The proposed task-based and stable telenanomanipulation in the NVE can be used for an augmented human machine interface for the manipulation of nanoscale objects with the atomic force microscope (AFM). In addition, it is beneficial for learning or performing nanoscale tasks, such as nanolithography, nanoindenting, nanofabrication, and cell manipulation. Also, the interaction with the NVE using haptic device provides a useful tool for researchers in a variety of disciplines, such as biology, chemistry, and physics. Moreover, it may even be applied to educational purposes. In future research, the developed stable haptic interface would be integrated with the AFM system as a slave manipulator for telenanomanipulation experiments, such as pushing a nanoparticle with precise positioning and nanoassembly.     

Hybrid client-server architecture and control techniques for collaborative product development using haptic interfaces

In this paper, a collaborative product development and prototyping framework is proposed by using distributed haptic interfaces along with deformable objects modeling. Collaborative Virtual Environment (CVE) is a promising technique for industrial product development and virtual prototyping. Network control problems such as network traffic and network delay in communication have greatly limited collaborative virtual environment applications. The problems become more difficult when high-update-rate haptic interfaces and computation intensive deformable objects modeling are integrated into CVEs for intuitive manipulation and enhanced realism. A hybrid network architecture is proposed to balance the computational burden of haptic rendering and deformable object simulation. Adaptive artificial time compensation is used to reduce the time discrepancy between the server and the client. Interpolation and extrapolation approaches are used to synchronize graphic and haptic data transmitted over the network. The proposed techniques can be used for collaborative product development, virtual assembly, remote product simulation and other collaborative virtual environments where both haptic interfaces and deformable object models are involved.     

Haptic rendering: introductory concepts

Haptic rendering allows users to "feel" virtual objects in a simulated environment. We survey current haptic systems and discuss some basic haptic-rendering algorithms. In the past decade we've seen an enormous increase in interest in the science of haptics. Haptics broadly refers to touch interactions (physical contact) that occur for the purpose of perception or manipulation of objects. These interactions can be between a human hand and a real object; a robot end-effector and a real object; a human hand and a simulated object (via haptic interface devices); or a variety of combinations of human and machine interactions with real, remote, or virtual objects. Rendering refers to the process by which desired sensory stimuli are imposed on the user to convey information about a virtual haptic object.     

Stable adaptive algorithm for Six Degrees-of-Freedom haptic rendering in a dynamic environment

Recently, physically-based simulations with haptics interaction attracted many researchers. In this paper, we propose an adaptive Six Degrees-of-Freedom (6-DOF) haptic rendering algorithm based on virtual coupling, which can automatically adjust virtual coupling parameters according to mass values of the simulated virtual tools. The algorithm can overcome the virtual tool displacement problem caused by the large mass values of the virtual tool and can provide stable force/torque display. The force/torque magnitude is saturated to the maximum force/torque values of the haptic device automatically. The implemented algorithm is tested on the simple and complex standard benchmarks. The experimental results confirm that the proposed adaptive 6-DOF haptic rendering algorithm displays good stability and accuracy for haptic rendering of dynamic virtual objects with mass values.     

Force modeling for tooth preparation in a dental training system

Feedback force is very important for novices to simulate tooth preparation by using the haptic interaction system (dental training system) in a virtual environment. In the process of haptic simulation, the fidelity of generated forces by a haptic device decides whether the simulation is successful. A force model computes feedback force, and we present an analytical force model to compute the force between a tooth and a dental pin during tooth preparation. The force between a tooth and a dental pin is modeled in two parts: (1) force to resist human’s operation and (2) friction to resist the rotation of the dental engine. The force to resist the human’s operation is divided into three parts in the coordinates that are constructed on the bottom center of the dental pin. In addition, we also consider the effects of dental-pin type, tooth stiffness, and contact geometry in the force model. To determine the parameters of the force model, we construct a measuring system by using machine vision and a force/torque sensor to track the human’s operations and measure the forces between the dental pins and teeth. Based on the measuring results, we construct the relation between the force and the human’s operation. The force model is implemented in the prototype of a dental training system that uses the Phantom as the haptic interface. Dentists performing virtual operations have confirmed the fidelity of feedback force.     

虚拟手抓持力觉生成算法真实性的评价

目的 虽然许多学者研发了多种虚拟手交互触力觉生成算法,但是如何评价虚拟手交互触力觉生成算法的真实性是一个富有挑战性的新问题,值得深入研究.方法 构建手指抓持力测量平台,设计3种抓持姿态下指尖静力抓持球体实验内容,测得各指尖作用力的实测值;通过虚拟手静力抓持力觉生成算法,求得这3种抓持姿态下各手指作用力的理论值;对实测值进行统计和分析,并与理论值进行对比和讨论;结果 日常抓持经验和实测值是完全相符的,实测值和理论值很接近且偏差均在可接受范围之内.单个手指作用力或多个手指合力的实测值与理论值的偏差均在1%6%.结论 本文实现了一种基于物理的实验方法,评价和分析了虚拟手静力抓持力觉生成算法的真实性,证实此算法可以逼真地生成虚拟手抓持力,可应用于具有力反馈的自然的虚拟手交互.     

Application of hidden Markov model to acquisition of manipulation skills from haptic rendered virtual environment

This paper explores the feasibility of reconstructing human manipulation skills in complex constrained motion by tracing and learning the manipulation performed by the operator. The peg-in-hole insertion problem is used as a case study, which represents a typical constrained motion force sensitive manufacturing task with the attendant issues of jamming, tight clearance and the need for quick assembly times. In the developed system, position and contact force and torque as well as orientation data generated in the haptic rendered virtual environment combined with a priori knowledge about the task are used to identify and learn the skills in the newly demonstrated task. The recorded training data is classified into contact states, which are identified with hidden Markov model (HMM) as human skills. The HMM parameters are obtained from the training data. By evaluating the controller's performance in each contact state from haptic rendered virtual environment, the robot develops the best trajectories to imitate the human behaviour. In this paper the significance of this research project is highlighted and the developed approach and the progress made so far on this project are reported.     

A large haptic device for aircraft engine maintainability

The virtual reality for maintainability (Revima) VR system supports maintainability simulation in aeronautics. Within this project we have developed and integrated a haptic device, the large haptic interface for aeronautic maintainability (LHIfAM). We use this device to track hand movements and provide force feedback within the large geometric models that describe aircraft engines. The user movements are the same as those that occur when testing physical mock-ups. An integrated haptic device and VR system for testing aircraft engines reduces development costs and avoids the necessity of physical mock-ups formaintainability.     

Robot-enabled tangible virtual assembly with coordinated midair object placement

The assembly in Virtual Reality (VR) enables users to fit virtual parts into existing 3D models immersively. However, users cannot physically feel the haptic feedback when connecting the parts with the virtual model. This work presents a robot-enabled tangible interface that dynamically moves a physical structure with a robotic arm to provide physical feedback for holding a handheld proxy in VR. This enables the system to provide force feedback during virtual assembly. The cooperation between the physical support and the handheld proxy produces realistic physical force feedback, providing a tangible experience for various virtual parts in virtual assembly scenarios. We developed a prototype system that allowed the operator to place a virtual part onto other models in VR by placing the proxy onto the matched structure attached to a robotic arm. We conducted a user evaluation to explore user performance and system usability in a virtual assembly task. The results indicated that the robot-enabled tangible support increased the task completion time but significantly improved the system usability and sense of presence with a more realistic haptic experience.     

Touch-enabled haptic modeling of deformable multi-resolution surfaces

Currently, interactive data exploration in virtual environments is mainly focused on vision-based and non-contact sensory channels such as visual/auditory displays. The lack of tactile sensation in virtual environments removes an important source of information to be delivered to the users. In this paper, we propose the touch-enabled haptic modeling of deformable multi-resolution surfaces in real time. The 6-DOF haptic manipulation is based on a dynamic model of Loop surfaces, where the dynamic parameters are computed easily without subdividing the control mesh recursively. A local deforming scheme is developed to approximate the solution of the dynamics equations, thus the order of the linear equations is reduced greatly. During each of the haptic interaction loop, the contact point is traced and reflected to the rendering of updated graphics and haptics. The sense of touch against the deforming surface is calculated according to the surface properties and the damping-spring force profile. Our haptic system supports the dynamic modeling of deformable Loop surfaces intuitively through the touch-enabled interactive manipulation.     

Haptic interaction and volume modeling techniques for realistic dental simulation

We present haptic simulation and volume modeling techniques for a virtual dental training system. The system allows dental students to learn dental procedures and master their skills with realistic tactual feelings. It supports various dental procedures, such as dental probing, to diagnose carious lesions, drilling operation for cavity preparation, and filling the prepared cavities with amalgam. The system requires fast and stable haptic rendering and volume modeling techniques working on the virtual tooth. Collision detection and force computation are implemented on an offset surface in volumetric representation to simulate reasonable physical interactions between dental tools with a certain volume and the teeth model. To avoid discrete haptic feeling due to the gap between the fast haptic process (1 KHz) and much slower visual update frequency (30 Hz) during drilling and filling the cavities, we employed an intermediate implicit surface to be animated between the original and target surfaces. The volumetric teeth model is converted into a geometric model by an adaptive polygonization method to maintain sharp features in every visual frame. Volumetric material properties are represented by stiffness and color values to simulate the resistance and texture information depending on anatomical tissues. Finally, we made a dental workbench to register sensory modalities like visual, auditory and haptic sensation.     

Development and Evaluation of a 7-DOF Haptic Interface

With the development of human robot interaction technologies, haptic interfaces are widely used for 3D applications to provide the sense of touch. These interfaces have been utilized in medical simulation, virtual assembly and remote manipulation tasks. However, haptic interface design and control are still critical problems to reproduce the highly sensitive touch sense of humans. This paper presents the development and evaluation of a 7-DOF (degree of freedom) haptic interface based on the modified delta mechanism. Firstly, both kinematics and dynamics of the modified mechanism are analyzed and presented. A novel gravity compensation algorithm based on the physical model is proposed and validated in simulation. A haptic controller is proposed based on the forward kinematics and the gravity compensation algorithm. To evaluate the control performance of the haptic interface, a prototype has been implemented. Three kinds of experiments:gravity compensation, static response and force tracking are performed respectively. The experimental results show that the mean error of the gravity compensation is less than 0.7 N and the maximum continuous force along the axis can be up to 6 N. This demonstrates the good performance of the proposed haptic interface.      

用于实时柔性触觉再现的平行菱形链连接模型

精度高且实时性好的柔性触觉变形模型是实现触觉再现系统的关键。提出了一种新的基于物理意义的平行菱形链连接触觉变形模型,系统中各个链结构单元相对位移的叠加对外等效为物体表面的变形,与之相连的弹簧弹性力的合力等效为物体表面的接触力。使用Delta 6-DOF手控器,建立了触觉再现实验系统,对柔性体的接触变形和实时虚拟触觉反馈进行仿真, 实验结果表明所提出的模型不仅计算简单,而且能够保证触觉接触力和形变计算具有较高精度,满足虚拟现实系统对精细作业和实时性的要求。     

The effect of sub-threshold forces on human performance in multi-modal computer-aided design

In traditional CAD (computer-aided design) systems, the manipulation of points and lines is often difficult because designers manipulate virtual objects through their vision system. Nowadays, designers can explore and manipulate virtual objects in haptic-enabled CAD systems using haptic devices. Haptic devices can present force feedbacks to pull or push the users’ hands into desirable targets. Of course the intent is for the user to experience the same sensations in the virtual realm as they would in the real world. Thus, sub-threshold forces, which cannot be perceived by users, should be incorporated in the control of users’ movements. As a result, our attention is directed to study the effect of sub-threshold forces on the accuracy of movement in a haptic-enabled virtual reality (VR) system. In this study, our goal is to manipulate users’ hands using controlled forces such that users cannot notice the forces. With this in mind, we have constructed a haptic-enabled virtual environment (VE) to carry out a multi-modal Fitts’ type task. In the task, subjects could see the position of the haptic probe in the VE where forces were applied on their hands. Basically, the accuracy of subjects was measured using a performance index when the intensity and direction of forces changed. A psychophysical method was utilized to ensure that the forces were below the force threshold of the human force perception. Results indicate that the accuracy is affected by the intensity and direction of sub-threshold forces even when users are allowed to control their actions through visual feedbacks.     

The moderating role of sensation seeking tendency in robotic haptic interfaces

This study examined the interaction effects between haptic force feedback and users’ sensation seeking tendency (i.e. need for sensations) on users’ feelings of presence (i.e. the state in which users experience virtual objects and virtual environments as if they were actual) in robotic haptic interfaces. Users with low sensation seeking tendency felt stronger physical presence and spatial presence in response to force feedback haptic stimuli (versus no force feedback), whereas users with high sensation seeking tendency did not show any difference between the two conditions, thus confirming the moderating role of the users’ sensation seeking tendency in the robotic haptic interface. Theoretical implications for human–computer interaction (HCI) research and managerial implications for the interactive media market are discussed.     

A Survey of Haptic Rendering Techniques

Computer Graphics technologies have developed considerably over the past decades. Realistic virtual environments can be produced incorporating complex geometry for graphical objects and utilising hardware acceleration for per pixel effects. To enhance these environments, in terms of the immersive experience perceived by users, the human's sense of touch, or haptic system, can be exploited. To this end haptic feedback devices capable of exerting forces on the user are incorporated. The process of determining a reaction force for a given position of the haptic device is known as haptic rendering. For over a decade users have been able to interact with a virtual environment with a haptic device. This paper focuses on the haptic rendering algorithms which have been developed to compute forces as users manipulate the haptic device in the virtual environment.     

Beyond the visuals: tactile augmentation and sensory enhancement in an arthroscopy simulator

This paper considers tactile augmentation, the addition of a physical object within a virtual environment (VE) to provide haptic feedback. The resulting mixed reality environment is limited in terms of the ease with which changes can be made to the haptic properties of objects within it. Therefore sensory enhancements or illusions that make use of visual cues to alter the perceived hardness of a physical object allowing variation in haptic properties are considered. Experimental work demonstrates that a single physical surface can be made to ‘feel’ both softer and harder than it is in reality by the accompanying visual information presented. The strong impact visual cues have on the overall perception of object hardness, indicates haptic accuracy may not be essential for a realistic virtual experience. The experimental results are related specifically to the development of a VE for surgical training; however, the conclusions drawn are broadly applicable to the simulation of touch and the understanding of haptic perception within VEs.     

Layered rhombus-chain-connected model for real-time haptic rendering

The modeling and simulation of deformable objects is a challenging topic in the field of haptic rendering between human and virtual environment. In this paper, a novel and efficient layered rhombus-chain-connected haptic deformation model based on physics is proposed for an excellent haptic rendering. During the modeling, the accumulation of relative displacements in each chain structure unit in each layer is equal to the deformation on the virtual object surface, and the resultant force of corresponding springs is equivalent to the external force. The layered rhombus-chain-connected model is convenient and fast to calculate, and can satisfy real-time requirement due to its simplicity. Experimental study in both homogenous and non-homogenous virtual human liver and lungs based on the proposed model are conducted, and the results demonstrate that our model provides stable and realistic haptic feeling in real time. Meanwhile, the display result is vivid.