Realistic haptic rendering of interacting deformable objects in virtual environments

A new computer haptics algorithm to be used in general interactive manipulations of deformable virtual objects is presented. In multimodal interactive simulations, haptic feedback computation often comes from contact forces. Subsequently, the fidelity of haptic rendering depends significantly on contact space modeling. Contact and friction laws between deformable models are often simplified in up to date methods. They do not allow a "realistic" rendering of the subtleties of contact space physical phenomena (such as slip and stick effects due to friction or mechanical coupling between contacts). In this paper, we use Signorini's contact law and Coulomb's friction law as a computer haptics basis. Real-time performance is made possible thanks to a linearization of the behavior in the contact space, formulated as the so-called Delassus operator, and iteratively solved by a Gauss-Seidel type algorithm. Dynamic deformation uses corotational global formulation to obtain the Delassus operator in which the mass and stiffness ratio are dissociated from the simulation time step. This last point is crucial to keep stable haptic feedback. This global approach has been packaged, implemented, and tested. Stable and realistic 6D haptic feedback is demonstrated through a clipping task experiment.     

Proposal of an Index for the Operator’s Haptic Sensation in a Master-Slave System with Force-Feedback Function

This article proposes an index to estimate the operator’s haptic sensation of the contact between the slave device and the environment in operating master–slave systems with force feedback function. The index value is derived from the velocity information of the master device before and after contact, which is hypothesized to represent the intensity of haptic sensation stimuli presented to the operator. Two characteristics of this index are discussed by means of psychophysics experiment, which are the statistical characteristics of the index value for different operators, and how the change in the operator’s haptic sensation is reflected on the index value. The index is validated by another psychophysics experiment. The experimental results show that the performance of operator’s haptic sensation can be predicted correctly based on the proposed index value. This index is expected to be applied in the parameter design of bilateral-control systems with force feedback function.     

Upper-body haptic system for snake robot teleoperation in pipelines

Snake robots have shown a great potential for operations in confined workplaces that are less accessible or dangerous to human workers, such as the in-pipe inspection. However, the snake robot teleoperation remains a nontrivial task due to the unique locomotion mechanism (e.g., helical motion) and the constraints of the workplaces including the low visibility and indistinguishable features. Most snake robot feedback systems are based on the live camera view only. It is hard for the human operator to develop a correct spatial understanding of the remote workplace, leading to problems such as disorientation and motion sickness in snake robot teleoperation. This study designs and evaluates an innovative haptic assistant system for snake robot teleoperation in the in-pipe inspection. An upper-body haptic suit with 40 vibrators on both the front and back sides of the human operator was developed to generate haptic feedback corresponding to the bottom and up sides of the snake robot, transferring the egocentric sensation of the snake robot to the human operator. A human-subject experiment (n = 31) was performed to evaluate the efficacy of the developed system. The results indicate that the proposed haptic assistant system outperformed other feedback systems in terms of both task performance and subjective workload and motion sickness evaluations. It inspires new control and feedback designs for the future snake robot in industrial operations.     

下颌骨拼接与导板设计的力反馈虚拟手术研究

传统的颅颌面术前规划与手术训练缺少力的触觉反馈，导致医生在手术建模与导板设计的过程效率低下。针对提高医生术前规划效率和训练的效果，设计了一种基于力反馈的颅颌面虚拟手术仿真平台。该系统采用力反馈机制与碰撞检测机制，能够对模型进行选取并移动而达到对下颌骨精确拼接的效果，同时能够设计出相应的导板。实验结果证明使用该平台进行颅颌面手术术前规划和导板设计更有效率，仿真结果的实时性和真实性也较高。     

Haptic force control based on impedance/admittance control aided by visual feedback

This paper demonstrates a haptic device for interaction with a virtual environment. The force control is added by visual feedback that makes the system more responsive and accurate. There are two popular control methods widely used in haptic controller design. First, is impedance control when user motion input is measured, and then, the reaction force is fed back to the operator. The alternative method is admittance control, when forces exerted by user are measured and motion is fed back to the user. Both, impedance and admittance control are also basic ways for interacting with a virtual environment. In this paper, several experiments were performed to evaluate the suitability of force-impedance control for haptic interface development. The difference between conventional application of impedance control in robot motion control and its application in haptic interface development is investigated. Open loop impedance control methodology is implemented for static case and a general-purpose robot under open loop impedance control was developed as a haptic device, while a closed loop model based impedance control was used for haptic controller design in both static and dynamic case. The factors that could affect to the performance of a haptic interface are also investigated experimentally using parametric studies. Experimental results for 1 DOF rotational motion and 2 DOF planar translational motion systems are presented. The results show that the impedance control aided by visual feedback broaden the applicability of the haptic device and makes the system more responsive and accurate.

Whole-hand kinesthetic feedback and haptic perception in dextrous virtual manipulation

One of the key requirements for a Virtual Reality system is the multimodal, real-time interaction between the human operator and a computer simulated and animated environment. This paper investigates problems related particularly to the haptic interaction between the human operator and a virtual environment. The work presented here focuses on two issues: 1) the synthesis of whole-hand kinesthetic feedback, based on the application of forces (torques) on individual phalanges (joints) of the human hand, and 2) the experimental evaluation of this haptic feedback system, in terms of human haptic perception of virtual physical properties (such as the weight of a virtual manipulated object), using psychophysical methods. The proposed kinesthetic feedback methodology is based on the solution of a generalized force distribution problem for the human hand during virtual manipulation tasks. The solution is computationally efficient and has been experimentally implemented using an exoskeleton force-feedback glove. A series of experiments is reported concerning the perception of weight of manipulated virtual objects and the obtained results demonstrate the feasibility of the concept. Issues related to the use of sensory substitution techniques for the application of haptic feedback on the human hand are also discussed.     

Prediction-based methods for teleoperation across delayed networks

The remote nature of telepresence scenarios can be seen as a strongpoint and also as a weakness. Although it enables the remote control of robots in dangerous or inaccessible environments, it necessarily involves some kind of communication mechanism for the transmission of control signals. This communication mechanism necessarily involves adverse network effects such as delay. Three mechanisms aimed at improving the effects of network delay are presented in this paper: (1) Motion prediction to partially compensate for network delays, (2) force prediction to learn a local force model, thereby reducing dependency on delayed force signals, and (3) haptic data compression to reduce the required bandwidth of high frequency data. The utilized motion prediction scheme was shown to improve operator performance, but had no influence on operator immersion. The force prediction provided haptic feedback through synchronous forces from the local model, thereby stabilizing the control loop. The developed haptic data compression scheme reduced the number of packets sent across the network by 90%, while improving the quality of the haptic feedback.     

Forbidden region virtual fixtures from streaming point clouds

Forbidden region virtual fixtures protect objects from unwanted contact with a robot. In this paper, we propose a method for creating forbidden region haptic virtual fixtures for teleoperation from streaming point clouds obtained by an RGB-D camera. Upon violating the protected area, the operator receives force feedback that opposes motion inside the forbidden region. Three architectures for creating virtual fixtures are presented and their advantages and disadvantages are described. The proposed methods have the ability to implement constraints and can handle dynamic environments in real-time. The effectiveness of the methods is demonstrated in experiments with a surgical robot.     

A haptic pedal for surgery assistance

The research and development of mechatronic aids for surgery is a persistent challenge in the field of robotic surgery. This paper presents a new haptic pedal conceived to assist surgeons in the operating room by transmitting real-time surgical information through the foot. An effective human–robot interaction system for medical practice must exchange appropriate information with the operator as quickly and accurately as possible. Moreover, information must flow through the appropriate sensory modalities for a natural and simple interaction. However, users of current robotic systems might experience cognitive overload and be increasingly overwhelmed by data streams from multiple modalities. A new haptic channel is thus explored to complement and improve existing systems. A preliminary set of experiments has been carried out to evaluate the performance of the proposed system in a virtual surgical drilling task. The results of the experiments show the effectiveness of the haptic pedal in providing surgical information through the foot.     

A haptic interface for computer-integrated endoscopic surgery and training

Haptic feedback has the potential to provide superior performance in computer-integrated surgery and training. This paper discusses the design of a user interface that is capable of providing force feedback in all the degrees of freedom (DOFs) available during endoscopic surgery. Using the Jacobian matrix of the haptic interface and its singular values, methods are proposed for analysis and optimization of the interface performance with regard to the accuracy of force feedback, the range of applicable forces, and the accuracy of control. The haptic user interface is used with a sensorized slave robot to form a master–slave test-bed for studying haptic interaction in a minimally invasive environment. Using the master–slave test-bed, teleoperation experiments involving a single degree of freedom surgical task (palpation) are conducted. Different bilateral control methods are compared based on the transparency of the master–slave system in terms of transmitting the critical task-related information to the user in the context of soft-tissue surgical applications.     

Construction of a supermicro sense of force feedback and vision for micro-objects: development of a haptic device

This research aims to develop a combined sense system that uses both the force feedback and visual feedback to establish the shape of microscopic features of a microsample. It is thought that the efficiency of minute procedures would be improved if the operator could obtain a sense of force while using a manipulator. We used a cantilever to touch a minute object and obtain a reaction force from the degree of bending. We made a haptic device which gives a sense of that force to the operator, who can feel the force when a user touches a sample with a cantilever. In addition, when the haptic device is used in simulations, the user can feel a force just as if the user had touched a sample.     

Virtual DesignWorks—designing 3D CAD models via haptic interaction

Haptic devices (force feedback interfaces) show great promise for use in the design of CAD models. However, current haptic systems are used primarily to verify rather than to interact with CAD systems to design a model.The paper discusses a haptic-based CAD system—Virtual DesignWorks. It is the world's first haptic application for the design of CAD models based on component technology (COM+). With haptic devices, the system allows engineers, in virtual space, to directly touch a native B-Rep CAD model, and deform it by pushing, pulling and dragging its surfaces in a natural three-dimensional environment. Virtual DesignWorks adopts the novel COM-based haptic model, which demonstrates significant advantages compared to the traditional haptic models. Force feedback gives designers the greatest flexibility for the design of complex surfaces, and haptics has the potential to become a critical interface for design applications.     

面向上颌骨骨折复位手术的虚拟系统设计

基于3D计算机触觉视觉交互(CHAI3D)和开放图形库(Open GL)等开源软件,设计了针对上颌骨复位手术的仿真系统。使用真实病例的CT图像搭建虚拟场景,通过Geomagic力反馈设备对虚拟模型进行三维操作并输出触觉反馈。在原有单点碰撞算法的基础上,提出了使用多个中介代理的多点碰撞算法,避免了虚拟手术工具的手柄插入虚拟器官的不实仿真;通过力反馈设备对头颅骨模型进行选择、移动和旋转,模拟手术中对头颅骨的移动和放置。系统可用于训练医学院学生,也可用于复杂手术的术前规划。     

Biological cell injection visual and haptic interface

This paper presents a new three-dimensional (3-D) biomicromanipulation system for biological objects such as embryos, cells or oocytes. As the cell is very small, kept in liquid and observed through a microscope, 2-D visual feedback makes accurate manipulation in the 3-D world difficult. To improve the manipulation work, we proposed an intelligent human–machine interface. The 3-D visual information is provided to the operator through a 3-D reconstruction method using vision-based tracking deformations of the cell embryo. In order to perform stable microinjection tasks, the operator needs force feedback and haptic assistance during penetration of the cell envelop — the chorion. Thus, realistic haptic rendering techniques have been implemented to validate stable insertion of a micropipette in a living cell. The proposed human–machine user's interface allows real-time realistic visual and haptic control strategies for constrained motion in image coordinates, virtual haptic rendering to constrain the path of insertion and removal in the 3-D scene or to avoid cell destruction by adequately controlling position, velocity and force parameters. Experiments showed that the virtualized reality interface acts as a tool for total guidance and assistance during microinjection tasks.     

Bilateral teleoperation of a group of mobile robots for cooperative tasks

A visual and force feedback-based teleoperation scheme is proposed for cooperative tasks. The bilateral teleoperation system includes a haptic device, an overhead camera and a group of wheeled robots. The commands of formation and average velocities of the multiple robots are generated by the operator through the haptic device. The state of the multiple robots and the working environment is sent to the human operator. The received information contains the feedback force through the haptic device and visual information returned by a depth camera. The feedback force based on the difference between the desired and actual average velocities is presented. The wave variable method is employed in the bilateral teleoperation of multiple mobile robots with time delays. The effectiveness of the bilateral teleoperation system is demonstrated by experiments. The robots in the slave side are able to follow the commands from the master side to interact with the environments, including moving in different formations and pushing a box. The results show that the scheme enables the operator to manipulate a group of robots to complete cooperative tasks freely.     

Effects of auditory,haptic and visual feedback on performing gestures by gaze or by hand

Modern interaction techniques like non-intrusive gestures provide means for interacting with distant displays and smart objects without touching them. We were interested in the effects of feedback modality (auditory, haptic or visual) and its combined effect with input modality on user performance and experience in such interactions. Therefore, we conducted two exploratory experiments where numbers were entered, either by gaze or hand, using gestures composed of four stroke elements (up, down, left and right). In Experiment 1, a simple feedback was given on each stroke during the motor action of gesturing: an audible click, a haptic tap or a visual flash. In Experiment 2, a semantic feedback was given on the final gesture: the executed number was spoken, coded by haptic taps or shown as text. With simultaneous simple feedback in Experiment 1, performance with hand input was slower but more accurate than with gaze input. With semantic feedback in Experiment 2, however, hand input was only slower. Effects of feedback modality were of minor importance; nevertheless, semantic haptic feedback in Experiment 2 showed to be useless at least without extensive training. Error patterns differed between both input modes, but again not dependent on feedback modality. Taken together, the results show that in designing gestural systems, choosing a feedback modality can be given a low priority; it can be chosen according to the task, context and user preferences.     

High-fidelity haptic synthesis of contact with deformable bodies

A method for synthesizing the haptic response of nonlinear deformable objects from data obtained by offline simulation helps create surgical simulators with high-fidelity haptic feedback. Haptic displays provide users with artificially created tactile sensations. One important use of these displays is to recreate the experience caused by contact between a tool and an object. This capability can be useful in several applications, such as surgical simulators, because users experience an enhanced sense of realism when a haptic simulation is combined with a graphic simulation. Haptic displays require two essential subsystems: a haptic device, which typically has a handle connected to sensors and actuators, and a computational system that interfaces with the device.     

Haptic teleoperation of a multirotor aerial robot using path planning with human intention estimation

Classical haptic teleoperation systems heavily rely on operators’ intelligence and efforts in aerial robot navigation tasks, thereby posing significantly users’ workloads. In this paper, a novel shared control scheme is presented facilitating a multirotor aerial robot haptic teleoperation system that exhibits autonomous navigation capability. A hidden Markov model filter is proposed to identify the intention state of operator based on human inputs from haptic master device, which is subsequently adopted to derive goal position for a heuristic sampling based local path planner. The human inputs are considered as commanded velocity for a trajectory servo controller to drive the robot along the planned path. In addition, vehicle velocity is perceived by the user via haptic feedback on master device to enhance situation awareness and navigation safety of the user. An experimental study was conducted in a simulated and a physical environment, and the results verify the effectiveness of the novel scheme in safe navigation of aerial robots. A user study was carried out between a classical haptic teleoperation system and the proposed approach in the identical simulated complex environment. The flight data and task load index (TLX) are acquired and analyzed. Compared with the conventional haptic teleoperation scheme, the proposed scheme exhibits superior performance in safe and fast navigation of the multirotor vehicle, and is also of low task and cognitive loads.

     

Effects of vision and friction on haptic perception

OBJECTIVE: Two experiments were conducted to examine the effects of vision and masking friction on contact perception and compliance differentiation thresholds in a simulated tissue-probing task. BACKGROUND: In minimally invasive surgery, the surgeon receives limited haptic feedback because of the current design of the instrumentation and relies on visual feedback to judge the amount of force applied to the tissues. It is suggested that friction forces inherent in the instruments contribute to errors in surgeons' haptic perception. This paper investigated the psychophysics of contact detection and cross-modal sensory processing in the context of minimally invasive surgery. METHOD: A within-subjects repeated measures design was used, with friction, vision, tissue softness, and order of presentation as independent factors, and applied force, detection time, error, and confidence as dependent measures. Eight participants took part in each experiment, with data recorded by a custom force-sensing system. RESULTS: In both detection and differentiation tasks, higher thresholds, longer detection times, and more errors were observed when vision was not available. The effect was more pronounced when haptic feedback was masked by friction forces in the surgical device (p < .05). CONCLUSION: Visual and haptic feedback were equally important for tissue compliance differentiation. APPLICATION: A frictionless endoscopic instrument can be designed to restore critical haptic information to surgeons without having to create haptic feedback artificially.