Force display using a hybrid haptic device composed of motors and brakes

In the virtual environment, force feedback to the human operator makes virtual experiences more realistic. However, the force feedback using active actuators such as motors can make the system active and sometimes unstable. To ensure the safe operation and enhance the haptic feeling, system stability should be guaranteed. Both active actuators such as motors and passive ones such as brakes are commonly used for haptic devices. Motors can generate a torque in any direction, but they can make the system active and thus, sometimes unstable during operation. On the other hand, brakes can generate a torque only against their rotation, but they dissipate energy during operation and this dissipation makes the system intrinsically stable. Consequently, motors and brakes are complementary to each other. In this research, a two degree-of-freedom (DOF) haptic device equipped with motors and brakes is designed, in which each DOF is actuated by a pair of motor and brake. Simultaneous operation of motors and brakes is analyzed. Models for some environments, virtual wall contact and frictional effect, are proposed. The results for the hybrid haptic system are compared with those for the active haptic system and the passivity based control system. The experimental results show that the hybrid haptic device is more suited to some applications than the other haptic systems.     

The virtual surgeon [virtual reality trainer]

Rapid improvements in computing power have opened the way for desktop virtual reality trainers that incorporate realistic graphics and, in some cases, the sense of touch. Affordable commercial simulators, for instance, are now available for practising such tasks as threading flexible endoscopes down a virtual patient's throat or manipulating the long surgical instruments used in laparoscopy. Companies and universities are also developing systems that simulate more complex procedures, such as suturing tissue and inserting a catheter into a vein using laparoscopic tools. These VR trainers can be adjusted to the user, to pinpoint areas of weakness, and they can be used at any time, without the need for supervision. What's more, they prepare the student psychologically for surgical tasks, because complications can be simulated in a safe manner. They can also give objective scores of a student's ability. Indeed, studies show that computer based training simulations are at least as good as standard training methods     

Wearable Soft Technologies for Haptic Sensing and Feedback

Virtual reality (VR) and augmented reality (AR) systems have garnered recent widespread attention due to increased accessibility, functionality, and affordability. These systems sense user inputs and typically provide haptic, audio, and visual feedback to blend interactive virtual environments with the real world for an enhanced or simulated reality experience. With applications ranging from immersive entertainment, to teleoperation, to physical therapy, further development of this technology has the potential for impact across multiple disciplines. However, VR/AR devices still face critical challenges that hinder integration into everyday life and additional applications; namely, the rigid and cumbersome form factor of current technology that is incompatible with the dynamic movements and pliable limbs of the human body. Recent advancements in the field of soft materials are uniquely suited to provide solutions to this challenge. Devices fabricated from flexible and elastic bio-compatible materials have significantly greater compatibility with the human body and could lead to a more natural VR/AR experience. This review reports state-of-the-art experimental studies in soft materials for wearable sensing and haptic feedback in VR/AR applications, explores emerging soft technologies for on-body devices, and identifies current challenges and future opportunities toward seamless integration of the virtual and physical world.     

A control-system architecture for robots used to simulate dynamicforce and moment interaction between humans and virtual objects

Haptic or kinesthetic feedback is essential in many important virtual reality and telepresence applications. Previous research focuses on simulating static forces such as those encountered when interacting with a stiff object such as a wall. Past studies usually employ custom-made devices that are not readily available to other researchers. Consequently, many of the results found in the haptic feedback literature cannot be replicated independently. With experimental results, the paper demonstrates that “off the shelf,” general purpose robotics equipment can be incorporated into an effective haptic/kinesthetic feedback system. Such a system can accommodate a wide variety of virtual reality applications including training and telerobotics. An admittance control scheme is utilized, which enables the simulation of dynamic force and moment interaction as well as contact with stiff objects. The paper shows that the mechanical deficiencies (e.g., friction, inertia, and backlash) often associated with general purpose manipulators can be overcome with a suitable control system architecture     

Force controlled and teleoperated endoscopic grasper for minimallyinvasive surgery-experimental performance evaluation

Minimally invasive surgery generates new user interfaces which create visual and haptic distortion when compared to traditional surgery. In order to regain the tactile and kinesthetic information that is lost, a computerized force feedback endoscopic surgical grasper (FREG) was developed with computer control and a haptic user interface. The system uses standard unmodified grasper shafts and tips. The FREG can control grasping forces either by surgeon teleoperation control, or under software control. The FREG performance was evaluated using an automated palpation function (programmed series of compressions) in which the grasper measures mechanical properties of the grasped materials. The material parameters obtained from measurements showed the ability of the FREG to discriminate between different types of normal soft tissues (small bowel, lung, spleen, liver, colon, and stomach) and different kinds of artificial soft tissue replication materials (latex/silicone) for simulation purposes. In addition, subjective tests of ranking stiffness of silicone materials using the FREG teleoperation mode showed significant improvement in the performance compared to the standard endoscopic grasper. Moreover, the FREG performance was closer to the performance of the human hand than the standard endoscopic grasper. The FREG as a tool incorporating the force feedback teleoperation technology may provide the basis for application in telesurgery, clinical endoscopic surgery, surgical training, and research.     

In vivo measurement of surgical gestures

Virtual reality techniques are now more and more widely used in the field of surgical training. However, the realism of the simulation devices requires a good knowledge of the mechanical behavior of the living organs. To provide perioperative measurement of laparoscopic surgical operations, we equipped a conventional operating grasper with a force sensor and a position sensor. The entire apparatus was connected to a PC that controlled the real-time data acquisition. After calibrating the sensors, we conducted three series of in vivo measurements on animals under video control. A standardized protocol was set up to perform various surgical gestures in a reproducible manner. Under these conditions, we can assess an original tool for a quantitative approach of surgical gestures' mechanics. The preliminary results will be extended by measurements during other operations and with other surgical instruments. The in vivo quantification of the mechanical interactions between operating instruments and anatomical structures is of great interest for the introduction of the force feedback in virtual surgery, for the modeling of the mechanical behavior of living organs, and for the design of new surgical instruments. This quantification of manipulations opens new prospects in the evaluation of surgical practices.     

Stereo‐Vision‐Based Human‐Computer Interaction with Tactile Stimulation

If a virtual object in a virtual environment represented by a stereo vision system could be touched by a user with some tactile feeling on his/her fingertip, the sense of reality would be heightened. To create a visual impression as if the user were directly pointing to a desired point on a virtual object with his/her own finger, we need to align virtual space coordinates and physical space coordinates. Also, if there is no tactile feeling when the user touches a virtual object, the virtual object would seem to be a ghost. Therefore, a haptic interface device is required to give some tactile sensation to the user. We have constructed such a human‐computer interaction system in the form of a simple virtual reality game using a stereo vision system, a vibro‐tactile device module, and two position/orientation sensors.     

Elastomeric Haptic Devices for Virtual and Augmented Reality

Since the modern concepts for virtual and augmented reality are first introduced in the 1960's, the field has strived to develop technologies for immersive user experience in a fully or partially virtual environment. Despite the great progress in visual and auditory technologies, haptics has seen much slower technological advances. The challenge is because skin has densely packed mechanoreceptors distributed over a very large area with complex topography; devising an apparatus as targeted as an audio speaker or television for the localized sensory input of an ear canal or iris is more difficult. Furthermore, the soft and sensitive nature of the skin makes it difficult to apply solid state electronic solutions that can address large areas without causing discomfort. The maturing field of soft robotics offers potential solutions toward this challenge. In this article, the definition and history of virtual (VR) and augmented reality (AR) is first reviewed. Then an overview of haptic output and input technologies is presented, opportunities for soft robotics are identified, and mechanisms of intrinsically soft actuators and sensors are introduced. Finally, soft haptic output and input devices are reviewed with categorization by device forms, and examples of soft haptic devices in VR/AR environments are presented.     

Pseudo-haptics for rigid tool/soft surface interaction feedback in virtual environments

This paper proposes a novel pseudo-haptic soft surface stiffness simulation technique achieved by displaying the deformation of the soft surface and maneuvering an indenter avatar over a virtual soft surface by means of a touch-sensitive tablet. The visual feedback of the surface deformation and the alterations to the indenter avatar behavior produced by the proposed technique create the illusion of interaction with a hard inclusion embedded in the virtual soft surface. The proposed pseudo-haptics technique is validated with a series of experiments conducted by employing a tablet computer with an S-pen input and a tablet computer with a bare finger input. Tablet computers provide unique opportunities for presenting the pseudo-haptic (indenter avatar speed), haptic (contact reaction force from the device surface) and visual cues (surface information) at the same active point of interaction which facilitates information fusion. Hence, here, we evaluate the performance of tablet computers in identification of hard inclusions within virtual soft objects and compare it with the performance of a touchpad input device. A direct hand-soft surface interaction is used for benchmarking of this study. We found that compared with using a touchpad, both the sensitivity and the positive predictive value of the hard inclusion detection can be significantly improved by 33.3% and 13.9%, respectively, by employing tablet computers. Using tablet computers could produce results comparable to the direct hand-soft surface interaction in detecting hard inclusions in a soft object. The experimental results presented here confirm the potential of the proposed technique for conveying haptic information in rigid tool/soft surface interaction in virtual environments.     

Haptic joystick with hybrid actuator using air muscles and spherical MR-brake

In this research, a new 2-DOF hybrid actuator concept is explored as a powerful and compact alternative to conventional haptic actuators. The actuator combines a spherical MR-brake and three air muscles and is integrated into a joystick that can apply forces in two degrees-of-freedom. The air muscles are used to create high active forces in a compact volume and the brake compensates for the “spongy” feeling associated with air muscles. To decrease the overall size of the system an inertial measurement unit has been implemented as a position measurement solution. As high as 16 N of total force output could be achieved at the tip of the joystick. Also, up to 16 times improvement in the stable virtual wall stiffness was obtained when the MR-brake was used to compensate for force errors. Experiments with an impedance-based haptic controller with force-feedback gave satisfactory wall following performance. This device can be employed in applications including computer games, military or medical training applications, rehabilitation and in teleoperation of equipment where high force feedback in 2-DOF in a compact work volume may be desirable while interacting with rigid or elastic virtual objects.     

A high bandwidth interface for haptic human computer interaction

Current force feedback, haptic interface devices are generally limited to the display of low frequency, high amplitude spatial data. A typical device consists of a low impedance framework of one or more degrees-of-freedom (dof), allowing a user to explore a pre-defined workspace via an end effector such as a handle, thimble, probe or stylus. The movement of the device is then constrained using high gain positional feedback, thus reducing the apparent dof of the device and conveying the illusion of hard contact to the user. Such devices are, however, limited to a narrow bandwidth of frequencies, typically below 30 Hz, and are not well suited to the display of surface properties, such as object texture. This paper details a device to augment an existing force feedback haptic display with a vibrotactile display, thus providing a means of conveying low amplitude, high frequency spatial information of object surface properties.     

Performance evaluation of a tactile and kinesthetic finger feedback system for teleoperation

Teleoperation during a catastrophic event requires an interface that can perform under frequently changing circumstances caused by unpredictable and dangerous conditions. Thus, teleoperation interfaces are under active development to provide both visual and haptic feedback to the fingers. However, studies of teleoperation systems with finger haptic feedback based on force profiles are difficult to conduct because of interface limitations. Therefore, in this paper, we introduce an intuitive teleoperation interface, an anthropomorphic teleoperated robot, and a hand-wearable force-feedback system that provides various feedbacks to the fingers. We combined these systems to compare and evaluated the performance of tactile and kinesthetic finger feedback using two experiments: maintaining appropriate grip force for variably fragile objects and following a force trajectory that changed in real time. Ten subjects participated in the experiments. The results were analyzed using repeated measures analysis of variance. Feedback factors differed significantly. Provision of force feedback to the user’s finger was most effective in both teleoperation experiments.     

Reality-based models for vibration feedback in virtual environments

Reality-based modeling of vibrations has been used to enhance the haptic display of virtual environments for impact events such as tapping, although the bandwidths of many haptic displays make it difficult to accurately replicate the measured vibrations. We propose modifying reality-based vibration parameters through a series of perceptual experiments with a haptic display. We created a vibration feedback model, a decaying sinusoidal waveform, by measuring the acceleration of the stylus of a three degree-of-freedom haptic display as a human user tapped it on several real materials. A series of perceptual experiments, where human users rated the realism of various parameter combinations, were performed to further enhance the realism of the vibration display for impact events. The results provided different parameters than those derived strictly from acceleration data. Additional experiments verified the effectiveness of these modified model parameters by showing that users could differentiate between materials in a virtual environment     

Kinematic and workspace-based dimensional optimization of a 2-DOF mechanism for a novel multipoint device

This paper presents the development of a novel haptic device that allows a user to interact with a computer using force feedback. The mechanism used in this work is expected to interact with a finger without the use of a fixture attached to the body, and by further including multiple identical mechanisms for each finger of the hand; it is possible to interact with virtual objects with gestures such as grabbing or pinching. This work investigates the theoretical workspace of a human index finger and compares it with an experimental workspace, which is obtained using a computer vision system, resulting in a modified theoretical model. Furthermore, the paper uses a two degree-of-freedom 7-bar linkage mechanism, whose dimensions are optimized according to the finger workspace. The paper obtains the mobility of the mechanism by using screw theory and analyzing the constraint screw system of the end-effector. Furthermore, the paper determines the closed-form solutions to the forward and inverse position, velocity and acceleration problems that are of interest in haptic simulations. In addition to this, the singular configurations are obtained by analyzing the Jacobian matrix and screw theory is further used to gain insight into the constraints applied to the end effector in these configurations. Theoretical workspaces are compared with experimental results, where a new finger workspace is proposed in order to account the difference between literature models and experiments. A prototype is built and tested as a proof of concept of the novel device, where the apparatus is comprised by a set of five 2-DOF mechanism; each mechanism sustains the proposed workspace but the dimensions of their components vary in order to fit all the end effectors within the fingers reach. The workspace of the constructed mechanism is compared with their theoretical models for assessing their effectiveness, and the feasibility of the use of accelerometers as position sensing instruments. The paper determines a closed-form solution to the applied force and performs an experiment to determine the force feedback that is experienced by the user. The torque produced by the motors of each mechanism is measured and the results extrapolated to cover the maximum current that can be applied to the motors.     

Haptic Glove With MR Brakes for Virtual Reality

Haptic gloves open up the world of force feedback by allowing the user to pick up and feel virtual objects in a natural way. In most of the existing gloves, a remote box houses a large number of actuators and sensors. Power to the glove is transmitted via cables. If the haptic gloves were smaller, lighter, and easier to use and control, they could become more common as human-machine interfaces. Recent developments show that actuators based on active fluids, such as the magnetorheological (MR) fluids, can be viable alternatives in haptics. But these devices are desk- or floor-mounted and use relatively large MR brakes. In this research, we developed a compact MR brake that is about 25 mm in diameter, weighs 84 g, and can apply up to 899 Nmiddotmm torque. The compact size was achieved by stacking steel and aluminum rings to create a serpentine flux path through the fluid. Six brakes were used to build a force feedback glove called MR glove. The glove weighs 640 g and does not require any remote actuators. Results of usability experiments showed that the MR glove improved task completion times in grasping virtual objects and could convey stiffness information to the user.     

Dynamic 3-D virtual fixtures for minimally invasive beating heart procedures

Two-dimensional or 3-D visual guidance is often used for minimally invasive cardiac surgery and diagnosis. This visual guidance suffers from several drawbacks such as limited field of view, loss of signal from time to time, and in some cases, difficulty of interpretation. These limitations become more evident in beating-heart procedures when the surgeon has to perform a surgical procedure in the presence of heart motion. In this paper, we propose dynamic 3-D virtual fixtures (DVFs) to augment the visual guidance system with haptic feedback, to provide the surgeon with more helpful guidance by constraining the surgeon's hand motions thereby protecting sensitive structures. DVFs can be generated from preoperative dynamic magnetic resonance (MR) or computed tomograph (CT) images and then mapped to the patient during surgery. We have validated the feasibility of the proposed method on several simulated surgical tasks using a volunteer's cardiac image dataset. Validation results show that the integration of visual and haptic guidance can permit a user to perform surgical tasks more easily and with reduced error rate. We believe this is the first work presented in the field of virtual fixtures that explicitly considers heart motion.     

The Rutgers Master II-new design force-feedback glove

The Rutgers Master II-ND glove is a haptic interface designed for dextrous interactions with virtual environments. The glove provides force feedback up to 16 N each to the thumb, index, middle, and ring fingertips. It uses custom pneumatic actuators arranged in a direct-drive configuration in the palm. Unlike commercial haptic gloves, the direct-drive actuators make unnecessary cables and pulleys, resulting in a more compact and lighter structure. The force-feedback structure also serves as position measuring exoskeleton, by integrating noncontact Hall-effect and infrared sensors. The glove is connected to a haptic-control interface that reads its sensors and servos its actuators. The interface has pneumatic servovalves, signal conditioning electronics, A/D/A boards, power supply and an imbedded Pentium PC. This distributed computing assures much faster control bandwidth than would otherwise be possible. Communication with the host PC is done over an RS232 line. Comparative data with the CyberGrasp commercial haptic glove is presented     

A Framework for the Design of a Novel Haptic-Based Medical Training Simulator

This paper presents a framework for the design of a haptic-based medical ultrasound training simulator. The proposed simulator is composed of a PHANToM haptic device and a modular software package that allows for visual feedback and kinesthetic interactions between an operator and multimodality image databases. The system provides real-time ultrasound images in the same fashion as a typical ultrasound machine, enhanced with corresponding augmented computerized tomographic (CT) and/or MRI images. The proposed training system allows trainees to develop radiology techniques and knowledge of the patient's anatomy with minimum practice on live patients, or in places or at times when radiology devices or patients with rare cases may not be available. Low-level details of the software structure that can be migrated to other similar medical simulators are described. A preliminary human factors study, conducted on the prototype of the developed simulator, demonstrates the potential usage of the system for clinical training.      

A General-Purpose 7 DOF Haptic Device: Applications Toward Robot-Assisted Surgery

A 7 DOF haptic device has been designed and developed with applications towards robot-assisted minimally invasive surgery. The device consists of four degrees of force feedback (X, Y, Z, and grasping) capability and seven degrees of position feedback capability. It has a closed kinematic chain with two halves (user interface and spatial mechanism) that connect together via a universal joint. The user interface contains four degrees of position feedback, namely, the roll, pitch, yaw, and linear motion of the hand and forearm. In addition, a grasping mechanism with two thimbles mounted at the end of the user interface provides force feedback to the fingers of the user. The spatial mechanism provides force feedback to the user interface through a universal joint located at the grasping mechanism. This paper presents the design and development of this haptic device. In addition, a kinematic and workspace analysis of the device has been completed to compute the position of the slave robot and end-effector tool. Friction estimation has been presented to enable a higher transparency of the haptic device. Finally, a simulation of needle insertion into soft tissue was developed to test the device.     

Constrained haptic-guided shared control for collaborative human–robot percutaneous nephrolithotomy training

Percutaneous nephrolithotomy is a procedure used to treat patients with large or irregularly shaped kidney stones. Surgical instruments are inserted through a small incision to access the kidney and remove the calculi. Surgeons who have less experience with the procedure manifest significantly higher rates of complications due to the procedure’s steep learning curve. This issue is further exacerbated by a lack of training opportunities in clinical settings. This paper introduces a teleoperative framework that can provide training to surgeons as well as assistance during procedures, based on two main components. Firstly, a type of constrained inverse kinematics that decouples the tooltip position from its orientation using a remote centre of motion, and incorporates the joint limits analytically. This reduces the workload of the procedure by having the surgeon control only the tooltip position rather than the position and the orientation while preventing the inverse kinematics from returning joint angles outside of the robot’s abilities. This kinematic framework also allows a three-degrees-of-freedom haptic device to control a six-degrees-of-freedom manipulator. Secondly, haptic feedback is provided to help guide and teach the surgeon during the procedure. Haptic feedback allows the surgeon to remain in full control during the procedure while still receiving haptic cues and assistance.Experimental results indicate that the haptic cues improved user’s accuracy, and they had shorter and smoother paths. This leads to a shorter procedure time overall. The results also indicate that the haptic assistance helped teach users the ideal trajectory of the procedure and that users who were taught with haptic feedback performed better than those who never experienced any haptic feedback.