Energy balance method for modelling of soft tissue deformation

This paper presents a novel methodology for modelling of soft tissue deformation, from the standpoint of work–energy balance based on the law of conservation of energy. The work done by an external force is always balanced against the strain energy due to the internal force of the object. A position-based incremental approach is established, in which the work–energy balance is achieved via an iterative position increment process for the new equilibrium state of the object. The position-based incremental approach is further combined with non-rigid mechanics of motion to govern the dynamics of soft tissue deformation. The proposed method employs nonlinear geometric and material formulations to account for the nonlinear soft tissue deformation. Soft tissue material properties can be accommodated by specifying strain energy density functions. Integration with a haptic device is also achieved for soft tissue deformation with haptic feedback for surgical simulation. Experimental results demonstrate that the deformations by the proposed method are in good agreement with those by a commercial package of finite element analysis. Isotropic and anisotropic deformations, as well as soft tissue viscoelastic behaviours, can be accommodated by the proposed methodology via strain energy density functions.     

A novel model for assessing sliding mechanics and tactile sensation of human-like fingertips during slip action

This paper presents a model for displaying friction and localized stick/slip of sliding inhomogeneous human-like fingertips, to understand how slippage occurs and its role in assessing tactile sensing mechanics. In the absence of friction, the fingertip slides, as on an ice surface, in the virtual world of haptic interfaces. Slippage of fingertip at very low velocity can reflect micro stick/slip on a contact area, which is challenging to represent in any friction model. To overcome these drawbacks, we propose that a Beam Bundle Model (BBM) can be used to model a human fingertip during pushing and sliding actions, especially during stick-to-slip transition. To construct its three-dimensional, non-homogeneous structure, we obtained a sequential series of magnetic resonance images, showing consecutive cross-sectional layers of a fingertip with distribution of skin, tissue, bone, and nail. Simulation results showed that this model could generate not only normal force distribution caused by pushing, but also response of friction during stick-to-slip transition. Secondly, and more interestingly, the model dynamically produced localized displacement phenomena on the contact area during stick-to-slip phase, indicating how slippage enlarges the contact area prior to total slippage of the fingertip. These findings may better assess the sliding processes of human fingertips, and how and when slippage occurs on the contact surface. This model may be a useful platform for studying tactile perception of fingertips.     

Real-Time Finite-Element Simulation of Linear Viscoelastic Tissue Behavior Based on Experimental Data

We propose an end-to-end solution to real-time and realistic finite-element modeling and simulation of viscoelastic soft tissue behavior. We provide an efficient numerical scheme for solving a linear viscoelastic FEM model derived from the generalized Maxwell solid, and present methods for measuring and integrating experimental data on the viscoelastic material properties of soft tissues into the model for realistic display of visual deformations and interaction forces. Our precomputation scheme and multilayer computational architecture enable the model's real-time execution with visual and haptic feedback to the user. Our approach includes time- and rate-dependent effects, which requires considering a node's loading history in our displacement computations at each cycle of the simulation     

Development of a real-time visual and force environment for a haptic medical training simulator

In this article, a real-time, visual and force environment for a 5-dof haptic urological training simulator is presented that deals with a low-force, high-deformation environment. A real-time graphical representation of the male urethra during the insertion of an endoscope is developed. Smooth urethra deformations are produced by a mesh of piece-wise Bézier interpolations, while its inner wall is simulated by realistic tissue textures. Efficient real-time techniques are developed that introduce endoscope camera depth-of-field effects. A novel particle-based model computes in real-time the forces fed to the haptic device. A 13 fps refresh rate is achieved on a 2-GHz computer with the depth-of-field effect activated, while the rate is doubled to 26 fps with this feature disabled. It is expected that the simulator will contribute to ethical, efficient, and modern surgical training.     

Friction model of fingertip sliding over wavy surface for friction-variable tactile feedback panel

A friction-variable touch panel is capable of presenting virtual bumps and holes on its flat surface through the control of the surface friction when a fingertip slides over it. To improve the presentation, we developed a friction model of a fingertip sliding over a sinusoidal surface with an amplitude of 0.5–2.5 mm and a spatial wavelength of 20–50 mm. When a metal ball rolls over a wavy surface with a low friction and contact area, the ratio of the horizontal force to the normal force is equal to the gradient of the surface (this is referred to as the ball bearing model) and is hardly affected by the normal load and rolling speed. In contrast, the profile of the force ratio of a sliding finger is substantially skewed and affected by the sliding direction and normal force exerted by the finger. To model this skewed force ratio, we formulated the asymmetric pressure distribution in the finger-surface contact area and used the effects of the adhesion friction to model the dependency of the force ratio on the normal force and sliding direction. The developed model of a bare finger with these features was found to sufficiently simulate the experimentally observed force ratios. The model can be easily applied to friction-variable touch panels and enables the achievement of a wide variety of haptic contents with macroscopically concave or convex surfaces.     

Real-time adaptive control for haptic telemanipulation with Kalman active observers

This paper discusses robotic telemanipulation with Kalman active observers and online stiffness estimation. Operational space techniques, feedback linearization, discrete state space methods, augmented states, and stochastic design are used to control a robotic manipulator with a haptic device. Stiffness estimation only based on force data (measured, desired, and estimated forces) is proposed, avoiding explicit position information. Stability and robustness to stiffness errors are discussed, as well as real-time adaptation techniques. Telepresence is analyzed. Experiments show high performance in contact with soft and hard surfaces.     

Principal Geodesic Analysis in the Space of Discrete Shells

Important sources of shape variability, such as articulated motion of body models or soft tissue dynamics, are highly nonlinear and are usually superposed on top of rigid body motion which must be factored out. We propose a novel, nonlinear, rigid body motion invariant Principal Geodesic Analysis (PGA) that allows us to analyse this variability, compress large variations based on statistical shape analysis and fit a model to measurements. For given input shape data sets we show how to compute a low dimensional approximating submanifold on the space of discrete shells, making our approach a hybrid between a physical and statistical model. General discrete shells can be projected onto the submanifold and sparsely represented by a small set of coefficients. We demonstrate two specific applications: model‐constrained mesh editing and reconstruction of a dense animated mesh from sparse motion capture markers using the statistical knowledge as a prior.     

Soft tissue deformation with reaction-diffusion process for surgery simulation

This paper presents a new methodology to conduct modelling and analysis of soft tissue deformation from the physicochemical viewpoint of soft tissues for surgery simulation. The novelty of this methodology is that soft tissue deformation is converted into a reaction-diffusion process coupled with a mechanical load, and thus reaction-diffusion of mechanical load and non-rigid mechanics of motion are combined to govern the dynamics of soft tissue deformation. The mechanical load applied to a soft tissue to cause a deformation is incorporated into the reaction-diffusion system and consequently distributed among mass points of the soft tissue. An improved reaction-diffusion model is developed to describe the distribution of the mechanical load in the tissue. A generic finite difference scheme is presented for construction of the reaction-diffusion model on a 3D tissue surface. A gradient method is established for derivation of internal forces from the distribution of the mechanical load. Real-time interactive deformation of virtual human organs with haptic feedback has been achieved by the proposed methodology for surgery simulation. The proposed methodology not only accommodates isotropic, anisotropic and inhomogeneous materials by simply modifying diffusion coefficients, but also accepts local and large-range deformations simultaneously.     

生物软组织力反馈触觉建模测试系统研制

设计了一种结构新颖的用于生物软组织力反馈触觉建模的测试系统。阐述了系统结构与工作原理,介绍了系统的机械设计、硬件电路和系统软件。系统采用计算机测量并控制带有力和位置传感器的单自由度机械手,通过插针实验与动物软组织交互得到软组织的力反馈触觉模型。以猪肝为标本进行实验,实验结果和现有的理论模型较为符合,该系统可以用于软组织触觉建模实验研究。     

Real-time area-based haptic rendering and the augmented tactile display device for a palpation simulator

Although people usually contact a surface with some area rather than a point, most haptic devices allow a user to interact with a virtual object at one point at a time and likewise most haptic rendering algorithms deal with such situations only. In a palpation procedure, medical doctors push and rub the organ's surface, and are provided the sensation of distributed pressure and contact force (reflecting force) for discerning doubtable areas of the organ. In this paper, we suggest real-time area-based haptic rendering to describe distributed pressure and contact force simultaneously, and present a haptic interface system to generate surface properties in accordance with the haptic rendering algorithm. We represent the haptic model using the shape-retaining chain link (S-chain) framework for a fast and stable computation of the contact force and distributed pressure from a volumetric virtual object. In addition, we developed a compact pin-array-type tactile display unit and attached it to the PHANToM^TM haptic device to complement each other. For the evaluation, experiments were conducted with non-homogenous volumetric cubic objects consisting of approximately 500 000 volume elements. The experimental results show that compared to the point contact, the area contact provides the user with more precise perception of the shape and softness of the object's composition, and that our proposed system satisfies the real-time and realism constraints to be useful for a virtual reality application.     

Touch-enabled haptic modeling of deformable multi-resolution surfaces

Currently, interactive data exploration in virtual environments is mainly focused on vision-based and non-contact sensory channels such as visual/auditory displays. The lack of tactile sensation in virtual environments removes an important source of information to be delivered to the users. In this paper, we propose the touch-enabled haptic modeling of deformable multi-resolution surfaces in real time. The 6-DOF haptic manipulation is based on a dynamic model of Loop surfaces, where the dynamic parameters are computed easily without subdividing the control mesh recursively. A local deforming scheme is developed to approximate the solution of the dynamics equations, thus the order of the linear equations is reduced greatly. During each of the haptic interaction loop, the contact point is traced and reflected to the rendering of updated graphics and haptics. The sense of touch against the deforming surface is calculated according to the surface properties and the damping-spring force profile. Our haptic system supports the dynamic modeling of deformable Loop surfaces intuitively through the touch-enabled interactive manipulation.     

Proposal of an Index for the Operator’s Haptic Sensation in a Master-Slave System with Force-Feedback Function

This article proposes an index to estimate the operator’s haptic sensation of the contact between the slave device and the environment in operating master–slave systems with force feedback function. The index value is derived from the velocity information of the master device before and after contact, which is hypothesized to represent the intensity of haptic sensation stimuli presented to the operator. Two characteristics of this index are discussed by means of psychophysics experiment, which are the statistical characteristics of the index value for different operators, and how the change in the operator’s haptic sensation is reflected on the index value. The index is validated by another psychophysics experiment. The experimental results show that the performance of operator’s haptic sensation can be predicted correctly based on the proposed index value. This index is expected to be applied in the parameter design of bilateral-control systems with force feedback function.     

High-fidelity haptic synthesis of contact with deformable bodies

A method for synthesizing the haptic response of nonlinear deformable objects from data obtained by offline simulation helps create surgical simulators with high-fidelity haptic feedback. Haptic displays provide users with artificially created tactile sensations. One important use of these displays is to recreate the experience caused by contact between a tool and an object. This capability can be useful in several applications, such as surgical simulators, because users experience an enhanced sense of realism when a haptic simulation is combined with a graphic simulation. Haptic displays require two essential subsystems: a haptic device, which typically has a handle connected to sensors and actuators, and a computational system that interfaces with the device.     

Measurement-based modeling of contact forces and textures for haptic rendering

Haptic texture represents the fine-grained attributes of an object's surface and is related to physical characteristics such as roughness and stiffness. We introduce an interactive and mobile scanning system for the acquisition and synthesis of haptic textures that consists of a visually tracked handheld touch probe. The most novel aspect of our work is an estimation method for the contact stiffness of an object based solely on the acceleration and forces measured during stroking of its surface with the handheld probe. We establish an experimental relationship between the estimated stiffness and the contact stiffness observed during compression. We also measure the height-displacement profile of an object's surface enabling us to generate haptic textures. We show an example of mapping the textures on to a coarse surface mesh obtained with an image-based technique, but the textures may also be combined with coarse surface meshes obtained by manual modeling.     

Neural dynamics-based Poisson propagation for deformable modelling

This paper presents a new methodology from the standpoint of energy propagation for real-time and nonlinear modelling of deformable objects. It formulates the deformation process of a soft object as a process of energy propagation, in which the mechanical load applied to the object to cause deformation is viewed as the equivalent potential energy based on the law of conservation of energy and is further propagated among masses of the object based on the nonlinear Poisson propagation. Poisson propagation of mechanical load in conjunction with non-rigid mechanics of motion is developed to govern the dynamics of soft object deformation. Further, these two governing processes are modelled with cellular neural networks to achieve real-time computational performance. A prototype simulation system with a haptic device is implemented for real-time simulation of deformable objects with haptic feedback. Simulations, experiments as well as comparisons demonstrate that the proposed methodology exhibits nonlinear force–displacement relationship, capable of modelling large-range deformation. It can also accommodate homogeneous, anisotropic and heterogeneous materials by simply changing the constitutive coefficient value of mass points.

     

A force reflected exoskeleton-type masterarm for human-robot interaction

Two human-robot interactions, including a haptic interaction and a teleoperated interaction, are explored with a new exoskeleton-type masterarm, in which the electric brakes with the torque sensor beams are used for force reflection. In the haptic interaction with virtual environment, the masterarm is used as a haptic device and tested to examine how the resistant torque of the electric brake for the force reflection is implemented in contact regime prior to conducting the teleoperated interaction. Two types of virtual environments, a rigid wall with high stiffness (hard contact with 10 [KN/m]) and a soft wall with low stiffness (soft contact with 0.1 [N/m]), are integrated with the masterarm for the haptic interaction. In hard contact, large force is fed back to the human operator, and makes the human operator hardly move. The electric brake with the torque sensor beam can detect the torque and its direction so that it allows free motion as well as contact motion by releasing or holding the movement of the operator. The experimental results show how the electric brake is switched from contact to free regime to allow the operator to move freely, especially when the operator intends to move toward the free regime in contact. In soft contact, the force applied to the human operator can be increased or decreased proportionally to the torque amount sensed by the torque sensor beam, thus the operator can feel the contact force proportional to the amount of the deformation during the contact. Finally, the masterarm is integrated with the humanoid robot, CENTAUR developed at Korea Institute of Science and Technology to conduct a pick-and-place task through the teleoperated interaction. It is examined that the CENTAUR as a slave robot can follow the movement of the operator.     

A mesh adaptivity procedure for CFD and fluid-structure interactions

We present a procedure to adapt and repair meshes in the general solution of Navier–Stokes incompressible and compressible fluid flows, including structural interactions. For fluid-structure interactions, FSI, the fluid is described by an arbitrary-Lagrangian–Eulerian formulation fully coupled to general solids and structures described by Lagrangian formulations. The solids and structures can undergo highly nonlinear response due to large deformations, nonlinear material behavior, contact and temperature. We focus on the need to adapt the fluid mesh in pure CFD solutions when high gradients are present or boundary layer effects are important, and FSI solutions when large structural deformations take place. The procedure is a practical scheme to solve complex problems. We illustrate the proposed scheme in various example solutions.     

Modeling of the condyle elements within a biomechanical knee model

The development of a computational multibody knee model able to capture some of the fundamental properties of the human knee articulation is presented. This desideratum is reached by including the kinetics of the real knee articulation. The research question is whether an accurate modeling of the condyle contact in the knee will lead to reproduction of the complex combination of flexion/extension, abduction/adduction, and tibial rotation observed in the real knee. The model is composed by two anatomic segments, the tibia and the femur, whose characteristics are functions of the geometric and anatomic properties of the real bones. The biomechanical model characterization is developed under the framework of multibody systems methodologies using Cartesian coordinates. The type of approach used in the proposed knee model is the joint surface contact conditions between ellipsoids, representing the two femoral condyles, and points, representing the tibial plateau and the menisci. These elements are closely fitted to the actual knee geometry. This task is undertaken by considering a parameter optimization process to replicate experimental data published in the literature, namely that by Lafortune and his coworkers in 1992. Then kinematic data in the form of flexion/extension patterns are imposed on the model corresponding to the stance phase of the human gait. From the results obtained, by performing several computational simulations, it can be observed that the knee model approximates the average secondary motion patterns observed in the literature. Because the literature reports considerable inter-individual differences in the secondary motion patterns, the knee model presented here is also used to check whether it is possible to reproduce the observed differences with reasonable variations of bone shape parameters. This task is accomplished by a parameter study, in which the main variables that define the geometry of condyles are taken into account. It was observed that the data reveal a difference in secondary kinematics of the knee in flexion versus extension. The likely explanation for this fact is the elastic component of the secondary motions created by the combination of joint forces and soft tissue deformations. The proposed knee model is, therefore, used to investigate whether this observed behavior can be explained by reasonable elastic deformations of the points representing the menisci in the model.     

用于实时柔性触觉再现的平行菱形链连接模型

精度高且实时性好的柔性触觉变形模型是实现触觉再现系统的关键。提出了一种新的基于物理意义的平行菱形链连接触觉变形模型,系统中各个链结构单元相对位移的叠加对外等效为物体表面的变形,与之相连的弹簧弹性力的合力等效为物体表面的接触力。使用Delta 6-DOF手控器,建立了触觉再现实验系统,对柔性体的接触变形和实时虚拟触觉反馈进行仿真, 实验结果表明所提出的模型不仅计算简单,而且能够保证触觉接触力和形变计算具有较高精度,满足虚拟现实系统对精细作业和实时性的要求。     

A fingerprint pointing device utilizing the deformation of the fingertip during the incipient slip

In this paper, a novel pointing device is proposed that utilizes the deformation of the fingertip. When a fingertip is pressed and slightly slid on a rigid plate, a partial slip, called an "incipient slip", occurs on the contact surface. While the deformation around the center of the contact area is small during the incipient slip, the boundary region moves to the sliding direction of the fingertip. The deformation changes depending on the sliding distance of the fingertip and the exerted force on the contact surface. The velocity of the pointer can be determined by the estimated distance and force based on the measurement of the deformation. In this study, the correlation between the sliding distance of the fingertip and the deformation and between the exerted force and the deformation are investigated. The degree of the deformation due to the sliding motion can be estimated based on the detected fingerprint center. The group delay spectrum tracking method is proposed for the detection of the fingerprint center. A prototype pointing device is developed to evaluate the operationality of the proposed device. Comparative experiments with conventional pointing devices are conducted. The validity of the proposed device is confirmed by the experiments.