Surface exploration using laparoscopic surgical instruments: the perception of surface roughness

During laparoscopic surgery video images are used to guide the movements of the hand and instruments, and objects in the operating field often obscure these images. Thus, surgeons often rely heavily on tactile information (sense of touch) to help guide their movements. It is important to understand how tactile perception is affected when using laparoscopic instruments, since many surgical judgements are based on how a tissue 'feels' to the surgeon, particularly in situations where visual inputs are degraded. Twelve na?ve participants used either their index finger or a laparoscopic instrument to explore sandpaper surfaces of various grits (60, 100, 150 and 220). These movements were generated with either vision or no vision. Participants were asked to estimate the roughness of the surfaces they explored. The normal and tangential forces of either the finger or instrument on the sandpaper surfaces were measured. Results showed that participants were able to judge the roughness of the sandpaper surfaces when using both the finger and the instrument. However, post hoc comparisons showed that perceptual judgements of surface texture were altered in the no vision condition compared to the vision condition. This was also the case when using the instrument, compared to the judgements provided when exploring with the finger. This highlights the importance of the completeness of the video images during laparoscopic surgery. More normal and tangential force was used when exploring the surfaces with the finger as opposed to the instrument. This was probably an attempt to increase the contact area of the fingertip to maximize tactile input. With the instrument, texture was probably sensed through vibrations of the instrument in the hand. Applications of the findings lie in the field of laparoscopic surgery simulation techniques and tactile perception.     

Visual field magnification and touch perception when exploring surfaces with the index finger and a rigid instrument

OBJECTIVE: The aim of this project was to compare texture discrimination when both touch and vision were perturbed. BACKGROUND: Texture discrimination is important in the workplace. How textures are identified with the finger and with instruments when vision is magnified with lenses or video cameras is unclear. METHOD: Sandpaper was explored with the index finger or a metal instrument (hemostat), using normal or magnified vision. The forces generated during exploration were measured, and participants rated surface roughness. RESULTS: With the finger, the perception of roughness was unaffected with magnification; with the instrument, magnified surfaces were perceived as rougher (p < .05). Forces during finger exploration were unaffected by magnification; forces with the instrument increased under magnification (p < .05). CONCLUSION: Visual characteristics of the working field can influence the exploration and perception of materials. With the finger, mechanoreceptors that directly detect textures are activated, and with the instrument, receptors sensitive to vibrations are stimulated. APPLICATION: The higher forces produced when using instruments under magnification could lead to material damage. Attenuated perception of texture when exploring with tools may lead to difficulty in accurate touch perception. This could create problems in industrial tasks such as grading wool or identifying surface imperfections on manufactured materials, as well as in clinical settings such as dentistry or surgery in which instruments are used during tissue identification.     

The influence of combined visual and tactile information on finger and eye movements during shape tracing.

Adaptation experiments in shape tracing were conducted to investigate finger and eye movements in various conditions of visual and tactile information. Maximum velocity, mean velocity, maximum acceleration and reacceleration point were calculated from finger movements. Number of eye fixations and lead time of eye fixation to finger position were calculated from eye movements. The results showed that for the finger movement the values of the indices studied were higher in the combined visual and tactile condition than in the visual only condition. The number of eye fixations decreased when subjects repeated the tracing and was more marked in the combined visual and tactile condition than in the visual only condition. The results suggest that finger movements become faster and use of vision is reduced when both visual and tactile information are given.     

The influence of combined visual and tactile information on finger and eye movements during shape tracing

Abstract

Adaptation experiments in shape tracing were conducted to investigate finger and eye movements in various conditions of visual and tactile information. Maximum velocity, mean velocity, maximum acceleration and reacceleration point were calculated from finger movements. Number of eye fixations and lead time of eye fixation to finger position were calculated from eye movements. The results showed that for the finger movement the values of the indices studied were higher in the combined visual and tactile condition than in the visual only condition. The number of eye fixations decreased when subjects repeated the tracing and was more marked in the combined visual and tactile condition than in the visual only condition. The results suggest that finger movements become faster and use of vision is reduced when both visual and tactile information are given.     

Factors Involved in Tactile Texture Perception Through Probes

An understanding of texture perception by robotic systems can be gained by examining human texture perception through a probe. As with texture perception in direct touch with the finger, texture perception by indirect touch by means of a probe is multidimensional, comprising rough, hard and sticky texture continua. In this study we describe variability among individual subjects in probe-mediated texture perception, and compare similarities and differences of texture perception between direct touch and indirect touch. The results show variability among subjects, as individual subjects may choose to rely on different texture dimensions, and examine the texture surface using different scanning velocities and contact forces. Despite the fact that the subjects use different scanning velocities and forces, they discriminate textures reliably and make subjective magnitude estimates consistently along a texture continuum when indirectly exploring texture surfaces with a probe. These data will be potentially useful in designing robotic sensory systems and understanding of sensory feedback, which is essential to teleoperations.     

图像局部纹理特性的静电力触觉渲染

目的 触摸触觉设备感知物体时,需要实现视觉-力触觉同步反馈,其中图像-力触觉反馈难点在于再现更真实的纹理触感的触觉渲染过程。本文提出了一种基于图像局部纹理特征的静电力触觉渲染模型,实现了更加清晰、触感真实的图像纹理的静电力触觉反馈。方法 首先,采用局部傅里叶变换方法强化局部纹理特征,提取傅里叶变换系数分离出表征形状和局部纹理、边缘的频域分量。其次,对局部纹理特征进行力触觉渲染,建立局部纹理特征与驱动信号的映射模型,采用比例模型将局部纹理特征值转化为同等级的静电力表达。最后,根据静电力与驱动信号的心理学模型,由局部纹理特征控制不同驱动信号的输出产生静电力触觉。结果 进行纹理触觉对比感知实验验证算法有效性,62.5%的实验参与者偏爱基于图像局部纹理的触觉渲染算法反馈的纹理触感,本文算法可以模拟多种图像的纹理、边缘的触感。结论 算法在频域分离图像局部纹理、边缘和形状特征,建立纹理-力触觉渲染模型,针对大多数图片可以有效地增强纹理触感,提升触觉再现交互技术的沉浸感。     

振动触觉纹理再现技术研究综述

纹理力触觉再现是通过特定的硬件装置模拟产生与物体纹理表面接触时的触感，使用户能感受到物体的粗糙度、软硬度等纹理特征信息。振动刺激作为再现物体触觉信息的一种刺激方式，在纹理触觉再现中被广泛运用，产生了不同的振动触觉表达装置和纹理触觉表达方法。从纹理触觉认知的角度，阐述了人对振动刺激的触觉感知生理学基础；介绍了纹理触觉再现的原理和方法；从振动与纹理特征的映射方法以及振动刺激方式两个方面分析了目前振动触觉纹理再现技术的发展现状；最后对相关研究的发展进行了总结展望。     

面向视频感知的静电力触觉渲染方法

针对视觉障碍的人获取视频等数字媒体信息受限的问题, 为扩展视频等数字媒体信息的触觉感知通道,提出一种面向视频感知的静电力触觉渲染方法。首先,采用基于像素点的视频帧处理算法,根据手指触摸位置获取当前视频帧的目标像素点,然后将目标像素点彩色信息从RGB模型转换为HSI模型,利用像素点色调分量来映射静电力激励信号频率参量,结合像素点亮度和饱和度分量来映射静电力激励信号幅度参量,合成静电力触觉激励信号,实现对实时视频的触觉渲染和感知。最后,设计动态色彩感知实验和亮度辨识感知实验,结果表明,该方法可实现对视频中物体信息的触觉感知,动态识别平均正确率达90.6%,色彩辨识平均正确率达69.4%,亮度辨识平均正确率达80.0%,所提方法能有效提取视频中的动态特征信息,增强视频触觉渲染的实时性。     

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70–0.95 (P < 0.05). A t-test was used for the statistical analysis.     

Remote Tactile Transmission with Time Delay for Robotic Master–Slave Systems

This study develops a method to compensate for the communication time delay for tactile transmission systems. For transmitting tactile information from remote sites, the communication time delay degrades the validity of feedback. However, so far time delay compensation methods for tactile transmissions have yet to be proposed. For visual or force feedback systems, local models of remote environments were adopted for compensating the communication delay. The local models cancel the perceived time delay in sensory feedback signals by synchronizing them with the users' operating movements. The objectives of this study are to extend the idea of the local model to tactile feedback systems and develop a system that delivers tactile roughness of textures from remote environments to the users of the system. The local model for tactile roughness is designed to reproduce the characteristic cutaneous deformations, including vibratory frequencies and amplitudes, similar to those that occur when a human finger scans rough textures. Physical properties in the local model are updated in real-time by a tactile sensor installed on the slave-side robot. Experiments to deliver the perceived roughness of textures were performed using the developed system. The results showed that the developed system can deliver the perceived roughness of textures. When the communication time delay was simulated, it was confirmed that the developed system eliminated the time delay perceived by the operators. This study concludes that the developed local model is effective for remote tactile transmissions.     

基于纹理分析的表面粗糙度等级识别

提出了一种利用图象纹理分析技术进行机械加工表面粗糙度检测的非接触检测方法,该方法首先根据统计方差对待测工件的表面粗糙度进行粗分类,然后,利用基于Ｇａｂｏｒ滤波器的纹理分类器,识别待测工件表面粗糙度等级。该新方法可简单、快速地实现表面粗糙度等级的自动识别,而且对图象旋转具有不变性,由于其纹理分类器的参数少,并且新方法成本低,参数标定方便,因而便于现场检测,如果与机床的控制系统相连,还可以实现加工的实     

Amplifying shear deformation of finger pad increases tracing distances

Human sensory inputs and motor outputs mutually affect one another. We pursue the idea that a tactile interface can influence human motor outputs by intervening in sensory–motor relationships. This study focuses on the shear deformation of a finger pad while a person traces a line or circle. During these tracing movements, the finger pads were deformed using a tactile interface. The tracing distances increased when the finger pad deformations were amplified by the tactile interface, which indicates that the intervention in the haptic sensorimotor loop affected the tracing movements. Elucidation of such interaction between the tracing movements and the shear deformations of finger pads enhances the understanding of human-assistive haptic techniques.     

A comparison of forearm and thumb muscle electromyographic responses to the use of laparoscopic instruments with either a finger grasp or a palm grasp

Laparoscopic techniques allow for less-invasive treatment of common surgical problems. Laparoscopic instruments are different from standard surgical instruments and generally incorporate a pistol-grip handle configuration with rings for the fingers. This handle configuration has been reported as being uncomfortable, leading to finger compression neuropathies in some cases. As an alternative, the surgeon can choose to grasp laparoscopic instruments using a more powerful palm grip during grasping motions. This study evaluates the hypothesis that the use of the palm grip requires less muscle tension than the finger-grip when grasping with laparoscopic instruments. Nine general surgeons used an Autosuture laparoscopic grasper with a ringed pistol-grip handle held in both a finger-in-ring (F) or palm (P) hand grip position to grasp and close two spring-loaded metal plates. The same task was performed with a surgical haemostat clamp (H) for comparison. Each subject performed the grasping task in a random sequence for the three instrument configurations at two grasping forces levels (0.7 and 4.2 N), and with the instrument at three angles to the subjects' sagittal plane (0 degree, 45 degrees and 90 degrees). Surface electromyographic (EMG) signals were acquired from the flexor carpi ulnaris (FCU), flexor digitorum profundus (FDP), flexor digitorum superficialis (FDS), extensor carpi ulnaris (ECU), extensor digitorum comunis (EDC) and the thenar compartment (TH). The peak root mean squared (RMS) EMG voltage was averaged for five repetitions at each instrument, force and angle condition. Statistical analysis was carried out by repeated measures ANOVA. The muscle EMG RMS amplitude while using the palm grip was decreased in the FDS, TH and EDC, was unchanged in the ECU and FCU, and was slightly higher in the FDP when compared with the finger grip. These differences were most prominent at 90 degrees to the sagittal plane where the subjects' wrists neared maximal flexion. It is concluded that the palm grip is more powerful than the finger grip when grasping with laparoscopic instruments, particularly at angles perpendicular to the surgeon's sagittal plane.     

Discriminability-based evaluation of transmission capability of tactile transmission systems

Tactile transmission systems deliver tactile information such as texture roughness to operators of robotic systems. Such systems are typically composed of tactile sensors that sense the physical characteristics of textures and tactile displays that present tactile stimuli to operators. One problem associated with tactile transmission systems is that when the system has a bottleneck, it is difficult to identify whether the tactile sensor, tactile display, or perceptual ability of the user is the cause because they have different performance criteria. To solve this problem, this study established an evaluation method that uses the discriminability index as an evaluation criterion. The method lets tactile sensors, displays, and human tactile perception be assessed in terms of the ability to transmit physical quantities; the same criterion is used for all three possible causes so that their abilities can be directly compared. The developed method was applied to a tactile-roughness transmission system (Okamoto et?al. 2009), and its tactile sensor was identified as the bottleneck of the system.     

Control of laparoscopic instrument motion in an inanimate bench model: implications for the training and the evaluation of technical skills

Computer-assisted analysis of wrist movement has recently emerged as an objective laparoscopic performance evaluation method. The first purpose of this study was to assess the differences in motion characteristics between the tip of the instrument and the wrist. The second purpose was to describe the control strategies used to move laparoscopic instruments. During a bead transfer task, motions of a laparoscopic needle driver's tip, heel, and the participants' wrist were monitored. Results showed that large amplitude movements were best described by movements of the wrist, and small amplitude movements were evidenced by motions of the instrument tip. Thus, for describing expertise, and for evaluation and feedback, motion of the tip of the laparoscopic instrument should be quantified, in addition to motion of the wrist. The motions of the instrument were controlled by utilizing the flexibility of the skin of the laparoscopic trainer in addition to using the fulcrum, and sliding through the trocar. In order to increase fidelity, virtual reality trainers should simulate the flexibility of the real structures around the insertion of the instrument.     

Vibratory tactile display of image-based textures

Virtual worlds must integrate tactile and force sensations to create a sense of presence. Our tactile display conveys the textural sensation of object surfaces to a user's fingertip. This vibratory tactile display contains a contact pin array that transmits vibrations to the human skin. This type of device has been investigated as a reading aid for the blind since the 1960s. A device called the Optacon, developed by J.G. Linvill and J.C. Bliss (1966) was intended for use by a visually impaired person. Therefore, methods for representing texture have not been discussed extensively for that device. We present techniques for virtually replicating surface texture sensations through a vibratory tactile display, using image data of objects. M. Minsky et al. (1990) demonstrated a method for presenting surface textures by force display, including-if needed-visual images of textures. Their system generated the images from geometrical shapes of depth maps. In contrast, we use pictures of real object surfaces to provide the distribution data for tactile stimulus     

An ontology based framework for mining dependence relationships between news and financial instruments

Before news is input into financial trading algorithms/models, it needs human judgements for exploring the market implications of news content, distinguishing significance extent of news, and finding out the impact of polar type of each kind of news on certain financial instrument trading activity. But Dawes and Faust (1989) reported that people usually rely on clinical judgements, especially it is hard for them to distinguish valid decision variables from invalid ones in decision making. Thus, in order to alleviate this problem and provide more objective decision making support about news in financial market, an ontology based framework is proposed, for investigating the actuarial dependence relationships between news and financial instruments trading activities as well as identifying more valid news for trading decision making. This framework is expected to help people in financial market how to consider weight for each kind of news when inputted in trading algorithms/models of certain financial instruments.     

Autonomous tactile perception: A combined improved sensing and Bayesian nonparametric approach

In recent years, autonomous robots have increasingly been deployed in unknown environments and required to manipulate or categorize unknown objects. In order to cope with these unfamiliar situations, improvements must be made both in sensing technologies and in the capability to autonomously train perception models. In this paper, we explore this problem in the context of tactile surface identification and categorization. Using a highly-discriminant tactile probe based upon large bandwidth, triple axis accelerometer that is sensitive to surface texture and material properties, we demonstrate that unsupervised learning for surface identification with this tactile probe is feasible. To this end, we derived a Bayesian nonparametric approach based on Pitman–Yor processes to model power-law distributions, an extension of our previous work using Dirichlet processes Dallaire et al. (2011). When tested against a large collection of surfaces and without providing the actual number of surfaces, the tactile probe combined with our proposed approach demonstrated near-perfect recognition in many cases and achieved perfect recognition given the right conditions. We consider that our combined improvements demonstrate the feasibility of effective autonomous tactile perception systems.     

An optimal motion planning method for computer-assisted surgical training

This paper focuses on the development and validation of an optimal motion planning method for computer-assisted surgical training. The context of this work is the development of new-generation systems that combine artificial intelligence and computer vision techniques in order to adjust the learning process to specific needs of a trainee, while preventing a trainee from the memorization of particular task settings. The problem described in the paper is the generation of shortest, collision-free trajectories for laparoscopic instrument movements in the rigid block world used for hand–eye coordination tasks. Optimal trajectories are displayed on a monitor to provide continuous visual guidance for optimal navigation of instruments. The key result of the work is a framework for the transition from surgical training systems in which users are dependent on predefined task settings and lack guidance for optimal navigation of laparoscopic instruments, to the so called intelligent systems that can potentially deliver the utmost flexibility to the learning process. A preliminary empirical evaluation of the developed optimal motion planning method has demonstrated the increase of total scores measured by total time taken to complete the task, and the instrument movement economy ratio. Experimentation with different task settings and the technical enhancement of the visual guidance are subjects of future research.     

Effects of vision and friction on haptic perception

OBJECTIVE: Two experiments were conducted to examine the effects of vision and masking friction on contact perception and compliance differentiation thresholds in a simulated tissue-probing task. BACKGROUND: In minimally invasive surgery, the surgeon receives limited haptic feedback because of the current design of the instrumentation and relies on visual feedback to judge the amount of force applied to the tissues. It is suggested that friction forces inherent in the instruments contribute to errors in surgeons' haptic perception. This paper investigated the psychophysics of contact detection and cross-modal sensory processing in the context of minimally invasive surgery. METHOD: A within-subjects repeated measures design was used, with friction, vision, tissue softness, and order of presentation as independent factors, and applied force, detection time, error, and confidence as dependent measures. Eight participants took part in each experiment, with data recorded by a custom force-sensing system. RESULTS: In both detection and differentiation tasks, higher thresholds, longer detection times, and more errors were observed when vision was not available. The effect was more pronounced when haptic feedback was masked by friction forces in the surgical device (p < .05). CONCLUSION: Visual and haptic feedback were equally important for tissue compliance differentiation. APPLICATION: A frictionless endoscopic instrument can be designed to restore critical haptic information to surgeons without having to create haptic feedback artificially.