Haptic interaction with objects in a picture based on pose estimation

In pictures, every object is displayed in 2D space. Seeing the 2D image, people can perceptually reconstruct and understand information regarding the scene. To enable users to haptically interact with an object that appears in the image, the present study proposes a geometry-based haptic rendering method. More specifically, our approach is intended to estimate haptic information from the object’s structure contained in an image while preserving the two-dimensional visual information. Of the many types of objects that can be seen in everyday pictures, this paper mainly deals with polyhedron figures or objects composed of rectangular faces, some of which might be shown in a slanted configuration in the picture. To obtain the geometric layout of the object being viewed from the image plane, we first estimate homographic information that describes a mapping from the object coordinate to the target image coordinate. Then, we transform the surface normals of the object face using the extrinsic part of homography that locates the face of the object we are viewing. Because the transformed normals are utilized for calculating the force in the image space, we call this process normal vector perturbation in the 2D image space. To physically represent the estimated normal vector without distorting the visual information, we employed a lateral haptic rendering scheme in that it fits with our interaction styles on 2D images. The active force value at a given position on the slanted faces is calculated during the interaction phase. To evaluate our approach, we conducted an experiment with different stimulus conditions, in which it was found that participants could reliably estimate the geometric layout that appears in the picture. We conclude with explorations of applications and a discussion of future work.     

A haptic-rendering technique based on hybrid surface representation

A novel haptic rendering technique using a hybrid surface representation addresses conventional limitations in haptic displays. A haptic interface lets the user touch, explore, paint, and manipulate virtual 3D models in a natural way using a haptic display device. A haptic rendering algorithm must generate a force field to simulate the presence of these virtual objects and their surface properties (such as friction and texture), or to guide the user along a specific trajectory. We can roughly classify haptic rendering algorithms according to the surface representation they use: geometric haptic algorithms for surface data, and volumetric haptic algorithms based on volumetric data including implicit surface representation. Our algorithm is based on a hybrid surface representation - a combination of geometric (B-rep) and implicit (V-rep) surface representations for a given 3D object, which takes advantage of both surface representations.     

Hybrid client-server architecture and control techniques for collaborative product development using haptic interfaces

In this paper, a collaborative product development and prototyping framework is proposed by using distributed haptic interfaces along with deformable objects modeling. Collaborative Virtual Environment (CVE) is a promising technique for industrial product development and virtual prototyping. Network control problems such as network traffic and network delay in communication have greatly limited collaborative virtual environment applications. The problems become more difficult when high-update-rate haptic interfaces and computation intensive deformable objects modeling are integrated into CVEs for intuitive manipulation and enhanced realism. A hybrid network architecture is proposed to balance the computational burden of haptic rendering and deformable object simulation. Adaptive artificial time compensation is used to reduce the time discrepancy between the server and the client. Interpolation and extrapolation approaches are used to synchronize graphic and haptic data transmitted over the network. The proposed techniques can be used for collaborative product development, virtual assembly, remote product simulation and other collaborative virtual environments where both haptic interfaces and deformable object models are involved.     

Out-of-core real-time haptic interaction on very large models

In this paper we address the problem of fast inclusion tests and distance calculation in very large models, an important issue in the context of environments involving haptic interaction or collision detection. Unfortunately, existing haptic rendering or collision detection toolkits cannot handle polygonal models obtained from 3D digitized point clouds unless the models are simplified up to a few thousand polygons, which leads to an important lack of detail for the scanned pieces. We propose a data structure that is able to manage very large polygonal models (over 25M polygons), and we explain how this can be used in order to compute the inclusion of a point into the solid surface very efficiently, performing several thousand point-in-solid tests per second. Our method uses a data structure called EBP-Octree (Extended Bounding-Planes Octree), which is a very tight hierarchy of convex bounding volumes. Based on a spatial decomposition of the model using an octree, at each node it defines a bounding volume using a subset of the planes of the portion of the polygonal model contained at that node. We use the EBP-Octree in a haptic interaction environment, where distance tests and the orientation of collided triangles must be accurate and fast. We also demonstrate that the proposed algorithm largely meets the interactive query rate demanded by a haptic interaction (1 kHz), despite being executed in a single CPU thread on a commonly available computer.     

An information-theoretic treatment of passive haptic media

Haptic rendering has been long considered as the process of estimating the force that stems from the interaction of a user and an object. Even if this approach follows the principles of natural haptic interaction, it places severe limitations in processing haptic media. This paper presents an information theoretic framework that aims to provide a new view of haptic rendering that can accommodate for open-loop synthetic haptic media, where interaction-based rendering is a special case. As a result, using the proposed information-theoretic approach, the haptic signal can be precomputed as a force field, stored and then filtered by taking into account device and perceptual capabilities of the receiver in order to lower the required bandwidth of the resulting stream, thus opening new possibilities for the representation and processing of haptic media.     

ROSy+: 3D Object Pose Normalization Based on PCA and Reflective Object Symmetry with Application in 3D Object Retrieval

A novel pose normalization method based on 3D object reflective symmetry is presented. It is a general purpose global pose normalization method; in this paper it is used to enhance the performance of a 3D object retrieval pipeline. Initially, the axis-aligned minimum bounding box of a rigid 3D object is modified by requiring that the 3D object is also in minimum angular difference with respect to the normals to the faces of its bounding box. To estimate the modified axis-aligned bounding box, a set of predefined planes of symmetry are used and a combined spatial and angular distance, between the 3D object and its symmetric object, is calculated. By minimizing the combined distance, the 3D object fits inside its modified axis-aligned bounding box and alignment with the coordinate system is achieved. The proposed method is incorporated in a hybrid scheme, that serves as the alignment method in a 3D object retrieval system. The effectiveness of the 3D object retrieval system, using the hybrid pose normalization scheme, is evaluated in terms of retrieval accuracy and demonstrated using both quantitative and qualitative measures via an extensive consistent evaluation on standard benchmarks. The results clearly show performance boost against current approaches.     

Object interaction and task programming by demonstration in visuo-haptic augmented reality

A visuo-haptic augmented reality system is presented for object manipulation and task learning from human demonstration. The proposed system consists of a desktop augmented reality setup where users operate a haptic device for object interaction. Users of the haptic device are not co-located with the environment where real objects are present. A three degrees of freedom haptic device, providing force feedback, is adopted for object interaction by pushing, selection, translation and rotation. The system also supports physics-based animation of rigid bodies. Virtual objects are simulated in a physically plausible manner and seem to coexist with real objects in the augmented reality space. Algorithms for calibration, object recognition, registration and haptic rendering have been developed. Automatic model-based object recognition and registration are performed from 3D range data acquired by a moving laser scanner mounted on a robot arm. Several experiments have been performed to evaluate the augmented reality system in both single-user and collaborative tasks. Moreover, the potential of the system for programming robot manipulation tasks by demonstration is investigated. Experiments show that a precedence graph, encoding the sequential structure of the task, can be successfully extracted from multiple user demonstrations and that the learned task can be executed by a robot system.     

A new desk-top encounter-type haptic device with an actively driven pen-tablet LCD panel

In conventional haptic devices for virtual reality (VR) systems, a user interacts with a scene by handling a tool (such as a pen) using a mechanical device (i.e. an end-effector-type haptic device). In the case that the device can ‘mimic’ a VR object, the user can interact directly with the VR object without the mechanical constraint of a device (i.e. an encounter-type haptic device). A new challenge of an encounter-type haptic device is displaying the visuals and haptic information simultaneously on a single device. We are proposing a new desk-top encounter-type haptic device with an actively driven pen-tablet LCD panel. The proposed device is capable of providing pseudo-3D visuals and haptic information on a single device. As the result, the system provides to the user a sense of interaction with a real object. To develop a proof-of-concept prototype, a compact parallel mechanism was developed and implemented. The aim of this research is to propose a new concept in haptic research. In this paper, the concept, the prototype, and some preliminary evaluation tests with the proposed system are presented.     

基于相似性学习的三维模型最优视图选择算法

针对已有最优视图度量难以适用于不同类型的三维模型,提出基于用户知识、在模型库中为各类模型建立最优视图样例,并在此基础上进行相似性学习,根据相似性度量获得输入模型最优视图的选择算法.首先采用AdaBoost算法对输入三维模型形状特征进行相似性学习,得到该模型的最优视图样例;然后将输入模型从不同视点得到的渲染视图和最优视图样例进行形状相似性分析,以相似度最高者作为输入模型的最优视图.实验结果表明,采用文中算法得到的最优视图不仅可以有效地逼近用户选择结果,而且具有较好的稳定性.     

Real-time area-based haptic rendering and the augmented tactile display device for a palpation simulator

Although people usually contact a surface with some area rather than a point, most haptic devices allow a user to interact with a virtual object at one point at a time and likewise most haptic rendering algorithms deal with such situations only. In a palpation procedure, medical doctors push and rub the organ's surface, and are provided the sensation of distributed pressure and contact force (reflecting force) for discerning doubtable areas of the organ. In this paper, we suggest real-time area-based haptic rendering to describe distributed pressure and contact force simultaneously, and present a haptic interface system to generate surface properties in accordance with the haptic rendering algorithm. We represent the haptic model using the shape-retaining chain link (S-chain) framework for a fast and stable computation of the contact force and distributed pressure from a volumetric virtual object. In addition, we developed a compact pin-array-type tactile display unit and attached it to the PHANToM^TM haptic device to complement each other. For the evaluation, experiments were conducted with non-homogenous volumetric cubic objects consisting of approximately 500 000 volume elements. The experimental results show that compared to the point contact, the area contact provides the user with more precise perception of the shape and softness of the object's composition, and that our proposed system satisfies the real-time and realism constraints to be useful for a virtual reality application.     

Priority-based haptic event filtering for transmission and error control in networked virtual environments

A data filtering scheme is proposed for transmission and error control of haptic events in haptic-based network virtual environments; this scheme is called as priority-based haptic event filtering. Because a high update rate of approximately 1 kHz is required for haptic rendering, sophisticated transmission rate control and reduction schemes are necessary for the haptic events. Although existing schemes can reduce the transmission rate without any perception impairment, they are very sensitive to packet losses. In this paper, we prioritize the haptic events according to the delay and loss effects. Utilizing the proposed haptic event prioritization, the proposed filtering scheme adapts the transmission rate and updates the predicted loss rate according to the current network state. Our simulation and experiment results confirm that the proposed scheme can effectively select important haptic events and guarantee an improved haptic interaction quality over a bandwidth-limited lossy network than existing transmission schemes tailored for networked haptics.     

From measured physical parameters to the haptic feeling of fabric

Abstract real-time cloth simulation involves the solution of many computational challenges, particularly in the context of haptic applications, where high frame rates are necessary for obtaining a satisfactory tactile experience. In this paper, we present a real-time cloth simulation system that offers a compromise between a realistic physically-based simulation of fabrics and a haptic application with high requirements in terms of computation speed. We place emphasis on architecture and algorithmic choices for obtaining the best compromise in the context of haptic applications. A first implementation using a haptic device demonstrates the features of the proposed system and leads to the development of new approaches for haptic rendering using the proposed approach.     

Effects of using a force feedback haptic augmented simulation on the attitudes of the gifted students towards studying chemical bonds in virtual reality environment

The aim of this study is to identify the effects of force feedback haptic applications developed in virtual reality environments (VREs), which is an important field of study in computer science and engineering, on gifted students’ attitudes towards chemistry education in learning process. A 3D 6 DOF (Degree of Freedom) haptik device (Phantom Omni?) has been used to develop the algorithm in this study. It can be used to transmit force and motion using a haptic device. Visual C++ was choosen as the software development environment. OpenGL? and Haptic Device Application Programming Interface have been used for rendering graphics. At the 3D image creation state Wrap 1200?, which is a kind of head-mounted display, has been chosen. The sample of this study consists of 52 students identified as gifted and are attending 6th and 7th grades at the Istanbul Science &; Art Center in Istanbul. The experimental group studied chemical bonds using an application developed by using a force feedback haptic device in VRE and the control group studied it by traditional teaching methods. The study reveals that there is a relation between using force feedback haptic applications which are developed in VREs and gifted students’ attitudes towards educational programs.     

虚拟柔性体实时形变仿真模型研究

为了在虚拟现实柔性体力触觉交互研究中得到稳定、连续、真实的力触感,提出一种基于球面调和函数表达的虚拟柔性体实时形变仿真模型,利用球面调和函数的正交归一、旋转不变、多尺度等特性实现物体的快速准确表达.在变形体的密度、杨氏模量、泊松比等参数已知的情况下,基于径向基函数神经网络模型预测柔性体受力形变后的SH模型.仿真结果表明,该方法不仅可以准确表达柔性体的实时形变,而且使得基于SH表达的柔性体形变的视觉刷新描述与柔性体反馈力的触觉刷新描述同步,从而满足虚拟手术仿真训练等虚拟柔性体力触觉交互研究要求.     

用于实时柔性触觉再现的平行菱形链连接模型

精度高且实时性好的柔性触觉变形模型是实现触觉再现系统的关键。提出了一种新的基于物理意义的平行菱形链连接触觉变形模型,系统中各个链结构单元相对位移的叠加对外等效为物体表面的变形,与之相连的弹簧弹性力的合力等效为物体表面的接触力。使用Delta 6-DOF手控器,建立了触觉再现实验系统,对柔性体的接触变形和实时虚拟触觉反馈进行仿真, 实验结果表明所提出的模型不仅计算简单,而且能够保证触觉接触力和形变计算具有较高精度,满足虚拟现实系统对精细作业和实时性的要求。     

面向磁悬浮视触觉交互的多速率系统框架

非接触式磁悬浮视触觉交互克服了机械式交互的固有摩擦,具有广阔应用前景,但存在交互过程中虚拟工具穿透物体、图形渲染与触觉渲染速率不一致等问题。针对上述问题,提出面向磁悬浮视触觉交互的多速率系统框架,通过扩展三自由度（3-DOF）单射线触觉渲染方法,利用多射线对虚拟工具进行建模,避免工具穿透,实现六自由度（6-DOF）触觉渲染;通过多速率并行,实现速率不一致模块间相互协同;通过构建映射滤波算法,实现视觉定位数据到虚拟工具位姿的稳定映射。实验结果表明,该系统能有效避免交互过程中的穿透现象,并提供稳定、真实的视触觉反馈。     

Improving virtual reality navigation tasks using a haptic vest and upper body tracking

Haptic feedback is an important component of immersive virtual reality (VR) applications that is often suggested to complement visual information through the sense of touch. This paper investigates the use of a haptic vest in navigation tasks. The haptic vest produces a repulsive vibrotactile feedback from nearby static virtual obstacles that augments the user spatial awareness. The tasks require the user to perform complex movements in a 3D cluttered virtual environment, like avoiding obstacles while walking backwards and pulling a virtual object. The experimental setup consists of a room-scale environment. Our approach is the first study where a haptic vest is tracked in real time using a motion capture device so that proximity-based haptic feedback can be conveyed according to the actual movement of the upper body of the user.User study experiments have been conducted with and without haptic feedback in virtual environments involving both normal and limited visibility conditions. A quantitative evaluation was carried out by measuring task completion time and error (collision) rate. Multiple haptic rendering techniques have also been tested. Results show that under limited visibility conditions proximity-based haptic feedback generated by a wearable haptic vest can significantly reduce the number of collisions with obstacles in the virtual environment.     

Layered rhombus-chain-connected model for real-time haptic rendering

The modeling and simulation of deformable objects is a challenging topic in the field of haptic rendering between human and virtual environment. In this paper, a novel and efficient layered rhombus-chain-connected haptic deformation model based on physics is proposed for an excellent haptic rendering. During the modeling, the accumulation of relative displacements in each chain structure unit in each layer is equal to the deformation on the virtual object surface, and the resultant force of corresponding springs is equivalent to the external force. The layered rhombus-chain-connected model is convenient and fast to calculate, and can satisfy real-time requirement due to its simplicity. Experimental study in both homogenous and non-homogenous virtual human liver and lungs based on the proposed model are conducted, and the results demonstrate that our model provides stable and realistic haptic feeling in real time. Meanwhile, the display result is vivid.     

A social interaction system based on cloud computing for mobile video telephony

As the use of smartphones become popular,people heavily depend on smartphone applications to deal with their social activities.For this reason,traditional message texting between mobile applications does not fulfill the versatile requirements of social networking.Many mobile applications use multimodality to deliver multimedia messages including sticker,voice and photo message,video call,and snap movie to enhance the communicative capability.However,without face-to-face interaction,people may fail to detect the other side’s non-verbal social behavior such as fine-grain facial expressions,body movements,or hand gesture.During social interaction,non-verbal behavior conveys information about the involved individuals and help the speakers express their social emotion in an implicit way.It is so important for real-world face-to-face interaction but is often blocked on the mobile telephony.To cope with this problem,we propose an afective computing model to assist the representation of social emotion and then help the progress of social interaction on the mobile telephony.In this model,for the purpose of real-time afective analysis,we delegate the computing loading to the cloud side service and enhance the system’s scalability and availability.The result of this experiment approves the feasibility of our system design for the applications of social intelligent.Also,the system provides a research framework of the social intelligent system on the mobile telephony.     

Model-Based Object Recognition Using Geometric Invariants of Points and Lines

In this paper, we derive new geometric invariants for structured 3D points and lines from single image under projective transform, and we propose a novel model-based 3D object recognition algorithm using them. Based on the matrix representation of the transformation between space features (points and lines) and the corresponding projected image features, new geometric invariants are derived via the determinant ratio technique. First, an invariant for six points on two adjacent planes is derived, which is shown to be equivalent to Zhu's result [1], but in simpler formulation. Then, two new geometric invariants for structured lines are investigated: one for five lines on two adjacent planes and the other for six lines on four planes. By using the derived invariants, a novel 3D object recognition algorithm is developed, in which a hashing technique with thresholds and multiple invariants for a model are employed to overcome the over-invariant and false alarm problems. Simulation results on real images show that the derived invariants remain stable even in a noisy environment, and the proposed 3D object recognition algorithm is quite robust and accurate.