Mesh cutting during real-time physical simulation

The ability to cut through meshes in real-time is an essential ingredient in a number of practical interactive simulations. Surgical simulation, cloth design, clay sculpting and many other related VR applications require the ability to introduce arbitrary discontinuities through models to separate, reposition, and reshape various pieces of the model as needed for the target application. In addition, in order to provide the necessary realism for these applications, model deformations must be computed from an underlying physically-based model—most commonly a continuum-based finite element model.In this work, we present a method for representing and computing, at interactive rates, the deformations of a mesh whose topology is being dynamically modified with multiple virtual tools. The method relies on introducing controlled discontinuities in the basis functions used to represent the geometry of deformation, and on fast incremental methods for updating global model deformations. The method can also generate the forces needed for force rendering in a haptic environment. The method is shown to scale well with problem size (linearly in the number of nonzeros of the Cholesky factor) allowing realistic interaction with fairly large models.     

Vision‐tangible interactive display method for mixed and virtual reality: Toward the human‐centered editable reality

Building a human‐centered editable world can be fully realized in a virtual environment. Both mixed reality (MR) and virtual reality (VR) are feasible solutions to support the attribute of edition. Based on the current development of MR and VR, we present the vision‐tangible interactive display method and its implementation in both MR and VR. We address the issue of MR and VR together because they are similar regarding the proposed method. The editable mixed and virtual reality system is useful for studies, which exploit it as a platform. In this paper, we construct a virtual reality environment based on the Oculus Rift, and an MR system based on a binocular optical see‐through head‐mounted display. In the MR system about manipulating the Rubik's cube, and the VR system about deforming the virtual objects, the proposed vision‐tangible interactive display method is utilized to provide users with a more immersive environment. Experimental results indicate that the vision‐tangible interactive display method can improve the user experience and can be a promising way to make the virtual environment better.     

一种基于三维建图和虚拟现实的人机交互系统

以提高人机共融水平为目的,以救援机器人为背景,提出并实现基于三维建图和虚拟现实(VR)技术的人机交互系统.在该系统中,救援机器人基于多线激光雷达和惯性测量单元(IMU)实时构建环境的三维点云地图,并将建图结果增量式地表示为3D-NDT地图,实时传输至操作台的虚拟现实系统中可视化;同时,操作人员利用虚拟现实系统的交互设备生成机器人的控制指令,控制机器人运动,构成一个完整的人在回路的人机交互系统.该系统在将机器人环境实时在虚拟现实中可视化的基础上,可以给操作人员以极强的沉浸感,有利于操作人员更直接地理解机器人所处环境.此外,该系统作为一种新的人机交互方式,为提高人与机器人的自然交互水平提供了新思路,对促进人机交互技术的发展具有重要意义.     

Research on environmental landscape design based on virtual reality technology and deep learning

Virtual Reality (VR) provides immersive visualization and intuitive interaction. The VR is used to enable any biomedical profession to develop a deep learning (DL) model for image classification. The Deep Neural Network (DNN) models can be used as a powerful tool for data analysis, but they are also challenging to understand and develop. To make deep learning more convenient and fast operation, it have established a landscape of DNN development environment based on virtual reality. In this environment, users can move concrete objects only to build neural networks with their own hands. It automatically transforms these configurations into a trainable model and reports on the real-time test dataset. In addition to realizing the insights users are developing into DNN models, it has also visually enriched the virtual environmental landscape objects with the parts of the model. In this way bridge the gap between professionals in different disciplines, providing a new perspective on the model analysis and data interaction. This system further demonstrates that learning and visualization technologies developed in Shenzhen can benefit from integrating virtual reality.     

基于VR和AR技术的工业产品包装工程设计研究

为了有效提高工业产品包装与产品的适应性,设计引入VR和AR技术,提出VRML工业产品包装工程设计方法。方法主要包括三个步骤,首先利用VR虚拟现实技术,在虚拟现实空间内,建立产品包装湍流模组,该模组主要用于求取与产品完全契合的包装内部数据,在VR技术的基础上应用AR增强现实,生成VRML虚拟交互空间,将虚拟空间节点进行覆盖连接,结合包装内部数据,生成工业产品包装虚拟外观,最后在虚拟空间内,对上述生成的产品包装细节层次和光照物化进行重新矫正,获取的最终结果即为包装工程最终设计结果'根据实验数据分析可以肯定,通过使用VRML工业产品包装工程设计方法,工业产品包装贴合度上升32%,抗压比提高了29%,证明该包装有效提高了产品抗震性能,具有重要使用价值。     

虚拟膝关节手术中的实时大范围形变方法

虚拟膝关节手术过程中要求对膝关节3D几何模型进行实时的大范围形变,而形变的特点,包括总体轮廓、细节保持、实时性、平滑性以及拓扑性等都影响到它的真实感。针对膝关节3D模型得到了一种新的形变方法,根据模型的横截面构建出模型的骨架,并利用柱坐标的变换来实现大范围形变,使形变能保持良好的局部细节特征,算法能满足虚拟手术的实时性和真实感的要求。该方法也能推广应用于其他类似的关节弯曲运动的实时变形。     

虚拟现实手部康复训练系统的设计与实现

将虚拟现实技术应用到康复医学领域,可有效克服传统康复训练方法的局限性,实现安全、舒适和主动的康复训 练。本文设计并实现了一套虚拟现实手部康复训练系统,系统由交互设备、人机交互软件和虚拟环境三部分组成。交互 设备采用 5DT 公司生产的 5DT Data Glove 14 Ultra 数据手套,而人机交互软件运用 Visual Studio 2012 作为开发工具,基于 MFC 编写,实现了用户管理、数据采集、手势信号分类、实时手势识别测试等功能。构建的虚拟场景使用 Flash 游戏, 通过 MFC 和 Flash 游戏之间通讯使用者能使用手势信号实现游戏操控。本文的实验结果表明：虚拟现实手部康复训练系 统能够指导使用者进行有效的手部康复训练,Flash 康复训练游戏能有效提高使用者进行康复训练的积极性和主动性。     

一种基于物理的实时细节保持变形算法

实时变形是计算机图形学研究的热点问题之一,复杂物体的实时变形至今仍未得到很好的解决.从物理变形方法和多分辨率网格编辑技术的优点出发,提出了一种适合于复杂弹性物体的实时变形算法.在预处理阶段,将原始精细网格模型进行简化以建立其基网格表示,基于基网格对模型的局部细节特征进行编码;在实时绘制阶段,在基网格上进行物理变形操作,并通过变形后的基网格和细节编码重构出变形后的精细网格.以上过程充分利用图形硬件的并行处理能力,利用像素处理器进行大部分计算操作.实验结果表明,该算法在变形过程中较好地保持了物体的局部特征,适合于表面细节复杂物体的实时变形应用.     

基于光学映射虚物体的并行绘制

尽管当前基于全局光照模型的图形绘制方法可以渲染出高质量的图象 ,但因为其计算量巨大 ,难以适用于诸如建筑物漫游、虚拟现实等对绘制速度有严格要求的场合 .为此引入光学映射虚物体的概念 ,利用构建在联网PC机上的集群系统 ,并行创建反射和折射虚物体 ,然后利用集群中各节点的图形加速硬件 ,像处理实际三维物体一样绘制这些虚物体 ,可以快速地绘制出反射 /折射效果的图象 .实验结果证明 ,该方法利用 CU P的计算能力和图形硬件的加速特性实现了真实感图形的高性能绘制 ,特别适用于诸如建筑物漫游、计算机动画和虚拟现实等要求交互式绘制的场合     

虚拟现实技术在柔性上肢康复机器人中的应用

根据末端牵引式和外骨骼式上肢康复机器人的特性,研制了一种新型的柔性上肢康复机器人。机器人的康复训练系统结合了主动和被动模式的要素,将虚拟现实（VR）技术引入上肢康复机器人,通过现实环境中的光学3D位置捕获以及VR环境中的3D位置感知,设计了虚拟动态模型交互性节点和碰撞检测实验,实现虚拟现实交互,提高虚拟模型运动实时效果和上肢康复训练精度。VR和新颖的康复机器人的集成为具有特定任务的患者提供了有效的训练。     

Research on hybrid synchronization methods in multi-user collaborative VR simulation medical surgery training system

The simulation training system based on VR technology has been applied in clinical medicine due to its immersive interactive experience, economy, and safety. In response to the limitation that most medical surgery training systems can only achieve single user operations, a hybrid synchronous model based multi-user collaborative framework design is proposed to meet the needs of collaborative training in virtual reality teaching processes. Based on the characteristics of frame synchronization and state synchronization, the synchronization method has been improved. Based on frame synchronization, specific state synchronization is used on interactive events to improve the consistency of operation details and real-time action. On this basis, a system prototype has been implemented using Unity 3D, and compared with traditional schemes, analysis shows that it has faster response speed for disconnected reconnection and more stable interaction state.     

A simple footskate removal method for virtual reality applications

Footskate is a common problem encountered in interactive applications dealing with virtual character animations. It has proven difficult to fix without the use of complex numerical methods, which require expert skills for their implementations, along with a fair amount of user interaction to correct a motion. On the other hand, deformable bodies are being increasingly used in virtual reality (VR) applications, allowing users to customize their avatar as they wish. This introduces the need of adapting motions without any help from a designer, as a random user seldom has the skills required to drive the existing algorithms towards the right solution. In this paper, we present a simple method to remove footskate artifacts in VR applications. Unlike previous algorithms, our approach does not rely on the skeletal animation to perform the correction but rather on the skin. This ensures that the final foot planting really matches the virtual character’s motion. The changes are applied to the root joint of the skeleton only so that the resulting animation is as close as possible to the original one. Eventually, thanks to the simplicity of its formulation, it can be quickly and easily added to existing frameworks.     

Development of a physically behaved robot work cell in virtual reality for task teaching

Robot task teaching on a real work cell is expensive and sometimes risky. This cost and risk can be avoided by using virtual reality technology. Using the simulated environment in virtual reality (VR), the operator can practise, explore and preview the operations for possible problems that might occur during implementation. It is therefore of practical importance to build the virtual robot work cell in VR that can facilitate the study of the performance of robotic tasks such as robotic assembly. This paper describes our work in incorporating physical behaviours of virtual objects into VR for robot task teaching. To facilitate the task teaching, we developed visual and audio cues which help visualise the dynamic interactions between virtual objects. Dynamic sensing capability is incorporated in the simulated environment. A simplified force sensor is modelled and simulated. The physical behaviours of the virtual objects are simulated using physics-based approach. A virtual robot work cell is built incorporating the developed features and an example for the task teaching is given. The implementation includes view tracking using virtual camera, visual and audio rendering, and the user interface developed in the VR. The current implementation was carried out on a PC-based VR platform, with the programs developed using Watcom C++.     

基于VR设备中IMU的无手交互方法研究

随着头戴式显示设备的发展,在基于虚拟现实(VR,virtual reality)的教育培训中,存在用户与设备间进行交互的场景;针对用户与VR视频中对象的模拟靠近与躲避问题,提出了姿态感知与步态识别相结合的方法;通过姿态感知算法解算用户头部姿态,通过步态识别算法识别出人体的静止与行走状态,进而在行走状态时,将计算所得航向角和固定步长代入三角函数公式进行位置更新,在静止状态时,保持位置不变;实验证明,提出的姿态感知算法可以有效的计算出使用者头部的姿态,与商用惯性测量单元提供的姿态角相比具有1.1×10-2的平均姿态偏差;提出的步态识别算法可以有效地识别出人体的静止与行走状态;所提出的两者结合的交互方法,可以有效地实现虚拟的靠近与躲避.     

A hierarchically structured and constraint-based data model for intuitive and precise solid modeling in a virtual reality environment

This article proposes a hierarchically structured and constraint-based data model for intuitive and precise solid modeling in a virtual reality (VR) environment. The data model integrates a high level constraint-based model for intuitive and precise manipulation, a middle level solid model for complete and precise representation and a low-level polygon mesh model for real-time interactions and visualization in a VR environment. The solid model is based on a hybrid B-rep/CSG data structure. Constraints are embedded in the solid model and are organized at hierarchical levels as feature constraints among internal feature elements, part constraints among internal features and assembly constraints between individual parts. In addition to providing a complete and precise model representation and the support for real-time visualization, the proposed data model permits intuitive and precise interaction through constraint-based manipulations for solid modeling in a VR environment. This is a critical issue for product design in a VR environment due to the limited resolutions of today's VR input and output devices.     

INTERVALES: INTERactive Virtual and Augmented framework for industriaL Environment and Scenarios

One of Industry 4.0’s greatest challenges for companies is the digitization of their processes and the integration of new related technologies such as virtual reality (VR) and augmented reality (AR), which can be used for training purposes, design, or assistance during industrial operations. Moreover, recent results and industrial proofs of concept show that these technologies demonstrate critical advantages in the industry. Nevertheless, the authoring and editing process of virtual and augmented content remains time-consuming, especially in complex industrial scenarios. While the use of interactive virtual environments through virtual and augmented reality presents new possibilities for many domains, a wider adoption of VR/AR is possible only if the authoring process is simplified, allowing for more rapid development and configuration without the need for advanced IT skills. To meet this goal, this study presents a new framework: INTERVALES. First, framework architecture is proposed, along with its different modules; this study then shows that the framework can be updated by not only IT workers, but also other job experts. The UML data model is presented to format and simplify the authoring processes for both VR and AR. This model takes into account virtual and augmented environments, the possible interactions, and ease operations orchestration. Finally, this paper presents the implementation of an industrial use case composed of collaborative robotic (cobotic) and manual assembly workstations in VR and AR based on INTERVALES data.     

A Physics-driven Neural Networks-based Simulation System (PhyNNeSS) for multimodal interactive virtual environments involving nonlinear deformable objects

BACKGROUND: While an update rate of 30 Hz is considered adequate for real time graphics, a much higher update rate of about 1 kHz is necessary for haptics. Physics-based modeling of deformable objects, especially when large nonlinear deformations and complex nonlinear material properties are involved, at these very high rates is one of the most challenging tasks in the development of real time simulation systems. While some specialized solutions exist, there is no general solution for arbitrary nonlinearities. METHODS: In this work we present PhyNNeSS - a Physics-driven Neural Networks-based Simulation System - to address this long-standing technical challenge. The first step is an off-line pre-computation step in which a database is generated by applying carefully prescribed displacements to each node of the finite element models of the deformable objects. In the next step, the data is condensed into a set of coefficients describing neurons of a Radial Basis Function network (RBFN). During real-time computation, these neural networks are used to reconstruct the deformation fields as well as the interaction forces. RESULTS: We present realistic simulation examples from interactive surgical simulation with real time force feedback. As an example, we have developed a deformable human stomach model and a Penrose-drain model used in the Fundamentals of Laparoscopic Surgery (FLS) training tool box. CONCLUSIONS: A unique computational modeling system has been developed that is capable of simulating the response of nonlinear deformable objects in real time. The method distinguishes itself from previous efforts in that a systematic physics-based pre-computational step allows training of neural networks which may be used in real time simulations. We show, through careful error analysis, that the scheme is scalable, with the accuracy being controlled by the number of neurons used in the simulation. PhyNNeSS has been integrated into SoFMIS (Software Framework for Multimodal Interactive Simulation) for general use.     

From visual simulation to virtual reality to games

During the past decades, the virtual reality community has based its development on a synthesis of earlier work in interactive 3D graphics, user interfaces, and visual simulation. Currently, the VR field is transitioning into work influenced by video games. Because much of the research and development being conducted in the games community parallels the VR community's efforts, it has the potential to affect a greater audience. Given these trends, VR researchers who want their work to remain relevant must realign to focus on game research and development. Leveraging technology from the visual simulation and virtual reality communities, serious games provide a delivery system for organizational video game instruction and training.     

Context-Aware Mixed Reality: A Learning-Based Framework for Semantic-Level Interaction

Mixed reality (MR) is a powerful interactive technology for new types of user experience. We present a semantic-based interactive MR framework that is beyond current geometry-based approaches, offering a step change in generating high-level context-aware interactions. Our key insight is that by building semantic understanding in MR, we can develop a system that not only greatly enhances user experience through object-specific behaviours, but also it paves the way for solving complex interaction design challenges. In this paper, our proposed framework generates semantic properties of the real-world environment through a dense scene reconstruction and deep image understanding scheme. We demonstrate our approach by developing a material-aware prototype system for context-aware physical interactions between the real and virtual objects. Quantitative and qualitative evaluation results show that the framework delivers accurate and consistent semantic information in an interactive MR environment, providing effective real-time semantic-level interactions.     

用户认知驱动的VR自然交互认知负荷研究

针对目前虚拟现实自然交互系统中用户认知负荷难以量化问题,以量化认知负荷为目的,提出一种基于概率神经网络的多通道信息融合模型。首先根据认知理论建立“认知—行为—环境”框架,并在此基础上研究用户认知的数学表征（方法,过程,机理）,构建多通道认知模型;然后将多通道信息融合方法引入虚拟现实自然交互系统,建立一种面向虚拟现实自然交互系统的认知负荷模型;通过VR隧道应急救援系统实验对系统中用户认知进行验证。验证结果表明,具体提出的认知负荷模型具有一定的可行性。