Assessing future landscapes using enhanced mixed reality with semantic segmentation by deep learning

Architecture, engineering, and construction projects need to be promoted in harmony with the natural environment and with the aim of preserving people’s living environment. At the planning and design stage, decision-makers and stakeholders share and assess landscape images during and after construction in order to avoid as much uncertainty as possible when performing environmental impact assessment. Given the lack of a standard visualization method for future landscapes that do not yet exist, mixed reality (MR), which overlays virtual content onto a real scene, has attracted attention in the field of landscape design. One challenge in MR is occlusion, which occurs when virtual objects obscure physical objects that should be rendered in the foreground. In MR-based landscape visualization, the distance between the MR camera and real objects located in front of the virtual objects might vary and might be large, causing difficulty for existing occlusion handling methods. In the process of landscape design, an evidence-based approach has also become important. Landscape index estimation using semantic segmentation by deep learning, which can recognize the surrounding environment, has been actively studied for landscape assessment. In this study, semantic segmentation by deep learning was integrated into an MR system to enable dynamic occlusion handling and landscape index estimation for both existing and designed landscape assessment. This system can be operated on a mobile device with video communication over the internet by connecting to real-time semantic segmentation on a high-performance personal computer. The applicability of the developed system is demonstrated through accuracy verification and case studies.     

The physiological mirror—a system for unconscious control of a virtual environment through physiological activity

This paper introduces a system for real-time physiological measurement, analysis, and metaphorical visualization within a virtual environment (VE). Our goal is to develop a method that allows humans to unconsciously relate to parts of an environment more strongly than to others, purely induced by their own physiological responses to the virtual reality (VR) displays. In particular, we exploit heart rate, respiration, and galvanic skin response in order to control the behavior of virtual characters in the VE. Such unconscious processes may become a useful tool for storytelling or assist guiding participants through a sequence of tasks in order to make the application more interesting, e.g., in rehabilitation. We claim that anchoring of subjective bodily states to a virtual reality (VR) can enhance a person’s sense of realism of the VR and ultimately create a stronger relationship between humans and the VR.     

A hierarchically structured and constraint-based data model for intuitive and precise solid modeling in a virtual reality environment

This article proposes a hierarchically structured and constraint-based data model for intuitive and precise solid modeling in a virtual reality (VR) environment. The data model integrates a high level constraint-based model for intuitive and precise manipulation, a middle level solid model for complete and precise representation and a low-level polygon mesh model for real-time interactions and visualization in a VR environment. The solid model is based on a hybrid B-rep/CSG data structure. Constraints are embedded in the solid model and are organized at hierarchical levels as feature constraints among internal feature elements, part constraints among internal features and assembly constraints between individual parts. In addition to providing a complete and precise model representation and the support for real-time visualization, the proposed data model permits intuitive and precise interaction through constraint-based manipulations for solid modeling in a VR environment. This is a critical issue for product design in a VR environment due to the limited resolutions of today's VR input and output devices.     

Vision‐tangible interactive display method for mixed and virtual reality: Toward the human‐centered editable reality

Building a human‐centered editable world can be fully realized in a virtual environment. Both mixed reality (MR) and virtual reality (VR) are feasible solutions to support the attribute of edition. Based on the current development of MR and VR, we present the vision‐tangible interactive display method and its implementation in both MR and VR. We address the issue of MR and VR together because they are similar regarding the proposed method. The editable mixed and virtual reality system is useful for studies, which exploit it as a platform. In this paper, we construct a virtual reality environment based on the Oculus Rift, and an MR system based on a binocular optical see‐through head‐mounted display. In the MR system about manipulating the Rubik's cube, and the VR system about deforming the virtual objects, the proposed vision‐tangible interactive display method is utilized to provide users with a more immersive environment. Experimental results indicate that the vision‐tangible interactive display method can improve the user experience and can be a promising way to make the virtual environment better.     

CAD模型转换到VR模型的实时可视性研究

实时3D 可视化是机械产品设计当中的一个主要问题,针对CAD几何学和装配以及机械产品的高可视化提出了MEMPHIS中间件框架,目的是连接实时虚拟现实应用,特别是CAD模型的管理操作设计。同时还保证了当进行模型转换时,在CAD模型上必须作相应的几何改变,系统会同步变化,这样避免了传统的从CAD模型转换到VR模型转换时人工加入虚拟现实特殊信息,如光照设置、材质、纹理、行为这些刻画VR模型可视化特性的信息属性后才能实现这种转换。从而大大提高了机械产品协同设计的效率和可视性性能。     

Effective virtual reality based building navigation using dynamic loading and path optimization

Although computer capabilities have been improved significantly, a large-scale virtual reality (VR) system demands much more in terms of memory and computation than the current computer systems can offer. This paper discusses two important issues related to VR performance and applications in building navigation. These are dynamic loading of models based on cell segmentation for the optimal VR operation, and the route optimization based on path planning for easy navigation. The VR model of engineering and information technology complex (EITC) building at the University of Manitoba is built as an example to show the feasibility of the proposed methods. The reality, enhanced by three-dimensional (3D) real-time interactivity and visualization, leads navigators into a state of the virtual building immersion.     

Development of a physically behaved robot work cell in virtual reality for task teaching

Robot task teaching on a real work cell is expensive and sometimes risky. This cost and risk can be avoided by using virtual reality technology. Using the simulated environment in virtual reality (VR), the operator can practise, explore and preview the operations for possible problems that might occur during implementation. It is therefore of practical importance to build the virtual robot work cell in VR that can facilitate the study of the performance of robotic tasks such as robotic assembly. This paper describes our work in incorporating physical behaviours of virtual objects into VR for robot task teaching. To facilitate the task teaching, we developed visual and audio cues which help visualise the dynamic interactions between virtual objects. Dynamic sensing capability is incorporated in the simulated environment. A simplified force sensor is modelled and simulated. The physical behaviours of the virtual objects are simulated using physics-based approach. A virtual robot work cell is built incorporating the developed features and an example for the task teaching is given. The implementation includes view tracking using virtual camera, visual and audio rendering, and the user interface developed in the VR. The current implementation was carried out on a PC-based VR platform, with the programs developed using Watcom C++.     

Tele-immersive product evaluation: a review and an implementation framework

A novel virtual tele-immersive product evaluation environment is conceived. The components for the tele-immersive virtual environment for product evaluation include a robust virtual reality (VR) hardware system, associated VR driving software, development tool for the tele-immersive virtual environment, networking software, user representation scheme and tools for developing 3D models and incorporating dynamic properties into the models. We have developed a model to allow users to collaboratively evaluate products using the CAVE^TM, Performer, CAVERN, CAVEActors, Pro/ENGINEER, and ADAMS.     

一种基于三维建图和虚拟现实的人机交互系统

以提高人机共融水平为目的,以救援机器人为背景,提出并实现基于三维建图和虚拟现实(VR)技术的人机交互系统.在该系统中,救援机器人基于多线激光雷达和惯性测量单元(IMU)实时构建环境的三维点云地图,并将建图结果增量式地表示为3D-NDT地图,实时传输至操作台的虚拟现实系统中可视化;同时,操作人员利用虚拟现实系统的交互设备生成机器人的控制指令,控制机器人运动,构成一个完整的人在回路的人机交互系统.该系统在将机器人环境实时在虚拟现实中可视化的基础上,可以给操作人员以极强的沉浸感,有利于操作人员更直接地理解机器人所处环境.此外,该系统作为一种新的人机交互方式,为提高人与机器人的自然交互水平提供了新思路,对促进人机交互技术的发展具有重要意义.     

面向边缘设备的高能效深度学习任务调度策略

近年来,深度学习在图像和自然语言处理等诸多领域表现出色,与深度学习相关的各类移动应用发展迅速,但由于移动网络状态的不稳定性及网络带宽的限制,基于云计算的深度模型任务可能出现较大响应延迟,严重影响用户体验.与此同时,深度模型对设备的计算及存储能力有较高的要求,无法直接在资源受限的移动设备中进行部署.因此,亟须设计一种新的计算模式,使得基于深度模型的移动应用能够满足用户对快速响应、低能耗及高准确率的期望.本文提出一种面向边缘设备的深度模型分类任务调度策略,该策略通过协同移动设备与边缘服务器,充分利用智能移动终端的便捷性和边缘服务器强大的计算能力,综合考虑分类任务的复杂度和用户期望,完成深度模型在移动设备和边缘服务器中的动态部署,并对推理任务进行动态调度,从而提升任务执行效率,降低深度学习模型推理开销.本文以基于卷积神经网络的图像识别应用为例,实验结果表明,在移动环境中,相比于准确率最高的深度模型,本文提出的高能效调度策略的推理能耗可降低93.2%、推理时间降低91.6%,同时准确率提升3.88%.     

A scene graph based visualization method for representing continuous simulation data

This paper proposes a scene graph based visualization method that can verify time-varying continuous analysis simulation in a virtual reality (VR) environment by using the computer-aided engineering (CAE) data of structural analysis in product development. In previous research, the use of CAE analysis data has been problematic because of the lack of any interactive simulation controls for visualizing continuous simulation data. Moreover, the research on post-processing methods for real-time evaluation of CAE analysis data has not been sufficient. We therefore propose a scene graph based visualization method for representing continuous simulation data. The method can continuously visualize static analysis data independently of any timeline; it can also continuously visualize dynamic analysis data that varies in relation to the timeline. The visualization system for continuous simulation data, which includes CAE2VR Middleware that interfaces with various formats of CAE analysis data as well as functions for visualizing continuous simulation data and operational functions, enables users to verify simulation results with more realistic scenes.     

基于 BIM+VR 技术的钢结构桥梁 教学实践改革研究

为改变当前本科院校钢结构桥梁课程教学效果不佳的现状,主动适应教育、建筑 行业信息化改革发展的新趋势,针对该课程专业性和实践性较强特点,我校尝试结合建筑信息 模型(BIM)技术和虚拟现实(VR)技术进行课程教学实践改革探索,将 BIM 技术和 VR 技术融入 到课堂教学和实践教学的各环节中,建立学生主动探索学习,教师协作引导的新型教学体系。 通过以钢桁梁课程实践设计为例,论证了引入 BIM+VR 技术的必要性、可行性和有效性,介绍 了应用 Revit、Navisworks 和 Fuzor 软件建立 BIM 和 VR 体验环境的实践过程,总结了引入 BIM 技术和 VR 技术到钢结构桥梁教学实践课程中存在的困难并提出部分对策,以期改善教学困境, 为相关课程的教学实践改革探索提供参考和借鉴。     

VRML数据模型的三维可视化研究

为了把虚拟现实技术应用到三维地理信息系统-3DGIS软件中，本在分析虚拟现实建模语言(Virtual Reality Modeling Language)数据结构的基础上，采用面向对象的方法创建VRML结构类，并借助OpenGL在3DGIS软件开发中实现了VRML数据模型的读取和三维可视化，以及漫游路线回放等交互操作功能。     

带有力感觉的显微外科手术血管的仿真研究

随着虚拟现实技术的进步，带有力感觉的仿真研究迅速发展，特别是在医学上得到了广泛应用。利用虚拟现实技术取代效率低下的常规培训，可以大大减少所需的培训时间和昂贵的动物实验费用。针对复杂的显微外科手术的血管缝合，作者建立了可用于大夫培训、手术规划的虚拟现实系统。其中的交互设备采用了具有力反馈功能的PHANTOM—Desktop，使用与之配套的GHOSTSDK软件开发包，建立虚拟环境。考虑到柔性体仿真的实时性和真实性要求，采用了质量一弹簧模型的建模方法。在完成了对虚拟环境中柔性平面仿真的基础上，成功实现了带有力感觉的显微外科血管穿刺的虚拟仿真。     

新冠肺炎CT影像的DNN对抗攻击研究

在深度学习应用于新型冠状肺炎CT智能识别的研究中,大量研究人员通过构建深度神经网络训练模型,从而理解医学影像数据内容,辅助新冠肺炎诊断。提出AMDRC-Net架构,其中的残差结构,通过恒等映射解决了网络退化问题,与此同时,针对残差结构阻碍新特征探索的新问题,受到注意力机制等最新研究启发,研究了长短注意力引导机制。关注深度学习模型安全性问题,讨论基于梯度上升的对抗攻击方法;为了解决其单一性问题,通过长短注意力机制,增加有效对抗扰动的同时减少冗余扰动,紧接着,提出的对抗攻击算法A-IM-FGSM,将对抗攻击问题转化为自适应约束问题,即可微变换思想用于迭代攻击中,探究注意力引导机制与DNN对抗攻击的相互关系。最后进行的实验中,在新型冠状肺炎CT数据集上,通过AMDRC-Net进行模型训练,设计对比实验、可视化实验、对抗攻击实验。     

利用标注者相关性的深度生成式众包学习

传统监督学习需要训练样本的真实标记信息,而在很多情况下,真实标记并不容易收集.与之对比,众包学习从多个可能犯错的非专家收集标注,通过某种融合方式估计样本的真实标记.注意到现有深度众包学习工作对标注者相关性建模不足,而非深度众包学习方面的工作表明,标注者相关性建模利用有助于改善学习效果.提出一种深度生成式众包学习方法,以...     

A virtual reality-based experimentation environment for the verification of human-related factors in assembly processes

This paper investigates the use of virtual reality (VR)-based methods for the verification of performance factors related to manual assembly processes. An immersive and interactive virtual environment has been created to provide functionality for realistic process experimentation. Ergonomic models and functions have been embedded into the VR environment to support verification and constrain experimentation to ergonomically acceptable conditions. A specific assembly test case is presented, for which a semi-empirical time model is developed employing statistical design experimentation in the virtual environment. The virtual experimentation results enable the quantification and prediction of the influence of a number of process parameters and their combination at the process cycle time.     

深度语音信号与信息处理：研究进展与展望

论文首先对深度学习进行简要的介绍,然后就其在语音信号与信息处理研究领域的主要研究方向,包括语音识别、语音合成、语音增强的研究进展进行了详细的介绍。语音识别方向主要介绍了基于深度神经网络的语音声学建模、大数据下的模型训练和说话人自适应技术;语音合成方向主要介绍了基于深度学习模型的若干语音合成方法;语音增强方向主要介绍了基于深度神经网络的若干典型语音增强方案。论文的最后我们对深度学习在语音信与信息处理领域的未来可能的研究热点进行展望。     

A Comprehensive Review of Redirected Walking Techniques: Taxonomy,Methods, and Future Directions

Virtual reality (VR) allows users to explore and experience a computer-simulated virtual environment so that VR users can be immersed in a totally artificial virtual world and interact with arbitrary virtual objects. However, the limited physical tracking space usually restricts the exploration of large virtual spaces, and VR users have to use special locomotion techniques to move from one location to another. Among these techniques, redirected walking (RDW) is one of the most natural locomotion techniques to solve the problem based on near-natural walking experiences. The core idea of the RDW technique is to imperceptibly guide users on virtual paths, which might vary from the paths they physically walk in the real world. In a similar way, some RDW algorithms imperceptibly change the structure and layout of the virtual environment such that the virtual environment fits into the tracking space. In this survey, we first present a taxonomy of existing RDW work. Based on this taxonomy, we compare and analyze both contributions and shortcomings of the existing methods in detail, and find view manipulation methods offer satisfactory visual effect but the experience can be interrupted when users reach the physical boundaries, while virtual environment manipulation methods can provide users with consistent movement but have limited application scenarios. Finally, we discuss possible future research directions, indicating combining artificial intelligence with this area will be effective and intriguing.     

Design and development of a maintenance and virtual training system for ancient Chinese architecture

Ancient Chinese architecture is an important aspect of traditional Chinese culture and has been studied by many scholars around the world via historical documents, photographs, and three-dimensional models. In this paper, a building information model (BIM) and virtual reality (VR) and video analysing technology are used to develop a maintenance and virtual training system for ancient architecture. A digital ancient architecture model that includes a three-dimensional model and attributes is established, and the model can be visualized using a VR video processing system. Based on this system, we propose a method of fire detection in the maintenance system to ensure the safety of ancient buildings. After performing lightweight processing of the three-dimensional model, the Forge platform, which can achieve high-speed browsing via Web browsers, is used to perform the virtual construction, dismantling and other functions. By providing an immersive experience, users will develop a deeper understanding of ancient architectural structures and construction processes, which will accelerate research on ancient architecture.