基于TUI的场景规划系统中的交互技术

单摄像头条件下,基于桌面TUI,划分出4类具有不同场景语义的用户界面操作实物;分析实物用户界面中执行通用交互任务的特点,建立支持4类语义的3层系统设计框架;结合场景规划系统自身特征,设计了4类承载上述场景语义的标记,并提出标记任务分配策略;为解决大场景与小视域之间的矛盾,结合paddle技术,提出动态比例空间、动态成组、双手交互和遮挡处理等交互技术;借鉴认知心理学原理,提出时空复用交互技术;开发基于TUI的交互工具箱,并进行了应用验证.     

酷软与插件

“美国编号为2008001 3860的专利技术“构建三维立体用户界面”是由微软公司于2006年7月14日提交的。一个创建三维用户界面的系统在一个3D的场景中为一个3D的物体模型绘制一个2D的交互式用户界面。一旦绘制完成,这个2D的用户界面将作为那个3D物体的外壳而成为一个基本的用户界面。一个3D的用户界面创建工具将使用一个2D用户界面的模型的图像资源结合一个3D的线框物体模型来创建一个3D的用户界面。”以上这些拗口的话来自……真不知道下一代的操作系统会是什么样子。     

协作笔式用户界面开发工具研究

笔式用户界面是一种重要的Post-WIMP(window icon menu pointer)界面,它给用户提供了自然的交互方式.然而,当前的笔式用户界面工具箱大多是面向单用户任务的,不能很好地支持协作应用场景.通过对笔式交互特征和协作环境功能需求的分析,设计并实现了一个工具箱CoPen Toolkit,用于支持协作笔式用户界面的开发.它提供了灵活的架构和可扩展的组件,支持笔迹描述、事件处理和网络协作等功能.基于CoPen Toolkit,构造了多个原型系统,实践表明,它能够很好地支持协作笔式用户界面的开发.     

场景语义约束的三维交互技术

针对复杂虚拟环境下的交互需求,从面向高层应用的角度对交互任务进行分析,提出三维交互任务的层次模型。结合应用特点,将虚拟场景中的语义约束引入到三维交互技术的设计中,实现了多种面向高层应用的导航技术和选择操作技术,并在不同交互技术和操作隐喻模式之间实现了自然的融合和平滑的转换。用户界面利用这些语义约束更好地感知和理解用户的交互意图,实现更为直观、高效的三维交互过程。     

基于VRML和Java3D的虚拟漫游系统研究

采用3DSMAX对别墅场景进行建模,提出将VRML与Java3D相结合为该场景添加交互功能的方案;阐述了建模方法、用Java3D控制场景的方法和具体步骤,以及用LOD技术提高场景漫游速度的方法;将Java Swing和Java3D结合构建用户平台.在该平台上实现了别墅场景的虚拟漫游.     

基于Web3D的虚拟场景漫游的设计与实现

介绍了Java3D与VR相结合的Web3D应用;采用了三层式的Client/Server结构和分布计算模型设计系统体系结构,提出了在Java3D中获取场景对象实例的方法,给出了在Java3D场景中实现虚拟场景中漫游的方法。采用Java3D实现与用户交互虚拟三维场景,并给出了部分实现细节。     

基于Web3D的虚拟场景漫游的设计与实现

介绍了Java3D与VR相结合的Web3D应用；采用了三层式的Client／Server结构和分布计算模型设计系统体系结构，提出了在Java3D中获取场景对象实例的方法，给出了在Java3D场景中实现虚拟场景中漫游的方法。采用Java3D实现与用户交互虚拟三维场景，并给出了部分实现细节。     

一种基于多层语义特征的图像理解方法

视觉场景理解包括检测和识别物体、推理被检测物体之间的视觉关系以及使用语句描述图像区域.为了实现对场景图像更全面、更准确的理解,将物体检测、视觉关系检测和图像描述视为场景理解中3种不同语义层次的视觉任务,提出一种基于多层语义特征的图像理解模型,并将这3种不同语义层进行相互连接以共同解决场景理解任务.该模型通过一个信息传递图将物体、关系短语和图像描述的语义特征同时进行迭代和更新,更新后的语义特征被用于分类物体和视觉关系、生成场景图和描述,并引入融合注意力机制以提升描述的准确性.在视觉基因组和COCO数据集上的实验结果表明,所提出的方法在场景图生成和图像描述任务上拥有比现有方法更好的性能.     

基于OpenGL的可复用软构件库与三维交互设计

Intra3D是基于OpenGL的交互式三维图形软件开发工具,其核心是集成了场景数据结构、图形对象、三维交互算法和图形用户界面的C 类库与COM对象库,论述了Intra3D可复用软构件库的体系结构和三维交互设计。软构件库分为基础对象与函数、图形对象、场景图与节点和绘制与交互4层,三维交互着重论述节点的图形交换、鼠标跟踪球算法和3D用户界面设计。     

一种新的面向电影场景搭建的自然手势交互方法

电影虚拟预演所需的三维场景搭建要求能够高效地初步将导演的构想付诸于影像,但传统计算机辅助设计工具的交互方式存在效率低的问题,限制了导演等非专业三维设计人士的场景构思.因此提出一种新的面向电影场景搭建的自然手势交互方法.根据电影虚拟预演特性和相关手势理论,设计一组有效的三维场景搭建手势集;在Unity3 D中构建基于虚拟现实的电影场景搭建徒手手势用户界面;通过用户实验对该方法进行评估,实验结果表明该方法是可行的,相比传统方法更具有自然易用性.     

三维组态软件设计中场景快速组态及交互方法研究

针对三维组态软件三维监控场景组态困难以及人机交互复杂的问题,提出一种基于本体的三维场景快速组态方法以及分层三维交互模型。在三维组态软件设计中给监控设备模型加入语义信息,使组态过程智能化,从而使监控场景的组态更加简便,人机交互更加自然。采用分层的人机交互模型,实现了高层交互的设备无关性,有利于系统的扩展。采用以上方法增强了三维组态软件开发及使用的有效性。     

A study of detecting and combating cybersickness with fuzzy control for the elderly within 3D virtual stores

Elderly individuals can access online 3D virtual stores from their homes to make purchases. However, most virtual environments (VEs) often elicit physical responses to certain types of movements in the VEs. Some users exhibit symptoms that parallel those of classical motion sickness, called cybersickness, both during and after the VE experience. This study investigated the factors that contribute to cybersickness among the elderly when immersed in a 3D virtual store. The results of the first experiment show that the simulator sickness questionnaire (SSQ) scores increased significantly by the reasons of navigational rotating speed and duration of exposure. Based on these results, a warning system with fuzzy control for combating cybersickness was developed. The results of the second and third experiments show that the proposed system can efficiently determine the level of cybersickness based on the fuzzy sets analysis of operating signals from scene rotating speed and exposure duration, and subsequently combat cybersickness.     

Separating semantics from rendering: a scene graph based architecture for graphics applications

A large number of rendering and graphics applications developed in research and industry are based on scene graphs. Traditionally, scene graphs encapsulate the hierarchical structure of a complete 3D scene, and combine both semantic and rendering aspects. In this paper, we propose a clean separation of the semantic and rendering parts of the scene graph. This leads to a generally applicable architecture for graphics applications that is loosely based on the well-known Model-View-Controller (MVC) design pattern for separating the user interface and computation parts of an application. We explore the benefits of this new design for various rendering and modeling tasks, such as rendering dynamic scenes, out-of-core rendering of large scenes, generation of geometry for trees and vegetation, and multi-view rendering. Finally, we show some of the implementation details that have been solved in the process of using this software architecture in a large framework for rapid development of visualization and rendering applications.     

基于实例和约束的三维场景概念设计和系统实现

3D场景设计是VR建模中最重要也最耗人力的工作之一,如何解决该问题是一项重要的课题.本文提出一种新的基于实例和约束的三维场景概念设计方法.首先,根据所获取的场景实例中所蕴涵的不同层次的语义信息,对其进行场景元素过滤和场景语义标注,并由此构建三维场景实例库.相应地,对场景中可能出现的元素类别进行分类和语义标注,并建立场景元素库.接着,应用基于实例的方法,并结合场景设计过程中所涉及到的知识(约束),给出三维场景概念设计的一般方法;最后以一个实例对该方法做进一步的说明.研究表明,该方法符合人们对场景设计的一般过程.此外,由于场景实例和场景元素具有丰富的语义信息,使得所生成的场景除了常规的可视效果,还可以直接应用于虚拟现实环境以及三维动画等数字内容制作中.     

Geometric calibration of head-mounted displays and its effects on distance estimation

Head-mounted displays (HMDs) allow users to observe virtual environments (VEs) from an egocentric perspective. However, several experiments have provided evidence that egocentric distances are perceived as compressed in VEs relative to the real world. Recent experiments suggest that the virtual view frustum set for rendering the VE has an essential impact on the user's estimation of distances. In this article we analyze if distance estimation can be improved by calibrating the view frustum for a given HMD and user. Unfortunately, in an immersive virtual reality (VR) environment, a full per user calibration is not trivial and manual per user adjustment often leads to mini- or magnification of the scene. Therefore, we propose a novel per user calibration approach with optical see-through displays commonly used in augmented reality (AR). This calibration takes advantage of a geometric scheme based on 2D point - 3D line correspondences, which can be used intuitively by inexperienced users and requires less than a minute to complete. The required user interaction is based on taking aim at a distant target marker with a close marker, which ensures non-planar measurements covering a large area of the interaction space while also reducing the number of required measurements to five. We found the tendency that a calibrated view frustum reduced the average distance underestimation of users in an immersive VR environment, but even the correctly calibrated view frustum could not entirely compensate for the distance underestimation effects.     

结合语义分割与模型匹配的室内场景重建方法

目的 由于室内点云场景中物体的密集性、复杂性以及多遮挡等带来的数据不完整和多噪声问题,极大地限制了室内点云场景的重建工作,无法保证场景重建的准确度。为了更好地从无序点云中恢复出完整的场景,提出了一种基于语义分割的室内场景重建方法。方法 通过体素滤波对原始数据进行下采样,计算场景三维尺度不变特征变换（3D scale-invariant feature transform,3D SIFT）特征点,融合下采样结果与场景特征点从而获得优化的场景下采样结果;利用随机抽样一致算法（random sample consensus,RANSAC）对融合采样后的场景提取平面特征,将该特征输入PointNet网络中进行训练,确保共面的点具有相同的局部特征,从而得到每个点在数据集中各个类别的置信度,在此基础上,提出了一种基于投影的区域生长优化方法,聚合语义分割结果中同一物体的点,获得更精细的分割结果;将场景物体的分割结果划分为内环境元素或外环境元素,分别采用模型匹配的方法、平面拟合的方法从而实现场景的重建。结果 在S3DIS （Stanford large-scale 3D indoor space dataset）数据集上进行实验,本文融合采样算法对后续方法的效率和效果有着不同程度的提高,采样后平面提取算法的运行时间仅为采样前的15%;而语义分割方法在全局准确率（overall accuracy,OA）和平均交并比（mean intersection over union,mIoU）两个方面比PointNet网络分别提高了2.3%和4.2%。结论 本文方法能够在保留关键点的同时提高计算效率,在分割准确率方面也有着明显提升,同时可以得到高质量的重建结果。     

基于注意力感知和语义感知的RGB-D室内图像语义分割算法

近年来,全卷积神经网络有效提升了语义分割任务的准确率.然而,由于室内环境的复杂性,室内场景语义分割仍然是一个具有挑战性的问题.随着深度传感器的出现,人们开始考虑利用深度信息提升语义分割效果.以往的研究大多简单地使用等权值的拼接或求和操作来融合RGB特征和深度特征,未能充分利用RGB特征与深度特征之间的互补信息.本文提出...     

Building 3D semantic maps for mobile robots using RGB-D camera

The wide availability of affordable RGB-D sensors changes the landscape of indoor scene analysis. Years of research on simultaneous localization and mapping (SLAM) have made it possible to merge multiple RGB-D images into a single point cloud and provide a 3D model for a complete indoor scene. However, these reconstructed models only have geometry information, not including semantic knowledge. The advancements in robot autonomy and capabilities for carrying out more complex tasks in unstructured environments can be greatly enhanced by endowing environment models with semantic knowledge. Towards this goal, we propose a novel approach to generate 3D semantic maps for an indoor scene. Our approach creates a 3D reconstructed map from a RGB-D image sequence firstly, then we jointly infer the semantic object category and structural class for each point of the global map. 12 object categories (e.g. walls, tables, chairs) and 4 structural classes (ground, structure, furniture and props) are labeled in the global map. In this way, we can totally understand both the object and structure information. In order to get semantic information, we compute semantic segmentation for each RGB-D image and merge the labeling results by a Dense Conditional Random Field. Different from previous techniques, we use temporal information and higher-order cliques to enforce the label consistency for each image labeling result. Our experiments demonstrate that temporal information and higher-order cliques are significant for the semantic mapping procedure and can improve the precision of the semantic mapping results.     

Visual creation of inhabited 3D environments

The creation of virtual reality applications and 3D environments is a complex task that requires good programming skills and expertise in computer graphics and many other disciplines. The complexity increases when we want to include complex entities such as virtual characters and animate them. In this paper we present a system that assists in the tasks of setting up a 3D scene and configuring several parameters affecting the behavior of virtual entities like objects and autonomous virtual humans. Our application is based on a visual programming paradigm, supported by a semantic representation, an ontology for virtual environments. The ontology allows us to store and organize the components of a 3D scene, together with the knowledge associated with them. It is also used to expose functionalities in the given 3D engine. Based on a formal representation of its components, the proposed architecture provides a scalable VR system. Using this system, non-experts can set up interactive scenarios with minimum effort; no programming skills or advanced knowledge is required.     

基于点云数据的三维目标识别和模型分割方法

三维模型的深度特征表示是三维目标识别和三维模型语义分割的关键和前提,在 机器人、自动驾驶、虚拟现实、遥感测绘等领域有着广泛的应用前景。然而传统的卷积神经网 络需要以规则化的数据作为输入,对于点云数据需要转换为视图或体素网格来处理,过程复杂 且损失了三维模型的几何结构信息。借助已有的可以直接处理点云数据的深度网络,针对产生 的特征缺少局部拓扑信息问题进行改进,提出一种利用双对称函数和空间转换网络获得更鲁棒、 鉴别力更强的特征。实验表明,通过端到端的方式很好地解决缺少局部信息问题,在三维目标 识别、三维场景语义分割任务上取得了更好的实验效果,并且相比于 PointNet++在相同精度的 情况下训练时间减少了 20%。