水彩画风格实时增强现实技术

在传统的AR系统中,虚拟物体和真实场景在视觉上存在较为明显的差异,达不到虚拟物体和真实场景无缝结合的要求。将增强现实技术与NPR有机地结合起来,减小这种视觉差异,研究并实现了水彩画风格的增强现实系统。     

The road to surgical simulation and surgical navigation

The recent advantage of the power of graphic workstations has made it possible to handle 3D human structures in an interactive way. Real-time imaging of medical 3D or 4D images can be used not only for diagnosis, but also for various novel medical treatments. By elaborating on the history of the establishment of our laboratory, which focuses on medical virtual reality, we describe our experience of developing surgery simulation and surgery navigation systems according to our research results. In the case of surgical simulation, we mention two kinds of virtual surgery simulators that produce the haptic sensation of surgical maneuvers in the user’s fingers. Regarding surgical navigation systems, we explain the necessity of the augmented reality function for the encouragement of the ability of robotic surgery and its trial for clinical case.     

A generalized registration method for augmented reality systems

In augmented reality systems, registration is one of the most difficult problems currently limiting their applications. In this paper, we propose a generalized registration method using projective reconstruction technique in computer vision. This registration method is composed of embedding and tracking. Embedding involves specifying four points to build the world coordinate system on which a virtual object will be superimposed. In this stage, any arbitrary two unrelated images or any 3×4 projective matrices with rank 3 can be used to calculate the 3D pseudo-projective coordinates of the four specified points. In the tracking process, these 3D pseudo-projective coordinates are used to track the four specified points to compute the registration matrix for augmentation. The proposed method is simple, as only four points need to be specified at the embedding stage, and the virtual object can then be easily augmented onto a real scene from a video sequence. One advantage is that the virtual objects can still be superimposed on the specified regions even when the regions are occluded in the video sequence. Another advantage of the proposed method is that the registration errors can be adjusted in real-time to ensure that they are less than certain thresholds that have been specified at the initial embedding stage. Several experiments have been conducted to validate the performance of the proposed generalized method.     

Augmented reality with automatic illumination control incorporatingellipsoidal models

In applications of augmented reality like virtual studio TV production, multisite video conference applications using a virtual meeting room and synthetic/natural hybrid coding according to the new ISO/MPEG-4 standard, a synthetic scene is mixed into a natural scene to generate a synthetic/natural hybrid image sequence. For realism, the illumination in both scenes should be identical. In this paper, the illumination of the natural scene is estimated automatically and applied to the synthetic scene. The natural scenes are restricted to scenes with nonoccluding, simple, moving, mainly rigid objects. For illumination estimation, these natural objects are automatically segmented in the natural image sequence and three-dimensionally (3-D) modeled using ellipsoid-like models. The 3-D shape, 3-D motion, and the displaced frame difference between two succeeding images are evaluated to estimate three illumination parameters. The parameters describe a distant point light source and ambient light. Using the estimated illumination parameters, the synthetic scene is rendered and mixed to the natural image sequence. Experimental results with a moving virtual object mixed into real video telephone sequences show that the virtual object appears naturally having the same shading and shadows as the real objects. Further, shading and shadow allows the viewer to understand the motion trajectory of the objects much better     

Video image target monitoring based on RNN-LSTM

Traditional image object classification and detection algorithms and strategies cannot meet the problem of video image acquisition and processing. Deep learning deliberately simulates the hierarchical structure of human brain, and establishes the mapping from low-level signals to high-level semantics, so as to achieve hierarchical feature representation of data. Deep learning technology has powerful visual information processing ability, which has become the forefront technology and domestic and international research hotspots to deal with this challenge. In order to solve the problem of target space location in video surveillance system, time-consuming and other problems, in this paper, we propose the algorithm based on RNN-LSTM deep learning. At the same time, according to the principle of OpenGL perspective imaging and photogrammetry consistency, we use 3D scene simulation imaging technology, relying on the corresponding relationship between video images and simulation images we locate the target object. In the 3D virtual scene, we set up the virtual camera to simulate the imaging processing of the actual camera, and the pixel coordinates in the video image of the surveillance target are substituted into the simulation image, next, the spatial coordinates of the target are inverted by the inverse process of the virtual imaging. The experimental results show that the detection of target objects has high accuracy, which has an important reference value for outdoor target localization through video surveillance images.

     

增强现实技术在虚拟演播室系统中的应用

虚拟演播室是虚拟现实技术和视频合成技术相结合的产物,其场景是计算机生成的三维场景,由于人们对虚拟场景和复杂度的无限要求,使得场景的实时显示十分困难,使用基于图像的绘制技术构造虚拟空间能够较好地解决这个问题。在虚拟演播室中,演员需要与三维运动虚拟物体进行交互,运用增强现实技术,可以将三维虚拟物体与基于图象绘制的虚拟场景融合在一起。     

基于多视频的虚实融合系统

提出了一种基于多视频的虚实融合可视化系统的构建方法,旨在将真实世界中的图像和视频融合到虚拟场景中,用视频图像中的纹理和动态信息去丰富虚拟场景,提高虚拟环境的真实性,得到一种增强的虚拟环境.利用无人机采集图像来重建虚拟场景,并借助图像特征点的匹配来实现视频图像的注册.然后利用投影纹理映射技术,将图像投影到虚拟场景中.视频中的动态物体由于在虚拟环境中缺失对应的三维模型,直接投影,当视点发生变化时会产生畸变.首先检测和追踪这些物体,然后尝试使用多种显示方式来解决畸变问题.此外,系统还考虑有重叠区域的多视频之间的融合.实验结果表明,所构造的虚实融合环境是十分有益的.     

基于标志点注册及探测球的光源追踪算法

增强现实系统中,解决虚拟物体表面光照情况与真实环境匹配问题关键的一步是进行光源追踪。基于标志点注册及具有漫反射特性的探测球提出一种光源追踪算法。该算法仅对一幅被单一光源照射的标志立方体和探测球图像进行分析,利用图中标志点确定探测球相对照相机的位置、姿态关系,利用探测球表面的亮度信息推算光源向量。该探测球图像要进行一系列图像处理程序,其中等亮度线的提取及拟合是关键步骤。实验结果表明,本文算法能够达到预期效果,较好地实现了光源追踪,适用于各种位置的单一光源照射情况及基于标志点注册的增强现实系统。     

增强现实中三维跟踪注册技术概述

三维跟踪注册技术是增强现实领域中的重要关键技术，通过对现实场景中的图像或物体进行跟踪与定位，将虚拟物体按照正确的空间透视关系叠加到真实场景当中。综述了增强现实中的三维跟踪注册技术（基于计算机视觉的跟踪注册技术、基于硬件传感器的跟踪注册技术以及混合跟踪注册技术），详细阐述了不同跟踪注册技术的优势与不足，并介绍了不同跟踪注册技术的当前应用现状，随后探讨了跟踪注册技术的发展趋势与存在的难题，对三维跟踪注册技术的进一步研究进行了展望。     

Fluid in Video: Augmenting Real Video with Simulated Fluids

We present a technique for coupling simulated fluid phenomena that interact with real dynamic scenes captured as a binocular video sequence. We first process the binocular video sequence to obtain a complete 3D reconstruction of the scene, including velocity information. We use stereo for the visible parts of 3D geometry and surface completion to fill the missing regions. We then perform fluid simulation within a 3D domain that contains the object, enabling one‐way coupling from the video to the fluid. In order to maintain temporal consistency of the reconstructed scene and the animated fluid across frames, we develop a geometry tracking algorithm that combines optic flow and depth information with a novel technique for “velocity completion”. The velocity completion technique uses local rigidity constraints to hypothesize a motion field for the entire 3D shape, which is then used to propagate and filter the reconstructed shape over time. This approach not only generates smoothly varying geometry across time, but also simultaneously provides the necessary boundary conditions for one‐way coupling between the dynamic geometry and the simulated fluid. Finally, we employ a GPU based scheme for rendering the synthetic fluid in the real video, taking refraction and scene texture into account.     

Stereoscopic visualization of laparoscope image using depth information from 3D model

Laparoscopic surgery is indispensable from the current surgical procedures. It uses an endoscope system of camera and light source, and surgical instruments which pass through the small incisions on the abdomen of the patients undergoing laparoscopic surgery. Conventional laparoscope (endoscope) systems produce 2D colored video images which do not provide surgeons an actual depth perception of the scene. In this work, the problem was formulated as synthesizing a stereo image of the monocular (conventional) laparoscope image by incorporating into them the depth information from a 3D CT model. Various algorithms of the computer vision including the algorithms for the feature detection, matching and tracking in the video frames, and for the reconstruction of 3D shape from shading in the 2D laparoscope image were combined for making the system. The current method was applied to the laparoscope video at the rate of up to 5 frames per second to visualize its stereo video. A correlation was investigated between the depth maps calculated with our method with those from the shape from shading algorithm. The correlation coefficients between the depth maps were within the range of 0.70–0.95 (P < 0.05). A t-test was used for the statistical analysis.     

Video mosaics for virtual environments

As computer-based video becomes ubiquitous with the expansion of transmission, storage, and manipulation capabilities, it will offer a rich source of imagery for computer graphics applications. This article looks at one way to use video as a new source of high-resolution, photorealistic imagery for these applications. If you walked through an environment, such as a building interior, and filmed a video sequence of what you saw you could subsequently register and composite the video images together into large mosaics of the scene. In this way, you can achieve an essentially unlimited resolution. Furthermore, since you can acquire the images using any optical technology, you can reconstruct any scene regardless of its range or scale. Video mosaics can be used in many different applications, including the creation of virtual reality environments, computer-game settings, and movie special effects. I present algorithms that align images and composite scenes of increasing complexity-beginning with simple planar scenes and progressing to panoramic scenes and, finally, to scenes with depth variation. I begin with a review of basic imaging equations and conclude with some novel applications of the virtual environments created using the algorithms presented     

基于Kinect的民族弓弦乐器虚拟演奏系统

利用Kinect相机结合增强现实技术和手势识别方法设计并实现了一个弓弦乐器虚拟演奏系统——以二胡为例.将Kinect获取的现实场景和虚拟乐器融合在一起绘制成增强现实场景.通过Kinect得到的深度数据和贝叶斯肤色模型将用户的左手分割出来,并再次绘制在增强图像上形成新的图像,从而解决虚拟演奏场景中的虚实遮挡问题.利用基于反向动力学和马尔可夫模型的三维虚拟手势拟合方法,对演奏过程中的左手手势进行识别,并结合右手的运动状态完成乐器的虚拟演奏.     

复杂机电产品线缆虚实融合装配体系构建及其关键技术

针对传统的基于实物样机的线缆模拟装配实验方法造成的效率低、精确度低、装配 质量和可靠性难以得到保证等问题,提出一种面向复杂机电产品的线缆虚实融合装配方法。首先 在分析了增强现实技术应用于线缆装配的优势基础上,构建了基于增强现实的线缆虚实融合装配 体系;其次,对装配体系中涉及到的关键技术进行了介绍,重点对线缆三维注册技术、线缆虚实 遮挡技术、线缆虚实光照技术以及相机标定与跟踪技术等关键技术进行了阐述;最后介绍了线缆 虚实融合装配研究内容,给出了基于增强现实的线缆虚实融合装配系统运行流程。     

A new desk-top encounter-type haptic device with an actively driven pen-tablet LCD panel

In conventional haptic devices for virtual reality (VR) systems, a user interacts with a scene by handling a tool (such as a pen) using a mechanical device (i.e. an end-effector-type haptic device). In the case that the device can ‘mimic’ a VR object, the user can interact directly with the VR object without the mechanical constraint of a device (i.e. an encounter-type haptic device). A new challenge of an encounter-type haptic device is displaying the visuals and haptic information simultaneously on a single device. We are proposing a new desk-top encounter-type haptic device with an actively driven pen-tablet LCD panel. The proposed device is capable of providing pseudo-3D visuals and haptic information on a single device. As the result, the system provides to the user a sense of interaction with a real object. To develop a proof-of-concept prototype, a compact parallel mechanism was developed and implemented. The aim of this research is to propose a new concept in haptic research. In this paper, the concept, the prototype, and some preliminary evaluation tests with the proposed system are presented.     

面向上颌骨骨折复位手术的虚拟系统设计

基于3D计算机触觉视觉交互(CHAI3D)和开放图形库(Open GL)等开源软件,设计了针对上颌骨复位手术的仿真系统。使用真实病例的CT图像搭建虚拟场景,通过Geomagic力反馈设备对虚拟模型进行三维操作并输出触觉反馈。在原有单点碰撞算法的基础上,提出了使用多个中介代理的多点碰撞算法,避免了虚拟手术工具的手柄插入虚拟器官的不实仿真;通过力反馈设备对头颅骨模型进行选择、移动和旋转,模拟手术中对头颅骨的移动和放置。系统可用于训练医学院学生,也可用于复杂手术的术前规划。     

基于图像分解的光照估计算法

增强现实技术的目的在于将计算机生成的虚拟物体叠加到真实场景中.实现良好的虚实融合需要对场景光照进行估算,针对高光场景,利用场景中的不同反射光信息对场景进行有效的光照估计,首先通过基于像素聚类方法的图像分解对图像进行反射光的分解,得到漫反射图和镜面反射图,对漫反射图进行进一步的本征图像分解,得到反照率图和阴影图;之后结合...     

基于图像的室内虚拟环境的研究

基于图像的建模和绘制技术,提出了一个构造室内虚拟环境的完整方案,用户只需要输入少数照片,即可重建室内场景的全景图像,方案主要包括以下几点：首先由用户交互确定图像中的匹配象素,通过运动分析算法恢复整个场景的几何结构,然后,将原始图像变换至平面的参数坐标系,抽取纹理图像,并在参数空间对纹理图像进行拼接;最后生成场景的全景图像,算法对拍摄条件和设备没有苛刻要求,运算量较小,有较强的稳定性。     

虚拟环境建模及实时性改善方法

提出了虚拟现实软件实现方法的概念,利用交互方式的,比较,阐明了虚拟现实与仿真的区别,通过计算机三维实体造型和真实感环境泻染技术,实现了虚拟环境建模,通过融入增强现实的思想,把握虚实结合尺度,将基于图象的方法与基于建模的相结合,从而改善了眯拟环境下交互的实时性。     

Haptics in minimally invasive surgical simulation and training

Haptics is a valuable tool in minimally invasive surgical simulation and training. We discuss important aspects of haptics in MISST, such as haptic rendering and haptic recording and playback. Minimally invasive surgery has revolutionized many surgical procedures over the last few decades. MIS is performed using a small video camera, a video display, and a few customized surgical tools. In procedures such as gall bladder removal (laparoscopic cholesystectomy), surgeons insert a camera and long slender tools into the abdomen through small skin incisions to explore the internal cavity and manipulate organs from outside the body as they view their actions on a video display. Because the development of minimally invasive techniques has reduced the sense of touch compared to open surgery, surgeons must rely more on the feeling of net forces resulting from tool-tissue interactions and need more training to successfully operate on patients.