3D augmented reality teleoperated robot system based on dual vision

A 3D augmented reality navigation system using stereoscopic images is developed for teleoperated robot systems.The accurate matching between the simulated model and the video image of the actual robot can be realized,which helps the operator to accomplish the remote control task correctly and reliably.The system introduces the disparity map translation transformation method to take parallax images for stereoscopic displays,providing the operator an immersive 3D experience.Meanwhile,a fast and accurate registration method of dynamic stereo video is proposed,and effective integration of a virtual robot and the real stereo scene can be achieved.Preliminary experiments show that operation error of the system is maintained at less than 2.2 mm and the average error is 0.854 7,0.909 3 and 0.697 2 mm at x,y,z direction respectively.Lots of experiments such as pressing the button,pulling the drawer and so on are also conducted to evaluate the performance of the system.The feasibility studies show that the depth information of structure can be rapidly and recognized in remote environment site.The augmented reality of the image overlay system could increase the operating accuracy and reduce the procedure time as a result of intuitive 3D viewing.     

An original mechatronic design of a kinaesthetic hand exoskeleton for virtual reality-based applications

Within the new industrial era, the interaction between humans and virtual reality is spreading across our lives. The development of exoskeleton designed to enhance the immersivity of virtual reality environments has a potentially considerable social impact and arises as a hot research topic. The presented work dwells well with the subject by describing the mechatronic design process of a kinaesthetic hand exoskeleton system meant to reproduce proprioceptive stimuli coming from the interaction with a virtual reality. The presented prototype is a modular device, equipped with force and pose sensors, and driven by a Bowden-cable-based remote actuation system. Unlike similar devices, the proposed exoskeleton is specifically thought for VR interaction and is designed to be reversible while exerting up to 15 N per finger. For a more accurate rendering of kinetostatic finger stimuli, a procedure for reconstructing HMI force as a function of measured force and position signals by employing a system’s kinematic and dynamic model is presented, detailed, and followed by some preliminary tests. The results showed that the model can trace forces back to the end-effector with a percentage error below 15%.     

基于AR技术的实时互动体验电商系统

电商已经不仅仅满足于线上的商品交易,越来越多地开始关注于线下用户体验的改善.本文以虚拟试鞋为切入口,借助于Kinect硬件设备和3d Max、unity3d软件平台,提出并搭建了基于AR增强现实技术的实时互动体验电商系统.本文对该系统的关键技术进行阐述,包括三维模型的跟踪定位、手势交互和RGB-D抠像.该系统综合应用三维场景虚拟化、三维互动识别以及增强现实融合等技术,硬件成本和算法复杂度低,操作简单,适用于新型电商利用AR技术对品牌进行线上展示和线下用户体验.     

Learning latent geometric consistency for 6D object pose estimation in heavily cluttered scenes

6D object pose (3D rotation and translation) estimation from RGB-D image is an important and challenging task in computer vision and has been widely applied in a variety of applications such as robotic manipulation, autonomous driving, augmented reality etc. Prior works extract global feature or reason about local appearance from an individual frame, which neglect the spatial geometric relevance between two frames, limiting their performance for occluded or truncated objects in heavily cluttered scenes. In this paper, we present a dual-stream network for estimating 6D pose of a set of known objects from RGB-D images. Our novelty stands in contrast to prior work that learns latent geometric consistency in pairwise dense feature representations from multiple observations of the same objects in a self-supervised manner. We show in experiments that our method outperforms state-of-the-art approaches on 6D object pose estimation in two challenging datasets, YCB-Video and LineMOD.     

Wearable Sensors-Enabled Human–Machine Interaction Systems: From Design to Application

In comparison to traditional bulky and rigid electronic devices, the human–machine interaction (HMI) system with flexible and wearable components is an inevitable future trend. To achieve effective, intuitive, and seamless manipulation of high-performance wearable HMI systems, it is important to develop effective strategies for designing material microstructures on flexible sensors and electric devices with excellent mechanical flexibility and stretchability. The real-time acquisition of human physiology and surrounding signals through accurate and flexible sensors is the basis of wearable HMIs. Herein, the construction of a wearable HMI system that utilizes sensors, communication modes, and actuators is reviewed. The mechanisms and strategies for designing various flexible sensors based on different mechanisms are analyzed and discussed. The functional mechanism, material selection, and novel design strategies of each part are summarized in detail. The different communication modes in interactive systems and the manufacturing technology of soft machines are also introduced. Additionally, the most advanced applications of wearable HMI systems in intelligent identification and security, interactive controls for robots, augmented reality, and virtual reality have been highlighted. The review concludes with an overview of the remaining key challenges and several ideas regarding the further improvement of wearable HMI systems.     

Emerging Thermal Technology Enabled Augmented Reality

In the past decade, remarkable progress has been made in the domain of augmented reality/virtual reality (AR/VR). The need for realistic and immersive augmentation has propelled the development of haptics interfaces-enabled AR/VR. The haptics interfaces facilitate direct interaction and manipulation with both real and virtual objects, thus augmenting the perception and experiences of the users. The level of augmentation can be significantly improved by thermal stimulation or sensing, which facilitates a higher degree of object identification and discrimination. This review discusses the thermal technology-enabled augmented reality and summarizes the recent progress in the development of different thermal technology such as thermal haptics including thermo-resistive heater and Peltier devices, thermal sensors including resistive, pyroelectric, and thermoelectric sensors, which can be utilized to improve the realism of augmentation. The fundamental mechanism, design strategies, and the rational guidelines for the adoption of these technologies in AR/VR is explicitly discussed. The conclusion provides an outlook on the existing challenges and outlines the future roadmap for the realization of next-generation thermo-haptics enabled augmented reality.     

基于增强现实的文物模型展示系统设计

基于移动设备快速发展,但文物浏览方式单一的现状,文中提出了以 Unity3D 为开发平台,应用C#开发语言,以 ARFoundation 为接口,连接 ARKit 软件开发包,构建增强现实平台,并设计开发了一款基于增强现实技术的文物模型展示系统。该系统可将虚拟的文物模型叠加放置到真实场景中,用户可通过点击屏幕上的真实场景平面改变模型位置,双指点击模型并旋转来改变模型方向,完成人机交互。同时,用户可通过移动自己的设备来多角度观察文物。增强现实技术与文物的结合,能为观赏者提供新奇的浏览体验,激发游览者的兴趣,为需要特殊保存的文物提供新的展出思路。     

移动增强现实应用中“毁伤”功能的实现

文中介绍了基于移动终端的增强现实应用“毁伤”功能。首先介绍“毁伤”的概念,接着介绍了增强现实应用开发过程中“毁伤”功能的抽象模型、构造方法与对应的实现方式,借助Unity3D引擎与Vuforia增强现实插件在Android移动设备上进行实验,最后给出了一种实现“毁伤”功能的具体方法。     

Elastomeric Haptic Devices for Virtual and Augmented Reality

Since the modern concepts for virtual and augmented reality are first introduced in the 1960's, the field has strived to develop technologies for immersive user experience in a fully or partially virtual environment. Despite the great progress in visual and auditory technologies, haptics has seen much slower technological advances. The challenge is because skin has densely packed mechanoreceptors distributed over a very large area with complex topography; devising an apparatus as targeted as an audio speaker or television for the localized sensory input of an ear canal or iris is more difficult. Furthermore, the soft and sensitive nature of the skin makes it difficult to apply solid state electronic solutions that can address large areas without causing discomfort. The maturing field of soft robotics offers potential solutions toward this challenge. In this article, the definition and history of virtual (VR) and augmented reality (AR) is first reviewed. Then an overview of haptic output and input technologies is presented, opportunities for soft robotics are identified, and mechanisms of intrinsically soft actuators and sensors are introduced. Finally, soft haptic output and input devices are reviewed with categorization by device forms, and examples of soft haptic devices in VR/AR environments are presented.     

Effect of Interaction Based on Augmented Context in Immersive Virtual Reality Environment

Virtual reality has recently rapidly emerged into the global spotlight. With the current increasing interest in head mounted display (HMD) as the next-generation content platform to replace smartphones, many companies are endeavoring to gain an early foothold in the virtual reality market. In HMD-based virtual reality environments, users are completely isolated from the real world, and are thus unable to perform even simple and familiar interactions with the real world. This paper proposes interaction based on context information augmented with stories, visuals, and sounds, as opposed to an interface that replicates the existing real world, in order to enable easy and natural interactions by users in HMD-based immersive virtual reality environments. The results of a user study conducted using our proposed augmented context-based interactions in an immersive virtual reality environment verify that it provides users with higher levels of immersion, accuracy, and emotional empathy than a virtual reality environment that simply replicates the existing real world.     

Registration algorithm based on image matching for outdoor AR system with fixed viewing position

An image registration method based on the Fourier-Mellin transform is introduced for an outdoor augmented reality (AR) system. For this type of AR system, the observation position is fixed, and a complex 3-D registration problem can be reduced to a 2-D image registration for this fixed viewing position system. An observation globe model for this method is proposed. Under this supposition, a Fourier-Mellin transform is used in image registration, and the architecture of this system is illustrated. Experimental results show that this image registration algorithm is accurate and robust. It is effective for an outdoor AR system with a fixed viewing position.     

Mobile Real-Time Collaboration for Semantic Multimedia

Advanced mobile applications that enable new ways of interaction with digital objects become increasingly important for on-site professional communities. These new ways of interaction, e.g. in Mobile Augmented Reality (MAR) via position and 3D movement, are real needs for fieldwork domains such as cultural heritage management and the construction industry. In addition, on-site professional communities generate shared knowledge bases with multimedia content and semantic annotations through collaboration. However, current MAR applications lack real-time collaboration features. In practice, blending multimedia semantics in mobile real-time collaboration is challenging due to the limitations of mobile devices, the lack of mature dedicated designed communication infrastructures and the constraints of the remote environments. This paper presents a mobile real-time collaboration system for semantic multimedia annotations with augmented reality features. We use XMPP as a real-time protocol for the secure, scalable and interoperable processing of XML-based semantic multimedia metadata described in MPEG-7. Our prototype was evaluated in the digital documentation of historical sites for cultural heritage management. The evaluation results indicate potential for increased productivity and enhanced mutual awareness in on-site professional communities.     

Augmented System for Immersive 3D Expansion and Interaction

In the field of augmented reality technologies, commercial optical see‐through‐type wearable displays have difficulty providing immersive visual experiences, because users perceive different depths between virtual views on display surfaces and see‐through views to the real world. Many cases of augmented reality applications have adopted eyeglasses‐type displays (EGDs) for visualizing simple 2D information, or video see‐through‐type displays for minimizing virtual‐ and real‐scene mismatch errors. In this paper, we introduce an innovative optical see‐through‐type wearable display hardware, called an EGD. In contrast to common head‐mounted displays, which are intended for a wide field of view, our EGD provides more comfortable visual feedback at close range. Users of an EGD device can accurately manipulate close‐range virtual objects and expand their view to distant real environments. To verify the feasibility of the EGD technology, subject‐based experiments and analysis are performed. The analysis results and EGD‐related application examples show that EGD is useful for visually expanding immersive 3D augmented environments consisting of multiple displays.     

Simulating 3-D lung dynamics using a programmable graphics processing unit.

Medical simulations of lung dynamics promise to be effective tools for teaching and training clinical and surgical procedures related to lungs. Their effectiveness may be greatly enhanced when visualized in an augmented reality (AR) environment. However, the computational requirements of AR environments limit the availability of the central processing unit (CPU) for the lung dynamics simulation for different breathing conditions. In this paper, we present a method for computing lung deformations in real time by taking advantage of the programmable graphics processing unit (GPU). This will save the CPU time for other AR-associated tasks such as tracking, communication, and interaction management. An approach for the simulations of the three-dimensional (3-D) lung dynamics using Green's formulation in the case of upright position is taken into consideration. We extend this approach to other orientations as well as the subsequent changes in breathing. Specifically, the proposed extension presents a computational optimization and its implementation in a GPU. Results show that the computational requirements for simulating the deformation of a 3-D lung model are significantly reduced for point-based rendering.     

基于增强现实的硬件维护系统研究与设计

传感器、多媒体、机器学习等技术的快速发展,促进了增强现实在多领域得到普及和应用。航空设备硬件运行维护也是一个重要领域,可以基于增强现实构建一个硬件维护系统,渲染一个虚拟的硬件运行环境,模拟实际的故障发生情况,加强对硬件设备的装配和维修,进一步提高航空设备硬件运行维护的水平。本文详细地描述了增强现实在硬件维护系统中的应用功能,分别是虚拟环境仿真功能、装配维护功能、3D模型设计功能,同时构建一个强大的硬件维护系统,该系统沉浸感强、娱乐性足,能够激发硬件维护人员的学习和应用兴趣,具有重要的作用和意义。     

融合特征法与直接法的RGB-D稠密视觉SLAM算法

为了保持直接法的快速性与特征法的高精度和闭环能力,提出了一种融合直接法与特征法的RGB-D同时定位与地图创建(SLAM)算法。该算法主要包含3个并行线程:跟踪线程、局部建图线程和闭环线程。在跟踪线程中对非关键帧进行跟踪,通过最小化光度图像误差来进行相机的初始位姿估计以及像素点的对应关系计算,利用最小化局部地图点重投影误差进一步优化相机位姿,实现快速准确的跟踪与定位;在局部建图线程中对关键帧进行提取并匹配ORB特征,执行局部BA(光束平差法),对局部关键帧位姿和局部地图点的位置进行优化,提高SLAM的局部一致性;在闭环线程中执行对关键帧的闭环检测和优化,从而保证SLAM全局一致性。另外,根据RGB-D图像和相机位姿信息,通过基于Octomap的建图框架,构建完整准确的3D稠密环境地图。在TUM数据集下的实验表明,所提出的方法可以得到与基于特征法相当的精度,且所需时间更少。     

一种高效的分布式多传感器多目标跟踪算法

现有的多传感器多目标跟踪算法大都基于马尔科夫-贝叶斯模型,需要诸如目标运动、杂波、传感器检测概率等先验信息,但是在恶劣的环境中,这些先验信息不准确并导致目标跟踪精度下降.为了解决该情况下的多目标跟踪问题,我们提出了一个高效的分布式多目标跟踪算法,该算法通过泛洪(Flooding)共识算法在分布式网络的传感器之间迭代的传...     

Consideration on three-dimensional visual communication systems

A novel head-tracking technique for a full-color stereoscopic video display system is presented. Using this head-tracking system, the stereoscopic area is increased by a factor of three and stabilized stereoscopic images are observed. Two kinds of large-size rear-projection stereoscopic display systems that do not require special glasses are proposed. Experimental tests, confirming the effectiveness of the proposed systems, are described. By incorporating these techniques into visual communication systems, such as TV telephone systems, video conference systems, or virtual reality systems, a very natural and accurate depth sensation which cannot be obtained by conventional systems that require the viewer to wear special glasses may be possible     

IEC立体显示终端测量方法综述

介绍了IEC起草的佩戴眼镜式的立体显示终端测量方法草案。立体显示测量包括亮度、对比度、色域及两眼间串扰等。两眼间串扰是当前研究最为关心的参数,该草案提出了灰阶串扰测量方法。最后总结了我国制定立体显示测量行业标准的进展情况。