Stereoscopic video images for telerobotic applications

This article addresses the use of stereoscopic images in teleoperated tasks. Depth perception is a key point in the ability to skillfully manipulate in remote environments. Displaying three‐dimensional images is a complex process but it is possible to design a teleoperation interface that displays stereoscopic images to assist in manipulation tasks. The appropriate interface for image viewing must be chosen and the stereoscopic video cameras must be calibrated so that the image disparity is natural for the observer. Attention is given to the calculation of stereoscopic image disparity, and suggestions are made as to the limits within which adequate stereoscopic image perception takes place. The authors have designed equipment for image visualization in teleoperated systems. These devices are described and their performance evaluated. Finally, an architecture for the transmission of stereoscopic video images via network is proposed, which in the future will substitute for current image processing devices. © 2005 Wiley Periodicals, Inc.     

考虑双目竞争视觉现象的非对称失真立体图像质量评价方法

目的 现有方法存在特征提取时间过长、非对称失真图像预测准确性不高的问题,同时少有工作对非对称失真与对称失真立体图像的分类进行研究,为此提出了基于双目竞争的非对称失真立体图像质量评价方法。方法 依据双目竞争的视觉现象,利用非对称失真立体图像两个视点的图像质量衰减程度的不同,生成单目图像特征的融合系数,融合从左右视点图像中提取的灰度空间特征与HSV （hue-saturation-value）彩色空间特征。同时,量化两个视点图像在结构、信息量和质量衰减程度等多方面的差异,获得双目差异特征。并且将双目融合特征与双目差异特征级联为一个描述能力更强的立体图像质量感知特征向量,训练基于支持向量回归的特征—质量映射模型。此外,还利用双目差异特征训练基于支持向量分类模型的对称失真与非对称失真立体图像分类模型。结果 本文提出的质量预测模型在4个数据库上的SROCC （Spearman rank order correlation coefficient）和PLCC （Pearson linear correlation coefficient）均达到0.95以上,在3个非对称失真数据库上的均方根误差（root of mean square error,RMSE）取值均优于对比算法。在LIVE-II（LIVE 3D image quality database phase II）、IVC-I（Waterloo-IVC 3D image qualityassessment database phase I）和IVC-II （Waterloo-IVC 3D image quality assessment database phase II）这3个非对称失真立体图像测试数据库上的失真类型分类测试中,对称失真立体图像的分类准确率分别为89.91%、94.76%和98.97%,非对称失真立体图像的分类准确率分别为95.46%,92.64%和96.22%。结论 本文方法依据双目竞争的视觉现象融合左右视点图像的质量感知特征用于立体图像质量预测,能够提升非对称失真立体图像的评价准确性和鲁棒性。所提取双目差异性特征还能够用于将对称失真与非对称失真立体图像进行有效分类,分类准确性高。     

基于单目和双目视觉信息的全参考立体图像质量评价模型

在立体图像质量评价领域,有效地模拟人类视觉系统对图像质量进行评价具有重要意义,考虑到人眼的视觉感知特性,基于单目和双目视觉信息构建一种立体图像质量评价模型MB-FR-SIQA。采用基于结构相似性的立体视差算法得到参考和失真立体图像的视差矩阵,结合Gabor能量响应图、显著性图和视差矩阵生成中间视图,并优化左右眼加权系数计算方法,以提高生成中间视图的准确性。分别利用单目图像和中间视图提取单目和双目视觉信息,计算单目质量分数和双目质量分数,并融合得到立体图像的质量分数,达到评价立体图像质量的目的。实验结果表明,MB-FR-SIQA模型在LIVE-I数据库上具有较高的预测精度,其斯皮尔曼等级相关系数、皮尔森线性相关系数、均方根误差分别为0.945、0.951、5.318,且预测的质量分数符合人类主观评估。     

VR-Based Teleoperation for Robot Compliance Control

Robots governed by remote human operators are excellent candidates for work in hazardous or uncertain environments such as nuclear plants or outer space. For successful teleoperation, it is important to let the operator feel physically present at the remote site. When the telerobotic system is used to execute compliance tasks in which simultaneous control of both position and force may be demanded and inevitable contact with environments is encountered, information about the interactions between the robot manipulator and the environment are especially crucial for the operator to make proper decisions. This paper proposes a VR-based telerobotic system for such compliance tasks. The proposed system provides both visual and haptic information. A local intelligence controller, capable of surface tracking and force regulation, is equipped on the robot manipulator to tackle the time-delay problem usually present in teleoperation and to share control load with the operator. The proposed telerobotic system is developed in a virtual environment due to recent gains in the capabilities and popularity of virtual reality to generate realistic telepresence. Experiments based on the surface-tracking and peg-in-hole compliance tasks demonstrate the effectiveness of the proposed system.     

Stereo Viewing and Virtual Reality Technologies in Mobile Robot Teleguide

The use of 3-D stereoscopic visualization may provide a user with higher comprehension of remote environments in teleoperation when compared with 2-D viewing, in particular, a higher perception of environment depth characteristics, spatial localization, remote ambient layout, faster system learning, and decision performance. Works in the paper have demonstrated how stereo vision contributes to the improvement of the perception of some depth cues, often for abstract tasks, while it is hard to find works addressing stereoscopic visualization in mobile robot teleguide applications. This paper intends to contribute to this aspect by investigating the stereoscopic robot teleguide under different conditions, including typical navigation scenarios and the use of synthetic and real images. This paper also investigates how user performance may vary when employing different display technologies. Results from a set of test trials run on seven virtual reality systems, from laptop to large panorama and from head-mounted display to Cave automatic virtual environment (CAVE), emphasized few aspects that represent a base for further investigations as well as a guide when designing specific systems for telepresence.      

Cognitive processes in complex tasks: introduction and discussion

This paper points out that the fusional limits of binocular stereoscopic viewing are an important factor, not only determining the reproduction range of stereoscopic images, but also the conformity between the stereoscopic display and stereoscopic vision. Experimental results showed that fusional limits increase in proportion to the field of view angle, and that they are affected not only by the size of the viewing target, but also by the effects that the environment has on the target. These findings explain the differences between binocular vision in real space and binocular vision in a stereoscopic display. Finally, the conditions under which observers arc able to view images of stereoscopic displays without excessive visual strain are discussed.     

A Portable Bio-Inspired Architecture for Efficient Robotic Vergence Control

In stereoscopic vision, the ability of perceiving the three-dimensional structure of the surrounding environment is subordinated to a precise and effective motor control for the binocular coordination of the eyes/cameras. If, on the one side, the binocular coordination of camera movements is a complicating factor, on the other side, a proper vergence control, acting on the binocular disparity, facilitates the binocular fusion and the subsequent stereoscopic perception process. In real-world situations, an effective vergence control requires further features other than real time capabilities: real robot systems are indeed characterized by mechanical and geometrical imprecision that affect the binocular vision, and the illumination conditions are changeable and unpredictable. Moreover, in order to allow an effective visual exploration of the peripersonal space, it is necessary to cope with different gaze directions and provide a large working space. The proposed control strategy resorts to a neuromimetic approach that provides a distributed representation of disparity information. The vergence posture is obtained by an open-loop and a closed-loop control, which directly interacts with saccadic control. Before saccade, the open-loop component is computed in correspondence of the saccade target region, to obtain a vergence correction to be applied simultaneously with the saccade. At fixation, the closed-loop component drives the binocular disparity to zero in a foveal region. The obtained vergence servos are able to actively drive both the horizontal and the vertical alignment of the optical axes on the object of interest, thus ensuring a correct vergence posture. Experimental tests were purposely designed to measure the performance of the control in the peripersonal space, and were performed on three different robot platforms. The results demonstrated that the proposed approach yields real-time and effective vergence camera movements on a visual stimulus in a wide working range, regardless of the illumination in the environment and the geometry of the system.     

Vision-based virtual force guidance for tele-robotic system

In order to improve operator performance and understanding within remote environment, a vision-based virtual forced guidance control methodology for tele-robotic system is presented. The remote operation of the construction robot is achieved by manipulating the graphic robot in a virtual environment. Based on binocular vision, the ground surface is modeled as an elevation map, and the task objects are recognized from video images and reconstructed using the Power Crust algorithm. The virtual guidance forces consisting of a pair of attractive force and repulsive force from the objects and obstacles are used to enhance the multi-task manipulation of the tele-robotic system.     

Stereoscopic image discomfort prediction using dual-stream multi-level interactive network

Existing stereoscopic image discomfort prediction methods may fail to work well because they are difficult to extract discomfort features from stereoscopic image’s statistical information since the mechanism of human binocular vision is very complex. In this work, we propose a dual-stream multi-level interactive network that is completely end-to-end trainable for stereoscopic image discomfort prediction. This method first extracts multi-level fusion and difference features from stereoscopic images through a multi-level interaction network. Then, the low-, medium- and high-level feature maps are concatenated to simulate the complicated visual interaction mechanism of the human visual system (HVS). Finally, two fully connected layers are used as a non-linear regression function that maps the feature vectors to stereoscopic image discomfort scores. Extensive experiments demonstrate that our approach performs favorably against the existing prediction models on the IEEE-SA dataset and NBU-S3D dataset.     

利用远程网络技术的机器人遥操作系统分析

本文系统地讨论了如何构建一个基于远程网络（Int ernet）技术的机器人遥操作系统．通过描述此系统,指出了构建系统中所遇到的问题,并 提出了解决此问题的基本策略与方法．     

Visual comfort enhancement study based on visual attention detection for stereoscopic displays

Although numerous potential causes may lead to visual discomfort when viewing content on three‐dimensional (3D) displays, vergence–accommodation conflict is a particular cause of binocular parallax‐based stereoscopic displays, and it is unavoidable. Based on the study of 3D content visual attention, we proposed a novel stereoscopic depth adjustment method to improve the visual comfort and enhance perceived naturalness. The proposed method combined the 3D image saliency and specific viewing condition to establish a novel model for computing the optimum zero‐disparity plane of stereoscopic image. The results of perception experiments, focused on visual comfort and stereoscopic sensation, supported that the proposed method can significantly enhance stereoscopic viewing comfort and even can improve the stereoscopic sensation by insuring the 3D image fusion.     

用户多维感知的3D图像体验质量评价

目的 符合用户视觉特性的3维图像体验质量评价方法有助于准确、客观地体现用户观看3D图像或视频时的视觉感知体验,从而给优化3维内容提供一定的思路。现有的评价方法仅从图像失真、深度感知和视觉舒适度中的一个维度或两个维度出发对立体图像进行评价,评价结果的准确性有待进一步提升。为了更加全面和准确地评价3D图像的视觉感知体验,提出了一种用户多维感知的3D图像体验质量评价算法。方法 首先对左右图像的差异图像和融合图像提取自然场景统计参数表示失真特征;然后对深度图像提取敏感区域,对敏感区域绘制失真前后深度变换直方图,统计深度变化情况以及利用尺度不变特征变换（SIFT）关键点匹配算法计算匹配点数目,两者共同表示深度感知特征;接下来对视觉显著区域提取视差均值、幅值表示舒适度特征;最后综合考虑图像失真、深度感知和视觉舒适度3个维度特征,将3个维度特征归一化后联合成体验质量特征向量,采用支持向量回归（SVR）训练评价模型,并得到最终的体验质量得分。结果 在LIVE和Waterloo IVC数据库上的实验结果表明,所提出的方法与人们的主观感知的相关性达到了0.942和0.858。结论 该方法充分利用了立体图像的特性,评价结果优于比较的几种经典算法,所构建模型的评价结果与用户的主观体验有更好的一致性。     

Vision Enhancement Using Stereoscopic Telepresence For Remotely Operated Underwater Robotic Vehicles

Remotely operated Underwater Robotic Vehicles (URV) are widely used for subsea tasks such as cleaning, repair and inspection of long objects such as pipelines. The ability to perform such inspections by a remotely located human pilot, over extended periods, is highly desirable. Due to underwater currents and the inability to maintain a fixed distance from the object, produces a constantly varying image for the observer. Incorporation of stereoscopic imaging, with the option of maintaining a constant image size, was perceived to be desirable. This effect was implemented by dynamic compensation of the camera zoom, and shown to negate the negative motion effects and allowed the operator to perform close inspection for extended periods of time. This paper describes the design, structure, and preliminary evaluation of an experimental Telepresence Viewing System (TVS) for Underwater Robotic Vehicle (URV) to enhance the observation capabilities of its remote pilot during an inspection task. The Telepresence Viewing System enables an automatic projection of suitably prepared and adapted underwater stereoscopic video images from the video cameras of the URV, into the operator’s visual field. The processed images aid the coupling of the operator’s vestibular and kinesthetic sensations to the visual information from the URV.     

Markerless human–robot interface for dual robot manipulators using Kinect sensor

Remote teleoperation of robot manipulators is often necessary in unstructured, dynamic, and dangerous environments. However, the existing mechanical and other contacting interfaces require unnatural, or hinder natural, human motions. At present, the contacting interfaces used in teleoperation for multiple robot manipulators often require multiple operators. Previous vision-based approaches have only been used in the remote teleoperation for one robot manipulator as well as require the special quantity of illumination and visual angle that limit the field of application. This paper presents a noncontacting Kinect-based method that allows a human operator to communicate his motions to the dual robot manipulators by performing double hand–arm movements that would naturally carry out an object manipulation task. This paper also proposes an innovative algorithm of over damping to solve the problem of error extracting and dithering due to the noncontact measure. By making full use of the human hand–arm motion, the operator would feel immersive. This human–robot interface allows the flexible implementation of the object manipulation task done in collaboration by dual robots through the double hand–arm motion by one operator.     

Telerobotic systems design based on real‐time CORBA

A new class of telerobotic applications is making its way into research laboratories, fine arts or science museums, and industrial installations. Virtual laboratories and remote equipment maintenance are examples of these applications, which are built exploiting distributed computing systems and Internet technologies. Distributed computing technologies provide several advantages to telerobotic applications, such as dynamic and multiuser access to remote resources and arbitrary user locations. Nonetheless, building these applications remains a substantial endeavor, especially when performance requirements must be met. The aim of this paper is to investigate how mainstream and advanced features of the CORBA object‐oriented middleware can be put to work to meet the requirements of novel telerobotic applications. We show that Real‐Time CORBA extensions and asynchronous method invocation of CORBA services can be relied upon to meet performance and functional requirements, thereby enabling teleoperation on local area networks. Furthermore, CORBA services for concurrency control and large‐scale data distribution enable geographic‐scale access for robot teleprogramming. Limitations in the currently available implementations of the CORBA standard are also discussed, along with their implications. The effectiveness and suitability for telerobotic applications of several CORBA mechanisms are tested first individually and then by means of a software framework exploiting CORBA services and ensuring component‐based development, software reuse, low development cost, fully portable real‐time and communication support. A comprehensive telerobotic application built based on the framework is described in the paper and evaluated on both local and wide area networks. The application includes a robot manipulator and several sensory subsystems under concurrent access by multiple competing or collaborating operators, one of which is equipped with a multimodal user interface acting as the master device. © 2005 Wiley Periodicals, Inc.     

Motion and stereoscopic tilt perception

Abstract— Stereoscopic perception of tilt about a vertical or horizontal axis is influenced by size and shear disparities, respectively. Other researchers have reported that, under certain conditions, stereoscopic perception deficits occur when the dots in a random-dot stereogram move at a velocity that produces optokinetic nystagmus. Here we examine how size disparity and shear disparity affect stereoscopic tilt perception under various motion conditions. We hypothesized that visual stimulus motion may interact with these disparities to affect tilt perception. Our results indicate that shear disparity and size disparity effects under static conditions are maintained under motion conditions. A possible explanation for the conflict between the current and previous results is discussed, as are implications for binocular head-mounted display applications.     

基于JAVA-互联网环境的移动机器人远程操作接口

以互联网为应用环境，开发了一个基于B／S模式的移动机器人远程操作控制接口，系统包括浏览器接口管理和服务器端操作接口管理两部分，采用JAVA Applet的技术，通过合理设计服务器端网页控制和JAVA程序，实现机器人状态、图像监视和直接行为控制以及路径设定控制。     

基于视差重映射的立体图像视觉舒适度提升

目的 针对人眼观看立体图像内容可能存在的视觉不舒适性,基于视差对立体图像视觉舒适度的影响,提出了一种结合全局线性和局部非线性视差重映射的立体图像视觉舒适度提升方法。方法 首先,考虑双目融合限制和视觉注意机制,分别结合空间频率和立体显著性因素提取立体图像的全局和局部视差统计特征,并利用支持向量回归构建客观的视觉舒适度预测模型作为控制视差重映射程度的约束;然后,通过构建的预测模型对输入的立体图像的视觉舒适性进行分析,就欠舒适的立体图像设计了一个两阶段的视差重映射策略,分别是视差范围的全局线性重映射和针对提取的潜在欠舒适区域内视差的局部非线性重映射;最后,根据重映射后的视差图绘制得到舒适度提升后的立体图像。结果 在IVY Lab立体图像舒适度测试库上的实验结果表明,相较于相关有代表性的视觉舒适度提升方法对于欠舒适立体图像的处理结果,所提出方法在保持整体场景立体感的同时,能更有效地提升立体图像的视觉舒适度。结论 所提出方法能够根据由不同的立体图像特征构建的视觉舒适度预测模型来自动实施全局线性和局部非线性视差重映射过程,达到既改善立体图像视觉舒适度、又尽量减少视差改变所导致的立体感削弱的目的,从而提升立体图像的整体3维体验。     

Disparity-based just-noticeable-difference model for perceptual stereoscopic video coding using depth of focus blur effect

Human 3D perception provides an important clue to the removal of redundancy in stereoscopic 3D (S3D) videos. Because objects outside the binocular fusion limit cannot be fused on retina, the human visual system (HVS) makes them blur according to the depth-of-focus (DOF) effect to increase the binocular fusion limit and suppress diplopia, i.e. double vision. Based on human depth perception, we propose a disparity-based just-noticeable-difference model (DJND) to save bit-rate and improve visual comfort in S3D videos. We combine the DOF blur effect with conventional JND models in the pixel domain into DJND. Firstly, we use disparity information to get the average disparity value of each block. Then, we integrate the DOF blur effect into luminance JND (LJND) by a selective low pass Gaussian filter to minimize the visual stimulus in S3D videos. Finally, we incorporate disparity information into the filtered JND models to obtain DJND. Experimental results demonstrate that the proposed method successfully improves both image quality and visual comfort in viewing S3D videos without increasing the bit-rate.     

Gaze perception and awareness in smart devices

Eye contact and gaze awareness play a significant role for conveying emotions and intentions during face-to-face conversation. Humans can perceive each other's gaze quite naturally and accurately. However, the gaze awareness/perception are ambiguous during video teleconferencing performed by computer-based devices (such as laptops, tablet, and smart-phones). The reasons for this ambiguity are the (i) camera position relative to the screen and (ii) 2D rendition of 3D human face i.e., the 2D screen is unable to deliver an accurate gaze during video teleconferencing. To solve this problem, researchers have proposed different hardware setups with complex software algorithms. The most recent solution for accurate gaze perception employs 3D interfaces, such as 3D screens and 3D face-masks. However, today commonly used video teleconferencing devices are smart devices with 2D screens. Therefore, there is a need to improve gaze awareness/perception in these smart devices. In this work, we have revisited the question; how to improve a remote user's gaze awareness among his/her collaborators. Our hypothesis is that ‘an accurate gaze perception can be achieved by the ‘3D embodiment’ of a remote user's head gesture during video teleconferencing’. We have prototyped an embodied telepresence system (ETS) for the 3D embodiment of a remote user's head. Our ETS is based on a 3-DOF neck robot with a mounted smart device (tablet PC). The electromechanical platform in combination with a smart device is a novel setup that is used for studying gaze awareness/perception in 2D screen-based smart devices during video teleconferencing. Two important gaze-related issues are considered in this work; namely (i) ‘Mona-Lisa Gaze Effect’ – the gaze is always directed at the person independent of his position in the room, and (ii) ‘Gaze Awareness/Faithfulness’ – the ability to perceive an accurate spatial relationship between the observing person and the object by an actor. Our results confirm that the 3D embodiment of a remote user head not only mitigates the Mona Lisa gaze effect but also supports three levels of gaze faithfulness, hence, accurately projecting the human gaze in distant space.