Intelligent control of a robot manipulator by visual feedback

We present a new approach to visual feedback control using image-based visual servoing with stereo vision. In order to control the position and orientation of a robot with respect to an object, a new technique is proposed using binocular stereo vision. The stereo vision enables us to calculate an exact image Jacobian not only at around a desired location, but also at other locations. The suggested technique can guide a robot manipulator to the desired location without needing such a priori knowledge as the relative distance to the desired location or a model of the object, even if the initial positioning error is large. We describes a model of stereo vision and how to generate feedback commands. The performance of the proposed visual servoing system is illustrated by a simulation and by experimental results, and compared with the conventional method for an assembling robot. This work was presented in part at the Fourth International Symposium on Artificial Life and Robotics, Oita, Japan, January 19–22, 1999     

结合机器视觉的采摘机械手的定位仿真研究

对水果采摘机械手空间定位机理进行了研究,分析了双目立体视觉系统的定位误差并建立视觉误差补偿机制,利用虚拟机械手开发软件和CCD视觉硬件构建了仿真系统,通过双目立体视觉获取空间位置数据映射到虚拟环境下引导机械手进行模拟采摘。该系统利用多领域知识融合实现了采摘机构与视觉关联精确定位的仿真,能有效地指导实际作业环境中采摘机械手精确定位的优化设计。     

基于模糊神经网络的双目视觉定位系统研究

针对移动机器人对未知不确定环境缺乏自适应性的缺点,在活动关节自由度的基础上构建了基于模糊神经网络的双目视觉定位系统.采用自组织学习和监督学习相结合的混合算法,从而避免了移动机器人在移动过程中的振动所带来的误差.此外,通过控制伺服电机实现摄像机的移动,降低了双目视觉定位系统的成本.仿真实验结果表明,该系统具有较高的精确度及响应速度.     

Robust image-based visual servoing using invariant visual information

This paper deals with the use of invariant visual features for visual servoing. New features are proposed to control the 6 degrees of freedom of a robotic system with better linearizing properties and robustness to noise than the state of the art in image-based visual servoing. We show in this paper that by using these features the behavior of image-based visual servoing in task space can be significantly improved. Several experimental results are provided and validate our proposal.     

Comparison of visual fatigue caused by head-mounted display for virtual reality and two-dimensional display using objective and subjective evaluation

This study aimed to evaluate objective and subjective visual fatigue experienced before and after performing a visual task while using a head-mounted display for virtual reality (VR-HMD) and two-dimensional (2D) display. Binocular fusion maintenance (BFM) was measured using a binocular open-view Shack–Hartmann wavefront aberrometer equipped with liquid crystal shutters. Twelve healthy subjects performed the BFM test and completed a questionnaire regarding subjective symptoms before and after performing a visual task that induces low visually induced motion sickness (VIMS). BFM (p = .87) and total subjective eye symptom scores (p = .38) were not significantly different between both groups, although these values were significantly lower after the visual task than before the task within both groups (p < .05). These findings suggest that visual fatigue after using a VR-HMD is not significantly different from that after using a 2D display in the presence of low-VIMS VR content.

Practitioner summary: Objective and subjective evaluation of visual fatigue were not significantly different with the use of a head-mounted display for virtual reality (VR-HMD) and two-dimensional display. These results should be valuable not only to engineers developing VR content but also to researchers involved in the evaluation of visual fatigue using VR-HMD.

Abbreviations: VR: virtual reality; VR-HMD: head-mounted display for virtual reality; BFM: binocular fusion maintenance; BWFA: binocular open-view Shack–Hartmann wavefront aberrometer     

旋翼飞行机械臂系统的混合视觉伺服控制

旋翼飞行机械臂是一种具有强耦合特性的机器人系统,借助视觉进行自主作业还存在诸多问题,如实时深度估计、目标极易丢失以及目标笛卡尔空间模型重建等.本文针对传统的基于图像与基于位置的视觉伺服的缺陷以及系统自身欠驱动等问题,建立了运动学模型和提出了基于力平衡原理的动力学联合建模,并通过欧几里得单应性矩阵分解设计出旋翼飞行机械臂系统的混合视觉伺服控制方法,在图像空间控制平移、笛卡尔空间控制旋转,减弱了平移与旋转之间的相互影响实现解耦效果,改善了系统对非结构因素的抗扰性能和全局稳定性.通过仿真和实验检验了系统鲁棒性和算法优越性.     

基于双目定位的离合器下料系统设计

机械臂在生产线上抓取组装完成的离合器后,需要准确地将离合器放置在自动导引车(AGV)的下料位置.针对AGV入库时位置不确定,机械臂无法准确实现下料问题,设计了一种基于机器视觉技术的离合器下料定位系统,并针对AGV尺寸较大、单相机定位存在精度差、定位不稳定的重要问题,采用双目定位提高精度和稳定性.实际使用结果表明:基于VisionPro视觉软件设计的AGV特征识别视觉定位系统具有很好的定位结果,定位精度为±0.5 mm,运行时间小于2 s;能够引导机械臂快速、准确地完成离合器下料任务,满足工业实际应用的需求.     

基于结构光的机器人弧焊混合视觉伺服控制

A novel hybrid visual servoing control method based on structured light vision is proposed for robotic arc welding with a general six degrees of freedom robot. It consists of a position control inner-loop in Cartesian space and two outer-loops. One is position-based visual control in Cartesian space for moving in the direction of weld seam, i.e., weld seam tracking, another is image-based visual control in image space for adjustment to eliminate the errors in the process of tracking. A new Jacobian matrix from image space of the feature point on structured light stripe to Cartesian space is provided for differential movement of the end-effector. The control system model is simplified and its stability is discussed. An experiment of arc welding protected by gas CO_2 for verifying is well conducted.     

Omnidirectional visual control of mobile robots based on the 1D trifocal tensor

The precise positioning of robotic systems is of great interest particularly in mobile robots. In this context, the use of omnidirectional vision provides many advantages thanks to its wide field of view. This paper presents an image-based visual control to drive a mobile robot to a desired location, which is specified by a target image previously acquired. It exploits the properties of omnidirectional images to preserve the bearing information by using a 1D trifocal tensor. The main contribution of the paper is that the elements of the tensor are introduced directly in the control law and neither any a priori knowledge of the scene nor any auxiliary image are required. Our approach can be applied with any visual sensor obeying approximately a central projection model, presents good robustness to image noise, and avoids the problem of a short baseline by exploiting the information of three views. A sliding mode control law in a square system ensures stability and robustness for the closed loop. The good performance of the control system is proven via simulations and real world experiments with a hypercatadioptric imaging system.     

基于深度学习的运动目标实时识别与定位

针对人体运动目标的实时检测与定位问题,采用深度学习的方法进行研究.在Caffe框架下,采用SSD （Single Shot multibox Detector）检测方法.以VGG16作为基础网络模型,增加额外特征卷积层,提取多尺度的卷积特征.然后对实验数据集进行迭代训练,得到运动目标检测模型.利用训练好的模型,通过2路摄像机检测运动目标,并双目视觉定位.实验结果表明,整个系统运行速度可达40 fps,在10 m×10 m的场景下,平均定位误差在6 cm以内,在速度和精度上均有很好的表现,为大中型场景的人体运动实时检测定位问题提供了有效的解决方案.     

Real-time gaze control of an active head-eye system without calibration

This article describes real-time gaze control using position-based visual servoing. The main control objective of the system is to enable a gaze point to track the target so that the image feature of the target is located at each image center. The overall system consists of two parts: the vision process and the control system. The vision system extracts a predefined color feature from images. An adaptive look-up table method is proposed in order to get the 3-D position of the feature within the video frame rate under varying illumination. An uncalibrated camera raises the problem of the reconstructed 3-D positions not being correct. To solve the calibration problem in the position-based approach, we constructed an end-point closed-loop system using an active head-eye system. In the proposed control system, the reconstructed position error is used with a Jacobian matrix of the kinematic relation. The system stability is locally guaranteed, like image-based visual servoing, and the gaze position was shown to converge to the feature position. The proposed approach was successfully applied to a tracking task with a moving target in some simulations and some real experiments. The processing speed satisfies the property of real time. This work was presented in part at the Sixth International Symposium on Artificial Life and Robotics, Tokyo, January 15–17, 2001     

Image based visual servo control for a class of aerial robotic systems

An image-based strategy for visual servo control of a class of dynamic systems is proposed. The class of systems considered includes dynamic models of unmanned aerial vehicles capable of quasi-stationary flight (hover and near hover flight). The control strategy exploits passivity-like properties of the dynamic model to derive a Lyapunov control algorithm using backstepping techniques. The paper extends earlier work (Hamel, T., & Mahony, R. (2002). Visual servoing of an under-actuated dynamic rigid-body system: An image based approach. IEEE Transactions on Robotics and Automation, 18(2), 187-198) where partial pose information was used in the construction of the visual error. In this paper the visual error is defined purely in terms of the image features derived from the camera input. Local exponential stability of the system is proved. An estimate of the basin of attraction for the closed-loop system is provided.     

Mechanical design and control of inflatable robotic arms for high positioning accuracy

In recent years, inflatable robotic arms have been developed so that physical contact with humans and working environments could be performed safety. In industry, there is a growing demand for safe industrial robots to collaborate with people and work in various environments. In general, however, the positioning accuracy of inflatable robotic arms has not been discussed. This paper proposes an inflatable link structure and a non-inflatable joint structure that could realize a high positioning accuracy for such robotic arms. This paper experimentally demonstrates that these structures can improve positioning accuracy. In addition to these structures, the joint torque characteristics of the inflatable robotic arms were investigated. In order to perform accurate motion control, a visual feedback control method was introduced for inflatable robotic arms. The mechanism and control system used in this paper can improve the positioning accuracy performance of inflatable robots. A 2-DOF inflatable robotic arm and a camera system were found to be able to achieve a high positioning accuracy (i.e. less than 1 mm).     

Measuring visual fatigue and visual discomfort associated with 3‐D displays

Abstract— This research identifies methods for signs of visual complaints associated with stereoscopic displays. Two potential causes for contradictory results concerning these signs in previous research are hypothesized: (1) not all clinical tests are equally appropriate and (2) there is a natural variation in susceptibility amongst people with normal vision. An optometric screening differentiated participants with moderate binocular status (MBS) from those with good binocular status (GBS). In a 2 × 2 within‐subjects design (2‐D vs. 3‐D and MBS vs. GBS), a questionnaire and eight optometric tests (i.e., binocular acuity, aligning prism, fixation disparity, heterophoria, convergent and divergent fusional reserves, vergence facility, and accommodation response) were each administered before and immediately after a reading task. Results revealed that only participants with MBS in 3‐D conditions showed clinically meaningful changes in fusion range, experienced more visual discomfort, and performed worse on the reading task. Acombination of fusion range measurements and self‐report is appropriate for evaluating visual complaints of stereoscopic stills, and people with a MBS are more susceptible to visual complaints associated with stereoscopic displays. A simple measurement tool, i.e., the ratio of reading performance between 2‐D and 3‐D, to categorize people based on their binocular status is proposed.     

Virtual manipulator-based binocular stereo vision positioning system and errors modelling

There is a bottleneck of mobile robots positioning technologies for uncertain goals in complex field environment. Owing to the disturbance of the environment, the objects are hard to be located precisely by robot manipulator. Aiming at the positioning problem, binocular stereo vision system and positioning principle of the picking manipulator in virtual environment (VE) were proposed and expatiated upon; in addition, the manipulator positioning model was built in VE, and the manipulator positioning simulation system was developed by Microsoft Visual C++ 6.0; and the binocular stereo vision system platform with a three-coordinate guideway positioning was constructed for test; what’s more the error sources of vision positioning system was analyzed, and the camera system error was established; the mathematical model of experimental error and the camera calibration matching error were also found; with the developed robot manipulator positioning simulation software and vision system hardware, an experimental platform of positioning system was constructed, and using the platform, the stereo vision data was mapped to the manipulator and was guiding the accurate positioning in VE. Finally, experiment of positioning error compensation was carried out. Results of simulation in VE and the experiment showed that the vision positioning method was feasible for positioning in the field environment; it can be applied to control robot operation and to correct the positioning errors in real-time, especially to the long-range precision modelling and error compensation of robots.     

基于OpenCV的遥操作工程机器人双目视觉定位技术

针对当前遥操作工程机器人双目视觉定位技术匹配精准度低,导致定位误差过大的问题,提出种基于OpenCV改进的遥操作工程机器人双目视觉定位技术。在分析HSV色彩空间后,建立适合工程机器人的颜色特征识别空间体系,通过分析图像特征及运动坐标确定圆形光点,利用提供的靶点目标,创建模板后通过双目视觉获取具有靶点特征的其他图像,将图像代入OpenCV技术函数库中。在OpenCV技术函数库中通过光流法对图像进行函数匹配,应用将目标的背景模型与图像的处理方式分割开来,提取背景与干扰因素的全部信息,利用二值化阈值处理运动目标的形态,实现无干扰图像显示,确保定位结果的准确性。实验结果表明,基于OpenCV的遥操作工程机器人双目视觉定位技术能够有效提高匹配精度,降低定位误差,具有一定的实际应用价值。     

基于粗精立体匹配的双目视觉目标定位方法

针对双目视觉系统定位精度较低的问题,提出一种基于粗-精立体匹配的双目视觉目标定位方法。该方法采用粗-精匹配策略：在粗匹配阶段使用基于Canny-Harris特征点的随机蕨算法对左右图中的目标进行识别,提取目标矩形区域的中心点,实现中心匹配;在精匹配阶段建立一种基于图像梯度信息的二值特征描述子,将中心匹配得到的右中心点作为估计值,设定像素搜索范围,于该区域中找出左中心点的最佳匹配点。最后,将得到的中心点匹配对代入平行双目视觉的数学模型中,实现目标定位。实验结果表明,在500 mm距离范围内,所提出定位方法的定位误差控制在7 mm内,平均相对定位误差为2.53%,相比其他方法具有定位精度高、运行时间短的优点。     

3-D positioning control by linear visual servoing

This article describes the performance of 3-D positioning control by linear visual servoing using binocular visual space in a human-like hand-eye system which has a similar kinetic structure to a human being. We approximate the nonlinear time-variant mapping from a binocular visual space to the joint space of the manipulator as a linear time-invariant mapping. We also investigate the effect of binocular visual space in linear mapping by comparing it with linear mapping using Cartesian space. Some experimental results are presented using the human-like hand-eye system to demonstrate the performance of 3-D positioning control. This work was presented, in part, at the Seventh International Symposium on Artificial Life and Robotics, Oita, Japan, January 16–18, 2002     

基于仿生双目的无人旋翼机自主着陆方法

基于仿生双目理论提出一种仿双眼异向运动的变视轴夹角双目视觉系统,通过对实时采集的目标图像的处理,研究其双目摄像机夹角的在线自标定,实现获取无人旋翼机的实时高度信息。实验结果表明,该方法获得的光轴夹角信息相对误差小于5%,能准确控制双目摄像机的转动,避免由于双目视觉盲区所造成的着陆误差,系统所标定出的旋翼机高度信息也为其安全着陆提供了有力保障。     

基于投影直方图匹配的双目视觉跟踪算法

为了在机器人机械手双目视觉伺服系统中跟踪并精确定位目标的空间位置,提出了一种利用投影直方图匹配和极线几何约束的目标跟踪方法。分别在2个视觉中人工标定目标,并提取目标在多颜色空间的水平、垂直投影直方图作为匹配模板;在一个视觉中利用目标的运动一致性原则和投影直方图匹配搜索并跟踪目标;在另一个视觉中依据双目视觉系统的极线几何原理限定目标搜索范围,搜索并定位目标。该方法利用水平、垂直投影直方图描述目标的结构信息,同时完成了双目视觉系统中的目标跟踪与配准功能,有利于目标的精确定位和视觉测量。实验结果表明,该方法可在双目视觉系统中有效跟踪目标,运算效率高,鲁棒性强。