Safe and efficient motion planning of multiple mobile robots based on artificial potential for human behavior and robot congestion

In order for robots to exist together with humans, safety for the humans has to be strictly ensured. On the other hand, safety might decrease working efficiency of robots. Namely, this is a trade-off problem between human safety and robot efficiency in a field of human–robot interaction. For this problem, we propose a novel motion planning technique of multiple mobile robots. Two artificial potentials are presented for generating repulsive force. The first potential is provided for humans. The von Mises distribution is used to consider the behavioral property of humans. The second potential is provided for the robots. The Kernel density estimation is used to consider the global robot congestion. Through simulation experiments, the effectiveness of the behavior and congestion potentials of the motion planning technique for human safety and robot efficiency is discussed. Moreover, a sensing system for humans in a real environment is developed. From experimental results, the significance of the behavior potential based on the actual humans is discussed. For the coexistence of humans and robots, it is important to evaluate a mutual influence between them. For this purpose, a virtual space is built using projection mapping. Finally, the effectiveness of the motion planning technique for the human–robot interaction is discussed from the point of view of not only robots but also humans.     

Human-robot coexistence and interaction in open industrial cells

Recent research results on human–robot interaction and collaborative robotics are leaving behind the traditional paradigm of robots living in a separated space inside safety cages, allowing humans and robot to work together for completing an increasing number of complex industrial tasks. In this context, safety of the human operator is a main concern. In this paper, we present a framework for ensuring human safety in a robotic cell that allows human–robot coexistence and dependable interaction. The framework is based on a layered control architecture that exploits an effective algorithm for online monitoring of relative human–robot distance using depth sensors. This method allows to modify in real time the robot behavior depending on the user position, without limiting the operative robot workspace in a too conservative way. In order to guarantee redundancy and diversity at the safety level, additional certified laser scanners monitor human–robot proximity in the cell and safe communication protocols and logical units are used for the smooth integration with an industrial software for safe low-level robot control. The implemented concept includes a smart human-machine interface to support in-process collaborative activities and for a contactless interaction with gesture recognition of operator commands. Coexistence and interaction are illustrated and tested in an industrial cell, in which a robot moves a tool that measures the quality of a polished metallic part while the operator performs a close evaluation of the same workpiece.     

Impedance Compensation of SUBAR for Back-Drivable Force-Mode Actuation

The Sogang University biomedical assistive robot (SUBAR), which is an advanced version of the exoskeleton for patients and the old by Songang (EXPOS) is a wearable robot developed to assist physically impaired people. It provides a person with assistive forces controlled by human intentions. If a standard geared dc motor is applied, however, the control efforts will be used mainly to overcome the resistive forces caused by the friction, the damping, and the inertia in actuators. In this paper, such undesired properties are rejected by applying a flexible transmission. With the proposed method, it is intended that an actuator exhibits zero impedance without friction while generating the desired torques precisely. Since the actuation system of SUBAR has a large model variation due to human–robot interaction, a control algorithm for the flexible transmission is designed based on a robust control method. In this paper, the mechanical design of SUBAR, including the flexible transmission and its associated control algorithm, are presented. They are also verified by experiments.      

Asymmetric Velocity Moderation for human-safe robot control

With the increasing physical proximity of human–robot interaction, ensuring that robots do not harm surrounding humans has become crucial. Therefore, we propose asymmetric velocity moderation as a low-level controller for robotic systems to enforce human-safe motions. While our method prioritizes human safety, it also maintains the robot’s efficiency. Our proposed method restricts the robot’s speed according to (1) the displacement vector between human and robot, and (2) the robot’s velocity vector. That is to say, both the distance and the relative direction of movement are taken into account to restrict the robot’s motion. Through real-robot and simulation experiments using simplified HRI scenarios and dangerous situations, we demonstrate that our method is able to maintain the robot’s efficiency without undermining human safety.     

Learning proactive behavior for interactive social robots

Learning human–robot interaction logic from example interaction data has the potential to leverage “big data” to reduce the effort and time spent on designing interaction logic or crafting interaction content. Previous work has demonstrated techniques by which a robot can learn motion and speech behaviors from non-annotated human–human interaction data, but these techniques only enable a robot to respond to human-initiated inputs, and do not enable the robot to proactively initiate interaction. In this work, we propose a method for learning both human-initiated and robot-initiated behavior for a social robot from human–human example interactions, which we demonstrate for a shopkeeper interacting with a customer in a camera shop scenario. This was achieved by extending an existing technique by (1) introducing a concept of a customer yield action, (2) incorporating interaction history, represented by sequences of discretized actions, as inputs for training and generating robot behavior, and (3) using an “attention mechanism” in our learning system for training robot behaviors, that learns which parts of the interaction history are more important for generating robot behaviors. The proposed method trains a robot to generate multimodal actions, consisting of speech and locomotion behaviors. We compared this study with the previous technique in two ways. Cross-validation on the training data showed higher social appropriateness of predicted behaviors using the proposed technique, and a user study of live interaction with a robot showed that participants perceived the proposed technique to produce behaviors that were more proactive, socially-appropriate, and better in overall quality.     

Physiological and subjective evaluation of a human-robot object hand-over task

In the context of task sharing between a robot companion and its human partners, the notions of safe and compliant hardware are not enough. It is necessary to guarantee ergonomic robot motions. Therefore, we have developed Human Aware Manipulation Planner (Sisbot et al., 2010), a motion planner specifically designed for human–robot object transfer by explicitly taking into account the legibility, the safety and the physical comfort of robot motions. The main objective of this research was to define precise subjective metrics to assess our planner when a human interacts with a robot in an object hand-over task. A second objective was to obtain quantitative data to evaluate the effect of this interaction. Given the short duration, the “relative ease” of the object hand-over task and its qualitative component, classical behavioral measures based on accuracy or reaction time were unsuitable to compare our gestures. In this perspective, we selected three measurements based on the galvanic skin conductance response, the deltoid muscle activity and the ocular activity. To test our assumptions and validate our planner, an experimental set-up involving Jido, a mobile manipulator robot, and a seated human was proposed. For the purpose of the experiment, we have defined three motions that combine different levels of legibility, safety and physical comfort values. After each robot gesture the participants were asked to rate them on a three dimensional subjective scale. It has appeared that the subjective data were in favor of our reference motion. Eventually the three motions elicited different physiological and ocular responses that could be used to partially discriminate them.     

Depth camera based collision avoidance via active robot control

A new type of depth cameras can improve the effectiveness of safety monitoring in human–robot collaborative environment. Especially on today's manufacturing shop floors, safe human–robot collaboration is of paramount importance for enhanced work efficiency, flexibility, and overall productivity. Within this context, this paper presents a depth camera based approach for cost-effective real-time safety monitoring of a human–robot collaborative assembly cell. The approach is further demonstrated in adaptive robot control. Stationary and known objects are first removed from the scene for efficient detection of obstacles in a monitored area. The collision detection is processed between a virtual model driven by real sensors, and 3D point cloud data of obstacles to allow different safety scenarios. The results show that this approach can be applied to real-time work cell monitoring.     

Wheeled inverted pendulum type assistant robot: design concept and mobile control

This paper describes the design concept of the human assistant robot I-PENTAR (Inverted PENdulum Type Assistant Robot) aiming at the coexistence of safety and work capability and its mobile control strategy. I-PENTAR is a humanoid type robot which consists of a body with a waist joint, arms designed for safety, and a wheeled inverted pendulum mobile platform. Although the arms are designed low-power and lightweight for safety, it is able to perform tasks that require high power by utilizing its self-weight, which is the feature of a wheeled inverted pendulum mobile platform. I-PENTAR is modeled as a three dimensional robot; with controls of inclination angle, horizontal position, and steering angle to achieve high mobile capability. The motion equation is derived considering the non-holonomic constraint of the two-wheeled mobile robot, and a state feedback control method is applied for basic mobile controls wherein the control gain is calculated by the LQR method. Through several experiments of balancing, linear running, and steering, it was confirmed that the robot could realize stable mobile motion in a real environment by the proposed controller.     

Robot adaptation to human physical fatigue in human–robot co-manipulation

In this paper, we propose a novel method for human–robot collaboration, where the robot physical behaviour is adapted online to the human motor fatigue. The robot starts as a follower and imitates the human. As the collaborative task is performed under the human lead, the robot gradually learns the parameters and trajectories related to the task execution. In the meantime, the robot monitors the human fatigue during the task production. When a predefined level of fatigue is indicated, the robot uses the learnt skill to take over physically demanding aspects of the task and lets the human recover some of the strength. The human remains present to perform aspects of collaborative task that the robot cannot fully take over and maintains the overall supervision. The robot adaptation system is based on the Dynamical Movement Primitives, Locally Weighted Regression and Adaptive Frequency Oscillators. The estimation of the human motor fatigue is carried out using a proposed online model, which is based on the human muscle activity measured by the electromyography. We demonstrate the proposed approach with experiments on real-world co-manipulation tasks: material sawing and surface polishing.     

Interaction forces beneath cuffs of physical assistant robots and their motion-based estimation

Most lower-limb physical assistant robots are fixed to wearers using cuffs. Hence, skin injuries beneath the cuffs are one of the major concerns of the users. A model that describes the relationship between the body posture and the interaction forces at the cuff was developed for use in assessing the risk of injury and improving user comfort. We measured the motion and interaction force beneath cuffs during the sitting and standing motions of subjects and a physical assistant robot which has been hardly reported thus far. Because of slippage and biomechanical motion, a traditional spring-damper model was found to be insufficient to describe the interaction forces associated with the measured motion of the cuffs. A parameter representing the motion or the knee joint angle was added to take into account these factors. Our model for estimation of the interaction forces using a spring, a damper, and the attitude of the lower leg fits the measured data especially well for the thigh cuff and is better than the traditional model. The applicability of this model was verified for several assist modes and wearers. The model was found to describe approximately 90% of the burden on the wearer, which reached a peak of approximately 60 N, the most hazardous condition. Having been validated for a commercial assistant robot, the model can be used to estimate the skin burdens beneath the cuffs without any force-sensitive elements.     

Brain signal-based safety measure activation for robotic systems

In this paper, we present our approach for using EEG signals to activate safety measures of a robot when an error or unexpected event is perceived by the human operator. In particular, we consider brain-based error perception while the operator passively observes the robot performing an action. Our approach consists of monitoring EEG signals and detecting a brain potential called error related negativity (ERN) that spontaneously occurs when the operator perceives an error made by the robot or when an unexpected event occurs. We detect ERN by pre-training two linear classifiers using data collected from a preliminary experiment based on a visual reaction task. We derive the probability of failure in demand (PFD), commonly used to assess functional safety for a two-channel verification system based on the combination of linear classifiers. Functional safety analysis was then performed on a BMI-based robotic framework in which a signal was sent to the robot to active its safety measures in when an ERN was detected. Using brain-based signals, we demonstrate that it is possible to send an emergency stop action during mobile navigation task when unexpected events occur with an accuracy of 75%.     

Compact laparoscopic assistant robot using a bending mechanism

This paper presents the development of a compact laparoscopic assistant robot. The robot was designed to increase convenience and reduce possible interference with surgical staff by confining the majority of motions inside the abdomen. Its size was miniaturized as much as possible for convenient handling. A bending mechanism composed of several articulated joints was introduced to produce motions inside the abdomen. The proposed assistant robot can generate 3-DOF motion, including 2-DOF internal bending motion and 1-DOF external linear motion. Since the robot itself functions as a laparoscope, a small CCD camera module and a bundle of optical fibers were integrated as part of the system. For accurate control, mathematical modeling of the bending mechanism and a method of hysteresis compensation were introduced and implemented. For the control of the robot, a voice interface and a visual-servoing method were implemented. The performance of the developed system was tested through solo-surgery of in vivo porcine cholecystectomy. It was found that the views generated by the bending mechanism were sufficient throughout the surgery. Since the robot has functions comparable to the previously developed systems, while retaining its compactness, it is expected to be a useful device for human cholecystectomy.     

颅颌面外科手术机器人空间配准及实验

针对颅颌面区域结构复杂、手术风险大、精度要求高的特点,研发了一套颅颌面外科手术机器人系统．为实现机器人系统高效、精确的空间配准,提出并建立了一套基于光学定位的颅颌面外科手术机器人空间配准方法,实现了机器人各子系统之间的坐标系转换．通过基于四元数的最近点迭代配准算法实现了点集之间的矩阵转换．在综合医学影像、机器人与各子系统空间配准的基础上,分别开展了配准精度和定位精度的验证实验,并完成了三叉神经射频热凝术的模型和尸体穿刺定位实验．配准精度验证实验的配准误差平均值均小于0.75mm,定位精度验证实验的误差平均值为0.56mm,模型实验的穿刺成功率为100%,尸体实验的穿刺成功率为95%．综合实验结果表明该空间配准方法的配准精度较高,具有一定的可行性,可以满足颅颌面外科手术机器人系统的需要．     

Shadow robot for teaching motion

Human-friendly robots have begun to spread in society. In the future such robots and intelligent machines should be autonomous in open situations. To give dexterity to a robot, teaching motion is a good candidate. However, there are some problems from the operational point of view due to gravity and friction effects.In this paper, a shadow robot is proposed for teaching motion instead of force sensors. The shadow robot is a novel disturbance compensation method that consists of a twin robot system. Two of the same type of robot are required and they are controlled with the same position, velocity, and acceleration by bilateral acceleration control based on a disturbance observer. One robot is in the teaching motion controlled by a human and the other is unconstrained. Thus the purity of the human force is extracted by subtracting the disturbance torque in the unconstrained robot from the constrained one. As a result, the shadow robot observes the human force with gravity and friction compensation. Since it is possible to apply this concept to a multi-degree-of-freedom system, the human operationality in teaching motion are improved.The experimental results show the viability of the proposed method.     

基于语义分割的简洁线条肖像画生成方法

针对目前主流的线条提取算法对于区域对比度不明显的边缘的检测能力较弱,且对于所有区域采用无差别、统一化的处理策略,所生成的线条画往往较复杂,非常不利于机器人机械臂绘图的问题,本文提出了一种基于语义分割的简洁线条肖像画生成方法(concise line portrait generation based on semantic segmentation, CLPG-SS)。首先,对人脸图像进行语义分割,将人脸划分为不同的区域,基于不同区域提取边缘轮廓与五官细节线条,进行边缘切向流优化,从而加强方向信息;在此基础上,利用线条图来生成调和图像,并利用优化后的边缘切向流、人脸语义分割结果以及调和图像,针对不同的分割区域调整线条提取方法的参数,实现对细节无关区域的线条过滤和细节重点区域的线条加强,生成简洁线条肖像画。实验结果表明:本文提出的CLPG-SS方法能够有效提取人脸主轮廓线条,并针对不同区域实现了对细节线条的针对性调节,提高了机器人机械臂的绘制效率。     

基于场景分析的系统形式化模型生成方法

采用形式化方法对系统的安全性进行分析与验证,是构造可靠安全软件系统的一个重要途径。当前的形式化安全分析方法,面临着系统的形式化建模难的问题。以铁路车站联锁系统中基本进路建立为例,提出基于场景分析的系统形式化模型生成方法。该方法首先采用OCL前/后置条件分析法对UML时序场景作一致性分析,然后将UML时序图中对象交互的行为序列转换成FSP进程代数模型,进而得到系统的形式化模型。该方法为系统的形式化建模提供了新思路,从安全质量方面改善了安全苛求软件的设计与开发,丰厚了基于模型的软件形式化开发方法。     

Development of a Face Verification System for a Home Service Robot

The robot providing services based on interaction with a human needs a fast verification process that knows whether the user belongs to a family or a guest group for providing the differentiated services to each group member. In this paper, we developed a verification system for one-to-group matching (e.g., user-to-family or user-to-guest) and tested on the database acquired using the robot in an uncontrolled environment. The proposed system consists of three parts: face detection using revised modified census transform and adaboost, feature extraction using multiple principle component analysis, and verification using the aggregating Gaussian mixture model–universal background model. Experimental results show that the proposed system works well on the deteriorated database with a high speed.     

四轮驱动滑动吸盘爬壁机器人的动力学研究

研究了四轮驱动滑动吸盘式爬壁机器人在滑动导向运动方式下的动力学问题．首先分析了机器人在壁面上安全移动的运动状态和约束条件, 然后对驱动轮支撑力分布、驱动轮与壁面间的横向摩擦力以及密封圈摩擦力进行分析,进而基于牛顿---欧拉法建立了机器人的动力学方程, 并用驱动力矩安全系数来表征驱动力矩的安全程度．通过动力学仿真,分析了吸附压力、驱动轮分布、密封圈刚度等对驱动力矩的影响, 为四轮驱动滑动吸盘式爬壁机器 人结构优化和安全运动控制提供理论依据．     

Fuzzy-based-admittance controller for safe natural human–robot interaction

ABSTRACT

In recent years, a great amount of research on physical human–robot interaction has been conducted, and mainly concentrated on safety issues to minimize the risk of accidents to the operator during the cooperation between human and robot. Unfortunately, the identification of inertia and damping matrices in the dynamic admittance model is time-consuming, which is still an open problem of previous admittance controllers. Additionally, the natural cooperation is that cooperative movements are implemented in every degree of freedom in space, which is rarely concerned while it is important to implement more complex cooperative movements, and to help operator feels naturally during the cooperation. This paper presents an alternative admittance controller based on inference mechanism of fuzzy logic to eliminate the identification of inertia and damping matrices during the process of controller formulation in which the end-effector’s velocity is adaptively adjusted via external wrench (force/torque measured by a sensor mounted on end-effector) and power transmitted by the robot. Moreover, the proposed controller also considers end-effector’s full DOF to guarantee the natural human–robot interaction. The fuzzy-admittance controller is evaluated by an experimental set-up of teaching task using 6-DOF manipulator in which manipulator moves passively via the human impact on real-time force/torque sensor mounted on end-effector.     

A deep learning-enhanced Digital Twin framework for improving safety and reliability in human–robot collaborative manufacturing

In Industry 5.0, Digital Twins bring in flexibility and efficiency for smart manufacturing. Recently, the success of artificial intelligence techniques such as deep learning has led to their adoption in manufacturing and especially in human–robot collaboration. Collaborative manufacturing tasks involving human operators and robots pose significant safety and reliability concerns. In response to these concerns, a deep learning-enhanced Digital Twin framework is introduced through which human operators and robots can be detected and their actions can be classified during the manufacturing process, enabling autonomous decision making by the robot control system. Developed using Unreal Engine 4, our Digital Twin framework complies with the Robotics Operating System specification, and supports synchronous control and communication between the Digital Twin and the physical system. In our framework, a fully-supervised detector based on a faster region-based convolutional neural network is firstly trained on synthetic data generated by the Digital Twin, and then tested on the physical system to demonstrate the effectiveness of the proposed Digital Twin-based framework. To ensure safety and reliability, a semi-supervised detector is further designed to bridge the gap between the twin system and the physical system, and improved performance is achieved by the semi-supervised detector compared to the fully-supervised detector that is simply trained on either synthetic data or real data. The evaluation of the framework in multiple scenarios in which human operators collaborate with a Universal Robot 10 shows that it can accurately detect the human and robot, and classify their actions under a variety of conditions. The data from this evaluation have been made publicly available, and can be widely used for research and operational purposes. Additionally, a semi-automated annotation tool from the Digital Twin framework is published to benefit the collaborative robotics community.