AR-based interaction for human-robot collaborative manufacturing

Industrial standards define safety requirements for Human-Robot Collaboration (HRC) in industrial manufacturing. The standards particularly require real-time monitoring and securing of the minimum protective distance between a robot and an operator. This paper proposes a depth-sensor based model for workspace monitoring and an interactive Augmented Reality (AR) User Interface (UI) for safe HRC. The AR UI is implemented on two different hardware: a projector-mirror setup and a wearable AR gear (HoloLens). The workspace model and UIs are evaluated in a realistic diesel engine assembly task. The AR-based interactive UIs provide 21–24% and 57–64% reduction in the task completion and robot idle time, respectively, as compared to a baseline without interaction and workspace sharing. However, user experience assessment reveal that HoloLens based AR is not yet suitable for industrial manufacturing while the projector-mirror setup shows clear improvements in safety and work ergonomics.     

Considerations of potential runaway motion and physical interaction for speed and separation monitoring

In the field of human-robot collaboration (HRC), the speed and separation monitoring (SSM) collaboration have attracted much attention owing to its non-contact safety strategy. 3D sensing applications are currently of interest for industrial automation technology and are considered to be a promising method for maximizing the efficiency of SSM. However, little attention has been given to the runaway space of the robot or the potential contact due to the foreseeable misuse of the operator. In this study, experiments are conducted using a radar system as an example of a 3D safety-related sensor, and battery assembly scenarios are carried out for the comparison. In the experiment, two different orientations of the robot are tested, considering the potential runaway motion of the robot. Also, the maximum permissible speed of the robot is calculated by geometrical transfer energy, which is based on effective mass and the velocity of the manipulator and human injury criteria. From the experimental results, it is evident that it is better to avoid placing the vertical articulated robot in front of the operator from the perspective of minimizing the effect of runaway motion into the safety distance. Finally, the proposed framework of speed limitation is thought to be an effective method to link SSM and power and force limiting safety function.     

Vision‐based control architecture for human–robot hand‐over applications

Human–robot collaboration will be an essential part of the production processes in the factories of tomorrow. In this paper, a human–robot hand‐over control strategy is presented. Human and robot can be both giver and receiver. A path‐planning algorithm drives the robotic manipulator towards the hand of the human and permits to adapt the pose of the tool center point of the robot to the pose of the hand of the human worker. The movements of the operator are acquired with a multi 3D‐sensors setup so to avoid any possible occlusion related to the presence of the robot or other dynamic obstacles. Estimation of the predicted position of the hand is performed to reduce the waiting time of the operator during the hand‐over task. The hardware setup is described, and the results of experimental tests, conducted to verify the effectiveness of the control strategy, are presented and discussed.     

Upper-body haptic system for snake robot teleoperation in pipelines

Snake robots have shown a great potential for operations in confined workplaces that are less accessible or dangerous to human workers, such as the in-pipe inspection. However, the snake robot teleoperation remains a nontrivial task due to the unique locomotion mechanism (e.g., helical motion) and the constraints of the workplaces including the low visibility and indistinguishable features. Most snake robot feedback systems are based on the live camera view only. It is hard for the human operator to develop a correct spatial understanding of the remote workplace, leading to problems such as disorientation and motion sickness in snake robot teleoperation. This study designs and evaluates an innovative haptic assistant system for snake robot teleoperation in the in-pipe inspection. An upper-body haptic suit with 40 vibrators on both the front and back sides of the human operator was developed to generate haptic feedback corresponding to the bottom and up sides of the snake robot, transferring the egocentric sensation of the snake robot to the human operator. A human-subject experiment (n = 31) was performed to evaluate the efficacy of the developed system. The results indicate that the proposed haptic assistant system outperformed other feedback systems in terms of both task performance and subjective workload and motion sickness evaluations. It inspires new control and feedback designs for the future snake robot in industrial operations.     

动态仓储环境下的多机器人路径规划方法

针对动态仓储环境下多机器人运动过程中出现的拥塞死锁问题,利用路径长度、转弯数、路径惩罚函数建立小车单任务耗时模型。模型引入阻塞惩罚函数,移除可能发生阻塞的路径增加罚值。同时针对传统遗传算法路径规划操作过程中路径交叉变异导致路径中断不可用的情况,设计重复点交叉算子,在变异操作后检查路径合法性,使算法都是在可行的解空间上进行搜索。仿真实验表明,算法能指导机器人获得动态环境下的最优路径,同时算法收敛速度大大提高。     

Biological movement increases acceptance of humanoid robots as human partners in motor interaction

The automatic tendency to anthropomorphize our interaction partners and make use of experience acquired in earlier interaction scenarios leads to the suggestion that social interaction with humanoid robots is more pleasant and intuitive than that with industrial robots. An objective method applied to evaluate the quality of human–robot interaction is based on the phenomenon of motor interference (MI). It claims that a face-to-face observation of a different (incongruent) movement of another individual leads to a higher variance in one’s own movement trajectory. In social interaction, MI is a consequence of the tendency to imitate the movement of other individuals and goes along with mutual rapport, sense of togetherness, and sympathy. Although MI occurs while observing a human agent, it disappears in case of an industrial robot moving with piecewise constant velocity. Using a robot with human-like appearance, a recent study revealed that its movements led to MI, only if they were based on human prerecording (biological velocity), but not on constant (artificial) velocity profile. However, it remained unclear, which aspects of the human prerecorded movement triggered MI: biological velocity profile or variability in movement trajectory. To investigate this issue, we applied a quasi-biological minimum-jerk velocity profile (excluding variability in the movement trajectory as an influencing factor of MI) to motion of a humanoid robot, which was observed by subjects performing congruent or incongruent arm movements. The increase in variability in subjects’ movements occurred both for the observation of a human agent and for the robot performing incongruent movements, suggesting that an artificial human-like movement velocity profile is sufficient to facilitate the perception of humanoid robots as interaction partners.     

Task analysis methods in industry

Task analysis is one of the basic tools used by ergonomists in investigating and designing tasks. It provides a formal comparison between the demands which the task places on the human operator and the capabilities the human operator possesses to deal with these demands. Three types of task analysis are described: sequential, branching and process control. Alternative formats are presented and examples given of their use in an industrial setting.     

Assistant force compensation for hand movement of patients by using exogenous signals and/or neurophysiological signals

Because functional diseases of the brain can cause disabilities related to human movement control, a compensation method was developed for improving the performance of hand movements. The compensation for human hand movements can be carried out by adding an assistant force that is generated from artificial equipment attached to a human arm. From the experiment on visual target tracking, it was found that the tracking trajectory was adequately represented by a dynamic model of the motion of an articulated industrial robot arm, and the different abilities for movement control among healthy people and patients were classified by different model parameters as position loop gain, velocity loop gain, and response delay. Dynamic force compensation was approached by considering the different control features of the patients. The effectiveness of the proposed compensation method was verified in a simulation study on an actual industrial robot arm. A human-machine interface, e.g., a brain-computer interface (BCI), for realizing the control of artificial equipment to compensate for human hand movements is also presented and discussed.     

Markerless human–robot interface for dual robot manipulators using Kinect sensor

Remote teleoperation of robot manipulators is often necessary in unstructured, dynamic, and dangerous environments. However, the existing mechanical and other contacting interfaces require unnatural, or hinder natural, human motions. At present, the contacting interfaces used in teleoperation for multiple robot manipulators often require multiple operators. Previous vision-based approaches have only been used in the remote teleoperation for one robot manipulator as well as require the special quantity of illumination and visual angle that limit the field of application. This paper presents a noncontacting Kinect-based method that allows a human operator to communicate his motions to the dual robot manipulators by performing double hand–arm movements that would naturally carry out an object manipulation task. This paper also proposes an innovative algorithm of over damping to solve the problem of error extracting and dithering due to the noncontact measure. By making full use of the human hand–arm motion, the operator would feel immersive. This human–robot interface allows the flexible implementation of the object manipulation task done in collaboration by dual robots through the double hand–arm motion by one operator.     

Proactive human–robot collaboration: Mutual-cognitive,predictable, and self-organising perspectives

Human–Robot Collaboration (HRC) has a pivotal role in smart manufacturing for strict requirements of human-centricity, sustainability, and resilience. However, existing HRC development mainly undertakes either a human-dominant or robot-dominant manner, where human and robotic agents reactively perform operations by following pre-defined instructions, thus far from an efficient integration of robotic automation and human cognition. The stiff human–robot relations fail to be qualified for complex manufacturing tasks and cannot ease the physical and psychological load of human operators. In response to these realistic needs, this paper presents our arguments on the obvious trend, concept, systematic architecture, and enabling technologies of Proactive HRC, serving as a prospective vision and research topic for future work in the human-centric smart manufacturing era. Human–robot symbiotic relation is evolving with a 5C intelligence — from Connection, Coordination, Cyber, Cognition to Coevolution, and finally embracing mutual-cognitive, predictable, and self-organising intelligent capabilities, i.e., the Proactive HRC. With proactive robot control, multiple human and robotic agents collaboratively operate manufacturing tasks, considering each others’ operation needs, desired resources, and qualified complementary capabilities. This paper also highlights current challenges and future research directions, which deserve more research efforts for real-world applications of Proactive HRC. It is hoped that this work can attract more open discussions and provide useful insights to both academic and industrial practitioners in their exploration of human–robot flexible production.     

Optimal robot arm control using the minimum variance model

Models of human movement from computational neuroscience provide a starting point for building a system that can produce flexible adaptive movement on a robot. There have been many computational models of human upper limb movement put forward, each attempting to explain one or more of the stereotypical features that characterize such movements. While these models successfully capture some of the features of human movement, they often lack a compelling biological basis for the criteria they choose to optimize. One that does provide such a basis is the minimum variance model (and its extension—task optimization in the presence of signal‐dependent noise). Here, the variance of the hand position at the end of a movement is minimized, given that the control signals on the arm's actuators are subject to random noise with zero mean and variance proportional to the amplitude of the signal. Since large control signals, required to move fast, would have higher amplitude noise, the speed‐accuracy trade‐off emerges as a direct result of the optimization process. We chose to implement a version of this model that would be suitable for the control of a robot arm, using an optimal control scheme based on the discrete‐time linear quadratic regulator. This implementation allowed us to examine the applicability of the minimum variance model to producing humanlike movement. In this paper, we describe our implementation of the minimum variance model, both for point‐to‐point reaching movements and for more complex trajectories involving via points. We also evaluate its performance in producing humanlike movement and show its advantages over other optimization based models (the well‐known minimum jerk and minimum torque‐change models) for the control of a robot arm. © 2005 Wiley Periodicals, Inc.     

A framework and method for Human-Robot cooperative safe control based on digital twin

Human-robot collaboration (HRC) combines the robot’s mechanical properties and predictability with human experience, logical thinking, and strain capabilities to alleviate production efficiency. However, ensuring the safety of the HRC process in-real time has become an urgent issue. Digital twin extends functions of virtual models in the design phase of the physical counterpart in the production phase through virtual-real interactive feedback, data fusion analysis, advanced computational features, etc. This paper proposes an HRC safety control framework and corresponding method based on the digital twin. In the design phase, virtual simulation and virtual reality technology are integrated to construct virtual twins of various HRC scenarios for testing and analyzing potential safety hazards. In the production phase, the safety distance between humans and robots of the HRC scene is monitored and calculated by an iterative algorithm according to machine vision and a convolutional neural network. Finally, the virtual twin is driven based on real-scene data, real-time online visual monitoring, and optimization of the HRC’s overall process. A case study using ABB-IRB1600 is presented to verify the feasibility of the proposed approach.     

Estimating probability of human hand intrusion for speed and separation monitoring using interference theory

Human Robot Collaboration (HRC) has attracted high attention in modern manufacturing. Recently, a safety standard for collaborative robots has been launched (ISO/TS 15066). It cover the safety function of Speed and Separation Monitoring (SSM) which includes velocities and positions of the human and the robot. However, risk should be discussed with probability, while the SSM is not. Therefore, the probability of intrusion was calculated according to the stopping time of the robot and the maximum speed of the robot. Since the SSM was testified along the simulation by far, actual adhesive applying task was carried out in this paper.     

Models of human movement: Trajectory planning and inverse kinematics studies

The seemingly simple everyday actions of moving limb and body to accomplish a motor task or interact with the environment are incredibly complex. To reach for a target we first need to sense the target’s position with respect to an external coordinate system; we then need to plan a limb trajectory which is executed by issuing an appropriate series of neural commands to the muscles. These, in turn, exert appropriate forces and torques on the joints leading to the desired movement of the arm. Here we review some of the earlier work as well as more recent studies on the control of human movement, focusing on behavioral and modeling studies dealing with task space and joint-space movement planning. At the task level, we describe studies investigating trajectory planning and inverse kinematics problems during point-to-point reaching movements as well as two-dimensional (2D) and three-dimensional (3D) drawing movements. We discuss models dealing with the two-thirds power law, particularly differential geometrical approaches dealing with the relation between path geometry and movement velocity. We also discuss optimization principles such as the minimum-jerk model and the isochrony principle for point-to-point and curved movements.We next deal with joint-space movement planning and generation, discussing the inverse kinematics problem and common solutions to the problems of kinematic redundancy. We address the question of which reference frames are used by the nervous system and review studies examining the employment of kinematic constraints such as Donders’ and Listing’s laws. We also discuss optimization approaches based on Riemannian geometry.One principle of motor coordination during human locomotion emerging from this body of work is the intersegmental law of coordination. However, the nature of the coordinate systems underlying motion planning remains of interest as they are related to the principles governing the control of human arm movements.     

Speed-accuracy characteristics of human-machine cooperative manipulation using virtual fixtures with variable admittance

This work explores the effect of virtual fixture admittance on the performance, defined by error and time, of task execution with a human-machine cooperative system. A desired path is obtained using computer vision, and virtual fixtures for assistance in planar path following were implemented on an admittance-controlled robot. The admittance controller uses a velocity gain, so that the speed of the robot is proportional to the force applied by the operator. The level of virtual fixture guidance is determined by the admittance ratio, which is the ratio of the admittance gain of the force components orthogonal to the path to the gain of the force components parallel to the path. In Experiment 1, we found a linear relationship between admittance ratio and performance. In Experiment 2, we examined the effect of admittance ratio on the performance of three tasks: path following, off-path targeting, and obstacle avoidance. An algorithm was developed to select an appropriate admittance ratio based on the nature of the task. Automatic admittance ratio tuning is recommended for next-generation virtual fixtures. Actual or potential applications of this research include surgery, assembly, and manipulation at the macro and micro scales.     

Fuzzy-based-admittance controller for safe natural human–robot interaction

ABSTRACT

In recent years, a great amount of research on physical human–robot interaction has been conducted, and mainly concentrated on safety issues to minimize the risk of accidents to the operator during the cooperation between human and robot. Unfortunately, the identification of inertia and damping matrices in the dynamic admittance model is time-consuming, which is still an open problem of previous admittance controllers. Additionally, the natural cooperation is that cooperative movements are implemented in every degree of freedom in space, which is rarely concerned while it is important to implement more complex cooperative movements, and to help operator feels naturally during the cooperation. This paper presents an alternative admittance controller based on inference mechanism of fuzzy logic to eliminate the identification of inertia and damping matrices during the process of controller formulation in which the end-effector’s velocity is adaptively adjusted via external wrench (force/torque measured by a sensor mounted on end-effector) and power transmitted by the robot. Moreover, the proposed controller also considers end-effector’s full DOF to guarantee the natural human–robot interaction. The fuzzy-admittance controller is evaluated by an experimental set-up of teaching task using 6-DOF manipulator in which manipulator moves passively via the human impact on real-time force/torque sensor mounted on end-effector.     

Robot apology as a post-accident trust-recovery control strategy in industrial human-robot interaction

Due to safety requirements for Human-Robot Interaction (HRI), industrial robots have to meet high standards of safety requirements (ISO 10218). However, even if robots are incapable of causing serious physical harm, they still may influence people's mental and emotional wellbeing, as well as their trust, behaviour and performance in close collaboration. This work uses an HTC Vive Virtual Reality headset to study the potential of using robot control strategies to positively influence human post-accident behaviour. In the designed scenario, a virtual industrial robot first makes sudden unexpected movements, after which it either does or does not attempt to apologise for them. The results show that after the robot tries to communicate with the participants, the robot is reported to be less scary, more predictable and easier to work with. Furthermore, postural analysis shows that the participants who were the most affected by the robot's sudden movement recover 74% of their postural displacement within 60 s after the event if the robot apologised, and only 34% if it did not apologise. It is concluded, that apologies, which are commonly used as a trust-recovery strategy in social robotics, can positively influence people engaged with industrial robotics as well.Relevance to industryFindings can be used as guidelines for designing robot behaviour and trust-recovery control strategies meant to speed up human recovery after a trust-violating event in industrial Human-Robot Interaction.     

Human-robot coexistence and interaction in open industrial cells

Recent research results on human–robot interaction and collaborative robotics are leaving behind the traditional paradigm of robots living in a separated space inside safety cages, allowing humans and robot to work together for completing an increasing number of complex industrial tasks. In this context, safety of the human operator is a main concern. In this paper, we present a framework for ensuring human safety in a robotic cell that allows human–robot coexistence and dependable interaction. The framework is based on a layered control architecture that exploits an effective algorithm for online monitoring of relative human–robot distance using depth sensors. This method allows to modify in real time the robot behavior depending on the user position, without limiting the operative robot workspace in a too conservative way. In order to guarantee redundancy and diversity at the safety level, additional certified laser scanners monitor human–robot proximity in the cell and safe communication protocols and logical units are used for the smooth integration with an industrial software for safe low-level robot control. The implemented concept includes a smart human-machine interface to support in-process collaborative activities and for a contactless interaction with gesture recognition of operator commands. Coexistence and interaction are illustrated and tested in an industrial cell, in which a robot moves a tool that measures the quality of a polished metallic part while the operator performs a close evaluation of the same workpiece.