Proactive human–robot collaboration: Mutual-cognitive,predictable, and self-organising perspectives

Human–Robot Collaboration (HRC) has a pivotal role in smart manufacturing for strict requirements of human-centricity, sustainability, and resilience. However, existing HRC development mainly undertakes either a human-dominant or robot-dominant manner, where human and robotic agents reactively perform operations by following pre-defined instructions, thus far from an efficient integration of robotic automation and human cognition. The stiff human–robot relations fail to be qualified for complex manufacturing tasks and cannot ease the physical and psychological load of human operators. In response to these realistic needs, this paper presents our arguments on the obvious trend, concept, systematic architecture, and enabling technologies of Proactive HRC, serving as a prospective vision and research topic for future work in the human-centric smart manufacturing era. Human–robot symbiotic relation is evolving with a 5C intelligence — from Connection, Coordination, Cyber, Cognition to Coevolution, and finally embracing mutual-cognitive, predictable, and self-organising intelligent capabilities, i.e., the Proactive HRC. With proactive robot control, multiple human and robotic agents collaboratively operate manufacturing tasks, considering each others’ operation needs, desired resources, and qualified complementary capabilities. This paper also highlights current challenges and future research directions, which deserve more research efforts for real-world applications of Proactive HRC. It is hoped that this work can attract more open discussions and provide useful insights to both academic and industrial practitioners in their exploration of human–robot flexible production.     

Human–robot interaction in industrial collaborative robotics: a literature review of the decade 2008–2017

ABSTRACT

Currently, a large number of industrial robots have been deployed to replace or assist humans to perform various repetitive and dangerous manufacturing tasks. However, based on current technological capabilities, such robotics field is rapidly evolving so that humans are not only sharing the same workspace with robots, but also are using robots as useful assistants. Consequently, due to this new type of emerging robotic systems, industrial collaborative robots or cobots, human and robot co-workers have been able to work side-by-side as collaborators to accomplish tasks in industrial environments. Therefore, new human–robot interaction systems have been developed for such systems to be able to utilize the capabilities of both humans and robots. Accordingly, this article presents a literature review of major recent works on human–robot interactions in industrial collaborative robots, conducted during the last decade (between 2008 and 2017). Additionally, the article proposes a tentative classification of the content of these works into several categories and sub-categories. Finally, this paper addresses some challenges of industrial collaborative robotics and explores future research issues.     

Structural design and kinematics of a new parallel reconfigurable robot

Reconfigurable robots can be defined as a group of robots that can have different geometries, thus obtaining different structures derived from the basic one, having different degrees of freedom and workspaces. Thanks to the optimum dexterity they offer, the user can accomplish a large variety of industrial tasks, using a structurally optimized robot leading towards better energy control and efficiency especially in case of batch size production lines where the task (for the robot) may vary periodically. Reconfigurable systems are a challenge for numerous scientists, due to the advantage of dealing with changes and uncertainties on the ever-changing manufacturing market. One of the main problems of reconfigurable robots is the proper structural geometry determination, so that the resulting structure is able to perform a variety of tasks. This paper presents the structural design of an innovative parallel robot with six degrees of freedom and its proposed configurations with five, four, three and two degrees of freedom. The kinematic analysis and the workspace representations of all the presented configurations of the parallel robot, called Recrob, are also presented.     

Difficulty and complexity definitions for assembly task allocation and assignment in human–robot collaborations: A review

This paper presents a literature review on the different aspects of task allocation and assignment problems in human–robot collaboration (HRC) tasks in industrial assembly environments. In future advanced industrial environments, robots and humans are expected to share the same workspace and collaborate to efficiently achieve shared goals. Difficulty- and complexity-aware HRC assembly is necessary for human-centric manufacturing, which is a goal of Industry 5.0. Therefore, the objective of this study is to clarify the definitions of difficulty and complexity used to encourage effective collaboration between humans and robots to leverage the adaptability of humans and the autonomy of robots. To achieve this goal, a systematic review of the following relevant databases for computer science was performed: IEEE Xplore, ScienceDirect, SpringerLink, ACM Digital Library, and ASME Digital Collection. The results extracted from 74 peer-reviewed research articles published until July 2022 were summarized and categorized into four taxonomies for 145 difficulty and complexity definitions from the perspectives of (1) definition-use objectives, (2) evaluation objectives, (3) evaluation factors, and (4) evaluation variables. Next, existing definitions were primarily classified according to the following two criteria to identify potential future studies on the formulation of new definitions for human-centric manufacturing: (1) agent specificity and (2) common aspects in manual and robotic assemblies.     

Improving human robot collaboration through Force/Torque based learning for object manipulation

Human–Robot Collaboration (HRC) is a term used to describe tasks in which robots and humans work together to achieve a goal. Unlike traditional industrial robots, collaborative robots need to be adaptive; able to alter their approach to better suit the situation and the needs of the human partner. As traditional programming techniques can struggle with the complexity required, an emerging approach is to learn a skill by observing human demonstration and imitating the motions; commonly known as Learning from Demonstration (LfD). In this work, we present a LfD methodology that combines an ensemble machine learning algorithm (i.e. Random Forest (RF)) with stochastic regression, using haptic information captured from human demonstration. The capabilities of the proposed method are evaluated using two collaborative tasks; co-manipulation of an object (where the human provides the guidance but the robot handles the objects weight) and collaborative assembly of simple interlocking parts. The proposed method is shown to be capable of imitation learning; interpreting human actions and producing equivalent robot motion across a diverse range of initial and final conditions. After verifying that ensemble machine learning can be utilised for real robotics problems, we propose a further extension utilising Weighted Random Forest (WRF) that attaches weights to each tree based on its performance. It is then shown that the WRF approach outperforms RF in HRC tasks.     

Framework for Assessing Robotic Dexterity within Flexible Manufacturing

With growing demand for flexibility in manufacturing processes, interest in dexterous industrial robots is increasing. To facilitate benchmarking, and to assess the suitability of these robots for flexible manufacturing tasks, there is a need to develop new methods of capturing the relevant performance characteristics of industrial robots. This research aims to show that the Boothroyd-Dewhurst (B-D) Design-For-Assembly method, an established method for optimizing manufacturing processes, can be effectively adopted to form the basis of a comprehensive robotic dexterity assessment within flexible manufacturing. A comparative study is conducted which shows that the B-D classification tables offer the most comprehensive solution due to the range of operations and artifacts considered. Building on these tables, a framework is developed for determining the suitability of a robot system within flexible manufacturing operations. In a sample test-case scenario involving a pick-and-place operation, the framework is shown to produce an accurate estimate of robot performance that can be easily compared to human data. The framework establishes a link between manufacturing operations and robot performance metrics, which addresses the current difficulty in robot integration and highlights the framework’s potential for adoption within flexible manufacturing.     

Human motion prediction for human-robot collaboration

In human-robot collaborative manufacturing, industrial robots would work alongside human workers who jointly perform the assigned tasks seamlessly. A human-robot collaborative manufacturing system is more customised and flexible than conventional manufacturing systems. In the area of assembly, a practical human-robot collaborative assembly system should be able to predict a human worker’s intention and assist human during assembly operations. In response to the requirement, this research proposes a new human-robot collaborative system design. The primary focus of the paper is to model product assembly tasks as a sequence of human motions. Existing human motion recognition techniques are applied to recognise the human motions. Hidden Markov model is used in the motion sequence to generate a motion transition probability matrix. Based on the result, human motion prediction becomes possible. The predicted human motions are evaluated and applied in task-level human-robot collaborative assembly.     

Effects of movement speed and predictability in human–robot collaboration

Human–robot collaboration (HRC) is characterized by a spatiotemporal overlap between the workspaces of the human and the robot and has become a viable option in manufacturing and other industries. However, for companies considering employing HRC it remains unclear how best to configure such a setup, because empirical evidence on human factors requirements remains inconclusive. As robots execute movements at high levels of automation, they adapt their speed and movement path to situational demands. This study therefore experimentally investigated the effects of movement speed and path predictability of an industrial collaborating robot on the human operator. Participants completed tasks together with a robot in an industrial workplace simulated in virtual reality. A lower level of predictability was associated with a loss in task performance, while faster movements resulted in higher‐rated values for task load and anxiety, indicating demands on the operator exceeding the optimum. Implications for productivity and safety and possible advancements in HRC workplaces are discussed.     

Design of service robots

According to the International Federation of Robotics (IFR), "a service robot is a robot which operates semi or fully autonomously to perform services useful to the well being of human and equipment, excluding manufacturing operations" [1]. These devices are typically complex systems requiring the input of knowledge from numerous disciplines. The authors have been using different software engineering techniques for the last 15 years, integrating new paradigms in the service robot development process as they emerged. This has made it possible to achieve rapid development of applications and subsequent maintenance. During the early years (1993?1998), our effortswere directed at the development of software for various kinds of teleoperated robots to performmaintenance tasks in nuclear power plants [2]; during a second phase (1999?2006),we built applications for ship-hull cleaning robots [3]. All this time, we have been applying all the possibilities of software engineering, from the use of paradigms for structured and object-based programming in early developments to the adoption of the current model-driven approach [model-driven engineering (MDE)] [4].     

Effects of automation and task load on task switching during human supervision of multiple semi-autonomous robots in a dynamic environment

The present study assessed the impact of task load and level of automation (LOA) on task switching in participants supervising a team of four or eight semi-autonomous robots in a simulated ‘capture the flag’ game. Participants were faster to perform the same task than when they chose to switch between different task actions. They also took longer to switch between different tasks when supervising the robots at a high compared to a low LOA. Task load, as manipulated by the number of robots to be supervised, did not influence switch costs. The results suggest that the design of future unmanned vehicle (UV) systems should take into account not simply how many UVs an operator can supervise, but also the impact of LOA and task operations on task switching during supervision of multiple UVs.

The findings of this study are relevant for the ergonomics practice of UV systems. This research extends the cognitive theory of task switching to inform the design of UV systems and results show that switching between UVs is an important factor to consider.     

Ergonomics considerations and management action in the implementation of industrial robots

To obtain maximum benefit from the implementation of industrial robots, it is necessary to identify specific ergonomics problems and provide answers to such problems. Special features of a typical industrial robot are described. Specific ergonomics problems are identified and discussed: sociopsychological factors, systems safety design, communications, training, and workplace design. For the successful implementation of industrial robots, management should take timely action with regard to advanced planning procedures, user involvement plans, communication channels, company labor policies, and continuous training programs. The technological change from conventional to advanced manufacturing, such as industrial robots, must be jointly supported by all levels of management and workers. © 2001 John Wiley & Sons, Inc.     

Describing Upper-Body Motions Based on Labanotation for Learning-from-Observation Robots

We have been developing a paradigm that we call learning-from-observation for a robot to automatically acquire a robot program to conduct a series of operations, or for a robot to understand what to do, through observing humans performing the same operations. Since a simple mimicking method to repeat exact joint angles or exact end-effector trajectories does not work well because of the kinematic and dynamic differences between a human and a robot, the proposed method employs intermediate symbolic representations, tasks, for conceptually representing what-to-do through observation. These tasks are subsequently mapped to appropriate robot operations depending on the robot hardware. In the present work, task models for upper-body operations of humanoid robots are presented, which are designed on the basis of Labanotation. Given a series of human operations, we first analyze the upper-body motions and extract certain fixed poses from key frames. These key poses are translated into tasks represented by Labanotation symbols. Then, a robot performs the operations corresponding to those task models. Because tasks based on Labanotation are independent of robot hardware, different robots can share the same observation module, and only different task-mapping modules specific to robot hardware are required. The system was implemented and demonstrated that three different robots can automatically mimic human upper-body operations with a satisfactory level of resemblance.     

Multiobjective layout optimization of robotic cellular manufacturing systems

This paper proposes a multiobjective layout optimization method for the conceptual design of robot cellular manufacturing systems. Robot cellular manufacturing systems utilize one or more flexible robots which can carry out a large number of operations, and can conduct flexible assemble processes. The layout design stage of such manufacturing systems is especially important since fundamental performances of the manufacturing system under consideration are determined at this stage. In this paper, the design criteria for robot cellular manufacturing system layout designs are clarified, and objective functions are formulated. Next, layout design candidates are represented using a sequence-pair scheme to avoid interference between assembly system components, and the use of dummy components is proposed to represent layout areas where components are sparse. A multiobjective genetic algorithm is then used to obtain Pareto optimal solutions for the layout optimization problems. Finally, several numerical examples are provided to illustrate the effectiveness and usefulness of the proposed method.     

An intelligent knowledge-based system for robotic cell design

As research fields in AI accelerate and a greater number of experts are demanded by industry, Expert y tem play an important role in meeting the technological sophistication required in today's competitive world. Industries are demanding the assistance of human experts for solving complicated problems. However, there is a shortage of experts due to this demand. Expert Systems are rapidly becoming one of the major approaches to solve engineering and manufacturing problems. They have been implemented for several practical applications in many decision making problems. Expert Systems are helping major companies to diagnose processes in real time, schedule operations, maintain machinery and to design service and production facilities.

Robots are an integral part of today's manufacturing environment. New tasks are being defined for robots in order to meet the challenges of Flexible Manufacturing Systems. Along with this growth there is an increasing variety of robots to choose from. One of the major problems facing the potential robot user will be his/her choice of an optimum robot for a particular task. Various parameters should be considered and the user should choose an industrial robot whose characteristics satisfy the requirements of the intended task. This paper will present a solution to the problem of selecting an optimum robot by building a Knowledge-Based Expert System. It uses the knowledge base and the rules to determine an optimum robot.     

Emerging research fields in safety and ergonomics in industrial collaborative robotics: A systematic literature review

Human–robot collaboration is a main technology of Industry 4.0 and is currently changing the shop floor of manufacturing companies. Collaborative robots are innovative industrial technologies introduced to help operators to perform manual activities in so called cyber-physical production systems and combine human inimitable abilities with smart machines strengths. Occupational health and safety criteria are of crucial importance in the implementation of collaborative robotics. Therefore, it is necessary to assess the state of the art for the design of safe and ergonomic collaborative robotic workcells. Emerging research fields beyond the state of the art are also of special interest. To achieve this goal this paper uses a systematic literature review methodology to review recent technical scientific bibliography and to identify current and future research fields. Main research themes addressed in the recent scientific literature regarding safety and ergonomics (or human factors) for industrial collaborative robotics were identified and categorized. The emerging research challenges and research fields were identified and analyzed based on the development of publications over time (annual growth).     

Hybrid machine learning for human action recognition and prediction in assembly

As one of the critical elements for smart manufacturing, human-robot collaboration (HRC), which refers to goal-oriented joint activities of humans and collaborative robots in a shared workspace, has gained increasing attention in recent years. HRC is envisioned to break the traditional barrier that separates human workers from robots and greatly improve operational flexibility and productivity. To realize HRC, a robot needs to recognize and predict human actions in order to provide assistance in a safe and collaborative manner. This paper presents a hybrid approach to context-aware human action recognition and prediction, based on the integration of a convolutional neural network (CNN) and variable-length Markov modeling (VMM). Specifically, a bi-stream CNN structure parses human and object information embedded in video images as the spatial context for action recognition and collaboration context identification. The dependencies embedded in the action sequences are subsequently analyzed by a VMM, which adaptively determines the optimal number of current and past actions that need to be considered in order to maximize the probability of accurate future action prediction. The effectiveness of the developed method is evaluated experimentally on a testbed which simulates an assembly environment. High accuracy in both action recognition and prediction is demonstrated.     

How human-robot collaboration impacts construction productivity: An agent-based multi-fidelity modeling approach

Though construction robots have drawn attention in research and practice for decades, human-robot collaboration (HRC) remains important to conduct complex construction tasks. Considering its complexity and uniqueness, it is still unclear how HRC process will impact construction productivity, which is difficult to handle with conventional methods such as field tests, mathematical modeling and physical simulation approaches. To this end, an agent-based (AB) multi-fidelity modeling approach is introduced to simulate and evaluate how HRC influences construction productivity. A high-fidelity model is first proposed for a scenario with one robot. Then, a low-fidelity model is established to extract key parameters that capture the inner relationship among scenarios. The multi-fidelity models work together to simulate complex scenarios. Based on the simulation model, the twofold influence of HRC on productivity, namely the supplement strategy on the worker side, and the design for proactive interaction on the robot side, are fully investigated. Experimental results show that: 1) the proposed approach is feasible and flexible for simulation of complex HRC processes, and can cover multiple collaboration and interaction modes; 2) the influence of the supplement strategy is simple when there is only one robot, where lower Check Interval (CI) and higher Supplement Limit (SL) will improve productivity. But the influence becomes much more complicated when there are more robots due to the internal competition among robots for the limited time of workers; 3) HRC has a scale effect on productivity per robot, which means the productivity improves if there are more robots and workers, even if the human-robot ratio remains the same; 4) introducing proactive interaction between robots and workers could improve productivity significantly, up to 22% in our experiments, which further depends on the supplement strategy and the human-robot ratio. Overall, this research contributes an integrated approach to simulate and evaluate HRC’s impacts on productivity as well as valuable insights on how to optimize HRC for better performance and occupational health. The proposed approach is also useful for the evaluation and development of new robots.     

Human perception of robot safe speed and idle time

Operators and users of robotic systems perform tasks which require close proximity to dangerous moving parts. Two experiments were performed to assess human perception of safe robot arm speed and idling times. Experiment 1 was designed to determine the maximum safe speed of robots. Subjects were asked to adjust the robot speeds. Perceived safe speeds were indicated for two different types of robots. Experiment 2 was designed to determine safe programmed idle time of robots. Subjects were asked to enter the robot work envelope when a programmed idle was perceived to be caused by a malfunction. Safe idle times were reported for two different robot speeds during operational cycles.     

How to include User eXperience in the design of Human-Robot Interaction

In recent years Human-Robot Collaboration (HRC) has become a strategic research field, considering the emergent need for common collaborative execution of manufacturing tasks, shared between humans and robots within the modern factories. However, the majority of the research focuses on the technological aspects and enabling technologies, mainly directing to the robotic side, and usually neglecting the human factors. This work deals with including the needs of the humans interacting with robots in the design in human-robot interaction (HRI). In particular, the paper proposes a user experience (UX)-oriented structured method to investigate the human-robot dialogue to map the interaction with robots during the execution of shared tasks, and to finally elicit the requirements for the design of valuable HRI. The research adopted the proposed method to an industrial case focused on assembly operations supported by collaborative robots and AGVs (Automated Guided Vehicles). A multidisciplinary team was created to map the HRI for the specific case with the final aim to define the requirements for the design of the system interfaces. The novelty of the proposed approach is the inclusion of typically interaction design tools focusing in the analysis of the UX into the design of the system components, without merely focusing on the technological issues. Experimental results highlighted the validity of the proposed method to identify the interaction needs and to drive the interface design.