Automation of SME production with a Cobot system powered by learning-based vision

The features of collaborative robots (cobots), like lightweight, easy programming, and flexibility, meet the production automation requirements in SMEs. However, SME productions are usually in semi-structured or cluttered environments, which raises major challenges in implementing cobot systems in SME production, for instance, increasing the visual perception of cobots, handling diverse tasks, and fast deploying cobot systems, etc. Therefore, we propose an automation framework for SME production by addressing these challenges with cobots to facilitate their production. First, the learning-based vision system is developed and implemented with the You Only Look Once (YOLOv5) for object detection, and with the Convolutional Neural Network cascaded with a Support Vector Machine (CNN-SVM) for quality control of products. Then, the multi-functional gripper system is designed and fabricated to be capable of performing multiple operations and tasks without tool changing, and be able to tolerate a certain level of changes in the environment. After that, a digital twin of the robotic system is developed, which enables the system developer to save time in troubleshooting and debugging, and the customers to have a customized model with all the elements and functions required before system deployment. Finally, the onsite testing of the integrated system is conducted in collaboration with our SME industrial partner, and the test results show that the cobot system can perform the automated production process well and accurately. It is feasible to extend the application of such a cobot system to other SME productions.     

A mixed perception-based human-robot collaborative maintenance approach driven by augmented reality and online deep reinforcement learning

Owing to the fact that the number and complexity of machines is increasing in Industry 4.0, the maintenance process is more time-consuming and labor-intensive, which contains plenty of refined maintenance operations. Fortunately, human-robot collaboration (HRC) can integrate human intelligence into the collaborative robot (cobot), which can realize not merely the nimble and sapiential maintenance operations of personnel but also the reliable and repeated maintenance manipulation of cobots. However, the existing HRC maintenance lacks the precise understand of the maintenance intention, the efficient HRC decision-making for executing robotized maintenance tasks (e.g., repetitive manual tasks) and the convenient interaction interface for executing cognitive tasks (e.g., maintenance preparation and guidance job). Hence, a mixed perception-based human-robot collaborative maintenance approach consisting of three-hierarchy structures is proposed in this paper, which can help reduce the severity of the mentioned problems. In the first stage, a mixed perception module is proposed to help the cobot recognize human safety and maintenance request according to human actions and gestures separately. During the second stage, an improved online deep reinforcement learning (DRL)-enabled decision-making module with the asynchronous structure and the function of anti-disturbance is proposed in this paper, which can realize the execution of robotized maintenance tasks. In the third stage, an augmented reality-assisted (AR) user-friendly interaction interface is designed to help the personnel interact with the cobot and execute the auxiliary maintenance task without the limitation of spatial and human factors. In addition, the auxiliary of maintenance operation can also be supported by the AR-assisted visible guidance. Finally, comparative numerical experiments are implemented in a typical machining workshop, and the experimental results show a competitive performance of the proposed HRC maintenance approach compared with other state-of-the-art methods.     

Research on the acceptance of collaborative robots for the industry 5.0 era -- The mediating effect of perceived competence and the moderating effect of robot use self-efficacy

Collaborative robots (Cobots), an important component of the Industry 5.0 era, have been rapidly entering a variety of industrial application scenarios. However, employees working with them are reluctant to accept cobots into the workplace. Therefore, the traditional technology acceptance model (TAM) is unsuitable for research on the acceptance of cobots with artificial intelligence and the human-robot interaction process. In addition, anthropomorphism cannot explain the lower employee acceptance with the increase of cobots anthropomorphic from the mechanistic level. Therefore, based on the human-robot interaction phenomenon in the emerging industrial field, combined with the Uncanny Valley effect and intergroup threat theory, 300 subjects were invited to conduct an empirical study using experimental vignette methodology (EVM). The findings are as follows: 1) Perceived competence plays a mediating role in the relationship between cobots anthropomorphic and acceptance of cobots; 2) Perceived competence and perceived threat serially mediates the relationship between cobots anthropomorphic and acceptance of cobots; 3) The cobot use self-efficacy plays a moderating role in the relationship between perceived competence and perceived threat. The research results provide a mechanistic explanation for alleviating the low acceptance of cobots, give measures and methods to improve acceptance of cobots and provide solutions for the promotion and application of cobots in the industrial field.     

Investigation of Motion Guidance With Scooter Cobot and Collaborative Learning

This paper investigates how collaborative robots (cobots) can assist a human by mechanically constraining motion to software-defined guide paths, and introduces simple and efficient tools to design ergonomic paths. Analysis of the movements of seven subjects with the Scooter cobot reveals significant differences between guided movements (GM) and free movements (FM). While FM requires learning for each novel task, movements in GM are satisfying from the first trial, require little effort, are faster, smoother, and with fewer back and forth corrections than in FM. Operators rely on path guidance to rotate the Scooter and direct it along curved trajectories. While these advantages demonstrate the strength of the cobot concept, they do not show how guide paths should be defined. We introduce tools to enable the cobot and its operator to collaboratively learn ergonomic guide paths and adapt to changes in the environment. By relying on the haptic sensing, vision, and planning capabilities of the human operator, we can avoid equipping the cobot with complex sensor processing. Experiments with human subjects demonstrate the efficiency and complementarity of these guide paths design tools     

iTP-LfD: Improved task parametrised learning from demonstration for adaptive path generation of cobot

An approach of Task-Parameterised Learning from Demonstration (TP-LfD) aims at automatically adapting the movements of collaborative robots (cobots) to new settings using knowledge learnt from demonstrated paths. The approach is suitable for encoding complex relations between a cobot and its surrounding, i.e., task-relevant objects. However, further efforts are still required to enhance the intelligence and adaptability of TP-LfD for dynamic tasks. With this aim, this paper presents an improved TP-LfD (iTP-LfD) approach to program cobots adaptively for a variety of industrial tasks. iTP-LfD comprises of three main improvements over other developed TP-LfD approaches: 1) detecting generic visual features for frames of reference (frames) in demonstrations for path reproduction in new settings without using complex computer vision algorithms, 2) minimising redundant frames that belong to the same object in demonstrations using a statistical algorithm, and 3) designing a reinforcement learning algorithm to eliminate irrelevant frames. The distinguishing characteristic of the iTP-LfD approach is that optimal frames are identified from demonstrations by simplifying computational complexity, overcoming occlusions in new settings, and boosting the overall performance. Case studies for a variety of industrial tasks involving different objects and scenarios highlight the adaptability and robustness of the iTP-LfD approach.     

Safety assurance mechanisms of collaborative robotic systems in manufacturing

Collaborative robots (cobots) are robots that are designed to collaborate with humans in an open workspace. In contrast to industrial robots in an enclosed environment, cobots need additional mechanisms to assure humans’ safety in collaborations. It is especially true when a cobot is used in manufacturing environment; since the workload or moving mass is usually large enough to hurt human when a contact occurs. In this article, we are interested in understanding the existing studies on cobots, and especially, the safety requirements, and the methods and challenges of safety assurance. The state of the art of safety assurance of cobots is discussed at the aspects of key functional requirements (FRs), collaboration variants, standardizations, and safety mechanisms. The identified technological bottlenecks are (1) acquiring, processing, and fusing diversified data for risk classification, (2) effectively updating the control to avoid any interference in a real-time mode, (3) developing new technologies for the improvement of HMI performances, especially, workloads and speeds, and (4) reducing the overall cost of safety assurance features. To promote cobots in manufacturing applications, the future researches are expected for (1) the systematic theory and methods to design and build cobots with the integration of ergonomic structures, sensing, real-time controls, and human-robot interfaces, (2) intuitive programming, task-driven programming, and skill-based programming which incorporate the risk management and the evaluations of biomechanical load and stopping distance, and (3) advanced instrumentations and algorithms for effective sensing, processing, and fusing of diversified data, and machine learning for high-level complexity and uncertainty. The needs of the safety assurance of integrated robotic systems are specially discussed with two development examples.     

Optimised Learning from Demonstrations for Collaborative Robots

The approach of Learning from Demonstrations (LfD) can support human operators especially those without much programming experience to control a collaborative robot (cobot) in an intuitive and convenient means. Gaussian Mixture Model and Gaussian Mixture Regression (GMM and GMR) are useful tools for implementing such a LfD approach. However, well-performed GMM/GMR require a series of demonstrations without trembling and jerky features, which are challenging to achieve in actual environments. To address this issue, this paper presents a novel optimised approach to improve Gaussian clusters then further GMM/GMR so that LfD enabled cobots can carry out a variety of complex manufacturing tasks effectively. This research has three distinguishing innovative characteristics: 1) a Gaussian noise strategy is designed to scatter demonstrations with trembling and jerky features to better support the optimisation of GMM/GMR; 2) a Simulated Annealing-Reinforcement Learning (SA-RL) based optimisation algorithm is developed to refine the number of Gaussian clusters in eliminating potential under-/over-fitting issues on GMM/GMR; 3) a B-spline based cut-in algorithm is integrated with GMR to improve the adaptability of reproduced solutions for dynamic manufacturing tasks. To verify the approach, cases studies of pick-and-place tasks with different complexities were conducted. Experimental results and comparative analyses showed that this developed approach exhibited good performances in terms of computational efficiency, solution quality and adaptability.     

Design and analysis of kinematically redundant parallel manipulators with configurable platforms

Redundancy can, in general, improve the ability and performance of parallel manipulators by implementing the redundant degrees of freedom to optimize a secondary objective function. Almost all published researches in the area of parallel manipulators redundancy were focused on the design and analysis of redundant parallel manipulators with rigid (nonconfigurable) platforms and on grasping hands to be attached to the platforms. Conventional grippers usually are not appropriate to grasp irregular or large objects. Very few studies focused on the idea of using a configurable platform as a grasping device. This paper highlights the idea of using configurable platforms in both planar and spatial redundant parallel manipulators, and generalizes their analysis. The configurable platform is actually a closed kinematic chain of mobility equal to the degree of redundancy of the manipulator. The additional redundant degrees of freedom are used in reconfiguring the shape of the platform itself. Several designs of kinematically redundant planar and spatial parallel manipulators with configurable platform are presented. Such designs can be used as a grasping device especially for irregular or large objects or even as a micro-positioning device after grasping the object. Screw algebra is used to develop a general framework that can be adapted to analyze the kinematics of any general-geometry planar or spatial kinematically redundant parallel manipulator with configurable platform.     

Frontiers in User-Computer Interaction

In this age of (near-)adequate computing power, the power and usability of the user interface is as key to an application's success as its functionality. Most of the code in modern desktop productivity applications resides in the user interface. But despite its centrality, the user interface field is currently in a rut: the WIMP (Windows, Icons, Menus, Point-and-Click GUI based on keyboard and mouse) has evolved little since it was pioneered by Xerox PARC in the early '70s. Computer and display form factors will change dramatically in the near future and new kinds of interaction devices will soon become available. Desktop environments will be enriched not only with PDAs such as the Newton and Palm Pilot, but also with wearable computers and large-screen displays produced by new projection technology, including office-based immersive virtual reality environments. On the input side, we will finally have speech-recognition and force-feedback devices. Thus we can look forward to user interfaces that are dramatically more powerful and better matched to human sensory capabilities than those dependent solely on keyboard and mouse. 3D interaction widgets controlled by mice or other interaction devices with three or more degrees of freedom are a natural evolution from their two-dimensional WIMP counterparts and can decrease the cognitive distance between widget and task for many tasks that are intrinsically 3D, such as scientific visualization and MCAD. More radical post-WIMP UIs are needed for immersive virtual reality where keyboard and mouse are absent. Immersive VR provides good driving applications for developing post-WIMP UIs based on multimodal interaction that involve more of our senses by combining the use of gesture, speech, and haptics.     

User performance with trackball-mice

Trackball-mice are devices that include both a trackball and a mouse. In this paper we discuss our experiences in building and testing trackball-mouse prototypes. We report four experiments on user performance with the prototypes used as trackball-mice, conventional mice, and in two-handed configuration with a separate trackball for the non-dominant hand. The results show that user performance with the two-handed configuration was better than in one-handed operation of a trackball-mouse and in one-handed operation of a mouse. Trackball-mouse use and conventional mouse use were more evenly matched. However, Trackball-mouse operation involves a skill that most users do not have whereas mouse operation is familiar to most. Therefore, widespread introduction of trackball-mice does not appear to be justified on performance grounds alone. However, trackball-mice can be used as regular mice by ignoring the ball. This makes them compatible with traditional graphical user interfaces while offering two extra degrees of freedom in tasks where they are beneficial.     

Characterization of crosstalk in stereoscopic display devices

Many different types of stereoscopic display devices are used for commercial and research applications. Stereoscopic displays offer the potential to improve performance in detection tasks for medical imaging diagnostic systems. Due to the variety of stereoscopic display technologies, it remains unclear how these compare with each other for detection and estimation tasks. Different stereo devices have different performance trade‐offs due to their display characteristics. Among them, crosstalk is known to affect observer perception of 3D content and might affect detection performance. We measured and report the detailed luminance output and crosstalk characteristics for three different types of stereoscopic display devices. We recorded the effect of other issues on recorded luminance profiles such as viewing angle, use of different eye wear, and screen location. Our results show that the crosstalk signature for viewing 3D content can vary considerably when using different types of 3D glasses for active stereo displays. We also show that significant differences are present in crosstalk signatures when varying the viewing angle from 0 degrees to 20 degrees for a stereo mirror 3D display device. Our detailed characterization can help emulate the effect of crosstalk in conducting computational observer image quality assessment evaluations that minimize costly and time‐consuming human reader studies.     

On 3D input devices

We provide an overview of some of our input device developments, which we designed in response to the need for more advanced 3D interfaces. Some of our devices are more task-specific and others are more general, but all of them support six or more degrees of freedom (DOF) and work in three dimensions. In our work, we try to understand the essential requirements of individual tasks and task combinations to develop corresponding devices and interaction techniques. This is our way of developing input devices for the 3D domain that work better for certain application areas than 2D mouses, gloves, and wands.     

An Attempt to Raise the Level of Software Abstraction in Assembly Robotics through an Apposite Choice of Underlying Mechatronics

Robot manipulators were meant to be the production engineer"s flexible friend. Assembly robots, however, have failed to fulfill their promise. The problem that has continuously plagued robotic assembly is that of spatial uncertainty. It is our thesis that the ubiquitous problem of spatial uncertainty is an artefact of the fact that current industrial manipulators are designed for an operational paradigm that assumes position control is of primary importance. In this paper we propound an alternative approach based on sliding as the primary motion primitive. We first present a model that uses sliding to allow us to raise the level of abstraction of robot programming tasks. We then describe an inherently accommodating, (planar) three degree of freedom, direct-drive robot arm that was constructed to test our approach. Finally, we present data collected from representative (planar) manipulation tasks that substantiate our claims.     

Augmenting Tactile 3D Data Navigation With Pressure Sensing

We present a pressure‐augmented tactile 3D data navigation technique, specifically designed for small devices, motivated by the need to support the interactive visualization beyond traditional workstations. While touch input has been studied extensively on large screens, current techniques do not scale to small and portable devices. We use phone‐based pressure sensing with a binary mapping to separate interaction degrees of freedom (DOF) and thus allow users to easily select different manipulation schemes (e. g., users first perform only rotation and then with a simple pressure input to switch to translation). We compare our technique to traditional 3D‐RST (rotation, scaling, translation) using a docking task in a controlled experiment. The results show that our technique increases the accuracy of interaction, with limited impact on speed. We discuss the implications for 3D interaction design and verify that our results extend to older devices with pseudo pressure and are valid in realistic phone usage scenarios.     

Selective Degree Elevation for Multi‐Sided Bézier Patches

This paper presents a method to selectively elevate the degree of an S‐Patch of arbitrary dimension. We consider not only S‐Patches with 2D domains but 3D and higher‐dimensional domains as well, of which volumetric cage deformations are a subset. We show how to selectively insert control points of a higher degree patch into a lower degree patch while maintaining the polynomial reproduction order of the original patch. This process allows the user to elevate the degree of only one portion of the patch to add new degrees of freedom or maintain continuity with adjacent patches without elevating the degree of the entire patch, which could create far more degrees of freedom than necessary. Finally we show an application to cage‐based deformations where we increase the number of control points by elevating the degree of a subset of cage faces. The result is a cage deformation with higher degree triangular Bézier functions on a subset of cage faces but no interior control points.     

Functional webs for freeform architecture

Rationalization and construction‐aware design dominate the issue of realizability of freeform architecture. The former means the decomposition of an intended shape into parts which are sufficiently simple and efficient to manufacture; the latter refers to a design procedure which already incorporates rationalization. Recent contributions to this topic have been concerned mostly with small‐scale parts, for instance with planar faces of meshes. The present paper deals with another important aspect, namely long‐range parts and supporting structures. It turns out that from the pure geometry viewpoint this means studying families of curves which cover surfaces in certain well‐defined ways. Depending on the application one has in mind, different combinatorial arrangements of curves are required. We here restrict ourselves to so‐called hexagonal webs which correspond to a triangular or tri‐hex decomposition of a surface. The individual curve may have certain special properties, like being planar, being a geodesic, or being part of a circle. Each of these properties is motivated by manufacturability considerations and imposes constraints on the shape of the surface. We investigate the available degrees of freedom, show numerical methods of optimization, and demonstrate the effectivity of our approach and the variability of construction solutions derived from webs by means of actual architectural designs.?     

Fusion Strategies for Minimizing Sensing-Level Uncertainty in Manipulator Control

Humanoid robotic applications require robot to act and behave like human being. Following soft computing like approach human being can think, decide and control himself in unstructured dynamic surroundings, where a great degree of uncertainty exists in the information obtained through sensory organs. In the robotics domain also, one of the key issues in extracting useful knowledge from sensory data is that of coping with information as well as sensory uncertainty at various levels. In this paper a generalized fusion based hybrid classifier (ANN-FDD-FFA) has been developed and applied for validating on generated synthetic data from observation model as well as from real hardware robot. The fusion goal, selected here, is primarily to minimize uncertainties in robotic manipulation tasks that are based on internal (joint sensors) as well as external (vision camera) sensory information. The effectiveness of present methodology has been extensively studied with a specially configured experimental robot having five degrees of freedom and a simulated model of a vision guided manipulator. In the present investigation main uncertainty handling approach includes weighted parameter selection (of geometric fusion) by a trained neural network that is not available in standard manipulator robotic controller designs. These approaches in hybrid configuration has significantly reduce the uncertainty at different levels for faster and more accurate manipulator control as demonstrated here through rigorous simulations and experimentations.     

Impedance controlled human–robot collaborative tooling for edge chamfering and polishing applications

Surface finishing, as the final stage in the manufacturing pipeline, is a key process in determining the quality and life span of a product. Such a task is characterized by low contact forces and minimal material removal from the object surface. Despite the advancements in machine learning and artificial intelligence, human workforce is still irreplaceable in performing such tasks due to superior dexterity and adaptability, but this is often prone to risks such as hand-arm vibration syndrome due to hand-held tools. Therefore, we propose a collaborative approach to assist the human in carrying out such tasks with the help of two case studies: Human–Robot-Collaborative edge chamfering and polishing tasks, based on an impedance controlled collaborative curve tracing technique.We propose a collaborative framework, where the robot assists an operator to guide the end-effector/tool along a pre-defined parametric curve. The algorithm is demonstrated in two scenarios. In the first case, we address a collaborative chamfering task whereas the second case focuses on a polishing application (for straight edges). For these kinds of tasks, the curve to be traced assumes the shape of a straight line along the edge. We make use of the compliant feature of a cobot, which allows the user to physically guide the robot in the task space, to generate a mathematical model for the tool path. From the end-user perspective, this is more intuitive than the classical programming-based path planning approaches. In the process of machining, to enhance the path tracking accuracy and to ensure constant tool-surface contact, we implement guidance virtual fixtures through impedance control. As a result, the machining error is reduced.     

Kinematics and the implementation of an elephant's trunk manipulator and other continuum style robots

Traditionally, robot manipulators have been a simple arrangement of a small number of serially connected links and actuated joints. Though these manipulators prove to be very effective for many tasks, they are not without their limitations, due mainly to their lack of maneuverability or total degrees of freedom. Continuum style (i.e., continuous "back-bone") robots, on the other hand, exhibit a wide range of maneuverability, and can have a large number of degrees of freedom. The motion of continuum style robots is generated through the bending of the robot over a given section; unlike traditional robots where the motion occurs in discrete locations, i.e., joints. The motion of continuum manipulators is often compared to that of biological manipulators such as trunks and tentacles. These continuum style robots can achieve motions that could only be obtainable by a conventionally designed robot with many more degrees of freedom. In this paper we present a detailed formulation and explanation of a novel kinematic model for continuum style robots. The design, construction, and implementation of our continuum style robot called the elephant trunk manipulator is presented. Experimental results are then provided to verify the legitimacy of our model when applied to our physical manipulator. We also provide a set of obstacle avoidance experiments that help to exhibit the practical implementation of both our manipulator and our kinematic model.     

Microcavity white‐emitting OLED devices

Abstract— Microcavity designs for OLED devices with an unpatterned white emitter have the potential to provide greater brightness and larger color gamut than non‐microcavity designs while still enabling lower‐cost large‐format manufacturing. In this paper, such microcavity and non‐microcavity designs are compared. Color filters must still be employed to provide an adequate color gamut. Top‐emitter structures have somewhat greater on‐axis luminance and color gamut, but increased angular change, than bottom‐emitter designs. In a single‐stack bottom‐emitter active‐matrix TFT device using an RGBW format, the use of microcavities is estimated to reduce the average power usage by 35% and the peak power by 58%, while increasing the NTSC ratio for color gamut area by about 10%. Angular luminance and color change is likely to be acceptable, especially for hand‐held applications. Tandem devices employing multiple emitter stacks increase the lifetime of OLED devices but require larger driving voltages; for such devices, microcavity structures are useful although the percentage reduction obtained in power usage is not quite as large. Generally, tandem devices with microcavities have a slightly stronger cavity effect yielding slightly larger color gamut, but also greater angular color and luminance shift. Therefore, microcavity architectures are less appealing for tandem devices.