Reinforcement learning in robotic applications: a comprehensive survey

In recent trends, artificial intelligence (AI) is used for the creation of complex automated control systems. Still, researchers are trying to make a completely autonomous system that resembles human beings. Researchers working in AI think that there is a strong connection present between the learning pattern of human and AI. They have analyzed that machine learning (ML) algorithms can effectively make self-learning systems. ML algorithms are a sub-field of AI in which reinforcement learning (RL) is the only available methodology that resembles the learning mechanism of the human brain. Therefore, RL must take a key role in the creation of autonomous robotic systems. In recent years, RL has been applied on many platforms of the robotic systems like an air-based, under-water, land-based, etc., and got a lot of success in solving complex tasks. In this paper, a brief overview of the application of reinforcement algorithms in robotic science is presented. This survey offered a comprehensive review based on segments as (1) development of RL (2) types of RL algorithm like; Actor-Critic, DeepRL, multi-agent RL and Human-centered algorithm (3) various applications of RL in robotics based on their usage platforms such as land-based, water-based and air-based, (4) RL algorithms/mechanism used in robotic applications. Finally, an open discussion is provided that potentially raises a range of future research directions in robotics. The objective of this survey is to present a guidance point for future research in a more meaningful direction.

     

人机智能技术及系统研究进展综述

人机智能系统是能够实现人机智能协作的机器人系统,近年来成为了机器人领域的研究热点,具有广泛的应用前景。针对人机智能系统技术和应用的国内外研究现状,从人机智能系统的关键技术和典型应用领域两方面进行了进展综述。重点综述了与传统机器人系统存在差异性的人机智能系统关键技术,从建模、交互、协同和优化4个方面的研究进展分别展开论述,对涉及的典型应用领域及典型人机智能系统进行总结,并对人机智能系统发展的挑战和未来研究方向进行了展望。     

Repeatability of Real World Training Experiments: A Case Study

We present a case study of reinforcement learning on a real robot that learns how to back up a trailer and discuss the lessons learned about the importance of proper experimental procedure and design. We identify areas of particular concern to the experimental robotics community at large. In particular, we address concerns pertinent to robotics simulation research, implementing learning algorithms on real robotic hardware, and the difficulties involved with transferring research between the two.     

复杂制造系统建模与优化研究现状及展望

近年来,复杂制造系统及其自动化、智能化和定制化等优势在汽车制造、芯片制造、机器人等领域得到了广泛关注,其建模与优化问题也已成为国内外的研究热点。本文首先介绍了复杂制造系统的研究现状和典型场景应用,比如设备管理、生产过程自动化和生产调度管理。紧接着汇总了常见的建模与优化方法以及所解决的实际工程问题,特别是深度学习、强化学习和合作博弈等方法在复杂制造系统建模与优化中的应用。最后,对复杂制造系统建模和优化问题进行了展望。     

An Anthropomorphic Robotic Platform for Progressive and Adaptive Sensorimotor Learning

In recent years, advances and improvements in engineering and robotics have in part been due to strengthened interactions with the biological sciences. Robots that mimic the complexity and adaptability of biological systems have become a central goal in research and development in robotics. Usually, such a collaboration is addressed to a 2-fold perspective of (i) setting up anthropomorphic platforms as test beds for studies in neuroscience and (ii) promoting new mechatronic and robotic technologies for the development of bio-inspired or humanoid high-performance robotic platforms. This paper provides a brief overview of recent studies on sensorimotor coordination in human motor control and proposes a novel paradigm of adaptive learning for sensorimotor control, based on a multi-network high-level control architecture. The proposed neurobiologically inspired model has been applied to a robotic platform, purposely designed to provide anthropomorphic solutions to neuroscientific requirements. The goal of this work is to use the bio-inspired robotic platform as a test bed for validating the proposed model of high-level sensorimotor control, with the aim of demonstrating adaptive and modular control based on acquired competences, with a higher degree of flexibility and generality than conventional robotic controllers, while preserving their robustness. To this purpose, a set of object-dependent, visually guided reach-and-grasp tasks and the associated training phases were first implemented in a multi-network control architecture in simulation. Subsequently, the offline learning realized in simulation was used to produce the input command of reach-and-grasp to the low-level position control of the robotic platform. Experimental trials demonstrated that the adaptive and modular high-level control allowed reaching and grasping of objects located at different positions and objects of variable size, shape and orientation. A future goal would be to address autonomous and progressive learning based on growing competences.     

Surgical robotics: Reviewing the past,analysing the present,imagining the future

This paper presents an overview of the surgical robotics field, highlighting significant milestones and grouping the various propositions into cohorts. The review does not aim to be exhaustive but rather to highlight how surgical robotics is acting as an enabling technology for minimally invasive surgery. As such, there is a focus on robotic surgical solutions which are commercially available; research efforts which have not gained regulatory approval or entered clinical use are mostly omitted. The practice of robotic surgery is currently largely dominated by the da Vinci system of Intuitive Surgical (Sunnyvale, CA, USA) but other commercial players have now entered the market with surgical robotic products or are appearing in the horizon with medium and long term propositions. Surgical robotics is currently a vibrant research topic and new research directions may lead to the development of very different robotic surgical devices in the future—small, special purpose, lower cost, possibly disposable robots rather than the current large, versatile and capital expensive systems. As the trend towards minimally invasive surgery (MIS) increases, surgery becomes more technically demanding for surgeons and more challenging for medical device technologists and it is clear that surgical robotics has now an established foothold in medicine as an enabling technology of MIS.     

Cognitive neuroscience and robotics: Advancements and future research directions

In recent years, brain-based technologies that capitalise on human abilities to facilitate human–system/robot interactions have been actively explored, especially in brain robotics. Brain–computer interfaces, as applications of this conception, have set a path to convert neural activities recorded by sensors from the human scalp via electroencephalography into valid commands for robot control and task execution. Thanks to the advancement of sensor technologies, non-invasive and invasive sensor headsets have been designed and developed to achieve stable recording of brainwave signals. However, robust and accurate extraction and interpretation of brain signals in brain robotics are critical to reliable task-oriented and opportunistic applications such as brainwave-controlled robotic interactions. In response to this need, pervasive technologies and advanced analytical approaches to translating and merging critical brain functions, behaviours, tasks, and environmental information have been a focus in brain-controlled robotic applications. These methods are composed of signal processing, feature extraction, representation of neural activities, command conversion and robot control. Artificial intelligence algorithms, especially deep learning, are used for the classification, recognition, and identification of patterns and intent underlying brainwaves as a form of electroencephalography. Within the context, this paper provides a comprehensive review of the past and the current status at the intersection of robotics, neuroscience, and artificial intelligence and highlights future research directions.     

Survey on frontiers of language and robotics

ABSTRACT

The understanding and acquisition of a language in a real-world environment is an important task for future robotics services. Natural language processing and cognitive robotics have both been focusing on the problem for decades using machine learning. However, many problems remain unsolved despite significant progress in machine learning (such as deep learning and probabilistic generative models) during the past decade. The remaining problems have not been systematically surveyed and organized, as most of them are highly interdisciplinary challenges for language and robotics. This study conducts a survey on the frontier of the intersection of the research fields of language and robotics, ranging from logic probabilistic programming to designing a competition to evaluate language understanding systems. We focus on cognitive developmental robots that can learn a language from interaction with their environment and unsupervised learning methods that enable robots to learn a language without hand-crafted training data.     

Search and tracking algorithms for swarms of robots: A survey

Target search and tracking is a classical but difficult problem in many research domains, including computer vision, wireless sensor networks and robotics. We review the seminal works that addressed this problem in the area of swarm robotics, which is the application of swarm intelligence principles to the control of multi-robot systems. Robustness, scalability and flexibility, as well as distributed sensing, make swarm robotic systems well suited for the problem of target search and tracking in real-world applications. We classify the works we review according to the variations and aspects of the search and tracking problems they addressed. As this is a particularly application-driven research area, the adopted taxonomy makes this review serve as a quick reference guide to our readers in identifying related works and approaches according to their problem at hand. By no means is this an exhaustive review, but an overview for researchers who are new to the swarm robotics field, to help them easily start off their research.     

蛇形机器人研究现况与进展

仿生技术与机器人技术的结合，使机器人从结构设计到运动模式的选择都有了 新的进展，这大大扩大了机器人的应用领域．本文阐述了仿蛇形机器人的应用背景和研究现 状，并展望了其未来的发展．     

Evolvable Hardware in Evolutionary Robotics

In recent decades the research on Evolutionary Robotics (ER) has developed rapidly. This direction is primarily concerned with the use of evolutionary computing techniques in the design of intelligent and adaptive controllers for robots. Meanwhile, much attention has been paid to a new set of integrated circuits named Evolvable Hardware (EHW), which is capable of reconfiguring its architectures unlimited time based on artificial evolution techniques. This paper surveys the application of evolvable hardware in evolutionary robotics. The evolvable hardware is an emerging research field concerning the development of evolvable robot controllers at the hardware level to adapt to dynamic changes in environments. The context of evolvable hardware and evolutionary robotics is reviewed, and a few representative experiments in the field of robotic hardware evolution are presented. As an alternative to conventional robotic controller designs, the potentialities and limitations of the EHW-based robotic system are discussed and summarized.     

Guest Editorial Special Issue on Rehabilitation Robotics

The 16 papers in this special section have been logically organized into four different groups: enabling technologies for rehabilitation robotics; two robotic systems used for assisted diagnosis of pathologies of interest in the rehabilitation domain; different robotic systems applied to assisted physical rehabilitation; and robotic systems used for assisted psychophysiological rehabilitation.     

Underwater robotics

As the ocean attracts great attention on environmental issues and resources as well as scientific and military tasks, the need for and use of underwater robotic systems has become more apparent. Underwater robotics represents a fast growing research area and promising industry as advanced technologies in various subsystems develop and potential application areas are explored. Great efforts have been made in developing autonomous underwater vehicles (AUVs) to overcome challenging scientific and engineering problems caused by the unstructured and hazardous ocean environment. With the development of new materials, advanced computing and sensory technology, as well as theoretical advancements, R & D activities in the AUV community have increased. This paper describes current state-of-the art in the area of underwater robotics focusing on some key subsystems.     

Developmental learning for autonomous robots

Developmental robotics is concerned with the design of algorithms that promote robot adaptation and learning through qualitative growth of behaviour and increasing levels of competence.This paper uses ideas and inspiration from early infant psychology (up to three months of age) to examine how robot systems could discover the structure of their local sensory-motor spaces and learn how to coordinate these for the control of action.An experimental learning model is described and results from robotic experiments using the model are presented and discussed.     

A review on reinforcement learning for contact-rich robotic manipulation tasks

Research and application of reinforcement learning in robotics for contact-rich manipulation tasks have exploded in recent years. Its ability to cope with unstructured environments and accomplish hard-to-engineer behaviors has led reinforcement learning agents to be increasingly applied in real-life scenarios. However, there is still a long way ahead for reinforcement learning to become a core element in industrial applications. This paper examines the landscape of reinforcement learning and reviews advances in its application in contact-rich tasks from 2017 to the present. The analysis investigates the main research for the most commonly selected tasks for testing reinforcement learning algorithms in both rigid and deformable object manipulation. Additionally, the trends around reinforcement learning associated with serial manipulators are explored as well as the various technological challenges that this machine learning control technique currently presents. Lastly, based on the state-of-the-art and the commonalities among the studies, a framework relating the main concepts of reinforcement learning in contact-rich manipulation tasks is proposed. The final goal of this review is to support the robotics community in future development of systems commanded by reinforcement learning, discuss the main challenges of this technology and suggest future research directions in the domain.     

Operational learning with sensory feedback for controlling a robotic thumb using the posterior auricular muscle

Extra robotic limbs in a robotic system that are designed to augment and expand human abilities have received attention in the field of wearable robotics. We aim to develop a robotic system that is controlled by body parts that are not used on a daily basis in order to augment and expand human abilities. This paper presents operational learning experiments for manipulating a robotic thumb using the posterior auricular muscle, a body part that is not used in everyday life. In these experiments, reaching motions were executed using sensory feedback in the robotic thumb through a device that continuously displays its position. The experimental results indicate the proposed operational learning experiments improve the ability to contract the posterior auricular muscles. In addition, the results indicate the operability of a robotic thumb could be improved by acquiring internal models through repetitive operational learning.     

Robotics in construction: A critical review of the reinforcement learning and imitation learning paradigms

The reinforcement and imitation learning paradigms have the potential to revolutionise robotics. Many successful developments have been reported in literature; however, these approaches have not been explored widely in robotics for construction. The objective of this paper is to consolidate, structure, and summarise research knowledge at the intersection of robotics, reinforcement learning, and construction. A two-strand approach to literature review was employed. A bottom-up approach to analyse in detail a selected number of relevant publications, and a top-down approach in which a large number of papers were analysed to identify common relevant themes and research trends. This study found that research on robotics for construction has not increased significantly since the 1980s, in terms of number of publications. Also, robotics for construction lacks the development of dedicated systems, which limits their effectiveness. Moreover, unlike manufacturing, construction's unstructured and dynamic characteristics are a major challenge for reinforcement and imitation learning approaches. This paper provides a very useful starting point to understating research on robotics for construction by (i) identifying the strengths and limitations of the reinforcement and imitation learning approaches, and (ii) by contextualising the construction robotics problem; both of which will aid to kick-start research on the subject or boost existing research efforts.     

Robots in space into the 21st century

Describes the technological developments which are establishing the foundation for an exciting era of in situ exploration missions to planets, comets and asteroids with advanced robotic systems. Also outlines important concurrent terrestrial applications and spinoffs of the space robotics technology. These include high-precision robotic manipulators for microsurgical operations and dexterous arm control systems.     

A literature survey of the robotic technologies during the COVID-19 pandemic

Since the late 2019, the COVID-19 pandemic has been spread all around the world. The pandemic is a critical challenge to the health and safety of the general public, the medical staff and the medical systems worldwide. It has been globally proposed to utilise robots during the pandemic, to improve the treatment of patients and leverage the load of the medical system. However, there is still a lack of detailed and systematic review of the robotic research for the pandemic, from the technologies’ perspective. Thus a thorough literature survey is conducted in this research and more than 280 publications have been reviewed, with the focus on robotics during the pandemic. The main contribution of this literature survey is to answer two research questions, i.e. 1) what the main research contributions are to combat the pandemic from the robotic technologies’ perspective, and 2) what the promising supporting technologies are needed during and after the pandemic to help and guide future robotics research. The current achievements of robotic technologies are reviewed and discussed in different categories, followed by the identification of the representative work’s technology readiness level. The future research trends and essential technologies are then highlighted, including artificial intelligence, 5 G, big data, wireless sensor network, and human-robot collaboration.     

Software engineering for robotics [From the Guest Editors]

This special issue on software engineering (SE) for robotics captures a snapshot of current research issues and state-of-the-practice in robotic software development, a topic that has recently received increasing attention from the robotics community, thanks to some popular initiatives such as the Microsoft move into robotics.