Three-dimensional swimming tadpole mini-robot using three-axis Helmholtz coils

Electromagnetic-actuated robotic systems have been studied recently for special purposes. Because these systems use external magnetic fields to control their robots, the robots can have simple structures and move with much freedom. In particular, these electromagnetic actuation (EMA) systems are being widely adopted for the actuation of biomedical mini-robots and microrobots for minimally invasive surgery (MIS) and diagnosis. We previously reported, as a feasible biomedical robot, the biomimetic swimming tadpole mini-robot, which can only swim above water. Indeed, the two-dimensional (D) plane swimming tadpole mini-robot is limited in its use because of its motility in the 2D plane. Therefore, this paper proposes a 3D swimming tadpole mini-robot that can move freely in water. First, in the proposed 3D swimming tadpole mini-robot, the buoyancy force was regulated for subaqueous swimming, and the permanent magnet was rearranged for precise movement. Second, to attain a 3D swimming motion, the actuation mechanism of the robot was developed using an EMA system. Finally, various experiments verified that the proposed 3D swimming tadpole mini-robot can swim freely in a 3D water environment.     

Multi-Agent Dynamic Area Coverage Based on Reinforcement Learning with Connected Agents

Dynamic area coverage with small unmanned aerial vehicle (UAV) systems is one of the major research topics due to limited payloads and the difficulty of decentralized decision-making process. Collaborative behavior of a group of UAVs in an unknown environment is another hard problem to be solved. In this paper, we propose a method for decentralized execution of multi-UAVs for dynamic area coverage problems. The proposed decentralized decision-making dynamic area coverage (DDMDAC) method utilizes reinforcement learning (RL) where each UAV is represented by an intelligent agent that learns policies to create collaborative behaviors in partially observable environment. Intelligent agents increase their global observations by gathering information about the environment by connecting with other agents. The connectivity provides a consensus for the decision-making process, while each agent takes decisions. At each step, agents acquire all reachable agents’ states, determine the optimum location for maximal area coverage and receive reward using the covered rate on the target area, respectively. The method was tested in a multi-agent actor-critic simulation platform. In the study, it has been considered that each UAV has a certain communication distance as in real applications. The results show that UAVs with limited communication distance can act jointly in the target area and can successfully cover the area without guidance from the central command unit.     

Software Agent with Reinforcement Learning Approach for Medical Image Segmentation

Many image segmentation solutions are problem-based. Medical images have very similar grey level and texture among the interested objects. Therefore, medical image segmentation requires improvements although there have been researches done since the last few decades. We design a self-learning framework to extract several objects of interest simultaneously from Computed Tomography (CT) images. Our segmentation method has a learning phase that is based on reinforcement learning (RL) system. Each RL agent works on a particular sub-image of an input image to find a suitable value for each object in it. The RL system is define by state, action and reward. We defined some actions for each state in the sub-image. A reward function computes reward for each action of the RL agent. Finally, the valuable information, from discovering all states of the interest objects, will be stored in a Q-matrix and the final result can be applied in segmentation of similar images. The experimental results for cranial CT images demonstrated segmentation accuracy above 95%.     

The student experience of a collaborative e-learning university module

The aim of this paper is to present a picture of student experience of a collaborative e-learning module in an asynchronous e-learning environment. A distance learning module on music education worth five credit points for a bachelor online degree for primary school educating teachers was assessed using a self-evaluation questionnaire that gathered quantitative and qualitative data about student satisfaction of the collaborative e-learning activity. The quantitative part of the questionnaire consisted of 27 closed questions on a 10-point Likert scale and offered data about satisfaction with the module. The qualitative part of the questionnaire provided an insight into the participant perspective of the online collaborative experience. General open questions on satisfaction and dissatisfaction were analyzed with an inductive analysis which showed the evaluation criteria used by 92 students. Results of the analysis showed five themes of the participants’ perspectives, which were interpreted by the researcher as: teamwork, cognitive, operating, organizing, and emotive/ethic for the positive aspects and teamwork, operating, organizing, and emotive/ethic for the aspects to be improved. The aspects that were associated with satisfaction include: collaborating, comparing ideas, sharing knowledge and skills to support each other, peer learning, analyzing and integrating different points of view, the usability of the platform, group planning and workload management. Aspects of the student learning experience that should inform the improvements of e-learning include: more collaboration between students since some students engage differently; more coordination and organization, the workload management in the group activities, some technical problems such as updating modifications. The participants’ results in the module increased their didactic potential as primary school teachers. The findings are discussed in relation to their potential impact on developing collaborative activities addressed to teacher education in distance learning. Implications for future research are also considered.     

An ontology engineering approach to the realization of theory-driven group formation

One of the main difficulties during the design of collaborative learning activities is adequate group formation. In any type of collaboration, group formation plays a critical role in the learners’ acceptance of group activities, as well as the success of the collaborative learning process. Nevertheless, to propose both an effective and pedagogically sound group formation is a complex issue due to multiple factors that influence group arrangement. The current (and previous) learner’s knowledge and skills, the roles and strategies used by learners to interact among themselves, and the teacher’s preferences are some examples of factors to be considered while forming groups. To identify which factors are essential (or desired) in effective group formation, a well-structured and formalized representation of collaborative learning processes, supported by a strong pedagogical basis, is desirable. Thus, the main goal of this paper is to present an ontology that works as a framework based on learning theories that facilitate group formation and collaborative learning design. The ontology provides the necessary formalization to represent collaborative learning and its processes, while learning theories provide support in making pedagogical decisions such as gathering learners in groups and planning the scenario where the collaboration will take place. Although the use of learning theories to support collaborative learning is open for criticism, we identify that they provide important information which can be useful in allowing for more effective learning. To validate the usefulness and effectiveness of this approach, we use this ontology to form and run group activities carried out by four instructors and 20 participants. The experiment was utilized as a proof-of-concept and the results suggest that our ontological framework facilitates the effective design of group activities, and can positively affect the performance of individuals during group learning.     

A system design framework-driven implementation of a learning collaboratory

This paper describes a design process to support the development of a learning collaboratory, a distributed, computer-based, virtual space for learning and work. A learning collaboratory, as a distributed distance learning environment, offers great opportunities to expand the way people teach and learn and to broaden educational opportunities to an ever increasing range of learners. The challenge is to design distance learning technologies that engender meaningful learning experiences that take full advantage of the power of computer-mediated communication to support innovative learner-centered and collaborative interactions between students, teachers, subject experts, and resources. First, the paper describes the learning collaboratory design framework (LUCIDIFY), a design process that integrates methods and concepts from cognitive systems engineering, theories of learning and instruction, distributed computing, and computer-supported collaborative learning to guide the principled design of learning collaboratories. Next, the paper describes how LUCIDIFY was used in the design and implementation of the collaborative learning environment for operational systems (CLEOS), a learning collaboratory for teachers, students, and practitioners in the physical sciences. CLEOS features two virtual instrument tutorials, an asynchronous messaging system, a project-based design and management application, and a collaborative multi-user domain infrastructure.     

Supervised fuzzy reinforcement learning for robot navigation

This paper addresses a new method for combination of supervised learning and reinforcement learning (RL). Applying supervised learning in robot navigation encounters serious challenges such as inconsistent and noisy data, difficulty for gathering training data, and high error in training data. RL capabilities such as training only by one evaluation scalar signal, and high degree of exploration have encouraged researchers to use RL in robot navigation problem. However, RL algorithms are time consuming as well as suffer from high failure rate in the training phase. Here, we propose Supervised Fuzzy Sarsa Learning (SFSL) as a novel idea for utilizing advantages of both supervised and reinforcement learning algorithms. A zero order Takagi–Sugeno fuzzy controller with some candidate actions for each rule is considered as the main module of robot's controller. The aim of training is to find the best action for each fuzzy rule. In the first step, a human supervisor drives an E-puck robot within the environment and the training data are gathered. In the second step as a hard tuning, the training data are used for initializing the value (worth) of each candidate action in the fuzzy rules. Afterwards, the fuzzy Sarsa learning module, as a critic-only based fuzzy reinforcement learner, fine tunes the parameters of conclusion parts of the fuzzy controller online. The proposed algorithm is used for driving E-puck robot in the environment with obstacles. The experiment results show that the proposed approach decreases the learning time and the number of failures; also it improves the quality of the robot's motion in the testing environments.     

一种面向个性化协同学习的任务生成方法

现有协同学习应用无法很好地支持学习任务的生成以及学习者的个性化学习.针对此问题,提出了一种面向个性化协同学习的学习任务生成方法.该方法在学习任务形式化描述的基础上,通过学习者分组、确定学习资源、分解学习单元、分配学习模式以及生成事件序列等步骤,生成既符合学习者群体认知水平,又符合个体学习者个性特征的协同学习任务.根据此任务,可以较好地实现网络环境下群体学习者的个性化协同学习.目前,该方法已在Smart-Realcalss网络教学系统中得到应用.     

一种基于知识点结构的分布式协同学习一致性模型分析

现代远程教育是基于互联网和终端实现教学活动,协同学习是提高远程学习效能感的重要方式,需要深入研究基于协同学习语义的一致性维护、协同感知等关键技术,以达到足够高效的、自然的互动化及个性化学习。提出一种远程教育特点下协同学习的一致性模型及算法分析,在此模型基础上,形成基于知识点结构的特有的操作转换算法来保持一致性。     

基于协同多目标算法的多机器人路径规划

多机器人路径规划是群体机器人协同工作的前提,其特点是在防碰撞与避障的前提下追求多方面资源的最小消耗.针对这一特点,提出协同非支配排序遗传算法,解决具有多个优化目标的多机器人路径规划问题;运用改进的多目标优化算法,克服多目标优化取权值的不足,同时考虑机器人能源与时间两大资源,以多机器人的路径总长度、总平滑度、总耗时为规划目标.同时引入合作型协同算法框架,将难以求解的多变量问题分组求解.每个机器人的路径视为子种群,子种群通过带精英策略的非支配排序遗传算法,进化并筛选出子种群的部分进入协同进化,每次迭代更新外部的精英解集,最终生成一组非支配路径解.仿真结果表明,在栅格地图环境下,本文算法可有效实现多移动机器人的多优化目标路径规划.     

Real-time collaboration of virtual laboratories through the Internet

Web-based learning environments are becoming increasingly popular in higher education. One of the most important web-learning resources is the virtual laboratory (VL), which gives students an easy way for training and learning through the Internet. Moreover, on-line collaborative communication represents a practical method to transmit the knowledge and experience from the teacher to students overcoming physical distance and isolation. Considering these facts, the authors of this document have developed a new dynamic collaborative e-learning system which combines the main advantages of virtual laboratories and collaborative learning practices. In this system, the virtual laboratories are based on Java applets which have embedded simulations developed in Easy Java Simulations (EJS), an open-source tool for teachers who do not need complex programming skills. The collaborative e-learning is based on a real-time synchronized communication among these Java applets. Therefore, this original approach provides a new tool which integrates virtual laboratories inside a synchronous collaborative e-learning framework. This paper describes the main features of this system and its successful application in a distance education environment among different universities from Spain.     

Agents-supported adaptive group awareness: smart distance andWWWaware

We study how agents can facilitate and mediate interaction, communication, and cooperation among people. We propose the concepts of a smart distance and an awareness network in a distributed collaborative environment. We illustrate the architecture of an agent-mediated collaborative system - the agent-buddy system that can create a sense of group presence and, at the same time, preserve the privacy of each user. Virtual springs systems are used to model the awareness degrees among team members. Each agent makes decisions by considering multiple factors. The goal of the multi-agent team is to minimize the global awareness frustrations with respect to different kinds of tasks. Empirical studies were conducted to analyze the influence of individual behavior on global performance for various kinds of tasks     

Reinforcement learning in spacecraft control applications: Advances,prospects, and challenges

This paper presents and analyzes Reinforcement Learning (RL) based approaches to solve spacecraft control problems. Different application fields are considered, e.g., guidance, navigation and control systems for spacecraft landing on celestial bodies, constellation orbital control, and maneuver planning in orbit transfers. It is discussed how RL solutions can address the emerging needs of designing spacecraft with highly autonomous on-board capabilities and implementing controllers (i.e., RL agents) robust to system uncertainties and adaptive to changing environments. For each application field, the RL framework core elements (e.g., the reward function, the RL algorithm and the environment model used for the RL agent training) are discussed with the aim of providing some guidelines in the formulation of spacecraft control problems via a RL framework. At the same time, the adoption of RL in real space projects is also analyzed. Different open points are identified and discussed, e.g., the availability of high-fidelity simulators for the RL agent training and the verification of RL-based solutions. This way, recommendations for future work are proposed with the aim of reducing the technological gap between the solutions proposed by the academic community and the needs/requirements of the space industry.     

分层强化学习综述

强化学习（reinforcement learning）是机器学习和人工智能领域的重要分支,近年来受到社会各界和企业的广泛关注。强化学习算法要解决的主要问题是,智能体如何直接与环境进行交互来学习策略。但是当状态空间维度增加时,传统的强化学习方法往往面临着维度灾难,难以取得好的学习效果。分层强化学习（hierarchical reinforcement learning）致力于将一个复杂的强化学习问题分解成几个子问题并分别解决,可以取得比直接解决整个问题更好的效果。分层强化学习是解决大规模强化学习问题的潜在途径,然而其受到的关注不高。本文将介绍和回顾分层强化学习的几大类方法。     

Integration of imitation learning using GAIL and reinforcement learning using task-achievement rewards via probabilistic graphical model

The integration of reinforcement learning (RL) and imitation learning (IL) is an important problem that has long been studied in the field of intelligent robotics. RL optimizes policies to maximize the cumulative reward, whereas IL attempts to extract general knowledge about the trajectories demonstrated by experts, i.e, demonstrators. Because each has its own drawbacks, many methods combining them and compensating for each set of drawbacks have been explored thus far. However, many of these methods are heuristic and do not have a solid theoretical basis. This paper presents a new theory for integrating RL and IL by extending the probabilistic graphical model (PGM) framework for RL, control as inference. We develop a new PGM for RL with multiple types of rewards, called probabilistic graphical model for Markov decision processes with multiple optimality emissions (pMDP-MO). Furthermore, we demonstrate that the integrated learning method of RL and IL can be formulated as a probabilistic inference of policies on pMDP-MO by considering the discriminator in generative adversarial imitation learning (GAIL) as an additional optimality emission. We adapt the GAIL and task-achievement reward to our proposed framework, achieving significantly better performance than policies trained with baseline methods.     

Students’ experiences with collaborative learning in asynchronous Computer-Supported Collaborative Learning environments

This paper describes an explorative study carried out to gain response from distance students on their experiences with collaborative learning in asynchronous computer supported collaborative learning (CSCL) environments. In addition, this study also attempts to have a good grip of crucial aspects concerning collaborative learning. The study was undertaken among distance learners from the Open University of the Netherlands who were working in groups of 4–11 persons. During and after the course students’ experiences with collaborative learning were measured and after the course also students’ satisfaction with collaborative learning was assessed. The finding revealed that distance learners appreciate the opportunities to work collaboratively. They show positive experiences and are quite satisfied with collaborative learning. This study also sought to explore individuals as well as course characteristics that influenced aspects of collaborative learning, and to search aspects of collaborative learning that influenced students’ satisfaction. The findings suggested that a group product influences group process regulation and group cohesion influences students’ satisfaction with collaborative learning.     

Evolutionary Development of Hierarchical Learning Structures

Hierarchical reinforcement learning (RL) algorithms can learn a policy faster than standard RL algorithms. However, the applicability of hierarchical RL algorithms is limited by the fact that the task decomposition has to be performed in advance by the human designer. We propose a Lamarckian evolutionary approach for automatic development of the learning structure in hierarchical RL. The proposed method combines the MAXQ hierarchical RL method and genetic programming (GP). In the MAXQ framework, a subtask can optimize the policy independently of its parent task's policy, which makes it possible to reuse learned policies of the subtasks. In the proposed method, the MAXQ method learns the policy based on the task hierarchies obtained by GP, while the GP explores the appropriate hierarchies using the result of the MAXQ method. To show the validity of the proposed method, we have performed simulation experiments for a foraging task in three different environmental settings. The results show strong interconnection between the obtained learning structures and the given task environments. The main conclusion of the experiments is that the GP can find a minimal strategy, i.e., a hierarchy that minimizes the number of primitive subtasks that can be executed for each type of situation. The experimental results for the most challenging environment also show that the policies of the subtasks can continue to improve, even after the structure of the hierarchy has been evolutionary stabilized, as an effect of Lamarckian mechanisms     

A web based collaborative testing environment

This paper presents a computer supported collaborative testing system built upon the Siette web-based assessment environment. The application poses the same set of questions to a group of students. Each student in the group should answer the same question twice. An initial response is given individually, without knowing the answers of others. Then the system provides some tools to show the other partners' responses, to support distance collaboration. Finally a second individual answer is requested. In this way assessment and collaboration activities are interlaced. At the end of a collaborative testing session, each student will have two scores: the initial score and the final score. Three sets of experiments have been carried out: (1) a set of experiments designed to evaluate and fine tune the application, improve usability, and to collect users' feelings and opinions about the system; (2) a second set of experiments to analyze the impact of collaboration in test results, comparing individual and group performance, and analyzing the factors that correlate to those results; and (3) a set of experiments designed to measure individual short-term learning directly related to the collaborative testing activity. We study whether the use of the system is associated with actual learning, and whether this learning is directly related to collaboration between students. Our studies confirm previous results and provide the following evidence (1) the performance increase is directly related to the access to other partners' answers; (2) a student tends to reach a common answer in most cases; and (3) the consensus is highly correlated with the correct response. Moreover, we have found evidence indicating that most of the students really do learn from collaborative testing. High-performing students improve by self-reflection, regardless the composition of the group, but low-performing students need to be in a group with higher-performing students in order to improve.     

Synergizing reinforcement learning and game theory—A new direction for control

Reinforcement learning (RL) has now evolved as a major technique for adaptive optimal control of nonlinear systems. However, majority of the RL algorithms proposed so far impose a strong constraint on the structure of environment dynamics by assuming that it operates as a Markov decision process (MDP). An MDP framework envisages a single agent operating in a stationary environment thereby limiting the scope of application of RL to control problems. Recently, a new direction of research has focused on proposing Markov games as an alternative system model to enhance the generality and robustness of the RL based approaches. This paper aims to present this new direction that seeks to synergize broad areas of RL and Game theory, as an interesting and challenging avenue for designing intelligent and reliable controllers. First, we briefly review some representative RL algorithms for the sake of completeness and then describe the recent direction that seeks to integrate RL and game theory. Finally, open issues are identified and future research directions outlined.     

Optimization of 3.5-in. HDD spindle motors for OP-vibration performance: theoretical prediction and experimental verification

The purpose of this paper is to optimize OP-vibration performance of 3.5-in. hard disk drive (HDD) spindle motors through theoretical prediction and experimental verification. OP-vibration performance of HDD is closely related to the first rocking vibration of spindle motors because excited frequencies of 3.5-in. HDD from the environment are mostly below 500 Hz and the first rocking vibration is the only resonance in the corresponding frequencies. Therefore, minimizing first rocking vibration leads to improve OP-vibration performance of the spindle motors. In order to minimize the first rocking vibration key parameters of FDB spindle motors were selected from a previous work done by Heo and Shen (Microsyst Technol 11:1204–1213, 2005). Then, the selected parameters have been optimized to minimize the first rocking vibration through a theoretical model developed at University of Washington. Then, experiments with ten prototype FDB spindle motors have been conducted to verify the theoretical results. Each prototype motor has different spindle parameter configurations including bearing coefficients, bearing locations, and center of gravity location, etc. Also, this paper demonstrated that radial measurements of spindle rocking vibration have better correlation with OP-vibration performance than axial measurements through PES measurements. Finally, the optimized design has been manufactured by a motor maker and has also successfully verified the theoretical prediction experimentally.