Learning strategies for task delegation in norm-governed environments

How do I choose whom to delegate a task to? This is an important question for an autonomous agent collaborating with others to solve a problem. Were similar proposals accepted from similar agents in similar circumstances? What arguments were most convincing? What are the costs incurred in putting certain arguments forward? Can I exploit domain knowledge to improve the outcome of delegation decisions? In this paper, we present an agent decision-making mechanism where models of other agents are refined through evidence from past dialogues and domain knowledge, and where these models are used to guide future delegation decisions. Our approach combines ontological reasoning, argumentation and machine learning in a novel way, which exploits decision theory for guiding argumentation strategies. Using our approach, intelligent agents can autonomously reason about the restrictions (e.g., policies/norms) that others are operating with, and make informed decisions about whom to delegate a task to. In a set of experiments, we demonstrate the utility of this novel combination of techniques. Our empirical evaluation shows that decision-theory, machine learning and ontology reasoning techniques can significantly improve dialogical outcomes.     

A comparison of learning strategies for biologically constrained development of gaze control on an iCub robot

Gaze control requires the coordination of movements of both eyes and head to fixate on a target. We present a biologically constrained architecture for gaze control and show how the relationships between the coupled sensorimotor systems can be learnt autonomously from scratch, allowing for adaptation as the system grows or changes. Infant studies suggest developmental learning strategies, which can be applied to sensorimotor learning in humanoid robots. We examine two strategies (sequential and synchronous) for the learning of eye and head coupled mappings, and give results from implementations on an iCub robot. The results show that the developmental approach can give fast, cumulative, on-line learning of coupled sensorimotor systems.     

Autonomous Path Planning and Experiment Study of Free-floating Space Robot for Target Capturing

Space robotic systems are expected to play an increasingly important role in the future. The robotic on-orbital service, whose key is the capturing technology, becomes research hot in recent years. In this paper, the authors propose an autonomous path planning method for target capturing. The task is described in Cartesian space and it can drive the manipulator to approach the target along the closest path. Firstly, the target feature is extracted based on the measured information via the hand-eye camera, and the target pose (position and orientation) and velocities (linear velocity and angular velocity) are estimated using Kalman filtering technology. Then, a numerically feasible approach is presented to plan the manipulator motion and avoid the dynamic singularities, which are transformed into real-time kinematic singularities avoiding problem. Thirdly, the potential disturbance on the base due to the manipulator’s motion is estimated, and the joint rates are autonomously adjusted to reduce the disturbance if it is beyond the allowed bound. At last, a ground experiment system is set up based on the concept of dynamic emulation and kinematic equivalence. With the experiment system, the autonomous target capturing experiments are conducted. The experiment results validate the proposed algorithm.     

Robot arm reaching through neural inversions and reinforcement learning

We present a neural method that computes the inverse kinematics of any kind of robot manipulators, both redundant and non-redundant. Inverse kinematics solutions are obtained through the inversion of a neural network that has been previously trained to approximate the manipulator forward kinematics. The inversion provides difference vectors in the joint space from difference vectors in the workspace. Our differential inverse kinematics (DIV) approach can be viewed as a neural network implementation of the Jacobian transpose method for arm kinematic control that does not require previous knowledge of the arm forward kinematics. Redundancy can be exploited to obtain a special inverse kinematic solution that meets a particular constraint (e.g. joint limit avoidance) by inverting an additional neural network The usefulness of our DIV approach is further illustrated with sensor-based multilink manipulators that learn collision-free reaching motions in unknown environments. For this task, the neural controller has two modules: a reinforcement-based action generator (AG) and a DIV module that computes goal vectors in the joint space. The actions given by the AG are interpreted with regard to those goal vectors.     

Reduction of state space in reinforcement learning by sensor selection

Much research has been conducted on the application of reinforcement learning to robots. Learning time is a matter of concern in reinforcement learning. In reinforcement learning, information from sensors is projected on to a state space. A robot learns the correspondence between each state and action in state space and determines the best correspondence. When the state space is expanded according to the number of sensors, the number of correspondences learnt by the robot is increased. Therefore, learning the best correspondence becomes time consuming. In this study, we focus on the importance of sensors for a robot to perform a particular task. The sensors that are applicable to a task differ for different tasks. A robot does not need to use all installed sensors to perform a task. The state space should consist of only those sensors that are essential to a task. Using such a state space consisting of only important sensors, a robot can learn correspondences faster than in the case of a state space consisting of all installed sensors. Therefore, in this paper, we propose a relatively fast learning system in which a robot can autonomously select those sensors that are essential to a task and a state space for only such important sensors is constructed. We define the measure of importance of a sensor for a task. The measure is the coefficient of correlation between the value of each sensor and reward in reinforcement learning. A robot determines the importance of sensors based on this correlation. Consequently, the state space is reduced based on the importance of sensors. Thus, the robot can efficiently learn correspondences owing to the reduced state space. We confirm the effectiveness of our proposed system through a simulation.     

A feature selection method for a sample-based stochastic policy

Stochastic policy gradient methods have been applied to a variety of robot control tasks such as robot’s acquisition of motor skills because they have an advantage in learning in high-dimensional and continuous feature spaces by combining some heuristics like motor primitives. However, when we apply one of them to a real-world task, it is difficult to represent the task well by designing the policy function and the feature space due to the lack of enough prior knowledge about the task. In this research, we propose a method to extract a preferred feature space autonomously to achieve a task using a stochastic policy gradient method for a sample-based policy. We apply our method to a control of linear dynamical system and the computer simulation result shows that a desirable controller is obtained and that the performance of the controller is improved by the feature selection.     

A Monte Carlo EM approach for partially observable diffusion processes: theory and applications to neural networks

We present a Monte Carlo approach for training partially observable diffusion processes. We apply the approach to diffusion networks, a stochastic version of continuous recurrent neural networks. The approach is aimed at learning probability distributions of continuous paths, not just expected values. Interestingly, the relevant activation statistics used by the learning rule presented here are inner products in the Hilbert space of square integrable functions. These inner products can be computed using Hebbian operations and do not require backpropagation of error signals. Moreover, standard kernel methods could potentially be applied to compute such inner products. We propose that the main reason that recurrent neural networks have not worked well in engineering applications (e.g., speech recognition) is that they implicitly rely on a very simplistic likelihood model. The diffusion network approach proposed here is much richer and may open new avenues for applications of recurrent neural networks. We present some analysis and simulations to support this view. Very encouraging results were obtained on a visual speech recognition task in which neural networks outperformed hidden Markov models.     

Model-free learning on robot kinematic chains using a nested multi-agent topology

This paper proposes a model-free learning scheme for the developmental acquisition of robot kinematic control and dexterous manipulation skills. The approach is based on a nested-hierarchical multi-agent architecture that intuitively encapsulates the topology of robot kinematic chains, where the activity of each independent degree-of-freedom (DOF) is finally mapped onto a distinct agent. Each one of those agents progressively evolves a local kinematic control strategy in a game-theoretic sense, that is, based on a partial (local) view of the whole system topology, which is incrementally updated through a recursive communication process according to the nested-hierarchical topology. Learning is thus approached not through demonstration and training but through an autonomous self-exploration process. A fuzzy reinforcement learning scheme is employed within each agent to enable efficient exploration in a continuous state–action domain. This paper constitutes in fact a proof of concept, demonstrating that global dexterous manipulation skills can indeed evolve through such a distributed iterative learning of local agent sensorimotor mappings. The main motivation behind the development of such an incremental multi-agent topology is to enhance system modularity, to facilitate extensibility to more complex problem domains and to improve robustness with respect to structural variations including unpredictable internal failures. These attributes of the proposed system are assessed in this paper through numerical experiments in different robot manipulation task scenarios, involving both single and multi-robot kinematic chains. The generalisation capacity of the learning scheme is experimentally assessed and robustness properties of the multi-agent system are also evaluated with respect to unpredictable variations in the kinematic topology. Furthermore, these numerical experiments demonstrate the scalability properties of the proposed nested-hierarchical architecture, where new agents can be recursively added in the hierarchy to encapsulate individual active DOFs. The results presented in this paper demonstrate the feasibility of such a distributed multi-agent control framework, showing that the solutions which emerge are plausible and near-optimal. Numerical efficiency and computational cost issues are also discussed.     

Joint workspace of parallel kinematic machines

Robot workspace is the set of positions a robot can reach. Workspace is one of the most useful measures for the evaluation of a robot. Workspace is usually defined as the reachable space of the end-effector in Cartesian coordinate system. However, it can be defined in joint coordinate system in terms of joint motions. In this paper, workspace of the end-effector is called task workspace, and workspace of the joint motions is called joint workspace. Joint workspace of a parallel kinematic machine (PKM) is focused, and a tripod machine tool with three degrees of freedom (DOF) is taken as an example. To study the joint workspace of this tripod machine tool, the forward kinematic model is established, and an interpolating approach is proposed to solve this model. The forward kinematic model is used to determine the joint workspace, which occupies a portion of the domain of joint motions. The following contributions have been made in this paper include: (i) a new concept so-called joint workspace has been proposed for design optimization and control of a PKM; (ii) an approach is developed to determine joint workspace based on the structural constraints of a PKM; (iii) it is observed that the trajectory planning in the joint coordinate system is not reliable without taking into considerations of cavities or holes in the joint workspace.     

Correspondence mapping induced state and action metrics for robotic imitation.

This paper addresses the problem of body mapping in robotic imitation where the demonstrator and imitator may not share the same embodiment [degrees of freedom (DOFs), body morphology, constraints, affordances, and so on]. Body mappings are formalized using a unified (linear) approach via correspondence matrices, which allow one to capture partial, mirror symmetric, one-to-one, one-to-many, many-to-one, and many-to-many associations between various DOFs across dissimilar embodiments. We show how metrics for matching state and action aspects of behavior can be mathematically determined by such correspondence mappings, which may serve to guide a robotic imitator. The approach is illustrated and validated in a number of simulated 3-D robotic examples, using agents described by simple kinematic models and different types of correspondence mappings.     

Shared Kernel Information Embedding for discriminative inference

Latent variable models, such as the GPLVM and related methods, help mitigate overfitting when learning from small or moderately sized training sets. Nevertheless, existing methods suffer from several problems: 1) complexity, 2) the lack of explicit mappings to and from the latent space, 3) an inability to cope with multimodality, and 4) the lack of a well-defined density over the latent space. We propose an LVM called the Kernel Information Embedding (KIE) that defines a coherent joint density over the input and a learned latent space. Learning is quadratic, and it works well on small data sets. We also introduce a generalization, the shared KIE (sKIE), that allows us to model multiple input spaces (e.g., image features and poses) using a single, shared latent representation. KIE and sKIE permit missing data during inference and partially labeled data during learning. We show that with data sets too large to learn a coherent global model, one can use the sKIE to learn local online models. We use sKIE for human pose inference.     

Gaussian process dynamical models for human motion

We introduce Gaussian process dynamical models (GPDM) for nonlinear time series analysis, with applications to learning models of human pose and motion from high-dimensionalmotion capture data. A GPDM is a latent variable model. It comprises a low-dimensional latent space with associated dynamics, and a map from the latent space to an observation space. We marginalize out the model parameters in closed-form, using Gaussian process priors for both the dynamics and the observation mappings. This results in a non-parametric model for dynamical systems that accounts for uncertainty in the model. We demonstrate the approach, and compare four learning algorithms on human motion capture data in which each pose is 50-dimensional. Despite the use of small data sets, the GPDM learns an effective representation of the nonlinear dynamics in these spaces.     

Fault tolerance of parallel manipulators using task space and kinematic redundancy

When a parallel manipulator suffers from failures, its performance can be significantly affected. Thus, fault tolerance is essential for task-critical applications or applications in which maintenance is hard to implement. In this paper, we consider three types of common strut failures corresponding to stuck joints, unactuated actuators, or the complete loss of struts, respectively. The impacts of different failures on the kinematics of a manipulator are examined, and the task space redundancy and kinematic redundancies are used to help overcome these failures. In addition, local measures of fault tolerance and their properties are analyzed. These measures can be helpful in architecture design and path planning.     

Efficient management of uncertainty in XML schema matching

Despite advances in machine learning technologies a schema matching result between two database schemas (e.g., those derived from COMA++) is likely to be imprecise. In particular, numerous instances of ??possible mappings?? between the schemas may be derived from the matching result. In this paper, we study problems related to managing possible mappings between two heterogeneous XML schemas. First, we study how to efficiently generate possible mappings for a given schema matching task. While this problem can be solved by existing algorithms, we show how to improve the performance of the solution by using a divide-and-conquer approach. Second, storing and querying a large set of possible mappings can incur large storage and evaluation overhead. For XML schemas, we observe that their possible mappings often exhibit a high degree of overlap. We hence propose a novel data structure, called the block tree, to capture the commonalities among possible mappings. The block tree is useful for representing the possible mappings in a compact manner and can be efficiently generated. Moreover, it facilitates the evaluation of a probabilistic twig query (PTQ), which returns the non-zero probability that a fragment of an XML document matches a given query. For users who are interested only in answers with k-highest probabilities, we also propose the top-k PTQ and present an efficient solution for it. An extensive evaluation on real-world data sets shows that our approaches significantly improve the efficiency of generating, storing, and querying possible mappings.     

A Neuro-Sliding Mode Control Scheme for Constrained Robots with Uncertain Jacobian

The joint robot control requires to map desired cartesian tasks into desired joint trajectories, by using the ill-posed inverse kinematics mapping. In order to avoid inverse kinematics, the control problem is formulated directly in task space to gives rise to cartesian robot control. In addition, when the robot is constrained due to its kinematic mappings yields a stiff system and an additional complexity arises to implement cartesian control for constrained robots. In this paper, an alternative approach is proposed to guarantee global convergence of force and position cartesian tracking errors under the assumption that the jacobian is not exactly known. A neuro-sliding mode controller is presented, where a small size adaptive neural network compensates approximately for the inverse dynamics and an inner control loop induces second order sliding modes to guarantee tracking. The sliding mode variable tunes the online adaptation of the weights. A passivity analysis yields the energy Lyapunov function to prove boundedness of all closed-loop signals and variable structure control theory is used to finally conclude convergence of position and force tracking errors. Experimental results are provided to visualize the expected performance.     

A feedback control structure for on-line learning tasks

This paper addresses adaptive control architectures for systems that respond autonomously to changing tasks. Such systems often have many sensory and motor alternatives and behavior drawn from these produces varying quality solutions. The objective is then to ground behavior in control laws which, combined with resources, enumerate closed-loop behavioral alternatives. Use of such controllers leads to analyzable and predictable composite systems, permitting the construction of abstract behavioral models. Here, discrete event system and reinforcement learning techniques are employed to constrain the behavioral alternatives and to synthesize behavior on-line. To illustrate this, a quadruped robot learning a turning gait subject to safety and kinematic constraints is presented.     

Function Approximation With XCS: Hyperellipsoidal Conditions, Recursive Least Squares, and Compaction

An important strength of learning classifier systems (LCSs) lies in the combination of genetic optimization techniques with gradient-based approximation techniques. The chosen approximation technique develops locally optimal approximations, such as accurate classification estimates, Q-value predictions, or linear function approximations. The genetic optimization technique is designed to distribute these local approximations efficiently over the problem space. Together, the two components develop a distributed, locally optimized problem solution in the form of a population of expert rules, often called classifiers. In function approximation problems, the XCSF classifier system develops a problem solution in the form of overlapping, piecewise linear approximations. This paper shows that XCSF performance on function approximation problems additively benefits from: 1) improved representations; 2) improved genetic operators; and 3) improved approximation techniques. Additionally, this paper introduces a novel closest classifier matching mechanism for the efficient compaction of XCS's final problem solution. The resulting compaction mechanism can boil the population size down by 90% on average, while decreasing prediction accuracy only marginally. Performance evaluations show that the additional mechanisms enable XCSF to reliably, accurately, and compactly approximate even seven dimensional functions. Performance comparisons with other, heuristic function approximation techniques show that XCSF yields competitive or even superior noise-robust performance.     

Combined Direct and Inverse Kinematic Control for Articulated Figure Motion Editing

A new approach for the animation of articulated figures is presented. We propose a system of articulated motion design which offers a full combination of both direct and inverse kinematic control of the joint parameters. Such an approach allows an animator to specify interactively goal-directed changes to existing sampled joint motions, resulting in a more general and expressive class of possible joint motions. The fundamental idea is to consider any desired-joint space motion as a reference model inserted into the secondary task of an inverse kinematic control scheme. This approach profits from the use of half-space Cartesian main tasks in conjunction with a parallel control of the articulated figure called the coach-trainee metaphor. In addition, a transition function is introduced so as to guarantee the continuity of the control. The resulting combined kinematic control scheme leads to a new methodology of joint-motion editing which is demonstrated through the improvement of a functional model of human walking.     

Generalization in the XCSF classifier system: analysis, improvement, and extension

We analyze generalization in XCSF and introduce three improvements. We begin by showing that the types of generalizations evolved by XCSF can be influenced by the input range. To explain these results we present a theoretical analysis of the convergence of classifier weights in XCSF which highlights a broader issue. In XCSF, because of the mathematical properties of the Widrow-Hoff update, the convergence of classifier weights in a given subspace can be slow when the spread of the eigenvalues of the autocorrelation matrix associated with each classifier is large. As a major consequence, the system's accuracy pressure may act before classifier weights are adequately updated, so that XCSF may evolve piecewise constant approximations, instead of the intended, and more efficient, piecewise linear ones. We propose three different ways to update classifier weights in XCSF so as to increase the generalization capabilities of XCSF: one based on a condition-based normalization of the inputs, one based on linear least squares, and one based on the recursive version of linear least squares. Through a series of experiments we show that while all three approaches significantly improve XCSF, least squares approaches appear to be best performing and most robust. Finally we show how XCSF can be extended to include polynomial approximations.     

Kinematic control of redundant robots with guaranteed joint limit avoidance

A novel approach for addressing the inverse differential kinematics of redundant manipulators in the presence of hard joint position constraints is presented. A prescribed performance signal for joint limit avoidance guarantees is proposed that can be utilized with both planned and on-line generated trajectories. In the first case, it is a null space velocity for the primary task velocity mapping while in the second case, it modifies the generated reference by acting on the whole velocity space producing a feasible path to the target. Experimental results utilizing a 7DOF KUKA LWR4+ arm demonstrate the performance of the proposed kinematic controller.