Integrating temporal difference methods and self-organizing neural networks for reinforcement learning with delayed evaluative feedback

This paper presents a neural architecture for learning category nodes encoding mappings across multimodal patterns involving sensory inputs, actions, and rewards. By integrating adaptive resonance theory (ART) and temporal difference (TD) methods, the proposed neural model, called TD fusion architecture for learning, cognition, and navigation (TD-FALCON), enables an autonomous agent to adapt and function in a dynamic environment with immediate as well as delayed evaluative feedback (reinforcement) signals. TD-FALCON learns the value functions of the state-action space estimated through on-policy and off-policy TD learning methods, specifically state-action-reward-state-action (SARSA) and Q-learning. The learned value functions are then used to determine the optimal actions based on an action selection policy. We have developed TD-FALCON systems using various TD learning strategies and compared their performance in terms of task completion, learning speed, as well as time and space efficiency. Experiments based on a minefield navigation task have shown that TD-FALCON systems are able to learn effectively with both immediate and delayed reinforcement and achieve a stable performance in a pace much faster than those of standard gradient-descent-based reinforcement learning systems.     

基于多模态多标记迁移学习的早期阿尔茨海默病诊断

针对当前基于机器学习的早期阿尔茨海默病（AD）诊断中训练样本不足的问题,提出一种基于多模态特征数据的多标记迁移学习方法,并将其应用于早期阿尔茨海默病诊断。所提方法框架主要包括两大模块：多标记迁移学习特征选择模块和多模态多标记分类回归学习器模块。首先,通过稀疏多标记学习模型对分类和回归学习任务进行有效结合;然后,将该模型扩展到来自多个学习领域的训练集,从而构建出多标记迁移学习特征选择模型;接下来,针对异质特征空间的多模态特征数据,采用多核学习技术来组合多模态特征核矩阵;最后,为了构建能同时用于分类与回归的学习模型,提出多标记分类回归学习器,从而构建出多模态多标记分类回归学习器。在国际老年痴呆症数据库（ADNI）进行实验,分类轻度认知功能障碍（MCI）最高平均精度为79.1%,预测神经心理学量表测试评分值最大平均相关系数为0.727。实验结果表明,所提多模态多标记迁移学习方法可以有效利用相关学习领域训练数据,从而提高早期老年痴呆症诊断性能。     

面向多模态模型训练的高效样本检索技术研究

深度学习中多模态模型的训练通常需要大量高质量不同类型的标注数据，如图像、文本、音频等. 然而，获取大规模的多模态标注数据是一项具有挑战性和昂贵的任务.为了解决这一问题，主动学习作为一种有效的学习范式被广泛应用，能够通过有针对性地选择最有信息价值的样本进行标注，从而降低标注成本并提高模型性能. 现有的主动学习方法往往面临着低效的数据扫描和数据位置调整问题，当索引需要进行大范围的更新时，会带来巨大的维护代价. 为解决这些问题，本文提出了一种面向多模态模型训练的高效样本检索技术So-CBI. 该方法通过感知模型训练类间边界点，精确评估样本对模型的价值；并设计了半有序的高效样本索引，通过结合数据排序信息和部分有序性，降低了索引维护代价和时间开销. 在多组多模态数据集上通过与传统主动学习训练方法实验对比，验证了So-CBI方法在主动学习下的训练样本检索问题上的有效性.     

Joint patch clustering-based dictionary learning for multimodal image fusion

Constructing a good dictionary is the key to a successful image fusion technique in sparsity-based models. An efficient dictionary learning method based on a joint patch clustering is proposed for multimodal image fusion. To construct an over-complete dictionary to ensure sufficient number of useful atoms for representing a fused image, which conveys image information from different sensor modalities, all patches from different source images are clustered together with their structural similarities. For constructing a compact but informative dictionary, only a few principal components that effectively describe each of joint patch clusters are selected and combined to form the over-complete dictionary. Finally, sparse coefficients are estimated by a simultaneous orthogonal matching pursuit algorithm to represent multimodal images with the common dictionary learned by the proposed method. The experimental results with various pairs of source images validate effectiveness of the proposed method for image fusion task.     

Deep imitation learning for 3D navigation tasks

Deep learning techniques have shown success in learning from raw high-dimensional data in various applications. While deep reinforcement learning is recently gaining popularity as a method to train intelligent agents, utilizing deep learning in imitation learning has been scarcely explored. Imitation learning can be an efficient method to teach intelligent agents by providing a set of demonstrations to learn from. However, generalizing to situations that are not represented in the demonstrations can be challenging, especially in 3D environments. In this paper, we propose a deep imitation learning method to learn navigation tasks from demonstrations in a 3D environment. The supervised policy is refined using active learning in order to generalize to unseen situations. This approach is compared to two popular deep reinforcement learning techniques: deep-Q-networks and Asynchronous actor-critic (A3C). The proposed method as well as the reinforcement learning methods employ deep convolutional neural networks and learn directly from raw visual input. Methods for combining learning from demonstrations and experience are also investigated. This combination aims to join the generalization ability of learning by experience with the efficiency of learning by imitation. The proposed methods are evaluated on 4 navigation tasks in a 3D simulated environment. Navigation tasks are a typical problem that is relevant to many real applications. They pose the challenge of requiring demonstrations of long trajectories to reach the target and only providing delayed rewards (usually terminal) to the agent. The experiments show that the proposed method can successfully learn navigation tasks from raw visual input while learning from experience methods fail to learn an effective policy. Moreover, it is shown that active learning can significantly improve the performance of the initially learned policy using a small number of active samples.

     

Under canopy light detection and ranging‐based autonomous navigation

This paper describes a light detection and ranging (LiDAR)‐based autonomous navigation system for an ultralightweight ground robot in agricultural fields. The system is designed for reliable navigation under cluttered canopies using only a 2D Hokuyo UTM‐30LX LiDAR sensor as the single source for perception. Its purpose is to ensure that the robot can navigate through rows of crops without damaging the plants in narrow row‐based and high‐leaf‐cover semistructured crop plantations, such as corn (Zea mays) and sorghum ( Sorghum bicolor). The key contribution of our work is a LiDAR‐based navigation algorithm capable of rejecting outlying measurements in the point cloud due to plants in adjacent rows, low‐hanging leaf cover or weeds. The algorithm addresses this challenge using a set of heuristics that are designed to filter out outlying measurements in a computationally efficient manner, and linear least squares are applied to estimate within‐row distance using the filtered data. Moreover, a crucial step is the estimate validation, which is achieved through a heuristic that grades and validates the fitted row‐lines based on current and previous information. The proposed LiDAR‐based perception subsystem has been extensively tested in production/breeding corn and sorghum fields. In such variety of highly cluttered real field environments, the robot logged more than 6 km of autonomous run in straight rows. These results demonstrate highly promising advances to LiDAR‐based navigation in realistic field environments for small under‐canopy robots.     

稀疏奖励下基于情感的异构多智能体强化学习

在强化学习中,当处于奖励分布稀疏的环境时,由于无法获得有效经验,智能体收敛速度和效率都会大幅下降.针对此类稀疏奖励,文中提出基于情感的异构多智能体强化学习方法.首先,建立基于个性的智能体情感模型,为异构多智能体提供激励机制,作为外部奖励的有效补充.然后,基于上述激励机制,融合深度确定性策略,提出稀疏奖励下基于内在情感激...     

Labeling sensing data for mobility modeling

In urban environments, sensory data can be used to create personalized models for predicting efficient routes and schedules on a daily basis; and also at the city level to manage and plan more efficient transport, and schedule maintenance and events. Raw sensory data is typically collected as time-stamped sequences of records, with additional activity annotations by a human, but in machine learning, predictive models view data as labeled instances, and depend upon reliable labels for learning. In real-world sensor applications, human annotations are inherently sparse and noisy. This paper presents a methodology for preprocessing sensory data for predictive modeling in particular with respect to creating reliable labeled instances. We analyze real-world scenarios and the specific problems they entail, and experiment with different approaches, showing that a relatively simple framework can ensure quality labeled data for supervised learning. We conclude the study with recommendations to practitioners and a discussion of future challenges.     

Autonomous Navigation for Micro Aerial Vehicles in Complex GNSS-denied Environments

Micro aerial vehicles, such as multirotors, are particular well suited for the autonomous monitoring, inspection, and surveillance of buildings, e.g., for maintenance in industrial plants. Key prerequisites for the fully autonomous operation of micro aerial vehicles in restricted environments are 3D mapping, real-time pose tracking, obstacle detection, and planning of collision-free trajectories. In this article, we propose a complete navigation system with a multimodal sensor setup for omnidirectional environment perception. Measurements of a 3D laser scanner are aggregated in egocentric local multiresolution grid maps. Local maps are registered and merged to allocentric maps in which the MAV localizes. For autonomous navigation, we generate trajectories in a multi-layered approach: from mission planning over global and local trajectory planning to reactive obstacle avoidance. We evaluate our approach in a GNSS-denied indoor environment where multiple collision hazards require reliable omnidirectional perception and quick navigation reactions.     

Structure inference for Bayesian multisensory scene understanding

We investigate a solution to the problem of multi-sensor scene understanding by formulating it in the framework of Bayesian model selection and structure inference. Humans robustly associate multimodal data as appropriate, but previous modelling work has focused largely on optimal fusion, leaving segregation unaccounted for and unexploited by machine perception systems. We illustrate a unifying, Bayesian solution to multi-sensor perception and tracking which accounts for both integration and segregation by explicit probabilistic reasoning about data association in a temporal context. Such explicit inference of multimodal data association is also of intrinsic interest for higher level understanding of multisensory data. We illustrate this using a probabilistic implementation of data association in a multi-party audio-visual scenario, where unsupervised learning and structure inference is used to automatically segment, associate and track individual subjects in audiovisual sequences. Indeed, the structure inference based framework introduced in this work provides the theoretical foundation needed to satisfactorily explain many confounding results in human psychophysics experiments involving multimodal cue integration and association.     

An Efficient Reinforcement Learning Game Framework for UAV-Enabled Wireless Sensor Network Data Collection

With the developing demands of massive-data services, the applications that rely on big geographic data play crucial roles in academic and industrial communities. Unmanned aerial vehicles (UAVs), combining with terrestrial wireless sensor networks (WSN), can provide sustainable solutions for data harvesting. The rising demands for efficient data collection in a larger open area have been posed in the literature, which requires efficient UAV trajectory planning with lower energy consumption methods. Currently, there are amounts of inextricable solutions of UAV planning for a larger open area, and one of the most practical techniques in previous studies is deep reinforcement learning (DRL). However, the overestimated problem in limited-experience DRL quickly throws the UAV path planning process into a locally optimized condition. Moreover, using the central nodes of the sub-WSNs as the sink nodes or navigation points for UAVs to visit may lead to extra collection costs. This paper develops a data-driven DRL-based game framework with two partners to fulfill the above demands. A cluster head processor (CHP) is employed to determine the sink nodes, and a navigation order processor (NOP) is established to plan the path. CHP and NOP receive information from each other and provide optimized solutions after the Nash equilibrium. The numerical results show that the proposed game framework could offer UAVs low-cost data collection trajectories, which can save at least 17.58% of energy consumption compared with the baseline methods.     

From signals to knowledge: A conceptual model for multimodal learning analytics

Multimodality in learning analytics and learning science is under the spotlight. The landscape of sensors and wearable trackers that can be used for learning support is evolving rapidly, as well as data collection and analysis methods. Multimodal data can now be collected and processed in real time at an unprecedented scale. With sensors, it is possible to capture observable events of the learning process such as learner's behaviour and the learning context. The learning process, however, consists also of latent attributes, such as the learner's cognitions or emotions. These attributes are unobservable to sensors and need to be elicited by human‐driven interpretations. We conducted a literature survey of experiments using multimodal data to frame the young research field of multimodal learning analytics. The survey explored the multimodal data used in related studies (the input space) and the learning theories selected (the hypothesis space). The survey led to the formulation of the Multimodal Learning Analytics Model whose main objectives are of (O1) mapping the use of multimodal data to enhance the feedback in a learning context; (O2) showing how to combine machine learning with multimodal data; and (O3) aligning the terminology used in the field of machine learning and learning science.     

Enabling Multimodal Human–Robot Interaction for the Karlsruhe Humanoid Robot

In this paper, we present our work in building technologies for natural multimodal human-robot interaction. We present our systems for spontaneous speech recognition, multimodal dialogue processing, and visual perception of a user, which includes localization, tracking, and identification of the user, recognition of pointing gestures, as well as the recognition of a person's head orientation. Each of the components is described in the paper and experimental results are presented. We also present several experiments on multimodal human-robot interaction, such as interaction using speech and gestures, the automatic determination of the addressee during human-human-robot interaction, as well on interactive learning of dialogue strategies. The work and the components presented here constitute the core building blocks for audiovisual perception of humans and multimodal human-robot interaction used for the humanoid robot developed within the German research project (Sonderforschungsbereich) on humanoid cooperative robots.     

A sparse probabilistic approach with chaotic artificial bee colony optimization for sea clutter soft computing

A novel soft computing method of sea clutter based on sparse probabilistic learning frameworks with an optimizing approach is proposed, where a probabilistic dynamic computing method of electromagnetic signals by relevance vector machine (RVM) is developed with sensor parameters optimization using a novel chaotic artificial bee colony (CABC) algorithm. LS-SVM, WLS-SVM and ABC-RVM soft computing models of sea clutter are also developed as the comparative basis. The experimental results show that new optimizing method outperforms the basic ABC both in convergence speed and calculation precision, and then an efficient CABC-RVM approach for computing sea clutter is presented and confirmed through real sea clutter data. Furthermore, the performance of CABC-RVM is analyzed and compared to above sea clutter sensors and literature reported sea clutter sensors in detail. The research results show effectiveness of the proposed approach.     

融合认知行为模型的深度强化学习框架及算法

面对高维连续状态空间或稀疏奖励等复杂任务时,仅依靠深度强化学习算法从零学习最优策略十分困难,如何将已有知识表示为人与学习型智能体之间相互可理解的形式,并有效地加速策略收敛仍是一个难题.对此,提出一种融合认知行为模型的深度强化学习框架,将领域内先验知识建模为基于信念-愿望-意图(belief- desire-intention, BDI)的认知行为模型,用于引导智能体策略学习.基于此框架,分别提出融合认知行为模型的深度Q学习算法和近端策略优化算法,并定量化设计认知行为模型对智能体策略更新的引导方式.最后,通过典型gym环境和空战机动决策对抗环境,验证所提出算法可以高效利用认知行为模型加速策略学习,有效缓解状态空间巨大和环境奖励稀疏的影响.     

数据受限条件下的多模态处理技术综述

随着多媒体技术的发展,可获取的媒体数据在种类和量级上大幅提升。受人类感知方式的启发,多种媒体数据互相融合处理,促进了人工智能在计算机视觉领域的研究发展,在遥感图像解译、生物医学和深度估计等方面有广泛的应用。尽管多模态数据在描述事物特征时具有明显优势,但仍面临着较大的挑战。1)受到不同成像设备和传感器的限制,难以收集到大规模、高质量的多模态数据集;2)多模态数据需要匹配成对用于研究,任一模态的缺失都会造成可用数据的减少;3)图像、视频数据在处理和标注上需要耗费较多的时间和人力成本,这些问题使得目前本领域的技术尚待攻关。本文立足于数据受限条件下的多模态学习方法,根据样本数量、标注信息和样本质量等不同的维度,将计算机视觉领域中的多模态数据受限方法分为小样本学习、缺乏强监督标注信息、主动学习、数据去噪和数据增强5个方向,详细阐述了各类方法的样本特点和模型方法的最新进展。并介绍了数据受限前提下的多模态学习方法使用的数据集及其应用方向(包括人体姿态估计、行人重识别等),对比分析了现有算法的优缺点以及未来的发展方向,对该领域的发展具有积极的意义。     

Face recognition based on dictionary learning and subspace learning

Dictionary learning plays an important role in sparse representation based face recognition. Many dictionary learning algorithms have been successfully applied to face recognition. However, for corrupted data because of noise or face variations (e.g. occlusion and large pose variation), their performances decline due to the disparity between domains. In this paper, we propose a face recognition algorithm based on dictionary learning and subspace learning (DLSL). In DLSL, a new subspace learning algorithm (SL) is proposed by using sparse constraint, low-rank technology and our label relaxation model to reduce the disparity between domains. Meanwhile, we propose a high-performance dictionary learning algorithm (HPDL) by constructing the embedding term, non-local self-similarity term, and time complexity drop term. In the obtained subspace, we use HPDL to classify these mapped test samples. DLSL is compared with other 28 algorithms on FRGC, LFW, CVL, Yale B and AR face databases. Experimental results show that DLSL achieves better performance than those 28 algorithms, including many state-of-the-art algorithms, such as recurrent regression neural network (RRNN), multimodal deep face recognition (MDFR) and projective low-rank representation (PLR).     

一种在源数据稀疏情况下的流形学习算法研究

传统的流形学习算法能有效地学习出高维采样数据的低维嵌入坐标,但也存在一些不足,如不能处理稀疏的样本数据.针对这些缺点,提出了一种基于局部映射的直接求解线性嵌入算法(Solving Directly Linear Embedding,简称SDLE).通过假定低维流形的整体嵌入函数,将流形映射赋予局部光滑的约束,应用核方法将高维空间的坐标投影到特征空间,最后构造出在低维空间的全局坐标.SDLE算法解决了在源数据稀疏情况下的非线性维数约简问题,这是传统的流形学习算法没有解决的问题.通过实验说明了SDLE算法研究的有效性.     

An integrated model based on deep multimodal and rank learning for point-of-interest recommendation

Modeling point-of-Interest (POI) for recommendations is vital in location-based social networks, yet it is a challenging task due to data sparsity and cold-start problems. Most existing approaches incorporate content features into a probabilistic matrix factorization model using unsupervised learning, which results in inaccuracy and weak robustness of recommendations when data is sparse, and the cold-start problems remain unsolved. In this paper, we propose a deep multimodal rank learning (DMRL) model that improves both the accuracy and robustness of POI recommendations. DMRL exploits temporal dynamics by allowing each user to have time-dependent preferences and captures geographical influences by introducing spatial regularization to the model. DMRL jointly learns ranking for personal preferences and supervised deep learning models to create a semantic representation of POIs from multimodal content. To make model optimization converge more rapidly while preserving high effectiveness, we develop a ranking-based dynamic sampling strategy to sample adverse or negative POIs for model training. We conduct experiments to compare our DMRL model with existing models that use different approaches using two large-scale datasets obtained from Foursquare and Yelp. The experimental results demonstrate the superiority of DMRL over the other models in creating cold-start POI recommendations and achieving excellent and highly robust results for different degrees of data sparsity.

     

A Fuzzy Perceptual Model for Ultrasound Sensors Applied to Intelligent Navigation of Mobile Robots

The way of understanding the role of perception along the intelligent robotic systems has evolved greatly since classic approaches to the reactive behavior-based approaches. Classic approaches tried to model the environment using a high level of accuracy while in reactive systems usually the perception is related to the actions that the robot needs to undertake so that such complex models are not generally necessary. Regarding hybrid approaches is likewise important to understand the role that has been assigned to the perception in order to assure the success of the system. In this work a new perceptual model based on fuzzy logic is proposed to be used in a hybrid deliberative-reactive architecture. This perceptual model deals with the uncertainty and vagueness underlying to the ultrasound sensor data, it is useful to carry out the data fusion from different sensors and it allows us to establish various levels of interpretation in the sensor data. Furthermore, using this perceptual model an approximate world model can be built so that the robot can plan its motions for navigating in an office-like environment. Then the navigation is accomplished using the hybrid deliberative-reactive architecture and taking into account the perceptual model to represent the robot's beliefs about the world. Experiments in simulation and in an real office-like environment are shown for validating the perceptual model integrated into the navigation architecture.