关系强化学习方法的初步研究

强化学习方法是人工智能领域中比较重要的方法之一,自从其提出以来已经有了很大的发展,并且能用来解决很多的问题。但是在遇到大规模状态空间问题时,使用普通的强化学习方法就会产生“维数灾”现象,所以提出了关系强化学习,把强化学习应用到关系领域可以在一定的程度上解决“维数灾”难题。在此基础上,简单介绍关系强化学习的概念以及相关的算法,以及以后有待解决的问题。     

Empirically evaluating the application of reinforcement learning to the induction of effective and adaptive pedagogical strategies

For many forms of e-learning environments, the system??s behavior can be viewed as a sequential decision process wherein, at each discrete step, the system is responsible for selecting the next action to take. Pedagogical strategies are policies to decide the next system action when there are multiple ones available. In this project we present a Reinforcement Learning (RL) approach for inducing effective pedagogical strategies and empirical evaluations of the induced strategies. This paper addresses the technical challenges in applying RL to Cordillera, a Natural Language Tutoring System teaching students introductory college physics. The algorithm chosen for this project is a model-based RL approach, Policy Iteration, and the training corpus for the RL approach is an exploratory corpus, which was collected by letting the system make random decisions when interacting with real students. Overall, our results show that by using a rather small training corpus, the RL-induced strategies indeed measurably improved the effectiveness of Cordillera in that the RL-induced policies improved students?? learning gains significantly.     

基于强化学习的多技能项目调度算法

多技能项目调度存在组合爆炸的现象, 其问题复杂度远超传统的单技能项目调度, 启发式算法和元启发式 算法在求解多技能项目调度问题时也各有缺陷. 为此, 根据项目调度的特点和强化学习的算法逻辑, 本文设计了基 于强化学习的多技能项目调度算法. 首先, 将多技能项目调度过程建模为符合马尔科夫性质的序贯决策过程, 并依 据决策过程设计了双智能体机制. 而后, 通过状态整合和行动分解, 降低了价值函数的学习难度. 最后, 为进一步提 高算法性能, 针对资源的多技能特性, 设计了技能归并法, 显著降低了资源分配算法的时间复杂度. 与启发式算法的 对比实验显示, 本文所设计的强化学习算法求解性能更高, 与元启发式算法的对比实验表明, 该算法稳定性更强, 且 求解速度更快.     

基于半结构化数据的项目备份方法研究

对软件项目管理系统的项目数据备份进行分析,提出了一种基于半结构化数据的项目备份方法SDB-Method.该方法通过对系统的数据模型进行分析,建立关系数据模型和半结构化数据模型OEM(对象交换模型)之间的映射,实现关系数据和半结构化数据的相互转换,从而解决项目的导入和导出问题.该方法应用于项目管理系统SoftPM中,支持软件项目的多分支开发,迭代开发以及移植,有效地解决了软件项目管理系统的项目备份问题.     

Heterogeneous relational reasoning in knowledge graphs with reinforcement learning

Path-based relational reasoning over knowledge graphs has become increasingly popular due to a variety of downstream applications such as question answering in dialogue systems, fact prediction, and recommendation systems. In recent years, reinforcement learning (RL) based solutions for knowledge graphs have been demonstrated to be more interpretable and explainable than other deep learning models. However, the current solutions still struggle with performance issues due to incomplete state representations and large action spaces for the RL agent. We address these problems by developing HRRL (Heterogeneous Relational reasoning with Reinforcement Learning), a type-enhanced RL agent that utilizes the local heterogeneous neighborhood information for efficient path-based reasoning over knowledge graphs. HRRL improves the state representation using a graph neural network (GNN) for encoding the neighborhood information and utilizes entity type information for pruning the action space. Extensive experiments on real-world datasets show that HRRL outperforms state-of-the-art RL methods and discovers more novel paths during the training procedure, demonstrating the explorative power of our method.     

Reinforcement learning via approximation of the Q-function

Relational reinforcement learning (RRL) combines traditional reinforcement learning (RL) with a strong emphasis on a relational (rather than attribute-value) representation. Earlier work used RRL on a learning version of the classic Blocks World planning problem (a version where the learner does not know what the result of taking an action will be) and the Tetris game. Learning results based on the structure of training examples were obtained, such as learning in a mixed 3–5 block environment and being able to perform in a 3 or 10 block environment. Here, we instead take a function approximation approach to RL for the Blocks World problem. We obtain similar learning accuracies, with better running times, allowing us to consider much larger problem sizes. For instance, we can train on 15 blocks and then perform well on worlds with 100–800 blocks–using less running time than the relational method required to perform well for 3–10 blocks.     

Relational database as a tool in industrial design. Experience from a human factors engineering analysis of a module for manned space-flight

A human factors engineering analysis was carried out to verify that the CUPOLA—a module developed by the European Space Agency for the International Space Station—complies with an extensive set of human factors requirements. Analysis was carried out in three steps: task analysis, computer simulation of tasks and empirical test of tasks in a physical mock up. In advance to each step the method of assessment and type of information to be obtained were determined in detail and the way of documentation was established as input forms to a relational database. Advantages and drawbacks of this early formalisation as seen by project members are discussed in the present paper.

Relevance to industry

The method of early formalisation by a relational database used throughout the project showed to be a suitable tool in this human factors analysis of a working place under design. Early formalisation of the results and documentation served as a quality system supporting a homogeneous high standard of data collection and documentation. In industry, the use of relational databases underlying Product Data Management systems increases. This makes it relevant to try and link human factors work to these methods of documentation in order to promote the integration of human factors work at an early stage of product and production development.     

IBTIS．NET对象／关系映射机制的研究与应用

在基于C#和关系型数据库的开发中,利用对象／关系映射机制,构建对象持久层,灵活地解决了关系数据库和面向对象开发之间的不匹配。研究了IBTIS．NET对象／关系映射机制,并在项目开发中应用IBTIS．NET构建对象持久层。实践证明使用IBTIS．NET可以简化对数据库的访问。     

第三方物流的多维数据模型

现有第三方物流企业的管理信息系统普遍采用传统的关系型数据库.随着大量关系数据的聚集,借助关系模型分析数据已明显不足.基于此,提出了采用数据仓库技术建立数据中心的解决方案,为企业决策层提供所需的数据环境.此外,按照维度建模方法,为第三方物流运输管理提供了具体的多维数据模型和应用分析.     

An implementation of remote laboratory for secondary science education

This paper reports our classroom implementation of a new remote laboratory (RL) system, which was developed by using innovative ideas and methods for applying technology‐enhanced learning to secondary school science education (Grades 7–9 or Ages 12–14). The newly developed RL system, which involves 8 remote experiments, was tested with 32 secondary school students from a local public school in order to evaluate its usability, learning, and their perception. The present study was carried out by using a mixed research method, including a questionnaire survey (open‐ended questions) and interviews. The corresponding research tools were specifically developed to collect data on students' perceptions and the implementation issues of the RL system. The survey results revealed that there was no major refinement required for the RL system, which was good enough for further adoption in secondary science education because its methods used to conduct online experiments could (a) extend/enhance the existing practices (with virtual/simulation experiments only) of e‐learning and (b) largely induce students' favourable views and perceptions in their own learning. Besides, negative comments and suggestions for improvement were purposely collected during a follow‐up period with an aim to pinpoint any ways in which the RL system and its design could be refined. They turned out to be very minor and easily fixed, and so the refined RL system is educationally suitable for use in laboratory activities and demonstrations to enhance the learning and teaching of science within and outside the secondary school environment.     

The MIEL system: Uniform interrogation of structured and weakly-structured imprecise data

We present an information system developed to help assessing the microbiological risk in food. That information system contains experimental results in microbiology, mainly extracted from scientific publications. The increasing amount of the experimental results available and the difficulty to integrate them into a classic relational database schema led us to design a system composed of two distinct subsystems queried through a common interface. The first subsystem is a classic relational database. The second subsystem is a database containing weakly-structured pieces of information expressed in terms of conceptual graphs. The data stored in both bases can be fuzzy ones in order to take into account the specificities of the biological information. The uniform query language used on both relational database and conceptual graph database allows the users to express preferences by using fuzzy sets in their queries. The MIEL system is now operational and used by the microbiologists involved in the Sym’Previus French project.     

Reinforcement learning system based on heuristics free state focusing

We discuss the role of state focus in reinforcement learning (RL) systems that are applicable to mechanical systems including robots. Although the concept of the state focus is similar to attention/focusing in visual domains, its implementation requires some theoretical background based on RL. We propose an RL system that effectively learns how to choose the focus simultaneously with how to achieve a task. This RL system does not need heuristics for the adaptation of its focus. We conducted a capture experiment to compare the learning speed between the proposed system and the traditional systems, SARSAs, and conducted a navigation experiment to confirm the applicability of the proposed system to a realistic task. In the capture experiment, the proposed system learned faster than SARSAs. We visualized the developmental process of the focusing strategy in the proposed system using a Q-value analysis technique. In the navigation task, the proposed system demonstrated faster learning than SARSAs in the realistic task. The proposed system is applicable to a wide class of RLs that are applicable to mechanical systems including robots.     

基于面积的灰色关联决策模型

以灰色关联决策理论为基础,分析经典灰色关联决策方法的优缺点.从两曲线相邻点间多边形面积的角度度量曲线在距离上的接近性和几何形状的相似性,提出以被选方案与理想方案间两相邻点的多边形面积作为关联系数,构建了灰色关联度公式.为了解决信息利用不充分和变化趋势不一致性问题,拟考虑被选方案与理想方案和负理想方案的关联度,构建了灰色关联相对贴近度模型.通过算例验证了所提出的灰色关联决策模型的合理性和算法的有效性.     

Reinforcement learning in robotic applications: a comprehensive survey

In recent trends, artificial intelligence (AI) is used for the creation of complex automated control systems. Still, researchers are trying to make a completely autonomous system that resembles human beings. Researchers working in AI think that there is a strong connection present between the learning pattern of human and AI. They have analyzed that machine learning (ML) algorithms can effectively make self-learning systems. ML algorithms are a sub-field of AI in which reinforcement learning (RL) is the only available methodology that resembles the learning mechanism of the human brain. Therefore, RL must take a key role in the creation of autonomous robotic systems. In recent years, RL has been applied on many platforms of the robotic systems like an air-based, under-water, land-based, etc., and got a lot of success in solving complex tasks. In this paper, a brief overview of the application of reinforcement algorithms in robotic science is presented. This survey offered a comprehensive review based on segments as (1) development of RL (2) types of RL algorithm like; Actor-Critic, DeepRL, multi-agent RL and Human-centered algorithm (3) various applications of RL in robotics based on their usage platforms such as land-based, water-based and air-based, (4) RL algorithms/mechanism used in robotic applications. Finally, an open discussion is provided that potentially raises a range of future research directions in robotics. The objective of this survey is to present a guidance point for future research in a more meaningful direction.

     

DLEJena: A practical forward-chaining OWL 2 RL reasoner combining Jena and Pellet

This paper describes DLEJena, a practical reasoner for the OWL 2 RL profile that combines the forward-chaining rule engine of Jena and the Pellet DL reasoner. This combination is based on rule templates, instantiating at run-time a set of ABox OWL 2 RL/RDF Jena rules dedicated to a particular TBox that is handled by Pellet. The goal of DLEJena is to handle efficiently, through instantiated rules, the OWL 2 RL ontologies under direct semantics, where classes and properties cannot be at the same time individuals. The TBox semantics are treated by Pellet, reusing in that way efficient and sophisticated TBox DL reasoning algorithms. The experimental evaluation shows that DLEJena achieves more scalable ABox reasoning than the direct implementation of the OWL 2 RL/RDF rule set in the Jena’s production rule engine, which is the main target of the system. DLEJena can be also used as a generic framework for applying an arbitrary number of entailments beyond the OWL 2 RL profile.     

Reinforcement learning in spacecraft control applications: Advances,prospects, and challenges

This paper presents and analyzes Reinforcement Learning (RL) based approaches to solve spacecraft control problems. Different application fields are considered, e.g., guidance, navigation and control systems for spacecraft landing on celestial bodies, constellation orbital control, and maneuver planning in orbit transfers. It is discussed how RL solutions can address the emerging needs of designing spacecraft with highly autonomous on-board capabilities and implementing controllers (i.e., RL agents) robust to system uncertainties and adaptive to changing environments. For each application field, the RL framework core elements (e.g., the reward function, the RL algorithm and the environment model used for the RL agent training) are discussed with the aim of providing some guidelines in the formulation of spacecraft control problems via a RL framework. At the same time, the adoption of RL in real space projects is also analyzed. Different open points are identified and discussed, e.g., the availability of high-fidelity simulators for the RL agent training and the verification of RL-based solutions. This way, recommendations for future work are proposed with the aim of reducing the technological gap between the solutions proposed by the academic community and the needs/requirements of the space industry.     

An advanced database accelerator

The authors report on a research project launched to develop a complete database management system (DBMS) to support emerging applications. They developed a hardware architecture consisting of a general-purpose host computer and the Delta Driven Computer (DDC), a parallel system that plays the role of a relational database accelerator. They also used compilation and I/O reduction to improve the efficiency of the DBMS and proposed a production-rule language that could be compiled into programs with explicit parallelism. The authors present their research objectives, give a general overview of the DDC, and discuss the elements of a complete DBMS     

基于强化学习的智能I/O调度算法

利用机器学习方法解决存储领域中若干技术难题是目前存储领域的研究热点之一。强化学习作为一种以环境反馈作为输入、自适应环境的特殊的机器学习方法,能通过观测环境状态的变化,评估控制决策对系统性能的影响来选择最优的控制策略,基于强化学习的智能RAID控制技术具有重要的研究价值。本文针对高性能计算应用特点,将机器学习领域中的强化学习技术引入RAID控制器中,提出了基于强化学习的智能I/O调度算法RL-scheduler,利用Q-学习策略实现了面向并行应用的自治调度策略。RL-scheduler综合考虑了调度的公平性、磁盘寻道时间和MPI应用的I/O访问效率,并提出多Q-表交叉组织方法提高Q-表的更新效率。实验结果表明,RL-scheduler缩短了并行应用的平均I/O服务时间,提高了大规模并行计算系统的I/O吞吐率。     

基于SQL Server 2000的关系数据与XML的集成

文章以“９８５轿车虚拟制造”项目中的虚拟产品开发平台为工程应用背景,对其中的一项关键技术──关系数据与 ＸＭＬ的集成机制进行了研究。提出了关系数据与 ＸＭＬ的映射规则,基于ＳＱＬ Ｓｅｒｖｅｒ２０００,结合实际的工程应用对象,阐明了关系数据与ＸＭＬ进行集成的具体实现。     

Reinforcement Learning Versus Model Predictive Control: A Comparison on a Power System Problem

This paper compares reinforcement learning (RL) with model predictive control (MPC) in a unified framework and reports experimental results of their application to the synthesis of a controller for a nonlinear and deterministic electrical power oscillations damping problem. Both families of methods are based on the formulation of the control problem as a discrete-time optimal control problem. The considered MPC approach exploits an analytical model of the system dynamics and cost function and computes open-loop policies by applying an interior-point solver to a minimization problem in which the system dynamics are represented by equality constraints. The considered RL approach infers in a model-free way closed-loop policies from a set of system trajectories and instantaneous cost values by solving a sequence of batch-mode supervised learning problems. The results obtained provide insight into the pros and cons of the two approaches and show that RL may certainly be competitive with MPC even in contexts where a good deterministic system model is available.