XCSG在多机器人强化学习中的应用

XCS分类器在解决机器人强化学习方面已显示出较强的能力,但在多机器人领域仅局限于MDP环境,只能解决环境空间较小的学习问题。提出了XCSG来解决多机器人的强化学习问题。XCSG建立低维的逼近函数,梯度下降技术利用在线知识建立稳定的逼近函数,使Q-表格一直保持在稳定低维状态。逼近函数Q不仅所需的存储空间更小,而且允许机器人在线对已获得的知识进行归纳一般化。仿真实验表明,XCSG算法很好地解决了多机器人学习空间大、学习速度慢、学习效果不确定等问题。     

The stability of long action chains in XCS

 XCS [1, 2] represents a new form of learning classifier system [3] that uses accuracy as a means of guiding fitness for selection within a Genetic Algorithm. The combination of accuracy-based selection and a dynamic niche-based deletion mechanism achieve a long sought-after goal–the reliable production, maintenance, and proliferation of the sub-population of optimally general accurate classifiers that map the problem domain [4]. Wilson [2] and Lanzi [5, 6] have demonstrated the applicability of XCS to the identification of the optimal action-chain leading to the optimum trade-off between reward distance and magnitude. However, Lanzi [6] also demonstrated that XCS has difficulty in finding an optimal solution to the long action-chain environment Woods-14 [7]. Whilst these findings have shed some light on the ability of XCS to form long action-chains, they have not provided a systematic and, above all, controlled investigation of the limits of XCS learning within multiple-step environments. In this investigation a set of confounding variables in such problems are identified. These are controlled using carefully constructed FSW environments [8, 9] of increasing length. Whilst investigations demonstrate that XCS is able to establish the optimal sub-population [O] [4] when generalisation is not used, it is shown that the introduction of generalisation introduces low bounds on the length of action-chains that can be identified and chosen between to find the optimal pathway. Where these bounds are reached a form of over-generalisation caused by the formation of dominant classifiers can occur. This form is further investigated and the Domination Hypothesis introduced to explain its formation and preservation.     

Combining accuracy and success-rate to improve the performance of eXtended Classifier System (XCS) for data-mining and control applications

The emergence of eXtended Classifier Systems (XCS) raised the bar for Learning Classifier Systems by incorporating the accuracies of the rules in the LCS's traditional reinforcement mechanism. However, neither XCS nor its extensions take into account the nature of a classifier's experience of attending the action set. We introduce an experience–evaluation mechanism that, once added to the traditional XCS, would assigns to each member of the action set a success rate indicating how effectively the classifier has contributed to the correct responding of the system to the environment's queries. Application of the augmented system (called SRXCS) to several benchmark problems shows that the proposed mechanism enhances XCS' classification capability and its rate of convergence at the same time. Application results indicate that SRXCS performs notably better on both pattern association and pattern recognition tasks. The applicability and efficiency of the proposed mechanism is further demonstrated through solving a fairly complex path planning problem for an autonomous mobile robot in a dynamic environment.     

What should a classifier system learn and how should we measure it?

 We consider the issues of how a classifier system should learn to represent a Boolean function, and how we should measure its progress in doing so. We identify four properties which may be desirable of a representation; that it be complete, accurate, minimal and non-overlapping. We distinguish two categories of learning metric, introduce new metrics and evaluate them. We demonstrate the superiority of population state metrics over performance metrics in two situations, and in the process find evidence of XCS's strong bias against overlapping rules.     

An accuracy based corporate classifier system

 Previously a corporate classifier system has been implemented based on the ideas of Wilson and Goldberg which demonstrates that rule-linkage can, for a certain class of problems, offer benefits to a system based on the zeroth-level classifier system (ZCS). In this work it is shown that similar benefits can be gained when similar rule-linkage mechanisms are applied to XCS. The intention is that the resultant system (CXCS) will exhibit XCS's generalization capabilities along with CCS's abilities to solve non-Markov tasks.     

FPGA器件XCS40XL及其在多画面处理器中的应用

FPGA器件XCS40XL是Xilinx公司推出的低价格、高性能现场可编程门阵列 ,文中详细讨论了XCS40XL中三大模块(CLB、IOB、布线通道)的结构和功能,同时给出了XCS40XL器件在多画面处理器中的应用情况     

Strong,Stable, and Reliable Fitness Pressure in XCS due to Tournament Selection

Recent analysis of the XCS classifier system have shown that successful genetic learning strongly depends on the amount of fitness pressure towards accurate classifiers. Since the traditionally used proportionate selection is dependent on fitness scaling and fitness distribution, the resulting evolutionary fitness pressure may be neither stable nor sufficiently strong. Thus, we apply tournament selection to XCS. In particular, we exhibit the weakness of proportionate selection and suggest tournament selection as a more reliable alternative. We show that tournament selection results in a learning classifier system that is more parameter independent, noise independent, and more efficient in exploiting fitness guidance in single-step problems as well as multistep problems. The evolving population is more focused on promising subregions of the problem space and thus finds the desired accurate, maximally general representation faster and more reliably.     

Learning classifier systems from a reinforcement learning perspective

 We analyze learning classifier systems in the light of tabular reinforcement learning. We note that although genetic algorithms are the most distinctive feature of learning classifier systems, it is not clear whether genetic algorithms are important to learning classifiers systems. In fact, there are models which are strongly based on evolutionary computation (e.g., Wilson's XCS) and others which do not exploit evolutionary computation at all (e.g., Stolzmann's ACS). To find some clarifications, we try to develop learning classifier systems “from scratch”, i.e., starting from one of the most known reinforcement learning technique, Q-learning. We first consider thebasics of reinforcement learning: a problem modeled as a Markov decision process and tabular Q-learning. We introduce a formal framework to define a general purpose rule-based representation which we use to implement tabular Q-learning. We formally define generalization within rules and discuss the possible approaches to extend our rule-based Q-learning with generalization capabilities. We suggest that genetic algorithms are probably the most general approach for adding generalization although they might be not the only solution.     

Intrusion detection with evolutionary learning classifier systems

Evolutionary Learning Classifier Systems (LCSs) combine reinforcement learning or supervised learning with effective genetics-based search techniques. Together these two mechanisms enable LCSs to evolve solutions to decision problems in the form of easy to interpret rules called classifiers. Although LCSs have shown excellent performance on some data mining tasks, many enhancements are still needed to tackle features like high dimensionality, huge data sizes, non-uniform distribution of classes, etc. Intrusion detection is a real world problem where such challenges exist and to which LCSs have not previously been applied. An intrusion detection problem is characterised by huge network traffic volumes, difficult to realize decision boundaries between attacks and normal activities and highly imbalanced attack class distribution. Moreover, it demands high accuracy, fast processing times and adaptability to a changing environment. We present the results and analysis of two classifier systems (XCS and UCS) on a subset of a publicly available benchmark intrusion detection dataset which features serious class imbalances and two very rare classes. We introduce a better approach for handling the situation when no rules match an input on the test set and recommend this be adopted as a standard part of XCS and UCS. We detect little sign of overfitting in XCS but somewhat more in UCS. However, both systems tend to reach near-best performance in very few passes over the training data. We improve the accuracy of these systems with several modifications and point out aspects that can further enhance their performance. We also compare their performance with other machine learning algorithms and conclude that LCSs are a competitive approach to intrusion detection.

Imitation guided learning in learning classifier systems

In this paper, we study the means of developing an imitation process allowing to improve learning in the framework of learning classifier systems. We present three different approaches in the way a behavior observed may be taken into account through a guidance interaction: two approaches using a model of this behavior, and one without modelling. Those approaches are evaluated and compared in different environments when they are applied to three major classifier systems: ZCS, XCS and ACS. Results are analyzed and discussed. They highlight the importance of using a model of the observed behavior to enable an efficient imitation. Moreover, they show the advantages of taking this model into account by a specialized internal action. Finally, they bring new results of comparison between ZCS, XCS and ACS.