Cascade ARTMAP: integrating neural computation and symbolic knowledge processing

This paper introduces a hybrid system termed cascade adaptive resonance theory mapping (ARTMAP) that incorporates symbolic knowledge into neural-network learning and recognition. Cascade ARTMAP, a generalization of fuzzy ARTMAP, represents intermediate attributes and rule cascades of rule-based knowledge explicitly and performs multistep inferencing. A rule insertion algorithm translates if-then symbolic rules into cascade ARTMAP architecture. Besides that initializing networks with prior knowledge can improve predictive accuracy and learning efficiency, the inserted symbolic knowledge can be refined and enhanced by the cascade ARTMAP learning algorithm. By preserving symbolic rule form during learning, the rules extracted from cascade ARTMAP can be compared directly with the originally inserted rules. Simulations on an animal identification problem indicate that a priori symbolic knowledge always improves system performance, especially with a small training set. Benchmark study on a DNA promoter recognition problem shows that with the added advantage of fast learning, cascade ARTMAP rule insertion and refinement algorithms produce performance superior to those of other machine learning systems and an alternative hybrid system known as knowledge-based artificial neural network (KBANN). Also, the rules extracted from cascade ARTMAP are more accurate and much cleaner than the NofM rules extracted from KBANN.     

基于神经网络结构学习的知识求精方法

知识求精是知识获取中必不可少的步骤．已有的用于知识求精的ＫＢＡＮＮ（ｋｎｏｗ ｌｅｄｇｅ ｂａｓｅｄ ａｒｔｉｆｉｃｉａｌｎｅｕｒａｌｎｅｔｗ ｏｒｋ）方法,主要局限性是训练时不能改变网络的拓扑结构．文中提出了一种基于神经网络结构学习的知识求精方法,首先将一组规则集转化为初始神经网络,然后用训练样本和结构学习算法训练初始神经网络,并提取求精的规则知识．网络拓扑结构的改变是通过训练时采用基于动态增加隐含节点和网络删除的结构学习算法实现的．大量实例表明该方法是有效的     

Combining models from neural networks and inductive learning algorithms

The knowledge-based artificial neural network (KBANN) is composed of phases involving the expression of domain knowledge, the abstraction of domain knowledge at neural networks, the training of neural networks, and finally, the extraction of rules from trained neural networks. The KBANN attempts to open up the neural network black box and generates symbolic rules with (approximately) the same predictive power as the neural network itself. An advantage of using KBANN is that the neural network considers the contribution of the inputs towards classification as a group, while rule-based algorithms like C5.0 measure the individual contribution of the inputs one at a time as the tree is grown. The knowledge consolidation model (KCM) combines the rules extracted using KBANN (NeuroRule), frequency matrix (which is similar to the Naïve Bayesian technique), and C5.0 algorithm. The KCM can effectively integrate multiple rule sets into one centralized knowledge base. The cumulative rules from other single models can improve overall performance as it can reduce error-term and increase R-square. The key idea in the KCM is to combine a number of classifiers such that the resulting combined system achieves higher classification accuracy and efficiency than the original single classifiers. The aim of KCM is to design a composite system that outperforms any individual classifier by pooling together the decisions of all classifiers. Another advantage of KCM is that it does not need the memory space to store the dataset as only extracted knowledge is necessary in build this integrated model. It can also reduce the costs from storage allocation, memory, and time schedule. In order to verify the feasibility and effectiveness of KCM, personal credit rating dataset provided by a local bank in Seoul, Republic of Korea is used in this study. The results from the tests show that the performance of KCM is superior to that of the other single models such as multiple discriminant analysis, logistic regression, frequency matrix, neural networks, decision trees, and NeuroRule. Moreover, our model is superior to a previous algorithm for the extraction of rules from general neural networks.     

Dynamically adding symbolically meaningful nodes to knowledge-based neural networks

Traditional connectionist theory-refinement systems map the dependencies of a domain-specific rule base into a neural network, and then refine this network using neural learning techniques. Most of these systems, however, lack the ability to refine their network's topology and are thus unable to add new rules to the (reformulated) rule base. Therefore, with domain theories that lack rules, generalization is poor, and training can corrupt the original rules — even those that were initially correct. The paper presents TopGen, an extension to the KBANN algorithm, which heuristically searches for possible expansions to the KBANN network. TopGen does this by dynamically adding hidden nodes to the neural representation of the domain theory, in a manner that is analogous to the adding of rules and conjuncts to the symbolic rule base. Experiments indicate that the method is able to heuristically find effective places to add nodes to the knowledge bases of four real-world problems, as well as an artificial chess domain. The experiments also verify that new nodes must be added in an intelligent manner. The algorithm showed statistically significant improvements over the KBANN algorithm in all five domains.     

Fast incremental learning methods inspired by biological learning behavior

Model-based learning systems such as neural networks usually “forget” learned skills due to incremental learning of new instances. This is because the modification of a parameter interferes with old memories. Therefore, to avoid forgetting, incremental learning processes in these learning systems must include relearning of old instances. The relearning process, however, is time-consuming. We present two types of incremental learning method designed to achieve quick adaptation with low resources. One approach is to use a sleep phase to provide time for learning. The other one involves a “meta-learning module” that acquires learning skills through experience. The system carries out “reactive modification” of parameters not only to memorize new instances, but also to avoid forgetting old memories using a meta-learning module.This work was presented, in part, at the 9th International Symposium on Artificial Life and Robotics, Oita, Japan, January 28–30, 2004     

FILIP: A fuzzy intelligent information system with learning capabilities

FILIP (fuzzy intelligent learning information processing) system is designed with the goal to model human information processing. The issues addressed are uncertain knowledge representation and approximate reasoning based on fuzzy set theory, and knowledge acquisition by “being told” or by “learning from examples”. Concepts that can be “learned” by the system can be imprecise (fuzzy), or the knowledge can be incomplete. In the latter case, FILIP uses the concept of similarity to extrapolate the knowledge to cases that were not covered by examples provided by the user. Concepts are stored in the Knowledge Base and employed in intelligent query processing, based on flexible inference that supports approximate matches between the data in the database and the query.

The architecture of FILIP is discussed, the learning algorithm is described, and examples of the system's performance in the knowledge acquisition and querying modes, together with its explanatory capabilities are shown.     

Application of Machine Learning in Water Distribution Networks Assisted by Domain Experts

The human-assisted application of machine learning techniques in the domain of water distribution networks is presented, corresponding to a research work done in the context of the European Esprit project WATERNET. One part of this project is a learning system that intends to capture knowledge from historic information collected during the operation of water distribution networks. The captured knowledge is expected to contribute to the improvement of the operation of the network. Presented ideas correspond to the first development phase of the learning system, focusing specially on the adopted methodology. The interactions between different classes of human experts and the learning system are also discussed. Finally some experimental results are presented.     

一种面向多源领域的实例迁移学习

在迁移学习最大的特点就是利用相关领域的知识来帮助完成目标领域中的学习任务,它能够有效地在相似的领域或任务之间进行信息的共享和迁移,使传统的从零开始的学习变成可积累的学习,具有成本低、效率高等优点.针对源领域数据和目标领域数据分布类似的情况,提出一种基于多源动态TrAdaBoost的实例迁移学习方法.该方法考虑多个源领域知识,使得目标任务的学习可以充分利用所有源领域信息,每次训练候选分类器时,所有源领域样本都参与学习,可以获得有利于目标任务学习的有用信息,从而避免负迁移的产生.理论分析验证了所提算法较单源迁移的优势,以及加入动态因子改善了源权重收敛导致的权重熵由源样本转移到目标样本的问题.实验结果验证了此算法在提高识别率方面的优势.     

Hybrid learning concepts based on self-organizing neural networks for adaptive control of walking machines

Online learning of complex control behaviour of autonomous mobile robots like walking machines is one of the current research topics. In this article, a hybrid learning architecture based on reinforcement learning (RL) and self-organizing neural networks for online adaptivity is presented. The hybrid concept integrates different learning methods and task-oriented representations as well as available domain knowledge. The proposed concept is used for RL of control strategies on different control levels on a walking machine.     

An incremental neural learning framework and its application to vehicle diagnostics

This paper presents a framework for incremental neural learning (INL) that allows a base neural learning system to incrementally learn new knowledge from only new data without forgetting the existing knowledge. Upon subsequent encounters of new data examples, INL utilizes prior knowledge to direct its incremental learning. A number of critical issues are addressed including when to make the system learn new knowledge, how to learn new knowledge without forgetting existing knowledge, how to perform inference using both the existing and the newly learnt knowledge, and how to detect and deal with aged learnt systems. To validate the proposed INL framework, we use backpropagation (BP) as a base learner and a multi-layer neural network as a base intelligent system. INL has several advantages over existing incremental algorithms: it can be applied to a broad range of neural network systems beyond the BP trained neural networks; it retains the existing neural network structures and weights even during incremental learning; the neural network committees generated by INL do not interact with one another and each sees the same inputs and error signals at the same time; this limited communication makes the INL architecture attractive for parallel implementation. We have applied INL to two vehicle fault diagnostics problems: end-of-line test in auto assembly plants and onboard vehicle misfire detection. These experimental results demonstrate that the INL framework has the capability to successfully perform incremental learning from unbalanced and noisy data. In order to show the general capabilities of INL, we also applied INL to three general machine learning benchmark data sets. The INL systems showed good generalization capabilities in comparison with other well known machine learning algorithms.