Integrative connectionist learning systems inspired by nature: current models, future trends and challenges

The so far developed and widely utilized connectionist systems (artificial neural networks) are mainly based on a single brain-like connectionist principle of information processing, where learning and information exchange occur in the connections. This paper extends this paradigm of connectionist systems to a new trend—integrative connectionist learning systems (ICOS) that integrate in their structure and learning algorithms principles from different hierarchical levels of information processing in the brain, including neuronal-, genetic-, quantum. Spiking neural networks (SNN) are used as a basic connectionist learning model which is further extended with other information learning principles to create different ICOS. For example, evolving SNN for multitask learning are presented and illustrated on a case study of person authentification based on multimodal auditory and visual information. Integrative gene-SNN are presented, where gene interactions are included in the functioning of a spiking neuron. They are applied on a case study of computational neurogenetic modeling. Integrative quantum-SNN are introduced with a quantum Hebbian learning, where input features as well as information spikes are represented by quantum bits that result in exponentially faster feature selection and model learning. ICOS can be used to solve more efficiently challenging biological and engineering problems when fast adaptive learning systems are needed to incrementally learn in a large dimensional space. They can also help to better understand complex information processes in the brain especially how information processes at different information levels interact. Open questions, challenges and directions for further research are presented.     

Combining Symbolic and Neural Learning

Conclusion Connectionist machine learning has proven to be a fruitful approach, and it makes sense to investigate systems that combine the strengths of the symbolic and connectionist approaches to AI. Over the past few years, researchers have successfully developed a number of such systems. This article summarizes one view of this endeavor, a framework that encompasses the approaches of several different research groups. This framework (see Figure 1) views the combination of symbolic and neural learning as a three-stage process: (1) the insertion of symbolic information into a neural network, thereby (partially) determining the topology and initial weight settings of a network, (2) the refinement of this network using a numeric optimization method such as backpropagation, possibly under the guidance of symbolic knowledge, and (3) the extraction of symbolic rules that accurately represent the knowledge contained in a trained network. These three components form an appealing, complete picture—approximately-correct symbolic information in, more-accurate symbolic information out—however, these three stages can be independently studied. In conclusion, the research summarized in this paper demonstrates that combining symbolic and connectionist methods is a promising approach to machine learning.     

On connectionism, rule extraction, and brain-like learning

There is a growing body of work that shows that both fuzzy and symbolic rule systems can be implemented using neural networks. This body of work also shows that these fuzzy and symbolic rules can be retrieved from these networks, once they have been learned by procedures that generally fall under the category of rule extraction. The paper argues that the idea of rule extraction from a neural network involves certain procedures, specifically the reading of parameters from a network, that are not allowed by the connectionist framework that these neural networks are based on. It argues that such rule extraction procedures imply a greater freedom and latitude about the internal mechanisms of the brain than is permitted by connectionism, but that such latitude is permitted by the recently proposed control theoretic paradigm for the brain. The control theoretic paradigm basically suggests that there are parts of the brain that control other parts and has far less restrictions on the kind of procedures that can be called “brain like”. The paper shows that this control theoretic paradigm is supported by new evidence from neuroscience about the role of neuromodulators and neurotransmitters in the brain. In addition, it shows that the control theoretic paradigm is also used in connectionist algorithms, although never acknowledged explicitly. The paper suggests that far better learning and rule extraction algorithms can be developed using these control theoretic notions and they would be consistent with the more recent understanding of how the brain works and learns     

Schemas,logics, and neural assemblies

To implement schemas and logics in connectionist models, some form of basic-level organization is needed. This paper proposes such an organization, which is termed a discrete neural assembly. Each discrete neural assembly is in turn made up of discrete neurons (nodes), that is, a node that processes inputs based on a discrete mapping instead of a continuous function. A group of discrete neurons (nodes) closely interconnected form an assembly and carry out a basic functionality. Some substructures and superstructures of such assemblies are developed to enable complex symbolic schemas to be represented and processed in connectionist networks. The paper shows that logical inference can be performed precisely, when necessary, in these networks and with certain genaralization, more flexible inference (fuzzy inference) can also be performed. The development of various connectionist constructs demonstrates the possibility of implementing symbolic schemas, in their full complexity, in connectionist networks.     

Connectionism and Novel Combinations of Skills: Implications for Cognitive Architecture

In the late 1980s, there were many who heralded the emergence of connectionism as a new paradigm – one which would eventually displace the classically symbolic methods then dominant in AI and Cognitive Science. At present, there remain influential connectionists who continue to defend connectionism as a more realistic paradigm for modeling cognition, at all levels of abstraction, than the classical methods of AI. Not infrequently, one encounters arguments along these lines: given what we know about neurophysiology, it is just not plausible to suppose that our brains are digital computers. Thus, they could not support a classical architecture. I argue here for a middle ground between connectionism and classicism. I assume, for argument's sake, that some form(s) of connectionism can provide reasonably approximate models – at least for lower-level cognitive processes. Given this assumption, I argue on theoretical and empirical grounds that most human mental skills must reside in separate connectionist modules or sub-networks. Ultimately, it is argued that the basic tenets of connectionism, in conjunction with the fact that humans often employ novel combinations of skill modules in rule following and problem solving, lead to the plausible conclusion that, in certain domains, high level cognition requires some form of classical architecture. During the course of argument, it emerges that only an architecture with classical structure could support the novel patterns of information flow and interaction that would exist among the relevant set of modules. Such a classical architecture might very well reside in the abstract levels of a hybrid system whose lower-level modules are purely connectionist.     

The Explanatory Role of Computation in Cognitive Science

Which notion of computation (if any) is essential for explaining cognition? Five answers to this question are discussed in the paper. (1) The classicist answer: symbolic (digital) computation is required for explaining cognition; (2) The broad digital computationalist answer: digital computation broadly construed is required for explaining cognition; (3) The connectionist answer: sub-symbolic computation is required for explaining cognition; (4) The computational neuroscientist answer: neural computation (that, strictly, is neither digital nor analogue) is required for explaining cognition; (5) The extreme dynamicist answer: computation is not required for explaining cognition. The first four answers are only accurate to a first approximation. But the ??devil?? is in the details. The last answer cashes in on the parenthetical ??if any?? in the question above. The classicist argues that cognition is symbolic computation. But digital computationalism need not be equated with classicism. Indeed, computationalism can, in principle, range from digital (and analogue) computationalism through (the weaker thesis of) generic computationalism to (the even weaker thesis of) digital (or analogue) pancomputationalism. Connectionism, which has traditionally been criticised by classicists for being non-computational, can be plausibly construed as being either analogue or digital computationalism (depending on the type of connectionist networks used). Computational neuroscience invokes the notion of neural computation that may (possibly) be interpreted as a sui generis type of computation. The extreme dynamicist argues that the time has come for a post-computational cognitive science. This paper is an attempt to shed some light on this debate by examining various conceptions and misconceptions of (particularly digital) computation.     

SYMCON—A Hybrid Symbolic/Connectionist System for Word Sense Disambiguation

Connectionist methods and knowledge-based techniques are two largely complementary approaches to natural language processing (NLP). However, they both have some potential problems which preclude their being a general purpose processing method. Research reveals that a hybrid processing approach that combines connectionist with symbolic techniques may be able to use the strengths of one processing paradigm to address the weakness of the other one. Hence, a system that effectively combines the two different approaches can be superior to either one in isolation. This paper describes a hybrid system—SYMCON (SYMbolic and CONnectionist) which integrates symbolic and connectionist techniques in an attempt to solve the problem of word sense disambiguation (WSD), which is arguably one of the most fundamental and difficult issues in NLP. It consists of three sub-systems: first, a distributed simple recurrent network (SRN) is trained by using the standard back-propagation algorithm to learn the semantic relationships among concepts, thereby generating categorical constraints that are supplied to the other two sub-systems as the initial results of pre-processing. The second sub-system of SYMCON is a knowledge-based symbolic component consisting of a knowledge base containing general inferencing rules in a certain application domain. Third, a localist network is used to select the best interpretation among multiple alternatives and potentially ambiguous inference paths by spreading activation throughout the network. The structure, initial states, and connection weights of the network are determined by the processing outcome in the other two sub-systems. This localist network can be viewed as a medium between the distributed network and the symbolic sub-system. Such a hybrid symbolic/connectionist system combines information from all three sources to select the most plausible interpretation for ambiguous words.     

Exploring the power of genetic search in learning symbolicclassifiers

In this paper we show, in a constructive way, that there are problems for which the use of genetic algorithm based learning systems can be at least as effective as traditional symbolic or connectionist approaches. To this aim, the system REGAL is briefly described, and its application to two classical benchmarks for machine learning is discussed, by comparing the results with the best ones published in the literature     

A connectionist computational model for epistemic and temporal reasoning

The importance of the efforts to bridge the gap between the connectionist and symbolic paradigms of artificial intelligence has been widely recognized. The merging of theory (background knowledge) and data learning (learning from examples) into neural-symbolic systems has indicated that such a learning system is more effective than purely symbolic or purely connectionist systems. Until recently, however, neural-symbolic systems were not able to fully represent, reason, and learn expressive languages other than classical propositional and fragments of first-order logic. In this article, we show that nonclassical logics, in particular propositional temporal logic and combinations of temporal and epistemic (modal) reasoning, can be effectively computed by artificial neural networks. We present the language of a connectionist temporal logic of knowledge (CTLK). We then present a temporal algorithm that translates CTLK theories into ensembles of neural networks and prove that the translation is correct. Finally, we apply CTLK to the muddy children puzzle, which has been widely used as a test-bed for distributed knowledge representation. We provide a complete solution to the puzzle with the use of simple neural networks, capable of reasoning about knowledge evolution in time and of knowledge acquisition through learning.     

A neural fuzzy system with fuzzy supervised learning

A neural fuzzy system learning with fuzzy training data (fuzzy if-then rules) is proposed in this paper. This system is able to process and learn numerical information as well as linguistic information. At first, we propose a five-layered neural network for the connectionist realization of a fuzzy inference system. The connectionist structure can house fuzzy logic rules and membership functions for fuzzy inference. We use alpha-level sets of fuzzy numbers to represent linguistic information. The inputs, outputs, and weights of the proposed network can be fuzzy numbers of any shape. Furthermore, they can be hybrid of fuzzy numbers and numerical numbers through the use of fuzzy singletons. Based on interval arithmetics, a fuzzy supervised learning algorithm is developed for the proposed system. It extends the normal supervised learning techniques to the learning problems where only linguistic teaching signals are available. The fuzzy supervised learning scheme can train the proposed system with desired fuzzy input-output pairs which are fuzzy numbers instead of the normal numerical values. With fuzzy supervised learning, the proposed system can be used for rule base concentration to reduce the number of rules in a fuzzy rule base. Simulation results are presented to illustrate the performance and applicability of the proposed system.     

A general framework for adaptive processing of data structures

A structured organization of information is typically required by symbolic processing. On the other hand, most connectionist models assume that data are organized according to relatively poor structures, like arrays or sequences. The framework described in this paper is an attempt to unify adaptive models like artificial neural nets and belief nets for the problem of processing structured information. In particular, relations between data variables are expressed by directed acyclic graphs, where both numerical and categorical values coexist. The general framework proposed in this paper can be regarded as an extension of both recurrent neural networks and hidden Markov models to the case of acyclic graphs. In particular we study the supervised learning problem as the problem of learning transductions from an input structured space to an output structured space, where transductions are assumed to admit a recursive hidden state-space representation. We introduce a graphical formalism for representing this class of adaptive transductions by means of recursive networks, i.e., cyclic graphs where nodes are labeled by variables and edges are labeled by generalized delay elements. This representation makes it possible to incorporate the symbolic and subsymbolic nature of data. Structures are processed by unfolding the recursive network into an acyclic graph called encoding network. In so doing, inference and learning algorithms can be easily inherited from the corresponding algorithms for artificial neural networks or probabilistic graphical model.     

Levels of description and explanation in cognitive science

The notion of levels has been widely used in discussions of cognitive science, especially in discussions of the relation of connectionism to symbolic modeling of cognition. I argue that many of the notions of levels employed are problematic for this purpose, and develop an alternative notion grounded in the framework of mechanistic explanation. By considering the source of the analogies underlying both symbolic modeling and connectionist modeling, I argue that neither is likely to provide an adequate analysis of processes at the level at which cognitive theories attempt to function: One is drawn from too low a level, the other from too high a level. If there is a distinctly cognitive level, then we still need to determine what are the basic organizational principles at that level.     

An historical overview of natural language processing systems that learn

A fundamental issue in natural language processing is the prerequisite of an enormous quantity of preprogrammed knowledge concerning both the language and the domain under examination. Manual acquisition of this knowledge is tedious and error prone. Development of an automated acquisition process would prove invaluable.This paper references and overviews a range of the systems that have been developed in the domain of machine learning and natural language processing. Each system is categorised into either a symbolic or connectionist paradigm, and has its own characteristics and limitations described.     

Interruptibility as a constraint on hybrid systems

It is widely mooted that a plausible computational cognitive model should involve both symbolic and connectionist components. However, sound principles for combining these components within a hybrid system are currently lacking; the design of such systems is oftenad hoc. In an attempt to ameliorate this we provide a framework of types of hybrid systems and constraints therein, within which to explore the issues. In particular, we suggest the use of system independent constraints, whose source lies in general considerations about cognitive systems, rather than in particular technological or task-based considerations. We illustrate this through a detailed examination of an interruptibility constraint: handling interruptions is a fundamental facet of cognition in a dynamic world. Aspects of interruptions are delineated, as are their precise expression in symbolic and connectionist systems. We illustrate the interaction of the various constraints from interruptibility in the different types of hybrid systems. The picture that emerges of the relationship between the connectionist and the symbolic within a hybrid system provides for sufficient flexibility and complexity to suggest interesting general implications for cognition, thus vindicating the utility of the framework.     

An integrated symbolic/connectionist architecture for parsing italian sentences containing pp-attachment ambiguities

This paper describes a hybrid (symbolic/connectionist) system that performs PP-attachment disambiguation by taking advantage of three distinguishing features of neutral networks: distributed representation, functional compositionality, and inductive learning. The connectionist part of the system follows all the steps performed by the symbolic parser, and drives the parser's behavior by inducing a bias towards the most semantically plausible attachment choices. The sentence to be parsed is read one word at a time. When the symbolic parser has more than one production to apply, the connectionist module has already developed an inner representation of the sentence and a distribution of probabilities over the possible choices. The parser continues its work according to such a distribution.     

Heterogeneous knowledge representation: integrating connectionist and symbolic computation

Heterogeneous knowledge representation allows combination of several knowledge representation techniques. For instance, connectionist and symbolic systems are two different computational paradigms and knowledge representations. Unfortunately, the integration of different paradigms and knowledge representations is not easy and very often is informal. In this paper, we propose a formal approach to integrate these two paradigms where as a symbolic system we consider a (logic) rule-based system. The integration is operated at language level between neural networks and rule languages. The formal model that allows the integration is based on constraint logic programming and provides an integrated framework to represent and process heterogeneous knowledge. In order to achieve this we define a new language that allows expression and modelling in a natural and intuitive way the above issues together with the operational semantics.     

Connectionist meta-representation for propositional attitudes

Abstract

This paper discusses the representation of propositional attitudes (beliefs, etc.) in connectionist systems that do not implement symbolic representations. One prominent way of symbolically representing attitudes is through meta-representational schemes. These have representational expressions that themselves refer to representational expressions. Meta-representation is one of the most expressively powerful symbolic approaches for attitude representation. Therefore: could non-implementational connectionist systems use an analogous approach? Unfortunately, it is not straightforward to devise a plausible analogy. The paper looks at three main possibilities: (i) the representational activation patterns of the non-implementational connectionist system refer to the system's own activation patterns; (ii) the activation patterns refer to formal symbolic expressions; and (iii) the activation patterns refer to natural-language expressions. These approaches, which are not claimed to be exhaustive, concentrate respectively on the following facts about symbolic meta-representation schemes: (1) they typically refer to their own representational expressions; (2) the schemes typically refer to formal symbolic expressions; although (3) some of the schemes refer to natural language expressions. The article briefly argues that possibility (iii) avoids some of the problems of (i) and (ii). There are also two independent, non-connectionism-derived reasons for considering (iii). One is that it is strongly related to a prevalent commonsense metaphor of beliefs and so on as internal, natural language utterances. (The other is to do with the heightened difficulty of handling vague quantification within propositional attitude contexts, but is discussed elsewhere and not in the present paper.) The paper as a whole highlights the point that even a non-implementational connectionist system must be able to think about complex symbolic constructs such as logic expressions and natural language phrases, even though it does not think with them.