Path selection in multi-layer networks

Multi-layer networks are computer networks where the configuration of the network can be changed dynamically at multiple layers. However, in practice, technologies at different layers may be incompatible to each other, which necessitates a careful choice of a multi-layer network model. Not much work has been done on path selection in multi-layer networks. In this paper, we describe how to represent a multi-layer network and we provide algorithms for selecting paths in them. Throughout the paper we will use examples drawn from practical experience with routing in hybrid optical networks.     

Generalization by neural networks

The authors discuss the requirements of learning for generalization, where the traditional methods based on gradient descent have limited success. A stochastic learning algorithm based on simulated annealing in weight space is presented. The authors verify the convergence properties and feasibility of the algorithm. An implementation of the algorithm and validation experiments are described     

Generalization ability of folding networks

The information theoretical learnability of folding networks, a very successful approach capable of dealing with tree structured inputs, is examined. We find bounds on the VC, pseudo-, and fat shattering dimension of folding networks with various activation functions. As a consequence, valid generalization of folding networks can be guaranteed. However, distribution independent bounds on the generalization error cannot exist in principle. We propose two approaches which take the specific distribution into account and allow us to derive explicit bounds on the deviation of the empirical error from the real error of a learning algorithm. The first approach requires the probability of large trees to be limited a priori and the second approach deals with situations where the maximum input height in a concrete learning example is restricted     

Limitations of multi-layer perceptron networks - steps towards genetic neural networks

In this paper we investigate multi-layer perceptron networks in the task domain of Boolean functions. We demystify the multi-layer perceptron network by showing that it just divides the input space into regions constrained by hyperplanes. We use this information to construct minimal training sets. Despite using minimal training sets, the learning time of multi-layer perceptron networks with backpropagation scales exponentially for complex Boolean functions. But modular neural networks which consist of independentky trained subnetworks scale very well. We conjecture that the next generation of neural networks will be genetic neural networks which evolve their structure. We confirm Minsky and Papert: “The future of neural networks is tied not to the search for some single, universal scheme to solve all problems at once, bu to the evolution of a many-faceted technology of network design.”     

Generalization and selection of examples in feedforward neural networks

In this work, we study how the selection of examples affects the learning procedure in a boolean neural network and its relationship with the complexity of the function under study and its architecture. We analyze the generalization capacity for different target functions with particular architectures through an analytical calculation of the minimum number of examples needed to obtain full generalization (i.e., zero generalization error). The analysis of the training sets associated with such parameter leads us to propose a general architecture-independent criterion for selection of training examples. The criterion was checked through numerical simulations for various particular target functions with particular architectures, as well as for random target functions in a nonoverlapping receptive field perceptron. In all cases, the selection sampling criterion lead to an improvement in the generalization capacity compared with a pure random sampling. We also show that for the parity problem, one of the most used problems for testing learning algorithms, only the use of the whole set of examples ensures global learning in a depth two architecture. We show that this difficulty can be overcome by considering a tree-structured network of depth 2log2(N)-1.     

Application of multi-layer perceptron neural networks to vision problems

This paper discusses the application of a class of feed-forward Artificial Neural Networks (ANNs) known as Multi-Layer Perceptrons(MLPs) to two vision problems: recognition and pose estimation of 3D objects from a single 2D perspective view; and handwritten digit recognition. In both cases, a multi-MLP classification scheme is developed that combines the decisions of several classifiers. These classifiers operate on the same feature set for the 3D recognition problem whereas different feature types are used for the handwritten digit recognition. The backpropagationlearning rule is used to train the MLPs. Application of the MLP architecture to other vision problems is also briefly discussed.     

网络泛化能力与随机扩展训练集

针对神经网络的过拟合和泛化能力差的问题, 研究了样本数据的输入输出混合概率密度函数的局部最大熵密度估计, 提出了运用Chebyshev不等式的样本参数按类分批自校正方法, 以此估计拉伸样本集, 得到新的随机扩充训练集. 使估计质量更高, 效果更好. 仿真结果证明用这种方法训练的前馈神经网络具有较好的泛化性能.     

ML-SOR: Message routing using multi-layer social networks in opportunistic communications

Opportunistic networks are a generalization of DTNs in which disconnections are frequent and encounter patterns between mobile devices are unpredictable. In such scenarios, message routing is a fundamental issue. Social-based routing protocols usually exploit the social information extracted from the history of encounters between mobile devices to find an appropriate message relay. Protocols based on encounter history, however, take time to build up a knowledge database from which to take routing decisions. While contact information changes constantly and it takes time to identify strong social ties, other types of ties remain rather stable and could be exploited to augment available partial contact information. In this paper, we start defining a multi-layer social network model combining the social network detected through encounters with other social networks and investigate the relationship between these social network layers in terms of node centrality, community structure, tie strength and link prediction. The purpose of this analysis is to better understand user behavior in a multi-layered complex network combining online and offline social relationships. Then, we propose a novel opportunistic routing approach ML-SOR (Multi-layer Social Network based Routing) which extracts social network information from such a model to perform routing decisions. To select an effective forwarding node, ML-SOR measures the forwarding capability of a node when compared to an encountered node in terms of node centrality, tie strength and link prediction. Trace driven simulations show that a routing metric combining social information extracted from multiple social network layers allows users to achieve good routing performance with low overhead cost.     

Generalization properties of modular networks: implementing theparity function

The parity function is one of the most used Boolean function for testing learning algorithms because both of its simple definition and its great complexity. We construct a family of modular architectures that implement the parity function in which, every member of the family can be characterized by the fan-in max of the network, i.e., the maximum number of connections that a neuron can receive. We analyze the generalization ability of the modular networks first by computing analytically the minimum number of examples needed for perfect generalization and then by numerical simulations. Both results show that the generalization ability of these networks is systematically improved by the degree of modularity of the network. We also analyze the influence of the selection of examples in the emergence of generalization ability, by comparing the learning curves obtained through a random selection of examples to those obtained through examples selected accordingly to a general algorithm we (2000) recently proposed.     

基于队列约束的多层无线网状网络容量研究

在常规和随机多层无线网络中,有研究已经得到了网络可以达到的吞吐量数量级,然而对单个节点吞吐量和端到端时延的研究却很少.为了解决这一问题,对多跳常规多层无线网状网络中的单节点最大吞吐量和端到端时延进行了研究.推导了网络中的分组吸收概率;根据网络的排队模型,在这个分组吸收概率的基础上,使用扩散近似法得到了单节点可达吞吐量和端到端时延.仿真分析了业务模式和网络拓扑对端到端时延和单节点最大吞吐量的影响,通过仿真结果可以发现使网络处理大量节点的最优辅助节点数,这有助于优化网络资源分配,且减少网络的拥塞.     

面向图像识别的多层脉冲神经网络学习算法综述

相较于第1代和第2代神经网络,第3代神经网络的脉冲神经网络是一种更加接近于生物神经网络的模型,因此更具有生物可解释性和低功耗性。基于脉冲神经元模型,脉冲神经网络可以通过脉冲信号的形式模拟生物信号在神经网络中的传播,通过脉冲神经元的膜电位变化来发放脉冲序列,脉冲序列通过时空联合表达不仅传递了空间信息还传递了时间信息。当前面向模式识别任务的脉冲神经网络模型性能还不及深度学习,其中一个重要原因在于脉冲神经网络的学习方法不成熟,深度学习中神经网络的人工神经元是基于实数形式的输出,这使得其可以使用全局性的反向传播算法对深度神经网络的参数进行训练,脉冲序列是二值性的离散输出,这直接导致对脉冲神经网络的训练存在一定困难,如何对脉冲神经网络进行高效训练是一个具有挑战的研究问题。本文首先总结了脉冲神经网络研究领域中的相关学习算法,然后对其中主要的方法:直接监督学习、无监督学习的算法以及ANN2SNN的转换算法进行分析介绍,并对其中代表性的工作进行对比分析,最后基于对当前主流方法的总结,对未来更高效、更仿生的脉冲神经网络参数学习方法进行展望。     

基于函数逼近的多层前馈神经网络灵敏度分析

神经网络灵敏度分析对网络结构设计、硬件实现等具有重要的指导意义,已有的灵敏度计算公式对权值和输入扰动有一定限制或者计算误差较大。基于Piché的随机模型,通过使用两个逼近函数对神经网络一类Sigmoid激活函数进行高精度逼近,获得了新的神经网络灵敏度计算公式,公式取消了对权值扰动和输入扰动的限制,与其他方法相比提高了计算精度,实验证明了公式的正确性和精确性。     

A multi-layer experimental study of multimedia and QoS communication in wireless mesh networks

A significant issue in Mesh networks is to support multimedia transmissions while providing Quality of Service (QoS) guarantees to mobile users. For real-time multimedia streaming, unstable throughput or insufficient bandwidth will incur unexpected delay or jitter, and it remains difficult to provide comprehensive service guarantees for a wireless mesh environment. In this paper, we target the problem of providing multimedia QoS in wireless mesh networks. We design and implement a campus test-bed for supporting multimedia traffic in mobile wireless mesh networks, and investigate in detail some possible improvements on a number of layers to enable multimedia transmission over wireless networks with QoS support. We first study a number of improvements of some existing routing protocols to support multimedia transmissions. Some new admission control and rate control mechanisms are studied and their performance gains are verified in our experiments. In our new cross-layer adaptive rate control (CLARC) mechanism, we adaptively change the video encoder’s output bit rate based on the available network bandwidth to improve the quality of the received video. We also design a mobile gateway protocol to connect the MANET to Internet and a wireless LAN management protocol to automatically manage WLAN to provide some QoS.     

Generalization performance of support vector machines and neural networks in runoff modeling

Effective one-day lead runoff prediction is one of the significant aspects of successful water resources management in arid region. For instance, reservoir and hydropower systems call for real-time or on-line site-specific forecasting of the runoff. In this research, we present a new data-driven model called support vector machines (SVMs) based on structural risk minimization principle, which minimizes a bound on a generalized risk (error), as opposed to the empirical risk minimization principle exploited by conventional regression techniques (e.g. ANNs). Thus, this stat-of-the-art methodology for prediction combines excellent generalization property and sparse representation that lead SVMs to be a very promising forecasting method. Further, SVM makes use of a convex quadratic optimization problem; hence, the solution is always unique and globally optimal. To demonstrate the aforementioned forecasting capability of SVM, one-day lead stream flow of Bakhtiyari River in Iran was predicted using the local climate and rainfall data. Moreover, the results were compared with those of ANN and ANN integrated with genetic algorithms (ANN-GA) models. The improvements in root mean squared error (RMSE) and squared correlation coefficient (R²) by SVM over both ANN models indicate that the prediction accuracy of SVM is at least as good as that of those models, yet in some cases actually better, as well as forecasting of high-value discharges.     

Developing higher-order networks with empirically selected units

Introduces a class of simple polynomial neural network classifiers, called mask perceptrons. A series of algorithms for practical development of such structures is outlined. It relies on ordering of input attributes with respect to their potential usefulness and heuristic driven generation and selection of hidden units (monomial terms) in order to combat the exponential explosion in the number of higher-order monomial terms to choose from. Results of tests for two popular machine learning benchmarking domains (mushroom classification and faulty LED-display), and for two nonstandard domains (spoken digit recognition and article category determination) are given. All results are compared against a number of other classifiers. A procedure for converting a mask perceptron to a classical logic production rule is outlined and shown to produce a number of 100% percent accurate simple rules after training on 6-20% of a database.     

Repositioning of distinguishable urban service units on networks

This paper examines the problem of repositioning mobile emergency service units on the urban Transportation network. Repositioning problems deal with real-time movements of available servers to better anticipate short-term future requests for service. It is assumed in the paper that q nodes of the network are designated as “home locations” for q distinguishable units. Depending on the status of other servers (busy or available), any particular available server can be moved to other locations (not necessarily home locations) in the network. Using Markovian Decision Theory, the policy space consists of decisions on where and when to move servers for any possible state. The paper includes an analysis of two cases based on the quality of information on the real-time location of non-stationary service units. In one case the assumption is that the dispatcher has perfect information whereas that in the other one it is assumed that no such information is available. The objective is to find the repositioning policy which minimizes the expected cost of operating the system in the long term.     

Generalization in interactive networks: the benefits of inhibitory competition and Hebbian learning

Computational models in cognitive neuroscience should ideally use biological properties and powerful computational principles to produce behavior consistent with psychological findings. Error-driven backpropagation is computationally powerful and has proven useful for modeling a range of psychological data but is not biologically plausible. Several approaches to implementing backpropagation in a biologically plausible fashion converge on the idea of using bidirectional activation propagation in interactive networks to convey error signals. This article demonstrates two main points about these error-driven interactive networks: (1) they generalize poorly due to attractor dynamics that interfere with the network's ability to produce novel combinatorial representations systematically in response to novel inputs, and (2) this generalization problem can be remedied by adding two widely used mechanistic principles, inhibitory competition and Hebbian learning, that can be independently motivated for a variety of biological, psychological, and computational reasons. Simulations using the Leabra algorithm, which combines the generalized recirculation (GeneRec), biologically plausible, error-driven learning algorithm with inhibitory competition and Hebbian learning, show that these mechanisms can result in good generalization in interactive networks. These results support the general conclusion that cognitive neuroscience models that incorporate the core mechanistic principles of interactivity, inhibitory competition, and error-driven and Hebbian learning satisfy a wider range of biological, psychological, and computational constraints than models employing a subset of these principles.     

Gear dynamics monitoring using discrete wavelet transformation and multi-layer perceptron neural networks

This paper presents a multi-stage algorithm for the dynamic condition monitoring of a gear. The algorithm provides information referred to the gear status (fault or normal condition) and estimates the mesh stiffness per shaft revolution in case that any abnormality is detected. In the first stage, the analysis of coefficients generated through discrete wavelet transformation (DWT) is proposed as a fault detection and localization tool. The second stage consists in establishing the mesh stiffness reduction associated with local failures by applying a supervised learning mode and coupled with analytical models. To do this, a multi-layer perceptron neural network has been configured using as input features statistical parameters sensitive to torsional stiffness decrease and derived from wavelet transforms of the response signal. The proposed method is applied to the gear condition monitoring and results show that it can update the mesh dynamic properties of the gear on line.     

Objective functions for training new hidden units in constructiveneural networks

In this paper, we study a number of objective functions for training new hidden units in constructive algorithms for multilayer feedforward networks. The aim is to derive a class of objective functions the computation of which and the corresponding weight updates can be done in O(N) time, where N is the number of training patterns. Moreover, even though input weight freezing is applied during the process for computational efficiency, the convergence property of the constructive algorithms using these objective functions is still preserved. We also propose a few computational tricks that can be used to improve the optimization of the objective functions under practical situations. Their relative performance in a set of two-dimensional regression problems is also discussed.     

A property of neural networks of associative memory with replacing units

A memory capacity exists for artificial neural networks of associative memory. The addition of new memories beyond the capacity overloads the network system and makes all learned memories irretrievable (catastrophic forgetting) unless there is a provision for forgetting old memories. This article describes a property of associative memory networks in which a number of units are replaced when networks learn. In our network, every time the network learns a new item or pattern, a number of units are erased and the same number of units are added. It is shown that the memory capacity of the network depends on the number of replaced units, and that there exists a optimal number of replaced units in which the memory capacity is maximized. The optimal number of replaced units is small, and seems to be independent of the network size. This work was presented in part at the 12th International Symposium on Artificial Life and Robotics, Oita, Japan, January 25–27, 2007