Parallel growing and training of neural networks using outputparallelism

In order to find an appropriate architecture for a large-scale real-world application automatically and efficiently, a natural method is to divide the original problem into a set of subproblems. In this paper, we propose a simple neural-network task decomposition method based on output parallelism. By using this method, a problem can be divided flexibly into several subproblems as chosen, each of which is composed of the whole input vector and a fraction of the output vector. Each module (for one subproblem) is responsible for producing a fraction of the output vector of the original problem. The hidden structure for the original problem's output units are decoupled. These modules can be grown and trained in parallel on parallel processing elements. Incorporated with a constructive learning algorithm, our method does not require excessive computation and any prior knowledge concerning decomposition. The feasibility of output parallelism is analyzed and proved. Some benchmarks are implemented to test the validity of this method. Their results show that this method can reduce computational time, increase learning speed and improve generalization accuracy for both classification and regression problems.     

Spectral–spatial based sub-pixel mapping of remotely sensed imagery with multi-scale spatial dependence

Sub-pixel mapping (SPM) is a technique used to obtain a land-cover map with a finer spatial resolution than input remotely sensed imagery. Spectral–spatial based SPM can directly apply original remote-sensing images as input to produce fine-resolution land-cover maps. However, the existing spectral–spatial based SPM algorithms only use the maximal spatial dependence principle (calculated at the sub-pixel scale) as the spatial term to describe the local spatial distribution of different land-cover features, which always results in an over-smoothed and discontinuous land-cover map. The spatial dependence can also be calculated at the coarse-pixel scale to maintain the holistic land-cover pattern information of the resultant fine-resolution land-cover map. In this article, a novel spectral–spatial based SPM algorithm with multi-scale spatial dependence is proposed to overcome the limitation in the existing spectral–spatial based SPM algorithms. The objective function of the proposed SPM algorithm is composed of three parts, namely spectral term, sub-pixel scale based spatial term, and coarse-pixel scale based spatial term. Synthetic multi-spectral, degraded Landsat multi-spectral and real IKONOS multi-spectral images are employed in the experiments to validate the performance of the proposed SPM algorithm. The proposed algorithm is evaluated visually and quantitatively by comparing with the hard-classification method and two traditional SRM algorithms including pixel-swapping (PS) and Markov-random-field (MRF) based SPM. The results indicate that the proposed algorithm can generate fine-resolution land-cover maps with higher accuracies and more detailed spatial information than other algorithms.     

Self-organization of the batch Kohonen network under quantization effects

In this work, we investigate the effect of quantized weights and inputs on the self-organizing properties of the batch variant of Kohonen's self-organizing map algorithm. In particular, we examine necessary and sufficient conditions that ensure self-organization of the batch SOM algorithm for one-dimensional (1D) networks mapping a quantized 1D input space. Using Markov chain formalism, it is shown that the existing analysis for the original algorithm can be extended to also include the more general batch variant. Finally, simulations verify the theoretical results, relate the speed of weight ordering to the distribution of the inputs, extend the results to the 2D case, and show the existence of metastable states of the Markov chain.     

Steady-state weights solution to affine projection algorithm

A new expression of the weights update equation for the affine projection algorithm (APA) is proposed that improves the convergence rate of an adaptive filter,particularly for highly colored input sign...     

Three-dimensional optical architecture and data-parallel algorithmsfor massively parallel computing

A 3-D optical architecture currently under investigation is described. This model, a single-instruction, multiple-data (SIMD) system, exploits spatial parallelism and processes 2-D binary images as fundamental computational entities using symbolic substitution logic. This system effectively implements highly structured data-parallel algorithms, such as signal and image processing, partial differential equations, multidimensional numerical transforms, and numerical supercomputing. The model includes a hierarchical mapping technique that helps design the algorithms and maps them onto the proposed optical architecture. The symbolic substitution logic and the mapping of data-parallel algorithms are discussed. The theoretical performance of the optical system was estimated and compared with that of electronic SIMD array processors. Preliminary results show that the system provides greater computational throughput and efficiency than its electronic counterparts     

Three-dimensional iterative closest point-based outdoor SLAM using terrain classification

To navigate in an unknown environment, a robot should build a model for the environment. For outdoor environments, an elevation map is used as the main world model. We considered the outdoor simultaneous localization and mapping (SLAM) method to build a global elevation map by matching local elevation maps. In this research, the iterative closest point (ICP) algorithm was used to match local elevation maps and estimate a robot pose. However, an alignment error is generated by the ICP algorithm due to false selection of corresponding points. Therefore, we propose a new method to classify environmental data into several groups, and to find the corresponding points correctly and improve the performance of the ICP algorithm. Different weights are assigned according to the classified groups because certain groups are very sensitive to the viewpoint of the robot. Three-dimensional (3-D) environmental data acquired by tilting a 2-D laser scanner are used to build local elevation maps and to classify each grid of the map. Experimental results in real environments show the increased accuracy of the proposed ICP-based matching and a reduction in matching time.     

Simple modification of Oja rule limits L1-norm of weight vector and leads to sparse connectivity

This letter describes a simple modification of the Oja learning rule, which asymptotically constrains the L1-norm of an input weight vector instead of the L2-norm as in the original rule. This constraining is local as opposed to commonly used instant normalizations, which require the knowledge of all input weights of a neuron to update each one of them individually. The proposed rule converges to a weight vector that is sparser (has more zero weights) than the vector learned by the original Oja rule with or without the zero bound, which could explain the developmental synaptic pruning.     

Piecewise Linear Projection Based on Self-Organizing Map

A piecewise linear projection algorithm, based on kohonen's Self-Organizing Map, is presented. Using this new algorithm, neural network is able to adapt its neural weights to accommodate with input space, while obtaining reduced 2-dimensional subspaces at each neural node. After completion of learning process, first project input data into their corresponding 2-D subspaces, then project all data in the 2-D subspaces into a reference 2-D subspace defined by a reference neural node. By piecewise linear projection, we can more easily deal with large data sets than other projection algorithms like Sammon's nonlinear mapping (NLM). There is no need to re-compute all the input data to interpolate new input data to the 2-D output space.     

Detection of anomalous episodes in urban Ozone maps

In addition to classification and regression, outlier detection has emerged as a relevant activity in deep learning. In comparison with previous approaches where the original features of the examples were used for separating the examples with high dissimilarity from the rest of the examples, deep learning can automatically extract useful features from raw data, thus removing the need for most of the feature engineering efforts usually required with classical machine learning approaches. This requires training the deep learning algorithm with labels identifying the examples or with numerical values. Although outlier detection in deep learning has been usually undertaken by training the algorithm with categorical labels—classifier—, it can also be performed by using the algorithm as regressor. Nowadays numerous urban areas have deployed a network of sensors for monitoring multiple variables about air quality. The measurements of these sensors can be treated individually—as time series—or collectively. Collectively, a variable monitored by a network of sensors can be transformed into a map. Maps can be used as images in machine learning algorithms—including computer vision algorithms—for outlier detection. The identification of anomalous episodes in air quality monitoring networks allows later processing this time period with finer‐grained scientific packages involving fluid dynamic and chemical evolution software, or the identification of malfunction stations. In this work, a Convolutional Neural Network is trained—as a regressor—using as input Ozone‐urban images generated from the Air Quality Monitoring Network of Madrid (Spain). The learned features are processed by Density‐based Spatial Clustering of Applications with Noise (DBSCAN) algorithm for identifying anomalous maps. Comparisons with other deep learning architectures are undertaken, for instance, autoencoders—undercomplete and denoizing—for learning salient features of the maps and later to use as input of DBSCAN. The proposed approach is able efficiently find maps with local anomalies compared to other approaches based on raw images or latent features extracted with autoencoders architectures with DBSCAN.     

Model selection of extreme learning machine based on multi-objective optimization

As a novel learning algorithm for single-hidden-layer feedforward neural networks, extreme learning machines (ELMs) have been a promising tool for regression and classification applications. However, it is not trivial for ELMs to find the proper number of hidden neurons due to the nonoptimal input weights and hidden biases. In this paper, a new model selection method of ELM based on multi-objective optimization is proposed to obtain compact networks with good generalization ability. First, a new leave-one-out (LOO) error bound of ELM is derived, and it can be calculated with negligible computational cost once the ELM training is finished. Furthermore, the hidden nodes are added to the network one-by-one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this LOO bound and the norm of output weight simultaneously in order to avoid over-fitting. Experiments on five UCI regression data sets are conducted, demonstrating that the proposed algorithm can generally obtain better generalization performance with more compact network than the conventional gradient-based back-propagation method, original ELM and evolutionary ELM.     

基于轻量级分组注意力模块的图像分类算法

针对图像分类任务中现有神经网络模型对分类对象特征表征能力不足,导致识别精度不高的问题,提出一种基于轻量级分组注意力模块（LGAM）的图像分类算法。该模块从输入特征图的通道和空间两个方向出发重构特征图：首先,将输入特征图沿通道方向进行分组并生成每个分组对应的通道注意力权重,同时采用阶梯型结构解决分组间信息不流通的问题;然后,基于各分组串联成的新特征图生成全局空间注意力权重,通过两种注意力权重加权得到重构特征图;最后,将重构特征图与输入特征图融合得到增强的特征图。以分类Top-1错误率作为评估指标,基于Cifar10和Cifar100数据集以及部分ImageNet2012数据集,对经LGAM增强之后的ResNet、Wide-ResNet、ResNeXt进行对比实验。实验结果表明,经LGAM增强之后的神经网络模型其Top-1错误率均低于增强之前1至2个百分点。因此LGAM能够提升现有神经网络模型的特征表征能力,从而提高图像分类的识别精度。     

Controllability improvement for multi-agent systems: leader selection and weight adjustment

For an uncontrollable system, adding leaders and adjusting edge weights are two methods to improve controllability. In this paper, the controllability of multi-agent systems under directed topologies is studied, especially on the leader selection problem and the weight adjustment problem. For a multi-agent system, necessary and sufficient algebraic conditions for controllability with fewest leaders are proposed. To improve controllability by adjusting edge weights, the system is supposed to be structurally controllable, which holds if and only if the communication topology contains a spanning tree. It is also proved that the number of the fewest edges needed to be assigned on new weights equals the rank deficiency of controllability matrix. In addition, a leader selection algorithm and a weight adjustment algorithm are presented. Simulation examples are provided to illustrate the theoretical results.     

基于随机特征映射的四层多核学习方法

针对单核网络模型的核函数选择无理论依据以及基于随机特征映射的四层神经网络(FRMFNN)节点规模过大的问题,提出了一种基于随机特征映射的四层多核学习神经网络(MK-FRMFNN)算法.首先,把原始输入特征通过特定的随机映射算法转化为随机映射特征;然后,经过不同的随机核映射生成多个基本核矩阵;最后,将基本核矩阵组成合成核...     

计算机兵棋作战实体轨迹聚类算法

针对计算机兵棋系统的实际应用,提出计算机兵棋实体轨迹聚类算法——CTECW(clustering trajectoriesof entities in computer wargames).算法分为3部分:轨迹预处理、轨迹分段聚类以及可视化表现.轨迹预处理将实体原始轨迹转化成实体简化轨迹,再进一步处理成轨迹分段;在DBSCAN算法的基本框架下引入DENCLUE算法中密度函数的概念,并基于提出的相似性度量函数对轨迹分段进行聚类;可视化表现将轨迹分段聚类的结果以赋有军事涵义的形式展现给参与兵棋推演的受训指挥员,体现出算法的实际应用价值.理论分析与实验结果表明,CTECW算法能够得到与TRACLUS算法比较接近的聚类结果,但计算效率却比TRACLUS算法要高,并且聚类结果不依赖于用户参数的仔细选择.     

Sensitivity analysis of multilayer perceptron with differentiableactivation functions

In a neural network, many different sets of connection weights can approximately realize an input-output mapping. The sensitivity of the neural network varies depending on the set of weights. For the selection of weights with lower sensitivity or for estimating output perturbations in the implementation, it is important to measure the sensitivity for the weights. A sensitivity depending on the weight set in a single-output multilayer perceptron (MLP) with differentiable activation functions is proposed. Formulas are derived to compute the sensitivity arising from additive/multiplicative weight perturbations or input perturbations for a specific input pattern. The concept of sensitivity is extended so that it can be applied to any input patterns. A few sensitivity measures for the multiple output MLP are suggested. For the verification of the validity of the proposed sensitivities, computer simulations have been performed, resulting in good agreement between theoretical and simulation outcomes for small weight perturbations.     

Analog implementation of a Kohonen map with on-chip learning.

Kohonen maps are self-organizing neural networks that classify and quantify n-dimensional data into a one- or two-dimensional array of neurons. Most applications of Kohonen maps use simulations on conventional computers, eventually coupled to hardware accelerators or dedicated neural computers. The small number of different operations involved in the combined learning and classification process, however, makes the Kohonen model particularly suited to a dedicated VLSI implementation, taking full advantage of the parallelism and speed that can be obtained on the chip. A fully analog implementation of a one-dimensional Kohonen map, with on-chip learning and refreshment of on-chip analog synaptic weights, is proposed. The small number of transistors in each cell allows a high degree of parallelism in the operations, which greatly improves the computation speed compared to other implementations. The storage of analog synaptic weights, based on the principle of current copiers, is emphasized. It is shown that this technique can be used successfully for the realization of VLSI Kohonen maps.     

A new robust training algorithm for a class of single-hidden layer feedforward neural networks

A robust training algorithm for a class of single-hidden layer feedforward neural networks (SLFNs) with linear nodes and an input tapped-delay-line memory is developed in this paper. It is seen that, in order to remove the effects of the input disturbances and reduce both the structural and empirical risks of the SLFN, the input weights of the SLFN are assigned such that the hidden layer of the SLFN performs as a pre-processor, and the output weights are then trained to minimize the weighted sum of the output error squares as well as the weighted sum of the output weight squares. The performance of an SLFN-based signal classifier trained with the proposed robust algorithm is studied in the simulation section to show the effectiveness and efficiency of the new scheme.     

Topology constraint free fuzzy gated neural networks for patternrecognition

A novel topology constraint free neural network architecture using a generalized fuzzy gated neuron model is presented for a pattern recognition task. The main feature is that the network does not require weight adaptation at its input and the weights are initialized directly from the training pattern set. The elimination of the need for iterative weight adaptation schemes facilitates quick network set up times which make the fuzzy gated neural networks very attractive. The performance of the proposed network is found to be functionally equivalent to spatio-temporal feature maps under a mild technical condition. The classification performance of the fuzzy gated neural network is demonstrated on a 12-class synthetic three dimensional (3-D) object data set, real-world eight-class texture data set, and real-world 12 class 3-D object data set. The performance results are compared with the classification accuracies obtained from a spatio-temporal feature map, an adaptive subspace self-organizing map, multilayer feedforward neural networks, radial basis function neural networks, and linear discriminant analysis. Despite the network's ability to accurately classify seen data and adequately generalize validation data, its performance is found to be sensitive to noise perturbations due to fine fragmentation of the feature space. This paper also provides partial solutions to the above robustness issue by proposing certain improvements to various modules of the proposed fuzzy gated neural network.     

社交网络隐私保护中的随机算法

提出了一种基于高斯随机乘法的社交网络隐私保护方法.该算法利用无向有权图表示社交网络,通过高斯随机乘法来扰乱其边的权重,保持网络最短路径不变并使其长度应与初始网络的路径长度尽可能接近,以实现对社交网络的隐私保护.从理论上证明了算法的可行性及完美算法的不存在性.采用这种随机乘法得到的仿真结果符合理论分析结果.