Deep multi-layered GMDH-type neural network using revised heuristic self-organization and its application to medical image diagnosis of liver cancer

In this study, the deep multi-layered group method of data handling (GMDH)-type neural network algorithm using revised heuristic self-organization method is proposed and applied to medical image diagnosis of liver cancer. The deep GMDH-type neural network can automatically organize the deep neural network architecture which has many hidden layers. The structural parameters such as the number of hidden layers, the number of neurons in hidden layers and useful input variables are automatically selected to minimize prediction error criterion defined as Akaike’s information criterion (AIC) or prediction sum of squares (PSS). The architecture of the deep neural network is automatically organized using the revised heuristic self-organization method which is a type of the evolutionary computation. This new neural network algorithm is applied to the medical image diagnosis of the liver cancer and the recognition results are compared with the conventional 3-layered sigmoid function neural network.     

基于特征可视化分析深度神经网络的内部表征

基于可视化的方式理解深度神经网络能直观地揭示其工作机理,即提供了黑盒模型做出决策的解释,在医疗诊断、自动驾驶等领域尤其重要。大部分现有工作均基于激活值最大化框架,即选定待观测神经元,通过优化输入值(如隐藏层特征图谱、原始图片),定性地将待观测神经元产生最大激活值时输入值的改变作为一种解释。然而,这种方法缺乏对深度神经网络深入的定量分析。文中提出了结构可视化和基于规则可视化两种可视化的元方法。结构可视化从浅至深依层可视化,发现浅层神经元具有一般性的全局特征,而深层神经元更针对细节特征。基于规则可视化包括交集与差集规则,可以帮助发现共享神经元与抑制神经元的存在,它们分别学习了不同类别的共有特征与抑制不相关的特征。实验针对代表性卷积网络VGG和残差网络ResNet在ImageNet和微软COCO数据集上进行了分析。通过量化分析发现,ResNet和VGG均有很高的稀疏性,通过屏蔽一些低激活值的“噪音”神经元,发现其对深度神经网络分类准确率均没有影响,甚至有一定程度的提高作用。文中通过可视化和量化分析深度神经网络的隐藏层特征,揭示其内部特征表达,从而为高性能深度神经网络的设计提供指导和借鉴。     

Research of stacked denoising sparse autoencoder

Learning results depend on the representation of data, so how to efficiently represent data has been a research hot spot in machine learning and artificial intelligence. With the deepening of the deep learning research, studying how to train the deep networks to express high dimensional data efficiently also has been a research frontier. In order to present data more efficiently and study how to express data through deep networks, we propose a novel stacked denoising sparse autoencoder in this paper. Firstly, we construct denoising sparse autoencoder through introducing both corrupting operation and sparsity constraint into traditional autoencoder. Then, we build stacked denoising sparse autoencoders which has multi-hidden layers by layer-wisely stacking denoising sparse autoencoders. Experiments are designed to explore the influences of corrupting operation and sparsity constraint on different datasets, using the networks with various depth and hidden units. The comparative experiments reveal that test accuracy of stacked denoising sparse autoencoder is much higher than other stacked models, no matter what dataset is used and how many layers the model has. We also find that the deeper the network is, the less activated neurons in every layer will have. More importantly, we find that the strengthening of sparsity constraint is to some extent equal to the increase in corrupted level.

     

Entropy-based generation of supervised neural networks for classification of structured patterns

Sperduti and Starita proposed a new type of neural network which consists of generalized recursive neurons for classification of structures. In this paper, we propose an entropy-based approach for constructing such neural networks for classification of acyclic structured patterns. Given a classification problem, the architecture, i.e., the number of hidden layers and the number of neurons in each hidden layer, and all the values of the link weights associated with the corresponding neural network are automatically determined. Experimental results have shown that the networks constructed by our method can have a better performance, with respect to network size, learning speed, or recognition accuracy, than the networks obtained by other methods.     

Stacked autoencoder based deep random vector functional link neural network for classification

Extreme learning machine (ELM), which can be viewed as a variant of Random Vector Functional Link (RVFL) network without the input–output direct connections, has been extensively used to create multi-layer (deep) neural networks. Such networks employ randomization based autoencoders (AE) for unsupervised feature extraction followed by an ELM classifier for final decision making. Each randomization based AE acts as an independent feature extractor and a deep network is obtained by stacking several such AEs. Inspired by the better performance of RVFL over ELM, in this paper, we propose several deep RVFL variants by utilizing the framework of stacked autoencoders. Specifically, we introduce direct connections (feature reuse) from preceding layers to the fore layers of the network as in the original RVFL network. Such connections help to regularize the randomization and also reduce the model complexity. Furthermore, we also introduce denoising criterion, recovering clean inputs from their corrupted versions, in the autoencoders to achieve better higher level representations than the ordinary autoencoders. Extensive experiments on several classification datasets show that our proposed deep networks achieve overall better and faster generalization than the other relevant state-of-the-art deep neural networks.     

Heuristic principles for the design of artificial neural networks

Artificial neural networks were used to support applications across a variety of business and scientific disciplines during the past years. Artificial neural network applications are frequently viewed as black boxes which mystically determine complex patterns in data. Contrary to this popular view, neural network designers typically perform extensive knowledge engineering and incorporate a significant amount of domain knowledge into artificial neural networks. This paper details heuristics that utilize domain knowledge to produce an artificial neural network with optimal output performance. The effect of using the heuristics on neural network performance is illustrated by examining several applied artificial neural network systems. Identification of an optimal performance artificial neural network requires that a full factorial design with respect to the quantity of input nodes, hidden nodes, hidden layers, and learning algorithm be performed. The heuristic methods discussed in this paper produce optimal or near-optimal performance artificial neural networks using only a fraction of the time needed for a full factorial design.     

Revised GMDH-type neural network algorithm self-selecting optimum neural network architecture

In this study, the revised group method of data handling (GMDH)-type neural network (NN) algorithm self-selecting the optimum neural network architecture is applied to the identification of a nonlinear system. In this algorithm, the optimum neural network architecture is automatically organized using two kinds of neuron architecture, such as the polynomial- and sigmoid function-type neurons. Many combinations of the input variables, in which the high order effects of the input variables are contained, are generated using the polynomial-type neurons, and useful combinations are selected using the prediction sum of squares (PSS) criterion. These calculations are iterated, and the multilayered architecture is organized. Furthermore, the structural parameters, such as the number of layers, the number of neurons in the hidden layers, and the useful input variables, are automatically selected in order to minimize the prediction error criterion defined as PSS.     

基于模糊神经网络的粗糙集在股市预测中的应用

提出在模糊神经网络中使用粗糙集理论进行网络的设计.在模糊神经网络中引入粗糙集理论,不仅可以去除模糊神经网络中输入层的冗余神经元而且可以确定隐含层神经元的数目,从而使模糊神经网络具有更准确的逼近收敛能力和较高的精度.最后应用于股票市场,在股票买卖时机预测中取得了良好的效果.     

Principal component analysis of fuzzy data using autoassociative neural networks

This paper describes an extension of principal component analysis (PCA) allowing the extraction of a limited number of relevant features from high-dimensional fuzzy data. Our approach exploits the ability of linear autoassociative neural networks to perform information compression in just the same way as PCA, without explicit matrix diagonalization. Fuzzy input values are propagated through the network using fuzzy arithmetics, and the weights are adjusted to minimize a suitable error criterion, the inputs being taken as target outputs. The concept of correlation coefficient is extended to fuzzy numbers, allowing the interpretation of the new features in terms of the original variables. Experiments with artificial and real sensory evaluation data demonstrate the ability of our method to provide concise representations of complex fuzzy data.     

Supervised Learning of Logical Operations in Layered Spiking Neural Networks with Spike Train Encoding

Few algorithms for supervised training of spiking neural networks exist that can deal with patterns of multiple spikes, and their computational properties are largely unexplored. We demonstrate in a set of simulations that the ReSuMe learning algorithm can successfully be applied to layered neural networks. Input and output patterns are encoded as spike trains of multiple precisely timed spikes, and the network learns to transform the input trains into target output trains. This is done by combining the ReSuMe learning algorithm with multiplicative scaling of the connections of downstream neurons. We show in particular that layered networks with one hidden layer can learn the basic logical operations, including Exclusive-Or, while networks without hidden layer cannot, mirroring an analogous result for layered networks of rate neurons. While supervised learning in spiking neural networks is not yet fit for technical purposes, exploring computational properties of spiking neural networks advances our understanding of how computations can be done with spike trains.     

Three-dimensional medical image analysis of the heart by the revised GMDH-type neural network self-selecting optimum neural network architecture

In this study, a revised group method of data handling (GMDH)-type neural network algorithm which self-selects the optimum neural network architecture is applied to 3-dimensional medical image analysis of the heart. The GMDH-type neural network can automatically organize the neural network architecture by using the heuristic self-organization method, which is the basic theory of the GMDH algorism. The heuristic self-organization method is a kind of evolutionary computation method. In this revised GMDH-type neural network algorithm, the optimum neural network architecture was automatically organized using the polynomial and sigmoid function neurons. Furthermore, the structural parameters, such as the number of layers, the number of neurons in the hidden layers, and the useful input variables, are selected automatically in order to minimize the prediction error criterion, defined as the prediction sum of squares (PSS).     

基于序列输入的神经网络模型算法及应用

为提高神经网络的逼近能力,提出一种基于序列输入的神经网络模型及算法。模型隐层为序列神经元,输出层为普通神经元。输入为多维离散序列,输出为普通实值向量。先将各维离散输入序列值按序逐点加权映射,再将这些映射结果加权聚合之后映射为隐层序列神经元的输出,最后计算网络输出。采用Levenberg-Marquardt算法设计了该模型学习算法。仿真结果表明,当输入节点和序列长度比较接近时,模型的逼近能力明显优于普通神经网络。     

Combining seasonal time series ARIMA method and neural networks with genetic algorithms for predicting the production value of the mechanical industry in Taiwan

Supplying industrial firms with an accurate method of forecasting the production value of the mechanical industry to facilitate decision makers in precise planning is highly desirable. Numerous methods, including the autoregressive integrated-moving average (ARIMA) model and artificial neural networks can make accurate forecasts based on historical data. The seasonal ARIMA (SARIMA) model and artificial neural networks can also handle data involving trends and seasonality. Although neural networks can make predictions, deciding the most appropriate input data, network structure and learning parameters are difficult. Therefore, this article presents a hybrid forecasting method that combines the SARIMA model and neural networks with genetic algorithms. Analytical results generated by the SARIMA model are inputted as the input data of a neural network. Subsequently, the number of neurons in the hidden layer and the number of learning parameters of the neural network architecture are globally optimized using genetic algorithms. This model is subsequently adopted to forecast seasonal time series data of the production value of the mechanical industry in Taiwan. The results presented here provide a valuable reference for decision makers in industry.     

A Cross-layer based mapping for spiking neural network onto network on chip

Abstract

Network-on-Chip provides a packet-based and scalable inter-connected structure for spiking neural networks. However, existing neural mapping methods just distribute all neurons of a population into an on-chip network core or nearby cores sequentially. As there is no connection among population, the population based mapping degrades inter-neuron communicating performance between different cores. This paper presents a Cross-LAyer based neural MaPping method that maps synaptic connected neurons belonging to adjacent layers into the same on-chip network node. In order to adapt to various input patterns, the strategy also takes input spike rate into consideration and remap neurons for improving mapping efficiency. The method helps to reduce inter-core communication cost. The experimental results demonstrate the efficient results of the proposed mapping strategy in the aspect of spike transfer latency as well as dynamic energy cost improvement. In the applications of handwritten digits and edge extraction, in which the type of interconnection among neurons is different, the neural mapping algorithm reduces spike average transfer latency by maximum 42.83%, and reduces dynamic energy by maximum 36.29%.     

A context layered locally recurrent neural network for dynamic system identification

This paper puts forward a novel recurrent neural network (RNN), referred to as the context layered locally recurrent neural network (CLLRNN) for dynamic system identification. The CLLRNN is a dynamic neural network which appears in effective in the input–output identification of both linear and nonlinear dynamic systems. The CLLRNN is composed of one input layer, one or more hidden layers, one output layer, and also one context layer improving the ability of the network to capture the linear characteristics of the system being identified. Dynamic memory is provided by means of feedback connections from nodes in the first hidden layer to nodes in the context layer and in case of being two or more hidden layers, from nodes in a hidden layer to nodes in the preceding hidden layer. In addition to feedback connections, there are self-recurrent connections in all nodes of the context and hidden layers. A dynamic backpropagation algorithm with adaptive learning rate is derived to train the CLLRNN. To demonstrate the superior properties of the proposed architecture, it is applied to identify not only linear but also nonlinear dynamic systems. The efficiency of the proposed architecture is demonstrated by comparing the results to some existing recurrent networks and design configurations. In addition, performance of the CLLRNN is analyzed through an experimental application to a dc motor connected to a load to show practicability and effectiveness of the proposed neural network. Results of the experimental application are presented to make a quantitative comparison with an existing recurrent network in the literature.     

基于深度学习算法的无线传感器异常数据检测

提出了基于深度学习的异常数据检测的方法,精准检测到无线传感器异常数据并直观展现检测结果。基于无线传感器网络模型分簇原理,通过异常数据驱动的簇内数据融合机制,去除无线传感器网络中的无效数据,获取无线传感器网络有效数据融合结果。构建了具有4层隐含层的深度卷积神经网络,将预处理后的无线传感器网络数据作为模型输入,通过隐含层完成数据特征提取和映射后,由输出层输出异常数据检测结果。实验证明：该方法可有效融合不同类型数据,且网络节点平均能耗较低;包含4层隐含层的深度卷积神经网络平均分类精度高达98.44%,1000次迭代后隐含层的训练损失均趋于0,可实现无线传感器异常数据实时、直观、准确检测。     

一种基于粗糙集理论的神经网络图像分割方法

神经网络用于分割图像时需要大量的训练数据,由于数据量大,计算速度相当慢。不适合实时数据处理。基于此,将粗糙集理论与神经网络相结合,提出基于粗糙集的神经网络图像分割方法。利用粗糙集理论中的约简的计算方法,从图像属性中获取精简的规则,根据这些规则构造神经网络各层的神经元个数,并根据粗糙集理论中的属性重要性来修正神经网络的权值。实验结果表明,该方法抗噪能力强,提高了精度,在大大缩短网络训练时间的同时改善了分割效果。满足图像处理的实时性要求。     

BP神经网络合理隐结点数确定的改进方法

合理选择隐含层结点个数是BP神经网络构造中的关键问题,对网络的适应能力、学习速率都有重要的影响.在此提出一种确定隐结点个数的改进方法.该方法基于隐含层神经元输出之间的线性相关关系与线性无关关系,对神经网络隐结点个数进行削减,缩减网络规模.以零件工艺过程中的加工参数作为BP神经网络的输入,加工完成的零件尺寸作为BP神经网络的输出建立模型,把该方法应用于此神经网络模型中,其训练结果证明了该方法的有效性.     

Human activity recognition via wearable devices using enhanced ternary weight convolutional neural network

Human activity recognition (HAR) has been known as an active area for more than a decade, and there are still crucial aspects that are intended as challenging problems. Providing detailed and appropriate information about the activities and behaviors of people is one of the most important fields in ubiquitous computing. There are numerous applications in this field, among which healthcare, security, and entertainment scenarios can be listed. Human activity recognition can be carried out with the assistance of smartphone sensors such as accelerometers and gyroscopes or images captured from webcams. Today, the application of deep neural networks in this domain has received much attention and has led to more accurate and effective results compared to traditional techniques. The deep neural network performs arithmetic operations on a various number of hidden layers. In this article, a new approach called HAR-CT is proposed to enhance the accuracy of human activity recognition in various classes by adopting a convolutional neural network (CNN). Subsequently, an optimization technique using the TWN model is also suggested to reduce the complexity of the deep neural network approach that decreases the energy consumption of mobile devices. To this end, the float precision weights of the convolutional neural network are quantized and converted into ternary weights, while the decline in the accuracy is very low compared to the initial deep neural network. The evaluation results of both networks demonstrate that the proposed methods outperform the recently published approaches in human activity recognition.     

A novel weight initialization with adaptive hyper-parameters for deep semantic segmentation

The semantic segmentation process divides an image into its constituent objects and background by assigning a corresponding class label to each pixel in the image. Semantic segmentation is an important area in computer vision with wide practical applications. The contemporary semantic segmentation approaches are primarily based on two types of deep neural networks architectures i.e., symmetric and asymmetric networks. Both types of networks consist of several layers of neurons which are arranged in two sections called encoder and decoder. The encoder section receives the input image and the decoder section outputs the segmented image. However, both sections in symmetric networks have the same number of layers and the number of neurons in an encoder layer is the same as that of the corresponding layer in the decoder section but asymmetric networks do not strictly follow such one-one correspondence between encoder and decoder layers. At the moment, SegNet and ESNet are the two leading state-of-the-art symmetric encoder-decoder deep neural network architectures. However, both architectures require extensive training for good generalization and need several hundred epochs for convergence. This paper aims to improve the convergence and enhance network generalization by introducing two novelties into the network training process. The first novelty is a weight initialization method and the second contribution is an adaptive mechanism for dynamic layer learning rate adjustment in training loop. The proposed initialization technique uses transfer learning to initialize the encoder section of the network, but for initialization of decoder section, the weights of the encoder section layers are copied to the corresponding layers of the decoder section. The second contribution of the paper is an adaptive layer learning rate method, wherein the learning rates of the encoder layers are updated based on a metric representing the difference between the probability distributions of the input images and encoder weights. Likewise, the learning rates of the decoder layers are updated based on the difference between the probability distributions of the output labels and decoder weights. Intensive empirical validation of the proposed approach shows significant improvement in terms of faster convergence and generalization.