Evaluation of an optimal design method for a multilayer perceptron by using the design of experiments

We evaluated the performance of an optimal design method for a multilayer perceptron (MLP) by using the design of experiments (DOE). In our previous work, we proposed an optimal design method for MLPs in order to determine the optimal values of such parameters as the number of neurons in the hidden layers and the learning rates. In this article, we evaluate the performance of the proposed design method through a comparison with a genetic algorithm (GA)-based design method. We target an optimal design of MLPs with six layers. We also evaluate the proposed designed method in terms of calculating the amount of optimization. Through the above-mentioned evaluation and analysis, we aim at improving the proposed design method in order to obtain an optimal MLP with less effort.     

Lur'e systems with multilayer perceptron and recurrent neuralnetworks: absolute stability and dissipativity

Sufficient conditions for absolute stability and dissipativity of continuous-time recurrent neural networks with two hidden layers are presented. In the autonomous case this is related to a Lur'e system with multilayer perceptron nonlinearity. Such models are obtained after parametrizing general nonlinear models and controllers by a multilayer perceptron with one hidden layer and representing the control scheme in standard plant form. The conditions are expressed as matrix inequalities and can be employed for nonlinear H_∞ control and imposing closed-loop stability in dynamic backpropagation     

The multilayer perceptron as an approximation to a Bayes optimaldiscriminant function

The multilayer perceptron, when trained as a classifier using backpropagation, is shown to approximate the Bayes optimal discriminant function. The result is demonstrated for both the two-class problem and multiple classes. It is shown that the outputs of the multilayer perceptron approximate the a posteriori probability functions of the classes being trained. The proof applies to any number of layers and any type of unit activation function, linear or nonlinear.     

Acceleration of multilayer perceptron training with CUDA

CUDA acceleration of Broyden-Fletcher-Goldfarb-Shanno (BFGS) training algorithm is described. Speedup in comparison with reference single thread CPU realization is ～18.     

Single imputation with multilayer perceptron and multiple imputation combining multilayer perceptron and k-nearest neighbours for monotone patterns

The knowledge discovery process is supported by data files information gathered from collected data sets, which often contain errors in the form of missing values. Data imputation is the activity aimed at estimating values for missing data items. This study focuses on the development of automated data imputation models, based on artificial neural networks for monotone patterns of missing values. The present work proposes a single imputation approach relying on a multilayer perceptron whose training is conducted with different learning rules, and a multiple imputation approach based on the combination of multilayer perceptron and k-nearest neighbours. Eighteen real and simulated databases were exposed to a perturbation experiment with random generation of monotone missing data pattern. An empirical test was accomplished on these data sets, including both approaches (single and multiple imputations), and three classical single imputation procedures – mean/mode imputation, regression and hot-deck – were also considered. Therefore, the experiments involved five imputation methods. The results, considering different performance measures, demonstrated that, in comparison with traditional tools, both proposals improve the automation level and data quality offering a satisfactory performance.     

An on-line hybrid learning algorithm for multilayer perceptron in identification problems

A hybrid learning algorithm for multilayered perceptrons (MLPs) and pattern-by-pattern training, based on optimized instantaneous learning rates and the recursive least squares method, is proposed. This hybrid solution is developed for on-line identification of process models based on the use of MLPs, and can speed up the learning process of the MLPs substantially, while simultaneously preserving the stability of the learning process. For illustration and test purposes the proposed algorithm is applied to the identification of a non-linear dynamic system.     

Sensitivity analysis of multilayer perceptron with differentiableactivation functions

In a neural network, many different sets of connection weights can approximately realize an input-output mapping. The sensitivity of the neural network varies depending on the set of weights. For the selection of weights with lower sensitivity or for estimating output perturbations in the implementation, it is important to measure the sensitivity for the weights. A sensitivity depending on the weight set in a single-output multilayer perceptron (MLP) with differentiable activation functions is proposed. Formulas are derived to compute the sensitivity arising from additive/multiplicative weight perturbations or input perturbations for a specific input pattern. The concept of sensitivity is extended so that it can be applied to any input patterns. A few sensitivity measures for the multiple output MLP are suggested. For the verification of the validity of the proposed sensitivities, computer simulations have been performed, resulting in good agreement between theoretical and simulation outcomes for small weight perturbations.     

Optimal experiment design based on local model networks and multilayer perceptron networks

This paper addresses the topic of model based design of experiments for the identification of nonlinear dynamic systems. Data driven modeling decisively depends on informative input and output data obtained from experiments. Design of experiments is targeted to generate informative data and to reduce the experimentation effort as much as possible. Furthermore, design of experiments has to comply with constraints on the system inputs and the system output, in order to prevent damage to the real system and to provide stable operational conditions during the experiment. For that purpose a model based approach is chosen for the optimization of excitation signals in this paper. Two different modeling architectures, namely multilayer perceptron networks and local model networks are chosen and the experiment design is based on the optimization of the Fisher information matrix of the associated model architecture. The paper presents and discusses feasible problem formulations and solution approaches for the constrained dynamic design of experiments. In this context the effects of the Fisher information matrix in the static and the dynamic configurations are discussed. The effectiveness of the proposed method is demonstrated on a complex nonlinear dynamic engine simulation model and an analysis as well as a comparison of the presented model architectures for model based experiment design is given.     

Sensitivity analysis of multilayer perceptron to input and weightperturbations

An important issue in the design and implementation of a neural network is the sensitivity of its output to input and weight perturbations. In this paper, we discuss the sensitivity of the most popular and general feedforward neural networks-multilayer perceptron (MLP). The sensitivity is defined as the mathematical expectation of the output errors of the MLP due to input and weight perturbations with respect to all input and weight values in a given continuous interval. The sensitivity for a single neuron is discussed first and an analytical expression that is a function of the absolute values of input and weight perturbations is approximately derived. Then an algorithm is given to compute the sensitivity for the entire MLP. As intuitively expected, the sensitivity increases with input and weight perturbations, but the increase has an upper bound that is determined by the structural configuration of the MLP, namely the number of neurons per layer and the number of layers. There exists an optimal value for the number of neurons in a layer, which yields the highest sensitivity value. The effect caused by the number of layers is quite unexpected. The sensitivity of a neural network may decrease at first and then almost keeps constant while the number increases.     

A multiple multilayer perceptron neural network with an adaptive learning algorithm for thyroid disease diagnosis in the internet of medical things

The Journal of Supercomputing - Medical information systems such as Internet of Medical Things (IoMT) are gained special attention over recent years. X-ray and MRI images are important sources of...     

A quantified sensitivity measure for multilayer perceptron to input perturbation

The sensitivity of a neural network's output to its input perturbation is an important issue with both theoretical and practical values. In this article, we propose an approach to quantify the sensitivity of the most popular and general feedforward network: multilayer perceptron (MLP). The sensitivity measure is defined as the mathematical expectation of output deviation due to expected input deviation with respect to overall input patterns in a continuous interval. Based on the structural characteristics of the MLP, a bottom-up approach is adopted. A single neuron is considered first, and algorithms with approximately derived analytical expressions that are functions of expected input deviation are given for the computation of its sensitivity. Then another algorithm is given to compute the sensitivity of the entire MLP network. Computer simulations are used to verify the derived theoretical formulas. The agreement between theoretical and experimental results is quite good. The sensitivity measure can be used to evaluate the MLP's performance.     

Comments on "Lur'e systems with multilayer perceptron and recurrent neural networks: absolute stability and dissipativity"

The authors point out that some of the claims and expressions in the paper mentioned in the title by Suykens et al. (ibid., vol.44 (1999)) are incorrect.     

Comparing evolutionary hybrid systems for design and optimization of multilayer perceptron structure along training parameters

In this paper we present a comparative study of several methods that combine evolutionary algorithms and local search to optimize multilayer perceptrons: A method that optimizes the architecture and initial weights of multilayer perceptrons; another that searches for training algorithm parameters, and finally, a co-evolutionary algorithm, introduced here, that handles the architecture, the network’s initial weights and the training algorithm parameters. Our aim is to determine how the co-evolutive method can obtain better results from the point of view of running time and classification ability. Experimental results show that the co-evolutionary method obtains similar or better results than the other approaches, requiring far less training epochs and thus, reducing running time.     

具有多层感知器力矩补偿的机器人自抗扰控制

本文针对机器人系统的控制特性,提出了一种基于自抗扰控制(ADRC)的关节控制算法,该算法可以克服传统控制算法中存在的如系统抗干扰能力弱,控制性能受限于建模精度,动态性能与稳态性能难以兼顾,控制律设计较为复杂等问题.针对受控系统特性给出了一套实际控制器的完整设计方法与参数整定方法,并根据控制性能指标设计优化函数完成了最优控制参数的优化,在系统参数辨识的基础上利用多层感知器(MLP)设计了对建模不确定性的补偿网络.数值仿真和实验结果均表明该算法能够实现机器人快速稳定的轨迹跟踪,具有良好的控制精度与很强的抗干扰能力,此外该算法不依赖于精确的系统模型,降低了实际设计和应用的难度,具有很好的工程应用价值.     

A multilayer perceptron solution to the match phase problem inrule-based artificial intelligence systems

In rule-based artificial intelligence (AI) planning, expert, and learning systems, it is often the case that the left-hand-sides of the rules must be repeatedly compared to the contents of some working memory. Normally, the intent is to determine which rules are relevant to the current situation (i.e., to find the conflict set). A technique using a multilayer perceptron to solve the match phase problem for rule-based AI systems is presented. A syntax for premise formulas (i.e., the left-hand-sides of the rules) is defined, and working memory is specified. From this, it is shown how to construct a multilayer perceptron that finds all of the rules which can be executed for the current situation in working memory. The complexity of the constructed multilayer perceptron is derived in terms of the maximum number of nodes and the required number of layers. A method for reducing the number of layers to at most three is presented     

一种多层前馈神经网络的快速修剪算法

针对目前神经网络在应用中难于确定隐层神经元数的问题,提出了一种神经网络结构的快速修剪算法.该算法在最优脑外科算法（OBS）的基础上,通过直接剔除冗余的隐层神经元实现神经网络结构自组织设计.实验结果表明,快速修剪算法与常规的最优脑外科算法相比,具有更简单的网络结构和更快的学习速度.     

Correction of array failure using grey wolf optimizer hybridized with an interior point algorithm

We design a grey wolf optimizer hybridized with an interior point algorithm to correct a faulty antenna array. If a single sensor fails, the radiation power pattern of the entire array is disturbed in terms of sidelobe level (SLL) and null depth level (NDL), and nulls are damaged and shifted from their original locations. All these issues can be solved by designing a new fitness function to reduce the error between the preferred and expected radiation power patterns and the null limitations. The hybrid algorithm has been designed to control the array’s faulty radiation power pattern. Antenna arrays composed of 21 sensors are used in an example simulation scenario. The MATLAB simulation results confirm the good performance of the proposed method, compared with the existing methods in terms of SLL and NDL.     

A modified gravitational search algorithm for slope stability analysis

This paper first proposes an effective modification for the gravitational search algorithm. The new strategy used an adaptive maximum velocity constraint, which aims to control the global exploration ability of the original algorithm, increase its convergence rate and thereby to obtain an acceptable solution with a lower number of iterations. We testify the performance of the modified gravitational search algorithm (MGSA) on a suite of five well-known benchmark functions and provide comparisons with standard gravitational search algorithm (SGSA). The simulated results illustrate that the modified GSA has the potential to converge faster, while improving the quality of solution. Thereafter, the proposed MGSA is employed to search for the minimum factor of safety and minimum reliability index in both deterministic and probabilistic slope stability analysis. The factor of safety is formulated using a concise approach of the Morgenstern and Price method and the advanced first-order second-moment (AFOSM) method is adopted as the reliability assessment model. The numerical experiments demonstrate that the modified algorithm significantly outperforms the original algorithm and some other methods in the literature.     

CATSMLP: toward a robust and interpretable multilayer perceptron with sigmoid activation functions.

Enhancing the robustness and interpretability of a multilayer perceptron (MLP) with a sigmoid activation function is a challenging topic. As a particular MLP, additive TS-type MLP (ATSMLP) can be interpreted based on single-stage fuzzy IF-THEN rules, but its robustness will be degraded with the increase in the number of intermediate layers. This paper presents a new MLP model called cascaded ATSMLP (CATSMLP), where the ATSMLPs are organized in a cascaded way. The proposed CATSMLP is a universal approximator and is also proven to be functionally equivalent to a fuzzy inference system based on syllogistic fuzzy reasoning. Therefore, the CATSMLP may be interpreted based on syllogistic fuzzy reasoning in a theoretical sense. Meanwhile, due to the fact that syllogistic fuzzy reasoning has distinctive advantage over single-stage IF-THEN fuzzy reasoning in robustness, this paper proves in an indirect way that the CATSMLP is more robust than the ATSMLP in an upper-bound sense. Several experiments were conducted to confirm such a claim.     

Detecting faulty sensors in an array using symmetrical structure and cultural algorithm hybridized with differential evolution

The detection of fully and partially defective sensors in a linear array composed of N sensors is addressed. First, the symmetrical structure of a linear array is proposed. Second, a hybrid technique based on the cultural algorithm with differential evolution is developed. The symmetrical structure has two advantages: (1) Instead of finding all damaged patterns, only (N–1)/2 patterns are needed; (2) We are required to scan the region from 0° to 90° instead of from 0° to 180°. Obviously, the computational complexity can be reduced. Monte Carlo simulations were carried out to validate the performance of the proposed scheme, compared with existing methods in terms of computational time and mean square error.