Information-Theoretic Competitive Learning with Inverse Euclidean Distance Output Units

In this paper, we propose a new information theoretic competitive learning method. We first construct a learning method in single-layered networks, and then we extend it to supervised multi-layered networks. Competitive unit outputs are computed by the inverse of Euclidean distance between input patterns and connection weights. As distance is smaller, competitive unit outputs are stronger. In realizing competition, neither the winner-take-all algorithm nor the lateral inhibition is used. Instead, the new method is based upon mutual information maximization between input patterns and competitive units. In maximizing mutual information, the entropy of competitive units is increased as much as possible. This means that all competitive units must equally be used in our framework. Thus, no under-utilized neurons or dead neurons are generated. When using multi-layered networks, we can improve noise-tolerance performance by unifying information maximization and minimization. We applied our method with single-layered networks to a simple artificial data problem and an actual road classification problem. In both cases, experimental results confirmed that the new method can produce the final solutions almost independently of initial conditions, and classification performance is significantly improved. Then, we used multi-layered networks, and applied them to a character recognition problem and a political data analysis. In these problem, we could show that noise-tolerance performance was improved by decreasing information content on input patterns to certain points.     

Supposed maximum information for comprehensible representations in SOM

In this paper, we propose a new information-theoretic method to simplify the computation of information and to unify several methods in one framework. The new method is called “supposed maximum information,” used to produce humanly comprehensible representations in competitive learning by taking into account the importance of input units. In the new learning method, by supposing the maximum information of input units, the actual information of input units is estimated. Then, the competitive network is trained with the estimated information in input units. The method is applied not to pure competitive learning, but to self-organizing maps, because it is easy to demonstrate visually how well the new method can produce more interpretable representations. We applied the method to three well-known sets of data, namely, the Kohonen animal data, the SPECT heart data and the voting data from the machine learning database. With these data, we succeeded in producing more explicit class boundaries on the U-matrices than did the conventional SOM. In addition, for all the data, quantization and topographic errors produced by our method were lower than those by the conventional SOM.     

Regression with small data sets: a case study using code surrogates in additive manufacturing

There has been an increasing interest in recent years in the mining of massive data sets whose sizes are measured in terabytes. However, there are some problems where collecting even a single data point is very expensive, resulting in data sets with only tens or hundreds of samples. One such problem is that of building code surrogates, where a computer simulation is run using many different values of the input parameters and a regression model is built to relate the outputs of the simulation to the inputs. A good surrogate can be very useful in sensitivity analysis, uncertainty analysis, and in designing experiments, but the cost of running expensive simulations at many sample points can be high. In this paper, we use a problem from the domain of additive manufacturing to show that even with small data sets we can build good quality surrogates by appropriately selecting the input samples and the regression algorithm. Our work is broadly applicable to simulations in other domains and the ideas proposed can be used in time-constrained machine learning tasks, such as hyper-parameter optimization.     

Cooperative information maximization with Gaussian activation functions for self-organizing maps

In this paper, we propose a new information-theoretic method to produce explicit self-organizing maps (SOMs). Competition is realized by maximizing mutual information between input patterns and competitive units. Competitive unit outputs are computed by the Gaussian function of distance between input patterns and competitive units. A property of this Gaussian function is that, as distance becomes smaller, a neuron tends to fire strongly. Cooperation processes are realized by taking into account the firing rates of neighboring neurons. We applied our method to uniform distribution learning, chemical compound classification and road classification. Experimental results confirmed that cooperation processes could significantly increase information content in input patterns. When cooperative operations are not effective in increasing information, mutual information as well as entropy maximization is used to increase information. Experimental results showed that entropy maximization could be used to increase information and to obtain clearer SOMs, because competitive units are forced to be equally used on average.     

Uncertainty evaluation and model selection of extreme learning machine based on Riemannian metric

Considering the uncertainty of hidden neurons, choosing significant hidden nodes, called as model selection, has played an important role in the applications of extreme learning machines(ELMs). How to define and measure this uncertainty is a key issue of model selection for ELM. From the information geometry point of view, this paper presents a new model selection method of ELM for regression problems based on Riemannian metric. First, this paper proves theoretically that the uncertainty can be characterized by a form of Riemannian metric. As a result, a new uncertainty evaluation of ELM is proposed through averaging the Riemannian metric of all hidden neurons. Finally, the hidden nodes are added to the network one by one, and at each step, a multi-objective optimization algorithm is used to select optimal input weights by minimizing this uncertainty evaluation and the norm of output weight simultaneously in order to obtain better generalization performance. Experiments on five UCI regression data sets and cylindrical shell vibration data set are conducted, demonstrating that the proposed method can generally obtain lower generalization error than the original ELM, evolutionary ELM, ELM with model selection, and multi-dimensional support vector machine. Moreover, the proposed algorithm generally needs less hidden neurons and computational time than the traditional approaches, which is very favorable in engineering applications.     

A robust super-efficiency data envelopment analysis model for ranking of provincial gas companies in Iran

Conventional super-efficiency data envelopment analysis (DEA) models require the exact information of inputs or outputs. However, in many real world applications this simple assumption does not hold. Stochastic super-efficiency is one of recent methods which could handle uncertainty in data. Stochastic super-efficiency DEA models are normally formulated based on chance constraint programming. The method is used to estimate the efficiency of various decision making units (DMUs). In stochastic chance constraint super-efficiency DEA, the distinction of probability distribution function for input/output data is difficult and also, in several cases, there is not enough data for estimating of distribution function. We present a new method which incorporates the robust counterpart of super-efficiency DEA. The perturbation and uncertainty in data is assumed as ellipsoidal set and the robust super-efficiency DEA model is extended. The implementation of the proposed method of this paper is applied for ranking different gas companies in Iran.     

Natural computing methods in bioinformatics: A survey

Often data analysis problems in Bioinformatics concern the fusion of multisensor outputs or the fusion of multisource information, where one must integrate different kinds of biological data. Natural computing provides several possibilities in Bioinformatics, especially by presenting interesting nature-inspired methodologies for handling such complex problems. In this article we survey the role of natural computing in the domains of protein structure prediction, microarray data analysis and gene regulatory network generation. We utilize the learning ability of neural networks for adapting, uncertainty handling capacity of fuzzy sets and rough sets for modeling ambiguity, and the search potential of genetic algorithms for efficiently traversing large search spaces.     

Uncertainty visualization using HDR volume rendering

In this paper, we explore a novel idea of using high dynamic range (HDR) technology for uncertainty visualization. We focus on scalar volumetric data sets where every data point is associated with scalar uncertainty. We design a transfer function that maps each data point to a color in HDR space. The luminance component of the color is exploited to capture uncertainty. We modify existing tone mapping techniques and suitably integrate them with volume ray casting to obtain a low dynamic range (LDR) image. The resulting image is displayed on a conventional 8-bits-per-channel display device. The usage of HDR mapping reveals fine details in uncertainty distribution and enables the users to interactively study the data in the context of corresponding uncertainty information. We demonstrate the utility of our method and evaluate the results using data sets from ocean modeling.     

Robust global sensitivity analysis under deep uncertainty via scenario analysis

Complex social-ecological systems models typically need to consider deeply uncertain long run future conditions. The influence of this deep (i.e. incalculable, uncontrollable) uncertainty on model parameter sensitivities needs to be understood and robustly quantified to reliably inform investment in data collection and model refinement. Using a variance-based global sensitivity analysis method (eFAST), we produced comprehensive model diagnostics of a complex social-ecological systems model under deep uncertainty characterised by four global change scenarios. The uncertainty of the outputs, and the influence of input parameters differed substantially between scenarios. We then developed sensitivity indicators that were robust to this deep uncertainty using four criteria from decision theory. The proposed methods can increase our understanding of the effects of deep uncertainty on output uncertainty and parameter sensitivity, and incorporate the decision maker's risk preference into modelling-related activities to obtain greater resilience of decisions to surprise.     

Stochastic support vector regression with probabilistic constraints

Support Vector Regression (SVR) solves regression problems based on the concept of Support Vector Machine (SVM). In this paper, we introduce a novel model of SVR in which any training samples containing inputs and outputs are considered the random variables with known or unknown distribution functions. Constraints occurrence have a probability density function which helps to obtain maximum margin and achieve robustness. The optimal hyperplane regression can be obtained by solving a quadratic optimization problem. The proposed method is illustrated by several experiments including artificial data sets and real-world benchmark data sets.     

Performance measure of switched control systems based on covariance tensor approach

In this paper, controller performance measure is considered for switched systems. The covariance tensor-based method is proposed for the controller performance measure for this systems evolution with the discrete-valued switching dynamics and local models for continuous dynamics. We define a measurement tensor to construct the measured process outputs data, and employ the data processing technique of higher-order singular value decomposition. Applying the higher-order singular value decomposition, we can obtain the sets of singular values from the measurement tensor, which are used jointly to evaluate the controller performance of the overall switched systems significantly. We develop the covariance tensor-based performance assessment method for the multivariate switched control systems with characteristic information being mined from the measurement tensor and derive the calculation approach base on the sets of singular values from the measurement tensor. It is shown that by applying higher-order singular value decomposition for measured outputs tensor data, the performance assessment results can exactly reflect the controller performance under the overall dynamical process. Finally, two cases study of the numerical simulation examples and a typical industrial process system well demonstrate the effectiveness of the proposed method.     

Double enhancement learning for explicit internal representations: unifying self-enhancement and information enhancement to incorporate information on input variables

In this paper, we propose a new type of information-theoretic method called “double enhancement learning,” in which two types of enhancement, namely, self-enhancement and information enhancement, are unified. Self-enhancement learning has been developed to create targets spontaneously within a network, and its performance has proven to be comparable with that of conventional competitive learning and self-organizing maps. To improve the performance of the self-enhancement learning, we try to include information on input variables in the framework of self-enhancement learning. The information on input variables is computed by information enhancement in which a specific input variable is used to enhance competitive unit outputs. This information is again used to train a network with the self-enhancement learning. We applied the method to three problems, namely, an artificial data, a student survey and the voting attitude problem. In all three problems, quantization errors were significantly decreased with the double enhancement learning. The topographic errors were relatively higher, but the smallest number of topographic errors was also obtained by the double enhancement learning. In addition, we saw that U-matrices for all problems showed explicit boundaries reflecting the importance of input variables.     

Identification of marginal and joint CDFs using Bayesian method for RBDO

In RBDO, input uncertainty models such as marginal and joint cumulative distribution functions (CDFs) need to be used. However, only limited data exists in industry applications. Thus, identification of the input uncertainty model is challenging especially when input variables are correlated. Since input random variables, such as fatigue material properties, are correlated in many industrial problems, the joint CDF of correlated input variables needs to be correctly identified from given data. In this paper, a Bayesian method is proposed to identify the marginal and joint CDFs from given data where a copula, which only requires marginal CDFs and correlation parameters, is used to model the joint CDF of input variables. Using simulated data sets, performance of the Bayesian method is tested for different numbers of samples and is compared with the goodness-of-fit (GOF) test. Two examples are used to demonstrate how the Bayesian method is used to identify correct marginal CDFs and copula.     

Inhibition in multiclass classification

The role of inhibition is investigated in a multiclass support vector machine formalism inspired by the brain structure of insects. The so-called mushroom bodies have a set of output neurons, or classification functions, that compete with each other to encode a particular input. Strongly active output neurons depress or inhibit the remaining outputs without knowing which is correct or incorrect. Accordingly, we propose to use a classification function that embodies unselective inhibition and train it in the large margin classifier framework. Inhibition leads to more robust classifiers in the sense that they perform better on larger areas of appropriate hyperparameters when assessed with leave-one-out strategies. We also show that the classifier with inhibition is a tight bound to probabilistic exponential models and is Bayes consistent for 3-class problems. These properties make this approach useful for data sets with a limited number of labeled examples. For larger data sets, there is no significant comparative advantage to other multiclass SVM approaches.     

Disparity Estimation by Pooling Evidence From Energy Neurons

In this paper, we propose an algorithm for disparity estimation from disparity energy neurons that seeks to maintain simplicity and biological plausibility, while also being based upon a formulation that enables us to interpret the model outputs probabilistically. We use the Bayes factor from statistical hypothesis testing to show that, in contradiction to the implicit assumption of many previously proposed biologically plausible models, a larger response from a disparity energy neuron does not imply more evidence for the hypothesis that the input disparity is close to the preferred disparity of the neuron. However, we find that the normalized response can be interpreted as evidence, and that information from different orientation channels can be combined by pooling the normalized responses. Based on this insight, we propose an algorithm for disparity estimation constructed out of biologically plausible operations. Our experimental results on real stereograms show that the algorithm outperforms a previously proposed coarse-to-fine model. In addition, because its outputs can be interpreted probabilistically, the model also enables us to identify occluded pixels or pixels with incorrect disparity estimates.      

Learning with Uncertain Kernel Matrix Set

We study support vector machines (SVM) for which the kernel matrix is not specified exactly and it is only known to belong to a given uncertainty set. We consider uncertainties that arise from two sources: (i) data measurement uncertainty, which stems from the statistical errors of input samples; (ii) kernel combination uncertainty, which stems from the weight of individual kernel that needs to be optimized in multiple kernel learning (MKL) problem. Much work has been studied, such as uncertainty sets that allow the corresponding SVMs to be reformulated as semi-definite programs (SDPs), which is very computationally expensive however. Our focus in this paper is to identify uncertainty sets that allow the corresponding SVMs to be reformulated as second-order cone programs (SOCPs), since both the worst case complexity and practical computational effort required to solve SOCPs is at least an order of magnitude less than that needed to solve SDPs of comparable size. In the main part of the paper we propose four uncertainty sets that meet this criterion. Experimental results are presented to confirm the validity of these SOCP reformulations.     

Type-2 fuzzy multi-objective DEA model: An application to sustainable supplier evaluation

In the last 10 years, sustainable supply chain management (SSCM) has become one of the important topics in business and academe. Sustainable supplier performance evaluation and selection play a significant role in establishing an effective SSCM. One of the techniques that can be used for evaluating sustainable supplier performance is data envelopment analysis (DEA). The conventional DEA methods require accurate measurement of both input and output variables present in the problem. In practice, the observed values of the input and output data present in real-world problems are often imprecise. To cope with this situation, fuzzy DEA models were constructed for expressing relative fuzzy efficiencies of decision-making units (DMUs). However, fuzzy DEA is still limited to fuzzy input/output data while some inputs and outputs might be affected by various factors of uncertainty and information granularity, meaning that they could be better modeled in terms of fuzzy sets of type-2. In this paper, we develop a multi-objective DEA model in a setting of type-2 fuzzy modeling to evaluate and select the most appropriate sustainable suppliers. In the proposed model, both efficiency and effectiveness are considered to describe the integrated productivity of suppliers. In sequel, chance constrained programming, critical value-based reduction methods and equivalent transformations are considered to solve the problem. A detailed case study is employed to show the advantages of the proposed model in terms of measuring effectiveness, efficiency and productivity in an uncertain environment expressed at different confidence levels. At the same time, the results demonstrate that the model is capable of helping decision makers to balance economic, social, and environmental factors when selecting sustainable suppliers.     

Optimal uncertainty reduction for multi-disciplinary multi-output systems using sensitivity analysis

We present a sensitivity analysis based uncertainty reduction approach, called Multi-dIsciplinary Multi-Output Sensitivity Analysis (MIMOSA), for the analysis model of a multi-disciplinary engineering system decomposed into multiple subsystems with each subsystem analysis having multiple inputs with reducible uncertainty and multiple outputs. MIMOSA can determine: (1) the sensitivity of system and subsystem outputs to input uncertainties at both system and subsystem levels, (2) the sensitivity of the system outputs to the variation from subsystem outputs, and (3) the optimal “investment” required to reduce uncertainty in inputs in order to obtain a maximum reduction in output variations at both the system and subsystem levels. A numerical and an engineering example with two and three subsystems, respectively, have been used to demonstrate the applicability of the MIMOSA approach.     

A Convolutional Deep Self-Organizing Map Feature extraction for machine learning

In this work we propose a new Unsupervised Deep Self-Organizing Map (UDSOM) algorithm for feature extraction, quite similar to the existing multi-layer SOM architectures. The principal underlying idea of using SOMs is that if a neuron is wins n times, these n inputs that activated this neuron are similar. The basic principle consists of an alternation of phases of splitting and abstraction of regions, based on a non-linear projection of high-dimensional data over a small space using Kohonen maps following a deep architecture. The proposed architecture consists of a splitting process, layers of alternating self-organizing, a rectification function RELU and an abstraction layer (convolution-pooling). The self-organizing layer is composed of a few SOMs with each map focusing on modelling a local sub-region. The most winning neurons of each SOM are then organized in a second sampling layer to generate a new 2D map. In parallel to this transmission of the winning neurons, an abstraction of the data space is obtained after the convolution-pooling module. The ReLU is then applied. This treatment is applied more than once, changing the size of the splitting window and the displacement step on the reconstructed input image each time. In this way, local information is gathered to form more global information in the upper layers by applying each time a convolution filter of the level. The architecture of the Unsupervised Deep Self-Organizing Map is unique and retains the same principle of deep learning algorithms. This architecture can be very interesting in a Big Data environment for machine learning tasks. Experiments have been conducted to discuss how the proposed architecture shows this performance.

     

Granular neural networks: A study of optimizing allocation of information granularity in input space

In this paper, we develop a granular input space for neural networks, especially for multilayer perceptrons (MLPs). Unlike conventional neural networks, a neural network with granular input is an augmented study on a basis of a well learned numeric neural network. We explore an efficient way of forming granular input variables so that the corresponding granular outputs of the neural network achieve the highest values of the criteria of specificity (and support). When we augment neural networks through distributing information granularities across input variables, the output of a network has different levels of sensitivity on different input variables. Capturing the relationship between input variables and output result becomes of a great help for mining knowledge from the data. And in this way, important features of the data can be easily found. As an essential design asset, information granules are considered in this construct. The quantification of information granules is viewed as levels of granularity which is given by the expert. The detailed optimization procedure of allocation of information granularity is realized by an improved partheno genetic algorithm (IPGA). The proposed algorithm is testified effective by some numeric studies completed for synthetic data and data coming from the machine learning and StatLib repositories. Moreover, the experimental studies offer a deep insight into the specificity of input features.