A comprehensive survey on optimizing deep learning models by metaheuristics

Deep neural networks (DNNs), which are extensions of artificial neural networks, can learn higher levels of feature hierarchy established by lower level features by transforming the raw feature space to another complex feature space. Although deep networks are successful in a wide range of problems in different fields, there are some issues affecting their overall performance such as selecting appropriate values for model parameters, deciding the optimal architecture and feature representation and determining optimal weight and bias values. Recently, metaheuristic algorithms have been proposed to automate these tasks. This survey gives brief information about common basic DNN architectures including convolutional neural networks, unsupervised pre-trained models, recurrent neural networks and recursive neural networks. We formulate the optimization problems in DNN design such as architecture optimization, hyper-parameter optimization, training and feature representation level optimization. The encoding schemes used in metaheuristics to represent the network architectures are categorized. The evolutionary and selection operators, and also speed-up methods are summarized, and the main approaches to validate the results of networks designed by metaheuristics are provided. Moreover, we group the studies on the metaheuristics for deep neural networks based on the problem type considered and present the datasets mostly used in the studies for the readers. We discuss about the pros and cons of utilizing metaheuristics in deep learning field and give some future directions for connecting the metaheuristics and deep learning. To the best of our knowledge, this is the most comprehensive survey about metaheuristics used in deep learning field.

     

State-dependent parameter modelling and identification of stochastic non-linear sampled-data systems

State-dependent parameter representations of stochastic non-linear sampled-data systems are studied. Velocity-based linearization is used to construct state-dependent parameter models which have a nominally linear structure but whose parameters can be characterized as functions of past outputs and inputs. For stochastic systems state-dependent parameter ARMAX (quasi-ARMAX) representations are obtained. The models are identified from input–output data using feedforward neural networks to represent the model parameters as functions of past inputs and outputs. Simulated examples are presented to illustrate the usefulness of the proposed approach for the modelling and identification of non-linear stochastic sampled-data systems.     

AN OPTIMIZATION APPROACH TO AUTOMATIC GENERIC DOCUMENT SUMMARIZATION

In this paper, we have presented an optimization approach to document summarization. The potential of optimization based document summarization models has not been well explored to date. This is partially the difficulty to formulate the criteria used for objective assessment. We modeled document summarization as the linear and nonlinear optimization problems. These models generally attempt simultaneously to balance coverage and diversity in the summary. To solve the optimization problem we developed a novel particle swarm optimization (PSO) algorithm. Experiments showed our linear and nonlinear models produce very competitive results, which significantly outperform the NIST baselines in both years. More important, although linear and nonlinear models are comparable to the top three systems S24, S15, and S12 in the DUC2006, they are even superior to the best participating system in the DUC2005.     

An optimization approach to design of cellular neural networks

In this paper, we consider the problem of realizing associative memories via cellular neural networks (CNNs). After introducing qualitative properties of the CNN model, we formulate the synthesis of CNNs that can store given binary vectors with improved performance as a constrained optimization problem. Next, we observe that this problem's constraints can be transformed into simple inequalities involving linear matrix inequalities. Finally, we reformulate the synthesis problem as a generalized eigenvalue problem, which can be efficiently solved by recently developed interior point methods. The validity of the proposed approach is illustrated by a design example.     

Non-regular feedback linearization of nonlinear systems via a normal form algorithm

In this paper, the problem of non-regular static state feedback linearization of affine nonlinear systems is considered. First of all, a new canonical form for non-regular feedback linear systems is proposed. Using this form, a recursive algorithm is presented, which yields a condition for single input linearization. Then the left semi-tensor product of matrices is introduced and several new properties are developed. Using the recursive framework and new matrix product, a formula is presented for normal form algorithm. Based on it, a set of conditions for single-input (approximate) linearizability is presented.     

A stochastic production planning problem with nonlinear cost

Most production planning models are deterministic and often assume a linear relation between production volume and production cost. In this paper, we investigate a production planning problem in a steel production process considering the energy consumption cost which is a nonlinear function of the production quantity. Due to the uncertain environment, the production demands are stochastic. Taking a scenario-based approach to express the stochastic demands according to the knowledge of planners on the demand distributions, we formulate the stochastic production planning problem as a mixed integer nonlinear programming (MINLP) model.Approximated with the piecewise linear functions, the MINLP model is transformed into a mixed integer linear programming model. The approximation error can be improved by adjusting the linearization ranges repeatedly. Based on the piecewise linearization, a stepwise Lagrangian relaxation (SLR) heuristic for the problem is proposed where variable splitting is introduced during Lagrangian relaxation (LR). After decomposition, one subproblem is solved by linear programming and the other is solved by an effective polynomial time algorithm. The SLR heuristic is tested on a large set of problem instances and the results show that the algorithm generates solutions very close to optimums in an acceptable time. The impact of demand uncertainty on the solution is studied by a computational discussion on scenario generation.     

Optimization of resource allocation for underlay device-to-device communications in cellular networks

Underlay device-to-device (D2D) communication in cellular networks has been considered as a promising technique that can improve the spectral efficiency of cellular systems and meet the growing demand for wireless local services. In underlay D2D, it is of primary importance to manage the mutual interference between cellular links and D2D links through effective resource allocation. While most of previous works on D2D resource allocation are developed based on the knowledge of the channel state information (CSI) on the interference channels as well as the desired channels, it is hard to obtain full CSI in practice. Accordingly, we consider D2D resource allocation schemes based on distance between nodes. In particular, we formulate two optimization problems for D2D resource allocation using the outage probability computed based on the distance information as cost functions. One is a linear sum assignment problem (LSAP) and the other is a linear bottleneck assignment problem (LBAP). By applying the graph theory, we provide efficient algorithms for solving the optimization problems. Numerical results are provided to show the effectiveness of the proposed optimization as compared to previously proposed distance-based resource allocation algorithms.     

Connectionist computations of intuitionistic reasoning

The construction of computational models with provision for effective learning and added reasoning is a fundamental problem in computer science. In this paper, we present a new computational model for integrated reasoning and learning that combines intuitionistic reasoning and neural networks. We use ensembles of neural networks to represent intuitionistic theories, and show that for each intuitionistic theory and intuitionistic modal theory there exists a corresponding neural network ensemble that computes a fixed-point semantics of the theory. This provides a massively parallel model for intuitionistic reasoning. In our model, the neural networks can be trained from examples to adapt to new situations using standard neural learning algorithms, thus providing a unifying foundation for intuitionistic reasoning, knowledge representation, and learning.     

On the construction and training of reformulated radial basis function neural networks

Presents a systematic approach for constructing reformulated radial basis function (RBF) neural networks, which was developed to facilitate their training by supervised learning algorithms based on gradient descent. This approach reduces the construction of radial basis function models to the selection of admissible generator functions. The selection of generator functions relies on the concept of the blind spot, which is introduced in the paper. The paper also introduces a new family of reformulated radial basis function neural networks, which are referred to as cosine radial basis functions. Cosine radial basis functions are constructed by linear generator functions of a special form and their use as similarity measures in radial basis function models is justified by their geometric interpretation. A set of experiments on a variety of datasets indicate that cosine radial basis functions outperform considerably conventional radial basis function neural networks with Gaussian radial basis functions. Cosine radial basis functions are also strong competitors to existing reformulated radial basis function models trained by gradient descent and feedforward neural networks with sigmoid hidden units.     

A model for a reliable single allocation hub network design under massive disruption

Hub facilities may fail to operate in networks because of accidental failures such as natural disasters. In this paper, a quadratic model was presented for a reliable single allocation hub network under massive random failure of hub facilities which more than one hub may be disrupted in a route. It determines the location of the hub facilities and the primal allocation of non-hub nodes. It also determines the backup allocation in case of failure of the primal hub. First, a new lexicographic form of a bi-objective quadratic model is presented where the first objective maximizes served demands or equivalently, minimizes lost flows and the second objective minimizes total cost under a to massive disruption in the network. Then, by adding a structure-based constraint, the model is transformed to a single objective one. A linearization technique reported in the literature is applied on the quadratic model to convert it into classic linear zero–one mixed integer model while enhancing it by finding tighter bounds. The tight bounds’ technique is compared with other techniques in terms of computational time and its better performance was approved in some problem instances. Finally, due to the NP-hardness of the problem, an iterated local search algorithm was developed to solve large sized instances in a reasonable computational time and the computational results confirm the efficiency of the proposed heuristic, ILS can solve all CAB and IAD data set instances in less than 15 and 24 seconds, respectively. Moreover, the proposed model was compared with the classical hub network using a network performance measure, and the results show the increased efficiency of the model.     

Design of information granule-oriented RBF neural networks and its application to power supply for high-field magnet

To realize effective modeling and secure accurate prediction abilities of models for power supply for high-field magnet (PSHFM), we develop a comprehensive design methodology of information granule-oriented radial basis function (RBF) neural networks. The proposed network comes with a collection of radial basis functions, which are structurally as well as parametrically optimized with the aid of information granulation and genetic algorithm. The structure of the information granule-oriented RBF neural networks invokes two types of clustering methods such as K-Means and fuzzy C-Means (FCM). The taxonomy of the resulting information granules relates to the format of the activation functions of the receptive fields used in RBF neural networks. The optimization of the network deals with a number of essential parameters as well as the underlying learning mechanisms (e.g., the width of the Gaussian function, the numbers of nodes in the hidden layer, and a fuzzification coefficient used in the FCM method). During the identification process, we are guided by a weighted objective function (performance index) in which a weight factor is introduced to achieve a sound balance between approximation and generalization capabilities of the resulting model. The proposed model is applied to modeling power supply for high-field magnet where the model is developed in the presence of a limited dataset (where the small size of the data is implied by high costs of acquiring data) as well as strong nonlinear characteristics of the underlying phenomenon. The obtained experimental results show that the proposed network exhibits high accuracy and generalization capabilities.     

多媒体信号处理的数学理论前沿进展

深度学习模型广泛应用于多媒体信号处理领域,通过引入非线性能够极大地提升性能,但是其黑箱结构无法解析地给出最优点和优化条件。因此如何利用传统信号处理理论,基于变换/基映射模型逼近深度学习模型,解析优化问题,成为当前研究的前沿问题。本文从信号处理的基础理论出发,分析了当前针对高维非线性非规则结构方法的数学模型和理论边界,主要包括：结构化稀疏表示模型、基于框架理论的深度网络模型、多层卷积稀疏编码模型以及图信号处理理论。详细描述了基于组稀疏性和层次化稀疏性的表示模型和优化方法,分析基于半离散框架和卷积稀疏编码构建深度/多层网络模型,进一步在非欧氏空间上扩展形成图信号处理模型,并对国内外关于记忆网络的研究进展进行了比较。最后,展望了多媒体信号处理的理论模型发展,认为图信号处理通过解析谱图模型的数学性质,解释其中的关联性,为建立广义的大规模非规则多媒体信号处理模型提供理论基础,是未来研究的重要领域之一。     

Modeling and navigation of social information networks in metric spaces

We are living in a world of various kinds of social information networks with small-world and scale-free characteristics. It is still an intriguing problem for researchers to explain how and why so many obviously different networks emerge and share common intrinsic characteristics such as short diameter, higher cluster and power-law degree distribution. Most previous works studied the topology formation and information navigation of complex networks in separated models. In this paper, we propose a metric based range intersection model to explore the topology evolution and information navigation in a synthetic way. We model the network as a set of nodes in a distance metric space where each node has an ID and a range of neighbor information around its ID in the metric space. The range of a node can be seen as the local knowledge or information that the node has around its position in the metric space. The topology is formed by setting up a link between two nodes that have intersected ranges. Information navigation over the network is modeled as a greedy routing process using neighbor links and the distance metric. Different from previous models, we do not assume that nodes join the network one by one and set up link according to the degree distribution of existing nodes or distances between nodes. Range of node is the key factor determining the topology and navigation properties of a network. Moreover, as the ranges of nodes grow, the network evolves from a set of totally isolated nodes to a connected network. Thus, we can easily model the network evolutions in terms of the network size and the individual node information range using the range intersection model. A set of experiments shows that networks constructed using the range intersection model have the scale-free degree distribution, high cluster, short diameter, and high navigability properties that are owned by the real networks.     

Coordinated multimedia object replacement in transcoding proxies

Finding replacement candidates for accommodating a new object is an important research issue in web caching. Due to the new emerging factors in the transcoding proxy and the aggregate effect of caching multiple versions of the same multimedia object, this problem becomes more important and complex as audio and video applications have proliferated over the Internet, especially in the environment of mobile computing systems. This paper addresses coordinated cache replacement in transcoding proxies. First, we propose an original model which determines cache replacement candidates on all candidate nodes in a coordinated fashion with the objective of minimizing the total cost loss for linear topology. We formulate this problem as an optimization problem and present a low-cost optimal solution for deciding cache replacement candidates. Second, we extend this problem to solve the same problem for tree networks. Finally, we conduct extensive simulations to evaluate the performance of our solutions by comparing with existing models.     

Multi-objective optimization of a stacked neural network using an evolutionary hyper-heuristic

The present paper deals with the development and optimization of a stacked neural network (SNN) through an evolutionary hyper-heuristic, called NSGA-II-QNSNN. The proposed hyper-heuristic is based on the NSGA-II (Non-dominated Sorting Genetic Algorithm - II) multi-objective optimization evolutionary algorithm which incorporates the Quasi-Newton (QN) optimization algorithm. QN is used for training each neural network from the stack. The final global optimal solution provided by NSGA-II-QNSNN algorithm is a Pareto optimal front. It represents all the equally good compromises that can be made between the structural complexity of the stacked neural network and its modelling performance. The set of decision variables, which led to obtaining the set of points in the Pareto optimal front, represents the optimum values for the parameters of the stacked neural network: the number of networks in the stack, the weights for every output of the composing networks, and the number of hidden neurons in each individual neural network. Each stacked neural network determined through the optimization process was trained and tested by applying it to a real world problem: the modelling of the polyacrylamide-based multicomponent hydrogels synthesis. The neural modelling established the influence of the reaction conditions on the reaction yield and the swelling degree. The results provided by NSGA-II-QNSNN were superior, not only in terms of performance, but also in terms of structural complexity, to those obtained in our previous works, where individual or aggregated neural networks were used, but the stacks were developed manually, based on successive trials.     

IG-based genetically optimized fuzzy polynomial neural networks with fuzzy set-based polynomial neurons

In this study, we introduce and investigate a new topology of fuzzy-neural networks—fuzzy polynomial neural networks (FPNN) that is based on a genetically optimized multiplayer perceptron with fuzzy set-based polynomial neurons (FSPNs). We also develop a comprehensive design methodology involving mechanisms of genetic optimization and information granulation. In the sequel, the genetically optimized FPNN (gFPNN) is formed with the use of fuzzy set-based polynomial neurons (FSPNs) composed of fuzzy set-based rules through the process of information granulation. This granulation is realized with the aid of the C-means clustering (C-Means). The design procedure applied in the construction of each layer of an FPNN deals with its structural optimization involving the selection of the most suitable nodes (or FSPNs) with specific local characteristics (such as the number of input variable, the order of the polynomial, the number of membership functions, and a collection of specific subset of input variables) and address main aspects of parametric optimization. Along this line, two general optimization mechanisms are explored. The structural optimization is realized via genetic algorithms (GAs) and HCM method whereas in case of the parametric optimization we proceed with a standard least square estimation (learning). Through the consecutive process of structural and parametric optimization, a flexible neural network is generated in a dynamic fashion. The performance of the designed networks is quantified through experimentation where we use two modeling benchmarks already commonly utilized within the area of fuzzy or neurofuzzy modeling.     

Identification of the rainfall–runoff relationship in urban drainage networks

The calibration of conceptual models for the design of urban drainage networks is an important and well-known problem in hydraulic engineering. In this paper the problem is analysed and the use of black-box identification methods is proposed and applied to experimental data. Both linear (ARX and state space) and nonlinear (polynomial and neural NARX) models are considered and their performance in the simulation and prediction of the network flow from rainfall measurements is evaluated.     

Linearization and state estimation of unknown discrete-timenonlinear dynamic systems using recurrent neurofuzzy networks

Model-based methods for the state estimation and control of linear systems have been well developed and widely applied. In practice, the underlying systems are often unknown and nonlinear. Therefore, data based model identification and associated linearization techniques are very important. Local linearization and feedback linearization have drawn considerable attention in recent years. In this paper, linearization techniques using neural networks are reviewed, together with theoretical difficulties associated with the application of feedback linearization. A recurrent neurofuzzy network with an analysis of variance (ANOVA) decomposition structure and its learning algorithm are proposed for linearizing unknown discrete-time nonlinear dynamic systems. It can be viewed as a method for approximate feedback linearization, as such it enlarges the class of nonlinear systems that can be feedback linearized using neural networks. Applications of this new method to state estimation are investigated with realistic simulation examples, which shows that the new method has useful practical properties such as model parametric parsimony and learning convergence, and is effective in dealing with complex unknown nonlinear systems.     

Network reliability optimization problem of interconnection network under node-edge failure model

The network reliability optimization problem for an interconnection network is to maximize the network reliability subjected to some constraints such as the total cost of the network. Even though, the problem is NP-Hard, many researchers have solved this problem in different ways but with a common assumption that nodes are perfect. But, this assumption is quite unrealistic in nature. In this paper, a new method based on artificial neural network is proposed to solve the network reliability optimization problem considering both the nodes and links of the interconnection networks to be imperfect. The problem is mapped onto an artificial neural network by constructing an energy function whose minimization process drives the neural network into one of its stable states. This stable state corresponds to a solution for the network reliability problem. Some existing methods are studied and compared with proposed method in evaluating the network reliability of some fully connected networks. The comparison reports the proposed method to be better than its counterparts in maximizing the network reliability. The proposed method is used to maximize the reliability of few fully connected networks subjected to some predefined total cost, where the node as well as the links of the networks may fail. Further, the behaviors of the cost as well as the time on the network reliability are discussed.     

Representing diverse mathematical problems using neural networks in hybrid intelligent systems

In recent years, artificial neural networks have attracted considerable attention as candidates for novel computational systems. Computer scientists and engineers are developing neural networks as representational and computational models for problem solving: neural networks are expected to produce new solutions or alternatives to existing models. This paper demonstrates the flexibility of neural networks for modeling and solving diverse mathematical problems including Taylor series expansion, Weierstrass's first approximation theorem, linear programming with single and multiple objectives, and fuzzy mathematical programming. Neural network representations of such mathematical problems may make it possible to overcome existing limitations, to find new solutions or alternatives to existing models, and to achieve synergistic effects through hybridization.