Microcode optimization with neural networks

Microcode optimization is an NP-complete combinatorial optimization problem. This paper proposes a new method based on the Hopfield neural network for optimizing the wordwidth in the control memory of a microprogrammed digital computer. We present two methodologies, viz., the maximum clique approach, and a cost function based method to minimize an objective function. The maximum clique approach albeit being near O(1) in complexity, is limited in its use for small problem sizes, since it only partitions the data based on the compatibility between the microoperations, and does not minimize the cost function. We thereby use this approach to condition the data initially (to form compatibility classes), and then use the proposed second method to optimize the cost function. The latter method is then able to discover better solutions than other schemes for the benchmark data set.     

Using artificial neural networks to enhance CART

Accuracy is a critical factor in predictive modeling. A predictive model such as a decision tree must be accurate to draw conclusions about the system being modeled. This research aims at analyzing and improving the performance of classification and regression trees (CART), a decision tree algorithm, by evaluating and deriving a new methodology based on the performance of real-world data sets that were studied. This paper introduces a new approach to tree induction to improve the efficiency of the CART algorithm by combining the existing functionality of CART with the addition of artificial neural networks (ANNs). Trained ANNs are utilized by the tree induction algorithm by generating new, synthetic data, which have been shown to improve the overall accuracy of the decision tree model when actual training samples are limited. In this paper, traditional decision trees developed by the standard CART methodology are compared with the enhanced decision trees that utilize the ANN’s synthetic data generation, or CART+. This research demonstrates the improved accuracies that can be obtained with CART+, which can ultimately improve the knowledge that can be extracted by researchers about a system being modeled.     

Using neural networks to learn hand-eye co-ordination

The work presented in this paper shows how the association of proprioceptive and exteroceptive stimuli can enable a Kohonen neural network, controlling a robot arm, to learn hand-eye co-ordination so that the arm can reach for and track a visually presented target. The approach presented in this work assumes no a priorimodel of arm kinematics or of the imaging characteristics of the cameras. No explicit representation, such as homogeneous transformations, is used for the specification of robot pose, and camera calibration and triangulation are done implicitly as the system adapts and learns its hand-eye co-ordination by experience. This research is validated on physical devices and not by simulation.     

Design of automotive joints: Using neural networks and optimization to translate performance requirements to physical design parameters

In the preliminary design stage of a car, targets are first set for the performance characteristics of the overall body and its components using optimization and engineering judgment. Then designers try to design components that meet these targets using empirical, trial-and-error procedures. This process usually yields poor results because it is difficult to find a feasible design that satisfies the targets by trial-and-error (a feasible design is one that satisfies packaging and manufacturing constraints). To improve this process, we need tools to link the performance targets with the physical design parameters that define the geometry of the components of a car body. A methodology is presented for developing two tools for design guidance of joints in car bodies. These tools translate the design parameters that define the geometry of a joint into performance characteristics of that joint and vice versa. The first tool, called translator A, rapidly predicts the performance characteristics of a given joint (at a fraction of a second). The second tool, called translator B, finds a joint design that meets or exceeds given performance targets and satisfies packaging and manufacturing constraints. The methodology is demonstrated on a joint of an actual car.     

An optimization approach to design of cellular neural networks

In this paper, we consider the problem of realizing associative memories via cellular neural networks (CNNs). After introducing qualitative properties of the CNN model, we formulate the synthesis of CNNs that can store given binary vectors with improved performance as a constrained optimization problem. Next, we observe that this problem's constraints can be transformed into simple inequalities involving linear matrix inequalities. Finally, we reformulate the synthesis problem as a generalized eigenvalue problem, which can be efficiently solved by recently developed interior point methods. The validity of the proposed approach is illustrated by a design example.     

Design optimization with back-propagation neural networks

A methodology with back-propagation neural network models is developed to explore the artificial neural nets (ANN) technology in the new application territory of design optimization. This design methodology could go beyond the Hopfield network model, Hopfield and Tank (1985), for combinatorial optimization problems In this approach, pattern classification with back-propagation network, the most demonstrated power of neural networks applications, is utilized to identify the boundaries of the feasible and the infeasible design regions. These boundaries enclose the multi-dimensional space within which designs satisfy all design criteria. A feedforward network is then incorporated to perform function approximation of the design objective function. This approximation is performed by training the feedforward network with objective functions evaluated at selected design sets in the feasible design regions. Additional optimum design sets in the classified feasible regions are calculated and included in the successive training sets to improve the function mapping. Iteration is continued until convergent criteria are satisfied. This paper demonstrates that the artificial neural nets technology provides a global perspective of the entire design space with good and near optimal solutions. ANN can indeed be a potential technology for design optimization.     

Modeling and qualitative analysis of continuous-time neural networks under pure structural variations

A qualitative analysis is developed for continuous-time neural networks subjected to random pure structural variations. Simple algebraic conditions are established for both structural exponential stability of x = 0 of the neural network and for estimates of its domain of attraction. Bounds on motions of the neural network in a forced regime are provided. They do not require any information about its actual structure, which can be completely unknown and may vary unpredictably.     

Multifocal electroretinogram diagnosis of glaucoma applying neural networks and structural pattern analysis

Glaucoma is a chronic ophthalmological disease that affects 5% of the 40-60-year-old population and can lead to irreversible blindness. The multifocal electroretinogram (mfERG) is a recently developed diagnostic technique that provides objective spatial data on the visual pathway and may be of potential benefit in early diagnosis of glaucoma. This paper analyses 13 morphological characteristics that define mfERG recordings and classifies them using a radial basis function network trained with the Extreme Learning Machine algorithm. When used to detect glaucomatous sectors, the method proposed produces sensitivity and specificity values of over 0.8.     

Structure optimization of neural networks for evolutionary design optimization

We study the use of neural networks as approximate models for the fitness evaluation in evolutionary design optimization. To improve the quality of the neural network models, structure optimization of these networks is performed with respect to two different criteria: One is the commonly used approximation error with respect to all available data, and the other is the ability of the networks to learn different problems of a common class of problems fast and with high accuracy. Simulation results from turbine blade optimizations using the structurally optimized neural network models are presented to show that the performance of the models can be improved significantly through structure optimization.We would like to thank the BMBF, grant LOKI, number 01 IB 001 C, for their financial support of our research.     

Deep neural networks and mixed integer linear optimization

Deep Neural Networks (DNNs) are very popular these days, and are the subject of a very intense investigation. A DNN is made up of layers of internal units (or neurons), each of which computes an affine combination of the output of the units in the previous layer, applies a nonlinear operator, and outputs the corresponding value (also known as activation). A commonly-used nonlinear operator is the so-called rectified linear unit (ReLU), whose output is just the maximum between its input value and zero. In this (and other similar cases like max pooling, where the max operation involves more than one input value), for fixed parameters one can model the DNN as a 0-1 Mixed Integer Linear Program (0-1 MILP) where the continuous variables correspond to the output values of each unit, and a binary variable is associated with each ReLU to model its yes/no nature. In this paper we discuss the peculiarity of this kind of 0-1 MILP models, and describe an effective bound-tightening technique intended to ease its solution. We also present possible applications of the 0-1 MILP model arising in feature visualization and in the construction of adversarial examples. Computational results are reported, aimed at investigating (on small DNNs) the computational performance of a state-of-the-art MILP solver when applied to a known test case, namely, hand-written digit recognition.     

Performance and fault-tolerance of neural networks for optimization

The fault-tolerance characteristics of time-continuous, recurrent artificial neural networks (ANNs) that can be used to solve optimization problems are investigated. The performance of these networks is illustrated by using well-known model problems like the traveling salesman problem and the assignment problem. The ANNs are then subjected to up to 13 simultaneous stuck-at-1 or stuck-at-0 faults for network sizes of up to 900 neurons. The effect of these faults on the performance is demonstrated, and the cause for the observed fault-tolerance is discussed. An application is presented in which a network performs a critical task for a real-time distributed processing system by generating new task allocations during the reconfiguration of the system. The performance degradation of the ANN under the presence of faults is investigated by large-scale simulations and the potential benefits of delegating a critical task to a fault-tolerant network are discussed.     

Polymer property prediction and optimization using neural networks

Prediction and optimization of polymer properties is a complex and highly nonlinear problem with no easy method to predict polymer properties directly and accurately. The problem is especially complicated with high molecular weight polymers such as engineering plastics which have the greatest use in industry. The effect of modifying a monomer (polymer repeat unit) on polymerization and the resulting polymer properties is not easy to investigate experimentally given the large number of possible changes. This severely curtails the design of new polymers with specific end-use properties. In this paper, we show how properties of modified monomers can be predicted using neural networks. We utilize a database of polymer properties and employ a variety of networks ranging from backpropagation networks to unsupervised self-associating maps. We select particular networks that accurately predict specific polymer properties. These networks are classified into groups that range from those that provide quick training to those that provide excellent generalization. We also show how the available polymer database can be used to accurately predict and optimize polymer properties.     

Calibrating artificial neural networks by global optimization

Artificial neural networks (ANNs) are used extensively to model unknown or unspecified functional relationships between the input and output of a “black box” system. In order to apply the generic ANN concept to actual system model fitting problems, a key requirement is the training of the chosen (postulated) ANN structure. Such training serves to select the ANN parameters in order to minimize the discrepancy between modeled system output and the training set of observations. We consider the parameterization of ANNs as a potentially multi-modal optimization problem, and then introduce a corresponding global optimization (GO) framework. The practical viability of the GO based ANN training approach is illustrated by finding close numerical approximations of one-dimensional, yet visibly challenging functions. For this purpose, we have implemented a flexible ANN framework and an easily expandable set of test functions in the technical computing system Mathematica. The MathOptimizer Professional global-local optimization software has been used to solve the induced (multi-dimensional) ANN calibration problems.     

Computer-assisted manufacturing process optimization with neural networks

Today's manufacturing methods are caught between the growing need for quality, high process safety, minimal manufacturing costs, and short manufacturing times. In order to meet these demands, process setting parameters have to be chosen in the best possible way, according to demand on quality. For such optimization it is necessary to represent the processes in a model. Due to the enormous complexity of many processes and the high number of influencing parameters, however, conventional approaches to modelling and optimization are no longer sufficient. In this article it is shown how, by means of applying neural networks for process modelling, even these highly complex interdependencies can be learned. That way both process and quality parameters can be assessed before or during processing. By connecting them with corresponding cost models, it is possible to optimize processes with the help of evolutionary algorithms. Using examples of different manufacturing processes, the possi bilities for process modelling and optimization with neural networks and evolutionary algorithms are demonstrated.     

Gradientless shape optimization using artificial neural networks

In this paper a new zero order method of structural shape optimization, in which material shrinks or grows perpendicular to the design boundary, has been proposed in order to satisfy fully stressed design criteria. To avoid mesh distortion that results in undesirable shape, design element concept and for nodal movement and convergence checking, fuzzy set theory have been used. To accelerate the convergence, artificial neural networks are employed. The proposed approach, named as GSN technique, has been incorporated in a FORTRAN software GSOANN. Using this software shape optimization of four structures are carried out. It is demonstrated that proposed technique overcomes most of the shortcomings of mundane zero order methods.     

Using JessTab to integrate Protege and Jess

Integration with external systems, such as problem solvers, is becoming increasingly important for ontology development and knowledge-modeling tools. The author's JessTab extension lets you write Jess programs that manage Protege ontologies and knowledge bases. Protege is a popular, modular ontology development and knowledge acquisition tool.     

Identification of structural systems by neural networks

A method based on the use of neural networks is developed for the identification of systems encountered in the field of structural dynamics. The methodology is applied to the identification of linear and nonlinear dynamic systems such as the damped Duffing oscillator and the Van der Pol equation. The “generalization” ability of the neural networks is used to predict the response of the identified systems under deterministic and stochastic excitations. It is shown that neural networks provide high fidelity models of unknown structural dynamic systems, which are used in applications such as structural control, health monitoring of structures, earthquake engineering, etc.     

Using artificial neural networks to detect unknown computer worms

Detecting computer worms is a highly challenging task. We present a new approach that uses artificial neural networks (ANN) to detect the presence of computer worms based on measurements of computer behavior. We compare ANN to three other classification methods and show the advantages of ANN for detection of known worms. We then proceed to evaluate ANN’s ability to detect the presence of an unknown worm. As the measurement of a large number of system features may require significant computational resources, we evaluate three feature selection techniques. We show that, using only five features, one can detect an unknown worm with an average accuracy of 90%. We use a causal index analysis of our trained ANN to identify rules that explain the relationships between the selected features and the identity of each worm. Finally, we discuss the possible application of our approach to host-based intrusion detection systems.     

Using neural networks in reliability prediction

It is shown that neural network reliability growth models have a significant advantage over analytic models in that they require only failure history as input and not assumptions about either the development environment or external parameters. Using the failure history, the neural-network model automatically develops its own internal model of the failure process and predicts future failures. Because it adjusts model complexity to match the complexity of the failure history, it can be more accurate than some commonly used analytic models. Results with actual testing and debugging data which suggest that neural-network models are better at endpoint predictions than analytic models are presented     

深度神经网络模型构建及优化策略

针对深度学习构建网络模型以及确定模型参数的问题,在分析神经网络基本结构和线性模型局限性的基础上,研究了深度神经网络设计的关键因素和优化策略。结合手写数字识别问题,对优化策略、动态衰减学习率、隐藏层节点数、隐藏层数等情形下的识别正确率进行了实验。结果表明,不同神经网络模型对最终正确率有质的影响,相同优化策略在不同参数取值时对最终正确率有很大影响,并进一步探究了具体选取优化策略和参数的方法。