Non-linear variable selection for artificial neural networks using partial mutual information

Artificial neural networks (ANNs) have been widely used to model environmental processes. The ability of ANN models to accurately represent the complex, non-linear behaviour of relatively poorly understood processes makes them highly suited to this task. However, the selection of an appropriate set of input variables during ANN development is important for obtaining high-quality models. This can be a difficult task when considering that many input variable selection (IVS) techniques fail to perform adequately due to an underlying assumption of linearity, or due to redundancy within the available data.This paper focuses on a recently proposed IVS algorithm, based on estimation of partial mutual information (PMI), which can overcome both of these issues and is considered highly suited to the development of ANN models. In particular, this paper addresses the computational efficiency and accuracy of the algorithm via the formulation and evaluation of alternative techniques for determining the significance of PMI values estimated during selection. Furthermore, this paper presents a rigorous assessment of the PMI-based algorithm and clearly demonstrates the superior performance of this non-linear IVS technique in comparison to linear correlation-based techniques.     

Protocol for developing ANN models and its application to the assessment of the quality of the ANN model development process in drinking water quality modelling

The application of Artificial Neural Networks (ANNs) in the field of environmental and water resources modelling has become increasingly popular since early 1990s. Despite the recognition of the need for a consistent approach to the development of ANN models and the importance of providing adequate details of the model development process, there is no systematic protocol for the development and documentation of ANN models. In order to address this shortcoming, such a protocol is introduced in this paper. In addition, the protocol is used to critically review the quality of the ANN model development and reporting processes employed in 81 journal papers since 2000 in which ANNs have been used for drinking water quality modelling. The results show that model architecture selection is the best implemented step, while greater focus should be given to input selection considering input independence and model validation considering replicative and structural validity.     

Methods used for the development of neural networks for the prediction of water resource variables in river systems: Current status and future directions

Over the past 15 years, artificial neural networks (ANNs) have been used increasingly for prediction and forecasting in water resources and environmental engineering. However, despite this high level of research activity, methods for developing ANN models are not yet well established. In this paper, the steps in the development of ANN models are outlined and taxonomies of approaches are introduced for each of these steps. In order to obtain a snapshot of current practice, ANN development methods are assessed based on these taxonomies for 210 journal papers that were published from 1999 to 2007 and focus on the prediction of water resource variables in river systems. The results obtained indicate that the vast majority of studies focus on flow prediction, with very few applications to water quality. Methods used for determining model inputs, appropriate data subsets and the best model structure are generally obtained in an ad-hoc fashion and require further attention. Although multilayer perceptrons are still the most popular model architecture, other model architectures are also used extensively. In relation to model calibration, gradient based methods are used almost exclusively. In conclusion, despite a significant amount of research activity on the use of ANNs for prediction and forecasting of water resources variables in river systems, little of this is focused on methodological issues. Consequently, there is still a need for the development of robust ANN model development approaches.     

Improving partial mutual information-based input variable selection by consideration of boundary issues associated with bandwidth estimation

Input variable selection (IVS) is vital in the development of data-driven models. Among different IVS methods, partial mutual information (PMI) has shown significant promise, although its performance has been found to deteriorate for non-Gaussian and non-linear data. In this paper, the effectiveness of different approaches to improving PMI performance is investigated, focussing on boundary issues associated with bandwidth estimation. Boundary issues, associated with kernel-based density and residual computations within PMI, arise from the extension of symmetrical kernels beyond the feasible bounds of potential inputs, and result in an underestimation of kernel-based marginal and joint probability distribution functions in the PMI. In total, the effectiveness of 16 different approaches is tested on synthetically generated data and the results are used to develop preliminary guidelines for PMI IVS. By using the proposed guidelines, the correct inputs can be identified in 100% of trials, even if the data are highly non-linear or non-Gaussian.     

Application of artificial neural networks for water quality prediction

The term “water quality” is used to describe the condition of water, including its chemical, physical, and biological characteristics. Modeling water quality parameters is a very important aspect in the analysis of any aquatic systems. Prediction of surface water quality is required for proper management of the river basin so that adequate measure can be taken to keep pollution within permissible limits. Accurate prediction of future phenomena is the life blood of optimal water resources management. The artificial neural network is a new technique with a flexible mathematical structure that is capable of identifying complex non-linear relationships between input and output data when compared to other classical modeling techniques. Johor River Basin located in Johor state, Malaysia, which is significantly degrading due to human activities and development along the river. Accordingly, it is very important to implement and adopt a water quality prediction model that can provide a powerful tool to implement better water resource management. Several modeling methods have been applied in this research including: linear regression models (LRM), multilayer perceptron neural networks and radial basis function neural networks (RBF-NN). The results showed that the use of neural networks and more specifically RBF-NN models can describe the behavior of water quality parameters more accurately than linear regression models. In addition, we observed that the RBF finds a solution faster than the MLP and is the most accurate and most reliable tool in terms of processing large amounts of non-linear, non-parametric data.

     

Neural networks for the prediction and forecasting of water resources variables: a review of modelling issues and applications

Artificial Neural Networks (ANNs) are being used increasingly to predict and forecast water resources variables. In this paper, the steps that should be followed in the development of such models are outlined. These include the choice of performance criteria, the division and pre-processing of the available data, the determination of appropriate model inputs and network architecture, optimisation of the connection weights (training) and model validation. The options available to modellers at each of these steps are discussed and the issues that should be considered are highlighted. A review of 43 papers dealing with the use of neural network models for the prediction and forecasting of water resources variables is undertaken in terms of the modelling process adopted. In all but two of the papers reviewed, feedforward networks are used. The vast majority of these networks are trained using the backpropagation algorithm. Issues in relation to the optimal division of the available data, data pre-processing and the choice of appropriate model inputs are seldom considered. In addition, the process of choosing appropriate stopping criteria and optimising network geometry and internal network parameters is generally described poorly or carried out inadequately. All of the above factors can result in non-optimal model performance and an inability to draw meaningful comparisons between different models. Future research efforts should be directed towards the development of guidelines which assist with the development of ANN models and the choice of when ANNs should be used in preference to alternative approaches, the assessment of methods for extracting the knowledge that is contained in the connection weights of trained ANNs and the incorporation of uncertainty into ANN models.     

Application of hybrid computational intelligence models in short-term bus load forecasting

Artificial neural networks (ANNs) are a favorable scheme in load forecasting applications mainly due to their endogenous capacity of robust modeling of data sets with highly non-linear relationship between inputs and outputs. Usually, the inputs correspond to historical load values, exogenous variables like temperature, day type identification codes and others. The outputs refer to the load values under examination. The majority of the load forecasting related literature focuses in aggregated load system level. While contemporary research efforts focus in smart grid technologies, there is need to study the characteristics of small scaled loads. Bus load forecasting refers to prediction of the demand patterns in buses of the transmission and distribution systems. Bus load exhibits low correlation with the aggregated system load, since it is characterized by a high level of stochasticity. Hence, a proper selection and formulation of the forecasting model is essential in order to keep the prediction accuracy within acceptable ranges. The treatment of bus load characteristics is held with computational intelligence techniques such as clustering and ANN. Neural network based systems are a favorable scheme in recent years in price and load predictions over traditional time series models. ANN can fully adapt expert knowledge and modify their parameters accordingly to simulate the problem`s attributions through training paradigms. Thus, ANN based systems are an essential choice, justified by the paper`s findings, for highly volatile time series. This work focuses on the short-term load forecasting (STLF) of a number of buses within the Greek interconnected system. Firstly, a modified version of the ANN already proposed for the aggregated load of the interconnected system is employed. To enhance the forecasting accuracy of the ANN, the load profiling methodology is used resulting to the formulation of two novel hybrid forecasting models. These models refer to the combination of the ANN with a clustering algorithm, resulting to superior performance. Simulation results indicate that the combination captures and successfully treats the special characteristics of the bus load patterns. The scope of the present paper is to develop efficient forecasting systems for short-term bus load predictions. This is a current research challenge due to the high interest for smart grids and demand side management applications by utilities, regulators, retailer and energy service companies. Bus load forecasting appears to be a more difficult engineering problem compared to forecasting of the total load of a country. No hybrid models for bus load predictions have been presented so far in the literature. Two novel clustering based tools are developed and successfully tested in a number of loads covering different types of electricity consumers and demand levels.     

Hybrid neural network models for hydrologic time series forecasting

The need for increased accuracies in time series forecasting has motivated the researchers to develop innovative models. In this paper, a new hybrid time series neural network model is proposed that is capable of exploiting the strengths of traditional time series approaches and artificial neural networks (ANNs). The proposed approach consists of an overall modelling framework, which is a combination of the conventional and ANN techniques. The steps involved in the time series analysis, e.g. de-trending and de-seasonalisation, can be carried out before gradually presenting the modified time series data to the ANN. The proposed hybrid approach for time series forecasting is tested using the monthly streamflow data at Colorado River at Lees Ferry, USA. Specifically, results from four time series models of auto-regressive (AR) type and four ANN models are presented. The results obtained in this study suggest that the approach of combining the strengths of the conventional and ANN techniques provides a robust modelling framework capable of capturing the non-linear nature of the complex time series and thus producing more accurate forecasts. Although the proposed hybrid neural network models are applied in hydrology in this study, they have tremendous scope for application in a wide range of areas for achieving increased accuracies in time series forecasting.     

Fixed-structure Gaussian process model

This article describes a method for modelling non-linear dynamic systems from measurement data. The method merges the linear local model blending approach in the velocity-based linearisation form with Bayesian Gaussian process (GP) modelling. The new Fixed-Structure GP (FSGP) model has a predetermined linear model structure with varying and probabilistic parameters represented by GP models. These models have several advantages for the modelling of local model parameters as they give us adequate results, even with small data sets. Furthermore, they provide a measure of the confidence in the prediction of the varying parameters and information about the dependence of the parameters on individual inputs. The FSGP model can be applied for the extended local linear equivalence class of non-linear systems. The obtained non-linear system model can be, for example, used for control-system design. The proposed modelling method is illustrated with a simple example of non-linear system modelling for control design.     

Generalized frequency response function matrix for MIMO non-linear systems

Recursive algorithms are derived to compute the generalized frequency response function matrix of multi-input multioutput (MIMO) non-linear systems as an analytical map from both non-linear differential equation models and NARX (Non-linear Auto Regressive Models with eXogenous inputs) models of the system. The algorithm is computationally compact and exposes the explicit relationship between the model parameters and the elements of the generalized frequency response function matrix and can thus provide important insights into the behaviour of non-linear systems.     

Process modelling for intelligent control

This contribution considers the problem of knowledge representation in intelligent control systems: the kind of knowledge that is required and how this may be encapsulated within an application framework. It is argued that rule-base techniques need to be supplemented by other methods in order to express certain pertinent aspects of the domain. After a brief discussion of qualitative modelling techniques, attention is focused upon numerical models and a relatively new qualitative modelling approach, Qualitative Transfer Functions.

The effectiveness of the latter model forms as repositories of deep knowledge are evaluated by application to a simulated distillation plant. The results indicate that although numerical models can provide accurate input-output representations, the integrated framework offered by Qualitative Transfer Functions is capable of carrying more information relevant to decision-making systems.     

Improved PMI-based input variable selection approach for artificial neural network and other data driven environmental and water resource models

Input variable selection (IVS) is one of the most important steps in the development of artificial neural network and other data driven environmental and water resources models. Partial mutual information (PMI) is one of the most promising approaches to IVS, but has the disadvantage of requiring kernel density estimates (KDEs) of the data to be obtained, which can become problematic when the data are non-normally distributed, as is often the case for environmental and water resources problems. In order to overcome this issue, preliminary guidelines for the selection of the most appropriate methods for obtaining the required KDEs are determined based on the results of 3780 trials using synthetic data with distributions of varying degrees of non-normality and six different KDE techniques. The validity of the guidelines is confirmed for two semi-real case studies developed based on the forecasting of river salinity and rainfall-runoff modelling problems.     

When do aquatic systems models provide useful predictions,what is changing,and what is next?

This article considers how aquatic systems modelling has changed since 1995 and how it must change in future if we are to continue to advance. A distinction is made between mechanistic and statistical models, and the relative merits of each are considered. The question of “when do aquatic systems models provide accurate and useful predictions?” is addressed, implying some guidelines for model development. It is proposed that, in general, ecological models only provide management-relevant predictions of the behaviour of real systems when there are strong physical (as opposed to chemical or ecological) drivers. Developments over the past 15 years have included changes in technology, changes in the modelling community and changes in the context in which modelling is conducted: the implications of each are briefly discussed. Current trends include increased uptake of best practice guidelines, increasing integration of models, operationalisation, data assimilation, development of improved tools for skill assessment, and application of models to new management questions and in new social contexts. Deeper merging of statistical and mechanistic modelling approaches through such techniques as Bayesian Melding, Bayesian Hierarchical Modelling and surrogate modelling is identified as a key emerging area. Finally, it is suggested that there is a need to systematically identify areas in which our current models are inadequate. We do not yet know for which categories of problems well-implemented aquatic systems models can (or cannot) be expected to accurately predict observational data and system behaviour. This can be addressed through better modelling and publishing practices.     

State-dependent parameter modelling and identification of stochastic non-linear sampled-data systems

State-dependent parameter representations of stochastic non-linear sampled-data systems are studied. Velocity-based linearization is used to construct state-dependent parameter models which have a nominally linear structure but whose parameters can be characterized as functions of past outputs and inputs. For stochastic systems state-dependent parameter ARMAX (quasi-ARMAX) representations are obtained. The models are identified from input–output data using feedforward neural networks to represent the model parameters as functions of past inputs and outputs. Simulated examples are presented to illustrate the usefulness of the proposed approach for the modelling and identification of non-linear stochastic sampled-data systems.     

Neural model input selection for a MIMO chemical process

In modelling non-linear systems using neural networks (NN), a commonly used method for the selection of network inputs, or to determine system order and time-delay, is to try different combinations of the system input–output data and choose the best one, giving minimum prediction error. The method is increasingly difficult to apply to industrial systems, due to their multivariable nature and complexity. A systematic method for the selection of model order and time-delay is developed in this paper, and applied to the neural modelling of a multivariable chemical process rig. The method is much simpler compared to the structure identification of the Non-linear Auto-Regressive with eXogenous inputs model (NARX), since the latter also needs to determine the significant terms from a linear-in-parameters polynomial. The orders and delays for system input and output are determined by identifying linearised models of the system. The method can also be applied to other approximations of a MIMO non-linear system, such as fuzzy logic models, etc. The application example demonstrates the selection procedure. Finally, the process rig is modelled using NNs according to the chosen structure, and the modelling error is compared with that of models with different structures to show the effectiveness of the method.     

An evaluation framework for input variable selection algorithms for environmental data-driven models

Input Variable Selection (IVS) is an essential step in the development of data-driven models and is particularly relevant in environmental modelling. While new methods for identifying important model inputs continue to emerge, each has its own advantages and limitations and no single method is best suited to all datasets and modelling purposes. Rigorous evaluation of new and existing input variable selection methods would allow the effectiveness of these algorithms to be properly identified in various circumstances. However, such evaluations are largely neglected due to the lack of guidelines or precedent to facilitate consistent and standardised assessment. In this paper, a new framework is proposed for the evaluation and inter-comparison of IVS methods which takes into account: (1) a wide range of dataset properties that are relevant to real world environmental data, (2) assessment criteria selected to highlight algorithm suitability in different situations of interest, and (3) a website for sharing data, algorithms and results (http://ivs4em.deib.polimi.it/). The framework is demonstrated on four IVS algorithms commonly used in environmental modelling studies and twenty-six datasets exhibiting different typical properties of environmental data. The main aim at this stage is to demonstrate the application of the proposed evaluation framework, rather than provide a definitive answer as to which of these algorithms has the best overall performance. Nevertheless, the results indicate interesting differences in the algorithms' performance that have not been identified previously.     

Modelling errors, entropy and the hydraulic reliability of water distribution systems

This paper reports on an investigation of the possible influence of modelling errors on the relationship between the entropy and hydraulic reliability of water distribution systems. The errors are due to minor differences between the design optimisation and subsequent simulation models, which lead to small discrepancies between the capacity of the network and the required supply. Pressure-dependent analysis was used for the hydraulic simulations. It is shown that any correlation between the redundancy or undercapacity due to the modelling errors and the hydraulic reliability is insignificant. The results, therefore, provide yet more evidence that the entropy-reliability relationship is strong.     

Towards integrated ADME prediction: past, present and future directions for modelling metabolism by UDP-glucuronosyltransferases

Undesirable absorption, distribution, metabolism, excretion (ADME) properties are the cause of many drug development failures and this has led to the need to identify such problems earlier in the development process. This review highlights computational (in silico) approaches that have been used to identify the characteristics of ligands influencing molecular recognition and/or metabolism by the drug-metabolising enzyme UDP-gucuronosyltransferase (UGT). Current studies applying pharmacophore elucidation, 2D-quantitative structure metabolism relationships (2D-QSMR), 3D-quantitative structure metabolism relationships (3D-QSMR), and non-linear pattern recognition techniques such as artificial neural networks and support vector machines for modelling metabolism by UGT are reported. An assessment of the utility of in silico approaches for the qualitative and quantitative prediction of drug glucuronidation parameters highlights the benefit of using multiple pharmacophores and also non-linear techniques for classification. Some of the challenges facing the development of generalisable models for predicting metabolism by UGT, including the need for screening of more diverse structures, are also outlined.     

Application of integrated soft computing techniques for optimisation of hybrid CO2 laser–MIG welding process

In this paper, artificial neural networks (ANNs), genetic algorithm (GA), simulated annealing (SA) and Quasi Newton line search techniques have been combined to develop three integrated soft computing based models such as ANN–GA, ANN–SA and ANN–Quasi Newton for prediction modelling and optimisation of welding strength for hybrid CO₂ laser–MIG welded joints of aluminium alloy. Experimental dataset employed for the purpose has been generated through full factorial experimental design. Laser power, welding speeds and wires feed rate are considered as controllable input parameters. These soft computing models employ a trained ANN for calculation of objective function value and thereby eliminate the need of closed form objective function. Among 11 tested networks, the ANN with best prediction performance produces maximum percentage error of only 3.21%. During optimisation ANN–GA is found to show best performance with absolute percentage error of only 0.09% during experimental validation. Low value of percentage error indicates efficacy of models. Welding speed has been found as most influencing factor for welding strength.