Variable selection procedures and efficient suboptimal mask search algorithms in fuzzy inductive reasoning

This paper describes two new suboptimal mask search algorithms for Fuzzy inductive reasoning (FIR), a technique for modelling dynamic systems from observations of their input/output behaviour. Inductive modelling is by its very nature an optimisation problem. Modelling large-scale systems in this fashion involves solving a high-dimensional optimisation problem, a task that invariably carries a high computational cost. Suboptimal search algorithms are therefore important. One of the two proposed algorithms is a new variant of a directed hill-climbing method. The other algorithm is a statistical technique based on spectral coherence functions. The utility of the two techniques is demonstrated by means of an industrial example. A garbage incinerator process is inductively modelled from observations of 20 variable trajectories. Both suboptimal search algorithms lead to similarly good models. Each of the algorithms carries a computational cost that is in the order of a few percent of the cost of solving the complete optimisation problem. Both algorithms can also be used to filter out variables of lesser importance, i.e. they can be used as variable selection tools.     

Fuzzy inductive reasoning for variable selection analysis and modelling of biological systems

Fuzzy inductive reasoning (FIR) is a qualitative inductive modelling and simulation methodology for dealing with dynamical systems. It has proven to be a powerful tool for qualitative model identification and prediction of future behaviour of various kinds of dynamical systems, especially from the soft sciences, such as biology, biomedicine and ecology. This paper focuses on modelling aspects of the FIR methodology. It is shown that the FIR variable selection analysis is a useful tool not only for FIR but also for other classical quantitative methodologies such as nonlinear autoregressive moving average modelling with external inputs (NARMAX). The tool allows us to obtain models that interpret a system under study in optimal ways, in the sense that these models are well suited for predicting the future behaviour of the system they represent. The FIR variable selection analysis turns out to work well even in those applications where standard statistical variable selection analysis does not provide any useful information. In this paper, the FIR variable selection analysis is applied to a real system stemming from biology, namely, shrimp farming. The main goal is the identification of a growth model for occidental white shrimp (Penaeus vannamei), which allows farmers to plan the dates for seeding and harvesting the ponds in order to maximise their profits. FIR and NARMAX shrimp growth models are identified, and their prediction capabilities are compared to each other.     

Non-linear variable selection for artificial neural networks using partial mutual information

Artificial neural networks (ANNs) have been widely used to model environmental processes. The ability of ANN models to accurately represent the complex, non-linear behaviour of relatively poorly understood processes makes them highly suited to this task. However, the selection of an appropriate set of input variables during ANN development is important for obtaining high-quality models. This can be a difficult task when considering that many input variable selection (IVS) techniques fail to perform adequately due to an underlying assumption of linearity, or due to redundancy within the available data.This paper focuses on a recently proposed IVS algorithm, based on estimation of partial mutual information (PMI), which can overcome both of these issues and is considered highly suited to the development of ANN models. In particular, this paper addresses the computational efficiency and accuracy of the algorithm via the formulation and evaluation of alternative techniques for determining the significance of PMI values estimated during selection. Furthermore, this paper presents a rigorous assessment of the PMI-based algorithm and clearly demonstrates the superior performance of this non-linear IVS technique in comparison to linear correlation-based techniques.     

Reconstruction analysis-based algorithm to decompose a complex system into subsystems

Two previous papers [Mirats et al. (2002a) “On the selection of variables for Qualitative Modelling of Dynamical Systems”, International Journal of General Systems 31(5) pp. 435–467; Mirats et al. (2002b) “Variable selection procedures and efficient suboptimal mask search algorithms in Fuzzy Inductive Reasoning”, International Journal of General Systems 31(5), pp. 469–498] were devoted to the selection of a set of variables that can best be used to model (reconstruct) a given output variable, whereby only static relations were analysed. Yet even after reducing the set of variables in this fashion, the number of remaining variables may still be formidable for large-scale systems. The present paper aims at tackling this problem by discovering substructures within the whole set of the system variables. Hence whereas previous research dealt with the problem of model reduction by means of reducing the set of variables to be considered for modelling, the present paper focuses on model structuring as a means to subdivide the overall modelling task into subtasks that are hopefully easier to handle. The second and third sections analyse this problem from a system-theoretic perspective, presenting the reconstruction analysis (RA) methodology, an informational approach to the problem of decomposing a large-scale system into subsystems. The fourth section uses the fuzzy inductive reasoning (FIR) methodology to find a possible structure of a system. The study performed in this paper only considers static relations.     

Fuzzy inductive reasoning: a consolidated approach to data-driven construction of complex dynamical systems

Fuzzy inductive reasoning (FIR) is a modelling and simulation methodology derived from the General Systems Problem Solver. It compares favourably with other soft computing methodologies, such as neural networks, genetic or neuro-fuzzy systems, and with hard computing methodologies, such as AR, ARIMA, or NARMAX, when it is used to predict future behaviour of different kinds of systems. This paper contains an overview of the FIR methodology, its historical background, and its evolution.     

Systems dynamics with fuzzy logic

Systems dynamics has been used to model and simulate a variety of environments, e.g. economic, social and political, which require quantification or some types of human behaviour. The lack of empirical verification of the relationships in the systems dynamics models has often been criticised. Nevertheless, the methodology is effective in dealing with time-varying (dynamic) interactions among components of the analysed system. The effectiveness of systems dynamics as a methodology for modelling, simulating and analysing real-life systems can be significantly increased if it is extended to deal with imprecise and vague variables or events. Such an extension requires: (1) treatment of imprecise and vague input variables as fuzzy variables: (2) use of fuzzy arithmetic in the level, rate and auxiliary equations when fuzzy numbers are involved; and (3) replacement of some of the relationships in the systems dynamics models either with conditional statements including fuzzy variables, or with fuzzy algorithms.     

Genetic polynomial regression as input selection algorithm for non-linear identification

The performance of non-linear identification techniques is often determined by the appropriateness of the selected input variables and the corresponding time lags. High correlation coefficients between candidate input variables in addition to a non-linear relation with the output signal induce the need for an appropriate input selection methodology. This paper proposes a genetic polynomial regression technique to select the significant input variables for the identification of non-linear dynamic systems with multiple inputs. Statistical tools are presented to visualize and to process the results from different selection runs. The evolutionary approach can be used for a wide range of identification techniques and only requires a minimal input and a priori knowledge from the user. The evolutionary selection algorithm has been applied on a real-world example to illustrate its performance. The engine load in a combine harvester is highly variable in time and should be kept below an allowable limit during automatic ground speed control mode. The genetic regression process has been used to select those measurement variables that have a significant impact on the engine load and that will act as measurement variables of a non-linear model-based engine load controller.     

The application of data mining techniques to model visual distraction of bicyclists

This paper presents a novel approach to modelling visual distraction of bicyclists. A unique bicycle simulator equipped with sensors capable of capturing the behaviour of the bicyclist is presented. While cycling two similar scenario routes, once while simultaneously interacting with an electronic device and once without any electronic device, statistics of the measured speed, head movements, steering angle and bicycle road position along with questionnaire data are captured. These variables are used to model the self-assessed distraction level of the bicyclist. Data mining techniques based on random forests, support vector machines and neural networks are evaluated for the modelling task. Out of the total 71 measured variables a variable selection procedure based on random forests is able to select a fraction of those and consequently improving the modelling performance. By combining the random forest-based variable selection and support vector machine-based modelling technique the best overall performance is achieved. The method shows that with a few observable variables it is possible to use machine learning to model, and thus predict, the distraction level of a bicyclist.     

Optimization of fuzzy partitions for inductive reasoning using genetic algorithms

Fuzzy Inductive Reasoning (FIR) is a data-driven methodology that uses fuzzy and pattern recognition techniques to infer system models and to predict their future behavior. It is well known that variations on fuzzy partitions have a direct effect on the performance of the fuzzy-rule-based systems. The FIR methodology is not an exception. The performance of the model identification and prediction processes of FIR is highly influenced by the discretization parameters of the system variables, i.e. the number of classes of each variable and the membership functions that define its semantics. In this work, we design two new genetic fuzzy systems (GFSs) that improve this modeling and simulation technique. The main goal of the GFSs is to learn the fuzzification parameters of the FIR methodology. The new approaches are applied to two real modeling problems, the human central nervous system and an electrical distribution problem.     

Application of partial mutual information variable selection to ANN forecasting of water quality in water distribution systems

Recent trends in the management of water supply have increased the need for modelling techniques that can provide reliable, efficient, and accurate representation of the complex, non-linear dynamics of water quality within water distribution systems. Statistical models based on artificial neural networks (ANNs) have been found to be highly suited to this application, and offer distinct advantages over more conventional modelling techniques. However, many practitioners utilise somewhat heuristic or ad hoc methods for input variable selection (IVS) during ANN development.This paper describes the application of a newly proposed non-linear IVS algorithm to the development of ANN models to forecast water quality within two water distribution systems. The intention is to reduce the need for arbitrary judgement and extensive trial-and-error during model development. The algorithm utilises the concept of partial mutual information (PMI) to select inputs based on the analysis of relationship strength between inputs and outputs, and between redundant inputs. In comparison with an existing approach, the ANN models developed using the IVS algorithm are found to provide optimal prediction with significantly greater parsimony. Furthermore, the results obtained from the IVS procedure are useful for developing additional insight into the important relationships that exist between water distribution system variables.     

Gaussian process approach for modelling of nonlinear systems

Parametric modelling principals such as neural networks, fuzzy models and multiple model techniques have been proposed for modelling of nonlinear systems. Research effort has focused on issues such as the selection of the structure, constructive learning techniques, computational issues, the curse of dimensionality, off-equilibrium behaviour, etc. To reduce these problems, the use of non-parametrical modelling approaches have been proposed. This paper introduces the Gaussian process (GP) prior approach for the modelling of nonlinear dynamic systems. The relationship between the GP model and the radial basis function neural network is explained. Issues such as selection of the dimension of the input space and the computation load are also discussed. The GP modelling technique is demonstrated on an example of the nonlinear hydraulic positioning system.     

Protocol modelling: A modelling approach that supports reusable behavioural abstractions

We describe a behavioural modelling approach based on the concept of a “Protocol Machine”, a machine whose behaviour is governed by rules that determine whether it accepts or refuses events that are presented to it. We show how these machines can be composed in the manner of mixins to model object behaviour and show how the approach provides a basis for defining reusable fine-grained behavioural abstractions. We suggest that this approach provides better encapsulation of object behaviour than traditional object modelling techniques when modelling transactional business systems. We relate the approach to work going on in model driven approaches, specifically the Model Driven Architecture initiative sponsored by the Object Management Group. Communicated by August-Wilhelm Scheer Ashley McNeile is a practitioner with over 25 years of experience in systems development and IT related management consultancy. His main areas of interest are requirements analysis techniques and model execution and in 2001 he founded Metamaxim Ltd. to pioneer new techniques in these areas. He has published and presented widely on object oriented development methodology and systems architecture. Nicholas Simons has been working with formal methods of system specification since their introduction, and has over 20 years experience in building tools for system design, code generation and reverse engineering. In addition, he lectures on systems analysis and design, Web programming and project planning. He is a co-founder and director of Metamaxim Ltd.     

The Construction of Smooth Models using Irregular Embeddings Determined by a Gamma Test Analysis

One of the key problems in forming a smooth model from input-output data is the determination of which input variables are relevant in predicting a given output. In this paper, we show how the Gamma test can be used to select that combination of input variables which can best be employed to form a smooth model of an output. For time series prediction this amounts to the selection of an appropriate irregular embedding. We give some simple zero noise examples of time series analysis, and illustrate how using these techniques a binary message encoded into a chaotic carrier can be retrieved without knowledge of the dynamics used to generate the carrier. Provided the underlying dynamics are such as to produce a smooth embedding model with bounded partial derivatives, the sampling distribution is dense in input space, and any associated distribution of measurement error has the first few moments bounded, so that the typical prerequisite conditions of the Gamma test are satisfied, we conclude that the Gamma test is an effective tool in the determination of irregular time series embeddings. These techniques can also be useful in practical applications which involve filtering seismic data to detect anomalous events.     

Application of neural networks and sensitivity analysis to improved prediction of trauma survival

The performance of trauma departments is widely audited by applying predictive models that assess probability of survival, and examining the rate of unexpected survivals and deaths. Although the TRISS methodology, a logistic regression modelling technique, is still the de facto standard, it is known that neural network models perform better. A key issue when applying neural network models is the selection of input variables. This paper proposes a novel form of sensitivity analysis, which is simpler to apply than existing techniques, and can be used for both numeric and nominal input variables. The technique is applied to the audit survival problem, and used to analyse the TRISS variables. The conclusions discuss the implications for the design of further improved scoring schemes and predictive models.     

Separating the wheat from the chaff: An intelligent sales recruitment and benchmarking system

The need for computer-based intelligent techniques for recruitment and retention of employees in a highly competitive global market has grown significantly in the last decade. Salesperson recruitment is a critical task for most organisations. Existing approaches for salesperson recruitment primarily rely on filtering of applications based on selection criteria followed by interviews. Some organisations also include personality testing based on psychometric techniques. The high turnover of salesperson in the industry suggests limited success of these procedures. Additionally, existing approaches lack benchmarking methods. In this paper we describe design and development of an intelligent sales recruitment and benchmarking system (ISRBS) for recruitment and benchmarking of salespersons. ISRBS design represents operation of the findings and outcomes based on actual field studies and random surveys of salespersons as well as development of models for measuring independent and dependent variables related to selling behaviour. The main contributions of the paper are (i) Developing an on line selling behaviour profiling technique based on integration of intelligent system techniques like expert systems and fuzzy sets, psychology based selling behaviour model, and AHP techniques, and (ii) an objective and novel selling behaviour benchmarking technique to facilitate modelling of organisation based benchmarks and cultural fits. An earlier version of this system has been commercially used in the industry in Australia. ISRBS integrates psychology based selling behaviour model with artificial intelligence techniques and soft computing methods for selling behaviour profiling and benchmarking.     

Robot introspection through learned hidden Markov models

In this paper we describe a machine learning approach for acquiring a model of a robot behaviour from raw sensor data. We are interested in automating the acquisition of behavioural models to provide a robot with an introspective capability. We assume that the behaviour of a robot in achieving a task can be modelled as a finite stochastic state transition system.Beginning with data recorded by a robot in the execution of a task, we use unsupervised learning techniques to estimate a hidden Markov model (HMM) that can be used both for predicting and explaining the behaviour of the robot in subsequent executions of the task. We demonstrate that it is feasible to automate the entire process of learning a high quality HMM from the data recorded by the robot during execution of its task.The learned HMM can be used both for monitoring and controlling the behaviour of the robot. The ultimate purpose of our work is to learn models for the full set of tasks associated with a given problem domain, and to integrate these models with a generative task planner. We want to show that these models can be used successfully in controlling the execution of a plan. However, this paper does not develop the planning and control aspects of our work, focussing instead on the learning methodology and the evaluation of a learned model. The essential property of the models we seek to construct is that the most probable trajectory through a model, given the observations made by the robot, accurately diagnoses, or explains, the behaviour that the robot actually performed when making these observations. In the work reported here we consider a navigation task. We explain the learning process, the experimental setup and the structure of the resulting learned behavioural models. We then evaluate the extent to which explanations proposed by the learned models accord with a human observer's interpretation of the behaviour exhibited by the robot in its execution of the task.     

A neural network approach for selection of powder metallurgy materials and process parameters

The artificial neural network (NN) methodology presented in this paper has been developed for selection of powder and process parameters for Powder Metallurgy (PM) part manufacture. This methodology differs from the statistical modelling of mechanical properties in that it is not necessary to make assumptions regarding the form of the functions relating input and output variables. Employment of a NN approach allows specification of multiple input criterion, and generation of multiple output recommendations. The inputs comprise the required mechanical properties for the PM material. The system employs this data within the NN in order to recommend suitable metal powder compositions and process settings. Comparison of predicted and experimental PM materials data has confirmed the accuracy of the NN approach, for predicting the materials and process settings needed for attainment of required process outcomes.     

Neurofuzzy identification of an autonomous underwater vehicle

Neurofuzzy modelling is ideally suited to many nonlinear system identification and data modelling applications. By combining the attractive attributes of fuzzy systems and neural networks transparent models of ill-defined systems can be identified. Available expert a priori knowledge is used to construct an initial model. Data modelling techniques from the neural network, statistical and conventional system identification communities are then used to adapt these models. As a result accurate parsimonious models which are transparent and easy to validate are identified. Recent advances in the datadriven identification algorithms have now made neurofuzzy modelling appropriate for high-dimensional problems for which the expert knowledge and data may be of a poor quality. In this paper neurofuzzy modelling techniques are presented. This powerful approach to system identification is demonstrated by its application to the identification of an Autonomous Underwater Vehicle (AUV).     

A hybrid stock selection model using genetic algorithms and support vector regression

In the areas of investment research and applications, feasible quantitative models include methodologies stemming from soft computing for prediction of financial time series, multi-objective optimization of investment return and risk reduction, as well as selection of investment instruments for portfolio management based on asset ranking using a variety of input variables and historical data, etc. Among all these, stock selection has long been identified as a challenging and important task. This line of research is highly contingent upon reliable stock ranking for successful portfolio construction. Recent advances in machine learning and data mining are leading to significant opportunities to solve these problems more effectively. In this study, we aim at developing a methodology for effective stock selection using support vector regression (SVR) as well as genetic algorithms (GAs). We first employ the SVR method to generate surrogates for actual stock returns that in turn serve to provide reliable rankings of stocks. Top-ranked stocks can thus be selected to form a portfolio. On top of this model, the GA is employed for the optimization of model parameters, and feature selection to acquire optimal subsets of input variables to the SVR model. We will show that the investment returns provided by our proposed methodology significantly outperform the benchmark. Based upon these promising results, we expect this hybrid GA-SVR methodology to advance the research in soft computing for finance and provide an effective solution to stock selection in practice.