Evaluation techniques for systems analysis and design modelling methods – a review and comparative analysis

Even though hundreds of modelling methods are in existence today, practitioners and researchers are zealously ‘producing’ new modelling methods. The ‘blooming’ production of modelling methods is not the problem; the lack of standardized techniques for evaluating them is. To complicate the matter even further, most of the modelling methods have been introduced based on common sense and intuition of the methods' developers. Many of these methods lack theoretical foundations and empirical evidence to demonstrate their worthiness. With the current state of affairs, studies on evaluation of modelling methods have become necessary and critical. Comparing modelling methods provides us with the necessary knowledge and understanding of the strengths and weaknesses of each method. This knowledge can also guide us in our quest for better modelling methods. This paper reviews various evaluation techniques used by both researchers and practitioners. The evaluation techniques are categorized into three classifications: feature comparison, theoretical and conceptual evaluation, and empirical evaluation. This research also analyses the underlying philosophies and assumptions of these evaluation techniques.     

Enhancement of robustness in observer-based fault detection†

A prerequisite for the feasibility of the technique of observer-based fault detection and isolation (FDI) in dynamic systems is a satisfactory robustness with respect to modelling uncertainties. This paper surveys the most relevant methods to increase the robustness in both the stage of residual generation and residual evaluation. Among these methods are the generalized observer scheme, the robust parity space check, the unknown input and H _∞ observer scheme, the decorrelation filter, and the concept of adaptive threshold selection. It is pointed out that the unknown input observer concept, which provides perfect decoupling from the modelling errors or its optimal approximation with the aid of H _∞ techniques, constitutes a general framework of robust residual generation that generalizes and unifies numerous other approaches, among them the parity space and detection filter approach. It is further shown that this FDI method can even be applied to a certain class of nonlinear systems.     

Doing critical information systems research – arguments for a critical research methodology

Critical information systems (IS) research, it is argued, does not have a distinct methodological identity. While some research methods are closely related to the positivist research paradigm (experiments, surveys, and structural equation modelling) and others to the interpretivist paradigm (field study, ethnography, and action research), the critical paradigm is not identified with specific ‘critical methods’ and typically relies on the appropriation of interpretivist methods (such as critical ethnography). The criticism of the critical research paradigm in IS has often focused on the lack of distinctly critical research methods and even the neglect of methodological issues (Klein; McGrath). This paper questions the notion of and the arguments behind the quest for ‘critical research methods’ defined in contrast to positivist and interpretivist methods. Instead, the paper argues that it is a critical research methodology – understood as an overall strategy of conceptualizing and conducting an inquiry, engaging with studied phenomena, and constructing and justifying socially relevant knowledge, which distinguishes critical from other research paradigms. Building on a Kleinian argument regarding the need for common principles across diverse critical IS inquiries (Klein; Myers & Klein) this paper proposes a framework that describes key dimensions of a critical research methodology that distinguish critical from other research paradigms and provide methodological guidance in the doing of critical research.     

Model fusion using fuzzy aggregation: Special applications to metal properties

To improve the modelling performance, one should either propose a new modelling methodology or make the best of existing models. In this paper, the study is concentrated on the latter solution, where a structure-free modelling paradigm is proposed. It does not rely on a fixed structure and can combine various modelling techniques in ‘symbiosis’ using a ‘master fuzzy system’. This approach is shown to be able to include the advantages of different modelling techniques altogether by requiring less training and by minimising the efforts relating optimisation of the final structure. The proposed approach is then successfully applied to the industrial problems of predicting machining induced residual stresses for aerospace alloy components as well as modelling the mechanical properties of heat-treated alloy steels, both representing complex, non-linear and multi-dimensional environments.     

Collaborative modelling or participatory modelling? A framework for water resources management

Decision Support Systems, and, more recently, participatory and collaborative modelling have emerged as a response to increased focus on stakeholder participation in modelling activities for certain fields like water resources management. Researchers and practitioners frequently use ‘buzzwords’ such as ‘participatory modelling’ and ‘collaborative modelling’. In some cases, both terms are used interchangeably, largely due to unclear distinction between them in literature. This article draws the line between participatory and collaborative modelling by using levels of participation and cooperation as conditioning dimensions. Based on this methodology, a new generic framework is presented. This framework can help identify determinant features of both modelling approaches currently used in water resources management. It permits analysis of these approaches in terms of context, specific use, information handling, stakeholder involvement, modelling team and means. The article concludes with an application of the framework to a collaborative modelling approach carried out for a groundwater study in the Netherlands.     

Paradigmatic analysis as a means of eliciting knowledge to assist multimedia methodological development

We have identified two paradigms in multimedia systems design based on the software engineering and graphic design approaches. Our aim is to better inform methodological development in the field. We argue that unless and until the underlying paradigm of each community is understood, attempts to develop specific multimedia methods will be sub-optimal. ‘Paradigm’ is defined in the Kuhnian sense of a ‘disciplinary matrix’ that is composed of those shared beliefs and values that guide a ‘community’ of theorists and practitioners. This paper is based on analyses of several textbooks from both disciplines and also on interviews with software engineers and graphic designers who are actively engaged in the development of digital interactive multimedia (DIM). The results indicate that (a) Kuhn's concept of paradigm is a useful mechanism through which to examine the knowledge and approaches of these two design disciplines vis-à-vis multimedia development; and (b) that the resulting paradigms provide a useful framework from which to inform methodological development within the multimedia field. The paper concludes with a discussion of how the paradigms can assist theorists and practitioners in developing a framework for method evaluation and integration in multimedia.     

Dynamic simulation of action at operations level

The attempt of using lumped or agent-based simulation models to support operations management in production systems puts action modelling to the fore. To fill the gap of classical decision-support systems ignoring human agents’ practices, a modelling framework of action at operations level is proposed. This framework aims at answering two questions: How to represent action? How to represent the management of action? Every action (i.e., what is actually done by an agent) is represented by a binary function of time governed by events detected upon processes of various kinds: artefacts (clocks or schedules), external processes occurring in the environment, other actions. In turn, every action exerts its effect on target processes. This modelling framework allows one to simulate the interpretation of ongoing actions by using temporal or propositional logics and operations management behaviors through plan specification and execution, action composition, and resource allocation to concurrent actions. It enables complex activity systems to be represented and management options to be tested by simulation. These capacities are illustrated on the example of a farming system. The main benefits and issues raised by this dynamical system approach close to the ‘situated’ (vs. ‘planned’) action paradigm are discussed in the light of related works in Artificial intelligence. Future directions of research are drawn, namely that of how to scale up this lower-level representation of action to the higher-level representation of agents endowed with skills relevant at the level of the individual (e.g., anticipation).     

A support system for optimization modelling

With widespread use of executive decision support systems, the activity of modelling itself needs to be supported by a user friendly interface. This paper outlines an effort in this direction with respect to optimization modelling using linear programs. The approach is through a synthesis of a modelling language (LAMP) under development, for manipulating the structure, and a commercially available database management system for manipulating the data. Such a synthesis provides a very powerful ‘what if’ analysis capability paving the way for a new generation of modelling systems.     

Graph-theoretic quantum system modelling for information/computation processing circuits

This paper introduces graph-theoretic quantum system modelling (GTQSM), which is facilitated by considering the fundamental unit of quantum computation and information, viz. a quantum bit or qubit as a basic building block. Unit directional vectors ‘ket 0’ and ‘ket 1’ constitute two distinct fundamental quantum across variable orthonormal basis vectors (for the Hilbert space) specifying the direction of propagation, as it were, of information (or computation data) while complementary fundamental quantum through (flow rate) variables specify probability parameters (or amplitudes) as surrogates for scalar quantum information measure (von Neumann entropy). Applications of GTQSM are presented for quantum information/computation processing circuits ranging from a simple qubit and superposition or product of two qubits through controlled NOT and Hadamard gate operations to a substantive case of 3-port, 5-stage circuit for quantum teleportation. An illustrative circuit for teleporting a qubit is modelled as a complex ‘system of systems’ resulting in four probable transfer function models. It has the potential of extending the applications of GTQSM further to systems at the higher end of complexity scale too. The key contribution of this paper lies in generalization or extension of the graph-theoretic system modelling framework, hitherto used for classical (mostly deterministic) systems, to quantum random systems. Further extension of the graph-theoretic system modelling framework to quantum field modelling is the subject of future work.     

Implementation and application of a distributed hydrological model using a component-based approach

The application of a model to different study areas often requires that the model be modified to conform to specific characteristics, but this can be challenging due to the poor readability and reusability of the legacy codes. Component-based programming supported by a modelling framework provides a generic means to develop and modify models. This paper describes the development of a distributed hydrological model using a component-based modelling framework, which is implemented as a set of functional components that are integrated at runtime. The model was applied to runoff simulation in a large scale and data scarce alpine basin, and was further improved by incorporating a simple empirical soil freezing-thawing component. The results show that the componentised model reproduced the daily and monthly flow hydrograph with ‘good’ accuracy. The framework is shown to be flexible enough for model development and model modification.     

A systemic framework and analysis of urban water energy

Energy impacts of urban water systems are substantial, but not typically analysed systemically. We develop a new system boundary framework including a utility, the ‘bulk water supply authority’ (SB1); the ‘urban water system’ including water use (SB2); and the ‘regional water system’ (SB3). We use the framework to review existing models and show that most address only one boundary. We apply the framework to quantify thermal equivalents of water-related energy in SB1 and SB2, and identify that over 96% of water-related energy in South East Queensland (SEQ) is outside SB1 and within SB2. Consideration of energy influenced by water use is paramount to systemic energy efficiency and optimisation in the urban water system. Clear articulation of system boundaries will improve modelling and management of the energy impact of urban water. Systemic modelling will help decision makers answer increasingly integrated and cross-system and sector questions regarding water and energy interactions.     

An ontological process modelling framework for stochastic systems

Most process modelling techniques exist without any firm theoretical foundation. This results in a lack of model validation, which can be in-terms of model consistency, feasibility and goal compliance. Moreover, these techniques are mostly deterministic in nature and not applicable to stochastic systems. In this paper, we propose an ontology-based stochastic process modelling framework that further provides a specialization to failure and reliability issues. The framework is notation independent, and is primarily rooted in Bunge’s ontology. The well-established theory of reliability constructs are also mapped to facilitate the modelling of failure prone systems.     

Simulating spectral images of natural waters and submerged objects from a bird’s-eye view

We describe a new approach to modelling two-dimensional ‘bird’s-eye’ images of natural waters, sensitive to the spectral absorption and scattering over the visual range. Constructed using the mathematical framework of invariant imbedding to efficiently and economically resolve the associated radiative transfer equations, this method provides a fast routine to build physically accurate simulations. We characterise the quality of the water through surface chlorophyll content, instead of the traditional Jerlov water types. Simulations of ‘Macbeth’ colour identification chart at different depths are compared with real, hyperspectral images recorded in a tank.     

Limnimeter and rain gauge FDI in sewer networks using an interval parity equations based detection approach and an enhanced isolation scheme

In this paper, a methodology for limnimeter and rain-gauge fault detection and isolation (FDI) in sewer networks is presented. The proposed model based FDI approach uses interval parity equations for fault detection in order to enhance robustness against modelling errors and noise. They both are assumed unknown but bounded, following the so-called interval (or set-membership) approach. On the other hand, fault isolation relies on an algorithm that reasons using several fault signature matrices that store additional information to the typical binary one used in standard FDI approaches. More precisely, the considered fault signature matrices contain information about residual fault sign/sensitivity and time/order of activation. The paper also proposes an identification procedure to obtain the interval models used in fault detection that delivers the nominal model plus parameter uncertainty is proposed. To exemplify the proposed FDI methodology, a case study based on the Barcelona sewer network is used.     

On-line fault detection in uncertain nonlinear systems using diagnostic observers: a survey

The paper discusses the principles of model-based fault detection and isolation (FDI) in nonlinear and time-varying uncertain dynamic systems. Such systems are typical for such complex plants as, for example, in the chemical process industries or in advanced transportation technology. For a model-based fault diagnosis in such situations, robust or even adaptive strategies are needed. In this paper the theory of robust linear observer-based residual generation for FDI is reviewed from a general point of view. The structural equivalence between the parity space approach and observer-based approach is shown in a new simple graphical way by showing that the observer-based FDI concept can easily be transformed into an equivalent extended parity space configuration, without claiming, however, equivalence of the underlying design techniques. The unknown input observer approach known as a most powerful and comprehensive framework for robust residual generation for FDI in uncertain linear systems is extended to classes of nonlinear and time-varying systems. For such plants an adaptive nonlinear unknown input observer scheme is proposed. Finally, appropriate residual evaluation techniques are outlined and suggestions are made to increase the robustness, for instance by using adaptive thresholds.     

Modelling of quality in physical system theory: an extended framework

Quality is a vital concept that has important bearing on the functioning of any system. In order to control the quality effectively, it has to be systematically planned. Further, the quality in the system has a significant impact on the cost. The cost and flow dimensions of any manufacturing or socioeconomic system can be very well modelled and analysed in the framework of physical system theory (PST), which is based on the concept of a linear graph and deals with two categories of variables; an ‘across’ variable and a ‘through’ variable. The “across’ variables are governed by the law of compatibility, whereas the ‘through’ variables are governed by the law of continuity. The quality variables are not governed by the strict definitions of the “across’ and ‘through’ variables and the constraints imposed on them. Hence, in order to utilize the PST framework for modelling the quality variables in addition to the cost and flow variables, we attempt to develop an extended framework with quality as a third category of variables. An overview of the PST concepts is provided and the need for an extended framework is discussed. Then, the extended framework of the PST is presented with the modelling of basic processes, i.e. transformation, transportation and storage, in the developed framework with additional quality equations and functions. The proposed framework is illustrated with the simple example of a manufacturing system with three stages of production.     

Comparing the accuracy of estimated terrain elevations across spatial resolution

Terrain is modelled in Geographic Information Science on a grid, assuming that elevation values are constant within any single pixel of a Digital Elevation Model (DEM). Pixels are considered flat and rigid, for computational simplicity (a ‘rigid pixel’ paradigm). This paradigm does not account for the slope and curvature of terrain within each pixel, generating imprecise measurements, particularly as pixel size increases or in uneven terrain. This paper relaxes the rigid pixel assumption, allowing for possible sub-pixel variations in slope and curvature (a ‘surface-adjusted’ paradigm). This paper compares different interpolation methods to investigate sub-pixel variations for estimating elevation of arbitrary points given a regular grid. Tests interpolating elevation values for 20,000 georeferenced off-centroid random points from a regular grid DEM are presented, using a variety of exact and inexact local deterministic interpolation methods within contiguity configurations incorporating first and second order neighbours. The paper examines the accuracy of surface-adjusted estimations across a progression of spatial resolutions (10 m, 30 m, 100 m, and 1,000 m DEMs) and a suite of terrain types varying in latitude, altitude, slope, and roughness, validating off-centre estimates against direct elevation measurements on 3 m resolution lidar DEM. Results illustrate that the Bi-quadratic and Bi-cubic interpolation methods outperform Weighted Average, Linear, and Bi-linear methods at coarse resolutions and in rough or non-uniform terrain. In smooth or flat terrain and at finest resolutions, the interpolation method impacts estimation accuracy less or not at all. The type of contiguity configuration appears to play a role in estimation errors as well, with tighter neighbourhoods exhibiting higher accuracy. The analysis also examined regularized mathematical surfaces, adding autocorrelated randomly distributed noise to simulate terrain. The results of experiments based on regularized smooth mathematical surfaces do not translate directly to terrain modelling. The analysis also considers the balance between the increased computation times needed to measure surface-adjusted elevation against improvements in accuracy.     

Fault detection,isolation and quantification from Gaussian residuals with application to structural damage diagnosis

Despite the general acknowledgment in the Fault Detection and Isolation (FDI) literature that FDI are typically accomplished in two steps, namely residual generation and residual evaluation, the second step is by far less studied than the first one. This paper investigates the residual evaluation method based on the local approach to change detection and on statistical tests. The local approach has the remarkable ability of transforming quite general residuals with unknown or non Gaussian probability distributions into a standard Gaussian framework, thanks to a central limit theorem. In this paper, the ability of the local approach for fault quantification will be exhibited, whereas previously it was only presented for fault detection and isolation. The numerical computation of statistical tests in the Gaussian framework will also be revisited to improve numerical efficiency. An example of vibration-based structural damage diagnosis will be presented to motivate the study and to illustrate the performance of the proposed method.     

A flexible,object-centric approach for business process modelling

Mainstream business process modelling techniques often promote a design paradigm wherein the activities that may be performed within a case, together with their usual execution order, form the backbone on top of which other aspects are anchored. This Fordist paradigm, while effective in standardised and production-oriented domains, breaks when confronted with processes in which case-by-case variations and exceptions are the norm. We contend that the effective design of flexible processes calls for a substantially different modelling paradigm. Motivated by requirements from the human services domain, we explore the hypothesis that a framework consisting of a small set of coordination concepts, combined with established object-oriented modelling principles, provides a suitable foundation for designing highly flexible processes. Several human service delivery processes have been designed using this framework, and the resulting models have been used to realise a system to support these processes in a pilot environment.