Bayesian structural equation modeling method for hierarchical model validation

A building block approach to model validation may proceed through various levels, such as material to component to subsystem to system, comparing model predictions with experimental observations at each level. Usually, experimental data becomes scarce as one proceeds from lower to higher levels. This paper presents a structural equation modeling approach to make use of the lower-level data for higher-level model validation under uncertainty, integrating several components: lower-level data, higher-level data, computational model, and latent variables. The method proposed in this paper uses latent variables to model two sets of relationships, namely, the computational model to system-level data, and lower-level data to system-level data. A Bayesian network with Markov chain Monte Carlo simulation is applied to represent the two relationships and to estimate the influencing factors between them. Bayesian hypothesis testing is employed to quantify the confidence in the predictive model at the system level, and the role of lower-level data in the model validation assessment at the system level. The proposed methodology is implemented for hierarchical assessment of three validation problems, using discrete observations and time-series data.     

Computational methods for model reliability assessment

This paper investigates various statistical approaches for the validation of computational models when both model prediction and experimental observation have uncertainties, and proposes two new methods for this purpose. The first method utilizes hypothesis testing to accept or reject a model at a desired significance level. Interval-based hypothesis testing is found to be more practically useful for model validation than the commonly used point null hypothesis testing. Both classical and Bayesian approaches are investigated. The second and more direct method formulates model validation as a limit state-based reliability estimation problem. Both simulation-based and analytical methods are presented to compute the model reliability for single or multiple comparisons of the model output and observed data. The proposed methods are illustrated and compared using numerical examples.     

Validation of reliability computational models using Bayes networks

This paper proposes a methodology based on Bayesian statistics to assess the validity of reliability computational models when full-scale testing is not possible. Sub-module validation results are used to derive a validation measure for the overall reliability estimate. Bayes networks are used for the propagation and updating of validation information from the sub-modules to the overall model prediction. The methodology includes uncertainty in the experimental measurement, and the posterior and prior distributions of the model output are used to compute a validation metric based on Bayesian hypothesis testing. Validation of a reliability prediction model for an engine blade under high-cycle fatigue is illustrated using the proposed methodology.     

Bayesian risk-based decision method for model validation under uncertainty

This paper develops a decision-making methodology for computational model validation, considering the risk of using the current model, data support for the current model, and cost of acquiring new information to improve the model. A Bayesian decision theory-based method is developed for this purpose, using a likelihood ratio as the validation metric for model assessment. An expected risk or cost function is defined as a function of the decision costs, and the likelihood and prior of each hypothesis. The risk is minimized through correctly assigning experimental data to two decision regions based on the comparison of the likelihood ratio with a decision threshold. A Bayesian validation metric is derived based on the risk minimization criterion. Two types of validation tests are considered: pass/fail tests and system response value measurement tests. The methodology is illustrated for the validation of reliability prediction models in a tension bar and an engine blade subjected to high cycle fatigue. The proposed method can effectively integrate optimal experimental design into model validation to simultaneously reduce the cost and improve the accuracy of reliability model assessment.     

Validation of models with multivariate output

This paper develops metrics for validating computational models with experimental data, considering uncertainties in both. A computational model may generate multiple response quantities and the validation experiment might yield corresponding measured values. Alternatively, a single response quantity may be predicted and observed at different spatial and temporal points. Model validation in such cases involves comparison of multiple correlated quantities. Multiple univariate comparisons may give conflicting inferences. Therefore, aggregate validation metrics are developed in this paper. Both classical and Bayesian hypothesis testing are investigated for this purpose, using multivariate analysis. Since, commonly used statistical significance tests are based on normality assumptions, appropriate transformations are investigated in the case of non-normal data. The methodology is implemented to validate an empirical model for energy dissipation in lap joints under dynamic loading.     

Uncertainty quantification and model validation of fatigue crack growth prediction

This paper presents a methodology for uncertainty quantification and model validation in fatigue crack growth analysis. Several models – finite element model, crack growth model, surrogate model, etc. – are connected through a Bayes network that aids in model calibration, uncertainty quantification, and model validation. Three types of uncertainty are included in both uncertainty quantification and model validation: (1) natural variability in loading and material properties; (2) data uncertainty due to measurement errors, sparse data, and different inspection results (crack not detected, crack detected but size not measured, and crack detected with size measurement); and (3) modeling uncertainty and errors during crack growth analysis, numerical approximations, and finite element discretization. Global sensitivity analysis is used to quantify the contribution of each source of uncertainty to the overall prediction uncertainty and to identify the important parameters that need to be calibrated. Bayesian hypothesis testing is used for model validation and the Bayes factor metric is used to quantify the confidence in the model prediction. The proposed methodology is illustrated using a numerical example of surface cracking in a cylindrical component.     

Model validation under epistemic uncertainty

This paper develops a methodology to assess the validity of computational models when some quantities may be affected by epistemic uncertainty. Three types of epistemic uncertainty regarding input random variables - interval data, sparse point data, and probability distributions with parameter uncertainty - are considered. When the model inputs are described using sparse point data and/or interval data, a likelihood-based methodology is used to represent these variables as probability distributions. Two approaches - a parametric approach and a non-parametric approach - are pursued for this purpose. While the parametric approach leads to a family of distributions due to distribution parameter uncertainty, the principles of conditional probability and total probability can be used to integrate the family of distributions into a single distribution. The non-parametric approach directly yields a single probability distribution. The probabilistic model predictions are compared against experimental observations, which may again be point data or interval data. A generalized likelihood function is constructed for Bayesian updating, and the posterior distribution of the model output is estimated. The Bayes factor metric is extended to assess the validity of the model under both aleatory and epistemic uncertainty and to estimate the confidence in the model prediction. The proposed method is illustrated using a numerical example.     

A Bayesian framework for accelerated reliability growth testing with multiple sources of uncertainty

Reliability growth tests are often used for achieving a target reliability for complex systems via multiple test‐fix stages with limited testing resources. Such tests can be sped up via accelerated life testing (ALT) where test units are exposed to harsher‐than‐normal conditions. In this paper, a Bayesian framework is proposed to analyze ALT data in reliability growth. In particular, a complex system with components that have multiple competing failure modes is considered, and the time to failure of each failure mode is assumed to follow a Weibull distribution. We also assume that the accelerated condition has a fixed time scaling effect on each of the failure modes. In addition, a corrective action with fixed ineffectiveness can be performed at the end of each stage to reduce the occurrence of each failure mode. Under the Bayesian framework, a general model is developed to handle uncertainty on all model parameters, and several special cases with some parameters being known are also studied. A simulation study is conducted to assess the performance of the proposed models in estimating the final reliability of the system and to study the effects of unbiased and biased prior knowledge on the system‐level reliability estimates.     

Bayesian uncertainty analysis with applications to turbulence modeling

In this paper, we apply Bayesian uncertainty quantification techniques to the processes of calibrating complex mathematical models and predicting quantities of interest (QoI's) with such models. These techniques also enable the systematic comparison of competing model classes. The processes of calibration and comparison constitute the building blocks of a larger validation process, the goal of which is to accept or reject a given mathematical model for the prediction of a particular QoI for a particular scenario. In this work, we take the first step in this process by applying the methodology to the analysis of the Spalart-Allmaras turbulence model in the context of incompressible, boundary layer flows. Three competing model classes based on the Spalart-Allmaras model are formulated, calibrated against experimental data, and used to issue a prediction with quantified uncertainty. The model classes are compared in terms of their posterior probabilities and their prediction of QoI's. The model posterior probability represents the relative plausibility of a model class given the data. Thus, it incorporates the model's ability to fit experimental observations. Alternatively, comparing models using the predicted QoI connects the process to the needs of decision makers that use the results of the model. We show that by using both the model plausibility and predicted QoI, one has the opportunity to reject some model classes after calibration, before subjecting the remaining classes to additional validation challenges.     

A model for Bayesian software reliability analysis

Two problems which are of great interest in relation to software reliability are the prediction of future times to failure and the calculation of the optimal release time. An important assumption in software reliability analysis is that the reliability grows whenever bugs are found and removed. In this paper we present a model for software reliability analysis using the Bayesian statistical approach in order to incorporate in the analysis prior assumptions such as the (decreasing) ordering in the assumed constant failure rates of prescribed intervals. We use as prior model the product of gamma functions for each pair of subsequent interval constant failure rates, considering as the location parameter of the first interval the failure rate of the following interval. In this way we include the failure rate ordering information. Using this approach sequentially, we predict the time to failure for the next failure using the previous information obtained. Using also the relevant predictive distributions obtained, we calculate the optimal release time for two different requirements of interest: (a) the probability of an in‐service failure in a prescribed time t; (b) the cost associated with a single or more failures in a prescribed time t. Finally a numerical example is presented. Copyright © 2000 John Wiley & Sons, Ltd.     

A novel Bayesian approach to reliability modeling: The benefits of uncertainty evaluation in the model selection procedure

This paper proposes a different likelihood formulation within the Bayesian paradigm for parameter estimation of reliability models. Moreover, the assessment of the uncertainties associated with parameters, the goodness of fit, and the model prediction of reliability are included in a systematic framework for better aiding the model selection procedure. Two case studies are appraised to highlight the contributions of the proposed method and demonstrate the differences between the proposed Bayesian formulation and an existing Bayesian formulation.     

Bayesian networks for multilevel system reliability

Bayesian networks have recently found many applications in systems reliability; however, the focus has been on binary outcomes. In this paper we extend their use to multilevel discrete data and discuss how to make joint inference about all of the nodes in the network. These methods are applicable when system structures are too complex to be represented by fault trees. The methods are illustrated through four examples that are structured to clarify the scope of the problem.     

Comparison of evidence theory and Bayesian theory for uncertainty modeling

This paper compares Evidence Theory (ET) and Bayesian Theory (BT) for uncertainty modeling and decision under uncertainty, when the evidence about uncertainty is imprecise. The basic concepts of ET and BT are introduced and the ways these theories model uncertainties, propagate them through systems and assess the safety of these systems are presented. ET and BT approaches are demonstrated and compared on challenge problems involving an algebraic function whose input variables are uncertain. The evidence about the input variables consists of intervals provided by experts. It is recommended that a decision-maker compute both the Bayesian probabilities of the outcomes of alternative actions and their plausibility and belief measures when evidence about uncertainty is imprecise, because this helps assess the importance of imprecision and the value of additional information. Finally, the paper presents and demonstrates a method for testing approaches for decision under uncertainty in terms of their effectiveness in making decisions.     

Objective Bayesian methods for one-sided testing

The one-sided testing problem can be naturally formulated as the comparison between two nonnested models. In an objective Bayesian setting, that is, when subjective prior information is not available, no general method exists either for deriving proper prior distributions on parameters or for computing Bayes factor and model posterior probabilities. The encompassing approach solves this difficulty by converting the problem into a nested model comparison for which standard methods can be applied to derive proper priors. We argue that the usual way of encompassing does not have a Bayesian justification. and propose a variant of this method that provides an objective Bayesian solution. The solution proposed here is further extended to the case where nuisance parameters are present and where the hypotheses to be tested are separated by an interval. Some illustrative examples are given for regular and non-regular sampling distributions. This paper has been supported by Ministerio de Ciencia y Tecnología under grant BEC20001-2982     

An engineering design methodology with multistage Bayesian surrogates and optimal sampling

This paper presents an adaptive, surrogate-based, engineering design methodology for the efficient use of numerical simulations of physical models. These surrogates are nonlinear regression models fitted with data obtained from deterministic numerical simulations using optimal sampling. A multistage Bayesian procedure is followed in the formulation of surrogates to support the evolutionary nature of engineering design. Information from computer simulations of different levels of accuracy and detail is integrated, updating surrogates sequentially to improve their accuracy. Data-adaptive optimal sampling is conducted by minimizing the sum of the eigenvalues of the prior covariance matrix. Metrics to quantify prediction errors are proposed and tested to evaluate surrogate accuracy given cost and time constraints. The proposed methodology is tested with a known analytical function to illustrate accuracy and cost tradeoffs. This methodology is then applied to the thermal design of embedded electronic packages with five design parameters. Temperature distributions of embedded electronic chip configurations are calculated using spectral element direct numerical simulations. Surrogates, built from 30 simulations in two stages, are used to predict responses of new design combinations and to minimize the maximum chip temperature.     

Mixture priors for Bayesian performance monitoring 1: fixed-constituent model

This paper addresses the problem of assessing the current performance of a system, given that performance can change over time. In many problems of interest, although a significant body of historical evidence is available, current performance information is too sparse to be the sole basis for an assessment of how well the system is currently performing. Therefore, it is desirable to apply current data within a Bayesian framework, making use of a broader body of evidence. However, both noninformative priors and simple informative priors have drawbacks for this application. The present work shows that ‘mixture’ priors have relatively desirable properties for performance assessment. These properties are illustrated using a simple example in assessment of reliability performance. It is also shown that one implementation of the mixture prior (the ‘fixed-constituent model’) is formally equivalent to methods used in signal detection, statistical decision rules in medical diagnosis, and many other applications. Building on the medical analogy, the potential benefits of an integrated treatment of reliability data and inspection results are illustrated. A companion paper develops a more sophisticated implementation of the mixture prior (the ‘variable-constituent model’), and extends the treatment to more complex examples.     

Assessing high reliability via Bayesian approach and accelerated tests

Sometimes the assessment of very high reliability levels is difficult for the following main reasons:

the high reliability level of each item makes it impossible to obtain, in a reasonably short time, a sufficient number of failures;

the high cost of the high reliability items to submit to life tests makes it unfeasible to collect enough data for ‘classical’ statistical analyses.

In the above context, this paper presents a Bayesian solution to the problem of estimation of the parameters of the Weibull–inverse power law model, on the basis of a limited number (say six) of life tests, carried out at different stress levels, all higher than the normal one.The over-stressed (i.e. accelerated) tests allow the use of experimental data obtained in a reasonably short time. The Bayesian approach enables one to reduce the required number of failures adding to the failure information the available a priori engineers' knowledge. This engineers' involvement conforms to the most advanced management policy that aims at involving everyone's commitment in order to obtain total quality.A Monte Carlo study of the non-asymptotic properties of the proposed estimators and a comparison with the properties of maximum likelihood estimators closes the work.     

A Bayesian model averaging method for software reliability modeling and assessment

The software reliability modeling is of great significance in improving software quality and managing the software development process. However, the existing methods are not able to accurately model software reliability improvement behavior because existing single model methods rely on restrictive assumptions and combination models cannot well deal with model uncertainties. In this article, we propose a Bayesian model averaging (BMA) method to model software reliability. First, the existing reliability modeling methods are selected as the candidate models, and the Bayesian theory is used to obtain the posterior probabilities of each reliability model. Then, the posterior probabilities are used as weights to average the candidate models. Both Markov Chain Monte Carlo (MCMC) algorithm and the Expectation–Maximization (EM) algorithm are used to evaluate a candidate model's posterior probability and for comparison purpose. The results show that the BMA method has superior performance in software reliability modeling, and the MCMC algorithm performs better than EM algorithm when they are used to estimate the parameters of BMA method.     

Bayesian networks inference algorithm to implement Dempster Shafer theory in reliability analysis

This paper deals with the use of Bayesian networks to compute system reliability. The reliability analysis problem is described and the usual methods for quantitative reliability analysis are presented within a case study. Some drawbacks that justify the use of Bayesian networks are identified. The basic concepts of the Bayesian networks application to reliability analysis are introduced and a model to compute the reliability for the case study is presented. Dempster Shafer theory to treat epistemic uncertainty in reliability analysis is then discussed and its basic concepts that can be applied thanks to the Bayesian network inference algorithm are introduced. Finally, it is shown, with a numerical example, how Bayesian networks’ inference algorithms compute complex system reliability and what the Dempster Shafer theory can provide to reliability analysis.     

Modelling the reliability of search and rescue operations with Bayesian Belief Networks

This paper uses a Bayesian Belief Networks (BBN) methodology to model the reliability of Search And Rescue (SAR) operations within UK Coastguard (Maritime Rescue) coordination centres. This is an extension of earlier work, which investigated the rationale of the government's decision to close a number of coordination centres. The previous study made use of secondary data sources and employed a binary logistic regression methodology to support the analysis. This study focused on the collection of primary data through a structured elicitation process, which resulted in the construction of a BBN. The main findings of the study are that statistical analysis of secondary data can be used to complement BBNs. The former provided a more objective assessment of associations between variables, but was restricted in the level of detail that could be explicitly expressed within the model due to a lack of available data. The latter method provided a much more detailed model, but the validity of the numeric assessments was more questionable. Each method can be used to inform and defend the development of the other. The paper describes in detail the elicitation process employed to construct the BBN and reflects on the potential for bias.