Expert elicitation approach for performing ATHEANA quantification

An expert elicitation approach has been developed to estimate probabilities for unsafe human actions (UAs) based on error-forcing contexts (EFCs) identified through the ATHEANA (A Technique for Human Event Analysis) search process. The expert elicitation approach integrates the knowledge of informed analysts to quantify UAs and treat uncertainty (‘quantification-including-uncertainty’). The analysis focuses on (a) the probabilistic risk assessment (PRA) sequence EFCs for which the UAs are being assessed, (b) the knowledge and experience of analysts (who should include trainers, operations staff, and PRA/human reliability analysis experts), and (c) facilitated translation of information into probabilities useful for PRA purposes. Rather than simply asking the analysts their opinion about failure probabilities, the approach emphasizes asking the analysts what experience and information they have that is relevant to the probability of failure. The facilitator then leads the group in combining the different kinds of information into a consensus probability distribution. This paper describes the expert elicitation process, presents its technical basis, and discusses the controls that are exercised to use it appropriately. The paper also points out the strengths and weaknesses of the approach and how it can be improved. Specifically, it describes how generalized contextually anchored probabilities (GCAPs) can be developed to serve as reference points for estimates of the likelihood of UAs and their distributions.     

Building prior distributions to support Bayesian reliability growth modelling using expert judgement

A sound methodology for the elicitation of subjective expert judgement is a pre-requisite for specifying prior distributions for the parameters of reliability growth models. In this paper, we describe an elicitation process that is developed to ensure valid data are collected by suggesting how possible bias might be identified and managed. As well as discussing the theory underpinning the elicitation process, the paper gives practical guidance concerning its implementation during reliability growth testing. The collection of subjective data using the proposed elicitation process is embedded within a Bayesian reliability growth modelling framework and reflections upon its practical use are described.     

Probability is perfect, but we can't elicit it perfectly

There are difficulties with probability as a representation of uncertainty. However, we argue that there is an important distinction between principle and practice. In principle, probability is uniquely appropriate for the representation and quantification of all forms of uncertainty; it is in this sense that we claim that ‘probability is perfect’. In practice, people find it difficult to express their knowledge and beliefs in probabilistic form, so that elicitation of probability distributions is a far from perfect process. We therefore argue that there is no need for alternative theories, but that any practical elicitation of expert knowledge must fully acknowledge imprecision in the resulting distribution.We outline a recently developed Bayesian technique that allows the imprecision in elicitation to be formulated explicitly, and apply it to some of the challenge problems.     

RISK DISK: A PC decision aid for human performance analysis

RISK DISK_CR is a set of IBM compatible PC diskettes that provide estimates from data, of the probabilities that certain tasks are finished within relevant time limits (windows) by different agents (crews). An important application is to performance of time-sensitive safety-related human interactions required during nuclear power plant operation.

The probability of satisfactory performance may be assessed from nuclear plant-specific data, augmented by other relevant data as desired, using modified log-normal and Weibull models. Procedures for assessing model fit, and for deriving informative error bounds, are made available.

There is no implication that RISK DISK actually calculates risk, in the sense of probability of an event multiplied by its consequence or penalty.     

Bayesian reliability analysis with imprecise prior probabilities

The Bayesian framework for statistical inference offers the possibility of taking expert opinions into account, and is therefore attractive in practical problems concerning the reliability of technical systems. Probability is the only language in which uncertainty can be consistently expressed, and this requires the use of prior distributions for reporting expert opinions. In this paper an extension of the standard Bayesian approach based on the theory of imprecise probabilities and intervals of measures is developed. It is shown that this is necessary to take the nature of experts' knowledge into account. The application of this approach in reliability theory is outlined. The concept of imprecise probabilities allows us to accept a range of possible probabilities from an expert for events of interest and thus makes the elicitation of prior information simpler and clearer. The method also provides a consistent way for combining the opinions of several experts.     

Reliability of drivers in urban intersections

The concept of human reliability has been widely used in industrial settings by human factors experts to optimise the person-task fit. Reliability is estimated by the probability that a task will successfully be completed by personnel in a given stage of system operation. Human Reliability Analysis (HRA) is a technique used to calculate human error probabilities as the ratio of errors committed to the number of opportunities for that error. To transfer this notion to the measurement of car driver reliability the following components are necessary: a taxonomy of driving tasks, a definition of correct behaviour in each of these tasks, a list of errors as deviations from the correct actions and an adequate observation method to register errors and opportunities for these errors. Use of the SAFE-task analysis procedure recently made it possible to derive driver errors directly from the normative analysis of behavioural requirements. Driver reliability estimates could be used to compare groups of tasks (e.g. different types of intersections with their respective regulations) as well as groups of drivers’ or individual drivers’ aptitudes. This approach was tested in a field study with 62 drivers of different age groups. The subjects drove an instrumented car and had to complete an urban test route, the main features of which were 18 intersections representing six different driving tasks. The subjects were accompanied by two trained observers who recorded driver errors using standardized observation sheets. Results indicate that error indices often vary between both the age group of drivers and the type of driving task. The highest error indices occurred in the non-signalised intersection tasks and the roundabout, which exactly equals the corresponding ratings of task complexity from the SAFE analysis. A comparison of age groups clearly shows the disadvantage of older drivers, whose error indices in nearly all tasks are significantly higher than those of the other groups. The vast majority of these errors could be explained by high task load in the intersections, as they represent difficult tasks. The discussion shows how reliability estimates can be used in a constructive way to propose changes in car design, intersection layout and regulation as well as driver training.     

Solving black box computation problems using expert knowledge theory and methods

The challenge problems for the Epistemic Uncertainty Workshop at Sandia National Laboratories provide common ground for comparing different mathematical theories of uncertainty, referred to as General Information Theories (GITs). These problems also present the opportunity to discuss the use of expert knowledge as an important constituent of uncertainty quantification. More specifically, how do the principles and methods of eliciting and analyzing expert knowledge apply to these problems and similar ones encountered in complex technical problem solving and decision making? We will address this question, demonstrating how the elicitation issues and the knowledge that experts provide can be used to assess the uncertainty in outputs that emerge from a black box model or computational code represented by the challenge problems. In our experience, the rich collection of GITs provides an opportunity to capture the experts' knowledge and associated uncertainties consistent with their thinking, problem solving, and problem representation. The elicitation process is rightly treated as part of an overall analytical approach, and the information elicited is not simply a source of data. In this paper, we detail how the elicitation process itself impacts the analyst's ability to represent, aggregate, and propagate uncertainty, as well as how to interpret uncertainties in outputs. While this approach does not advocate a specific GIT, answers under uncertainty do result from the elicitation.     

Safety analysis in process facilities: Comparison of fault tree and Bayesian network approaches

Safety analysis in gas process facilities is necessary to prevent unwanted events that may cause catastrophic accidents. Accident scenario analysis with probability updating is the key to dynamic safety analysis. Although conventional failure assessment techniques such as fault tree (FT) have been used effectively for this purpose, they suffer severe limitations of static structure and uncertainty handling, which are of great significance in process safety analysis. Bayesian network (BN) is an alternative technique with ample potential for application in safety analysis. BNs have a strong similarity to FTs in many respects; however, the distinct advantages making them more suitable than FTs are their ability in explicitly representing the dependencies of events, updating probabilities, and coping with uncertainties. The objective of this paper is to demonstrate the application of BNs in safety analysis of process systems. The first part of the paper shows those modeling aspects that are common between FT and BN, giving preference to BN due to its ability to update probabilities. The second part is devoted to various modeling features of BN, helping to incorporate multi-state variables, dependent failures, functional uncertainty, and expert opinion which are frequently encountered in safety analysis, but cannot be considered by FT. The paper concludes that BN is a superior technique in safety analysis because of its flexible structure, allowing it to fit a wide variety of accident scenarios.     

Bayesian analysis for the results of fatigue test using full-scale models to obtain the accurate failure probabilities of the Shinkansen vehicle axle

Bayesian analysis was performed to estimate an appropriate value of the uncertain propagation rate of cracks that can be initiated at the wheelseat of a Shinkansen vehicle axle. In the analysis, fatigue life distribution obtained by numerical simulation that employed the crack propagation rate obtained from small specimens was used as the prior distribution. Then it was modified by the results of the fatigue test of full-scale models as additional information to obtain the posterior distribution. It was indicated that the variances of fatigue life distribution reduced through the analysis. By using the crack propagation rate obtained from the posterior fatigue life distribution, the failure probabilities of the Shinkansen vehicle axle in operation, that were calculated previously by using the crack propagation rate due to the experiment of small specimens were recalculated. The resulting probabilities of failure were almost the same as those that were not modified, but were slightly lower. Although the difference was not so significant, it was thought that more confident values of the failure probability were obtained.     

Quantifying uncertainty under a predictive, epistemic approach to risk analysis

Risk analysis is a tool for investigating and reducing uncertainty related to outcomes of future activities. Probabilities are key elements in risk analysis, but confusion about interpretation and use of probabilities often weakens the message from the analyses. Under the predictive, epistemic approach to risk analysis, probabilities are used to express uncertainty related to future values of observable quantities like the number of fatalities or monetary loss in a period of time. The procedure for quantifying this uncertainty in terms of probabilities is, however, not obvious. Examples of topics from the literature relevant in this discussion are use of expert judgement, the effect of so-called heuristics and biases, application of historical data, dependency and updating of probabilities. The purpose of this paper is to discuss and give guidelines on how to quantify uncertainty in the perspective of these topics. Emphasis is on the use of models and assessment of uncertainties of similar quantities.     

A Total Operating Cost Model for Paced Lines with Stochastic Task Times

This paper presents a method for evaluating the often neglected component, the expected incompletion cost, of the operating cost of a paced line with stochastic task times. This incompletion cost is developed through a three stage process. First presented is a procedure for generating all possible combinations of incompleted tasks. Next is an evaluation of the probabilities and costs of their occurrence. Included are not only a mechanism for generating only those incompletion combinations with a significant probability of occurring, but also various techniques for reducing computer storage requirements and computation time. Finally, these make possible an evaluation of the operating cost of paced lines.     

Beliefs about beliefs, a theory for stochastic assessments of subjective probabilities

Summary Parameterized families of subjective probability distributions can be used to great advantage to model beliefs of experts, especially when such models include dependence on concomitant variables. In one such model, probabilities of simple events can be expressed in loglinear form. In another, a generalization of the multivariatet distribution has concomitant variables entering linearly through the location vector. Interactive interview methods for assessing this second model and matrix extensions thereof were given in recent joint work of the author with A.P. Dawid, J.B. Kadane and others. In any such verbal assessment method, elicited quantiles must be fitted by subjective probability models. The fitting requires the use of a further probability model for errors of elicitation. This paper gives new theory relating the form of the distribution of elicited probabilities and elicited quantiles to the form of the subjective probability distribution. The first and second order moment structures are developed to permit generalized least squares fits. Present affiliation: State University of New York, Albany     

Reliability of railway traffic personnel

The paper describes an analysis of reliability of railway traffic personnel. The human tasks critical to system safety are identified. Human error probability is chosen as a measure of human reliability in performing these tasks. The probabilities of some critical errors are estimated using the field data collected in the Slovenian railway traffic system. As stress conditions strongly affect the human performance, a simple model of the dependence of human error probability on stress levels is proposed.     

Introducing a heuristic approach to enhance the reliability of system safety assessment

Probabilistic risk assessment techniques are the important tools which can considerably improve the safety performance of the studied system and reduce the risk to an acceptable level. Typically, decision‐making process is an important part of risk assessment methods that accordingly bring the ambiguity inside. Decision makers as experts commonly express their subjective opinions about the occurrence of the root events in order to obtain the probability of the undesired event. Subsequently, the critical root events are identified, and possible intervention is performed to reduce the probability of the critical events. However, the serious point is the viability of the obtained probabilities and priority ranking of the critical events. In this study, a heuristic optimization model of linear mathematical programming using triangular intuitionistic fuzzy number (TrIFN) is proposed to obtain the feasible, optimum, and reliable results compared with available methods. The Spearman correlation is performed to examine the reliability and behavior of the proposed model. In order to show the effectiveness of the proposed approach, it is applied on a real case study. The application of the model confirms its robustness to prioritize critical root events over the conventional one.     

An application of logit models in analysing the behaviour of pedestrians and car drivers on pedestrian crossings

A multinomial logit model is used to examine pedestrian and driver reaction to "encounters" occurring on pedestrian crossings. The probabilities of a driver braking or weaving, and of a pedestrian continuing to cross in response to an encounter are identified for a variety of pedestrian, environmental, and traffic conditions. Results indicate that the most important explanatory variables included pedestrian distance from kerb, city size, number of pedestrians simultaneously crossing, vehicle speed, and vehicle platoon size. It is felt that the model performed well, should be applied in further studies, and could be a useful technique for identifying the most hazardous situations and locations within an area, for planning relevant safety measures, and for national research for developing traffic legislation.     

Bayesian reliability demonstration for failure-free periods

We study sample sizes for testing as required for Bayesian reliability demonstration in terms of failure-free periods after testing, under the assumption that tests lead to zero failures. For the process after testing, we consider both deterministic and random numbers of tasks, including tasks arriving as Poisson processes. It turns out that the deterministic case is worst in the sense that it requires most tasks to be tested. We consider such reliability demonstration for a single type of task, as well as for multiple types of tasks to be performed by one system. We also consider the situation, where tests of different types of tasks may have different costs, aiming at minimal expected total costs, assuming that failure in the process would be catastrophic, in the sense that the process would be discontinued. Generally, these inferences are very sensitive to the choice of prior distribution, so one must be very careful with interpretation of non-informativeness of priors.     

Aleatory and epistemic uncertainty in probability elicitation with an example from hazardous waste management

The quantification of a risk assessment model often requires the elicitation of expert judgments about quantities that cannot be precisely measured. The aims of the model being quantified provide important guidance as to the types of questions that should be asked of the experts. The uncertainties underlying a quantity may be classified as aleatory or epistemic according to the goals of the risk process. This paper discusses the nature of such a classification and how it affects the probability elicitation process and implementation of the resulting judgments. Examples from various areas of risk assessment are used to show the practical implications of how uncertainties are treated. An extended example from hazardous waste disposal is given.     

Treatment of uncertainties in space nuclear risk assessment with examples from Cassini mission applications

By means of several examples from a recent comprehensive space nuclear risk analysis of the Cassini mission, a scenario and consequence representational framework is presented for risk analysis of space nuclear power systems in the context of epistemic and aleatory uncertainties. The framework invites the use of probabilistic models for the calculation of both event probabilities and scenario consequences. Each scenario is associated with a frequency that may include both aleatory and epistemic uncertainties. The outcome of each scenario is described in terms of an end state vector. The outcome of each scenario is also characterized by a source term. In this paper, the source term factors of interest are number of failed clads in the space nuclear power system, amount of fuel released and amount of fuel that is potentially respirable. These are also subject to uncertainties. The 1990 work of Apostolakis is found to be a useful formalism from which to derive the relevant probabilistic models. However, an extension to the formalism was necessary to accommodate the situation in which aleatory uncertainty is represented by changes in the form of the probability function itself, not just its parameters. Event trees that show reasonable alternative accident scenarios are presented. A grouping of probabilities and consequences is proposed as a useful structure for thinking about uncertainties. An example of each category is provided. Concluding observations are made about the judgments involved in this analysis of uncertainties and the effect of distinguishing between aleatory and epistemic uncertainties.