Scenario-based assessment of nonfunctional requirements

This paper describes a method and a tool for validating nonfunctional requirements in complex socio-technical systems. The system requirements analyzer (SRA) tool validates system reliability and operational performance requirements using scenario-based testing. Scenarios are transformed into sequences of task steps and the reliability of human agents performing tasks with computerized technology is assessed using Bayesian belief network (BN) models. The tool tests system performance within an envelope of environmental variations and reports the number of tests that pass a benchmark threshold. The tool diagnoses problematic areas in scenarios representing pathways through system models, assists in the identification of their causes, and supports comparison of alternative requirements specifications and system designs. It is suitable for testing socio-technical systems where operational scenarios are sequential and deterministic, in domains where designs are incrementally modified so set up costs of the BNs can be defrayed over multiple tests.     

Reflecting on the Design Process for Virtual Reality Applications

A reflective analysis on the experience of virtual environment (VE) design is presented focusing on the human–computer interaction (HCI) challenges presented by virtual reality (VR). HCI design guidelines were applied to development of two VRs, one in marine archaeology and the other in situation awareness simulation experiments. The impact of methods and HCI knowledge on the VR design process is analyzed, leading to proposals for presenting HCI and cognitive knowledge in the context of design trade-offs in the choice of VR design techniques. Problems reconciling VE and standard Graphical User Interface (GUI) design components are investigated. A trade-off framework for design options set against criteria for usability, efficient operation, realism, and presence is proposed. HCI-VR design advice and proposals for further research aimed towards improving human factor-related design in VEs are discussed.     

Simulation-based evaluation of an in-vehicle smart situation awareness enhancement system

Situation awareness (SA) constitutes a critical factor in road safety, strongly related to accidents. This paper describes the evaluation of a proposed SA enhancement system (SAES) that exploits augmented reality through a head-up display (HUD). Two SAES designs were evaluation (information rich vs. minimal information) using a custom-made simulator and the Situation Awareness Global Assessment Technique with performance and EEG measures. The paper describes the process of assessing the SA of drivers using the SAES, through a series of experiments with participants in a Cave Automatic Virtual Environment. The effectiveness of the SAES was tested in a within-group research design. The results showed that the information rich (radar-style display) was superior to the minimal (arrow hazard indicator) design and that both SAES improved drivers’ SA and performance compared to the control (no HUD) design.

Practitioner Summary: Even though driver situation awareness is considered as one of the leading causes of road accidents, little has been done to enhance it. The current study demonstrates the positive effect of a proposed situation awareness enhancement system on driver situation awareness, through an experiment using virtual prototyping in a simulator.     

Workload prediction for improved design and reliability of complex systems

This paper describes a method and a tool for analysing and predicting workload for the design and reliability of complex socio-technical systems. It concentrates on the need to assess workload early in the design phase to prevent systems failures. This is a continuation of our previous work on workload assessment. The method is supported by a tool that enables scenario-based validation of prospective socio-technical systems designs such as command and control rooms of military vessels. The approach combines probabilistic measures of human performance with subjective estimates of workload. The causal relationships of performance shaping factors (PSF) are modelled in a Bayesian belief network (BBN) and used to assess the agent's operational performance and reliability. Workload for each agent is calculated based on demand placed upon agents in terms of behavioural response to tasks, communications and interactions between humans and technology. The approach uses scenarios to stress test prospective system designs, where each scenario is modelled as a sequence of events. Reliability is expressed in terms of human error and is dynamically assessed throughout test scenario executions using BBN technology. The innovation beyond traditional reliability analysis relies to the use of dynamic and static estimates of reliability inputs for better informed assessment. This method enables identification of performance bottlenecks to be addressed by the designer early in the design phase. A case study is presented that demonstrates the use of the method and tool for the design of the command and control room of a military vessel.     

The system reliability analyser tool

This paper describes the design and evaluation of a socio-technical design support system, the system reliability analyser (SRA). The tool is used to validate non-functional system requirements, such as system reliability. It employs a Bayesian belief network (BBN) model to assess system reliability (Pearl in Probabilistic reasoning in intelligent systems: networks of plausible information, 1988) based on a variety of high-level operational scenarios. The tool diagnoses problematic areas in future system models and assists in the identification of their causes. The evaluation of the tool demonstrated that it supported the task it was intended to do. The evaluation also identified usability problems in the current visualisations and illustrated their resolution.     

Scenario advisor tool for requirements engineering

This study investigates the usefulness of a scenario advisor tool which was built to help requirements engineers to generate sufficient sets of scenarios in the domain of socio-technical systems. The tool provides traceability between scenario models and requirements and helps to generate new scenarios and scenario variations. Through two series of evaluation sessions, we found that the scenario advisor tool helped users to write more sound scenarios without any domain knowledge, and to generate more variations on existing scenarios by providing specific scenario-generation hints for each scenario component. The tool should improve the reliability of requirements elicitation and validation.     

Automating Scenario Analysis of Human and System Reliability

The system reliability analyzer tool for analyzing the reliability of system designs is described and its use illustrated in a system engineering case study of a naval command and control system. The performance of systems consisting of human operators and technology components is assessed by Bayesian nets, which calculate error probabilities from inputs of agent properties and environmental conditions. The tool tests scenarios representing the system design and its operational behavior, which is modeled as cycles of command and control tasks. The tool indicates weak points in the scenario sequence and assesses the reliability of one or more system designs with a set of operational scenarios and a variety of environmental conditions     

Automated assistance for human factors analysis in complex systems

A tool and a method for scenario-based workload assessment and performance validation in complex socio-technical systems design, such as command and control rooms of military vessels, are described. We assess workload in terms of communication and the task load that each agent is able to handle. The method employs subjective task and communication estimates used to calculate the workload of human operators, using static and scenario-based analyses. This enables the identification of bottlenecks to be addressed by the designer with the appropriate allocation of function between humans and smart technology. This task is supported by the functional allocation adviser tool. A case study demonstrating the use of the tool for the design of the command and control room of a military vessel is presented.