Investigating software testing and maintenance reports: Case study

ContextAlthough many papers have been published on software development and defect prediction techniques, problem reports in real projects quite often differ from those described in the literature. Hence, there is still a need for deeper exploration of case studies from industry.ObjectiveThe aim of this study is to present the impact of fine-grained problem reports on improving evaluation of testing and maintenance processes. It is targeted at projects involving several releases and complex schemes of problem handling. This is based on our experience gained while monitoring several commercial projects.MethodExtracting certain features from detailed problem reports, we derive various measures and present analysis models which characterize and visualize the effectiveness of testing and problem resolution processes. The considered reports describe types of problems (e.g. defects), their locations in project versions and software modules, ways of their resolution, etc. The performed analysis is related to eleven projects developed in the same company. This study is an exploratory research with some explanatory features. Moreover, having identified some drawbacks, we present extensions of problem reports and their analysis which have been verified in another industrial case study project.ResultsFine-grained (accurate) problem handling reports provide a wider scope of possible measures to assess the relevant development processes. This is helpful in controlling single projects (local perspective) as well as in managing these processes in the whole company (global perspective).ConclusionDetailed problem handling reports extend the space and quality of statistical analysis, they provide significant enhancement in evaluation and refinement of software development processes as well as in reliability prediction.     

Judgemental errors in aviation maintenance

Cognition, Technology & Work - Aircraft maintenance is a critical success factor in the aviation sector, and incorrect maintenance actions themselves can be the cause of accidents. Judgemental...     

Eradicating root causes of aviation maintenance errors: introducing the AMMP

This is the second of two papers in the area of aviation maintenance safety enhancement utilizing human error risk management in engineering systems methodology. The first focused on the retrospective analysis of human error in aircraft maintenance, where the impact of human reliability on aviation maintenance safety was investigated. The purpose was to explore the actual mechanisms of maintenance errors initialization and propagation, with the aim of identifying requisite characteristics for future solutions. This second part focuses on a prospective approach to proactive safety techniques that can be applied to the field. It builds on the findings from the previous work and introduces a proposed Aviation Maintenance Monitoring Process (AMMP); an integrated process that is to be collectively implemented by industry to proactively monitor existence of human error causal factors. These factors are typically pre-initiated during design practices, manufacturing processes, or at later stages due to organizational, individuals, or workplace conditions within maintenance, repair, and overhaul organizations (MRO). As a result, such causal factors can be gradually eliminated. This generic AMMP is based on user software built applying the fuzzy Analytic Network Process (fuzzy ANP) theory, with a database of a communal total of 870 assessment criteria designed within the software. The full process is validated practically in real world within industry.     

Greener aviation with virtual sensors: a case study

The environmental impact of aviation is enormous given the fact that in the US alone there are nearly 6 million flights per year of commercial aircraft. This situation has driven numerous policy and procedural measures to help develop environmentally friendly technologies which are safe and affordable and reduce the environmental impact of aviation. However, many of these technologies require significant initial investment in newer aircraft fleets and modifications to existing regulations which are both long and costly enterprises. We propose to use an anomaly detection method based on Virtual Sensors to help detect overconsumption of fuel in aircraft which relies only on the data recorded during flight of most existing commercial aircraft, thus significantly reducing the cost and complexity of implementing this method. The Virtual Sensors developed here are ensemble-learning regression models for detecting the overconsumption of fuel based on instantaneous measurements of the aircraft state. This approach requires no additional information about standard operating procedures or other encoded domain knowledge. We present experimental results on three data sets and compare five different Virtual Sensors algorithms. The first two data sets are publicly available and consist of a simulated data set from a flight simulator and a real-world turbine disk. We show the ability to detect anomalies with high accuracy on these data sets. These sets contain seeded faults, meaning that they have been deliberately injected into the system. The second data set is from real-world fleet of 84 jet aircraft where we show the ability to detect fuel overconsumption which can have a significant environmental and economic impact. To the best of our knowledge, this is the first study of its kind in the aviation domain.     

Helicopter maintenance error analysis: Beyond the third order of the HFACS-ME

In this paper a statistical analysis of a sample of 58 helicopter maintenance-induced safety occurrences is conducted to study helicopter accidents and incidents’ survivability and the severity distribution of such occurrences. Analysis is also carried out to identify helicopter main and sub-systems mostly exposed to maintenance errors and to determine various types of such errors. Expected inherent relations between rotorcraft components affected and types of associated maintenance errors are investigated. Human factors-based triggers of these accidents and severe incidents are explored. The concept of Specific Failures (SFs) that immediately precede each of such occurrences is introduced for more detailed representation of the last breached individual and organizational safety barriers. Root causes of these safety occurrences were then sought utilizing the Human Factors Analysis and Classification System-Maintenance Extension (HFACS-ME) taxonomy with a refined focus on its third order categories’ list. The rotorcraft characteristics influencing individuals and organizational behaviours within Maintenance, Repair and Overhaul organizations (MROs) are discussed in the light of the root cause investigation results.

Relevance to industry

The study of human reliability within helicopter maintenance industry is waited to emphasise the understanding of causes and propagation mechanisms of maintainers' errors and their consequences on the overall aviation safety. Previous works often investigated maintenance errors and their roles in promoting aviation accidents of fixed-wing aircraft; this research is investigating the case of rotorcraft.     

The development of a model and simulation of an aviation maintenance technician task performance

This paper describes the model of an aircraft maintenance technician (AMT) behaviour working in a real world environment and the application of this model in a computer simulation. The model was derived from the combination of two existing models of cognition (SHELL and RMC/PIPE), based on the paradigm of an information processing system. The simulation structure was developed by transforming the model into an action-execution structure and implementing this model in a computer program based on an object-oriented programming language. An error generation system was coupled to a simple taxonomy of causal correlations between socio-technical and contextual factors operating in an environment defined by a comprehensive task analysis. The simulator can be applied to many aviation maintenance tasks to assist in the design of aviation maintenance training systems through the analysis of the interaction of performance influencing factors and possible AMT performances during the execution of normal maintenance operations. A number of sample outputs from the simulator show the potential of the results to assist in training development. A discussion is also made of other potential applications and the future directions the simulator structure can take.     

Distributed cognition in aviation operations: a gate-to-gate study with implications for distributed crewing

Abstract

The network analysis method, Event Analysis of Systemic Teamwork (EAST), was used to examine routine aviation operations from multiple perspectives from six key areas (i.e. Dispatch, ATC, ATM, Maintenance, Loading, and the Cockpit). Data was collected over a five-day observational field trial at an international air cargo operator. Researchers recorded the activities of agents operating within the six key areas over three outbound and two inbound flights. Three networks (i.e. social, information and task) were created for four key phases of flight: (i) pre-flight checks and engines start (ii) taxi, take-off and assent, (iii) descent, landing and taxi, and (iv) park and shut down. The networks represent a ‘work audit’ of short-haul cargo operations, which enabled a detailed understanding of the interactions and connections within the current system. Implications for the future of distributed crewing concepts are discussed.

Practitioner Summary: An analysis of the aviation system was undertaken using the amalgamated data from three outbound and two inbound flights. These analyses show the social, information and task interactions for cargo operations. This has been used to specify requirements for future distributed crewing options.     

Human error and commercial aviation accidents: an analysis using the human factors analysis and classification system

OBJECTIVE: The aim of this study was to extend previous examinations of aviation accidents to include specific aircrew, environmental, supervisory, and organizational factors associated with two types of commercial aviation (air carrier and commuter/ on-demand) accidents using the Human Factors Analysis and Classification System (HFACS). BACKGROUND: HFACS is a theoretically based tool for investigating and analyzing human error associated with accidents and incidents. Previous research has shown that HFACS can be reliably used to identify human factors trends associated with military and general aviation accidents. METHOD: Using data obtained from both the National Transportation Safety Board and the Federal Aviation Administration, 6 pilot-raters classified aircrew, supervisory, organizational, and environmental causal factors associated with 1020 commercial aviation accidents that occurred over a 13-year period. RESULTS: The majority of accident causal factors were attributed to aircrew and the environment, with decidedly fewer associated with supervisory and organizational causes. Comparisons were made between HFACS causal categories and traditional situational variables such as visual conditions, injury severity, and regional differences. CONCLUSION: These data will provide support for the continuation, modification, and/or development of interventions aimed at commercial aviation safety. APPLICATION: HFACS provides a tool for assessing human factors associated with accidents and incidents.     

Investigating service‐oriented system performance: a systematic study

The initial use of a service‐oriented system, which collates information from three databases, highlighted a performance problem with query response time for queries that selected more than ten attributes from a single data source. Our aim was to find the cause of the performance problem within the service‐oriented system. The challenges that were encountered when attempting to locate such a problem are described. Our approach to finding the performance problem, while addressing these challenges, was based on systematically devising and applying measurements to the system. We developed strategies that addressed the challenges and the inherent difficulty of measuring performance in a service‐oriented system. The cause of the performance problem was a buffer used by a JDBC‐ODBC driver to connect Java code to a Microsoft Access database. Three strategies successfully addressed the challenges of measuring performance in the service‐oriented system. In particular, routing queries to a local data source eliminated networking problems. Creating a model of the system allowed performance measurements to continue concurrently with demonstrations of the system, and with its further development. Finally, porting the system identified the part that had a detrimental effect on performance. Copyright © 2006 John Wiley & Sons, Ltd.     

Investigating multiple human performance measures in teleoperation task: A translation task in a tele‐sandblasting maintenance system

In sandblasting tasks for complex steel structure maintenance, teleoperation is required to keep humans away from occupational risk and hazard. On the other hand, teleoperation typically degrades system‐human performances, resulting in poor product quality and must be designed such that the performances remain as high as possible. However, designing the teleoperation system regarding to a single performance measure may lead to an improper design. In this article, we propose two novel loss‐function‐based human‐performance measures to incorporate with a widely used performance measure, movement time, to thoroughly represent performance: unfinished surface and damaged surface. We aim to investigate the effects of two main design parameters, viewing distance and path width. The results show that only path width is significant for overall performances. Furthermore, the effect of gender is significant such that men outperform women in cleaning the surface. Finally, the optimal setting conditions are suggested to achieve their optimal performances.     

Designing augmented reality for future commercial aviation: a user-requirement analysis with commercial aviation pilots

Virtual Reality - Augmented reality (AR) capable head-mounted displays (HMDs) have been proposed as technological enablers of several complex future flight concepts, which will bring accompanying...     

Investigating the use of moving windows to improve software effort prediction: a replicated study

To date most research in software effort estimation has not taken chronology into account when selecting projects for training and validation sets. A chronological split represents the use of a project’s starting and completion dates, such that any model that estimates effort for a new project p only uses as its training set projects that have been completed prior to p’s starting date. A study in 2009 (“S3”) investigated the use of chronological split taking into account a project’s age. The research question investigated was whether the use of a training set containing only the most recent past projects (a “moving window” of recent projects) would lead to more accurate estimates when compared to using the entire history of past projects completed prior to the starting date of a new project. S3 found that moving windows could improve the accuracy of estimates. The study described herein replicates S3 using three different and independent data sets. Estimation models were built using regression, and accuracy was measured using absolute residuals. The results contradict S3, as they do not show any gain in estimation accuracy when using windows for effort estimation. This is a surprising result: the intuition that recent data should be more helpful than old data for effort estimation is not supported. Several factors, which are discussed in this paper, might have contributed to such contradicting results. Some of our future work entails replicating this work using other datasets, to understand better when using windows is a suitable choice for software companies.     

Seasat-A: a retrospective

Abstract

This paper traces the highlights of the Seasat-A project from its earliest conceptual phases through the orbital operations phase. The primary objective of the project was to demonstrate the feasibility of remotely sensing fundamental oceanographic phenomena on a global, accurate, reliable and timely basis from an Earth-orbiting platform. The project successfully achieved its primary objective, in spite of a premature spacecraft failure, and thus provided a firm basis for the development of sensors for future ocean-observing space missions.     

Investigating the applicability of Agility assessment surveys: A case study

ContextAgile software development has become popular in the past decade without being sufficiently defined. The Agile principles can be instantiated differently which creates different perceptions of Agility. This has resulted in several frameworks being presented in the research literature to evaluate the level of Agility. However, the evidence of their actual use in practice is limited.ObjectiveThe objective is to identify online surveys that assess/profile Agility in practice, and to evaluate the surveys in an industrial setting.MethodThe Agility assessment surveys were identified through searching the web. Then, they were explored and two surveys were identified as most promising for our objective. The selected surveys were evaluated in a case study with three Agile teams in a software consultancy company.ResultsEach team and its customer separately judged the team's Agility. This outcome was compared with the two survey results in focus-group meetings, and finally one of the surveys was agreed to provide a more holistic assessment of Agility.ConclusionsDifferent surveys may judge Agility differently, which supports the viewpoint that Agile is not well defined. Therefore, practitioners must decide what Agile means to them and select the assessment survey that matches their definition.     

On-board preventive maintenance: a design-oriented analytic study for long-life applications

With respect to the long-life missions associated with NASA’s X2000 Advanced Deep-Space System Development Program, reliability implies a system’s continuous operation for many years in an unsurveyed radiation-intense environment. Further, the stringent constraints on the mass of a spacecraft and the power on-board create unprecedented challenges on the means for achieving the ultra-high mission reliability. In this paper, we present an approach to on-board preventive maintenance which rejuvenates a system by letting system components rotate between on-duty and off-duty shifts, slowing down a system’s aging process and thus enhancing mission reliability. By exploiting nondedicated system redundancy, hardware and software rejuvenation are realized simultaneously without significant performance penalty. Our design-oriented analysis confirms a potential for significant gains in mission reliability from on-board preventive maintenance and provides to us useful insights about the collective effect of age-dependent failure behavior, residual mission life, risk of unsuccessful maintenance and maintenance frequency on mission reliability.     

Measure, error and intrinsic fuzziness: a mathematical study of vagueness

 An operative method is proposed to evaluate degrees of membership, in situations when imprecise observations are made in a repeated way. The procedure is developed for limited subsets of the real axis R, but can be extended to R ^N . The analysis of a special case in R² shows the existence of sets whose membership function does not become a characteristic function, even when the inaccuracy of sampling vanishes and the observations are made in absolutely precise way.     

Investigating and promoting UX practice in industry: An experimental study

The efforts of addressing user experience (UX) in product development keep growing, as demonstrated by the proliferation of workshops and conferences bringing together academics and practitioners, who aim at creating interactive software able to satisfy their users. This special issue focuses on “Interplay between User Experience Evaluation and Software Development”, stating that the gap between human-computer interaction and software engineering with regard to usability has somewhat been narrowed. Unfortunately, our experience shows that software development organizations perform few usability engineering activities or none at all. Several authors acknowledge that, in order to understand the reasons of the limited impact of usability engineering and UX methods, and to try to modify this situation, it is fundamental to thoroughly analyze current software development practices, involving practitioners and possibly working from inside the companies. This article contributes to this research line by reporting an experimental study conducted with software companies. The study has confirmed that still too many companies either neglect usability and UX, or do not properly consider them. Interesting problems emerged. This article gives suggestions on how they may be properly addressed, since their solution is the starting point for reducing the gap between research and practice of usability and UX. It also provides further evidence on the value of the research method, called Cooperative Method Development, based on the collaboration of researchers and practitioners in carrying out empirical research; it has been used in a step of the performed study and has revealed to be instrumental for showing practitioners why to improve their development processes and how to do so.     

Mixed-fleet flying in commercial aviation: a joint cognitive systems perspective

Previous research investigating work activities and cognition in multi-crew airline flight decks has used a joint cognitive systems approach. However, is this approach suitable when some components—such as pilots—physically shift between differing aircraft, or joint cognitive systems? A current practice within the airline industry, known as mixed-fleet flying (MFF), allows pilots to fly aircraft of slightly differing configurations within the same working roster. The assumption held by aviation authorities is that pilots are permitted to fly in MFF configurations as long as relevant training occurs. Based on a cognitive anthropological study on pilots flying the same aircraft type—with differing flight deck configurations—we demonstrate that there are two different joint cognitive systems at work as each system involves different functional systems. The aim of this paper is to extend certain aspects of the joint cognitive systems approach to enable an analysis of real-world issues like MFF.