Evolutionary functional testing

The development and testing of software-based systems is an essential activity for the automotive industry. The 50–70 software-based systems with different complexities and developed by various suppliers are installed in today's premium vehicles, communicating with each other via different bus systems. The integration and testing of systems of this complexity is a very challenging task. The aim of testing is to detect faults in the systems under test and to convey confidence in the correct functioning of the systems if no faults are found during comprehensive testing. Faults not found in the different testing phases could have significant consequences that range from customer dissatisfaction to damage of physical property or, in safety-relevant areas, even to the endangering of human lives. Therefore, the thorough testing of developed systems is essential. Evolutionary testing tries to improve the effectiveness and efficiency of the testing process by transforming testing objectives into search problems, and applying evolutionary computation in order to solve them. The most important class of testing methods is functional testing. However, functional testing is difficult to automate by evolutionary testing. This work will describe how evolutionary testing could be applied to automate functional testing in general and the testing of complex automotive systems in particular. It presents two case studies and shows how evolutionary testing is effective at finding faults in the functional behaviour of these systems. In addition, a quantitative comparison with manual and random test case selection is done for one application.     

Ranking and comparing evolutionary algorithms with Hellinger-TOPSIS

When multiple algorithms are applied to multiple benchmarks as it is common in evolutionary computation, a typical issue rises, how can we rank the algorithms? It is a common practice in evolutionary computation to execute the algorithms several times and then the mean value and the standard deviation are calculated. In order to compare the algorithms performance it is very common to use statistical hypothesis tests. In this paper, we propose a novel alternative method based on the Technique for Order Preference by Similarity to Ideal Solution (TOPSIS) to support the performance comparisons. In this case, the alternatives are the algorithms and the criteria are the benchmarks. Since the standard TOPSIS is not able to handle the stochastic nature of evolutionary algorithms, we apply the Hellinger-TOPSIS, which uses the Hellinger distance, for algorithm comparisons. Case studies are used to illustrate the method for evolutionary algorithms but the approach is general. The simulation results show the feasibility of the Hellinger-TOPSIS to find out the ranking of algorithms under evaluation.     

A practical model‐based statistical approach for generating functional test cases: application in the automotive industry

With the growing complexity of industrial software applications, industrials are looking for efficient and practical methods to validate the software. This paper develops a model‐based statistical testing approach that automatically generates online and offline test cases for embedded software. It discusses an integrated framework that combines solutions for three major software testing research questions: (i) how to select test inputs; (ii) how to predict the expected results of a test; and (iii) when to stop testing software. The automatic selection of test inputs is based on a stochastic test model that accounts for the main particularity of embedded software: time sensitivity. Software test practitioners may design one or more test models when they generate random, user‐oriented, or fault‐oriented test inputs. A formal framework integrating existing and appropriate specification techniques was developed for the design of automated test oracles (executable software specifications) and the formal measurement of functional coverage. The decision to stop testing software is based on both test coverage objectives and cost constraints. This approach was tested on two representative case studies from the automotive industry. The experiment was performed at unit testing level in a simulated environment on a host personal computer (automatic test execution). The two software functionalities tested had previously been unit tested and validated using the test design approach conventionally used in the industry. Applying the proposed model‐based statistical testing approach to these two case studies, we obtained significant improvements in performing functional unit testing in a real and complex industrial context: more bugs were detected earlier and in a shorter time. Copyright © 2012 John Wiley & Sons, Ltd.     

软件结构化测试用例自动生成方法

软件测试和软件开发对软件质量具有同等重要的意义,其工作量很大,且其许多过程又适于自动化,于是软件测试自动化应运而生。测试用例的生成是软件测试过程的重要环节,近年来,人们开始尝试将人工智能技术运用于测试用例生成问题中,产生进化测试思想方法。文中对基于结构化的测试用例自动生成方法研究现状作了介绍,重点介绍了进化测试并对其基于不同进化计算算法的应用情况进行了比较和总结。应用结果表明进化测试方法是当前最为高效的软件测试用例自动生成方法。最后对进化测试进一步的研究方向提出了看法。     

Evolutionary Multiobjective Design in Automotive Development

This paper describes the use of evolutionary algorithms to solve multiobjective optimization problems arising at different stages in the automotive design process. The problems considered are black box optimization scenarios: definitions of the decision space and the design objectives are given, together with a procedure to evaluate any decision alternative with regard to the design objectives, e.g., a simulation model. However, no further information about the objective function is available. In order to provide a practical introduction to the use of multiobjective evolutionary algorithms, this article explores the three following case studies: design space exploration of road trains, parameter optimization of adaptive cruise controllers, and multiobjective system identification. In addition, selected research topics in evolutionary multiobjective optimization will be illustrated along with each case study, highlighting the practical relevance of the theoretical results through real-world application examples. The algorithms used in these studies were implemented based on the PISA (Platform and Programming Language Independent Interface for Search Algorithm) framework. Besides helping to structure the presentation of different algorithms in a coherent way, PISA also reduces the implementation effort considerably.     

ADC standard and testing in Japanese industry

This paper presents case studies of ADC standards and testing in Japanese industry from the viewpoints of ADC users, an ADC designer and an ADC tester manufacturer. These studies report the opinions of experienced ADC engineers in three Japanese companies (Iwatsu Electric, Agilent Technologies Japan, Sanyo Electric) about standards for ADCs. In most Japanese companies, ADC standards belong to engineers, and there is no comprehensive document that can be used as a Bible of ADC standards and testing methods, so ADC standards, which satisfy the requirements of engineers in industry, would be very useful.     

A review of the use of asset information standards for collaboration in the process industry

In this article the use of asset information standards for collaboration in the process industry is reviewed based on a survey of the literature and two case studies. The investigation shows that the process industry appears to have had only limited success in introducing such standards so far, despite significant efforts. Since information hand-over between asset life cycle phases is important, lack of information standardization suggests that collaboration costs are higher than necessary. Reported causes can be grouped into standard related causes (slow development of standards, stability, complexity, cost, quality/ontological problems), organization related causes (lack of direct financial incentives, organizational readiness, resistance to change) and business environment related causes (legal aspects, level of adoption, limited governmental enforcement and a lack of dominant actors in the process industry). It is also shown that initial local configuration of a standard may lead to successful acceptance of the standards, but may hinder later external use. The contribution of this article is insight into the use of asset information standards and the causes for lack of pervasiveness. This is necessary for improving the use of standards in collaboration in the process industry. The article concludes by suggesting future research directions.     

Testing restorable systems: formal definition and heuristic solution based on river formation dynamics

Given a finite state machine denoting the specification of a system, finding some short interaction sequences capable of reaching some/all states or transitions of this machine is a typical goal in testing methods. If these sequences are applied to an implementation under test, then equivalent states or transitions would be reached and observed in the implementation—provided that the implementation were actually defined as the specification. We study the problem of finding such sequences in the case where configurations previously traversed can be saved and restored (at some cost). In general, this feature enables sequences to reach the required parts of the machine in less time, because some repetitions can be avoided. However, we show that finding optimal sequences in this case is an NP-hard problem. We propose an heuristic method to approximately solve this problem based on an evolutionary computation approach, in particular river formation dynamics (RFD). Given finite state machine specifications and sets of states/transitions to be reached, we apply RFD to construct testing plans reaching these configurations. Experimental results show that being able to load previously traversed states generally reduces the time needed to cover the target configurations.     

Evaluating evolutionary testability for structure-oriented testing with software measurements

Test case design is the most important test activity with respect to test quality. For this reason, a large number of testing methods have been developed to assist the tester with the definition of appropriate, error-sensitive test data. Evolutionary testing is a promising approach for automating structure-oriented test case design completely. In many experiments, high coverage degrees were reached using evolutionary testing. However, evolutionary testing is not equally well applicable to different test objects. For example, evolutionary testing of a test object with complex predicates might fail. In order to assess the difficulty of a test object for evolutionary testing, software measures can be used. The knowledge provided by software measurements could lead to a significant increase in efficiency of evolutionary testing. In this paper, we investigate the suitability of structure-based complexity measures for the assessment of whether or not evolutionary testing can be performed successfully for a given test object.     

改进的粒子群算法多模态生物医学图像配准

多模态生物医学图像配准在医疗诊断、治疗方案的制定,以及身体机能的研究等方面起到越来越大的作用。如何将这些多模态信息融合在一起是目前研究的重点,目前,该融合主要基于图像强度信息的配准方法。该类方法通过最大化化图像间的相似度函数达到配准的目的,但配准过程中使用往往会出现参数变化非凸且不光滑的现象,因而,传统的局部最优方法通常不能得到较好的结果。粒子群算法是一种全局寻优算法,但传统的方法中受初始值的选取以及当前全局最优点的影响,易陷入局部最优。本文对其进行改进,使得即使在初始值离准确值较远时也能得到全局最优,并将该方法用于多模态医学图像配准中,得到了较好的结果。     

Industry case studies in the use of immersive virtual assembly

In this paper, we report on two engineering case studies that have been conducted as part of a Virtual Assembly Technology Consortium. The objectives of the case studies were to determine if immersive virtual assembly capabilities allow industry assembly situations to be modelled and studied realistically, and to demonstrate the downstream value of the virtual assembly capabilities in areas such as ergonomics, assembly installation, process planning, installation, and serviceability. What is of special significance is that instead of modelling simplified problems or perceived representative situations, the case studies were constructed from actual assembly floor projects and situations encountered at industry member sites and with considerable participation from industry engineers and manufacturing shop floor personnel. Based on the success of the case studies, the consortium members inferred that virtual assembly methods are poised to move out of the realm of special projects and test scenarios to deployment in the actual design and manufacturing cycle. However, in order to be truly accepted in industry, there are still issues to be addressed in terms of ease of use, portability of the applications, and preparation of the models for the evaluations. Thus, the case studies added a new dimension to the exploration and understanding of how this new technology could be of practical value in industry.     

Search Algorithms for Regression Test Case Prioritization

Regression testing is an expensive, but important, process. Unfortunately, there may be insufficient resources to allow for the reexecution of all test cases during regression testing. In this situation, test case prioritization techniques aim to improve the effectiveness of regression testing by ordering the test cases so that the most beneficial are executed first. Previous work on regression test case prioritization has focused on greedy algorithms. However, it is known that these algorithms may produce suboptimal results because they may construct results that denote only local minima within the search space. By contrast, metaheuristic and evolutionary search algorithms aim to avoid such problems. This paper presents results from an empirical study of the application of several greedy, metaheuristic, and evolutionary search algorithms to six programs, ranging from 374 to 11,148 lines of code for three choices of fitness metric. The paper addresses the problems of choice of fitness metric, characterization of landscape modality, and determination of the most suitable search technique to apply. The empirical results replicate previous results concerning greedy algorithms. They shed light on the nature of the regression testing search space, indicating that it is multimodal. The results also show that genetic algorithms perform well, although greedy approaches are surprisingly effective, given the multimodal nature of the landscape     

Classification knowledge discovery in mold tooling test using decision tree algorithm

The scale of Taiwan’s mold industry was ranked the sixth in the world. But, under the global competitive pressure, Taiwan has lost its competitive advantage gradually. The only chance of Taiwan’s mold industry lies in improving the competitive abilities in product research, development and design. In mold manufacturing cycle, mold tooling test plays a very important role at accelerating the speed of production. An experienced engineer can minimize the error rate of mold tooling test according to his rich experiences in parameter adjustment. However, this experience is mostly implicit without theoretical basis and its knowledge is difficult to be transmitted. Benefiting from the well development of data mining technologies, this study aimed at constructing an intelligent classification knowledge discovery system for mold tooling test based on decision tree algorithm, so as to explore and accumulate the experimental knowledge for the use of Taiwan’s mold industry. This study took the only high-alloy steel manufacturer in Taiwan for case study, and performed system validation with 66 record data. The results showed the accuracy rates of prediction of training data and testing data are 97.6 and 86.9%, respectively. In addition, this study explored two classification knowledge rules and proposed concrete proposals for tooling test parameter adjustment. Moreover, this study provided two ways, rule verification and effectiveness comparison of four mining algorithms, to conduct model verification. The experimental results showed the decision tree algorithm has an excellent discriminatory power of classification and is able to provide clear and simple reference rules for decisions.     

Methodologies and website development: a survey of practice

Website development work is a growing aspect of the IT activities within many organisations. However, the manner in which website development actually takes place within organisations is still largely uncertain. In this paper, we examine the results of a research exercise involving case studies in 25 UK organisations aimed at investigating the way in which website development activities are currently carried out within UK organisations. In particular, this paper discusses the activities that are typically involved in website development projects, and the techniques and standards actually used for website development found within 25 organisations studied. One of the main findings of the research project was that in 25 organisations studied there was only limited use of formalised website design techniques (mainly hierarchy charts and storyboards). However, roughly half of the organisations studied did use some form of website layout standards. Website documentation was only produced in roughly a third of the organisations and only roughly a quarter of the organisations had any formalised website testing procedures in place.     

Achieving software quality with testing coverage measures

Coverage testing helps the tester create a thorough set of tests and gives a measure of test completeness. The concepts of coverage testing are well-described in the literature. However, there are few tools that actually implement these concepts for standard programming languages, and their realistic use on large-scale projects is rare. In this article, we describe the uses of a dataflow coverage-testing tool for C programs-called ATAC for Automatic Test Analysis for C³-in measuring, controlling,and understanding the testing process. We present case studies of two real-world software projects using ATAC. The first study involves 12 program versions developed by a university/industry fault-tolerant software project for a critical automatic-flight-control system. The second study involves a Bellcore project of 33 program modules. These studies indicate that coverage analysis of programs during testing not only gives a clear measure of testing quality but also reveals important aspects of software structure. Understanding the structure of a program, as revealed in coverage testing, can be a significant component in confident assessment of overall software quality     

The role of information support systems in the joint optimization of work systems

Sociotechnical approach preaches the affinity of the social and the technical organization but divides organizations into social and technical subsystems. Thus, it has failed to anticipate the possibilities of new technologies within the social system. The social subsystem of a modern sociotechnical system does not consist of mere human beings but combines people and the technological artifacts they use. In this article, we discuss one such technology, information support, and how it can be used to expand functional redundancy of a sociotechnical system, even the cognitive redundancy of individual people. However, such technologies have often been applied specifically to limit functional redundancy of the sociotechnical system, that is, to reduce communication, learning, and utilization of human talent. The discussion is based on findings from case studies on multimedia‐based interactive task support systems used in lightweight assembly industry in Finland and in Denmark. The case studies provide both justification to the information support approach and questions to be addressed in the future. For example, why did the introduction of so many interactive task support systems fail? © 2000 John Wiley & Sons, Inc.     

Evolutionary testing of autonomous software agents

A system built in terms of autonomous software agents may require even greater correctness assurance than one that is merely reacting to the immediate control of its users. Agents make substantial decisions for themselves, so thorough testing is an important consideration. However, autonomy also makes testing harder; by their nature, autonomous agents may react in different ways to the same inputs over time, because, for instance they have changeable goals and knowledge. For this reason, we argue that testing of autonomous agents requires a procedure that caters for a wide range of test case contexts, and that can search for the most demanding of these test cases, even when they are not apparent to the agents’ developers. In this paper, we address this problem, introducing and evaluating an approach to testing autonomous agents that uses evolutionary optimisation to generate demanding test cases. We propose a methodology to derive objective (fitness) functions that drive evolutionary algorithms, and evaluate the overall approach with two simulated autonomous agents. The obtained results show that our approach is effective in finding good test cases automatically.     

Analyzing the relationships between inspections and testing to provide a software testing focus

ContextQuality assurance effort, especially testing effort, is frequently a major cost factor during software development. Consequently, one major goal is often to reduce testing effort. One promising way to improve the effectiveness and efficiency of software quality assurance is the use of data from early defect detection activities to provide a software testing focus. Studies indicate that using a combination of early defect data and other product data to focus testing activities outperforms the use of other product data only. One of the key challenges is that the use of data from early defect detection activities (such as inspections) to focus testing requires a thorough understanding of the relationships between these early defect detection activities and testing. An aggravating factor is that these relationships are highly context-specific and need to be evaluated for concrete environments.ObjectiveThe underlying goal of this paper is to help companies get a better understanding of these relationships for their own environment, and to provide them with a methodology for finding relationships in their own environments.MethodThis article compares three different strategies for evaluating assumed relationships between inspections and testing. We compare a confidence counter, different quality classes, and the F-measure including precision and recall.ResultsOne result of this case-study-based comparison is that evaluations based on the aggregated F-measures are more suitable for industry environments than evaluations based on a confidence counter. Moreover, they provide more detailed insights about the validity of the relationships.ConclusionWe have confirmed that inspection results are suitable data for controlling testing activities. Evaluated knowledge about relationships between inspections and testing can be used in the integrated inspection and testing approach In²Test to focus testing activities. Product data can be used in addition. However, the assumptions have to be evaluated in each new context.     

Compositionality and correctness of fault tolerant patterns in HOL4

In the development of critical systems, it is common practice to make use of redundancy in order to achieve higher levels of reliability. There are well established design patterns that introduce redundancy and that are widely documented and adopted by the industry. However there have been few attempts to formally verify them. In this work, we modelled in the HOL4 system such design patterns, which we call here fault tolerant patterns. We illustrate our approach by modelling three classical fault tolerant patterns: Homogeneous Redundancy, Heterogeneous Redundancy and Triple Modular Redundancy. Our model takes into account that the original system (without redundancy) computes a certain function with some delay and is amenable to random failures.We proved that our fault tolerant patterns preserve the behaviour of its replicated subsystems. The notion of correctness adopted makes use of interval arithmetic and is restricted to functional behaviour. Timing is not regarded as part of the functional behaviour in this work. Therefore, real-time systems are not the focus of our approach. We also proved that our fault tolerant patterns are compositional in the sense that we can apply fault tolerant patterns consecutively and for an arbitrary number of times. The consecutive application of our patterns still results in a system that computes a certain function with some delay and amenable to random failures. We developed a case study that verifies that a fault tolerant pattern applied to a simplified avionic Elevator Control System preserves its original behaviour. This work was done in collaboration with the Brazilian aircraft manufacturer Embraer.