SMArDT modeling for automotive software testing

Efficient testing is a crucial prerequisite to engineer reliable automotive software successfully. However, manually deriving test cases from ambiguous textual requirements is costly and error-prone. Model-based software engineering captures requirements in structured, comprehensible, and formal models, which enables early consistency checking and verification. Moreover, these models serve as an indispensable basis for automated test case derivation. To facilitate automated test case derivation for automotive software engineering, we conducted a survey with testing experts of the BMW Group and conceived a method to extend the BMW Group's specification method for requirements, design, and test methodology by model-based test case derivation. Our method is realized for a variant of systems modeling language activity diagrams tailored toward testing automotive software and a model transformation to derive executable test cases. Hereby, we can address many of the surveyed practitioners' challenges and ultimately facilitate quality assurance for automotive software.     

Automatic generation of path conditions for concurrent timed systems

This paper presents an automatic method for calculating the path condition for programs with real time constraints. We model concurrent systems using timed transition systems and translate them into extended timed automata. Then an acyclic extended timed automaton is constructed and the path condition is calculated backwards over it. This method can be used for semiautomatic verification of a unit of code in isolation, i.e., without providing the exact values of parameters with which it is called. It can also be used for test case generation for real-time systems. Such a symbolic model checking algorithm was implemented previous in the PET system [E. Gunter, D. Peled, Unit checking: Symbolic model checking for a unit of code, Verification: Theory and Practice 2003, Essays Dedicated to Zohar Manna on the Occasion of his 64th Birthday, Lecture Notes in Computer Science, vol. 2772, Springer, 548–567] for untimed systems. Our method can also be used for the automatic generation of test cases for unit testing. The current generalization of the calculation of path condition for the timed case turns out to be quite tricky, since not only the selected path contributes to the path condition, but also timing constraints of alternative choices in the code.     

Timed Wp-method: testing real-time systems

Real-time systems interact with their environment using time constrained input/output signals. Examples of real-time systems include patient monitoring systems, air traffic control systems, and telecommunication systems. For such systems, a functional misbehavior or a deviation from the specified time constraints may have catastrophic consequences. Therefore, ensuring the correctness of real-time systems becomes necessary. Two different techniques are usually used to cope with the correctness of a software system prior to its deployment, namely, verification and testing. In this paper, we address the issue of testing real-time software systems specified as a timed input output automaton (TIOA). TIOA is a variant of timed automaton. We introduce the syntax and semantics of TIOA. We present the potential faults that can be encountered in a timed system implementation. We study these different faults based on TIOA model and look at their effects on the execution of the system using the region graph. We present a method for generating timed test cases. This method is based on a state characterization technique and consists of the following three steps: First, we sample the region graph using a suitable granularity, in order to construct a subautomaton easily testable, called grid automaton. Then, we transform the grid automaton into a nondeterministic timed finite state machine (NTFSM). Finally, we adapt the generalized Wp-method to generate timed test cases from NTFSM. We assess the fault coverage of our test cases generation method and prove its ability to detect all the possible faults. Throughout the paper, we use examples to illustrate the various concepts and techniques used in our approach.     

基于UML的软件Markov链使用模型构造研究

软件统计测试要求基于软件使用模型产生测试例对软件系统进行测试,并根据测试结果评价软件可靠性,是高可靠软件测试的重要组成部分.由于统一建模语言(unified modeling language,简称UML)已经成为事实上的面向对象标准建模语言,因此,从软件UML模型构造软件使用模型就成为面向对象软件统计测试的关键.为此,定义了加入统计测试约束的UML用例图、序列图以及用例执行顺序关系,为基于UML的软件统计测试提供了一个形式化描述基础.在此基础上,给出一个从软件UML模型构造软件Markov链使用模型的算法,并给出了自动化支持工具UMGen的类图结构,基于一个卫星控制系统,说明了所提出方法的有效性.     

Testing and Monitoring Model-based Generated Program

We propose an integrated framework to test and monitor code generated from hybrid models for embedded systems. The framework consists of the following elements: First, we create a testing automaton as a controlled environment to produce test traces achieving the desired testing criteria; Second, we synthesize a monitoring automaton from the behavior specification to check the run-time behavior of the tested system in response to the test traces; Finally, since both automata are encoded in the same language as the system model, the same code generator may be used to generate a tester and a monitor from the testing automaton and the monitoring automaton. The tester and the monitor may be linked as needed with the code generated from the system model. Our approach yields self-testing and self-monitoring code which may be run both on the simulation level and on the code level. We discuss our approach in its full details through an example on a SONY AIBO robotic dog.     

Generating semantically valid test inputs using constrained input grammars

ContextGenerating test cases based on software input interface is a black-box testing technique that can be made more effective by using structured input models such as input grammars. Automatically generating grammar-based test inputs may lead to structurally valid but semantically invalid inputs that may be rejected in early semantic error checking phases of a system under test.ObjectiveThis paper aims to introduce a method for specifying a grammar-based input model with the model’s semantic constraints to be used in the generation of positive test inputs. It is also important that the method can generate effective test suites based on appropriate grammar-based coverage criteria.MethodFormal specification of both input structure and input semantics provides the opportunity to use model instantiation techniques to create model instances that satisfy all specified constraints. The input interface of a subject system can be specified using a high-level specification scheme such as attribute grammars, and a transformation function from this scheme to an instantiable formal modeling language can generate the desired model instances.ResultsWe propose a declarative grammar-based input specification method that is based on a variation of attribute grammars and allows the user to specify input constraints in addition to input structure. The model can be instantiated automatically to generate structurally and semantically valid test inputs. The proposed method has the capability to specify test requirements and coverage criteria and use them to generate valid test suites that satisfy test coverage criteria requirements.ConclusionThe work presented in this paper provides a black-box test generation method for grammar-based software inputs that can automatically generate criteria-covering test suites.     

Relatively quantified constraint satisfaction

The constraint satisfaction problem (CSP) is a convenient framework for modelling search problems; the CSP involves deciding, given a set of constraints on variables, whether or not there is an assignment to the variables satisfying all of the constraints. This paper is concerned with the more general framework of quantified constraint satisfaction, in which variables can be quantified both universally and existentially. We study the relatively quantified constraint satisfaction problem (RQCSP), in which the values for each individual variable can be arbitrarily restricted. We give a complete complexity classification of the cases of the RQCSP where the types of constraints that may appear are specified by a constraint language.     

Reformulation of Global Constraints Based on Constraints Checkers

This article deals with global constraints for which the set of solutions can be recognized by an extended finite automaton whose size is bounded by a polynomial in n, where n is the number of variables of the corresponding global constraint. By reducing the automaton to a conjunction of signature and transition constraints we show how to systematically obtain an automaton reformulation. Under some restrictions on the signature and transition constraints, this reformulation maintains arc-consistency. An implementation based on some constraints as well as on the metaprogramming facilities of SICStus Prolog is available. For a restricted class of automata we provide an automaton reformulation for the relaxed case, where the violation cost is the minimum number of variables to unassign in order to get back to a solution.     

Model-based test generation using extended symbolic grammars

A novel, model-based test case generation approach for validating reactive systems, especially those supporting richly structured data inputs and/or interactions, is presented. Given an executable system model and an extended symbolic grammar specifying plausible system inputs, the approach performs a model-based simulation to (i) ensure the consistency of the model with respect to the specified inputs, and (ii) generate corresponding test cases for validating the system. The model-based simulation produces a state transition diagram (STD) automatically justifying the model runtime behaviors within the test case coverage. The STD can further be transformed to produce an evolved symbolic grammar, which can then be used to incrementally generate a refined set of test cases. As a case study, we present a live sequence chart (LSC) model-based test generator, named LCT in short, for LSC simulation and consistency testing. The evolved symbolic grammar produced by the simulator can either be used to generate practical test cases for software testing, or be further refined by applying our model-based test generation approach again with additional test coverage criteria. We further show that LSCs can also be used to specify and test certain temporal system properties during the model simulation. Their satisfaction, reflected in the STD, can either be served as a directive for selective test generation, or a basis for further temporal property model checking.     

Extending symbolic execution for automated testing of stored procedures

Stored procedures in database management systems are often used to implement complex business logic. Correctness of these procedures is critical for flawless working of the system. However, testing them remains difficult due to many possible database states and constraints on data. This leads to mostly manual testing. Newer tools offer automated execution for unit testing of stored procedures but the test cases are still written manually. We propose an approach of using dynamic symbolic execution for generating automated test cases and corresponding database states for stored procedures. We model the constraints on data imposed by the schema and the SQL statements, treating values in database tables as symbolic. We use SMT solver to find values that will drive the stored procedure on a particular execution path. We instrument the internal execution plans generated by PostgreSQL to extract constraints. We use Z3 to generate test cases consisting of table data and procedure inputs. Our evaluation using stored procedures from a large business application and various GitHub repositories quantifies the evidence of effectiveness of our technique by generating test cases that lead to schema constraint violations and user-defined exceptions.     

A unified framework for consistency check

Constraint Satisfaction Problem (CSP) involves finding values for variables to satisfy a set of constraints. Consistency check is the key technique in solving this class of problems. Past research has developed many algorithms for such a purpose, e.g., node consistency, are consistency, generalized node and arc consistency, specific methods for checking specific constraints, etc. In this article, an attempt is made to unify these algorithms into a common framework. This framework consists of two parts. the first part is a generic consistency check algorithm, which allows and encourages each individual constraint to be checked by its specific consistency methods. Such an approach provides a direct way of practical implementation of the CSP model for real problem-solving. the second part is a general schema for describing the handling of each type of constraint. the schema characterizes various issues of constraint handling in constraint satisfaction, and provides a common language for expressing, discussing, and exchanging constraint handling techniques. © 1995 John Wiley & Sons, Inc.     

Generating customized yet factually consistent information: a constraint satisfaction approach

We present an information customization framework that leverages a hybrid of adaptive hypermedia and intelligent techniques, in particular constraint satisfaction methods, to generate customized and factually consistent information based on a user profile. Information customization is modeled as a constraint satisfaction problem, whereby a solution is derived by (a) satisfying user-model constraints to select a user-specific set of ‘information snippets’; and (b) establishing inter-snippet consistency to ensure that all snippets are compatible with each other. Our approach takes the unique step of establishing factually consistency – via the satisfaction of inter-snippet constraints – between heterogeneous information snippets. A customized information package is generated by systematically synthesizing the set of user-specific and factually consistent information snippets. The featured information customization framework incorporates variations of various search and constraint satisfaction methods. The work is applied in an E-Healthcare setting leading to the generation of customized healthcare information.     

Search-based automated testing of continuous controllers: Framework,tool support,and case studies

ContextTesting and verification of automotive embedded software is a major challenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink models, generating code from the models, and deploying the resulting code on hardware devices. Automotive software artifacts are subject to three rounds of testing corresponding to the three production stages: Model-in-the-Loop (MiL), Software-in-the-Loop (SiL) and Hardware-in-the-Loop (HiL) testing.ObjectiveWe study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed loop system. These controllers make up a large part of automotive functions, and monitor and control the operating conditions of physical devices.MethodWe identify a set of requirements characterizing the behavior of continuous controllers, and develop a search-based technique based on random search, adaptive random search, hill climbing and simulated annealing algorithms to automatically identify worst-case test scenarios which are utilized to generate test cases for these requirements.ResultsWe evaluated our approach by applying it to an industrial automotive controller (with 443 Simulink blocks) and to a publicly available controller (with 21 Simulink blocks). Our experience shows that automatically generated test cases lead to MiL level simulations indicating potential violations of the system requirements. Further, not only does our approach generate significantly better test cases faster than random test case generation, but it also achieves better results than test scenarios devised by domain experts. Finally, our generated test cases uncover discrepancies between environment models and the real world when they are applied at the Hardware-in-the-Loop (HiL) level.ConclusionWe propose an automated approach to MiL testing of continuous controllers using search. The approach is implemented in a tool and has been successfully applied to a real case study from the automotive domain.     

Program Understanding as Constraint Satisfaction: Representation and Reasoning Techniques

The process of understanding a source code in a high-level programming language involves complex computation. Given a piece of legacy code and a library of program plan templates, understanding the code corresponds to building mappings from parts of the source code to particular program plans. These mappings could be used to assist an expert in reverse engineering legacy code, to facilitate software reuse, or to assist in the translation of the source into another programming language. In this paper we present a model of program understanding using constraint satisfaction. Within this model we intelligently compose a partial global picture of the source program code by transforming knowledge about the problem domain and the program itself into sets of constraints. We then systematically study different search algorithms and empirically evaluate their performance. One advantage of the constraint satisfaction model is its generality; many previous attempts in program understanding could now be cast under the same spectrum of heuristics, and thus be readily compared. Another advantage is the improvement in search efficiency using various heuristic techniques in constraint satisfaction.     

Systematic scenario test case generation for nuclear safety systems

ContextThe current validation tests for nuclear software are routinely performed by random testing, which leads to uncertain test coverage. Moreover, validation tests should directly verify the system’s compliance with the original user’s needs. Unlike current model-based testing methods, which are generally based on requirements or design models, the proposed model is derived from the original user’s needs in text through domain-specific ontology, and then used to generate validation tests systematically.ObjectiveOur first goal is to develop an objective, repeatable, and efficient systematic validation test scheme that is effective for large systems, with analyzable test coverage. Our second goal is to provide a new model-based validation testing method that reflects the user’s original safety needs.MethodA model-based scenario test case generation for nuclear digital safety systems was designed. This was achieved by converting the scenarios described in natural language in a Safety Analysis Report (SAR) prepared by the power company for licensing review, to Unified Modeling Language (UML) sequence diagrams based on a proposed ontology of a related regulatory standard. Next, we extracted the initial environmental parameters and the described operational sequences. We then performed variations on these data to systematically generate a sufficient number of scenario test cases.ResultsTest coverage criteria, which are the equivalence partition coverage of initial environment, the condition coverage, the action coverage and the scenario coverage, were met using our method.ConclusionThe proposed model-based scenario testing can provide improved testing coverage than random testing. A test suite based on user needs can be provided.     

Modelling dynamic memory management in constraint-based testing

Constraint-based testing (CBT) is the process of generating test cases against a testing objective by using constraint solving techniques. When programs contain dynamic memory allocation and loops, constraint reasoning becomes challenging as new variables and new constraints should be created during the test data generation process. In this paper, we address this problem by proposing a new constraint model of C programs based on operators that model dynamic memory management. These operators apply powerful deduction rules on abstract states of the memory enhancing the constraint reasoning process. This allows to automatically generate test data respecting complex coverage objectives. We illustrate our approach on a well-known difficult example program that contains dynamic memory allocation/deallocation, structures and loops. We describe our implementation and provide preliminary experimental results on this example that show the highly deductive potential of the approach.     

LDPChecker一个实时和混成系统模型检验工具

混成系统是一类复杂系统,线性混成系统作为其重要子类,在形式方法中,人们通常使用线性混成自动机来对它建模．虽然线性混成自动机的模型检验问题总的来说还是不可判定的,但对于其中的正环闭合自动机．其对于线性时段性质的满足性能够通过线性规划方法加以检验．为了实现自动检验正环闭合自动机对线性时段性质的满足性,设计并实现了工具LDPChecker．工具LDPChecker能够识别正环闭合自动机并对其进行相应的检验,其主要特色在于它能够对实时和混成系统检验包含可达性在内的许多实时性质,并且能够自动给出诊断信息．     

Test generation from P systems using model checking

This paper presents some testing approaches based on model checking and using different testing criteria. First, test sets are built from different Kripke structure representations. Second, various rule coverage criteria for transitional, non-deterministic, cell-like P systems, are considered in order to generate adequate test sets. Rule based coverage criteria (simple rule coverage, context-dependent rule coverage and variants) are defined and, for each criterion, a set of LTL (Linear Temporal Logic) formulas is provided. A codification of a P system as a Kripke structure and the sets of LTL properties are used in test generation: for each criterion, test cases are obtained from the counterexamples of the associated LTL formulas, which are automatically generated from the Kripke structure codification of the P system. The method is illustrated with an implementation using a specific model checker, NuSMV.     

Deep learning-based extraction of construction procedural constraints from construction regulations

Construction procedural constraints are critical in facilitating effective construction procedure checking in practice and for various inspection systems. Nowadays, the manual extraction of construction procedural constraints is costly and time-consuming. The automatic extraction of construction procedural constraint knowledge (e.g., knowledge entities and interlinks/relationships between them) from regulatory documents is a key challenge. Traditionally, natural language processing is implemented using either rule-based or machine learning approaches. Limited efforts on rule-based extraction of construction regulations often rely on pre-defined vocabularies and involve heavy feature engineering. Based on characteristics of the knowledge expression of construction procedural constraints in Chinese regulations, this paper explores a hybrid deep neural network, combining the bidirectional long short-term memory (Bi-LSTM) and the conditional random field (CRF), for the automatic extraction of the qualitative construction procedural constraints. Based on the proposed deep neural network, the recognition and extraction of named entities and relations between them are realized. Unlike existing information extraction research efforts using rule-based methods, the proposed hybrid deep learning approach can be applied without complex handcrafted features engineering. Besides, the long distance dependency relationships between different entities in regulations are considered. The model implementation results demonstrate the good performance of the end-to-end deep neural network in the extraction of construction procedural constraints. This study can be considered as one of the early explorations of knowledge extraction from construction regulations.     

ProTest: An Automatic Test Environment for B Specifications

We present ProTest, an automatic test environment for B specifications. B is a model-oriented notation where systems are specified in terms of abstract states and operations on abstract states. ProTest first generates a state coverage graph of a B specification through exhaustive model checking, and the coverage graph is traversed to generate a set of test cases, each being a sequence of B operations. For the model checking to be exhaustive, some transformations are applied to the sets used in the B machine. The approach also works if it is not exhaustive; one can stop at any point in time during the state space exploration and generate test cases from the coverage graph obtained so far. ProTest then simultaneously performs animation of the B machine and the execution of the corresponding implementation in Java, and assigns verdicts on the test results. With some restrictions imposed on the B operations, the whole of the testing process is performed mechanically. We demonstrate the efficacy of our test environment by performing a small case study from industry. Furthermore, we present a solution to the problem of handling non-determinism in B operations.