航天嵌入式软件测试用例典型设计缺陷研究

作为动态测试充分性的基本评价指标,覆盖率分析只能帮助修正因输入不足而导致的测试用例设计缺陷。针对航天嵌入式软件测试过程中不影响覆盖率统计结果的用例设计缺陷,从测试步骤和预期结果两大测试用例核心要素开展研究,提出十个典型缺陷,分别予以分析,并进行缺陷修正。工程实践证明,这些缺陷的发生率高,具有典型性;修正这些缺陷后,可以有效检出软件设计缺陷;与用例执行后的覆盖率统计数据分析相结合,可以有效提高测试充分性。     

Mutant generation for embedded systems using kernel-based software and hardware fault simulation

Context

Mutation testing is a fault-injection-based technique to help testers generate test cases for detecting specific and predetermined types of faults.

Objective

Before mutation testing can be effectively applied to embedded systems, traditional mutation testing needs to be modified. To inject a fault into an embedded system without causing any system failure or hardware damage is a challenging task as it requires some knowledge of the underlying layers such as the kernel and the corresponding hardware.

Method

We propose a set of mutation operators for embedded systems using kernel-based software and hardware fault simulation. These operators are designed for software developers so that they can use the mutation technique to test the entire system after the software is integrated with the kernel and hardware devices.

Results

A case study on a programmable logic controller for a digital reactor protection system in a nuclear power plant is conducted. Our results suggest that the proposed mutation operators are useful for fault-injection and this is evidenced by the fact that faults not injected by us were discovered in the subject software as a result of the case study.

Conclusion

We conclude that our mutation operators are useful for integration testing of an embedded system.     

A state-based approach to integration testing based on UML models

Correct functioning of object-oriented software depends upon the successful integration of classes. While individual classes may function correctly, several new faults can arise when these classes are integrated together. In this paper, we present a technique to enhance testing of interactions among modal classes. The technique combines UML collaboration diagrams and statecharts to automatically generate an intermediate test model, called SCOTEM (State COllaboration TEst Model). The SCOTEM is then used to generate valid test paths. We also define various coverage criteria to generate test paths from the SCOTEM model. In order to assess our technique, we have developed a tool and applied it to a case study to investigate its fault detection capability. The results show that the proposed technique effectively detects all the seeded integration faults when complying with the most demanding adequacy criterion and still achieves reasonably good results for less expensive adequacy criteria.     

A property-based testing framework for encryption programs

In recent years, a variety of encryption algorithms were proposed to enhance the security of software and systems. Validating whether encryption algorithms are correctly implemented is a challenging issue. Software testing delivers an effective and practical solution, but it also faces the oracle problem (that is, under many practical situations, it is impossible or too computationally expensive to know whether the output for any given input is correct). In this paper, we propose a property-based approach to testing encryption programs in the absence of oracles. Our approach makes use of the so-called metamorphic properties of encryption algorithms to generate test cases and verify test results. Two case studies were conducted to illustrate the proposed approach and validate its effectiveness. Experimental results show that even without oracles, the proposed approach can detect nearly 50% inserted faults with at most three metamorphic relations (MRs) and fifty test cases.     

A mapping study on testing non-testable systems

The terms “Oracle Problem” and “Non-testable system” interchangeably refer to programs in which the application of test oracles is infeasible. Test oracles are an integral part of conventional testing techniques; thus, such techniques are inoperable in these programs. The prevalence of the oracle problem has inspired the research community to develop several automated testing techniques that can detect functional software faults in such programs. These techniques include N-Version testing, Metamorphic Testing, Assertions, Machine Learning Oracles, and Statistical Hypothesis Testing. This paper presents a Mapping Study that covers these techniques. The Mapping Study presents a series of discussions about each technique, from different perspectives, e.g. effectiveness, efficiency, and usability. It also presents a comparative analysis of these techniques in terms of these perspectives. Finally, potential research opportunities within the non-testable systems problem domain are highlighted within the Mapping Study. We believe that the aforementioned discussions and comparative analysis will be invaluable for new researchers that are attempting to familiarise themselves with the field, and be a useful resource for practitioners that are in the process of selecting an appropriate technique for their context, or deciding how to apply their selected technique. We also believe that our own insights, which are embedded throughout these discussions and the comparative analysis, will be useful for researchers that are already accustomed to the field. It is our hope that the potential research opportunities that have been highlighted by the Mapping Study will steer the direction of future research endeavours.     

A formal analysis of the subsume relation between software testadequacy criteria

Software test adequacy criteria are rules to determine whether a software system has been adequately tested. A central question in the study of test adequacy criteria is how they relate to fault detecting ability. We identify two idealized software testing scenarios. In the first scenario, which we call prior testing scenario, software testers are provided with an adequacy criterion in addition to the software under test. The knowledge of the adequacy criterion is used to generate test cases. In the second scenario, which we call posterior testing scenario, software testers are not provided with the knowledge of adequacy criterion. The criterion is only used to decide when to stop the generation of test cases. In 1993, Frankl and Weyuker proved that the subsume relation between software test adequacy criteria does not guarantee better fault detecting ability in the prior testing scenario. We investigate the posterior testing scenario and prove that in this scenario the subsume relation does guarantee a better fault detecting ability. Two measures of fault detecting ability will be used, the probability of detecting faults and the expected number of exposed errors     

基于Keil C51的嵌入式软件外设虚拟化设计与实现

航空航天控制领域嵌入式软件测试主要存在软件运行物理环境受限和软件测试覆盖性不足等问题,为解决以上问题,对嵌入式软件的外部设备进行了研究,构建了数字化测试平台替代实物环境的方案,设计并完成了平台的总线1553B、总线RS422、AD采集、I/O等外部设备;并模拟了平台嵌入式软件真实的运行环境,使测试过程不受实物环境制约,测试激励的注入不受任何限制,保障了测试的充分性;最后,以某电源下位机测试过程为例,实现了故障注入、边界测试,验证了平台在嵌入式软件测试中的有效性和可靠性。     

Evaluating Test Suites and Adequacy Criteria Using Simulation-Based Models of Distributed Systems

Test adequacy criteria provide the engineer with guidance on how to populate test suites. While adequacy criteria have long been a focus of research, existing testing methods do not address many of the fundamental characteristics of distributed systems, such as distribution topology, communication failure, and timing. Furthermore, they do not provide the engineer with a means to evaluate the relative effectiveness of different criteria, nor the relative effectiveness of adequate test suites satisfying a given criterion. This paper makes three contributions to the development and use of test adequacy criteria for distributed systems: (1) a testing method based on discrete-event simulations; (2) a fault-based analysis technique for evaluating test suites and adequacy criteria; and (3) a series of case studies that validate the method and technique. The testing method uses a discrete-event simulation as an operational specification of a system, in which the behavioral effects of distribution are explicitly represented. Adequacy criteria and test cases are then defined in terms of this simulation-based specification. The fault-based analysis involves mutation of the simulation-based specification to provide a foil against which test suites and the criteria that formed them can be evaluated. Three distributed systems were used to validate the method and technique, including DNS, the Domain Name System.     

Automating unit and integration testing with partial oracles

The oracle problem is an essential part in current research on automating software tests. Partial oracles seem to be a viable solution, but their suitability for different testing steps and general applicability for various systems remains still to be shown. This paper presents a study in which partial oracles are applied in order to automatically test a jpeg2000 encoder as an example for a modular software system with several integrated units and components. The effectiveness of the partial oracles is measured by means of mutation analysis to determine their adequacy for both unit and integration testing. Additionally, the paper presents possibilities of improving the effectiveness as well as the efficiency of the employed partial oracles. It shows how the knowledge of certain characteristics of the system to be tested, such as linearity or time-invariance, may lead to a better choice of partial oracles and thus to an improved effectiveness and efficiency.     

A methodology of testing high-level Petri nets

Petri nets have been extensively used in the modelling and analysis of concurrent and distributed systems. The verification and validation of Petri nets are of particular importance in the development of concurrent and distributed systems. As a complement to formal analysis techniques, testing has been proven to be effective in detecting system errors and is easy to apply. An open problem is how to test Petri nets systematically, effectively and efficiently. An approach to solve this problem is to develop test criteria so that test adequacy can be measured objectively and test cases can be generated efficiently, even automatically. In this paper, we present a methodology of testing high-level Petri nets based on our general theory of testing concurrent software systems. Four types of testing strategies are investigated, which include state-oriented testing, transition-oriented testing, flow-oriented testing and specification-oriented testing. For each strategy, a set of schemes to observe and record testing results and a set of coverage criteria to measure test adequacy are defined. The subsumption relationships and extraction relationships among the proposed testing methods are systematically investigated and formally proved.     

IPSETFUL: an iterative process of selecting test cases for effective fault localization by exploring concept lattice of program spectra

Fault localization is an important and challenging task during software testing. Among techniques studied in this field, program spectrum based fault localization is a promising approach. To perform spectrum based fault localization, a set of test oracles should be provided, and the effectiveness of fault localization depends highly on the quality of test oracles. Moreover, their effectiveness is usually affected when multiple simultaneous faults are present. Faced with multiple faults it is difficult for developers to determine when to stop the fault localization process. To address these issues, we propose an iterative fault localization process, i.e., an iterative process of selecting test cases for effective fault localization (IPSETFUL), to identify as many faults as possible in the program until the stopping criterion is satisfied. It is performed based on a concept lattice of program spectrum (CLPS) proposed in our previous work. Based on the labeling approach of CLPS, program statements are categorized as dangerous statements, safe statements, and sensitive statements. To identify the faults, developers need to check the dangerous statements. Meantime, developers need to select a set of test cases covering the dangerous or sensitive statements from the original test suite, and a new CLPS is generated for the next iteration. The same process is proceeded in the same way. This iterative process ends until there are no failing tests in the test suite and all statements on the CLPS become safe statements. We conduct an empirical study on several subject programs, and the results show that IPSETFUL can help identifymost of the faults in the program with the given test suite. Moreover, it can save much effort in inspecting unfaulty program statements compared with the existing spectrum based fault localization techniques and the relevant state of the art technique.     

浮力调节软件的单元测试技术研究

为了提高嵌入式软件的单元测试效率,同时能达到很好的测试效果,针对嵌入式软件制定单元测试充分性准则和单元测试策略,并使用测试工具对嵌入式软件进行单元测试具有实际意义;以浮力调节软件为例,通过研究基于控制流的单元测试充分性准则,结合浮力调节软件的单元测试需求,提出了针对浮力调节软件的单元测试充分性准则;同时,考虑到圈复杂度和函数节点数对函数正确实现的影响,制定了基于优先级的单元测试策略;利用自动化单元测试工具Testbed搭建了浮力调节软件动态测试环境,通过代码覆盖率分析,帮助创建测试用例以达到单元测试充分性要求,从而实现了浮力调节软件单元测试自动化;在自动化测试工具的帮助下,结合单元测试充分性准则和单元测试策略,最终实现严格而高效的单元测试。     

Deadline analysis of interrupt-driven software

Real-time, reactive, and embedded systems are increasingly used throughout society (e.g., flight control, railway signaling, vehicle management, medical devices, and many others). For real-time, interrupt-driven software, timely interrupt handling is part of correctness. It is vital for software verification in such systems to check that all specified deadlines for interrupt handling are met. Such verification is a daunting task because of the large number of different possible interrupt arrival scenarios. For example, for a Z86-based microcontroller, there can be up to six interrupt sources and each interrupt can arrive during any clock cycle. Verification of such systems has traditionally relied upon lengthy and tedious testing; even under the best of circumstances, testing is likely to cover only a fraction of the state space in interrupt-driven systems. This paper presents the Zilog architecture resource bounding infrastructure (ZARBI), a tool for deadline analysis of interrupt-driven Z86-based software. The main idea is to use static analysis to significantly decrease the required testing effort by automatically identifying and isolating the segments of code that need the most testing. Our tool combines multiresolution static analysis and testing oracles in such a way that only the oracles need to be verified by testing. Each oracle specifies the worst-case execution time from one program point to another, which is then used by the static analysis to improve precision. For six commercial microcontroller systems, our experiments show that a moderate number of testing oracles are sufficient to do precise deadline analysis.     

线性独立路径覆盖率的软件测试充分性判别方法

软件测试充分性判别准则是决定一个软件系统是否已经被充分测试的停止准则,而充分性判别准则的关键是它的揭错能力。对充分性判别准则进行了形式化描述,并且讨论了充分性判别准则的性质及准则之间的比较方法。为了给保障软件测试充分性提供理论依据,提出了一个软件测试充分性的度量准则。     

一种面向AADL架构的模型测试方法

鉴于模型在软件系统开发中日趋重要的地位和AADL模型在嵌入式软件建模中的良好应用前景,为了在嵌入式软件系统开发前期保证AADL模型的质量,提出了一种基于模型测试的AADL架构验证方法;该方法应用马尔可夫链描述AADL架构的行为,然后根据得到的马尔可夫链模型以及系统设计要求标准生成相应的测试用例和测试预言,并通过测试用例执行输出和期望值的比较判断AADL模型的正确性,实现对系统AADL模型的测试;最后通过案例分析证明了该方法的有效性。     

一种基于模型的测试充分性评估方法

测试充分性评估通常采用覆盖率的方法来评估测试对软件特征的覆盖充分程度。如今,传统的充分性评估方法难以满足复杂软件的测试评估需求。首先,代码覆盖准则难以准确验证软件需求;其次,软件测试还需考虑软件不同特征对系统测试充分性的不同影响。对此,提出一种基于接口的建模方法和基于该模型的综合覆盖充分性评估方法。该方法根据软件接口说明,对系统功能进行特征抽取、建模,并对接口模型的测试用例进行不同层级的充分性评估,对评佑结果进行归一化处理,得到系统的综合测试充分性。通过案例表明,这种评佑方法能够反映功能的测试充分性,对测试用例的设计和优化有一定指导意义。     

On the adoption of MC/DC and control-flow adequacy for a tight integration of program testing and statistical fault localization

ContextTesting and debugging consume a significant portion of software development effort. Both processes are usually conducted independently despite their close relationship with each other. Test adequacy is vital for developers to assure that sufficient testing effort has been made, while finding all the faults in a program as soon as possible is equally important. A tight integration between testing and debugging activities is essential.ObjectiveThe paper aims at finding whether three factors, namely, the adequacy criterion to gauge a test suite, the size of a prioritized test suite, and the percentage of such a test suite used in fault localization, have significant impacts on integrating test case prioritization techniques with statistical fault localization techniques.MethodWe conduct a controlled experiment to investigate the effectiveness of applying adequate test suites to locate faults in a benchmark suite of seven Siemens programs and four real-life UNIX utility programs using three adequacy criteria, 16 test case prioritization techniques, and four statistical fault localization techniques. We measure the proportion of code needed to be examined in order to locate a fault as the effectiveness of statistical fault localization techniques. We also investigate the integration of test case prioritization and statistical fault localization with postmortem analysis.ResultThe main result shows that on average, it is more effective for a statistical fault localization technique to utilize the execution results of a MC/DC-adequate test suite than those of a branch-adequate test suite, and is in turn more effective to utilize the execution results of a branch-adequate test suite than those of a statement-adequate test suite. On the other hand, we find that none of the fault localization techniques studied can be sufficiently effective in suggesting fault-relevant statements that can fit easily into one debug window of a typical IDE.ConclusionWe find that the adequacy criterion and the percentage of a prioritized test suite utilized are major factors affecting the effectiveness of statistical fault localization techniques. In our experiment, the adoption of a stronger adequacy criterion can lead to more effective integration of testing and debugging.     

测试用例描述语言研究

软件测试是软件开发生命周期的一项重要活动，也是保证软件质量和可靠性的重要手段。测试用例是软件测试的核心和关键。而迄今为止，在测试领域，还没有统一的测试用例描述语言。虽然目前大量的测试工具引入了测试用例描述技术，但也是五花八门、各自为政，这成为提高软件测试的效率和软件测试复用程度的瓶颈。在对国内外的测试用例描述语言进行了广泛调研后，作了简要的总结。提出了设计测试用例描述语言的准则，同时阐述了测试用例描述技术在嵌入式软件测试领域的应用情况。     

基于目标码的测试覆盖不可达分析方法

基于目标码的测试覆盖率分析是软件测试过程的必要关键步骤,不可达分析能够保证测试的完整性和充分性.给出嵌入式软件基于覆盖率测试的分析过程,在嵌入式虚拟测试平台的基础上,对程序目标代码插桩,采用语句和分支覆盖率分析准则,将黑盒测试和白盒测试相结合,分析不可达分支语句引入机理,并通过具体的反汇编代码实例分析来验证这种测试方法的可行性和有效性.专门针对覆盖率不可达的分析可以有效验证软件功能,发现软件缺陷,进一步提升软件质量.     

A practical model‐based statistical approach for generating functional test cases: application in the automotive industry

With the growing complexity of industrial software applications, industrials are looking for efficient and practical methods to validate the software. This paper develops a model‐based statistical testing approach that automatically generates online and offline test cases for embedded software. It discusses an integrated framework that combines solutions for three major software testing research questions: (i) how to select test inputs; (ii) how to predict the expected results of a test; and (iii) when to stop testing software. The automatic selection of test inputs is based on a stochastic test model that accounts for the main particularity of embedded software: time sensitivity. Software test practitioners may design one or more test models when they generate random, user‐oriented, or fault‐oriented test inputs. A formal framework integrating existing and appropriate specification techniques was developed for the design of automated test oracles (executable software specifications) and the formal measurement of functional coverage. The decision to stop testing software is based on both test coverage objectives and cost constraints. This approach was tested on two representative case studies from the automotive industry. The experiment was performed at unit testing level in a simulated environment on a host personal computer (automatic test execution). The two software functionalities tested had previously been unit tested and validated using the test design approach conventionally used in the industry. Applying the proposed model‐based statistical testing approach to these two case studies, we obtained significant improvements in performing functional unit testing in a real and complex industrial context: more bugs were detected earlier and in a shorter time. Copyright © 2012 John Wiley & Sons, Ltd.