A survey of new trends in symbolic execution for software testing and analysis

Symbolic execution is a well-known program analysis technique which represents program inputs with symbolic values instead of concrete, initialized, data and executes the program by manipulating program expressions involving the symbolic values. Symbolic execution has been proposed over three decades ago but recently it has found renewed interest in the research community, due in part to the progress in decision procedures, availability of powerful computers and new algorithmic developments. We provide here a survey of some of the new research trends in symbolic execution, with particular emphasis on applications to test generation and program analysis. We first describe an approach that handles complex programming constructs such as input recursive data structures, arrays, as well as multithreading. Furthermore, we describe recent hybrid techniques that combine concrete and symbolic execution to overcome some of the inherent limitations of symbolic execution, such as handling native code or availability of decision procedures for the application domain. We follow with a discussion of techniques that can be used to limit the (possibly infinite) number of symbolic configurations that need to be analyzed for the symbolic execution of looping programs. Finally, we give a short survey of interesting new applications, such as predictive testing, invariant inference, program repair, analysis of parallel numerical programs and differential symbolic execution.     

Unit Test Data Generation for C Using Rule-Directed Symbolic Execution

Unit testing is widely used in software development. One important activity in unit testing is automatic test data generation. Constraint-based test data generation is a technique for automatic generation of test data, which uses symbolic execution to generate constraints. Unit testing only tests functions instead of the whole program, where individual functions typically have preconditions imposed on their inputs. Conventional symbolic execution cannot detect these preconditions, let alone converting these preconditions into constraints. To overcome these limitations, we propose a novel unit test data generation approach using rule-directed symbolic execution for dealing with functions with missing input preconditions. Rule-directed symbolic execution uses predefined rules to detect preconditions in the individual function, and generates constraints for inputs based on preconditions. We introduce implicit constraints to represent preconditions, and unify implicit constraints and program constraints into integrated constraints. Test data generated based on integrated constraints can explore previously unreachable code and help developers find more functional faults and logical faults. We have implemented our approach in a tool called CTS-IC, and applied it to real-world projects. The experimental results show that rule-directed symbolic execution can find preconditions (implicit constraints) automatically from an individual function. Moreover, the unit test data generated by our approach achieves higher coverage than similar tools and efficiently mitigates missing input preconditions problems in unit testing for individual functions.

     

An approach to creating concretized test scenarios within test automation technology for industrial software projects

The paper describes an approach for creation of effective test automation technology for industrial software projects based on a formal model of the system under test, performing its automatic symbolic verification of the model and generation of symbolic traces, then their automatic concretization, generation and execution of test suites. The technology includes means for test results analysis and tests correction and update. High quality of software products is ensured by a balance between static (verification) and dynamic (testing) analysis techniques starting from the earlier development phases and through automation of all phases of software development. Special emphasis is made on presentation of the algorithm of concretization and configuring test scenarios.     

基于内存建模的复杂结构类型测试数据自动生成方法

针对链表、树和图等这类复杂结构类型的测试数据自动生成问题,提出一种面向路径的基于内存建模的测试数据生成方法.采用一种将结构变量和数值变量分别建模的抽象内存模型,并利用此模型辅助符号执行被测路径;把路径执行过程中语句的语义操作映射到对抽象内存的操作,解决指针引起的别名问题,并且在抽象内存中精准地记录了路径的约束条件;最后通过约束求解得到测试数据.文中方法已应用于自主开发的自动单元测试系统——UATS,通过实验证明了该方法的可行件.     

Using symbolic execution to guide test generation

Although a number of weaknesses of symbolic execution, when used for software testing, have been highlighted in the literature, the recent resurgence of strongly‐typed languages has created opportunities for re‐examining symbolic execution to determine whether these shortfalls can be overcome. This paper discusses symbolic execution in general and makes two contributions: (a) overcoming one of the key problems, analysing programs with indexed arrays; and (b) describing the incorporation of a symbolic execution module for test case generation into an integrated testing tool. For methods which index arrays, a new approach determines all the possible values of each array index, allowing the generation of equivalence classes for every possible combination of array element aliases. An incremental simplification approach, which converts path expressions to canonical forms in order to identify infeasible paths at the earliest opportunity and thus reduce the analysis time, is also described. Symbolic execution is most effective when included in an integrated test and analysis environment: a component test bench was built with a symbolic execution module integrated into it, providing a toolbox of software component test and code analysis methods aimed at programmers at all levels. Copyright © 2000 John Wiley & Sons, Ltd.     

An empirical study on constraint optimization techniques for test generation

Constraint solving is a frequent, but expensive operation with symbolic execution to generate tests for a program. To improve the efficiency of test generation using constraint solving, four optimization techniques are usually applied to existing constraint solvers, which are constraint independence, constraint set simplification, constraint caching, and expression rewriting. In this paper, we conducted an empirical study, using these four constraint optimization techniques in a well known test generation tool KLEE with 77 GNU Coreutils applications, to systematically investigate how these optimization techniques affect the efficiency of test generation. The experimental results show that these constraint optimization techniques as well as their combinations cannot improve the efficiency of test generation significantly for ALL-SIZED programs. Moreover, we studied the constraint optimization techniques with respect to two static metrics, lines of code (LOC) and cyclomatic complexity (CC), of programs. The experimental results show that the “constraint set simplification” technique can improve the efficiency of test generation significantly for the programs with high LOC and CC values. The “constraint caching” optimization technique can improve the efficiency of test generation significantly for the programs with low LOC and CC values. Finally, we propose four hybrid optimization strategies and practical guidelines based on different static metrics.     

Automatic Generation of Floating-Point Test Data

For numerical programs, or more generally for programs with floating-point data, it may be that large savings of time and storage are made possible by using numerical maximization methods instead of symbolic execution to generate test data. Two examples, a matrix factorization subroutine and a sorting method, illustrate the types of data generation problems that can be successfully treated with such maximization techniques.     

路由协议的符号化测试生成

协议一致性测试用于验证协议实现的正确性.文中根据路由协议的消息复杂特点,提出基于on-the-fly策略符号测试生成与动态执行的算法:建立了一种新的统一符号语义模型,该模型把数据操作和控制都抽象为动作行为;以该语义模型为基础,扩展了行为之间的关系以及一致性测试关系;给出了基于on-the-fly策略符号测试生成与符号动态执行的算法,在符号动态执行中,使用了统计的聚类算法来进行符号的数据选择.论文最后用具体的测试例说明该算法在OSPFv3协议一致性测试上的应用.     

形状分析符号执行引擎中的状态合并

符号执行技术以其良好的精确度控制和代码覆盖率被广泛应用于静态程序分析和高覆盖率测试用例自动生成。 符号执行 在分析程序时,以模拟真实的程序执行过程的方式分析程序的数据流和控制流信息,并检查程序可能出现的所有状态,得到程序的分析结果。高精确度和高覆盖率要求对程序状态描述具体而完备,这会导致符号执行过程中常见的状态爆炸问题。首先提出在不同的执行路径上对具体内存状态进行合并的算法,然后对内存模型进行适度的抽象,扩大状态合并算法的适用范围,最后讨论状态合并所带来的实际效果,并提出了状态合并的优化解决方案。所提出的算法在符号执行引擎ShapeChecker上实现,并取得了良好的实验结果。     

Symbolic PathFinder: integrating symbolic execution with model checking for Java bytecode analysis

Symbolic PathFinder (SPF) is a software analysis tool that combines symbolic execution with model checking for automated test case generation and error detection in Java bytecode programs. In SPF, programs are executed on symbolic inputs representing multiple concrete inputs and the values of program variables are represented by expressions over those symbolic inputs. Constraints over these expressions are generated from the analysis of different paths through the program. The constraints are solved with off-the-shelf solvers to determine path feasibility and to generate test inputs. Model checking is used to explore different symbolic program executions, to systematically handle aliasing in the input data structures, and to analyze the multithreading present in the code. SPF incorporates techniques for handling input data structures, strings, and native calls to external libraries, as well as for solving complex mathematical constraints. We describe the tool and its application at NASA, in academia, and in industry.     

EFSM模型的字符串类型测试数据自动生成

基于软件描述模型的测试数据自动生成研究中,字符串类型测试数据生成是一个研究热点和难点。EFSM模型是一种重要的软件描述模型。分析了EFSM模型的特点,针对面向EFSM模型目标路径的字符串测试数据生成,建立了字符串输入变量模型和操作模型,结合静态测试的特点,给出了通过字符串变量模型在目标路径上的符号执行结果生成字符串类型测试数据的方法。实验结果表明,该方法能够达到预期效果,提高测试生成效率。     

Automatically performing weak mutation with the aid of symbolic execution,concolic testing and search-based testing

Automating software testing activities can increase the quality and drastically decrease the cost of software development. Toward this direction, various automated test data generation tools have been developed. The majority of existing tools aim at structural testing, while a quite limited number aim at a higher level of testing thoroughness such as mutation. In this paper, an attempt toward automating the generation of mutation-based test cases by utilizing existing automated tools is proposed. This is achieved by reducing the killing mutants’ problem into a covering branches one. To this extent, this paper is motivated by the use of state of the art techniques and tools suitable for covering program branches when performing mutation. Tools and techniques such as symbolic execution, concolic execution, and evolutionary testing can be easily adopted toward automating the test input generation activity for the weak mutation testing criterion by simply utilizing a special form of the mutant schemata technique. The propositions made in this paper integrate three automated tools in order to illustrate and examine the method’s feasibility and effectiveness. The obtained results, based on a set of Java program units, indicate the applicability and effectiveness of the suggested technique. The results advocate that the proposed approach is able to guide existing automating tools in producing test cases according to the weak mutation testing criterion. Additionally, experimental results with the proposed mutation testing regime show that weak mutation is able to speedup the mutant execution time by at least 4.79 times when compared with strong mutation.     

Scaling symbolic execution using staged analysis

Recent advances in constraint solving technology and raw computation power have led to a substantial increase in the effectiveness of techniques based on symbolic execution for systematic bug finding. However, scaling symbolic execution remains a challenging problem. We present a novel approach to increase the efficiency of symbolic execution for systematic testing of object-oriented programs. Our insight is that we can apply symbolic execution in stages, rather than the traditional approach of applying it all at once, to compute abstract symbolic inputs that can later be shared across different methods to test them systematically. For example, a class invariant can provide the basis of generating abstract symbolic tests that are then used to symbolically execute several methods that require their inputs to satisfy the invariant. We present an experimental evaluation to compare our approach against KLEE, a state-of-the-art implementation of symbolic execution. Results show that our approach enables significant savings in the cost of systematic testing using symbolic execution.     

Scenario-based testing from UML/OCL behavioral models

We present in this article a way to produce test suites applied to the POSIX mini-challenge based on a behavioral model of a file system manager written in UML/OCL. We illustrate the limitations of a fully automated test generation approach, which justifies the use of test scenarios as a complement to a functional testing approach. Scenarios are expressed through regular expressions describing sequences of operations, possibly punctuated by intermediate states that have to be reached by the execution of the model. Scenarios are unfolded into extended sequences of operations that are played on the model using symbolic animation techniques. We experimented our approach by testing the conformance of two different file systems w.r.t. the POSIX standard: a recent Linux distribution and a customized Java implementation of POSIX used to evaluate the relevance of our approach and its complementarity with a structural test generation approach.     

面向漏洞挖掘的基于符号分治区的测试用例生成技术

在漏洞挖掘中,符号执行技术是一种常用的测试用例生成技术。但当软件中包含加解密、校验和检验等复杂数学运算函数时,使用符号执行技术生成测试用例存在无法有效求解约束表达式的问题,导致漏洞挖掘效率低下。针对该问题,文章结合分治算法的思想提出基于符号分治区的测试用例生成技术。首先通过静态分析技术识别软件中的加解密、校验和检验等函数;然后以程序中的加解密、校验和检验函数为分界点对软件进行分区,符号执行引擎每执行到软件的一个分治区,就在本区引入一个新的符号变元进行约束构建,在约束求解时从软件最后一个分治区开始递归求解。基于该方法,文章在符号执行平台S2E上实现了漏洞挖掘原型系统Divide,并与现有的符号执行生成测试用例技术进行对比实验。实验结果表明,文章方法能够快速、有效地生成测试用例,提高漏洞挖掘的效率。     

基于符号执行和N-scope复杂度的代码混淆度量方法

代码混淆可有效对抗逆向工程等各类MATE攻击威胁,作为攻击缓和性质的内生安全技术发展较为成熟,对代码混淆效果的合理度量具有重要价值。代码混淆度量研究相对较少,针对代码混淆弹性的度量方法与泛化性、实用性度量方法相对缺乏。符号执行技术广泛应用于反混淆攻击,其生成遍历程序完整路径输入测试集的难度可为混淆弹性度量提供参考,然而基于程序嵌套结构的对抗技术可显著降低符号执行效率,增加其混淆弹性参考误差。针对上述问题,提出结合符号执行技术和N-scope复杂度的代码混淆度量方法,该方法首先基于程序符号执行时间定义程序混淆弹性;其次提出适配符号执行的N-scope复杂度,定义程序混淆强度同时增强符号执行对多层嵌套结构程序的混淆弹性度量鲁棒性;进而提出结合动态分析与静态分析的混淆效果关联性分析,通过对程序进行符号执行与控制流图提取量化混淆效果。面向C程序构建了该度量方法的一种实现框架并验证,实验对3个公开程序集及其混淆后程序集约4 000个程序进行混淆效果度量,度量结果表明,提出的度量方法在较好地刻画混淆效果的同时拥有一定的泛化能力与实用价值;模拟真实混淆应用场景给出了该度量方法的使用样例,为混淆技术使...     

基于符号执行与模糊测试的混合测试方法

软件测试是保障软件质量的常用方法,如何获得高覆盖率是测试中十分重要且具有挑战性的研究问题.模糊测试与符号执行作为两大主流测试技术已被广泛研究并应用到学术界与工业界中,这两种技术都具有一定的优缺点：模糊测试随机变异生成测试用例并动态执行程序,可以执行并覆盖到较深的分支,但其很难通过变异的方法生成覆盖到复杂条件分支的测试用例.而符号执行依赖约束求解器,可以生成覆盖复杂条件分支的测试用例,但在符号化执行过程中往往会出现状态爆炸问题,因此很难覆盖到较深的分支.有工作已经证明,将符号执行与模糊测试相结合可以获得比单独使用模糊测试或者符号执行更好的效果.分析符号执行与模糊测试的优缺点,提出了一种基于分支覆盖将两种方法结合的混合测试方法——Afleer,结合双方优点从而可以生成具有更高分支覆盖率的测试用例.具体来说,模糊测试（例如AFL）为程序快速生成大量可以覆盖较深分支的测试用例,符号执行（例如KLEE）基于模糊测试的覆盖信息进行搜索,仅为未覆盖到的分支生成测试用例.为了验证Afleer的有效性,选取标准程序集LAVA-M以及实际项目oSIP作为评测对象,以漏洞检测能力以及覆盖能力作为评测指标.实验结果表明：（1）在漏洞检测能力上,Afleer总共可以发现755个漏洞,而AFL仅发现1个;（2）在覆盖能力上,Afleer在标准程序集上以及实际项目中都有不同程度的提升.其中,在oSIP中,Afleer比AFL在分支覆盖率上提高2.4倍,在路径覆盖率上提升6.1倍.除此之外,Afleer在oSIP中还检测出一个新的漏洞.     

一种安全关键软件系统符号执行优化方法

在航空、核电和国防军工领域当中,安全关键系统(Safety-Critical System,SCS)的软件非常重要,其可靠性必须通过测试或形式化方法来保障。符号执行作为一种高效的测试用例生成方法被广泛使用,然而,SCS软件系统的模块之间的耦合性较高,使得符号执行约束求解困难。本文针对这类软件系统提出一种带权最小割集的解耦方法,为安全关键软件系统的自动化测试提供了一种新思路。