General test result checking with log file analysis

We describe and apply a lightweight formal method for checking test results. The method assumes that the software under test writes a text log file; this log file is then analyzed by a program to see if it reveals failures. We suggest a state-machine-based formalism for specifying the log file analyzer programs and describe a language and implementation based on that formalism. We report on empirical studies of the application of log file analysis to random testing of units. We describe the results of experiments done to compare the performance and effectiveness of random unit testing with coverage checking and log file analysis to other unit testing procedures. The experiments suggest that writing a formal log file analyzer and using random testing is competitive with other formal and informal methods for unit testing.     

面向构件化软件的合约检查测试框架

基于构件软件开发的主要思想是使用现存的构件来建构软件系统。而这样的系统由于构件本身的特点导致了许多测试困难。B. Meyer将构件与其客户代码之间的关系形式化地定义为一种合约,它严格限定了构件对象之间的交互规则。通过对合约的监视和检查,可以容易地发现构件之间的交互错误,从而达到集成测试构件化软件的目的。该文提出了一种基于合约检查的构件集成测试框架 (contract-checking test framework,CCTF)。讨论了该框架合约检查的思想、5大功能模块以及其测试流程,并介绍了将CCTF应用到构件化软件测试平台实现的一些关键技术。     

一个可证明安全的代理签名方案

代理签名是一方将自己签名的能力授权给另一方，是一种很重要的密码协议，目前已知的可证明安全的代理签名还很少。该文利用间隙Diffie-Hellman(GDH)群的特点构造了一个新的代理签名方案，新方案在随机预言模型下是可证明安全的。     

On the value of static analysis for fault detection in software

No single software fault-detection technique is capable of addressing all fault-detection concerns. Similarly to software reviews and testing, static analysis tools (or automated static analysis) can be used to remove defects prior to release of a software product. To determine to what extent automated static analysis can help in the economic production of a high-quality product, we have analyzed static analysis faults and test and customer-reported failures for three large-scale industrial software systems developed at Nortel Networks. The data indicate that automated static analysis is an affordable means of software fault detection. Using the orthogonal defect classification scheme, we found that automated static analysis is effective at identifying assignment and checking faults, allowing the later software production phases to focus on more complex, functional, and algorithmic faults. A majority of the defects found by automated static analysis appear to be produced by a few key types of programmer errors and some of these types have the potential to cause security vulnerabilities. Statistical analysis results indicate the number of automated static analysis faults can be effective for identifying problem modules. Our results indicate static analysis tools are complementary to other fault-detection techniques for the economic production of a high-quality software product.     

支持形状分析的符号执行引擎的设计与实现

目前提高软件可靠性的方法有3种:动态测试、静态分析和程序验证。动态测试的结果依赖于测试集的设计,误报率低,漏报率高,分析结果不稳定。程序验证可以对程序的各种性质进行完备的验证。但目前程序验证通常都需要手动证明,分析成本最高。而程序静态分析可以更早、更全面、较高效和低成本地检测到程序中的缺陷。其中符号执行技术是一种比较有应用前景的静态分析技术,可以很好地控制 精确度。针对符号执行可伸缩性差和容易产生路径爆炸的问题,在符号执行过程中利用形状分析技术实现自动推导循环不变式和构建函数行为规范,实现了一个较为实用的C程序分析工具。     

State-rich model checking

In this paper we survey the area of formal verification techniques, with emphasis on model checking due to its wide acceptance by both academia and industry. The major approaches and their characteristics are presented, together with the main problems faced while trying to apply them. With the increased complexity of systems, as well as interest in software correctness, the demand for more powerful automatic techniques is pushing the theories and tools towards integration. We discuss the state of the art in combining formal methods tools, mainly model checking with theorem proving and abstract interpretation. In particular, we present our own recent contribution on an approach to integrate model checking and theorem proving to handle state-rich systems specified using a combination of Z and CSP.     

Model checking driven static analysis for the real world: designing and tuning large scale bug detection

Model checking and static analysis are traditionally seen as two separate approaches to software analysis and verification. In this work we define a model, checking approach for the static analysis of large C/C++ source code bases to detect potential run-time issues such as program crashes, security vulnerabilities and memory leaks. Working on the intersection of software model checking and automated static bug detection for real-life systems, we address a number of issues: how to scale for real-life systems of 1,000,000 LoC or more, how to quickly write new checks, and most importantly how to distinguish between relevant and irrelevant bugs and fine tune the analysis accordingly. We define our model checking-based static analysis approach implemented in our tool Goanna, illustrate a number of design and implementation decisions to obtain practical outcomes and relevant results, and present our findings by empirical data obtained from regularly analyzing large industrial and open source code bases such as the Firefox Web browser.     

SMArDT modeling for automotive software testing

Efficient testing is a crucial prerequisite to engineer reliable automotive software successfully. However, manually deriving test cases from ambiguous textual requirements is costly and error-prone. Model-based software engineering captures requirements in structured, comprehensible, and formal models, which enables early consistency checking and verification. Moreover, these models serve as an indispensable basis for automated test case derivation. To facilitate automated test case derivation for automotive software engineering, we conducted a survey with testing experts of the BMW Group and conceived a method to extend the BMW Group's specification method for requirements, design, and test methodology by model-based test case derivation. Our method is realized for a variant of systems modeling language activity diagrams tailored toward testing automotive software and a model transformation to derive executable test cases. Hereby, we can address many of the surveyed practitioners' challenges and ultimately facilitate quality assurance for automotive software.     

Dynamic testing of knowledge bases using the heuristic testing approach

How to develop knowledge-based and expert systems today is becoming more and more well understood; how to test these systems still poses some challenges. There has been considerable progress in developing techniques for static testing of these systems, checking for problems via formal examination methods; but there has been almost no work on dynamic testing, testing the systems under operating conditions. A novel approach for the dynamic testing of expert system rule bases is presented. This approach, Heuristic Testing, is based on the idea of first testing systems for disastrous safety and integrity problems before testing for primary functions and other classes of problems, and a prioritized series of 10 classes of faults are identified. The Heuristic Testing approach is intended to assure software reliability rather than simply find defects; the reliability is based on the 10 fault clones called compotent reliability. General procedures for conceptualizing and generating test cases were developed for all fault classes, including a Generic Testing Method for generating key test-case values. One of the classes, error-metric, illustrates how complexity-metrics, now used for predicting conventional software problems, could be developed for expert system rule bases. Two key themes are automation (automatically generating test cases) and fix-as-you-go testing (fixing a problem before continuing to test). The overall approach may be generalizable to static rule base testing, to testing of other expert system components, to testing of other nonconventional systems such as neural network and object-oriented systems, and even to conventional software.     

Model learning: a survey of foundations,tools and applications

Software systems are present all around us and playing their vital roles in our daily life. The correct functioning of these systems is of prime concern. In addition to classical testing techniques, formal techniques like model checking are used to reinforce the quality and reliability of software systems. However, obtaining of behavior model, which is essential for model-based techniques, of unknown software systems is a challenging task. To mitigate this problem, an emerging black-box analysis technique, called Model Learning, can be applied. It complements existing model-based testing and verification approaches by providing behavior models of blackbox systems fully automatically. This paper surveys the model learning technique, which recently has attracted much attention from researchers, especially from the domains of testing and verification. First, we review the background and foundations of model learning, which form the basis of subsequent sections. Second, we present some well-known model learning tools and provide their merits and shortcomings in the form of a comparison table. Third, we describe the successful applications of model learning in multidisciplinary fields, current challenges along with possible future works, and concluding remarks.     

Automated formal analysis and verification: an overview

This paper provides an overview of various existing approaches to automated formal analysis and verification. The most space is devoted to the approach of model checking, including its basic principles as well as the different techniques that have been proposed for dealing with the state space explosion problem in model checking. This paper, however, includes a brief discussion of theorem proving and static analysis too. All of the discussed approaches are introduced mostly on an informal level, with an attempt to provide the reader with their basic ideas and references to works where more details can be found.     

Model Checking Programs

The majority of work carried out in the formal methods community throughout the last three decades has (for good reasons) been devoted to special languages designed to make it easier to experiment with mechanized formal methods such as theorem provers, proof checkers and model checkers. In this paper we will attempt to give convincing arguments for why we believe it is time for the formal methods community to shift some of its attention towards the analysis of programs written in modern programming languages. In keeping with this philosophy we have developed a verification and testing environment for Java, called Java PathFinder (JPF), which integrates model checking, program analysis and testing. Part of this work has consisted of building a new Java Virtual Machine that interprets Java bytecode. JPF uses state compression to handle big states, and partial order and symmetry reduction, slicing, abstraction, and runtime analysis techniques to reduce the state space. JPF has been applied to a real-time avionics operating system developed at Honeywell, illustrating an intricate error, and to a model of a spacecraft controller, illustrating the combination of abstraction, runtime analysis, and slicing with model checking.     

Allocation and analysis of reliability: multiple levels: system, subsystem, and module

We model the reliability allocation and prediction process across a hierarchical software system comprised of modules, subsystems, and system. We experiment in modeling complex reliability software systems using several software reliability models to test the feasibility of the process and to evaluate the accuracy of the models for this application. This is a subject deserving research and experimentation because this type of system is implemented in safety-critical projects, such as National Aeronautics and Space Administration (NASA) flight software modules, that we use in our experiments. Given the reliability requirement of a software system in the software planning or design stage, we predict each module’s reliability and their relationships (e.g., reliability interactions among modules, subsystems, and system), Our critical interfaces and components are failure-mode sequences and the modules that comprise these sequences, respectively. In addition, we evaluate how sensitive the achievement of reliability goals is to predicted component reliabilities that do not meet expectations.     

基于不可满足核的近似逼近可达性分析

近年来,形式化验证技术受到了越来越多的关注,它在保障安全关键领域系统的安全性和正确性方面发挥着重要的作用.模型检测作为形式化验证中自动化程度较高的分支,具有十分广阔的发展前景.本文中我们研究并提出了一种新的模型检测技术,可以有效地对迁移系统进行模型检测,包括不安全性检测和证明安全性.与现有的模型检测算法不同,我们提出的这种方法——基于不可满足核（unsatisfiable core,UC）的近似逼近可达性分析（UC-based approximate incremental reachability,UAIR）,主要利用不可满足核来求解一系列的候选安全不变式直至生成最终的不变式,以此来实现安全性证明和不安全性检测（漏洞查找）.在基于SAT求解器的符号模型检测中,我们使用由可满足性求解器得到的UC构造候选安全不变式,如果迁移系统本身是安全的,我们得到的初始不变式只是安全不变式的一个近似.然后,我们在检查安全性的同时,逐步改进候选安全不变式,直到找到一个真正的不变式,证明系统是安全的;如果系统是不安全的,我们的方法最终可以找到一个反例证明系统是不安全的.作为一种全新的方法,我们利用不可满足核进行安全性模型检测,取得了相当好的效果.众所周知,模型检测领域没有绝对最好的方法,尽管我们的方法在基准的可解数量上无法超越当前的成熟方法例如IC3、CAR等,但是我们的方法却可以解出3个其他方法都无法解出的案例,相信本方法可以作为模型检测工具集很有价值的补充.     

Verifying Haskell programs by combining testing, model checking and interactive theorem proving

We propose a program verification method that combines random testing, model checking and interactive theorem proving. Testing and model checking are used for debugging programs and specifications before a costly interactive proof attempt. During proof development, testing and model checking quickly eliminate false conjectures and generate counterexamples which help to correct them. With an interactive theorem prover we also ensure the correctness of the reduction of a top level problem to subproblems that can be tested or proved. We demonstrate the method using our random testing tool and binary decision diagrams-based (BDDs) tautology checker, which are added to the Agda/Alfa interactive proof assistant for dependent type theory. In particular we apply our techniques to the verification of Haskell programs. The first example verifies the BDD checker itself by testing its components. The second uses the tautology checker to verify bitonic sort together with a proof that the reduction of the problem to the checked form is correct.     

Concolic testing of the multi-sector read operation for flash storage platform software

In today’s information society, flash memory has become a virtually indispensable component, particularlyfor mobile devices. In order for mobile devices to operate successfully, it is essential that flash memorybe controlled correctly through flash storage platform software such as the file system, flash translation layer,and low-level device drivers. However, as is typical for embedded software, conventional testing methods oftenfail to detect hidden flaws in the software due to the difficulty of creating effective test cases. As a differentapproach, model checking techniques guarantee a complete analysis, but only on a limited scale. In this paper,we describe an empirical study wherein a concolic testing method is applied to the multi-sector read operation for flash storage platform software. This method combines a concrete dynamic execution and a symbolic execution to automatically generate test cases for full path coverage. Through the experiments, we analyze the advantages and weaknesses of the concolic testing approach on the flash storage platform software.     

Towards Monitoring-Oriented Programming: A Paradigm Combining Specification and Implementation

With the explosion of software size, checking conformance of implementation to specification becomes an increasingly important but also hard problem. Current practice based on ad-hoc testing does not provide correctness guarantees, while highly confident traditional formal methods like model checking and theorem proving are still too expensive to become common practice. In this paper we present a paradigm for combining formal specification with implementation, called monitoring-oriented programming (MoP), providing a light-weighted formal method to check conformance of implementation to specification at runtime. System requirements are expressed using formal specifications given as annotations inserted at various user selected places in programs. Efficient monitoring code using the same target language as the implementation is then automatically generated during a pre-compilation stage. The generated code has the same effect as a logical checking of requirements and can be used in any context, in particular to trigger user defined actions, when requirements are violated. Our proposal is language- and logic- independent, and we argue that it smoothly integrates other interesting system development paradigms, such as design by contract and aspect oriented programming. A prototype has been implemented for Java, which currently supports requirements expressed using past time and future time linear temporal logics, as well as extended regular expressions.     

The hidden models of model checking

In the past, applying formal analysis, such as model checking, to industrial problems required a team of formal methods experts and a great deal of effort. Model checking has become popular, because model checkers have evolved to allow domain-experts, who lack model checking expertise, to analyze their systems. What made this shift possible and what roles did models play in this? That is the main question we consider here. We survey approaches that transform domain-specific input models into alternative forms that are invisible to the user and which are amenable to model checking using existing techniques—we refer to these as hidden models. We observe that keeping these models hidden from the user is in fact paramount to the success of the domain-specific model checker. We illustrate the value of hidden models by surveying successful examples of their use in different areas of model checking (hardware and software) and how a lack of suitable models hamper a new area (biological systems).     

无人机飞控软件系统建模与测试用例生成研究

软件规模与复杂度的迅速增长已成为设计与检验现代高质量无人机飞行控制软件(FCS)系统的重要挑战。采用模型驱动工程(MDE)的框架,使用嵌入式实时系统建模语言(MARTE)建立起某型无人机飞控软件系统的模型,给出了基于时间自动机的系统动态行为的形式化模型实例;结合无人机FCS系统的应用背景,建立了基于时间自动机模型的测试用例生成方法,包括建立测试用例生成框架、测试用例生成规则以及用例生成策略等;对某型无人机飞控软件系统中的主控模块进行了建模与测试用例生成的实例分析研究。