Bytecode fault injection for Java software

Developers using third party software components need to test them to satisfy quality requirements. In the past, researchers have proposed fault injection testing approaches in which the component state is perturbed and the resulting effects on the rest of the system are observed. Non-availability of source code in third-party components makes it harder to perform source code level fault injection. Even if Java decompilers are used, they do not work well with obfuscated bytecode. We propose a technique that injects faults in Java software by manipulating the bytecode. Existing test suites are assessed according to their ability to detect the injected faults and improved accordingly. We present a case study using an open source Java component that demonstrates the feasibility and effectiveness of our approach. We also evaluate the usability of our approach on obfuscated bytecode.     

Establishing structural testing criteria for Java bytecode

This paper describes intra‐method control‐flow and data‐flow testing criteria for the Java bytecode language. Six testing criteria are considered for the generation of testing requirements: four control‐flow and two data‐flow based. The main reason to work at a lower level is that, even when there is no source code, structural testing requirements can still be derived and used to assess the quality of a given test set. It can be used, for instance, to perform structural testing on third‐party Java components. In addition, the bytecode can be seen as an intermediate language, so the analysis performed at this level can be mapped back to the original high‐level language that generated the bytecode. To support the application of the testing criteria, we have implemented a tool named JaBUTi (Java Bytecode Understanding and Testing). JaBUTi is used to illustrate the application of the ideas developed in this paper. Copyright © 2006 John Wiley & Sons, Ltd.     

软构件的可测试性研究

构件的可测试性是决定构件质量的关键因素,若能在构件设计阶段就考虑构件的可测试性问题,改善和提高构件的可测试性,那么构件质量就能得到很好的保障,进而减少系统开发时的测试成本。文中针对这个问题,讨论了影响构件可测试性的几个因素,分析了构件测试中存在的问题和构件测试要达到的目标,提出一种构造可测试性构件的通用体系结构,即在原有构件的基础上增加测试工具,把可测试性构件当作对包含嵌入式测试和跟踪工具的扩展单元。     

Effective test case selection for component customization and its application to Enterprise JavaBeans

Component users need to customize components they obtain from providers, because providers usually develop components for general use. Although the customization is accomplished by modifying the interface of a component, faults from customization appear when the implementation part of a component and the interfaces interact. The implementation part is a black‐box, whose source code is not available to a component user, while the interface is a white‐box, whose source code is available for customization. Therefore, customization faults should be tested using both the black‐box part and the white‐box part of a component. This paper proposes a new technique to test customization faults using software fault injection and mutation testing, and the technique is tailored to Enterprise JavaBeans. Test cases are selected by injecting faults not into the entire interface but into specific parts of the component's interface. The specific parts that are chosen control the effectiveness of the test cases. An empirical study to evaluate the technique is reported. Copyright © 2003 John Wiley & Sons, Ltd.     

MuJava: an automated class mutation system

Several module and class testing techniques have been applied to object‐oriented (OO) programs, but researchers have only recently begun developing test criteria that evaluate the use of key OO features such as inheritance, polymorphism, and encapsulation. Mutation testing is a powerful testing technique for generating software tests and evaluating the quality of software. However, the cost of mutation testing has traditionally been so high that it cannot be applied without full automated tool support. This paper presents a method to reduce the execution cost of mutation testing for OO programs by using two key technologies, mutant schemata generation (MSG) and bytecode translation. This method adapts the existing MSG method for mutants that change the program behaviour and uses bytecode translation for mutants that change the program structure. A key advantage is in performance: only two compilations are required and both the compilation and execution time for each is greatly reduced. A mutation tool based on the MSG/bytecode translation method has been built and used to measure the speedup over the separate compilation approach. Experimental results show that the MSG/bytecode translation method is about five times faster than separate compilation. Copyright © 2004 John Wiley & Sons, Ltd.     

Deriving a Fault Architecture to Guide Testing

Defect analysis of software components can be used to guide testing, with the goal of focusing on parts of the software that were fault-prone in earlier releases or earlier life cycle phases, such as development. We replicate a study that adapted a reverse architecting technique using defect reports to derive fault architectures. A fault architecture determines and visualizes components that are fault-prone in their relationships with other components, as well as those that are locally fault-prone. Our case study uses defect data from three releases of a large medical record system to identify relationships among system components, based on whether they are involved in the same defect report.We investigate measures that assess the fault-proneness of components and component relationships. Component relationships are used to derive a fault architecture. The resulting fault architecture indicates what the most fault-prone relationships are in a release. We also apply the technique in a new way. Not only do we derive fault architectures for each release, we derive fault architectures for the development, system test and post release phases within each release. Comparing across releases, makes it possible to see whether some components are repeatedly in fault-prone relationships. Comparing across phases, makes it possible to see whether development fault architectures can be used to identify those parts of the software that need to be tested more. We validate our predictions using system test data from the same release. We also use the development and system test fault architectures to identify fault-prone components after release, and validate our predictions using post release data.     

Lightweight Bytecode Verification

In this paper, we provide a theoretical foundation for and improvements to the existing bytecode verification technology, a critical component of the Java security model, for mobile code used with the Java “micro edition” (J2ME), which is intended for embedded computing devices. In Java, remotely loaded “bytecode” class files are required to be bytecode verified before execution, that is, to undergo a static type analysis that protects the platform's Java run-time system from so-called type confusion attacks such as pointer manipulation. The data flow analysis that performs the verification, however, is beyond the capacity of most embedded devices because of the memory requirements that the typical algorithm will need. We propose to take a proof-carrying code approach to data flow analysis in defining an alternative technique called “lightweight analysis” that uses the notion of a “certificate” to reanalyze a previously analyzed data flow problem, even on poorly resourced platforms. We formally prove that the technique provides the same guarantees as standard bytecode safety verification analysis, in particular that it is “tamper proof” in the sense that the guarantees provided by the analysis cannot be broken by crafting a “false” certificate or by altering the analyzed code. We show how the Java bytecode verifier fits into this framework for an important subset of the Java Virtual Machine; we also show how the resulting “lightweight bytecode verification” technique generalizes and simulates the J2ME verifier (to be expected as Sun's J2ME “K-Virtual machine” verifier was directly based on an early version of this work), as well as Leroy's “on-card bytecode verifier,” which is specifically targeted for Java Cards.     

Training fuzzy logic based software components by combining adaptation algorithms

 A training framework of an effective method for off-line training of a class of control software components (e.g., for first-order nonlinear feedback control systems) using combinations of three kinds of adaptation algorithms is presented. Each control software component is represented at the abstract level by means of a set of adaptive fuzzy logic (FL) rules and at the concrete level by means of fuzzy membership functions (MBFs). At the concrete representation level adaptation algorithms specified for use in adapting MBFs are: genetic algorithms, neural net algorithms, and Monte Carlo algorithms. We specify effective combinations of these three existing adaptation algorithms to train a faulty FL rule-based software component for the tracker problem. In the framework, training consists of two phases: testing and adapting. In the testing phase, a test driver generates an effective fault scenario ( fs) and locates the faulty fuzzy elements (FFEs) by using each or a combination of three adaptation algorithms. In the adapting phase, for each fault scenario adaptation algorithms and their combinations are used to modify the MBFs of the component. Effectiveness of the two phase training is determined in terms of testability, flexibility, adaptability, and stability. An initial design of the simulation environment is presented. In the experiment, for a given circumstance (environment and fuzzy rules) we apply a combination of a genetic algorithm GA) and a neural network (NN) with an error back-propagation algorithm (BP) in the testing phase for generating fault scenarios. Then we apply GA-only method in the adapting phase for adapting the faulty software component. Simulation results on effectiveness and efficiency are discussed.     

Integration testing of object‐oriented components using finite state machines

In object‐oriented terms, one of the goals of integration testing is to ensure that messages from objects in one class or component are sent and received in the proper order and have the intended effect on the state of the objects that receive the messages. This research extends an existing single‐class testing technique to integration testing of multiple classes. The single‐class technique models the behaviour of a single class as a finite state machine, transforms the representation into a data flow graph that explicitly identifies the definitions and uses of each state variable of the class, and then applies conventional data flow testing to produce test case specifications that can be used to test the class. This paper extends those ideas to inter‐class testing by developing flow graphs, finding paths between pairs of definitions and uses, detecting some infeasible paths and automatically generating tests for an arbitrary number of classes and components. It introduces flexible representations for message sending and receiving among objects and allows concurrency among any or all classes and components. Data flow graphs are stored in a relational database and database queries are used to gather def‐use information. This approach is conceptually simple, mathematically precise, quite powerful and general enough to be used for traditional data flow analysis. This testing approach relies on finite state machines, database modelling and processing techniques and algorithms for analysis and traversal of directed graphs. The paper presents empirical results of the approach applied to an automotive system. This work was prepared by U.S. Government employees as part of their official duties and is, therefore, a work of the U.S. Government and not subject to copyright. Published in 2006 by John Wiley & Sons, Ltd.     

Applying a dynamic testability technique to debugging certain classes of software faults

Testability, the tendency for software to reveal its faults during testing, is an important issue for verification and quality assurance. But testability can also be used to good advantage as a debugging technique. Although this concept is more general, we will illustrate it with a specific example: propagation analysis.Propagation Analysis (PA) is a technique for predicting the probability that a data state error affects program output. PA is a technique that produces information about a piece of software's testability. PA bases its prediction on empirical measurement of the probability that an artificial data state error affects program output. After obtaining propagation analysis information for a program and obtaining a failure probability estimate for the program during execution we build a model that can be used to identify possible sites of missing-assignment faults of the form x f(x). Thus we can apply the testability technique PA as a debugging tool.This work supported by a National Research Council NASA-Langley Resident Research Associateship and NASA-Langley Grant NAG-1-884.     

Polymorphic bytecode instrumentation

Bytecode instrumentation is a widely used technique to implement aspect weaving and dynamic analyses in virtual machines such as the Java virtual machine. Aspect weavers and other instrumentations are usually developed independently and combining them often requires significant engineering effort, if at all possible. In this article, we present polymorphic bytecode instrumentation(PBI), a simple but effective technique that allows dynamic dispatch amongst several, possibly independent instrumentations. PBI enables complete bytecode coverage, that is, any method with a bytecode representation can be instrumented. We illustrate further benefits of PBI with three case studies. First, we describe how PBI can be used to implement a comprehensive profiler of inter‐procedural and intra‐procedural control flow. Second, we provide an implementation of execution levels for AspectJ, which avoids infinite regression and unwanted interference between aspects. Third, we present a framework for adaptive dynamic analysis, where the analysis to be performed can be changed at runtime by the user. We assess the overhead introduced by PBI and provide thorough performance evaluations of PBI in all three case studies. We show that pure Java profilers like JP2 can, thanks to PBI, produce accurate execution profiles by covering all code, including the core Java libraries. We then demonstrate that PBI‐based execution levels are much faster than control flow pointcuts to avoid interference between aspects and that their efficient integration in a practical aspect language is possible. Finally, we report that PBI enables adaptive dynamic analysis tools that are more reactive to user inputs than existing tools that rely on dynamic aspect‐oriented programming with runtime weaving. These experiments position PBI as a widely applicable and practical approach for combining bytecode instrumentations. © 2015 The Authors. Software: Practice and Experience Published by John Wiley & Sons Ltd.     

航电处理机相关性建模与故障方程生成方法

航电产品的相关性建模方法为验证故障检测/隔离能力、提高产品的测试性提供了实用方法,由相关性模型生成的故障方程能够有效地进行故障诊断.针对工程实际需求,介绍了一阶相关性模型的数据获取及模型建立方法.利用测试性建模软件建立了某型航电处理机的一阶相关性模型,并获得了相应的测试性参数及系统D矩阵.同时,探讨了由一阶相关性模型生成系统故障方程的方法,包括D矩阵压缩方法和故障方程逻辑合成两个步骤.     

采用Java技术开发可复用的Web表示层构件

JSP和Servlet技术为Web应用系统的构建提供了强大的支持.但是,许多开发者常常会忽略掉一些关键的设计原则,开发出难以维护、难以测试的系统.基于分层次的结构,以Java技术为基础,开发控制器构件、值传递构件、用户会话管理构件、格式构件和JSP页面测试驱动构件,最后组装成Web表示层构件.该构件化的开发方法,提高了系统的可复用度,并且可以在较短的时间内组装出符合用户需求的新系统,从而降低了系统的开发成本及升级维护费用,并能增强系统的可测试性和可维护性.为基于Web的应用系统开发提供了新的思路和方法.     

电子装备测试性融合评估方法

测试性是装备通用质量特性之一，其设计水平直接影响了装备保障效能的发挥，如何更加真实的评估装备的测试性水平是当前研究的热点。在研制阶段装备实物测试性试验数据数量少、获取难、费用高，可认为是“小子样”数据，为了更加全面客观地评价装备的测试性水平，需要充分利用测试性仿真试验获取仿真数据进行融合评估。进行测试性仿真试验时，研究电子装备的仿真模型、故障模型、仿真故障注入方法，获取测试性仿真数据；在此基础上，将仿真试验数据作为验前信息并进行的处理，结合“小子样”装备实物试验数据，采用贝叶斯方法进行测试性试验数据的融合来评估装备的测试性水平，提高评估结果的客观性和可信度。通过案例分析，验证了测试性融合评估方法的有效性。     

Can fault‐exposure‐potential estimates improve the fault detection abilities of test suites?

Code‐coverage‐based test data adequacy criteria typically treat all coverable code elements (such as statements, basic blocks or outcomes of decisions) as equal. In practice, however, the probability that a test case can expose a fault in a code element varies: some faults are more easily revealed than others. Thus, several researchers have suggested that if one could estimate the probability that a fault in a code element will cause a failure, one could use this estimate to determine the number of executions of a code element that are required to achieve a certain level of confidence in that element's correctness. This estimate, in turn, could be used to improve the fault‐detection effectiveness of test suites and help testers distribute testing resources more effectively. This conjecture is intriguing; however, like many such conjectures it has never been directly examined empirically. If empirical evidence were to support this conjecture, it would motivate further research into methodologies for obtaining fault‐exposure‐potential estimates and incorporating them into test data adequacy criteria. This paper reports the results of experiments conducted to investigate the effects of incorporating an estimate of fault‐exposure probability into the statement coverage test data adequacy criterion. The results of these experiments, however, ran contrary to the conjectures of previous researchers. Although incorporation of the estimates did produce statistically significant increases in the fault‐detection effectiveness of test suites, these increases were quite small, suggesting that the approach might not be able to produce the gains hoped for and might not be worth the cost of its employment. Copyright © 2002 John Wiley & Sons, Ltd.     

TPS自动生成平台上的模糊故障字典法

目前数字系统的故障诊断和测试性设计技术已经比较成熟,而模拟系统的故障诊断,尽管起步较早,但进展缓慢。这主要由于参数值的连续变化、反馈和非线性等原因,使得模拟系统的故障诊断相对比较复杂。目前已经提出了多种诊断方法,故障字典法即为其中的一种,它通常用于直流电路硬故障的诊断。由于故障因素和容差因素互相交迭,具有一定的模糊性,这给故障字典的应用带来一定的困难。提出了一种基于模糊故障字典的模拟电路故障诊断方法,用模糊故障字典取代传统的故障字典,适用于模拟电路单故障的故障诊断及故障隔离,便于计算机辅助诊断。     

Bytecode verification on Java smart cards

This article presents a novel approach to the problem of bytecode verification for Java Card applets. By relying on prior off‐card bytecode transformations, we simplify the bytecode verifier and reduce its memory requirements to the point where it can be embedded on a smart card, thus increasing significantly the security of post‐issuance downloading of applets on Java Cards. This article describes the on‐card verification algorithm and the off‐card code transformations, and evaluates experimentally their impact on applet code size. Copyright © 2002 John Wiley & Sons, Ltd.     

Testing‐based process for component substitutability

Software components have emerged to ease the assembly of software systems. However, updates of systems by substitution or upgrades of components demand careful management due to stability risks of deployed systems. Replacement components must be properly evaluated to identify if they provide the expected behaviour affected by substitution. To address this problem, this paper proposes a substitutability assessment process in which the regular compatibility analysis is complemented with the use of black‐box testing criteria. The purpose is to observe the components' behaviour by analysing their internal functions of data transformation, which fulfils the observability testing metric. The approach is conceptually based on the technique Back‐to‐Back testing. When a component should be replaced, a specific Test Suite TS is built in order to represent its behavioural facets, viz. a Component Behaviour TS. This TS is later exercised on candidate upgrades or replacement components with the purpose of identifying the required compatibility. Automation of the process is supported through the testooj tool, which constrains the conditions and steps of the whole process in order to provide a rigorous and reliable approach. Copyright © 2010 John Wiley & Sons, Ltd.     

From bytecode to JavaScript: the Js_of_ocaml compiler

We present the design and implementation of a compiler from OCaml bytecode to JavaScript. The compiler first translates the bytecode into a static single‐assignment intermediate representation on which optimizations are performed, before generating JavaScript. We believe that taking bytecode as an input instead of a high‐level language is a sensible choice. Virtual machines provide a very stable API. Such a compiler is thus easy to maintain. It is also convenient to use, and it can just be added to an existing installation of the development tools. Already‐compiled libraries can be used directly, with no need to reinstall anything. Finally, some virtual machines are the target of several languages. A bytecode to JavaScript compiler would make it possible to retarget all these languages to Web browsers at once. Copyright © 2013 John Wiley & Sons, Ltd.