Data mining for defects in multicore applications: an entropy‐based call‐graph technique

Multicore computers are ubiquitous. Expert developers as well as developers with little experience in parallelism are now asked to create multithreaded software to exploit parallelism in mainstream shared‐memory hardware. However, finding and fixing parallel programming errors is a complex and arduous task. Programmers thus rely on tools such as race detectors that typically focus on reporting errors due to incorrect usage of synchronization constructs or due to missing synchronization. This arsenal of debugging techniques, however, is incomplete. This article presents a new perspective and addresses a largely unexplored direction of defect localization where a wrong usage of nonparallel programming constructs might cause wrong parallel application behavior. In particular, we make a contribution by showing how to use data‐mining techniques to locate defects in multithreaded shared‐memory programs. Our technique analyzes execution anomalies in a condensed representation of the dynamic call graphs of a multithreaded object‐oriented application and identifies methods that contain a defect. Compared with race detectors that concentrate on finding incorrect synchronization, our method is able to reveal a wider range of defects that affect the control flow of a parallel program. Results from controlled experiments show that our data‐mining approach finds not only race conditions in different types of multicore applications but also other errors that cause incorrect parallel program behavior. Data‐mining techniques offer a fruitful new ground for parallel program debugging, and we also discuss long‐term directions for this interesting field. Copyright © 2012 John Wiley & Sons, Ltd.     

面向RISC-V的汇编程序语义等价性自动化测试系统

本文设计并实现了一套面向RISC-V的汇编程序语义等价性自动化测试系统.在面向RISC-V开发软件时,尤其是基于扩展指令(例如向量指令)编写高效的程序时,很难避免以手写汇编的方式编写代码.例如,为标准的C函数库编写相应的向量版函数.与编译器自动生成的代码不同,手写的汇编代码虽然可以最大限度地提高程序的效率,但因绕过了编译时对程序的约束(如类型检查、寄存器分配等)而对开发者提出了更高的要求.能否对新版本与标准版本的汇编程序进行快速地、自动化的语义等价性测试,将大大影响代码的正确性和软件开发和调试的效率.已有面向RISC-V的测试框架缺乏对语义等价性测试的支持,也未考虑程序执行带来的副作用.本研究基于模拟器的动态测试环境,设计并实现了一套面向RISC-V的汇编程序语义等价性自动化测试系统.系统通过跟踪机器状态,捕获程序执行的副作用,并结合用户定义的测试目标生成测试报告.实验表明,本系统相比已有的测试系统,能够有效地对RISC-V汇编程序的语义等价性进行测试.     

A systematic review of approaches for testing concurrent programs

Concurrent programs are replacing the sequential programs as they utilize the true capabilities of multicore architecture. The extensive use of multicore systems and multithreaded paradigms warrants more attention to the testing of the concurrent programs. The testing concurrent program is not a new field as it has been more than 40 years because the first problem related to the testing concurrent program was addressed by the researchers. The field covers various domains, which include concurrency problems, testing approaches, techniques, graphical representations, tools, and subject systems. This paper aims at providing an overview of research in the domain of testing concurrent programs by classifying it into eight categories: (a) reachability testing, (b) structural testing, (c) model‐based testing, (d) mutation‐based testing, (e) slicing‐based testing, (f) formal methods, (g) random testing, and (h) search‐based testing. The survey is focused on the techniques applied, methodologies followed, and tools used in these aforementioned approaches. Furthermore, the gaps are also identified in different approaches. The paper concludes with the consolidation of various testing parameters along with the future directions. Copyright © 2015 John Wiley & Sons, Ltd.     

IPSETFUL: an iterative process of selecting test cases for effective fault localization by exploring concept lattice of program spectra

Fault localization is an important and challenging task during software testing. Among techniques studied in this field, program spectrum based fault localization is a promising approach. To perform spectrum based fault localization, a set of test oracles should be provided, and the effectiveness of fault localization depends highly on the quality of test oracles. Moreover, their effectiveness is usually affected when multiple simultaneous faults are present. Faced with multiple faults it is difficult for developers to determine when to stop the fault localization process. To address these issues, we propose an iterative fault localization process, i.e., an iterative process of selecting test cases for effective fault localization (IPSETFUL), to identify as many faults as possible in the program until the stopping criterion is satisfied. It is performed based on a concept lattice of program spectrum (CLPS) proposed in our previous work. Based on the labeling approach of CLPS, program statements are categorized as dangerous statements, safe statements, and sensitive statements. To identify the faults, developers need to check the dangerous statements. Meantime, developers need to select a set of test cases covering the dangerous or sensitive statements from the original test suite, and a new CLPS is generated for the next iteration. The same process is proceeded in the same way. This iterative process ends until there are no failing tests in the test suite and all statements on the CLPS become safe statements. We conduct an empirical study on several subject programs, and the results show that IPSETFUL can help identifymost of the faults in the program with the given test suite. Moreover, it can save much effort in inspecting unfaulty program statements compared with the existing spectrum based fault localization techniques and the relevant state of the art technique.     

A Hardware Memory Race Recorder for Deterministic Replay

As hardware vendors transition to multicore chips, software vendors face increased software reliability challenges. To effectively debug software in this new world, developers must be able to replay executions that exhibit a bug so that they can zero in on concurrency bugs - especially intermittent ones. Such deterministic replay also aids fault detection and recovery, intrusion detection, and the like. Deterministic replay requires both a recorder and a replayer. The recorder logs information during multithreaded program execution that is sufficient to enable deterministic replay. In the postmortem analysis, the replayer uses the logged information, together with the program binary, to faithfully replay the original execution. The replay will always exercise the same bugs and produce the same outputs. Clearly, an investment in modest chip resources has great potential to ease the challenges of debugging the multithreaded software that both users and vendors will continue to demand     

Statistical Debugging: A Hypothesis Testing-Based Approach

Manual debugging is tedious, as well as costly. The high cost has motivated the development of fault localization techniques, which help developers search for fault locations. In this paper, we propose a new statistical method, called SOBER, which automatically localizes software faults without any prior knowledge of the program semantics. Unlike existing statistical approaches that select predicates correlated with program failures, SOBER models the predicate evaluation in both correct and incorrect executions and regards a predicate as fault-relevant if its evaluation pattern in incorrect executions significantly diverges from that in correct ones. Featuring a rationale similar to that of hypothesis testing, SOBER quantifies the fault relevance of each predicate in a principled way. We systematically evaluate SOBER under the same setting as previous studies. The result clearly demonstrates the effectiveness: SOBER could help developers locate 68 out of the 130 faults in the Siemens suite by examining no more than 10 percent of the code, whereas the cause transition approach proposed by Holger et al. [2005] and the statistical approach by Liblit et al. [2005] locate 34 and 52 faults, respectively. Moreover, the effectiveness of SOBER is also evaluated in an "imperfect world", where the test suite is either inadequate or only partially labeled. The experiments indicate that SOBER could achieve competitive quality under these harsh circumstances. Two case studies with grep 2.2 and bc 1.06 are reported, which shed light on the applicability of SOBER on reasonably large programs     

Anomaly detection in concurrent Java programs using dynamic data flow analysis

Concurrency constructs are widely used when developing complex software such as real-time, networking and multithreaded client–server applications. Consequently, testing a program, which includes concurrency constructs is a very elaborate and complex process. In this work, we first identify the different classes of synchronization anomalies that may occur in concurrent Java programs. We then consider testing concurrent Java programs against synchronization anomalies using dynamic data flow analysis techniques. Moreover, we show how the data flow analysis technique can be extended to detect such anomalies.     

自适应推导下的统一化调试加速技术

在传统调试过程中,缺陷定位通常作为程序修复的前置步骤.最近,一种新型调试框架(统一化调试)被提出.不同于传统调试中缺陷定位和程序修复的单向连接方式,统一化调试首次建立了定位与修复之间的双向连接机制,从而达到同时提升两个领域的效果.作为首个统一化调试技术,ProFL利用程序修复过程中伴随产生的大量补丁执行信息逆向地提升已...     

基于线程调度顺序控制的多线程程序测试

随着多核技术越来越普及,多线程程序的编程也越来越流行。但是多线程程序的正确性问题已经严重影响软件可靠性,且现有的测试技术不能很好地满足多线程程序的需求。本文重点研究多线程程序中最常见的一种bug,即数据竞争,提出一种基于线程调度顺序控制的测试方法。该方法混合静态方法和动态方法,能够有效地找到多线程程序中存在的数据竞争,且能够区分出哪些数据竞争是有害的,需要程序员优先修复。实验结果显示,对于数据竞争的触发概率,本文的方法使其平均从0.53%提高到79.2%,且本文所引入的运行时开销平均只有80%,与相关方法所引入370%的开销相比更优。     

基于长并行距离优先的确定性多线程调度

随着多核技术的不断发展,多线程技术更加广泛地应用于计算机软件中.但由于执行的不确定性,多线程程序的排错和调试存在着很大的困难.确定性多线程系统可以使多线程程序以确定的方式执行,即多次执行同一个多线程程序的顺序和结果是相同的,这可以大大简化多线程程序的排错和调试.但是,确定性多线程系统会导致多线程程序性能的下降.本文提出一种基于长并行距离优先的确定性多线程调度算法,优先执行并行距离长的线程,减少线程总体等待时间,从而提高多线程程序的效率.实验结果表明,本文方法可以使多线程程序的性能提升10%,并且具有很好的可扩展性.     

Using SPIN for automated debugging of infinite executions of Java programs

This paper presents an approach for the automated debugging of reactive and concurrent Java programs, combining model checking and runtime monitoring. Runtime monitoring is used to transform the Java execution traces into the input for the model checker, the purpose of which is twofold. First, it checks these execution traces against properties written in linear temporal logic (LTL), which represent desirable or undesirable behaviors. Second, it produces several execution traces for a single Java program by generating test inputs and exploring different schedulings in multithreaded programs. As state explosion is the main drawback to model checking, we propose two abstraction approaches to reduce the memory requirements when storing Java states. We also present the formal framework to clarify which kinds of LTL safety and liveness formulas can be correctly analysed with each abstraction for both finite and infinite program executions. A major advantage of our approach comes from the model checker, which stores the trace of each failed execution, allowing the programmer to replay these executions to locate the bugs. Our current implementation, the tool TJT, uses Spin as the model checker and the Java Debug Interface (JDI) for runtime monitoring. TJT is presented as an Eclipse plug-in and it has been successfully applied to debug complex public Java programs.     

Predictive Mitigation of Timing Channels - Threat Defense for Machine Codes

The security of software systems can be threatened by many internal and external threats, including data leakages due to timing channels. Even if developers manage to avoid security threats in the source code or bytecode during development and testing, new threats can arise as the compiler generates machine codes from representations at the binary code level during execution on the processor or due to operating system specifics. Current approaches either do not allow the neutralization of timing channels to be achieved comprehensively with a sufficient degree of security or require an unjustifiable amount of time and/or resources. Herein, a method is demonstrated for the protected execution of software based on a secure virtual execution environment (VEE) that combines the results from dynamic and static analyses to find timing channels through the application of code transformations. This solution complements other available techniques to prevent timing channels from being exploited. This approach helps control the appearance and neutralization of timing channels via just-in-time code modifications during all stages of program development and usage. This work demonstrates the identification of threats using timing channels as an example. The approach presented herein can be expanded to the neutralization of other types of threats.     

Contextual debugging and analysis of multithreaded applications

Multithreaded programs are especially difficult to test and debug. The aim of the paper is to present a new concept of multithreaded program analysis and debugging based on contextual visualisation of the program components that influence thread execution. For this purpose, a dedicated software package called MTV (multithreading viewer) has been designed and implemented. It performs above the run-time library level, and hence only a programmer's view of multiple threads of control execution may be analyzed. The paper presents tested program code instrumentation, communication and synchronization between the instrumented program and MTV. Next, a general concept of contextual visualisation of multithreaded programs has been elaborated. A scheme of the MTV cooperation with the monitored program is discussed. The user interface has been described. A representation of the multithreaded program state has been shown, and the capability of MTV for certain classes of error recognition has been specified and illustrated by a few examples. These examples have been not intended to be exhaustive, but they rather indicate the opportunities to exploit MTV for analysis of complex applications. Short evaluation of the proposed contextual visualisation techniques with application to multithreaded program analysis concludes the paper. © 1997 by John Wiley & Sons, Ltd.     

Techniques for Classifying Executions of Deployed Software to Support Software Engineering Tasks

There is an increasing interest in techniques that support analysis and measurement of fielded software systems. These techniques typically deploy numerous instrumented instances of a software system, collect execution data when the instances run in the field, and analyze the remotely collected data to better understand the system's in-the-field behavior. One common need for these techniques is the ability to distinguish execution outcomes (e.g., to collect only data corresponding to some behavior or to determine how often and under which condition a specific behavior occurs). Most current approaches, however, do not perform any kind of classification of remote executions and either focus on easily observable behaviors (e.g., crashes) or assume that outcomes' classifications are externally provided (e.g., by the users). To address the limitations of existing approaches, we have developed three techniques for automatically classifying execution data as belonging to one of several classes. In this paper, we introduce our techniques and apply them to the binary classification of passing and failing behaviors. Our three techniques impose different overheads on program instances and, thus, each is appropriate for different application scenarios. We performed several empirical studies to evaluate and refine our techniques and to investigate the trade-offs among them. Our results show that 1) the first technique can build very accurate models, but requires a complete set of execution data; 2) the second technique produces slightly less accurate models, but needs only a small fraction of the total execution data; and 3) the third technique allows for even further cost reductions by building the models incrementally, but requires some sequential ordering of the software instances' instrumentation.     

Is non-parametric hypothesis testing model robust for statistical fault localization?

Fault localization is one of the most difficult activities in software debugging. Many existing statistical fault-localization techniques estimate the fault positions of programs by comparing the program feature spectra between passed runs and failed runs. Some existing approaches develop estimation formulas based on mean values of the underlying program feature spectra and their distributions alike. Our previous work advocates the use of a non-parametric approach in estimation formulas to pinpoint fault-relevant positions. It is worthy of further study to resolve the two schools of thought by examining the fundamental, underlying properties of distributions related to fault localization. In particular, we ask: Can the feature spectra of program elements be safely considered as normal distributions so that parametric techniques can be soundly and powerfully applied? In this paper, we empirically investigate this question from the program predicate perspective. We conduct an experimental study based on the Siemens suite of programs. We examine the degree of normality on the distributions of evaluation biases of the predicates, and obtain three major results from the study. First, almost all examined distributions of evaluation biases are either normal or far from normal, but not in between. Second, the most fault-relevant predicates are less likely to exhibit normal distributions in terms of evaluation biases than other predicates. Our results show that normality is not common as far as evaluation bias can represent. Furthermore, the effectiveness of our non-parametric predicate-based fault-localization technique weakly correlates with the distributions of evaluation biases, making the technique robust to this type of uncertainty in the underlying program spectra.     

面向软件错误定位与理解的测试执行离散特征筛选

软件错误定位与错误理解是软件调试过程中的重要步骤,然而调试人员利用基于覆盖分析的软件错误定位获取的可疑度,从高到低静态分析每条程序语句的检查方式,与实际软件调试过程并不相符。为了能够筛选更有助于理解错误根源的测试执行,尤其是致使程序失效的失效执行,帮助调试人员进行动态差异化分析,针对失效执行提出基于高可疑度覆盖率、揭示错误潜力和覆盖语句可疑度离散特征的3种优先级策略,针对成功执行提出加权余弦相似度匹配策略。通过将3种失效执行优先级策略与随机选择在常用错误定位技术中进行实验对比,验证了基于覆盖语句可疑度离散特征的失效执行筛选策略能够对筛选前后的错误理解工作量变化产生更强的积极影响和更弱的消极影响,并能够在相同工作量下理解更多的错误,进而更有助于将错误定位结果应用于错误根源的理解。     

A class library for implementing, testing, and debugging concurrent programs

We describe the Modern Multithreading (MM) class library. MM is a class library consisting of thread and synchronization classes that provide significant support for testing and debugging multithreaded programs. The synchronization classes implement commonly used synchronization objects such as semaphores, monitors, and asynchronous and synchronous message passing channels, for programs that run on a single computer or on a distributed system. MM uses controlled executions to provide program tracing and replay and to support a number of implementation-based and specification-based testing techniques, including non-deterministic and deterministic testing and several forms of reachability testing. MM is portable and easy to use, and has been implemented in Java and C++, with C++ versions for the POSIX Pthreads library and for the Windows Win32 API.     

Generating educational mobile applications using UIDPs identified by artificial intelligence techniques

Commercial mobile devices vary in brand, size, and functionalities, but they all allow people to interact with educational applications. In software engineering, application development techniques, approaches, methodologies, and processes (e.g., BBD, FDD, RAD, DDD) are often time consuming, costly, or aim at expert developers – which implies that users outside the software development field (e.g., teachers) need great practice to become experienced application developers. This work proposes an artificial-intelligence-based process for generating educational mobile apps from freehand-generated images. The images’ design is based on User Interface Design Pattern (UIDP) representations. As a proof of concept, we introduce EduMatic, an application development tool. To test our process, we assessed Wiki-Comp, an application built with EduMatic, along with three other external educational applications: Khan Academy, Wiki encyclopedia, and Kiwix. According to the evaluation results, Wiki-Comp outstands in functionality, usability, and performance aspects.     

Mutation-oriented test data augmentation for GUI software fault localization

ContextFault localization lies at the heart of program debugging and often proceeds by contrasting the statistics of program constructs executed by passing and failing test cases. A vital issue here is how to obtain these “suitable” test cases. Techniques presented in the literature mostly assume the existence of a large test suite a priori. However, developers often encounter situations where a failure occurs, but where no or no appropriate test suite is available for use to localize the fault.ObjectiveThis paper aims to alleviate this key limitation of traditional fault localization techniques for GUI software particularly, namely, it aims at enabling cost-effective fault localization process for GUI software in the described scenario.MethodTo address this scenario, we propose a mutation-oriented test data augmentation technique, which actually is directed by the “similarity” criterion in GUI software’s test case context towards the generation of test suite with excellent fault localization capabilities. More specifically, the technique mainly uses four proposed novel mutation operators to iteratively mutate some failing GUI test cases’ event sequences to derive new test cases potentially useful to localize the specific encountered fault. We then compare the fault localization performance of the test suite generated using this technique with that of an original provided large event-pair adequate test suite on some GUI applications.ResultsThe results indicate that the proposed technique is capable of generating a test suite that has comparable, if not better, fault localization effectiveness to the event-pair adequate test suite, but it is much smaller and it is generated immediately once a failure is encountered by developers.ConclusionIt is concluded that the proposed technique can truly enable quick-start cost-effective fault localization process under the investigated all-too-common scenario, greatly alleviating one key limitation of traditional fault localization techniques and prompting the test–diagnose–repair cycle.