Learning dependency-based change impact predictors using independent change histories

Context: Recent studies showed that combining present data, which are derived from the current software version, with past data, which are derived from previous software versions, can improve the accuracy of change impact predictions. However, for a specific program, existing combined techniques can rely only on version history of that program, if available, and the prediction results depend on the variety of available change impact examples.Objective: We propose a hybrid probabilistic approach that predicts the change impact for a software entity using, as training data, existing version histories of whatever software systems.Method: Change-impact predictors are learned from past change impact graphs (CIGs), extracted from the version history, along with their associations with different influencing factors of change propagation. The learning examples in CIGs are not specific to the software entities that are covered in those examples, and the change propagation influencing factors are structural and conceptual dependencies between software entities. Once our predictors are trained, they can predict change impacts regardless of the version history of the software under-analysis. We evaluate our approach using four systems in two scenarios. First, we use as training data the CIGs extracted from previous versions of the system under-analysis. Second, for each analyzed system, we use only the training data extracted from the other systems.Results: Our approach produces accurate predictions in terms of both precision and recall. Moreover, when training our classifiers with a large variety of CIGs extracted from the change histories of different projects, the recall scores of predicted impact sets were significantly improved.Conclusion:Our approach produces accurate predictions for new classes without recorded change histories, as well as for old classes. For the systems considered in our evaluation, once our approach is trained with a variety of CIGs it can predict change impacts with good recall scores.     

An empirical study of the impact of modern code review practices on software quality

Software code review, i.e., the practice of having other team members critique changes to a software system, is a well-established best practice in both open source and proprietary software domains. Prior work has shown that formal code inspections tend to improve the quality of delivered software. However, the formal code inspection process mandates strict review criteria (e.g., in-person meetings and reviewer checklists) to ensure a base level of review quality, while the modern, lightweight code reviewing process does not. Although recent work explores the modern code review process, little is known about the relationship between modern code review practices and long-term software quality. Hence, in this paper, we study the relationship between post-release defects (a popular proxy for long-term software quality) and: (1) code review coverage, i.e., the proportion of changes that have been code reviewed, (2) code review participation, i.e., the degree of reviewer involvement in the code review process, and (3) code reviewer expertise, i.e., the level of domain-specific expertise of the code reviewers. Through a case study of the Qt, VTK, and ITK projects, we find that code review coverage, participation, and expertise share a significant link with software quality. Hence, our results empirically confirm the intuition that poorly-reviewed code has a negative impact on software quality in large systems using modern reviewing tools.     

An empirical study on the use of mutant traces for diagnosis of faults in deployed systems

Debugging deployed systems is an arduous and time consuming task. It is often difficult to generate traces from deployed systems due to the disturbance and overhead that trace collection may cause on a system in operation. Many organizations also do not keep historical traces of failures. On the other hand earlier techniques focusing on fault diagnosis in deployed systems require a collection of passing–failing traces, in-house reproduction of faults or a historical collection of failed traces. In this paper, we investigate an alternative solution. We investigate how artificial faults, generated using software mutation in test environment, can be used to diagnose actual faults in deployed software systems. The use of traces of artificial faults can provide relief when it is not feasible to collect different kinds of traces from deployed systems. Using artificial and actual faults we also investigate the similarity of function call traces of different faults in functions. To achieve our goal, we use decision trees to build a model of traces generated from mutants and test it on faulty traces generated from actual programs. The application of our approach to various real world programs shows that mutants can indeed be used to diagnose faulty functions in the original code with approximately 60–100% accuracy on reviewing 10% or less of the code; whereas, contemporary techniques using pass–fail traces show poor results in the context of software maintenance. Our results also show that different faults in closely related functions occur with similar function call traces. The use of mutation in fault diagnosis shows promising results but the experiments also show the challenges related to using mutants.     

Code-based prioritization: a pre-testing effort to minimize post-release failures

Improving the efficiency of the testing process is a challenging goal. Prior work has shown that often a small number of errors account for the majority of software failures; and often, most errors are found in a small portion of a source code. We argue that prioritizing code elements before conducting testing can help testers focus their testing effort on the parts of the code most likely to expose errors. This can, in turn, promote more efficient testing of software. Keeping this in view, we propose a testing effort prioritization method to guide tester during software development life cycle. Our approach considers five factors of a component such as Influence value, Average execution time, Structural complexity, Severity and Value as inputs and produce the priority value of the component as an output. Once all components of a program have been prioritized, testing effort can be apportioned so that the components causing more frequent and/or more severe failures will be tested more thoroughly. Our proposed approach is effective in guiding testing effort as it is linked to external measure of defect severity and business value, internal measure of frequency and complexity. As a result, the failure rate is decreased and the chance of severe type of failures is also decreased in the operational environment. We have conducted experiments to compare our scheme with a related scheme. The results establish that our proposed approach that prioritizes the testing effort within the source code is able to minimize highly severed types of failures and also number of failures at the post-release time of a software system.     

Software reliability from the customer view

We augment our defect-prevention activities with analyses of what our customers experience - software failures. NED provides a suite of hardware and software products for business-critical applications, including the operating system, database management system, communication protocols, transaction monitors, and application-development tools. We update this suite quarterly in a release version update. An RVU is an aggregation of all product changes plus the previous version of all unchanged products, which totals several million lines of code. We track the failure rates for each RVU so that we can measure the customer experience with that update. We have been analyzing software failure data for the past several years to provide feedback on the effectiveness of our defect-prevention activities.     

On the impact of software evolution on software clustering

The evolution of a software project is a rich data source for analyzing and improving the software development process. Recently, several research groups have tried to cluster source code artifacts based on information about how the code of a software system evolves. The results of these evolutionary approaches seem promising, but a direct comparison to traditional software clustering approaches based on structural code dependencies is still missing. To fill this gap, we conducted several clustering experiments with an established software clustering tool comparing and combining the evolutionary and the structural approach. These experiments show that the evolutionary approach could produce meaningful clustering results. While the traditional approach provides better results because of a more reliable data density of the structural data, the combination of both approaches is able to improve the overall clustering quality. A review of related studies shows that this approach of combining dependency information is also successful in other software engineering applications.     

The Linux kernel as a case study in software evolution

We use 810 versions of the Linux kernel, released over a period of 14 years, to characterize the system’s evolution, using Lehman’s laws of software evolution as a basis. We investigate different possible interpretations of these laws, as reflected by different metrics that can be used to quantify them. For example, system growth has traditionally been quantified using lines of code or number of functions, but functional growth of an operating system like Linux can also be quantified using the number of system calls. In addition we use the availability of the source code to track metrics, such as McCabe’s cyclomatic complexity, that have not been tracked across so many versions previously. We find that the data supports several of Lehman’s laws, mainly those concerned with growth and with the stability of the process. We also make some novel observations, e.g. that the average complexity of functions is decreasing with time, but this is mainly due to the addition of many small functions.     

The exception handling effectiveness of POSIX operating systems

Operating systems form a foundation for robust application software, making it important to understand how effective they are at handling exceptional conditions. The Ballista testing system was used to characterize the handling of exceptional input parameter values for up to 233 POSIX functions and system calls on each of 15 widely used operating system (OS) implementations. This identified ways to crash systems with a single call, ways to cause task hangs within OS code, ways to cause abnormal task termination within OS and library code, failures to implement defined POSIX functionality, and failures to report unsuccessful operations. Overall, only 55 percent to 76 percent of the exceptional tests performed generated error codes, depending on the operating system being tested. Approximately 6 percent to 19 percent of tests failed to generate any indication of error despite exceptional inputs. Approximately 1 percent to 3 percent of tests revealed failures to implement defined POSIX functionality for unusual, but specified, situations. Between 18 percent and 33 percent of exceptional tests caused the abnormal termination of an OS system call or library function, and five systems were completely crashed by individual system calls with exceptional parameter values. The most prevalent sources of these robustness failures were illegal pointer values, numeric overflows, and end-of-file overruns     

A new hierarchical software architecture towards safety-critical aspects of a drone system

A new hierarchical software architecture is proposed to improve the safety and reliability of a safety-critical drone system from the perspective of its source code. The proposed architecture uses formal verification methods to ensure that the implementation of each module satisfies its expected design specification, so that it prevents a drone from crashing due to unexpected software failures. This study builds on top of a formally verified operating system kernel, certified kit operating system (CertiKOS). Since device drivers are considered the most important parts affecting the safety of the drone system, we focus mainly on verifying bus drivers such as the serial peripheral interface and the inter-integrated circuit drivers in a drone system using a rigorous formal verification method. Experiments have been carried out to demonstrate the improvement in reliability in case of device anomalies.

     

The impact of requirements changes on specifications and state machines

Requirements change both during and after a phase of development for a variety of reasons, including error correction and feature changes. Requirements change management is one of the most complex and difficult problems to deal with in requirements elicitation and tracking. It is generally not understood how a specific change propagates through the specification and into the code. In this paper we capture requirements changes as series of atomic changes in specifications. Using a rigorous specification method called sequence‐based specification, we propose a set of algorithms for managing all possible atomic requirements changes. The algorithms have been formulated within an axiom system for sequence‐based specification and proven for correctness. They have also been implemented in a prototype tool with which users are able to push requirements changes through to changes in specifications, maintain old specifications over time and evolve them into new specifications with the least amount of human interaction and rework. The approach of utilizing state machines to model and manage requirements changes guarantees strong evidence about the correctness and completeness of the proposed theory that will lead to more reliable software in the presence of change, especially with embedded systems and safety‐critical systems. The solution described is general enough for adoption by software and system developers, and well suited for incremental development. Copyright © 2008 John Wiley & Sons, Ltd.