An empirical study of program quality during testing and maintenance

In this paper, we report the results of a study conducted on a large commercial software system written in assembly language. Unlike studies of the past, our data represent the unit test, integration, and all categories of the maintenance phase: adaptive, perfective, and corrective. The results confirm that faults and change activity are related to software measurements. In addition, we report the relationship between the number of design change requests and software measurements. This new observation has the potential to aid the software engineering management process. Finally, we demonstrate the value of multiple regression models over simple regression models.     

An empirical study of testing and integration strategies usingartificial software systems

There has been much discussion about the merits of various testing and integration strategies. Top-down, bottom-up, big-bang, and sandwich integration strategies are advocated by various authors. Also, some authors insist that modules be unit tested, while others believe that unit testing diverts resources from more effective verification processes. This article addresses the ability of the aforementioned integration strategies to detect defects, and produce reliable systems. It also explores the efficacy of spot unit testing, and compares phased and incremental versions of top-down and bottom-up integration strategies. Relatively large artificial software systems were constructed using a code generator with ten basic module templates. These systems were seeded with known defects and tested using the above testing and integration strategies. A number of experiments were then conducted using a simulator whose validity was established by comparing results against these artificial systems. The defect detection ability and resulting system reliability were measured for each strategy. Results indicated that top-down integration strategies are generally most effective in terms of defect correction. Top-down and big-bang strategies produced the most reliable systems. Results favored neither those strategies that incorporate spot unit testing nor those that do not; also, results favored neither phased nor incremental strategies     

An empirical evaluation of P system testing techniques

This paper presents the existing techniques for P system testing and performs an empirical evaluation of their fault-detection efficiency. The comparison is performed using mutation testing and, based on the results obtained, some improved testing methodologies are proposed.     

An intuitive polygon morphing

We present a new algorithm for morphing simple polygons that is inspired by growing forms in nature. While previous algorithms require user-assisted definition of complicated correspondences between the morphing objects, our algorithm defines the correspondence by overlapping the input polygons. Once the morphing of one object into another is defined, very little or no user interaction is necessary to achieve intuitive results. Our algorithm is suitable namely for growth-like morphing. We present the basic algorithm and its three variations. One of them is suitable mainly for convex polygons, the other two are for more complex polygons, such as curved or spiral polygonal forms.     

An empirical study of aspect-oriented metrics

Metrics for aspect-oriented software have been proposed and used to investigate the benefits and the disadvantages of crosscutting concerns modularisation. Some of these metrics have not been rigorously defined nor analytically evaluated. Also, there are few empirical data showing typical values of these metrics in aspect-oriented software. In this paper, we provide rigorous definitions, usage guidelines, analytical evaluation, and empirical data from ten open source projects, determining the value of six metrics for aspect-oriented software (lines of code, weighted operations in module, depth of inheritance tree, number of children, crosscutting degree of an aspect, and coupling on advice execution). We discuss how each of these metrics can be used to identify shortcomings in existing aspect-oriented software.     

An empirical study of interest-based negotiation

While argumentation-based negotiation has been accepted as a promising alternative to game-theoretic or heuristic-based negotiation, no evidence has been provided to confirm this theoretical advantage. We propose a model of bilateral negotiation extending a simple monotonic concession protocol by allowing the agents to exchange information about their underlying interests and possible alternatives to achieve them during the negotiation. We present an empirical study that demonstrates (through simulation) the advantages of this interest-based negotiation approach over the more classic monotonic concession approach to negotiation.     

An empirical study of FORTRAN programs

A sample of programs, written in FORTRAN by a wide variety of people for a wide variety of applications, was chosen ‘at random’ in an attempt to discover quantitatively ‘what programmers really do’. Statistical results of this survey are presented here, together with some of their apparent implications for future work in compiler design. The principal conclusion which may be drawn is the importance of a program ‘profile’, namely a table of frequency counts which record how often each statement is performed in a typical run; there are strong indications that profile-keeping should become a standard practice in all computer systems, for casual users as well as system programmers. This paper is the report of a three month study undertaken by the author and about a dozen students and representatives of the software industry during the summer of 1970. It is hoped that a reader who studies this report will obtain a fairly clear conception of how FORTRAN is being used, and what compilers can do about it.     

Advanced empirical testing

In today’s industrial applications, we see that knowledge systems are successfully implemented. However, critical domains require the elaborate and thoughtful validation of the knowledge bases before the deployment. Empirical testing, also known as regression testing, denotes the most popular validation technique, where predefined test cases are used to simulate and review the correct behavior of the system. In this paper, we motivate that the classic notions of a test case and the corresponding measures are not sufficient in many application scenarios. We present enhanced notions generalizing the standard test case, and we show appropriate extensions of the measures precision and recall, that work on these test case notions. Furthermore, the effective inspection of test runs is important whenever test cases fail. We introduce a novel visualization technique that allows for the effective and intuitive analysis of test cases and test run outcomes. The new visualization is useful for debugging a knowledge base and test case, respectively, but it also provides an intuitive overview of the status of the entire test suite. A case study reports on the (repeated) validation of a medical decision-support system and demonstrates the practical relevance of the presented work.     

The cost of data flow testing: an empirical study

A family of test data adequacy criteria employing data-flow information was previously proposed, and a theoretical complexity analysis was performed. The author describes an empirical study to determine the actual cost of using these criteria. The aim is to establish the practical usefulness of these criteria in testing software and provide a basis for predicting the amount of testing needed for a given program. The first goal of the study is to confirm the belief that the family of software testing criteria considered is practical to use. An attempt is made to show that even as the program size increases, the amount of testing, expressed in terms of the number of test cases sufficient to satisfy a given criterion, remains modest. Several ways of evaluating this hypothesis are explored. The second goal is to provide the prospective user of these criteria with a way of predicting the number of test cases that will be needed to satisfy a given criterion for a given program. This provides testers with a basis for selecting the most comprehensive criterion that they can expect to satisfy. Several plausible bases for such a prediction are considered     

An empirical, path-oriented approach to software analysis and testing

Error flow analysis and testing techniques focus on the introduction of errors through code faults into data states of an executing program, and their subsequent cancellation or propagation to output. The goals and limitations of several error flow techniques are discussed, including mutation analysis, fault-based testing, PIE analysis, and dynamic impact analysis. The attributes desired of a good error flow technique are proposed, and a model called dynamic error flow analysis (DEFA) is described that embodies many of these attributes. A testing strategy is proposed that uses DEFA information to select an optimal set of test paths and to quantify the results of successful testing. An experiment is presented that illustrates this testing strategy. In this experiment, the proposed testing strategy outperforms mutation testing in catching arbitrary data state errors.     

The indecomposability problem in binary morphology: An algorithmic approach

An indecomposable shape is like a prime number. It cannot be decomposed further as a Minkowski sum of two simpler shapes. With respect to Minkowski addition (dilation), therefore, the indecomposable shapes are the fundamental building blocks of all geometric shapes. However, just as it is difficult to identify whether a given number is a prime number or not, it is equally or more difficult to say whether a given shape is indecomposable or not. In this paper we take up a subdomain of binary images, called the weakly taxicab convex image domain, and show how the indecomposability problem in that shape domain can be approached in a manner closely analogous to the number theoretic way. Apart from our attempt to show that the indecomposability problem is an extremely interesting mathematical problem, our algorithmic treatment of the problem also leads to an efficient method of computing Minkowski addition and decomposition of binary images.     

An empirical study of programming language trends

Predicting software engineering trends is a strategically important asset for both developers and managers, but it's also difficult, due to the wide range of factors involved and the complexity of their interactions. This paper reveals some interesting trends and a method for studying other important software engineering trends. This article trades breadth for depth by focusing on a small, compact set of trends involving 17 high-level programming languages. We quantified many of their relevant factors, and then collected data on their evolution over 10 years. By applying statistical methods to this data, we aim to gain insight into what does and does not make a language successful.     

An empirical study of software release notes

Release notes are an important source of information about a new software release. Such notes contain information regarding what is new, changed, and/or got fixed in a release. Despite the importance of release notes, they are rarely explored in the research literature. Little is known about the contained information, e.g., contents and structure, in release notes. To better understand the types of contained information in release notes, we manually analyzed 85 release notes across 15 different software systems. In our manual analysis, we identify six different types of information (e.g., caveats and addressed issues) that are contained in release notes. Addressed issues refer to new features, bugs, and improvements that were integrated in that particular release. We observe that most release notes list only a selected number of addressed issues (i.e., 6-26 % of all addressed issues in a release). We investigated nine different factors (e.g., issue priority and type) to better understand the likelihood of an issue being listed in release notes. The investigation is conducted on eight release notes of three software systems using four machine learning techniques. Results show that certain factors, e.g., issue type, have higher influence on the likelihood of an issue to be listed in release notes. We use machine learning techniques to automatically suggest the issues to be listed in release notes. Our results show that issues listed in all release notes can be automatically determined with an average precision of 84 % and an average recall of 90 %. To train and build the classification models, we also explored three scenarios: (a) having the user label some issues for a release and automatically suggest the remaining issues for that particular release, (b) using the previous release notes for the same software system, and (c) using prior releases for the current software system and the rest of the studied software systems. Our results show that the content of release notes vary between software systems and across the versions of the same software system. Nevertheless, automated techniques can provide reasonable support to the writers of such notes with little training data. Our study provides developers with empirically-supported advice about release notes instead of simply relying on adhoc advice from on-line inquiries.     

An empirical study of fine-grained software modifications

Software is typically improved and modified in small increments (we refer to each of these increments as a modification record—MR). MRs are usually stored in a configuration management or version control system and can be retrieved for analysis. In this study we retrieved the MRs from several mature open software projects. We then concentrated our analysis on those MRs that fix defects and provided heuristics to automatically classify them. We used the information in the MRs to visualize what files are changed at the same time, and who are the people who tend to modify certain files. We argue that these visualizations can be used to understand the development stage of in which a project is at a given time (new features are added, or defects are being fixed), the level of modularization of a project, and how developers might interact between each other and the source code of a system.     

An empirical study of subjective road categorization

Abstract

Driver behaviour will partially be a function of the cognitive schemata by which drivers represent features, functions and operations of traffic system components. This paper uses the Repertory Grid Technique to address the questions: (1) How do drivers internally represent different categories of road? (2) How are those representations mapped on to ‘official’ road categories?

A factor analysis of data obtained from 40 subjects revealed, in answer to question (1), that four factors, namely, ‘safety’; ‘urban/rural’; ‘road boundaries/slip-roads’; and ‘markings’; accounted for 82·3% of the variance. Multidimensional scaling and clustering analyses showed that subjective categorizations of roads are closely related to their objective physical characteristics, but less-well determined by ‘official’ road categories.

Certain features (e.g. slip-roads, markings, obstacle-free boundaries) appear to contribute little to the subjective safety of roads. This suggests that the potential benefits of these safety features may be fully achieved since risk compensation mechanisms will not operate.     

An empirical study into COBOL type inferencing

In a typical COBOL program, the data division consists of 50% of the lines of code. Automatic type inference can help to understand the large collections of variable declarations contained therein, showing how variables are related based on their actual usage. The most problematic aspect of type inference is pollution, the phenomenon that types become too large, and contain variables that intuitively should not belong to the same type. The aim of the paper is to provide empirical evidence for the hypothesis that the use of subtyping is an effective way for dealing with pollution. The main results include a tool set to carry out type inference experiments, a suite of metrics characterizing type inference outcomes, and the experimental observation that only one instance of pollution occurs in the case study conducted.     

An empirical study of Android Wear user complaints

Wearable apps are becoming increasingly popular in recent years. Nevertheless, to date, very few studies have examined the issues that wearable apps face. Prior studies showed that user reviews contain a plethora of insights that can be used to understand quality issues and help developers build better quality mobile apps. Therefore, in this paper, we mine user reviews in order to understand the user complaints about wearable apps. We manually sample and categorize 2,667 reviews from 19 Android wearable apps. Additionally, we examine the replies posted by developers in response to user complaints. This allows us to determine the type of complaints that developers care about the most, and to identify problems that despite being important to users, do not receive a proper response from developers. Our findings indicate that the most frequent complaints are related to Functional Errors, Cost, and Lack of Functionality, whereas the most negatively impacting complaints are related to Installation Problems, Device Compatibility, and Privacy & Ethical Issues. We also find that developers mostly reply to complaints related to Privacy & Ethical Issues, Performance Issues, and notification related issues. Furthermore, we observe that when developers reply, they tend to provide a solution, request more details, or let the user know that they are working on a solution. Lastly, we compare our findings on wearable apps with the study done by Khalid et al. (2015) on handheld devices. From this, we find that some complaint types that appear in handheld apps also appear in wearable apps; though wearable apps have unique issues related to Lack of Functionality, Installation Problems, Connection & Sync, Spam Notifications, and Missing Notifications. Our results highlight the issues that users of wearable apps face the most, and the issues to which developers should pay additional attention to due to their negative impact.     

An empirical study of evaluating software development environmentquality

A study that evaluates new-paradigm-oriented software development environments which have been developed in the five-year formal approach to software environment technology (FASET) project is reviewed. For this study, a software environment evaluation technology based on a software quality evaluation process model defined in ISO/IEC 9126 has been developed. The evaluation technology has been applied to the R&D project at the middle and final phase of development. The evaluation results provide useful information to develop a widely acceptable evaluation technology and to improve the new-paradigm-oriented software development environments that are based on various specification methods: the algebraic specification method, function-oriented specification method, declarative specification method, natural-language-oriented specification method, diagrammatic specification method, state-transition-oriented specification method, and model-based specification method     

An empirical study of delay jitter management policies

This paper presents an empirical study of several policies for managing the effect of delay jitter on the playout of audio and video in computer-based conferences. The problem addressed is that of managing the fundamental trade-off between display with low latency and display with few gaps. We describe a particular policy calledqueue monitoring which observes delay jitter over time and dynamically adjusts display latency in order to support low-latency conferences with an acceptable gap rate. Queue monitoring is evaluated by comparing it with two policies from the literature in a study based on measurements from a computer-based conferencing system. Our results show that queue monitoring performs as well or better than the other policies over the range of observed network loads. More importantly, we show that queue monitoring performs better on those network loads for which the other policies exhibit poor performance.This work was supported by the National Science Foundation (grant numbers CCR-9110938 and ICI-9015443), and by the IBM and Intel Corporations