Empirical evaluation of optimization algorithms when used in goal-oriented automated test data generation techniques

Software testing is an essential process in software development. Software testing is very costly, often consuming half the financial resources assigned to a project. The most laborious part of software testing is the generation of test-data. Currently, this process is principally a manual process. Hence, the automation of test-data generation can significantly cut the total cost of software testing and the software development cycle in general. A number of automated test-data generation approaches have already been explored. This paper highlights the goal-oriented approach as a promising approach to devise automated test-data generators. A range of optimization techniques can be used within these goal-oriented test-data generators, and their respective characteristics, when applied to these situations remain relatively unexplored. Therefore, in this paper, a comparative study about the effectiveness of the most commonly used optimization techniques is conducted.

Building measure-based prediction models for UML class diagram maintainability

The usefulness of measures for the analysis and design of object oriented (OO) software is increasingly being recognized in the field of software engineering research. In particular, recognition of the need for early indicators of external quality attributes is increasing. We investigate through experimentation whether a collection of UML class diagram measures could be good predictors of two main subcharacteristics of the maintainability of class diagrams: understandability and modifiability. Results obtained from a controlled experiment and a replica support the idea that useful prediction models for class diagrams understandability and modifiability can be built on the basis of early measures, in particular, measures that capture structural complexity through associations and generalizations. Moreover, these measures seem to be correlated with the subjective perception of the subjects about the complexity of the diagrams. This fact shows, to some extent, that the objective measures capture the same aspects as the subjective ones. However, despite our encouraging findings, further empirical studies, especially using data taken from real projects performed in industrial settings, are needed. Such further study will yield a comprehensive body of knowledge and experience about building prediction models for understandability and modifiability.

Using information retrieval based coupling measures for impact analysis

Coupling is an important property of software systems, which directly impacts program comprehension. In addition, the strength of coupling measured between modules in software is often used as a predictor of external software quality attributes such as changeability, ripple effects of changes and fault-proneness. This paper presents a new set of coupling measures for Object-Oriented (OO) software systems measuring conceptual coupling of classes. Conceptual coupling is based on measuring the degree to which the identifiers and comments from different classes relate to each other. This type of relationship, called conceptual coupling, is measured through the use of Information Retrieval (IR) techniques. The proposed measures are different from existing coupling measures and they capture new dimensions of coupling, which are not captured by the existing coupling measures. The paper investigates the use of the conceptual coupling measures during change impact analysis. The paper reports the findings of a case study in the source code of the Mozilla web browser, where the conceptual coupling metrics were compared to nine existing structural coupling metrics and proved to be better predictors for classes impacted by changes.

An experimental investigation of personality types impact on pair effectiveness in pair programming

In this paper, pair programming is empirically investigated from the perspective of developer personalities and temperaments and how they affect pair effectiveness. A controlled experiment was conducted to investigate the impact of developer personalities and temperaments on communication, pair performance and pair viability-collaboration. The experiment involved 70 undergraduate students and the objective was to compare pairs of heterogeneous developer personalities and temperaments with pairs of homogeneous personalities and temperaments, in terms of pair effectiveness. Pair effectiveness is expressed in terms of pair performance, measured by communication, velocity, design correctness and passed acceptance tests, and pair collaboration-viability measured by developers’ satisfaction, knowledge acquisition and participation. The results have shown that there is important difference between the two groups, indicating better communication, pair performance and pair collaboration-viability for the pairs with heterogeneous personalities and temperaments. In order to provide an objective assessment of the differences between the two groups of pairs, a number of statistical tests and stepwise Discriminant Analysis were used.

An industrial case study in reconstructing requirements views

Requirements views, such as coverage and status views, are an important asset for monitoring and managing software development projects. We have developed a method that automates the process of reconstructing these views, and we have built a tool, ReqAnalyst, that supports this method. This paper presents an investigation as to which extent requirements views can be automatically generated in order to monitor requirements in industrial practice. The paper focuses on monitoring the requirements in test categories and test cases. In order to retrieve the necessary data, an information retrieval technique, called Latent Semantic Indexing, was used. The method was applied in an industrial study. A number of requirements views were defined and experiments were carried out with different reconstruction settings for generating these views. Finally, we explored how these views can help the developers during the software development process.

On guiding the augmentation of an automated test suite via mutation analysis

Mutation testing has traditionally been used as a defect injection technique to assess the effectiveness of a test suite as represented by a “mutation score.” Recently, mutation testing tools have become more efficient, and industrial usage of mutation analysis is experiencing growth. Mutation analysis entails adding or modifying test cases until the test suite is sufficient to detect as many mutants as possible and the mutation score is satisfactory. The augmented test suite resulting from mutation analysis may reveal latent faults and provides a stronger test suite to detect future errors which might be injected. Software engineers often look for guidance on how to augment their test suite using information provided by line and/or branch coverage tools. As the use of mutation analysis grows, software engineers will want to know how the emerging technique compares with and/or complements coverage analysis for guiding the augmentation of an automated test suite. Additionally, software engineers can benefit from an enhanced understanding of efficient mutation analysis techniques. To address these needs for additional information about mutation analysis, we conducted an empirical study of the use of mutation analysis on two open source projects. Our results indicate that a focused effort on increasing mutation score leads to a corresponding increase in line and branch coverage to the point that line coverage, branch coverage and mutation score reach a maximum but leave some types of code structures uncovered. Mutation analysis guides the creation of additional “common programmer error” tests beyond those written to increase line and branch coverage. We also found that 74% of our chosen set of mutation operators is useful, on average, for producing new tests. The remaining 26% of mutation operators did not produce new test cases because their mutants were immediately detected by the initial test suite, indirectly detected by test suites we added to detect other mutants, or were not able to be detected by any test.

On detecting faults for Boolean expressions

Fault based testing aims at detecting hypothesized faults based on specifications or program source. There are some fault based techniques for testing Boolean expressions which are commonly used to model conditions in specifications as well as logical decisions in program source. The MUMCUT strategy has been proposed to generate test cases from Boolean expressions. Moreover, it detects eight common types of hypothesized faults provided that the original expression is in irredundant disjunctive normal form, IDNF. Software practitioners are more likely to write the conditions and logical decisions in general form rather than IDNF. Hence, it is interesting to investigate the fault detecting capability of the MUMCUT strategy with respect to general form Boolean expressions. In this article, we perform empirical studies to investigate the fault detection capability of the MUMCUT strategy with respect to general form Boolean expressions as well as mutated expressions. A mutated expression can be obtained from the original given Boolean expression by making a syntactic change based on a particular type of fault.

Presenting software engineering results using structured abstracts: a randomised experiment

When conducting a systematic literature review, researchers usually determine the relevance of primary studies on the basis of the title and abstract. However, experience indicates that the abstracts for many software engineering papers are of too poor a quality to be used for this purpose. A solution adopted in other domains is to employ structured abstracts to improve the quality of information provided. This study consists of a formal experiment to investigate whether structured abstracts are more complete and easier to understand than non-structured abstracts for papers that describe software engineering experiments. We constructed structured versions of the abstracts for a random selection of 25 papers describing software engineering experiments. The 64 participants were each presented with one abstract in its original unstructured form and one in a structured form, and for each one were asked to assess its clarity (measured on a scale of 1 to 10) and completeness (measured with a questionnaire that used 18 items). Based on a regression analysis that adjusted for participant, abstract, type of abstract seen first, knowledge of structured abstracts, software engineering role, and preference for conventional or structured abstracts, the use of structured abstracts increased the completeness score by 6.65 (SE 0.37, p < 0.001) and the clarity score by 2.98 (SE 0.23, p < 0.001). 57 participants reported their preferences regarding structured abstracts: 13 (23%) had no preference; 40 (70%) preferred structured abstracts; four preferred conventional abstracts. Many conventional software engineering abstracts omit important information. Our study is consistent with studies from other disciplines and confirms that structured abstracts can improve both information content and readability. Although care must be taken to develop appropriate structures for different types of article, we recommend that Software Engineering journals and conferences adopt structured abstracts.

An experiment on the role of graphical elements in architecture visualization

The evolution and maintenance of large-scale software systems requires first an understanding of its architecture before delving into lower-level details. Tools facilitating the architecture comprehension tasks by visualization provide different sets of configurable, graphical elements to present information to their users. We conducted a controlled experiment that exemplifies the critical role of such graphical elements when aiming at understanding the architecture. In our setting, a different configuration of graphical elements had significant influence on program comprehension tasks. In particular, a 63% gain in effectiveness in architectural analysis tasks was achieved simply by changing the configuration of the graphical elements of the same tool. Based on the results, we claim that significant effort should be spent on the configuration of architecture visualization tools and that configurability should be a requirement for such tools.

Comparing distributed and face-to-face meetings for software architecture evaluation: A controlled experiment

Scenario-based methods for evaluating software architecture require a large number of stakeholders to be collocated for evaluation meetings. Collocating stakeholders is often an expensive exercise. To reduce expense, we have proposed a framework for supporting software architecture evaluation process using groupware systems. This paper presents a controlled experiment that we conducted to assess the effectiveness of one of the key activities, developing scenario profiles, of the proposed groupware-supported process of evaluating software architecture. We used a cross-over experiment involving 32 teams of three 3rd and 4th year undergraduate students. We found that the quality of scenario profiles developed by distributed teams using a groupware tool were significantly better than the quality of scenario profiles developed by face-to-face teams (p < 0.001). However, questionnaires indicated that most participants preferred the face-to-face arrangement (82%) and 60% thought the distributed meetings were less efficient. We conclude that distributed meetings for developing scenario profiles are extremely effective but that tool support must be of a high standard or participants will not find distributed meetings acceptable.

An information retrieval process to aid in the analysis of code clones

The advent of new static analysis tools has automated the searching for code clones, which are duplicated or similar code fragments in a program. However, clone detection tools can report many clones if the source code that is being searched is large. Programmers may have difficulty comprehending the extensive results from the detection tool, which may inhibit the ability to maintain the identified clones. Latent Semantic Indexing (LSI) is an information retrieval technique that attempts to find relationships in a corpus based on the analysis of the documents in the corpus and the terms in the documents. In this paper, LSI is used to cluster clone classes that have been identified initially by a clone detection tool. The goal of this paper is to detect trends and associations among the clustered clone classes and determine if they provide further comprehension to assist in the maintenance of clones. Experimental evaluation of the approach is reported from a sequence of tools that are chained together to perform an analysis of clones detected in the Microsoft Windows NT kernel source code.

On the relative value of cross-company and within-company data for defect prediction

We propose a practical defect prediction approach for companies that do not track defect related data. Specifically, we investigate the applicability of cross-company (CC) data for building localized defect predictors using static code features. Firstly, we analyze the conditions, where CC data can be used as is. These conditions turn out to be quite few. Then we apply principles of analogy-based learning (i.e. nearest neighbor (NN) filtering) to CC data, in order to fine tune these models for localization. We compare the performance of these models with that of defect predictors learned from within-company (WC) data. As expected, we observe that defect predictors learned from WC data outperform the ones learned from CC data. However, our analyses also yield defect predictors learned from NN-filtered CC data, with performance close to, but still not better than, WC data. Therefore, we perform a final analysis for determining the minimum number of local defect reports in order to learn WC defect predictors. We demonstrate in this paper that the minimum number of data samples required to build effective defect predictors can be quite small and can be collected quickly within a few months. Hence, for companies with no local defect data, we recommend a two-phase approach that allows them to employ the defect prediction process instantaneously. In phase one, companies should use NN-filtered CC data to initiate the defect prediction process and simultaneously start collecting WC (local) data. Once enough WC data is collected (i.e. after a few months), organizations should switch to phase two and use predictors learned from WC data.

Theory of relative defect proneness

In this study, we investigated the functional form of the size-defect relationship for software modules through replicated studies conducted on ten open-source products. We consistently observed a power-law relationship where defect proneness increases at a slower rate compared to size. Therefore, smaller modules are proportionally more defect prone. We externally validated the application of our results for two commercial systems. Given limited and fixed resources for code inspections, there would be an impressive improvement in the cost-effectiveness, as much as 341% in one of the systems, if a smallest-first strategy were preferred over a largest-first one. The consistent results obtained in this study led us to state a theory of relative defect proneness (RDP): In large-scale software systems, smaller modules will be proportionally more defect-prone compared to larger ones. We suggest that practitioners consider our results and give higher priority to smaller modules in their focused quality assurance efforts.

Assessing IR-based traceability recovery tools through controlled experiments

We report the results of a controlled experiment and a replication performed with different subjects, in which we assessed the usefulness of an Information Retrieval-based traceability recovery tool during the traceability link identification process. The main result achieved in the two experiments is that the use of a traceability recovery tool significantly reduces the time spent by the software engineer with respect to manual tracing. Replication with different subjects allowed us to investigate if subjects’ experience and ability play any role in the traceability link identification process. In particular, we made some observations concerning the retrieval accuracy achieved by the software engineers with and without the tool support and with different levels of experience and ability.

The internal consistency and precedence of key process areas in the capability maturity model for software

Evaluating the reliability of maturity level (ML) ratings is crucial for providing confidence in the results of software process assessments. This study investigates the dimensions underlying the maturity construct in the Capability Maturity Model (CMM) for Software (SW-CMM) and estimates the internal consistency of each dimension. The results suggest that SW-CMM maturity is a three-dimensional construct, with “Project Implementation” representing the ML 2 key process areas (KPAs), “Organization Implementation” representing the ML 3 KPAs, and “Quantitative Process Implementation” representing the KPAs at MLs 4 and 5. The internal consistency for each of the three dimensions as estimated by Cronbach’s alpha exceeds the recommended value of 0.9. Based on those results, this study builds and tests a theoretical model which posits that the achievement of lower ML KPAs sustains the implementation of higher ML KPAs. Results of path analysis using partial least squares (PLS) support the theoretical model and provide detailed understanding of the process improvement path. The analysis is based on 676 CMM-Based Appraisal for Internal Process Improvement (CBA IPI) assessments.

Software architecture evaluation methods based on cost benefit analysis and quantitative decision making

Since many parts of the architecture evaluation steps of the Cost Benefit Analysis Method (CBAM) depend on the stakeholders’ empirical knowledge and intuition, it is very important that such an architecture evaluation method be able to faithfully reflect the knowledge of the experts in determining Architectural Strategy (AS). However, because CBAM requires the stakeholders to make a consensus or vote for collecting data for decision making, it is difficult to accurately reflect the stakeholders’ knowledge in the process. In order to overcome this limitation of CBAM, we propose the two new CBAM-based methods for software architecture evaluation, which respectively adopt the Analytic Hierarchy Process (AHP) and the Analytic Network Process (ANP). Since AHP and ANP use pair-wise comparison they are suitable for a cost and benefit analysis technique since its purpose is not to calculate correct values of benefit and cost but to decide AS with highest return on investment. For that, we first define a generic process of CBAM and develop variations from the generic process by applying AHP and ANP to obtain what we call the CBAM+AHP and CBAM+ANP methods. These new methods not only reflect the knowledge of experts more accurately but also reduce misjudgments. A case study comparison of CBAM and the two new methods is conducted using an industry software project. Because the cost benefit analysis process that we present is generic, new cost benefit analysis techniques with capabilities and characteristics different from the three methods we examine here can be derived by adopting various different constituent techniques.

Evaluation of preliminary data analysis framework in software cost estimation based on ISBSG R9 Data

Previous research has argued that preliminary data analysis is necessary for software cost estimation. In this paper, a framework for such analysis is applied to a substantial corpus of historical project data (ISBSG R9 data), selected without explicit bias. The consequent analysis yields sets of dominant variables, which are then used to construct project effort estimation models. Performance of the predictors on the raw variables and the extracted sets of variables is then measured in terms of Mean Magnitude of Relative Error (MMRE), Median of Magnitude of Relative Error (MdMRE) and prediction at levels 0.05, 0.1, and 0.25. The results from the comparative evaluation suggest that more accurate prediction models can be constructed for the selected prediction techniques. The framework processed predictor variables are statistically significant, at the 95% confidence level for both parametric techniques and one non-parametric technique. The results are also compared with the latest published results obtained by other research based on the same data set. The comparison indicates that, the models constructed using framework processed data are generally more accurate.

Quadtree-based eigendecomposition for pose estimation in the presence of occlusion and background clutter

Eigendecomposition-based techniques are popular for a number of computer vision problems, e.g., object and pose estimation, because they are purely appearance based and they require few on-line computations. Unfortunately, they also typically require an unobstructed view of the object whose pose is being detected. The presence of occlusion and background clutter precludes the use of the normalizations that are typically applied and significantly alters the appearance of the object under detection. This work presents an algorithm that is based on applying eigendecomposition to a quadtree representation of the image dataset used to describe the appearance of an object. This allows decisions concerning the pose of an object to be based on only those portions of the image in which the algorithm has determined that the object is not occluded. The accuracy and computational efficiency of the proposed approach is evaluated on 16 different objects with up to 50% of the object being occluded and on images of ships in a dockyard.