Evaluating defect prediction approaches: a benchmark and an extensive comparison

Reliably predicting software defects is one of the holy grails of software engineering. Researchers have devised and implemented a plethora of defect/bug prediction approaches varying in terms of accuracy, complexity and the input data they require. However, the absence of an established benchmark makes it hard, if not impossible, to compare approaches. We present a benchmark for defect prediction, in the form of a publicly available dataset consisting of several software systems, and provide an extensive comparison of well-known bug prediction approaches, together with novel approaches we devised. We evaluate the performance of the approaches using different performance indicators: classification of entities as defect-prone or not, ranking of the entities, with and without taking into account the effort to review an entity. We performed three sets of experiments aimed at (1) comparing the approaches across different systems, (2) testing whether the differences in performance are statistically significant, and (3) investigating the stability of approaches across different learners. Our results indicate that, while some approaches perform better than others in a statistically significant manner, external validity in defect prediction is still an open problem, as generalizing results to different contexts/learners proved to be a partially unsuccessful endeavor.     

Incorporating context into recommender systems: an empirical comparison of context-based approaches

Recently, there has been growing interest in recommender systems (RSs) and particularly in context-aware RSs. Methods for generating context-aware recommendations were classified into the pre-filtering, post-filtering and contextual modeling approaches. This paper focuses on comparing the pre-filtering, the post-filtering, the contextual modeling and the un-contextual approaches and on identifying which method dominates the others and under which circumstances. Although some of these methods have been studied independently, no prior research compared the relative performance to determine which of them is better. This paper proposes an effective method of comparing the three methods to incorporate context and selecting the best alternatives. As a result, it provides analysts with a practical suggestion on how to pick a good approach in an effective manner to improve the performance of a context-aware recommender system.     

Remote sensing of phytoplankton pigments: A comparison of empirical and theoretical approaches

Algorithms that have been used on a routine basis for remote sensing of the phytoplankton pigment, chlorophyll- a, from ocean colour data from satellite sensors such as the CZCS (Coastal Zone Color Scanner), SeaWiFS (Sea Viewing Wide Field-of-View Sensor) and OCTS (Ocean Colour and Temperature Scanner) are all of an empirical nature. However, there exist theoretical models that allow ocean colour to be expressed as a function of the inherent optical properties of seawater, such as the absorption coefficient and the backscattering coefficient. These properties can in turn be expressed as functions of chlorophyll- a, at least for the so-called Case 1 waters in which phytoplankton may be considered to be the single, independent variable responsible for most of the variations in the marine optical properties. Here, we use such a theoretical approach to model variations in ocean colour as a function of chlorophyll- a concentration, and compare the results with some empirical models in routine use. The parameters of phytoplankton absorption necessary for the implementation of the ocean colour model are derived from our database of over 700 observations of phytoplankton absorption spectra and concurrent measurements of phytoplankton pigments by HPLC (High Performance Liquid Chromatography) techniques. Since there are reports in the literature that significant differences exist in the performance of the algorithms in polar regions compared with lower latitudes, the model is first implemented using observations made at latitudes less than 50. It is then applied to the Labrador Sea, a high-latitude environment. Our results show that there are indeed differences in the performance of the algorithm at high latitudes, and that these differences may be attributed to changes in the optical characteristics of phytoplankton that accompany changes in the taxonomic composition of their assemblages. The sensitivities of the model to assumptions made regarding absorption by coloured dissolved organic matter (or yellow substances) and backscattering by particles are examined. The importance of Raman scattering on ocean colour and its influence on the algorithms are also investigated.     

Evaluating the ordering of the SPICE capability levels: an empirical study

The Standard ISO/IEC PDTR 15504 (Software Process Assessment) defines process attributes (PAs) and associated practices must be implemented according to a process capability level. This definition implies that PA practices at lower capability levels must be implemented before moving to higher capability levels. The purpose of this study is to evaluate empirically whether the ordering of set of PAs, as measures of capability, is consistent with the Standard. For this purpose, the study estimates the Coefficient of Reproducibility (CR) statistic that measures the extent to which the observed ratings are identical to the pattern inferred by the Standard. Our analyses based on ratings of 689 process instances show that generally PA order of capability levels is consistent with that inferred by the Standard. However, our results also show that the definition of PA3.2 (Process resource) could be improved. This evaluation is capable of providing a substantiated basis for using the notion of capability, as well as providing information for necessary improvements to the Standard.     

Evaluating business information systems: reflections on an empirical study

Abstract. A leading vendor of information technology (IT) systems commissioned the authors to investigate how its clients made their decision to acquire new systems, what benefits they expected to gain, and the kind of evaluation procedures used both before taking the decision to go ahead and subsequently when the systems were in operation. The 16 organizations participating in the study had all purchased major systems and were willing to talk frankly about their experiences. The results of the research have been extensively reported elsewhere. The purpose of this paper is to draw out the implications for IT management. The paper opens with a summary of the findings of the original research. The main part of the paper consists of six case studies from which the management issues are drawn out.     

External information source and channel effectiveness and the diffusion of CASE innovations: an empirical study

This paper examines the relationship between perceived ‘knowledge transfer’ effectiveness of different information channels for computer-aided software engineering (CASE) technology and CASE innovation diffusion. Specifically, the knowledge transfer effectiveness of seven information sources and channels is assessed. CASE innovation behaviour is modelled using three stages: initiation, adoption and implementation. Three types of CASE innovations are examined: two technical process innovations and one administrative innovation. Data for the empirical study were collected through a large-scale national survey of senior IS managers. A total of 405 usable responses were received. The perceived knowledge transfer effectiveness of the information sources and channels is associated differently with the initiation, adoption and implementation phases of the innovation process. Further, these associations are not uniform across the three categories of CASE innovations. The study has implications for how knowledge barriers can be alleviated during the CASE diffusion process.     

The joint impact of service-oriented architectures and business process management on business process quality: an empirical evaluation and comparison

What is the interplay between business process management (BPM) and service-oriented architecture (SOA) in achieving high business process quality? In this paper, we empirically investigate the impact of both SOA and BPM on business process quality in terms of straight-through processing, business-to-business integration, quality control, as well as standardization and consolidation of business processes. For the empirical evaluation of our model, we use the data of 157 German service firms. The results show that the SOA paradigm has still received rather low adoption rates in the industry. However, SOA, BPM, and related information technologies have a direct positive impact on business process quality and the analysis provides evidence for the complementarities of BPM and SOA since interaction effects between them have an additional significant impact on business process quality. Consequently, firms having adopted SOA can more effectively apply BPM.     

Pull-down versus traditional menu types: an empirical comparison

An experiment was conducted to test the performance of pull-down versus traditional or explicit menus. Sixty subjects, including novice and experienced computer users, manipulated both types of menus to complete banking tasks similar to those found on Automatic Teller M achines. The order of the menus was randomly varied to control for learning effects. Across both types of users, traditional-style menus elicited fewer errors than did pull-down menus; however, no significant difference was found in the time to complete the banking task. Experienced users outperformed novice users in the amount of time taken to complete the task regardless of menu type, though no difference was found in the number of errors committed by both user types.     

Evaluating the usage of fault localization in automated program repair: an empirical study

Fault localization techniques are originally proposed to assist in manual debugging by generally producing a rank list of suspicious locations.With the increasing popularity of automated program repair,the fault localization techniques have been introduced to effectively reduce the search space of automated program repair.Unlike developers who mainly focus on the rank information,current automated program repair has two strategies to use the fault localization information:suspiciousness-first algorithm(SFA)based on the suspiciousness accuracy and rank-first algorithm(RFA)relying on the rank accuracy.However,despite the fact that the two different usages are widely adopted by current automated program repair and may result in different repair results,little is known about the impacts of the two strategies on automated program repair.In this paper we empirically compare the performance of SFA and RFA in the context of automated program repair.Specifically,we implement the two strategies and six well-studied fault localization techniques into four state-of-the-art automated program repair tools,and then use these tools to perform repair experiments on 60 real-world bugs from Defects4J.Our study presents a number of interesting findings:RFA outperforms SFA in 70.02%of cases when measured by the number of candidate patches generated before a valid patch is found(NCP),while SFA performs better in parallel repair and patch diversity;the performance of SFA can be improved by increasing the suspiciousness accuracy of fault localization techniques;finally,we use SimFix that deploys SFA to successfully repair four extra Defects4J bugs which cannot be repaired by SimFix originally using RFA.These observations provide a new perspective for future research on the usage and improvement of fault localization in automated program repair.     

Survey and empirical comparison of different approaches for text extraction from scholarly figures

Different approaches have been proposed in the past to address the challenge of extracting text from scholarly figures. However, until recently, no comparative evaluation of the different approaches had been conducted. Thus, we performed an extensive study of the related work and evaluated in total 32 different approaches. In this work, we perform a more detailed comparison of the 7 most relevant approaches described in the literature and extend to 37 systematic linear combinations of methods for extracting text from scholarly figures. Our generic pipeline, consisting of six steps, allows us to freely combine the different possible methods and perform a fair comparison. Overall, we have evaluated 44 different linear pipeline configurations and systematically compared the different methods. We then derived two non-linear configurations and a two-pass approach. We evaluate all pipeline configurations over four datasets of scholarly figures of different origin and characteristics. The quality of the extraction results is assessed using F-measure and Levenshtein distance, and we measure the runtime performance. Our experiments showed that there is a linear configuration that overall shows the best text extraction quality on all datasets. Further experiments showed that the best configuration can be improved by extending it to a two-pass approach. Regarding the runtime, we observed huge differences from very fast approaches to those running for several weeks. Our experiments found the best working configuration for text extraction from our method set. However, they also showed that further improvements regarding region extraction and classification are needed.     

In silico prediction methods of self-interacting proteins: an empirical and academic survey

In silico prediction of self-interacting proteins (SIPs) has become an important part of proteomics. There is an urgent need to develop effective and reliable prediction methods to overcome the disadvantage of high cost and labor intensive in traditional biological wet-lab experiments. The goal of our survey is to sum up a comprehensive overview of the recent literature with the computational SIPs prediction, to provide important references for actual work in the future. In this review, we first describe the data required for the task of DTIs prediction. Then, some interesting feature extraction methods and computational models are presented on this topic in a timely manner. Afterwards, an empirical comparison is performed to demonstrate the prediction performance of some classifiers under different feature extraction and encoding schemes. Overall, we conclude and highlight potential methods for further enhancement of SIPs prediction performance as well as related research directions.     

Evaluating time series forecasting models: an empirical study on performance estimation methods

Machine Learning - Performance estimation aims at estimating the loss that a predictive model will incur on unseen data. This process is a fundamental stage in any machine learning project. In this...     

Evaluating the use and impact of lecture recording in undergraduates: Evidence for distinct approaches by different groups of students

Lecture recordings are increasingly used to supplement lecture attendance within higher education, but their impact on student learning remains unclear. Here we describe a study to evaluate student use of lecture recordings and quantify their impact on academic performance. Questionnaire responses and online monitoring of student's access to recordings indicate that ∼75% students use this material, the majority in a targeted manner. In contrast, a small subset of students (∼5%) are highly dependent on recordings downloading every lecture, and viewing the material for long periods, such that this represents a large proportion of their independent study. This ‘high user’ group is atypical, as it contains a high proportion of dyslexic and Non-English Speaking Background students. Despite high usage, lecture recordings do not have a significant impact on academic performance, either across the cohort or with students that use the recordings. Overall, this approach appears to be beneficial, but may reduce lecture attendance and encourage surface learning approaches in a minority of students.     

Evaluating the usefulness of design rationale in CASE

Advanced systems development is composed of constant alternation between designing artifacts and debating the designs. This article evaluates the usefulness of the Debate Browser design rationale system in the MetaEdit+ CASE environment in one field setting. This usefulness is evaluated in terms of both the system's utility and usability. The research is composed of coaching, observation, and interviews, and it is supplemented through studying the design output. The evaluation suggests that the usefulness of the system is high. It points to advantages in reading design documents, via improved author-reader communication and the understanding of design solutions. This reduces the possibility of errors in design.     

The relative success of alternative approaches to strategic information systems planning: an empirical analysis

Strategic information systems planning (SISP) is an exercise or ongoing activity that enables organisations to develop priorities for information systems development. It has been suggested that the ‘SISP approach’, a combination of method, process and implementation, is the most complete way of describing SISP activity. Based upon questionnaire responses from 267 IT Directors, four distinct approaches to SISP have been derived using cluster analysis. A comparison of these four approaches with five approaches of Earl, M.J., 1993. Experiences in SISP, MIS Quarterly, (March), 1–24, indicates that three bear strong similarities to the ‘organisational’, ‘business-led’, and ‘administrative’ approaches, whilst the fourth cluster is related to both Earl's ‘method-driven’ and ‘technological’ approaches. An analysis of the relationship between SISP approach and SISP success demonstrates that the ‘organisational approach’ is significantly more successful than the other three approaches.     

Creativity: a survey of AI approaches

In this paper we critically survey the AI programs that have been developed to exhibit some aspect of creative behaviour. We describe five necessary characteristics of models of creativity, and we apply these characteristics to help assess the programs surveyed. These characteristic features also provide a basis for a new theory of creative behavior: an emergent memory model. The survey is concluded with an assessment of an implementation of this latest theory.

相似文献   

Detecting binocular half-occlusions: empirical comparisons of five approaches

Binocular half-occlusion points are those that are visible in one of the two views provided by a binocular imaging system. Due to their importance in binocular matching as well as, subsequent interpretation tasks, a number of approaches have been developed for dealing with such points. In the current paper, we consider five methods that explicitly detect half-occlusions and report on a more uniform comparison than has previously been performed. Taking a disparity image and its associated match goodness image as input, we generate images that show the half-occluded points in the underlying scene. We quantitatively and qualitatively compare these methods under a variety of conditions.     

Performance comparison of multi-container deployment schemes for HPC workloads: an empirical study

The Journal of Supercomputing - The high-performance computing (HPC) community has recently started to use containerization to obtain fast, customized, portable, flexible, and reproducible...