A process for managing interaction between experimenters to get useful similar replications

ContextA replication is the repetition of an experiment. Several efforts have been made to adopt replication as a common practice in software engineering. There are different types of replications, depending on their purpose. Similar replications keep the experimental conditions as alike as possible to the original ones. External similar replications, where the replicating experimenters are not the same people as the original experimenters, have been a stumbling block. Several attempts at combining the results of replications have resulted in failure. Software engineering does not appear to be well suited to such replications, because it works with complex experimentally immature contexts. Software engineering settings have a large number of variables, and the role that many of them play is unknown. A successful (or useful) similar replication helps to better understand the phenomenon under study by verifying results and/or identifying contextual variables that could influence (or not) the results, through the combination of experimental results.ObjectiveTo be able to get successful similar replications, there needs to be interaction between original and replicating experimenters. In this paper, we propose an interaction process for achieving successful similar replications.MethodThis process consists of: an adaptation meeting, where experimenters tailor the experiment to the new setting; querying, to settle occasional inquiries while the experiment is being run; and a combination meeting, where experimenters meet to discuss the combination of replication outcomes with previous results. To check its effectiveness, the process has been tested on three different replications of the same experiment.ResultsThe proposed interaction process has helped to identify new contextual variables that could potentially influence (or not) the experimental results in the three replications run. Additionally, the interaction process has helped to uncover certain problems and deviations that occurred during some of the replications that we would have not been aware of otherwise.ConclusionsThere are signs that suggest that it is possible to get successful similar replications in software engineering experimentation, when there is appropriate interaction among experimenters.     

The role of replications in Empirical Software Engineering

Replications play a key role in Empirical Software Engineering by allowing the community to build knowledge about which results or observations hold under which conditions. Therefore, not only can a replication that produces similar results as the original experiment be viewed as successful, but a replication that produce results different from those of the original experiment can also be viewed as successful. In this paper we identify two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replications, in which the same research question is evaluated by using a different experimental procedure. The focus of this paper is on exact replications. We further explore them to identify two sub-categories: dependent replications, where researchers attempt to keep all the conditions of the experiment the same or very similar and independent replications, where researchers deliberately vary one or more major aspects of the conditions of the experiment. We then discuss the role played by each type of replication in terms of its goals, benefits, and limitations. Finally, we highlight the importance of producing adequate documentation for an experiment (original or replication) to allow for replication. A properly documented replication provides the details necessary to gain a sufficient understanding of the study being replicated without requiring the replicator to slavishly follow the given procedures.

Effectiveness for detecting faults within and outside the scope of testing techniques: an independent replication

The verification and validation activity plays a fundamental role in improving software quality. Determining which the most effective techniques for carrying out this activity are has been an aspiration of experimental software engineering researchers for years. This paper reports a controlled experiment evaluating the effectiveness of two unit testing techniques (the functional testing technique known as equivalence partitioning (EP) and the control-flow structural testing technique known as branch testing (BT)). This experiment is a literal replication of Juristo et al. (2013). Both experiments serve the purpose of determining whether the effectiveness of BT and EP varies depending on whether or not the faults are visible for the technique (InScope or OutScope, respectively). We have used the materials, design and procedures of the original experiment, but in order to adapt the experiment to the context we have: (1) reduced the number of studied techniques from 3 to 2; (2) assigned subjects to experimental groups by means of stratified randomization to balance the influence of programming experience; (3) localized the experimental materials and (4) adapted the training duration. We ran the replication at the Escuela Politécnica del Ejército Sede Latacunga (ESPEL) as part of a software verification & validation course. The experimental subjects were 23 master’s degree students. EP is more effective than BT at detecting InScope faults. The session/program and group variables are found to have significant effects. BT is more effective than EP at detecting OutScope faults. The session/program and group variables have no effect in this case. The results of the replication and the original experiment are similar with respect to testing techniques. There are some inconsistencies with respect to the group factor. They can be explained by small sample effects. The results for the session/program factor are inconsistent for InScope faults. We believe that these differences are due to a combination of the fatigue effect and a technique x program interaction. Although we were able to reproduce the main effects, the changes to the design of the original experiment make it impossible to identify the causes of the discrepancies for sure. We believe that further replications closely resembling the original experiment should be conducted to improve our understanding of the phenomena under study.     

Understanding replication of experiments in software engineering: A classification

ContextReplication plays an important role in experimental disciplines. There are still many uncertainties about how to proceed with replications of SE experiments. Should replicators reuse the baseline experiment materials? How much liaison should there be among the original and replicating experimenters, if any? What elements of the experimental configuration can be changed for the experiment to be considered a replication rather than a new experiment?ObjectiveTo improve our understanding of SE experiment replication, in this work we propose a classification which is intend to provide experimenters with guidance about what types of replication they can perform.MethodThe research approach followed is structured according to the following activities: (1) a literature review of experiment replication in SE and in other disciplines, (2) identification of typical elements that compose an experimental configuration, (3) identification of different replications purposes and (4) development of a classification of experiment replications for SE.ResultsWe propose a classification of replications which provides experimenters in SE with guidance about what changes can they make in a replication and, based on these, what verification purposes such a replication can serve. The proposed classification helped to accommodate opposing views within a broader framework, it is capable of accounting for less similar replications to more similar ones regarding the baseline experiment.ConclusionThe aim of replication is to verify results, but different types of replication serve special verification purposes and afford different degrees of change. Each replication type helps to discover particular experimental conditions that might influence the results. The proposed classification can be used to identify changes in a replication and, based on these, understand the level of verification.     

A Replicated Experiment to Assess Requirements Inspection Techniques

This paper presents the independent replication of a controlled experiment which compared three defect detection techniques (Ad Hoc, Checklist, and Defect-based Scenario) for software requirements inspections, and evaluated the benefits of collection meetings after individual reviews. The results of our replication were partially different from those of the original experiment. Unlike the original experiment, we did not find any empirical evidence of better performance when using scenarios. To explain these negative findings we provide a list of hypotheses. On the other hand, the replication confirmed one result of the original experiment: the defect detection rate is not improved by collection meetings.The independent replication was made possible by the existence of an experimental kit provided by the original investigators. We discuss what difficulties we encountered in applying the package to our environment, as a result of different cultures and skills. Using our results, experience and suggestions, other researchers will be able to improve the original experimental design before attempting further replications.     

Further Experiences with Scenarios and Checklists

Software inspection is one of the best methods of verifying software documents. Software inspection is a complex process, with many possible variations, most of which have received little or no evaluation. This paper reports on the evaluation of one component of the inspection process, detection aids, specifically using Scenario or Checklist approaches. The evaluation is by subject-based experimentation, and is currently one of three independent experiments on the same hypothesis. The paper describes the experimental process, the resulting analysis of the experimental data, and attempts to compare the results in this experiment with the other experiments. This replication is broadly supportive of the results from the original experiment, namely, that the Scenario approach is superior to the Checklist approach; and that the meeting component of a software inspection is not an effective defect detection mechanism. This experiment also tentatively proposes additional relationships between general academic performance and individual inspection performance; and between meeting loss and group inspection performance.     

Replication and comparison of computational experiments in applied evolutionary computing: Common pitfalls and guidelines to avoid them

Replicating and comparing computational experiments in applied evolutionary computing may sound like a trivial task. Unfortunately, it is not so. Namely, many papers do not document experimental settings in sufficient detail, and hence replication of experiments is almost impossible. Additionally, some work fails to satisfy the thumb rules for Experimentation throughout all disciplines, such that all experiments should be conducted and compared under the same or stricter conditions. Also, because of the stochastic properties inherent in evolutionary algorithms (EAs), experimental results should always be rich enough with respect to Statistics. Moreover, the comparisons conducted should be based on suitable performance measures and show the statistical significance of one approach over others. Otherwise, the derived conclusions may fail to have scientific merits. The primary objective of this paper is to offer some preliminary guidelines and reminders for assisting researchers to conduct any replications and comparisons of computational experiments when solving practical problems, by the use of EAs in the future. The common pitfalls are explained, that solve economic load dispatch problems using EAs from concrete examples found in some papers.     

Building knowledge through families of experiments

Experimentation in software engineering is necessary but difficult. One reason is that there are a large number of context variables and, so, creating a cohesive understanding of experimental results requires a mechanism for motivating studies and integrating results. It requires a community of researchers that can replicate studies, vary context variables, and build models that represent the common observations about the discipline. The paper discusses the experience of the authors, based upon a collection of experiments, in terms of a framework for organizing sets of related studies. With such a framework, experiments can be viewed as part of common families of studies, rather than being isolated events. Common families of studies can contribute to important and relevant hypotheses that may not be suggested by individual experiments. A framework also facilitates building knowledge in an incremental manner through the replication of experiments within families of studies. To support the framework, the paper discusses the experiences of the authors in carrying out empirical studies, with specific emphasis on persistent problems encountered in experimental design, threats to validity, criteria for evaluation, and execution of experiments in the domain of software engineering     

Solving RCPSP/max by lazy clause generation

We present a generic exact method for minimizing the project duration of the resource-constrained project scheduling problem with generalized precedence relations (Rcpsp/max). This is a very general scheduling model with applications areas such as project management and production planning. Our method uses lazy clause generation, i.e., a hybrid of finite domain and Boolean satisfiability solving, in order to apply no-good learning and conflict-driven search to the solution generation. Our experiments show the benefit of lazy clause generation for finding an optimal solution and proving its optimality in comparison to other state-of-the-art exact and non-exact methods. In comparison to other methods, our method is able to find better solutions faster on the Rcpsp/max benchmarks. Indeed, our method closes 573 open problem instances and generates better solutions in most of the remaining instances. Surprisingly, although ours is an exact method, it outperforms the published non-exact methods on these benchmarks in terms of the quality of solutions.     

Stability analyses of group decision making

The importance of multicriteria models in a group decision making is increasingly being emphasized by researchers. One of the most significant of these models is the analytic hierarchy process (AHP). Through the AHP, decision makers are able to conduct a series of pairwise comparisons on pairs of criteria and priority indices can thereby be derived. However, the judgmental process can often be subjective as researchers have not put adequate emphasis on the stability and reliability of group weights observed through the process.

This paper develops a method of replication coupled with the use of the quality confidence intervals in order to generate invigorating debates on a particular issue before weight assignments are made. The replicated assignments are used to determine the group's priority indices. The aim is to enhance the ability of decision makers to make the same decisions when provided with similar environmental conditions. Ultimately, this will provide greater reliability to the derived outcomes. Obviously, the use of multicriteria modelling in subjective assessments would be meaningless if the decisions made were not consistent under the same conditions. In other words, these models would provide no guidance or benefit to decision makers.

In the paper, two case studies relating to the ranking of factors for selection of advanced technologies and issues for achieving competitiveness are analyzed utilizing an experimental group. Paired t-tests show that there are no differences between the rankings observed for the three replications for each respective case for almost all the criteria. Thus, the procedure provided here offers utility in group decision making.     

Repeatable software engineering experiments for comparing defect-detection techniques

Techniques for detecting defects in source code are fundamental to the success of any software development approach. A software development organization therefore needs to understand the utility of techniques such as reading or testing in its own environment. Controlled experiments have proven to be an effective means for evaluating software engineering techniques and gaining the necessary understanding about their utility. This paper presents a characterization scheme for controlled experiments that evaluate defect-detection techniques. The characterization scheme permits the comparison of results from similar experiments and establishes a context for cross-experiment analysis of those results. The characterization scheme is used to structure a detailed survey of four experiments that compared reading and testing techniques for detecting defects in source code. We encourage educators, researchers, and practitioners to use the characterization scheme in order to develop and conduct further instances of this class of experiments. By repeating this experiment we expect the software engineering community will gain quantitative insights about the utility of defect-detection techniques in different environments.This work was conducted while the author was with the Department of Computer Science, University of Kaiserslautern, 67653 Kaiserslautern, Germany.     

Large-scale information retrieval in software engineering - an experience report from industrial application

Software Engineering activities are information intensive. Research proposes Information Retrieval (IR) techniques to support engineers in their daily tasks, such as establishing and maintaining traceability links, fault identification, and software maintenance. We describe an engineering task, test case selection, and illustrate our problem analysis and solution discovery process. The objective of the study is to gain an understanding of to what extent IR techniques (one potential solution) can be applied to test case selection and provide decision support in a large-scale, industrial setting. We analyze, in the context of the studied company, how test case selection is performed and design a series of experiments evaluating the performance of different IR techniques. Each experiment provides lessons learned from implementation, execution, and results, feeding to its successor. The three experiments led to the following observations: 1) there is a lack of research on scalable parameter optimization of IR techniques for software engineering problems; 2) scaling IR techniques to industry data is challenging, in particular for latent semantic analysis; 3) the IR context poses constraints on the empirical evaluation of IR techniques, requiring more research on developing valid statistical approaches. We believe that our experiences in conducting a series of IR experiments with industry grade data are valuable for peer researchers so that they can avoid the pitfalls that we have encountered. Furthermore, we identified challenges that need to be addressed in order to bridge the gap between laboratory IR experiments and real applications of IR in the industry.     

Replicating software engineering experiments: a poisoned chalice or the Holy Grail

Recently, software engineering has witnessed a great increase in the amount of work with an empirical component; however, this work has often little or no established empirical framework within the topic to draw upon. Frequently, researchers use frameworks from other disciplines in an attempt to alleviate this deficiency. A common underpinning in these frameworks is that experimental replication is available as the cornerstone of knowledge discovery within the discipline. This paper investigates the issues involved in accepting this premise as a fundamental building block with empirical software engineering and recommends extending the traditional view of replication to improve the effectiveness of this essential process within our domain.     

Comparison of fuzzy logic based models for the multi-response surface problems with replicated response measures

A replicated multi-response experiment is a process that includes more than one responses with replications. One of the main objectives in these experiments is to estimate the unknown relationship between responses and input variables simultaneously. In general, classical regression analysis is used for modeling of the responses. However, in most practical problems, the assumptions for regression analysis cannot be satisfied. In this case, alternative modeling methods such as fuzzy logic based modeling approaches can be used. In this study, fuzzy least squares regression (FLSR) and fuzzy clustering based modeling methods, which are switching fuzzy C-regression (SFCR) and Takagi–Sugeno (TS) fuzzy model, are preferred. The novelty of the study is presenting the applicability of SFCR to the multi-response experiment data set with replicated response measures. Three real data set examples are given for application purposes. In order to compare the prediction performance of modeling approaches, root mean square error (RMSE) criteria is used. It is seen from the results that the SFCR gives the better prediction performance among the other fuzzy modeling approaches for the replicated multi-response experimental data sets.     

Experimental design and analysis in software engineering

The paper presents key activities necessary for designing and analyzing an experiment in software engineering. After explaining how to choose an appropriate research technique to fit project goals, the paper shows how to state a hypothesis and determine how much control is needed over the variables involved. If control is not possible, then a formal experiment is not possible; a case study may be a better approach. Next, the six stages of an experiment (conception, design, preparation, execution, analysis and dissemination) are described, with design examined in detail. Design considerations such as replication, randomization and local control are discussed, and design techniques such as crossing and nesting are explained. Finally, data analysis is shown to be a function both of the experimental design and the distribution of the data. Throughout, examples are given to show how the techniques are interpreted and used in software engineering.     

Knowledge-Sharing Issues in Experimental Software Engineering

Recently the awareness of the importance of replicating studies has been growing in the empirical software engineering community. The results of any one study cannot simply be extrapolated to all environments because there are many uncontrollable sources of variation between different environments. In our work, we have reasoned that the availability of laboratory packages for experiments can encourage better replications and complementary studies. However, even with effectively specified laboratory packages, transfer of experimental know-how can still be difficult. In this paper, we discuss the collaboration structures we have been using in the Readers’ Project, a bilateral project supported by the Brazilian and American national science agencies that is investigating replications and transfer of experimental know-how issues. In particular, we discuss how these structures map to the Nonaka–Takeuchi knowledge sharing model, a well-known paradigm used in the knowledge management literature. We describe an instantiation of the Nonaka–Takeuchi Model for software engineering experimentation, establishing a framework for discussing knowledge sharing issues related to experimental software engineering. We use two replications to illustrate some of the knowledge sharing issues we have faced and discuss the mechanisms we are using to tackle those issues in Readers’ Project.     

Mining transposed motifs in music

The discovery of frequent musical patterns (motifs) is a relevant problem in musicology. This paper introduces an unsupervised algorithm to address this problem in symbolically-represented musical melodies. Our algorithm is able to identify transposed patterns including exact matchings, i.e., null transpositions. We have tested our algorithm on a corpus of songs and the results suggest that our approach is promising, specially when dealing with songs that include non-exact repetitions.     

Investigation of individual factors impacting the effectiveness of requirements inspections: a replicated experiment

This paper presents a replication of an empirical study regarding the impact of individual factors on the effectiveness of requirements inspections. Experimental replications are important for verifying results and investigating the generality of empirical studies. We utilized the lab package and procedures from the original study, with some changes and additions, to conduct the replication with 69 professional developers in three different companies in Turkey. In general the results of the replication were consistent with those of the original study. The main result from the original study, which is supported in the replication, was that inspectors whose degree is in a field related to software engineering are less effective during a requirements inspection than inspectors whose degrees are in other fields. In addition, we found that Company, Experience, and English Proficiency impacted inspection effectiveness.     

Fast replication of out-of-plane microlens with polydimethylsiloxane and curable polymer (NOA73)

Out-of-plane microlens, as its in-plane counterpart, is an important micro optics component that can be used in building integrated micro-optic systems for many applications. In earlier publications from our group, an ultra violet (UV) lithography based technique for out-of-plane microlens fabrication was reported. In this paper, we report a replication technology for time-efficient fabrication of out-of-plane microlens made of a curable polymer, NOA73. Microlens of cured SU-8 polymer was fabricated using a unique tilted UV lithography process, polydimethylsiloxane (PDMS) was molded using the resulting SU-8 master to form a negative mold, curable polymer NOA73 was then casted in the PDMS mold and out-of-plane microlens replica made of NOA73 was finally obtained after curing. The entire replication process took less than 5 h. Since PDMS negative mold was reusable, multiple replications of the microlens could be done with the same mold and each replication only took about 30 min. Scanning electron microscopic (SEM) images showed that NOA73 microlens replica had almost identical shape as the SU-8 master. In Comparison to the SU-8 microlens, microlens replica of UV curable polymer had slightly longer focal length and smaller numerical aperture due to the lower refractive index of NOA73. In addition, NOA73 microlens replica also had improved spectral transmission. Because of its compatibility with soft lithography technique, the reported replication process may also be used to integrate out-of-plane microlens into micro-opto-electro-mechanical-systems (MOEMS) and BioMEMS chips.     

A product configuration analysis method for emotional design using a personal construct theory

Identifying emotion-related product attributes (perceived by consumers) is no easy task in the realm of emotional design. Conventionally, this process relies heavily on the researchers who conduct the Kansei experiments selecting product attributes such as color, form, and texture for Kansei studies. However, in so doing, other product attributes that also play a vital role in product-emotion associations might be neglected by the researchers. More importantly, the identification of product attributes should be based on consumer's point of view (and feelings). Accordingly, a personal construct theory based product configuration analysis method is proposed in this work. The method develops the customer's mind map for each Kansei tag in order to capture replications of candidate products. A means-value chain is used to generate targets which are later compared with candidate products by consumers. The comparison results could suggest product attributes that are relevant to the desired Kansei. The proposed approach is presented and illustrated using a case study of Graffiti designs on notebooks. Results obtained are discussed. It appears that the proposed method is promising in identifying product attributes with desired Kansei impacts.