A process for managing interaction between experimenters to get useful similar replications

ContextA replication is the repetition of an experiment. Several efforts have been made to adopt replication as a common practice in software engineering. There are different types of replications, depending on their purpose. Similar replications keep the experimental conditions as alike as possible to the original ones. External similar replications, where the replicating experimenters are not the same people as the original experimenters, have been a stumbling block. Several attempts at combining the results of replications have resulted in failure. Software engineering does not appear to be well suited to such replications, because it works with complex experimentally immature contexts. Software engineering settings have a large number of variables, and the role that many of them play is unknown. A successful (or useful) similar replication helps to better understand the phenomenon under study by verifying results and/or identifying contextual variables that could influence (or not) the results, through the combination of experimental results.ObjectiveTo be able to get successful similar replications, there needs to be interaction between original and replicating experimenters. In this paper, we propose an interaction process for achieving successful similar replications.MethodThis process consists of: an adaptation meeting, where experimenters tailor the experiment to the new setting; querying, to settle occasional inquiries while the experiment is being run; and a combination meeting, where experimenters meet to discuss the combination of replication outcomes with previous results. To check its effectiveness, the process has been tested on three different replications of the same experiment.ResultsThe proposed interaction process has helped to identify new contextual variables that could potentially influence (or not) the experimental results in the three replications run. Additionally, the interaction process has helped to uncover certain problems and deviations that occurred during some of the replications that we would have not been aware of otherwise.ConclusionsThere are signs that suggest that it is possible to get successful similar replications in software engineering experimentation, when there is appropriate interaction among experimenters.     

Investigations about replication of empirical studies in software engineering: A systematic mapping study

ContextTwo recent mapping studies which were intended to verify the current state of replication of empirical studies in Software Engineering (SE) identified two sets of studies: empirical studies actually reporting replications (published between 1994 and 2012) and a second group of studies that are concerned with definitions, classifications, processes, guidelines, and other research topics or themes about replication work in empirical software engineering research (published between 1996 and 2012).ObjectiveIn this current article, our goal is to analyze and discuss the contents of the second set of studies about replications to increase our understanding of the current state of the work on replication in empirical software engineering research.MethodWe applied the systematic literature review method to build a systematic mapping study, in which the primary studies were collected by two previous mapping studies covering the period 1996–2012 complemented by manual and automatic search procedures that collected articles published in 2013.ResultsWe analyzed 37 papers reporting studies about replication published in the last 17 years. These papers explore different topics related to concepts and classifications, presented guidelines, and discuss theoretical issues that are relevant for our understanding of replication in our field. We also investigated how these 37 papers have been cited in the 135 replication papers published between 1994 and 2012.ConclusionsReplication in SE still lacks a set of standardized concepts and terminology, which has a negative impact on the replication work in our field. To improve this situation, it is important that the SE research community engage on an effort to create and evaluate taxonomy, frameworks, guidelines, and methodologies to fully support the development of replications.     

Research in software engineering: an analysis of the literature

In this paper, we examine the state of software engineering (SE) research from the point of view of the following research questions:

• 1.What topics do SE researchers address?
• 2.What research approaches do SE researchers use?
• 3.What research methods do SE researchers use?
• 4.On what reference disciplines does SE research depend?
• 5.At what levels of analysis do SE researchers conduct research?

To answer those questions, we examined 369 papers in six leading research journals in the SE field, answering those research questions for each paper.From that examination, we conclude that SE research is diverse regarding topic, narrow regarding research approach and method, inwardly-focused regarding reference discipline, and technically focused (as opposed to behaviorally focused) regarding level of analysis.We pass no judgment on the SE field as a result of these findings. Instead, we present them as groundwork for future SE research efforts.     

The role of replications in Empirical Software Engineering

Replications play a key role in Empirical Software Engineering by allowing the community to build knowledge about which results or observations hold under which conditions. Therefore, not only can a replication that produces similar results as the original experiment be viewed as successful, but a replication that produce results different from those of the original experiment can also be viewed as successful. In this paper we identify two types of replications: exact replications, in which the procedures of an experiment are followed as closely as possible; and conceptual replications, in which the same research question is evaluated by using a different experimental procedure. The focus of this paper is on exact replications. We further explore them to identify two sub-categories: dependent replications, where researchers attempt to keep all the conditions of the experiment the same or very similar and independent replications, where researchers deliberately vary one or more major aspects of the conditions of the experiment. We then discuss the role played by each type of replication in terms of its goals, benefits, and limitations. Finally, we highlight the importance of producing adequate documentation for an experiment (original or replication) to allow for replication. A properly documented replication provides the details necessary to gain a sufficient understanding of the study being replicated without requiring the replicator to slavishly follow the given procedures.

Personality,emotional intelligence and work preferences in software engineering: An empirical study

ContextThere is an increasing awareness among Software Engineering (SE) researchers and practitioners that more focus is needed on understanding the engineers developing software. Previous studies show significant associations between the personalities of software engineers and their work preferences.ObjectiveVarious studies on personality in SE have found large, small or no effects and there is no consensus on the importance of psychometric measurements in SE. There is also a lack of studies employing other psychometric instruments or using larger datasets. We aim to evaluate our results in a larger sample, with software engineers in an earlier state of their career, using advanced statistics.MethodAn operational replication study where extensive psychometric data from 279 master level students have been collected in a SE program at a Swedish University. Personality data based on the Five-Factor Model, Trait Emotional Intelligence Questionnaire and Self-compassion have been collected. Statistical analysis investigated associations between psychometrics and work preferences and the results were compared to our previous findings from 47 SE professionals.ResultsAnalysis confirms existence of two main clusters of software engineers; one with more “intense” personalities than the other. This corroborates our earlier results on SE professionals. The student data also show similar associations between personalities and work preferences. However, for other associations there are differences due to the different population of subjects. We also found connections between the emotional intelligence and work preferences, while no associations were found for self-compassion.ConclusionThe associations can help managers to predict and adapt projects and tasks to available staff. The results also show that the Emotional Intelligence instrument can be predictive. The research methods and analytical tools we employ can detect subtle associations and reflect differences between different groups and populations and thus can be important tools for future research as well as industrial practice.     

Effectiveness for detecting faults within and outside the scope of testing techniques: an independent replication

The verification and validation activity plays a fundamental role in improving software quality. Determining which the most effective techniques for carrying out this activity are has been an aspiration of experimental software engineering researchers for years. This paper reports a controlled experiment evaluating the effectiveness of two unit testing techniques (the functional testing technique known as equivalence partitioning (EP) and the control-flow structural testing technique known as branch testing (BT)). This experiment is a literal replication of Juristo et al. (2013). Both experiments serve the purpose of determining whether the effectiveness of BT and EP varies depending on whether or not the faults are visible for the technique (InScope or OutScope, respectively). We have used the materials, design and procedures of the original experiment, but in order to adapt the experiment to the context we have: (1) reduced the number of studied techniques from 3 to 2; (2) assigned subjects to experimental groups by means of stratified randomization to balance the influence of programming experience; (3) localized the experimental materials and (4) adapted the training duration. We ran the replication at the Escuela Politécnica del Ejército Sede Latacunga (ESPEL) as part of a software verification & validation course. The experimental subjects were 23 master’s degree students. EP is more effective than BT at detecting InScope faults. The session/program and group variables are found to have significant effects. BT is more effective than EP at detecting OutScope faults. The session/program and group variables have no effect in this case. The results of the replication and the original experiment are similar with respect to testing techniques. There are some inconsistencies with respect to the group factor. They can be explained by small sample effects. The results for the session/program factor are inconsistent for InScope faults. We believe that these differences are due to a combination of the fatigue effect and a technique x program interaction. Although we were able to reproduce the main effects, the changes to the design of the original experiment make it impossible to identify the causes of the discrepancies for sure. We believe that further replications closely resembling the original experiment should be conducted to improve our understanding of the phenomena under study.     

The role of non-exact replications in software engineering experiments

In no science or engineering discipline does it make sense to speak of isolated experiments. The results of a single experiment cannot be viewed as representative of the underlying reality. Experiment replication is the repetition of an experiment to double-check its results. Multiple replications of an experiment increase the confidence in its results. Software engineering has tried its hand at the identical (exact) replication of experiments in the way of the natural sciences (physics, chemistry, etc.). After numerous attempts over the years, apart from experiments replicated by the same researchers at the same site, no exact replications have yet been achieved. One key reason for this is the complexity of the software development setting, which prevents the many experimental conditions from being identically reproduced. This paper reports research into whether non-exact replications can be of any use. We propose a process aimed at researchers running non-exact replications. Researchers enacting this process will be able to identify new variables that are possibly having an effect on experiment results. The process consists of four phases: replication definition and planning, replication operation and analysis, replication interpretation, and analysis of the replication’s contribution. To test the effectiveness of the proposed process, we have conducted a multiple-case study, revealing the variables learned from two different replications of an experiment.     

Software engineering article types: An analysis of the literature

The software engineering (SE) community has recently recognized that the field lacks well-established research paradigms and clear guidance on how to write good research reports. With no comprehensive guide to the different article types in the field, article writing and reviewing heavily depends on the expertise and the understanding of the individual SE actors.In this work, we classify and describe the article types published in SE with an emphasis on what is required for publication in journals and conference proceedings. Theoretically, we consider article types as genres, because we assume that each type of article has a specific function and a particular communicative purpose within the community, which the members of the community can recognize. We draw on written sources available, i.e. the instructions to authors/reviewers of major SE journals, the calls for papers of major SE conferences, and previous research published on the topic.Despite the fragmentation and limitations of the sources studied, we are able to propose a classification of different SE article types. Such classification helps in guiding the reader through the SE literature, and in making the researcher reflect on directions for improvements.     

A Replicated Experiment to Assess Requirements Inspection Techniques

This paper presents the independent replication of a controlled experiment which compared three defect detection techniques (Ad Hoc, Checklist, and Defect-based Scenario) for software requirements inspections, and evaluated the benefits of collection meetings after individual reviews. The results of our replication were partially different from those of the original experiment. Unlike the original experiment, we did not find any empirical evidence of better performance when using scenarios. To explain these negative findings we provide a list of hypotheses. On the other hand, the replication confirmed one result of the original experiment: the defect detection rate is not improved by collection meetings.The independent replication was made possible by the existence of an experimental kit provided by the original investigators. We discuss what difficulties we encountered in applying the package to our environment, as a result of different cultures and skills. Using our results, experience and suggestions, other researchers will be able to improve the original experimental design before attempting further replications.     

Triangulation as a basis for knowledge discovery in software engineering

Everyone agrees that reliable empirical “facts” must be derived from a series of investigations. However, the question of what is an appropriate framework for this series is an unresolved issue. Other articles in this issue present replication as the basis for this framework (Shull FJ, Carver JC, Vegas S, Juristo N (2008) The role of replications in Empirical Software Engineering, J Empir Softw Eng (in press)); however, alternatives exist! This brief note argues triangulation should be considered as the basis for this knowledge discovery (from empirical evaluation) strategy.

Evaluating the productivity of a reference-based programming approach: A controlled experiment

ContextDomain engineering aims at facilitating software development in an efficient and economical way. One way to measure that is through productivity indicators, which refer to the ability of creating a quality software product in a limited period and with limited resources. Many approaches have been devised to increase productivity; however, these approaches seem to suffer from a tension between expressiveness on the one hand, and applicability (or the lack of it) in providing guidance for developers.ObjectiveThis paper evaluates the applicability and efficiency of adopting a domain engineering approach, called Application-based DOmain Modeling (ADOM), in the context of the programming task with Java, and thus termed ADOM-Java, for improving productivity in terms of code quality and development time.MethodTo achieve that objective we have qualitatively evaluate the approach using questionnaires and following a text analysis procedure. We also set a controlled experiment in which 50 undergraduate students performed a Java-based programming task using either ADOM-Java or Java alone.ResultsThe qualitative evaluation reveal that the approach is easy to uses and provides valuable guidance. Nevertheless, it requires training. The outcomes of the experiment indicate that the approach is applicable and that the students that used ADOM-Java achieved better code quality, as well as better functionality and within less time than the students who used only Java.ConclusionThe results of the experiments imply that by providing a code base equipped with reuse guidelines for programmers can increase programming productivity in terms of quality and development time. These guidelines may also enforce coding standards and architectural design.     

Highly-cited papers in software engineering: The top-100

ContextAccording to the search reported in this paper, as of this writing (May 2015), a very large number of papers (more than 70,000) have been published in the area of Software Engineering (SE) since its inception in 1968. Citations are crucial in any research area to position the work and to build on the work of others. Identification and characterization of highly-cited papers are common and are regularly reported in various disciplines.ObjectiveThe objective of this study is to identify the papers in the area of SE that have influenced others the most as measured by citation count. Studying highly-cited SE papers helps researchers to see the type of approaches and research methods presented and applied in such papers, so as to be able to learn from them to write higher quality papers which will likely receive high citations.MethodTo achieve the above objective, we conducted a study, comprised of five research questions, to identify and classify the top-100 highly-cited SE papers in terms of two metrics: total number of citations and average annual number of citations.ResultsBy total number of citations, the top paper is "A metrics suite for object-oriented design", cited 1817 times and published in 1994. By average annual number of citations, the top paper is "QoS-aware middleware for Web services composition", cited 154.2 times on average annually and published in 2004.ConclusionIt is concluded that it is important to identify the highly-cited SE papers and also to characterize the overall citation landscape in the SE field. We hope that this paper will encourage further discussions in the SE community towards further analysis and formal characterization of the highly-cited SE papers.     

细粒度图像分类的互补注意力方法

目的 由于分类对象具有细微类间差异和较大类内变化的特点,细粒度分类一直是一个具有挑战性的任务。绝大多数方法利用注意力机制学习目标中显著的局部特征。然而,传统的注意力机制往往只关注了目标最显著的局部特征,同时抑制其他区域的次级显著信息,但是这些抑制的信息中通常也含有目标的有效特征。为了充分提取目标中的有效显著特征,本文提出了一种简单而有效的互补注意力机制。方法 基于SE（squeeze-and-excitation）注意力机制,提出了一种新的注意力机制,称为互补注意力机制（complemented SE,CSE）。既从原始特征中提取主要的显著局部特征,也从抑制的剩余通道信息中提取次级显著特征,这些特征之间具有互补性,通过融合这些特征可以得到更加高效的特征表示。结果 在CUB-Birds（Caltech-UCSD Birds-200-2011）、Stanford Dogs、Stanford Cars和FGVC-Aircraft（fine-grained visual classification of aircraft）4个细粒度数据集上对所提方法进行验证,以ResNet50为主干网络,在测试集上的分类精度分别达到了87.9%、89.1%、93.9%和92.4%。实验结果表明,所提方法在CUB-Birds和Stanford Dogs两个数据集上超越了当前表现最好的方法,在Stanford Cars和FGVC-Aircraft数据集的表现也接近当前主流方法。结论 本文方法着重提升注意力机制提取特征的能力,得到高效的目标特征表示,可用于细粒度图像分类和特征提取相关的计算机视觉任务。     

A systematic review of quasi-experiments in software engineering

Background:Experiments in which study units are assigned to experimental groups nonrandomly are called quasi-experiments. They allow investigations of cause–effect relations in settings in which randomization is inappropriate, impractical, or too costly.Problem outline:The procedure by which the nonrandom assignments are made might result in selection bias and other related internal validity problems. Selection bias is a systematic (not happening by chance) pre-experimental difference between the groups that could influence the results. By detecting the cause of the selection bias, and designing and analyzing the experiments accordingly, the effect of the bias may be reduced or eliminated.Research method:To investigate how quasi-experiments are performed in software engineering (SE), we conducted a systematic review of the experiments published in nine major SE journals and three conference proceedings in the decade 1993–2002.Results:Among the 113 experiments detected, 35% were quasi-experiments. In addition to field experiments, we found several applications for quasi-experiments in SE. However, there seems to be little awareness of the precise nature of quasi-experiments and the potential for selection bias in them. The term “quasi-experiment” was used in only 10% of the articles reporting quasi-experiments; only half of the quasi-experiments measured a pretest score to control for selection bias, and only 8% reported a threat of selection bias. On average, larger effect sizes were seen in randomized than in quasi-experiments, which might be due to selection bias in the quasi-experiments.Conclusion:We conclude that quasi-experimentation is useful in many settings in SE, but their design and analysis must be improved (in ways described in this paper), to ensure that inferences made from this kind of experiment are valid.     

Citations,research topics and active countries in software engineering: A bibliometrics study

Context: An enormous number of papers (more than 70,000) have been published in the area of Software Engineering (SE) since its inception in 1968. To better characterize and understand this massive research literature, there is a need for comprehensive bibliometrics assessments in this vibrant field.Objective: The objective of this study is to utilize automated citation and topic analysis to characterize the software engineering research literature over the years. While a few bibliometrics studies have appeared in the field of SE, this article aims to be the most comprehensive bibliometrics assessments in this vibrant field.Method: To achieve the above objective, we report in this paper a bibliometrics study with data collected from Scopus database consisting of over 70,000 articles. For thematic analysis, we used topic modeling to automatically generate the most probable topic distributions given the data.Results: We found that number of papers published per year has grown tremendously and currently 6000–7000 papers are published every year. At the same time, nearly half of the papers are not cited at all. Using text mining of articles titles, we found that currently the hot research topics in software engineering are: (1) web services, (2) mobile and cloud computing, (3) industrial (case) studies, (4) source code and (5) test generation. Finally, we found that a small share of large countries produce the majority of the papers in SE while small European countries are proportionally the most active in the area of SE, based on the number of papers.Conclusion: Due to large volumes of research in SE, we suggest using the automated analysis of bibliometrics as we have done in this paper. By picking out the most cited papers, we can present the land marks of SE and, with thematic analysis, we can characterize the entire field. This can be useful for students and other new comers to SE and for presenting our achievements to other disciplines. In particular, we see and report the value of such an analysis in situations where performing a full scale SLR is not feasible due to restrictions on time or to lack of exact research questions.     

A HCI technique for improving requirements elicitation

ContextTo develop usable software we need to understand the users that will interact with the system. Personas is a HCI technique that gathers information about users in order to comprehend their characteristics. This information is used to define fictitious persons on which development should focus. Personas provides an understanding of the user, often overlooked in SE developments.ObjectiveThe goal of our research is to modify Personas to readily build the technique into the requirements stage of regular SE developments.MethodWe tried to apply Cooper’s version of the Personas technique and we found shortcomings in both the definition of the procedure to be enacted and the formalization of the product resulting from the execution of each step of the Personas technique. For each of these limitations (up to a total of 11), we devised an improvement to be built into Personas. We have incorporated these improvements into a SE version of Personas. The improved Personas avoid the weaknesses encountered by an average software developer unfamiliar with HCI techniques applying the original Personas.ResultsWe aim to improve requirements elicitation through the use of Personas. We have systematized and formalized Personas in the SE tradition in order to build this new version of the technique into the requirements stage. We have applied our proposal in an application example.ConclusionThe integration of Personas into the SE requirements stage might improves the understanding of what the software product should do and how it should behave. We have modified the HCI Personas technique to comply with the levels of systematization required by SE. We have enriched the SE requirements process by incorporating Personas activities into requirements activities. Requirements elicitation and requirements analysis are the RE activities most affected by incorporating Personas.     

What went right? An analysis of the protective factors in aviation near misses

Abstract

Learning from successful safety outcomes, or what went right, is an important emerging component of maintaining safe systems. Accordingly, there are increasing calls to study normal performance in near misses as a part of safety management activities. Despite this, there is limited guidance on how to accomplish this in practice. This article presents a study in which using Rasmussen’s risk management framework to analyse 16 serious incidents from the aviation domain. The findings show that a network of protective factors prevents accidents with factors identified across the sociotechnical system. These protective networks share many properties with those identified in accidents. The article demonstrates that is possible to identify these networks of protective factors from incident investigation reports. The theoretical implications of these results and future research opportunities are discussed.

Practitioner Statement: The analysis of near misses is an important part of safety management activities. This article demonstrates that Rasmussen?s risk management framework can be used to identify networks of protective factors which prevent accidents. Safety practitioners can use the framework described to discover and support the system-wide networks of protective factors.     

PBL 教学法在验证性实验中的应用研究

验证性实验是通信工程专业实验课程的一个重要组成部分,简单易操作,但常因技术含量不高而不被重视。 实际上结合PBL问题教学法,只要对验证性实验进行合理的设计和有效的组织,虽不能像综合设计实验那样,能够直接提高学 生的设计能力和创新能力,但可以巩固通信基础理论,而且通过实验箱测得的信号都是实际信号,某种程度上还可以更好地验 证了通信系统的工程特性,对基础理论进行相对真实的模拟。     

Automatic web accessibility metrics: Where we are and where we can go

The fact that several web accessibility metrics exist may be evidence of a lack of a comparison framework that highlights how well they work and for what purposes they are appropriate. In this paper we aim at formulating such a framework, demonstrating that it is feasible, and showing the findings we obtained when we applied it to seven existing automatic accessibility metrics. The framework encompasses validity, reliability, sensitivity, adequacy and complexity of metrics in the context of four scenarios where the metrics can be used. The experimental demonstration of the viability of the framework is based on applying seven published metrics to more than 1500 web pages and then operationalizing the notions of validity-as-conformance, adequacy and complexity. Our findings lead us to conclude that the Web Accessibility Quantitative Metric, Page Measure and Web Accessibility Barrier are the metrics that achieve the highest levels of quality (out of the seven that we examined). Finally, since we did not analyse reliability, sensitivity and validity-in-use, this paper provides guidance to address them in what are new research avenues.