A Replicated Experiment to Assess Requirements Inspection Techniques

This paper presents the independent replication of a controlled experiment which compared three defect detection techniques (Ad Hoc, Checklist, and Defect-based Scenario) for software requirements inspections, and evaluated the benefits of collection meetings after individual reviews. The results of our replication were partially different from those of the original experiment. Unlike the original experiment, we did not find any empirical evidence of better performance when using scenarios. To explain these negative findings we provide a list of hypotheses. On the other hand, the replication confirmed one result of the original experiment: the defect detection rate is not improved by collection meetings.The independent replication was made possible by the existence of an experimental kit provided by the original investigators. We discuss what difficulties we encountered in applying the package to our environment, as a result of different cultures and skills. Using our results, experience and suggestions, other researchers will be able to improve the original experimental design before attempting further replications.     

Further Experiences with Scenarios and Checklists

Software inspection is one of the best methods of verifying software documents. Software inspection is a complex process, with many possible variations, most of which have received little or no evaluation. This paper reports on the evaluation of one component of the inspection process, detection aids, specifically using Scenario or Checklist approaches. The evaluation is by subject-based experimentation, and is currently one of three independent experiments on the same hypothesis. The paper describes the experimental process, the resulting analysis of the experimental data, and attempts to compare the results in this experiment with the other experiments. This replication is broadly supportive of the results from the original experiment, namely, that the Scenario approach is superior to the Checklist approach; and that the meeting component of a software inspection is not an effective defect detection mechanism. This experiment also tentatively proposes additional relationships between general academic performance and individual inspection performance; and between meeting loss and group inspection performance.     

A Methodology for Constructing User-Oriented Requirements Specifications for Large-Scale Systems Using Electronic Hypermedia

requirements specifications are developed for large-scale systems, the final specification is usually an abstraction of the original requirements data into a text-based form that is often foreign to end-users. A method was developed for representing requirements through use of electronic multimedia. The resulting specification is capable of representing requirements and requirements data in a manner that is more representative of the real-world problem space than traditional specifications. This paper presents a method for incorporating multimedia exhibits, notably the results of rapid prototyping activities and animated simulation, into a requirements specification for large-scale C2I systems. To examine the effectiveness of the method, a multimedia requirements specification was developed based on an existing text specification for a real-world system. An experiment was also performed that showed the product of the methodology to be effective in increasing the understandability of the specification over that obtained from the text specification alone.     

Comparing Detection Methods For Software Requirements Inspections: A Replication Using Professional Subjects

Software requirements specifications (SRS) are often validated manually. One such process is inspection, in which several reviewers independently analyze all or part of the specification and search for faults. These faults are then collected at a meeting of the reviewers and author(s).Usually, reviewers use Ad Hoc or Checklist methods to uncover faults. These methods force all reviewers to rely on nonsystematic techniques to search for a wide variety of faults. We hypothesize that a Scenario-based method, in which each reviewer uses different, systematic techniques to search for different, specific classes of faults, will have a significantly higher success rate.In previous work we evaluated this hypothesis using 48 graduate students in computer science as subjects.We now have replicated this experiment using 18 professional developers from Lucent Technologies as subjects. Our goals were to (1) extend the external credibility of our results by studying professional developers, and to (2) compare the performances of professionals with that of the graduate students to better understand how generalizable the results of the less expensive student experiments were.For each inspection we performed four measurements: (1) individual fault detection rate, (2) team fault detection rate, (3) percentage of faults first identified at the collection meeting (meeting gain rate), and (4) percentage of faults first identified by an individual, but never reported at the collection meeting (meeting loss rate).For both the professionals and the students the experimental results are that (1) the Scenario method had a higher fault detection rate than either Ad Hoc or Checklist methods, (2) Checklist reviewers were no more effective than Ad Hoc reviewers, (3) Collection meetings produced no net improvement in the fault, and detection rate—meeting gains were offset by meeting losses,Finally, although specific measures differed between the professional and student populations, the outcomes of almost all statistical tests were identical. This suggests that the graduate students provided an adequate model of the professional population and that the much greater expense of conducting studies with professionals may not always be required.     

Using abstraction and model checking to detect safety violations inrequirements specifications

Exposing inconsistencies can uncover many defects in software specifications. One approach to exposing inconsistencies analyzes two redundant specifications, one operational and the other property-based, and reports discrepancies. This paper describes a “practical” formal method, based on this approach and the SCR (software cost reduction) tabular notation, that can expose inconsistencies in software requirements specifications. Because users of the method do not need advanced mathematical training or theorem-proving skills, most software developers should be able to apply the method without extraordinary effort. This paper also describes an application of the method which exposed a safety violation in the contractor-produced software requirements specification of a sizable, safety-critical control system. Because the enormous state space of specifications of practical software usually renders direct analysis impractical, a common approach is to apply abstraction to the specification. To reduce the state space of the control system specification, two “pushbutton” abstraction methods were applied, one which automatically removes irrelevant variables and a second which replaces the large, possibly infinite, type sets of certain variables with smaller type sets. Analyzing the reduced specification with the model checker Spin uncovered a possible safety violation. Simulation demonstrated that the safety violation was not spurious but an actual defect in the original specification     

Replication of an experiment on linguistic tool support for consolidation of requirements from multiple sources

Large market-driven software companies continuously receive large numbers of requirements and change requests from multiple sources. The task of analyzing those requests against each other and against already analyzed or implemented functionality then recording similarities between them, also called the requirements consolidation task, may be challenging and time consuming. This paper presents a replicated experiment designed to further investigate the linguistic tool support for the requirements consolidation task. In this replication study, 45 subjects, working in pairs on the same set of requirements as in the original study, were assigned to use two methods for the requirements consolidation: (1) lexical similarity and (2) searching and filtering. The results show that the linguistic method used in this experiment is not more efficient in consolidating requirements than the searching and filtering method, which contradicts the findings of the original study. However, we confirm the previous results that the assisted method (lexical similarity) can deliver more correct links and miss fewer links than the manual method (searching and filtering).     

基于OTT策略的组合测试用例优先级排序方法

在组合测试用例优先级排序问题中,通常采用组合覆盖率为排序标准,该方法能够尽快满足覆盖率的要求,但其排序影响因子较为单一,缺陷检测能力不稳定。针对此问题,该文结合One-Test-at-a-Time（OTT）策略提出了一种在线调整的组合测试用例优先级排序方法,引入多重待覆盖率、测试用例失效率和测试用例重要程度3个影响因子用于衡量组合测试用例优先级。根据测试用例的在线测试反馈信息,该方法能够实时调整组合测试用例的优先级,增加了优先级排序的准确性。实验结果表明,相较于original、基于单一覆盖和多重覆盖的组合测试优先级排序方法,该方法在缺陷检测能力和稳定性上具有更强的竞争力。     

Integrating software specifications into intrusion detection

There exist a number of Intrusion Detection Systems (IDSs) that detect computer attacks based on some defined attack scenarios. The attack scenarios or security requirements in some of these IDSs are specified in attack specification languages that are different from software specification languages. The use of two different languages for software specification and attack specification may generate redundant and conflicting requirements. The advantage of using the same language for both functional specifications and attacks specifications is that software designers can address the two different issues without learning two types of languages. We present a method of integrating Abstract State Machine Language (AsmL) and Unified Modeling Language (UML) state charts that are extended finite state machine based software specification languages, with an open source IDS Snort. This work provides AsmL and UML users an IDS that they can use without knowing how to write Snort rules. We automatically translate attack scenarios written in AsmL and UML state charts into Snort rules with context information. The original Snort is modified so that it can use the rules automatically generated by the translator. Adding context information to Snort rules improves the detection capability of Snort. To show the efficacy of the presented approach, we have built a prototype and evaluated it using a number of well-known attack scenarios.     

基于凸包的电池片边缘缺陷检测算法

为了实现电池片分选机的快速分拣,需要加快电池片图像各项检测项目的速度。其中,电池片边缘缺损与破片的检测是把控产品质量的首要检测项目。已有的检测方法图像处理时间较长,处理过程可复用性差,处理结果受环境影响大。因此,提出一种通过凸包判断电池片边缘是否存在残缺,同时计算缺陷位置像素深度的快速边缘缺损检测办法。该方法对成像环境要求低,且可以提供电池片像素中心及偏转角度等电池片姿态信息,缩短了其他检测过程的时间。最后,实验证明该算法的平均时间消耗小于50ms,同时,大量电池片检测结果的误检率和漏检率均小于0.5%。     

Effectiveness for detecting faults within and outside the scope of testing techniques: an independent replication

The verification and validation activity plays a fundamental role in improving software quality. Determining which the most effective techniques for carrying out this activity are has been an aspiration of experimental software engineering researchers for years. This paper reports a controlled experiment evaluating the effectiveness of two unit testing techniques (the functional testing technique known as equivalence partitioning (EP) and the control-flow structural testing technique known as branch testing (BT)). This experiment is a literal replication of Juristo et al. (2013). Both experiments serve the purpose of determining whether the effectiveness of BT and EP varies depending on whether or not the faults are visible for the technique (InScope or OutScope, respectively). We have used the materials, design and procedures of the original experiment, but in order to adapt the experiment to the context we have: (1) reduced the number of studied techniques from 3 to 2; (2) assigned subjects to experimental groups by means of stratified randomization to balance the influence of programming experience; (3) localized the experimental materials and (4) adapted the training duration. We ran the replication at the Escuela Politécnica del Ejército Sede Latacunga (ESPEL) as part of a software verification & validation course. The experimental subjects were 23 master’s degree students. EP is more effective than BT at detecting InScope faults. The session/program and group variables are found to have significant effects. BT is more effective than EP at detecting OutScope faults. The session/program and group variables have no effect in this case. The results of the replication and the original experiment are similar with respect to testing techniques. There are some inconsistencies with respect to the group factor. They can be explained by small sample effects. The results for the session/program factor are inconsistent for InScope faults. We believe that these differences are due to a combination of the fatigue effect and a technique x program interaction. Although we were able to reproduce the main effects, the changes to the design of the original experiment make it impossible to identify the causes of the discrepancies for sure. We believe that further replications closely resembling the original experiment should be conducted to improve our understanding of the phenomena under study.     

New Object-Oriented PROGRES for Specifying the Conceptual Design Tool GraCAD

This paper deals with the application of graph transformations for the specification of conceptual design tools. We show how the graph rewriting system PROGRES is used for specifying the graph part of the conceptual method for architects in which functional requirements of the building to be designed are elicited by means of graph structures. The consistency of the specified requirements and whether a design matches those requirements is verified with graph constraint checkers. We consider how the new object-oriented extensions of the PROGRES language, i.e. packages and node objects with redefinable methods can be used to achieve the required constraint monitoring and preserving functions in the form of graph checker objects. The prototype for our method, called GraCAD, is created with UPGRADE - the recently developed Java framework for developing visual applications based on a PROGRES specification, and the commercial system for architects ArchiCAD.     

Ellipsoidal decision regions for motif-based patterned fabric defect detection

This paper presents a study of using ellipsoidal decision regions for motif-based patterned fabric defect detection, the result of which is found to improve the original detection success using max-min decision region of the energy-variance values. In our previous research, max-min decision region was found to be effective in distinct cases but ill detect the ambiguous false-positive and false-negative cases. To alleviate this problem, we first assume that the energy-variance values can be described by a Gaussian mixture model. Second, we apply k-means clustering to roughly identify the various clusters that make up the entire data population. Third, convex hull of each cluster is employed as a basis for fitting an ellipsoidal decision region over it. Defect detection is then based on these ellipsoidal regions. To validate the method, three wallpaper groups are evaluated using the new ellipsoidal regions, and compared with those results obtained using the max-min decision region. For the p2 group, success rate improves from 93.43% to 100%. For the pmm group, success rate improves from 95.9% to 96.72%, while the p4 m group records the same success rate at 90.77%. This demonstrates the superiority of using ellipsoidal decision regions in motif-based defect detection.     

基于自编码和知识蒸馏的表面缺陷检测方法

针对传统的表面缺陷检测方法只能对具有高对比度或低噪声的明显缺陷轮廓进行检测的问题,提出了一种基于自编码和知识蒸馏的表面缺陷检测方法来准确定位和分类从实际工业环境捕获的输入图像中出现的缺陷。首先,设计了一种级联自动编码器（CAE）架构用于分割和定位缺陷,其目的是将输入的原始图像转换为基于CAE的预测蒙版;其次,利用阈值模块对预测结果进行二值化以获得准确的缺陷轮廓;然后,把缺陷区域检测器提取并裁剪出来的缺陷区域视为下一个模块的输入;最后,将CAE分割结果的缺陷区域通过知识蒸馏进行类别分类。实验结果表明,与其他几种表面缺陷检测方法相比,所提出的方法综合性能最好,其缺陷检测平均准确率为97.00%。该方法能够有效地对较小的、边缘不清晰的缺陷进行分割,满足对物品表面缺陷实时分割检测的工程要求。     

An automated vision based on-line novel percept detection method for a mobile robot

It is generally agreed that learning, either supervised or unsupervised, can provide the best possible specification of known classes and offer inference for outlier detection by a dissimilarity threshold from the nominal feature space. Novel percept detection can take a step further by investigating whether these outliers form new dense clusters in both the feature space and the image space. By defining a novel percept to be a pattern group that has not been seen before in the feature space and the image space, in this paper, a non-conventional approach is proposed for multiple-novel-percept detection problem in robotic applications. Based on a computer vision system inspired loosely by neurobiological evidence, our approach can work in near real time for highly sparse high-dimensional feature vectors extracted from image patches while maintaining robustness to image transformations. Experiments conducted in an indoor environment and an outdoor environment demonstrate the efficacy of our method.     

应用代数规约和数据变异策略测试类

在应用基于代数规约测试类时,虽然可以有效解决测试数据生成和测试结果判断这两个测试难题,但是,因为该方法常常随机生成测试数据,会导致对于同一待测试类两次测试会得到不同的测试效果。为克服这个缺陷,提出应用数据变异方法,对已生成的测试用例进行变异。变异测试实验表明：该方法能够有效地改善代数测试方法的测试效果。     

Controlled experiment replication in evaluation of e-learning system’s educational influence

We believe that every effectiveness evaluation should be replicated at least in order to verify the original results and to indicate evaluated e-learning system’s advantages or disadvantages. This paper presents the methodology for conducting controlled experiment replication, as well as, results of a controlled experiment and an internal replication that investigated the effectiveness of intelligent authoring shell eXtended Tutor–Expert System (xTEx-Sys). The initial and the replicated experiment were based on our approach that combines classical two-group experimental design and with factoral design. A trait that distinguishes this approach from others is the existence of arbitrary number of checkpoint-tests to determine the effectiveness in intermediate states. We call it a pre-and-post test control group experimental design with checkpoint-tests. The gained results revealed small or even negative effect sizes, which could be explained by the fact that the xTEx-Sys’s domain knowledge presentation is rather novel for students and therefore difficult to grasp and apply in earlier phases of the experiment. In order to develop and improve the xTEx-Sys, further experiments must be conducted.