Tester interactivity makes a difference in search-based software testing: A controlled experiment

Context: Search-based software testing promises to provide users with the ability to generate high quality test cases, and hence increase product quality, with a minimal increase in the time and effort required.The development of the Interactive Search-Based Software Testing (ISBST) system was motivated by a previous study to investigate the application of search-based software testing (SBST) in an industrial setting. ISBST allows users to interact with the underlying SBST system, guiding the search and assessing the results. An industrial evaluation indicated that the ISBST system could find test cases that are not created by testers employing manual techniques. The validity of the evaluation was threatened, however, by the low number of participants.Objective: This paper presents a follow-up study, to provide a more rigorous evaluation of the ISBST system.Method: To assess the ISBST system a two-way crossover controlled experiment was conducted with 58 students taking a Verification and Validation course. The NASA Task Load Index (NASA-TLX) is used to assess the workload experienced by the participants in the experiment.Results:The experimental results validated the hypothesis that the ISBST system generates test cases that are not found by the same participants employing manual testing techniques. A follow-up laboratory experiment also investigates the importance of interaction in obtaining the results.In addition to this main result, the subjective workload was assessed for each participant by means of the NASA-TLX tool. The evaluation showed that, while the ISBST system required more effort from the participants, they achieved the same performance.Conclusions: The paper provides evidence that the ISBST system develops test cases that are not found by manual techniques, and that interaction plays an important role in achieving that result.     

A survey on search-based software design

This survey investigates search-based approaches to software design. The basics of the most popular meta-heuristic algorithms are presented as background to the search-based viewpoint. Software design is considered from a wide viewpoint, including topics that can also be categorized as software maintenance or re-engineering. Search-based approaches have been used in research from the high architecture design level to software clustering and finally software refactoring. Enhancing and predicting software quality with search-based methods is also taken into account as a part of the design process. The background for the underlying software engineering problems is discussed, after which search-based approaches are presented. Summarizing remarks and tables collecting the fundamental issues of approaches for each type of problem are given. The choices regarding critical decisions, such as representation and fitness function, when used in meta-heuristic search algorithms, are emphasized and discussed in detail. Ideas for future research directions are also given.     

A search-based framework for automatic testing of MATLAB/Simulink models

Search-based test-data generation has proved successful for code-level testing but almost no search-based work has been carried out at higher levels of abstraction. In this paper the application of such approaches at the higher levels of abstraction offered by MATLAB/Simulink models is investigated and a wide-ranging framework for test-data generation and management is presented. Model-level analogues of code-level structural coverage criteria are presented and search-based approaches to achieving them are described. The paper also describes the first search-based approach to the generation of mutant-killing test data, addressing a fundamental limitation of mutation testing. Some problems remain whatever the level of abstraction considered. In particular, complexity introduced by the presence of persistent state when generating test sequences is as much a challenge at the Simulink model level as it has been found to be at the code level. The framework addresses this problem. Finally, a flexible approach to test sub-set extraction is presented, allowing testing resources to be deployed effectively and efficiently.     

特定领域软件复用技术的研究与应用

围绕特定领域软件体系结构的设计,利用平台化分层体系结构的设计思想以及应用集成设计机制,对面向领域的软构件开发模型进行了探讨与设计,并以某石油公司企业MIS的设计为开发背景,选取其中一个特殊的子系统模块,对业务构件的组合过程——复用技术进行了详细的设计与阐述。     

A systematic review of search-based testing for non-functional system properties

Search-based software testing is the application of metaheuristic search techniques to generate software tests. The test adequacy criterion is transformed into a fitness function and a set of solutions in the search space are evaluated with respect to the fitness function using a metaheuristic search technique. The application of metaheuristic search techniques for testing is promising due to the fact that exhaustive testing is infeasible considering the size and complexity of software under test. Search-based software testing has been applied across the spectrum of test case design methods; this includes white-box (structural), black-box (functional) and grey-box (combination of structural and functional) testing. In addition, metaheuristic search techniques have also been applied to test non-functional properties. The overall objective of undertaking this systematic review is to examine existing work into non-functional search-based software testing (NFSBST). We are interested in types of non-functional testing targeted using metaheuristic search techniques, different fitness functions used in different types of search-based non-functional testing and challenges in the application of these techniques. The systematic review is based on a comprehensive set of 35 articles obtained after a multi-stage selection process and have been published in the time span 1996–2007. The results of the review show that metaheuristic search techniques have been applied for non-functional testing of execution time, quality of service, security, usability and safety. A variety of metaheuristic search techniques are found to be applicable for non-functional testing including simulated annealing, tabu search, genetic algorithms, ant colony methods, grammatical evolution, genetic programming (and its variants including linear genetic programming) and swarm intelligence methods. The review reports on different fitness functions used to guide the search for each of the categories of execution time, safety, usability, quality of service and security; along with a discussion of possible challenges in the application of metaheuristic search techniques.     

面向上下文无关语言的测试工具的设计和实现

软件测试是软件开发生命周期的重要组成部分,也是目前和今后相当长一段时间内保证软件质量和可靠性的主要手段,首先以两个已经实现了的分别面向VB和C/C 语言的计算机辅助软件测试工具为基础,将该系列测试工具的系统结构推广至适用于任何一种上下文无关语言。而该系列测试式工具高度的模块可重用性特征为这种推广提供了必要的基础。为实现这种高度的模块可重用性特征。从各个角度论述了增强该特征的设计,提出了基于块的程序划分机制。这种机制从根本上保证了对多种上下文无关语言的测试语义的一致性,最后,还给出了两个不同语言写就的实例的运行结果,以说明这种一致性。     

Search-based automated testing of continuous controllers: Framework,tool support,and case studies

ContextTesting and verification of automotive embedded software is a major challenge. Software production in automotive domain comprises three stages: Developing automotive functions as Simulink models, generating code from the models, and deploying the resulting code on hardware devices. Automotive software artifacts are subject to three rounds of testing corresponding to the three production stages: Model-in-the-Loop (MiL), Software-in-the-Loop (SiL) and Hardware-in-the-Loop (HiL) testing.ObjectiveWe study testing of continuous controllers at the Model-in-Loop (MiL) level where both the controller and the environment are represented by models and connected in a closed loop system. These controllers make up a large part of automotive functions, and monitor and control the operating conditions of physical devices.MethodWe identify a set of requirements characterizing the behavior of continuous controllers, and develop a search-based technique based on random search, adaptive random search, hill climbing and simulated annealing algorithms to automatically identify worst-case test scenarios which are utilized to generate test cases for these requirements.ResultsWe evaluated our approach by applying it to an industrial automotive controller (with 443 Simulink blocks) and to a publicly available controller (with 21 Simulink blocks). Our experience shows that automatically generated test cases lead to MiL level simulations indicating potential violations of the system requirements. Further, not only does our approach generate significantly better test cases faster than random test case generation, but it also achieves better results than test scenarios devised by domain experts. Finally, our generated test cases uncover discrepancies between environment models and the real world when they are applied at the Hardware-in-the-Loop (HiL) level.ConclusionWe propose an automated approach to MiL testing of continuous controllers using search. The approach is implemented in a tool and has been successfully applied to a real case study from the automotive domain.     

Learning from optimization: A case study with Apache Ant

ContextSoftware architecture degrades when changes violating the design-time architectural intents are imposed on the software throughout its life cycle. Such phenomenon is called architecture erosion. When changes are not controlled, erosion makes maintenance harder and negatively affects software evolution.ObjectiveTo study the effects of architecture erosion on a large software project and determine whether search-based module clustering might reduce the conceptual distance between the current architecture and the design-time one.MethodTo run an exploratory study with Apache Ant. First, we characterize Ant’s evolution in terms of size, change dispersion, cohesion, and coupling metrics, highlighting the potential introduction of architecture and code-level problems that might affect the cost of changing the system. Then, we reorganize the distribution of Ant’s classes using a heuristic search approach, intending to re-emerge its design-time architecture.ResultsIn characterizing the system, we observed that its original, simple design was lost due to maintenance and the addition of new features. In optimizing its architecture, we found that current models used to drive search-based software module clustering produce complex designs, which maximize the characteristics driving optimization while producing class distributions that would hardly be acceptable to developers maintaining Ant.ConclusionThe structural perspective promoted by the coupling and cohesion metrics precludes observing the adequate software module clustering from the perspective of software engineers when considering a large open source system. Our analysis adds evidence to the criticism of the dogma of driving design towards high cohesion and low coupling, at the same time observing the need for better models to drive design decisions. Apart from that, we see SBSE as a learning tool, allowing researchers to test Software Engineering models in extreme situations that would not be easily found in software projects.     

Who tested my software? Testing as an organizationally cross-cutting activity

There is a recognized disconnect between testing research and industry practice, and more studies are needed on understanding how testing is conducted in real-world circumstances instead of demonstrating the superiority of specific methods. Recent literature indicates that testing is a cross-cutting activity that involves various organizational roles rather than the sole involvement of specialized testers. This research empirically investigates how testing involves employees in varying organizational roles in software product companies. We studied the organization and values of testing using an exploratory case study methodology through interviews, defect database analysis, workshops, analyses of documentation, and informal communications at three software product companies. We analyzed which employee groups test software in the case companies, and how many defects they find. Two companies organized testing as a team effort, and one company had a specialized testing group because of its different development model. We found evidence that testing was not an action conducted only by testing specialists. Testing by individuals with customer contact and domain expertise was an important validation method. We discovered that defects found by developers had the highest fix rates while those revealed by specialized testers had the lowest. The defect importance was susceptible to organizational competition of resources (i.e., overvaluing defects of reporter’s own products or projects). We conclude that it is important to understand the diversity of individuals participating in software testing and the relevance of validation from the end users’ viewpoint. Future research is required to evaluate testing approaches for diverse organizational roles. Finally, to improve defect information, we suggest increasing automation in defect data collection.     

Ergonomic aspects of man-computer problem solving

The paper reports a survey of interactive computer usage by designers, engineers, economists and other specialists in several industrial organisations. Sixty-nine of these specialists were interviewed about both hardware interface problems, eg, keyboard layout, and software interface problems, eg, how system organisation affects the way the specialist tackles his work. The results of the survey show that interactive computing can be of considerable assistance to specialists, although there are often many difficulties in using the computer system to full advantage.     

完善验证的凝聚力测量方法以提升软件品质

无适当使用软件测量将可能引起软件低品质且高成本的窘态.凝聚力是软件品质重要因子之一如同维护度,可靠度和再利用度.软件模组品质的优劣必影响整体系统之品质的优劣.为了设计和维护高品质软件,软件专案经理人和软件工程师无可避免需引用软件凝聚力测量以衡量和产生高品质软件.提出以活路跃变量及视觉化变量纵距为分析基础之功能导向凝聚力测量方法.进而,以一系列实际案例来作实验验证,并以一组性质来作理论辩证所提的测量方法.因此一经完善定义,完善实验和完善辩证之凝聚力测量方法被提出用于当软件凝聚力强度的指标和因此增进软件品质.这凝聚力测量方法能容易嵌入CASE以帮助软件工程师确保软件品质.     

Interactive Explanation of Software Systems

This paper describes an effort to provide automated support for the interactive inquiry and explanation process that is at the heart of software understanding. A hypermedia tool called I-Doc allows software engineers to post queries about a software system, and generates focused explanations in response. These explanations are task oriented, i.e., they are sensitive to the software engineering task being performed by the user that led to the query. Task orientation leads to more effective explanations, and is particularly helpful for understanding large software systems. Empirical studies of inquiry episodes were conducted in order to investigate this claim: the kinds of questions users ask, their relation to the user's task and level of expertise. The I-Doc tool is being developed to embody these principles, employing knowledge-based techniques. The presentation mechanism employs World Wide Web (WWW) technology, making it suitable for widespread use.     

Kineo CAM: a success story of motion planning algorithms

Kineo computer aided motion (CAM) is a start-up company that develops and markets software technologies dedicated to motion planning and control and destined for the industrial domain of product lifecycle management (PLM). In addition to this product strategy, Kineo CAM's expertise covers various domains ranging from medical robotics to transportation via graphics. This article gives an account of the different phases of the product design from the seminal research conducted at LAAS-CNRS, the company's creator, during the 1990s to the first success stories of treating difficult industrial problems.     

Lightweight coarse-grained coordination: a scalable system-level approach

In this paper, our solution to the problem of modelling functionally complex communication systems at the application level, based on lightweight coordination, is extended to seamlessly capture system-level testing as well. This extension could be realized simply by self-application: the bulk of the work for integrating system-level testing into our development environment, the ABC, concerned domain modelling, which can be done using the ABC. Therefore, the extension of the ABC to cover system-level testing was merely an application development on the basis of the ABC, illustrated here in the domain of Computer Telephony Integration. Here the adoption of a coarse-grained approach to test design, which is central to the scalability of the overall testing environment, is the enabling aspect for system-level test automation. Together with our lightweight coordination approach this induces an understandable modelling paradigm of system-wide test cases that is adequate for the needs and requirements of industrial test engineers. In particular, it enables test engineers to graphically design complex test cases that, in addition, can even be automatically checked for their intended purposes via model checking.     

A practical model‐based statistical approach for generating functional test cases: application in the automotive industry

With the growing complexity of industrial software applications, industrials are looking for efficient and practical methods to validate the software. This paper develops a model‐based statistical testing approach that automatically generates online and offline test cases for embedded software. It discusses an integrated framework that combines solutions for three major software testing research questions: (i) how to select test inputs; (ii) how to predict the expected results of a test; and (iii) when to stop testing software. The automatic selection of test inputs is based on a stochastic test model that accounts for the main particularity of embedded software: time sensitivity. Software test practitioners may design one or more test models when they generate random, user‐oriented, or fault‐oriented test inputs. A formal framework integrating existing and appropriate specification techniques was developed for the design of automated test oracles (executable software specifications) and the formal measurement of functional coverage. The decision to stop testing software is based on both test coverage objectives and cost constraints. This approach was tested on two representative case studies from the automotive industry. The experiment was performed at unit testing level in a simulated environment on a host personal computer (automatic test execution). The two software functionalities tested had previously been unit tested and validated using the test design approach conventionally used in the industry. Applying the proposed model‐based statistical testing approach to these two case studies, we obtained significant improvements in performing functional unit testing in a real and complex industrial context: more bugs were detected earlier and in a shorter time. Copyright © 2012 John Wiley & Sons, Ltd.     

Studying the co-evolution of production and test code in open source and industrial developer test processes through repository mining

Many software production processes advocate rigorous development testing alongside functional code writing, which implies that both test code and production code should co-evolve. To gain insight in the nature of this co-evolution, this paper proposes three views (realized by a tool called TeMo) that combine information from a software project’s versioning system, the size of the various artifacts and the test coverage reports. We validate these views against two open source and one industrial software project and evaluate our results both with the help of log messages, code inspections and the original developers of the software system. With these views we could recognize different co-evolution scenarios (i.e., synchronous and phased) and make relevant observations for both developers as well as test engineers.     

T-CBESD:一个构件化嵌入式软件设计模型验证工具

现代复杂嵌入式软件系统的高可靠性需要有效的基于模型的设计与分析技术.传统的嵌入式软件可靠性保障技术主要关注于系统开发后期.本文在Eclipse平台上设计并实现了一个基于接口自动机模型的构件化嵌入式软件设计的形式化验证原型工具T-CBESD(Tool for Component-Based Embedded Software Designs).工具直接使用UML顺序图模型作为系统规约,可以检验系统设计模型与场景式规约之间多种行为一致性问题;并使用消息事件的时间约束不等式,检验实时接口自动机网络与带时间约束的顺序图模型之间的实时行为一致性问题.工具设计与实现内容包括:输入输出接口、顺序图模型的预处理转换、状态空间数据结构设计、抽象验证算法的实现以及通信构件组合系统的实例应用分析.     

需求工程的探讨

软件工程被分为需求、设计、实现、测试等几个阶段,其中需求阶段是一个项目的开端,也是项目成功的基石。在软件工程中,需求分析指的是在建立一个新的或改变一个现存的电脑系统时描写新系统的目的、范围、定义和功能时所要做的所有的工作。需求分析是软件工程中的一个关键过程。在这个过程中,系统分析员和软件工程师确定顾客的需要。只有在确定了这些需要后他们才能够分析和寻求新系统的解决方法。在软件工程的历史中,很长时间里人们一直认为需求分析是整个软件工程中最简单的一个步骤,但在过去十年中越来越多的人认识到它是整个过程中最关键的一个过程。假如在需求分析时分析者们未能正确地认识到顾客的需要的话,那么最后的软件实际上不可能达到顾客的需要,或者软件无法在规定的时间里完工。     

Markov控制转换多模块软件可靠性测评方法

针对软件多模块化发展的趋势以及当前软件可靠性测评把软件当成一个整块进行处理的缺点,本文提供了一种Markov控制转换多模块软件的可靠性测评方法.方法的主要思想包括建立基于软件体系结构的可靠性模型,并依据该模型把系统级的可靠性指标优化分配到模块级,实现在模块级对整个软件系统可靠性的监控,并在系统级的测评中进行确认.文中针对系统级测评缺少失效数据的情况,重点阐述了如何基于最大熵原理求解系统可靠性参数分布函数,以实现对系统可靠性指标的验证.