Tools: An environment for time-shared computing and programming

The Yale Tools environment provides integrated modern computing facilities to a mixed group of programmers and users on time-shared machines. Hundreds of people use the environment every day for systems and applications programming, word-processing and university business. The success of the environment is in its integration and engineering detail. Building such a system without a high-level language and a run-time library would have been impossible.     

Investigating software testing and maintenance reports: Case study

ContextAlthough many papers have been published on software development and defect prediction techniques, problem reports in real projects quite often differ from those described in the literature. Hence, there is still a need for deeper exploration of case studies from industry.ObjectiveThe aim of this study is to present the impact of fine-grained problem reports on improving evaluation of testing and maintenance processes. It is targeted at projects involving several releases and complex schemes of problem handling. This is based on our experience gained while monitoring several commercial projects.MethodExtracting certain features from detailed problem reports, we derive various measures and present analysis models which characterize and visualize the effectiveness of testing and problem resolution processes. The considered reports describe types of problems (e.g. defects), their locations in project versions and software modules, ways of their resolution, etc. The performed analysis is related to eleven projects developed in the same company. This study is an exploratory research with some explanatory features. Moreover, having identified some drawbacks, we present extensions of problem reports and their analysis which have been verified in another industrial case study project.ResultsFine-grained (accurate) problem handling reports provide a wider scope of possible measures to assess the relevant development processes. This is helpful in controlling single projects (local perspective) as well as in managing these processes in the whole company (global perspective).ConclusionDetailed problem handling reports extend the space and quality of statistical analysis, they provide significant enhancement in evaluation and refinement of software development processes as well as in reliability prediction.     

Relative debugging in an integrated development environment

Relative Debugging allows a user to compare the internal state of two programs as they run, making it possible to test whether two programs perform the same function given the same input. When implemented with a command line user interface, a relative debugger looks like traditional debugging tools with the addition of commands that describe which structures should be equivalent in the two programs. In this paper, we discuss relative debugging within an integrated development environment, and show that there are significant advantages over a command line form. We describe a pluggable, modular, architecture that works with a variety of different products, including Microsoft's Visual Studio, SUN's NetBeans, and IBM's Eclipse. Copyright © 2009 John Wiley & Sons, Ltd.     

Global software testing under deadline pressure: Vendor-side experiences

ContextIn the era of globally-distributed software engineering, the practice of global software testing (GST) has witnessed increasing adoption. Although there have been ethnographic studies of the development aspects of global software engineering, there have been fewer studies of GST, which, to succeed, can require dealing with unique challenges.ObjectiveTo address this limitation of existing studies, we conducted, and in this paper, report the findings of, a study of a vendor organization involved in one kind of GST practice: outsourced, offshored software testing.MethodWe conducted an ethnographically-informed study of three vendor-side testing teams over a period of 2 months. We used methods, such as interviews and participant observations, to collect the data and the thematic-analysis approach to analyze the data.FindingsOur findings describe how the participant test engineers perceive software testing and deadline pressures, the challenges that they encounter, and the strategies that they use for coping with the challenges. The findings reveal several interesting insights. First, motivation and appreciation play an important role for our participants in ensuring that high-quality testing is performed. Second, intermediate onshore teams increase the degree of pressure experienced by the participant test engineers. Third, vendor team participants perceive productivity differently from their client teams, which results in unproductive-productivity experiences. Lastly, participants encounter quality-dilemma situations for various reasons.ConclusionThe study findings suggest the need for (1) appreciating test engineers’ efforts, (2) investigating the team structure’s influence on pressure and the GST practice, (3) understanding culture’s influence on other aspects of GST, and (4) identifying and addressing quality-dilemma situations.     

Automatic testing environment for multi-core embedded software—ATEMES

Software testing during the development process of embedded software is not only complex, but also the heart of quality control. Multi-core embedded software testing faces even more challenges. Major issues include: (1) how demanding efforts and repetitive tedious actions can be reduced; (2) how resource restraints of embedded system platform such as temporal and memory capacity can be tackled; (3) how embedded software parallelism degree can be controlled to empower multi-core CPU computing capacity; (4) how analysis is exercised to ensure sufficient coverage test of embedded software; (5) how to do data synchronization to address issues such as race conditions in the interrupt driven multi-core embedded system; (6) high level reliability testing to ensure customer satisfaction. To address these issues, this study develops an automatic testing environment for multi-core embedded software (ATEMES). Based on the automatic mechanism, the system can parse source code, instrument source code, generate testing programs for test case and test driver, support generating primitive, structure and object types of test input data, multi-round cross-testing, and visualize testing results. To both reduce test engineer's burden and enhance his efficiency when embedded software testing is in process, this system developed automatic testing functions including unit testing, coverage testing, multi-core performance monitoring. Moreover, ATEMES can perform automatic multi-round cross-testing benchmark testing on multi-core embedded platform for parallel programs adopting Intel TBB library to recommend optimized parallel parameters such as pipeline tokens. Using ATEMES on the ARM11 multi-core platform to conduct testing experiments, the results show that our constructed testing environment is effective, and can reduce burdens of test engineer, and can enhance efficiency of testing task.     

Design and implementation of an adaptable software environment

In any major software project, a large volume of project-related information must be maintained and manipulated throughout the software life-cycle. Often this information is scattered, in different media ranging from scraps of papers to program listings, poorly cross-referenced, and inaccessible. TRIAD, an adaptable, integrated software development environment, has been designed to integrate and manage all this information in an organized and structured manner.

The TRIAD tool-box environment supports different methodologies to guide the users during development and maintenance of project related information. The environment can be customized by selecting a set of forms which enforce a particular methodology. These forms help develop, integrate, and maintain project information. Forms for a methodology can be tuned to reflect a project-team's experience. When customized by the tuned methodology, TRIAD can provide better project oriented support.

TRIAD uses grammar forms as its organizational model. This paper will discuss the advantages of using grammar forms in the modeling and implementation of an adaptable software information editor, within the TRIAD tool-box environment.     

A requirements engineering environment within a tightly integrated SDE

This paper describes the outside functionality of an RE environment within an integrated software development environment. Furthermore, an integrator tool for the transition to software system architecture modelling is presented. The tools discussed are editors, analysers, executors, monitors, and integration tools of different characteristics for horizontal integration (within RE) and vertical integration (to architecture modelling). All tools are tightly integrated and work incrementally, therefore allowing different forms of construction and modification processes and giving substantial support.     

An initial industrial evaluation of interactive search-based testing for embedded software

Search-based software testing promises the ability to generate and evaluate large numbers of test cases at minimal cost. From an industrial perspective, this could enable an increase in product quality without a matching increase in the time and effort required to do so.Search-based software testing, however, is a set of quite complex techniques and approaches that do not immediately translate into a process for use with most companies.For example, even if engineers receive the proper education and training in these new approaches, it can be hard to develop a general fitness function that covers all contingencies. Furthermore, in industrial practice, the knowledge and experience of domain specialists are often key for effective testing and thus for the overall quality of the final software system. But it is not clear how such domain expertise can be utilized in a search-based system.This paper presents an interactive search-based software testing (ISBST) system designed to operate in an industrial setting and with the explicit aim of requiring only limited expertise in software testing. It uses SBST to search for test cases for an industrial software module, while also allowing domain specialists to use their experience and intuition to interactively guide the search.In addition to presenting the system, this paper reports on an evaluation of the system in a company developing a framework for embedded software controllers. A sequence of workshops provided regular feedback and validation for the design and improvement of the ISBST system. Once developed, the ISBST system was evaluated by four electrical and system engineers from the company (the ‘domain specialists’ in this context) used the system to develop test cases for a commonly used controller module. As well as evaluating the utility of the ISBST system, the study generated interaction data that were used in subsequent laboratory experimentation to validate the underlying search-based algorithm in the presence of realistic, but repeatable, interactions.The results validate the importance that automated software testing tools in general, and search-based tools, in particular, can leverage input from domain specialists while generating tests. Furthermore, the evaluation highlighted benefits of using such an approach to explore areas that the current testing practices do not cover or cover insufficiently.     

参数依赖型软件参数更动测试策略和方法

参数依赖型软件是指初始化时读取并解析配置参数,并据此进行任务处理的软件,航天测控软件是典型的参数依赖型软件;航天测控软件具有明显的领域软件特征,多采用领域工程分析技术,实现业务处理逻辑和具体任务参数的分离,达到仅通过修改任务配置参数而适应高强度型号任务的目的;通过对参数依赖型软件架构、应用模式的分析,提出一种对参数依赖特性进行验收测试、参数更动测试的流程、策略和方法;并基于该方法,对远程数据交互软件进行了参数依赖特性测试,测试结果表明,该方法具有测试覆盖性强、测试重点突出、测试效率高的特点。     

Model-based testing of global properties on large-scale distributed systems

ContextLarge-scale distributed systems are becoming commonplace with the large popularity of peer-to-peer and cloud computing. The increasing importance of these systems contrasts with the lack of integrated solutions to build trustworthy software. A key concern of any large-scale distributed system is the validation of global properties, which cannot be evaluated on a single node. Thus, it is necessary to gather data from distributed nodes and to aggregate these data into a global view. This turns out to be very challenging because of the system’s dynamism that imposes very frequent changes in local values that affect global properties. This implies that the global view has to be frequently updated to ensure an accurate validation of global properties.ObjectiveIn this paper, we present a model-based approach to define a dynamic oracle for checking global properties. Our objective is to abstract relevant aspects of such systems into models. These models are updated at runtime, by monitoring the corresponding distributed system.MethodWe conduce real-scale experimental validation to evaluate the ability of our approach to check global properties. In this validation, we apply our approach to test two open-source implementations of distributed hash tables. The experiments are deployed on two clusters of 32 nodes.ResultsThe experiments reveal an important defect on one implementation and show clear performance differences between the two implementations. The defect would not be detected without a global view of the system.ConclusionTesting global properties on distributed software consists of gathering data from different nodes and building a global view of the system, where properties are validated. This process requires a distributed test architecture and tools for representing and validating global properties. Model-based techniques are an expressive mean for building oracles that validate global properties on distributed systems.     

嵌入式实时系统软件测试实践

软件质量是软件的生命,软件测试是保证软件产品质量的重要手段。本文从嵌入式系统的特点出发,阐述了嵌入式系统软件测试的实现方法。并结合某无线通信电台软件测试的实例,详细叙述了如何利用Testbed和RTInsight搭建嵌入式实时系统的软件测试平台,说明了在此平台上进行软件覆盖率测试和性能分析的全过程,同时给出了部分测试结果。     

嵌入式实时系统软件测试实践

软件质量是软件的生命,软件测试是保证软件产品质量的重要手段。本文从嵌入式系统的特点出发,阐述了嵌入式系统软件测试的实现方法。并结合某无线通信电台软件测试的实例,详细叙述了如何利用Testbed和RTInsight搭建嵌入式实时系统的软件测试平台,说明了在此平台上进行软件覆盖率测试和性能分析的全过程,同时给出了部分测试结果。     

A web based collaborative testing environment

This paper presents a computer supported collaborative testing system built upon the Siette web-based assessment environment. The application poses the same set of questions to a group of students. Each student in the group should answer the same question twice. An initial response is given individually, without knowing the answers of others. Then the system provides some tools to show the other partners' responses, to support distance collaboration. Finally a second individual answer is requested. In this way assessment and collaboration activities are interlaced. At the end of a collaborative testing session, each student will have two scores: the initial score and the final score. Three sets of experiments have been carried out: (1) a set of experiments designed to evaluate and fine tune the application, improve usability, and to collect users' feelings and opinions about the system; (2) a second set of experiments to analyze the impact of collaboration in test results, comparing individual and group performance, and analyzing the factors that correlate to those results; and (3) a set of experiments designed to measure individual short-term learning directly related to the collaborative testing activity. We study whether the use of the system is associated with actual learning, and whether this learning is directly related to collaboration between students. Our studies confirm previous results and provide the following evidence (1) the performance increase is directly related to the access to other partners' answers; (2) a student tends to reach a common answer in most cases; and (3) the consensus is highly correlated with the correct response. Moreover, we have found evidence indicating that most of the students really do learn from collaborative testing. High-performing students improve by self-reflection, regardless the composition of the group, but low-performing students need to be in a group with higher-performing students in order to improve.     

An experimental study of adaptive testing for software reliability assessment

Adaptive testing is a new form of software testing that is based on the feedback and adaptive control principle and can be treated as the software testing counterpart of adaptive control. Our previous work has shown that adaptive testing can be formulated and guided in theory to minimize the variance of an unbiased software reliability estimator and to achieve optimal software reliability assessment. In this paper, we present an experimental study of adaptive testing for software reliability assessment, where the adaptive testing strategy, the random testing strategy and the operational profile based testing strategy were applied to the Space program in four experiments. The experimental results demonstrate that the adaptive testing strategy can really work in practice and may noticeably outperform the other two. Therefore, the adaptive testing strategy can serve as a preferable alternative to the random testing strategy and the operational profile based testing strategy if high confidence in the reliability estimates is required or the real-world operational profile of the software under test cannot be accurately identified.     

Testing and verification aspects of Pascal-like languages

This paper addresses aspects of programming language design that affect the ease with which programs written in a language can be subjected to systematic testing and/or program verification. The discussion focuses of Pascal and on several languages that have been derived primarily from Pascal, particularly Euclid and PLAIN. Specific language issues addressed include translation-time checking, program readability, flow of control, support for program modularity, data flow, and program immutability. The relative ease of validating such programs is then determined by the style in which the programs are written. The paper presents some guidelines for writing programs in Pascal-like languages for testability and verifiability.     

An approach to testing commercial embedded systems

A wide range of commercial consumer devices such as mobile phones and smart televisions rely on embedded systems software to provide their functionality. Testing is one of the most commonly used methods for validating this software, and improved testing approaches could increase these devices’ dependability. In this article we present an approach for performing such testing. Our approach is composed of two techniques. The first technique involves the selection of test data; it utilizes test adequacy criteria that rely on dataflow analysis to distinguish points of interaction between specific layers in embedded systems and between individual software components within those layers, while also tracking interactions between tasks. The second technique involves the observation of failures: it utilizes a family of test oracles that rely on instrumentation to record various aspects of a system's execution behavior, and compare observed behavior to certain intended system properties that can be derived through program analysis. Empirical studies of our approach show that our adequacy criteria can be effective at guiding the creation of test cases that detect faults, and our oracles can help expose faults that cannot easily be found using typical output-based oracles. Moreover, the use of our criteria accentuates the fault-detection effectiveness of our oracles.     

An adaptive and trustworthy software testing framework on the grid

Grid computing, which is characterized by large-scale sharing and collaboration of dynamic distributed resources has quickly become a mainstream technology in distributed computing and is changing the traditional way of software development. In this article, we present a grid-based software testing framework for unit and integration test, which takes advantage of the large-scale and cost-efficient computational grid resources to establish a testbed for supporting automated software test in complex software applications. Within this software testing framework, a dynamic bag-of-tasks model using swarm intelligence is developed to adaptively schedule unit test cases. Various high-confidence computing mechanisms, such as redundancy, intermediate value checks, verification code injection, and consistency checks are employed to verify the correctness of each test case execution on the grid. Grid workflow is used to coordinate various test units for integration test. Overall, we expect that the grid-based software testing framework can provide efficient and trustworthy services to significantly accelerate the testing process with large-scale software testing.

基于MVC模式的嵌入式软件测试开发环境设计

嵌入式软件仿真测试开发环境(ESTDE,Embedded Software Testing Development Environment)是嵌入式软件仿真测试平台(ESSTP,Embedded Software Simulation Testing Platform)的重要组成部分,是有效实现嵌入式软件实时、闭环测试的基础。MVC(Model-View-Controller)是一种开发交互式软件系统的典型体系结构模式,在分析和研究ESTDE功能、组成和工作原理的基础上,将MVC应用于ESTDE的设计当中,不仅保证了ESSTP的成功开发,而且提高了系统的可复用性和可适应性。     

An overview of TEDIUM

TEDIUM was developed to manage the entire life cycle of a software application from initial analysis through final retirement. In this sense, TEDIUM is different from a Programming Support Environment (PSE), which applies automated tools to aid in program implementation and project control within the context of an established process model. In the case of the PSE, computer technology is used to aid in the management of large, complex software projects. TEDIUM, on the other hand, uses its own software process model and is constrained to moderate-sized information system applications. After a brief history of the environment, the author offers an evaluation of TEDIUM, and some observations that may be of value for other development paradigms or application classes.