On the effectiveness of the test-first approach to programming

Test-driven development (TDD) is based on formalizing a piece of functionality as a test, implementing the functionality such that the test passes, and iterating the process. This paper describes a controlled experiment for evaluating an important aspect of TDD: in TDD, programmers write functional tests before the corresponding implementation code. The experiment was conducted with undergraduate students. While the experiment group applied a test-first strategy, the control group applied a more conventional development technique, writing tests after the implementation. Both groups followed an incremental process, adding new features one at a time and regression testing them. We found that test-first students on average wrote more tests and, in turn, students who wrote more tests tended to be more productive. We also observed that the minimum quality increased linearly with the number of programmer tests, independent of the development strategy employed.     

The effectiveness of test-driven development: an industrial case study

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an industrial environment. Three comparable medium-sized projects were observed during their development cycle. Two projects were driven without TDD practice, while the third one introduced TDD into the development process. The effectiveness of TDD was expressed in terms of external code quality, productivity, and maintainability. Our results indicate that the TDD developers produced higher quality code that is easier to maintain, although we did observe a reduction in productivity.     

Does Test-Driven Development Really Improve Software Design Quality?

Support for test-driven development [TDD] is growing in many development contexts beyond its common association with extreme programming. By focusing on how TDD influences design characteristics, we hope to raise awareness of TDD as a design approach and assist others in decisions on whether and how to adopt TDD. Our results indicate that test-first programmers are more likely to write software in more and smaller units that are less complex and more highly tested. We weren't able to confirm claims that TDD improves cohesion while lowering coupling, but we anticipate ways to clarify the questions these design characteristics raised. In particular, we're working to eliminate the confounding factor of accessor usage in the cohesion metrics.     

Operational definition and automated inference of test-driven development with Zorro

Test-driven development (TDD) is a style of development named for its most visible characteristic: the design and implementation of test cases prior to the implementation of the code required to make them pass. Many claims have been made for TDD: that it can improve implementation as well as design quality, that it can improve productivity, that it results in 100% coverage, and so forth. However, research to validate these claims has yielded mixed and sometimes contradictory results. We believe that at least part of the reason for these results stems from differing interpretations of the TDD development style, along with an inability to determine whether programmers actually follow whatever definition of TDD is in use. Zorro is a system designed to automatically determine whether a developer is complying with an operational definition of Test-Driven Development (TDD) practices. Automated recognition of TDD can benefit the software development community in a variety of ways, from inquiry into the “true nature” of TDD, to pedagogical aids to support the practice of test-driven development, to support for more rigorous empirical studies on the effectiveness of TDD in both laboratory and real world settings.     

Driving Software Quality: How Test-Driven Development Impacts Software Quality

Recently, software development teams using agile processes have started widely adopting test-driven development. Despite its name, "test driven" or "test first" development isn't really a testing technique. Also known as test-driven design, TDD works like this: For each small bit of functionality the programmers code, they first write unit tests. Then they write the code that makes those unit tests pass. This forces the programmer to think about many aspects of the feature before coding it. It also provides a safety net of tests that the programmers can run with each update to the code, ensuring that refactored, updated, or new code doesn't break existing functionality     

A test case refactoring approach for pattern-based software development

In the current trend, Extreme Programing methodology is widely adopted by small and medium-sized projects for dealing with rapidly or indefinite changing requirements. Test-first strategy and code refactoring are the important practices of Extreme Programing for rapid development and quality support. The test-first strategy emphasizes that test cases are designed before system implementation to keep the correctness of artifacts during software development; whereas refactoring is the removal of “bad smell” code for improving quality without changing its semantics. However, the test-first strategy may conflict with code refactoring in the sense that the original test cases may be broken or inefficient for testing programs, which are revised by code refactoring. In general, the developers revise the test cases manually since it is not complicated. However, when the developers perform a pattern-based refactoring to improve the quality, the effort of revising the test cases is much more than that in simple code refactoring. In our observation, a pattern-based refactoring is composed of many simple and atomic code refactorings. If we have the composition relationship and the mapping rules between code refactoring and test case refactoring, we may infer a test case revision guideline in pattern-based refactoring. Based on this idea, in this research, we propose a four-phase approach to guide the construction of the test case refactoring for design patterns. We also introduce our approach by using some well-known design patterns and evaluate its feasibility by means of test coverage.     

An industry experiment on the effects of test-driven development on external quality and productivity

Existing empirical studies on test-driven development (TDD) report different conclusions about its effects on quality and productivity. Very few of those studies are experiments conducted with software professionals in industry. We aim to analyse the effects of TDD on the external quality of the work done and the productivity of developers in an industrial setting. We conducted an experiment with 24 professionals from three different sites of a software organization. We chose a repeated-measures design, and asked subjects to implement TDD and incremental test last development (ITLD) in two simple tasks and a realistic application close to real-life complexity. To analyse our findings, we applied a repeated-measures general linear model procedure and a linear mixed effects procedure. We did not observe a statistical difference between the quality of the work done by subjects in both treatments. We observed that the subjects are more productive when they implement TDD on a simple task compared to ITLD, but the productivity drops significantly when applying TDD to a complex brownfield task. So, the task complexity significantly obscured the effect of TDD. Further evidence is necessary to conclude whether TDD is better or worse than ITLD in terms of external quality and productivity in an industrial setting. We found that experimental factors such as selection of tasks could dominate the findings in TDD studies.     

Impact of test-driven development on productivity, code and tests: A controlled experiment

Context

Test-driven development is an approach to software development, where automated tests are written before production code in highly iterative cycles. Test-driven development attracts attention as well as followers in professional environment; however empirical evidence of its superiority regarding its effect on productivity, code and tests compared to test-last development is still fairly limited. Moreover, it is not clear if the supposed benefits come from writing tests before code or maybe from high iterativity/short development cycles.

Objective

This paper describes a family of controlled experiments comparing test-driven development to micro iterative test-last development with emphasis on productivity, code properties (external quality and complexity) and tests (code coverage and fault-finding capabilities).

Method

Subjects were randomly assigned to test-driven and test-last groups. Controlled experiments were conducted for two years, in an academic environment and in different developer contexts (pair programming and individual programming contexts). Number of successfully implemented stories, percentage of successful acceptance tests, McCabe’s code complexity, code coverage and mutation score indicator were measured.

Results

Experimental results and their selective meta-analysis show no statistically significant differences between test-driven development and iterative test-last development regarding productivity (χ²(6) = 4.799, p = 1.0, r = .107, 95% CI (confidence interval): −.149 to .349), code complexity (χ²(6) = 8.094, p = .46, r = .048, 95% CI: −.254 to .341), branch coverage (χ²(6) = 13.996, p = .059, r = .182, 95% CI: −.081 to .421), percentage of acceptance tests passed (one experiment, Mann-Whitney U = 125.0, p = .98, r = .066) and mutation score indicator (χ²(4) = 3.807, p = .87, r = .128, 95% CI: −.162 to .398).

Conclusion

According to our findings, the benefits of test-driven development compared to iterative test-last development are small and thus in practice relatively unimportant, although effects are positive. There is an indication of test-driven development endorsing better branch coverage, but effect size is considered small.     

Test-Driven Development of Relational Databases

In test-first development, developers formulate and implement a detailed design iteratively, one test at a time. Test-driven development (also called test-driven design) combines TFD with refactoring, wherein developers make small changes (refactorings) to improve code design without changing the code's semantics. When developers decide to use TDD to implement a new feature, they must first ask whether the current design is the easiest possible design to enable the feature's addition. Implementing test-driven database design involves database refactoring, regression testing, and continuous integration. TDDD is an integrated part of the overall development process, not a standalone activity that data professionals perform in parallel with application TDD. Although from a technical view point, TDDD is relatively straightforward, we must overcome several challenges to its whole sale adoption throughout the IT community     

Considering rigor and relevance when evaluating test driven development: A systematic review

ContextTest driven development (TDD) has been extensively researched and compared to traditional approaches (test last development, TLD). Existing literature reviews show varying results for TDD.ObjectiveThis study investigates how the conclusions of existing literature reviews change when taking two study quality dimension into account, namely rigor and relevance.MethodIn this study a systematic literature review has been conducted and the results of the identified primary studies have been analyzed with respect to rigor and relevance scores using the assessment rubric proposed by Ivarsson and Gorschek 2011. Rigor and relevance are rated on a scale, which is explained in this paper. Four categories of studies were defined based on high/low rigor and relevance.ResultsWe found that studies in the four categories come to different conclusions. In particular, studies with a high rigor and relevance scores show clear results for improvement in external quality, which seem to come with a loss of productivity. At the same time high rigor and relevance studies only investigate a small set of variables. Other categories contain many studies showing no difference, hence biasing the results negatively for the overall set of primary studies. Given the classification differences to previous literature reviews could be highlighted.ConclusionStrong indications are obtained that external quality is positively influenced, which has to be further substantiated by industry experiments and longitudinal case studies. Future studies in the high rigor and relevance category would contribute largely by focusing on a wider set of outcome variables (e.g. internal code quality). We also conclude that considering rigor and relevance in TDD evaluation is important given the differences in results between categories and in comparison to previous reviews.     

测试驱动开发在Net环境中的应用

软件测试是保障软件质量的关键，开发者通常都是先写代码后测试，但当时间很紧迫时，测试就变得很困难并且经常被忽略。极限编程最佳实践之一的测试驱动开发在写代码之前先写测试，解决了这一问题，提供了高质量、可测试的代码。介绍了测试驱动开发在．Net环境中的应用。     

Mapping canopy gaps in an indigenous subtropical coastal forest using high-resolution WorldView-2 data

Invasive species usually colonize canopy gaps in tropical and subtropical forests, which results in a loss of native species. Therefore, an understanding of the location and distribution of canopy gaps will assist in predicting the occurrence of invasive species in such canopy gaps. We tested the utility of WorldView-2 (WV-2) with eight spectral bands at 2 m spatial resolution to delineate forest canopy gaps in a subtropical Dukuduku coastal forest in South Africa. We compared the four conventional visible-near-infrared bands with the eight-band WV-2 image. The eight-band WV-2 image yielded a higher overall accuracy of 86.90% (kappa coefficient = 0.82) than the resampled conventional four-band image that yielded an overall accuracy of 74.64% (kappa coefficient = 0.63) in pixel-based classification. We further compared the vegetation indices that were derived from four conventional bands with those derived from WV-2 bands. The enhanced vegetation index yielded the highest overall accuracy in the category of conventional indices (85.59% at kappa coefficient = 0.79), while the modified plant senescence reflectance index involving the red-edge band showed the highest overall accuracy (93.69%) in the category of indices derived from eight-band WV-2 imagery in object-based classification. Overall, the study shows that the unique high-resolution WV-2 data can improve the delineation of canopy gaps as compared to the conventional multispectral bands.     

Besouro: A framework for exploring compliance rules in automatic TDD behavior assessment

ContextThe improvements promoted by Test-Driven Design (TDD) have not been confirmed by quantitative assessment studies. To a great extent, the problem lies in the lack of a rigorous definition for TDD. An emerging approach has been to measure the conformance of TDD practices with the support of automated systems that embed an operational definition, which represent the specific TDD process assumed and the validation tests used to determine its presence and quantity. The empirical construction of TDD understanding and consensus building requires the ability of comparing different definitions, evaluating them with regard to practitioners’ perception, and exploring code information for improvement of automatic assessment.ObjectiveThis paper describes Besouro, a framework targeted at the development of systems for automatic TDD behavior assessment. The main rationale of Besouro’s design is the ability to compare distinct operational definitions, evaluate them with regard to users’ perception, and explore code information for further analysis and conformance assessment improvement.MethodWe developed an architecture with clear separation of concerns, which enables to vary: (a) the atomic events and respective metrics to be collected from developing and testing environments; (b) the organization of atomic events in streams of actions or processes; and (c) the classification and assessment components for each set of operational definitions adopted. The architecture also includes a mechanism for on-line user assessment awareness and feedback, and integrates event-related information with the respective code in a code version system.ResultsWe illustrate the usefulness of Besouro’s features for understanding the actions and processes underlying TDD through a prototype developed to support an experiment based on user feedback. We show how it was possible to compare variations of a same operational definition by exploring users’ feedback, and use source code to improve the automatic classification of TDD practices.ConclusionUnderstanding the actions and processes underlying successful TDD application is key for leveraging TDD benefits. In the absence of a rigorous definition for TDD, the proposed approach aims at building consensus from experimentation and empirical validation.     

Test-Driven Development of a PID Controller

The development of embedded control systems in Simulink usually continues with automatic code generation, the build process, and several tests: software-in-the-loop (SiL), hardware-in-the-loop (HiL), and system tests in the real environment of the controller, the vehicle. Our test-driven control-system design cycle integrates into this process because it doesn't interrupt the system's model-based architecture. Furthermore, the next development cycle usually considers the results of SiL, HiL, or the system tests. With TDD, such changes cause one or more new tests, which describe the modified requirements either with a modified or different process model or with a stimulus from measurements. All previously defined tests ensure the controller's basic behavior, leading to higher quality for the complete system. In addition, we believe we can extend our approach to different types of real-time systems and evaluate how to use TDD for their development     

Recognizing authors: an examination of the consistent programmer hypothesis

Software developers have individual styles of programming. This paper empirically examines the validity of the consistent programmer hypothesis: that a facet or set of facets exist that can be used to recognize the author of a given program based on programming style. The paper further postulates that the programming style means that different test strategies work better for some programmers (or programming styles) than for others. For example, all‐edges adequate tests may detect faults for programs written by Programmer A better than for those written by Programmer B. This has several useful applications: to help detect plagiarism/copyright violation of source code, to help improve the practical application of software testing, and to help pursue specific rogue programmers of malicious code and source code viruses. This paper investigates this concept by experimentally examining whether particular facets of the program can be used to identify programmers and whether testing strategies can be reasonably associated with specific programmers. Copyright © 2009 John Wiley & Sons, Ltd.     

Effects of sediments and coloured dissolved organic matter on remote sensing of chlorophyll-a using Landsat TM/ETM+ over turbid waters

Remote sensing of chlorophyll-a is challenging in water containing inorganic suspended sediments (i.e. non-volatile suspended solids, NVSS) and coloured dissolved organic matter (CDOM). The effects of NVSS and CDOM on empirical remote-sensing estimates of chlorophyll-a in inland waters have not been determined on a broad spatial and temporal scale. This study evaluated these effects using a long-term (1989–2012) data set that included chlorophyll-a, NVSS, and CDOM from 39 reservoirs across Missouri (USA). Model comparisons indicated that the machine-learning algorithm BRT (boosted regression trees, validation Nash–Sutcliffe coefficient = 0.350) was better than linear regression (validation Nash–Sutcliffe coefficient = 0.214) for chlorophyll-a estimate using Landsat Thematic Mapper (TM) and Enhanced Thematic Mapper Plus (ETM+) imagery. Only a small proportion of BRT model residuals could be explained by sediments or CDOM, and the observed trends in BRT residuals were different from the theoretical effects expected from NVSS and CDOM. Our results also indicated a small systematic bias by the BRT model, but it was not likely caused by NVSS or CDOM.     

An experimental investigation into the process of knowledge-based systems development

This paper presents an experimental study into the processes that may contribute to building a better knowledge-based system. A model that defines quality to be composed of two related aspects, internal and external quality, is introduced. To test the model, 24 subjects developed a knowledge-based system for MBA course planning in an experimental setting over a 7 week period. Subjects were factored by development methodology (structured vs rapid prototyping), knowledge representation scheme used (rule-based vs hybrid) and programmer quality (naive vs experienced). The major finding is that an appropriate mix of development methodology, knowledge representation and personnel is necessary. No single development methodology or knowledge representation scheme is best, and a considerable number of interactions was observed in the experiment. Rapid prototyping combined with rule-based representation produced the best external quality in terms of functionality, but when combined with hybrid representation produced the worst. Similarly, rapid prototyping combined with hybrid representation produced the worst usability. Programmer quality had a positive effect on coding productivity, which in turn resulted in an increase in system usability. As with conventional software, increasing programmer quality can be very beneficial to both process and content. The study presented provides some evidence of the anomalies that are generated in the course of system development, and how they relate to internal quality. As might be expected, experienced programmers produced significantly fewer anomalies. A relationship was found between internal quality and usability, but not functionality.     

CScout: A refactoring browser for C

Despite its maturity and popularity, the C programming language still lacks tool support for reliably performing even simple refactoring, browsing, or analysis operations. This is primarily due to identifier scope complications introduced by the C preprocessor. The CScout refactoring browser analyses complete program families by tagging the original identifiers with their precise location and classifying them into equivalence classes orthogonal to the C language’s namespace and scope extents. A web-based user interface provides programmers with an intuitive source code analysis and navigation front-end, while an sql-based back-end allows more complex source code analysis and manipulation. CScout has been successfully applied to many medium and large-sized proprietary and open-source projects identifying thousands of modest refactoring opportunities.     

Evaluating the productivity of a reference-based programming approach: A controlled experiment

ContextDomain engineering aims at facilitating software development in an efficient and economical way. One way to measure that is through productivity indicators, which refer to the ability of creating a quality software product in a limited period and with limited resources. Many approaches have been devised to increase productivity; however, these approaches seem to suffer from a tension between expressiveness on the one hand, and applicability (or the lack of it) in providing guidance for developers.ObjectiveThis paper evaluates the applicability and efficiency of adopting a domain engineering approach, called Application-based DOmain Modeling (ADOM), in the context of the programming task with Java, and thus termed ADOM-Java, for improving productivity in terms of code quality and development time.MethodTo achieve that objective we have qualitatively evaluate the approach using questionnaires and following a text analysis procedure. We also set a controlled experiment in which 50 undergraduate students performed a Java-based programming task using either ADOM-Java or Java alone.ResultsThe qualitative evaluation reveal that the approach is easy to uses and provides valuable guidance. Nevertheless, it requires training. The outcomes of the experiment indicate that the approach is applicable and that the students that used ADOM-Java achieved better code quality, as well as better functionality and within less time than the students who used only Java.ConclusionThe results of the experiments imply that by providing a code base equipped with reuse guidelines for programmers can increase programming productivity in terms of quality and development time. These guidelines may also enforce coding standards and architectural design.     

Cooperative aspect-oriented programming

Aspect-oriented programming (AOP) seeks to improve software modularity via the separation of cross-cutting concerns. AOP proponents often advocate a development strategy where programmers write the main application (base code), ignoring cross-cutting concerns, and then aspect programmers, domain experts in their specific concerns, weave in the logic for these more specialized cross-cutting concerns. This purely oblivious strategy, however, has empirically been shown to tightly couple aspects to base code in many cases, hindering aspect modularity and reuse. In essence, the more intricate the weaving between the cross-cutting concern and the base code (lexically and/or semantically), the harder it becomes to: (a) robustly specify how to weave the aspects in at the required points, (b) capture interactions between aspects and base code, and (c) preserve the correct weaving as the base code evolves.We propose an alternate methodology, termed cooperative aspect-oriented programming (Co-AOP), where complete lexical separation of concerns is not taken as an absolute requirement. Instead, cross-cutting concerns are explicitly modeled as abstract interfaces through explicit join points (EJPs). Programmers specify where these interfaces interact with base code either through explicit lexical references or via traditional oblivious aspects. This explicit awareness allows base code and aspects to cooperate in ways that were previously not possible: arbitrary blocks of code can be advised, advice can be explicitly parameterized, base code can guide aspects in where to apply advice, and aspects can statically enforce new constraints upon the base code that they advise. These new techniques allow aspect modularity and program safety to increase, and bring us towards a cooperative AOP paradigm.We illustrate our methodology via an example on transactions, and also give an initial evaluation of cooperative AOP through an empirical study on program extensibility comparing both the traditional and cooperative AOP methodologies. Initial results show that cooperative AOP techniques result in code that is less complex with lower overall coupling, facilitating extensibility.