The effectiveness of test-driven development: an industrial case study

Test-driven development (TDD) is a software development practice, where test cases are incrementally written before implementing the production code. This paper presents the results of a multi-case study investigating the effectiveness of TDD within an industrial environment. Three comparable medium-sized projects were observed during their development cycle. Two projects were driven without TDD practice, while the third one introduced TDD into the development process. The effectiveness of TDD was expressed in terms of external code quality, productivity, and maintainability. Our results indicate that the TDD developers produced higher quality code that is easier to maintain, although we did observe a reduction in productivity.     

On the role of tests in test-driven development: a differentiated and partial replication

Background: Test-Driven Development (TDD) is claimed to have positive effects on external code quality and programmers’ productivity. The main driver for these possible improvements is the tests enforced by the test-first nature of TDD as previously investigated in a controlled experiment (i.e. the original study). Aim: Our goal is to examine the nature of the relationship between tests and external code quality as well as programmers’ productivity in order to verify/ refute the results of the original study. Method: We conducted a differentiated and partial replication of the original setting and the related analyses, with a focus on the role of tests. Specifically, while the original study compared test-first vs. test-last, our replication employed the test-first treatment only. The replication involved 30 students, working in pairs or as individuals, in the context of a graduate course, and resulted in 16 software artifacts developed. We performed linear regression to test the original study’s hypotheses, and analyses of covariance to test the additional hypotheses imposed by the changes in the replication settings. Results: We found significant correlation (Spearman coefficient = 0.66, with p-value = 0.004) between the number of tests and productivity, and a positive regression coefficient (p-value = 0.011). We found no significant correlation (Spearman coefficient = 0.41 with p-value = 0.11) between the number of tests and external code quality (regression coefficient p-value = 0.0513). For both cases we observed no statistically significant interaction caused by the subject units being individuals or pairs. Further, our results are consistent with the original study although there were changes in the timing constraints for finishing the task and the enforced development processes. Conclusions: This replication study confirms the results of the original study concerning the relationship between the number of tests vs. external code quality and programmer productivity. Moreover, this replication allows us to identify additional context variables, for which the original results still hold; namely the subject unit, timing constraint and isolation of test-first process. Based on our findings, we recommend practitioners to implement as many tests as possible in order to achieve higher baselines for quality and productivity.     

Empirical evaluation of the effects of experience on code quality and programmer productivity: an exploratory study

Context

There is a widespread belief in both SE and other branches of science that experience helps professionals to improve their performance. However, cases have been reported where experience not only does not have a positive influence but sometimes even degrades the performance of professionals.

Aim

Determine whether years of experience influence programmer performance.

Method

We have analysed 10 quasi-experiments executed both in academia with graduate and postgraduate students and in industry with professionals. The experimental task was to apply ITLD on two experimental problems and then measure external code quality and programmer productivity.

Results

Programming experience gained in industry does not appear to have any effect whatsoever on quality and productivity. Overall programming experience gained in academia does tend to have a positive influence on programmer performance. These two findings may be related to the fact that, as opposed to deliberate practice, routine practice does not appear to lead to improved performance. Experience in the use of productivity tools, such as testing frameworks and IDE also has positive effects.

Conclusion

Years of experience are a poor predictor of programmer performance. Academic background and specialized knowledge of task-related aspects appear to be rather good predictors.

     

The effect of experience on the test-driven development process

We conducted a quasi-experiment to compare the characteristics of experts’ and novices’ test-driven development processes. Our novices were 11 computers science students who participated in an Extreme Programming lab course, the expert group consisted of seven professionals who had industrial experience in test-driven development. The novices as well as two of the experts worked in a laboratory environment whereas the remaining five experts worked in their office. The experts complied more to the rules of test-driven development and had shorter test-cycles than the novices. The tests written by the experts were of higher quality in terms of statement and block coverage as well. All reported results are statistically significant on the 5% level. We conclude that the results of studies which evaluate performance of test-driven development using subjects inexperienced in TDD are not easily generalisable.     

The effectiveness of pair programming: A meta-analysis

Several experiments on the effects of pair versus solo programming have been reported in the literature. We present a meta-analysis of these studies. The analysis shows a small significant positive overall effect of pair programming on quality, a medium significant positive overall effect on duration, and a medium significant negative overall effect on effort. However, between-study variance is significant, and there are signs of publication bias among published studies on pair programming. A more detailed examination of the evidence suggests that pair programming is faster than solo programming when programming task complexity is low and yields code solutions of higher quality when task complexity is high. The higher quality for complex tasks comes at a price of considerably greater effort, while the reduced completion time for the simpler tasks comes at a price of noticeably lower quality. We conclude that greater attention should be given to moderating factors on the effects of pair programming.     

Are Two Better Than One? A Comparison Between Single- and Dual-Monitor Work Stations in Productivity and User’s Windows Management Style

Multiple monitors are commonly used in the workplace nowadays. This study compares user productivity and windows management style (WMS) on single- and dual-monitor work stations for engineering tasks of three complexity levels. Four productivity measures including task time, cursor movement, the number of window switches, and the number of mouse clicks were compared. The results showed that dual-monitor setting resulted in significantly less window switches and mouse clicks. Most users preferred dual-monitor setting. To understand how users manage multiple windows in completing their tasks, a new WMS categorization method is proposed, toggler and resizer, and user behavior was categorized into one of these two styles. More users adopted “toggler” style, but as the task complexity level increased, some “toggler” style users switched to “resizer” style.     

Considering rigor and relevance when evaluating test driven development: A systematic review

ContextTest driven development (TDD) has been extensively researched and compared to traditional approaches (test last development, TLD). Existing literature reviews show varying results for TDD.ObjectiveThis study investigates how the conclusions of existing literature reviews change when taking two study quality dimension into account, namely rigor and relevance.MethodIn this study a systematic literature review has been conducted and the results of the identified primary studies have been analyzed with respect to rigor and relevance scores using the assessment rubric proposed by Ivarsson and Gorschek 2011. Rigor and relevance are rated on a scale, which is explained in this paper. Four categories of studies were defined based on high/low rigor and relevance.ResultsWe found that studies in the four categories come to different conclusions. In particular, studies with a high rigor and relevance scores show clear results for improvement in external quality, which seem to come with a loss of productivity. At the same time high rigor and relevance studies only investigate a small set of variables. Other categories contain many studies showing no difference, hence biasing the results negatively for the overall set of primary studies. Given the classification differences to previous literature reviews could be highlighted.ConclusionStrong indications are obtained that external quality is positively influenced, which has to be further substantiated by industry experiments and longitudinal case studies. Future studies in the high rigor and relevance category would contribute largely by focusing on a wider set of outcome variables (e.g. internal code quality). We also conclude that considering rigor and relevance in TDD evaluation is important given the differences in results between categories and in comparison to previous reviews.     

Learning and transfer effects in simulated industrial information processing tasks

Practice and alternation among a set of jobs are the characteristics of any jobbing industry. Learning and transfer of learning, which are the main human factors issues in practice and alternation, were investigated through laboratory simulation of two typical industrial information processing tasks. In the first experiment a location task was examined while a search task was investigated in experiment two. In both experiments two levels of task complexity, two groups of subjects, and two positions were combined in a 2 × 2 latin square formation with 800 trials in each task level. The results show that learning pattern appears to be task dependent, with quicker learning in location tasks than in search tasks. Learning also transfers differently for the tasks considered. Implications for the industrial training program for assembly tasks are discussed.     

In search of evidence for model-driven development claims: An experiment on quality,effort, productivity and satisfaction

ContextModel-Driven Development (MDD) is a paradigm that prescribes building conceptual models that abstractly represent the system and generating code from these models through transformation rules. The literature is rife with claims about the benefits of MDD, but they are hardly supported by evidences.ObjectiveThis experimental investigation aims to verify some of the most cited benefits of MDD.MethodWe run an experiment on a small set of classes using student subjects to compare the quality, effort, productivity and satisfaction of traditional development and MDD. The experiment participants built two web applications from scratch, one where the developers implement the code by hand and another using an industrial MDD tool that automatically generates the code from a conceptual model.ResultsOutcomes show that there are no significant differences between both methods with regard to effort, productivity and satisfaction, although quality in MDD is more robust to small variations in problem complexity. We discuss possible explanations for these results.ConclusionsFor small systems and less programming-experienced subjects, MDD does not always yield better results than a traditional method, even regarding effort and productivity. This contradicts some previous statements about MDD advantages. The benefits of developing a system with MDD appear to depend on certain characteristics of the development context.     

The effect of individual and task characteristics on decision aid reliance

This research explores reliance behaviours of decision-makers using a decision aid. Objective and subjective task characteristics in the form of task complexity and task difficulty, respectively, are examined, along with the effect of the individual characteristic of expertise. A total of 130 subjects (65 novices and 65 experienced practitioners) completed a lab experiment using a decision aid (Insolve-DG) to help them make decisions for two insolvency tasks with differing levels of complexity. The research finds that the objective task characteristic (task complexity) and individual characteristic (expertise) both affect reliance behaviours; however, their effects are fully mediated by the subjective task characteristic (task difficulty). Expertise and task complexity are both associated with the degree of task difficulty experienced by an individual user: increasing task complexity increases task difficulty, and increasing expertise reduces task difficulty. Task difficulty and task complexity are established as different constructs; and importantly it is task difficulty, not task complexity, that ultimately affects reliance.     

Operational definition and automated inference of test-driven development with Zorro

Test-driven development (TDD) is a style of development named for its most visible characteristic: the design and implementation of test cases prior to the implementation of the code required to make them pass. Many claims have been made for TDD: that it can improve implementation as well as design quality, that it can improve productivity, that it results in 100% coverage, and so forth. However, research to validate these claims has yielded mixed and sometimes contradictory results. We believe that at least part of the reason for these results stems from differing interpretations of the TDD development style, along with an inability to determine whether programmers actually follow whatever definition of TDD is in use. Zorro is a system designed to automatically determine whether a developer is complying with an operational definition of Test-Driven Development (TDD) practices. Automated recognition of TDD can benefit the software development community in a variety of ways, from inquiry into the “true nature” of TDD, to pedagogical aids to support the practice of test-driven development, to support for more rigorous empirical studies on the effectiveness of TDD in both laboratory and real world settings.     

基于过程模型随机仿真的TDD模块选取建模方法研究*

为了帮助项目经理合理选择TDD实施模块,基于进程代数方法对测试驱动软件开发过程和非测试驱动软件开发过程建立过程仿真模型。通过用例度量软件模块的复杂性来获取随机变量参数对模型调参,并采用该模型得到仿真结果。提出TDD模块选取算法来分析仿真结果并得出最佳TDD实施策略,最终为项目经理提供合理的决策。     

Relations between complexity,quality and cognitive automation in mixed-model assembly

Modern assembly systems are adapting to the increased mass customization resulting in shorter cycle times, more variants and increased complexity for final assembly operators. More variants require additional support functions targeted for assembly operators to avoid assembly errors. An industrial case study was carried out in order to examine the relations between complexity, quality and cognitive automation. Complexity was quantified by the measure Operator Choice Complexity (OCC). Assembly errors were measured from historical data and cognitive automation was assessed on a task level. Results show that complexity positively correlates with assembly errors and that the usage of cognitive support is low. Over 60% of all tasks in the study were performed based on own experience. Increased usage of cognitive automation is needed in final assembly to minimize the negative effects of complexity.     

Effect of Job Rotation Types on Productivity,Accident Rate,and Satisfaction in the Automotive Assembly Line Workers

This study investigates different autonomous job rotation types to analyze their impacts on productivity, accident rate, and worker's satisfaction. The subjects of research were 422 assembly‐line units in 3 production plants from an automobile company. The preferred rotation types and workers’ satisfaction scores in the 422 units were surveyed by the average worker's experience and work productivity, quality and accident rate scores were traced over a 5‐year period. Results showed that workers with little work experience preferred to work in shorter cycles composed of a small variety of tasks, whereas workers with more experience preferred longer cycles composed of a larger variety of tasks. In addition, autonomously chosen rotation systems proved to boost productivity and work satisfaction when compared to units that did not implement job rotation. In contrast, only 4 out of 10 rotation types showed improvements in decreased number of accidents. Of the 10 rotation types, ones with daily cycles with a small range of tasks and weekly cycles with a wide range of tasks displayed the most promising results for productivity, work satisfaction, and accident prevention. This study implies that the preferred types of rotation do not necessarily lead to higher productivity, safety, and satisfaction. The results of this study are expected to serve as a basic guideline for job design.     

Exploring factors underlying effective office information systems

Office Automation (OA) traditionally emphasized productivity of the clerical workforce, ignoring higher-level office tasks. Office Information Systems (OIS) are meant to support office tasks done by managers and professionals within a company. By surveying local firms through telephone interviews, we investigate the factors responsible for effective OIS. We analyzed the results by partitioning responses by type of firm, firm size, degree of integration of OIS tools, top management support, and usage of OIS tools. Some findings are: professional service firms reported a greater improvement of productivity due to the introduction of OIS than non-professional service firms. A higher degree of system integration led to higher success rates of OIS in the perspective organization.     

Assembly failures and action cost in relation to complexity level and assembly ergonomics in manual assembly (part 2)

Earlier studies have demonstrated strong relationships between manual assembly at high physical load levels and increased amounts of quality defects compared to assembly at low physical load levels. A recent Swedish interview study of engineers in design and manufacturing engineering indicated that assembly complexity factors are of additional importance for the assembly quality. The objective of this study was therefore to examine the significance of high and low complexity criteria and the relationships between assembly ergonomics and assembly complexity and quality failures by analyzing manual assembly tasks in car manufacturing. In total, 47 000 cars were analyzed and the results showed several significant correlations between assembly ergonomics and assembly complexity, assembly time, failures and action costs. The action costs for high complexity tasks were 22.4 times increased per task per car compared to low complexity tasks.     

Review visual attention and spatial memory in building inspection: Toward a cognition-driven information system

With the increasing complexity of modern buildings, it is becoming more challenging for the professionals in the Architecture, Engineering, and Construction (AEC) industry to effectively digest complex engineering and design information and develop an accurate spatial memory that is critical to their daily tasks. As emerging visualization technologies, such as Virtual Reality, are considered as a promising solution, there is a pressing need to understand the mechanism by which different information visualization methods affect AEC task performance. Cognition literature has discovered a strong relationship between attention and memory development, but little has been done to understand how the visual attention patterns during the design documents review affect the effectiveness of spatial memory in AEC tasks. To fill the knowledge gap, this paper presents a human-subject experiment (n = 63) to test how spatial knowledge is acquired in a building inspection task and how the different visual attention patterns affect the development of spatial memory. Participants were asked to review the design information of a real building on campus. To trigger different attention patterns, they were randomly assigned to one of the three groups based on the forms of information given in the review session, including 2D, 3D, and VR groups. After a brief review session, participants were asked to go to the real building to identify discrepancies (based on memory) that were intentionally inserted by the authors. The inspection performance was used to evaluate the spatial memory development. The results indicate that in general there is a positive relationship between test subjects’ visual attention (fixation time) and spatial memory, but the increasing rate varies across the three groups, suggesting that visual context plays a critical role in the development efficiency of spatial memory. The findings also indicate that the visual attention – spatial memory relationship may be mediated by the use of different spatial knowledge acquisition strategies. This study is expected to contribute to the construction information technology literature by setting the cornerstone of a cognition-driven information system that tailors into the spatial cognitive process of AEC professionals.     

Impact of training standard complexity on inspection performance

Research in the area of visual inspection has shown that various factors influence inspection performance. Task factors have been identified as one of the primary classes of factors influencing the complexity of inspection tasks. If inspection task complexity is to be reduced it is essential to understand the influence of various task factors and prescribe interventions based on the impact of these factors. Moreover, historical work in this area has shown that the greater the difficulty of a vigilance task, the more engaged operators may become. Therefore, this research studies the influence of the following task factors: number of defect types, defect standard complexity, defect probability, and defect distribution on both the visual search and decision‐making components of a contact lens inspection task. This study was conducted using a computer simulation of a real world contact lens inspection task using 28 student subjects. Performance was measured on both the visual search and decision‐making components of the task. The results revealed a negative influence of defect standard complexity and a positive influence of defect probability on both the visual search and decision‐making components of the inspection task. © 2006 Wiley Periodicals, Inc. Hum Factors Man 16: 109–132, 2006.     

Kneeling trunk kinematics during simulated sloped roof shingle installation

Trunk musculoskeletal disorders are common among residential roofers. Addressing this problem requires a better understanding of the movements required to complete working tasks, such as affixing shingles on a sloped residential roof. We analyzed the extent to which the trunk kinematics during a shingling process are altered due to different angles of roof slope. Eight male subjects completed a kneeling shingle installation process on three differently sloped roof surfaces. The magnitude of the trunk kinematics was significantly influenced by both slope and task phase of the shingling process, depending on the metric. The results unequivocally point to roof slope and task phase as significant factors altering trunk kinematics. However, extension of the results to roofing workers should be done carefully, depending on the degree to which the study protocol represents the natural setting. Future studies on shingle installation in residential roofing should absolutely consider capturing a wider array of shingling procedures in order to encapsulate all the possible methods that are used due to the lack of a standardized procedure.     

Test-Driven Development of Relational Databases

In test-first development, developers formulate and implement a detailed design iteratively, one test at a time. Test-driven development (also called test-driven design) combines TFD with refactoring, wherein developers make small changes (refactorings) to improve code design without changing the code's semantics. When developers decide to use TDD to implement a new feature, they must first ask whether the current design is the easiest possible design to enable the feature's addition. Implementing test-driven database design involves database refactoring, regression testing, and continuous integration. TDDD is an integrated part of the overall development process, not a standalone activity that data professionals perform in parallel with application TDD. Although from a technical view point, TDDD is relatively straightforward, we must overcome several challenges to its whole sale adoption throughout the IT community