Designing and Conducting an Empirical Study on Test Management Automation

Test management aims at organizing, documenting and executing test cases, and at generating execution reports. The adoption of a support tool and of a standard process for such activities is expected to improve the current practice. ITALO is a European project devoted to the evaluation of the benefits coming from test management automation. In this paper the experiences collected and the lessons learned during ITALO are summarized. A formal methodology was adopted for the selection of a support tool among those available from the market. A survey of the current practice in component testing was conducted to adapt the existing process model so as to obtain the greatest benefits from automation. An empirical study was then designed to measure the effects that are expected to be produced by the new test process complemented with the introduction of the support tool. Three pilot projects were conducted to measure the benefits obtained from tool usage and process modification. Results are presented and discussed in this paper.