Evaluating the impact of CASE: an empirical comparison of retrospective and cross-sectional survey approaches

The purpose of this paper is to evaluate two methods of assessing the productivity and quality impact of Computer Aided Software Engineering (CASE) and Fourth Generation Language (4GL) technologies: (1) by the retrospective method; and (2) the cross-sectional method. Both methods involve the use of questionnaire surveys. Developers' perceptions depend on the context in which they are expressed and this includes expectations about the effectiveness of a given software product. Consequently, it is generally not reliable to base inferences about the relative merits of CASE and 4GLs on a cross-sectional comparison of two separate samples of users. The retrospective method that requires each respondent to directly compare different products is shown to be more reliable. However, there may be scope to employ cross-sectional comparisons of the findings from different samples where both sets of respondents use the same reference point for their judgements, and where numerical rather than verbal rating scales are used to measure perceptions.