Construction of a criterion-referenced, diagnostic test for an individualized instruction program.

In the absence of a well-defined technology for the construction of criterion-referenced tests, a methodology was devised for use in developing and evaluating a diagnostic test keyed to 14 modules of individualized, US Navy shipboard instruction. The test was constructed in 2 phases. In the 1st phase, preinstruction and postinstruction groups each consisted of 100 Navy boiler technicians; in the 2nd phase, each group consisted of 75 Navy boiler technicians. In both phases, 25 members of each instruction group were chosen randomly to form cross-validation samples. The main construction procedures included (a) writing and refining an item pool, (b) selecting items that best discriminated between instruction groups, (c) determining cutoff scores, (d) validating items on cross-validation samples, and (e) estimating test–retest reliability. High face validity was achieved by using materials that were encountered on the job and by having job experts write the items. In the final construction phase, the amount of agreement between actual instruction-group membership of the cross-validation sample and that diagnosed by test scores ranged from 68 to 92%. Also, the discrimination ability of refined items improved significantly. Across a test and retest during a tryout phase, the agreement in diagnostic decisions ranged from 71 to 96%. (10 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)