Easier Tasks Can Have Better Discriminating Power: The Case of Verbal Fluency.

Loren and Jean Chapman highlighted the importance that the psychometric characteristics of a task may have when comparing different populations (L. J. Chapman & J. P. Chapman, 1973, 1978). Specifically, they pointed out that when comparing 2 tasks measuring different constructs, there is a greater likelihood of finding larger deficits on a task with higher discriminating power, irrespective of what the task measures. While researchers have addressed this issue, they typically have assumed that more difficult tasks have greater discriminating power, often without actually measuring discriminating power. In this article, the authors present data from 2 studies using phonologic and category fluency tasks to demonstrate that the critical factor for task selection and matching is discriminating power, not task difficulty. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A process-oriented approach for averting confounds resulting from general performance deficiencies in schizophrenia.

The most pervasive and least well-addressed problem in cognitive studies of schizophrenia is the propensity of schizophrenia patients to show inferior performance on a variety of cognitive tasks. Consequently, apparent specific cognitive abnormalities may actually reflect the interaction of task discriminating power with generalized deficit. L. J. Chapman and J. P. Chapman (1973) suggested psychometric approaches for eliminating such artifactual group differences. Unfortunately, their solution neglects important issues of process specification and does not provide a viable strategy for process-oriented investigators. Psychometric remediation of artifactual Group?×?Task interactions inevitably confounds the processes being measured, resulting in theoretically ambiguous findings. Moreover, evidence that changes in measurement reliability can both increase and decrease group discrimination challenges a basic underlying assumption of the Chapmans' matching solution. This article presents a process-oriented approach to solving this problem in schizophrenia research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some perspectives on "the attenuation paradox in test theory."

Clarification is given of the "paradox" posed by Loevinger (see 29: 3299), i.e., "validity" (the product-moment r of test score with the common factor of test items) decreases upon the increase beyond a certain point of item intercorrelations in a test whose items are of equal difficulty. Four points are made: (1) the "region of paradox" is reduced using a curvilinear r rather than the usual product-moment "validity"; (2) the paradox's incidence in aptitude and achievement tests is not as great as is frequently believed; (3) over-all "validity" for a group is inferior to the test's discriminating power for a specified decision problem for a specific examinee; and (4) the conception that "… for optimum results the percentage of correct answers to each item should equal the percentage of examinees to be selected" is correct only if the latter percentage is 50. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Problems in the measurement of cognitive deficits.

Considers that artifacts cloud the findings in a majority of the studies on differential cognitive deficit because of differences in discriminating power of tasks coupled with generalized deficit of patients. With 2 tests of differing reliability, the test with the higher reliability will yield greater performance deficit for the less able Ss. The advisability, in studies of differential cognitive deficit, is discussed of matching tasks on reliability, shape of the distribution of scores, and mean, variance, and shape of the distribution of item difficulty, using normal Ss alone as a standardization group. Data from a study with 49 severely disturbed schizophrenics and 206 normals is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Demonstrating specific cognitive deficits: A psychometric perspective.

Cognitive deficits associated with psychopathology often do not occur in isolation. Consequently, identifying a specific deficit in a disorder requires comparing the magnitude of group differences on theoretically relevant measures with those on control tasks measuring other constructs. L. J. Chapman and J. P. Chapman (1973) noted that common forms of such Group?×?Task interactions are theoretically ambiguous unless performance measures have comparable discriminating power. The principles of psychometric matching for discriminating power developed in the Chapmans' research program are reviewed, and both criticisms and alternative psychometric approaches are evaluated. Psychometric matching can be mindful of threats to the construct validity of measures and frequently remains methodologically necessary. Otherwise, interactions involving measures that vary in sensitivity to individual differences may be misinterpreted as evidence for specific deficits. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Schizophrenic response to affectivity in word definition.

120 drug-free schizophrenics and 167 nonpsychotic Ss were given affect-laden and affectively neutral free-response vocabulary items. The 2 subtests were matched on psychometric characteristics that affect the power of the subtest to discriminate among ability levels of nonpsychotic Ss. These characteristics (coefficient alpha, mean and standard deviation of test score, standard deviation of item difficulty, and shape of the distribution of item difficulty) were matched using nonpsychotic Ss with the same sum of the subtest scores as the schizophrenics. Both chronic and newly admitted schizophrenics were no less accurate on the affective subtest than on the neutral subtest. It is concluded that most published findings on schizohprenic response to affective stimuli appear to be artifacts of unmatched tests. (33 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Associative intrusions in schizophrenic verbal behavior.

Developed 2 multiple-choice vocabulary subtests in which the items of 1 subtest contained associates to the stimulus word as incorrect alternatives, while the items of the other subtest contained no associative distractors. Using a standardization sample of normal Ss (65 firemen with average IQ and 27 prison inmates with below-average IQ), the subtests were matched on coefficient alpha, mean, variance of item difficulty, and shape of the distribution of item difficulty. The subtests were then given to 42 chronic schizophrenics. Results reveal that schizophrenics, unlike normal Ss, showed a significantly greater deficit on the with-associates subtest than on the no-associates subtest. This finding demonstrates heightened susceptibility to associative distraction in schizophrenia, without the possible artifact of generalized deficit coupled with differential discriminating power of tests. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity generalization results for two job groups in the petroleum industry.

Estimated the transituational generalizability (or "transportability") of the validities of 4 types of cognitive tests (Bennett Mechanical Comprehension Test, Richardson-Bellows-Henry Chemical Comprehension and RBH Arithmetic Reasoning Tests, and general intelligence) and a weighted biographical information blank for performance in 2 petroleum industry job groups. Generalizability was strongly supported for mechanical and chemical comprehension tests for both jobs. In the case of the chemical comprehension tests, virtually all variance of observed validity coefficients was accounted for by artifacts, and thus the hypothesis of situational specificity was rejected. Support for generalizability was substantial for general mental ability and arithmetic reasoning tests. It was found, however, that corrections for variance due to sampling error accounted for an average of 90% of all variance due to artifacts, indicating the relative unimportance of differences between sudies in criterion reliability and in range restriction in accounting for variation in observed validities. Generalizable multivariate validities were estimated for various test batteries using beta and unit weights. Finally, true score beta weights were used to estimate the causal role of the 4 cognitive abilities in job performance. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of aging, distraction, and response pressure on the binding of actors and actions.

Two experiments provide evidence for an age-related deficit in the binding of actors with actions that is distinct from binding deficits associated with distraction or response pressure. Young and older adults viewed a series of actors performing different actions. Participants returned 1 week later for a recognition test. Older adults were more likely than young adults to falsely recognize novel conjunctions of familiar actors and actions. This age-related binding deficit occurred even when older adults could discriminate old items from new items just as well as could young adults. Young adults who experienced distraction or time pressure also had difficulty discriminating old items from conjunction items, but this deficit was accompanied by a deficit at discriminating old and new items. These results suggest that distraction and response pressure lead to deficits in memory for stimulus components, with any deficits in binding ability commensurate with these deficits in component memory. Aging, in turn, may lead to binding difficulties that are independent of attention-demanding executive processes involved in maintaining individual stimulus components in working memory, likely reflecting declines in hippocampally mediated associative processes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development and validation of the Neuropsychology Behavior and Affect Profile.

This series of 4 studies describes the psychometric properties of the Neuropsychology Behavior and Affect Profile, which consists of 5 peer-rated scales (106 items) designed to measure personality change in brain-impaired individuals. Study 1 pertains to item derivation. Study 2 used relatives of 61 Ss identified as demented to determine the test's internal consistency. Results showed moderate levels of internal consistency across the 5 scales, with slightly higher coefficients (.68–.82) obtained for present (vs. premorbid) emotional status. High test–retest reliability was demonstrated in Study 3 (intraclass correlation coefficients ranged from .92 to .99). Study 4 established discriminant validity; the instrument differentiated 61 demented Ss from 88 normal elderly controls on the basis of present behavioral affective style. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Test difficulty, reliability, and discrimination as functions of item difficulty order.

In an attempt to evaluate the often stated rule that test items should be arranged in an increasing order of difficulty, the effect of item difficulty order on total test reliability, difficulty, and discrimination was investigated in a series of 4 experiments. Each experiment involved a comparison of 2 or more tests, containing the same 40 items and differing only with respect to the order of those items. The differently ordered forms did not, in any of the experiments, differ significantly in test difficulty or test reliability. The results with respect to discrimination were not as clear-cut. However, the results tend to lead to the conclusion that item difficulty order on a power test of facts and principles given in the normal college classroom will not significantly affect these 3 test statistics. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Note on the reliability of experimental measures and the power of significance tests.

The statistical theory of the power of significance tests, combined with the classical theory of the reliability of measurement, reveals that the power of a statistical test sometimes increases and sometimes decreases as the reliability coefficient of a dependent variable increases. This seeming paradox arises because the relation between statistical power and the reliability coefficient is not a functional relation unless another variable—either true variance or error variance—remains constant. The authors show that the paradox disappears if widely accepted, elementary results in statistical theory and measurement theory are considered together. This approach explains why some authors have reached different conclusions about how reliability influences significance tests. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Self-adapted testing: A performance-improving variant of computerized adaptive testing.

We conducted an experiment that contrasted a variant of computerized adaptive testing, self-adapted testing, with two traditional tests, a relatively difficult one and a relatively easy one, that were constructed from the same bank of verbal ability items. In a self-adapted test, the examinee, rather than a computerized algorithm, chooses the difficulty of the next item to be presented. Participants completed a self-report of text anxiety and were randomly assigned to take one of the three tests of verbal ability. Analyses of variance using Rasch estimates of ability and the standard error of those estimates as dependent measures demonstrated that the self-adapted test led to higher ability estimates and minimized the effect of test anxiety without any overall loss of measurement precision. Analysis of the item choices in the self-adapted test suggested that, in general, participants chose more difficult items as the test progressed. Anxiety was negatively associated with the difficulty of the initial choice but not associated with the rate of progress to higher difficulty items. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Adult age differences in recall of performed and nonperformed items.

Memory for performed cognitive activities (e.g., psychometric tests of intelligence), for performed brief actions (e.g., hand wave), and for nonperformed items (e.g., written words) was assessed for 102 older and 101 younger adults. Although enactment improved recall, the beneficial effects of enactment were the same for both age groups. In fact, more than 80% of the age-related variance in memory for performed items was shared with memory for nonperformed items. Working memory and perceptual speed were important to the age differences in memory for both types of items. Performed and nonperformed items showed different serial position effects. However, the correlation between memory for the 2 types of items was high, especially for older adults, suggesting that the 2 types of memory share many common processes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bifactor and item response theory analyses of interviewer report scales of cognitive impairment in schizophrenia.

A psychometric analysis of 2 interview-based measures of cognitive deficits was conducted: the 21-item Clinical Global Impression of Cognition in Schizophrenia (CGI-CogS; Ventura et al., 2008), and the 20-item Schizophrenia Cognition Rating Scale (SCoRS; Keefe et al., 2006), which were administered on 2 occasions to a sample of people with schizophrenia. Traditional psychometrics, bifactor analysis, and item response theory methods were used to explore item functioning and dimensionality and to compare instruments. Despite containing similar item content, responses to the CGI-CogS demonstrated superior psychometric properties (e.g., higher item intercorrelations, better spread of ratings across response categories) relative to the SCoRS. The authors argue that these differences arise mainly from the differential use of prompts and how the items are phrased and scored. Bifactor analysis demonstrated that although both measures capture a broad range of cognitive functioning (e.g., working memory, social cognition), the common variance on each is overwhelmingly explained by a single general factor. Item response theory analyses of the combined pool of 41 items showed that measurement precision is peaked in the mild to moderate range of cognitive impairment. Finally, simulated adaptive testing revealed that only about 10 to 12 items are necessary to achieve latent trait level estimates with reasonably small standard errors for most individuals. This suggests that these interview-based measures of cognitive deficits could be shortened without loss of measurement precision. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Some methodological issues in the development of quality of life measures for the evaluation of medical interventions

This paper discusses a series of important methodological issues in developing targeted health-related quality of life measures in studies of the effects of medical interventions. Such measures cannot be developed unless the evaluator understands the life domains that medical interventions affect. Qualitative discovery methods are needed to obtain this understanding. Once domains are targeted for measurement, careful and systematic laboratory pilot work should be used to select initial scale items. Psychometric evaluation of response patterns in subsequent field tests is needed to assess the measures. Less concern should be directed to internal consistency reliability of scales in the psychometric evaluation and more to the ability of short scales to reproduce total scale variance and to provide precise measurement within the range of the outcome where effects are expected. The paper closes with a discussion of modern methods of item response scaling that can be used to address these issues.     

Computer-adaptive testing: The impact of test characteristics on perceived performance and test takers' reactions.

This study examined how specific features of adaptive tests are related to test takers' reactions. Participants took a computer-adaptive test in which 2 features, difficulty of the initial item and difficulty of subsequent items, were manipulated, then responded to questionnaires assessing their reactions to the test. The data show that the relationship between a test's objective difficulty, which was determined by the 2 manipulated test characteristics, and reactions was fully mediated by perceived performance. Additional analyses evaluated the impact of feedback on reactions to the adaptive test. In general, feedback that was consistent with perceptions of performance was positively related to reactions. The results suggest that minor changes to the design of an adaptive test may potentially enhance examinees' reactions, (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The dot pattern expectancy task: Reliability and replication of deficits in schizophrenia.

The dot pattern expectancy (DPX) task was created to efficiently assess context-processing deficits in patients with schizophrenia. Three studies investigated the characteristics of the DPX relevant for clinical applications. To answer questions regarding the psychometric properties of the task, performance on this task was studied in 2 healthy samples. Acceptable reliability and internal consistency and retest reliability were found for most measures of context processing, but not for a difficulty control condition. These characteristics were also found in a suggested brief version (DPXbrf), which may be more practical for clinical purposes. In a 3rd study, schizophrenia patients showed a specific deficit in context processing, replicating previous findings. Findings of these studies indicated some promise for use of this task in measuring context processing and also identified characteristics of this task that need to be strengthened to increase reliability, feasibility, and single-subject interpretability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)