Incremental Validity Principles in Test Construction.

The authors articulate 5 basic principles for enhancing incremental validity, both among elements within a test and between tests, during test construction: (a) careful, precise articulation of each element or facet within the content domain; (b) reliable measurement of each facet through use of multiple, alternate-form items; (c) examination of incremental validity at the facet level rather than the broad construct level; (d) use of items that represent single facets rather than combinations of facets; and (e) empirical examination of whether there is a broad construct or a combination of separate constructs. Using these principles, the authors offer specific suggestions for modifications in 3 classic test construction approaches: (a) criterion keying, (b) inductive test construction, and (c) deductive test construction. Implementation of these suggestions is likely to provide theoretical clarification and improved prediction. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On Construct Validity: Issues of Method and Measurement.

Fifty years ago, L. J. Cronbach and P. E. Meehl (1955) advocated for the concept of construct validity, noting that psychologists study hypothetical, inferred entities and that validating measures of such entities involves basic theory testing. Three important developments in clinical assessment following that seminal article are noteworthy. First, clinical research has benefited from greater theoretical integration and subsequent differentiation among related constructs. Second, implementation of ongoing, critical evaluation of all aspects of the construct validity process, including theory development, hypothesis specification, research design, and empirical evaluation, has improved clinical assessment. Third, improvement in evaluating fit between hypotheses and observations has been sought. Improved means of evaluating multitrait, multimethod designs, and ways to increase their clinical representativeness, are one encouraging development. Ongoing efforts to improve the construct validity process reflect the legacy of L. J. Cronbach and P. E. Meehl. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity versus reliability.

After reviewing recent studies involving the selection of items for interest scales, in which scales with higher validity (and fewer items) generally had lower reliability, the author presents the original odd-even reliabilities and recently-collected test-retest reliabilities (over an average 18-year interval) for 15 scales of the Strong VIB. The test-retest reliabilities were all lower than the odd-even reliabilities, and the shrinkage was greatest for those scales with the lowest original reliabilities. It is concluded that, for prediction in the distant future, scale reliability is important. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

"Validity" of short-form intelligence tests.

A short-form intelligence test is commonly "validated" by demonstrating a high correlation between it and the full scale from which it is taken. It has been argued, however, that the correlation between the short form and the full scale is less meaningful than the extent to which they agree in classifying individuals as to intellectual level; the latter depends in part upon the width of the categories in the classification system. A table is provided which shows the theoretical agreement between the short form and the full scale as a function of their correlation and category width. Empirical values from two studies approximate fairly closely the values given in this table. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the Complexity of Quantifying Construct Validity: Reply.

To validate a measure of a construct is to validate a theory: The process includes theory specification, theory elaboration, choice of methods, choice of research design, reliance on necessary auxiliary theories, and empirical observation. Evaluating the success of the process is likely to be enhanced through quantification. To date, scientists have quantified the final step of empirical observation, but earlier steps in the validation process appear more difficult to measure. In this issue, D. Westen and R. Rosenthal (2005; see record 2005-16347-003) emphasize that threats to validity are often methodological ones that precede empirical observation. Yet, their earlier attempt to quantify construct validity focused only on the final, empirical observation step of the process. For that reason, it can produce positive results despite real theoretical and methodological threats to validity. Nonetheless, the laudable attempt by these distinguished authors to quantify construct validity reminds us of the complexity of the construct validation process. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity and Meaning: Comment.

Comments on an article by Robert L. Ebel (see record 1962-05654-001). This article discusses basic problems in psychological testing and measurement, and as such the author touches upon basic problems of scientific psychology as well. According to the commentator the problems the author raises are very well taken. For a while it seems that he will come up with what points to a solution but some reflection shows that although he approached the "truth," he failed to reach it. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Incremental Validity and the Assessment of Psychopathology in Adults.

Results from studies on incremental validity are described for (a) interviews, (b) personality inventories, (c) projective techniques, and (d) brief self-rated and clinician-rated measures. In some of the studies (clinical judgment studies), psychologists were given increasing amounts of information. In other studies (statistical prediction studies), increasing amounts of assessment information were entered into a statistical prediction rule. Although relatively little research has been conducted on incremental validity, results that have been obtained tend to favor the use of interviews, personality inventories, and brief self-rated measures. Results are generally less encouraging for projective techniques. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity data on the psychotic reaction profile.

Psychotic Reaction Profile (PRP) ratings were obtained on random samples from 4 identifiable groups of hospitalized psychotics: closed-ward patients without ground privileges (N = 59), closed-ward patients with privileges (N = 60), open-ward patients (N = 52), and open-ward patients involved in predischarge planning (N = 25). It was hypothesized that if these groups reflect an ordering in terms of degree of psychiatric incapacitation, then the mean PRP-scale scores for these groups should show the same ordering. Significant between-group differences were found on 3 scales (Withdrawal, Thinking Disorganization, and Paranoid Belligerence). Comparisons made between the means of the "adjacent" groups, using Duncan's test, indicated the scales to be sensitive enough to differentiate in most cases. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Lowry test: A simple status-free measure of intellectual ability.

The validity of an intelligence test is discussed. "The Lowry Reasoning Test Combination has been found to be relatively free of social status bias and to measure intellectual function. It is easily administered and simply scored and does not depend upon a high level of verbal ability. Variance in concept difficulty is obtained by altering combinations of constructs while keeping the verbal material on a uniformly simple level. Whereever such a discriminative and effective selection device is needed the present writers would recommend that the Lowry test be tried." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The measurement of creativity in machine design.

"(1) A battery of four measuring devices [Power Source Apparatus Test, Application of Mechanisms Test, Personal Inventory, Personal History Form] for the discrimination of creative from noncreative machine designers has been developed. (2) The concurrent validity of this battery, within companies, is such that it would probably predict the correct classification of about three-fourths of the members of two equal groups of creative and noncreative designers." "Creatives" and "noncreatives" are not differentiated by general mental ability, and creativity seems to be somewhat specific to the field in question. 18 references. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Incremental Validity of Psychological Testing and Assessment: Conceptual, Methodological, and Statistical Issues.

There has been insufficient effort in most areas of applied psychology to evaluate incremental validity. To further this kind of validity research, the authors examined applicable research designs, including those to assess the incremental validity of test instruments, of test-informed clinical inferences, and of newly developed measures. The authors also considered key statistical and measurement issues that can influence incremental validity findings, including the entry order of predictor variables, how to interpret the size of a validity increment, and possible artifactual effects in the criteria selected for incremental validity research. The authors concluded by suggesting steps for building a cumulative research base concerning incremental validity and by describing challenges associated with applying nomothetic research findings to individual clinical cases. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The development and validation of a test of creativity in engineering.

"Two forms of a 20-item test of creativity were developed through analyses of item response data of 345 engineering students at Purdue University. Three scores were developed for the test: Fluency score, Flexibility score, and Originality score. Investigations of the validity, reliability, interscorer agreement, relationships with other tests, and 'face validity' of the Creativity scores were made with 64 product development engineers and process engineers in a large automobile accessories manufacturing company." Significant validity was found (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity and utility of a multidimensional model of received support in intimate relationships.

The authors of the present study statistically address the largely conceptual debate about the multidimensional nature of received support in intimate relationships. The Support in Intimate Relationships Rating Scale (SIRRS) was factor analyzed in 3 samples of dating and married couples. A novel, 4-factor structure of support types was generated that constituted esteem/emotional, physical comfort, informational, and tangible support types. This structure was reliable and valid in dating and marital relationships, across men and women, and across time. Each support type also demonstrated incremental validity for explaining marital adjustment, depression symptoms, and anxiety symptoms longitudinally. This study is among the first to generate and cross-validate a factor analytically derived set of support types for received support and the first to do so regarding partner support specifically. This is also the first study to replicate a distinct set of support types across different types of intimate relationships, across both sexes, and over time within relationships. Implications for enhancing functional theories of support and for augmenting the construct validity of a multidimensional model of received support are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Incremental Validity in the Psychological Assessment of Children and Adolescents.

Incremental validity in the process of psychological assessment of children and adolescents is explored. The authors highlight the dependence of the incremental validity of assessment information on factors such as goal of assessment, other information available, base rate of the problem or outcome, age or gender of the child, and type of problem being assessed. The authors discuss the incremental validity of assessment information from alternate sources, methods, and constructs. In view of the limited number of studies directly relevant to incremental validity in child clinical assessments, the authors call for more clinically relevant research. To have the greatest impact on child and adolescent services, this research must be readily generalized and immediately relevant to actual clinical practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Must All Tests Be Valid?

Comments on the article "Must All Tests Be Valid?" by R. L. Ebel (see record 1962-05654-001.) The author finds Ebel's conclusions provocative, but open to disagreement and proposes that: (a) validity is not a confusing concept which needs to be replaced; (b) the standard of meaningfulness is not new, but only a necessary element of validity; and (c) meaningfulness is not a suitable substitute for traditional predictive validity. The author of this comment concludes that meaningfulness is not a substitute for validity, although validity does not exclude Ebel's meaningfulness, which is, in fact, a well defined part of what is meant by validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Incremental Validity of New Clinical Assessment Measures.

The authors address conceptual and methodological foundations of incremental validity in the evaluation of newly developed clinical assessment measures. Incremental validity is defined as the degree to which a measure explains or predicts a phenomenon of interest, relative to other measures. Incremental validity can be evaluated on several dimensions, such as sensitivity to change, diagnostic efficacy, content validity, treatment design and outcome, and convergent validity. Indices of incremental validity can vary depending on the criterion measures, comparison measures, and individual differences in samples. The authors review the rationale for, principles, and methods of incremental validation, including the selection of comparison and criterion measures, and address data analytic strategies and the conditional nature of incremental validity evaluations in the selection of measures. Incremental validity contributes to, but is different from, cost-benefits, which reflect the cost of acquiring the data and the benefits from the data. The impact of an incremental validity index on whether a measure is selected will be moderated by the cost of acquiring the new data, the importance of the measured phenomenon, and the clinical utility of the new data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The adjective check list as a personality assessment technique with children: Validity of the scales.

Adjective Check List (ACL) ratings of personality characteristics were made by the mothers of 52 pairs of grade-school twin girls. The 24 ACL scales were factor analyzed, yielding 3 factors named extroversion-introversion, social desirability, and personality traits associated with intelligence. The factors were found to be similar to those extracted from other personality tests. In addition to the ACL, the TAT, the Vineland Social Maturity Scale, the Fels Behavior Scales, 2 WISC subtests, the Goodenough drawings, an experimental Toys game, an activity interview, and background information were scored for the sample. The ACL scales correlated systematically with both direct and indirect measures of the twins' behavior, lending confidence in the validity of the ACL scales for a population of young children. (17 ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Creativity Research and Test Theory.

Having (just barely) survived 20 years of sentimentality in which half of psychology attempted to supplant competence by personal adjustment, we now appear to be entering another equally unproductive period in which creativity will be lauded above all other human attributes, particularly that of intelligence. I have just read a study of creativity in which the investigators again found little relationship between intelligence and creativity. Their measurements of intelligence were based upon whatever happened to be already available. Is it too much to expect that experimenters in this area be at least minimally sophisticated in test construction, analysis, and theory if they base most of their conclusions on test data? Before creativity research gets completely out of control, some effort must be devoted to definitions of terms. Creativity research will flounder until the measuring instruments and procedures are improved, at least to the level of current test theory. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The definition and measurement of job involvement.

The purpose of the present research was to define job involvement, develop a scale for measuring it, gather evidence on the reliability and validity of the scale, and to learn something about the nature of job involvement through its correlation with other job attitudes. This paper describes the development and validation of a scale measuring job involvement, the resulting scales, the relation between job involvement, and other job attitudes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Static structuralism.

R. L. Ebel (see record 1962-05654-001) examines present concepts regarding test validity and comes to the conclusion that psychologists concerned with this problem are asking essentially unanswerable questions. This startling point of view is bolstered by evidence that many useful tests do not have adequate validity data published in their support and that reviewers of tests seem unconcerned by the lack of such data. Ebel then courageously calls into question the accepted "face validity" of the concept of validity and suggests that the meaningfulness of a test be substituted as a criterion. In a later issue of this journal, Jordan (see record 2005-11890-001) and Skager (see record 2005-11890-002), react. Skager is conservative and implies that the problem is a technical one. For him, the correct questions to be asked are known. Problems occur because we fail to ask the obvious and familiar question regarding validity or because technique is insufficiently developed to generate measures with adequate validity but then we know our limitations. Jordan points out that Ebel's suggestion of substituting meaningfulness for validity begs the question. He argues that meaningfulness rests on an appropriate theoretical analysis of the problem which must precede measurement operations and that, if the theoretical analysis is meaningful and valid, the measurement operations will follow of their own accord. The merit of Ebel's paper is that he calls into question glib pronunciamentos that have characterized the theory of test construction. He is aware that something is amiss and implicates the validity issue. Skager clings to the traditional approach and, in effect, denies the problem. Jordan agrees that the problem exists and pinpoints the area of difficulty but neither he nor the other writers suggest the explicit theoretical contents that may give rise to the whole problem in the first place. I would like to suggest that the real culprit is that dominant theoretical bias, more or less explicit in most test construction, which I will term "static structuralism." (PsycINFO Database Record (c) 2010 APA, all rights reserved)