Integrity tests: Facts and unresolved issues.

Describes 2 independent reports, recently completed by the American Psychological Association (APA) and the US Congress Office of Technology Assessment, that examined many scientific and measurement issues concerning integrity testing (e.g., validity, criterion relevance). Background data are offered on a variety of tests collected by a survey of test publishers, providing a view of the industry's scope (e.g., test audience, user screening, score reporting) not available elsewhere. In the light of APA's Ethical Principles for Psychologists and Code of Conduct (APA, 1992), unresolved concerns are addressed that have a wide range of implications for the profession of psychology, the testing industry, and public policy (e.g., cutting scores, user screening and training, and test marketing practices). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Stereotype Threat Might Explain the Black-White Test-Score Difference.

Comments on an article by Paul Sackett, Chaitra Hardison and Michael Cullen entitled On Interpreting Stereotype Threat as Accounting for African American-White Differences on Cognitive Tests (see record 2004-10043-001). In their correction of the ostensibly widespread misinterpretation of Steele and Aronson's (see record 2000-16592-021) seminal study of the effects of stereotype threat on intellectual test scores, Sackett, Hardison, and Cullen expressed dismay and puzzlement that so many erudite people consistently have gone so far astray in their understanding of this matter. The gist of Sackett et al.'s correction was that interpreters of Steele and Aronson's results have ignored the researchers' statistical adjustment of their dependent measure for SAT scores and, consequently, have wrongly concluded that racial-group (i.e., Black-White) differences in test scores disappear when stereotype threat is removed. In their justification for this much needed clarification, Sackett et al. (2004) implied that the interpretation that stereotype threat explains the Black-White test score disparity is not plausible. Yet whether or not the construct of stereotype threat generally can account for the Black-White test-score disparity was not the question that was directly addressed by either Steele and Aronson's (1995) original study or Sackett et al.'s (2004) critique of it. It appears that Sackett et al. (2004), as well as the many people responsible for the allegedly faulty interpretation, essentially want an answer to the question, What causes or explains racial-group difference(s) in Black-White test scores? This question logically flows from (a) reviews demonstrating the chronic resistance of these differences to psychoeducational interventions, (b) general recognition that racial-group membership cannot cause behavior (e.g., differences in test scores), as well as (c) acknowledgment that use of test scores for high-stakes decision making under prevailing circumstances amounts to "racial profiling" condoned by society and the law. Therefore, if stereotype threat or analogous race or culture-related psychological constructs could be shown to account for the Black-White testscore disparity, then society would be relieved of the burden of unfair testing practices, and Sackett et al. would be relieved of the burden of "heading off future interpretive errors" (p. 11) regarding Steele and Aronson's results. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Fair employment, sound psychometric practice, and reality: A dilemma and a partial solution.

Considers that while many of the tests used in employee selection are valid nomothetically, they penalize the individual. The fairness of certain validity studies (e.g., the biodata approach and standardized tests of mental ability) is questioned. It is suggested that the use of job simulation tests may remedy this problem. It is noted that such tests would be more expensive and might not apply to certain jobs, but would allow the direct assessment of the behavior desired and permit the candidate to learn more about the position. It is proposed that the American Psychological Association publicly endorse the use of job simulations, consulting psychologists stress the advantages of job simulations and that business and educational institutions be encouraged to provide validity data for their specific situations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Implications for social policy of variability in racial groups.

Social policy and federal and state legislation require the use of single cut scores when tests of cognitive ability, knowledge, or skills (CAKS) are used to make high-stakes assessment decisions, such as whether students or employees may be promoted. Rationales offered for the requirement are that cut scores provide objective standards and are fairer than using subjective criteria, such as racial group membership. It is argued that failure to consider threats to statistical conclusion validity, such as differences in variability between groups, obscures the differential impact of using a common cut score as the basis for highstakes decisions. Analyses of 40 Black and White samples revealed that (a) Whites might be considerably advantaged and Blacks might be considerably disadvantaged by the same cut score and (b) depending on where the cut score is set, decisions based on ratios of numbers of Whites numbers of Blacks might be fairer than use of CAKS test cut scores. Implications for assessment practice and social policy are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Being creative with the predictors and criteria for success.

Sackett, Borneman, and Connelly (see record 2008-05553-001) argued that several common criticisms of cognitively laden tests are not well supported by the literature. The authors’ systematic exploration of research surrounding seven specific criticisms is laudable, and we do not find fault with their conclusions as presented. In evaluating the seven concerns, however, the authors largely neglected the criteria that such tests are intended to predict. As a result, readers may come away with the erroneous conclusion that all is well in the mass testing world of cognitive ability. We wish to expand on Sackett et al.’s review by raising concerns about traditional approaches to defining academic and organizational success. In doing so, we argue for the importance of creativity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The golden rule in psychological testing: Please, please don’t do it unto me.

In William Angoff's (1987) recent article in Theoretical and Philosophical Psychology, he presented several thought-provoking philosophical issues of concern to measurement-oriented psychologists. The first of these issues, the notion of test and item bias, is the topic of the present paper. In his article, he reports that average score differences in the performance on many cognitive tests have appeared among racial or ethnic groups. Twenty to thirty years ago, differential psychologists, summarizing and critically evaluating studies demonstrating performance differences on tests and in other behavior among cultural, ethnic and racial groups, agreed that such group differences indeed existed, but that causal interpretations differed markedly among the investigators at the time. More recently, the focus has shifted to what has been termed test fairness, with the focus falling more upon tests and their inadequacies than the behavior which underlies the tests. Similar findings and conclusions were frequently advanced in physiological tests of strength and endurance which are often used for civil service positions such as those in the fire fighting service and which frequently disadvantage women. The question raised is simply whether the discrimination is fair or not; that is, do given test score differences reflect variation in performance between groups which are reflective of true behavioral differences or are these differences artifactual and embedded in the assessment device itself, whether intentional or not. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

(Over and) beyond high-stakes testing.

Sackett, Borneman, and Connelly’s (see record 2008-05553-001) article and recent meta-analyses (e.g., Kuncel & Hezlett, 2007) should lay to rest any doubt over whether high-stakes standardized tests predict important academic and professional outcomes—they do. The challenge now is to identify noncognitive individual differences that determine the same outcomes. Noncognitive is, of course, a misnomer. Every psychological process is cognitive in the sense of relying on the processing of information of some kind. Why do so many psychologists, including myself, resort to the term noncognitive despite its obvious inappropriateness? (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The absence of underprediction does not imply the absence of measurement bias.

Sackett, Borneman, and Connelly (see record 2008-05553-001) recently discussed several criticisms that are often raised against the use of cognitive tests in selection. One criticism concerns the issue of measurement bias in cognitive ability tests with respect to specific groups in society. Sackett et al. (2008) stated that “absent additional information, one cannot determine whether mean differences [in test scores] reflect true differences in the developed ability being measured or bias in the measurement of that ability” (p. 222). Their discussion of measurement bias appears to suggest that measurement bias in tests can be accurately detected through the study of differential prediction of criteria across groups. In this comment, we argue that this assertion is incorrect. In fact, it has been known for more than a decade that tests of differential regression are not generally diagnostic of measurement bias (Millsap, 1997, 1998, 2008). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Racial differences in validity of employment tests: Reality or illusion?

Tested the fit of data on single-group validity of 19 employment tests to a statistical model assuming equal true validites for blacks and whites. For both subjective and objective criterion measures, observed frequencies of both kinds of single-group validity (significant for whites but not for blacks and vice versa) were not significantly different from those predicted by the null differences model. These findings cast serious doubt on the existence of single-group and differential validity as substantive phenomena. It is concluded that psychologists concerned with the applicability of employment tests to minority groups should direct their future efforts to the study and determination of test fairness rather than to the problem of racial differences in test validity. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Implementing fairness in racial-group assessment requires assessment of individuals.

Replies to comments by R. J. Griffore (see record 2007-18356-020) and D. A. Newman et al (see record 2007-18356-021) on the author's original article (see record 2006-21079-024) on test validity and cultural bias in racial-group assessment. Helms notes that, given that within-group variance exceeds between-groups variance, racial groups are probably simulating a psychological construct that is more strongly related to individuals' test scores than to their respective racial group's mean test scores. Therefore, models of individual differences, such as her Helms individual-differences (HID) model, that remove construct-irrelevant racial variance, are needed to make the testing process fair at the level of individual African American, Latino/Latina American, and Native American test takers. Her HID model is intended to focus attention on identifying the factors responsible for the racial-group-level differences and, thereby, assist test users to look beyond presumed physical appearance (e.g., racial-group designations) for explanations of individuals' cognitive abilities, knowledge, or skills test scores. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Speaking of fairness in testing.

In an article based on her presentation to the 114th Annual Convention of the American Psychological Association, Janet Helms (see record 2006-21079-024) described a concept of test fairness. Helms's approach to fairness appears to be based on the premise that socialization consists of experiences received from one's environment, which are simply internalized as personal attributes. From an ecological perspective, this is incorrect, misleading, and oddly evocative of social-cognitive and psychodynamic notions. Individuals are not passive recipients of culture, and they do not necessarily simply reflect their experiences. Although Helms intended her model to apply to racial and cultural groups, it seems that she has overlooked the potential of applying her concept of fairness to another important level of social organization: the family. R. J. Griffore suggests that it is unfair to avoid using valid tests that are said to be unfair because of racially or culturally based construct-irrelevant variance if those tests can help individuals make decisions that lead to success outcomes. The use of race or culture as a variable that in any way jeopardizes the prediction of success for individuals or groups is unfair for all groups for whom the test is used. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The impact of Canadian employment selection issues on validation research.

Canadian industrial/organizational psychologists are being challenged to justify the use of employment selection tests to ensure adherence to fair employment practice legislation. The author examines 4 issues related to validation research: the definition of terms, such as validity, test bias, and test fairness; sample size; the adverse impact of employment tests; and the legal responsibilities of test users. Strategies of concurrent validation, validity generalization, synthetic validation, and latent trait theory are discussed. (French abstract) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Responses to issues raised about validity, bias, and fairness in high-stakes testing.

We are pleased that our article (see record 2008-05553-001) prompted this series of four commentaries and that we have this opportunity to respond. We address each in turn. Duckworth (see record 2009-06923-012) and Kaufman and Agars (see record 2009-06923-013) discussed, respectively, two broad issues concerning the validity of selection systems, namely, the expansion of the predictor domain to include noncognitive predictors of performance and the expansion of the criterion domain to include additional criteria (e.g., creativity). We agree with these arguments, noting that they expand on points made in our original article. Wicherts and Millsap (see record 2009-06923-014) rightly noted the distinction between measurement bias and predictive bias and the fact that a finding of no predictive bias does not rule out the possibility that measurement bias still exists. They took issue with a statement we cited from Cullen, Hardison, and Sackett (2004) that if motivational mechanisms, such as stereotype threat, result in minority group members obtaining lower observed scores than true scores (i.e., a form of measurement bias), then the performance of minority group members should be under predicted. Our characterization of Cullen et al.’s (2004) statement was too cryptic; what was intended was a statement to the effect that if the regression lines for majority and minority groups are identical at the level of true predictor scores, then a biasing factor resulting in lower observed scores than true scores for minority group members would shift the minority group regression line to result in under prediction for that group. We do agree with Helms’s (see record 2009-06923-015) call for studying the reasons why racial- group differences are found and encourage this line of research; however, we view the study of racial-group differences and the study of determinants of those differences as complementary. We thank the authors for contributing these commentaries and for stimulating this discussion. Duckworth (2009) and Kaufman and Agars (2009) discussed important issues regarding expanding the predictor and criterion domains. Wicherts and Millsap (2009) correctly noted distinctions between predictive and measurement bias and used stereotype threat as a mechanism to discuss these issues. Helms (2009) raised several issues regarding the validity and fairness of standardized tests. In all cases, we welcomed the opportunity to discuss these topics and provide more detail on issues relating to high-stakes standardized testing. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Racial/ethnic differences in the criterion-related validity of cognitive ability tests: A qualitative and quantitative review.

The correlation between cognitive ability test scores and performance was separately meta-analyzed for Asian, Black, Hispanic, and White racial/ethnic subgroups. Compared to the average White observed correlation ( = .33, N = 903,779), average correlations were lower for Black samples ( = .24, N = 112,194) and Hispanic samples ( = .30, N = 51,205) and approximately equal for Asian samples ( = .33, N = 80,705). Despite some moderating effects (e.g., type of performance criterion, decade of data collection, job complexity), validity favored White over Black and Hispanic test takers in almost all conditions that included a sizable number of studies. Black–White validity comparisons were possible both across and within the 3 broad domains that use cognitive ability tests for high-stakes selection and placement: civilian employment, educational admissions, and the military. The trend of lower Black validity was repeated in each domain; however, average Black–White validity differences were largest in military studies and smallest in educational and employment studies. Further investigation of the reasons for these validity differences is warranted. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Fairness is not validity or cultural bias in racial-group assessment: A quantitative perspective.

When test scores that differ by racial groups are used for assessment purposes, resulting decisions regarding members of the lower scoring group are potentially unfair. Fairness is defined as the removal from test scores of systematic variance attributable to experiences of racial or cultural socialization, and it is differentiated from test-score validity and cultural bias. Two fairness models for identifying, quantifying, and removing from test scores construct-irrelevant variance attributable to racial or cultural psychological attributes are presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Revival of test bias research in preemployment testing.

We developed a new analytic proof and conducted Monte Carlo simulations to assess the effects of methodological and statistical artifacts on the relative accuracy of intercept- and slope-based test bias assessment. The main simulation design included 3,185,000 unique combinations of a wide range of values for true intercept- and slope-based test bias, total sample size, proportion of minority group sample size to total sample size, predictor (i.e., preemployment test scores) and criterion (i.e., job performance) reliability, predictor range restriction, correlation between predictor scores and the dummy-coded grouping variable (e.g., ethnicity), and mean difference between predictor scores across groups. Results based on 15 billion 925 million individual samples of scores and more than 8 trillion 662 million individual scores raise questions about the established conclusion that test bias in preemployment testing is nonexistent and, if it exists, it only occurs regarding intercept-based differences that favor minority group members. Because of the prominence of test fairness in the popular media, legislation, and litigation, our results point to the need to revive test bias research in preemployment testing. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Modeling aptitude test validity from cognitive components.

Cognitive component analysis of aptitude tests has developed in response to concerns about the nature of the capabilities that are reflected in test scores; as yet cognitive components have not been examined as ability measures in their own right. The current study with test data from 104 undergraduates (i.e., 45 verbal items from the Cognitive Ability Test and the Lorge-Thorndyke Intelligence Test and 35 verbal analogies as aptitude and component scores, and American College Testing Program Examination scores as measures of achievement) examined component validity by using covariance modeling to test hypotheses about the relationships between components, aptitude, and achievement. Support was found for cognitive components to (a) model individual differences in verbal aptitude, (b) decompose test validity (i.e., explain correlations with external measures), and (c) differentially predict achievement. Across-item equivalency of comparable components was not supported, even when content differences were controlled. (18 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Can I retake it? Exploring subgroup differences and criterion-related validity in promotion retesting.

Despite recent interest in the practice of allowing job applicants to retest, surprisingly little is known about how retesting affects 2 of the most critical factors on which staffing procedures are evaluated: subgroup differences and criterion-related validity. We examined these important issues in a sample of internal candidates who completed a job-knowledge test for a within-job promotion. This was a useful context for these questions because we had job-performance data on all candidates (N = 403), regardless of whether they passed or failed the promotion test (i.e., there was no direct range restriction). We found that retest effects varied by subgroup, such that females and younger candidates improved more upon retesting than did males and older candidates. There also was some evidence that Black candidates did not improve as much as did candidates from other racial groups. In addition, among candidates who retested, their retest scores were somewhat better predictors of subsequent job performance than were their initial test scores (rs = .38 vs. .27). The overall results suggest that retesting does not negatively affect criterion-related validity and may even enhance it. Furthermore, retesting may reduce the likelihood of adverse impact against some subgroups (e.g., female candidates) but increase the likelihood of adverse impact against other subgroups (e.g., older candidates). (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

The search for predictors with high validity and low adverse impact: Compatible or incompatible goals?

Many researchers and personnel selection specialists appear to believe that validity must often be sacrificed to reduce adverse impact. This belief may be bolstered by an interpretation of the Uniform Guidelines on Employee Selection Procedures (Equal Employment Opportunity Commission, 1978) that alternative selection methods should be sought in an effort to reduce adverse impact as long as the accompanying reduction in validity is not too large. The authors show that, contrary to popular belief, within the universe of fair tests (as defined by T. A. Cleary, 1968), the most valid selection method will necessarily produce the least adverse impact. Although a less valid selection method can have less adverse impact than the most valid fair method, such an alternative necessarily fails to meet Cleary's fairness criterion. Thus, for fair tests, maximizing validity also minimizes adverse impact. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

When do procedural fairness and outcome fairness interact to influence employees’ work attitudes and behaviors? The moderating effect of uncertainty.

Prior research has shown that procedural fairness interacts with outcome fairness to influence employees’ work attitudes (e.g., organizational commitment) and behaviors (e.g., job performance, organizational citizenship behavior), such that employees’ tendencies to respond more positively to higher procedural fairness are stronger when outcome fairness is relatively low. In the present studies, we posited that people’s uncertainty about their standing as organizational members would have a moderating influence on this interactive relationship between procedural fairness and outcome fairness, in that the interactive relationship was expected to be more pronounced when uncertainty was high. Using different operationalizations of uncertainty of standing (i.e., length of tenure as a proxy, along with self-reports and coworkers’ reports), we found support for this hypothesis in 4 field studies spanning 3 different countries. (PsycINFO Database Record (c) 2010 APA, all rights reserved)