Progress in validity generalization: Comments on Callender and Osburn and further developments.

Examined the simulation study of J. C. Callender and H. G. Osburn (see record 1981-27033-001) aimed at determining the relative accuracy of equations used in validity generalization (transportability) research to estimate the variance of true validities. Certain incorrect statements about equations derived and used by the present authors and their associates (1980, 1981) shown to be predicated on the mistaken impression that their derivational assumptions were the same as those of Callender and Osburn. The following conclusions are made: There is only 1 model for validity generalization, although there are different equations for estimating its parameters. There are now 3 accurate equations available for estimating the variance of true validities: All 3 equations are correctly derived and are of necessity approximations rather than perfect reflections of reality, although for different reasons. Validity generalization is such a robust phenomenon that it is possible based on only corrections for simple sampling error. (11 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bias in validity generalization variance estimates: A reply to Hoben Thomas.

In a recent article H. Thomas (see record 1989-10691-001) derived the expected value of the true validity variance estimate used in validity generalization studies. Based on computations of the expected values for certain scenarios, Thomas made a number of critical assertions regarding the variance estimate. This article shows that Thomas's arguments regarding deficiencies in the variance estimate used in the validity generalization studies are misleading. Contrary to Thomas's extremely negative assessment of the situation, there is no really convincing reason to doubt or abandon the estimates of true validity variance obtained in applied research from the Callender-Osborn and other closely related methods. Rather, there is strong evidence to indicate that populations of true validities with meaningful differences in mean and variance can be reliably distinguished, provided that a sufficient amount of base data are available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A likelihood-based model for validity generalization.

The pair (ri, ni) is an observed correlation coefficient, ri, with sample size ni. Given s independent pairs, each arising from a setting in which the test and criterion measurement are bivariate normal, a mixture model may be used to make the following inferences about the unknown population parameters: (a) The number t of different population correlation coefficients, ρj, j?=?1,…, t, supported by the data, (b) point values of the ρj, (c) the proportion of ri associated with each ρj, and (d) the variance among the ρj all may be estimated from the s pairs. Maximum likelihood estimation equations are given. Approximate confidence intervals and tests may be constructed. Monte Carlo examples and a real-data example are provided. Procedures for correcting artifacts of range restriction and unreliability are briefly discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on validity generalization procedures.

Reviews validity generalization (VG) procedures and finds them to be subject to the logical fallacy of "affirming the consequent." It is argued that alternative models may explain variation in validity coefficients as well as the cross-situational consistency model espoused by many users of the VG approach. Moreover, some of the assumptions that form the statistical foundation of VG work are open to question. It is suggested that Fisher z transformations, which remove most of these problems and usually produce more conservative estimates of the degree to which sampling error may account for variability in correlations, be used. It is also recommended that a more stringent criterion than the 75% decision rule for rejecting the situational specificity hypothesis be used. (64 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

ROC curves and measures of discrimination accuracy: A reply to Swets.

Attempts to clarify differences between J. A. Swets's (see record 1986-13499-001) and the present author's (see record 1984-11109-001) approach to the evaluation of measures of discrimination accuracy. Swets's approach relied on the assumption that discrimination accuracy remains constant across all decision thresholds (i.e., along the relative-operating-characteristic curve [ROC]). The empirical validity of that assumption in psychology is questioned. Because Swets used that assumption while generating theoretical ROCs for various measures, discrepancies between those ROCs and empirical ROCs may be due to a lack of the psychological constancy he assumed. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Children's responses to ambiguous messages: A response to Ackerman's reply.

Responds to B. P. Ackerman's (see record 1987-03694-001) reply to the present authors' (see record 1985-25142-001) test of Ackerman's (see record 1982-05434-001) conclusions concerning the ability of young children to understand that verbal messages can be ambiguous, particularly the idea that children's understanding is masked in standard referential communication tasks because of performative bias. (4 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of a general solution to the problem of validity generalization.

Personnel psychologists have traditionally believed that employment test validities are situation specific. This study presents a Bayesian statistical model that is based on the alternate hypothesis that variation in validity outcomes from study to study for similar jobs and tests is artifactual in nature. Certain outcomes using this model permit validity generalization to new settings without carrying out a validation study of any kind. Where such generalization is not justified, the procedure is considered to provide an improved method of data analysis and decision making for the necessary situational validity study. Application to 4 distributions of empirical validity coefficients is presented to demonstrate the power of the model. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessing the validity of ape-human comparisons: A reply to Boesch (2007).

Boesch (2007) criticizes research comparing ape and human cognition on the basis of both internal and external validity. The authors show here that most of those criticisms are not valid because: (i) most threats to internal validity (e.g., conspecific experimenters for humans but not apes) are controlled for experimentally; (ii) externally, there is no empirical evidence that captive apes have fewer cognitive skills than wild apes and indeed some evidence (especially from human-raised apes) that they have more; and (iii) externally, there is no empirical evidence that Western middle-class children have different cognitive skills from other children at very early ages in basic cognitive domains. Although difficult, with appropriate methodological care, experimental cross-species comparisons may be validly made. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

As for the kings: A reply with regard to depression subtypes and antidepressant response.

The notion that depression subtyping would sharpen the antidepressant treatment advantage over placebo has been around for years. This article questions whether the endogenous–neurotic distinction has proven useful by briefly surveying the evidence on diagnostic reliability and predictability of treatment response. The issue of subtyping is then related to data from the meta-analysis on antidepressant efficacy (see record 1993-06735-001). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An empirical Monte Carlo test of the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization.

An empirical Monte Carlo study with a large data base (N?=?84,808) was conducted to determine the accuracy of the correlation, covariance, and regression slope models for assessing validity generalization (VG). This study has resulted in the following three major conclusions: (1) The correlation between sampling errors and population parameters appears to be close to zero; (2) when the sampling error is the only artifact, all three models do well in estimating the relevant parameters across three sample sizes and 10 different numbers of validity studies per VG study; and (3) when the predictor reliability, criterion reliability, and range restriction are also included as artifacts, the accuracy of estimation depends on how closely the hypothetical and true distributions of artifacts match. All three models performed inadequately when the match between the two sets of distributions was poor. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Refinements in validity generalization methods: Implications for the situational specificity hypothesis.

Using a large database, this study examined 3 refinements of validity generalization procedures: (1) a more accurate procedure for correcting the residual standard deviation (SD) for range restriction to estimate SDp, (2) use of r? instead of study-observed rs in the formula for sampling error variance, and (3) removal of non-Pearson rs. The 1st procedure does not affect the amount of variance accounted for by artifacts. The addition of the 2nd and 3rd procedures increased the mean percentage of validity variance accounted for by artifacts from 70 to 82%, a 17% increase. The cumulative addition of all 3 procedures decreased the mean SDp estimate from .150 to .106, a 29% decrease. Six additional variance-producing artifacts were identified that could not be corrected for. In light of these it was concluded that the obtained estimates of mean SDp and mean validity variance accounted for were consistent with the hypothesis that the true mean SDp value is close to zero. These findings provide further evidence against the situational specificity hypothesis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bootstrap estimates of standard errors in validity generalization.

Bootstrapping is introduced as a method for approximating the standard errors of validity generalization (VG) estimates. A Monte Carlo study was conducted to evaluate the accuracy of bootstrap validity-distribution parameter estimates, bootstrap standard error estimates, and nonparametric bootstrap confidence intervals. In the simulation study the authors manipulated the sample sizes per correlation coefficient, the number of coefficients per VG analysis, and the variance of the distribution of true correlation coefficients. The results indicate that the standard error estimates produced by the bootstrapping procedure were very accurate. It is recommended that the bootstrap standard-error estimates and confidence intervals be used in the interpretation of the results of VG analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analysis of the relation between personality traits and leadership perceptions: An application of validity generalization procedures.

Reexamines, via meta-analysis, the relation between personality traits and leadership perceptions or extent of leader emergence, arguing that prior research on trait theories and leadership has been misinterpreted as applying to a leader's effect on performance when it actually pertains to the relation of leadership traits to leadership emergence. Further, based on current theories of social perceptions, several traits were expected to be strongly related to leadership perceptions. The meta-analytic technique of validity generalization was used with the 15 articles identified by R. D. Mann (see record 1960-04194-001) as investigating the relationship between personality traits and leadership. These studies were then pooled with 9 subsequent studies in an additional set of meta-analyses. Results support the expectation in that intelligence, masculinity–femininity, and dominance were significantly related to leadership perceptions. Findings show that variability across studies in the relation of these traits to leadership perceptions could be explained largely by methodological factors, indicating that contingency theories of leadership perceptions may not be needed. Both of these results contrast with the conclusions of earlier nonquantitative literature reviews on traits and leadership perceptions and with conventional thinking in the leadership area. (62 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Another view of progress in validity generalization: Reply to Schmidt, Hunter, and Pearlman.

Refutes the F. L. Schmidt et al (see record 1983-07150-001) argument that the present authors' (see record 1981-27033-001) work on validity generalization models contained erroneous conclusions. Modifications to past practices in computing and reporting variance estimates are recommended. (3 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the sampling variance of the mean uncorrected correlation in meta-analysis and validity generalization.

Compares the accuracy of several formulas for the standard error of the mean uncorrected correlation in meta-analytic and validity generalization studies. The effect of computing the mean correlation by weighting the correlation in each study by its sample size is also studied. On the basis of formal analysis and simulation studies, it is concluded that the common formula for the sampling variance of the mean correlation, Vr ?=?Vr/K where K is the number of studies in the meta-analysis, gives reasonably accurate results. This formula gives accurate results even when sample sizes and ρs are unequal and regardless of whether or not the statistical artifacts vary from study to study. It is also shown that using sample-size weighting may result in underestimation of the standard error of the mean uncorrected correlation when there are outlier sample sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Statistical power of tests of the situational specificity hypothesis in validity generalization studies: A cautionary note.

Evaluated the statistical power of the Callender-Osburn method for testing the situational specificity hypothesis in validity generalization studies. The Schmidt-Hunter 75% rule for testing the situational specificity hypothesis was also studied with regard to its sensitivity for detecting both Type I and Type II errors. Results show that both the Callender-Osburn procedure and Schmidt-Hunter 75% rule lacked sufficient statistical power to detect low-to-moderate true validity variance when sample size was below 100. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The role of facial response in the experience of emotion: A reply to Tourangeau and Ellsworth.

Replies to R. Tourangeau and P. C. Ellsworth (see record 1981-00499-001), who tested a hypothesis about the role of voluntarily innervated facial responses in the experience of emotion and disconfirmed that hypothesis. The present author's theory would also have predicted that their hypothesis would be disconfirmed. The value of the technique of voluntary simulation of facial responses for the study of innate affects is seriously questioned. (5 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the validity of inferences about personality constructs: A response to Friedman and Spence and Helmreich.

Argues that J. T. Spence and R. L. Helmreich (see record 1984-00243-001) and H. S. Friedman (see record 1984-00168-001) incorrectly attribute to the present authors the suggestion that judgments of the face validity of scales should replace the process of construct validation. The present authors advocate inspection of scale content to see if it is possible that associations between scales reflect, to a greater or lesser degree, overlapping content. If this is possible, researchers can face a dilemma that prevents them from using their scales to test their hypotheses. (6 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distorting the point: A reply to Wolpe.

Replies to J. Wolpe's (1982) critique of the present author's (1971) review of 100 cases that yielded a 36% relapse rate after behavior therapy. It is maintained that Wolpe misleads the reader to believe that this was a reference to "multimodal therapy" when in fact this was developed later by the present author (1983). The 36% relapse rate was found in those individuals who were treated by traditional behavior therapy. Also questioned is Wolpe's reference to a follow-up rate of 3% or less as characteristic of behavior therapy. (6 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)