Incremental validity of situational judgment tests.

Using 3 different samples, the authors assessed the incremental validity of situational judgment inventories (SJIs), relative to job knowledge, cognitive ability, job experience, and conscientiousness, in the prediction of job performance. The SJI was a valid predictor in all 3 samples and incrementally so in 2 samples. Relative to the other predictors, SJI's partial correlation with performance, controlling for the other 4 predictors, was superior in most comparisons. Subgroup differences on the SJI also appear to be less than those for cognitive ability and job knowledge, but greater than differences in conscientiousness. The SJI should prove to be a valuable additional measure in the prediction of job performance, but several additional areas of research are suggested. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Video-based versus written situational judgment tests: A comparison in terms of predictive validity.

In this study, the authors examined whether video-based situational judgment tests (SJTs) have higher predictive validity than written SJTs (keeping verbal content constant). The samples consisted of 1,159 students who completed a video-based version of an SJT and 1,750 students who completed the same SJT in a written format. The study was conducted in a high stakes testing context. The video-based version of an interpersonally oriented SJT had a lower correlation with cognitive ability than did the written version. It also had higher predictive and incremental validity for predicting interpersonally oriented criteria than did the written version. In this high stakes context, applicants also reacted relatively favorably to the SJTs, although there was no significant difference in face validity between the formats. These findings suggest that SJT format changes be made with caution and that validation evidence is needed when changes are proposed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Toward an understanding of situational judgment item validity and group differences.

This paper evaluates 2 adjustments to common scoring approaches for situational judgment tests (SJTs). These adjustments can result in substantial improvements to item validity, reductions in mean racial differences, and resistance to coaching designed to improve scores. The first adjustment, applicable to SJTs that use Likert scales, controls for elevation and scatter (Cronbach & Gleser, 1953). This adjustment improves item validity. Also, because there is a White–Black mean difference in the preference for extreme responses on Likert scales (Bachman & O'Malley, 1984), these adjustments substantially reduce White–Black mean score differences. Furthermore, this adjustment often eliminates the score elevation associated with the coaching strategy of avoiding extreme responses (Cullen, Sackett, & Lievens, 2006). Item validity is shown to have a U-shaped relationship with item means. This holds both for SJTs with Likert score response formats and for SJTs where respondents identify the best and worst response option. Given the U-shaped relationship, the second adjustment is to drop items with midrange item means. This permits the SJT to be shortened, sometimes dramatically, without necessarily harming validity. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Job satisfaction, job performance, and situational characteristics.

How are job satisfaction and performance related to the variables of the situation? S were approximately 2500 workers in 72 warehouses of 1 company. Job satisfaction was measured by questionnaire items. Factor analysis was used to relate the variables of quantity, quality, profitability, turnover, size of workforce, city size, wage rate and unionization and percentage of males. Job satisfaction was highest with the greatest productivity and profit. Small town culture had the most satisfaction and performance. From Psyc Abstracts 36:01:3LH65K. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Differentiating specific job knowledge from implicit trait policies in procedural knowledge measured by a situational judgment test.

This study compared validities of situational judgment test (SJT) scoring keys that were presumed to be differentially saturated with specific knowledge about effective job behavior and general knowledge about effective trait expression. The keys were based on subject matter experts’ effectiveness judgments, undergraduates’ effectiveness judgments, and graduate students’ trait judgments. We used data reported earlier by Motowidlo, Dunnette, and Carter (1990) with managerial incumbents in telecommunication companies. All keys yielded valid relations with supervisory performance ratings. The key based on subject matter experts’ judgments, however, explained criterion variance beyond the variance explained by the other keys. These results suggest that specific knowledge about effective job behavior and general knowledge about effective trait expression (i.e., implicit trait policies) contribute independently to variance in job performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection.

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

The effects of response instructions on situational judgment test performance and validity in a high-stakes context.

This study fills a key gap in research on response instructions in situational judgment tests (SJTs). The authors examined whether the assumptions behind the differential effects of knowledge and behavioral tendency SJT response instructions hold in a large-scale high-stakes selection context (i.e., admission to medical college). Candidates (N = 2,184) were randomly assigned to a knowledge or behavioral tendency response instruction SJT, while SJT content was kept constant. Contrary to prior research in low-stakes settings, no meaningfully important differences were found between mean scores for the response instruction sets. Consistent with prior research, the SJT with knowledge instructions correlated more highly with cognitive ability than did the SJT with behavioral tendency instructions. Finally, no difference was found between the criterion-related validity of the SJTs under the two response instruction sets. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The definition and measurement of job involvement.

The purpose of the present research was to define job involvement, develop a scale for measuring it, gather evidence on the reliability and validity of the scale, and to learn something about the nature of job involvement through its correlation with other job attitudes. This paper describes the development and validation of a scale measuring job involvement, the resulting scales, the relation between job involvement, and other job attitudes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Team Role Test: Development and validation of a team role knowledge situational judgment test.

The main objectives in this research were to introduce the concept of team role knowledge and to investigate its potential usefulness for team member selection. In Study 1, the authors developed a situational judgment test, called the Team Role Test, to measure knowledge of 10 roles relevant to the team context. The criterion-related validity of this measure was examined in 2 additional studies. In a sample of academic project teams (N = 93), team role knowledge predicted team member role performance (r = .34). Role knowledge also provided incremental validity beyond mental ability and the Big Five personality factors in the prediction of role performance. The results of Study 2 revealed that the predictive validity of role knowledge generalizes to team members in a work setting (N = 82, r = .30). The implications of the results for selection in team environments are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity and fairness of physical ability tests for predicting performance in craft jobs: Correction to Reilly, Zedeck, and Tenopyr.

Reports an error in the original article by Richard R. Reilly, Sheldon Zedeck, and Mary L. Tenopyr (Journal of Applied Psychology, 1979, Vol. 64, No. 3, pp. 262-274). In the Results section of Experiment 2 several results were reported incorrectly. The corrected results are provided. (The following abstract of this article originally appeared in record 1980-26872-001.) Problems relating to performance, accidents, and turnover in outdoor telephone craft jobs stimulated 2 experiments aimed at developing and validating a physical test battery. Based on job analysis results, a battery of 9 measures was administered to a sample of 128 Ss (83 males and 45 females) in Exp I. A 2-test battery (dynamic arm strength and reaction time), valid for predicting job task performance and turnover, was selected. Regression equations for males and females were not significantly different. Exp II included a sample of 210 Ss (132 males and 78 females). A 3-test battery consisting of a body density measure, a balance test, and a static strength test was selected based on relationships with training performance. No significant differences were found in the regression equations for males compared to females. The Exp II battery was also significantly related to field performance, training completion, and accidents and was valid for the Exp I criteria. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Psychomotor abilities tests as predictors of training performance.

Examined the predictive and incremental validity of three psychomotor ability measures (manual dexterity, finger dexterity, and motor coordination) in 209 Canadian Forces personnel being trained in technical and mechanical occupations. For both types of occupations combined, manual dexterity predicted training performance (r=.18), as it did for the Technical (r=.22) and Mechanical (r=.17) families, separately Finger Dexterity (r=.02) and Motor Coordination (r=.05) did not predict training performance for either the combined group or for each family by itself. The addition of the three psychomotor measures increased validity beyond what was predicted by cognitive ability in the combined occupations (δ R2=.05); however, only Manual Dexterity (β=.26) made a significant contribution to the regression model. Similar, though nonsignificant, increases in predictive validity occurred within each family. The results from this study suggest that specific abilities, when determined through a job analysis, improve predictions based solely on cognitive ability. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Use of the General Aptitude Test Battery to determine aptitude changes with age and to predict job performance.

"It was found that Aptitudes G, V, N, and S were related to age in a curvilinear manner." 400 Ss were divided into 4 age groups. Aptitudes and supervisory ratings were plotted against age. "… only Aptitude K contributed significantly to predicting the variance in the criterion." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits.

Researchers of broad and narrow traits have debated whether narrow traits are important to consider in the prediction of job performance. Because personality-performance relationship meta-analyses have focused almost exclusively on the Big Five, the predictive power of narrow traits has not been adequately examined. In this study, the authors address this question by meta-analytically examining the degree to which the narrow traits of conscientiousness predict above and beyond global conscientiousness. Results suggest that narrow traits do incrementally predict performance above and beyond global conscientiousness, yet the degree to which they contribute depends on the particular performance criterion and occupation in question. Overall, the results of this study suggest that there are benefits to considering the narrow traits of conscientiousness in the prediction of performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Traditional tests and job simulations: Minority and majority performance and test validities.

Performance of 565 minority and 414 majority job applicants on traditional paper-and-pencil tests and on a telephone simulation that measured similar constructs was assessed. Models of the measurement characteristics of these 2 test batteries indicated larger subgroup mean differences on the traditional tests than on the simulation. Correlations between traditional tests were lower in the majority sample than in the minority sample, and the variance of the minority candidates' scores on the traditional tests was much larger than the variance of majority applicants' scores on the same measures. The validity of the simulation was lower than the validity of the traditional tests. This study replicates previous laboratory research that has indicated smaller subgroup differences on simulations than on paper-and-pencil tests and extends this research by providing evidence of the relative validity of these 2 types of measures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Job performance over time is a function of latent trajectories and previous performance.

Previous literature addressing job performance over time notes that past performance can affect future performance and that individuals often have distinct latent performance trajectories. However, no research to date has modeled these 2 aspects of job performance in tandem. Drawing on previous literature, the authors note that current performance may act as performance feedback, influencing future performance directly (i.e., autoregression), and that individuals differ in their performance trajectories due to individual-difference factors (i.e., latent trajectories). The authors demonstrate an autoregressive latent trajectory (ALT) model to show how both autoregressive and latent trajectory parameters may be incorporated in modeling job performance over time. Also discussed are the implications of the ALT model for future studies examining job performance longitudinally. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Does stereotype threat affect test performance of minorities and women? A meta-analysis of experimental evidence.

A meta-analysis of stereotype threat effects was conducted and an overall mean effect size of |.26| was found, but true moderator effects existed. A series of hierarchical moderator analyses evidenced differential effects of race- versus gender-based stereotypes. Women experienced smaller performance decrements than did minorities when tests were difficult: mean ds = |.36| and |.43|, respectively. For women, subtle threat-activating cues produced the largest effect, followed by blatant and moderately explicit cues: ds = |.24|, |.18|, and |.17|, respectively; explicit threat-removal strategies were more effective in reducing stereotype threat effects than subtle ones: ds = |.14| and |.33|, respectively. For minorities, moderately explicit stereotype threat-activating cues produced the largest effect, followed by blatant and subtle cues: ds = |.64|, |.41|, and |.22|, respectively; explicit removal strategies enhanced stereotype threat effects compared with subtle strategies: ds = |.80| and |.34|, respectively. In addition, stereotype threat affected moderately math-identified women more severely than highly math-identified women: ds = |.52| and |.29|, respectively; low math-identified women suffered the least from stereotype threat: d= |.11|. Theoretical and practical implications of these findings are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The contribution of interview and situational performance procedures to the selection of supervisory personnel.

2 groups of supervisors (each N = 40), selected to be as similar as possible on a set of control predictor variables "known to be predictive of supervisory performance" (basic abilities test score, supervisory practices score, age, service, job level) and to be maximally different on criterion scores of supervisory effectiveness ratings, were used to try out 5 experimental supervisory performance predictors: panel interview, individual interview, group discussion problem, role playing situation, and small-job management. Each group was divided into a high and low subgroup on the basis of the control predictor variables. The predictor battery validity increased from an average of .27 for the control predictors alone to .30-.34 when various of the experimental predictors were added. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative importance of task, citizenship, and counterproductive performance to global ratings of job performance: A policy-capturing approach.

A review of research on job performance suggests 3 broad components: task, citizenship, and counterproductive performance. This study examined the relative importance of each component to ratings of overall performance by using an experimental policy-capturing design. Managers in 5 jobs read hypothetical profiles describing employees' task, citizenship, and counterproductive performance and provided global ratings of performance. Within-subjects regression analyses indicated that the weights given to the 3 performance components varied across raters. Hierarchical cluster analyses indicated that raters' policies could be grouped into 3 homogeneous clusters: (a) task performance weighted highest, (b) counterproductive performance weighted highest, and (c) equal and large weights given to task and counterproductive performance. Hierarchical linear modeling indicated that demographic variables were not related to raters' weights. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Racial/ethnic differences in the criterion-related validity of cognitive ability tests: A qualitative and quantitative review.

The correlation between cognitive ability test scores and performance was separately meta-analyzed for Asian, Black, Hispanic, and White racial/ethnic subgroups. Compared to the average White observed correlation ( = .33, N = 903,779), average correlations were lower for Black samples ( = .24, N = 112,194) and Hispanic samples ( = .30, N = 51,205) and approximately equal for Asian samples ( = .33, N = 80,705). Despite some moderating effects (e.g., type of performance criterion, decade of data collection, job complexity), validity favored White over Black and Hispanic test takers in almost all conditions that included a sizable number of studies. Black–White validity comparisons were possible both across and within the 3 broad domains that use cognitive ability tests for high-stakes selection and placement: civilian employment, educational admissions, and the military. The trend of lower Black validity was repeated in each domain; however, average Black–White validity differences were largest in military studies and smallest in educational and employment studies. Further investigation of the reasons for these validity differences is warranted. (PsycINFO Database Record (c) 2011 APA, all rights reserved)