Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales.

A procedure was tested for the construction of evaluative rating scales anchored by examples of expected behavior. Expectations, based on having observed similar behavior, were used to permit rating in a variety of situations without sacrifice of specificity. Examples, submitted by head nurses as illustrations of nurses' behavior related to a given dimension were retained only if reallocated to that dimension by other head nurses, and then scaled as to desirability. Agreement for a number of examples was high, and scale reliabilities ranged above .97. Similar content validity should be obtained in other rating situations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reliability of short rating scales and the heterogeneity of the rated stimuli.

"Three lists of 10 food stimuli were selected so that the lists varied in the heterogeneity of the stimuli. Preference ratings were collected from 278 Ss using rating scales with 2, 3, 4, or 5 categories. Rating reliability was highest with the most heterogeneous list and with the 4-category scale and was lowest with the most homogeneous list and the 2-category scale. Rater bias results were more tentative, with the list of intermediate stimuli heterogeneity and the 4-category scale most subject to systematic rater error on the part of the Ss." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multitrait-multisource confirmatory factor analytic approach to the construct validity of ADHD rating scales.

Confirmatory factor analysis was used to model a multitrait-multisource design to evaluate the construct validity of attention-deficit/hyperactivity disorder (ADHD) rating scales. The 2 trait factors were the ADHD inattention and hyperactivity/impulsivity dimensions. The 2 source factors were parents and teachers. In Study 1, parents and teachers rated 1,475 Australian elementary school children on the ADHD symptoms. In Study 2, parents and teachers rated 285 Brazilian elementary school children on the ADHD symptoms. Similar results occurred in both studies with most of the ADHD symptoms containing more source than trait variance, thus providing weak evidence for the convergent and discriminant validity of the symptoms as measured by rating scales. The study outlines the implications of such strong source effects for understanding ADHD. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance.

Although evidence supports the unique contribution of task performance and contextual performance to overall evaluations, little is known about the relative contribution that specific dimensions of contextual performance make to overall performance judgments. This study evaluated the extent to which supervisors consider task and contextual performance by using relative weights (J. W. Johnson, 2000) to statistically describe the relative importance of specific dimensions of each type of performance to overall performance ratings. Within each of 8 job families in a large organization, each of 4 dimensions of contextual performance made not only a unique contribution but a relatively important contribution to the overall evaluation. Evidence also supports the adaptive performance dimension of handling work stress as an aspect of contextual performance and job–task conscientiousness as an aspect of both task and contextual performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Developing a measure of sluggish cognitive tempo for children: Content validity, factor structure, and reliability.

Sluggish cognitive tempo (SCT) is a construct that some researchers believe may be extremely useful in understanding the inattentive subtype of attention-deficit/hyperactivity disorder, and may even help define a completely new disorder. However, the construct of SCT is as yet inadequately operationally or theoretically defined. The authors took the first steps toward developing an empirically supported measure of SCT in children. In Study 1, potential items to measure SCT were identified from a literature review, content validity of the items was evaluated by a group of experts, and a preliminary set of SCT items were selected. In Study 2, ratings completed by parents and teachers of 335 children (ages 4–13) were used to further develop and evaluate the SCT items by computing factor analyses, item-level analyses, reliability analyses, and preliminary validity analyses. The final SCT scale (14 items) produced a total scale score and 3 subscale scores: Slow, Sleepy, and Daydreamer. These scales were constructed with good content validity and were found to have strong reliability. Future directions include replication, extension into a clinical population, and further examination of validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The validity and incremental validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations for predicting job performance in advanced-level high-stakes selection.

In high-stakes selection among candidates with considerable domain-specific knowledge and experience, investigations of whether high-fidelity simulations (assessment centers; ACs) have incremental validity over low-fidelity simulations (situational judgment tests; SJTs) are lacking. Therefore, this article integrates research on the validity of knowledge tests, low-fidelity simulations, and high-fidelity simulations in advanced-level high-stakes settings. A model and hypotheses of how these 3 predictors work in combination to predict job performance were developed. In a sample of 196 applicants, all 3 predictors were significantly related to job performance. Both the SJT and the AC had incremental validity over the knowledge test. Moreover, the AC had incremental validity over the SJT. Model tests showed that the SJT fully mediated the effects of declarative knowledge on job performance, whereas the AC partially mediated the effects of the SJT. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

A meta-analytic investigation of conscientiousness in the prediction of job performance: Examining the intercorrelations and the incremental validity of narrow traits.

Researchers of broad and narrow traits have debated whether narrow traits are important to consider in the prediction of job performance. Because personality-performance relationship meta-analyses have focused almost exclusively on the Big Five, the predictive power of narrow traits has not been adequately examined. In this study, the authors address this question by meta-analytically examining the degree to which the narrow traits of conscientiousness predict above and beyond global conscientiousness. Results suggest that narrow traits do incrementally predict performance above and beyond global conscientiousness, yet the degree to which they contribute depends on the particular performance criterion and occupation in question. Overall, the results of this study suggest that there are benefits to considering the narrow traits of conscientiousness in the prediction of performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Impact of Job Complexity and Performance Measurement on the Temporal Consistency, Stability, and Test-Retest Reliability of Employee Job Performance Ratings.

Although research has shown that individual job performance changes over time, the extent of such changes is unknown. In this article, the authors define and distinguish between the concepts of temporal consistency, stability, and test-retest reliability when considering individual job performance ratings over time. Furthermore, the authors examine measurement type (i.e., subjective and objective measures) and job complexity in relation to temporal consistency, stability, and test-retest reliability. On the basis of meta-analytic results, the authors found that the test-retest reliability of these ratings ranged from .83 for subjective measures in low-complexity jobs to .50 for objective measures in high-complexity jobs. The stability of these ratings over a 1-year time lag ranged from .85 to .67. The analyses also reveal that correlations between performance measures decreased as the time interval between performance measurements increased, but the estimates approached values greater than zero. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development, reliability, and validity of the Measure of Sexual Identity Exploration and Commitment (MOSIEC).

Four studies were conducted on the development and validation of the Measure of Sexual Identity Exploration and Commitment (MoSIEC). Exploratory factor analysis of an initial item pool yielded a 22-item measure with 4 distinct factors assessing commitment, exploration, sexual orientation identity uncertainty, and synthesis/integration. Exploratory factor analysis findings support the argument that sexual identity is a broad, multidimensional construct and that the MoSIEC assesses the construct of sexual identity in a manner consistent with J. E. Marcia's (1966) model of identity development. Confirmatory factor analyses demonstrated the stability of the MoSIEC factor structure, providing evidence of construct validity. Test-retest stability, internal consistency, and validity coefficients supported the use and continued development of the MoSIEC. Significant differences in levels of exploration and sexual orientation identity uncertainty were found among different sexual orientation identity groups, establishing the criterion-related validity of the MoSIEC. Implications for further development and use of the MoSIEC are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Suicide Attempt Self-Injury Interview (SASII): Development, reliability, and validity of a scale to assess suicide attempts and intentional self-injury.

The authors describe the development of the Suicide Attempt Self-Injury Interview (SASII), an instrument designed to assess the factors involved in nonfatal suicide attempts and intentional self-injury. Using 4 cohorts of participants, authors generated SASII items and evaluated them with factor and content analyses and internal consistency statistics. The final measure was assessed for reliability and validity with collateral measures. The SASII assesses variables related to method, lethality and impulsivity of the act, likelihood of rescue, suicide intent or ambivalence and other motivations, consequences, and habitual self-injury. The SASII was found to have very good interrater reliability and adequate validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Beyond alpha: An empirical examination of the effects of different sources of measurement error on reliability estimates for measures of individual-differences constructs.

On the basis of an empirical study of measures of constructs from the cognitive domain, the personality domain, and the domain of affective traits, the authors of this study examine the implications of transient measurement error for the measurement of frequently studied individual-differences variables. The authors clarify relevant reliability concepts as they relate to transient error and present a procedure for estimating the coefficient of equivalence and stability (L. J. Cronbach, 1947), the only classical reliability coefficient that assesses a 3 major sources of measurement error (random response, transient, and specific factor errors). The authors conclude that transient error exists in all 3 trait domains and is especially large in the domain of affective traits. Their findings indicate that the nearly universal use of the coefficient of equivalence (Cronbach's alpha; L. J. Cronbach, 1951), which fails to assess transient error, leads to overestimates of reliability and undercorrections for biases due to measurement error. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Severity Indices of Personality Problems (SIPP-118): Development, factor structure, reliability, and validity.

This article describes a series of studies involving 2,730 participants on the development and validity testing of the Severity Indices of Personality Problems (SIPP), a self-report questionnaire covering important core components of (mal)adaptive personality functioning. Results show that the 16 facets constituted homogeneous item clusters (i.e., unidimensional and internally consistent parcels) that fit well into 5 clinically interpretable, higher order domains: self-control, identity integration, relational capacities, social concordance, and responsibility. These domains appeared to have good concurrent validity across various populations, good convergent validity in terms of associations with interview ratings of the severity of personality pathology, and good discriminant validity in terms of associations with trait-based personality disorder dimensions. Furthermore, results suggest that the domain scores are stable over a time interval of 14-21 days in a student sample but are sensitive to change over a 2-year follow-up interval in a treated patient population. Taken together, the final instrument, the SIPP-118, provides a set of 5 reliable, valid, and efficient indices of the core components of (mal)adaptive personality functioning. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Predictors of work performance among higher education employees: An examination using the ASSET Model of Stress.

This study used university-based statistics of performance and self-rated employee productivity to examine the relationship between stress levels, organizational commitment, health, and performance. The authors conducted a secondary analysis of data from staff in 13 higher education institutions. In common with earlier research, the authors found that stressors had a negative linear relationship with all the performance measures used. However, this relationship was also influenced by physical health, psychological well-being, and organizational commitment, and by the measure of performance used. In addition, the authors found variations in the relationship between performance and stress by category of staff, which suggests the influence of job factors. These findings are discussed in relation to previous research and their implications for English academic institutions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

New subscales for an anchored version of the Brief Psychiatric Rating Scale: Construction, reliability, and validity in acute psychiatric admissions.

Attending psychiatrists completed an anchored version of the 18-item Brief Psychiatric Rating Scale (BPRS-A) based on admission and evaluation information on a total of 2,921 adult patients treated at 1 public sector acute psychiatric teaching hospital. Exploratory factor analysis was applied to a 6-month sample to construct 4 nonoverlapping subscales: Resistance, Positive Symptoms, Negative Symptoms, and Psychological Discomfort. Confirmatory factor analysis compared these new subscales to 3 other published subscale models using a second 6-month sample. Internal consistency, rater influence, and interrater agreement were estimated in separate studies. Discriminant validity was explored by comparison of diagnosis-based samples. Application of the BPRS-A as a debriefing instrument in the study of symptomatic change and the multiple challenges inherent in psychometric study of such a rating scale in realistic hospital practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Teachers' perceptions of students' reading abilities: An examination of the relationship between teachers' judgments and students' performance across a continuum of rating methods.

Teacher perceptions about students' academic abilities are important for several reasons (e.g., instructional decision making, special education entitlement decisions). Not surprisingly, researchers have investigated the accuracy of teachers' decisions. Although some data reveal that teachers are relatively good judges of academic performance, other findings have suggested otherwise. A likely explanation for conflicting findings is the varying assessment methods (e.g., direct vs. indirect, norm-referenced vs. peer-independent) and different data analysis procedures that have been used across studies. The purpose of this study was to investigate a continuum of teacher-perception assessment methods as they corresponded to students' oral reading fluency performance. Participants included 10 teachers and 87 first, second, and third grade students from a suburban school in the northeast. Overall results suggested that teachers were generally accurate when estimating students' performance when students had strong oral reading fluency skills, but teachers had more difficulty judging students with average to low oral reading fluency. Further, data interpretation of teachers' judgment accuracy differed somewhat depending upon the statistical method employed. Implications, limitations, and suggestions for future research related to this study are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

An anxiety scale for the Strong Vocational Interest Inventory: Development, cross-validation, and subsequent tests of validity.

A level of adjustment dimension utilizing a combination of Taylor MAS and Winne Scale of Neuroticism scales has been developed for the Strong VIB. 400 graduate psychology students served as Ss while a second validation used 200 male freshmen. For the 33 items split-half reliability on the first cross-validation was .73. Criterion cross-validations yielded corrected correlations of .44 and .51. 27 references. (PsycINFO Database Record (c) 2010 APA, all rights reserved)