Relationship between observational accuracy and accuracy in evaluating performance.

Examined relationships among components of accuracy in tasks of performance rating and observational accuracy. Measures of elevation, differential elevation, stereotype accuracy, and differential accuracy were obtained in a teacher rating task. 44 undergraduates viewed videotapes of 4 graduate student lecturers and rated each using the Frequency Rating scale to measure accuracy in behavioral observation, as well as standard teacher evaluation forms (the Performance Evaluation scale). Measures of elevation, differential elevation, and differential accuracy in rating the frequency of behaviors previously defined as critical incidents were related to corresponding measures of accuracy in evaluating performance. Data suggest that accuracy in observing or evaluating performance is not a generalized ability, but different types of accuracy are independent. Frequency ratings are tied to specific behaviors chosen on the basis of clarity and relevance, while performance evaluation requires complex, abstract judgments about performance quality. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings, judgment decomposition, and their impact on interrater agreement and rating accuracy.

One strategy suggested for improving the accuracy of the complex evaluative judgments involved in performance evaluation is to decompose them into a series of simpler judgments. Another is to collect observations in a distributional rating scheme in which raters estimate the frequencies of different classes of behavior and performance is assessed in terms of the relative frequencies of effective and ineffective behaviors. Distributional ratings were compared to Likert-type ratings of videotaped lectures at 3 levels of dimensional decomposition; ratings were evaluated in terms of interrater agreement and rating accuracy. Decomposition led to increased agreement and accuracy, but the use of distributional ratings did not. The practical implications of the results are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Role of Children's Ethnicity in the Relationship Between Teacher Ratings of Attention-Deficit/Hyperactivity Disorder and Observed Classroom Behavior.

Significant ethnic differences have been consistently documented on attention-deficit/hyperactivity disorder (ADHD) teacher rating scales. Whether these ethnic differences result from a teacher rating bias or reflect actual classroom behavior patterns is unknown. Ethnic differences between Caucasian and African American (AA) elementary schoolchildren on teacher ratings and codings of observed classroom behavior were examined with latent variables. In structural equation models, correlations between teacher ratings and observed classroom behavior suggested nonbiased teacher ratings of AA schoolchildren with diagnosed ADHD. Ethnic differences were documented for both teacher ratings of ADHD and classroom behavior. Differences in classroom behavior were attenuated when the behavior of an average child in the classroom was taken into account. Multiple explanations for this pattern of results are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating format research revisited: Format effectiveness and acceptability depend on rater characteristics.

Rating format research has largely been ignored since F. J. Landy and J. L. Farr's (see record 1980-08528-001) call for a moratorium over a decade ago. Their conclusion that ratings were not affected by changes in scale format was based on research that treated all raters alike. However, individuals differ in the way in which they perceive and integrate information. This article investigates the proposition that differences in rating accuracy associated with different rating formats are contingent on rater characteristics. The study tested the rating accuracy and affective reactions toward performance appraisal of field-dependent (FD) and field-independent (FIN) raters on 4 different performance measures. As hypothesized, FINs were more accurate raters than FDs only when scale formats were holistic, and only FDs' ratings were significantly affected by the level of structure in the scale format. FIN raters were also more confident in their ratings and less frustrated and confused with the rating task than were FDs. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Attitudes, self-monitoring, and appraisal behaviors.

This study examined the influence of attitudes and self-monitoring on leniency (elevation accuracy) of performance ratings and personnel decisions. In addition, moderating effects of self-monitoring on the relationship between attitudes and accuracy of ratings and decisions were investigated. Attitudes and self-monitoring tendency of 210 managers–professionals were measured, and ratings provided and decisions made by them were used to test 3 sets of hypotheses. Moderated regression and follow-up split-group analyses indicated that self-monitoring moderated the relationship between attitudes toward accurate appraisal and rating accuracy. Self-monitoring significantly influenced rating and decision accuracy such that accuracy declined with increasing level of self-monitoring. Results highlight the influence of rater's personality on appraisal behaviors. Implications of results and directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Retrieval processes underlying confidence in comprehension judgments.

People's ratings of confidence in ability to answer questions about previously studied expository passages are virtually uncorrelated with test performance. This correlation would be attenuated if confidence ratings were influenced by momentary accessibility (e.g., latency to initial access of probed information) and if test performance were influenced by other aspects of retrieval (e.g., total recall over an interval). To see if momentary accessibility influences confidence ratings, Experiment 1 manipulated accessibility by using self- versus other-generated title probes. Both confidence and accessibility measures varied with generation condition. Experiment 2 provided three findings. First it assessed whether accessibility is the only common factor underlying confidence and domain familiarity ratings by observing whether confidence and familiarity covary wth accessibility partialed out. Marginal covariation implied either some common factor other than accessibility, imperfect measurement of accessibility, or both. Second, negative correlations of rating latencies with rating magnitudes implied that access latency, momentary access rate, or both, affect the ratings. Third, calibration of comprehension was poor, but nonzero. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the Independence of Ratings of Hyperactivity, Conduct Problems, and Attention Deficits in Children: A Multiple Regression Analysis: Correction to Lahey, Green, and Forehand.

Reports an error in the original article by Benjamin B. Lahey, Kenneth D. Green, and Rex Forehand (Journal of Consulting and Clinical Psychology, 1980, Vol. 48, No. 5, pp. 566-574). It was incorrectly stated on page 571 that Loney, Langhorne, and Paternite derived factor scores from the product of all factor loadings and teacher ratings. Instead, they used normalized z scores derived from ratings of only those items that loaded most strongly on each factor. Differences in results between the two studies, therefore, are probably not due to differences in the methods of calculating factor scores. (The following abstract of this article originally appeared in record 1980-33426-001.) Teacher ratings, direct observational measures, and peer evaluations were obtained on 109 3rd-grade children. Factor scores on the Conners Teacher Rating Scale were correlated with the following measures: observation of time on task; frequency of teacher interaction; positive and negative peer interactions; peer ratings of acceptance, rejection, and dislike; and 3 measures of academic performance. Arguments are made for combining hyperactivity and conduct problem scales in rating child behavior problems. A factor resembling the attention deficit disorder without hyperactivity (inattentive-passive) was found to account for a substantial portion of independent variance in the multiple regression analyses and should, therefore, be considered as a separate rating scale factor. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Initial decisions and subsequent performance ratings.

Examined how information initially encountered for one decision was later used for evaluating ratee performance. 101 undergraduates viewed videotaped performances of 4 male carpenters performing 4 tasks. Half of the Ss were instructed to view the performance with the intent of rating the deservedness of each worker for outside contracting work; the other half were asked to designate the one best worker for this work. Two days later, 70 of the 101 Ss were asked to rate (from memory) each worker's performance. Findings indicate that raters initially designating one worker for a positive outcome rated all workers higher than raters making initial deservedness ratings. This elevation in ratings, which occurred for both subsequent overall evaluations and task ratings, may have reflected both the tendency to inflate ratings given the individual receiving the initial treatment and leniency toward the other ratees for whom representations of ability may not have been well established or remembered. Overall delayed performance ratings were influenced by these initial contextual factors. Results suggest that the pattern in which information is presented to raters and the nature of previous decisions may affect memory-based performance ratings. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The necessity of a teacher background for school psychologists.

Studied principals' and teachers' satisfaction with the services of their respective school psychologists via a statewide survey of 160 elementary and 90 junior high schools. The survey's purpose was to determine if teacher certification and teaching experience contributed to significantly higher ratings in a number of designated skill areas. With 1 exception, no significant differences were found in teacher and principal ratings of school psychologists with and without teacher certification. When 4 school psychologist groups with varying amounts of teaching experience were compared, principal ratings of the groups produced no significant differences. Teacher ratings of the groups, however, produced significant differences in 3 of 5 skill areas. In all cases, the difference favored fewer years of teaching experience. Findings support the view that prolonged teaching experience negatively affects performance ratings and that teaching experience per se does not significantly improve performance ratings. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct validity of multiple measures in teaching, research, and service and reliability of peer ratings.

The construct validity measures in teaching, research, and service were determined using a multitrait–multimethod matrix. The measures in research (research publications, research grants, research awards, and peer ratings in research) and three of the measures in teaching (student evaluations, teaching awards, and peer ratings in teaching) supported convergent and divergent validity. However, the three-pronged model in academia was not validated, and exploratory factor analysis identified 5 and possibly 6 domains of behavior (research, classroom teaching, writing about teaching, community service, writing about service, and internal service). For one measure, peer ratings, reliability varied across academic areas and was moderately high for only high-confident ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relation between types of aggression and sociometric status: Peer and teacher perceptions.

Examined the relation between aggressive behavior and sociometric status (the extent to which a child is liked by his or her peers) in third and fourth graders. 47 boys and 51 girls from a northern New England elementary school were given a group-administered peer rating of social status and a peer nomination measure assessing five types of aggressive behavior. Teachers of the same children were given modified versions of these measures. Significant negative correlations were found between social status and all categories of aggressive behavior for both sexes except provoked physical aggression in boys for which the correlation was not significant. Indirect aggression (tattling, stealing, or breaking others' property) was the type of aggressive behavior that correlated most highly with low peer ratings. There were significant differences between boys and girls on all five categories of aggressive behavior. Teachers' ratings of peer social status correlated more highly with boys' ratings than with girls' ratings, and teacher perceptions of aggressive behavior correlated significantly and positively with peers' ratings on only two categories, unprovoked physical aggression and indirect aggression. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effects of a monetary incentive on the accuracy of post-training self-reinforcement behavior.

96 paid undergraduates classified as high or low self-reinforcers during a baseline phase were randomly assigned to 1 of 2 criterion groups and trained to a 40 or an 80% criterion on a 4-choice discrimination task. Ss' self-reinforcement behavior was assessed during a posttraining phase, and a rating scale evaluating confidence in the accuracy of their choice during the posttraining phase was given. The higher training level was associated with a higher self-reinforcement (SR) accuracy, while high baserate was associated with a greater SR frequency and more indiscriminate SR behavior. Incentive did not improve SR accuracy. Confidence ratings of females were related to training level, baseline grouping, and incentive, while in males the ratings were related to training level and baseline grouping. (French summary) (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Dimensions that make a difference: Examining the impact of in-role and extrarole behaviors on supervisory ratings.

Levels of in-role and extrarole performance were experimentally manipulated in a 3?×?2 within-Ss design. Dimensions capturing both in-role and extrarole behaviors (i.e., citizenship behaviors directed toward other individuals) strongly influenced various measures of rater search strategies as well as the ratings given by 116 supervisors evaluating secretarial performance. Furthermore, as predicted by D. W. Organ (1990), dimension ratings assigned to ratees with high levels of extrarole behavior displayed significantly more halo than ratings given to ratees with neutral levels of extrarole behavior. Tests for the effects of level of extrarole behavior on 2 measures of rater accuracy produced contradictory results. Implications are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of assessment center ratings for the prediction of performance ratings and school climate of school administrators.

Describes the validation of an assessment center used to select school administrators on the basis of ratings of 153 school administrators who had participated in the center between 1976 and 1981. Behaviorally anchored rating scales were obtained from supervisors, teachers, and support staff; and measures of 7 school climate dimensions were provided by teachers, students, and support staff. Results show a significant relationship between an overall assessment center placement recommendation and supervisory, teacher, and support staff ratings on most performance dimensions. Climate measures were significantly related to few assessment-center ratings regardless of the source of the climate measure. It is suggested that the assessment center proved valid for a sample from widely distant geographic areas and school districts of differing sizes and levels. (16 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student evaluations of college professors: Are female and male professors rated differently?

Over 1,000 male and female college students of 16 male and female professors (matched for course division, years of teaching, and tenure status) evaluated their instructors in terms of teaching effectiveness and sex-typed characteristics. Male students gave female professors significantly poorer ratings than they gave male professors on the six teaching evaluation measures; their ratings of female professors were poorer than those of female students on four of the six measures. Female students also evaluated female professors less favorably than male professors on three measures. Student perceptions of a professor's instrumental/active and expressive/nurturant traits, which were positively related to student ratings of teaching, accounted for only a few of these gender-related effects. Student major and student class standing also played a role in the evaluation of professors. The importance of gender variables in teacher evaluation studies is discussed, and implications for future research are noted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessing Measurement Equivalence Across Rating Sources: A Multitrait-Multirater Approach.

Research examining the structure of multisource performance ratings has demonstrated that ratings are a direct function of both who is doing the rating (rating source) as well as what is being rated (performance dimension). A separate line of research has focused on the extent to which performance ratings are equivalent across sources. To date no research has examined the measurement equivalence of multisource ratings within the context of both dimension and rating source direct effects on ratings. We examine the impact of both performance dimension and rating source as well as the degree of measurement equivalence across sources. Results indicate that (a) the impact of the underlying performance dimension is the same across rating sources, (b) the impact of rating source is substantial and only slightly smaller than the impact of the underlying performance dimension, and (c) the impact of rating source differs substantially depending on the source. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of the behavioral rating accuracy of groups and individuals.

Recent efforts to improve the accuracy of performance appraisal ratings have concentrated on individuals, with little attention given to the role groups might play as performance raters. Thus, the goal of this study was to compare the behavioral rating accuracy of groups vs individuals. Using a behavioral checklist, 191 Ss rated the behavior of a police officer individually or in 4-person groups. Ratings were made immediately or after a 5-day delay. Measures of memory sensitivity and decision criteria indicated that in the delayed rating condition, groups remembered the behaviors more accurately than did individuals, whereas in the immediate rating condition, groups and individuals did not differ. However, groups also demonstrated greater response bias, adopting a more liberal decision criterion than individuals. Results suggest that groups can be a help, but they are not a panacea for the problems of rating accuracy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Construct validity of measures of college teaching effectiveness.

Compared evaluation form, student, colleague, trained observer, former student, and self-rating teacher assessments of 43 university instructors. Data show that student and former student ratings displayed substantially greater validity coefficients of teaching effectiveness than self-report, colleague, and trained observer ratings. Advantages of student rating techniques (i.e., greater exposure to instructor's teaching), various teaching assessment methods, and problems in the literature due to limitations of research approaches typically used are discussed. (27 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)