The predictability of ratings as a function of interrater agreement.

"The hypothesis tested was that high agreement among the ratings assigned the same men by different raters does not necessarily imply predictable ratings." 3 superior officers rated 100 submariners on personal adjustment and technical competence. Each rating group was divided into 4 samples for interrater agreement. Correlations were computed with 3 predictor variables. The results support the hypothesis given. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Distributional ratings of performance: Further examination of a new rating format.

Examined several aspects of construct validity evidence for a distributional format by comparing it with a Behavioral Observation Scale (BOS) rating form and determining whether raters were sensitive to differences in performance variability. Raters were assigned to 1 of 2 instructional procedures, 1 of 2 form orderings (BOS or distributional rating first), and 1 of 5 conditions of performance variability. Ss rated an instructor's performance after viewing 4 videotaped excerpts of his lectures. Mean ratings were lower using the distributional format relative to the BOS format. The distributional ratings indicated that Ss were sensitive to the different variability conditions. The potential of distributional ratings for providing a richer source of performance information than more traditional ratings is discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Friends and strangers: Acquaintanceship, agreement, and the accuracy of personality judgment.

We examined the effect of acquaintanceship on interjudge agreement in personality ratings. Approximately 150 undergraduates described their own personalities using the Q-sort. They were also described by two close acquaintances and by two "strangers" who knew them only via a single, spontaneous interaction viewed on videotape. The effect of acquaintanceship was powerful: Judgments by close acquaintances agreed with each other and with subjects' self-judgments much better than did judgments by strangers, even though strangers' judgments agreed with each other and with subjects' self-judgments beyond a chance level. This result implies that agreement among acquaintances' judgments must derive at least partly from experience with and observation of the person who is judged. The same traits that yielded better agreement among acquaintances also yielded better agreement among strangers and tended to be rated higher in subjective visibility, suggesting that people are intuitively knowledgeable about the traits they can judge with more and less agreement. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessor training strategies and their effects on accuracy, interrater reliability, and discriminant validity.

This study compares the effects of data-driven assessor training with schema-driven assessor training and control training. The sample consisted of 229 industrial and organizational psychology students and 161 managers who were randomly assigned to 1 of these training strategies. Participants observed and rated candidates in an assessment center exercise. The data-driven and schema-driven assessor training approaches outperformed the control training on all 3 dependent variables. The schema-driven assessor training resulted in the largest values of interrater reliability, dimension differentiation, and accuracy. Managers provided significantly more accurate ratings than students but distinguished less between the dimensions. Practical implications regarding the design of assessor trainings and the composition of assessor teams are proposed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Similarity of post-interview trait rating intercorrelations as a contributor to interrater agreement in a structured employment interview.

Examined the influence of degree of interview structure, use of scaled-expectation rating scales, and similarity of postinterview trait rating intercorrelations on interrater agreement in employment interviews. 9 nursing interviewers sat as a selection board and interviewed and independently rated 54 senior nursing students. The interviews were highly structured, and ratings were recorded on scaled-expectation scales for general staff nursing positions. Although all interviewers shared essentially the same structure among their postinterview trait ratings, interrater agreement was no better than in previous studies. This finding shows the power of halo in the interview setting (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Systematic distortions in memory-based behavior ratings and performance evaluations: Consequences for rating accuracy.

To test the hypothesis that memory-based ratings should be less accurate than ratings collected under conditions that minimize demands on memory, data were obtained from 82 undergraduates who had rated 4 videotapes of graduate student lecturers. Half of the tapes were rated immediately after they were viewed; Ss returned the following day and rated the remaining tapes from memory. Memory-based behavior ratings and performance evaluations showed higher intercorrelations (more halo) than did ratings that were collected immediately after viewing the ratee's performance. However, ratings were systematically more accurate in the delayed-rating condition than in the immediate-rating condition. It is argued that (1) under certain conditions, raters may depend on their general impressions of ratees rather than on their memory for specific details; and (2) these schematic evaluations may preserve a greater proportion of valid information, as compared with irrelevant detail, than is available immediately after observing ratee behavior. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of comparative performance information on the accuracy of self-ratings and agreement between self- and supervisor ratings.

The manner in which social comparison performance information affects the accuracy of self-ratings and the agreement between self- and supervisor ratings was investigated in a laboratory experiment using 163 undergraduates. Ss proofread a series of articles for 30 min and then made self-ratings of work performance. Half of the subjects received social comparison information prior to making self-ratings. Correlations between self- and supervisor evaluations and between self-evaluations and objective performance indicators were significantly larger when self-raters were presented with the same comparative performance information that was available to supervisors. Implications of the findings for future research on self- and supervisor evaluations are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater errors and rating accuracy.

Meta-analysis was used to determine the relationship between rater error measures and measures of rating accuracy. Data from 10 studies (N?=?1,096) were used to estimate correlations between measures of halo, leniency, and range restriction and L. J. Cronbach's (1955) four measures of accuracy. The average correlation between error and accuracy was .05. No moderators of the error–accuracy relationship were found. Furthermore, the data are not consistent with the hypothesis that error measures are sometimes valid indicators of accuracy. The average value of the 90th percentile of the distribution of correlations (corrected for attenuation and range restriction) was .11. The use of rater error measures as indirect indicators of accuracy is not recommended. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Diversity of decision-making models and the measurement of interrater agreement.

Several papers have appeared criticizing the kappa coefficient because of its tendency to fluctuate with sample base rates. The importance of these criticisms is difficult to evaluate because they are presented with regard to a highly specific model of diagnostic decision making. In this article, diagnostic decision making is viewed as a special case of signal detection theory. Each diagnostic process is characterized by a function that relates the probability of a case receiving a positive diagnosis to the severity or salience of symptoms. The shape of this diagnosability curve greatly affects the value of kappa obtained in a study of interrater reliability, how it changes in response to variation in the base rates, and how closely it corresponds to the validity of diagnostic decisions. The common practice of evaluating a diagnostic procedure, when criterion diagnoses for comparison are unavailable, on the basis of the magnitude of the kappa coefficient observed in a reliability study is questionable. New methods for measuring interrater agreement are necessary, and possible directions for research in this area are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Understanding performance ratings: Dynamic performance, attributions, and rating purpose.

The present two studies integrate and extend the literatures on dynamic performance, performance attributions, and rating purpose, making several important contributions. First, examining attributions of dynamic performance, Study 1 predicted that performance mean and trend would affect judged ratee ability and effort and that performance variation would affect locus of causality; both predictions were supported by the results. Second, investigating the interaction between dynamic performance and rating purpose, Study 2 predicted that performance mean would have a stronger impact on administrative than on developmental ratings, whereas performance trend and variation would have a stronger impact on developmental than on administrative ratings; again, both predictions were borne out by the results. Third, both studies found that performance trend interacted with performance mean and variability to predict overall ratings. Fourth, both studies replicated main effects of dynamic performance characteristics on ratings in a different culture and, in Study 2, a sample of more experienced managers. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Do the ways in which psychotherapy process ratings are made make a difference? The effects of mode of presentation, segment, and rating format on interrater reliability.

52 Ss who had experienced the death of a parent or husband underwent 12 weekly psychotherapy sessions with 9 therapists. 208 sessions were rated. Compared were (a) ratings made from audiotapes vs videotapes; (b) ratings made from sessions 2, 5, 8, or 11; (c) ratings made from the 1st half vs ratings made from the whole session; (d) ratings made using a global response format vs a tally method of making judgments; and (e) ratings of therapist actions vs patient actions. Results indicate that small but significant differences in conclusions about process studies may emerge due to the operations of the conduct of the study and that these differences may have little relationship to the constructs being measured or their construct validity network. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Direct, indirect, and controlled observation and rating accuracy.

Videotaping of assessment center exercises has become an increasingly common practice, yet little is known about the impact of video technology on rating accuracy. This study compared ratings of a group discussion made after live observation (direct), after viewing a video (indirect), or after viewing a video with opportunities to pause and rewind (controlled). Results indicated some differences in observational accuracy but not in rating accuracy. Implications for the use of video technology in assessment centers are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of variations in rating scale format on judgment.

The purpose of this study was to determine whether mean occupation evaluation ratings would differ as a function of 7 variations in rating-scale format. 60 basic airmen rated 15 occupations on 9 occupation-requirement factors for each format. A 3-way analysis of variance (occupations, factors, scale format) resulted in statistically significant terms for each of the main effects and for all 4 interaction terms. It was concluded that rating-scale format was a determiner of the judgment of raters in this sample and that selection of an optimal format should be based upon capability to predict a criterion. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Counselor-client agreement on session impact.

We propose a new framework for understanding studies of counselor–client agreement about their counseling. The framework includes five factors: the scope of counseling being studied (process, impact, or outcome), the dimension (index) being rated (in this study, session Depth, Smoothness, Positivity, or Arousal), the measure used to assess agreement (correlations or absolute differences), the level at which the analysis is conducted (session, client, or counselor), and the type of agreement—(a) consensus, the similarity of counselors' own ratings to clients' own ratings; (b) counselor awareness, the similarity of counselors' perceptions of their clients to clients' own ratings; (c) client awareness, the similarity of counselors' own ratings to clients' perceptions of their counselors; and (d) matched awareness, the similarity of counselors' perceptions of their clients to clients' perceptions of their counselors. In a study of session impact (scope), degree of agreement was found to vary substantially with each of the other factors—type of agreement, dimension rated, measure of agreement, and level of analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Perceived similarity and accuracy of peer ratings.

The hypothesis that raters will be more accurate in rating peers perceived to be similar to themselves, suggested by Mumford (1983) and derived from social comparison theory, was examined. Subjects were 681 Israeli entrants to a military training program. Shortly after course inception, subjects were asked to review the performance of squad members and to forecast their final grade. Subjects also judged peers' similarity to self overall (general similarity), in course achievement (foreground similarity), and in military experience (background similarity). Analysis revealed that accuracy was markedly lower in the evaluation of dissimilar others. Although the same patterns of results was observed for all forms of similarity, stronger effects resulted when similarity was measured in terms of general and foreground characteristics. Implications for future theory and research as well as for the practical application of peer assessment are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Impediments to accurate clinical judgment and possible ways to minimize their impact.

Recent research in cognitive psychology has begun to uncover some of the factors that make clinical judgment a difficult task. Five impediments to accurate judgment are discussed: inability to assess covariation, influence of preconceived notions, lack of awareness of one's judgmental processes, overconfidence, and the hindsight bias. To minimize the impact of these impediments, 3 strategies are suggested: active consideration of alternative outcomes, increased attention to certain types of usually ignored data, and minimization of the role of memory. (43 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)