Using frame-of-reference training to understand the implications of rater idiosyncrasy for rating accuracy.

Frame-of-reference (FOR) rater training is one technique used to impart a theory of work performance to raters. In this study, the authors explored how raters' implicit performance theories may differ from a normative performance theory taught during training. The authors examined how raters' level and type of idiosyncrasy predicts their rating accuracy and found that rater idiosyncrasy negatively predicts rating accuracy. Moreover, although FOR training may improve rating accuracy even for trainees with lower performance theory idiosyncrasy, it may be more effective in improving errors of omission than commission. The discussion focuses on the roles of idiosyncrasy in FOR training and the implications of this research for future FOR research and practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater goals on rating patterns: Evidence from an experimental field study.

The goal-based perspective of performance appraisals suggests that raters who pursue different goals give different performance ratings. Yet previous studies have not provided strong empirical evidence that there are different impacts of different goals on mean rating and discriminability, nor have they provided evidence of a goal-rating causality. The authors extend the literature by manipulating rater goals in the context of peer evaluations of graded group projects with a sample of 104 undergraduate students. They find that (a) pursuing a harmony goal increased mean rating and decreased discriminability, and (b) pursuing a fairness goal increased mean rating and decreased discriminability when the group projects had not ended and increased mean rating but did not change discriminability when the group projects had ended. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The roles of rater goals and ratee performance levels in the distortion of performance ratings.

The goal-directed perspective of performance appraisal suggests that raters with different goals will give different ratings. Considering the performance level as an important contextual factor, we conducted 2 studies in a peer rating context and in a nonpeer rating context and found that raters do use different rating tactics to achieve specific goals. Raters inflated their peer ratings under the harmony, fairness, and motivating goal conditions (Study 1, N = 103). More important, raters inflated their ratings more for low performers than for high and medium performers. In a nonpeer rating context, raters deflated ratings for high performers to achieve the fairness goal, and they inflated ratings for low performers to motivate them (Study 2, N = 120). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The quantification of the frame of reference in labor-management communication.

"A semantic barrier to communication between labor and management was quantified by establishing the frames of reference of labor and management criterion groups on the semantic differential, using concepts selected from the area of labor-management relations." The meaning of a concept is measured by asking S to mark on a 7-point scale between 2 logically or psychologically opposing terms the point at which he perceives the meaning to be. The score for a concept is the sum of scores on the scales related to it. "Significant semantic distance between the two groups was revealed. Labor stereotyped more than management, and assumed more extreme scale positions. Thus, the semantic distance seemed to have resulted more from labor's position than from management's. The greater standard deviations of the responses of the management group on three concepts suggested that management's position on these concepts was changing." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The need for theoretical and methodological integrations in family research.

Comments on the article by Carlson and Grotevant (see record 1989-00066-001) who have provided a welcome and thorough evaluation of the adequacy of eight family rating systems, with detailed discussions of optimal scale properties, rater competence and training issues, psychometric reliability, and validity. The present author realizes that it may be unfair to comment on issues that are not part of Carlson and Grotevant's central concerns, but after reading their article, focused entirely on method, he began to worry about the pervasive lack of integration between theory and method in family research. He also notes as worrying, the tendency of family researchers to focus on one perspective and to ignore the possible gains to be derived from integrating theoretical and methodological points of view. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A new frame for frame-of-reference training: Enhancing the construct validity of assessment centers.

The authors undertook a comprehensive examination of the construct validity of an assessment center in this study by (1) gathering many different types of evidence to evaluate the strength of the inference between predictor measures and constructs (e.g., reliability, accuracy, convergent and discriminant relationships), (2) introducing a theoretically relevant intervention (frame-of-reference [FOR] training) aimed at improving construct validity, and (3) examining the effect of this intervention on criterion-related validity (something heretofore unexamined in the assessment center literature). Results from 58 assessees and 122 assessors suggested that FOR training was effective at improving the reliability, accuracy, convergent and discriminant validity, and criterion-related validity of assessment center ratings. Findings are discussed in terms of implications and future directions for both FOR training and assessment center practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Dimensional comparisons: An experimental approach to the internal/external frame of reference model.

According to the internal/external frame of reference model (H. W. Marsh, 1986, 1990a), students not only use social comparisons to evaluate their performance (external frame of reference) but they also use dimensional comparisons (internal frame of reference), comparing their own achievement in one subject with that in other subjects. Three experimental studies were conducted to investigate the psychological processes underlying the effects of achievement in one domain on self-perceived competence in another. In Study 1 (N?=?36), high achievement in one domain led to lower self-perceived competence in the other domain. Study 2 (N?=?45) showed inverse effects on self-perceived competence when achievement feedback included explicit dimensional comparison information about students' achievement in both tasks. In Study 3 (N?=?90), dimensional comparison effects were shown even when additional social comparison information was presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

"Learning potential" as an assessment approach to the adolescent mentally retarded.

Studies with the educable mentally retarded have shown that, given a period of sufficient training and practice on motor tasks, the final level of performance achieved overlaps that of Ss with average IQs. The present study reports data fitting with this result using a reasoning task, the Kohs Blocks. Institutionalized adolescents, Stanford-Binet IQ range 42 to 75 were used. Experimental Ss were coached on a parallel series of block designs. Initial performance level on the task approximated the level predicted from the Binet IQs. Scores of the coached Ss on the Kohs series administered 1 day and 1 mo. following coaching demonstrated markedly increased proficiency. The discussion questioned the adequacy of present criteria used in categorizing the retarded. It suggested that the approach of this study, i.e., obtaining an estimate of potential for profiting from a nonschool related reasoning task on which they are appropriately coached in an ego-supportive context, may prove to be a useful supplementary technique to predict adult status. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Advancing the science of psychological assessment: The Rorschach Inkblot Method as exemplar.

This article comments on a series of 5 articles, concerning the utility of the Rorschach Inkblot Method (RTM; R. M. Dawes, see record 1999-11130-006; J. Hiller et al, see record 1999-11130-005; J. Hunsley and J. M. Bailey, see record 1999-11130-004; G. Stricker and J. R. Gold, see record 1999-11130-002; and D. J. Vigilone, see record 1999-11130-003). Two of the articles provide extensive empirical evidence that the RIM has been standardized, normed, made reliable, and validated in ways that exemplify sound scientific principles for developing an assessment instrument. A 3rd article reports a meta-analysis, indicating that the RIM and the Minnesota Multiphasic Personality Inventory have almost identical validity effect sizes, both large enough to warrant confidence in using these measures. The other 2 articles adduce sketchy data and incomplete literature reviews as a basis for questioning the psychometric soundness of Rorschach assessment. Unwarranted skepticism should not be given credence as an adequate platform from which to challenge abundant evidence that the RIM works very well for its intended purposes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further evidence for the validity of assessment center dimensions: A meta-analysis of the incremental criterion-related validity of dimension ratings.

This study investigates the incremental variance in job performance explained by assessment center (AC) dimensions over and above personality and cognitive ability. The authors extend previous research by using meta-analysis to examine the relationships between AC dimensions, personality, cognitive ability, and job performance. The results indicate that the 7 summary AC dimensions postulated by W. Arthur, Jr., E. A. Day, T. L. McNelly, & P. S. Edens (2003) are distinguishable from popular individual difference constructs and explain a sizeable proportion of variance in job performance beyond cognitive ability and personality. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relative importance of task and contextual performance dimensions to supervisor judgments of overall performance.

Although evidence supports the unique contribution of task performance and contextual performance to overall evaluations, little is known about the relative contribution that specific dimensions of contextual performance make to overall performance judgments. This study evaluated the extent to which supervisors consider task and contextual performance by using relative weights (J. W. Johnson, 2000) to statistically describe the relative importance of specific dimensions of each type of performance to overall performance ratings. Within each of 8 job families in a large organization, each of 4 dimensions of contextual performance made not only a unique contribution but a relatively important contribution to the overall evaluation. Evidence also supports the adaptive performance dimension of handling work stress as an aspect of contextual performance and job–task conscientiousness as an aspect of both task and contextual performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The men's center approach to addressing the health crisis of college men.

College men's health is in crisis, yet men are reluctant to seek mental health services. How can psychologists provide interventions to engage and empower college men to address their health needs? What are the components of culturally-tailored interventions for college men? We describe the origins and operations of a university-based Men's Center devoted to helping college men lead healthy lives. The Men's Center has evolved into a therapeutic and training approach that guides campus psychologists toward unique roles to intervene effectively with college men. Key components of the Men's Center Approach (MCA) include acceptance, nonjudgment and unconditional positive regard, respect for diversity, working from the inside-out, power sharing, strategic use of the planning process, therapeutic experiences in nontherapy settings, and fostering and strengthening commitments to social justice and activism. Central to these components is our notion of possible masculinity, in which we focus on men's aspirations and future goals for their identities and behaviors based on what men need to become healthy, responsible, and nurturing in their families and communities. Practical applications of these components are presented through examples of two Men's Center interventions. We conclude by discussing how psychologists can implement the MCA in their clinical practice with men to increase cultural competence with men while working across various settings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Identification and meta-analytic assessment of psychological constructs measured in employment interviews.

There has been a growing interest in understanding what constructs are assessed in the employment interview and the properties of those assessments. To address these issues, the authors developed a comprehensive taxonomy of 7 types of constructs that the interview could assess. Analysis of 338 ratings from 47 actual interview studies indicated that basic personality and applied social skills were the most frequently rated constructs in this taxonomy, followed by mental capability and job knowledge and skills. Further analysis suggested that high- and low-structure interviews tend to focus on different constructs. Taking both frequency and validity results into consideration, the findings suggest that at least part of the reason why structured interviews tend to have higher validity is because they focus more on constructs that have a stronger relationship with job performance. Limitations and directions for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Retranslation of expectations: An approach to the construction of unambiguous anchors for rating scales.

A procedure was tested for the construction of evaluative rating scales anchored by examples of expected behavior. Expectations, based on having observed similar behavior, were used to permit rating in a variety of situations without sacrifice of specificity. Examples, submitted by head nurses as illustrations of nurses' behavior related to a given dimension were retained only if reallocated to that dimension by other head nurses, and then scaled as to desirability. Agreement for a number of examples was high, and scale reliabilities ranged above .97. Similar content validity should be obtained in other rating situations. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Athletic classmates, physical self-concept, and free-time physical activity: A longitudinal study of frame of reference effects.

The systematic analysis of factors that promote or impede physical activity in children is an urgent task for educational researchers. The present study investigated the reciprocal relationship between physical self-concept, teacher-assigned grades in physical education classes, and free-time physical activity, and analyzed positive and negative consequences of being in a class with high class-average physical ability. Data from a large, representative sample of 1,095 preadolescents from 66 classrooms were examined within a longitudinal framework. Multilevel analyses showed that membership in a class with high class-average physical ability was associated with lower physical self-concept and free-time physical activity and highlighted the significant role of teacher-assigned grades in the development of physical self-concept and physical activity. Furthermore, as predicted, there were positive reciprocal effects between physical self-concept and physical activity levels. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The adjective check list as a personality assessment technique with children: Validity of the scales.

Adjective Check List (ACL) ratings of personality characteristics were made by the mothers of 52 pairs of grade-school twin girls. The 24 ACL scales were factor analyzed, yielding 3 factors named extroversion-introversion, social desirability, and personality traits associated with intelligence. The factors were found to be similar to those extracted from other personality tests. In addition to the ACL, the TAT, the Vineland Social Maturity Scale, the Fels Behavior Scales, 2 WISC subtests, the Goodenough drawings, an experimental Toys game, an activity interview, and background information were scored for the sample. The ACL scales correlated systematically with both direct and indirect measures of the twins' behavior, lending confidence in the validity of the ACL scales for a population of young children. (17 ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)