Rating formats and rater training redux: A context-specific approach for enhancing the effectiveness of performance management.

In this paper, we critically examine previous research on rating formats and rater training in the context of performance appraisal. Historically, the goal of this body of research has been to search for ways of maximizing the psychometric quality of performance evaluation data. Our central thesis is that there are a number of avenues for broadening this research. Accordingly, we propose a conceptual model that hopefully serves as a conceptual framework for future work in these 2 traditional performance appraisal research streams. For example, both rating formats and rater training research may be useful for facilitating and improving the feedback and employee development process, as well as reducing the potential existence of rater biases. In addition, format and training research may focus upon ways of enhancing both rater and ratee reactions to the appraisal system. A key feature of our model is the integration of national culture as a moderator of the relations between specific formats, training programs, and various outcomes. We consider both the national culture of raters and ratees, and focus specifically on comparisons between Western and East Asian cultures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Can performance-feedback accuracy be improved? Effects of rater priming and rating-scale format on rating accuracy.

Performance appraisal information is often used for employee feedback and development. Research has found that assessments that are global (i.e., based on broad aspects of performance) and comparative (i.e., explicit interratee comparisons) may be most accurate in terms of Cronbach's (1955) differential accuracy, a type of accuracy that is directly relevant to the provision of feedback. Unfortunately, a global-comparative assessment may not give recipients the most useful diagnostic feedback. In this experiment, an innovative rater-priming manipulation was developed and tested on a sample of 109 participants. The priming manipulation had the effect of improving differential accuracy and providing diagnostic feedback. A 2nd independent variable involving 2 different Behavioral Observation Scale formats also was investigated. Explanations of findings, limitations of this experiment, directions for future research, and implications for performance appraisal practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of rater training on rater accuracy: Levels-of-processing theory and social facilitation theory perspectives.

We use levels-of-processing theory and social facilitation theory to explain the effect of training format and group size on distance and correlation accuracy, leniency-severity, halo, retention of training and pretraining information, and subject arousal. The training factor included frame-of-reference (FOR) training, information only (INFO) training, and no training (NOT). Group size was n?=?1, n?=?6, and n?=?12, respectively. A total of 108 subjects, randomly assigned to one of nine Training?×?Group Size conditions, viewed and rated videotaped lectures. Results indicated that FOR training effected improved retention of training information, improved distance accuracy, and less halo over INFO training or NOT (p     

Consultant effectiveness and treatment acceptability: An examination of consultee requests and consultant responses.

There has emerged a very strong supposition in the literature and in practice that consultation be a collaborative venture between coequal professionals. Although a collaborative approach has been generally assumed beneficial, an emerging body of research is calling this into question. This study was designed to elucidate the collaborative process. Teachers were exposed to the manipulation of two independent variables: type of Teacher Request and type of Consultant Response. Teachers viewed videotaped scenarios in which a consultee presented a consultant with one of two types of requests for help: (a) a specific request for assistance or (b) a vague request for process clarification. Teachers in videotapes received one of three types of responses from consultants: (a) specific expert advice, (b) a problem-solving process, or (c) a request for the teacher to collect baseline information. Analyses of group differences were performed yielding a main effect for type of Teacher Request and a main effect for type of Consultant Response. A significant interaction was yielded with the deletion of the attention control Consultant Response groups. Results were interpreted as related to the collaborative consultation literature. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bobo revisited: What the research says.

[Correction Notice: An erratum for this article was reported in Vol 17(2) of International Journal of Play Therapy (see record 2008-14897-001). A reference was incomplete. The correct reference is Trotter, K., Eshelman, D., & Landreth, G. (2003). A place for Bobo in play therapy. International Journal of Play Therapy, 12, 117-139.] There has been some discussion in the play therapy literature regarding whether to use a Bobo doll (bop bag) in the play therapy room. The following article reviews research studies from the fields of personality, learning, and social psychology, and underlying theories to help inform and assist play therapists in their decision-making regarding Bobo. Suggestions are offered regarding future empirical research in play therapy outcomes and the choices of play materials in the playroom. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Planning of research on effectiveness of psychotherapy.

Describes the difficulties inherent in conducting research into the effectiveness of psychotherapy (PST) and provides recommendations for planning and design. Because PST is a treatment modality that has not been defined, requires long-term treatment and follow-up, and can be influenced by many variables, little systematic empirical knowledge about its effectiveness exists. Because of these considerations, such research should be closely linked to theory and concepts and use standardized measurement procedures as well as sound experimental or correlational designs. To ensure replicability and reliability, the methodology, treatment, setting, patient, therapist, and other variables should be described thoroughly. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The syntagmatic-paradigmatic shift revisited: A review of research and theory.

Discusses 4 classes of theories about the significance of the syntagmatic-paradigmatic shift in children's word associations: (a) associative strength, (b) syntactic, (c) semantic, and (d) cognitive theories. Qualifications regarding the generality of the phenomenon are pointed out. It is suggested that both a shift in conceptual organization and a change in children's interpretation of the task may account for the data. (71 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology.

[Correction Notice: An erratum for this article was reported in Vol 16(1) of School Psychology Quarterly (see record 2007-17958-001). In Table 1, the following errors appeared for the Colton and Sheridan (1998) entry: the information should be "3" (case scenarios), "naturalistic" (experimental design), "multiple baseline" (experimental methods), and "postassessment" (assessment interval).] Reviews conceptual, methodological, and paradigmatic issues related to the acceptability of school-based practices from a behavioral orientation. First, the authors provide an overview of the acceptability construct from a behavioral perspective including (a) the historical development of the construct of acceptability, (b) the behavioral conceptualization and definition of the construct of acceptability, and (c) the prevailing conceptual models of acceptability. Second, the authors illustrate the methodology typically used when examining consumers' acceptability of school-based practices from a behavioral perspective using a cross-source, cross-method approach. A review of empirical studies examining the acceptability of consultation, assessment, and intervention practices using a variety of school-based consumers (e.g., school psychologists, parents, teachers, children) is conducted. The importance of examining the acceptability of school-based practices from this perspective is discussed. Third, the authors discuss important methodological issues that need to be considered in conducting acceptability research. Finally, the advantages and limitations of examining acceptability within a behavioral context are reviewed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of rater participation in scale contruction on the psychometric characteristics of two rating scale formats.

Investigated the effect of participation in scale construction on selected characteristics of the rating scales and certain rater perceptions regarding use of the scales. 25 US Air Force ROTC seniors were randomly assigned to 1 of 3 experimental conditions--one in which Ss developed behaviorally anchored rating scales via the process of P. C. Smith and L. M. Kendall (1963), one in which Ss developed graphic rating scales, and one in which Ss did not participate in scale development. Ss then rated a common set of ratees on 2 occasions using both sets of scales. Participation in scale development led to increased convergent validity and less halo. Discriminant validity was not affected. The treatment groups did not significantly differ with respect to perceptions regarding scale use as measured by questionnaire. (32 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater training and diary-keeping on psychometric error in ratings.

Compared the psychometric properties of ratings on behavioral expectation scales (BES) across 4 groups totalling 156 undergraduate raters. Groups differed with respect to amount of prior training (1 hr or more), the nature of psychometric errors, and the extent of exposure to scales (read scales and recorded observed critical incidents, discussed general scale dimensions, or no exposure to scales). Three Ss from each group rated 1 of 13 instructors during the last week of a 10-wk term. Significantly less leniency error and halo effect, plus higher interrater reliability, were found for the group that had received the hour of training and full exposure to the BES. Ss who had received only training had significantly less halo error than those that had received no training. The need for rater training prior to observation and the use of BES as a context for observation are discussed. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Therapeutic Bond Scales: Psychometric characteristics and relationship to treatment effectiveness.

The Therapeutic Bond Scales assess the quality of the therapeutic relationship from the patient's perspective. The therapeutic bond is composed of 3 aspects: working alliance, empathic resonance, and mutual affirmation. Scales were developed to measure these aspects and the therapeutic bond as a whole. The correlations between these scales and 2 measures of outcome (session quality assessed by the patient and termination outcome evaluated by nonparticipant raters) were examined. All scales were significantly correlated with session quality. Therapeutic bond was significantly correlated with termination outcome in both a linear and a curvilinear fashion, suggesting that, at least in the initial phase of therapy, the therapeutic bond can be too high as well as too low. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of rater training programs: Error training and accuracy training.

108 undergraduates were randomly assigned to 1 of 4 experimental groups to rate videotaped performances of several managers talking with a problem subordinate. The research employed a single-factor experimental design in which rater error training (RET), rater accuracy training (RAT), rating error and accuracy training (RET/RAT), and no training were compared for 2 rating errors (halo and leniency) and accuracy of performance evaluations. Differences in program effectiveness for various performance dimensions were also assessed. Results show that RAT yielded the most accurate ratings and no-training the least accurate ratings. The presence of error training (RET or RET/RAT) was associated with reduced halo, but the presence of accuracy training (RAT or RET/RAT) was associated with less leniency. Dimensions?×?Training interactions revealed that training was not uniformly effective across the rating dimensions. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

"Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology": Errata.

Reports an error in "Behavioral conceptions and applications of acceptability: Issues related to service delivery and research methodology" by Tanya L. Eckert and John M. Hintze (School Psychology Quarterly, 2000[Sum], Vol 15[2], 123-148). In Table 1, the following errors appeared for the Colton and Sheridan (1998) entry: the information should be "3" (case scenarios), "naturalistic" (experimental design), "multiple baseline" (experimental methods), and "postassessment" (assessment interval). (The following abstract of the original article appeared in record 2000-00107-002.) Reviews conceptual, methodological, and paradigmatic issues related to the acceptability of school-based practices from a behavioral orientation. First, the authors provide an overview of the acceptability construct from a behavioral perspective including (a) the historical development of the construct of acceptability, (b) the behavioral conceptualization and definition of the construct of acceptability, and (c) the prevailing conceptual models of acceptability. Second, the authors illustrate the methodology typically used when examining consumers' acceptability of school-based practices from a behavioral perspective using a cross-source, cross-method approach. A review of empirical studies examining the acceptability of consultation, assessment, and intervention practices using a variety of school-based consumers (e.g., school psychologists, parents, teachers, children) is conducted. The importance of examining the acceptability of school-based practices from this perspective is discussed. Third, the authors discuss important methodological issues that need to be considered in conducting acceptability research. Finally, the advantages and limitations of examining acceptability within a behavioral context are reviewed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rater goals on rating patterns: Evidence from an experimental field study.

The goal-based perspective of performance appraisals suggests that raters who pursue different goals give different performance ratings. Yet previous studies have not provided strong empirical evidence that there are different impacts of different goals on mean rating and discriminability, nor have they provided evidence of a goal-rating causality. The authors extend the literature by manipulating rater goals in the context of peer evaluations of graded group projects with a sample of 104 undergraduate students. They find that (a) pursuing a harmony goal increased mean rating and decreased discriminability, and (b) pursuing a fairness goal increased mean rating and decreased discriminability when the group projects had not ended and increased mean rating but did not change discriminability when the group projects had ended. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The role of method in treatment effectiveness research: Evidence from meta-analysis.

A synthesis of 319 meta-analyses of psychological, behavioral, and educational treatment research was conducted to assess the influence of study method on observed effect sizes relative to that of substantive features of the interventions. An index was used to estimate the proportion of effect size variance associated with various study features. Study methods accounted for nearly as much variability in study outcomes as characteristics of the interventions. Type of research design and operationalization of the dependent variable were the method features associated with the largest proportion of variance. The variance as a result of sampling error was about as large as that associated with the features of the interventions studied. These results underscore the difficulty of detecting treatment outcomes, the importance of cautiously interpreting findings from a single study, and the importance of meta-analysis in summarizing results across studies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Morphology and behavior: Phrenology revisited.

Research conducted by evolutionary psychologists and biologists shows that subtle individual differences in body morphology can be related to surprising and important differences in human behavior and reproductive success. The authors summarize recent work on these effects as they relate to fluctuating asymmetry, facial attractiveness, finger digit morphology, sexually dimorphic differences in body configuration, and head circumference. Examples include the discovery that women who have sex with bilaterally symmetrical men report more orgasms; men with attractive faces have higher quality sperm; the length of the index finger in relation to the ring finger is related to verbal fluency, spatial ability, and the risk of autism; women with an hourglass figure have more regular menstrual cycles and are more fertile; and the sound of a person's voice predicts his or her sexual behavior. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Frame-of-reference training and cognitive categorization: An empirical investigation of rater memory issues.

Considered the effects of frame-of-reference (FOR) training on raters' ability to correctly classify ratee performance as well as their ability to recognize previously observed behaviors. The purpose was to examine the cognitive changes associated with FOR training to better understand why such training generally improves rating accuracy. 93 college students (mean age 22 yrs) trained using either FOR or control procedures, observed 3 managers on videotape, and rated the managers on 3 performance dimensions. Results supported the hypothesis that, compared with control training, FOR training led to better rating accuracy and better classification accuracy. Also consistent with predictions, FOR training resulted in lower decision criteria (i.e., higher bias) and lower behavioral accuracy on a recognition memory task involving impression-consistent behaviors. The implications of these results are discussed, particularly in terms of the ability of FOR-trained raters to provide accurate performance feedback to ratees. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Expectations of obese trainees: How stigmatized trainee characteristics influence training effectiveness.

This study identifies stigma as a potential precursor to self-fulfilling prophecies in training interactions. Expectations held by leaders often result in actions that elicit expectancy-confirming behaviors from their subordinates. The results of the present study suggest that trainee weight (manipulated with a photograph depicting the trainee as either obese or average weight for height) influenced female trainer expectations and evaluations of the training and trainee. Furthermore, the results suggest that negative expectations held by trainers were related to trainee evaluations of the training and the trainer and, for less flexible trainers, to decrements in trainee performance on the trained task. Overall, the results suggest that trainer expectations can be influenced by stereotypes held about trainee characteristics, thus undermining training effectiveness. (PsycINFO Database Record (c) 2010 APA, all rights reserved)