Effectiveness of performance feedback from behaviorally anchored rating scales.

Compared a behaviorally anchored rating scale (BARS) developed according to the procedure of H. Bernardin et al (see record 1976-08614-001) with a summated rating scale. Using both scale formats (designed to evaluate teaching effectiveness), 859 undergraduates rated 32 instructors during spring classes, and 314 undergraduates rated 19 instructors during summer classes. Students rated instructors halfway through the course and at its end. Instructors received feedback from the 1st rating period on either the BARS (including written behavioral observations) or on the summated rating format. Analyses of covariance revealed no format effect on the performance ratings obtained during the 2nd rating period in the spring experiment. BARS feedback, however, produced greater behavioral change among instructors than did feedback from the alternative format used during the summer experiment. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A Guttman approach to the development of behaviorally based rating scales for systems analysts and programmer/analysts.

Developed behaviorally-based rating scales for programer/analysts and systems analysts. The scales exhibited adequate Guttman scaling properties and also demonstrated reasonably good convergent and discriminant validity. In an attempt to identify relatively poor raters using Guttman scaling procedure, 103 supervisors rated 113 programer/analysts and 87 systems analysts. There was only slight evidence that supervisors who rate poorly on 1 dimension also rate poorly on other dimensions, or that supervisors who rate 1 individual poorly also rate another individual poorly on the same dimension. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of behaviorally anchored rating scales as a function of organizational level.

2 levels in a large medical center-8 head nurses (supervisory level) and 9 registered nurses (subordinate level)-independently participated in the development of behaviorally anchored rating scales. Although the 2 organizational levels identified similar behavioral dimensions, the behavioral incidents defining the dimensions were valued differently. Alternative explanations for the obtained differences are offered, and a discussion of behavioral expectation scale methodology is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Development of behaviorally anchored scales for the evaluation of faculty teaching.

Presented a procedure and rationale for evaluating college teaching using behaviorally anchored rating scales. In Stage 1 (n = 38 undergraduates), 9 independent dimensions important for teaching evaluation and representative behavioral incidents were identified. In Stage 2 (n = 54 undergraduates), incidents were allocated to dimensions. In Stage 3 (n = 139 undergraduates), incidents were evaluated on a scale representing effective teaching. Items with low standard deviations were retained for the final scales. The underlying notions of the resulting scales and the advantages of using the behavioral expectation procedure relative to other procedures are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Computer-administered interviews and rating scales.

To evaluate the value of computer-administered interviews and rating scales, the following topics are reviewed in the present article: (a) strengths and weaknesses of structured and unstructured assessment instruments, (b) advantages and disadvantages of computer administration, and (c) the validity and utility of computer-administered interviews and rating scales. Computer-administered evaluations are more comprehensive and reliable and less biased than evaluations routinely conducted in clinical practice. Also, the use of continuous monitoring systems, which increasingly entail the use of computer administration, has been related to improved treatment outcome. However, the use of computer-administered interviews and rating scales will sometimes lead to false positive diagnoses, and for this reason, it is recommended that computer assessment be combined with clinical judgment. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Children's and teachers' perspectives on children's self-control: The development of two rating scales.

Three teachers rated 28 4th graders on 15 self-control items using a 5-point frequency scale, and children responded to a yes/no format for each of their 11 self-control items. Both the teachers' and children's scales were factored based on responses of 763 4th and 5th graders from 36 classes. Children's IQ was assessed using the Lorge-Thorndike Intelligence Tests, psychosocial adjustment was evaluated using the Child Behavior Rating Scale, and verbal and quantitative academic achievement was measured by the Stanford Achievement Test. The teachers' scale consisted of 2 factors replicated across halved subsamples: (a) Behavioral/Interpersonal and (b) Cognitive/Personal Self-Control. The children's scale had 4 orthogonal factors: (a) Personal Self-Control, (b) Interpersonal Self-Control, (c) Self-Evaluation, and (d) Consequential Thinking. Full-scale and factor scores had high reliability. The validity of each scale was supported, although the 2 were only weakly related to each other. Self-control ratings by teachers and children related to naturalistic observations and to teacher ratings of frustration tolerance and acting-out/aggressive problems. Teachers' (but not children's) ratings of self-control related to IQ and achievement. Girls were rated as better self-controlled than boys by both groups. There were no grade-level differences. Uses of the scales in large-scale social-systems studies are discussed. (38 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Behaviorally based rating scales need not be job specific.

Used P. C. Smith and L. M. Kendall's (1963) procedure for designing behaviorally based rating scales to develop a set of 10 rating scales for all jobs (except nursing) of an entire hospital. Results show that 20 supervisors reached acceptable levels of agreement in allocating performance incidents to scales and assigning scale values to the incidents. This agreement was particularly encouraging since any given incident described job performance outside the work area of a majority of the supervisors. The practical implications of developing behaviorally based rating scales for various jobs, instead of just one, are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The use of parametric vs. nonparametric tests in the statistical evaluation of rating scales

In psychiatric studies, treatment efficacy is usually measured by rating scales. These scales have ordinal (rank) level and the statistical evaluation of the scale scores should be performed with nonparametric rather than parametric tests. In recent years, nonparametric statistical procedures for repeated measures have been developed for the evaluation of clinical trials in psychiatry. To assess the frequency of application of nonparametric tests in psychiatric trials, an analysis was performed on all treatment studies with panic disorder patients (DSM-III/III-R) that could be traced in the literature. This survey revealed that nonparametric tests were used in only 16.7% of all studies.     

A comparison of behavioral expectation scales and graphic rating scales.

Compared ratings derived from behavioral expectation scales developed by 147 personnel management students with ratings based on graphic rating scales. The ratees were 4 college professors, and the raters were the 183 students in their classes. The behaviorally anchored scales resulted in less halo error, or alternatively, more independence in ratings of different dimensions of performance. The behaviorally anchored scales did not correct for leniency in ratings. These results were observed both among raters who participated in developing the behavioral expectation scales and among similar raters who did not take part in this process. The factor structures of the 2 rating formats were essentially equivalent in "cleanness." Neither solution was judged superior to the other. However, the behavioral expectation scale format possessed greater discriminant validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Four semantic rating scales compared.

In the construction of semantic rating scales, what format works best? "The questionnaire used in this investigation consisted of 24 eight-step rating scales… . All the scales were bipolar, and were defined by words or phrases at the scale poles, in the manner of the Semantic Differential… . The ratings were made by approximately 400 housewives… interviewed in their own homes by 11 professional interviewers." Rating and interviewer reports were the criteria. Methods were compared. "Whatever method is chosen, the choice should be made with a thoughtful evaluation of the influences these details of format and administration may have upon results." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Psychometric structure of psychiatric rating scales.

This study reports an exploratory principal-components analysis of four common psychiatric rating scales that yield a total of 22 separate scores. Five components were extracted accounting for 78% of the initial variance; oblique and orthogonal rotations produced identical interpretations of the components: (a) minimal functioning, (b) hostility/paranoia, (c) disturbed communication, (d) psychosis, and (e) social activity. Some limits on the present study are discussed, and suggestions for future research are presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Item weights in employee rating scales.

"Data from three rating forms were analyzed to determine the extent of superiority of statistically determined weights to arbitrarily assigned weights. Correlations between statistically determined and arbitrarily assigned weights were so high that they can be considered to be one and the same. Split-half reliability showed no difference between statistical and arbitrary weights, correlations between two rating forms were the same for both, and inter-rater reliability showed no difference between weights… ." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Observational rating scales: Supplementary comments.

Comments on the article by Carlson and Grotevant (see record 1988-17683-001). Carlson and Grotevant's article in this issue provides an excellent and thoughtful overview of observational rating scales as applied to entire family units. Although originally viewed as "clinical" in nature, and lacking in respectable scientific rigor and precision, these assessment devices now have become a useful and flexible approach to the complicated process of family assessment. Notwithstanding the authors' scholarly discussion of issues in utilizing these methods, and their critical review of existing rating formats, I wish to raise three points for discussion: The first concerns the relationship of micro- to macroanalytic procedures, the second focuses on reliability, and the third raises a troubling theoretical issue. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Choosing response categories for summated rating scales.

Argues that the selection of response categories for summated rating (Likert) scales is often made arbitrarily; however, the equal interval properties of the response continuum is assumed even though this assumption may in fact, be false. To avoid the problems that unequal intervals might cause, it is suggested that equal interval categories be used. To this end, 3 lists of commonly used categories (agreement, evaluation, and frequency) were scaled, using data from 107 undergraduates. The scale values are presented to aid the researcher in choosing response categories; however, it is suggested that separate scale values be derived for each population sampled. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A multimethod validation of two psychiatric rating scales.

Investigated the concurrent validity of the Psychotic Inpatient Profile (PIP) and the Nurses Observational Scale for Inpatient Evaluation (NOSIE-30) with 207 18–69 yr old prechronic psychiatric inpatients. Aides and professional observers also provided behavioral observations using the Time Sample Behavioral Checklist. Results show fairly good convergent and discriminant validity across instruments and across different raters. It is concluded that psychiatric rating scales, while an important approach to assessment, may not provide appropriate data for research requiring more direct assessment of patient's behavior on the ward. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validation of rating scales in psychiatry

Statistical analysis of psychiatric research requires the translation of qualitative data into quantitative form. This can be achieved using rating scales. However, the reliability and the validity, i.e. the validation of these scales, presents significant problems; these shall be outlined in this article.     

Defenses in rating scales and therapy success

Evaluation of psychotherapeutic or psychopharmacological treatment frequently uses rating scales without regard for patient's defensiveness. This study demonstrates on 309 psychotherapy patients treated with inpatient client-centered therapy and followed-up 1 year after treatment that defensiveness, measured with validity scales of personality inventories MMPI-K, FPI-R and Giessen-Test, influences ratings (scales CGI, HAMA, HAMD, BRMES) before and after therapy. Patients who deny psychopathology on admission do so at discharge and to some extent at follow-up. The extent of effect is significantly diminished with an increase in frankness in these patients, indicated by the significant improvement in validity scales. Inpatient client-centered psychotherapy improved defensiveness in the case of initial self-criticism and diminished it in the case of initial retentiveness.     

A clarification of some issues regarding the development and use of behaviorally anchored ratings scales (BARS).

Attempts to clarify some issues regarding the development and use of behaviorally anchored rating scales (BARS). The P. C. Smith and L. M. Kendall (1963) format is distinguished from other approaches to BARS, and research comparing their relative effectiveness is reviewed. Their format is discussed as a method that is designed to enhance future observations and to foster a common frame of reference in observer raters. Emphasis is also placed on BARS as an observation-rating system that provides data for the assessment of estimates of accuracy for individual raters. Responses are made to criticisms dealing with the rating process of BARS, the relative effectiveness of BARS vs summated scales, and the role and characteristics of the behavioral anchors. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A multidimensional set of rating scales for television commercials.

Uses attribute ratings and factor analysis as a method of multidimensional scaling to determine on what dimensions viewers can rate TV commercials. An initial pool of 525 words was reduced to a final set of 45 words by a series of steps designed to eliminate words that: (a) Ss do not spontaneously use, (b) do not discriminate among commercials, and (c) do not represent a basic dimension emerging from a factor analysis. The final analysis resulted in 7 factors. The 3 main ones were overall level of energy or stimulation, personal relevance, and familiarity. 4 other factors also emerged at a lower level of explanatory power: sensual, novel, authoritative, and disliked. Results are surprising in that there is no positive evaluative factor. It is suggested that positive evaluation is expressed in more specific ways. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Variable agreement between visual rating scales for white matter hyperintensities on MRI. Comparison of 13 rating scales in a poststroke cohort

BACKGROUND AND PURPOSE: Previous reports on the frequency, extent, and clinical correlates of white matter hyperintensities (WMHIs) have been contradictory. The purpose of this study was to test whether part of this variation could be explained by the different properties of the visual WMHI rating scales used. METHODS: The periventricular (PVHIs) and deep white matter (DWMHIs) hyperintensities of 395 poststroke patients were systematically analyzed and transformed to correspond to 13 different rating scales. The scales were compared with the use of Goodman-Kruskal measures of association. The relative frequencies, means, and medians of PVHI and DWMHI grades as well as Spearman rank correlations between WMHI grade and hypertension were calculated. RESULTS: At best more than 80% of the patients received an equivalent WMHI grade by different scales, but at worst the corresponding values were only 0.4% for PVHI and 18% for DWMHI. At best different scales categorized patients similarly in regard to WMHI grade, but at worst the corresponding values were 8% for PVHI and 57% for DWMHI ratings. The distribution of WMHI grades also varied, and when the effect of age on WMHI was assessed, some of the scales had a ceiling effect and some had a floor effect. Only 1 of the 7 PVHI, 5 of the 9 DWMHI, and 1 of the 3 combined rating scales showed a significant correlation with arterial hypertension, a putative risk factor for WMHIs. CONCLUSIONS: Some of the inconsistencies in previous studies of WMHIs are due to differences in visual rating scales. Our findings may warrant international debate regarding harmonization of WMHI ratings.