Comparison of thought-listing rating methods.

Compared 3 methods of rating thoughts: self-ratings, rating by experts with thoughts presented randomly, and rating by experts with thoughts presented in context. 107 university students enrolled in rhetoric classes listed their thoughts prior to giving a speech and completed a personal report of confidence as a speaker. Results indicate that all 3 methods of rating thoughts were equal in predictions of speech anxiety and performance. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some reflections on general performance dimensions and halo rating error.

Criticizes an article by F. Landy et al (see record 1981-00274-001), which assumed that a general factor present in the intercorrelations of ratings of performance was generated by halo rating errors. A number of decision rules were used to generate an analytic procedure for extracting halo errors, correcting ratings, and interpreting factors present in the corrected correlations. A number of the decision rules, as well as the initial assumption, seem to be rules of thumb or to depend on implicit theories of ratings and not on empirical data. Reanalysis of the rating data to allow both general 2nd- and 1st-order factors to be expressed in terms of item loadings recovered the structure present in the correlation of the original ratings as well as the psychological meanings of the 1st-order factors. General factors in rating data resemble general factors in measures of human ability. It is argued that removing general factors as if they were halo rather than true score may eliminate more of the variance from rating data than is justifiable. (8 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of rating format on goal-setting dimensions: A field experiment.

Atotal of 10 managers and their 62 subordinates participated in a field experiment that examined the effects of performance appraisal feedback deriving from two rating formats—behavioral observation scales (BOS) and graphic rating scales—on three goal-setting dimensions. Multivariate analyses of variance and a priori t tests revealed, as hypothesized, that the BOS format yielded significantly higher levels of goal clarity, goal acceptance, and goal commitment, the three goal-setting dimensions. Results are discussed in terms of systemic characteristics, that is, a "built-in" focus on specific behaviors, rather than a translation of generic performance dimensions into specific work behaviors. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Substantive dimensions of psychopathology derived from MMPI content scales and the Differential Personality Inventory.

Data from 282 male and 129 female state psychiatric patients admitted for acute alcoholism show convergence for MMPI and Differential Personality Inventory (DPI) scales in the areas of health problems, interpersonal conflict, depression and withdrawal, and familial problems. DPI scales yield somewhat more differentiated characterizations of psychopathology, particularly in the areas of impulse expression and denial. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparative reliability of numerically anchored versus job-task anchored rating scales.

Compared the effectiveness with which job-task anchored equal-appearing interval scales could be used in contrast with scales anchored only by simple numerical benchmarks. 2 groups of judges rated identical lists of job-task statements in terms of both types of scales. Ratings were made on 5 sensory/physical dimensions of job activities. The reliabilities of ratings for all scales were computed by an analysis of variance approach. In a test of statistical significance across all 5 scale dimensions, it was found that job-task anchored scales could generally be used with significantly greater reliability than simple numerically anchored scales. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Color versus shape coding in information displays.

8 Ss counted objects of a specified color or shape on displays of 20, 60, or 100 items. Counting time and errors increased with increasing display density. Counting based on a 5-valued color code was faster and more accurate than counting using any of 3 shape codes. Color counting was not affected by the particular shape code on which the colors were superimposed. Shape counting was somewhat faster and/or more accurate when color did not vary on the display, and vice versa. Differences in counting performance appeared among the 3 shape codes and among certain of the symbols within shape codes, and small differences were confirmed among the particular code colors used. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Salient-features coding in the translation between orthogonal stimulus and response dimensions.

Stimulus-response compatibility effects typically are attributed to translation processes. However, studies of compatibility effects obtained with orthogonal stimulus and response arrangements have attributed such effects to mechanisms other than translation processes. The present study evaluated whether recent theoretical advances regarding cognitive coding render a translation-based account viable for orthogonal stimulus-response arrangements. Across all experiments, a preference was shown for the mapping of up-to-right and down-to-left. This preference occurred regardless of whether the response set was manual or vocal, unimanual or bimanual, and whether the stimuli were spatial or symbolic. The results are consistent with the general literature on stimulus-response compatibility, which places the locus of most compatibility phenomena in translation processes. Moreover, the results support the notion that these translation processes operate according to a salient-features coding principle. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Primary therapist response modes: Comparison of six rating systems.

Six therapist response-mode rating systems were compared in order to delineate a set of primary modes that would best summarize the domain of therapist actions. Ratings of seven diverse therapy sessions showed that, in spite of differences in measurement assumptions and rater characteristics, interrater reliabilities generally were similar. When categories in different rating systems were collapsed to the same level of specificity, moderate to strong convergence was found for the six modes rated in all systems: question, information, advisement, reflection, interpretation, and self-disclosure. These modes discriminated among the seven contrasting therapeutic approaches. Each therapist was characterized by a unique pattern of response modes that differed significantly from the others. Researchers interested in assessing therapist in-session behaviors should consider incorporating measures that include these six modes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Work observation versus recall in developing behavioral examples for rating scales.

Used the retranslation method with 32 supervisors from a national airline to develop rating forms. 13 supervisors employed the work observation method and 19 used the recall method in collecting behavioral examples. Results indicate that the difference in method did not influence supervisor agreement in reclassifying behavioral examples with regard to (a) dimension of job performance illustrated or (b) level of performance illustrated. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Mixed standard rating scale: A consistent system for numerically coding inconsistent response combinations.

A revised system for numerically coding mixed standard scale (MSS) response combinations is proposed, and the psychometric implications of the new system are examined in the context of a comparative empirical study. The MSS ratings and graphic scale ratings obtained from 18 police supervisors for a total of 92 police patrol officers suggest that the proposed system does not substantially alter the findings of such comparative studies. The data further suggest that contradictions in the literature regarding the relative psychometric strengths and weaknesses of MSS ratings are probably not a function of the coding system originally proposed by F. Blanz and E. E. Ghiselli (1972) for translating MSS responses into numerical ratings. Adoption of the revised system is nevertheless recommended on the basis of (a) anticipated reliability increments and (b) greater "face validity" of the coding system, thereby rendering the MSS format potentially more acceptable to raters and ratees. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Perceiving group members: A comparison of derived and imposed dimensions.

Three studies with 54 18–55 yr olds compared 2 methods for spatially representing the perceived internal structure of small groups. R. F. Bales's (1980) Systematic Multiple Level Observation of Groups (SYMLOG) imposes 3 a-priori dimensions of interpersonal behavior—dominant vs submissive, friendly vs unfriendly, and instrumentally controlled vs emotionally expressive. Multidimensional scaling (MDS) derives dimensions on the basis of perceived similarity of group members. Each of the 3 studies demonstrated substantial and statistically significant overlap between the SYMLOG and MDS dimensions. In particular, the 1st MDS dimension was highly correlated with the axis of greatest differentiation in the SYMLOG space. Findings are discussed in terms of psychological salience and the behavioral attributes underlying the 3 SYMLOG dimensions. (46 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Asymmetry of coding versus noncoding strand in coding sequences of different genomes

We have used the asymmetry between the coding and noncoding strands in different codon positions of coding sequences of DNA as a parameter to evaluate the coding probability for open reading frames (ORFs). The method enables an approximation of the total number of coding ORFs in the set of analyzed sequences as well as an estimation of the coding probability for the ORFs. The asymmetry observed in the nucleotide composition of codons in coding sequences has been used successfully for analysis of the genomes completed at the time of this analysis.     

Systematically derived work dimensions: Factor analyses of the Occupation Analysis Inventory.

Developed the Occupation Analysis Inventory (OAI) for use as a taxonomic tool, derived a broad set of human work dimensions (factors) based on that questionnaire, and established some degree of construct validity for the resultant dimensions. 602 OAI work elements (items) were subjected to several factor analyses based on the ratings of 1,414 jobs on the elements and ratings of the elements on their requirements for 102 defined human attributes. The resultant factors were significantly related to the tested abilities of relevant job holders. It is concluded that job-rating factors should be (a) fairly reflective of the various types of work activities and conditions extant in the world of work and (b) unique in their coverage of information relevant to occupational education and career decision making. (79 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are rating dimensions in assessment centers transparent for participants? Consequences for criterion and construct validity.

Investigates whether the extent to which participants recognize rating dimensions in assessment centers has an effect on performance. Results showed that people who more accurately identify dimensions perform better. Convergent validity of dimension ratings is greater when participants accurately perceive that the same dimension is being evaluated in 2 exercises. In the discussion, the author considers how the ability to adequately perceive rating dimensions (i.e., transparency) may influence criterion validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effectiveness of performance feedback from behaviorally anchored rating scales.

Compared a behaviorally anchored rating scale (BARS) developed according to the procedure of H. Bernardin et al (see record 1976-08614-001) with a summated rating scale. Using both scale formats (designed to evaluate teaching effectiveness), 859 undergraduates rated 32 instructors during spring classes, and 314 undergraduates rated 19 instructors during summer classes. Students rated instructors halfway through the course and at its end. Instructors received feedback from the 1st rating period on either the BARS (including written behavioral observations) or on the summated rating format. Analyses of covariance revealed no format effect on the performance ratings obtained during the 2nd rating period in the spring experiment. BARS feedback, however, produced greater behavioral change among instructors than did feedback from the alternative format used during the summer experiment. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Integrating information from separable psychological dimensions.

This article examines decision processes in the perception and categorization of stimuli composed of the separable psychological dimensions, orientation and size. The randomization technique (F. G. Ashby and R. Gott; see record 1989-07388-001) of general recognition theory, which allows accurate estimation of a subject's decision boundary in a categorization task, is used in 4 experiments. Even though the stimulus components are clearly separable, it was found that Ss were not constrained to use separable response strategies, nor were they constrained to attend to distance to the prototypes. Instead, they used decision rules that were nearly optimal, even if this required information integration or for the Ss to attend to higher level category properties such as component correlation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Variable agreement between visual rating scales for white matter hyperintensities on MRI. Comparison of 13 rating scales in a poststroke cohort

BACKGROUND AND PURPOSE: Previous reports on the frequency, extent, and clinical correlates of white matter hyperintensities (WMHIs) have been contradictory. The purpose of this study was to test whether part of this variation could be explained by the different properties of the visual WMHI rating scales used. METHODS: The periventricular (PVHIs) and deep white matter (DWMHIs) hyperintensities of 395 poststroke patients were systematically analyzed and transformed to correspond to 13 different rating scales. The scales were compared with the use of Goodman-Kruskal measures of association. The relative frequencies, means, and medians of PVHI and DWMHI grades as well as Spearman rank correlations between WMHI grade and hypertension were calculated. RESULTS: At best more than 80% of the patients received an equivalent WMHI grade by different scales, but at worst the corresponding values were only 0.4% for PVHI and 18% for DWMHI. At best different scales categorized patients similarly in regard to WMHI grade, but at worst the corresponding values were 8% for PVHI and 57% for DWMHI ratings. The distribution of WMHI grades also varied, and when the effect of age on WMHI was assessed, some of the scales had a ceiling effect and some had a floor effect. Only 1 of the 7 PVHI, 5 of the 9 DWMHI, and 1 of the 3 combined rating scales showed a significant correlation with arterial hypertension, a putative risk factor for WMHIs. CONCLUSIONS: Some of the inconsistencies in previous studies of WMHIs are due to differences in visual rating scales. Our findings may warrant international debate regarding harmonization of WMHI ratings.     

The observation of specific affect in marital interactions: Psychometric properties of a coding system and a rating system.

The Specific Affect Coding System (SPAFF; J. M. Gottman & L. J. Krokoff, 1989) has led to conclusions about which types of dyadic affect predict positive and negative outcomes in marriage, yet the lack of information about collinearity among the codes limits interpretation of SPAFF results. Psychometric properties of SPAFF were examined by assessing the interactions of 172 newlywed couples with SPAFF and with an affect rating system developed for this study. For husbands and wives, factor analysis indicated 4 distinct factors of affect, representing anger/contempt, sadness, anxiety, and humor/affection. Anger/contempt and humor/affection were associated with marital satisfaction, relationship beliefs, and appraisals of the interactions. Correlations were in the expected directions. The strengths, limitations, and implications of the data are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)