Double, double, toil and trouble: Pitfalls of suppositional inquiries of journal ratings.

Three recent comments in the September 1976 issue (Buss & McDermott; Levin & Kratochwill; Porter (see record 1990-57250-001; see record 1990-57248-001; and see record 1990-57249-001), attempting to deal with the difficult area of assessing journal "reputations," raised some interesting questions concerning our study reporting journal rankings (Koulack & Keselman, November 1975; see record 1976-24649-001). We are in agreement with Buss and McDermott (1976) that citations and rankings might not be measuring the same things, but we are in disagreement with Porter, who suggests that "fine ordering among journals is whimsical" (p. 675). In fact, as we suggest in our introduction and have demonstrated in the body of our article (Koulack & Keselman, 1975), journal rankings change as a function of type of work and area of interest. Perhaps Porter's (1976) findings might be a bit whimsical because of the procedure he used to obtain his correlations. Moreover, it is impossible to probe further because Porter does not present the rankings of the two journals chosen from the APA members' top 50, which appeared in either of the citation measures' top 50. Such data might provide some insight into the low correlations obtained between journal citations and rankings. For example, extremely low citation rankings on either citation index for these two journals, given their relatively high position in the APA membership rankings, would diminish the size of the correlation coefficients. The Levin and Kratochwill (1976) comment is somewhat annoying because it distorts a line from Shakespeare as well as misrepresents our presentation. They imply that (a) we thought our rankings represented a definitive approach to the journal rating problem, (b) we neglected to place emphasis on a table presented in the paper, and (c) respondents chose to ignore our instructions and in fact, rated journals on the basis of familiarity. In conclusion, we appreciate the fact that there are numerous ways of examining journal reputations (e.g., rankings by departmental chairpersons, rankings by APA membership, citations obtained from 77 psychology journals published in 1969, citations obtained from 3 psychology journals published from July 1973 to June 1975) and that each of them has potential value. However, comments that are not based on empirical investigation, such as those of Levin and Kratochwill (1976), are mere suppositions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Ratings and misratings: A reply to Mace and Warner.

Comments on K. C. Mace and H. D. Warner's (1973) study purporting to assess the extent of agreement among psychologists in their rating of journals. The authors criticize the initial selection of journals, the rating procedure, and the selection raters. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the form of ROCs constructed from confidence ratings.

A classical question for memory researchers is whether memories vary in an all-or-nothing, discrete manner (e.g., stored vs. not stored, recalled vs. not recalled), or whether they vary along a continuous dimension (e.g., strength, similarity, or familiarity). For yes-no classification tasks, continuous- and discrete-state models predict nonlinear and linear receiver operating characteristics (ROCs), respectively (D. M. Green & J. A. Swets, 1966; N. A. Macmillan & C. D. Creelman, 1991). Recently, several authors have assumed that these predictions are generalizable to confidence ratings tasks (J. Qin, C. L. Raye, M. K. Johnson, & K. J. Mitchell, 2001; S. D. Slotnick, S. A. Klein, C. S. Dodson, & A. P. Shimamura, 2000; and A. P. Yonelinas, 1999). This assumption is shown to be unwarranted by showing that discrete-state ratings models predict both linear and nonlinear ROCs. The critical factor determining the form of the discrete-state ROC is the response strategy adopted by the classifier. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Supervisory ratings and attitudes.

Do supervisors tend to be biased in their ranking and rating of workers? 42 janitors were ranked and rated by 4 supervisors and rated by the division chief. The highest and lowest were compared by 10 factor scores. "Supervisors of our hospital maintenance men valued most the abilities least prevalent in the men (leadership and executive ability) and valued least the attributes most abundant in these men (social adjustment and personal charm). (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

On the nature of self-monitoring: Construct explication with Q-sort ratings.

To explicate M. Snyder's (1987) construct of self-monitoring (SM), a new Q-sort prototype is introduced. Analyses of Q-sorts by both observers and self demonstrated cross-method convergent validity for the revised 18-item Self-Monitoring Scale (SMS-R) and its Public Performing subscale; however, neither scale showed discriminant validity against measures of extraversion. The Other-Directedness items remaining on the SMS-R correlated neither with the other measures of SM nor with extraversion. These findings suggest that the scale revision led to a conceptual shift toward extraverted (and away from other-directed) features of self-presentation. To adequately assess the conceptual domain of SM phenomena, researchers should administer the original 25-item SMS (not the abbreviated 18-item SMS-R) and score Public Performing and Other-Directedness separately to examine their individual and joint effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater bias in performance ratings: Superior, self-, and peer ratings.

Leniency errors, halo effects, and differential dimensionality were explored in an analysis of superior, self-, and peer performance ratings of 107 managerial and 76 professional employees in a medium-sized manufacturing location, representing 95% of the managerial and professional staff. Self-ratings showed greater leniency effects than superior or peer ratings. A multitrait–multimethod (MTMM) analysis supported the presence of strong halo effect and significant convergent validity but not discriminant validity. The dimensional analysis supported the presence of strong halo effect. A statistical control procedure for the halo effect was developed that involved calculating residuals of the performance items, controlling for the "overall effectiveness" variance component in each item. The procedure did not reduce the significant halo effect, nor did it improve the nonsignificant discriminant validity in the MTMM analysis. It did, however, clarify the dimensional structure of ratings by superiors. Data from 4 previously published studies were also reanalyzed using the statistical control procedure. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Initial decisions and subsequent performance ratings.

Examined how information initially encountered for one decision was later used for evaluating ratee performance. 101 undergraduates viewed videotaped performances of 4 male carpenters performing 4 tasks. Half of the Ss were instructed to view the performance with the intent of rating the deservedness of each worker for outside contracting work; the other half were asked to designate the one best worker for this work. Two days later, 70 of the 101 Ss were asked to rate (from memory) each worker's performance. Findings indicate that raters initially designating one worker for a positive outcome rated all workers higher than raters making initial deservedness ratings. This elevation in ratings, which occurred for both subsequent overall evaluations and task ratings, may have reflected both the tendency to inflate ratings given the individual receiving the initial treatment and leniency toward the other ratees for whom representations of ability may not have been well established or remembered. Overall delayed performance ratings were influenced by these initial contextual factors. Results suggest that the pattern in which information is presented to raters and the nature of previous decisions may affect memory-based performance ratings. (26 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student ratings and instructor-defined extenuating circumstances.

Examined the relationships between student ratings of instructor performance and 7 instructor-defined extenuating circumstances. Data were collected in 195 courses from a total of 3,625 students taught by 138 faculty members. For each extenuating circumstance, mean ratings of the group of faculty who said it applied to their course were compared with mean ratings of the group who said that it did not apply. The only extenuating circumstance that produced a statistically significant difference was "Class is too large to adequately present the material." Findings suggest that this set of instructors' rationalizations for not receiving better evaluations does not really explain them, even though there was a tendency for class ratings to be lower for the groups responding "Yes, the extenuating circumstances did apply to this course." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On averaging judges' ratings to increase their correlation with an external criterion.

It is well-known that the ratings of judges become more reliable as more judges' ratings are averaged. Furthermore, more reliable measures are expected to be more valid. However, empirical data do not exist which show that the averaged ratings of more judges are in fact more valid. In this study, 2 groups of 10 judges (clinicians and students) rated a patient's mood from speech samples taken at different times during psychotherapy. The judges' ratings were averaged and then correlated with an objective measure of anxiety based upon the patient's speech disruptions. The correlations did increase (up to a theoretical asymptote) as the number of judges increased. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A reply to Mace and Pittenger on "Directly perceiving Gibson".

A reply to Mace and Pittenger on "Directly Perceiving Gibson" (see record 2005-09162-001). W. M. Mace and J. B. Pittenger address themselves to a false issue when they argue that J. J. Gibson has both a theory of stimulus structure and a theory of attention. The point of the 1972 article by J. W. Gyr is that these two theories are not as well integrated as they might be or as integrated as Gibson intended them to be. Moreover, a point also stressed by Gyr and overlooked by Mace and Pittenger is that Gibson's findings of stimulus structure are also consistent with theories that are not theories of direct visual perception, and therefore such findings by themselves cannot be used to support Gibson's theory. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The journal wedge.

Describes how peripheral (e.g., subdisciplinary) psychological foci lead to the creation of new organizational divisions and journals, and warns against the potential unintended consequence of driving a wedge, i.e., the "journal wedge," between related fields of inquiry. One suggestion for lessening the impact of the journal wedge is for organizations to maintain more complete and up-to-date home pages on the World Wide Web, well linked to other home pages and accessible to nonmembers, so that it would be easier for nonmembers to keep up with fields of concern. If organizations can get and keep the peripheral psychologist interested by providing information of this sort, they may be better able to recruit new members who may then ultimately help control journal costs as well as other aspects of organizational operating budgets. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Cognitive categorization and quality of performance ratings.

The effects of cognitive categorization of raters on accuracy, leniency, and halo of performance evaluations were investigated in a field setting. One hundered seventy-four subordinates evaluated the performance of their managers on three performance dimensions. Managers were categorized as congruent or incongruent based on subordinates' perceptions of the extent to which the manager's behavior met the subordinates' expectations. The results indicated that the quality of ratings assigned by subordinates was related to the cognitive categories used. As hypothesized, ratings of managers who were categorized as congruent were found to be more accurate and also to contain more leniency and halo tendency than the ratings of managers who were categorized as incongruent. Implications of these findings for performance-appraisal research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Coder gender and potential for hostility ratings.

This study examined the effects of coder gender on Potential for Hostility ratings. Six trained coders (3 men and 3 women) who were unaware of the coder gender effect hypothesis coded 30 male and 30 female undergraduates for Potential for Hostility. Although reliability estimates as calculated by Cronbach's alpha suggested that all coders were consistent, an analysis of variance revealed a significant main effect for coder gender, wherein female coders rated participants as displaying significantly less Potential for Hostility than did male coders. This significant difference was also meaningful, as coder gender accounted for 32% of the variance in Potential for Hostility scores. Thus, future Potential for Hostility investigations need to consider the gender of those coding, as this factor both significantly and substantially influences reported Potential for Hostility ratings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rater and technique contamination in criterion ratings.

Rater contamination may occur when the same rater makes both prediction and criterion ratings; technique contamination may occur when both the prediction and criterion ratings are on the same form. 400 Army officers provided ratings on a graphic scale and two versions of a forced-choice scale. Each officer rated 20 of his fellow officers on the graphic scale, and eight days later rerated two of them on the graphic scale and one of the forced-choice scales. Intercorrelations for the same raters on the same (graphic) form were .69-.82. Inter-correlations for the same rater on different forms were .52-.57. For different raters, inter-correlations were below .3, for same or different forms. It is concluded that the rater is the principal source of contamination. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Audio tape ratings: Comments and guidelines.

Audiotape ratings play an important role in research and evaluation in the counseling profession. It is commented that procedures in obtaining such ratings have been inconsistent and, in some cases, methodologically unsound and that these inconsistencies are due in part to the lack of published guidelines for obtaining audiotape ratings. Guidelines for 4 major components of the audiotape rating process are offered. (44 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Success, failure, and self-assessment ratings.

Investigated the effects of success and failure experiences on subsequent self-ratings of positive and negative social-psychological characteristics. 60 teacher trainees were assigned to success, failure, or control experiences on 3 achievement tasks ostensibly testing intellectual ability. Posttest minus pretest gains were assessed on variables of self-reported self-confidence, self-worth, opportunity, efficacy, and true unto self, which were categorized as positive dimensions. Similar gains were assessed on variables of self-reported need for isolation, worried and afraid, need for help, troubles and obstacles, and cynicism, categorized as negative dimensions. As hypothesized, Ss who experienced success made significantly greater gains in positive self-assessments, and failure Ss made significantly greater gains in negative self-assessments. No significant Success-Failure * Sex interaction was found. Implications for clinicians, psychotherapists, and procedures employed in self-rating scales are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Navigating student ratings of instruction.

Many colleges and universities have adopted the use of student ratings of instruction as one (often the most influential) measure of instructional effectiveness. In this article, the authors present evidence that although effective instruction may be multidimensional, student ratings of instruction measure general instructional skill, which is a composite of 3 subskills: delivering instruction, facilitating interactions, and evaluating student learning. The authors subsequently report the results of a meta-analysis of the multisection validity studies that indicate that student ratings are moderately valid; however, administrative, instructor, and course characteristics influence student ratings of instruction. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Accuracy of accurate empathy ratings.

Studied the accuracy of accurate empathy ratings among 31 counseling graduate students by comparing 2 sets of ratings made under different experimental conditions with each other and with a predefined criterion of accuracy. 10 counselor-client exchanges representing excerpts from a single interview were deliberately constructed to represent 2 interactions at each level of the 5-point Accurate Empathy Scale. 2 groups of raters differing in amount of prior client information rated the exchanges. Data indicated that the 17 informed raters were significantly more accurate in their ratings than 14 noninformed raters and significant discrepancies in ratings occurred at the higher scale points of the Accurate Empathy Scale. Implications and suggestions for using this scale are discussed. (17 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some early measurements and ratings in psychology.

Major discussion headings are statistics, vision, memory, attention, thought, probability, and esthetics. "Quantitative data concerning vision may be found as early as the literature of the 18th century… .All early measurements of memory were concerned with cases of extraordinary memory.… In the very middle of the 18th century, a little work appeared which apparently represents the 1st attempt to measure the 'velocity of thought.'… The 1st reliably known psychological measurements and ratings made in the interests of science do not go back further than the beginning of the 18th century." (PsycINFO Database Record (c) 2010 APA, all rights reserved)