Implicit theories and semantic similarities: Reply to Marsh and Groves (1987).

The role of implicit theories in behavioral judgments continues to be a source of debate. In a study exploring the factorial validity of student ratings of instruction, Cadwell and Jenkins (1985) suggested that students' implicit theories of instructor behavior may account for the robust factor structure underlying student ratings. They argued that students rely on the semantic similarity of the rating items, inflating the observed relations among the items. Marsh and Groves cite methodological and conceptual difficulties with the Cadwell and Jenkins study, but most of these difficulties may arise from Marsh and Groves's misconceptions about the role of implicit theories and semantic similarities and, more generally, the cognitive processes underlying judgment tasks. I review these issues and address various points of Marsh and Groves's critique. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student written comments: Dimensions of instructional quality.

Students' written comments to 4 open-ended questions used in end-of-semester student evaluations of 60 courses from 10 different fields, taught by instructors of different ranks, were content analyzed and classified into 22 categories. Approximately 50% of the comments were about the instructor, with 1 of 4 comments pertaining to the instructor's pedogogical skills. The degree of favorableness of written comments about the instructor and course was sufficiently convergent with ratings based on fixed alternative items measuring overall instructor performance and course quality. Students provided similar evaluations of course and instructor quality on both open-ended and fixed alternative items; nevertheless, each type of item provides useful and different information. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Congruency of student evaluative information collected by three methods.

Investigated the structural corroboration of instructional evaluation information collected from 1 source (students) by 3 different methods: student responses to objective questionnaire items, written student comments to open-ended questions, and group interview results. Pearson product-moment correlations calculated between instructor and course ratings across 14 class sections based on each of the 3 data collection methods revealed a high degree of structural corroboration of overall teaching performance and course quality. Results indicate that the 3 types of evaluative information examined presented a similar general impression of instructor quality. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student ratings and instructor-defined extenuating circumstances.

Examined the relationships between student ratings of instructor performance and 7 instructor-defined extenuating circumstances. Data were collected in 195 courses from a total of 3,625 students taught by 138 faculty members. For each extenuating circumstance, mean ratings of the group of faculty who said it applied to their course were compared with mean ratings of the group who said that it did not apply. The only extenuating circumstance that produced a statistically significant difference was "Class is too large to adequately present the material." Findings suggest that this set of instructors' rationalizations for not receiving better evaluations does not really explain them, even though there was a tendency for class ratings to be lower for the groups responding "Yes, the extenuating circumstances did apply to this course." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of students' evaluations of college teaching: A multitrait–multimethod analysis.

College instructors in 329 classes evaluated their own teaching effectiveness with the same 35-item rating form used by their students. There was student–instructor agreement in courses taught by teaching assistants (r?=?.46), undergraduate courses taught by faculty (r?=?.41), and graduate level courses (r?=?.39). Separate factor analyses of the student and instructor ratings demonstrated that the same 9 evaluation factors (e.g., work load, organization, interaction) underlay both sets of ratings. A multitrait–multimethod analysis supported convergent and divergent validity of these rating factors. Not only were correlations between student and instructor ratings on the same factors statistically significant for each of 9 factors, but correlations between their ratings on different factors were low. Findings demonstrate student–instructor agreement on evaluation of teaching effectiveness, support the validity of student ratings for both graduate and undergraduate courses, and emphasize the importance of using multifactor rating scales derived through the application of factor analysis. (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Using global student rating items for summative evaluation.

Research has established the multidimensional nature of student ratings of teaching, but debate continues concerning the use of multiple- vs single-item ratings for summative evaluation. In this study, the usefulness of global items in predicting weighted-composite evaluations of teaching was evaluated with a sample of 17,183 classes from 105 institutions. In separate regression analyses containing 2 global items—one concerning the instructor, the other concerning the course—each global item accounted for more than 50% of the variance in the weighted-composite criterion measure. Student, class, and method items generally added much less variance. Results suggested that, because global items accounted for a substantial amount of the variance, a short and economical form could capture much of the information needed for summative evaluation and longer diagnostic forms could be reserved for teaching improvement. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Relationship of student and instructor educational orientations with course ratings.

In a study with 359 undergraduates and 9 university faculty members, student and instructor educational orientations were examined in relation to course ratings in 9 classes. With 4 rating dimension scales (i.e., Teacher–Student Relationships, Course Objectives and Organization, Quality of Lectures, and Quality of Reading Assignments—Student Instructional Report), multiple regression analyses yielded an overall R–2 of 10 to 28%; instructor orientations were relatively more useful than student orientations in the regression equations. Spearman rhos between a measure of class–instructor orientation disparity and ratings were not significant. However, the negative sign of these coefficients is consistent with the congruence hypothesis of a person–environment interaction model. Discussion centers on the need for further study of student and instructor orientations and their "fit" in relation to outcomes for students. (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Gain–loss versus reinforcement–affect ordering of student ratings of teaching: Effect of rating instructions.

Investigated whether student rating instructions would influence the rank ordering by 83 undergraduates of 4 lecture quality sequences, (i.e., whether instructions influence primacy and recency effects). It was also investigated whether affect, self-esteem, and liking would show the same ordering for the 4 lecture quality sequences as ratings. In a laboratory analog of a classroom, using videotaped lectures, initial testing (good or poor Lecture 1), final teaching (good or poor Lecture 2), and student rating instructions (consider only Lecture 2, consider Lectures 1 and 2) were manipulated in a 2?×?2?×?2 design. Effects were measured on final ratings of the instructor, liking for the instructor, S affect, and S self-esteem. For Ss considering only Lecture 2, ratings and liking varied moderately and inversely with Lecture 1 quality (negative primacy effect) and greatly with Lecture 2 quality (positive recency effect), consistent with gain–loss theory. For Ss considering both Lectures 1 and 2, ratings and liking varied moderately with Lecture 1 quality (positive primacy effect) and greatly with Lecture 2 quality (positive recency effect), consistent with reinforcement–affect theory. Evidence failed to show that the effect of lecture quality on liking and ratings was mediated by affect or self-esteem. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student/instructor attitude similarity, student ratings, and course performance.

Investigated the validity of student ratings of instructional effectiveness by exploring the relations among student ratings, course performance, and the perceived attitude similarity of students and their instructors. 349 college students enrolled in 22 classes first rated instructor effectiveness on a 28-item rating form and then rated their own attitudes plus instructor attitudes on an attitude scale adapted from D. Byrne (1971). Another 296 college students enrolled in 19 classes rated instructor effectiveness and then their own attitudes plus instructor attitudes on an attitude scale adapted from F. E. Hofman and L. Kremer (see record 1980-33352-001). For both S groups, official end-of-term grades were also collected. Consistent with prior research, modest correlations were found between student ratings and perceived attitude similarity. There was also a modest relation between similarity and course grades. These relations were greatly reduced (56.2 and 44.5%, respectively) when the large influence of instructors on ratings and achievement was removed. Findings fail to support the claim that perceived attitude similarity is a substantial source of bias in student ratings. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Do teacher standards for assigning grades affect student evaluations of instruction?

Conducted 2 experiments to determine the effects of teacher standards for assigning grades on student ratings of teachers and student achievement. 143 undergraduates participated in Exp 1, in which instructor expressiveness (high, low), lecture content (high, low), and grading standards (B, C+, C) were factorially manipulated. In Exp II with 278 undergraduates, student incentive (high, low), instructor expressiveness (high, low), and grading standards (B+, B, C+, C) were factorially manipulated. Standards failed to affect student achievement in either experiment. Significant effects of standards on ratings were found, but not consistently for all types of ratings, instructors, or differences in standards. The size of the differences are considered to be relatively unimportant when ratings are used to make gross distinctions between teachers. (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Another look at Dr. Fox: Effect of stated purpose for evaluation, lecturer expressiveness, and density of lecture content on student ratings.

Investigated the extent to which the student's perception of the purpose for evaluating an instructor, the instructor's expressiveness, and the density of content presented in a lecture influenced student ratings and student achievement. 161 college students were randomly assigned to view lectures that systematically differed in lecturer expressiveness and density of content. The perceived purpose for evaluating the instructor had no effect on the Ss' ratings. All 5 student-rating subscale scores were significantly higher for the expressive lectures than for the nonexpressive lectures. On the dimension of instructor explanations, medium-content lectures received higher ratings than high-content lectures. (12 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity and reliability of scaffolded peer assessment of writing from instructor and student perspectives.

Although peer reviewing of writing is a way to create more writing opportunities in college and university settings, the validity and reliability of peer-generated grades are a major concern. This study investigated the validity and reliability of peer-generated writing grades of 708 students across 16 different courses from 4 universities in a particular scaffolded reviewing context: Students were given guidance on peer assessment, used carefully constructed rubrics, and were provided clear incentives to take the assessment task seriously. Distinguishing between instructor and student perspectives of reliability and validity, the analyses suggest that the aggregate ratings of at least 4 peers on a piece of writing are both highly reliable and as valid as instructor ratings while (paradoxically) producing very low estimates of reliability and validity from the student perspective. The results suggest that instructor concerns about peer evaluation reliability and validity should not be a barrier to implementing peer evaluations, at least with appropriate scaffolds. Future research needs to investigate how to address student concerns about reliability and validity and to identify scaffolds that may ensure high levels of reliability and validity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of teacher expressiveness: Mediated by teacher sex-typing?

Teacher expressiveness typically results in favorable student ratings, although teacher gender may affect these results. Eighty students viewed a videotape of either a male or female instructor acting either expressively or nonexpressively. Expressive instructors received the most positive student ratings, although expressiveness interacted with teacher gender on the rating of scholarship: The positive effect of expressiveness occurred only for the female instructor. Expressiveness significantly interacted with both teacher gender and student gender on achievement scores because of the differential effect of female expressiveness on male and female students. Student-perceived teacher sex-typing partially accounts for the effect of expressiveness on student ratings but not on student achievement. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Sex and gender in the college classroom: A quantitative analysis of faculty–student interactions and perceptions.

Students and instructors from 24 classrooms across 8 departments at a major university were observed in this study to (a) assess for sex differences in faculty–student interactions and in students' perceptions of their college classroom environment, (b) compare student perceptions of their college classroom interaction patterns with observed faculty–student interactions, and (c) assess a variety of demographic characteristics together to determine their singular and/or interactive effects on faculty–student interaction patterns and student perceptions. Male and female students did not differ in their classroom participation or perceptions, and instructors did not interact differently with the male and female students. Student perceptions strongly correlated with their own behaviors and with instructor behaviors. Classroom interactions and student perceptions varied on the basis of different demographic characteristics including instructor sex, class size, instructor monitoring of gender–race equity in the classroom, gender relevance of the course, and the sex ratio of the class. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Satisfaction with processes of collecting student opinions about instruction: The student perspective.

This study used a 2?×?2?×?2 factorial experiment to examine student satisfaction with eight processes of collecting student ratings of instruction by varying (a) method (group interviews vs. individual standardized rating forms), (b) timing (midterm vs. end of course), and (c) amount of instructor reaction to student ratings (restricted vs. extended). Consistent with predictions drawn from reactance and social comparison theories, students were more satisfied with interview methods at midterm followed by extended instructor reaction than with traditional approaches for collecting student opinions about instruction (i.e., standardized rating forms administered at the end of a course). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Personality, classroom behavior, and student ratings of college teaching effectiveness: A path analysis.

Tested the hypothesis that classroom teaching behavior mediates the relation typically found between personality and college teaching effectiveness. Colleagues rated 37 full-time college instructors on 29 personality traits, and trained observers assessed the frequency with which the same instructors exhibited 95 specific classroom teaching behaviors. Instructional effectiveness was measured by global end-of-term student ratings averaged over a 5-yr period. Path analyses revealed that approximately 50% of the relation between personality and teaching effectiveness was mediated by classroom behavior. The highly rated teacher was found to exhibit 2 types of personality traits: achievement orientation and interpersonal orientation. Results are discussed in terms of the validity of student ratings of teaching and in relation to M. J. Dunkin and B. J. Biddle's (1974) model of classroom teaching. (31 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of teacher sex, student sex, and teacher warmth on the evaluation of college instructors.

Examined the influence of teacher sex, student sex, and teacher warmth as perceived by students and teachers on teacher evaluation. Male and female instructors in the same department were matched on the level of course taught. 22 pairs of courses evaluated by 838 college students were obtained. Twenty Instructional Improvement Questionnaire items that directly evaluate instructor performance were analyzed using a 3-factor analysis of variance and the .0025 level of significance. No interactions between faculty sex, student sex, and teacher warmth were found. When Ss rated their instructor's interest and warmth, teachers who were warmer and primarily interested in students received higher ratings in teaching effectiveness. When teachers rated themselves on warmth and interest, self-ratings interacted with faculty sex. Generally, female teachers received higher effectiveness ratings than did male teachers when they considered themselves low in warmth or interested in course content. Male teachers who rated themselves high in warmth or primarily interested in students received higher ratings than did male teachers who rated themselves low in warmth or primarily interested in course content, respectively. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Grading leniency is a removable contaminant of student ratings

It is well established that students' evaluative ratings of instruction correlate positively with expected course grades. The authors identify 4 additional data patterns that, collectively, discriminate among 5 theories of the grades--ratings correlation. The presence of all 4 of these markers in student ratings data (obtained at University of Washington) was most consistent with the theory that the grades--ratings correlation is due to an unwanted influence of instructors' grading leniency on ratings. This conclusion justifies use of a statistical correction--illustrated here with actual ratings data--to remove the unwanted inflation of ratings produced by lenient grading. Additional research can profitably seek other inappropriate influences on ratings to identify more opportunities for validity-enhancing adjustments.     

Instructor reputation: An expectancy relationship involving student ratings and achievement.

Instructor reputation, defined as student expectations of an instructor's teaching ability, was investigated to determine its influence on student ratings and achievement. Two teaching behaviors, instructor expressiveness and lecture content, were combined with reputation in a 2–3 factorial design to assess interaction effects. 198 undergraduates read an introductory statement about an instructor (reputation), viewed a videotape lecture, rated the instructor, and took a test. Results indicate that reputation interacted with expressiveness but not content, in which students rated the positive, high-expressive instructor more favorably than the negative, high-expressive instructor. Results suggest a complex relationship in which reputation interacts with some teaching behaviors and affects some student outcomes. (24 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)