Impact of work experience on the validity of student evaluations of teaching effectiveness.

Examined whether student evaluations of teaching effectiveness changed significantly once the students had left the university and had gained work experience. Students in their final year at a university were asked to grade lecturers on 9 items relating to teaching effectiveness and course effectiveness; 723 students responded. After the students had graduated and had been working for 1 yr, they were asked again to give a rating of teaching effectiveness for their former lecturers and to nominate the "best" and "worst" teachers; 506 replied. Comparisons of the results show that there was little difference in the 2 ratings of lecturers by their teaching scores. Increasing maturity and job experience did not significantly alter student evaluations of relative teaching effectiveness. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effectiveness of a counseling intervention program for teaching career decision-making skills.

Developed a career decision-making model and program to provide a counseling intervention and teaching materials to aid students in improving their career maturity and decision-making skills. The model and program were piloted and evaluated to determine program effectiveness. 334 academic and nonacademic student groups were part of a quasi-experimental–control design to measure career maturity and decision making. Data indicate that students in the program significantly increased their Career Maturity Inventory scores, career maturity was significantly related to decision making, and more nonacademic program than academic groups increased their decision-making scores. Students reported they found the program useful in evaluating future career choices. (30 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Students' evaluations of university teaching: Dimensionality, reliability, validity, potential baises, and utility.

Reviews findings and research designs used to study students' evaluations of teaching effectiveness. The focus of the investigation is on the author's own research that has led to the development of a students' evaluations of educational quality measure, but it also incorporates a wide range of other research. Based on this overview, class-average student ratings are (a) multidimensional; (b) reliable and stable; (c) primarily a function of the instructor who teaches a course rather than the course that is taught; (d) relatively valid against a variety of indicators of effective teaching; (e) relatively unaffected by a variety of variables hypothesized as potential biases; and (f) seen to be useful by faculty as feedback about their teaching, by students for use in course selection, and by administrators for use in personnel decisions. It is suggested that in future research a construct validation approach should be used in which it is recognized that effective teaching and students' evaluations designed to reflect it are multifaceted, that there is no single criterion of effective teaching, and that tentative interpretations of relations with validity criteria and with potential biases must be scrutinized in different context and examine multiple criteria of effective teaching. (136 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student evaluations of teaching measure the intervention, not the effect.

Comments on the article by W. J. McKeachie (see record 1997-43129-006) that discusses 4 additional articles in the November 1997 issue of American Psychologist concerning student evaluations of teaching. The current author does not comment on the consensus belief that certain forms of student ratings are useful for improving teaching. Rather, he addresses the dangerous entrenched doctrine that considers student ratings or measurements of teaching behaviors to be the best way to measure or define teaching effectiveness. An mathematical alternative to relying on students' evaluations is presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Do university teachers become more effective with experience? A multilevel growth model of students' evaluations of teaching over 13 years.

Do university teachers, like good wine, improve with age? The purpose of this methodological/substantive study is to apply a multiple-level growth modeling approach to the long-term stability of students' evaluations of teaching effectiveness (SETs). For a diverse cohort of 195 teachers who were evaluated continuously over 13 years (6,024 classes, an average of 30.9 classes per teacher), there was little evidence that teachers became either more or less effective with added experience. This stability of SETs generalized reasonably well over undergraduate- and graduate-level courses, early career teachers, and teachers who differed substantially in terms of their teaching effectiveness overall. Whereas there were substantial individual differences between teachers in terms of their teaching effectiveness, these individual differences were also highly stable over time. Although highly supportive of the use of SETs for many purposes, the results provide a serious challenge for existing programs that assume that SET feedback alone is sufficient to improve teaching effectiveness. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Use of student ratings to benchmark universities: Multilevel modeling of responses to the Australian Course Experience Questionnaire (CEQ).

Recently graduated university students from all Australian Universities rate their overall departmental and university experiences (DUEs), and their responses (N = 44,932, 41 institutions) are used by the government to benchmark departments and universities. We evaluate this DUE strategy of rating overall departments and universities rather than individual teachers, and we juxtapose it with the traditional use of student ratings to evaluate individual teachers (SETs). Multilevel analyses of DUE overall ratings were not able to discriminate well between universities or departments—few universities or departments differed significantly from the grand mean. Although the a priori 5-factor structure for this DUE instrument was reasonably well-defined at the individual student level, none of the 5 factors separately or in combination discriminated well between departments or universities. In contrast to this pattern of results, we review studies showing that SETs do reliably differentiate between teachers and are valid in relation to many criteria of effective teaching. However, casual reviews of these research literatures should not use this support for SETs to justify the use of DUE-type strategies. We conclude that DUE-type ratings should be used with great caution, if at all, and should not be seen as an alternative to SETs. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Validity of students' evaluations of college teaching: A multitrait–multimethod analysis.

College instructors in 329 classes evaluated their own teaching effectiveness with the same 35-item rating form used by their students. There was student–instructor agreement in courses taught by teaching assistants (r?=?.46), undergraduate courses taught by faculty (r?=?.41), and graduate level courses (r?=?.39). Separate factor analyses of the student and instructor ratings demonstrated that the same 9 evaluation factors (e.g., work load, organization, interaction) underlay both sets of ratings. A multitrait–multimethod analysis supported convergent and divergent validity of these rating factors. Not only were correlations between student and instructor ratings on the same factors statistically significant for each of 9 factors, but correlations between their ratings on different factors were low. Findings demonstrate student–instructor agreement on evaluation of teaching effectiveness, support the validity of student ratings for both graduate and undergraduate courses, and emphasize the importance of using multifactor rating scales derived through the application of factor analysis. (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On "Construct validity of measures of college teaching effectiveness."

In their recent study Howard, Conway, and Maxwell (1985) proffered evidence of the validity of student evaluations of teaching effectiveness, particularly the convergence of student and former-student ratings. The authors claimed that these two measures are "especially effective evaluation methods" (Howard et al., 1985, p. 187). In this article, I raise questions on conceptual and methodological grounds about the quality of the Howard et al. validation evidence. The main points of concern are whether (a) the Howard et al. evidence can support the hypothesis that their student evaluation measure is tapping a construct other than teaching effectiveness and (b) the convergent validation offered in their article more closely resembles reliability analysis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Teaching evaluations: Concerns for psychologists?

Comments on articles in the November 1997 issue of American Psychologist concerning student evaluations of teaching performance. It is argued that the articles highlight the fact that there is still much to learn about the validity of these measures and how they might best be used to improve teaching and personnel decision making. It is contended that there may be unique issues facing teachers in psychology that call for the exploration of teaching evaluations in a different light. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student and department chairman views of the performance of university professors.

Compared evaluations of students for 713 university courses taught over 4 semesters by 347 professors to rankings made by department chairmen of their faculty. The faculty were ranked by professional visibility, current research, teaching impact, communication ability, and departmental contributions. 16 of 27 rhos computed for visibility and student evaluations of teaching were negative. A substantial number of relationships for research were around zero. Relationships for teaching and communications were moderately positive. One-year stability coefficients of rankings by chairmen were high for a single chairman but considerably lower when a change of chairman took place. In a chairman's view, research and visibility are highly related, but effective teaching is only moderately related to these performance criteria. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Current concerns are past concerns.

Replies to comments by J. S. Armstrong (see record 1998-11971-007), D. Buck (see record 1998-11971-008), J. Friedrich (see record 1998-11971-009), and R. E. Redding (see record 1998-11971-010) regarding articles in the November 1997 issues of American Psychologist concerning student evaluations of teaching (SETs) (see records 1997-43129-002, 1997-43129-003, 1997-43129-004, and 1997-43129-005). The current authors contend that the challenges explored in the comments are weak and suggestions are problematic. Abrami and D'Apollonia cite their review of research evidence from multisection validity studies and found that ratings explain instructor impacts on student learning to a moderate extend (corrected r?=?.47). They also state that the relationship between ratings and teacher-produced learning is somewhat variable. They also contend that it is harmful to disregard teaching effectiveness as a criterion for promotion and tenure, urge prudence in decision making, and advocate the use of ratings to make only gross distinctions regarding teaching effectiveness. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of teacher sex, student sex, and teacher warmth on the evaluation of college instructors.

Examined the influence of teacher sex, student sex, and teacher warmth as perceived by students and teachers on teacher evaluation. Male and female instructors in the same department were matched on the level of course taught. 22 pairs of courses evaluated by 838 college students were obtained. Twenty Instructional Improvement Questionnaire items that directly evaluate instructor performance were analyzed using a 3-factor analysis of variance and the .0025 level of significance. No interactions between faculty sex, student sex, and teacher warmth were found. When Ss rated their instructor's interest and warmth, teachers who were warmer and primarily interested in students received higher ratings in teaching effectiveness. When teachers rated themselves on warmth and interest, self-ratings interacted with faculty sex. Generally, female teachers received higher effectiveness ratings than did male teachers when they considered themselves low in warmth or interested in course content. Male teachers who rated themselves high in warmth or primarily interested in students received higher ratings than did male teachers who rated themselves low in warmth or primarily interested in course content, respectively. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Teaching tomorrow's psychology students: Who pays the piper?

Pressures likely to influence the teaching of psychology in the 1980s include the declining number of 18-yr-old students, student interest in clinical psychology, the emergence of professional schools of psychology, the awarding of credit for life experience to midcareer adult students, and the tightening of the job market for doctoral students. The relevance of these pressures to the use of student evaluations of teaching, to the training of graduate students, and to the undergraduate curriculum is discussed. (6 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of professional versus student ratings of job interviewee behavior.

Designed a replication of research by T. McGovern and H. Tinsley (1978) which asked professional personnel representatives to rate a videotaped job interviewee. Nonprofessional undergraduates were used as the raters of job interviewee behavior. Content of the interviews was constant; nonverbal behavior was experimentally manipulated. The results obtained with the student raters closely parallel those of professionals. The student Ss were able to discriminate nonverbal behaviors such as eye contact, voice tone, and body movement in making their evaluations. The implications for future research with students as Ss and the development of peer counseling programs for job interview preparation are discussed. (9 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interindividual consistencies in determining behavior-based dimensions of teaching effectiveness.

Two experiments examined the issue of the universality of meanings of behavior-based teaching effectiveness dimensions within and across subgroups of undergraduates classified by sex, age, and major and a group of teachers. In Exp I, 565 pertinent critical incidents were gathered and categorized by students and 3 teachers. Exp II utilized 1,963 incidents independently categorized by 14 students. An unstructured situation format was used to avoid predetermination of dimensions by the researcher. The resulting dimensions were subjected to overlap analysis, yielding a measure of the extent of agreement among sets of critical incidents assigned to dimension. Results indicate a low extent of agreement within and between subgroups of categorizers. However, cluster analysis indicated that the present procedure is capable of identifying dimensions of effective performance and exhibiting consensus regarding interpretations of actual behaviors. Disagreements found between teacher and student subgroups in abstracting incidents into statements imply the desirability of having students define dimensions of effective teaching when they are to be the ultimate raters. (27 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student evaluations of college professors: Are female and male professors rated differently?

Over 1,000 male and female college students of 16 male and female professors (matched for course division, years of teaching, and tenure status) evaluated their instructors in terms of teaching effectiveness and sex-typed characteristics. Male students gave female professors significantly poorer ratings than they gave male professors on the six teaching evaluation measures; their ratings of female professors were poorer than those of female students on four of the six measures. Female students also evaluated female professors less favorably than male professors on three measures. Student perceptions of a professor's instrumental/active and expressive/nurturant traits, which were positively related to student ratings of teaching, accounted for only a few of these gender-related effects. Student major and student class standing also played a role in the evaluation of professors. The importance of gender variables in teacher evaluation studies is discussed, and implications for future research are noted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of teacher, student, and class characteristics on the evaluation of college instructors.

Investigated whether student evaluations of faculty would be affected by the characteristics of the teacher, the student, and the class. Each instructor at a single university was asked to answer questions indicating personal warmth, professorial rank, years of teaching experience, sex, and class size. Students in 174 classes were asked to complete the Instructional Improvement Questionnaire (IIQ). The 20 questions on the IIQ that directly evaluate instructor performance were analyzed. Only the results for the 1st set of canonical functions are presented. An instructor who received high scores on this canonical function would be rated as encouraging student participation in the course, showing an interest in students, knowing when students understood her or him, being available to students, increasing appreciation for the course, and accepting criticism and suggestions. The classes that received high values on this function were small in size, were taught by instructors who rated themselves as warm, and had students that expected high grades. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The proposal to certify nurse anesthetists

RATIONALE AND OBJECTIVES: The application of information technology to teaching radiology will profoundly change the way learning is mediated to students. In this project, the integration of veterinary medical students' knowledge of sonography was promoted by a computer-assisted instruction program and a subsequent practical class. METHODS: The computer-assisted instruction program emphasized the physical principles of clinical sonography and contained simulations and user-active experiments. In the practical class, the students used an actual sonographic machine for the first time and made images of a tissue-equivalent phantom. Students' responses to questionnaires were analyzed. RESULTS: On completing the overall project, 96% of the students said that they now understood sonographic concepts very or reasonably well, and 98% had become very or moderately interested in clinical sonography. CONCLUSIONS: The teaching and learning initiatives enhanced an integrated approach to learning, stimulated student interest and curiosity, improved understanding of sonographic principles, and contributed to an increased confidence and skill in using sonographic equipment.     

Effectiveness of an interactive map display in tutoring geography.

Evaluated the teaching effectiveness of different aspects of the SCHOLAR computer-assisted instruction system. The experiment compared how well students learn using SCHOLAR with (a) the interactive map display of Map-SCHOLAR, (b) a static labeled map, and (c) an unlabeled map. The 9 high school and 9 college student Ss learned significantly more with the interactive map display than with either the labeled map or the unlabeled map. A new method called backtrace analysis was used to assess the effectiveness of specific aspects of the tutoring strategy and the map system used in the experiment. (PsycINFO Database Record (c) 2010 APA, all rights reserved)