Predicting performance during clinical years from the new Medical College Admission Test

The results of a predictive validity study of the new Medical College Admission Test (MCAT) using criteria from the clinical years of undergraduate medical education are presented and discussed. The criteria included course grades and faculty ratings of clerks in internal medicine, surgery, obstetrics and gynecology, pediatrics, and psychiatry; scores on a comprehensive test of clinical knowledge (including patient management type examinations); and scores on Part II of the examinations of the National Board of Medical Examiners (NBME). While the validity coefficients of the MCAT with the Part II examinations ranged from .03 to .47, they were higher than those of undergraduate grade-point averages with the same criteria. The implications of the small-to-medium size validity coefficients for admissions are discussed.     

Radiology faculty evaluation: bi-institutional implementation of an experimental appraisal instrument

RATIONALE AND OBJECTIVES: The authors evaluated the reliability and validity of an experimental radiology faculty appraisal instrument. MATERIALS AND METHODS: Residents from the University of Wisconsin (UW) and Indiana University (IU) were asked to use a previously developed behaviorally based, 53-item experimental faculty appraisal instrument. Twenty UW residents evaluated 29 UW faculty members and 37 IU residents evaluated 31 IU faculty members by using the experimental instrument. Residents also evaluated faculty by using their institution's existing appraisal instrument. RESULTS: Correlations between existing and experimental forms were .69 and .87 for UW and IU, respectively. Existing form reliabilities were .89 and .94 and experimental form reliabilities were .98 and .98 for UW and IU, respectively. Experimental form length was reduced to 30 items by eliminating the questions that correlated the least with section scores. Reliabilities of scores on the shortened form were .97 and .98 and correlated .65 and .88 with scores on the longer form for UW and IU, respectively. CONCLUSION: Ratings obtained with the existing forms correlated substantially with the experimental form, attesting to the experimental form's validity. Shortening the experimental form had a minimal effect on the reliability and validity of the data obtained. The behavior-based form was used to rate behaviors that residents believed discriminated between effective and ineffective instructors, enabling an objective and relevant assessment to be made.     

Chinese herbs: a clinical review of Astragalus, Ligusticum, and Schizandrae

BACKGROUND AND OBJECTIVES: The objective structured clinical exam (OSCE) is increasingly being used to evaluate student clinical performance. However, scant literature exists pertinent to this approach in evaluating family medicine clerkship performance. In this study, we assess 8 years' experience with a family medicine clerkship OSCE. METHODS: Eight annual clerkship OSCEs and the performance of 696 students are described. Comparisons of faculty evaluation, written exam, and OSCE performance are made for 335 students. Post-OSCE student and faculty feedback regarding OSCE validity and utility is also presented. RESULTS: Student performance is highest in medical history taking and physical examination and lowest in information-sharing stations. OSCE results appear to be relatively consistent on a year-to-year basis. OSCE, faculty evaluation, and written exam results have low overall levels of correlation, particularly in assessing performance that differs substantially from the mean. Students and faculty agree that the OSCE experience reflects skills that students should possess, but there is less agreement that the OSCE reflects clerkship-related learning and actual student performance. Both students and faculty derive insight from the OSCE regarding the definition of specific learning needs. CONCLUSIONS: The family medicine clerkship OSCE we describe appears to provide consistent measures of student performance. Although content validity is high, further assessment is needed to assure construct validity. The OSCE experience provides students with a rich resource for defining clerkship-related learning needs. Study results strongly suggest that OSCEs, faculty evaluations, and written exams provide differing measures of student performance. The reasons for these differences merit further exploration.     

Evaluation of students' clinical competence.

In this article we describe a clinical proficiency review required of 3rd-year graduate students, and we present data from a study of its psychometric properties and utility. Students' oral and written case presentations were evaluated on seven dimensions by a six-member panel composed of faculty and students. Moderately high interjudge reliabilities were achieved, although the level of agreement between rater pairs was related to the level of similarity of their theoretical orientation. Significant correlations were obtained between panel ratings and those of current field placement supervisors, but not with supervisor's ratings from prior or subsequent placements. Performance on the clinical proficiency review was relatively independent of academic performance measures such as course grades and scores on a six-part comprehensive examination. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Gamma interferon prevents diabetes in the BB rat

OBJECTIVE: Although increased evidence of disproportionate psychosocial risk and other health problems encountered by lesbian, gay male, and bisexual (LGB) youths has emerged, no study has described how the topic of homosexuality is addressed within child and adolescent residency psychiatry training. METHOD: Residency training directors in U.S. child and adolescent psychiatry programs were asked questions about instruction on the topic of homosexuality and the care of LGB patients, the department's view of whether homosexuality represents a pathological condition, the director's awareness of LGB colleagues and residents, and the director's opinion of LGB residents' disclosure of their homosexuality to their patients and patients' families. Asking similar questions facilitated a comparison of survey results with those of an earlier study of general psychiatry training directors. RESULTS: The reported departmental attitudes about whether homosexuality represents a pathological condition were essentially equivalent in general and child programs. Child and adolescent training directors were, however, less likely to have a favorable view of disclosure of sexual orientation to patients, less likely to know LGB residents or faculty, and less likely to report LGB residents an asset to their departments. CONCLUSIONS: The prediction that the majority of child and adolescent training programs would reflect a heightened awareness of the vulnerability of LGB youths was not confirmed.     

Performance of "standardized examinees" in a standardized-patient examination of clinical skills

PURPOSE: As a first step in testing the utility of using trained "standardized examinees" (SEs) as a quality-assurance measure for the scoring process in a standardized-patient (SP) examination, to test whether medical residents could simulate students in an SP examination and perform consistently to specified levels under test conditions. METHOD: Fourth-year students from the Baltimore-Washington Consortium for SPs participated in a National Board of Medical Examiners Prototype Examination of clinical skills consisting of twelve 15-minute student-patient encounters in 1994-95. For this examination, internal medicine residents were trained to act as ordinary candidates and to achieve target scores by performing to a set level on specific checklist items used by SPs for recording interviewing, physical-examination, and communication skills. The "strong SEs" were trained to score 80% correct on six of the examination's 12 cases (study cases), and the "weak SEs" were trained to score 40% correct on the same six cases. The strong and weak SEs' checklist scores on the study cases were compared through independent, two-tailed t-tests. When there was less than 85% agreement on specific checklist items in each case between the SE training and the SP recording, videotapes of the cases were reviewed; in such cases an SE's performance was the final score agreed upon after review. RESULTS: Seven SEs took the SP examination and were not detected by the SPs. There was a total of 84 discrepancies between predicted and recorded checklist scores across 659 checklist items in 40 encounters scored by the SPs. After correcting the discrepancies based on videotape review, the estimated actual mean score was 77.3% for the strong SEs and 44.0% for the weak SEs, and was higher for the strong SEs in each study case. The overall fidelity of the SEs to their training was estimated to be 97%, and the overall SP accuracy was estimated to be 91%. The videotape review revealed 47 training-scoring discrepancies, most in the area of communication skills. CONCLUSION: This study suggests that SEs can be trained to specific performance levels and may be an effective internal control for a high-stakes SP examination. They may also provide a mechanism for refining scoring checklists and for exploring the validity of SP examinations.     

Client initial disturbance and outcome in psychotherapy.

Evaluated 34 clients seen by 19 therapists in a counseling center by 6 different criteria of outcome, including pre- and posttherapy MMPI difference scores, and ratings by clients, therapists, and supervisors completed at the end of therapy. Measures of initial disturbance consisted of self-report inventories and ratings by therapists, supervisors, and judges. It was found that in general, none of the measures of rated disturbance were related to outcome criteria, but small to modest inverse relationships were obtained between the measures of subjective disturbance and global ratings of outcome. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity indices for the Heston Personal Adjustment Inventory.

For a sample of 88 female freshmen, scores on the Heston PAI were correlated with ratings (faculty, counselor, dormitory head) on six personality traits named after the Heston scales, and raters were asked to note certain types of behavior considered symptomatic of difficulty in adjustment to college. "The correlations… indicative of association between trait scores and faculty ratings are considerably lower than similar indices reported from other studies cited in the… Manual… . Further, the inventory failed to discriminate between students who seemed to have considerable difficulty adjusting to college and those who did not… one must conclude that the… Inventory is of limited practical value for predicting adjustment to college life." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validity of students' evaluations of college teaching: A multitrait–multimethod analysis.

College instructors in 329 classes evaluated their own teaching effectiveness with the same 35-item rating form used by their students. There was student–instructor agreement in courses taught by teaching assistants (r?=?.46), undergraduate courses taught by faculty (r?=?.41), and graduate level courses (r?=?.39). Separate factor analyses of the student and instructor ratings demonstrated that the same 9 evaluation factors (e.g., work load, organization, interaction) underlay both sets of ratings. A multitrait–multimethod analysis supported convergent and divergent validity of these rating factors. Not only were correlations between student and instructor ratings on the same factors statistically significant for each of 9 factors, but correlations between their ratings on different factors were low. Findings demonstrate student–instructor agreement on evaluation of teaching effectiveness, support the validity of student ratings for both graduate and undergraduate courses, and emphasize the importance of using multifactor rating scales derived through the application of factor analysis. (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Does clinical training facilitate feigning schizophrenia on the MMPI-2?

This study assessed the effect of knowledge of psychopathology and psychological testing on the ability to feign schizophrenia on the Minnesota Multiphasic Personality Inventory—2 (MMPI-2). Two groups of clinically trained participants—psychiatric residents and fellows (n?=?26) and clinical psychology graduate students (n?=?28)—and a group of undergraduate students with no clinical training (n?=?24) completed the MMPI-2 under instructions to feign schizophrenia. A sample of outpatients with schizophrenia (n?=?51) responded under standard instruction. Scores from the clinical scales and several validity indicators were compared. Although all participants asked to malinger had validity indicator scores indicating faking, the clinically trained participants produced generally lower scores on the clinical scales and validity indicators than did the undergraduate students. In particular, the scores for the clinical scales 6 (Paranoia) and 8 (Schizophrenia) were higher for all groups asked to feign schizophrenia when compared with the schizophrenic sample. The validity indicators F and the F-K index produced the largest effect sizes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The graduate record examination and selection of graduate students.

The sample was comprised of 50 students who entered the graduate program of the University of Florida between 1955 and 1959. The GRE Verbal and Quantitative aptitude scores were available on all of these students, and a complete undergraduate record on 38 of the 50 students… . Nine of the 11 full-time faculty were asked to rate each student." Only the GPA in math-science courses correlated significantly (.37) with the staff ratings. The "combination of X̄ GRE and GPA (math-science) yielded a multiple correlation of .46 with faculty ratings. Thus, combining these two predictors would be a definite improvement over selection based on either one alone." From Psyc Abstracts 36:04:4KJ48R. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Objective Structured Clinical Examination (OSCE) in the clinical clerkship: an overview

The Objective Structured Clinical Examination (OSCE) for student assessment is well established, with an extensive body of research documenting that this is a valid means to assess clinical skills that are fundamental to the practice of medicine. The OSCE consists of a circuit of stations which tests a range of skills and learning to assess undergraduate medical students. A well-constructed OSCE provides important information about candidate performance and the quality of training. It is used at the University of South Dakota School of Medicine (USDSM) in assessment of third year medical students during their Obstetrics Clerkship, and as a teaching tool in the Pediatric Clerkship. On August 10, 1996, the USDSM administered an OSCE for the first time to third year medical students. The purpose of this article is to present state of the art information about setting up OSCE based on our recent experience and to provide practical examples of OSCE questions which can be addressed in the clinical setting. The narrative, references and examples give guidelines for the preparation of OSCE testing. The OSCE provided a standardized way of assessing clinical competence. Both students and faculty were very satisfied with the examination, and felt that the material tested was relevant and appropriate. The OSCE process does serve to identify areas of weakness in the curriculum and/or teaching methods, and thus can serve as a mechanism to improve educational effectiveness.     

Construct validity of multiple job performance measures using confirmatory factor analysis.

The construct validity of job performance ratings obtained from self, supervisors, and peers was assessed relative to performance data obtained from an objective test of task proficiency. Data were gathered from a sample of 256 first-term jet engine mechanics in the U.S. Air Force. Widaman's (1985) approach to assessment of convergent and discriminant validity of multitrait–multimethod data using confirmatory factor analysis to contrast alternative models was used. As expected, all analyses revealed support for a model with correlated performance traits and uncorrelated methods. Significant convergence among the three rating sources was found. In addition, convergence was found between task ratings and scores from the objective proficiency test. Significant discriminant validity was found in all of the analyses. Results revealed that self, supervisors, and peers can be equally valid as sources of performance information, although performance of some tasks may be more validly evaluated by one source than another. Needs for further research on the effects of task characteristics on the validity of ratings are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Student perceptions of and expectations for male and female instructors: Evidence relating to the question of gender bias in teaching evaluation.

253 undergraduates completed a course evaluation questionnaire that included formal teaching performance ratings, perceptual orientation scales, and indicators of degree and context of student–instructor contact. Ss represented 11 female-instructed courses and 28 male-instructed courses. Male and female instructors were found to be placed within a unitary perceptual frame of reference. Female instructors were perceived as warmer and more potent individuals, but were required by the Ss to offer greater interpersonal support and were judged more closely than male instructors in providing it. Results suggest that although direct gender bias may not be observed in formal student evaluations of their instructors, female faculty members are nonetheless subject to culturally conditioned gender stereotypes. (22 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Marital Satisfaction Questionnaire for Older Persons.

Conducted 5 studies to construct and psychometrically evaluate a marital satisfaction questionnaire for older persons. The questionnaire content was generated and refined in Study 1, temporal stability was evaluated in Study 2, and criterion-related validity was examined in Study 3. Norms, factor structure, and construct validity of the questionnaire were established in Study 4. The degree to which scores on the questionnaire predicted observer ratings of marital interactions during an analog problem-solving situation was examined in Study 5. The final questionnaire consists of 24 items with 1 major and 2 minor factors. Factor scores and a total Marital Satisfaction Scale score were found to be homogeneous, temporally stable, and significantly correlated with multiple measures of marital adjustment, life satisfaction, perceived spouse behaviors, and observer ratings of marital communication. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Validating an indirect measure of clarity of feelings: Evidence from laboratory and naturalistic settings.

This study analyzed the reliability and validity of an indirect measure of clarity of feelings that is based on response latencies (RTs) of mood ratings. Fifty-two participants completed a laboratory session and an experience-sampling week with 6 measurement occasions per day. Shorter RT of mood ratings measured in the laboratory (but not self-reported dispositional clarity) predicted higher overall mood regulation success during the experience-sampling week. As a new indirect ambulatory measure of clarity, RTs of mood ratings were measured on handheld devices during the experience-sampling week. The new ambulatory RT measure of clarity demonstrated good psychometric properties. Within-occasions reliability (internal consistency) was satisfactory, and between-occasions reliability (consistency of aggregated scores) was high. Ambulatory RT of mood ratings demonstrated moderate to high convergence with RT of mood ratings measured in the laboratory session. Both RT measures were unrelated to self-reported dispositional clarity of feelings. However, momentary RT converged with a self-report measure of momentary clarity on the within-persons level: Participants were faster to rate those mood items that they were more certain about. Evidence for the predictive validity of the new ambulatory RT measure was provided by the finding that on the within-persons level, shorter RT (but not self-reported momentary clarity) predicted higher mood regulation success and better mood at subsequent measurement occasions. The results suggest that RT of mood ratings can be used as a reliable and valid indicator of an individual's clarity of feelings in laboratory and experience-sampling studies. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Personality, classroom behavior, and student ratings of college teaching effectiveness: A path analysis.

Tested the hypothesis that classroom teaching behavior mediates the relation typically found between personality and college teaching effectiveness. Colleagues rated 37 full-time college instructors on 29 personality traits, and trained observers assessed the frequency with which the same instructors exhibited 95 specific classroom teaching behaviors. Instructional effectiveness was measured by global end-of-term student ratings averaged over a 5-yr period. Path analyses revealed that approximately 50% of the relation between personality and teaching effectiveness was mediated by classroom behavior. The highly rated teacher was found to exhibit 2 types of personality traits: achievement orientation and interpersonal orientation. Results are discussed in terms of the validity of student ratings of teaching and in relation to M. J. Dunkin and B. J. Biddle's (1974) model of classroom teaching. (31 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Average thresholds in the 8 to 20 kHz range in young adults

RATIONALE AND OBJECTIVES: The Comprehensive Clinical Assessment (CCA) examination at the University of Michigan Medical School is a series of test stations through which the mastery of clinical skills is evaluated. The purpose of this study was to determine whether student performance on the radiology station improved in years when radiology faculty presented case-of-the-week unknowns to the 3rd-year students. MATERIALS AND METHODS: The authors compared four separate classes of medical students in examination years 1993, 1994, 1995, and 1996 by using the total CCA examination score, the radiology station score, and radiology station pass/fail rates. Radiology case-of-the-week presentations were given by the radiology faculty only in academic years 1993-1994 and 1994-1995 (examination years 1994 and 1995). RESULTS: The means and standard deviations of the radiology station scores for the examination years when case-of-the-week presentations were not given, 1993 and 1996, were 78.92 +/- 13.62 and 79.76 +/- 13.62, respectively. In the years case-of-the-week presentations were given, 1994 and 1995, the radiology station scores averaged 90.83 +/- 8.58 and 89.97 +/- 9.66, respectively (P < .001, global alpha = .05). Total CCA percentage correct scores were similar for all years studied. In 1993 and 1996, 7.6% and 5.3% of students, respectively, failed the radiology station. In 1994 and 1995, 0.4% and 0% of students, respectively, failed (P < .0001). CONCLUSION: Case-of-the-week presentations by radiology faculty increased 3rd-year students' basic radiologic knowledge as evidenced by increased scores on the radiology station of the CCA examination.     

Dimensions of perceived competence in professional psychology.

Appraised levels of general professional competence for 126 trainees in professional psychology by supervisors who had worked closely with the students. After rating the students, supervisors were asked to describe the dominant characteristics of "outstanding" and "incompetent" trainees. The most commonly mentioned quality of outstanding practitioner-trainees was high intelligence. The most commonly mentioned characteristic of incompetent trainees was lack of knowledge. A rating schedule composed of the 28 most commonly used terms was then employed by supervisors in rating students the following year. Four factors—Professional Responsibility, Interpersonal Warmth, Intelligence, and Experience—emerged from analysis of intercorrelations among variables. Intercorrelations of factor scores with ratings of overall competence showed that behaviorally oriented supervisors gave less weight to warmth in evaluating general competence than did supervisors with psychodynamic or eclectic viewpoints. (14 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interviewer validity as a function of interview structure, biographical data, and interviewee order.

Investigated effects of degree of interview structure, presence or absence of interviewee biographical information, and interviewee order on interviewer validity. 54 undergraduates in personnel management and 36 social-worker supervisors serving as judges rated 6 currently employed social workers assuming the role of job applicants in videotaped interviews. Low validities, calculated using interview ratings and performance-measure criterion scores based on a job analysis of the social-worker position, were obtained in all treatment conditions. Analysis of variance revealed that only interviewee order had an appreciable effect on interviewer validity. Analysis of the interview ratings revealed the presence of halo error and low interrater reliability. Possible design limitations and future research suggestions are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)