Evaluating the predictive validity of MCAT scores across diverse applicant groups

PURPOSE: To examine the predictive validity of MCAT scores, alone and in combination with other preadmission data, for medical students grouped by race/ethnicity and sex. METHOD: This study included two samples: 1,109 students who entered in 1992 any of the 14 medical schools participating in the MCAT Predictive Validity Study; and all 11,279 students who entered medical school in 1992 and took the USMLE Step 1 in June 1994. Criterion measures included each student's cumulative GPA in the first two years of medical school and his or her pass/fail status on Step 1. Differential predictive validity was examined by comparing prediction errors across racial/ethnic and sex groups. For cumulative GPA; residuals were compared, and for Step 1, classification errors were studied. RESULTS: The patterns of prediction errors observed across the groups indicated that, on average, (1) no difference between the sexes in prediction errors was evident; (2) performances of the three racial/ethnic minority groups tended to be overpredicted, with significant findings for Asians and Hispanics; and (3) Caucasians' performance tended to be underpredicted, although the magnitude of this underprediction was quite small. When USMLE Step 1 scores were the criterion for success in medical school, the majority of errors were overprediction errors. CONCLUSION: The authors caution that although MCAT scores, alone and in combination with undergraduate GPA, are good predictors of medical school performance, they are not perfect. The authors encourage future research exploring additional predictor variables, such as diligence, motivation, communication skills, study habits, and other relevant characteristics. Similarly, they indicate that high grades and Step 1 scores are not the only indicators of success in the medical profession and call for studies examining other important qualities, such as integrity, interpersonal skills, capacity for caring, willingness to commit to lifelong learning, and desire to serve in underserved areas.     

The Gateway program: ten-year lessons about outcomes and admission measures

The Gateway to Higher Education is a comprehensive program that provides selected minority students from five New York public high schools with rigorous high school preparation for college and professional careers in medicine, science, engineering, and technology. In existence for over ten years, the program has begun to accumulate solid outcomes data. The authors briefly describe the Gateway program and discuss students' graduation rates, career plans, and other outcomes. They then describe the correlation they have established between students' scores on the Stanford mathematics test (a standardized test administered to all Gateway ninth-grade students) and their subsequent SAT scores. Given the correlation recently established by the Association of American Medical Colleges between SAT scores and MCAT scores, this information provides another useful--and early--predictor of the future success in medical school of underrepresented minority students.     

Emergency medicine career change: associations with performances in medical school and in the first postgraduate year and with indebtedness

OBJECTIVE: Emergency medicine has been identified as the specialty that has gained the most young physicians who have changed their careers. To identify factors that may have contributed to such career changes, the authors compared the characteristics of three groups of physicians trained at their medical school: those who chose and stayed in emergency medicine, those who migrated into emergency medicine from other specialties, and those who moved out of emergency medicine. METHODS: A prospective longitudinal study was conducted. The sample consisted of physicians who chose emergency medicine as their careers at graduation and stayed in the specialty (n = 24), those who migrated from other specialties into emergency medicine (n = 51), and those who moved out of emergency medicine (n = 10). This sample was obtained from a total of 2,173 graduates of Jefferson Medical College between 1978 and 1987. The three groups of physicians were compared according to their academic performances both during medical school and after graduation. The dependent variables were freshmen and sophomore grade point averages (GPAs), written clinical examination scores, scores on National Board of Medical Examiners examination (Parts I, II, and III), and residency program directors' ratings. Age and indebtedness at medical school graduation and board certification status also were examined. RESULTS: Those physicians who stayed in emergency medicine and those who migrated from other specialties into emergency medicine had similar measures of academic performance, but both of these groups had higher academic performance measures and higher board certification rates than did the physicians who moved out of emergency medicine. Those who stayed in emergency medicine had the highest mean debt in the senior year of medical school. CONCLUSIONS: High academic performance and high indebtedness are factors associated with choosing or staying in the specialty of emergency medicine.     

Predicting performance during clinical years from the new Medical College Admission Test

The results of a predictive validity study of the new Medical College Admission Test (MCAT) using criteria from the clinical years of undergraduate medical education are presented and discussed. The criteria included course grades and faculty ratings of clerks in internal medicine, surgery, obstetrics and gynecology, pediatrics, and psychiatry; scores on a comprehensive test of clinical knowledge (including patient management type examinations); and scores on Part II of the examinations of the National Board of Medical Examiners (NBME). While the validity coefficients of the MCAT with the Part II examinations ranged from .03 to .47, they were higher than those of undergraduate grade-point averages with the same criteria. The implications of the small-to-medium size validity coefficients for admissions are discussed.     

Self-concept and peer acceptance in students with learning disabilities: A four- to five-year prospective study.

Examined the peer relations and self-concepts of students prior to and following their identification by the school district as learning disabled (LD) in a 4- to 5-yr prospective study. Self-concept ratings (kindergarten–4th grade) and peer acceptance ratings (kindergarten–3rd grade), as well as academic achievement scores, were compared across 3 groups: LD students who were placed in resource special education programs during 2nd grade, low-achieving (LA) students, and average-achieving/high-achieving (AA/HA) students. For peer acceptance, AA/HA students' scores were higher than LA students' scores only. No between-groups differences were obtained during any school year on the self-concept measure. Findings suggest that LD students' self-perceptions are not negatively affected by academic and social difficulties in the early grades or by the identification and labeling process. Though generalization is limited by the small sample size, few studies have examined students with learning disabilities longitudinally or prior to and following their identification. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of standardized patients and faculty in teaching medical interviewing

PURPOSE: This randomized controlled study compared the interviewing skills of first-year medical students receiving feedback primarily from standardized patients (SPs) with those of students receiving feedback primarily from faculty. METHOD: All 154 first-year students at the University of Oklahoma College of Medicine in 1993-94 were video-taped to assess baseline and post-instruction interviewing skills. All the students, randomized to one of three study groups, attended two four-hour workshops on interviewing skills. Instruction in the groups was as similar as possible except in the matter of who provided feedback. Two rating systems were used to rate the videotaped interviews for performances of targeted skills. RESULTS: Complete, usable data were available for 120 (78%) of the students. Skill ratings using the Arizona Clinical Interview Rating Scale were significantly higher for the "types of questions used" and "use of empathy" items in the SP-led feedback group. No significant difference in ratings was detected among the groups as measured by the Rotor Interactional Analysis System. CONCLUSION: The SPs were at least as effective as the faculty in effecting behavioral changes in the first-year medical students' interviewing skills.     

An evaluation of interviewed versus noninterviewed clinical psychology students.

Over an 8-yr period, half of the entering classes of 1 American Psychological Association-accredited clinical program were interviewed prior to admission. 35 interviewed and 40 noninterviewed students were compared. Faculty judges were asked to rate the strength of their probable endorsement of these students for internship, clinical employment, and academic employment. Results show no significant differences between groups on the ratings for internship, clinical position, or academic position, nor were there any group differences on traditional admissions criteria (e.g., Graduate Record Examination scores). (7 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The cat vertebral column: stance configuration and range of motion

PURPOSE: To evaluate whether clinical-teaching skills could be improved by providing teachers with augmented student feedback. METHOD: A randomized, controlled trial in 1994 included 42 attending physicians and 39 residents from the Department of Medicine at the Indiana University School of Medicine who taught 110 students on medicine ward rotations for one-month periods. Before teaching rotations, intervention group teachers received norm-referenced, graphic summaries of their teaching performances as rated by students. At mid-month, intervention group teachers received students' ratings augmented by individualized teaching-effectiveness guidelines based on the Stanford Faculty Development Program framework. Linear models were used to analyze the students' mean ratings of teaching behaviors at mid-month and end-of-month. Independent variables included performance ratings, intervention status, teacher status, teaching experience, and interactions with baseline ratings. RESULTS: Complex interactions with baseline performance were found for most teaching categories at mid-month and end-of-month. The intervention-group teachers who had high baseline performance scores had higher student ratings than did the control group teachers with similar baseline scores; the intervention group teachers who had low baseline performance scores were rated lower than were the control group teachers with comparable baseline scores. The residents who had medium or high baseline scores were rated higher than were the attending physicians with comparable baseline scores; the performance of the residents who had low baseline scores was similar to that of the attending physicians with comparable baseline scores. CONCLUSION: Baseline performance is important for targeting those teachers most likely to benefit from augmented student feedback. Potential deterioration in teaching performance warrants a reconsideration of distributing students' ratings to teachers with low baseline performance scores.     

Teachers' judgments of students' academic functioning: A comparison of actual and predicted performances.

The relationship between teachers' judgments of students' academic achievement and students' performances on an achievement test was investigated. The study included 12 teachers and 47 1st–4th graders students randomly selected from Wisconsin public schools. Teachers filled out the Academic Competence Scale from the Social Skills Rating System—Teacher version and 1 questionnaire for each student, which required teachers to predict how students would do on each item of the Kaufman Test of Educational Achievement, Brief form (K-TEA). Students were then administered the K-TEA by a qualified examiner. Teachers' judgments of students' academic achievement on the Academic Competence Scale were correlated moderately highly with students' actual K-TEA scores. Furthermore, mean percent agreement between teachers' item predictions and students' actual performances on the K-TEA was moderately high. Lastly, there was partial support for the prediction that teachers were better predictors of higher achieving than lower achieving students. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Neuropsychologic functioning of survivors of childhood medulloblastoma randomized to receive conventional or reduced-dose craniospinal irradiation: a Pediatric Oncology Group study

PURPOSE: The purpose of this study was to test the hypothesis that survivors of medulloblastoma who were younger at diagnosis and those who received standard-dose cranial irradiation (SRT) of 36 Gy would have a lower performance on standardized tests of cognitive function and achievement than children who were older and those treated with reduced-dose cranial irradiation (RRT) of 23.4 Gy. PATIENTS AND METHODS: Eligible patients had been treated on Pediatric Oncology Group (POG) study 8631 for low-risk medulloblastoma that randomized patients to receive RRT or SRT after surgical resection. Those who were alive and free of progressive disease 6.1 to 9.9 years from completion of treatment were eligible for this study. Of the 35 eligible patients, 22 patients (13 SRT, nine RRT) participated in a battery of tests that included intellectual and academic development as well as ratings of health-related quality of life. RESULTS: Patients were stratified by treatment group (SRT v RRT) and into younger (Y) and older (O) groups by the median age at diagnosis (8.85 years), which resulted in four groups that we hypothesized would show neuropsychologic test scores in the following order: Y/SRT less than Y/RRT less than O/SRT less than O/RRT. Evidence to support the hypothesized ordering of groups in terms of neuropsychologic toxicity was obtained with regard to Performance Intelligence Quotient (IQ), Full Scale IQ, Attention, Reading, and Arithmetic. CONCLUSION: Children treated for medulloblastoma experienced less severe neuropsychologic toxicity when treated with 23.4 Gy instead of 36 Gy cranial irradiation. Older children experienced less toxicity than children who were younger at the time of irradiation.     

Achievement strivings, scholastic aptitude, and academic performance: A follow-up to "Impatience versus achievement strivings in the Type A pattern."

A recent study of college students by J. T. Spence et al (see record 1988-09108-001) demonstrated that the Jenkins Activity Survey measure of the Type A pattern consisted of two relatively independent factors labeled Achievement Strivings (AS) and Impatience–Irritability (II). Scores on the AS scale but not the II scale were significantly correlated with cumulative grade point average (GPA) on the basis of 2 or more semesters of work. Follow-up data on the academic performance of these students are reported here. Correlations between AS scores and updated GPAs that were based on at least 4 semesters of work were comparable in value with those reported by Spence et al. Scholastic Aptitude Test (SAT) scores, obtained from students' records, were also significantly related to GPA. However, SAT and AS scores were nonsignificantly correlated and combined additively to account for a substantial portion of the variability in GPAs. The R–2s in four independent samples ranged from .22 to .36. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Correspondences among parent, teacher, and student perceptions of adolescents' learning disabilities

A group of 26 adolescents with learning disabilities (Grades 9 through 12), their parents, and their special education teachers were asked to rate the students' skills in each of 21 specific areas covering general ability, oral language, reading, written language, math, study skills, motivation, social skills, attention, and nonverbal skills. Correspondences in the absolute and relative ratings of parents, teachers, and students across the 21 skill areas were examined. The parents' ratings were consistent with those of the teachers in 16 areas and significantly lower than the teachers in 5 areas. The students' ratings were generally higher than those of their parents and teachers. The student-teacher differences were significant in 6 areas, whereas the student-parent differences were significant in 11 areas. Although generally lower in absolute terms than the ratings of their children, the parents' relative ratings were strikingly parallel to their children's ratings across skill areas (r = 80). Differences in the reference groups used for the ratings did not seem to account for the discrepant ratings. Possible implications of the differing perceptions of students' learning disabilities for students' self-esteem and academic progress are discussed.     

Should we spark interest in rotational atherectomy for in-stent restenosis?

PURPOSE: To determine whether it is possible for faculty to arrive at consistent, non-idiosyncratic grades in a problem-based learning (PBL) course. METHOD: Integrated Case Studies and Medical Decision Making (ICS) is the final course of the second year at the University of Pittsburgh School of Medicine. In ICS, 16 groups of nine students work in a PBL format over seven weeks. Each group is led by three faculty facilitators who, at the end of the course, independently give each student ratings for overall performance in the course and for each of seven performance categories. In 1993-94 and 1994-95, concordance in grades among the facilitators was determined by computing the intraclass correlation coefficients [ICC (3,1)] for the overall scores, the seven performance category scores, and all eight scores in aggregate. An ICC (3,1) of > or = .1 was considered indicative of statistically significant interrater concordance. An ICC (3,1) of > or = .7 was considered indicative of concordance of practical significance. RESULTS: Because the facilitators occasionally did not rate every student in every performance category, complete information was not available for all 32 groups. Statistically significant concordance was achieved in the aggregate scores in 100% of 23 groups, and in the overall scores in 90% of 18 groups. In six of the seven performance categories, concordance was achieved in at least 75% of the groups (n = 16-20). Practically significant concordance was achieved in the aggregate scores in 83% of 23 groups. CONCLUSION: The study results show that, given specific criteria by which to judge students' performances, it is possible to arrive at consistent, non-idiosyncratic grades for students in PBL courses.     

Average thresholds in the 8 to 20 kHz range in young adults

RATIONALE AND OBJECTIVES: The Comprehensive Clinical Assessment (CCA) examination at the University of Michigan Medical School is a series of test stations through which the mastery of clinical skills is evaluated. The purpose of this study was to determine whether student performance on the radiology station improved in years when radiology faculty presented case-of-the-week unknowns to the 3rd-year students. MATERIALS AND METHODS: The authors compared four separate classes of medical students in examination years 1993, 1994, 1995, and 1996 by using the total CCA examination score, the radiology station score, and radiology station pass/fail rates. Radiology case-of-the-week presentations were given by the radiology faculty only in academic years 1993-1994 and 1994-1995 (examination years 1994 and 1995). RESULTS: The means and standard deviations of the radiology station scores for the examination years when case-of-the-week presentations were not given, 1993 and 1996, were 78.92 +/- 13.62 and 79.76 +/- 13.62, respectively. In the years case-of-the-week presentations were given, 1994 and 1995, the radiology station scores averaged 90.83 +/- 8.58 and 89.97 +/- 9.66, respectively (P < .001, global alpha = .05). Total CCA percentage correct scores were similar for all years studied. In 1993 and 1996, 7.6% and 5.3% of students, respectively, failed the radiology station. In 1994 and 1995, 0.4% and 0% of students, respectively, failed (P < .0001). CONCLUSION: Case-of-the-week presentations by radiology faculty increased 3rd-year students' basic radiologic knowledge as evidenced by increased scores on the radiology station of the CCA examination.     

Age stereotyping at work: the role of rater and contextual factors on evaluations of job applicants

Using refinements of hypotheses by L. M. Finkelstein, M. J. Burke, and N. S. Raju (1995), the authors examined the effects of rater age, age salience, and job-relevant information on 324 managers' ratings of an older or a younger hypothetical applicant's interpersonal skills, economic worth, and likelihood of being interviewed. They hypothesized that age identity would interact with age salience to produce ingroup biases that would lead raters to favor workers from their own age groups. There was a main effect of target age on all dependent variables, with the strongest effects on the ratings of economic worth: The participants rated the older target as less economically beneficial than the younger target. When age was highly salient and when the raters identified psychologically with their age groups, older raters actually disfavored older workers in ratings of economic worth. The authors also discuss directions of future research into the roles of the target's economic worth and the rater's age identity in age stereotyping and age discrimination in employment decisions.     

Students' evaluations of a university counseling center and their intentions to use its programs.

Two representative samples of 801 university students were telephoned in 1977 and asked to evaluate a university counseling service and to indicate how likely they would be to participate in a variety of developmental workshops. Results indicate that both groups of Ss used the service with more regularity and with greater satisfaction than did students in the 1960s. Ss were most inclined to participate in workshops that assisted them in career planning, negotiating the university system, and coping with financial and academic concerns. Ss most receptive to specific services tended to be younger and undergraduate, to have lower GPAs, to live in a residence hall, and to be Black. (17 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the importance and validity of medical accreditation standards

In late 1997, the authors conducted a national survey of communities of interest about the importance and clarity of 44 accreditation standards applied to teaching, learning, and evaluation in medical schools by the Liaison Committee on Medical Education (LCME). Questionnaires were mailed to deans and educational administrators at U.S. medical schools; current LCME members and surveyors and those who had served during the preceding five years; a random selection of residency program directors drawn from both general practice and speciality disciplines; sample groups of medical students and residents; and a cohort of practicing physicians not affiliated with academic medical institutions. Altogether 1,659 questionnaires were mailed, and 701 responses were received (42%). The recipients were asked to use a five-point Likert scale to rate each of the 44 standards both for its perceived importance as an indicator of the quality of undergraduate medical education and for the clarity with which the standard's intent was conveyed. Although the mean ratings of importance all fell in the "moderately important" and "highly important" areas across the respondent groups, the ratings divided into three groups, semantically and statistically. At the high end for importance are standards dealing with fundamental qualities of students, instruction, and the structuring of resources. At the low end of the importance scale are standards dealing largely with matters of process. The ratings for clarity were systematically lower than the ratings for importance, and in some cases the rating for clarity were even more widely discrepant with the ratings for importance. Individual comments by respondents about certain standards were critical of their complicated construction and of confusion about their meaning and measures of compliance. One or more of these hallmarks--being rated of lower importance or clarity, and being the target of criticism by survey respondents--distinguished most of the standards that earlier study had shown are often neglected by surveyors. The predictive validity of each of a number of standards was examined by testing the association between the standard (or its derivative) and outcomes expressed in annual student and school questionnaires and compiled in databases of the Association of American Medical Colleges and the American Medical Association. The result was a mixed bag, confounded by the absence of specific dimensions of many accreditation standards (independent variables) and the lack of discriminating measures of outcome (dependent variables). Nevertheless, the LCME's accreditation standards are believed to be important by those most affected by them. And beyond validating that medical accreditation is guided by relevant standards for teaching, learning, and evaluation, the results of this study point to ways by which the process can be made more precise and useful.     

Personality scale validities increase throughout medical school.

Admissions and personnel decisions rely on stable predictor–criterion relationships. The authors studied the validity of Big Five personality factors and their facets for predicting academic performance in medical school across multiple years, investigating whether criterion-related validities change over time. In this longitudinal investigation, an entire European country’s 1997 cohort of medical students was studied throughout their medical school career (Year 1, N = 627; Year 7, N = 306). Over time, extraversion, openness, and conscientiousness factor and facet scale scores showed increases in operational validity for predicting grade point averages. Although there may not be any advantages to being open and extraverted for early academic performance, these traits gain importance for later academic performance when applied practice increasingly plays a part in the curriculum. Conscientiousness, perhaps more than any other personality trait, appears to be an increasing asset for medical students: Operational validities of conscientiousness increased from .18 to .45. In assessing the utility of personality measures, relying on early criteria might underestimate the predictive value of personality variables. Implications for personality measures to predict work performance are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Confidence in Individual and Group Decision Making: When "Two Heads" Are Worse Than One.

The authors investigated students' accuracy and confidence judgments for course-related material in college classrooms. Under conditions of group work and instructor feedback, students produced higher exam accuracy scores working in groups than alone but at a cost of increased confidence for groups' wrong answers. Groups' high confidence for wrong answers generated the case when "two heads are worse than one." Students participating in groups that arrived at wrong exam answers gave higher confidence when wrong and lower confidence when correct for repeated items on a final exam. "Two heads" groups when wrong had no adverse effect on students' accuracy for repeated exam items. An intervention of lecture and readings on confidence calibration, metamemory, and overconfidence did not improve the students' accuracy-confidence judgments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Motivational level and response to academic encouragement among low-achieving college males.

Investigated the relationship between academic achievement motivation and response to academic encouragement, using 2 different counseling letters. Ss were 150 male college freshmen who had received 1 or 2 failing grades in their 1st semester. Motivation was measured by Biggs' College Opinion Survey Questionnaire. No differences were found between the experimental letter groups and no-contact controls on either posttreatment GPA or on the number of helping sources contacted by the Ss. However, the high-motivation experimental Ss had significantly higher posttreatment GPAs and fewer "F" grades than low-motivation experimental Ss. There were no differences in posttreatment GPAs or number of "F" grades between high- and low-motivation controls. Implications for the role of the university counseling center in future attempts at helping low-achieving college students are discussed. (20 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)