Using working memory theory to investigate the construct validity of multiple-choice reading comprehension tests such as the SAT.

When taking multiple-choice tests of reading comprehension such as the Scholastic Assessment Test (SAT), test takers use a range of strategies that vary in the extent to which they emphasize reading the questions versus reading the passages. Researchers have challenged the construct validity of these tests because test takers can achieve better-than-chance performance even if they do not read the passages at all. By using an individual-differences approach that compares the relative power of working memory span to predict SAT performance for different test-taking strategies, the authors show that the SAT appears to be tapping reading comprehension processes as long as test takers engage in at least some reading of the passages themselves. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of study strategies for passages: Rereading, answering questions, and generating questions.

Students are often encouraged to generate and answer their own questions on to-be-remembered material, because this interactive process is thought to enhance memory. But does this strategy actually work? In three experiments, all participants read the same passage, answered questions, and took a test to get accustomed to the materials in a practice phase. They then read three passages and did one of three tasks on each passage: reread the passage, answered questions set by the experimenter, or generated and answered their own questions. Passages were 575-word (Experiments 1 and 2) or 350-word (Experiment 3) texts on topics such as Venice, the Taj Mahal, and the singer Cesaria Evora. After each task, participants predicted their performance on a later test, which followed the same format as the practice phase test (a short-answer test in Experiments 1 and 2, and a free recall test in Experiment 3). In all experiments, best performance was predicted after generating and answering questions. We show, however, that generating questions led to no improvement over answering comprehension questions, but that both of these tasks were more beneficial than rereading. This was the case on an immediate short-answer test (Experiment 1), a short-answer test taken 2 days after study (Experiment 2), and an immediate free recall test (Experiment 3). Generating questions took at least twice as long as answering questions in all three experiments, so although it is a viable alternative to answering questions in the absence of materials, it is less time-efficient. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Test anxiety and intellectual performance.

High school students were administered 2 anxiety scales, the Test Anxiety Scale and a Need for Achievement scale. Scores on these tests were related to scores on the School and College Ability Test (SCAT). Test anxiety was found to be negatively correlated with SCAT scores. The negative correlations obtained tended to be larger for female than for male Ss. The Need for Achievement scale showed only a slight tendency to correlate negatively with SCAT scores. The results were interpreted as being consistent with the conception of anxiety as an interfering nonintellectual influence on intellectual performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The Positive and Negative Consequences of Multiple-Choice Testing.

Multiple-choice tests are commonly used in educational settings but with unknown effects on students' knowledge. The authors examined the consequences of taking a multiple-choice test on a later general knowledge test in which students were warned not to guess. A large positive testing effect was obtained: Prior testing of facts aided final cued-recall performance. However, prior testing also had negative consequences. Prior reading of a greater number of multiple-choice lures decreased the positive testing effect and increased production of multiple-choice lures as incorrect answers on the final test. Multiple-choice testing may inadvertently lead to the creation of false knowledge. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Stereotype Threat Research and the Assumptions Underlying Analysis of Covariance.

Comments on an article by Paul Sackett, Chaitra Hardison and Michael Cullen entitled On Interpreting Stereotype Threat as Accounting for African American-White Differences on Cognitive Tests (see record 2004-10043-001). Sackett, Hardison, and Cullen discussed the role of covariates in Steele and Aronson's (see record 2000-16592-021) seminal research on the effects of stereotype threat on scores of African American test takers. Besides highlighting some common misinterpretations that stem from the use of covariance-adjusted means in reporting Steele and Aronson's (Study 2) experimental results, Sackett et al. argued that these results indicate that Black-White testscore difference within the no-stereotype threat (i.e., nondiagnostic) condition actually reflects the test score difference on the SAT (i.e., the covariate). This implies that stereotype threat effects add to the often found Black-White test score gap instead of partly accounting for it (Sackett et al., 2004). Here the author comments on the use of analysis of covariance (ANCOVA) in stereotype threat (ST) experiments, because ST theory implies violations of the assumptions underlying ANCOVA. Such violations could result in incorrect Type I error rates and distortions in the adjustment of means. Because of this, ANCOVA appears inappropriate for analyzing (quasi-) experimental results of ST research. In addition, the interpretation proposed by Sackett et al. of Steele and Aronson's results may be due to distortions of mean adjustments caused by violations of model assumptions. While avoiding technical detail, the author provides the assumptions underlying ANCOVA and discuss why these assumptions do not sit well with several aspects of ST theory. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Stereotype Threat and the Social and Scientific Contexts of the Race Achievement Gap.

Comments on an article by Paul Sackett, Chaitra Hardison and Michael Cullen entitled On Interpreting Stereotype Threat as Accounting for African American-White Differences on Cognitive Tests (see record 2004-10043-001). In their article, Sackett, Hardison, and Cullen (see record 2000-16592-021) critiqued misrepresentations of the original stereotype threat findings presented by Steele and Aronson. They criticized representations of the research that suggest that stereotype threat explains all the racial achievement gap in academic performance when, in fact, the original studies statistically equated the ability of Black students and White students by using SAT scores as a covariate. As Sackett et al. acknowledged, Steele and Aronson did not claim that stereotype threat explains all the racial achievement gap, though as they suggested in their critique, it may have been a claim made implicitly and even explicitly in some media and textbook coverage of the work. The authors of this comment wish to make three points that Sackett and colleagues did not make. These points highlight the social and scientific contexts in which Sackett et al.'s critical commentary, and stereotype threat research in general, can be interpreted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The relation of ninth grade test scores to twelfth grade test scores and high school rank.

From a population of high school students who had been tested in state-wide programs at the ninth and twelfth grade levels, a sample of 2185 was used to compute correlations between performance at these scholastic levels. At the ninth grade the tests were the ACE (Highschool Edition) and the Cooperative English Test (Form Y); at the twelfth grade level the tests were the ACE (College Edition) and the Cooperative English Test (Form S). HS percentile ranks were also computed for the Ss. The correlation between different forms of the same test was .8, for each test. The correlations between ACE forms and the Coop forms were .7 or higher. HSR correlated .63 with ninth grade ACE score, .71 with ninth grade English. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effects and perceived consequences of testing accommodations on math and science performance assessments.

The present study examined the effect accommodations have on test results of students with and without disabilities and documented experts’ judgments about the appropriateness of testing accommodations. Test score data were collected from 218 fourth-grade students with and without disabilities on mathematics and science performance tasks and from eight testing experts who evaluated the fairness and validity of a sample of testing accommodations used with these students. Results indicated that, for most students with disabilities and some students without disabilities, packages of testing accommodations had a moderate to large effect on performance task scores. Expert reviewers rated most accommodations for a student with disabilities as being both valid and fair, and they gave accommodations listed on a student’s individualized education program (IEP) significantly higher validity and fairness ratings than accommodations that were not listed on the student’s IEP. Interpretations of these data are provided and implications for practice and future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The influence of test familiarity and student disability status upon teachers' judgments of students' test performance.

Two questions motivated this study: (a) Does test familiarity influence teachers' judgments of their students' test performance? and (b) Does the disability status of students influence their teachers' judgments? Teachers (n=19) judged item performances for one student with disabilities and one student without disabilities (n pairs=19) from their fourth-grade classrooms. Teachers made judgments using (a) a mathematics test from the research version of the TerraNova CTBS Multiple Assessments edition, which is similar to the large-scale achievement test administered in numerous states, and (b) classroom-based math tests. Judgment accuracy was higher (a) on classroom tests and (b) for students without disabilities. Among less accurate judgments, teachers consistently underestimated the performances of students with disabilities. Students with disabilities performed lower on both types of tests. Student test performance accounted for a significant proportion of the variance in teacher judgment accuracy. Implications of these findings for future research and practice are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of successful and unsuccessful students in the medical school at the University of Minnesota.

Test performance of 215 passing medical students was compared with that of 29 drop-outs because of scholastic failure. Mean score differences were computed for 9 measures, including high school rank, 2 honor-point ratio (grade) measures, ACE, Coop English Test, Medical College Admission Test, Minn. Medical Aptitude Test, SVIB, MMPI. Successful students make significantly higher pre-med course grades, Minn. Medical Aptitude Test scores, ACE scores, and lower scores on the L-scale of the MMPI. The other variable yielded only uncertain or no differentiation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Detection of feigned ADHD in college students.

Significant motivations and incentives exist for young-adult students to seek a diagnosis of attention-deficit/hyperactivity disorder (ADHD). With ADHD information readily accessible on the Internet, today's students are likely to be symptom educated prior to evaluation. This may result in false-positive diagnoses, particularly when students are motivated to convey symptoms. We evaluated the utility of ADHD symptom checklists, neurocognitive tests, and measures initially developed to detect feigned neurocognitive or psychiatric dysfunction (symptom validity tests [SVTs]). The performance of 31 undergraduates financially motivated and coached about ADHD via Internet-derived information was compared to that of 29 ADHD undergraduates following medication washout and 14 students not endorsing symptomatology. Results indicated malingerers readily produced ADHD-consistent profiles. Symptom checklists, including the ADHD Rating Scale and Conners's Adult ADHD Rating Scale–Self-Rating Form: Long, were particularly susceptible to faking. Conners's Continuous Performance Test—II findings appeared more related to motivation than condition. Promising results were seen with all cognitive SVTs (Test of Memory Malingering [TOMM], Digit Memory Test, Letter Memory Test, and Nonverbal–Medical Symptom Validity Test), particularly TOMM Trial 1 when scored using Trial 2 criteria. All SVTs demonstrated very high specificity for the ADHD condition and moderate sensitivity to faking, which translated into high positive predictive values at rising base rates of feigning. Combining 2 or more failures resulted in only modest declines in sensitivity but robust specificity. Results point to the need for a thorough evaluation of history, cognitive and emotional functioning, and the consideration of exaggerated symptomatology in the diagnosis of ADHD. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Age appropriateness and motivation, engagement, and performance in high school: Effects of age within cohort, grade retention, and delayed school entry.

The present study examined the relative salience of age within cohort, grade retention, and delayed school entry (3 dimensions of age appropriateness) in 3,684 high school students' academic motivation, engagement, and performance. Structural equation modeling revealed that after the effects of demographic characteristics and grade retention were taken into account, little significant variance was explained by the linear effects of age within cohort. However, subsequent modeling incorporating nonlinear effects showed that the markedly older-for-cohort students (i.e., over the "standard" 12-month age range for a given cohort) and delayed-entry students (i.e., academic "red shirts") experienced some academic disadvantage in motivation, engagement, and performance while the age-appropriate students (particularly the younger ones) fared best. Over and above demographic and age-within-cohort effects, the effects of grade retention were consistently negative. Taken together, data suggest that there appear to be little or no motivation, engagement, or performance advantages to being markedly older-for-cohort, having delayed-entry status, or being retained in a grade. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Social comparison and big-fish-little-pond effects on self-concept and other self-belief constructs: Role of generalized and specific others.

Two studies integrate the big-fish-little-pond effect (BFLPE; negative effects of class-average achievement on academic self-concept, ASC), which is based upon educational psychological research, with related social psychological research that is based on social comparison theory. Critical distinctions are the nature of the social comparison processes that are based on generalized-other (class- or school-average) or individual (target comparison classmate) comparisons, and the nature of self-belief constructs that invoke normative (social comparison) or absolute frames of reference. In a large cross-national study (26 countries; 3,851 schools; 103,558 students), school-average ability negatively affected ASC but had little effect on 4 other self-belief constructs that did not invoke social comparison processes. In Study 2 (64 classes; 764 students), 2 sources of social comparison information (class-average achievement and achievement of an individually selected target comparison classmate) each had distinct, substantial negative effects on agency self-beliefs that invoked social comparison processes but not on metacognitive responses that did not invoke these processes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Unskilled, unaware, or both? The better-than-average heuristic and statistical regression predict errors in estimates of own performance.

People who score low on a performance test overestimate their own performance relative to others, whereas high scorers slightly underestimate their own performance. J. Kruger and D. Dunning (1999) attributed these asymmetric errors to differences in metacognitive skill. A replication study showed no evidence for mediation effects for any of several candidate variables. Asymmetric errors were expected because of statistical regression and the general better-than-average (BTA) heuristic. Consistent with this parsimonious model, errors were no longer asymmetric when either regression or the BTA effect was statistically removed. In fact, high rather than low performers were more error prone in that they were more likely to neglect their own estimates of the performance of others when predicting how they themselves performed relative to the group. (PsycINFO Database Record (c) 2010 APA, all rights reserved)