Effects of information reliability in predicting task performance using ability and effort.

Past research concluded that predictions of performance are a multiplicative function of ability and effort, although these studies were not designed to test between the averaging and multiplying models. The present study with 65 undergraduates competitively tested these models by manipulating the reliability of information about effort and ability. Results show that the greater the reliability of the ability information, the greater was its effect on judged performance. Similarly, the greater the reliability of the effort information, the greater was its effect on judged performance; in addition, the greater the reliability of 1 type of information, the less the effect of the other type of information. Findings are inconsistent with a multiplying model, but they are consistent with an averaging model in which the reliability of information influences its weight. Individual differences in weighting of effort and ability information were also found, and these differences were predictable from self-reports of the relative importance of the variables. (37 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effect of dimensional salience and salience of variability on problem solving: A developmental study.

Presented a matching task to 120 6-20 yr olds to investigate the relative influence of dimensional salience and salience of variability on problem solving. The task included 4 dimensions: form, color, number, and position. On each problem, 1 dimension was relevant and the other 3, which could be either constant or variable, were irrelevant for solution. Ss were asked to find which of 2 stimuli was more similar to a standard stimulus. Results show the same dimensional hierarchy for both reaction time and error scores. The effect of variability, as measured by reaction time and error scores, increased with increasing number of variable irrelevant dimensions and decreased with age. The effect of variability proved to be greater than the effect of dimensions. The implications of the relative influence of dimensional salience and salience variability for problem solving are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

One determinant of the capacity to free-associate in psychotherapy.

The hypothesis tested was that "clients whose behavior when asked to free associate is highly variable, flexible, or productive exhibit significantly greater variability when judging the extent of autokinetic movement on successive trials than Ss whose behavior at the outset of psychotherapy is highly inflexible, stereotyped, or rigid." 7 neurotic college student analysands in the extremely flexible group were matched with 10 Ss of the extremely rigid group for relevant personal variables. Sherif's autokinetic apparatus was used, and each S's judgmental variability was determined. The highly significant difference between autokinetic variability scores of the 2 groups lends strong support to the hypothesis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Revival of test bias research in preemployment testing.

We developed a new analytic proof and conducted Monte Carlo simulations to assess the effects of methodological and statistical artifacts on the relative accuracy of intercept- and slope-based test bias assessment. The main simulation design included 3,185,000 unique combinations of a wide range of values for true intercept- and slope-based test bias, total sample size, proportion of minority group sample size to total sample size, predictor (i.e., preemployment test scores) and criterion (i.e., job performance) reliability, predictor range restriction, correlation between predictor scores and the dummy-coded grouping variable (e.g., ethnicity), and mean difference between predictor scores across groups. Results based on 15 billion 925 million individual samples of scores and more than 8 trillion 662 million individual scores raise questions about the established conclusion that test bias in preemployment testing is nonexistent and, if it exists, it only occurs regarding intercept-based differences that favor minority group members. Because of the prominence of test fairness in the popular media, legislation, and litigation, our results point to the need to revive test bias research in preemployment testing. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

When bad isn't all bad: Strategic use of sample information in generalization and stereotyping.

The influence of the reliability and valence of sample information on generalization to in-group and out-group target populations was investigated in 2 studies. In both studies the valence of the information was either favorable (i.e., a positive in-group sample and a negative out-group sample) or unfavorable. The reliability of sample information was manipulated in terms of sample variability (Study 1) and sample size (Study 2). In both studies the perceived central tendency of favorable sample information was directly generalized, but unfavorable and unreliable sample information was not. Moreover, generalizations based on unfavorable and unreliable samples were characterized by enhanced variability estimates for both in-group and out-group populations, thereby undermining unfavorable group differences. These results are discussed in terms of strategic use of sample information in the formation of social stereotypes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Test of the cultural bias hypothesis: Some Israeli findings.

The major aim of this study was to examine the cross-cultural validity of the test bias contention, with particular concern for possible sociocultural group differences in the construct and predictive validity of college entrance scholastic aptitude tests in Israel. The analyses were based on the test scores of 1,538 Israeli college student candidates of varying ethnic group membership, applying for admission to a major Israeli campus. The psychometric properties of the test battery were compared by ethnic group via a variety of internal (factor structure, reliability, etc.) and external (predictive validity, homogeneity of regression, etc.) test bias criteria. On the whole, the data provided little evidence for differential construct or predictive validity of aptitude test scores as a function of ethnic group membership, thus lending a greater deal of generality to previous research on test bias, generally negating the cultural bias hypothesis. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Characterization of stability of performance in patients with traumatic brain injury: Variability and consistency on reaction time tests.

The stability of performance of reaction times (RTs) on 3 visual discrimination tasks was assessed in patients with varying severity of traumatic brain injury (TBI) and in matched control Ss. Results were analyzed for group and individual variability and consistency of performance over time. There was significantly greater intersubject (group) variability in the TBI group. Individual patients (intrasubject) tended to be more variable in performance and less consistent over time than control Ss, but this occurred only with specific measures, which suggests that increased lack of stability in performance is not a general impairment after brain damage but requires specific analyses for identification of performance differences. This intrasubject variability was indepedent of test–retest and split-half test reliability. No obvious factors such as severity of brain injury were related to the variability and inconsistency of performance. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Chance guessing in a forced-choice recognition task and the detection of malingering.

Guessing occurs on forced-choice (FC) tests for which responses cannot be based on relevant knowledge. Its importance is in inverse relation to the level of knowledge being measured, so that it becomes an increasing component of test scores as the level of knowledge decreases. It is also used as a benchmark to detect simulated impairment. This investigation examined the role of guessing in a 2-alternative FC face recognition test. Chance groups shown only the test items were asked to guess which were the targets, to measure the variation in scores likely to be found with pure guessing. Controls performed normally, and two simulation groups tried to fake amnesia. Results suggested that simple guessing in the chance group produced variable scores that overlapped both low genuine and "malingering" performance. Low control scores were hidden by the guessing "chance bonus," which ameliorated the apparent level of decline in memory. Simulators told of their role at the outset (before presentation of the target items) produced more convincing "amnesic" scores than those told only just before the test, who produced the expected below-chance level of score. It is suggested that guessing variability should be taken into account in interpreting FC scores. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Interrater agreement and predictive validity.

Ss were 549 illiterate Iranian truck drivers rated for intelligence and given an individual intelligence test. To relate unreliability of ratings to validity, correlations were made between intelligence test scores and 4 groups of criterion ratings differing in reliability. The authors conclude that in the construction of rating scales, weighting of ratings by their agreement is better than by their disagreement. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Time-dependent decisions in dogs (Canis familiaris).

Investigated whether the foraging decisions of dogs (Canis familiaris) were time dependent and consistent with a temporal weighting rule (TWR) for maximizing the reliability of information. Dogs were given information about patches whose qualities varied over time. To stimulate natural conditions, interruptions at selected points in foraging were interposed. Patch choices were time dependent and closely matched the predictions of the TWR. Dogs relied on very recent information when available, but with increasing delays they used patch averages. The TWR may be a general solution to problems faced by foragers in variable environments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Optimizing the reliability of speech recognition scores

Speech recognition assessment involves a dilemma because clinicians want a test that is short and reliable, but statistical principles dictate that a short test is unreliable. Curves representing the variability of test scores based on the binomial model reveal that approximately 450 scorable items are needed in order to optimize the reliability of a speech recognition test. A testing approach was developed to achieve this sample size while retaining the principal features of the most commonly accepted speech recognition tests (i.e., monosyllabic words presented in an open-set format, verbal responses, and right/wrong scoring). It involves the use of an interactive computer program to present CNC words in 50 three-word groups, which are scored phonemically, resulting in 450 scorable items. Normal performance is described as a function of both presentation level and signal-to-noise ratio. Comparisons of test and retest scores for 100 individuals with normal hearing and 100 persons with sensorineural losses revealed that the approach achieves the degree of reliability predicted by the binomial model for both groups. Phoneme scores accounted for 99% of the variance of word scores for most of the performance range encountered in clinical practice, making it possible for test outcomes based on phonemic scoring to be expressed in terms of equivalent word recognition scores.     

Diagnostic accuracy of four approaches to interpreting neuropsychological test data.

The diagnostic accuracy of 4 approaches to interpreting neuropsychological test results are evaluated in 672 cognitively normal and 407 cognitively impaired persons using the Mayo Cognitive Factor Scales (G. E. Smith et al., 1994). The interpretation approaches studied are absolute scores, difference scores, profile variability, and change scores at 1- to 2-year test-retest intervals. All dependent measures were "highly significant" when diagnostic groups were compared on null hypothesis significance testing analyses. In contrast, varied accuracy rates were obtained when each measure's ability to correctly classify individuals was evaluated relative to overall diagnostic accuracy. Odds ratios were also highly varied and ranged from ≤ 1.0 (i.e., chance) to 34.9. The clinical usefulness of absolute scores and difference scores in data interpretation is supported. Neither profile variability measures nor measures of change over time were diagnostically useful. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Revision and extension of a multidimensional measure of sexual minority identity: The Lesbian, Gay, and Bisexual Identity Scale.

Two studies were conducted to investigate a revised and extended version of the Lesbian and Gay Identity Scale (Mohr & Fassinger, 2000): the 27-item Lesbian, Gay, and Bisexual Identity Scale (LGBIS). This revision features more inclusive and less stigmatizing language than the previous version and includes 2 new subscales assessing identity affirmation and centrality. In Study 1, an exploratory factor analysis (n = 297) and a confirmatory factor analysis (n = 357) supported an 8-factor solution assessing acceptance concerns, concealment motivation, identity uncertainty, internalized homonegativity, difficulty with the identity development process, identity superiority, identity affirmation, and identity centrality. Predicted associations with measures of identity-related constructs and psychosocial functioning provided preliminary validity evidence for LGBIS scores in a college student population. Study 2 (N = 51) provided evidence of the test–retest and internal consistency reliability of LGBIS scores. These studies suggest that the LGBIS may offer researchers an efficient means of assessing multiple dimensions of sexual orientation minority identity. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Factors and clusters for the Brazelton Scale: An investigation of the dimensions of neonatal behavior.

B. M. Lester et al (see record 1982-28799-001) proposed a set of 7 a priori clusters (e.g., orientation and Reflexes) for reducing data from the Neonatal Behavioral Assessment Scale. In the present study, the distributional and psychometric properties of these clusters were examined in 162 neonates. The distributions of 5 clusters were reasonably normal, but 2 were significantly positively skewed. The 2 clusters whose items were recoded so that midrange optimal scores would be high and extreme scores low exhibited poor internal consistency. In addition, 3 clusters contained some items that were only weakly related to others in their respective clusters. A revised set of clusters was constructed based on data from 6 independent samples. The revised clusters exhibited greater internal consistency than the Lester clusters, comparable or stronger test–retest reliability, and a greater degree of orthogonality. The 2 sets of clusters performed similarly in 4 tests of concurrent validity that used examiner persistence, general irritability, birthweight, and gestational age as criterion variable. (31 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Long-term follow-up of surgical patients with single-ventricle physiology: prognostic anatomical determinants

The objectives of this study were to compare the reliability and validity of written test formats that are widely used in medical education (multiple choice, uncued, extended matching, and true/false) and evaluate the effects of uncued examinations on long-term retention of medical knowledge. Uncued tests were introduced into a traditional course in general and systemic pathology (six interim tests). In the following year, students were given eight tests written in the four formats, each being used twice. The academic achievement of students in these 2 years was compared with that of students in 2 previous years, in which multiple choice tests were used. Measures of academic achievement included performance on a final comprehensive examination and the United States Medical Licensing Examination (USMLE). Student performance on uncued tests was consistent over time (i.e., there was no learning curve). Mean scores ranged from 77% to 84%, and coefficient alpha reliability estimates on 100-item tests were excellent (0.79 to 0.90). Extended matching tests were also reliable, with a mean coefficient alpha of 0.90. There was no significant relationship between test format and student performance on subsequent comprehensive examinations. Our results indicate that extended matching and uncued tests have considerable advantages over multiple choice and true/false examinations. They are more reliable, better able to discriminate the well-prepared from the marginal student, and well suited for tested core knowledge. Contrary to our expectation, extended matching questions with 20 choices presented to the student were as statistically reliable and valid as uncued queries with several hundred choices.     

Drinking Water Treatment Plant Design Incorporating Variability and Uncertainty

Both inherent natural variability and model parameter uncertainty must be considered in the development of robust and reliable designs for drinking water treatment. This study presents an optimization framework for investigating the effects of five variable influent parameters and three uncertain model parameters on the least-cost treatment plant configuration (contact, direct, or nonsweep conventional filtration) that reliably satisfies an effluent particulate matter concentration constraint. Incorporating variability and uncertainty within the decision-making framework generates information for investigating: (1) impacts on total cost and treatment reliability; (2) shifts on the least-cost treatment configuration for providing reliable treatment; and (3) the importance of the individual variable and uncertain parameter distributions for reliably satisfying an effluent water quality constraint. Increasing the magnitude of influent variability and model parameter uncertainty results in a greater expected design cost due, generally, to increases in process sizing required to reliably satisfy the effluent concentration constraint. The inclusion of variability and uncertainty can also produce a shift in the locations of the least-cost configuration regions, which are dependent on the expected influent water quality and the magnitude of variability and uncertainty. The additional information provided by incorporating the variable and uncertain parameters illustrates that parameter distributions related to the primary removal mechanism are critical, and that contact and direct filtration are more sensitive to variability and uncertainty than conventional filtration.     

Motivated skepticism: Use of differential decision criteria for preferred and nonpreferred conclusions.

Three experiments show that information consistent with a preferred conclusion is examined less critically than information inconsistent with a preferred conclusion, and consequently, less information is required to reach the former than the latter. In Study 1, Ss judged which of 2 students was most intelligent, believing they would work closely with the one they chose. Ss required less information to decide that a dislikable student was less intelligent than that he was more intelligent. In Studies 2 and 3, Ss given an unfavorable medical test result took longer to decide their test result was complete, were more likely to retest the validity of their result, cited more life irregularities that might have affected test accuracy, and rated test accuracy as lower than did Ss receiving more favorable diagnoses. Results suggest that a core component of self-serving bias is the differential quantity of cognitive processing given to preference-consistent and preference-inconsistent information. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Effects of categorization and imagery on recognition and recall by process and reactive schizophrenics.

Imagery and categorization were orthogonally varied in an assessment of recognition and recall of 16 process schizophrenics (process group), 16 reactive schizophrenics (reactive group), and 16 controls (student group). Schizophrenic Ss were selected from the inpatient population at Bellevue Hospital, and "students" were volunteers who had all received a bachelor's degree. For all groups, recognition was uninfluenced by categorization but was directly related to imagery. For the student and reactive groups, recall was directly related both to categorization and to imagery. For the process group, however, neither variable influenced recall. In addition, the student group had higher intertrial repetition scores than the reactive group, which had higher scores than the process group. Taken together, findings suggest that the process schizophrenic's inabiity to retrieve information from memory is related to faulty mnemonic organization of to-be-remembered materials. (34 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reliability of the Revised Psychopathy Checklist in substance abuse patients.

Examined the 1-mo test–retest reliability of the revised Psychopathy Checklist (PCL—R) in 88 methadone-maintained male patients (average age nearly 40 yrs) as well as the effects of increased information (interview, clinical chart review, partial and complete criminal records) on PCL—R scores and on 2 diagnostic derivations of the PCL—R. PCL—R scores and diagnostic proportions were not found to differ significantly between baseline and 1 mo. However, increased amounts of rater information resulted in significantly higher PCL—R scores and more positive diagnoses. Test–retest reliabilities of PCL—R scores were found to be excellent, and the reliabilities of each of its 2 factors were also quite good. Diagnostic reliabilities were also good, based on the interview alone, but appeared to improve with increasing information available to the rater. The findings suggest that the PCL—R continues to show promise as a research tool. However, because clinical and criminal records may be difficult to obtain for nonprison populations and settings, there may be limitations in the use of the PCL—R to diagnose psychopathy. (PsycINFO Database Record (c) 2010 APA, all rights reserved)