From the editors.

Presents issue highlights which includes the first debate in the journal's history. Several commentaries are presented in response to the target article "Assessing creativity with divergent thinking tasks: Exploring the reliability and validity of new subjective scoring methods," by Paul J. Silvia, Beate P. Winterstein, John T. Willse, Christopher Barona, Joshua Cram, Karl I. Hess, Jenna L. Martinez, and Crystal A. Richard (see record 2008-05954-001). These responses raised numerous insightful questions, new ideas, and pointed critiques to the lead article. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: Reliability and validity of uniqueness scoring in creativity assessment.

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. In Study 1, Silvia et al. (2008) criticized the uniqueness scoring of Wallach and Kogan (1965). The uniqueness scoring has a virtue that single rater may be enough to rate, and it is characterized by the assignment of points to uncommon responses in a pool of sample's responses. The first criticism for uniqueness scoring is that uniqueness scores increase as a subject produces more responses, resulting in confounding of uniqueness and fluency. The second criticism relates to the ambiguity of statistical rarity pursued by uniqueness scoring in that uniqueness does not guarantee creativity. When a mundane unique response is misperceived as creative, reliability is threatened. Some bizarre, grotesque, or inappropriate responses in the pool of responses may be assigned a point, causing the validity to be threatened. The third criticism raised by the authors is that the uniqueness scoring system penalizes large samples in that it is less probable for a response in a larger sample of people to appear unique. However, the subjective scoring system has other deficits and is never free from the first two criticisms. The third criticism is, however unfounded; rather, the uniqueness scoring system is in a better position to capture the construct of creativity through better accessibility to large samples. The authors' (Silvia et al., 2008) three criticisms will be discussed one by one. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: Divergent thinking tests have problems, but this is not the solution.

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. It is certainly true, as Silvia et al. (2008) write, that "after half a century of research, the evidence for global creative ability ought to be better" (p. 68). The authors believe--incorrectly, I think--that the reason that divergent thinking tests have not done a better job can be found in the various scoring systems that have been used when assessing divergent thinking ability. I have presented evidence elsewhere that creativity is not a general ability or set of traits or dispositions that can be applied across domains (Baer, 1991, 1993, 1994a, 1994b, 1998). In those studies, I used Amabile's (1982, 1996) Consensual Assessment Technique (which is the basis for the subjective scoring technique proposed by Silvia et al. [2008]) to judge the creativity of a wide range of artifacts. What I found was that there is little correlation among the creativity ratings received by subjects across domains, and what little there is tends to disappear if an IQ test is also given and variance attributable to intelligence is first removed. The evidence presented thus far for Silvia et al.'s (2008) proposed method for scoring responses to divergent thinking tasks has far too many flaws to allow any confidence in its use. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: The Torrance Tests of Creative Thinking already overcome many of the perceived weaknesses that Silvia et al.'s (2008) methods are intended to correct.

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. Silvia et al.'s (2008) primary motivations for exploring and proposing their subjective scoring method are their perceived deficiencies of current divergent thinking tests. First, scores on divergent thinking tests frequently correlate highly with general intelligence. Second, the scoring of divergent thinking tests has changed little since the 1960s. Third, the necessity of instructing people to be creative prior to taking divergent thinking tests is integral to obtaining useful responses and needs to be reaffirmed. Fourth, and finally, the problems posed by uniqueness scoring--confounding with fluency, ambiguity of rarity, and the seeming "penalty" imposed on large samples--need to be addressed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Further explorations of perceptual speed abilities in the context of assessment methods, cognitive abilities, and individual differences during skill acquisition.

Measures of perceptual speed ability have been shown to be an important part of assessment batteries for predicting performance on tasks and jobs that require a high level of speed and accuracy. However, traditional measures of perceptual speed ability sometimes have limited cost-effectiveness because of the requirements for administration and scoring of paper-and-pencil tests. There have also been concerns about the validity of previous computer approaches to administering perceptual speed tests (e.g., see Mead & Drasgow, 1993). The authors developed two sets of computerized perceptual speed tests, with touch-sensitive monitors, that were designed to parallel several paper-and-pencil tests. The reliability and validity of the tests were explored across three empirical studies (N = 167, 160, and 117, respectively). The final study included two criterion tasks with 4.67 and 10 hours of time-on-task practice, respectively. Results indicated that these new measures provide both high levels of reliability and substantial validity for performance on the two skill-learning tasks. Implications for research and application for computerized perceptual speed tests are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Discernment and creativity: How well can people identify their most creative ideas?

How well can people judge the creativity of their ideas? The distinction between generating ideas and evaluating ideas appears in many theories of creativity, but the massive literature on generation has overshadowed the question of evaluation. After critically reviewing the notion of accuracy in creativity judgments, this article explores whether (1) people in general are discerning and (2) whether some people are more discerning than others. University students (n = 226) completed four divergent thinking tasks and then decided which responses were their most creative. Judges then rated the creativity of all of the responses. Multilevel latent-variable models found that people's choices strongly agreed with judges' ratings of the responses; overall, people were discerning in their decisions. But some people were more discerning than others: people high in openness to experience, in particular, had stronger agreement between their decisions and the judges' ratings. Creative people are thus doubly skilled: they are better at generating good ideas and at picking their best ideas. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A closer look at the frame-of-reference effect in personality scale scores and validity.

This article contributes to the understanding of why the use of a frame-of-reference leads to increased criterion-related validity of personality inventories. Two competing explanations are described and tested. A between-subjects (N = 337) and a within-subject (N = 105) study are conducted to test the hypothesized effects of use of a frame of reference on reliability and validity. Regarding the effects on reliability, use of a frame of reference reduces within-person inconsistency (instead of between-person variability) in responding to generic items. Use of a frame of reference further leads to higher validity as a result of the reduction of between-person variability and within-person inconsistency. Yet, reducing these inconsistencies is not enough. It is also important to use a frame of reference that is conceptually relevant to the criterion. Besides implications for contextualized personality inventories, these results provide an explanation for the moderate validities of generic personality inventories. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Rejoinder: The madness to our method: Some thoughts on divergent thinking.

Replies to comments by M. D. Mumford et al. (see record 2008-05954-002), J. Baer (see record 2008-05954-003), M. A. Runco (see record 2008-05954-004), K. H. Kim (see record 2008-05954-005), N. Kogan (see record 2008-05954-006), and S. Lee (see record 2008-05954-007) on the current authors' original article on divergent thinking (see record 2008-05954-001). In this reply, the authors examine the madness to their method in light of the comments. Overall, the authors agree broadly with the comments; many of the issues will be settled only by future research. The authors disagree, though, that past research has proven past scoring methods--including the Torrance methods--to be satisfactory or satisfying. The authors conclude by offering their own criticisms of their method, of divergent thinking, and of the concept of domain-general creative abilities. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Job applicant training and work experience evaluation: An empirical comparison of four methods.

Assessed the reliability, quasi validity, and scoring time required for each of 4 training and experience (TAE) evaluation methods for 1st-line supervisory positions in 3 substantially different occupational areas. The TAE scoring procedures were applied to application forms by 4 undergraduates who received formal training regarding TAE methods. Ss were assigned to pairs, and application packets for each of 3 target jobs (TJs) were randomly split into 2 groups of equal size and distributed to Ss, so that each pair rated half the application forms from each of the TJs. Results indicate that the greatest amount of variation was obtained for the behavioral consistency method, the 2nd most for the task-based method (TBM), the 3rd most for the point method, and the least for the grouping method (GM). The TBM yielded significantly higher reliability coefficients, as compared with the other 3 methods, and the GM yielded significantly lower reliability coefficients. However, the TBM indicated high interrater reliability coefficients due to the straightforward nature of the scoring procedure. (13 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: Divergent thinking is not synonymous with creativity.

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. On several occasions I have suggested that a modified scientific method be used in studies of creativity (Runco, 1994a, 1999, 2006). This is a fairly contrarian suggestion because it implies a less-than-maximally objective perspective. Yet creativity will never be fully understood using the traditional scientific approach. This is in part because creativity requires originality, and the novelty that signifies originality is typically unpredictable, or at least not predictable with much precision. Perhaps more important for a modified scientific approach is the fact that creativity depends on affect, intuition, and other processes which cannot be accurately described using only objective terms. Yet at the same time, we should be as objective as possible. And although I am intrigued by generalizability theory, as described by Silvia, Winterstein, Willse, Barona, Cram, Hess, Martinez, and Richard (2008), I am concerned about their decision to use subjective scoring of divergent thinking tests. Their rationale is weak, to be blunt about it, and they have overlooked some critical research on the topic. In this commentary, I could describe the attraction of generalizability theory, but Silvia et al. do a more than adequate job of that. So instead, I will try to fill some gaps in their review of the research on divergent thinking. I also have a few questions with several of their claims and methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A biochemical correlate of divergent thinking.

Rated 254 undergraduates on divergent thinking, using M. A. Wallach and N. Kogan's Pattern Meanings Test. Subsequently, serum uric acid (SUA) determinations were carried out for a subgroup of 55 Ss (32 mean and 23 women), of whom 34 had obtained very high divergent thinking scores, and 21 very low scores. It was found that the highly divergent Ss had significantly lower SUA levels, the difference being much more pronounced in the males than the females. These differences are interpreted as supporting the construct validity of the concept of divergent thinking. (French summary) (22 ref.) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: Measuring divergent thinking: Is there really one solution to the problem?

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. Although their findings appear reasonable, we have two key concerns with regard to these conclusions, 1) the strength of the available construct validity evidence, and 2) the substantive logic underlying the study. The next generation of measures should be applying a domain, strategy, process approach rather than the domain-free, output-based approach recommended by Silvia et al. (2008). (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Selecting items for interest inventory keys.

Several methods of scoring an interest inventory so as to maximize the separation of workers in an occupation from workers in general were applied to samples of electricians (compared with civilian workers) and aviation machinists' mates (compared with Navy men-in-general.) Criteria of a good key were (a) its ability to separate groups (per cent overlap), and (b) its test-retest reliability. It was found that (1) using unit item weights an optimum number of items can be found for scoring, (2) units weights with an optimum number of items yielded more discriminating keys than Strong scoring weights, (3) selecting items by a method designed to increase item heterogeneity, the validity of the key is increased but test-retest reliability is somewhat decreased. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Individual differences in adult decision-making competence.

The authors evaluated the reliability and validity of a set of 7 behavioral decision-making tasks, measuring different aspects of the decision-making process. The tasks were administered to individuals from diverse populations. Participants showed relatively consistent performance within and across the 7 tasks, which were then aggregated into an Adult Decision-Making Competence (A-DMC) index that showed good reliability. The validity of the 7 tasks and of overall A-DMC emerges in significant relationships with measures of socioeconomic status, cognitive ability, and decision-making styles. Participants who performed better on the A-DMC were less likely to report negative life events indicative of poor decision making, as measured by the Decision Outcomes Inventory. Significant predictive validity remains when controlling for demographic measures, measures of cognitive ability, and constructive decision-making styles. Thus, A-DMC appears to be a distinct construct relevant to adults' real-world decisions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The construct validity of egocentrism.

Briefly explains construct validation and applies it to egocentrism. Conceptual and operational referents of this construct are organized into 3 categories: visual/spatial egocentrism (what does the other see), affective egocentrism (what does the other feel), and cognitive/communicative egocentrism (what is the other thinking). Several kinds of reliability information are reported, and construct validity is evaluated primarily by examination of the relationships among measures of egocentrism within and between categories. Although interrater reliability and interrater agreement were found to be uniformly high for all egocentrism measures examined in a review, and the measurement reliability was usually adequate, a few tasks were not internally consistent. Overall, the construct validity of egocentrism was not supported, since most task intercorrelations were low and often nonsignificant. An alternative interpretation of the data based on cognitive constructs and task- and response-specific variables is proposed. (2? p ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Emotional intelligence: in search of an elusive construct

The view that emotional intelligence should be included within the traditional cognitive abilities framework was explored in 3 studies (total N = 530) by investigating the relations among measures of emotional intelligence, traditional human cognitive abilities, and personality. The studies suggest that the status of the emotional intelligence construct is limited by measurement properties of its tests. Measures based on consensual scoring exhibited low reliability. Self-report measures had salient loadings on well-established personality factors, indicating a lack of divergent validity. These data provide controvertible evidence for the existence of a separate Emotion Perception factor that (perhaps) represents the ability to monitor another individual's emotions. This factor is narrower than that postulated within current models of emotional intelligence.     

Reliability estimated by scoring key equivalence across samples.

Estimated reliability of a biographical inventory (BI) by correlating scores from preexistent BI scoring keys with scores from empirically constructed BI scoring keys. Exp. I demonstrated that a preexistent scoring key, developed on National Aeronautics and Space Administration scientists to predict creativity, was correlated .87 (n = 294) and .91 (n = 296) with scoring keys empirically constructed to predict creativity for 2 samples of industrially employed scientists and engineers. Exp. II demonstrated that preexistent scoring keys, constructed on 11,246 university freshmen to predict GPA, were correlated .88 with scoring keys empirically constructed to predict GPA for high school students. Test-retest reliability estimates and validity generalization estimates support the above results. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Commentary: Divergent-thinking research and the Zeitgeist.

Commentary on an article by P. J. Silvia et al. (see record 2008-05954-001) which discusses the topic of divergent thinking. More than 40 years have passed since the publication of the Wallach and Kogan (1965) volume, and yet it continues to draw both praise and criticism from researchers in the creativity field. The Silvia et al. (2008) article tilts more strongly to criticism than to praise, and accordingly, one of the editors of this journal (JCK) kindly offered me an opportunity to respond. I do so with some hesitancy, as I am no longer an active divergent-thinking (DT) researcher. This gap in active involvement as a DT researcher was not a severe handicap for me in appraising the Silvia et al. (2008) article. Issues of reliability and validity of DT measures--the central concern of that article--have preoccupied investigators ever since Guilford's (1950) original formulation, a preoccupation that I shared. At the outset, I should state that I consider the Silvia et al. (2008) treatment of those fundamental issues to be methodologically sound. My intent in the present commentary, rather, is to demonstrate that the Zeitgeist at the time of the Wallach and Kogan (1965) study was quite different from that prevailing today. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Reliability of scores on the Stroke Rehabilitation Assessment of Movement (STREAM) measure

BACKGROUND AND PURPOSE: The Stroke Rehabilitation Assessment of Movement (STREAM) is a new clinical measurement tool for evaluating the recovery of voluntary movement and basic mobility following stroke. This article presents the results of 3 substudies examining the reliability (interrater and intrarater) and internal consistency of STREAM scores. SUBJECTS AND METHODS: A "direct-observation reliability study" was conducted on 20 patients who had strokes and were in a rehabilitation setting. Pairs of raters from a group of 6 participating therapists provided data to judge interrater agreement. A "videotaped assessments reliability study" was done to assess intrarater and interrater agreement on the scoring of videotaped performances using the STREAM measure and involved 4 videotaped assessments that were viewed and rated on 2 occasions by 20 physical therapists. The internal consistency of the STREAM scores was evaluated for 26 patients who had strokes and who demonstrated the full range of motor ability. RESULTS: The reliability of the STREAM scores was demonstrated by generalizability correlation coefficients of .99 for total scores and of .96 to .99 for subscale scores. The internal consistency of the STREAM scores was demonstrated by Cronbach alphas of greater than .98 on the subscales and overall. CONCLUSION AND DISCUSSION: These high levels of reliability support the use of the STREAM instrument for the measurement of motor recovery following stroke. Further work on the validity and responsiveness of the STREAM measure is in progress.     

Accuracy-based phonological awareness tasks: Are they reliable, efficient, and sensitive to growth?

This two-experiment study examined the efficiency and sensitivity of five accuracy-based phonological awareness tasks for monitoring the development of these skills in kindergarten and Grade 1 students. The first experiment examined responses to different numbers and types of items included in each phonological awareness task for their correspondence to responses obtained from a larger, more inclusive item pool. Results suggested that an internally consistent and valid measure of each skill included 10 items per task, each representing a different linguistic combination. The second experiment examined the interscorer reliability and concurrent validity of the 5 measures, and compared their sensitivity to growth. Sensitivity was examined by administering 12 alternate forms of the tasks once per week to 32 kindergarten and 35 Grade 1 students. Mean slopes computed for each task suggested positive growth across all tasks and grades. Mean kindergarten slopes were significantly steeper than mean Grade 1 slopes for each of the 5 tasks, whereas the most sensitive task for both kindergarten and grade I students was Segmentation. (PsycINFO Database Record (c) 2011 APA, all rights reserved)