A comparison of economic utility estimates for alternative SDy estimation procedures.

Compared the per-selectee and total utility estimates for an assessment center that was used to select midlevel sales managers when 2 consensus-seeking procedures, the F. L. Schmidt et al (see record 1981-02231-001) procedure and 40% and 70% of mean salary, were used for estimating the standard deviation of job performance in dollars (SDy). Using the Schmidt et al procedure as the base, the overall utility estimates varied from 21 to 125% of the Schmidt et al values. For most cases, the different SDy estimation procedures produced somewhat similar utility estimates. The resulting estimated dollar gains from the use of the assessment center to select sales managers were substantial in all cases. The dollar gains from the current selection strategy were, at a maximum, only 63% of estimated top-down selection utility gains. Findings illustrate the value of economic utility analysis for evaluating human resource programs regardless of the type of SDy estimation procedure used. (21 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Direct estimates of SDy and the implications for utility analysis.

Utility analysis suggests that human resources policies can have an economically significant impact on business organizations. Confidence in such conclusions, however, requires an accurate estimate of SDy. This article provides a validity check on prevailing subjective methods of SDy estimation by directly estimating SDy from unique field data. Using both simulated and field data, the range of potential bias associated with predictor unreliability in regression analyis is illustrated, and how to calculate corrected values is shown. The methodological problems of directly estimating SDy with organizational data is discussed, and a range of estimates for SDy is provided. Direct estimation of SDy yielded values ranging from 74% to 100% of mean salary, which are considerably greater than conventional subjective judgments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Efficacy of modified {crepid} SDys on the basis of archival organizational data.

This study examined the effect of estimating {crepid} (Cascio & Ramos, 1986) SDy using the original procedure or using modified versions that replaced data obtained specifically for {crepid} SDy estimation with the following organizational archival data: performance evaluations, job analysis ratings, or both evaluations and ratings. Those four SDy estimates and the estimate obtained using Hunter and Schmidt's (1982) 40% rule converged at a level nearly one-fifth of that for the Procedure-B (Burke & Frederick, 1984) SDy. Moreover, judges preferred supplying Procedure-B percentile estimates, but perceived that their {crepid} ratings were more accurate. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A model-averaging approach to replication: The case of prep.

The purpose of the recently proposed prep statistic is to estimate the probability of concurrence, that is, the probability that a replicate experiment yields an effect of the same sign (Killeen, 2005a). The influential journal Psychological Science endorses prep and recommends its use over that of traditional methods. Here we show that prep overestimates the probability of concurrence. This is because prep was derived under the assumption that all effect sizes in the population are equally likely a priori. In many situations, however, it is advisable also to entertain a null hypothesis of no or approximately no effect. We show how the posterior probability of the null hypothesis is sensitive to a priori considerations and to the evidence provided by the data; and the higher the posterior probability of the null hypothesis, the smaller the probability of concurrence. When the null hypothesis and the alternative hypothesis are equally likely a priori, prep may overestimate the probability of concurrence by 30% and more. We conclude that prep provides an upper bound on the probability of concurrence, a bound that brings with it the danger of having researchers believe that their experimental effects are much more reliable than they actually are. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessing interrater agreement on the job relevance of a test: A comparison of CVI, T, rWG(J)}, and r*WG(J)} indexes.

This article examines the problem of estimating interrater agreement about the job relevance of test items by examining 4 different indexes: C. H. Lawshe's (1975) Content Validity Index (CVI); H. E. A. Tinsley and D. J. Weiss's (1975) T; L. R. James, R. G. Demaree, and G. Wolf's (1984) r{wg(j)}; and M. K. Lindell, C. J. Brandt, and D. J. Whitney's (1999) r*{wg(j)}. These 4 indexes are shown to be systematically related to each other, but r*{wg(j)} is computationally simpler and is more generally applicable than the other indexes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Killeen's (2005) prep coefficient: Logical and mathematical problems.

In his article, “An alternative to null-hypothesis significance tests,” Killeen (2005) urged the discipline to abandon the practice of pobs-based null hypothesis testing and to quantify the signal-to-noise characteristics of experimental outcomes with replication probabilities. He described the coefficient that he invented, prep, as the probability of obtaining “an effect of the same sign as that found in an original experiment” (Killeen, 2005, p. 346). The journal Psychological Science quickly came to encourage researchers to employ prep, rather than pobs, in the reporting of their experimental findings. In the current article, we (a) establish that Killeen's derivation of prep contains an error, the result of which is that prep is not, in fact, the probability that Killeen set out to derive; (b) establish that prep is not a replication probability of any kind but, rather, is a quasi-power coefficient; and (c) suggest that Killeen has mischaracterized both the relationship between replication probabilities and statistical inference, and the kinds of claims that are licensed by knowledge of the value assumed by the replication probability that he attempted to derive. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

rwg: An assessment of within-group interrater agreement.

F. L. Schmidt and J. E. Hunter (1989) critiqued the within-group interrater reliability statistic (rwg) described by L. R. James et al (1984). S. W. Kozlowski and K. Hattrup (1992) responded to the Schmidt and Hunter critique and argued that rwg is a suitable index of interrater agreement. This article focuses on the interpretation of rwg as a measure of agreement among judges' ratings of a single target. A new derivation of rwg is given that underscores this interpretation. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

From H = log sn to conceptual framework: A short history of information.

"Information" has become a widely used term in psychology, especially within cognitive psychology. However, despite its status as a technical term, the word now rarely receives explicit definition. By contrast, when information entered the vocabulary of psychologists in the late 1940s, it had an explicit mathematical definition largely derived from developments in information theory. This article examines how information entered psychology, how its meaning changed, and how it remained a technical term in the vocabulary of psychologists in the second part of the 20th century. "Information" became a term that was required to speak to ever more diverse theoretical concerns and its earliest definitions in psychology could not sustain such uses. As a consequence, "information" became a term whose technical uses became increasingly difficult to differentiate from its everyday meanings. I argue that this has not necessarily made "information" a worthless term but one whose lack of specificity may now be unsettling to some psychologists. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Regarding prep: Comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010).

The sense that replicability is an important aspect of empirical science led Killeen (2005a) to define prep, the probability that a replication will result in an outcome in the same direction as that found in a current experiment. Since then, several authors have praised and criticized prep, culminating in the 3 articles in the current issue of Psychological Methods. In this article, Killeen's prep is reviewed, and the contributions of the current articles are summarized and discussed. An examination of the role of a measure of theoretical support such as prep in the acquisition of knowledge leads me to concur with Senn (2002) that prep is of little epistemological value. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

ppep Replicates: Comment prompted by Iverson, Wagenmakers, and Lee (2010); Lecoutre, Lecoutre, and Poitevineau (2010); and Maraun and Gabriel (2010).

Lecoutre, Lecoutre, and Poitevineau (2010) have provided sophisticated grounding for prep. Computing it precisely appears, fortunately, no more difficult than doing so approximately. Their analysis will help move predictive inference into the mainstream. Iverson, Wagenmakers, and Lee (2010) have also validated prep as a boundary condition in a model-averaging approach. I argue that the boundary is good; that the proper place for their assignment of subjective priors is to update personal belief; and that such assignment has no place in the evaluation of evidence, on which priors should be flat. Maraun and Gabriel (2010) have provided clarification, context, and critique of the derivation of prep. They concluded that prediction can never be precise without knowledge of population parameters and joint empirical distributions; that nature evolves; that a large initial effect will encourage replication attempts, which are therefore not independent of it; and that there is no substitute for a sustained program of replication attempts—all of which are prudent cautions. If statistics is to be the handmaiden of science and policy, however, rather than history, it ineluctably must speak to the future. The posterior predictive distribution is an underexploited tool in the analyst's kit that will serve that end, and prep is but one valid implementation of it. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Analysis of nonorthogonal designs: The 2k factorial experiment.

Asimple unweighted means ANOVA of a 2k factorial experiment with unequal and disproportionate cell frequencies resulted in the same sums of squares and F tests as the more complex Method 1 analysis recommended by J. E. Overall et al (see record 1975-20136-001). (6 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing the significance of Kendall's τ and Spearman's r[s]..

An overview of procedures for testing the significance of Kendall's τ and Spearman's r[s], as presented in well-known textbooks for the behavioral sciences, is given. Testing Kendall's τ proceeds in an almost uniform way and seldom leads to erroneous conclusions. For N?>?20, the exact test can be replaced by an approximate z test. Testing Spearman's r[s] varies from textbook to textbook. Some textbooks present tables with significant values that are in error; this seems partly due to the discrete character of r[s]. The well-known Statistical Package for the Social Sciences computer package presents false results, both for τ and for r[s], when N is small. Although tables for exact tests of r[s] for N?=?12(1)16 exist, they seem to be unknown to textbook writers, except Zar (1984). Testing ΣD–2 instead of r[s] prevents a clear view of the degree of dependence. For these reasons, a table with exact values of r[s] for α?=?.01 and .05, one- and two-tailed, is presented for N?=?4(1)16. For N?>?16, an approximate t test may be used, although Zar's (1972, 1984) approximation yields slightly better results. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Presentation modality and indirect performance information: Effects on ratings, reactions, and memory.

The authors investigated whether raters integrate indirect (second-hand) information from an employee's co-worker with their direct observations when completing performance evaluations. Performance levels of direct and indirect information and presentation modality (auditory vs. textual) were manipulated (N=220). Results showed that indirect information was perceived to be of highest utility when the performance levels of the direct and indirect information were consistent. Confidence in performance ratings was lowest when the indirect source delivered negative performance feedback that was contrary to the rater's own positive observations. Indirect information was only reflected in the performance ratings when direct observations were positive. There was a significant 3-way interaction between performance level of the direct information, performance level of the indirect information, and presentation modality on memory for performance incidents. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Review of Social intervention: Theory and practice.

Reviews the book, Social intervention: Theory and practice, edited by Edward M. Bennett (see record 1987-98585-000). Scholarly literature in the field of social change is extensive, but knowledge of the nature of that change and the process of how it is implemented and carried out is limited. According to Bennett, this lacuna is due to the fact that writings in the field have dealt with interventions within a system and not with change of the system itself. The scarcity of meaningful study of social intervention, Bennett believes, is due to the unwillingness of social scientists to 1) function together in teams to address what is in essence a problem, 2) to pair-up with practitioners working on social intervention projects in joint ventures, and 3) to address the limitations of the logical positivist traditions of their university bases. To overcome these obstacles, Bennett has called upon both academics and practitioners from various disciplines to present both theory and interventions dealing with social problems. The book contains 14 chapters divided into four parts: I, Introduction; II, Cultural, Ideological and Educational Approaches to Social Intervention; III, Legal, Policy and Political Approaches to Social Intervention; and IV, Community Economic Development Approaches to Social Intervention. This book represents a Herculean task. Any text which would include the diversity of professions and disciplines in an effort to integrate theory and practice relating to social intervention is a challenge few would undertake. In doing so, Bennett has produced a volume which does indeed represent a serious attempt to come to grips with how to change a system instead of just the parts of that system. He has achieved this by insisting on a workable conceptual structure, theme and focus for all contributors regardless of the content areas covered by the authors. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Review of Symptom analysis: A method of brief therapy.

Reviews the book, Symptom analysis: A method of brief therapy by M. Gerald Edelstien (see record 1990-97543-000). This book compares different methods of psychotherapy to find their common thread of effectiveness. According to the reviewer, the author has presented a bold, elegantly simple and quite useful theory. After a discussion of various theories about the origin of psychiatric symptoms and showing how the trauma theory may be seen to underlie various seemingly disparate therapeutic approaches, the author goes on to discuss the general values of brief therapy. Symptom Analysis is a bright book that is full of the author's clinical observations and sometimes challenges so-called "conventional wisdom" without having scientific "hard data" to support its suggestions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Laterality, alternation, and perseveration relationships on the T-maze test.

The T-maze test has been used to study several entirely different issues: spontaneous alternation behavior (SAB), perseveration behavior (PB), and behavioral lateralization. Despite the fact that in this test the behavior studied is always the same one (i.e., side choice), the possible relationships among SAB, PB, and lateralization have not been previously evaluated. The present study investigated the relationships among these functions. The results demonstrated that (1) shock increases PB and lateralization decreases SAB, (2) practice increases lateralization and decreases SAB but does not modify PB, and (3) there are sex differences for alternation and SAB. Because these functions are expressed by the same behavioral pattern, they must be quantified simultaneously to avoid mistaken conclusions when the T-maze test is used. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Review of Health enhancement, disease prevention, and early intervention: Biobehavioral perspectives.

Reviews the book, Health enhancement, disease prevention, and early intervention: Biobehavioral perspectives by Kenneth Craig and Stephen Weiss (see record 1990-98088-000). Recent years have witnessed a dramatic increase in the recognition that important relations exist between health and behavioural variables. The introduction of the biopsychosocial model (Engel, 1977) and its adoption in preference to the earlier germ theory of disease have led to the identification of powerful behavioural and lifestyle influences on physical health and well-being. Important consequences of this rapidly evolving body of knowledge have included the development of behavioural interventions and their incorporation into routine clinical practice. Innovative behavioural prevention efforts have also been developed. This work has addressed a broad range of health care objectives, including symptom control (e.g., hypertension, pain), self-management of chronic conditions (e.g., arthritis, asthma), adherence to therapeutic regimens (e.g., medications, dietary and fluid-intake limitations), and prevention of disease through the modification of behavioural risk factors (e.g., smoking, obesity, alcohol abuse, unsafe sexual practices). Intervention efforts have been directed at individuals, dyads, small groups, and even entire communities, with very encouraging results. This book makes a useful contribution to this literature. More than a single volume will be needed, however, to provide comprehensive coverage of this rapidly growing and changing field. Inasmuch as the present volume is the third in the Banff series to deal with topics in behavioural medicine, one can be encouraged that leading developments and innovations in the field will continue to receive a forum. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Review of Child sexual abuse: Critical perspectives on prevention, intervention, and treatment.

Reviews the book, Child sexual abuse: Critical perspectives on prevention, intervention, and treatment edited by Christopher R. Bagley and Ray J. Thomlison (1991). This book is a compilation of a series of literature reviews originally commissioned by Health and Welfare Canada in 1987 and completed in 1988. The individual papers included in this edited version represent a cross section of Canadian academicians, clinicians, and case workers who are integrally involved in the policies and practices regarding child sexual abuse in Canada. The work provides a comprehensive perspective on prevention at the primary, secondary, and tertiary levels. The list of authors is impressive in terms of their expertise and experience. At a time when numerous books on child sexual abuse are appearing on the market, it is nice to see a book that has some unusual aspects. Aside from its distinctly Canadian perspective, the book addresses several important, yet frequently ignored, topics. The book provides an up-to-date review of several core issues: conceptualization of the problem, prevention strategies, impact of sexual abuse, investigative interviewing, treatment outcome studies, and treatment issues for child molesters. The more unique topics include a review of the strategies used to evaluate prevention programs, prevalence rates among a number of special populations, the role of medical practitioners in preventing and intervening in child sexual abuse, and sexual abuse and exploitation among disabled individuals. Overall, I can recommend this book for clinicians and researchers in Canada and elsewhere who are interested in child sexual abuse. This compilation of literature reviews highlights the leading role that Canadian social service agencies have taken in developing programs for sexually abused children and their families. (PsycINFO Database Record (c) 2010 APA, all rights reserved)