A method for exploring the effects of attrition in randomized experiments with dichotomous outcomes.

Attrition from conditions in randomized experiments is common. Yet it is difficult to assess the possible effects of attrition because the outcome status of the dropouts is usually unknown. This article develops methods to assess those effects in studies with dichotomous outcomes, illustrating the methods with randomized experiments in drug abuse treatment, smoking cessation treatment, and alcoholism treatment. The methods include computing the lowest and highest possible effect sizes that could have been observed, enumerating the percent of possible study outcomes below a given threshold, estimating the probability that an outcome beyond any given threshold would be observed if all participants were measured, and constructing attrition analysis plots showing the effects of attrition under varied assumptions. For the kind of study to which they apply, these methods should replace the treatment of missing participants as failures in an "intent-to-treat" analysis. A user-friendly personal computer program is available to implement all of these analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On blowing trumpets to the tulips: To prove or not to prove the null hypothesis--Comment on B?sch, Steinkamp, and Boller (2006).

The H. B?sch, F. Steinkamp, and E. Boller (see record 2006-08436-001) meta-analysis reaches mixed and cautious conclusions about the possibility of psychokinesis. The authors argue that, for both methodological and philosophical reasons, it is nearly impossible to draw any conclusions from this body of research. The authors do not agree that any significant effect at all, no matter how small, is fundamentally important (B?sch et al., 2006, p. 517), and they suggest that psychokinesis researchers focus either on producing larger effects or on specifying the conditions under which they would be willing to accept the null hypothesis. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assignment methods in experimentation: When do nonrandomized experiments approximate answers from randomized experiments?

This meta-analysis compares effect size estimates from 51 randomized experiments to those from 47 nonrandomized experiments. These experiments were drawn from published and unpublished studies of Scholastic Aptitude Test coaching, ability grouping of students within classrooms, presurgical education of patients to improve postsurgical outcome, and drug abuse prevention with juveniles. The raw results suggest that the two kinds of experiments yield very different answers. But when studies are equated for crucial features (which is not always possible), nonrandomized experiments can yield a reasonably accurate effect size in comparison with randomized designs. Crucial design features include the activity level of the intervention given the control group, pretest effect size, selection and attrition levels, and the accuracy of the effect-size estimation method. Implications of these results for the conduct of meta-analysis and for the design of good nonrandomized experiments are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues.

Meta-analysis has mostly been used to summarize the size of an effect averaged over multiple studies, but meta-analysis has not been much applied to the study of causal mediating processes through which an effect is produced. This lacuna has limited the contribution of meta-analysis to the explanatory theories that play such a key role in science. Fortunately, meta-analysts can explore causal processes. This article reviews several examples of how this has been done in past meta-analyses, using these examples to introduce the methodological, statistical, and conceptual problems that are raised when meta-analysis is applied to the task. Meta-analysts are encouraged to adapt such methods to their work to improve the capacity of their work to contribute to scientific theory, and statisticians are encouraged to solve the remaining statistical problems that current meta-analytic mediational analyses incur. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Costs and effects in mental health policy.

Argues that C. A. Kiesler (see record 1982-31029-001) inadvertently overstates the expense of institutionalization in present mental health policy and hence overestimates the relative fiscal viability of alternative care. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

PhDs, PsyDs, and real-world constraints on scholarly activity: Another look at the Boulder Model.

We report the results of a random sample survey of the scholarly activities of PhD and PsyD-trained clinical psychologists. Like past researchers, we found that practicing clinicians produce few publications, although they are active in research in many other ways and are active consumers of research. The best predictors of scholarly activity are measures of the resources and incentives available in the worksite for engaging in such activities, but a positive attitude toward research was also a good predictor. In general, PhDs report more scholarly involvement than do PsyDs. But this difference diminishes in settings where fewer resources and incentives are available, to the point at which no differences in scholarly activity exist between PhDs and PsyDs in private practice or in settings where no paid hours are allowed to be devoted to research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

"How interchangeable are different estimators of effect size?": Correction to Ray and Shadish (1996).

Reports an error in the original article by J. W. Ray and W. R. Shadish (Journal of Consulting and Clinical Psychology, 1996[Dec], Vol 64(6), 1316–1325). On page 1325, a correction is made to column 1, lines 25–26. (The following abstract of this article originally appeared in record 1996-07086-021). The computation of effect sizes is a key feature of meta-analysis. In treatment outcome meta-analyses, the standardized mean difference statistic on posttest scores (d) is usually the effect size statistic used. However, when primary studies do not report the statistics needed to compute d, many methods for estimating d from other data have been developed. Little is known about the accuracy of these estimates, yet meta-analysts frequently use them on the assumption that they are estimating the same population parameter as d. This study investigates that assumption empirically. On a sample of 140 psychosocial treatment or prevention studies from a variety of areas, the present study shows that these estimates yield results that are often not equivalent to d in either mean or variance. The frequent mixing of d and other estimates of d in past meta-analyses, therefore, may have led to biased effect size estimates and inaccurate significance tests. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The buck stops here: A survey of the financial status of PhD graduate students in psychology, 1966–1987.

Results of a survey of 454 graduate PhD candidates in psychology (1966–1987) suggest that current graduate students must assume a greater proportion of the cost of their education. Several strategies for increasing graduate student stipends are presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A randomized experiment comparing random and cutoff-based assignment.

In this article, we review past studies comparing randomized experiments to regression discontinuity designs, mostly finding similar results, but with significant exceptions. The latter might be due to potential confounds of study characteristics with assignment method or with failure to estimate the same parameter over methods. In this study, we correct the problems by randomly assigning 588 participants to be in a randomized experiment or a regression discontinuity design in which they are otherwise treated identically, comparing results estimating both the same and different parameters. Analysis includes parametric, semiparametric, and nonparametric methods of modeling nonlinearities. Results suggest that estimates from regression discontinuity designs approximate the results of randomized experiments reasonably well but also raise the issue of what constitutes agreement between the 2 estimates. (PsycINFO Database Record (c) 2011 APA, all rights reserved)