When small effects are impressive.

Effect size is becoming an increasingly popular measure of the importance of an effect, both in individual studies and in meta-analyses. However, a large effect size is not the only way to demonstrate that an effect is important. This article describes 2 alternative methodological strategies, in which importance is a function of how minimal a manipulation of the independent variable or how difficult-to-influence a dependent variable will still produce an effect. These methodologies demonstrate the importance of an independent variable or psychological process, even though they often yield effects that are small in statistical terms. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Efficacy of self-administered treatments for depression and anxiety.

Self-administered treatments (SATs) are widely used by the general public and mental health professionals. Previous reviews of the efficacy of SATs have included under this category interventions for nonclinical problems, group interventions, and interventions involving significant amounts of therapist contact. The efficacy of SATs for clinical levels of depression and anxiety with minimal therapeutic contact was examined by meta-analyzing 24 studies. The results show large effects for SATs when compared with no-treatment control groups (d=1.00). However, unlike previous meta-analyses that found nonsignificant differences between SATs and therapist-administered treatments, in this sample SATs resulted in significantly poorer outcomes (d=-0.31). Some differences in effect size were observed between the clinical targets of depression and anxiety. However, there were high correlations between clinical target, methodological quality of the study, and amount of contact. This makes it impossible to determine whether the observed differences could be explained by the nature of the disorders, methodological quality, or the amount of contact with a member of the research team. The implications of the findings for the clinical use of SATs and for future research are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The efficacy of violence prediction: A meta-analytic comparison of nine risk assessment tools.

Actuarial risk assessment tools are used extensively to predict future violence, but previous studies comparing their predictive accuracies have produced inconsistent findings as a result of various methodological issues. We conducted meta-analyses of the effect sizes of 9 commonly used risk assessment tools and their subscales to compare their predictive efficacies for violence. The effect sizes were extracted from 28 original reports published between 1999 and 2008, which assessed the predictive accuracy of more than one tool. We used a within-subject design to improve statistical power and multilevel regression models to disentangle random effects of variation between studies and tools and to adjust for study features. All 9 tools and their subscales predicted violence at about the same moderate level of predictive efficacy with the exception of Psychopathy Checklist—Revised (PCL-R) Factor 1, which predicted violence only at chance level among men. Approximately 25% of the total variance was due to differences between tools, whereas approximately 85% of heterogeneity between studies was explained by methodological features (age, length of follow-up, different types of violent outcome, sex, and sex-related interactions). Sex-differentiated efficacy was found for a small number of the tools. If the intention is only to predict future violence, then the 9 tools are essentially interchangeable; the selection of which tool to use in practice should depend on what other functions the tool can perform rather than on its efficacy in predicting violence. The moderate level of predictive accuracy of these tools suggests that they should not be used solely for some criminal justice decision making that requires a very high level of accuracy such as preventive detention. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Straying in the methodology. I. Introduction

In recent decades clinical studies with patients have gained qualitative and quantitative importance. This has led to growing interest in methodological quality. At present, doctors need more methodological knowledge. Sophisticated software enables almost everyone to carry out the most exotic of statistical and epidemiological analyses, but the methodological implications are too often incompletely understood. In a series of column-like short articles methodological items will be discussed, in order to point out frequent errors, fallacies and practical problems.     

Does quality of reports of randomised trials affect estimates of intervention efficacy reported in meta-analyses?

BACKGROUND: Few meta-analyses of randomised trials assess the quality of the studies included. Yet there is increasing evidence that trial quality can affect estimates of intervention efficacy. We investigated whether different methods of quality assessment provide different estimates of intervention efficacy evaluated in randomised controlled trials (RCTs). METHODS: We randomly selected 11 meta-analyses that involved 127 RCTs on the efficacy of interventions used for circulatory and digestive diseases, mental health, and pregnancy and childbirth. We replicated all the meta-analyses using published data from the primary studies. The quality of reporting of all 127 clinical trials was assessed by means of component and scale approaches. To explore the effects of quality on the quantitative results, we examined the effects of different methods of incorporating quality scores (sensitivity analysis and quality weights) on the results of the meta-analyses. FINDINGS: The quality of trials was low. Masked assessments provided significantly higher scores than unmasked assessments (mean 2.74 [SD 1.10] vs 2.55 [1.20]). Low-quality trials (score < or = 2), compared with high-quality trials (score > 2), were associated with an increased estimate of benefit of 34% (ratio of odds ratios [ROR] 0.66 [95% CI 0.52-0.83]). Trials that used inadequate allocation concealment, compared with those that used adequate methods, were also associated with an increased estimate of benefit (37%; ROR=0.63 [0.45-0.88]). The average treatment benefit was 39% (odds ratio [OR] 0.61 [0.57-0.65]) for all trials, 52% (OR 0.48 [0.43-0.54]) for low-quality trials, and 29% (OR 0.71 [0.65-0.77]) for high-quality trials. Use of all the trial scores as quality weights reduced the effects to 35% (OR 0.65 [0.59-0.71]) and resulted in the least statistical heterogeneity. INTERPRETATION: Studies of low methodological quality in which the estimate of quality is incorporated into the meta-analyses can alter the interpretation of the benefit of intervention, whether a scale or component approach is used in the assessment of trial quality.     

The need for evidence-based, spiritually oriented psychotherapies.

Despite a proliferation of spiritually oriented psychotherapies during the past 2 decades in the mental health professions, outcome research is lacking. In this present article, 6 published outcome reviews that shed light on the efficacy of these psychotherapies are discussed. Although there is general support for the efficacy of spiritually oriented treatment approaches, the data base is relatively small and has methodological limitations. Spiritually oriented cognitive approaches for religious clients with depression and anxiety meet evidence-based standards of efficacy. Several other spiritually oriented approaches are probably efficacious but need additional investigation. Methodological recommendations for improving the quality of future studies in this domain are made. Philosophically and methodologically pluralistic research strategies that do not reduce spiritual phenomena into a naturalistic and materialistic framework are recommended. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Are interventions theory-based? Development of a theory coding scheme.

Objective: Interventions are increasingly described as theory-based; however, the basis for this is often not clear. Advancing behavioral science requires a good understanding of how interventions are informed by, and test, theory. This study aims to develop a reliable method for assessing the extent to which behavioral interventions are theory-based. Design: The reliability, usability, and comprehensiveness of an initial coding scheme were improved in 13 iterative stages on the basis of its application to 29 papers, from a systematic review of interventions to promote physical activity and healthy eating. Results: The final Theory Coding Scheme contained 19 items, each with satisfactory interrater reliabilities, coding whether a theory or model was mentioned, how theories were used in intervention design, how intervention evaluations tested theory, and the implications of the results for future theory development. Conclusion: The Theory Coding Scheme is an important methodological innovation, providing a research tool to reliably describe the theoretical base of interventions, inform evidence syntheses within literature reviews and meta-analyses, and stimulate the use of empirical data for theory development. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Viability of Designing for Construction Worker Safety

Designing for construction safety entails consideration of the safety of construction workers in the design of a project. Research studies have identified the design aspect of projects as being a significant contributing factor to construction site accidents. Designing to eliminate or avoid hazards prior to exposure on the jobsite is also listed as the top priority in the hierarchy of controls common to the safety and health professions. Widespread implementation of the concept in the United States by engineering and architecture firms, however, is lacking due to perceived industry and project barriers. Given its absence from standard design practice, a question arises as to the viability of designing for safety as an intervention in the construction industry. This paper presents a pilot study that was conducted to investigate the practice of addressing construction worker safety when designing a project and to determine the feasibility and practicality of such an intervention. Through interviews of architects and engineers, the study found that a large percentage of design professionals are interested and willing to implement the concept in practice. Among the perceived impacts of implementation, project cost and schedule were mentioned most often along with limitations being placed on design creativity. The results of the pilot study indicate that designing for safety is a viable intervention in construction. The factors that impact the consideration of safety in the design of a project do not entirely prohibit its implementation or make its implementation extremely impractical and therefore not feasible. Additionally, the outcomes of implementation provide sufficient motivation to implement the concept in practice. The paper describes the key changes needed for implementation of the concept in practice which include: a change in designer mindset toward safety; establishment of a motivational force to promote designing for safety; increase designer knowledge of the concept; incorporate construction safety knowledge in the design phase; utilize designers knowledgeable about design-for-safety modifications; make design for safety tools and guidelines available for use and reference; and mitigate designer liability exposure.     

Effectiveness of home care programmes for patients with incurable cancer on their quality of life and time spent in hospital: systematic review

OBJECTIVE: To investigate whether for patients with incurable cancer comprehensive home care programmes are more effective than standard care in maintaining the patients' quality of life and reducing their "readmission time" (percentage of days spent in hospital from start of care till death). DESIGN: Systematic review. METHODS: A computer aided search was conducted using the databases of Medline, Embase, CancerLit, and PsychLit. The search for studies and the assessment of the methodological quality of the relevant studies were performed by two investigators, blinded from each other. Prospective, controlled studies investigating the effects of a home care intervention programme on patients' quality of life or on readmission time were included in the analyses. RESULTS: Only 9 prospective controlled studies were found; eight were performed in the United States and 1 in the United Kingdom. Their methodological quality was judged to be moderate (median rating 62 on a 100 point scale). None of the studies showed a negative influence of home care interventions on quality of life. A significantly positive influence on the outcome measures was seen in 2 out of the 5 studies measuring patients' satisfaction with care, in 3/7 studies measuring physical dimensions of quality of life, in 1/6 studies measuring psychosocial dimensions, and in 2/5 studies measuring readmission time. The incorporation of team members' visits to patients at home or regular multidisciplinary team meetings into the intervention programme seemed to be related to positive results. CONCLUSIONS: The effectiveness of comprehensive home care programmes remains unclear. Given the enormity of the problems faced by society in caring for patients with terminal cancer, further research is urgently needed.     

Conducting Psychological Research in Medical Settings: Challenges, Limitations, and Recommendations for Effectiveness Research.

Efficacy research is insufficient and ineffective in changing practice, despite a large supporting psychotherapy efficacy literature. Concurrently, demand for supporting data and the burden on everyday practice for generating these data is increasing. The disconnect between efficacy and effectiveness is due to a lack of conceptual and methodological tools for testing efficacious procedures in real-world settings, particularly medical contexts. This article articulates the position that effectiveness studies pose more complex questions that require an alteration of conceptual and methodological frames of reference in order to make behavioral services available to the broadest patient base. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Local and Global Judgments of Confidence.

Studies of calibration have shown that people's mean confidence in their answers (local confidence) tends to be greater than their overall estimate of the percentage of correct answers (global confidence). Moreover, whereas the former exhibits overconfidence, the latter often exhibits underconfidence. Three studies present evidence that global underconfidence reflects a failure to make an allowance for correct answers that are likely to result from mere guessing and can be eliminated by informing participants of the dubious normative status of estimates below 50% (i.e., chance). Previously reported discrepancies between global and local confidence, it is concluded, arise less from possible methodological artifacts in assessment of local confidence than from normatively inappropriate assessments of global confidence. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Differential validity revived.

Analyzed 31 investigations of test validity in samples of Black and White workers to test null hypotheses that differences in validity between the 2 ethnic groups do not occur more often than chance expectation. The analyses differed from others recently addressing the same issue by dealing both with single-group validity and differential validity, by including additional data, and by making certain methodological changes. Because of low power and other deficiencies, the accumulated studies cannot furnish an unequivocal test of the issue. On the whole, however, their implications do not disconfirm the null hypothesis regarding chance differences in single-group validity, whereas they do appear to reject it with respect to differential validity. Nevertheless, tests do not consistently work better in one ethnic population than the other, even though they may function differently in one than the other. Ethnic differences in test validity are not a "pseudoproblem." (23 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A systematic and transparent approach for assessing the methodological quality of intervention effectiveness research: The Study Design and Implementation Assessment Device (Study DIAD).

Assessments of studies meant to evaluate the effectiveness of interventions, programs, and policies can serve an important role in the interpretation of research results. However, evidence suggests that available quality assessment tools have poor measurement characteristics and can lead to opposing conclusions when applied to the same body of studies. These tools tend to (a) be insufficiently operational, (b) rely on arbitrary post-hoc decision rules, and (c) result in a single number to represent a multidimensional construct. In response to these limitations, a multilevel and hierarchical instrument was developed in consultation with a wide range of methodological and statistical experts. The instrument focuses on the operational details of studies and results in a profile of scores instead of a single score to represent study quality. A pilot test suggested that satisfactory between-judge agreement can be obtained using well-trained raters working in naturalistic conditions. Limitations of the instrument are discussed, but these are inherent in making decisions about study quality given incomplete reporting and in the absence of strong, contextually based information about the effects of design flaws on study outcomes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The effectiveness of play therapy: Responding to the critics.

Notes that play therapy has long been criticized for a lack of adequate research base to prove its efficacy. For 6 decades, while play therapists conducted small research studies, critics challenged the utility and efficacy of play therapy as a viable psychotherapy intervention. The purpose of this study was to conduct a meta-analysis of 94 research studies focusing on the efficacy of play therapy, filial therapy, and combined play therapy and filial therapy. The following electronic databases were used in the authors 3-yr search for studies on play therapy: PsycLit, PsycINFO, ERIC, FirstSearch, and Dissertation Abstracts. Meta-analysis revealed a large positive effect on treatment outcomes with children. Play therapy appeared effective across modality, age, gender, clinical vs nonclinical populations, setting, and theoretical schools of thought. Additionally, positive play therapy effects were found to be greatest when there was parent involvement in treatment and an optimal number of sessions provided. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Evaluating alliance-focused intervention for potential treatment failures: A feasibility study and descriptive analysis.

This article describes a pilot study evaluating the feasibility of an approach developed to test the efficacy of a therapeutic intervention (brief relational therapy) for patients with whom it is difficult to establish a therapeutic alliance. In the first phase of the study, 60 patients were randomly assigned to either short-term dynamic therapy (STDP) or short-term cognitive therapy (CBT), and their progress in the first eight sessions of treatment was monitored. On the basis of a number of empirically derived criteria, 18 potential treatment failures were identified. In the second phase of the study, these identified patients were offered the option of being reassigned to another treatment. The 10 patients who agreed to switch treatments were reassigned either to the alliance-focused treatment, referred to as brief relational therapy (BRT), or a control condition. For patients coming from CBT, the control condition was STDP. For patients coming from STDP, the control condition was CBT. The results provide preliminary evidence supporting the potential value of BRT as an intervention that is useful in the context of alliance ruptures. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Contingency management for behavior change: Applications to promote brief smoking cessation among opioid-maintained patients.

Cigarette smoking is highly prevalent among patients who are being treated for opioid-dependence, yet there have been limited scientific efforts to promote smoking cessation in this population. Contingency management (CM) is a behavioral treatment that provides monetary incentives contingent upon biochemical evidence of drug abstinence. This paper discusses the results of two studies that utilized CM to promote brief smoking cessation among opioid-maintained patients. Participants in a pilot study were randomly assigned for a 2-week period to a Contingent group that earned monetary vouchers for providing biochemical samples that met criteria for smoking abstinence, or a Noncontingent group that earned monetary vouchers independent of smoking status (Dunn et al., 2008). Results showed Contingent participants provided significantly more smoking-negative samples than Noncontingent participants (55% vs. 5%, respectively). A second randomized trial that utilized the same 2-week intervention and provided access to the smoking cessation pharmacotherapy bupropion replicated the results of the pilot study (55% and 17% abstinence in Contingent and Noncontingent groups, respectively; Dunn et al, 2010). Relapse to illicit drug use was also evaluated prospectively and no association between smoking abstinence and relapse to illicit drug use was observed (Dunn et al., 2009). It will be important for future studies to evaluate participant characteristics that might predict better treatment outcome, to assess the contribution that pharmacotherapies might have alone or in combination with a CM intervention on smoking cessation and to evaluate methods for maintaining the abstinence that is achieved during this brief intervention for longer periods of time. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Quality and homogeneity monitoring of radiologists in the Turin mammographic screening. Comparison of the pilot phase results

The Turin mammographic screening program was designed to be carried out in several independent screening centers because of the large proportion of population involved (76,000 women aged 50 to 59 years). The first center began working in 1992 and the second center was opened in 1995. The latter center carried out an early pilot study in which 1024 women were examined to assess the homogeneity and the quality of the results. The results were then compared with European quality standards and with the results of both the pilot and the active screening periods in the first center. The results from the second center were very good as far as detection rate is concerned (7.8/1000-2.9/1000 in carcinoma < or = 1 cm). This rate is higher than the so-called "desirable" European standard. Recall rate and benign/malignant biopsy ratio were higher than the so-called "fair" European standard (recall rate: 7.1%; B/M biopsy ratio: 0.62). These results are slightly superior to those of the pilot period in the first center and slightly inferior to the results of active screening in the same center. The improvement relative to the pilot period in the first center (1991) is probably related to technical progress, such as the introduction of the double mammographic projection. The difference relative to the results from the active screening period in the first center (1992-1994) reflects different specific experience. Even though pilot periods have no statistical significance, they can be used for comparison, and in our experience they have proved, with positive results, the quality of the mammographic screening program provided to the Turin population.