Mixed-model pairwise multiple comparisons of repeated measures means.

One approach to the analysis of repeated measures data allows researchers to model the covariance structure of the data rather than presume a certain structure, as is the case with conventional univariate and multivariate test statistics. This mixed-model approach was evaluated for testing all possible pairwise differences among repeated measures marginal means in a Between-Subjects?×?Within-Subjects design. Specifically, the authors investigated Type I error and power rates for a number of simultaneous and stepwise multiple comparison procedures using SAS (1999) PROC MIXED in unbalanced designs when normality and covariance homogeneity assumptions did not hold. J. P. Shaffer's (1986) sequentially rejective step-down and Y. Hochberg's (1988) sequentially acceptive step-up Bonferroni procedures, based on an unstructured covariance structure, had superior Type I error control and power to detect true pairwise differences across the investigated conditions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparisons of methods for multiple hypothesis testing in neuropsychological research.

Hypothesis testing with multiple outcomes requires adjustments to control Type I error inflation, which reduces power to detect significant differences. Maintaining the prechosen Type I error level is challenging when outcomes are correlated. This problem concerns many research areas, including neuropsychological research in which multiple, interrelated assessment measures are common. Standard p value adjustment methods include Bonferroni-, Sidak-, and resampling-class methods. In this report, the authors aimed to develop a multiple hypothesis testing strategy to maximize power while controlling Type I error. The authors conducted a sensitivity analysis, using a neuropsychological dataset, to offer a relative comparison of the methods and a simulation study to compare the robustness of the methods with respect to varying patterns and magnitudes of correlation between outcomes. The results lead them to recommend the Hochberg and Hommel methods (step-up modifications of the Bonferroni method) for mildly correlated outcomes and the step-down minP method (a resampling-based method) for highly correlated outcomes. The authors note caveats regarding the implementation of these methods using available software. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs.

When a meta-analysis on results from experimental studies is conducted, differences in the study design must be taken into consideration. A method for combining results across independent-groups and repeated measures designs is described, and the conditions under which such an analysis is appropriate are discussed. Combining results across designs requires that (a) all effect sizes be transformed into a common metric, (b) effect sizes from each design estimate the same treatment effect, and (c) meta-analysis procedures use design-specific estimates of sampling variance to reflect the precision of the effect size estimates. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Multiphase mixed-effects models for repeated measures data.

Behavior that develops in phases may exhibit distinctively different rates of change in one time period than in others. In this article, a mixed-effects model for a response that displays identifiable regimes is reviewed. An interesting component of the model is the change point. In substantive terms, the change point is the time when development switches from one phase to another. In a mixed-effects model, the change point can be a random coefficient. This possibility allows individuals to make the transition from one phase to another at different ages or after different lengths of time in treatment. Two examples are reviewed in detail, both of which can be estimated with software that is widely available. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

A note on repeated measures in the study of short-term memory.

Keppel's (1965) objection to averaging over repeated measures is discussed. Several possibilities for minimizing proactive interference are suggested, including increasing time between tests. Consideration is given to acceptance of a constant amount of proactive interference over conditions in which other variables are studied. The advantages of using repeated measures in regard to efficiency as well as interpretation are noted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiplicity, directional (Type III) errors, and the Null Hypothesis.

L. V. Jones and J. W. Tukey (2000) pointed out that the usual 2-sided, equal-tails null hypothesis test at level ot can be reinterpreted as simultaneous tests of 2 directional inequality hypotheses, each at level α/2, and that the maximum probability of a Type I error is α/2 if the truth of the null hypothesis is considered impossible. This article points out that in multiple testing with familywise error rate controlled at ot, the directional error rate (assuming all null hypotheses are false) is greater than α/2 and can be arbitrarily close to α. Single-step, step-down, and step-up procedures are analyzed, and other error rates, including the false discovery rate, are discussed. Implications for confidence interval estimation and hypothesis testing practices are considered. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Longitudinal prediction of child outcomes from differing measures of parenting in a low-income sample.

This study examined predictions from preschool parenting measures to middle childhood cognitive and socioemotional child outcomes to explore whether parenting assessment methodologies that require more time, training, and expense yield better predictions of child outcomes than less intensive methodologies. Mother-child dyads (N=278) in low-income African American families were assessed when the child was in preschool, using maternal report, the Home Observation for Measurement of the Environment-Short Form (P. Baker & F. Mott, 1989; R. Bradley & B. Caldwell, 1984), and structured observational measures of parenting. Child outcomes reported by children, mothers, teachers, and direct assessment were collected 4 years later. All parenting methodologies showed some predictive value; however, observational parenting measures showed the strongest and most consistent predictions of child outcomes. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A general method of analysis of frequency data for multiple classification designs.

There are many research areas for which the data can only be obtained in terms of counts or frequencies within specified categories rather than in terms of measures. The techniques of analysis of variance are not amenable to this kind of data. However, the use of "multiple contingency analysis" permits the investigator to make use of factorial designs even though his data are in the form of frequencies. The development of the method and an illustrative problem are presented. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Lower level mediation in multilevel models.

Multilevel models are increasingly used to estimate models for hierarchical and repeated measures data. The authors discuss a model in which there is mediation at the lower level and the mediational links vary randomly across upper level units. One repeated measures example is a case in which a person's daily stressors affect his or her coping efforts, which affect his or her mood, and both links vary randomly across persons. Where there is mediation at the lower level and the mediational links vary randomly across upper level units, the formulas for the indirect effect and its standard error must be modified to include the covariance between the random effects. Because no standard method can estimate such a model, the authors developed an ad hoc method that is illustrated with real and simulated data. Limitations of this method and characteristics of an ideal method are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Basic statistics and the inconsistency of multiple comparison procedures.

This paper has two main themes. First, the various statistical measures used in this journal are summarized and their interrelationships described by way of a flow chart. These are the pooled standard deviation, the pooled variance or mean square error (MSE), the standard error of each treatment mean (SEM) and of the difference between two treatment means (SED), and the least difference between two means which is significant at (e.g.) the 5% level of significance (LSD[5%]). The last three measures can be displayed as vertical bars in graphs, and the relationship between the lengths of these bars is graphically illustrated. It is suggested that the LSD is the most useful of these three measures. Second, when the experimenter has no prior hypotheses to be tested using analysis of variance "contrasts," a multiple comparison procedure (MCP) that examines all pair-wise differences between treatment means, may be appropriate. In this paper a fictitious experimental data set is used to compare several well-known MCPs by focussing on a particular operating known characteristic, the consistency of the results between an overall analysis of all treatments and an analysis of a subset of the experimental treatments... (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Confidence intervals in repeated-measures designs: The number of observations principle.

Since the publication of Loftus and Masson’s (1994) method for computing confidence intervals (CIs) in repeated-measures (RM) designs, there has been uncertainty about how to apply it to particular effects in complex factorial designs. Masson and Loftus (2003) proposed that RM CIs for factorial designs be based on number of observations rather than number of participants. However, determining the correct number of observations for a particular effect can be complicated, given the variety of effects occurring in factorial designs. In this paper the authors define a general “number of observations” principle, explain why it obtains, and provide step-by-step instructions for constructing CIs for various effect types. The authors illustrate these procedures with numerical examples. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparing intervention outcomes in smokers treated for single versus multiple behavioral risks.

This study examined treatment outcomes among smokers with single versus multiple behavioral risks. Data were drawn from smokers (N = 2,326) participating in 3 population-based, multibehavioral studies targeting cigarette smoking, high-fat diets, and high-risk sun exposure. Outcomes were compared for participants at risk for smoking only (13%), smoking plus 1 additional risk factor (37%), and smoking plus 2 additional risk factors (50%). The smoking only group tended to be female, older, more educated, lighter smokers, in the preparation stage of change for smoking, with more previous quit attempts and longer periods of abstinence. At 12 and 24 months follow-up, treatment of 1 or 2 coexisting risk factors did not decrease the effectiveness of smoking cessation treatment, and treatment for the coexisting factors was effective. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparative outcomes for individual cognitive-behavior therapy, supportive-expressive group psychotherapy, and sertraline for the treatment of depression in multiple sclerosis.

This study compared the efficacy of 3 16-week treatments for depression in 63 patients with multiple sclerosis (MS) and major depressive disorder (MDD): individual cognitive- behavioral therapy (CBT), supportive expressive group therapy (SEG), and the antidepressant sertraline. Significant reductions were seen from pre- to posttreatment in all measures of depression. Intent-to-treat and completers analyses using the Beck Depression Inventory (BDI; A. T. Beck, C. H. Ward, M. Medelson, J. Mock, & J. Erbaugh, 1961) and MDD diagnosis found that CBT and sertraline were more effective than SEG at reducing depression. These results were largely supported by the BDI-18, which eliminates BDI items confounded with MS. However, the Hamilton Rating Scale for Depression (M. Hamilton, 1960) did not show consistent differences between treatments. Reasons for this inconsistency are discussed. These findings suggest that CBT or sertraline is more likely to be effective in treating MDD in MS compared with supportive group treatments. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Item Response Theory and Computerized Adaptive Testing: Implications for Outcomes Measurement in Rehabilitation.

Objective: To evaluate computerized adaptive testing (CAT) measures of rehabilitation outcomes. Study Design: Physical functioning questions were calibrated via item response theory (IRT) and administered with CAT software. Subjects: 485 adults interviewed during postacute care rehabilitation (simulation study) and 26 adults who completed CAT and personal interviews (CAT pilot study). Main Outcome Measures: Patient acceptance and respondent burden, reliability, and discriminant validity. Results: In the simulation study, CAT-based estimates correlated highly (r=.93 and r=.98) with criteria, minimized ceiling and floor effects, and reduced respondent burden while achieving high reliability. Pilot study patients preferred self-administered CAT surveys, and CAT scores discriminated well across severity levels. Conclusions: CAT software has considerable potential to improve physical functioning measurement in rehabilitation settings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A note on the inconsistency inherent in the necessity to perform multiple comparisons.

The comparison of multiple groups raises the question of the appropriate unit in which to evaluate the research. Thus it would be possible to hold errors constant per comparison, per hypothesis, per experiment, per group, or even per S. "… the most defensible decision is to divide our work into separate tests of hypotheses and to hold constant the expected number of errors per hypothesis tested." (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Maternal employment and child cognitive outcomes: The importance of analytic approach.

J. Brooks-Gunn, W. J. Han, and J. Waldfogel (2002; see record 2002-17576-005) and the National Institute of Child Health and Human Development Early Child Care Research Network (ECCRN; 2000b; see record 2000-02736-015) came to different conclusions about the effects of maternal employment--although they were addressing similar questions using the same data set. Brooks-Gunn et al. concluded that maternal employment in infancy has a negative effect on children's cognitive abilities at age 3, whereas the ECCRN found that early nonmaternal care is not related to children's cognitive abilities in their first 3 years. The authors account for this difference by comparing 2 approaches to data analysis: a top-down testing of continuous variables (the approach used by the ECCRN, 2000b) and an a priori comparison approach that involves pairwise testing of specific dichotomous contrasts (the approach used by Brooks-Gunn et al., 2002). This comparison illustrates the critical importance of analytic approach. It also suggests that Brooks-Gunn et al.'s conclusion from this data set is overstated and should not be used on its own as the basis for practical or policy decisions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Training in the use of psychotherapy outcome assessment measures at psychology internship sites.

American Psychological Association accredited psychology internship training programs (N = 407) were surveyed concerning their attitudes, beliefs, and practices with regard to outcome assessment measures. Results indicated that 47% of surveyed sites use outcome measures for assessment, and 66% used these measures for diagnostic purposes. In addition, 79% of respondents supported using outcome assessment measures to evaluate client progress, 61% supported training interns in the use of outcome assessment measures, and 87% felt outcome assessment measures would increase in importance in the future. The discrepancy between support for outcome assessment measures and actual use is discussed and recommendations provided. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing the multiple in the multiple read-out model of visual word recognition.

This study provided a test of the multiple criteria concept used for lexical decision, as implemented in J. Grainger and A. M. Jacobs's (1996) multiple read-out model. This account predicts more inhibition (or less facilitation) from a masked neighbor when accuracy is stressed more but more facilitation (or less inhibition) when the speed of responding is emphasized more. The authors tested these predictions by stressing accuracy (Experiment 1) and response speed (Experiment 2). The results of Experiment 1 showed a stronger neighbor-inhibition effect in the stress-on-accuracy condition than in the control condition. The results of Experiment 2 showed facilitation because of the neighbor prime in the stress-on-speed condition relative to the control condition. These results corroborate the multiple criteria account. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Unraveling the differential effects of motivational and skills, social, and self-management measures from traditional predictors of college outcomes.

The authors report on a large-scale study examining the effects of self-reported psychosocial factors on 1st-year college outcomes. Using a sample of 14,464 students from 48 institutions, the authors constructed hierarchical regression models to measure the predictive validity of the Student Readiness Inventory, a measure of psychosocial factors. Controlling for institutional effects and traditional predictors, the authors tested the effects of motivational and skill, social, and self-management measures on academic performance and retention. Academic Discipline was incrementally predictive of academic performance (grade-point average) and retention. Social Activity and Emotional Control also helped predict academic performance and retention, whereas Commitment to College and Social Connection offered incremental prediction of retention. This study elaborates recent meta-analytic findings (S. Robbins et al., 2004), demonstrating the salience of a subset of motivational, social, and self-management factors. Future research questions include how measures of psychosocial factors can be used to aid students, the salience of these measures over the entire college experience and for predicting job performance, and the need for testing theoretical models for explaining postsecondary educational outcomes incorporating traditional, motivational, self-management, and social engagement factors. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing for Suspected Impairments and Dissociations in Single-Case Studies in Neuropsychology: Evaluation of Alternatives Using Monte Carlo Simulations and Revised Tests for Dissociations.

In neuropsychological single-case studies, a patient is compared with a small control sample. Methods of testing for a deficit on Task X, or a significant difference between Tasks X and Y, either treat the control sample statistics as parameters (using z and zD) or use modified t tests. Monte Carlo simulations demonstrated that if z is used to test for a deficit, the Type I error rate is high for small control samples, whereas control of the error rate is essentially perfect for a modified t test. Simulations on tests for differences revealed that error rates were very high for zD. A new method of testing for a difference (the revised standardized difference test) achieved good control of the error rate, even with very small sample sizes. A computer program that implements this new test (and applies criteria to test for classical and strong dissociations) is made available. (PsycINFO Database Record (c) 2010 APA, all rights reserved)