Omnibus Hypothesis Testing in Dominance-Based Ordinal Multiple Regression.

Often quantitative data in the social sciences have only ordinal justification. Problems of interpretation can arise when least squares multiple regression (LSMR) is used with ordinal data. Two ordinal alternatives are discussed, dominance-based ordinal multiple regression (DOMR) and proportional odds multiple regression. The Q2 statistic is introduced for testing the omnibus null hypothesis in DOMR. A simulation study is discussed that examines the actual Type I error rate and power of Q2 in comparison to the LSMR omnibus F test under normality and non-normality. Results suggest that Q2 has favorable sampling properties as long as the sample size-to-predictors ratio is not too small, and Q2 can be a good alternative to the omnibus F test when the response variable is non-normal. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of fixed-effects and mixed (random-effects) models for meta-analysis tests of moderator variable effects.

The growing popularity of meta-analysis has focused increased attention on the statistical models analysts are using and the assumptions underlying these models. Although comparisons often have been limited to fixed-effects (FE) models, recently there has been a call to investigate the differences between FE and random-effects (RE) models, differences that may have substantial theoretical and applied implications (National Research Council, 1992). Three FE models (including L. V. Hedges & I. Olkin's, 1985, and R. Rosenthal's, 1991, tests) and 2 RE models were applied to simulated correlation data in tests for moderator effects. The FE models seriously underestimated and the RE models greatly overestimated sampling error variance when their basic assumptions were violated, which caused biased confidence intervals and hypothesis tests. The implications of these and other findings are discussed as are methodological issues concerning meta-analyses. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A Monte Carlo study of the inferential properties of three methods of shape comparison

Three inferential morphometric methods, Euclidean distance matrix analysis (EDMA), Bookstein's edge-matching method (EMM), and the Procrustes method, were applied to facial landmark data. A Monte Carlo simulation was conducted with three sample sizes, ranging from n = 10 to 50, to assess type I error rates and the power of the tests to detect group differences for two- and three-dimensional representations of forms. Type I error rates for EMM were at or below nominal levels in both two and three dimensions. Procrustes in 2D and EDMA in 2D and 3D produced inflated type I error rates in all conditions, but approached acceptable levels with moderate cell sizes. Procrustes maintained error rates below the nominal levels in 2D. The power of EMM was high compared with the other methods in both 2D and 3D, but, conflicting EMM decisions were provided depending on which pair (2D) or triad (3D) of landmarks were selected as reference points. EDMA and Procrustes were more powerful in 2D data than for 3D data. Interpretation of these results must take into account that the data used in this simulation were selected because they represent real data that might have been collected during a study or experiment. These data had characteristics which violated assumptions central to the methods here with unequal variances about landmarks, correlated errors, and correlated landmark locations; therefore these results may not generalize to all conditions, such as cases with no violations of assumptions. This simulation demonstrates, however, limitations of each procedure that should be considered when making inferences about shape comparisons.     

Meta-analysis of multiple outcomes by regression with random effects

Earlier work showed how to perform fixed-effects meta-analysis of studies or trials when each provides results on more than one outcome per patient and these multiple outcomes are correlated. That fixed-effects generalized-least-squares approach analyzes the multiple outcomes jointly within a single model, and it can include covariates, such as duration of therapy or quality of trial, that may explain observed heterogeneity of results among the trials. Sometimes the covariates explain all the heterogeneity, and the fixed-effects regression model is appropriate. However, unexplained heterogeneity may often remain, even after taking into account known or suspected covariates. Because fixed-effects models do not make allowance for this remaining unexplained heterogeneity, the potential exists for bias in estimated coefficients, standard errors and p-values. We propose two random-effects approaches for the regression meta-analysis of multiple correlated outcomes. We compare their use with fixed-effects models and with separate-outcomes models in a meta-analysis of periodontal clinical trials. A simulation study shows the advantages of the random-effects approach. These methods also facilitate meta-analysis of trials that compare more than two treatments.     

Effect of error variance heterogeneity on the power of tests for regression slope differences.

In moderated regression analysis with both a continuous predictor and nominal-level (group membership) variables, there are conditions in which the hypothesis of equal slopes of the regression of Y onto X across groups is equivalent to the hypothesis of equality of X–Y correlations across groups. This research uses those conditions to investgate the impact of heterogeneity of error variance on the power accuracy of the F test for equality of regression slopes. The results show that even when sample sizes are equal, the test is not robust and, under unequal sample sizes, the pattern of excessively high or excessively low rejection rates can be severe. (PsycINFO Database Record (c) 2011 APA, all rights reserved)     

Meta-analytic interval estimation for bivariate correlations.

The currently available meta-analytic methods for correlations have restrictive assumptions. The fixed-effects methods assume equal population correlations and exhibit poor performance under correlation heterogeneity. The random-effects methods do not assume correlation homogeneity but are based on an equally unrealistic assumption that the selected studies are a random sample from a well-defined superpopulation of study populations. The random-effects methods can accommodate correlation heterogeneity, but these methods do not perform properly in typical applications where the studies are nonrandomly selected. A new fixed-effects meta-analytic confidence interval for bivariate correlations is proposed that is easy to compute and performs well under correlation heterogeneity and nonrandomly selected studies. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Stability of validity indices through test score ranges.

Effects of correction for restriction in range and variability of validity evidence throughout the predictor score range were examined in a sample of 16,230 inexperienced life insurance agents. The predictor variable was the Aptitude Index Battery (a biographical inventory) and the criterion was 1-year objective sales production. Data displayed nonlinearity and heteroscedasticity. Although variations in the validity coefficients and the slopes within the restricted samples were observed, these were generally accounted for by sampling error. When analyzed with linear regression techniques, data showed stronger relationships between the predictor and criterion at the upper score range; however, when analyzed with techniques appropriate for nonlinear, heteroscedastic data, validity evidence indicated better predictability at lower score ranges. Increasing the truncation of the distribution yielded increasingly overestimated slopes and corrected correlations. Influences of nonlinearity and heteroscedasticity and implications for the use of the direct restriction of range correction formula are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measurement error, instrumental variables and corrections for attenuation with applications to meta-analyses

MacMahon et al. present a meta-analysis of the effect of blood pressure on coronary heart disease, as well as new methods for estimation in measurement error models for the case when a replicate or second measurement is made of the fallible predictor. The correction for attenuation used by these authors is compared to others already existing in the literature, as well as to a new instrumental variable method. The assumptions justifying the various methods are examined and their efficiencies are studied via simulation. Compared to the methods we discuss, the method of MacMahon et al. may have bias in some circumstances because it does not take into account: (i) possible correlations among the predictors within a study; (ii) possible bias in the second measurement; or (iii) possibly differing marginal distributions of the predictors or measurement errors across studies. A unifying asymptotic theory using estimating equations is also presented.     

The predictive power of ridge regression: Some quasi-simulation results: Correction to Cattin.

Reports an error in the original article by Philippe Cattin (Journal of Applied Psychology, 1981, Vol. 66, No. 3, pp. 282-290). The fifth sentence in the first paragraph on page 284 contains an error. The sentence should read: "For intermediate values of k?, the ridge regression weights are 'weighted sums' of the OLS regression weights [not models] and of the zero-order sample correlations and tend to decrease in absolute value as k? increases." (The following abstract of this article originally appeared in record 1981-27117-001.) Reviews research indicating that ridge regression tends to improve the mean square error of prediction obtained with ordinary least squares (OLS) regression in a wide range of conditions. MMPI profiles of 861 psychiatric patients and diagnoses from 29 psychologists, as collected by P. E. Meehl (see record 1960-04396-001), were used to examine the gains in cross-validated multiple correlation obtained with ridge regression compared with OLS and equal weights as a function of sample size and ridge constant. A simple formula for estimating the ridge constant was also evaluated. Results that are related to recent developments concerning the use of Bayesian regression procedures show that ridge regression improves both the mean square error of prediction and the cross-validated multiple correlation obtained with OLS when the ratio sample size to number of predictor variables is relatively small. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Parametric statistics and levels of measurement: Factorial designs and multiple regression.

Let Y be a continuous, ordinal measure of a latent variable Θ. In general, for factorial designs, an analysis of variance of the observed variable Y cannot be used to draw inferences about main effects and interactions on the latent variable Θ even when the standard normality and equality of variance assumptions hold. If Y is a continuous, ordinal measure of a latent variable Θ; X?,…, Xn are continuous, ordinal measures of latent variables Φ?,…, Φn; and the observed measures have a multivariate normal distribution, then a multiple regression analysis of the observed criterion measure Y and predictors X?,…, Xn can be used to test hypotheses about multivariate associations among the latent variables. Furthermore, the predicted values Y′ are unbiased estimates of quantities that are monotonically related to predicted values on the latent criterion variable Θ. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Multiple comparison procedures for trimmed means.

Stepwise multiple comparison procedures (MCPs) based on least squares and trimmed estimators were compared for their rates of Type I error and their ability to detect true pairwise group differences. The MCPs were compared in unbalanced one-way completely randomized designs when normality and homogeneity of variance assumptions were violated. Results indicated that MCPs based on trimmed means and Winsorized variances controlled rates of Type I error, whereas MCPs based on least squares estimators typically could not, particularly when the data were highly skewed. However, MCPs based on least squares estimators were substantially more powerful than their counterparts based on trimmed means and Winsorized variances when the data were only moderately skewed, a finding which qualifies recommendations on the use of trimmed estimators offered in the literature. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

More Powerful Tests of Predictor Subsets in Regression Analysis Under Nonnormality.

It is well-known that for normally distributed errors parametric tests are optimal statistically, but perhaps less well-known is that when normality does not hold, nonparametric tests frequently possess greater statistical power than parametric tests, while controlling Type I error rate. However, the use of nonparametric procedures has been limited by the absence of easily performed tests for complex experimental designs and analyses and by limited information about their statistical behavior for realistic conditions. A Monte Carlo study of tests of predictor subsets in multiple regression analysis indicates that various nonparametric tests show greater power than the F test for skewed and heavy-tailed data. These nonparametric tests can be computed with available software. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Unsquared genetic correlations.

Demonstrates that correlations commonly used to provide an estimate of the ratio of genotypic to phenotypic variance require restrictive (and probably unreasonable) assumptions. When these assumptions are violated, the correlation in question will provide a biased estimate of the desired ratio. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Type I error rates for moderated multiple regression analysis.

The method of moderated multiple regression is increasingly being applied in the search for moderator variables in industrial and organizational psychology. Because of frequent failures of the method in revealing moderator effects in empirical studies—in which such effects are strongly expected—it has been suggested that the procedure may lack statistical power with respect to hypothesis tests about moderating effects and, therefore, is inappropriate for the purposes of conventional moderator analyses. We evaluated this conclusion with computer simulation data. Our study indicated that the method is not overly conservative and that the Type I error rate of moderated multiple regression is approximately .05 at α?=?.05. Moreover, a proposed alternative multivariate procedure, principal component regression, is shown to have a Type I error rate that approaches unity under ordinary conditions when applied to the evaluation of moderator effects. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The homogeneity assumption in differential prediction analysis: Does it really matter?

In simulation studies, the F test for differences in regression slopes has tended to distort nominal Type I and II error rates when the 2 subgroup error variances exceeded a 1.50:1 ratio. This study examines the frequency and extent that this ratio is violated within data sets relevant to applied psychology. The General Aptitude Test Battery (GATB) validity study database contained ability data and overall job performance ratings. The Project A military database contained both ability and personality data, along with job performance factor scores and an overall job performance rating. Results suggest that subgroup (White-Black, male-female) error variances are often homogeneous enough to support F test results from past empirical work. Enough heterogeneity was found, however, to urge applied psychologists investigating differential prediction to explore their data and consider the possibility of alternative statistical tests. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A generally robust approach for testing hypotheses and setting confidence intervals for effect sizes.

Standard least squares analysis of variance methods suffer from poor power under arbitrarily small departures from normality and fail to control the probability of a Type I error when standard assumptions are violated. This article describes a framework for robust estimation and testing that uses trimmed means with an approximate degrees of freedom heteroscedastic statistic for independent and correlated groups designs in order to achieve robustness to the biasing effects of nonnormality and variance heterogeneity. The authors describe a nonparametric bootstrap methodology that can provide improved Type I error control. In addition, the authors indicate how researchers can set robust confidence intervals around a robust effect size parameter estimate. In an online supplement, the authors use several examples to illustrate the application of an SAS program to implement these statistical methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Awareness of interpersonal style and self-evaluation

Multiple regression models are commonly used to control for confounding in epidemiologic research. Parametric regression models, such as multiple logistic regression, are powerful tools to control for multiple covariates provided that the covariate-risk associations are correctly specified. Residual confounding may result, however, from inappropriate specification of the confounder-risk association. In this paper, we illustrate the order of magnitude of residual confounding that may occur with traditional approaches to control for continuous confounders in multiple logistic regression, such as inclusion of a single linear term or categorization of the confounder, under a variety of assumptions on the confounder-risk association. We show that inclusion of the confounder as a single linear term often provides satisfactory control for confounding even in situations in which the model assumptions are clearly violated. In contrast, categorization of the confounder may often lead to serious residual confounding if the number of categories is small. Alternative strategies to control for confounding, such as polynomial regression or linear spline regression, are a useful supplement to the more traditional approaches.     

The predictive power of ridge regression: Some quasi-simulation results.

Reviews research indicating that ridge regression tends to improve the mean square error of prediction obtained with ordinary least squares (OLS) regression in a wide range of conditions. MMPI profiles of 861 psychiatric patients and diagnoses from 29 psychologists, as collected by P. E. Meehl (see record 1960-04396-001), were used to examine the gains in cross-validated multiple correlation obtained with ridge regression compared with OLS and equal weights as a function of sample size and ridge constant. A simple formula for estimating the ridge constant was also evaluated. Results that are related to recent developments concerning the use of Bayesian regression procedures show that ridge regression improves both the mean square error of prediction and the cross-validated multiple correlation obtained with OLS when the ratio sample size to number of predictor variables is relatively small. (19 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Testing hypotheses about ordinal interactions: Simulations and further comments.

This paper elaborates on several issues related to testing for the presence of ordinal interactions, as described by Bobko (1986). First, the philosophy underlying Bobko's approach is explicitly stated and compared with the traditional approach to testing for the presence of interactions. Second, two modifications of Bobko's approach are described. Third, the procedures for testing ordinal interactions are compared (on the basis of Type I and Type II error rates) with each other as well as to the traditional analysis of variance (ANOVA) approach. All variants of Bobko's procedure have comparable power across different sample sizes and experimental effect sizes. These procedures differ, however, in their likelihood of falsely concluding that an ordinal pattern is present. The traditional ANOVA approach (a) is noticeably lacking in power for detecting ordinal interactions and (b) commonly identifies significant main effects but not an interaction when, in fact, an ordinal interaction is present in the population. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Unbiased estimation of sampling variance of correlations.

In a recent article, P. E. Spector and E. L. Levine (1987) asserted that the estimate of sampling error variance used in validity generalization studies is biased when the number of correlations is relatively small. In addition, Spector and Levine maintained that the bias is such that the sampling error variance estimate seriously overestimates the actual variance of observed correlations. A partial replication of Spector and Levine's study showed that the alleged bias was due to a distributional artifact and that the sampling error estimate is not seriously biased. We review evidence from several Monte Carlo studies indicating that the sampling error estimate is quite accurate. (PsycINFO Database Record (c) 2010 APA, all rights reserved)