Degeneracy and discrete receiver operating characteristic rating data

RATIONALE AND OBJECTIVES: Observer performance studies sometimes use too few cases for estimating diagnostic accuracy from binormal receiver operating characteristic (ROC) curves. One important problem is degenerate data sets. We compared a new algorithm, RSCORE4, with the exact-solution approach to degeneracy in ROCFIT and with the Wilcoxon statistic. METHODS: Degenerate ROC solutions result from empty cells in the data matrix. We addressed this problem by adding a small constant to empty cells in a maximum-likelihood program, RSCORE4. When this method failed, the program branched to a pattern-search algorithm. We tested the program in a series of Monte Carlo studies. RESULTS: RSCORE4 converged to nondegenerate solutions in every case and gave results closer to population values than ROCFIT or Wilcoxon. ROCFIT converged to exact-fit degenerate solutions, those with zero or infinite parameter values, in more than 40% of the samples. The Wilcoxon statistic was biased. CONCLUSION: RSCORE4 seems to outperform other currently recommended methods for dealing with degeneracy.     

Multireader receiver operating characteristic studies: a comparison of study designs

RATIONALE AND OBJECTIVES: Traditionally, multireader receiver operating characteristic (ROC) studies have used a "paired-case, paired-reader" design. The statistical power of such a design for inferences about the relative accuracies of the tests was assessed and compared with alternative designs. METHODS: The noncentrality parameter of an F statistic was used to compute power as a function of the reader and patient sample sizes and the variability and correlation between readings. RESULTS: For a fixed-power and Type I error rate, the traditional design reduces the number of verified cases required. A hybrid design, in which each reader interprets a different sample of patients, reduces the number of readers, total readings, and reading required per reader. The drawback is a substantial increase in the number of verified cases. CONCLUSION: The ultimate choice of study design depends on the nature of the tests being compared, limiting resources, a priori knowledge of the magnitude of the correlations and variability and logistic complexity.     

Confidence intervals for the receiver operating characteristic area in studies with small samples

RATIONALE AND OBJECTIVES: The authors performed this study to address two practical questions. First, how large does the sample size need to be for confidence intervals (CIs) based on the usual asymptotic methods to be appropriate? Second, when the sample size is smaller than this threshold, what alternative method of CI construction should be used? MATERIALS AND METHODS: The authors performed a Monte Carlo simulation study where 95% CIs were constructed for the receiver operating characteristic (ROC) area and for the difference between two ROC areas for rating and continuous test results--for ROC areas of moderate and high accuracy--by using both parametric and nonparametric estimation methods. Alternative methods evaluated included several bootstrap CIs and CIs with the Student t distribution. RESULTS: For the difference between two ROC areas, CIs based on the asymptotic theory provided adequate coverage even when the sample size was very small (20 patients). In contrast, for a single ROC area, the asymptotic methods do not provide adequate CI coverage for small samples; for ROC areas of high accuracy, the sample size must be large (more than 200 patients) for the asymptotic methods to be applicable. The recommended alternative (bootstrap percentile, bootstrap t, or bootstrap bias-corrected accelerated method) depends on the estimation approach, format of the test results, and ROC area. CONCLUSION: Currently, there is not a single best alternative for constructing CIs for a single ROC area for small samples.     

Practical application of receiver operating characteristic curve on laboratory diagnosis

The receiver operating characteristic (ROC) curve represents characteristics specific to an examination (diagnostic sensitivity and specificity) and is useful for evaluation and comparison of the diagnostic accuracy. However, the ROC curve is not widely used at present. In this symposium, we showed how to draw this curve and its practical utilization, using as examples the diagnosis of the diabetic and impaired glucose tolerance group and the diagnosis of deep-seated fungal infection and acute myocardial infarction. In the ROC curve, true positive is plotted on the vertical axis and false positive on the horizontal axis. This curve is readily drown and visually shows the diagnostic accuracy that can not be clarified by histograms. The advantages of this curve are as follows. 1. Diagnostic accuracy can be compared. 2. The significance of the reference interval in diagnosis can be evaluated. 3. The diagnostic cut-off value can be determined using this curve. 4. Combined with prevalence, the diagnostic probability can be represented quantitatively. The points that require attention are differences in the ROC curve according to selection of subjects (including controls), the time factor (disease stage) and severity (disease condition). By paying attention to these points, the ROC curve can be used as a simple and useful method in laboratory diagnosis. We hope that this curve will be widely used.     

Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design

RATIONALE AND OBJECTIVES: The authors conducted a series of null-case Monte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) method for comparing modalities with multireader receiver operating characteristic (ROC) discrete rating data. MATERIALS AND METHODS: Monte Carlo simulations were performed by using discrete ratings on fully crossed factorial designs with two modalities and three, five, and 10 hypothetical readers. The null hypothesis was true for all simulations. The population ROC areas, latent variable structures, case sample sizes, and normal/abnormal case sample ratios used in another study were used in these simulations. RESULTS: For equal allocation ratios and small (Az = 0.702) and moderate (Az = 0.855) ROC areas, the empirical type I error rate closely matched the nominal alpha level. For very large ROC areas (Az = 0.961), however, the empirical type I error rate was somewhat smaller than the nominal alpha level. This conservatism increased with decreasing case sample size and asymmetric normal/abnormal case allocation ratio. The empirical type I error rate was sometimes slightly larger than the nominal alpha level with many cases and few readers, where there was large residual, relatively small treatment-by-case interaction and relatively large treatment-by-reader interaction. CONCLUSION: The results suggest that the DBM method provides trustworthy alpha levels with discrete ratings when the ROC area is not too large and case and reader sample sizes are not too small. In other situations, the test tends to be somewhat conservative or slightly liberal.     

Recollection and familiarity deficits in amnesia: Convergence of remember-know, process dissociation, and receiver operating characteristic data.

Previous studies using the process dissociation and the remember–know procedures led to conflicting conclusions regarding the effects of anterograde amnesia on recollection and familiarity. We argue that these apparent contradictions arose because different models were used to interpret the results and because differences in false-alarm rates between groups biased the estimates provided by those models. A reanalysis of those studies with a dual-process signal-detection model that incorporates response bias revealed that amnesia led to a pronounced reduction in recollection and smaller but consistent reduction in familiarity. To test the assumptions of the model and to further assess recognition deficits in amnesics, we examined receiver operating characteristics (ROCs) in amnesics and controls. The ROCs of the controls were curved and asymmetrical, whereas those of the amnesics were curved and symmetrical. The results supported the predictions of the model and indicated that amnesia was associated with deficits in both recollection and familiarity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Three approaches to regression analysis of receiver operating characteristic curves for continuous test results

Individual gastric glands of the stomach are composed of cells of different phenotypes. These are derived from multipotent progenitor stem cells located at the isthmus region of the gland. Previous cell lineage analyses suggest that gastric glands, as in the colon and small intestine, are invariably monoclonal by adult stages. However, little is known about the ontogenetic progression of glandular clonality in the stomach. To examine this issue, we employed an in situ cell lineage marker in female mice heterozygous for an X-linked transgene. We found that stomach glands commence development as polyclonal units, but by adulthood (6 weeks), the majority progressed to monoclonal units. Our analysis suggests that at least three progenitor cells are required to initiate the development of individual gastric glands if they are analyzed just after birth. Hence, unlike the colon and small intestine, stomachs showed a significant fraction (10-25%) of polyclonal glands at adult stages. We suggest that these glands persist from polyclonal glands present in the embryonic stomach and hypothesize that they represent a subpopulation of glands with larger numbers of self-renewing stem cells.     

Word frequency and receiver operating characteristic curves in recognition memory: Evidence for a dual-process interpretation.

Dual-process models of the word-frequency mirror effect posit that low-frequency words are recollected more often than high-frequency words, producing the hit rate differences in the word-frequency effect, whereas high-frequency words are more familiar, producing the false-alarm-rate differences. In this pair of experiments, the authors demonstrate that the analysis of receiver operating characteristic (ROC) curves provides critical information in support of this interpretation. Specifically, when participants were required to discriminate between studied nouns and their plurality reversed complements, the ROC curve was accurately described by a threshold model that is consistent with recollection-based recognition. Further, the plurality discrimination ROC curves showed characteristics consistent with the interpretation that participants recollected low-frequency items more than high-frequency items. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Assessment of dysregulated children using the Child Behavior Checklist: A receiver operating characteristic curve analysis.

Disorders of self-regulatory behavior are common reasons for referral to child and adolescent clinicians. Here, the authors sought to compare 2 methods of empirically based assessment of children with problems in self-regulatory behavior. Using parental reports on 2,028 children (53% boys) from a U.S. national probability sample of the Child Behavior Checklist (CBCL; T. M. Achenbach & L. A. Rescorla, 2001), the receiver operating characteristic curve analysis was applied to compare scores on the Posttraumatic Stress Problems Scale (PTSP) of the CBCL with the CBCL Dysregulation Profile (DP), identified using latent class analysis of the Attention Problems, Aggressive Behavior, and Anxious/Depressed scales of the CBCL. The CBCL–PTSP score demonstrated an area under the curve of between .88 and .91 for predicting membership in the CBCL–DP profile for boys and for girls. These findings suggest that the CBCL–PTSP, which others have shown does not uniquely identify children who have been traumatized, does identify the same profile of behavior as the CBCL–DP. Therefore, the authors recommend renaming the CBCL–PTSP the Dysregulation Short Scale and provide some guidelines for the use of the CBCL–DP scale and the CBCL–PTSP in clinical practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Aging selectively impairs recollection in recognition memory for pictures: Evidence from modeling and receiver operating characteristic curves.

Young and older adults were tested on recognition memory for pictures. The Yonelinas high threshold (YHT) model, a formal implementation of 2-process theory, fit the response distribution data of both young and older adults significantly better than a normal unequal variance signal-detection model. Consistent with this finding, nonlinear z-transformed receiver operating characteristic curves were obtained for both groups. Estimates of recollection from the YHT model were significantly higher for young than for older adults. This deficit was not a consequence of a general decline in memory; older adults showed comparable overall accuracy and in fact a nonsignificant increase in their familiarity scores. Implications of these results for theories of recognition memory and the mnemonic deficit associated with aging are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Analysis of prevention program effectiveness with clustered data using generalized estimating equations.

Experimental studies of prevention programs often randomize clusters of individuals rather than individuals to treatment conditions. When the correlation among individuals within clusters is not accounted for in statistical analysis, the standard errors are biased, potentially resulting in misleading conclusions about the significance of treatment effects. This study demonstrates the generalized estimating equations (GEE) method, focusing specifically on the GEE-independent method, to control for within-cluster correlation in regression models with either continuous or binary outcomes. The GEE-independent method yields consistent and robust variance estimates. Data from Project DARE, a youth substance abuse prevention program, are used for illustration. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment

Interstitial deletions in chromosome 22 and features associated with CATCH-22 syndrome have been reported in patients with conotruncal congenital heart anomalies. Absent pulmonary valve syndrome is characterized by absent or rudimentary pulmonary valve cusps, absent ductus arteriosus, conoventricular septal defect, and massive dilation of the pulmonary arteries. Because absence of the ductus arteriosus is a key element in the pathogenesis of this syndrome and aortic arch malformations are frequently seen in patients with CATCH-22 syndrome, we hypothesized that patients with absent pulmonary valve syndrome would have a high incidence of deletions in the critical region of chromosome 22. Eight patients with absent pulmonary valve syndrome were studied. Metaphase preparations were examined with fluorescent in situ hybridization of the N25 (D22S75) probe to the critical region of chromosome 22q11.2. Deletions were detected in 6 of 8 patients. The presence of deletions in chromosome 22 in most of the patients we have examined with a diagnosis of absent pulmonary valve syndrome supports a specific genetic and embryologic mechanism involving the interaction of the neural crest and the primitive aortic arches as one cause of congenital absence of the pulmonary valve.     

Between- and within-cluster covariate effects in the analysis of clustered data

Standard methods for the regression analysis of clustered data postulate models relating covariates to the response without regard to between- and within-cluster covariate effects. Implicit in these analyses is the assumption that these effects are identical. Example data show that this is frequently not the case and that analyses that ignore differential between- and within-cluster covariate effects can be misleading. Consideration of between- and within-cluster effects also helps to explain observed and theoretical differences between mixture model analyses and those based on conditional likelihood methods. In particular, we show that conditional likelihood methods estimate purely within-cluster covariate effects, whereas mixture model approaches estimate a weighted average of between- and within-cluster covariate effects.     

Assessing gamma frailty models for clustered failure time data

Proportional hazards frailty models use a random effect, so called frailty, to construct association for clustered failure time data. It is customary to assume that the random frailty follows a gamma distribution. In this paper, we propose a graphical method for assessing adequacy of the proportional hazards frailty models. In particular, we focus on the assessment of the gamma distribution assumption for the frailties. We calculate the average of the posterior expected frailties at several followup time points and compare it at these time points to 1, the known mean frailty. Large discrepancies indicate lack of fit. To aid in assessing the goodness of fit, we derive and estimate the standard error of the mean of the posterior expected frailties at each time point examined. We give an example to illustrate the proposed methodology and perform sensitivity analysis by simulations.     

余热锅炉结构形式及运行特性研究综述

余热锅炉是冶金工业中回收余热的重要设备,本文综述了余热锅炉结构形式的发展,烟气侧流动换热的解析计算、数值计算和实验研究,以及基于动态特性模拟的结构优化.在此基础上指出了余热锅炉现有研究中的不足,对下一步的研究方向进行了探讨.     

Scintigraphic diagnosis of cirrhosis: a receiver operator characteristic analysis of the common interpretative criteria

Receiver operator characteristic (ROC) techniques were used to determine relative importance of heterogeneity, bone marrow uptake, hepatomegaly, splenomegaly, and focal lesions in the scintigraphic diagnosis of cirrhosis, as well as to determine whether each of the criteria should be used as an identifier (high sensitivity) or as a discriminator (high specificity). Heterogeneity, splenomegaly, and bone marrow uptake were found to be good identifiers of hepatic disease. However, heterogeneity was found to be a poor discriminator for cirrhosis, splenomegaly showed some value as a discriminator, and bone marrow uptake was found to be a reliable discriminator for alcoholic liver disease. The presence of focal lesions was a good discriminator for metastatic disease, and was not specific for cirrhosis. Hepatomegaly also was not specific for cirrhosis. This method of analysis provides a relatively simple method for assembling a hierarchical guide to diagnostic criteria for the interpreting physician.     

Statistical methodology: III. Receiver operating characteristic (ROC) curves

Measures including sensitivity, specificity, and positive and negative predictive values have been traditionally used to assess a diagnostic test's ability to detect the presence or absence of disease. Receiver operating characteristic (ROC) curve analysis allows visual evaluation of the trade-offs between sensitivity and specificity associated with different values of the test result, or different "cutpoints" for defining a positive result. The purpose of this article is to define, construct, and interpret a ROC curve using a hypothetical example applicable to emergency medicine practice.     

Random-effects regression models for clustered data with an example from smoking prevention research.

Proposes a random-effects regression model for analysis of clustered data. Unlike ordinary regression analysis of clustered data, random-effects regression models do not assume that each observation is independent but do assume that data within clusters are dependent to some degree. The degree of this dependency is estimated along with estimates of the usual model parameters, thus adjusting these effects for the dependency resulting from the clustering of the data. A maximum marginal likelihood solution is described, and available statistical software for the model is discussed. An analysis of a dataset in which students are clustered within classrooms and schools is used to illustrate features of random-effects regression analysis, relative to both individual-level analysis that ignores the clustering of the data, and classroom-level analysis that aggregates the individual data. (PsycINFO Database Record (c) 2010 APA, all rights reserved)