Receiver operating characteristic curves for analysis of the results of cervicovaginal smears. A useful quality improvement tool

OBJECTIVES: To assess the value of receiver operating characteristic curves (ROCs) as a tool for improvement in the evaluation of cervicovaginal smear results. DESIGN: We reviewed the results of cervicovaginal smear interpretation for 1 year by 10 different cytopathologists working in the same laboratory (total number of cases, 45,356) and compared them with results of histologic evaluation of corresponding biopsies (n = 2090, 4.6%), the latter taken to be the gold standard. Receiver operating characteristic curves were generated for the laboratory as a whole and for individual cytopathologists. RESULTS: Analysis of these receiver operating characteristic curves was surprisingly informative. They showed that cytopathologists with different diagnostic thresholds could be distinguished from each other. A difference in diagnostic threshold could be distinguished from a difference in diagnostic accuracy. CONCLUSIONS: We conclude that while receiver operating characteristic curves for cervicovaginal smear interpretation have limitations, the results can be used for quality improvement.     

Word frequency and receiver operating characteristic curves in recognition memory: Evidence for a dual-process interpretation.

Dual-process models of the word-frequency mirror effect posit that low-frequency words are recollected more often than high-frequency words, producing the hit rate differences in the word-frequency effect, whereas high-frequency words are more familiar, producing the false-alarm-rate differences. In this pair of experiments, the authors demonstrate that the analysis of receiver operating characteristic (ROC) curves provides critical information in support of this interpretation. Specifically, when participants were required to discriminate between studied nouns and their plurality reversed complements, the ROC curve was accurately described by a threshold model that is consistent with recollection-based recognition. Further, the plurality discrimination ROC curves showed characteristics consistent with the interpretation that participants recollected low-frequency items more than high-frequency items. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Aging selectively impairs recollection in recognition memory for pictures: Evidence from modeling and receiver operating characteristic curves.

Young and older adults were tested on recognition memory for pictures. The Yonelinas high threshold (YHT) model, a formal implementation of 2-process theory, fit the response distribution data of both young and older adults significantly better than a normal unequal variance signal-detection model. Consistent with this finding, nonlinear z-transformed receiver operating characteristic curves were obtained for both groups. Estimates of recollection from the YHT model were significantly higher for young than for older adults. This deficit was not a consequence of a general decline in memory; older adults showed comparable overall accuracy and in fact a nonsignificant increase in their familiarity scores. Implications of these results for theories of recognition memory and the mnemonic deficit associated with aging are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Analysis of clustered data in receiver operating characteristic studies

Clustered data is not simply correlated data, but has its own unique aspects. In this paper, various methods for correlated receiver operating characteristic (ROC) curve data that have been extended specifically to clustered data are reviewed. For those methods that have not yet been extended, suggestions for their application to clustered ROC studies are provided. Various methods with respect to their ability to meet either of two objectives of the analysis of clustered ROC data are compared to consider a variety of ROC indices and their accessibility to researchers. The available statistical methods for clustered data vary in the range of indices that can be considered and in their accessibility to researchers. Parametric models permit all indices to be considered but, owing to computational complexity, are the least accessible of available methods. Nonparametric methods are much more accessible, but only permit estimation and inference about ROC curve area. The jackknife method is the most accessible and permits any index to be considered. Future development of methods for clustered ROC studies should consider the continuation ratio model, which will permit the application of widely available software for the analysis of mixed generalized linear models. Another area of development should be in the adoption of bootstrapping methods to clustered ROC data.     

Practical application of receiver operating characteristic curve on laboratory diagnosis

The receiver operating characteristic (ROC) curve represents characteristics specific to an examination (diagnostic sensitivity and specificity) and is useful for evaluation and comparison of the diagnostic accuracy. However, the ROC curve is not widely used at present. In this symposium, we showed how to draw this curve and its practical utilization, using as examples the diagnosis of the diabetic and impaired glucose tolerance group and the diagnosis of deep-seated fungal infection and acute myocardial infarction. In the ROC curve, true positive is plotted on the vertical axis and false positive on the horizontal axis. This curve is readily drown and visually shows the diagnostic accuracy that can not be clarified by histograms. The advantages of this curve are as follows. 1. Diagnostic accuracy can be compared. 2. The significance of the reference interval in diagnosis can be evaluated. 3. The diagnostic cut-off value can be determined using this curve. 4. Combined with prevalence, the diagnostic probability can be represented quantitatively. The points that require attention are differences in the ROC curve according to selection of subjects (including controls), the time factor (disease stage) and severity (disease condition). By paying attention to these points, the ROC curve can be used as a simple and useful method in laboratory diagnosis. We hope that this curve will be widely used.     

Multireader receiver operating characteristic studies: a comparison of study designs

RATIONALE AND OBJECTIVES: Traditionally, multireader receiver operating characteristic (ROC) studies have used a "paired-case, paired-reader" design. The statistical power of such a design for inferences about the relative accuracies of the tests was assessed and compared with alternative designs. METHODS: The noncentrality parameter of an F statistic was used to compute power as a function of the reader and patient sample sizes and the variability and correlation between readings. RESULTS: For a fixed-power and Type I error rate, the traditional design reduces the number of verified cases required. A hybrid design, in which each reader interprets a different sample of patients, reduces the number of readers, total readings, and reading required per reader. The drawback is a substantial increase in the number of verified cases. CONCLUSION: The ultimate choice of study design depends on the nature of the tests being compared, limiting resources, a priori knowledge of the magnitude of the correlations and variability and logistic complexity.     

Confidence intervals for the receiver operating characteristic area in studies with small samples

RATIONALE AND OBJECTIVES: The authors performed this study to address two practical questions. First, how large does the sample size need to be for confidence intervals (CIs) based on the usual asymptotic methods to be appropriate? Second, when the sample size is smaller than this threshold, what alternative method of CI construction should be used? MATERIALS AND METHODS: The authors performed a Monte Carlo simulation study where 95% CIs were constructed for the receiver operating characteristic (ROC) area and for the difference between two ROC areas for rating and continuous test results--for ROC areas of moderate and high accuracy--by using both parametric and nonparametric estimation methods. Alternative methods evaluated included several bootstrap CIs and CIs with the Student t distribution. RESULTS: For the difference between two ROC areas, CIs based on the asymptotic theory provided adequate coverage even when the sample size was very small (20 patients). In contrast, for a single ROC area, the asymptotic methods do not provide adequate CI coverage for small samples; for ROC areas of high accuracy, the sample size must be large (more than 200 patients) for the asymptotic methods to be applicable. The recommended alternative (bootstrap percentile, bootstrap t, or bootstrap bias-corrected accelerated method) depends on the estimation approach, format of the test results, and ROC area. CONCLUSION: Currently, there is not a single best alternative for constructing CIs for a single ROC area for small samples.     

Assessment of dysregulated children using the Child Behavior Checklist: A receiver operating characteristic curve analysis.

Disorders of self-regulatory behavior are common reasons for referral to child and adolescent clinicians. Here, the authors sought to compare 2 methods of empirically based assessment of children with problems in self-regulatory behavior. Using parental reports on 2,028 children (53% boys) from a U.S. national probability sample of the Child Behavior Checklist (CBCL; T. M. Achenbach & L. A. Rescorla, 2001), the receiver operating characteristic curve analysis was applied to compare scores on the Posttraumatic Stress Problems Scale (PTSP) of the CBCL with the CBCL Dysregulation Profile (DP), identified using latent class analysis of the Attention Problems, Aggressive Behavior, and Anxious/Depressed scales of the CBCL. The CBCL–PTSP score demonstrated an area under the curve of between .88 and .91 for predicting membership in the CBCL–DP profile for boys and for girls. These findings suggest that the CBCL–PTSP, which others have shown does not uniquely identify children who have been traumatized, does identify the same profile of behavior as the CBCL–DP. Therefore, the authors recommend renaming the CBCL–PTSP the Dysregulation Short Scale and provide some guidelines for the use of the CBCL–DP scale and the CBCL–PTSP in clinical practice. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

On the generalization of the receiver operating characteristic analysis to the population of readers and cases with the jackknife method: an assessment

Interstitial deletions in chromosome 22 and features associated with CATCH-22 syndrome have been reported in patients with conotruncal congenital heart anomalies. Absent pulmonary valve syndrome is characterized by absent or rudimentary pulmonary valve cusps, absent ductus arteriosus, conoventricular septal defect, and massive dilation of the pulmonary arteries. Because absence of the ductus arteriosus is a key element in the pathogenesis of this syndrome and aortic arch malformations are frequently seen in patients with CATCH-22 syndrome, we hypothesized that patients with absent pulmonary valve syndrome would have a high incidence of deletions in the critical region of chromosome 22. Eight patients with absent pulmonary valve syndrome were studied. Metaphase preparations were examined with fluorescent in situ hybridization of the N25 (D22S75) probe to the critical region of chromosome 22q11.2. Deletions were detected in 6 of 8 patients. The presence of deletions in chromosome 22 in most of the patients we have examined with a diagnosis of absent pulmonary valve syndrome supports a specific genetic and embryologic mechanism involving the interaction of the neural crest and the primitive aortic arches as one cause of congenital absence of the pulmonary valve.     

Monte Carlo validation of a multireader method for receiver operating characteristic discrete rating data: factorial experimental design

RATIONALE AND OBJECTIVES: The authors conducted a series of null-case Monte Carlo simulations to evaluate the Dorfman-Berbaum-Metz (DBM) method for comparing modalities with multireader receiver operating characteristic (ROC) discrete rating data. MATERIALS AND METHODS: Monte Carlo simulations were performed by using discrete ratings on fully crossed factorial designs with two modalities and three, five, and 10 hypothetical readers. The null hypothesis was true for all simulations. The population ROC areas, latent variable structures, case sample sizes, and normal/abnormal case sample ratios used in another study were used in these simulations. RESULTS: For equal allocation ratios and small (Az = 0.702) and moderate (Az = 0.855) ROC areas, the empirical type I error rate closely matched the nominal alpha level. For very large ROC areas (Az = 0.961), however, the empirical type I error rate was somewhat smaller than the nominal alpha level. This conservatism increased with decreasing case sample size and asymmetric normal/abnormal case allocation ratio. The empirical type I error rate was sometimes slightly larger than the nominal alpha level with many cases and few readers, where there was large residual, relatively small treatment-by-case interaction and relatively large treatment-by-reader interaction. CONCLUSION: The results suggest that the DBM method provides trustworthy alpha levels with discrete ratings when the ROC area is not too large and case and reader sample sizes are not too small. In other situations, the test tends to be somewhat conservative or slightly liberal.     

A comparison of different approaches for fitting centile curves to birthweight data

Centile charts, plotted by week of gestational age, are widely used clinically to decide whether a child is 'small-for-dates', indicating that there may have been some cause of intra-uterine growth retardation. A variety of different statistical techniques have been proposed for constructing centile charts. Some methods are distribution free and work on the basis of obtaining and then smoothing the empirical centiles. In others the centiles are calculated assuming a Normal distribution either of the raw data or of transformed data. In Scotland the details of each maternity admission are entered onto an SMR2 form and these records are stored centrally by the Scottish Health Service. A data set consisting of all live births occurring in Scottish hospitals between 1980 and 1992 is used to compare different approaches to constructing centile charts.     

Receiver operator characteristic analysis of endoscopy as a test for gastritis

Endoscopic evaluation of the presence or absence of gastritis is often performed in lieu of biopsy and histologic diagnosis. The purpose of our study was to assess the value of endoscopic examination as a diagnostic test for gastritis. Two endoscopists prospectively assessed the antrum of 73 patients undergoing upper gastrointestinal endoscopy and graded, on a scale of 0-4 (0 = completely absent, 4 = definitely present), the likelihood of gastritis. The following features were also assessed at the time of endoscopy: erythema, nodularity, erosion, edema, and friability. Two concomitant antral biopsies (3 cm from the pylorus on the greater curvature of the stomach) were performed regardless of the endoscopic impression. The histologic findings were graded independently on a scale of 0-3 by two pathologists who were not aware of the endoscopic findings. The following histologic features were graded: acute inflammation, chronic inflammation, lymphoid aggregates, intestinal metaplasia, and quantity of Helicobacter pylori organisms. Receiver operator characteristic analysis, a method derived from signal detection theory, assesses the trade-off of sensitivity and specificity over all cutoff points of a test and is considered the best method by which to compare tests and determine the diagnostic utility of a given test. Receiver operator characteristic analysis gave an area of 0.65 +/- 0.01 SE for endoscopy as a test for gastritis (0.5 = chance, 1 = perfect) as defined by the histologic presence of inflammation. Additionally, endoscopy as a test for the presence of histologically proven Helicobacter pylori gave an area of 0.55 +/- 0.01 SE. All endoscopically graded features treated as separate tests for gastritis and/or H. pylori gave areas of approximately 0.44-0.61, indicative of a poor test. While H. pylori was always associated with at least some degree of inflammation, linear regression analysis revealed no correlation among any of the histologic features or of any histologic feature with any endoscopic feature. We conclude that a tissue diagnosis is essential for the proper diagnosis of gastritis.     

Diagnostic power of lecithin/sphingomyelin ratio and fluorescence polarization assays for respiratory distress syndrome compared by relative operating characteristic curves

Gluten-free diets have been used in the treatment of patients with dermatitis herpetiformis in our department since 1967. Of the 212 patients with dermatitis herpetiformis attending between 1967 and 1992, 133 managed to take the diet, and 78 of these achieved complete control of their rash by diet alone. Of the remaining 55 patients taking a gluten-free diet, all but three were taking partial diets; over half of these patients managed to substantially reduce the dose of medication required. Of the 77 patients taking a normal diet, eight entered spontaneous remission, giving a remission rate of 10%; a further two patients who had been taking gluten-free diets were found to have remitted when they resumed normal diets. Loss of IgA from the skin was observed in 10 of 41 (24%) patients taking strict gluten-free diets. These patients had been taking their diets for an average of 13 years (range 5-24 years), and their rash had been controlled by diet alone for an average of 10 years (range 3-16 years). The advantages of a gluten-free diet in the management of patients with dermatitis herpetiformis are: (i) the need for medication is reduced or abolished; (ii) there is resolution of the enteropathy, and (iii) patients experience a feeling of well-being after commencing the diet. Thus, we propose that a gluten-free diet is the most appropriate treatment for patients with dermatitis herpetiformis.     

A comparison of four approaches to robust regression.

Reviews the vulnerability of the ordinary least squares estimates to extreme or abnormal observations and presents 4 alternative approaches to linear robust regression analysis. All are set up so that the final estimates of the regression coefficients are minimally influenced by the observations with large residuals. Under consideration are the procedures advanced by Huber, Hampel, Andrews (the sine estimate), and Tukey (the biweight estimate). Numerical examples are given that document the adequate performance of the alternative estimates in the presence of influential observations. In addition, when no outliers are apparent, only marginal differences exist between the robust estimates and those based on the least squares procedure. (15 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Recollection and familiarity deficits in amnesia: Convergence of remember-know, process dissociation, and receiver operating characteristic data.

Previous studies using the process dissociation and the remember–know procedures led to conflicting conclusions regarding the effects of anterograde amnesia on recollection and familiarity. We argue that these apparent contradictions arose because different models were used to interpret the results and because differences in false-alarm rates between groups biased the estimates provided by those models. A reanalysis of those studies with a dual-process signal-detection model that incorporates response bias revealed that amnesia led to a pronounced reduction in recollection and smaller but consistent reduction in familiarity. To test the assumptions of the model and to further assess recognition deficits in amnesics, we examined receiver operating characteristics (ROCs) in amnesics and controls. The ROCs of the controls were curved and asymmetrical, whereas those of the amnesics were curved and symmetrical. The results supported the predictions of the model and indicated that amnesia was associated with deficits in both recollection and familiarity. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Measurement error in the analysis of interaction effects between continuous predictors using multiple regression: Multiple indicator and structural equation approaches.

Unreliability of measures produces bias in regression coefficients. Such measurement error is particularly problematic with the use of product terms in multiple regression because the reliability of the product terms is generally quite low relative to its component parts. The use of confirmatory factor analysis as a means of dealing with the problem of unreliability was explored in a simulation study. The design compared traditional regression analysis (which ignores measurement error) with approaches based on latent variable structural equation models that used maximum-likelihood and weighted least squares estimation criteria. The results showed that the latent variable approach coupled with maximum-likelihood estimation methods did a satisfactory job of interaction analysis in the presence of measurement error in terms of Type I and Type II errors. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Scintigraphic diagnosis of cirrhosis: a receiver operator characteristic analysis of the common interpretative criteria

Receiver operator characteristic (ROC) techniques were used to determine relative importance of heterogeneity, bone marrow uptake, hepatomegaly, splenomegaly, and focal lesions in the scintigraphic diagnosis of cirrhosis, as well as to determine whether each of the criteria should be used as an identifier (high sensitivity) or as a discriminator (high specificity). Heterogeneity, splenomegaly, and bone marrow uptake were found to be good identifiers of hepatic disease. However, heterogeneity was found to be a poor discriminator for cirrhosis, splenomegaly showed some value as a discriminator, and bone marrow uptake was found to be a reliable discriminator for alcoholic liver disease. The presence of focal lesions was a good discriminator for metastatic disease, and was not specific for cirrhosis. Hepatomegaly also was not specific for cirrhosis. This method of analysis provides a relatively simple method for assembling a hierarchical guide to diagnostic criteria for the interpreting physician.