Roaming through methodology. VII. Reproducibility of measurements

Reproducibility measurements are important for a proper interpretation of medical data. Kappa is the most adequate measure for categorical variables. Kappa adjusts the observed agreement for chance agreement. The interpretation of kappa is rather difficult. The kappa value is influenced by the number of categories used for classification and the prevalence of scores of the observers. For continuous variables the Pearson correlation coefficient can be used, keeping in mind its ignoring systematic errors and its dependence on the heterogeneity of the data. Another method to assess reproducibility for continuous variables is the method of limits of agreement. This method distinguishes systematic and random errors, and quantifies the differences in the dimension of the measurements. In general, the interpretation of the various measures of agreement is helped by a visual presentation of the data in a table or figure.     

Roaming through methodology. IX. The interpretation of subgroup analyses]

The results of subgroup analyses can be very informative, in particular for clinicians interested in evidence pertaining most directly to a particular patient. However, subgroup analyses may also be potentially misleading. The strength of evidence for subgroup effects depends on the question whether hypotheses have been defined prior to analysis, whether potential problems regarding multiple comparisons have been considered and whether there is biological plausibility of the effects. Using these guidelines the reader of a trial report should be able to decide if presented subgroup effects are of clinical importance or if the overall result is a better estimate of treatment effect.     

Roaming through methodology. VI. Medical perspective on the causes of death

The Central Statistics Office (CBS) annually publishes the causes of death of decreased Dutch residents. It is generally assumed that the registration of causes of death is unreliable and provides only limited additional information. It is a fact that the physicians who perform the post mortems may supply incomplete or incorrect information. Hence, individual causes of death are sometimes coded incorrectly. This necessitates specific instruction of medical students. For studies of large groups, misclassification of individual causes of death is of less importance. However, the systematics of the registration has to be taken into account. It is important that causes of death are registered uniformly facilitating (inter)national comparisons. With the aid of a causes of death register, characteristics of (groups of) the population can be related to the distribution of the causes of death; at the level of the individual patient the causes of death may afford insight into causes and effects of disease. The latter is possible by a temporary link of patient data with the individual causes of death.     

Roaming through methodology. III. Randomization at the level of the physicians

In some studies randomization at the patient level is not possible; randomization at the doctor's level can then be applied. However, statistical analysis at the level of the individual patient data can lead to wrong conclusions in such a study design. By basing the analysis on the averages per doctor a statistical test can be performed which guarantees the correct significance level. The correct sample size can be found by means of a correction factor that depends on the variation between the doctors. The data of a study with randomization at the doctor's level can be analysed in a more sophisticated way by means of so-called 'mixed models' that take account of possible variation between doctors.     

Roaming through the methodology. VIII. Pilot studies: sense and nonsense

A pilot study is often performed before the start of a large study. Its aim is improving the methodological quality and evaluating the feasibility. The results of a pilot study are also often used to gain an impression of the efficacy of an intervention. For this purpose a pilot is absolutely unsuited. However, the estimate of the effect of an intervention in small studies such as pilot studies is determined to a large extent by chance. When conducting small studies the chance of publication bias is large. Small studies of adequate methodological quality should be published, whether the results are positive or negative, significant or non-significant, because combination in a later cumulative meta-analysis may lead to sufficient power to assess the efficacy of an experimental intervention.     

Roaming through methodology. V. The prevention paradox: few with a high risk versus many with a moderate risk

With a good primary prevention programme a number of healthy individuals will never know that they had a narrow escape from disease and maybe even death. Large numbers of individuals participate in such a programme, while only a few of those would have fallen ill without it. Nevertheless such a population approach often will yield a larger health benefit (on a population level) than an intervention that only aims at the high risk group in the population. That has been called the prevention paradox. The choice between a population or a high risk approach has prompted lengthy debates in the Netherlands. Epidemiology offers a methodology to estimate the effects of different approaches. The health impact will be determined by the distribution of exposure in the population and the risk function of exposure. In the final decision on a prevention strategy other considerations such as cost-effectiveness of interventions, negative side effects, possibilities of identification of high risk groups and ethical issues can play a part.     

Improved Bonferroni-type multiple testing procedures.

[Correction Notice: An erratum for this article was reported in Vol 104(2) of Psychological Bulletin (see record 2008-06007-001). An error was made in the author note on page 145. Correspondence should be addressed to Burt S. Holland, Department of Statistics, Temple University, Speakman Hall (006-00), Philadelphia, Pennsylvania 19122. Margaret DiPonzio Copenhaver is now at Merck Sharp & Dohme Research Laboratories, West Point, Pennsylvania.] The Bonferroni multiple comparisons procedure is customarily used when doing several simultaneous tests of significance in relatively nonstandard situations in which other methods do not apply. We review some new and improved competitors to the Bonferroni procedure, that although constraining generalized Type I error probability to be at most α, afford increased power in exchange for increased complexity in implementation. An improvement to the weighted form of the Bonferroni procedure is also presented. Several data sets are reanalyzed with the new methods. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Comparison of closed testing procedures for pairwise testing of means.

A Monte Carlo simulation was conducted to compare 9 pairwise multiple comparison procedures. Procedures were evaluated on the basis of any-pair power and all-pairs power. No procedure was found to be uniformly most powerful. A modification due to A. J. Hayter (1986) of Fisher's least significant difference was found to provide the best combination of ease of use and moderately high any-pair power in most cases. Pilot or exploratory studies can expect good power results with this relatively simple procedure. The greatest all-pairs power was usually provided by 1 of 2 partition-based versions of E. Peritz's (1970) procedure. Confirmatory studies will require such complex methods but may also need larger sample sizes than have been customary in psychological research. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Weight gain through self-control procedures.

21 underweight (at least 10%) undergraduates were assigned to either a self-reinforcement condition, a self-punishment condition, or to a discussion-reflection control condition. Ss received 1 weekly treatment session over a 5-wk period. Following treatment, the self-reinforcement groups gained significantly more pounds (kilograms) than either the self-punishment or the control group. This effect was maintained at a 12-wk follow-up. The punishment and control groups did not differ significantly at posttest or at follow-up. Similar results were obtained when percentage of weight change was analyzed, with the treatment effect significant at follow-up. (17 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

The teaching of diagnostic methodology in nursing schools]

The results of examining patients with parasagittal meningiomas were first compared by angiography and intraoperative ultrasound diagnosis. The significance of each method was defined. A correlation was found between the data obtained by the two techniques and the functional status of the upper sagittal sinus by taking into account each angiographic phase. It is concluded that intraoperative ultrasound diagnosis is of highly informative significance and it should be obligatorily used while removing parasagittal meningiomas in order to maintain the anatomic integrity of the upper sagittal sinus.     

On Carson's comment re diagnostic testing.

Comments on Robert Carson's article titled The status of diagnostic testing (see record 2005-07925-004). The article focuses on the importance of psychodiagnostic testing states very clearly and consistently a position shared by many in our profession, but one which is predicated on a number of assumptions which--as value judgments--lack empirical validity and, in fact, serve to engender and perpetuate interprofessional rivalry, professional mediocrity, and scientific stagnation. This writer does not deny the right to hold to and express value judgments per se; rather, he wishes to suggest alternatives to those underlying Carson's argument. Three assumptions are implied in the latter: (a) the "tripartite organization" of professions, i.e., psychiatry, clinical psychology, and psychiatric social work, constitutes a satisfactory state of affairs and should endure; (b) unique contributions to the service endeavor must arise from each profession and will obtain in the case of psychology via the psychodiagnostic enterprise; and (c) the service function (treatment?) is implemented, facilitated, or benefited as a result of the psychodiagnostic activity (testing) of psychologists. The commenter responds to the assumptions. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Some effects of testing methodology on children's cross-modal performance.

Tested 40 41-57 mo old children on the number of trials they took to learn the identical form-discrimination problem given in either visual or tactual modalities in either "held" (E gave S the reward) or "platform" (S himself picked up the reward) conditions. It appears that when testing methodology is equated in the 2 modalities, cross-modal transfer of a simple form discrimination is not asymmetrical, but rather is equivalent to the transfer from the tactual to the visual modality. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

A comparison of two procedures for testing multiple contrasts.

Scheffe's test for contrasts is frequently used when a relatively small set of contrasts is of interest. When these contrasts have been planned prior to the data analysis, the Bonferroni t statistic provides a reasonable alternative. Comparisons are made of critical values for the 2 tests. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Automation of testing procedures: Elwood replies to Moses.

Argues that J. L. Moses' (1969) response to D. L. Elwood's research (see record 1970-03011-001) into automated administration of the Wechsler Adult Intelligence Scale (WAIS) reflects a lack of appreciation of the use of such testing and a reaction against progress in the area of psychological assessment. Results indicated that automated testing was accurate, reliable, time-effective, and well-accepted. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Modification of study time and grades through self-control procedures.

Evaluated the efficacy of a self-control treatment package to increase daily study time and grades of 44 undergraduates. The additive contribution of individual elements to the treatment package, as well as the potential difference in setting proximal vs distal goals, were also investigated through the use of the following treatment groups: (1) self-monitoring, distal goals, financial contingency, and group meetings; (2) self-monitoring, distal goals, and financial contingency; (3) self-monitoring and distal goals; (4) self-monitoring and proximal goals; (5) self-monitoring alone; (6) waiting-list control. Results show that, without differing from each other, the 1st 3 groups were all superior to self-monitoring and waiting-list control in terms of time spent studying. No significant improvement was found for grades. Results are discussed in terms of implications for potential use in applied settings. (French abstract) (28 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Bedside diagnostic testing of body fluids

Numerous bedside diagnostic modalities are appropriate for the practice of emergency medicine. The proliferation of sophisticated technology is likely to increase both the availability and accuracy of commercial testing products. If health care reform in the United States results in a relaxation of the CLIA regulations, there will be a rapid expansion of research and development aimed at the biotechnology market. How much this would pertain to hospital-based emergency practice remains to be seen. Cost containment pressures may act in both directions on the utilization of available bedside technology. Although these tests are often less expensive than centralized laboratory determinations, the ready availability of near-patient testing may result in an increase in use that negates the lower cost. As with other diagnostic modalities, a thoughtful, considered approach based on scientific evidence will be necessary to formulate the appropriate use of bedside testing in individual emergency practice settings.     

The IPCS Collaborative Study on Neurobehavioral Screening Methods: II. Protocol design and testing procedures

This paper describes the development of the protocol for the International Programme on Chemical Safety (IPCS)-sponsored Collaborative Study on Neurobehavioral Screening Methods, including background on the methods and chemicals selected, as well as details concerning the conduct of the collaborative study, including proficiency testing, range-finding and main study. Participating laboratories in the collaborative study received training in the conduct and scoring of the behavioral tests and each laboratory received a video training film to train additional personnel as needed. Each of the eight laboratories that chose to participate in the study completed proficiency testing and assessed seven representative chemicals using a functional observational battery and automated motor activity assessment. The seven chemicals studied were acrylamide, bis-acrylamide, p,p'-DDT, lead acetate, parathion, toluene, and triethyl tin. Participants received coded samples of the chemicals from a common source. Each laboratory derived doses for single and repeated administration based on the determination of a within-laboratory acute "top dose." Animal strains were not standardized and laboratory conditions were standardized to a limited degree in order to judge the general utility and robustness of these procedures in a diversity of testing situations.