The Medical Journal of Australia Internet peer-review study

BACKGROUND: Peer review of medical papers is a confidential consultancy between the reviewer and the journal editor, and has been criticised for its potential bias and inadequacy. We explored the potential of the internet for open peer review to see whether this approach improved the quality and outcome of peer review. METHODS: Research and review articles that had been accepted for publication in The Medical Journal of Australia (MJA) were published together with the reviewers' reports on the worldwide web, with the consent of authors and referees. Selected readers' e-mailed comments were electronically published as additional commentary; authors could reply or revise their paper in response to readers' comments. Articles were edited and published in print after this open review. FINDINGS: 60 (81%) of 74 authors agreed to take part in the study, together with 150 (92%) of 162 reviewers. There was no significant difference in the performance of commissioned reviewers before and during the study. Four articles were not included because of insufficient time before print publication. Of the remaining 56 papers, 28 received 52 comments from 42 readers (2% of readers submitted comments). Most readers' comments were short and specific, and seven articles were changed by the authors in response. INTERPRETATION: Open peer review is acceptable to most authors and reviewers. Postpublication review by readers on the internet is no substitute for commissioned prepublication review, but can provide editors with valuable input from individuals who would not otherwise be consulted. Readers also gain insight into the processes of peer review and publication.     

Anonymous reviewing and the peer-review process.

Suggests that S. J. Ceci and D. Peters (see record 1985-29014-001) should have carried their investigation of anonymous peer review a step further to determine what proportion of the 35.6% of papers whose authors were identified by the reviewers was published. It is suggested that it is time to try a peer review in which the identities of author and reviewer are made known to each other. (1 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Omentoplasty in the management of filarial lymphoedema

OBJECTIVE: Peer review is an important part of the publication of scientific results. We analysed the peer review process of abstracts submitted to the European Association for the Study of the Liver (EASL) congress in Copenhagen in 1995. DESIGN: Comparative study. All abstracts were reviewed by all six members of the EASL scientific committee. Furthermore, each abstract was evaluated by three to six experts within 17 specific study fields, appointed by the scientific committee. All abstracts received a score on a scale from 1 to 5, one being the best. For each abstract the average was calculated for the scores of the committee members and the expert reviewers separately. MAIN OUTCOME MEASURE: Abstract score. RESULTS: There was a significant difference in the scores between the committee members, who scored 3.31 on average, and the expert reviewers, who scored 3.02 on average. Within most of the study fields the tendency was that the expert reviewers gave the highest scores. Moreover, there was a significant variation in the estimates among the individual reviewers as well as between the 2 groups of reviewers. Only 25% of the abstracts that received the best scores by the expert reviewers were classified in the same way by the committee members. The predictive value of a positive evaluation by the committee members was 0.6, the predictive value of a negative evaluation was 0.90. CONCLUSIONS: There was a considerable variation in the peer review process for abstracts submitted to the 1995 EASL congress depending on selection of reviewers. Various types of peer review strategies should be tested and evaluated to secure the highest possible scientific quality of published research results.     

An evaluation of the Aetna pilot peer review project.

Describes a 6-mo trial peer review project in which the 3rd-party payer requested treatment reports from psychologist providers, submitted 45 selected cases to reviewers designated by the American Psychological Association, and received recommendation from these reviewers. The characteristics of the patients, the type of treatment provided, and the characteristics of the reviewers are outlined. Differences in the judged adequacy of treatment were related to factors such as the reviewer's gender and theoretical orientation, the frequency of sessions, the therapeutic orientation of the provider, and the gender of the patient. Problems related to poor documentation and the potential of peer review in improving patient care are discussed. (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Agreement and differences between reviewers in a peer review procedure

In Germany the statutory pension insurance institutions have started a quality assurance programme. Our institute developed a peer review procedure for screening the process quality of rehabilitation care. The peer review was tested in a pilot study. Our article refers to the examination of interrater reliability, intrarater reliability and reviewer bias. First of all, experienced doctors were trained in reviewing reports routinely written by rehabilitation doctors at discharge of their patients. The peers had to judge on 56 process criteria belonging to six categories (e.g. case history). The reliability coefficients were calculated for the overall judgement of each category and the overall judgement of the process quality of rehabilitation care. The coefficients of interrater reliability and the coefficients of average intrarater reliability range from sufficient to good. Only few reviewers showed a general tendency to harsh or lenient rating. The objectivity of the tested peer review procedure seems definitely higher than in American studies of peer review of hospital charts.     

APA/CHAMPUS peer review project: Implications for research and practice.

Describes the history of the American Psychological Association (APA) peer review of psychological services provided under the Civilian Health and Medical Program of the Uniformed Services (CHAMPUS), the operational details of the system, and some of the criticisms of the project. Research is summarized that examined the characteristics of peer reviewers, the effects of the theoretical orientation of the reviewer, and the effects of documented treatment progress. Implications for care providers, patients, and the profession are discussed. (18 ref) (PsycINFO Database Record (c) 2010 APA, all rights reserved)     

Why review articles on the health effects of passive smoking reach different conclusions

OBJECTIVE: To determine whether the conclusions of review articles on the health effects of passive smoking are associated with article quality, the affiliations of their authors, or other article characteristics. DATA SOURCES: Review articles published from 1980 to 1995 were identified through electronic searches of MEDLINE and EMBASE and from a database of symposium proceedings on passive smoking. ARTICLE SELECTION: An article was included if its stated or implied purpose was to review the scientific evidence that passive smoking is associated with 1 or more health outcomes. Articles were excluded if they did not focus specifically on the health effects of passive smoking or if they were not written in English. DATA EXTRACTION: Review article quality was evaluated by 2 independent assessors who were trained, followed a written protocol, had no disclosed conflicts of interest, and were blinded to all study hypotheses and identifying characteristics of articles. Article conclusions were categorized by the 2 assessors and by one of the authors. Author affiliation was classified as either tobacco industry affiliated or not, based on whether the authors were known to have received funding from or participated in activities sponsored by the tobacco industry. Other article characteristics were classified by one of the authors using predefined criteria. DATA SYNTHESIS: A total of 106 reviews were identified. Overall, 37% (39/106) of reviews concluded that passive smoking is not harmful to health; 74% (29/39) of these were written by authors with tobacco industry affiliations. In multiple logistic regression analyses controlling for article quality, peer review status, article topic, and year of publication, the only factor associated with concluding that passive smoking is not harmful was whether an author was affiliated with the tobacco industry (odds ratio, 88.4; 95% confidence interval, 16.4-476.5; P<.001). CONCLUSIONS: The conclusions of review articles are strongly associated with the affiliations of their authors. Authors of review articles should disclose potential financial conflicts of interest, and readers of review articles should consider authors' affiliations when deciding how to judge an article's conclusions.     

Differences between Experts and Novices in the Review of Engineering Journal Papers

This paper reports on a controlled experiment carried out to investigate the differences between novices and experts during the review of papers for engineering journals. A total of 17 young faculty members with PhD degrees were given instruction on the peer review process, and then had to review a short paper, for which the actual reviews were available. The quality of the reviews was assessed using criteria developed by editors of medical journals. The evidence obtained from this study helps to identify the critical areas in which reviewers fail to perform a good review, and then to develop strategies to overcome such limitations.     

When reviews attack: Ethics, free speech, and the peer review process.

The peer review process, whether formally applied in publication and grant review, or informally, such as exchange of ideas in scientific and professional newsgroups, has sparked controversy. Writers in this area agree that scholarly reviews that are inappropriate in tone are not uncommon. Indeed commentators have suggested rules and guidelines that can be used to improve the review process and to make reviewers more accountable. In this paper, the authors examine the relevance and impact of ethical codes on the conduct of peer review. They contend that the peer review process can be improved, not by a new set of rules, but through closer attention to the ethical principles to which psychologists already subscribe. The discussion considers the four principles of the Canadian Psychological Association in relation to peer review and to the six principles presented in the code of the American Psychological Association. (PsycINFO Database Record (c) 2010 APA, all rights reserved)