Insiders, outsiders, and efficiency in a National Science Foundation panel.

Assessed the efficiency of the peer review system during 3 National Science Foundation panel meetings held in 1982 and 1983. During these meetings, nearly 200 proposals received almost 1,400 reviews from "insiders" (panel members) and "outsiders" (ad hoc reviewers). Although the ratio of ad hoc time to panelist time was about 2:1, ad hoc reviewers' quantitative ratings had scant independent influence on final proposal ratings. Nevertheless, it may be that ad hoc reviewers provided crucial qualitative information, acting more as expert witnesses than as judges. The outcome of about a third of the proposals (the very good and the very poor) could be reliably predicted by independent panelist assessments that occurred before the panel meetings. (PsycINFO Database Record (c) 2010 APA, all rights reserved)