Selecting and assessing challenge problems

Abstract

Organisations conducting research programs often focus the work of their scientists and technologists on challenge problems (CPs). These challenges are designed to ensure that progress is measurable and relevant to the goals of the program sponsor. Generating and selecting pertinent CPs is difficult, as is assessing their value. We describe a method of generating and selecting CPs and its application in a highly collaborative, multi-organisation research program. Thirty-eight biologists, chemists, mathematicians and computer scientists across academic, commercial and government organisations generated and ranked their top choices from among 12 richly described candidate challenge problems. A ranked-choice voting formula was applied. Five CPs were highly scored; the remaining seven were distributed across a lower range of scores. The program sponsor subsequently directed researchers to address six CPs, including the elected five. Analysis of the rationales that participants offered for their CP rankings revealed four domain-independent dimensions of value: capability, speed, impact and synergy. These dimensions of value can help managers of interdisciplinary research programs systematically select a portfolio of CPs that will efficiently apply utilise resources towards program goals and facilitate measurement of scientific progress.