Measuring progress in Premo order-verification |
| |
| Authors: | Ryan B Bond Curtis C Ober Patrick M Knupp |
| |
| Affiliation: | (1) Aerosciences, Dept. 1515, Sandia National Laboratories, P.O. Box 5800, MS 0825, Albuquerque, NM 87185, USA;(2) Exploratory Simulation Technologies, Dept. 1433, Sandia National Laboratories, P.O. Box 5800, MS 0316, Albuquerque, NM 87185, USA;(3) Optimization & Uncertainty Estimation, Dept. 1411, Sandia National Laboratories, P.O. Box 5800, MS 0316, Albuquerque, NM 87185, USA |
| |
| Abstract: | Since verification of computational simulation codes requires significant resources, the ability to measure progress in verification
is critical to assess whether resources are being applied appropriately. Additionally, potential users need to know what fraction
of the software has been order-verified. In this study, the procedures and progress measures presented by Knupp et al. (Measuring
progress order-verification within software development projects. Engineering with Computers, appears in this issue, 2007)
are demonstrated on the Premo software, which simulates compressible aerodynamics through and around general geometries. Premo
was selected for this demonstration because extensive order-verification tests have been previously performed, yet no systematic
effort has been made to assess test-suite completeness or progress. This effort was performed to identify the practical issues
encountered when attempting to apply the ideas by Knupp (Measuring progress order-verification within software development
projects. Engineering with Computers, appears in this issue, 2007) to existing production-quality software. In this work,
a non-specific order-verification exercise is considered, as opposed to an application-specific order-verification exercise,
since past and present Premo order-verification efforts have been motivated by the need to verify all of the code, rather
than portions relevant for specific applications. Constructing an order-verification test suite that verifies the order of
accuracy of all the code capabilities is a major step in measuring progress. A practical approach to test-suite construction
is described that helps create a complete test suite through a combination of coarse-grain code coverage, input-keyword inspection,
discretization-algorithm documentation, and expert knowledge. Some of the difficulties and issues encountered during the construction
of the test suite are described, along with recommendations on how to deal with them. Once the test suite is constructed,
the progress measures proposed by Knupp (Measuring progress order-verification within software development projects. Engineering
with Computers, appears in this issue, 2007) can be evaluated and used to reconstruct the history of progress in Premo verification
over the past several years. Gaps in Premo verification are identified and indicate future directions for making progress.
Additionally, a measure of Premo verification fitness is computed for selected applications commonly simulated in the aerospace
industry. It is hoped that this demonstration will provide a practical example for other software-development groups in measuring
their own verification progress.
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed-Martin Company, for the United States Department
of Energy’s National Nuclear Security Administration under Contract DE-AC04-94AL85000.
|
| |
| Keywords: | |
| 本文献已被 SpringerLink 等数据库收录！ |
|
