Moving SWAT model calibration and uncertainty analysis to an enterprise Hadoop-based cloud

With enhanced availability of high spatial resolution data, hydrologic models such as the Soil and Water Assessment Tool (SWAT) are increasingly used to investigate effects of management activities and climate change on water availability and quality. The advantages come at a price of greater computational demand and run time. This becomes challenging to model calibration and uncertainty analysis as these routines involve a large number of model runs. For efficient modelling, a cloud-based Calibration and Uncertainty analysis Tool for SWAT (CUT-SWAT) was implemented using Hadoop, an open source cloud platform, and the Generalized Likelihood Uncertainty Estimation method. Test results on an enterprise cloud showed that CUT-SWAT can significantly speedup the calibration and uncertainty analysis processes with a speedup of 21.7–26.6 depending on model complexity and provides a flexible and fault-tolerant model execution environment (it can gracefully and automatically handle partial failure), thus would be an ideal method to solve computational demand problems in hydrological modelling.     

SWAT-DayCent coupler: An integration tool for simultaneous hydro-biogeochemical modeling using SWAT and DayCent

Process-based numerical models in environmental science can help understand and quantify terrestrial material cycles in nature. However, the existing models usually focus on the cycles of one or more elements (e.g., water, carbon, or nitrogen). For example, hydrological models such as Soil and Water Assessment Tool (SWAT) focus on the water cycle and nutrient loadings at watershed scale, whereas biogeochemical models such as DayCent (i.e., daily CENTURY) emphasize carbon/nitrogen storage and fluxes of ecosystems at landscape scale. Therefore, using either one of the two categories of models is not enough for understanding/solving the current complex environmental issues that involve multiple aspects. Although use of both models (SWAT and DayCent) could be an expedient way toward treating the problem, creating separate model projects for a single area could be challenging and time consuming, and integration/analyses of model results have some limitations due to the non-uniformity of input spatial data between models. To overcome this issue, we developed an integrated model implementation coupler that aims to drive SWAT and DayCent—the two representative models in hydrology and biogeochemistry, respectively—just using a user's SWAT project without the need of any extra efforts such as developing a framework or preparing input data for DayCent modeling. This software is easy to use and would be promising for conducting comprehensive environmental impact assessment involving hydrological and biogeochemical cycles at watershed scale.     

An enhanced SWAT wetland module to quantify hydraulic interactions between riparian depressional wetlands,rivers and aquifers

This study develops a modified version of the Soil and Water Assessment Tool (SWAT) designed to better represent riparian depressional wetlands (SWATrw). It replaces existing unidirectional hydrological interactions between a wetland and a river/aquifer with a more robust bidirectional approach based on hydraulic principles. SWATrw incorporates a more flexible wetland morphometric formula and a connecting channel concept to model wetland-river interactions. SWAT and SWATrw were tested for the Barak-Kushiyara River Basin (Bangladesh and India). Although the two models showed small differences in simulated stream flow, SWATrw outperformed SWAT in reproducing river stages and the pre-monsoon river-spills into riparian wetlands. SWATrw showed that the observed presence of dry season water in the wetland was due to reduced seepage to the local groundwater table whilst continuous seepage simulated by SWAT resulted in the wetland drying out completely. The new model therefore more closely simulates the hydrological interactions between wetlands, rivers and groundwater.     

A model integration framework for linking SWAT and MODFLOW

Assessment of long-term anthropogenic impacts on agro-ecosystems requires comprehensive modelling capabilities to simulate water interactions between the surface and groundwater domains. To address this need, a modelling framework, called “SWATmf”, was developed to link and integrate the Soil Water Assessment Tool (SWAT), a widely used surface watershed model with the MODFLOW, a groundwater model. The SWATmf is designed to serve as a project manager, builder, and model performance evaluator, and to facilitate dynamic interactions between surface and groundwater domains at the watershed scale, thus providing a platform for simulating surface and groundwater interactions. Using datasets from the Fort Cobb Reservoir experimental watershed (located in Oklahoma, USA), the SWATmf to facilitate linkage and dynamic simulation of SWAT and MODFLOW models. Simulated streamflow and groundwater levels generally agreed with observations trends showing that the SWATmf can be used for simulating surface and groundwater interactions.     

SWATShare – A web platform for collaborative research and education through online sharing,simulation and visualization of SWAT models

Hydrologic models for a particular watershed or a region are created for addressing a specific research or management problem, and most of the models do not get reused after the project is completed. Similarly, multiple models may exist for a particular geographic location from different researchers or organizations. To avoid the duplication of efforts, and enable model reuse and enhancement through collaborative efforts, a prototype cyberinfrastructure, called SWATShare, is developed for sharing, execution and visualization of Soil and Water Assessment Tool (SWAT). The objective of this paper is to present the software architecture, functional capabilities and implementation of SWATShare as a collaborative environment for hydrology research and education using the models published and shared in the system. Besides the capability of publishing, sharing, discovery and downloading of SWAT models, some of the functions in SWATShare such as model calibration are supported by providing access to high performance computing resources including the XSEDE and cloud. Additionally, SWATShare can create dynamic spatial and temporal plots of model outputs at different scales. SWATShare can also be used as an educational tool within a classroom setting for comparing the hydrologic processes under different geographic and climatic settings. The utility of SWATShare for collaborative research and education is demonstrated by using three case studies. Even though this paper focuses on the SWAT model, the system’s architecture can be replicated for other models as well.     

Setting up a hydrological model of Alberta: Data discrimination analyses prior to calibration

Failure to setup a large-scale hydrological model correctly may not allow proper calibration and uncertainty analyses, leading to inaccurate model prediction. To build a model with accurate accounting of hydrological processes, a data discrimination procedure was applied in this study. The framework uses a hydrological model of Alberta built with the Soil and Water Assessment Tool (SWAT) program. The model was used to quantify the causes and extents of biases in predictions due to different types of input data. Data types represented different sources of errors, including input data (e.g., climate), conceptual model (e.g., potholes, glaciers), and control structure (e.g., reservoirs, dams). The results showed that accounting for these measures leads to a better physical accounting of hydrological processes, significantly improving the overall model performance. The procedure used in this study helps to avoid unnecessary and arbitrary adjustment of parameters to compensate for the errors in the model structure.     

Introducing a new open source GIS user interface for the SWAT model

The Soil and Water Assessment Tool (SWAT) model is a robust watershed modeling tool. It typically uses the ArcSWAT interface to create its inputs. ArcSWAT is public domain software which works in the licensed ArcGIS environment. The aim of this paper was to develop an open source user interface for the SWAT model. The interface, QSWAT, is written in the Python programming language and uses various functionalities of the open source geographic information system, QGIS. The current interface performs similar functions to ArcSWAT, but with additional enhanced features such as merging small subbasins and static and dynamic visualization of outputs. The interface is demonstrated through a case study in the Gumera watershed in the Lake Tana basin of Ethiopia, where it showed a successful performance. QSWAT will be a valuable tool for the SWAT scientific community, with improved availability and functionality compared with other options for creating SWAT models.     

Design and implementation of a general software library for using NSGA-II with SWAT for multi-objective model calibration

Calibrating watershed-scale hydrologic models remains a critical but challenging step in the modeling process. The Soil and Water Assessment Tool (SWAT) is one example of a widely used watershed-scale hydrologic model that requires calibration. The calibration algorithms currently available to SWAT modelers through freely available and open source software, however, are limited and do not include many multi-objective genetic algorithms (MOGAs). The Non-Dominated Sorting Genetic Algorithm II (NSGA-II) has been shown to be an effective and efficient MOGA calibration algorithm for a wide variety of applications including for SWAT model calibration. Therefore, the objective of this study was to create an open source software library for multi-objective calibration of SWAT models using NSGA-II. The design and implementation of the library are presented, followed by a demonstration of the library through a test case for the Upper Neuse Watershed in North Carolina, USA using six objective functions in the model calibration.     

An open source QGIS-based workflow for model application and experimentation with aquatic ecosystems

We present the new Water Ecosystems Tool (WET), a workflow implemented (as a plugin) in QGIS, for application and evaluation of aquatic ecosystem models. WET provides a Graphical User Interface (GUI) for the coupled one-dimensional hydrodynamic-ecosystem model GOTM-FABM-PCLake. WET is unique as it enables a standardized and easy-to-use workflow for an otherwise complex model application and is readily applicable to any individual lake and reservoir in the world. WET integrates a platform for model experimentation through scenario simulations – currently encompassing changes in climate and nutrient loads. WET also includes a link to the SWAT (Soil & Water Assessment Tool) watershed model, which can be used to simulate how land use changes affect aquatic ecosystems. The tool is open source and may therefore be readily expanded and adapted for additional model experimentations.     

基于克里金插值的SWAT参数率定大规模并行方法

SWAT（Soil and Water Assessment Tool）水文模型是近几年国内外使用最广泛的分布式水文模型,因其良好的精度、广泛的适应度得到了众多水文工作者的欢迎。但由于其参数率定部分过高的计算量,给水文工作者建模调参过程中造成了大量的不便。为了提高SWAT参数率定的运行效率,本文基于克里金插值理论,对于SWAT原有的优化算法进行改进,提出了一种基于随机过程的参数率定方法,通过一种“期望优化”的方式对拟合曲面进行寻优。并通过Intel MKL和一个对等式并行框架加以并行实现。大大提高了SWAT参数寻优的计算效率以及参数精度,并通过天山北坡流域的模拟验证了此并行模型的有效性,说明了克里金优化方法在水文模拟参数寻优的过程中能够达到良好的精度。     

Integrating statistical and hydrological models to identify implementation sites for agricultural conservation practices

Watershed models are scarcely used by watershed managers due to their complexity. This study facilitates information transfer by introducing simpler techniques related to easily obtained watershed characteristics, including distance to the watershed outlet and stream order. The Soil and Water Assessment Tool (SWAT) was calibrated for the Saginaw River Watershed, Michigan. Five agricultural best management practices (BMPs) were implemented in SWAT one at a time in each subbasin. Five statistical models were used to determine the pollution reduction at the watershed outlet using distance and BMP type, with results suggesting that a mixed effects model (model 5) was optimal. This model included subbasin as a random effect, while distance to watershed outlet and BMP type were fixed effects. Native grass and strip cropping were the most effective BMPs for reducing sediment and nutrient transport. Subbasins containing stream orders 1–3 were ideal for BMP implementation throughout the watershed.     

Identifying model consistency through stepwise calibration to capture streamflow variability

The consistency of calibrated hydrological models (whether the model is internally consistent) is often ignored as model calibration generally only evaluates performance. A correlation matrix is developed in this paper to assess model consistency by comparing different flow dynamics combined with a stepwise calibration approach targeting different flow signals (low, medium and high). The Soil Water Assessment tool (SWAT) was used as an example model and simulations were conducted using Sequential Uncertainty Fitting-2 (SUFI-2) algorithm. Critical evaluation of the method demonstrates that satisfactory model performance does not necessarily lead to a consistent model and this yields poor performance in validation. However, results may vary depending on climatic conditions and temporal scales. By calibrating on disaggregated flow signals and evaluating based on consistency, the proposed method improves model realism which will improve understanding of catchment functioning.     

A comprehensive approach to evaluating watershed models for predicting river flow regimes critical to downstream ecosystem services

Selection of strategies that help reduce riverine inputs requires numerical models that accurately quantify hydrologic processes. While numerous models exist, information on how to evaluate and select the most robust models is limited. Toward this end, we developed a comprehensive approach that helps evaluate watershed models in their ability to simulate flow regimes critical to downstream ecosystem services. We demonstrated the method using the Soil and Water Assessment Tool (SWAT), the Hydrological Simulation Program–FORTRAN (HSPF) model, and Distributed Large Basin Runoff Model (DLBRM) applied to the Maumee River Basin (USA). The approach helped in identifying that each model simulated flows within acceptable ranges. However, each was limited in its ability to simulate flows triggered by extreme weather events, owing to algorithms not being optimized for such events and mismatched physiographic watershed conditions. Ultimately, we found HSPF to best predict river flow, whereas SWAT offered the most flexibility for evaluating agricultural management practices.     

Model of pathogen transmission between livestock and white-tailed deer in fragmented agricultural and forest landscapes

The study summarizes the current knowledge on infection and recovery of white-tailed deer and cattle, and integrates this knowledge into the Soil and Water Assessment Tool (SWAT) via a new add-on module SIR (Susceptible - Infected - Recovered) for predicting pathogen transmission between livestock and deer. New processes modeled by the SWAT-SIR model include: (a) seasonal changes in deer population and habitat; (b) resource selection and seasonal changes in foliage consumption by deer; (c) ingestion of pathogens with water, foliage and via grooming soiled hide by deer and grazing cattle; (d) infection and recovery of deer and co-grazing cattle; (e) pathogen shedding by infected animals; (f) survival of pathogens in manure; (g) kinetic release of pathogens from applied manure and fecal material. The model output is linked with ARC-GIS to allow spatial and temporal analysis of pathogen distribution across the watershed for specific land use, weather and management scenarios.     

Comparison of variance-based and moment-independent global sensitivity analysis approaches by application to the SWAT model

Global Sensitivity Analysis (GSA) is an essential technique to support the calibration of environmental models by identifying the influential parameters (screening) and ranking them.In this paper, the widely-used variance-based method (Sobol') and the recently proposed moment-independent PAWN method for GSA are applied to the Soil and Water Assessment Tool (SWAT), and compared in terms of ranking and screening results of 26 SWAT parameters. In order to set a threshold for parameter screening, we propose the use of a “dummy parameter”, which has no influence on the model output. The sensitivity index of the dummy parameter is calculated from sampled data, without changing the model equations. We find that Sobol' and PAWN identify the same 12 influential parameters but rank them differently, and discuss how this result may be related to the limitations of the Sobol' method when the output distribution is asymmetric.     

Calibration of SWAT models using the cloud

This paper evaluates a recently created Soil and Water Assessment Tool (SWAT) calibration tool built using the Windows Azure Cloud environment and a parallel version of the Dynamically Dimensioned Search (DDS) calibration method modified to run in Azure. The calibration tool was tested for six model scenarios constructed for three watersheds of increasing size each for a 2 year and 10 year simulation duration. Results show significant speedup in calibration time and, for up to 64 cores, minimal losses in speedup for all watershed sizes and simulation durations. An empirical relationship is presented for estimating the time needed to calibration a SWAT model using the cloud calibration tool as a function of the number of Hydrologic Response Units (HRUs), time steps, and cores used for the calibration.     

A quantitative phosphorus loss assessment tool for agricultural fields

In the United States, government sponsored conservation programs are under increasing pressure to quantify the environmental benefits of practices they subsidize. To meet this objective, conservation planners need tools to accurately predict phosphorus (P) loss from agricultural lands. Existing P export coefficient based tools are easy to use, but do not adequately account for local conditions. Hydrologic and water quality models are more accurate, but are prohibitively complex for conservation planners to use. Pasture Phosphorus Management (PPM) Plus was developed as a user-friendly P and sediment loss prediction tool based on the Soil and Water Assessment Tool (SWAT), a popular comprehensive hydrologic and water quality model. PPM Plus is applicable under a wide variety of management options and conservation practices and simple enough for use by conservation planners. SWAT hydrologic components were calibrated to allow application anywhere in the State of Oklahoma. The SWAT model was modified to include soil P algorithm updates and improved representation of conservation practices. This tool was successfully validated using 286 field years of measured data from the southern United States. PPM Plus allows the development of more effective conservation plans by allowing planners to evaluate pollutant losses resulting from a particular management strategy prior to implementation.     

Development and evaluation of targeted marginal land mapping approach in SWAT model for simulating water quality impacts of selected second generation biofeedstock

Information about location of marginal lands in a watershed is of interest to those who view these areas as potential land for producing biofuel crops. However, representing marginal lands into a distributed model such as the Soil and Water Assessment Tool (SWAT) is a challenge due to a rigid framework used for watershed sub-division. In this study, we developed a Geographic Information System (GIS) based approach for implementing targeted land use i.e. marginal lands into the SWAT model and evaluated the applicability of the approach on a 8-digit Hydrologic Unit Code (HUC) watershed scale. Comparative results showed that conventional targeting approach overestimates the benefit of targeting marginal lands for Alamo switchgrass (Panicum virgatum, L) and giant miscanthus (Miscanthus × giganteus) production due to simulation of larger area under marginal land category. Compared to baseline condition, which corresponds to no biofuel crop production on marginal lands, the pollutant losses under new targeting approach with simulation of Alamo switchgrass and giant miscanthus on marginal lands were substantially lower. The new targeting approach advances the science behind landscape representation in the SWAT model – that has potential to be used in future targeting studies.     

Autocalibration experiments using machine learning and high performance computing

Using as example the Soil and Water Assessment Tool (SWAT) model and a Southern Ontario Canada watershed, we conduct a set of experiments on calibration using a manual approach, a parallelized version of the shuffled complex evolution (SCE), Generalized Likelihood Uncertainty Estimation (GLUE), Sequential Uncertainty Fitting (SUFI-2) and compare to a simple parallel search on a finite set of gridded input parameter values invoking the probably approximately correct (PAC) learning hypothesis. We derive an estimation of the error in fitting and a prior estimate of the probability of success, based on the PAC hypothesis. We conclude that from the equivalent effort expended on initial setup for the other named algorithms we can already find directly a good parameter set for calibration. We further note that, in this algorithm, simultaneous co-calibration of flow and chemistry (total nitrogen and total phosphorous) is more likely to produce acceptable results, as compared to flow first, even with a simple weighted multiobjective approach. This approach is especially suited to a parallel, distributed or cloud computational environment.