Ensemble data assimilation in the presence of cloud

In numerical weather prediction (NWP) data assimilation (DA) methods are used to combine available observations with numerical model estimates. This is done by minimising measures of error on both observations and model estimates with more weight given to data that can be more trusted. For any DA method an estimate of the initial forecast error covariance matrix is required. For convective scale data assimilation, however, the properties of the error covariances are not well understood.An effective way to investigate covariance properties in the presence of convection is to use an ensemble-based method for which an estimate of the error covariance is readily available at each time step. In this work, we investigate the performance of the ensemble square root filter (EnSRF) in the presence of cloud growth applied to an idealised 1D convective column model of the atmosphere. We show that the EnSRF performs well in capturing cloud growth, but the ensemble does not cope well with discontinuities introduced into the system by parameterised rain. The state estimates lose accuracy, and more importantly the ensemble is unable to capture the spread (variance) of the estimates correctly. We also find, counter-intuitively, that by reducing the spatial frequency of observations and/or the accuracy of the observations, the ensemble is able to capture the states and their variability successfully across all regimes.     

An evaluation of the nonlinear/non-Gaussian filters for the sequential data assimilation

This paper aims to investigate several new nonlinear/non-Gaussian filters in the context of the sequential data assimilation. The unscented Kalman filter (UKF), the ensemble Kalman filter (EnKF), the sampling importance resampling particle filter (SIR-PF) and the unscented particle filter (UPF) are described in the state-space model framework in the Bayesian filtering background. We first evaluated those methods with a simple highly nonlinear Lorenz model and a scalar nonlinear non-Gaussian model to investigate the filter stability and the error sensitivity, and then their abilities in the one-dimensional estimation of the soil moisture content with the synthetic microwave brightness temperature assimilation experiment in the land surface model VIC-3L. All the results are compared with the EnKF. The advantages and disadvantages of each filter are discussed.The results in the Lorenz model showed that the particle filters are suitable for the large measurement interval assimilation and that the Kalman filters were suitable for the frequent measurement assimilation as well as small measurement uncertainties. The EnKF also showed its feasibility for the non-Gaussian noise. The performance of the SIR-PF was actually not as good as that of the UKF or the EnKF regarding a very small observation noise level compared with the uncertainties in the system. In the one-dimensional brightness temperature assimilation experiment, the UKF, the EnKF and the SIR-PF all proved to be flexible and reliable nonlinear filter algorithms for the low dimensional sequential land data assimilation application. For the high dimensional land surface system that takes the horizontal error correlations into account, the UKF is restricted by its computational demand in the covariance propagation; we must use the EnKF, the SIR-PF and other covariance reduction algorithms. The large computational cost prevents the UPF from being applied in practice.     

Estimation of regional terrestrial water cycle using multi-sensor remote sensing observations and data assimilation

An integrated data assimilation system is implemented over the Red-Arkansas river basin to estimate the regional scale terrestrial water cycle driven by multiple satellite remote sensing data. These satellite products include the Tropical Rainfall Measurement Mission (TRMM), TRMM Microwave Imager (TMI), and Moderate Resolution Imaging Spectroradiometer (MODIS). Also, a number of previously developed assimilation techniques, including the ensemble Kalman filter (EnKF), the particle filter (PF), the water balance constrainer, and the copula error model, and as well as physically based models, including the Variable Infiltration Capacity (VIC), the Land Surface Microwave Emission Model (LSMEM), and the Surface Energy Balance System (SEBS), are tested in the water budget estimation experiments. This remote sensing based water budget estimation study is evaluated using ground observations driven model simulations. It is found that the land surface model driven by the bias-corrected TRMM rainfall produces reasonable water cycle states and fluxes, and the estimates are moderately improved by assimilating TMI 10.67 GHz microwave brightness temperature measurements that provides information on the surface soil moisture state, while it remains challenging to improve the results by assimilating evapotranspiration estimated from satellite-based measurements.     

Development of a coupled land-atmosphere satellite data assimilation system for improved local atmospheric simulations

This study developed a coupled land-atmosphere satellite data assimilation system as a new physical downscaling approach, by coupling a mesoscale atmospheric model with a land data assimilation system (LDAS). The LDAS consists of a land surface scheme as the model operator, a radiative transfer model as the observation operator, and the simulated annealing method for minimizing the difference between the observed and simulated microwave brightness temperature. The atmospheric model produces forcing data for the LDAS, and the LDAS produces better initial surface conditions for the modelling system. This coupled system can take into account land surface heterogeneities through assimilating satellite data for a better precipitation prediction. To assess the effectiveness of the new system, 3-dimensional numerical experiments were carried out in a mesoscale area of the Tibetan Plateau during the wet monsoon season. The results show significant improvement compared with a no assimilation regional atmospheric model simply nested from the global model. The surface soil moisture content and its distribution from the assimilation system were more consistent to in situ observations. These better surface conditions affect the land-atmosphere interactions through convection systems and lead to better atmospheric predictability as confirmed by satellite-based cloud observations and in situ sounding observations. Through the use of satellite brightness temperature, the developed coupled land-atmosphere assimilation system has shown potential ability to provide better initial surface conditions and its inputs to the atmosphere and to improve physical downscaling through regional models.     

State-space dynamics distance for clustering sequential data

This paper proposes a novel similarity measure for clustering sequential data. We first construct a common state space by training a single probabilistic model with all the sequences in order to get a unified representation for the dataset. Then, distances are obtained attending to the transition matrices induced by each sequence in that state space. This approach solves some of the usual overfitting and scalability issues of the existing semi-parametric techniques that rely on training a model for each sequence. Empirical studies on both synthetic and real-world datasets illustrate the advantages of the proposed similarity measure for clustering sequences.     

Improved Tabu Search and Simulated Annealing methods for nonlinear data assimilation

Nonlinear data assimilation can be a very challenging task. Four local search methods are proposed for nonlinear data assimilation in this paper. The methods work as follows: At each iteration, the observation operator is linearized around the current solution, and a gradient approximation of the three dimensional variational (3D-Var) cost function is obtained. Then, samples along potential steepest descent directions of the 3D-Var cost function are generated, and the acceptance/rejection criteria for such samples are similar to those proposed by the Tabu Search and the Simulated Annealing framework. In addition, such samples can be drawn within certain sub-spaces so as to reduce the computational effort of computing search directions. Once a posterior mode is estimated, matrix-free ensemble Kalman filter approaches can be implemented to estimate posterior members. Furthermore, the convergence of the proposed methods is theoretically proven based on the necessary assumptions and conditions. Numerical experiments have been performed by using the Lorenz-96 model. The numerical results show that the cost function values on average can be reduced by several orders of magnitudes by using the proposed methods. Even more, the proposed methods can converge faster to posterior modes when sub-space approximations are employed to reduce the computational efforts among iterations.     

A parallel data assimilation model for oceanographic observations

In this paper we describe the development of a program that aims at achieving the optimal integration of observed data in an oceanographic model describing the water transport phenomena in the Agulhas area at the tip of South Africa. Two parallel implementations, MPI and OpenMP, are described and experiments with respect to speed and scalability on a Compaq AlphaServer SC and an SGI Origin3000 are reported. Copyright © 2003 John Wiley & Sons, Ltd.     

A data-driven kernel method assimilation technique for geophysical modelling

Incorporating the quantity and variety of observations in atmospheric and oceanographic assimilation and prediction models has become an increasingly complex task. Data assimilation allows for uneven spatial and temporal data distribution and redundancy to be addressed so that the models can ingest massive data sets. Traditional data assimilation methods introduce Kalman filters and variational approaches. This study introduces a family of algorithms, motivated by advances in machine learning. These algorithms provide an alternative approach to incorporating new observations into the analysis forecast cycle. The application of kernel methods to processing the states of a quasi-geostrophic numerical model is intended to demonstrate the feasibility of the method as a proof-of-concept. The speed, efficiency, accuracy and scalability in recovering unperturbed state trajectories establishes the viability of machine learning for data assimilation.     

Enhanced biomass prediction by assimilating satellite data into a crop growth model

Complex crop growth models (CGM) require a large number of input parameters, which can cause large errors if they are uncertain. Furthermore, they often lack spatial information. The coupling of a CGM with a radiative transfer model offers the possibility to assimilate remote sensing data while taking into account uncertainties in input parameters. A particle filter was used to assimilate satellite data into a CGM coupled with a leaf-canopy radiative transfer model to update biomass simulations of maize. The synthetic experiment set up to test the reliability of the procedure, highlighted the importance of the acquisition time. The real case study with RapidEye observations confirmed these findings. Data assimilation increased the accuracy of biomass predictions in the majority of the six maize fields where biomass validation data was available, with improvements of up to 15%. The smallest and largest errors in biomass prediction after assimilation were 82 kg/ha and 2116 kg/ha, respectively. Furthermore, data assimilation enabled the production of biomass maps showing detailed spatial variability.     

航空遥感数据同化算法集成计算与可视化

遥感数据同化技术在动力模型框架内,使用数据同化算法对动力模型输出的定量(物理、化学量)数据与观测数据进行一致性处理与结果误差分析。将多源遥感数据同化到动力模型预测与参数估计中,可帮助改善地表、大气和海洋变化的分析和预测精度。以国家发改委"十二五"建设的国家航空遥感系统项目为依托,针对航空遥感系统10种传感器设计开发数据同化系统。因无法找到适用于该系统的3DVAR和EnKF算法程序,必须自主开发核心算法程序。介绍了研究开发的航空遥感数据同化算法集成计算与可视化系统及其核心算法的关键技术流程。实验结果证实,该系统可以有效地对航空遥感数据进行同化。     

面向HYCOM的高分辨率海洋数据同化并行算法优化

HYCOM(hybrid coordinate ocean model)海洋数值模式要求较高的吞吐量和相对较小的计算量,这给并行算法设计带来了巨大的挑战.针对具有高吞吐量的海洋数据同化问题,设计了一种基于区域分解的并行优化算法.首先,提出了一种灵活的文件访问方法,可以高效地从磁盘读取大量的数据,避免数据访问冲突,大幅降低磁盘寻址操作的频率.此外,设计了一种避免通信的策略,以一些额外的计算量为代价大幅减少进程间的通信量.最后,提出了一种基于管道流的通信策略,以实现无冲突的消息传递.实验结果表明,该算法与基线算法相比,总体性能提高了5倍,其中文件读取速度提升6倍,进程间的通信性能提升了2.7倍.     

Quadratic programming based data assimilation with passive drifting sensors for shallow water flows

We present a method for assimilating Lagrangian sensor measurement data into a shallow water equation model. The underlying estimation problem (in which the dynamics of the system are represented by a system of partial differential equations) relies on the formulation of a minimisation of an error functional, which represents the mismatch between the estimate and the measurements. The corresponding so-called variational data assimilation problem is formulated as a quadratic programming problem with linear constraints. For the hydrodynamics application of interest, data is obtained from drifting sensors that gather position and velocity. The data assimilation method refines the estimate of the initial conditions of the hydrodynamic system. The method is implemented using a new sensor network hardware platform for gathering flow information from a river, which is presented in this article for the first time. Validation of the results is performed by comparing them to an estimate derived from an independent set of static sensors, some of which were deployed as part of our field experiments.     

Automated test‐data generation for exception conditions

This paper presents a technique for automatically generating test‐data to test exceptions. The approach is based on the application of a dynamic global optimization based search for the required test‐data. The authors' work has focused on test‐data generation for safety‐critical systems. Such systems must be free from anomalous and uncontrolled behaviour. Typically, it is easier to prove the absence of any exceptions than proving that the exception handling is safe. A process for integrating automated testing with exception freeness proofs is presented as a way forward for tackling the special needs of safety critical systems. The results of a number of simple case‐studies are presented and show the technique to be effective. The major result shows the application of the technique to a commercial aircraft engine controller system as part of a proof of exception freeness. This illustrates how automated testing can be effectively integrated into a formal safety‐critical process to reduce costs and add value. Copyright © 2000 John Wiley & Sons, Ltd.     

Integrating hydrological modelling,data assimilation and cloud computing for real-time management of water resources

Online data acquisition, data assimilation and integrated hydrological modelling have become more and more important in hydrological science. In this study, we explore cloud computing for integrating field data acquisition and stochastic, physically-based hydrological modelling in a data assimilation and optimisation framework as a service to water resources management. For this purpose, we developed an ensemble Kalman filter-based data assimilation system for the fully-coupled, physically-based hydrological model HydroGeoSphere, which is able to run in a cloud computing environment. A synthetic data assimilation experiment based on the widely used tilted V-catchment problem showed that the computational overhead for the application of the data assimilation platform in a cloud computing environment is minimal, which makes it well-suited for practical water management problems. Advantages of the cloud-based implementation comprise the independence from computational infrastructure and the straightforward integration of cloud-based observation databases with the modelling and data assimilation platform.     

Heuristics-based infeasible path detection for dynamic test data generation

Automated test data generation plays an important part in reducing the cost and increasing the reliability of software testing. However, a challenging problem in path-oriented test data generation is the existence of infeasible program paths, where considerable effort may be wasted in trying to generate input data to traverse the paths. In this paper, we propose a heuristics-based approach to infeasible path detection for dynamic test data generation. Our approach is based on the observation that many infeasible program paths exhibit some common properties. Through realizing these properties in execution traces collected during the test data generation process, infeasible paths can be detected early with high accuracy. Our experiments show that the proposed approach efficiently detects most of the infeasible paths with an average precision of 96.02% and a recall of 100% of all the cases.     

面向安全苛求软件的测试用例自动生成*

测试用例的自动生成是验证安全苛求软件最关键的技术问题,然而目前的研究并没有充分考虑安全苛求软件的安全性需求,为此提出一种应用安全覆盖准则的安全苛求软件的测试用例自动生成策略,将该策略应用于铁路车站计算机连锁软件,并与全节点覆盖准则进行了比较。结果表明该策略对关键变迁有更高的安全性保证。     

Ensemble component selection for improving ICA based microarray data prediction models

Independent component analysis (ICA) has been widely used to tackle the microarray dataset classification problem, but there still exists an unsolved problem that the independent component (IC) sets may not be reproducible after different ICA transformations. Inspired by the idea of ensemble feature selection, we design an ICA based ensemble learning system to fully utilize the difference among different IC sets. In this system, some IC sets are generated by different ICA transformations firstly. A multi-objective genetic algorithm (MOGA) is designed to select different biologically significant IC subsets from these IC sets, which are then applied to build base classifiers. Three schemes are used to fuse these base classifiers. The first fusion scheme is to combine all individuals in the final generation of the MOGA. In addition, in the evolution, we design a global-recording technique to record the best IC subsets of each IC set in a global-recording list. Then the IC subsets in the list are deployed to build base classifier so as to implement the second fusion scheme. Furthermore, by pruning about half of less accurate base classifiers obtained by the second scheme, a compact and more accurate ensemble system is built, which is regarded as the third fusion scheme. Three microarray datasets are used to test the ensemble systems, and the corresponding results demonstrate that these ensemble schemes can further improve the performance of the ICA based classification model, and the third fusion scheme leads to the most accurate ensemble system with the smallest ensemble size.     

面向Deep Web数据自动抽取的模板生成方法*

Deep Web结果页面大多由网站根据请求从后台数据库读取数据并动态填充到通用模板而生成的。研究如何从一系列同模板生成的页面中生成该模板,并利用模板自动抽取数据。给出了模板生成问题的形式化描述,提出了一种新颖的模板生成方法,利用生成的模板从实例网页中抽取数据。与现有方法相比,该方法适用于列表页面和详细页面两种类型网页。通过在多个领域站点上实验,说明新方法在不降低准确率的情况下能大大提高召回率。