Ensemble data assimilation in the presence of cloud

In numerical weather prediction (NWP) data assimilation (DA) methods are used to combine available observations with numerical model estimates. This is done by minimising measures of error on both observations and model estimates with more weight given to data that can be more trusted. For any DA method an estimate of the initial forecast error covariance matrix is required. For convective scale data assimilation, however, the properties of the error covariances are not well understood.An effective way to investigate covariance properties in the presence of convection is to use an ensemble-based method for which an estimate of the error covariance is readily available at each time step. In this work, we investigate the performance of the ensemble square root filter (EnSRF) in the presence of cloud growth applied to an idealised 1D convective column model of the atmosphere. We show that the EnSRF performs well in capturing cloud growth, but the ensemble does not cope well with discontinuities introduced into the system by parameterised rain. The state estimates lose accuracy, and more importantly the ensemble is unable to capture the spread (variance) of the estimates correctly. We also find, counter-intuitively, that by reducing the spatial frequency of observations and/or the accuracy of the observations, the ensemble is able to capture the states and their variability successfully across all regimes.     

An evaluation of the nonlinear/non-Gaussian filters for the sequential data assimilation

This paper aims to investigate several new nonlinear/non-Gaussian filters in the context of the sequential data assimilation. The unscented Kalman filter (UKF), the ensemble Kalman filter (EnKF), the sampling importance resampling particle filter (SIR-PF) and the unscented particle filter (UPF) are described in the state-space model framework in the Bayesian filtering background. We first evaluated those methods with a simple highly nonlinear Lorenz model and a scalar nonlinear non-Gaussian model to investigate the filter stability and the error sensitivity, and then their abilities in the one-dimensional estimation of the soil moisture content with the synthetic microwave brightness temperature assimilation experiment in the land surface model VIC-3L. All the results are compared with the EnKF. The advantages and disadvantages of each filter are discussed.The results in the Lorenz model showed that the particle filters are suitable for the large measurement interval assimilation and that the Kalman filters were suitable for the frequent measurement assimilation as well as small measurement uncertainties. The EnKF also showed its feasibility for the non-Gaussian noise. The performance of the SIR-PF was actually not as good as that of the UKF or the EnKF regarding a very small observation noise level compared with the uncertainties in the system. In the one-dimensional brightness temperature assimilation experiment, the UKF, the EnKF and the SIR-PF all proved to be flexible and reliable nonlinear filter algorithms for the low dimensional sequential land data assimilation application. For the high dimensional land surface system that takes the horizontal error correlations into account, the UKF is restricted by its computational demand in the covariance propagation; we must use the EnKF, the SIR-PF and other covariance reduction algorithms. The large computational cost prevents the UPF from being applied in practice.     

Estimation of regional terrestrial water cycle using multi-sensor remote sensing observations and data assimilation

An integrated data assimilation system is implemented over the Red-Arkansas river basin to estimate the regional scale terrestrial water cycle driven by multiple satellite remote sensing data. These satellite products include the Tropical Rainfall Measurement Mission (TRMM), TRMM Microwave Imager (TMI), and Moderate Resolution Imaging Spectroradiometer (MODIS). Also, a number of previously developed assimilation techniques, including the ensemble Kalman filter (EnKF), the particle filter (PF), the water balance constrainer, and the copula error model, and as well as physically based models, including the Variable Infiltration Capacity (VIC), the Land Surface Microwave Emission Model (LSMEM), and the Surface Energy Balance System (SEBS), are tested in the water budget estimation experiments. This remote sensing based water budget estimation study is evaluated using ground observations driven model simulations. It is found that the land surface model driven by the bias-corrected TRMM rainfall produces reasonable water cycle states and fluxes, and the estimates are moderately improved by assimilating TMI 10.67 GHz microwave brightness temperature measurements that provides information on the surface soil moisture state, while it remains challenging to improve the results by assimilating evapotranspiration estimated from satellite-based measurements.     

Development of a coupled land-atmosphere satellite data assimilation system for improved local atmospheric simulations

This study developed a coupled land-atmosphere satellite data assimilation system as a new physical downscaling approach, by coupling a mesoscale atmospheric model with a land data assimilation system (LDAS). The LDAS consists of a land surface scheme as the model operator, a radiative transfer model as the observation operator, and the simulated annealing method for minimizing the difference between the observed and simulated microwave brightness temperature. The atmospheric model produces forcing data for the LDAS, and the LDAS produces better initial surface conditions for the modelling system. This coupled system can take into account land surface heterogeneities through assimilating satellite data for a better precipitation prediction. To assess the effectiveness of the new system, 3-dimensional numerical experiments were carried out in a mesoscale area of the Tibetan Plateau during the wet monsoon season. The results show significant improvement compared with a no assimilation regional atmospheric model simply nested from the global model. The surface soil moisture content and its distribution from the assimilation system were more consistent to in situ observations. These better surface conditions affect the land-atmosphere interactions through convection systems and lead to better atmospheric predictability as confirmed by satellite-based cloud observations and in situ sounding observations. Through the use of satellite brightness temperature, the developed coupled land-atmosphere assimilation system has shown potential ability to provide better initial surface conditions and its inputs to the atmosphere and to improve physical downscaling through regional models.     

State-space dynamics distance for clustering sequential data

This paper proposes a novel similarity measure for clustering sequential data. We first construct a common state space by training a single probabilistic model with all the sequences in order to get a unified representation for the dataset. Then, distances are obtained attending to the transition matrices induced by each sequence in that state space. This approach solves some of the usual overfitting and scalability issues of the existing semi-parametric techniques that rely on training a model for each sequence. Empirical studies on both synthetic and real-world datasets illustrate the advantages of the proposed similarity measure for clustering sequences.     

A data-driven kernel method assimilation technique for geophysical modelling

Incorporating the quantity and variety of observations in atmospheric and oceanographic assimilation and prediction models has become an increasingly complex task. Data assimilation allows for uneven spatial and temporal data distribution and redundancy to be addressed so that the models can ingest massive data sets. Traditional data assimilation methods introduce Kalman filters and variational approaches. This study introduces a family of algorithms, motivated by advances in machine learning. These algorithms provide an alternative approach to incorporating new observations into the analysis forecast cycle. The application of kernel methods to processing the states of a quasi-geostrophic numerical model is intended to demonstrate the feasibility of the method as a proof-of-concept. The speed, efficiency, accuracy and scalability in recovering unperturbed state trajectories establishes the viability of machine learning for data assimilation.     

航空遥感数据同化算法集成计算与可视化

遥感数据同化技术在动力模型框架内,使用数据同化算法对动力模型输出的定量(物理、化学量)数据与观测数据进行一致性处理与结果误差分析。将多源遥感数据同化到动力模型预测与参数估计中,可帮助改善地表、大气和海洋变化的分析和预测精度。以国家发改委"十二五"建设的国家航空遥感系统项目为依托,针对航空遥感系统10种传感器设计开发数据同化系统。因无法找到适用于该系统的3DVAR和EnKF算法程序,必须自主开发核心算法程序。介绍了研究开发的航空遥感数据同化算法集成计算与可视化系统及其核心算法的关键技术流程。实验结果证实,该系统可以有效地对航空遥感数据进行同化。     

Ensemble component selection for improving ICA based microarray data prediction models

Independent component analysis (ICA) has been widely used to tackle the microarray dataset classification problem, but there still exists an unsolved problem that the independent component (IC) sets may not be reproducible after different ICA transformations. Inspired by the idea of ensemble feature selection, we design an ICA based ensemble learning system to fully utilize the difference among different IC sets. In this system, some IC sets are generated by different ICA transformations firstly. A multi-objective genetic algorithm (MOGA) is designed to select different biologically significant IC subsets from these IC sets, which are then applied to build base classifiers. Three schemes are used to fuse these base classifiers. The first fusion scheme is to combine all individuals in the final generation of the MOGA. In addition, in the evolution, we design a global-recording technique to record the best IC subsets of each IC set in a global-recording list. Then the IC subsets in the list are deployed to build base classifier so as to implement the second fusion scheme. Furthermore, by pruning about half of less accurate base classifiers obtained by the second scheme, a compact and more accurate ensemble system is built, which is regarded as the third fusion scheme. Three microarray datasets are used to test the ensemble systems, and the corresponding results demonstrate that these ensemble schemes can further improve the performance of the ICA based classification model, and the third fusion scheme leads to the most accurate ensemble system with the smallest ensemble size.     

Heuristics-based infeasible path detection for dynamic test data generation

Automated test data generation plays an important part in reducing the cost and increasing the reliability of software testing. However, a challenging problem in path-oriented test data generation is the existence of infeasible program paths, where considerable effort may be wasted in trying to generate input data to traverse the paths. In this paper, we propose a heuristics-based approach to infeasible path detection for dynamic test data generation. Our approach is based on the observation that many infeasible program paths exhibit some common properties. Through realizing these properties in execution traces collected during the test data generation process, infeasible paths can be detected early with high accuracy. Our experiments show that the proposed approach efficiently detects most of the infeasible paths with an average precision of 96.02% and a recall of 100% of all the cases.     

面向安全苛求软件的测试用例自动生成

测试用例的自动生成是验证安全苛求软件最关键的技术问题,然而目前的研究并没有充分考虑安全苛求软件的安全性需求,为此提出一种应用安全覆盖准则的安全苛求软件的测试用例自动生成策略,将该策略应用于铁路车站计算机连锁软件,并与全节点覆盖准则进行了比较。结果表明该策略对关键变迁有更高的安全性保证。     

面向Deep Web数据自动抽取的模板生成方法*

Deep Web结果页面大多由网站根据请求从后台数据库读取数据并动态填充到通用模板而生成的。研究如何从一系列同模板生成的页面中生成该模板,并利用模板自动抽取数据。给出了模板生成问题的形式化描述,提出了一种新颖的模板生成方法,利用生成的模板从实例网页中抽取数据。与现有方法相比,该方法适用于列表页面和详细页面两种类型网页。通过在多个领域站点上实验,说明新方法在不降低准确率的情况下能大大提高召回率。     

Dynamic method for software test data generation

Test data generation in program testing is the process of identifying a set of test data which satisfies a given testing criterion. Existing pathwise test data generators proceed by selecting program paths that satisfy the selected criterion and then generating program inputs for these paths. One of the problems with this approach is that unfeasible paths are often selected; as a result, significant computational effort can be wasted in analysing those paths. In this paper, an approach to test data generation, referred to as a dynamic approach for test data generation, is presented. In this approach, the path selection stage is eliminated. Test data are derived based on the actual execution of the program under test and function minimization methods. The approach starts by executing a program for an arbitrary program input. During program execution for each executed branch, a search procedure decides whether the execution should continue through the current branch or an alternative branch should be taken. If an undesirable execution flow is observed at the current branch, then a real-valued function is associated with this branch, and function minimization search algorithms are used to locate values of input variables automatically, which will change the flow of execution at this branch.     

Using single layer networks for discrete, sequential data: An example from Natural Language Processing

Natural Language Processing (NLP) is concerned with processing ordinary, unrestricted text. This work takes a new approach to a traditional NLP task, using neural computing methods. A parser which has been successfully implemented is described. It is a hybrid system, in which neural processors operate within a rule based framework. The neural processing components belong to the class of Generalized Single Layer Networks (GSLN). In general, supervised, feed-forward networks need more than one layer to process data. However, in some cases data can be pre-processed with a non-linear transformation, and then presented in a linearly separable form for subsequent processing by a single layer net. Such networks offer advantages of functional transparency and operational speed. For our parser, the initial stage of processing maps linguistic data onto a higher order representation, which can then be analysed by a single layer network. This transformation is supported by information theoretic analysis. Three different algorithms for the neural component were investigated. Single layer nets can be trained by finding weight adjustments based on (a) factors proportional to the input, as in the Perceptron, (b) factors proportional to the existing weights, and (c) an error minimization method. In our experiments generalization ability varies little; method (b) is used for a prototype parser. This is available via telnet.     

一种用于测试数据生成的动态程序切片算法

介绍了程序切片技术的基本概念，提出了一种基于前向分析的动态程序切片算法，探讨了程序切片在软件测试数据生成中的应用，结果表明可以有效地提高基于路径的测试数据生成效率。     

多路径测试数据自动生成方法及应用

以程序结构测试自动生成为研究背景,提出了一种重叠路径结构用以描述程序路径,并以此为基础设计了一种多路径测试数据生成适应值算法,实现了一次搜索完成多条路径的测试数据生成。算法通过目标路径间共享遗传算法产生的中间个体减少单一路径搜索始于随机产生的无序个体的初期迭代,从而加快搜索收敛的速度。应用于常用的基准程序和取自实际项目的程序,该算法与典型的分支谓词距离算法相比平均消耗时间缩短了70.6%。     

EFSM模型的字符串类型测试数据自动生成

基于软件描述模型的测试数据自动生成研究中,字符串类型测试数据生成是一个研究热点和难点。EFSM模型是一种重要的软件描述模型。分析了EFSM模型的特点,针对面向EFSM模型目标路径的字符串测试数据生成,建立了字符串输入变量模型和操作模型,结合静态测试的特点,给出了通过字符串变量模型在目标路径上的符号执行结果生成字符串类型测试数据的方法。实验结果表明,该方法能够达到预期效果,提高测试生成效率。     

Web环境下基于SOAOffice中间件的合同生成与数据抽取方案

合同生成是合同管理系统中一个重要组成部分.通过分析以往Web环境下合同管理系统存在的缺陷,给出一种新的合同生成方法.本方法解决了以往合同管理系统存在的多个问题,将合同生成与合同管理有机地结合起来,实现了合同签订模板化、合同审批网络化、数据抽取自动化、合同管理高效化、准确化、标准化.这种方法是在设计合同生成及后期管理数据结构的基础上,制作各类word合同模板并通过SOAOffice中间件实现Web环境下数据抽取与组合,完成合同网上审批及管理功能.     

Regularization techniques for ill-posed inverse problems in data assimilation

Optimal state estimation from given observations of a dynamical system by data assimilation is generally an ill-posed inverse problem. In order to solve the problem, a standard Tikhonov, or L₂, regularization is used, based on certain statistical assumptions on the errors in the data. The regularization term constrains the estimate of the state to remain close to a prior estimate. In the presence of model error, this approach does not capture the initial state of the system accurately, as the initial state estimate is derived by minimizing the average error between the model predictions and the observations over a time window. Here we examine an alternative L₁ regularization technique that has proved valuable in image processing. We show that for examples of flow with sharp fronts and shocks, the L₁ regularization technique performs more accurately than standard L₂ regularization.