An automated scheme for the removal of cloud contamination from AVHRR radiances over western Europe

Abstract

Various cloud-detection schemes are applied to 1.1 km Advanced Very High Resolution Radiometer (AVHRR) day- and night-time data to determine an optimum automated scheme for deriving cloud-free radiances over both land and sea. A combination of the spatial coherence method at infrared wavelengths (11 μm) and dynamic visible threshold methods proved to be the most effective scheme for day-time use. Uniform thin cirrus (i.e. reflectance less than 15 per cent) was difficult to detect with all methods. Problems were also encountered over regions with a changing underlying surface type (e.g. coastal areas) where the automated scheme was not as effective as over uniform surfaces. At night a combination of the spatial coherence method and a scheme based on the differences in brightness temperatures between the three infrared channels at 37, II and 12 μm wavelength was successfully used. Results obtained by applying these algorithms to AVHRR data are presented and the different problems encountered with each algorithm are discussed.     

An unsupervised approach to online noisy-neighbor detection in cloud data centers

Resource sharing is an inherent characteristic of cloud data centers. Virtual Machines (VMs) and/or Containers that are co-located in the same physical server often compete for resources leading to interference. The noisy neighbor’s effect refers to an anomaly caused by a VM/container limiting resources accessed by another one. Our main contribution is an online, lightweight and application-agnostic solution for anomaly detection, that follows an unsupervised approach. It is based on comparing models for different lags: Dirichlet Process Gaussian Mixture Models to characterize the resource usage profile of the application, and distance measures to score the similarity among models. An alarm is raised when there is an abrupt change in short-term lag (i.e. high distance score for short-term models), while the long-term state remains constant. We test the algorithm for different cloud workloads: websites, periodic batch applications, Spark-based applications, and Memcached server. We are able to detect anomalies in the CPU and memory resource usage with up to 82–96% accuracy (recall) depending on the scenario. Compared to other baseline methods, our approach is able to detect anomalies successfully, while raising low number of false positives, even in the case of applications with unusual normal behavior (e.g. periodic). Experiments show that our proposed algorithm is a lightweight and effective solution to detect noisy neighbor effect without any historical info about the application, that could also be potentially applied to other kind of anomalies.     

An efficient privacy-preserving multi-keyword search over encrypted cloud data with ranking

Information search and retrieval from a remote database (e.g., cloud server) involves a multitude of privacy issues. Submitted search terms and their frequencies, returned responses and order of their relevance, and retrieved data items may contain sensitive information about the users. In this paper, we propose an efficient multi-keyword search scheme that ensures users’ privacy against both external adversaries including other authorized users and cloud server itself. The proposed scheme uses cryptographic techniques as well as query and response randomization. Provided that the security and randomization parameters are appropriately chosen, both search terms in queries and returned responses are protected against privacy violations. The scheme implements strict security and privacy requirements that essentially disallow linking queries featuring identical search terms. We also incorporate an effective ranking capability in the scheme that enables user to retrieve only the top matching results. Our comprehensive analytical study and extensive experiments using both real and synthetic datasets demonstrate that the proposed scheme is privacy-preserving, effective, and highly efficient.     

加密云数据下的关键词模糊搜索方案

对于加密云数据的搜索,传统的关键词模糊搜索方案虽然能搜索到相关文档,但是搜索的结果并不令人满意。在用户输入正确的情况下,无法完成近似搜索,当用户出现拼写错误时,返回的结果中包含大量无关关键词文档,严重浪费了带宽资源。针对目前在加密云数据下关键词模糊搜索的缺陷,提出了一种新型的关键词模糊搜索方案,通过对关键词计算相关度分数并对文档根据相关度分数进行排序,将top-k（即相关度最高的k个文档）个文档返回给搜索用户,减少了不必要的带宽浪费和用户寻找有效文档的时间消耗,提供了更加有效的搜索结果,并且通过引入虚假陷门集,增大了云服务器对文档关键词的分析难度,增加了系统的隐私性保护。     

An industrial engineering technique applied to pest management

Simulation of continuous systems has become a widely used tool for design and analysis in Industrial Engineering. This paper presents an application of the technique and the concept of systems dynamics to predict the status of bark beetle infestation in southern pine forests and to provide an estimate of subsequent timber loss. Such predictions are vital to the formulation of effective strategies for pest management.     

适于数据流组合分类的直推学习方法

在进行组合决策时,已有的组合分类方法需要对多个组合分类器均有效的公共已知标签训练样本。为了解决在没有已知标签样本的情况下数据流组合分类决策问题,提出一种基于约束学习的数据流组合分类器的融合策略。在判定测试样本上的决策时,根据直推学习理论设计满足每一个局部分类器约束度量的方法,保证了约束的可行性,解决了分布式分类聚集时最大熵的直推扩展问题。测试数据集上的实验证明,与已有的直推学习方法相比,此方法可以获得更好的决策精度,可以应用于数据流组合分类的融合。     

Non-linear statistical technique applied to data from baboon articular cartilage

Pattern recognition software was developed and applied together with statistical techniques to articular cartilage data from the knee joint of the baboon. The standard statistical method used for comparison was ANOVA which indicates linear discrimination. In addition a Karhunen-Loève expansion was performed to reduce the dimensionality of the data and provide independent uncorrelated variables. Nearest neighbour analysis, a non-linear method, when combined with bionomial probabilities gave discrimination that was not obtained by ANOVA. Use of pattern recognition and related techniques can improve and extend the analysis of biological data to include non-linear discrimination and classification.     

Satellite data cloud detection using deep learning supported by hyperspectral data

ABSTRACT

Deep learning methods can play an important role in satellite data cloud detection. The number and quality of training samples directly affect the accuracy of cloud detection based on deep learning. Therefore, selecting a large number of representative and high-quality training samples is a key step in cloud detection based on deep learning. For different satellite data sources, choosing sufficient and high-quality training samples has become an important factor limiting the application of deep learning in cloud detection. This paper presents a fast method for obtaining high-quality learning samples, which can be used for cloud detection of different satellite data with deep learning methods. AVIRIS (Airborne Visible Infrared Imaging Spectrometer) data, which have 224 continuous bands in the spectral range from 400–2500 nm, are used to provide cloud detection samples for different types of satellite data. Through visual interpretation, a sufficient number of cloud and clear sky pixels are selected from the AVIRIS data to construct a hyperspectral data sample library, which is used to simulate different satellite data (such as data from the Moderate Resolution Imaging Spectroradiometer (MODIS) and Landsat Operational Land Imager (OLI) satellites) as training samples. This approach avoids selecting training samples for different satellite sensors. Based on the Keras deep learning framework platform, a backpropagation (BP) neural network is employed for cloud detection from Landsat 8 OLI, National Polar-orbiting Partnership (NPP) Visible Infrared Imaging Radiometer Suite (VIIRS) and Terra MODIS data. The results are compared with cloud coverage results interpreted via artificial vision. The results demonstrate that the algorithm achieves good cloud detection results for the above data, and the overall accuracy is greater than 90%.     

Double-model adaptive fault detection and diagnosis applied to real flight data

The existing multiple model-based estimation algorithms for Fault Detection and Diagnosis (FDD) require the design of a model set, which contains a number of models matching different fault scenarios. To cope with partial faults or simultaneous faults, the model set can be even larger. A large model set makes the computational load intensive and can lead to performance deterioration of the algorithms. In this paper, a novel Double-Model Adaptive Estimation (DMAE) approach for output FDD is proposed, which reduces the number of models to only two, even for the FDD of partial and simultaneous output faults. Two Selective-Reinitialization (SR) algorithms are proposed which can both guarantee the FDD performance of the DMAE. The performance is tested using a simulated aircraft model with the objective of Air Data Sensors (ADS) FDD. Another contribution is that the ADS FDD using real flight data is addressed. Issues related to the FDD using real flight test data are identified. The proposed approaches are validated using real flight data of the Cessna Citation II aircraft, which verified their effectiveness in practice.     

An SDN-enhanced load-balancing technique in the cloud system

The vast majority of Web services and sites are hosted in various kinds of cloud services, and ordering some level of quality of service (QoS) in such systems requires effective load-balancing policies that choose among multiple clouds. Recently, software-defined networking (SDN) is one of the most promising solutions for load balancing in cloud data center. SDN is characterized by its two distinguished features, including decoupling the control plane from the data plane and providing programmability for network application development. By using these technologies, SDN and cloud computing can improve cloud reliability, manageability, scalability and controllability. SDN-based cloud is a new type cloud in which SDN technology is used to acquire control on network infrastructure and to provide networking-as-a-service (NaaS) in cloud computing environments. In this paper, we introduce an SDN-enhanced Inter cloud Manager (S-ICM) that allocates network flows in the cloud environment. S-ICM consists of two main parts, monitoring and decision making. For monitoring, S-ICM uses SDN control message that observes and collects data, and decision-making is based on the measured network delay of packets. Measurements are used to compare S-ICM with a round robin (RR) allocation of jobs between clouds which spreads the workload equitably, and with a honeybee foraging algorithm (HFA). We see that S-ICM is better at avoiding system saturation than HFA and RR under heavy load formula using RR job scheduler. Measurements are also used to evaluate whether a simple queueing formula can be used to predict system performance for several clouds being operated under an RR scheduling policy, and show the validity of the theoretical approximation.     

云存储服务中支持动态数据完整性检测方法

在云存储服务中,为了让用户可以验证存储在云存储服务器上数据的完整性,提出一种支持动态更新和公开验证的云存储数据完整性检测方法.通过引入双线性对和用户随机选择待检测数据块可以无限次验证数据完整性是否完好无损;可信第三方的引入解决了云用户与云存储供应商在数据完整性问题上产生的纠纷,实现数据完整性的公开验证;然后给出该方法的正确性、安全性以及性能分析,最后通过实验验证了该方法是高效可行的.     

云数据中心心跳检测与故障评估

为提高云数据中心的抗毁能力,针对软硬件日趋复杂化带来的诸多故障问题,以传统的服务器心跳检测机制为基础,结合云计算的特点,提出一种保证云数据中心有效运行的心跳检测与故障评估方案。在实验室搭建仿真平台进行实验,实验结果表明,该方案能够及时检测到节点故障,依据预先制定的故障等级评估标准,采用模糊综合法进行评估,为云数据中心的管理与运作提供参考依据与保障。     

Achieving fully privacy-preserving private range queries over outsourced cloud data

With the prevalence of cloud computing, data owners are motivated to outsource their databases to the cloud server. However, to preserve data privacy, sensitive private data have to be encrypted before outsourcing, which makes data utilization a very challenging task. Existing work either focus on keyword searches and single-dimensional range query, or suffer from inadequate security guarantees and inefficiency. In this paper, we consider the problem of multidimensional private range queries over encrypted cloud data. To solve the problem, we systematically establish a set of privacy requirements for multidimensional private range queries, and propose a multidimensional private range query (MPRQ) framework based on private block retrieval (PBR), in which data owners keep the query private from the cloud server. To achieve both efficiency and privacy goals, we present an efficient and fully privacy-preserving private range query (PPRQ) protocol by using batch codes and multiplication avoiding technique. To our best knowledge, PPRQ is the first to protect the query, access pattern and single-dimensional privacy simultaneously while achieving efficient range queries. Moreover, PPRQ is secure in the sense of cryptography against semi-honest adversaries. Experiments on real-world datasets show that the computation and communication overhead of PPRQ is modest.     

Split keyword fuzzy and synonym search over encrypted cloud data

Multimedia Tools and Applications - A substitute solution for various organizations of data owners to store their data in the cloud using storage as a service(SaaS). The outsourced sensitive data...     

Efficient multi-keyword ranked query over encrypted data in cloud computing

Cloud computing infrastructure is a promising new technology and greatly accelerates the development of large scale data storage, processing and distribution. However, security and privacy become major concerns when data owners outsource their private data onto public cloud servers that are not within their trusted management domains. To avoid information leakage, sensitive data have to be encrypted before uploading onto the cloud servers, which makes it a big challenge to support efficient keyword-based queries and rank the matching results on the encrypted data. Most current works only consider single keyword queries without appropriate ranking schemes. In the current multi-keyword ranked search approach, the keyword dictionary is static and cannot be extended easily when the number of keywords increases. Furthermore, it does not take the user behavior and keyword access frequency into account. For the query matching result which contains a large number of documents, the out-of-order ranking problem may occur. This makes it hard for the data consumer to find the subset that is most likely satisfying its requirements. In this paper, we propose a flexible multi-keyword query scheme, called MKQE to address the aforementioned drawbacks. MKQE greatly reduces the maintenance overhead during the keyword dictionary expansion. It takes keyword weights and user access history into consideration when generating the query result. Therefore, the documents that have higher access frequencies and that match closer to the users’ access history get higher rankings in the matching result set. Our experiments show that MKQE presents superior performance over the current solutions.     

云计算中加密数据的模糊关键字搜索方法

传统的可搜索加密方案仅支持精确匹配的搜索,在效率和性能上都不能适应云计算环境。用支持多种字符串相似性操作的R+树构建索引,实现了云计算中对加密数据的模糊关键字搜索;用编辑距离来量化关键字的相似度,提出了一种可以返回与关键字更接近的文件检索方法。通过字符串聚类提高了模糊关键字搜索的效率。     

Improving volcanic ash cloud detection by a robust satellite technique

Automated and reliable satellite-based techniques are strongly required for volcanic ash cloud detection and tracking. In fact, volcanic ash clouds pose a serious hazard for air traffic and the synoptic (and possibly frequent) coverage offered by satellites can provide exciting opportunities for monitoring activities as well as for risk mitigation purposes.A new, AVHRR-based technique for improved automatic detection of volcanic clouds by means of multi-temporal analysis of historical, long-term satellite records has been recently proposed. The technique basically rests on the Robust AVHRR Techniques (RAT) approach, which is an innovative strategy of satellite data analysis, devoted to a former characterisation of the measured signal, in terms of expected value and natural variability and a further recognition of signal anomalies by an automatic, unsupervised change detection step. In this work, an extension of this method to nighttime observations is presented, by using thermal infrared information coming from AVHRR bands centred approximately at 3.5, 11.0 and 12.0 μm. Results achieved for two recent eruptive events of Mount Etna (occurred in May 2000 and in July 2001) seem to be encouraging, showing clear improvements in terms of ash detection sensitivity as well as in terms of false alarms reduction. The technique performance is also evaluated by comparison with the traditional “split-window” brightness temperature difference method; this exercise revealed a general improvement obtained by the proposed approach, even though some common problems still remain unsolved. The main merits of such an approach are its intrinsic self-adaptability to different environmental/natural/observational conditions and its natural exportability also to different satellite sensors. The results here presented show the benefits of such a technique especially when different observational conditions (time of pass, seasonal period, atmospheric moisture, solar illumination, volcanic cloud composition, satellite angles of view, etc.) are considered.The future prospects, also in terms of possible operational scenarios, coming from the implementation of such an approach on the new generation of satellite sensors (like, for example, SEVIRI aboard Meteosat Second Generation platform) are also discussed.     

Using data mining technique to enhance tax evasion detection performance

Currently, tax authorities face the challenge of identifying and collecting from businesses that have successfully evaded paying the proper taxes. In solving the problem of tax evaders, tax authorities are equipped with limited resources and traditional tax auditing strategies that are time-consuming and tedious. These continued practices have resulted in the loss of a substantial amount of tax revenue for the government. The objective of the current study is to apply a data mining technique to enhance tax evasion detection performance. Using a data mining technique, a screening framework is developed to filter possible non-compliant value-added tax (VAT) reports that may be subject to further auditing. The results show that the proposed data mining technique truly enhances the detection of tax evasion, and therefore can be employed to effectively reduce or minimize losses from VAT evasion.     

An automated microfluidic colourimetric sensor applied in situ to determine nitrite concentration

A stand-alone sensor system with integrated sub-systems is demonstrated. The system is portable and capable of in situ reagent-based nutrient analysis. The system is based on a low cost optical detection method, together with an automated microfluidic delivery system that is able to detect nitrite with a limit of detection (LOD) of 15 nM. The sensor was operated in situ at Southampton Dockhead for 57 h (December 2010) and 375 measurements were taken.