A fast and effective method to find correlations among attributes in databases

The problem of identifying meaningful patterns in a database lies at the very heart of data mining. A core objective of data mining processes is the recognition of inter-attribute correlations. Not only are correlations necessary for predictions and classifications – since rules would fail in the absence of pattern – but also the identification of groups of mutually correlated attributes expedites the selection of a representative subset of attributes, from which existing mappings allow others to be derived. In this paper, we describe a scalable, effective algorithm to identify groups of correlated attributes. This algorithm can handle non-linear correlations between attributes, and is not restricted to a specific family of mapping functions, such as the set of polynomials. We show the results of our evaluation of the algorithm applied to synthetic and real world datasets, and demonstrate that it is able to spot the correlated attributes. Moreover, the execution time of the proposed technique is linear on the number of elements and of correlations in the dataset.     

高光谱大气红外探测器通道选择方法及试验研究

考虑到高光谱大气红外探测器通道之间的相关性、变分同化的时效性等,需要进行通道选择。利用主成分-逐步回归法进行AIRS通道选择研究。由于短波CO2通道易受太阳光影响,分白天和夜晚进行。具体执行过程中,首先进行通道预处理,然后分别对温度和湿度雅可比矩阵进行主成分分析,并采用逐步回归法找出对前几个主成分影响较大的通道得到入选的通道子集。进而根据经验和实际观测资料,为了达到全局最优并兼顾局部,基于分区的思想,采用主成分-双区逐步回归法进行通道选择。结果表明:①利用AIRS进行反演时,合理选择通道是非常必要的;②主成分-双区逐步回归法得到的通道组合进行温度、湿度反演的误差整体比基于信息熵分步迭代小。〖JP〗     

Analysis of collocated AIRS and MODIS data: a global investigation of correlations between clouds and atmosphere in 2004–2012

We used collocated observations from the Moderate Resolution Imaging Spectroradiometer (MODIS) and the Atmospheric Infrared Sounder (AIRS) to investigate correlations between cloud parameters and atmospheric stability. We focus on low clouds and specifically investigate the cloud parameters cloud cover and cloud optical thickness from MODIS. The selected atmospheric parameters from AIRS are maximum relative humidity (MRH), lower tropospheric stability (LTS), and water vapour gradient (QTS). The correlations were tested for temporal and regional variation on a global scale and over a time frame of 10 years. Cloud cover and MRH show weak correlations and strong variations on both the temporal and spatial scales. However, cloud cover and lower tropospheric stability show a high correlation in areas with low maritime clouds. The correlation is relatively stable, but slightly increased for the years 2009–2012. Correlations between cloud cover and QTS show a similar behaviour, but slightly stronger variations on the spatial and temporal scales, with better correlations in the East Pacific and from 2004 to 2012. The correlations with cloud optical thickness are weaker in all three cases. A more detailed analysis of the Southeast Pacific shows the influence of El Niño Southern Oscillation (ENSO) on most parameters, but a relatively stable behaviour for the connection of cloud fraction and LTS. Based on the analysis, we suggest that relative humidity is an insufficient approach to link atmospheric properties and low cloud cover. However, we find good correlations with respect to LTS and QTS. LTS in particular indicates low temporal fluctuations, even in the case of influence by ENSO.     

Comparison of atmospheric correction methods using ASTER data for the area of Crete,Greece

The purpose of atmospheric correction is to produce more accurate surface reflectance and to potentially improve the extraction of surface parameters from satellite images. To achieve this goal the influences of the atmosphere, solar illumination, sensor viewing geometry and terrain information have to be taken into account. Although a lot of information from satellite imagery can be extracted without atmospheric correction, the physically based approach offers advantages, especially when dealing with multitemporal data and/or when a comparison of data provided by different sensors is required. The use of atmospheric correction models is limited by the need to supply data related to the condition of the atmosphere at the time of imaging. Such data are not always available and the cost of their collection is considerable, hence atmospheric correction is performed with the use of standard atmospheric profiles. The use of these profiles results in a loss of accuracy. Therefore, site-dependent databases of atmospheric parameters are needed to calibrate and to adjust atmospheric correction methods for local level applications. In this article, the methodology and results of the project Adjustment of Atmospheric Correction Methods for Local Studies: Application in ASTER (Advanced Spaceborne Thermal Emission and Reflection Radiometer) (ATMOSAT) for the area of Crete are presented. ATMOSAT aimed at comparing several atmospheric correction methods for the area of Crete, as well as investigating the effects of atmospheric correction on land cover classification and change detection. Databases of spatio-temporal distributions of all required input parameters (atmospheric humidity, aerosols, spectral signatures, land cover and elevation) were developed and four atmospheric correction methods were applied and compared. The baseline for this comparison is the spatial distribution of surface reflectance, emitted radiance and brightness temperature as derived by ASTER Higher Level Products (HLPs). The comparison showed that a simple image based method, which was adjusted for the study area, provided satisfactory results for visible, near infrared and short-wave infrared spectral areas; therefore it can be used for local level applications. Finally, the effects of atmospheric correction on land cover classification and change detection were assessed using a time series of ASTER multispectral images acquired in 2000, 2002, 2004 and 2006. Results are in agreement with past studies, indicating that for this type of application, where a common radiometric scale is assumed among the multitemporal images, atmospheric correction should be taken into consideration in pre-processing.     

The implementation of land use and relief height data in predicting the evolution of the atmospheric boundary-layer within the urban environment

It is vitally important to understand the structure of the atmospheric boundary-layer within the urban environment and the local rural surroundings. Within the local environment the structure of the earth's boundary-layer is highly turbulent and diffusive, with distinctive inner and outer regions. The flow in the inner region is mainly influenced by the physical characteristics of the terrain, such as the relief and the land use, whereas in the outer region larger scale phenomena need to be taken into account. In this paper, novel methods are implemented to understand the structure of the atmospheric boundary-layer within the urban environment. These include a technique that allows the larger scale topographical effects on the atmospheric boundary-layer to be incorporated, yet still enabling fine resolution of the boundary-layer within the smaller scale of the urban environment. This is achieved by manipulating two large data sets for the land use and the relief height within the UK in conjunction with sophisticated mathematical modelling. In this paper a simple but effective algorithm is presented that allows these large data sets to be practically implemented within the framework of an atmospheric boundary-layer fluid flow model.     

Effective and efficient classification on a search-engine model

Traditional document classification frameworks, which apply the learned classifier to each document in a corpus one by one, are infeasible for extremely large document corpora, like the Web or large corporate intranets. We consider the classification problem on a corpus that has been processed primarily for the purpose of searching, and thus our access to documents is solely through the inverted index of a large scale search engine. Our main goal is to build the “best” short query that characterizes a document class using operators normally available within search engines. We show that surprisingly good classification accuracy can be achieved on average over multiple classes by queries with as few as 10 terms. As part of our study, we enhance some of the feature-selection techniques that are found in the literature by forcing the inclusion of terms that are negatively correlated with the target class and by making use of term correlations; we show that both of those techniques can offer significant advantages. Moreover, we show that optimizing the efficiency of query execution by careful selection of terms can further reduce the query costs. More precisely, we show that on our set-up the best 10-term query can achieve 93% of the accuracy of the best SVM classifier (14,000 terms), and if we are willing to tolerate a reduction to 89% of the best SVM, we can build a 10-term query that can be executed more than twice as fast as the best 10-term query.     

Adaptive Kernel Firefly Algorithm Based Feature Selection and Q-Learner Machine Learning Models in Cloud

CC’s (Cloud Computing) networks are distributed and dynamic as signals appear/disappear or lose significance. MLTs (Machine learning Techniques) train datasets which sometime are inadequate in terms of sample for inferring information. A dynamic strategy, DevMLOps (Development Machine Learning Operations) used in automatic selections and tunings of MLTs result in significant performance differences. But, the scheme has many disadvantages including continuity in training, more samples and training time in feature selections and increased classification execution times. RFEs (Recursive Feature Eliminations) are computationally very expensive in its operations as it traverses through each feature without considering correlations between them. This problem can be overcome by the use of Wrappers as they select better features by accounting for test and train datasets. The aim of this paper is to use DevQLMLOps for automated tuning and selections based on orchestrations and messaging between containers. The proposed AKFA (Adaptive Kernel Firefly Algorithm) is for selecting features for CNM (Cloud Network Monitoring) operations. AKFA methodology is demonstrated using CNSD (Cloud Network Security Dataset) with satisfactory results in the performance metrics like precision, recall, F-measure and accuracy used.     

A comparison of algorithms for Near‐Real Time water vapour retrieval from MODIS

When studying the Earth's surface from space it is important that the component of the signal measured by the satellite‐based sensor due to the atmosphere is accurately estimated and removed. Such atmospheric correction requires good knowledge of atmospheric parameters including precipitable water (PW), ozone concentration and aerosol optical depth. To make full use of the capabilities of satellite sensors such as the Moderate Resolution Imaging Spectroradiometer (MODIS) these parameters should be accurately estimated in Near‐Real Time (NRT) with complete global coverage approximately every two days. NRT retrieval of the required ancillary information facilitates the atmospheric correction of such direct broadcast data from the MODIS instrument in the operational environment. In this paper three Near Infrared (NIR) algorithms for PW retrieval from MODIS are compared to determine which is most suitable for use in an operational MODIS‐based process for the atmospheric correction of spectral reflectance data. Two of the algorithms estimate PW in NRT and gave RMS errors of approximately 0.48 g cm^?2 (23%) and 0.59 g cm^?2 (28%), respectively, when compared against radiosonde data and modelled PW fields over Western Australia. The third algorithm was the NIR PW product from MODIS (MOD05) archived by the Distributive Active Archive Centre (DAAC). For the same locations the MOD05 NIR PW dataset gave an RMS error of approximately 0.95 g cm^?2 (44%). In each of the cases the best results were obtained after optimal cloudmasking of the NIR data. In this paper, the accuracy and suitability of the three algorithms for use in the operational atmospheric correction of MODIS data are evaluated and the importance of an accurate cloudmask for atmospheric correction in NRT is discussed.     

多标签学习中基于互信息的快速特征选择方法

针对传统的基于启发式搜索的多标记特征选择算法时间复杂度高的问题，提出一种简单快速的多标记特征选择（EF-MLFS）方法。首先使用互信息（MI）衡量每个维度的特征与每一维标记之间的相关性，然后将所得相关性相加并排序，最后按照总的相关性大小进行特征选择。将所提方法与六种现有的比较有代表性的多标记特征选择方法作对比，如最大依赖性最小冗余性（MDMR）算法和基于朴素贝叶斯的多标记特征选择（MLNB）方法等。实验结果表明，EF-MLFS方法进行特征选择并分类的结果在平均准确率、覆盖率、海明损失等常见的多标记分类评价指标上均达最优；该方法无需进行全局搜索，因此时间复杂度相较于MDMR、对偶多标记应用（PMU）等方法也有明显降低。     

Dynamic service selection with QoS constraints and inter-service correlations using cooperative coevolution

Building business processes by Web services in cloud computing has become the hotspot of service applications. Due to the complexity and uncertainty of business environment, QoS violations of service processes often take place at run-time. To rapidly recover from failures and minimize their impacts on the original execution plan of service processes, dynamic service selection is urgently needed once potential QoS violations are detected. However, existing research works do not fully investigate QoS constraints and inter-service correlations, as well as the breach penalty caused by service adjustment. In this paper, we present a new cooperative coevolutionary approach for dynamic service selection with QoS constraints and inter-service correlations. First, a novel formal model for the dynamic service selection problem with QoS constraints and inter-service correlations is presented. Second, a Double Information based Cooperative Coevolutionary algorithm (DICC) is proposed which uses Potter’s cooperative coevolutionary framework and provides both local and global knowledge for the dynamic service selection optimization. Finally, we develop a prototype system to apply our approach and adopt different test cases to show that our DICC approach performs more effectively and efficiently than existing algorithms.     

语音合成中的韵律关联模型

基于大规模语音数据库的文语转换系统(Text-to-Speech , TTS)中,如何选取合适的语音基元是提高合成语音自然度的重要因素。本文研究了连续语流中的韵律关联现象,提出了包含韵律关联参数的汉语韵律特征参数集,基于数据挖掘中的关联规则模型(Association Rules Model)建立韵律关联模型,并将该模型应用于基元选取。实验表明,该方法有效地利用了语音基元的韵律及关联信息,符合人耳的知觉感受,使得合成语音自然度的主观评测MOS(Mean Opinion Score)得分与不考虑韵律关联时的结果相比提高了12.22%(3.49/3.11)。     

Estimation of Sea Surface Temperature from the Spinning Enhanced Visible and Infrared Imager, improved using numerical weather prediction

Most of the operational Sea Surface Temperature (SST) products derived from satellite infrared radiometry use multi-spectral algorithms. They show, in general, reasonable performances with root mean square (RMS) residuals around 0.5 K when validated against buoy measurements, but have limitations, particularly a component of the retrieval error that relates to such algorithms' limited ability to cope with the full variability of atmospheric absorption and emission. We propose to use forecast atmospheric profiles and a radiative transfer model to simulate the algorithmic errors of multi-spectral algorithms. In the practical case of SST derived from the Spinning Enhanced Visible and Infrared Imager (SEVIRI) onboard Meteosat Second Generation (MSG), we demonstrate that simulated algorithmic errors do explain a significant component of the actual errors observed for the non linear (NL) split window algorithm in operational use at the Centre de Météorologie Spatiale (CMS). The simulated errors, used as correction terms, reduce significantly the regional biases of the NL algorithm as well as the standard deviation of the differences with drifting buoy measurements. The availability of atmospheric profiles associated with observed satellite-buoy differences allows us to analyze the origins of the main algorithmic errors observed in the SEVIRI field of view: a negative bias in the inter-tropical zone, and a mid-latitude positive bias. We demonstrate how these errors are explained by the sensitivity of observed brightness temperatures to the vertical distribution of water vapour, propagated through the SST retrieval algorithm.     

考虑标记间协作的标记分布学习

近些年来,作为一种新的有监督学习范式,标记分布学习(LDL)已被应用到多个领域,如人脸年龄估计、头部姿态估计、电影评分预测、公共视频监控中的人群计数等,并且在这些领域的相关任务上取得了一定性能上的进展.最近几年,很多关于标记分布学习的算法在解决标记分布学习问题时考虑到了标记之间的相关性,但是现有方法大多将标记相关性作为...     

Building a robust linear model with forward selection and stepwise procedures

Classical step-by-step algorithms, such as forward selection (FS) and stepwise (SW) methods, are computationally suitable, but yield poor results when the data contain outliers and other contaminations. Robust model selection procedures, on the other hand, are not computationally efficient or scalable to large dimensions, because they require the fitting of a large number of submodels. Robust and computationally efficient versions of FS and SW are proposed. Since FS and SW can be expressed in terms of sample correlations, simple robustifications are obtained by replacing these correlations by their robust counterparts. A pairwise approach is used to construct the robust correlation matrix—not only because of its computational advantages over the d-dimensional approach, but also because the pairwise approach is more consistent with the idea of step-by-step algorithms. The proposed robust methods have much better performance compared to standard FS and SW. Also, they are computationally very suitable and scalable to large high-dimensional data sets.     

Packie图广泛应用于预测原油常压塔产品分馏的精度

Packie图广泛应用于预测原油常压塔产品分馏的精度.为了能够应用于计算机的模拟与计算,本文通过采用非线性同归和单纯形的加速算法拟合Packie图的"原油常压精馏塔塔顶产品与一线分馏精度图"和"原油常压精馏塔侧线产品分馏精度图",建立可用计算机计算的数学模型.该模型方程形式简单,并估计其模型参数误差.证明,该数学关联式的计算结果,跟两图得出的数据能够吻合,相关系数分别为0.9982697和0.999005,计算准确.     

Using a digital terrain model to calculate visual sunrise and sunset times

Since its advent the Digital Terrain Model (DTM) has been employed widely in the sciences for the solution of problems requiring a digital model of landforms. In this paper we describe a new use of the DTM in the calculation of highly accurate visual sunrise and sunset times that are required by the observant population in Israel. We have employed ray tracing to determine the effect of atmospheric refraction through a simplified layered atmosphere. A general analytic expression for the atmospheric refraction was determined from these calculations as a function of the observer’s height for two model atmospheres known as the subtropical summer and winter atmospheres. These expressions determine the general magnitude of the refraction as a function of apparent view angle of the observer. We also determined a simplified analytic expression for the effect of atmospheric refraction on the vertical angular profile of the mountainous horizon as calculated from the DTM (atmospheric refraction magnifies the mountainous features of the horizon). These expressions are then used in calculating the apparent vertical angular position of the Sun as a function of time. The time when the upper limb of the Sun first (last) appears to rise (set) over the horizon adjusted for the effect of refraction determines the time of the visible sunrise (sunset). Comparison with observations have shown that the visible sunrise and sunset times can be typically determined to better than 15 s using the 25 m DTM of Israel and eastern Jordan.     

Spatial and temporal patterns of remotely-sensed and field-measured rainfall in southern California

Quantification of spatial and temporal patterns of rainfall is an important step toward developing regional hydrological models. However, traditionally used rain gauge data are sparse and do not always provide adequate spatial representation of rainfall. In this study, we evaluated the daily 1-degree resolution remotely-sensed atmospheric precipitation data provided by Global Precipitation Climatology Project (GPCP) as an alternative to rain gauge-measured data. We analyzed data from the watersheds of southern California during the period of 1996-2003, focusing on the comparison of patterns of spatial, seasonal, and interannual rainfall dynamics. We used Empirical Orthogonal Functions to discern the patterns of precipitation and atmospheric circulation at different time scales, from synoptic to interannual. The correlation between the daily rain gauge-measured and remotely-sensed precipitation was poor and the resulting patterns of remotely-sensed precipitation are different than the temporal patterns of precipitation accumulated by rain gauges. These differences likely result from the fact that the precipitable water concentration measured by satellites is not always highly correlated to rainfall reaching the earth surface. Differences in the spatial resolution and coverage of the two methods and the differential influence of orographic effects and wind patterns on each also contribute to low correlations. We conclude that daily remotely-sensed precipitation produced at GPCP is not currently appropriate for use in assessing fine-scale hydrological processes in arid zones like southern California, and would not be a recommended surrogate for event-based hydrologic modeling. At the same time, the interannual variabilities of remotely-sensed and gauge-measured precipitation were highly correlated and the regional patterns of gauge-measured and remotely-sensed precipitation variability were similar; though the total precipitation estimated from satellite data was substantially lower than the gauge-measured data. Therefore, remotely-sensed precipitation data may be appropriate for use in long-term regional hydrologic or climate modeling focused on trends and patterns of rainfall in southern California. Both data sets showed that precipitation generally decreases from the northern to the southern watersheds. At interannual time-scale, the rainfall is related to the ENSO cycle. At synoptic time-scales, the rainfall patterns in southern California result from atmospheric moisture transport from the south-southwest.     

Atmospheric correction of optical imagery from MODIS and Reanalysis atmospheric products

In this paper we analyze the differences obtained in the atmospheric correction of optical imagery covering bands located in the Visible and Near Infra-Red (VNIR), Short-Wave Infra-Red (SWIR) and Themal-Infrared (TIR) spectral regions when atmospheric profiles extracted from different sources are used. In particular, three sensors were used, Compact High Resolution Imaging Spectrometer (CHRIS), Advanced Spaceborne Thermal Emission and Reflection radiometer (ASTER) and Landsat5 Thematic Mapper (TM), whereas four atmospheric profiles sources were considered: i) local soundings launched near the sensor overpass time, ii) Moderate Resolution Radiometer (MODIS) atmospheric profiles product (MOD07), iii) Atmospheric Correction Parameter Calculator (ACPC) generated by the National Center for Environmental Prediction (NCEP) and iv) Modified Atmospheric Profiles from Reanalysis Information (MAPRI), which includes data from NCEP and National Center of Atmospheric Research (NCAR) Reanalysis project but interpolated to 34 atmospheric levels and resampled to 0.5° × 0.5°. MODIS aerosol product (MOD04) was also used to extract Aerosol Optical Thickness (AOT) values at 550 nm. Analysis was performed for three test dates (12th July 2003, 18th July 2004 and 13th July 2005) over an agricultural area in Spain. Results showed that air temperature vertical profiles were similar for the four sources, whereas dew point temperature profiles showed significant differences at some particular levels. Atmospheric profiles were used as input to MODTRAN4 radiative transfer code in order to compute atmospheric parameters involved in atmospheric correction, with the aim of retrieving surface reflectances in the case of VNIR and SWIR regions, and Land Surface Temperature (LST) in the case of the TIR region. For the VNIR and SWIR region, significant differences depending on the atmospheric profile used were not found, particularly in the Visible region in which the AOT content is the main parameter involved in the atmospheric correction. In the case of TIR, differences depending on the atmospheric profile used were appreciable, since in this case the main parameter involved in the atmospheric correction is the water vapor content, which depends on the vertical profile. In terms of LST retrieval from ASTER data (2004 test case), all profiles provided satisfactory results compared to the ones obtained when using a local sounding, with errors of 0.3 K for ACPC and MAPRI cases and 0.7 K for MOD07. When retrieving LST from TM data (2005 test case), errors for MOD07 and MAPRI were 0.6 and 0.9 K respectively, whereas ACPC provided an error of 2 K. The results presented in this paper show that the different atmospheric profile sources are useful for accurate atmospheric correction when local soundings are not available. In particular, MOD07 product provides atmospheric information at the highest spatial resolution, 5 km, although its use is limited from 2000 to present, whereas MAPRI provides historical information from 1970 to present, but at lower spatial resolution.     

Remote sensing of the terrestrial environment using middle infrared radiation (3.0–5.0 µm)

The middle infrared (MIR) spectral region, between 3.0 and 5.0?µm in the electromagnetic spectrum, features a myriad of atmospheric windows. The favourable atmospheric penetration of electromagnetic radiation at MIR wavelengths has been readily noted. To exploit this atmospheric window, there have been several remote sensing instruments acquiring radiation from the terrestrial environment. However, these data remain under-utilized for terrestrial environmental studies. The principal reason for this is the hybrid nature of this spectral region, as it comprises a combination of both reflected and emitted radiation during daytime measurement. Despite this, there are many attributes of the MIR spectral region that should encourage its use. Studies focusing on a wide range of terrestrial environment characteristics using MIR radiation have advocated that these data be used. Moreover, there has been a progressive maturation of the methods for processing the radiation acquired by instruments in the MIR spectral region, allowing the informed use of the unique information provided by this part of the spectrum. In particular, these methods have allowed the retrieval of MIR reflectance from the full MIR radiation acquired by a sensor and this information has been demonstrated to be particularly useful for the study of the terrestrial environment and its change. It would appear that the full potential of MIR radiation is unrealized. This paper is a review that aims to motivate and advise of future research using MIR radiation, and in particular MIR reflectance. The paper outlines the physical principles determining MIR radiation interactions with the terrestrial environment; reviews current status of processing methods of MIR radiation to retrieve MIR reflectance; and how the retrieved data have been used. In conclusion, future research priorities are suggested.     

Temporal flexibility of permit trading when pollutants are correlated

It is now accepted that climate change is due to the cumulative and joint effect of multiple atmospheric trace constituents. The importance of adopting a comprehensive approach to global warming that would account for multiple interacting pollutants is increasingly recognized. In this paper we propose to extend the analysis of intertemporal permit trading to a framework encompassing multiple correlated pollutants. In doing so our aim is to assess the consequence of providing “when flexibility” to participants in a pollution market scheme with respect to the timing of use of their permits. In the same manner as local pollution hotspots have been identified as potential drawbacks of allowing intra-regional trade, temporal flexibility has the potential to induce a type of “temporal hotspot” of pollution that could undermine the efficiency of intertemporal trading schemes — especially when the stock of a particular pollution induces damage in itself but also impacts on the accumulation of another detrimental pollution stock. In a first step, we assess the impact of various types of correlations (technological and physical) on the socially optimal accumulation of regional and global pollutants. We illustrate that even in the case of a linear damage function the regional stocks may have ambiguous impacts on the global stock. In a second step, we show that it is possible for a global benevolent regulator to have recourse to a set of intertemporal trading schemes to induce individual agents to take socially optimal decisions over time. One requirement is to implement a set of time-dependent intertemporal trading rates. We also analyze the impact on pollutants accumulation of implementing non-optimal intertemporal trading rates.