Supporting early pruning in top-k query processing on massive data

This paper analyzes the execution behavior of “No Random Accesses” (NRA) and determines the depths to which each sorted file is scanned in growing phase and shrinking phase of NRA respectively. The analysis shows that NRA needs to maintain a large quantity of candidate tuples in growing phase on massive data. Based on the analysis, this paper proposes a novel top-k algorithm Top-K with Early Pruning (TKEP) which performs early pruning in growing phase. General rule and mathematical analysis for early pruning are presented in this paper. The theoretical analysis shows that early pruning can prune most of the candidate tuples. Although TKEP is an approximate method to obtain the top-k result, the probability for correctness is extremely high. Extensive experiments show that TKEP has a significant advantage over NRA.     

A survey of multilinear subspace learning for tensor data

Increasingly large amount of multidimensional data are being generated on a daily basis in many applications. This leads to a strong demand for learning algorithms to extract useful information from these massive data. This paper surveys the field of multilinear subspace learning (MSL) for dimensionality reduction of multidimensional data directly from their tensorial representations. It discusses the central issues of MSL, including establishing the foundations of the field via multilinear projections, formulating a unifying MSL framework for systematic treatment of the problem, examining the algorithmic aspects of typical MSL solutions, and categorizing both unsupervised and supervised MSL algorithms into taxonomies. Lastly, the paper summarizes a wide range of MSL applications and concludes with perspectives on future research directions.     

Projected sequential Gaussian processes: A C++ tool for interpolation of large datasets with heterogeneous noise

Heterogeneous datasets arise naturally in most applications due to the use of a variety of sensors and measuring platforms. Such datasets can be heterogeneous in terms of the error characteristics and sensor models. Treating such data is most naturally accomplished using a Bayesian or model-based geostatistical approach; however, such methods generally scale rather badly with the size of dataset, and require computationally expensive Monte Carlo based inference. Recently within the machine learning and spatial statistics communities many papers have explored the potential of reduced rank representations of the covariance matrix, often referred to as projected or fixed rank approaches. In such methods the covariance function of the posterior process is represented by a reduced rank approximation which is chosen such that there is minimal information loss. In this paper a sequential Bayesian framework for inference in such projected processes is presented. The observations are considered one at a time which avoids the need for high dimensional integrals typically required in a Bayesian approach. A C++ library, gptk, which is part of the INTAMAP web service, is introduced which implements projected, sequential estimation and adds several novel features. In particular the library includes the ability to use a generic observation operator, or sensor model, to permit data fusion. It is also possible to cope with a range of observation error characteristics, including non-Gaussian observation errors. Inference for the covariance parameters is explored, including the impact of the projected process approximation on likelihood profiles. We illustrate the projected sequential method in application to synthetic and real datasets. Limitations and extensions are discussed.     

Multi-class supervised classification of electrical borehole wall images using texture features

Electrical borehole wall images represent micro-resistivity measurements at the borehole wall. The lithology reconstruction is often based on visual interpretation done by geologists. This analysis is very time-consuming and subjective. Different geologists may interpret the data differently. In this work, linear discriminant analysis (LDA) in combination with texture features is used for an automated lithology reconstruction of ODP (Ocean Drilling Program) borehole 1203A drilled during Leg 197. Six rock groups are identified by their textural properties in resistivity data obtained by a Formation MircoScanner (FMS). Although discriminant analysis can be used for multi-class classification, non-optimal decision criteria for certain groups could emerge. For this reason, we use a combination of 2-class (binary) classifiers to increase the overall classification accuracy. The generalization ability of the combined classifiers is evaluated and optimized on a testing dataset where a classification rate of more than 80% for each of the six rock groups is achieved. The combined, trained classifiers are then applied on the whole dataset obtaining a statistical reconstruction of the logged formation. Compared to a single multi-class classifier the combined binary classifiers show better classification results for certain rock groups and more stable results in larger intervals of equal rock type.     

Potensoft: MATLAB-based software for potential field data processing, modeling and mapping

An open-source software including an easy-to-use graphical user interface (GUI) has been developed for processing, modeling and mapping of gravity and magnetic data. The program, called Potensoft, is a set of functions written in MATLAB. The most common application of Potensoft is spatial and frequency domain filtering of gravity and magnetic data. The GUI helps the user easily change all the required parameters. One of the major advantages of the program is to display the input and processed maps in a preview window, thereby allowing the user to track the results during the ongoing process. Source codes can be modified depending on the users' goals. This paper discusses the main features of the program and its capabilities are demonstrated by means of illustrative examples. The main objective is to introduce and ensure usage of the developed package for academic, teaching and professional purposes.     

CCS-MAC: Exploiting the overheard data for compression in wireless sensor networks

Both the overhearing and overhearing avoidance in a densely distributed sensor network may inevitably incur considerable power consumption. In this paper we propose a so-called CCS-MAC (collaborative compression strategy-based MAC) MAC protocol which facilitates to exploit those overheard data that is treated useless in traditional MAC protocols for the purpose of cost and energy savings. Particularly the CCS-MAC enables different sensor nodes to perform data compression cooperatively with regard to those overheard data, so that the redundancy of data prepared for the link layer transmission can be totally eliminated at the earliest. The problem of collaborative compression is analyzed and discussed along with a corresponding linear programming model formulated. Based on it a heuristic node-selection algorithm with a time complexity of (O(N²)) is proposed to the solve the linear programming problem. The node-selection algorithm is implemented in CCS-MAC at each sensor node in a distributed manner. The experiment results verify that the proposed CCS-MAC scheme can achieve a significant energy savings so as to prolong the lifetime of the sensor networks so far.     

Double-layered schema integration of heterogeneous XML sources

Schema integration aims to create a mediated schema as a unified representation of existing heterogeneous sources sharing a common application domain. These sources have been increasingly written in XML due to its versatility and expressive power. Unfortunately, these sources often use different elements and structures to express the same concepts and relations, thus causing substantial semantic and structural conflicts. Such a challenge impedes the creation of high-quality mediated schemas and has not been adequately addressed by existing integration methods. In this paper, we propose a novel method, named XINTOR, for automating the integration of heterogeneous schemas. Given a set of XML sources and a set of correspondences between the source schemas, our method aims to create a complete and minimal mediated schema: it completely captures all of the concepts and relations in the sources without duplication, provided that the concepts do not overlap. Our contributions are fourfold. First, we resolve structural conflicts inherent in the source schemas. Second, we introduce a new statistics-based measure, called path cohesion, for selecting concepts and relations to be a part of the mediated schema. The path cohesion is statistically computed based on multiple path quality dimensions such as average path length and path frequency. Third, we resolve semantic conflicts by augmenting the semantics of similar concepts with context-dependent information. Finally, we propose a novel double-layered mediated schema to retain a wider range of concepts and relations than existing mediated schemas, which are at best either complete or minimal, but not both. Performed on both real and synthetic datasets, our experimental results show that XINTOR outperforms existing methods with respect to (i) the mediated-schema quality using precision, recall, F-measure, and schema minimality; and (ii) the execution performance based on execution time and scale-up performance.     

Clustering data in an uncertain environment using an artificial immune system

Clustering of data in an uncertain environment can result into different partitions of the data at different points in time. Therefore, the initial formed clusters of non-stationary data can adapt over time which means that feature vectors associated with different clusters can follow different migration types to and from other clusters. This paper investigates different data migration types and proposes a technique to generate artificial non-stationary data which follows different migration types. Furthermore, the paper proposes clustering performance measures which are more applicable to measure the clustering quality in a non-stationary environment compared to the clustering performance measures for stationary environments. The proposed clustering performance measures in this paper are then used to compare the clustering results of three network based artificial immune models, since the adaptability and self-organising behaviour of the natural immune system inspired the modelling of network based artificial immune models for clustering of non-stationary data.     

A k-means type clustering algorithm for subspace clustering of mixed numeric and categorical datasets

Almost all subspace clustering algorithms proposed so far are designed for numeric datasets. In this paper, we present a k-means type clustering algorithm that finds clusters in data subspaces in mixed numeric and categorical datasets. In this method, we compute attributes contribution to different clusters. We propose a new cost function for a k-means type algorithm. One of the advantages of this algorithm is its complexity which is linear with respect to the number of the data points. This algorithm is also useful in describing the cluster formation in terms of attributes contribution to different clusters. The algorithm is tested on various synthetic and real datasets to show its effectiveness. The clustering results are explained by using attributes weights in the clusters. The clustering results are also compared with published results.