An empirical comparison of random forest-based and other learning-to-rank algorithms

Pattern Analysis and Applications - Random forest (RF)-based pointwise learning-to-rank (LtR) algorithms use surrogate loss functions to minimize the ranking error. In spite of their competitive...     

A performance comparison of event calendar algorithms: An empirical approach

We propose a suite of tests based on two-state Markov chains for experimentally assessing the dynamic performance of a variety of simulation event calendar implementations. In contrast to previous studies based on the standard hold model for evaluation of performance statically, the proposed Markov hold model is more general and can be used to examine how different implementations respond dynamically to dependent sequences of insertion and deletion requests. The Markov hold model is used to conduct tests based on random, stressed, and correlated input sequences of requests, with performance measures including completion times, sensitivity to correlations, sensitivity to duplication, and efficiency of data-handling. We apply these tests to fourteen different event calendar implementations. To demonstrate the utility of the proposed model, we also include a comparison of the event calendar algorithms on a token ring protocol with bursty Markovian packet-traffic.     

An empirical comparison of combinations of evolutionary algorithms and neural networks for classification problems.

There are numerous combinations of neural networks (NNs) and evolutionary algorithms (EAs) used in classification problems. EAs have been used to train the networks, design their architecture, and select feature subsets. However, most of these combinations have been tested on only a few data sets and many comparisons are done inappropriately measuring the performance on training data or without using proper statistical tests to support the conclusions. This paper presents an empirical evaluation of eight combinations of EAs and NNs on 15 public-domain and artificial data sets. Our objective is to identify the methods that consistently produce accurate classifiers that generalize well. In most cases, the combinations of EAs and NNs perform equally well on the data sets we tried and were not more accurate than hand-designed neural networks trained with simple backpropagation.     

An empirical comparison of learning algorithms for nonparametric scoring: the TreeRank algorithm and other methods

The TreeRank algorithm was recently proposed in [1] and [2] as a scoring-based method based on recursive partitioning of the input space. This tree induction algorithm builds orderings by recursively optimizing the Receiver Operating Characteristic curve through a one-step optimization procedure called LeafRank. One of the aim of this paper is the in-depth analysis of the empirical performance of the variants of TreeRank/LeafRank method. Numerical experiments based on both artificial and real data sets are provided. Further experiments using resampling and randomization, in the spirit of bagging and random forests are developed [3, 4] and we show how they increase both stability and accuracy in bipartite ranking. Moreover, an empirical comparison with other efficient scoring algorithms such as RankBoost and RankSVM is presented on UCI benchmark data sets.     

An up-to-date comparison of state-of-the-art classification algorithms

Current benchmark reports of classification algorithms generally concern common classifiers and their variants but do not include many algorithms that have been introduced in recent years. Moreover, important properties such as the dependency on number of classes and features and CPU running time are typically not examined. In this paper, we carry out a comparative empirical study on both established classifiers and more recently proposed ones on 71 data sets originating from different domains, publicly available at UCI and KEEL repositories. The list of 11 algorithms studied includes Extreme Learning Machine (ELM), Sparse Representation based Classification (SRC), and Deep Learning (DL), which have not been thoroughly investigated in existing comparative studies. It is found that Stochastic Gradient Boosting Trees (GBDT) matches or exceeds the prediction performance of Support Vector Machines (SVM) and Random Forests (RF), while being the fastest algorithm in terms of prediction efficiency. ELM also yields good accuracy results, ranking in the top-5, alongside GBDT, RF, SVM, and C4.5 but this performance varies widely across all data sets. Unsurprisingly, top accuracy performers have average or slow training time efficiency. DL is the worst performer in terms of accuracy but second fastest in prediction efficiency. SRC shows good accuracy performance but it is the slowest classifier in both training and testing.     

An empirical comparison of seven programming languages

Often heated, debates regarding different programming languages' effectiveness remain inconclusive because of scarce data and a lack of direct comparisons. The author addresses that challenge, comparatively analyzing 80 implementations of the phone-code program in seven different languages (C, C++, Java, Perl, Python, Rexx and Tcl). Further, for each language, the author analyzes several separate implementations by different programmers. The comparison investigates several aspects of each language, including program length, programming effort, runtime efficiency, memory consumption, and reliability. The author uses comparisons to present insight into program language performance     

An empirical time analysis of evolutionary algorithms as C programs

This article presents an empirical study devoted to characterize the computational efficiency behavior of an evolutionary algorithm (usually called canonical) as a C program. The study analyzes the effects of several implementation decisions on the execution time of the resulting evolutionary algorithm. The implementation decisions studied include: memory utilization (using dynamic vs. static variables and local vs. global variables), methods for ordering the population, code substitution mechanisms, and the routines for generating pseudorandom numbers within the evolutionary algorithm. The results obtained in the experimental analysis allow us to conclude that significant improvements in efficiency can be gained by applying simple guidelines to best program an evolutionary algorithm in C. Copyright © 2013 John Wiley & Sons, Ltd.     

An analytical comparison of two string searching algorithms

Average case analyses of two algorithms to locate the leftmost occurrence of a string Pattern in a string Text are conducted in this paper. One algorithm is based on a straightforward trial-and-error approach, the other one uses a sophisticated stragegy discovered by Knuth, Morris and Pratt (1977).Costs measured are the expected number of comparisons between individual characters. Let Naive and kmp denote the average case complexities of the two algorithms, respectively. We show that 1?(1/c)+(1/c²) is an accurate approximation for the ratio kmp/Naive, provided both Pattern and Text are random strings over an alphabet of size c.In both cases, the application of Markov chain theory is expedient for performing the analysis. However, in order to get rid of complex conditioning, the Markov chain model for the kmp algorithm is based on some heuristics. This approach is believed to be practically sound. Some indication on the complexity that might be involved in an exact average case analysis of the kmp algorithm can be found in the work by Guibas and Odlyzko (1981).     

An experimental comparison of range image segmentation algorithms

A methodology for evaluating range image segmentation algorithms is proposed. This methodology involves (1) a common set of 40 laser range finder images and 40 structured light scanner images that have manually specified ground truth and (2) a set of defined performance metrics for instances of correctly segmented, missed, and noise regions, over- and under-segmentation, and accuracy of the recovered geometry. A tool is used to objectively compare a machine generated segmentation against the specified ground truth. Four research groups have contributed to evaluate their own algorithm for segmenting a range image into planar patches     

An empirical comparison of nine pattern classifiers.

There are many learning algorithms available in the field of pattern classification and people are still discovering new algorithms that they hope will work better. Any new learning algorithm, beside its theoretical foundation, needs to be justified in many aspects including accuracy and efficiency when applied to real life problems. In this paper, we report the empirical comparison of a recent algorithm RM, its new extensions and three classical classifiers in different aspects including classification accuracy, computational time and storage requirement. The comparison is performed in a standardized way and we believe that this would give a good insight into the algorithm RM and its extension. The experiments also show that nominal attributes do have an impact on the performance of those compared learning algorithms.     

An empirical comparison of software fault tolerance and faultelimination

The authors compared two major approaches to the improvement of software-software fault elimination and software fault tolerance-by examination of the fault detection (and tolerance, where applicable) of five techniques: run-time assertions, multiversion voting, functional testing augmented by structural testing, code reading by stepwise abstraction, and static data-flow analysis. The focus was on characterizing the sets of faults detected by the techniques and on characterizing the relationships between these sets of faults. Two categories of questions were investigated: (1) comparison between fault elimination and fault tolerance techniques and (2) comparisons among various testing techniques. The results provide information useful for making decisions about the allocation of project resources, show strengths and weaknesses of the techniques studies, and indicate directions for future research     

Pull-down versus traditional menu types: An empirical comparison

An experiment was conducted to test the performance of pull-down versus traditional or explicit menus. Sixty subjects, including novice and experienced computer users, manipulated both types of menus to complete banking tasks similar to those found on Automatic Teller M achines. The order of the menus was randomly varied to control for learning effects. Across both types of users, traditional-style menus elicited fewer errors than did pull-down menus; however, no significant difference was found in the time to complete the banking task. Experienced users outperformed novice users in the amount of time taken to complete the task regardless of menu type, though no difference was found in the number of errors committed by both user types.     

An empirical comparison of commercial and open-source web vulnerability scanners

Web vulnerability scanners (WVSs) are tools that can detect security vulnerabilities in web services. Although both commercial and open-source WVSs exist, their vulnerability detection capability and performance vary. In this article, we report on a comparative study to determine the vulnerability detection capabilities of eight WVSs (both open and commercial) using two vulnerable web applications: WebGoat and Damn vulnerable web application. The eight WVSs studied were: Acunetix; HP WebInspect; IBM AppScan; OWASP ZAP; Skipfish; Arachni; Vega; and Iron WASP. The performance was evaluated using multiple evaluation metrics: precision; recall; Youden index; OWASP web benchmark evaluation; and the web application security scanner evaluation criteria. The experimental results show that, while the commercial scanners are effective in detecting security vulnerabilities, some open-source scanners (such as ZAP and Skipfish) can also be effective. In summary, this study recommends improving the vulnerability detection capabilities of both the open-source and commercial scanners to enhance code coverage and the detection rate, and to reduce the number of false-positives.     

An empirical comparison of generators for self similar simulated traffic

It is generally recognised that aggregated network traffic is self similar and that self similar traffic models should be used in simulation experiments when assessing the performance of a network. Many generators have been proposed to synthetically produce self similar simulation input; however most of them require the trace length to be known a priori. Four generators that allow continuous generation of self similar time series are evaluated in this work with respect to their ability to reproduce the desired level of self similarity. This extensive investigation uses ten times as many traces and twice the number of parameter values as previously reported. Three of the tested generators perform well but surprisingly the generator supplied with a widely used commercial network simulator is unusable. The reported results indicate that the generator based on multiplexing strictly alternating ON/OFF sources may perform better than generators based on chaotic maps, provided that more than 100 ON/OFF sources can be used. Three estimators for the degree of self similarity of a time series have been evaluated as part of the process, and the only acceptable one is based on a Wavelet decomposition of the traffic trace.     

An application of coroutines and backtracking in interactive systems

A software structure well-suited for the programming of interactive recognition and translation systems is described. This structure makes use of coroutines and backtracking in a highly coordinated and integrated fashion. A set of coroutine and backtracking primitives that supports this approach is defined. An example of the use of this approach is given.     

An empirical comparison of neural techniques for edge linking of images

Edge linking is a fundamental computer vision task, yet presents difficulties arising from the lack of information in the image. Viewed as a constrained optimisation problem, it is NP hard — being isomorphic to the classical travelling salesman problem. Self-learning neural techniques boast the ability to solve hard, ill-defined problems, and hence offer promise for such an application. This paper examines the suitability of four well-known unsupervised techniques for the task of edge linking, by applying them to a test bed of edge point images and then evaluating their performance both quantitatively and qualitatively. Techniques studied are the elastic net, active contours, Kohonen map and Burr's modified elastic net. Of these, only the elastic net and the Kohonen map are realistic contenders for general edge-linking tasks. However, the other two exhibit behaviour which may make them particularly suited to some specific image-processing and computer vision applications.     

An extensive empirical comparison of ensemble learning methods for binary classification

We present an extensive empirical comparison between nineteen prototypical supervised ensemble learning algorithms, including Boosting, Bagging, Random Forests, Rotation Forests, Arc-X4, Class-Switching and their variants, as well as more recent techniques like Random Patches. These algorithms were compared against each other in terms of threshold, ranking/ordering and probability metrics over nineteen UCI benchmark data sets with binary labels. We also examine the influence of two base learners, CART and Extremely Randomized Trees, on the bias–variance decomposition and the effect of calibrating the models via Isotonic Regression on each performance metric. The selected data sets were already used in various empirical studies and cover different application domains. The source code and the detailed results of our study are publicly available.     

An empirical comparison of data flow and mutation-based test adequacy criteria

Evaluation of the adequacy of a test set consisting of one or more test cases is a problem oftes encountered in software testing environments. Two test adequacy criiteria are considered, namely the data flow based all-uses criterion and a mutation based criterion. An empirical study was conducted to compare the ‘difficulty’ of satisfying the two criteria and their costs. Similar studies conducted in the past are discussed in the light of this study. A discussion is also presented of how and why the results of this study, when viewed in conjunction with the results of earlier comparisons of testing methods, are useful to a software test team.     

Document-level sentiment classification: An empirical comparison between SVM and ANN

Document-level sentiment classification aims to automate the task of classifying a textual review, which is given on a single topic, as expressing a positive or negative sentiment. In general, supervised methods consist of two stages: (i) extraction/selection of informative features and (ii) classification of reviews by using learning models like Support Vector Machines (SVM) and Na?¨ve Bayes (NB). SVM have been extensively and successfully used as a sentiment learning approach while Artificial Neural Networks (ANN) have rarely been considered in comparative studies in the sentiment analysis literature. This paper presents an empirical comparison between SVM and ANN regarding document-level sentiment analysis. We discuss requirements, resulting models and contexts in which both approaches achieve better levels of classification accuracy. We adopt a standard evaluation context with popular supervised methods for feature selection and weighting in a traditional bag-of-words model. Except for some unbalanced data contexts, our experiments indicated that ANN produce superior or at least comparable results to SVM’s. Specially on the benchmark dataset of Movies reviews, ANN outperformed SVM by a statistically significant difference, even on the context of unbalanced data. Our results have also confirmed some potential limitations of both models, which have been rarely discussed in the sentiment classification literature, like the computational cost of SVM at the running time and ANN at the training time.