Temporal generalization of sub-pixel vegetation mapping with multiple machine learning and atmospheric correction algorithms

Temporal generalization allows a trained classification algorithm to be applied to multiple images across time to derive reliable classification map products. It is a challenging remote-sensing research topic since the results are dependent on the selection of atmospheric correction methods, classification algorithms, validation processes, and their varying combinations. This study examined the temporal generalization of sub-pixel vegetation mapping using multiple Landsat images (1990, 1996, 2004, and 2010). All Landsat images were processed with two atmospheric correction methods: simple dark object subtraction (DOS) and the Landsat Ecosystem Disturbance Adaptive Processing System (LEDAPS) algorithm. For the sub-pixel vegetation mapping of the 2004 Landsat image, we used high-resolution OrbView-3 images as a training/validation data set and compared three machine learning algorithms (neural networks, random forests, and classification and regression trees) for their classification performance. The trained classifiers were then applied to other Landsat images (1990, 1996, and 2010) to derive sub-pixel vegetation map products. For the 2004 Landsat image classification, cross-validation shows similar classification results for neural networks (root mean square error (RMSE) = 0.099) and random forests (RMSE = 0.100) algorithms, and both are better than classification and regression trees (RMSE = 0.123). Pseudo-invariant pixels between 2004 and 2010 were used as validation points to evaluate the temporal generalizability of classification algorithms. Simple DOS and LEDAPS atmospheric correction resulted in similar accuracy statistics. The neural-network-based classifier performed best in generating reliable sub-pixel vegetation map products across time.     

Robustness and generalization

We derive generalization bounds for learning algorithms based on their robustness: the property that if a testing sample is “similar” to a training sample, then the testing error is close to the training error. This provides a novel approach, different from complexity or stability arguments, to study generalization of learning algorithms. One advantage of the robustness approach, compared to previous methods, is the geometric intuition it conveys. Consequently, robustness-based analysis is easy to extend to learning in non-standard setups such as Markovian samples or quantile loss. We further show that a weak notion of robustness is both sufficient and necessary for generalizability, which implies that robustness is a fundamental property that is required for learning algorithms to work.     

Problems, methods and specialization

The large aspiration to place the whole of software development alongside the established branches as one more branch of engineering is misconceived. The author discusses how our aspiration should be to develop specialized branches of software engineering, each meriting its own place alongside the specialized established branches     

Just-in-time value specialization

JavaScript emerges today as one of the most important programming languages for the development of client-side web applications. Therefore, it is essential that browsers be able to execute JavaScript programs efficiently. However, the dynamic nature of this programming language makes it very challenging to achieve this much needed efficiency. In this paper we propose parameter-based value specialization as a way to improve the quality of the code produced by JIT engines. We have empirically observed that almost 60% of the JavaScript functions found in the world's 100 most popular websites are called only once, or are called with the same parameters. Capitalizing on this observation, we adapt a number of classic compiler optimizations to specialize code based on the runtime values of function's actual parameters. We have implemented the techniques proposed in this paper in IonMonkey, an industrial quality JavaScript JIT compiler developed at the Mozilla Foundation. Our experiments, run across three popular JavaScript benchmarks, SunSpider, V8 and Kraken, show that, in spite of its highly speculative nature, our optimization pays for itself. As an example, we have been able to speed up V8 by 4.83%, and to reduce the size of its generated native code by 18.84%.     

Class hierarchy specialization

Many class libraries are designed with an emphasis on generality and extensibility. Applications often exercise only part of a library's functionality. As a result, the objects created by an application may contain unused (user-specified or compiler-generated) members. Redundant members in objects are undesirable because they increase an application's memory usage. We present an algorithm for specializing a class hierarchy with respect to its usage in a program . That is, the algorithm analyzes the member access patterns for 's variables, and creates distinct classes for variables that access different members. The algorithm addresses the inheritance mechanisms of C++ in their full generality, including multiple inheritance and virtual (shared) inheritance. Class hierarchy specialization reduces object size, and can be viewed as a space optimization. However, execution time may also be reduced through reduced object creation or destruction time, and caching and paging effects. Class hierarchy specialization may also create new opportunities for existing optimizations such as call devirtualization and inlining. In addition, specialization may be useful in tools for software maintenance and program understanding. Received: 11 May 1999 / 14 December 1999     

Abstraction and specialization of information granules

In his paper, we introduce a model of generalization and specialization of information granules. The information granules themselves are modeled as fuzzy sets or fuzzy relations. The generalization is realized by or-ing fuzzy sets while the specialization is completed through logic and operation. These two logic operators are realized using triangular norms (that is t- and a-norms). We elaborate on two (top-down and bottom-up) strategies of constructing information granules that arise as results of generalization and specialization. Various triangular norms are experimented with and some conclusions based on numeric studies are derived.     

Genetic programming and evolutionary generalization

In genetic programming (GP), learning problems can be classified broadly into two types: those using data sets, as in supervised learning, and those using an environment as a source of feedback. An increasing amount of research has concentrated on the robustness or generalization ability of the programs evolved using GP. While some of the researchers report on the brittleness of the solutions evolved, others proposed methods of promoting robustness/generalization. It is important that these methods are not ad hoc and are applicable to other experimental setups. In this paper, learning concepts from traditional machine learning and a brief review of research on generalization in GP are presented. The paper also identifies problems with brittleness of solutions produced by GP and suggests a method for promoting robustness/generalization of the solutions in simulating learning behaviors using GP     

Third-order generalization: A new approach to categorizing higher-order generalization

Generalization, in its most basic form, is an artificial neural network's (ANN's) ability to automatically classify data that were not seen during training. This paper presents a framework in which generalization in ANNs is quantified and different types of generalization are viewed as orders. The ordering of generalization is a means of categorizing different behaviours. These orders enable generalization to be evaluated in a detailed and systematic way. The approach used is based on existing definitions which are augmented in this paper. The generalization framework is a hierarchy of categories which directly aligns an ANN's ability to perform table look-up, interpolation, extrapolation, and hyper-extrapolation tasks.

The framework is empirically validated. Validation is undertaken with three different types of regression task: (1) a one-to-one (o–o) task, f(x):x_i→y_j; (2) the second, in its f(x):{x_i,x_i+1, …}→y_j formulation, maps a many-to-one (m–o) task; and (3) the third f(x):x_i→{y_j,y_j+1, …} a one-to-many (o–m) task. The first and second are assigned to feedforward nets, while the third, due to its complexity, is assigned to a recurrent neural net.

Throughout the empirical work, higher-order generalization is validated with reference to the ability of a net to perform symmetrically related or isomorphic functions generated using symmetric transformations (STs) of a net's weights. The transformed weights of a base net (BN) are inherited by a derived net (DN). The inheritance is viewed as the reuse of information. The overall framework is also considered in the light of alignment to neural models; for example, which order (or level) of generalization can be performed by which specific type of neuron model.

The complete framework may not be applicable to all neural models; in fact, some orders may be special cases which apply only to specific neuron models. This is, indeed, shown to be the case. Lower-order generalization is viewed as a general case and is applicable to all neuron models, whereas higher-order generalization is a particular or special case. This paper focuses on initial results; some of the aims have been demonstrated and amplified through the experimental work.     

Explicitly biased generalization

During incremental concept learning from examples, tentative hypotheses are formed and then modified to form new hypotheses. When there is a choice among hypotheses, bias is used to express a preference. Bias may be expressed by the choice of hypothesis language, it may be implemented as an evaluation function for selecting among hypotheses already generated, or it may consist of screening potential hypotheses prior to hypothesis generation. This paper describes the use of the third method. Bias is represented explicitly both as assumptions that reduce the space of potential hypotheses and as procedures for testing these assumptions. There are advantages gained by using explicit assumptions. One advantage is that the assumptions are meta-level hypotheses that are used to generate future, as well as to select between current, inductive hypotheses. By testing these meta-level hypotheses, a system gains the power to anticipate the form of future hypotheses. Furthermore, rigorous testing of these meta-level hypotheses before using them to generate inductive hypotheses avoids consistency checks of the inductive hypotheses. A second advantage of using explicit assumptions is that bias can be tested using a variety of learning methods.     

Costs and benefits of behavioral specialization

In this work, we study behavioral specialization in a swarm of autonomous robots. In the studied swarm, robots have to carry out tasks of different types that appear stochastically in time and space in a given environment. We consider a setting in which a robot working repeatedly on tasks of the same type improves its performance on them due to learning. Robots can exploit learning by adapting their task selection behavior, that is, by selecting with higher probability tasks of the type on which they have improved their performance. This adaptation of behavior is called behavioral specialization. We employ a simple task allocation strategy that allows a swarm of robots to behaviorally specialize. We study the influence of different environmental parameters on the performance of the swarm and show that the swarm can exploit learning successfully. However, there is a trade-off between the benefits and the costs of specialization. We study this trade-off in multiple experiments using different swarm sizes. Our experimental results indicate that spatiality has a major influence on the costs and benefits of specialization.     

Feedback-directed specialization of code

Based on feedback information, a large number of optimizations can be performed by the compiler. This information actually indicates the changing behavior of the applications and can be used to specialize code accordingly.Code specialization is a way to facilitate the compiler to perform optimizations by providing the information regarding variables in the code. It is however difficult to select the variables which maximize the benefit of specialization. Also the overhead of specialization and code size increase are the main issues while specializing code.This paper suggests a novel method for improving the performance using specialization based on feedback information and analysis. The code is iteratively specialized after selecting candidate variables by using a heuristic, followed by generation of optimized templates. These templates require a limited set of instructions to be specialized at runtime and are valid for a large number of values. The overhead of runtime specialization is further minimized through optimal software cache of template clones whose instantiation can be performed at static compile time.The experiments have been performed on Itanium-II(IA-64) and Pentium-IV processors using icc and gcc compilers. A significant improvement in terms of execution speed and reduction of code size has been achieved for SPEC and FFTW benchmarks.     

Inheritance and generalization in intelligent SQL

Inheritance provides a way to model complex real-world relationships in a database management system. This paper examines inheritance in Intelligent SQL, the access language for an intelligent database under development at Ashton-Tate. Intelligent SQL includes specialization and generalization as complementary aspects of inheritance and describes a set inclusion semantics for them. We illustrate how these concepts can be used to present a global view of distributed data, through bottom-up integration of existing, heterogeneous SQL databases.     

Representation and generalization properties of class-entropynetworks

Using conditional class entropy (CCE) as a cost function allows feedforward networks to fully exploit classification-relevant information. CCE-based networks arrange the data space into partitions, which are assigned unambiguous symbols and are labeled by class information. By this labeling mechanism the network can model the empirical data distribution at the local level. Region labeling evolves with the network-training process, which follows a plastic algorithm. The paper proves several theoretical properties about the performance of CCE-based networks, and considers both convergence during training and generalization ability at run-time. In addition, analytical criteria and practical procedures are proposed to enhance the generalization performance of the trained networks. Experiments on artificial and real-world domains confirm the accuracy of this class of networks and witness the validity of the described methods.     

Evolving team behaviors with specialization

This article evaluates Collective Neuro-Evolution (CONE), a cooperative co-evolutionary method for solving collective behavior tasks and increasing task performance via facilitating behavioral specialization in agent teams. Specialization is used as a problem solving mechanism, and its emergence is guided and regulated by CONE. CONE is comparatively evaluated with related methods in a simulated evolutionary robotics pursuit-evasion task. This task required multiple pursuer robots to cooperatively capture evader robots. Results indicate that CONE is appropriate for evolving specialized behaviors. The interaction of specialized behaviors produces behavioral heterogeneity in teams and collective prey capture behaviors that yield significantly higher performances compared to related methods.     

Software specialization via symbolic execution

A technique and an environment-supporting specialization of generalized software components are described. The technique is based on symbolic execution. It allows one to transform a generalized software component into a more specific and more efficient component. Specialization is proposed as a technique that improves software reuse. The idea is that a library of generalized components exists and the environment supports a designer in customizing a generalized component when the need arises for reusing it under more restricted conditions. It is also justified as a reengineering technique that helps optimize a program during maintenance. Specialization is supported by an interactive environment that provides several transformation tools: a symbolic executor/simplifier, an optimizer, and a loop refolder. The conceptual basis for these transformation techniques is described, examples of their application are given, and how they cooperate in a prototype environment for the Ada programming language is outlined     

Aliasing and adversarial robust generalization of CNNs

Many commonly well-performing convolutional neural network models have shown to be susceptible to input data perturbations, indicating a low model robustness. To reveal model weaknesses, adversarial attacks are specifically optimized to generate small, barely perceivable image perturbations that flip the model prediction. Robustness against attacks can be gained by using adversarial examples during training, which in most cases reduces the measurable model attackability. Unfortunately, this technique can lead to robust overfitting, which results in non-robust models. In this paper, we analyze adversarially trained, robust models in the context of a specific network operation, the downsampling layer, and provide evidence that robust models have learned to downsample more accurately and suffer significantly less from downsampling artifacts, aka. aliasing, than baseline models. In the case of robust overfitting, we observe a strong increase in aliasing and propose a novel early stopping approach based on the measurement of aliasing.

     

Learning and generalization with the information bottleneck

The Information Bottleneck is an information theoretic framework that finds concise representations for an ‘input’ random variable that are as relevant as possible for an ‘output’ random variable. This framework has been used successfully in various supervised and unsupervised applications. However, its learning theoretic properties and justification remained unclear as it differs from standard learning models in several crucial aspects, primarily its explicit reliance on the joint input–output distribution. In practice, an empirical plug-in estimate of the underlying distribution has been used, so far without any finite sample performance guarantees. In this paper we present several formal results that address these difficulties. We prove several finite sample bounds, which show that the information bottleneck can provide concise representations with good generalization, based on smaller sample sizes than needed to estimate the underlying distribution. The bounds are non-uniform and adaptive to the complexity of the specific model chosen. Based on these results, we also present a preliminary analysis on the possibility of analyzing the information bottleneck method as a learning algorithm in the familiar performance-complexity tradeoff framework. In addition, we formally describe the connection between the information bottleneck and minimal sufficient statistics.