Analysis of a complexity-based pruning scheme for classification trees

A complexity-based pruning procedure for classification trees is described, and bounds on its finite sample performance are established. The procedure selects a subtree of a (possibly random) initial tree in order to minimize a complexity penalized measure of empirical risk. The complexity assigned to a subtree is proportional to the square root of its size. Two cases are considered. In the first, the growing and pruning data sets are identical, and in the second, they are independent Using the performance bound, the Bayes risk consistency of pruned trees obtained via the procedure is established when the sequence of initial trees satisfies suitable geometric and structural constraints. The pruning method and its analysis are motivated by work on adaptive model selection using complexity regularization.     

C-fuzzy decision trees

This paper introduces a concept and design of decision trees based on information granules - multivariable entities characterized by high homogeneity (low variability). As such granules are developed via fuzzy clustering and play a pivotal role in the growth of the decision trees, they will be referred to as C-fuzzy decision trees. In contrast with "standard" decision trees in which one variable (feature) is considered at a time, this form of decision trees involves all variables that are considered at each node of the tree. Obviously, this gives rise to a completely new geometry of the partition of the feature space that is quite different from the guillotine cuts implemented by standard decision trees. The growth of the C-decision tree is realized by expanding a node of tree characterized by the highest variability of the information granule residing there. This paper shows how the tree is grown depending on some additional node expansion criteria such as cardinality (number of data) at a given node and a level of structural dependencies (structurability) of data existing there. A series of experiments is reported using both synthetic and machine learning data sets. The results are compared with those produced by the "standard" version of the decision tree (namely, C4.5).     

基于流形学习的新高光谱图像降维算法

提出了一种新的基于图像块距离的邻域选择方法,并将其应用于流形学习中,得到一类新的高光谱图像非线性降维算法。该类算法利用高光谱图像物理特性,结合图像的光谱信息和空间信息,在最大限度减小图像信息冗余的基础之上,很好地保持了原始数据集的特性。与其它高光谱图像的降维算法相比,改进的流形学习算法不仅考虑到高光谱图像本身的空间关系,而且利用图像块距离更好地保持了数据点之间的局部特性,从而有效地去除原始数据集光谱维和空间维的冗余信息。实际高光谱数据的实验结果表明,所提出的算法在应用于高光谱图像分类时,与其它方法相比具有更高的分类精度。     

Build better diagnostic decision trees

Enhancing the ability to perform diagnostics on a system that has failed can significantly impact maintenance and repair costs. A good diagnostic tool enables a user to analyze a failed system and identify the failed components. While the field of diagnostics is not a modern one, the way in which system diagnostics are performed is continuously changing. The automatic diagnosis based on reliability analysis (ADORA) methodology utilizes reliability information developed during the design phase to build a diagnostic map. Previous work on ADORA demonstrated how a diagnostics procedure can be performed on a system that has been analyzed using a static reliability model, particularly a fault tree (Assaf and Dugan, 2003). In this article, we extend the ADORA methodology to utilize reliability analysis of dynamic fault trees (DFTs), which are reliability models that capture sequences and combinations of component failures that induce system failure. DFTs are particularly well suited for analyzing computer-based systems.As an example the common rail fuel injection system is discussed.     

Multiple-instance learning algorithms for computer-aided detection

Many computer-aided diagnosis (CAD) problems can be best modelled as a multiple-instance learning (MIL) problem with unbalanced data, i.e., the training data typically consists of a few positive bags, and a very large number of negative instances. Existing MIL algorithms are much too computationally expensive for these datasets. We describe CH, a framework for learning a Convex Hull representation of multiple instances that is significantly faster than existing MIL algorithms. Our CH framework applies to any standard hyperplane-based learning algorithm, and for some algorithms, is guaranteed to find the global optimal solution. Experimental studies on two different CAD applications further demonstrate that the proposed algorithm significantly improves diagnostic accuracy when compared to both MIL and traditional classifiers. Although not designed for standard MIL problems (which have both positive and negative bags and relatively balanced datasets), comparisons against other MIL methods on benchmark problems also indicate that the proposed method is competitive with the state-of-the-art.     

The dispersion characteristics of oblique coordinate beampropagation algorithms

The use of an oblique coordinate system with the finite difference beam propagation method has previously been demonstrated to offer significant computational advantages over using rectangular coordinates for a wide range of practical optical structures. The effects of finite mesh resolution, step size, and order of the algorithm in terms of numerical dispersion and dissipation are here investigated and quantified for the first time     

Using decision trees for the minimization of multiple-valued functions

This paper presents a simple procedure for the approximate minimization of multiple-valued functions using multiple-valued decision trees. The proposed procedure is compared with a near-absolute procedure, using for the test four-valued functions of four variables. The results show a great advantage for our procedure with respect to the CPU time needed.     

Minimax-optimal classification with dyadic decision trees

Decision trees are among the most popular types of classifiers, with interpretability and ease of implementation being among their chief attributes. Despite the widespread use of decision trees, theoretical analysis of their performance has only begun to emerge in recent years. In this paper, it is shown that a new family of decision trees, dyadic decision trees (DDTs), attain nearly optimal (in a minimax sense) rates of convergence for a broad range of classification problems. Furthermore, DDTs are surprisingly adaptive in three important respects: they automatically 1) adapt to favorable conditions near the Bayes decision boundary; 2) focus on data distributed on lower dimensional manifolds; and 3) reject irrelevant features. DDTs are constructed by penalized empirical risk minimization using a new data-dependent penalty and may be computed exactly with computational complexity that is nearly linear in the training sample size. DDTs comprise the first classifiers known to achieve nearly optimal rates for the diverse class of distributions studied here while also being practical and implementable. This is also the first study (of which we are aware) to consider rates for adaptation to intrinsic data dimension and relevant features.     

Intelligent learning algorithms for adaptive digital filters

It is wellknown that gradient search fails in adaptive IIR filters, since their mean-square error surfaces may be multi-modal. In the letter a new approach based on learning algorithms is shown to be capable of performing global optimisation. The new algorithms are suitable for both adaptive FIR and IIR filters.<>     

Competitive learning algorithms for robust vector quantization

The efficient representation and encoding of signals with limited resources, e.g., finite storage capacity and restricted transmission bandwidth, is a fundamental problem in technical as well as biological information processing systems. Typically, under realistic circumstances, the encoding and communication of messages has to deal with different sources of noise and disturbances. We propose a unifying approach to data compression by robust vector quantization, which explicitly deals with channel noise, bandwidth limitations, and random elimination of prototypes. The resulting algorithm is able to limit the detrimental effect of noise in a very general communication scenario. In addition, the presented model allows us to derive a novel competitive neural networks algorithm, which covers topology preserving feature maps, the so-called neural-gas algorithm, and the maximum entropy soft-max rule as special cases. Furthermore, continuation methods based on these noise models improve the codebook design by reducing the sensitivity to local minima. We show an exemplary application of the novel robust vector quantization algorithm to image compression for a teleconferencing system     

CTU depth decision algorithms for HEVC: A survey

High Efficiency Video Coding (HEVC) surpasses its predecessors in encoding efficiency by introducing new coding tools at the cost of an increased encoding time-complexity. The Coding Tree Unit (CTU) is the main building block used in HEVC. In the HEVC standard, frames are divided into CTUs with the predetermined size of up to 64 × 64 pixels. Each CTU is then divided recursively into a number of equally sized square areas, known as Coding Units (CUs). Although this diversity of frame partitioning increases encoding efficiency, it also causes an increase in the time complexity due to the increased number of ways to find the optimal partitioning. To address this complexity, numerous algorithms have been proposed to eliminate unnecessary searches during partitioning CTUs by exploiting the correlation in the video. In this paper, existing CTU depth decision algorithms for HEVC are surveyed. These algorithms are categorized into two groups, namely statistics and machine learning approaches. Statistics approaches are further subdivided into neighboring and inherent approaches. Neighboring approaches exploit the similarity between adjacent CTUs to limit the depth range of the current CTU, while inherent approaches use only the available information within the current CTU. Machine learning approaches try to extract and exploit similarities implicitly. Traditional methods like support vector machines or random forests use manually selected features, while recently proposed deep learning methods extract features during training. Finally, this paper discusses extending these methods to more recent video coding formats such as Versatile Video Coding (VVC) and AOMedia Video 1(AV1).     

Asymptotically optimal soft decision decoding algorithms for Hamming codes

Asymptotically optimal soft decision decoding algorithms for d = 3 and d = 4 Hamming codes are given and analysed. Only error sequences with probability exponent larger than that of maximum-likelihood decoding are corrected. Upper bounds on the block error probability for the Gaussian channel are given.     

Stochastic correlative learning algorithms

This paper addresses stochastic correlative learning as the basis for a broadly defined class of statistical learning algorithms known collectively as the algorithm of pattern extraction (ALOPEX) family. Starting with the neurobiologically motivated Hebb's rule, the two conventional forms of the ALOPEX algorithm are derived, followed by a modified variant designed to improve the convergence speed. We next describe two more elaborate versions of the ALOPEX algorithm, which incorporate particle filtering that exemplifies a form of Monte Carlo simulation, to exchange computational complexity for an improved convergence and tracking behavior. In support of the different forms of the ALOPEX algorithm developed herein, we present three different experiments using synthetic and real-life data on binocular fusion of stereo images, on-line prediction, and system identification.     

Rapid backpropagation learning algorithms

One of the major drawbacks of the backpropagation algorithm is its slow rate of convergence. Researchers have tried several different approaches to speed up the convergence of backpropagation learning. In this paper, we present those rapid learning methods as three categories, and implement the representative methods of each category: (1) for the numerical method based approach, the Aitken's ² process, (2) for the heuristics based approach, the dynamic adaptation of learning rate, and (3) for the learning strategy based approach, the selective presentation of learning samples. Based on these implementations, the performance is evaluated with experiments and the merits and demerits are briefly discussed.This work was supported in part by a grant from the Korea Science and Engineering Foundation (KOSEF) and the Center for Artificial Intelligence Research (CAIR), the Engineering Research Center (ERC) of Excellence Program.     

Iterative learning algorithms for linear Gaussian observation models

In this paper, we consider a signal/parameter estimation problem that is based on a linear model structure and a given setting of statistical models with unknown hyperparameters. We consider several combinations of Gaussian and Laplacian models. We develop iterative algorithms based on two typical machine learning methods - the evidence-based method and the integration-based method - to deal with the hyperparameters. We have applied the proposed algorithms to adaptive prediction and wavelet denoising. In linear prediction, we show that the proposed algorithms are efficient tools for tackling a difficult problem of adapting simultaneously the order and the coefficients of the predictor. In wavelet denoising, we show that by using the proposed algorithms, the noisy wavelet coefficients are subject to shrinkage and thresholding.     

Learn decision trees with deep visual primitives

In this paper, we strive to propose a self-interpretable framework, termed PrimitiveTree, that incorporates deep visual primitives condensed from deep features with a conventional decision tree, bridging the gap between deep features extracted from deep neural networks (DNNs) and trees’ transparent decision-making processes. Specifically, we utilize a codebook, which embeds the continuous deep features into a finite discrete space (deep visual primitives) to distill the most common semantic information. The decision tree adopts the spatial location information and the mapped primitives to present the decision-making process of the deep features in a tree hierarchy. Moreover, the trained interpretable PrimitiveTree can inversely explain the constituents of the deep features, highlighting the most critical and semantic-rich image patches attributing to the final predictions of the given DNN. Extensive experiments and visualization results validate the effectiveness and interpretability of our method.     

Multi-criteria decision algorithms for efficient content delivery in content networks

Today’s Internet is prominently used for content distribution. Various platforms such as content delivery networks (CDNs) have become an integral part of the digital content ecosystem. Most recently, the information-centric networking (ICN) paradigm proposes the adoption of native content naming for secure and efficient content delivery. This further enhances the flexibility of content access where a content request can be served by any source within the Internet. In this paper, we propose and evaluate a multi-criteria decision algorithm for efficient content delivery applicable for content networks in general (among others, CDN and ICN). Our algorithm computes the best available source and path for serving content requests taking into account information about content transfer requirements, location of the consumer, location of available content servers, content server load and content delivery paths between content servers and consumer. The proposed algorithm exploits two closely related processes. The first level discovers multiple content delivery paths and gathers their respective transfer characteristics. This discovery process is based on long-term network measurements and performed offline. The second process is invoked for each content request to find the best combined content server and delivery path. The cooperation between both levels allows our algorithm to increase the number of satisfied content requests thanks to efficient utilisation of network and server resources. The proposed decision algorithm was evaluated by simulation using Internet scale network model. The results confirm the effectiveness gain of content network architectures that introduce network awareness. Moreover, the simulation process allows for a comparison between different routing algorithms and, especially, between single and multipath routing algorithms.     

Markov decision algorithms for dynamic routing [telephone networks]

The nature of the traffic-routing problem is described, and early studies of state-dependent routing are noted. A state-dependent scheme seeks to route each call so as to minimize the risk of blocking future calls, and thus responds to the current state of the network on the basis of certain assumptions about future traffic demands. State-dependent routing is considered as a Markov decision process. How the relative costs can be determined for the case of direct routing is shown     

New models and algorithms for future networks

In future networks, transmission and switching capacity will dominate processing capacity. The authors investigate the way in which distributed algorithms should be changed in order to operate efficiently in this new environment. They introduce a class of new models for distributed algorithms which make explicit the difference between switching and processing. Based on these new models they define new message and time complexity measures which, they believe, capture the costs in many high-speed networks more accurately then traditional measures. In order to explore the consequences of the new models, they examine three problems in distributed computation. For the problem of maintaining network topology they devise a broadcast algorithm which takes O(n) messages and O(log n) time for a single broadcast in the new measure. For the problem of leader election they present a simple algorithm that uses O(n) messages and O(n) time. The third problem, distributed computation of a “globally sensitive” function, demonstrates some important features and tradeoffs in the new models and emphasizes and differences with the traditional network model. The results of the present paper influenced later research, as well as the design of IBM Networking Broadband Services (NBBS)