A new clustering technique for function approximation

To date, clustering techniques have always been oriented to solve classification and pattern recognition problems. However, some authors have applied them unchanged to construct initial models for function approximators. Nevertheless, classification and function approximation problems present quite different objectives. Therefore it is necessary to design new clustering algorithms specialized in the problem of function approximation. This paper presents a new clustering technique, specially designed for function. approximation problems, which improves the performance of the approximator system obtained, compared with other models derived from traditional classification oriented clustering algorithms and input-output clustering techniques.     

A novel kernel-based framework for facial-image hallucination

In this paper, we present a kernel-based eigentransformation framework to hallucinate the high-resolution (HR) facial image of a low-resolution (LR) input. The eigentransformation method is a linear subspace approach, which represents an image as a linear combination of training samples. Consequently, those novel facial appearances not included in the training samples cannot be super-resolved properly. To solve this problem, we devise a kernel-based extension of the eigentransformation method, which takes higher-order statistics of the image data into account. To generate HR face images with higher fidelity, the HR face image reconstructed using this kernel-based eigentransformation method is treated as an initial estimation of the target HR face. The corresponding high-frequency components of this estimation are extracted to form a prior in the maximum a posteriori (MAP) formulation of the SR problem so as to derive the final reconstruction result. We have evaluated our proposed method using different kernels and configurations, and have compared these performances with some current SR algorithms. Experimental results show that our kernel-based framework, along with a proper kernel, can produce good HR facial images in terms of both visual quality and reconstruction errors.     

A kernel-based framework for image collection exploration

While search engines have been a successful tool to search text information, image search systems still face challenges. The keyword-based query paradigm used to search in image collection systems, which has been successful in text retrieval, may not be useful in scenarios where the user does not have the precise way to express a visual query. Image collection exploration is a new paradigm where users interact with the image collection to discover useful and relevant pictures. This paper proposes a framework for the construction of an image collection exploration system based on kernel methods, which offers a mathematically strong basis to address each stage of an image collection exploration system: image representation, summarization, visualization and interaction. In particular, our approach emphasizes a semantic representation of images using kernel functions, which can be seamlessly harnessed across all system components. Experiments were conducted with real users to verify the effectiveness and efficiency of the proposed strategy.     

A simple approximation for the Erlang loss function

In this paper we present a simple non-iterative computational procedure for approximating the Erlang loss function B(N, ). It is applicable to the practical range 10⁻⁵<B(N, )<10⁻¹ and gives results that are within 10% of the exact values. The formula can be computed on a pocket calculator in constant time and could be used to approximately compute B(N, ) for systems of practically any size.     

A new kernel-based approach for linear system identification

This paper describes a new kernel-based approach for linear system identification of stable systems. We model the impulse response as the realization of a Gaussian process whose statistics, differently from previously adopted priors, include information not only on smoothness but also on BIBO-stability. The associated autocovariance defines what we call a stable spline kernel. The corresponding minimum variance estimate belongs to a reproducing kernel Hilbert space which is spectrally characterized. Compared to parametric identification techniques, the impulse response of the system is searched for within an infinite-dimensional space, dense in the space of continuous functions. Overparametrization is avoided by tuning few hyperparameters via marginal likelihood maximization. The proposed approach may prove particularly useful in the context of robust identification in order to obtain reduced order models by exploiting a two-step procedure that projects the nonparametric estimate onto the space of nominal models. The continuous-time derivation immediately extends to the discrete-time case. On several continuous- and discrete-time benchmarks taken from the literature the proposed approach compares very favorably with the existing parametric and nonparametric techniques.     

A novel algebraic method for kernel-based object tracking

In the present paper, a new tracking method based on kernel tracking is proposed. The proposed method employs a novel algebraic algorithm to get the kernel movement. In contrast to the mean-shift method which uses a weighted kernel to reduce the effect of the background, the algebraic algorithm of the proposed method allows dividing the candidate area into two parts in order to identify the object and background regions. To detect the object and background regions, we propose measuring the similarity of weighted histogram for each part. The experiments show the superiority of the proposed method for the removal of the background. The effect of noise and background clutter is reduced by segmentation of the object which produces the narrow histogram. In conclusion, the ability of the proposed method for tracking in crowded and cluttered scenes is demonstrated.     

A kernel-based parametric method for conditional density estimation

A conditional density function, which describes the relationship between response and explanatory variables, plays an important role in many analysis problems. In this paper, we propose a new kernel-based parametric method to estimate conditional density. An exponential function is employed to approximate the unknown density, and its parameters are computed from the given explanatory variable via a nonlinear mapping using kernel principal component analysis (KPCA). We develop a new kernel function, which is a variant to polynomial kernels, to be used in KPCA. The proposed method is compared with the Nadaraya-Watson estimator through numerical simulation and practical data. Experimental results show that the proposed method outperforms the Nadaraya-Watson estimator in terms of revised mean integrated squared error (RMISE). Therefore, the proposed method is an effective method for estimating the conditional densities.     

A tree-structured adaptive network for function approximation inhigh-dimensional spaces

Nonlinear function approximation is often solved by finding a set of coefficients for a finite number of fixed nonlinear basis functions. However, if the input data are drawn from a high-dimensional space, the number of required basis functions grows exponentially with dimension, leading many to suggest the use of adaptive nonlinear basis functions whose parameters can be determined by iterative methods. The author proposes a technique based on the idea that for most of the data, only a few dimensions of the input may be necessary to compute the desired output function. Additional input dimensions are incorporated only where needed. The learning procedure grows a tree whose structure depends upon the input data and the function to be approximated. This technique has a fast learning algorithm with no local minima once the network shape is fixed, and it can be used to reduce the number of required measurements in situations where there is a cost associated with sensing. Three examples are given: controlling the dynamics of a simulated planar two-joint robot arm, predicting the dynamics of the chaotic Mackey-Glass equation, and predicting pixel values in real images from pixel values above and to the left.     

A constructive method for multivariate function approximation bymultilayer perceptrons

Mathematical theorems establish the existence of feedforward multilayered neural networks, based on neurons with sigmoidal transfer functions, that approximate arbitrarily well any continuous multivariate function. However, these theorems do not provide any hint on how to find the network parameters in practice. It is shown how to construct a perceptron with two hidden layers for multivariate function approximation. Such a network can perform function approximation in the same manner as networks based on Gaussian potential functions, by linear combination of local functions.     

Measure fields for function approximation

The computation of a piecewise smooth function that approximates a finite set of data points may be decomposed into two decoupled tasks: 1) the computation of the locally smooth models, and hence, the segmentation of the data into classes that consist of the sets of points best approximated by each model; 2) the computation of the normalized discriminant functions for each induced class (which maybe interpreted as relative probabilities). The approximating function may then be computed as the optimal estimator with respect to this measure field. For the first step, we propose a scheme that involves both robust regression and spatial localization using Gaussian windows. The discriminant functions are obtained fitting Gaussian mixture models for the data distribution inside each class. We give an efficient procedure for effecting both computations and for the determination of the optimal number of components. Examples of the application of this scheme to image filtering, surface reconstruction and time series prediction are presented.     

一种新的核化SVM多层分类方法

利用核化思想提出了一种新的SVM多层分类算法。该算法的基本思路是：先利用Mercer核,将输入空间非线性可分的训练样本映射到高维特征空间Hilbert中,使之线性可分,然后采用最小超球体类包含作为层次分类的依据来生成二叉决策树,从而实现在高维空间中的多类分类。实验表明,采用该算法进行多类分类,可以有效地解决输入空间非线性可分问题,并可在一定程度上提高分类器的分类精度。     

Incomplete series expansion for function approximation

We present an incomplete series expansion (ISE) as a basis for function approximation. The ISE is expressed in terms of an approximate Hessian matrix, which may contain second, third, and even higher order “main” or diagonal terms, but which excludes “interaction” or off-diagonal terms. From the ISE, a family of approximation functions may be derived. The approximation functions may be based on an arbitrary number of previously sampled points, and any of the function and gradient values at suitable previously sampled points may be enforced when deriving the approximation functions. When function values only are enforced, the storage requirements are minimal. However, irrespective of the conditions enforced, the approximate Hessian matrix is a sparse diagonal matrix. In addition, the resultant approximations are separable. Hence, the proposed approximation functions are very well-suited for use in gradient-based sequential approximate optimization requiring computationally expensive simulations; a typical example is structural design problems with many design variables and constraints. We derived a wide selection of approximations from the family of ISE approximating functions; these include approximations based on the substitution of reciprocal and exponential intervening variables. A comparison with popular approximating functions previously proposed illustrates the accuracy and flexibility of the new family of approximation functions. In fact, a number of popular approximating functions previously proposed for structural optimization applications derive from our ISE. Based on the similarly named paper presented at the Sixth World Congress on Structural and Multidisciplinary Optimization, Rio de Janeiro, Brazil, May 2005     

A fast kernel-based nonlinear discriminant analysis for multi-class problems

Nonlinear discriminant analysis may be transformed into the form of kernel-based discriminant analysis. Thus, the corresponding discriminant direction can be solved by linear equations. From the view of feature space, the nonlinear discriminant analysis is still a linear method, and it is provable that in feature space the method is equivalent to Fisher discriminant analysis. We consider that one linear combination of parts of training samples, called “significant nodes”, can replace the total training samples to express the corresponding discriminant vector in feature space to some extent. In this paper, an efficient algorithm is proposed to determine “significant nodes” one by one. The principle of determining “significant nodes” is simple and reasonable, and the consequent algorithm can be carried out with acceptable computation cost. Depending on the kernel functions between test samples and all “significant nodes”, classification can be implemented. The proposed method is called fast kernel-based nonlinear method (FKNM). It is noticeable that the number of “significant nodes” may be much smaller than that of the total training samples. As a result, for two-class classification problems, the FKNM will be much more efficient than the naive kernel-based nonlinear method (NKNM). The FKNM can be also applied to multi-class via two approaches: one-against-the-rest and one-against-one. Although there is a view that one-against-one is superior to one-against-the-rest in classification efficiency, it seems that for the FKNM one-against-the-rest is more efficient than one-against-one. Experiments on benchmark and real datasets illustrate that, for two-class and multi-class classifications, the FKNM is effective, feasible and much efficient.     

A kernel-based trend pattern tracking system for portfolio optimization

We propose a novel kernel-based trend pattern tracking (KTPT) system for portfolio optimization. It includes a three-state price prediction scheme, which extracts both of the following and reverting patterns from the asset price trend to make future price predictions. Moreover, KTPT is equipped with a novel kernel-based tracking system to optimize the portfolio, so as to capture a potential growth of the asset price effectively. The kernel measures the similarity between the current portfolio and the predicted price relative to control the influence of each asset when optimizing the portfolio, which is different from some previous kernels that measure the probability of occurrence of a price relative. Extensive experiments on 5 benchmark datasets from real-world stock markets with various assets in different time periods indicate that KTPT outperforms other state-of-the-art strategies in cumulative wealth and other risk-adjusted metrics, showing its effectiveness in portfolio optimization.     

A note on parametric spline function approximation

The application of a parametric spline function which depends on a parameter p > 0 to two linear partial differential equations is discussed. For p = 0, the parametric spline reduces to the ordinary cubic spline.     

A fast approximation to the sine function

A function approximating the sine function is given which was designed for high speed when coded into a computer routine. The domain is divided into a small number of intervals and in each of these a straight line approximation is used. The slopes of the straight line segments have denominators which are powers of 2, and so an implementation needs no floating point multiplication or division operations. With [0, 1/2£] divided into only four intervals, an absolute error of about 0.013 is achieved by a routine taking just over a third of the time used by a more conventional one. Such accuracy is adequate for certain graphics applications, especially moving displays.     

A kernel-based method for pattern extraction in random process signals

In many applications, one is interested to detect certain patterns in random process signals. We consider a class of random process signals which contain sub-similarities at random positions representing the texture of an object. Those repetitive parts may occur in speech, musical pieces and sonar signals. We suggest a warped time-resolved spectrum kernel for extracting the subsequence similarity in time series in general, and as an example in biosonar signals. Having a set of those kernels for similarity extraction in different size of subsequences, we propose a new method to find an optimal linear combination of those kernels. We formulate the optimal kernel selection via maximizing the kernel Fisher discriminant (KFD) criterion and use Mesh Adaptive Direct Search (MADS) method to solve the optimization problem. Our method is used for biosonar landmark classification with promising results.     

面向主动学习的模糊核聚类采样算法

针对主动学习中构造初始分类器难以选取代表性样本的问题,提出一种模糊核聚类采样算法。该算法首先通过聚类分析技术将样本集划分,然后分别在类簇中心和类簇边界区域选取样本进行标注,最后依此构造初始分类器。在该算法中,通过高斯核函数把原始样本空间中的点非线性变换到高维特征空间,以达到线性可聚的目的,并引入了一种基于局部密度的初始聚类中心选择方法,从而改善聚类效果。为了提高采样质量,结合划分后各类簇的样本个数设计了一种采样比例分配策略。同时,在采样结束阶段设计了一种后补采样策略,以确保采样个数达标。实验结果分析表明,所提算法可以有效地减少构造初始分类器所需的人工标注负担,并取得较高的分类正确率。     

A kernel-based centroid classifier using hypothesis margin

The centroid-based classifier is both effective and efficient for document classification. However, it suffers from over-fitting and linear inseparability problems caused by its fundamental assumptions. To address these problems, we propose a kernel-based hypothesis margin centroid classifier (KHCC). First, KHCC optimises the class centroids via minimising hypothesis margin under structural risk minimisation principle; second, KHCC uses the kernel method to relieve the problem of linear inseparability in the original feature space. Given the radial basis function, we further discuss a guideline for tuning the value of its parameter. The experimental results on four well-known data-sets indicate that our KHCC algorithm outperforms the state-of-the-art algorithms, especially for the unbalanced data-set.     

Synthetic minority oversampling for function approximation problems

Imbalanced data sets are a common occurrence in important machine learning problems. Research in improving learning under imbalanced conditions has largely focused on classification problems (ie, problems with a categorical dependent variable). However, imbalanced data also occur in function approximation, and far less attention has been paid to this case. We present a novel stratification approach for imbalanced function approximation problems. Our solution extends the SMOTE oversampling preprocessing technique to continuous-valued dependent variables by identifying regions of the feature space with a low density of examples and high variance in the dependent variable. Synthetic examples are then generated between nearest neighbors in these regions. In an empirical validation, our approach reduces the normalized mean-squared prediction error in 18 out of 21 benchmark data sets, and compares favorably with state-of-the-art approaches.