Neighborhood Guided Smoothed Emphasis for Real Adaboost Ensembles

Neural Processing Letters - The exceptional capabilities of the much celebrated Real Adaboost ensembles for solving decision and classification problems are universally recognized. These...     

Sample selection via clustering to construct support vector-likeclassifiers

Explores the possibility of constructing RBF classifiers which, somewhat like support vector machines, use a reduced number of samples as centroids, by means of selecting samples in a direct way. Because sample selection is viewed as a hard computational problem, this selection is done after a previous vector quantization: this way obtains also other similar machines using centroids selected from those that are learned in a supervised manner. Several forms of designing these machines are considered, in particular with respect to sample selection; as well as some different criteria to train them. Simulation results for well-known classification problems show very good performance of the corresponding designs, improving that of support vector machines and reducing substantially their number of units. This shows that our interest in selecting samples (or centroids) in an efficient manner is justified. Many new research avenues appear from these experiments and discussions, as suggested in our conclusions.     

Advantages of Unbiased Support Vector Classifiers for Data Mining Applications

Many learning algorithms have been used for data mining applications, including Support Vector Classifiers (SVC), which have shown improved capabilities with respect to other approaches, since they provide a natural mechanism for implementing Structural Risk Minimization (SRM), obtaining machines with good generalization properties. SVC leads to the optimal hyperplane (maximal margin) criterion for separable datasets but, in the nonseparable case, the SVC minimizes the L ₁ norm of the training errors plus a regularizing term, to control the machine complexity. The L ₁ norm is chosen because it allows to solve the minimization with a Quadratic Programming (QP) scheme, as in the separable case. But the L ₁ norm is not truly an “error counting” term as the Empirical Risk Minimization (ERM) inductive principle indicates, leading therefore to a biased solution. This effect is specially severe in low complexity machines, such as linear classifiers or machines with few nodes (neurons, kernels, basis functions). Since one of the main goals in data mining is that of explanation, these reduced architectures are of great interest because they represent the origins of other techniques such as input selection or rule extraction. Training SVMs as accurately as possible in these situations (i.e., without this bias) is, therefore, an interesting goal. We propose here an unbiased implementation of SVC by introducing a more appropriate “error counting” term. This way, the number of classification errors is truly minimized, while the maximal margin solution is obtained in the separable case. QP can no longer be used for solving the new minimization problem, and we apply instead an iterated Weighted Least Squares (WLS) procedure. This modification in the cost function of the Support Vector Machine to solve ERM was not possible up to date given the Quadratic or Linear Programming techniques commonly used, but it is now possible using the iterated WLS formulation. Computer experiments show that the proposed method is superior to the classical approach in the sense that it truly solves the ERM problem.     

A mixed neural-genetic algorithm for the broadcast scheduling problem

The broadcast scheduling problem (BSP) arises in frame design for packet radio networks (PRNs). The frame structure determines the main communication parameters: communication delay and throughput. The BSP is a combinatorial optimization problem which is known to be NP-hard. To solve it, we propose an algorithm with two main steps which naturally arise from the problem structure: the first one tackles the hardest contraints and the second one carries out the throughput optimization. This algorithm combines a Hopfield neural network for the constraints satisfaction and a genetic algorithm for achieving a maximal throughput. The algorithm performance is compared with that of existing algorithms in several benchmark cases; in all of them, our algorithm finds the optimum frame length and outperforms previous algorithms in the resulting throughput.     

Improving CELP coders by backward adaptive non-linear prediction

Non-linear prediction is a natural way to increase the quality of speech coders. In particular, low-delay CELP-type coders can incorporate this improvement because the predictor adaptation is backward. Consequently, there is the possibility of using neural networks as predictors, since their weights (usually a larger number than required in the linear approach) do not have to be transmitted. We apply a radial basis function (RBF) network for this purpose since it computes a regularized solution to the prediction problem. As a result, the stability of the non-linear autoregressive synthesis system can be guaranteed. Investigations of how to combine non-linear predictors with linear predictors indicate that a cascade of an RBF network and a linear filter is a suitable selection since it provides good results and its application to analysis-by-synthesis coders results in large computational advantages with respect to the parallel configuration. This hybrid predictor has been tested for a low-delay code-excited predictive coder, providing an average improvement of 0·4 dB with respect to a CELP coder. Additionally, subjective listening tests give the proposed coder a slight preference over the CELP coder. These results are encouraging because we consider that the proposed coder can be implemented in real time after some improvements, which are detailed as the subject of further work. © 1997 John Wiley & Sons. Ltd.     

Mean-square performance of a convex combination of two adaptive filters

Combination approaches provide an interesting way to improve adaptive filter performance. In this paper, we study the mean-square performance of a convex combination of two transversal filters. The individual filters are independently adapted using their own error signals, while the combination is adapted by means of a stochastic gradient algorithm in order to minimize the error of the overall structure. General expressions are derived that show that the method is universal with respect to the component filters, i.e., in steady-state, it performs at least as well as the best component filter. Furthermore, when the correlation between the a priori errors of the components is low enough, their combination is able to outperform both of them. Using energy conservation relations, we specialize the results to a combination of least mean-square filters operating both in stationary and in nonstationary scenarios. We also show how the universality of the scheme can be exploited to design filters with improved tracking performance.     

Efficient block training of multilayer perceptrons

The attractive possibility of applying layerwise block training algorithms to multilayer perceptrons MLP, which offers initial advantages in computational effort, is refined in this article by means of introducing a sensitivity correction factor in the formulation. This results in a clear performance advantage, which we verify in several applications. The reasons for this advantage are discussed and related to implicit relations with second-order techniques, natural gradient formulations through Fisher's information matrix, and sample selection. Extensions to recurrent networks and other research lines are suggested at the close of the article.     

New algorithms for improved adaptive convex combination of LMS transversal filters

Among all adaptive filtering algorithms, Widrow and Hoff's least mean square (LMS) has probably become the most popular because of its robustness, good tracking properties and simplicity. A drawback of LMS is that the step size implies a compromise between speed of convergence and final misadjustment. To combine different speed LMS filters serves to alleviate this compromise, as it was demonstrated by our studies on a two filter combination that we call combination of LMS filters (CLMS). Here, we extend this scheme in two directions. First, we propose a generalization to combine multiple LMS filters with different steps that provides the combination with better tracking capabilities. Second, we use a different mixing parameter for each weight of the filter in order to make independent their adaption speeds. Some simulation examples in plant identification and noise cancellation applications show the validity of the new schemes when compared to the CLMS filter and to other previous variable step approaches.     

Generalizing CMAC architecture and training.

The cerebellar model articulation controller (CMAC) is a simple and fast neural-network based on local approximations. However, its rigid structure reduces its accuracy of approximation and speed of convergence with heterogeneous inputs. In this paper, we propose a generalized CMAC (GCMAC) network that considers different degrees of generalization for each input. Its representation abilities are analyzed, and a set of local relationships that the output function must satisfy are derived. An adaptive growing method of the network is also presented. The validity of our approach and methods are shown by some simulated examples.     

Weighted least squares training of support vector classifiersleading to compact and adaptive schemes

An iterative block training method for support vector classifiers (SVCs) based on weighted least squares (WLS) optimization is presented. The algorithm, which minimizes structural risk in the primal space, is applicable to both linear and nonlinear machines. In some nonlinear cases, it is necessary to previously find a projection of data onto an intermediate-dimensional space by means of either principal component analysis or clustering techniques. The proposed approach yields very compact machines, the complexity reduction with respect to the SVC solution is especially notable in problems with highly overlapped classes. Furthermore, the formulation in terms of WLS minimization makes the development of adaptive SVCs straightforward, opening up new fields of application for this type of model, mainly online processing of large amounts of (static/stationary) data, as well as online update in nonstationary scenarios (adaptive solutions). The performance of this new type of algorithm is analyzed by means of several simulations.