Feature selection in MLPs and SVMs based on maximum output information

This paper presents feature selection algorithms for multilayer perceptrons (MLPs) and multiclass support vector machines (SVMs), using mutual information between class labels and classifier outputs, as an objective function. This objective function involves inexpensive computation of information measures only on discrete variables; provides immunity to prior class probabilities; and brackets the probability of error of the classifier. The maximum output information (MOI) algorithms employ this function for feature subset selection by greedy elimination and directed search. The output of the MOI algorithms is a feature subset of user-defined size and an associated trained classifier (MLP/SVM). These algorithms compare favorably with a number of other methods in terms of performance on various artificial and real-world data sets.     

F&H filter: a novel ultra-low power discrete time filter

A new technique for designing filters with long time constants in the discrete time domain is presented. The F&H (filter and hold) methodology halts the state of a continuous time filter every T seconds resulting in a filter implementation with time constants that can be controlled in three distinct ways: by the sampling period T, the duty cycle k=τ/T or the time constant of the continuous time filter prototype. The final filter can be constructed from a typical Gm-C technique with very low power consumption     

Local dynamic modeling with self-organizing maps and applicationsto nonlinear system identification and control

The technique of local linear models is appealing for modeling complex time series due to the weak assumptions required and its intrinsic simplicity. Here, instead of deriving the local models from the data, we propose to estimate them directly from the weights of a self-organizing map (SOM), which functions as a dynamic preserving model of the dynamics. We introduce one modification to the Kohonen learning to ensure good representation of the dynamics and use weighted least squares to ensure continuity among the local models. The proposed scheme is tested using synthetic chaotic time series and real-world data. The practicality of the method is illustrated in the identification and control of the NASA Langley wind tunnel during aerodynamic tests of model aircraft. Modeling the dynamics with an SOM lends to a predictive multiple model control strategy. Comparison of the new controller against the existing controller in test runs shows the superiority of our method     

The past, present, and future of neural networks for signalprocessing

The article provides a review of the fundamental of neural networks and reports recent progress. Topics covered include dynamic modeling, model-based neural networks, statistical learning, eigenstructure-based processing, active learning, and generalization capability. Current and potential applications of neural networks are also described in detail. Those applications include optical character recognition, speech recognition and synthesis, automobile and aircraft control, image analysis and neural vision, and several medical applications. Essentially, neural networks have become a very effective tool in signal processing, particularly in various recognition tasks     

Design and Implementation of Linear Phase FIR Filters for Biological Signal Processing

This paper presents a new method for designing digital linear phase, finite impulse response filters with loose frequency response characteristics, but with good time resolution as is required in biological signal conditioning. The design is very simple and has been used with success in the microcomputer implementation of filters for the automated processing of electroencephalographic (EEG) data. Examples and a discussion of possible filter implementations are included.     

Analog Hardware Implementation of Continuous-Time Adaptive Filter Structures

We have implemented a four-tap adaptive filter in a continuous-time analog VLSI circuit. Since an ideal delay is impossible to implement in continuous-time hardware, we implemented the delay line as a cascade of low-pass filters (called the gamma filter). Since many years of research in our lab has shown that the gamma filter outperforms the ideal delay line for a wide range of applications, the gamma filter should not be considered merely a crude approximation of the ideal delay line. We show measured results from an analog chip that solves the problem of system identification–identifying an unknown linear circuit from its input/output relationship. Furthermore, we believe that a cascade of all-pass filters (called the Laguerre filter) will potentially outperform the gamma. We have built an adaptive Laguerre filter and show that its measured convergence rate is superior to that of the gamma. Finally, rather than perform gradient descent on a multimodal error function to determine a single optimal time constant, we propose multi-scale realizations of these delay line structures.     

The Kernel Least-Mean-Square Algorithm

The combination of the famed kernel trick and the least-mean-square (LMS) algorithm provides an interesting sample-by-sample update for an adaptive filter in reproducing kernel Hilbert spaces (RKHS), which is named in this paper the KLMS. Unlike the accepted view in kernel methods, this paper shows that in the finite training data case, the KLMS algorithm is well posed in RKHS without the addition of an extra regularization term to penalize solution norms as was suggested by Kivinen [Kivinen, Smola and Williamson, ldquoOnline Learning With Kernels,rdquo IEEE Transactions on Signal Processing, vol. 52, no. 8, pp. 2165-2176, Aug. 2004] and Smale [Smale and Yao, ldquoOnline Learning Algorithms,rdquo Foundations in Computational Mathematics, vol. 6, no. 2, pp. 145-176, 2006]. This result is the main contribution of the paper and enhances the present understanding of the LMS algorithm with a machine learning perspective. The effect of the KLMS step size is also studied from the viewpoint of regularization. Two experiments are presented to support our conclusion that with finite data the KLMS algorithm can be readily used in high dimensional spaces and particularly in RKHS to derive nonlinear, stable algorithms with comparable performance to batch, regularized solutions.     

Fast RLS-Like Algorithm for Generalized Eigendecomposition and its Applications

Generalized eigendecomposition (GED) plays a vital role in many signal-processing applications. In this paper, we will propose a new method for computing the generalized eigenvectors, which is on-line and resembles the RLS algorithm for Wiener filtering. We further present a proof to show convergence to the exact solution and simulations have shown that the algorithm is faster than most of the traditional methods. This algorithm belongs to the class of fixed-point algorithms and hence does not require any external step-size parameters like the gradient-based methods. Simulations are performed on synthetic data and compared with other algorithms found in literature. Finally we will demonstrate the application of GED in the design of a CDMA receiver for direct-sequence spread spectrum signals.     

Adaptive blind deconvolution of linear channels using Renyi's entropy with Parzen window estimation

Blind deconvolution of linear channels is a fundamental signal processing problem that has immediate extensions to multiple-channel applications. In this paper, we investigate the suitability of a class of Parzen-window-based entropy estimates, namely Renyi's entropy, as a criterion for blind deconvolution of linear channels. Comparisons between maximum and minimum entropy approaches, as well as the effect of entropy order, equalizer length, sample size, and measurement noise on performance, will be investigated through Monte Carlo simulations. The results indicate that this nonparametric entropy estimation approach outperforms the standard Bell-Sejnowski and normalized kurtosis algorithms in blind deconvolution. In addition, the solutions using Shannon's entropy were not optimal either for super- or sub-Gaussian source densities.     

Quantifying cognitive state from EEG using dependence measures

The exquisite human ability to perceive facial features has been explained by the activity of neurons particularly responsive to faces, found in the fusiform gyrus and the anterior part of the superior temporal sulcus. This study hypothesizes and demonstrates that it is possible to automatically discriminate face processing from processing of a simple control stimulus based on processed EEGs in an online fashion with high temporal resolution using measures of statistical dependence applied on steady-state visual evoked potentials. Correlation, mutual information, and a novel measure of association, referred to as generalized measure of association (GMA), were applied on filtered current source density data. Dependences between channel locations were assessed for two separate conditions elicited by distinct pictures (a face and a Gabor grating) flickering at a rate of 17.5?Hz. Filter settings were chosen to minimize the distortion produced by bandpassing parameters on dependence estimation. Statistical analysis was performed for automated stimulus classification using the Kolmogorov-Smirnov test. Results show active regions in the occipito-parietal part of the brain for both conditions with a greater dependence between occipital and inferotemporal sites for the face stimulus. GMA achieved a higher performance in discriminating the two conditions. Because no additional face-like stimuli were examined, this study established a basic difference between one particular face and one nonface stimulus. Future work may use additional stimuli and experimental manipulations to determine the specificity of the current connectivity results.