一种基于加权标志点的流形学习算法

近年来出现的一系列进行维数约简的非线性方法——流形学习中等距映射（Isomap）是其中的代表,该算法高效、简单,但计算复杂度较高。基于标志点（Landmark Points）的L-Isomap减少了计算复杂度,但对于标志点的选取,大都采用随机的方法,致使该算法不稳定。考虑到样本点和近邻点相对位置,将对嵌入流形影响较大的样本点赋予较高的权重。然后根据权重大小选择标志点,同时考虑标志点之间的相对位置,使得选出的标志点不会出现过度集中的现象,近似直线分布的概率也大大降低,从而保证了算法的稳定性。实验结果表明,该算法在标志点数量较少的情况下,比L-Isomap稳定,且对缺失数据的不完整流形,也能获取和Isomap相差不大的结果。     

Semidefinite spectral clustering

Multi-way partitioning of an undirected weighted graph where pairwise similarities are assigned as edge weights, provides an important tool for data clustering, but is an NP-hard problem. Spectral relaxation is a popular way of relaxation, leading to spectral clustering where the clustering is performed by the eigen-decomposition of the (normalized) graph Laplacian. On the other hand, semidefinite relaxation, is an alternative way of relaxing a combinatorial optimization, leading to a convex optimization. In this paper we employ a semidefinite programming (SDP) approach to the graph equipartitioning for clustering, where sufficient conditions for strong duality hold. The method is referred to as semidefinite spectral clustering, where the clustering is based on the eigen-decomposition of the optimal feasible matrix computed by SDP. Numerical experiments with several data sets, demonstrate the useful behavior of our semidefinite spectral clustering, compared to existing spectral clustering methods.     

Landmark MDS ensemble

Landmark multidimensional scaling (LMDS) uses a subset of data (landmark points) to solve classical multidimensional scaling (MDS), where the scalability is increased but the approximation is noise-sensitive. In this paper we present an LMDS ensemble where we use a portion of the input in a piecewise manner to solve classical MDS, combining individual LMDS solutions which operate on different partitions of the input. Ground control points (GCPs) that are shared by partitions considered in the ensemble, allow us to align individual LMDS solutions in a common coordinate system through affine transformations. We incorporate priors into combining multiple LMDS solutions such that the weighted averaging by priors improves the noise-robustness of our method. Our LMDS ensemble is much less noise-sensitive while maintaining the scalability and the speed of LMDS. Experiments on synthetic data (noisy grid) and real-world data (similar image retrieval) confirm the high performance of the proposed LMDS ensemble.     

Polynomial summaries of positive semidefinite kernels

Polynomials have proven to be useful tools to tailor generic kernels to specific applications. Nevertheless, we had only restricted knowledge for selecting fertile polynomials which consistently produce positive semidefinite kernels. For example, the well-known polynomial kernel can only take advantage of a very narrow range of polynomials, that is, the univariate polynomials with positive coefficients. This restriction not only hinders intensive exploitation of the flexibility of the kernel method, but also causes misuse of indefinite kernels. Our main theorem significantly relaxes the restriction by asserting that a polynomial consistently produces positive semidefinite kernels, if it has a positive semidefinite coefficient matrix. This sufficient condition is quite natural, and hence, it can be a good characterization of the fertile polynomials. In fact, we prove that the converse of the assertion of the theorem also holds true in the case of degree 1. We also prove the effectiveness of our main theorem by showing three corollaries relating to certain applications known in the literature: the first and second corollaries, respectively, give generalizations of the polynomial kernel and the principal-angle (determinant) kernel. The third corollary shows extended and corrected sufficient conditions for the codon-improved kernel and the weighted-degree kernel with shifts to be positive semidefinite.     

Semidefinite programming for uncertain linear equations in static analysis of structures

This paper presents a method for computing a minimal bound that contains the solution set to the uncertain linear equations. Particularly, we aim at finding a bounding ellipsoid for static response of structures, where both external forces and elastic moduli of members are assumed to be imprecisely known and bounded. By using the S-lemma, we formulate a semidefinite programming (SDP) problem which provides an outer approximation of the minimal bounding ellipsoid. Bounding ellipsoids are computed for nodal displacements of uncertain braced frames as the solutions of the presented SDP problems by using the primal-dual interior-point method.     

Optimal non-iterative pose estimation via convex relaxation

In this paper we present a convex relaxation method that globally solves for the camera position and orientation given a set of image pixel measurements associated with a scene of reference points of known three-dimensional positions. The approach formulates the pose optimization problem as a semidefinite positive relaxation (SDR) program. A comprehensive comparative performance analysis, carried out in the computer simulations section, demonstrates the superior performance of the relaxation method over existing approaches. The computational complexity of the method is O(n)$O(n)$, where n is the number of reference points, and is applicable to both coplanar and non-coplanar reference point configurations. The average run-time recorded is 50 ms for an average point count of 100.     

Globalized and localized matrix-pattern-oriented classification machine

Inspired by the matrix-based methods used in feature extraction and selection, one matrix-pattern-oriented classification framework has been designed in our previous work and demonstrated to utilize one matrix pattern itself more effectively to improve the classification performance in practice. However, this matrix-based framework neglects the prior structural information of the whole input space that is made up of all the matrix patterns. This paper aims to overcome such flaw through taking advantage of one structure learning method named Alternative Robust Local Embedding (ARLE). As a result, a new regularization term R_gl is designed, expected to simultaneously represent the globality and the locality of the whole data domain, further boosting the existing matrix-based classification method. To our knowledge, it is the first trial to introduce both the globality and the locality of the whole data space into the matrixized classifier design. In order to validate the proposed approach, the designed R_gl is applied into the previous work matrix-pattern-oriented Ho-Kashyap classifier (MatMHKS) to construct a new globalized and localized MatMHKS named GLMatMHKS. The experimental results on a broad range of data validate that GLMatMHKS not only inherits the advantages of the matrixized learning, but also uses the prior structural information more reasonably to guide the classification machine design.     

Heterogeneous data fusion for predicting mild cognitive impairment conversion

In the clinical study of Alzheimer’s Disease (AD) with neuroimaging data, it is challenging to identify the progressive Mild Cognitive Impairment (pMCI) subjects from the stableMCI (sMCI) subjects (i.e., the pMCI/sMCI classification) in an individual level because of small inter-group differences between two groups (i.e., pMCIs and sMCIs) as well as high intra-group variations within each group. Moreover, there are a very limited number of subjects available, which cannot guarantee to find informative and discriminative patterns for achieving high diagnostic accuracy. In this paper, we propose a novel sparse regression method to fuse the auxiliary data into the predictor data for the pMCI/sMCI classification, where the predictor data is structural Magnetic Resonance Imaging (MRI) information of both pMCI and sMCI subjects and the auxiliary data includes the ages of the subjects, the Positron Emission Tomography (PET) information of the predictor data, and the structural MRI information of AD and Normal Controls (NC). Specifically, we incorporate the auxiliary data and the predictor data into a unified framework to jointly achieve the following objectives: i) jointly selecting informative features from both the auxiliary data and the predictor data; ii) robust to outliers from both the auxiliary data and the predictor data; and iii) reducing the aging effect due to the possible cause of brain atrophy induced by both the normal aging and the disease progression. As a result, our proposed method jointly selects the useful features from the auxiliary data and the predictor data by taking into account the influence of outliers and the age of the two kinds of data, i.e., the pMCI and sMCI subjects as well as the AD and NC subjects. We further employ the linear Support Vector Machine (SVM) with the selected features of the predictor data to conduct the pMCI/sMCI classification. Experimental results on the public data of Alzheimer’s Disease Neuroimaging Initiative (ADNI) show the proposed method achieved the best classification performance, compared to the best comparison method, in terms of four evaluation metrics.     

Synthesis of discrete H ∞-controllers with given stability margin radius and settling time

For linear multivariable systems, we construct discrete output controllers that guarantee a given stability margin radius on the input or output of a control plant. Besides, given control time is also taken into account. We show that solving such problems reduces to a certain specially constructed standard H _∞-optimization problem. A numerical solution has been implemented in MATLAB with the Robust Control Toolbox suite based on the method of linear matrix inequalities (LMI).     

Robust self-tuning semi-supervised learning

We investigate the issue of graph-based semi-supervised learning (SSL). The labeled and unlabeled data points are represented as vertices in an undirected weighted neighborhood graph, with the edge weights encoding the pairwise similarities between data objects in the same neighborhood. The SSL problem can be then formulated as a regularization problem on this graph. In this paper we propose a robust self-tuning graph-based SSL method, which (1) can determine the similarities between pairwise data points automatically; (2) is not sensitive to outliers. Promising experimental results are given for both synthetic and real data sets.     

Density-induced margin support vector machines

This paper proposes a new classifier called density-induced margin support vector machines (DMSVMs). DMSVMs belong to a family of SVM-like classifiers. Thus, DMSVMs inherit good properties from support vector machines (SVMs), e.g., unique and global solution, and sparse representation for the decision function. For a given data set, DMSVMs require to extract relative density degrees for all training data points. These density degrees can be taken as relative margins of corresponding training data points. Moreover, we propose a method for estimating relative density degrees by using the K nearest neighbor method. We also show the upper bound on the leave-out-one error of DMSVMs for a binary classification problem and prove it. Promising results are obtained on toy as well as real-world data sets.     

Sparse non Gaussian component analysis by semidefinite programming

Sparse non-Gaussian component analysis is an unsupervised linear method of extracting any structure from high-dimensional distributed data based on estimating a low-dimensional non-Gaussian data component. In this paper we discuss a new approach with known apriori reduced dimension to direct estimation of the projector on the target space using semidefinite programming. The new approach avoids the estimation of the data covariance matrix and overcomes the traditional separation of element estimation of the target space and target space reconstruction. This allows to reduced the sampling size while improving the sensitivity to a broad variety of deviations from normality. Moreover the complexity of the new approach is limited to O(dlogd). We also discuss the procedures which allows to recover the structure when its effective dimension is unknown.     

On the stability of the reduced basis method for Stokes equations in parametrized domains

We present an application of reduced basis method for Stokes equations in domains with affine parametric dependence. The essential components of the method are (i) the rapid convergence of global reduced basis approximations - Galerkin projection onto a space W_N spanned by solutions of the governing partial differential equation at N selected points in parameter space; (ii) the off-line/on-line computational procedures decoupling the generation and projection stages of the approximation process.The operation count for the on-line stage - in which, given a new parameter value, we calculate an output of interest - depends only on N (typically very small) and the parametric complexity of the problem; the method is thus ideally suited for the repeated and rapid evaluations required in the context of parameter estimation, design, optimization, and real-time control. Particular attention is given (i) to the pressure treatment of incompressible Stokes problem; (ii) to find an equivalent inf-sup condition that guarantees stability of reduced basis solutions by enriching the reduced basis velocity approximation space with the solutions of a supremizer problem; (iii) to provide algebraic stability of the problem by reducing the condition number of reduced basis matrices using an orthonormalization procedure applied to basis functions; (iv) to reduce computational costs in order to allow real-time solution of parametrized problem.     

Active learning with effective scoring functions for semi-supervised temporal action localization

Temporal Action Localization (TAL) aims to predict both action category and temporal boundary of action instances in untrimmed videos, i.e., start and end time. Existing works usually adopt fully-supervised solutions, however, one of the practical bottlenecks in these solutions is the large amount of labeled training data required. To reduce expensive human label cost, this paper focuses on a rarely investigated yet practical task named semi-supervised TAL and proposes an effective active learning method, named AL-STAL. We leverage four steps for actively selecting video samples with high informativeness and training the localization model, named Train, Query, Annotate, Append. Two scoring functions that consider the uncertainty of localization model are equipped in AL-STAL, thus facilitating the video sample ranking and selection. One takes entropy of predicted label distribution as measure of uncertainty, named Temporal Proposal Entropy (TPE). And the other introduces a new metric based on mutual information between adjacent action proposals, named Temporal Context Inconsistency (TCI). To validate the effectiveness of proposed method, we conduct extensive experiments on three benchmark datasets THUMOS’14, ActivityNet 1.3 and ActivityNet 1.2. Experiment results show that AL-STAL outperforms the existing competitors and achieves satisfying performance compared with fully-supervised learning.     

A new autocovariance least-squares method for estimating noise covariances

Industrial implementation of model-based control methods, such as model predictive control, is often complicated by the lack of knowledge about the disturbances entering the system. In this paper, we present a new method (constrained ALS) to estimate the variances of the disturbances entering the process using routine operating data. A variety of methods have been proposed to solve this problem. Of note, we compare ALS to the classic approach presented in Mehra [(1970). On the identification of variances and adaptive Kalman filtering. IEEE Transactions on Automatic Control, 15(12), 175-184]. This classic method, and those based on it, use a three-step procedure to compute the covariances. The method presented in this paper is a one-step procedure, which yields covariance estimates with lower variance on all examples tested. The formulation used in this paper provides necessary and sufficient conditions for uniqueness of the estimated covariances, previously not available in the literature. We show that the estimated covariances are unbiased and converge to the true values with increasing sample size. The proposed method also guarantees positive semidefinite covariance estimates by adding constraints to the ALS problem. The resulting convex program can be solved efficiently.     

F-BIDS: Federated-Blending based Intrusion Detection System

The rapid development of network communication along with the drastic increase in the number of smart devices has triggered a surge in network traffic, which can contain private data and in turn affect user privacy. Recently, Federated Learning (FL) has been proposed in Intrusion Detection Systems (IDS) to ensure attack detection, privacy preservation, and cost reduction, which are crucial issues in traditional centralized machine-learning-based IDS. However, FL-based approaches still exhibit vulnerabilities that can be exploited by adversaries to compromise user data. At the same time, meta-models (including the blending models) have been recognized as one of the solutions to improve generalization for attack detection and classification since they enhance generalization and predictive performances by combining multiple base models. Therefore, in this paper, we propose a Federated Blending model-driven IDS framework for the Internet of Things (IoT) and Industrial IoT (IIoT), called F-BIDS, in order to further protect the privacy of existing ML-based IDS. The proposition consists of a Decision Tree (DT) and Random Forest (RF) as base classifiers to first produce the meta-data. Then, the meta-classifier, which is a Neural Networks (NN) model, uses the meta-data during the federated training step, and finally, it makes the final classification on the test set. Specifically, in contrast to the classical FL approaches, the federated meta-classifier is trained on the meta-data (composite data) instead of user-sensitive data to further enhance privacy. To evaluate the performance of F-BIDS, we used the most recent and open cyber-security datasets, called Edge-IIoTset (published in 2022) and InSDN (in 2020). We chose these datasets because they are recent datasets and contain a large amount of network traffic including both malicious and benign traffic.     

An extended iterative format for the progressive-iteration approximation

Progressive-iteration approximation (PIA) is a new data fitting technique developed recently for blending curves and surfaces. Taking the given data points as the initial control points, PIA constructs a series of fitting curves (surfaces) by adjusting the control points iteratively, while the limit curve (surface) interpolates the data points. More importantly, progressive-iteration approximation has the local property, that is, the limit curve (surface) can interpolate a subset of data points by just adjusting a part of corresponding control points, and remaining others unchanged. However, the current PIA format requires that the number of the control points equals that of the data points, thus making the PIA technique inappropriate to fitting large scale data points. To overcome this drawback, in this paper, we develop an extended PIA (EPIA) format, which allows that the number of the control points is less than that of the given data points. Moreover, since the main computations of EPIA are independent, they can be performed in parallel efficiently, with storage requirement O(n), where n is the number of the control points. Therefore, due to its local property and parallel computing capability, the EPIA technique has great potential in large scale data fitting. Specifically, by the EPIA format, we develop an incremental data fitting algorithm in this paper. In addition, some examples are demonstrated in this paper, all implemented by the parallel computing toolbox of Matlab, and run on a PC with a four-core CPU.     

Comments on “Robust optimal design and convergence properties analysis of iterative learning control approaches” and “On the P-type and Newton-type ILC schemes for dynamic systems with non-affine input factors”

The papers by Xu and Tan [Robust optimal design and convergence properties analysis of iterative learning control approaches, Automatica 38 (2002) 1867-1880], and Xu and Tan [On the P-type and Newton-type ILC schemes for dynamic systems with non-affine input factors, Automatica 38 (2002) 1237-1242], give a convergence analysis for several iterative learning control approaches. Unfortunately, these papers contains several mathematical errors that render the proofs of the claimed results invalid. As there are no obvious ways to correct these errors, the results presented in these papers are questionable.     

Approximating Points by a Piecewise Linear Function

Approximating points by piecewise linear functions is an intensively researched topic in computational geometry. In this paper, we study, based on the uniform error metric, an array of variations of this problem in 2-D and 3-D, including points with weights, approximation with violations, using step functions or more generally piecewise linear functions. We consider both the min-# (i.e., given an error tolerance ?, minimizing the size k of the approximating function) and min-? (i.e., given a size k of the approximating function, minimizing the error tolerance ?) versions of the problems. Our algorithms either improve on the previously best-known solutions or are the first known results for the respective problems. Our approaches are based on interesting geometric observations and algorithmic techniques. Some data structures we develop are of independent interest and may find other applications.