Semi-supervised learning combining co-training with active learning

Co-training is a good paradigm of semi-supervised, which requires the data set to be described by two views of features. There are a notable characteristic shared by many co-training algorithm: the selected unlabeled instances should be predicted with high confidence, since a high confidence score usually implies that the corresponding prediction is correct. Unfortunately, it is not always able to improve the classification performance with these high confidence unlabeled instances. In this paper, a new semi-supervised learning algorithm was proposed combining the benefits of both co-training and active learning. The algorithm applies co-training to select the most reliable instances according to the two criterions of high confidence and nearest neighbor for boosting the classifier, also exploit the most informative instances with human annotation for improve the classification performance. Experiments on several UCI data sets and natural language processing task, which demonstrate our method achieves more significant improvement for sacrificing the same amount of human effort.     

Statistical analysis of kernel-based least-squares density-ratio estimation

The ratio of two probability densities can be used for solving various machine learning tasks such as covariate shift adaptation (importance sampling), outlier detection (likelihood-ratio test), feature selection (mutual information), and conditional probability estimation. Several methods of directly estimating the density ratio have recently been developed, e.g., moment matching estimation, maximum-likelihood density-ratio estimation, and least-squares density-ratio fitting. In this paper, we propose a kernelized variant of the least-squares method for density-ratio estimation, which is called kernel unconstrained least-squares importance fitting (KuLSIF). We investigate its fundamental statistical properties including a non-parametric convergence rate, an analytic-form solution, and a leave-one-out cross-validation score. We further study its relation to other kernel-based density-ratio estimators. In experiments, we numerically compare various kernel-based density-ratio estimation methods, and show that KuLSIF compares favorably with other approaches.     

Semi-supervised learning with an imperfect supervisor

Real-life applications may involve huge data sets with misclassified or partially classified training data. Semi-supervised learning and learning in the presence of label noise have recently emerged as new paradigms in the machine learning community to cope with this kind of problems. This paper describes a new discriminant algorithm for semi-supervised learning. This algorithm optimizes the classification maximum likelihood (CML) of a set of labeled–unlabeled data, using a discriminant extension of the Classification Expectation Maximization algorithm. We further propose to extend this algorithm by modeling imperfections in the estimated class labels for unlabeled data. The parameters of this label-error model are learned together with the semi-supervised classifier parameters. We demonstrate the effectiveness of the approach using extensive experiments on different datasets. Massih R. Amini is currently assistant professor in the University of Pierre and Marie Curie (Paris 6). He received an engineering degree in computer science from the Ecole Supérieure d'Informatique (Computer science engineering school) in Paris in 1995. He then accomplished his master thesis in science in artificial intelligence in 1997 and obtained his PhD in 2001 at University of Pierre and Marie Curie. His research interests include Statistical Learning and Text-Mining. Patrick Gallinari is currently professor in the University of Pierre and Marie Curie (Paris 6) and head of the Computer Science laboratory (LIP6). His main research activity has been in the field of statistical machine learning for the last 15 years. He has also contributed in developing machine learning techniques for different application domains like information retrieval and text mining, user modelling, man–machine interaction and pen interfaces.     

Semi-supervised learning by disagreement

In many real-world tasks, there are abundant unlabeled examples but the number of labeled training examples is limited, because labeling the examples requires human efforts and expertise. So, semi-supervised learning which tries to exploit unlabeled examples to improve learning performance has become a hot topic. Disagreement-based semi-supervised learning is an interesting paradigm, where multiple learners are trained for the task and the disagreements among the learners are exploited during the semi-supervised learning process. This survey article provides an introduction to research advances in this paradigm.     

Semi-supervised Unified Latent Factor learning with multi-view data

Explosive multimedia resources are generated on web, which can be typically considered as a kind of multi-view data in nature. In this paper, we present a Semi-supervised Unified Latent Factor learning approach (SULF) to learn a predictive unified latent representation by leveraging both complementary information among multiple views and the supervision from the partially label information. On one hand, SULF employs a collaborative Nonnegative Matrix Factorization formulation to discover a unified latent space shared across multiple views. On the other hand, SULF adopts a regularized regression model to minimize a prediction loss on partially labeled data with the latent representation. Consequently, the obtained parts-based representation can have more discriminating power. In addition, we also develop a mechanism to learn the weights of different views automatically. To solve the proposed optimization problem, we design an effective iterative algorithm. Extensive experiments are conducted for both classification and clustering tasks on three real-world datasets and the compared results demonstrate the superiority of our approach.     

Computational complexity of kernel-based density-ratio estimation: a condition number analysis

In this study, the computational properties of a kernel-based least-squares density-ratio estimator are investigated from the viewpoint of condition numbers. The condition number of the Hessian matrix of the loss function is closely related to the convergence rate of optimization and the numerical stability. We use smoothed analysis techniques and theoretically demonstrate that the kernel least-squares method has a smaller condition number than other M-estimators. This implies that the kernel least-squares method has desirable computational properties. In addition, an alternate formulation of the kernel least-squares estimator that possesses an even smaller condition number is presented. The validity of the theoretical analysis is verified through numerical experiments.     

Semi-supervised dictionary learning with label propagation for image classification

Computational Visual Media - Sparse coding and supervised dictionary learning have rapidly developed in recent years, and achieved impressive performance in image classification. However, there is...     

Semi-supervised tensor learning for image classification

In this paper, we propose a new tensor-based representation algorithm for image classification. The algorithm is realized by learning the parameter tensor for image tensors. One novelty is that the parameter tensor is learned according to the Tucker tensor decomposition as the multiplication of a core tensor with a group of matrices for each order, which endows that the algorithm preserved the spatial information of image. We further extend the proposed tensor algorithm to a semi-supervised framework, in order to utilize both labeled and unlabeled images. The objective function can be solved by using the alternative optimization method, where at each iteration, we solve the typical ridge regression problem to obtain the closed form solution of the parameter along the corresponding order. Experimental results of gray and color image datasets show that our method outperforms several classification approaches. In particular, we find that our method can implement a high-quality classification performance when only few labeled training samples are provided.     

Semi-supervised learning using hidden feature augmentation

Semi-supervised learning methods are conventionally conducted by simultaneously utilizing abundant unlabeled samples and a few labeled samples given. However, the unlabeled samples are usually adopted with assumptions, e.g., cluster and manifold assumptions, which degrade the performance when the assumptions become invalid. The reliable hidden features embedded in both the labeled and the unlabeled samples can potentially be used to tackle this issue. In this regard, we investigate the feature augmentation technique to improve the robustness of semi-supervised learning in this paper. By introducing an orthonormal projection matrix, we first transform both the unlabeled and labeled samples into a shared hidden subspace to determine the connections between the samples. Then we utilize the hidden features, the raw features, and zero vectors determined to develop a novel feature augmentation strategy. Finally, a hidden feature transformation (HTF) model is proposed to compute the desired projection matrix by applying the maximum joint probability distribution principle in the augmented feature space. The effectiveness of the proposed method is evaluated in terms of the hinge and square loss functions respectively, based on two types of semi-supervised classification formulations developed using only the labeled samples with their original features and hidden features. The experimental results have demonstrated the effectiveness of the proposed feature augmentation technique for semi-supervised learning.     

Semi-supervised clustering with deep metric learning and graph embedding

World Wide Web - As a common technology in social network, clustering has attracted lots of research interest due to its high performance, and many clustering methods have been presented. The most...     

Semi-supervised learning integrated with classifier combination for word sense disambiguation

Word sense disambiguation (WSD) is the problem of determining the right sense of a polysemous word in a certain context. This paper investigates the use of unlabeled data for WSD within a framework of semi-supervised learning, in which labeled data is iteratively extended from unlabeled data. Focusing on this approach, we first explicitly identify and analyze three problems inherently occurred piecemeal in the general bootstrapping algorithm; namely the imbalance of training data, the confidence of new labeled examples, and the final classifier generation; all of which will be considered integratedly within a common framework of bootstrapping. We then propose solutions for these problems with the help of classifier combination strategies. This results in several new variants of the general bootstrapping algorithm. Experiments conducted on the English lexical samples of Senseval-2 and Senseval-3 show that the proposed solutions are effective in comparison with previous studies, and significantly improve supervised WSD.     

Semi-supervised clustering with metric learning: An adaptive kernel method

Most existing representative works in semi-supervised clustering do not sufficiently solve the violation problem of pairwise constraints. On the other hand, traditional kernel methods for semi-supervised clustering not only face the problem of manually tuning the kernel parameters due to the fact that no sufficient supervision is provided, but also lack a measure that achieves better effectiveness of clustering. In this paper, we propose an adaptive Semi-supervised Clustering Kernel Method based on Metric learning (SCKMM) to mitigate the above problems. Specifically, we first construct an objective function from pairwise constraints to automatically estimate the parameter of the Gaussian kernel. Then, we use pairwise constraint-based K-means approach to solve the violation issue of constraints and to cluster the data. Furthermore, we introduce metric learning into nonlinear semi-supervised clustering to improve separability of the data for clustering. Finally, we perform clustering and metric learning simultaneously. Experimental results on a number of real-world data sets validate the effectiveness of the proposed method.     

Semi-supervised kernel density estimation for video annotation

Insufficiency of labeled training data is a major obstacle for automatic video annotation. Semi-supervised learning is an effective approach to this problem by leveraging a large amount of unlabeled data. However, existing semi-supervised learning algorithms have not demonstrated promising results in large-scale video annotation due to several difficulties, such as large variation of video content and intractable computational cost. In this paper, we propose a novel semi-supervised learning algorithm named semi-supervised kernel density estimation (SSKDE) which is developed based on kernel density estimation (KDE) approach. While only labeled data are utilized in classical KDE, in SSKDE both labeled and unlabeled data are leveraged to estimate class conditional probability densities based on an extended form of KDE. It is a non-parametric method, and it thus naturally avoids the model assumption problem that exists in many parametric semi-supervised methods. Meanwhile, it can be implemented with an efficient iterative solution process. So, this method is appropriate for video annotation. Furthermore, motivated by existing adaptive KDE approach, we propose an improved algorithm named semi-supervised adaptive kernel density estimation (SSAKDE). It employs local adaptive kernels rather than a fixed kernel, such that broader kernels can be applied in the regions with low density. In this way, more accurate density estimates can be obtained. Extensive experiments have demonstrated the effectiveness of the proposed methods.     

Semi-supervised learning through adaptive Laplacian graph trimming

Graph-based semi-supervised learning (GSSL) attracts considerable attention in recent years. The performance of a general GSSL method relies on the quality of Laplacian weighted graph (LWR) composed of the similarity imposed on input examples. A key for constructing an effective LWR is on the proper selection of the neighborhood size K or ε on the construction of KNN graph or ε-neighbor graph on training samples, which constitutes the fundamental elements in LWR. Specifically, too large K or ε will result in “shortcut” phenomenon while too small ones cannot guarantee to represent a complete manifold structure underlying data. To this issue, this study attempts to propose a method, called adaptive Laplacian graph trimming (ALGT), to make an automatic tuning to cut improper inter-cluster shortcut edges while enhance the connection between intra-cluster samples, so as to adaptively fit a proper LWR from data. The superiority of the proposed method is substantiated by experimental results implemented on synthetic and UCI data sets.     

Semi-supervised AUC optimization based on positive-unlabeled learning

Maximizing the area under the receiver operating characteristic curve (AUC) is a standard approach to imbalanced classification. So far, various supervised AUC optimization methods have been developed and they are also extended to semi-supervised scenarios to cope with small sample problems. However, existing semi-supervised AUC optimization methods rely on strong distributional assumptions, which are rarely satisfied in real-world problems. In this paper, we propose a novel semi-supervised AUC optimization method that does not require such restrictive assumptions. We first develop an AUC optimization method based only on positive and unlabeled data and then extend it to semi-supervised learning by combining it with a supervised AUC optimization method. We theoretically prove that, without the restrictive distributional assumptions, unlabeled data contribute to improving the generalization performance in PU and semi-supervised AUC optimization methods. Finally, we demonstrate the practical usefulness of the proposed methods through experiments.     

Semi-supervised machine learning approach for DDoS detection

Even though advanced Machine Learning (ML) techniques have been adopted for DDoS detection, the attack remains a major threat of the Internet. Most of the existing ML-based DDoS detection approaches are under two categories: supervised and unsupervised. Supervised ML approaches for DDoS detection rely on availability of labeled network traffic datasets. Whereas, unsupervised ML approaches detect attacks by analyzing the incoming network traffic. Both approaches are challenged by large amount of network traffic data, low detection accuracy and high false positive rates. In this paper we present an online sequential semi-supervised ML approach for DDoS detection based on network Entropy estimation, Co-clustering, Information Gain Ratio and Exra-Trees algorithm. The unsupervised part of the approach allows to reduce the irrelevant normal traffic data for DDoS detection which allows to reduce false positive rates and increase accuracy. Whereas, the supervised part allows to reduce the false positive rates of the unsupervised part and to accurately classify the DDoS traffic. Various experiments were performed to evaluate the proposed approach using three public datasets namely NSL-KDD, UNB ISCX 12 and UNSW-NB15. An accuracy of 98.23%, 99.88% and 93.71% is achieved for respectively NSL-KDD, UNB ISCX 12 and UNSW-NB15 datasets, with respectively the false positive rates 0.33%, 0.35% and 0.46%.     

SELM: Semi-supervised ELM with application in sparse calibrated location estimation

Indoor location estimation based on Wi-Fi has attracted more and more attention from both research and industry fields. It brings two significant challenges. One is requiring a vast amount of labeled calibration data. The other is real-time training and testing for location estimation task. Traditional machine learning methods cannot get high performance in both aspects. This paper proposed a novel semi-supervised learning method SELM (semi-supervised extreme learning machine) and applied it to sparse calibrated location estimation. There are two advantages of the proposed SELM. First, it employs graph Laplacian regularization to import large number of unlabeled samples which can dramatically reduce labeled calibration samples. Second, it inherits the good property of ELM on extreme training and testing speed. Comparative experiments show that with same number of labeled samples, our method outperforms original ELM and back propagation (BP) network, especially in the case that the calibration data is very sparse.     

Semi-supervised multi-view binary learning for large-scale image clustering

Applied Intelligence - Large-scale image clustering has attracted sustained attention in machine learning. The traditional methods based on real value representation often suffer from the data...     

Semi-supervised online structure learning for composite event recognition

Online structure learning approaches, such as those stemming from statistical relational learning, enable the discovery of complex relations in noisy data streams. However, these methods assume the existence of fully-labelled training data, which is unrealistic for most real-world applications. We present a novel approach for completing the supervision of a semi-supervised structure learning task. We incorporate graph-cut minimisation, a technique that derives labels for unlabelled data, based on their distance to their labelled counterparts. In order to adapt graph-cut minimisation to first order logic, we employ a suitable structural distance for measuring the distance between sets of logical atoms. The labelling process is achieved online (single-pass) by means of a caching mechanism and the Hoeffding bound, a statistical tool to approximate globally-optimal decisions from locally-optimal ones. We evaluate our approach on the task of composite event recognition by using a benchmark dataset for human activity recognition, as well as a real dataset for maritime monitoring. The evaluation suggests that our approach can effectively complete the missing labels and eventually, improve the accuracy of the underlying structure learning system.