Discriminative learning can succeed where generative learning fails

Generative algorithms for learning classifiers use training data to separately estimate a probability model for each class. New items are classified by comparing their probabilities under these models. In contrast, discriminative learning algorithms try to find classifiers that perform well on all the training data.We show that there is a learning problem that can be solved by a discriminative learning algorithm, but not by any generative learning algorithm. This statement is formalized using a framework inspired by previous work of Goldberg [P. Goldberg, When can two unsupervised learners achieve PAC separation?, in: Proceedings of the 14th Annual COLT, 2001, pp. 303-319].     

一种基于属性分割的产生式/判别式混合分类器

为了利用产生式和判别式方法各自的优势,研究了基于属性分割的产生式/判别式混合分类模型框架,提出了一种基于属性分割的产生式/判别式混合分类器学习算法GDGA。其利用遗传算法,将属性集X划分为两个子集XG和XD,并相应地将训练集D垂直分割为两个子集DG和DD,在两个训练子集上分别学习产生式分类器和判别式分类器;最后将两个分类器合并形成一个混合分类器。实验结果表明,在大多数数据集上,混合分类器的分类正确率优于其成员分类器。在训练数据不足或数据属性分布不清楚的情况下,该混合分类器具有特别的优势。     

kNN中局部生成模型测度学习

已有的关于k近邻测度学习算法的工作主要集中于纯区分模型。在假定隐含的生成模型已知的情况下,提出了一种通过分析样本的k个近邻点的概率密度学习测度的方法。实验表明,这种基于类的生成模型假设学习到的局部测度可以有效改善kNN区分模型的性能。     

Efficient parameter learning of Bayesian network classifiers

Recent advances have demonstrated substantial benefits from learning with both generative and discriminative parameters. On the one hand, generative approaches address the estimation of the parameters of the joint distribution—\(\mathrm{P}(y,\mathbf{x})\), which for most network types is very computationally efficient (a notable exception to this are Markov networks) and on the other hand, discriminative approaches address the estimation of the parameters of the posterior distribution—and, are more effective for classification, since they fit \(\mathrm{P}(y|\mathbf{x})\) directly. However, discriminative approaches are less computationally efficient as the normalization factor in the conditional log-likelihood precludes the derivation of closed-form estimation of parameters. This paper introduces a new discriminative parameter learning method for Bayesian network classifiers that combines in an elegant fashion parameters learned using both generative and discriminative methods. The proposed method is discriminative in nature, but uses estimates of generative probabilities to speed-up the optimization process. A second contribution is to propose a simple framework to characterize the parameter learning task for Bayesian network classifiers. We conduct an extensive set of experiments on 72 standard datasets and demonstrate that our proposed discriminative parameterization provides an efficient alternative to other state-of-the-art parameterizations.     

产生式与判别式线性混合分类器

产生式方法和判别式方法是解决分类问题的两种不同框架,具有各自的优势。为利用两种方法各自的优势,文中提出一种产生式与判别式线性混合分类模型,并设计一种基于遗传算法的产生式与判别式线性混合分类模型的学习算法。该算法将线性混合分类器混合参数的学习看作一个最优化问题,以两个基分类器对每个训练数据的后验概率值为数据依据,用遗传算法找出线性混合分类器混合参数的最优值。实验结果表明,在大多数数据集上,产生式与判别式线性混合分类器的分类准确率优于或近似于它的两个基分类器中的优者。     

Object tracking via collaborative multi-task learning and appearance model updating

In this paper, we propose a novel visual tracking algorithm using the collaboration of generative and discriminative trackers under the particle filter framework. Each particle denotes a single task, and we encode all the tasks simultaneously in a structured multi-task learning manner. Then, we implement generative and discriminative trackers, respectively. The discriminative tracker considers the overall information of object to represent the object appearance; while the generative tracker takes the local information of object into account for handling partial occlusions. Therefore, two models are complementary during the tracking. Furthermore, we design an effective dictionary updating mechanism. The dictionary is composed of fixed and variational parts. The variational parts are progressively updated using Metropolis–Hastings strategy. Experiments on different challenging video sequences demonstrate that the proposed tracker performs favorably against several state-of-the-art trackers.     

Learning semantic concepts from image database with hybrid generative/discriminative approach

Semantic gap has become a bottleneck of content-based image retrieval in recent years. In order to bridge the gap and improve the retrieval performance, automatic image annotation has emerged as a crucial problem. In this paper, a hybrid approach is proposed to learn the semantic concepts of images automatically. Firstly, we present continuous probabilistic latent semantic analysis (PLSA) and derive its corresponding Expectation–Maximization (EM) algorithm. Continuous PLSA assumes that elements are sampled from a multivariate Gaussian distribution given a latent aspect, instead of a multinomial one in traditional PLSA. Furthermore, we propose a hybrid framework which employs continuous PLSA to model visual features of images in generative learning stage and uses ensembles of classifier chains to classify the multi-label data in discriminative learning stage. Therefore, the framework can learn the correlations between features as well as the correlations between words. Since the hybrid approach combines the advantages of generative and discriminative learning, it can predict semantic annotation precisely for unseen images. Finally, we conduct the experiments on three baseline datasets and the results show that our approach outperforms many state-of-the-art approaches.     

A fast learning algorithm for deep belief nets

We show how to use "complementary priors" to eliminate the explaining-away effects that make inference difficult in densely connected belief nets that have many hidden layers. Using complementary priors, we derive a fast, greedy algorithm that can learn deep, directed belief networks one layer at a time, provided the top two layers form an undirected associative memory. The fast, greedy algorithm is used to initialize a slower learning procedure that fine-tunes the weights using a contrastive version of the wake-sleep algorithm. After fine-tuning, a network with three hidden layers forms a very good generative model of the joint distribution of handwritten digit images and their labels. This generative model gives better digit classification than the best discriminative learning algorithms. The low-dimensional manifolds on which the digits lie are modeled by long ravines in the free-energy landscape of the top-level associative memory, and it is easy to explore these ravines by using the directed connections to display what the associative memory has in mind.     

鲁棒可预测判别字典学习人脸识别方法

提出一种可预测判别K-SVD网络模型(DKSVDN)并用于人脸识别问题。该模型构造了一种新颖的字典结构,包含类别标签字典和描述字典,以兼顾判别和重构性能。相应的稀疏编码向量由标签编码向量和描述编码向量组成。针对样本稀疏编码时间效率低的问题,利用预测神经网络与判别字典学习模型协同训练的方法来加速预测稀疏编码。此外,针对DKSVDN还特别引入一种拟梦境的训练方法用于提升模型在训练集多样性不足时的鲁棒性。通过在主流人脸数据集上的对比实验证明了该模型的优良性能。     

Discriminative Learning for Dynamic State Prediction

We consider the problem of predicting a sequence of real-valued multivariate states that are correlated by some unknown dynamics, from a given measurement sequence. Although dynamic systems such as the State-Space Models are popular probabilistic models for the problem, their joint modeling of states and observations, as well as the traditional generative learning by maximizing a joint likelihood may not be optimal for the ultimate prediction goal. In this paper, we suggest two novel discriminative approaches to the dynamic state prediction: 1) learning generative state-space models with discriminative objectives and 2) developing an undirected conditional model. These approaches are motivated by the success of recent discriminative approaches to the structured output classification in discrete-state domains, namely, discriminative training of Hidden Markov Models and Conditional Random Fields (CRFs). Extending CRFs to real multivariate state domains generally entails imposing density integrability constraints on the CRF parameter space, which can make the parameter learning difficult. We introduce an efficient convex learning algorithm to handle this task. Experiments on several problem domains, including human motion and robot-arm state estimation, indicate that the proposed approaches yield high prediction accuracy comparable to or better than state-of-the-art methods.     

On the generative–discriminative tradeoff approach: Interpretation, asymptotic efficiency and classification performance

The interpretation of generative, discriminative and hybrid approaches to classification is discussed, in particular for the generative–discriminative tradeoff (GDT), a hybrid approach. The asymptotic efficiency of the GDT, relative to that of its generative or discriminative counterpart, is presented theoretically and, by using linear normal discrimination as an example, numerically. On real and simulated datasets, the classification performance of the GDT is compared with those of normal-based linear discriminant analysis (LDA) and linear logistic regression (LLR). Four arguments are made as follows. First, the GDT is a generative model integrating both discriminative and generative learning. It is therefore subject to model misspecification of the data-generating process and hindered by complex optimisation. Secondly, among the three approaches being compared, the asymptotic efficiency of the GDT is higher than that of the discriminative approach but lower than that of the generative approach, when no model misspecification occurs. Thirdly, without model misspecification, LDA performs the best; with model misspecification, LLR or the GDT with an optimal, large weight on its discriminative component may perform the best. Finally, LLR is affected by the imbalance between groups of data.     

Learning probabilistic models of tree edit distance

Nowadays, there is a growing interest in machine learning and pattern recognition for tree-structured data. Trees actually provide a suitable structural representation to deal with complex tasks such as web information extraction, RNA secondary structure prediction, computer music, or conversion of semi-structured data (e.g. XML documents). Many applications in these domains require the calculation of similarities over pairs of trees. In this context, the tree edit distance (ED) has been subject of investigations for many years in order to improve its computational efficiency. However, used in its classical form, the tree ED needs a priori fixed edit costs which are often difficult to tune, that leaves little room for tackling complex problems. In this paper, to overcome this drawback, we focus on the automatic learning of a non-parametric stochastic tree ED. More precisely, we are interested in two kinds of probabilistic approaches. The first one builds a generative model of the tree ED from a joint distribution over the edit operations, while the second works from a conditional distribution providing then a discriminative model. To tackle these tasks, we present an adaptation of the expectation–maximization algorithm for learning these distributions over the primitive edit costs. Two experiments are conducted. The first is achieved on artificial data and confirms the interest to learn a tree ED rather than a priori imposing edit costs; The second is applied to a pattern recognition task aiming to classify handwritten digits.     

Statistical topic models for multi-label document classification

Machine learning approaches to multi-label document classification have to date largely relied on discriminative modeling techniques such as support vector machines. A?drawback of these approaches is that performance rapidly drops off as the total number of labels and the number of labels per document increase. This problem is amplified when the label frequencies exhibit the type of highly skewed distributions that are often observed in real-world datasets. In this paper we investigate a class of generative statistical topic models for multi-label documents that associate individual word tokens with different labels. We investigate the advantages of this approach relative to discriminative models, particularly with respect to classification problems involving large numbers of relatively rare labels. We compare the performance of generative and discriminative approaches on document labeling tasks ranging from datasets with several thousand labels to datasets with tens of labels. The experimental results indicate that probabilistic generative models can achieve competitive multi-label classification performance compared to discriminative methods, and have advantages for datasets with many labels and skewed label frequencies.     

Efficient Learning of Relational Object Class Models

We present an efficient method for learning part-based object class models from unsegmented images represented as sets of salient features. A model includes parts’ appearance, as well as location and scale relations between parts. The object class is generatively modeled using a simple Bayesian network with a central hidden node containing location and scale information, and nodes describing object parts. The model’s parameters, however, are optimized to reduce a loss function of the training error, as in discriminative methods. We show how boosting techniques can be extended to optimize the relational model proposed, with complexity linear in the number of parts and the number of features per image. This efficiency allows our method to learn relational models with many parts and features. The method has an advantage over purely generative and purely discriminative approaches for learning from sets of salient features, since generative method often use a small number of parts and features, while discriminative methods tend to ignore geometrical relations between parts. Experimental results are described, using some bench-mark data sets and three sets of newly collected data, showing the relative merits of our method in recognition and localization tasks.     

Discriminative multi-task objects tracking with active feature selection and drift correction

In this paper, we propose a discriminative multi-task objects tracking method with active feature selection and drift correction. The developed method formulates object tracking in a particle filter framework as multi-Task discriminative tracking. As opposed to generative methods that handle particles separately, the proposed method learns the representation of all the particles jointly and the corresponding coefficients are similar. The tracking algorithm starts from the active feature selection scheme, which adaptively chooses suitable number of discriminative features from the tracked target and background in the dynamic environment. Based on the selected feature space, the discriminative dictionary is constructed and updated dynamically. Only a few of them are used to represent all the particles at each frame. In other words, all the particles share the same dictionary templates and their representations are obtained jointly by discriminative multi-task learning. The particle that has the highest similarity with the dictionary templates is selected as the next tracked target state. This jointly sparsity and discriminative learning can exploit the relationship between particles and improve tracking performance. To alleviate the visual drift problem encountered in object tracking, a two-stage particle filtering algorithm is proposed to complete drift correction and exploit both the ground truth information of the first frame and observations obtained online from the current frame. Experimental evaluations on challenging sequences demonstrate the effectiveness, accuracy and robustness of the proposed tracker in comparison with state-of-the-art algorithms.     

基于深度残差生成对抗网络的医学影像超分辨率算法

针对医学影像超分辨率重建过程中细节丢失导致的模糊问题,提出了一种基于深度残差生成对抗网络（GAN）的医学影像超分辨率算法。首先,算法包括生成器网络和判别器网络,生成器网络生成高分辨率图像,判别器网络辨别图像真伪。然后,通过设计生成器网络的上采样采用缩放卷积来削弱棋盘效应,并去掉标准残差块中的批量规范化层以优化网络;进一步增加判别器网络中特征图数量以加深网络等方面提高网络性能。最后,用生成损失和判别损失来不断优化网络,指导生成高质量的图像。实验结果表明,对比双线性内插、最近邻插值、双三次插值法、基于深度递归神经网络、基于生成对抗网络的超分辨率方法（SRGAN）,所提算法重建出了纹理更丰富、视觉更逼真的图像。相比SRGAN方法,所提算法在峰值信噪比（PSNR）和结构相似度（SSIM）上有0.21 dB和0.32%的提升。所提算法为医学影像超分辨率的理论研究提供了深度残差生成对抗网络的方法,在其实际应用中可靠、有效。     

Maximum margin Bayesian network classifiers

We present a maximum margin parameter learning algorithm for Bayesian network classifiers using a conjugate gradient (CG) method for optimization. In contrast to previous approaches, we maintain the normalization constraints on the parameters of the Bayesian network during optimization, i.e., the probabilistic interpretation of the model is not lost. This enables us to handle missing features in discriminatively optimized Bayesian networks. In experiments, we compare the classification performance of maximum margin parameter learning to conditional likelihood and maximum likelihood learning approaches. Discriminative parameter learning significantly outperforms generative maximum likelihood estimation for naive Bayes and tree augmented naive Bayes structures on all considered data sets. Furthermore, maximizing the margin dominates the conditional likelihood approach in terms of classification performance in most cases. We provide results for a recently proposed maximum margin optimization approach based on convex relaxation. While the classification results are highly similar, our CG-based optimization is computationally up to orders of magnitude faster. Margin-optimized Bayesian network classifiers achieve classification performance comparable to support vector machines (SVMs) using fewer parameters. Moreover, we show that unanticipated missing feature values during classification can be easily processed by discriminatively optimized Bayesian network classifiers, a case where discriminative classifiers usually require mechanisms to complete unknown feature values in the data first.     

Discriminative speaker recognition using large margin GMM

Most state-of-the-art speaker recognition systems are based on discriminative learning approaches. On the other hand, generative Gaussian mixture models (GMM) have been widely used in speaker recognition during the last decades. In an earlier work, we proposed an algorithm for discriminative training of GMM with diagonal covariances under a large margin criterion. In this paper, we propose an improvement of this algorithm, which has the major advantage of being computationally highly efficient, thus well suited to handle large-scale databases. We also develop a new strategy to detect and handle the outliers that occur in the training data. To evaluate the performances of our new algorithm, we carry out full NIST speaker identification and verification tasks using NIST-SRE’2006 data, in a Symmetrical Factor Analysis compensation scheme. The results show that our system significantly outperforms the traditional discriminative support vector machines (SVM)-based system of SVM-GMM supervectors, in the two speaker recognition tasks.     

Dynamic selection of generative–discriminative ensembles for off-line signature verification

In practice, each writer provides only a limited number of signature samples to design a signature verification (SV) system. Hybrid generative–discriminative ensembles of classifiers (EoCs) are proposed in this paper to design an off-line SV system from few samples, where the classifier selection process is performed dynamically. To design the generative stage, multiple discrete left-to-right Hidden Markov Models (HMMs) are trained using a different number of states and codebook sizes, allowing the system to learn signatures at different levels of perception. To design the discriminative stage, HMM likelihoods are measured for each training signature, and assembled into feature vectors that are used to train a diversified pool of two-class classifiers through a specialized Random Subspace Method. During verification, a new dynamic selection strategy based on the K-nearest-oracles (KNORA) algorithm and on Output Profiles selects the most accurate EoCs to classify a given input signature. This SV system is suitable for incremental learning of new signature samples. Experiments performed with real-world signature data (composed of genuine samples, and random, simple and skilled forgeries) indicate that the proposed dynamic selection strategy can significantly reduce the overall error rates, with respect to other EoCs formed using well-known dynamic and static selection strategies. Moreover, the performance of the SV system proposed in this paper is significantly greater than or comparable to that of related systems found in the literature.     

Structural Extension to Logistic Regression: Discriminative Parameter Learning of Belief Net Classifiers

Bayesian belief nets (BN_s) are often used for classification tasks—typically to return the most likely class label for each specified instance. Many BN-learners, however, attempt to find the BN that maximizes a different objective function—viz., likelihood, rather than classification accuracy—typically by first learning an appropriate graphical structure, then finding the parameters for that structure that maximize the likelihood of the data. As these parameters may not maximize the classification accuracy, “discriminative parameter learners” follow the alternative approach of seeking the parameters that maximize conditional likelihood (CL), over the distribution of instances the BN will have to classify. This paper first formally specifies this task, shows how it extends standard logistic regression, and analyzes its inherent sample and computational complexity. We then present a general algorithm for this task, ELR, that applies to arbitrary BN structures and that works effectively even when given incomplete training data. Unfortunately, ELR is not guaranteed to find the parameters that optimize conditional likelihood; moreover, even the optimal-CL parameters need not have minimal classification error. This paper therefore presents empirical evidence that ELR produces effective classifiers, often superior to the ones produced by the standard “generative” algorithms, especially in common situations where the given BN-structure is incorrect.