Independent factor analysis

We introduce the independent factor analysis (IFA) method for recovering independent hidden sources from their observed mixtures. IFA generalizes and unifies ordinary factor analysis (FA), principal component analysis (PCA), and independent component analysis (ICA), and can handle not only square noiseless mixing but also the general case where the number of mixtures differs from the number of sources and the data are noisy. IFA is a two-step procedure. In the first step, the source densities, mixing matrix, and noise covariance are estimated from the observed data by maximum likelihood. For this purpose we present an expectation-maximization (EM) algorithm, which performs unsupervised learning of an associated probabilistic model of the mixing situation. Each source in our model is described by a mixture of gaussians; thus, all the probabilistic calculations can be performed analytically. In the second step, the sources are reconstructed from the observed data by an optimal nonlinear estimator. A variational approximation of this algorithm is derived for cases with a large number of sources, where the exact algorithm becomes intractable. Our IFA algorithm reduces to the one for ordinary FA when the sources become gaussian, and to an EM algorithm for PCA in the zero-noise limit. We derive an additional EM algorithm specifically for noiseless IFA. This algorithm is shown to be superior to ICA since it can learn arbitrary source densities from the data. Beyond blind separation, IFA can be used for modeling multidimensional data by a highly constrained mixture of gaussians and as a tool for nonlinear signal encoding.     

多尺度多核高斯过程隐变量模型

高斯过程隐变量模型（GPLVM）作为一种无监督的贝叶斯非参数降维模型,无法有效利用数据所包含的语义标记信息,同时其建模过程中假设观测变量的各特征相互独立,忽略了特征之间的空间结构信息。为解决上述问题,采用图像池化操作获得不同尺度的特征表示,利用线性投影方式将不同尺度的图像投影到低维隐空间进行特征融合,并将融合特征和数据标记分别作为输入和输出,构建多尺度多核高斯过程隐变量模型（MSMK-GPLVM）,通过图像数据与数据标记的关联实现模型监督学习,同时对GPLVM和线性投影权重矩阵进行联合学习以提高分类性能。实验结果表明,MSMK-GPLVM能够有效利用图像空间结构信息和语义标记信息,相比其他隐变量模型具有更强的数据降维和分类能力。     

Data decomposition using independent component analysis with prior constraints

In many data analysis problems, it is useful to consider the data as generated from a set of unknown (latent) generators or sources. The observations we make of a system are then taken to be related to these sources through some unknown function. Furthermore, the (unknown) number of underlying latent sources may be less than the number of observations. Recent developments in independent component analysis (ICA) have shown that, in the case where the unknown function linking sources to observations is linear, such data decomposition may be achieved in a mathematically elegant manner. In this paper, we extend the general ICA paradigm to include a very flexible source model, prior constraints and conditioning on sets of intermediate variables so that ICA forms one part of a hierarchical system. We show that such an approach allows for efficient discovery of hidden representation in data and for unsupervised data partitioning.     

Multimodal Earth observation data fusion: Graph-based approach in shared latent space

Multiple and heterogenous Earth observation (EO) platforms are broadly used for a wide array of applications, and the integration of these diverse modalities facilitates better extraction of information than using them individually. The detection capability of the multispectral unmanned aerial vehicle (UAV) and satellite imagery can be significantly improved by fusing with ground hyperspectral data. However, variability in spatial and spectral resolution can affect the efficiency of such dataset's fusion. In this study, to address the modality bias, the input data was projected to a shared latent space using cross-modal generative approaches or guided unsupervised transformation. The proposed adversarial networks and variational encoder-based strategies used bi-directional transformations to model the cross-domain correlation without using cross-domain correspondence. It may be noted that an interpolation-based convolution was adopted instead of the normal convolution for learning the features of the point spectral data (ground spectra). The proposed generative adversarial network-based approach employed dynamic time wrapping based layers along with a cyclic consistency constraint to use the minimal number of unlabeled samples, having cross-domain correlation, to compute a cross-modal generative latent space. The proposed variational encoder-based transformation also addressed the cross-modal resolution differences and limited availability of cross-domain samples by using a mixture of expert-based strategy, cross-domain constraints, and adversarial learning. In addition, the latent space was modelled to be composed of modality independent and modality dependent spaces, thereby further reducing the requirement of training samples and addressing the cross-modality biases. An unsupervised covariance guided transformation was also proposed to transform the labelled samples without using cross-domain correlation prior. The proposed latent space transformation approaches resolved the requirement of cross-domain samples which has been a critical issue with the fusion of multi-modal Earth observation data. This study also proposed a latent graph generation and graph convolutional approach to predict the labels resolving the domain discrepancy and cross-modality biases. Based on the experiments over different standard benchmark airborne datasets and real-world UAV datasets, the developed approaches outperformed the prominent hyperspectral panchromatic sharpening, image fusion, and domain adaptation approaches. By using specific constraints and regularizations, the network developed was less sensitive to network parameters, unlike in similar implementations. The proposed approach illustrated improved generalizability in comparison with the prominent existing approaches. In addition to the fusion-based classification of the multispectral and hyperspectral datasets, the proposed approach was extended to the classification of hyperspectral airborne datasets where the latent graph generation and convolution were employed to resolve the domain bias with a small number of training samples. Overall, the developed transformations and architectures will be useful for the semantic interpretation and analysis of multimodal data and are applicable to signal processing, manifold learning, video analysis, data mining, and time series analysis, to name a few.     

Bridging systems theory and data science: A unifying review of dynamic latent variable analytics and process monitoring

This paper is concerned with data science and analytics as applied to data from dynamic systems for the purpose of monitoring, prediction, and inference. Collinearity is inevitable in industrial operation data. Therefore, we focus on latent variable methods that achieve dimension reduction and collinearity removal. We present a new dimension reduction expression of state space framework to unify dynamic latent variable analytics for process data, dynamic factor models for econometrics, subspace identification of multivariate dynamic systems, and machine learning algorithms for dynamic feature analysis. We unify or differentiate them in terms of model structure, objectives with constraints, and parsimony of parameterization. The Kalman filter theory in the latent space is used to give a system theory foundation to some empirical treatments in data analytics. We provide a unifying review of the connections among the dynamic latent variable methods, dynamic factor models, subspace identification methods, dynamic feature extractions, and their uses for prediction and process monitoring. Both unsupervised dynamic latent variable analytics and the supervised counterparts are reviewed. Illustrative examples are presented to show the similarities and differences among the analytics in extracting features for prediction and monitoring.     

Transfer latent variable model based on divergence analysis

Latent variable models are powerful dimensionality reduction approaches in machine learning and pattern recognition. However, this kind of methods only works well under a necessary and strict assumption that the training samples and testing samples are independent and identically distributed. When the samples come from different domains, the distribution of the testing dataset will not be identical with the training dataset. Therefore, the performance of latent variable models will be degraded for the reason that the parameters of the training model do not suit for the testing dataset. This case limits the generalization and application of the traditional latent variable models. To handle this issue, a transfer learning framework for latent variable model is proposed which can utilize the distance (or divergence) of the two datasets to modify the parameters of the obtained latent variable model. So we do not need to rebuild the model and only adjust the parameters according to the divergence, which will adopt different datasets. Experimental results on several real datasets demonstrate the advantages of the proposed framework.     

半监督学习研究的述评

监督学习需要利用大量的标记样本训练模型,但实际应用中,标记样本的采集费时费力。无监督学习不使用先验信息,但模型准确性难以保证。半监督学习突破了传统方法只考虑一种样本类型的局限,能够挖掘大量无标签数据隐藏的信息,辅助少量的标记样本进行训练,成为机器学习的研究热点。通过对半监督学习研究的总趋势以及具体研究内容进行详细的梳理与总结,分别从半监督聚类、分类、回归与降维以及非平衡数据分类和减少噪声数据共六个方面进行综述,发现半监督方法众多,但存在以下不足：（1）部分新提出的方法虽然有效,但仅通过特定数据集进行了实证,缺少一定的理论证明;（2）复杂数据下构建的半监督模型参数较多,结果不稳定且缺乏参数选取的指导经验;（3）监督信息多采用样本标签或成对约束形式,对混合约束的半监督学习需要进一步研究;（4）对半监督回归的研究匮乏,对如何利用连续变量的监督信息研究甚少。     

Partially supervised Independent Factor Analysis using soft labels elicited from multiple experts: application to railway track circuit diagnosis

Using a statistical model in a diagnosis task generally requires a large amount of labeled data. When ground truth information is not available, too expensive or difficult to collect, one has to rely on expert knowledge. In this paper, it is proposed to use partial information from domain experts expressed as belief functions. Expert opinions are combined in this framework and used with measurement data to estimate the parameters of a statistical model using a variant of the EM algorithm. The particular application investigated here concerns the diagnosis of railway track circuits. A noiseless Independent Factor Analysis model is postulated, assuming the observed variables extracted from railway track inspection signals to be generated by a linear mixture of independent latent variables linked to the system component states. Usually, learning with this statistical model is performed in an unsupervised way using unlabeled examples only. In this paper, it is proposed to handle this learning process in a soft-supervised way using imperfect information on the system component states. Fusing partially reliable information about cluster membership is shown to significantly improve classification results.     

混合高斯变分自编码器的聚类网络

目的 经典的聚类算法在处理高维数据时存在维数灾难等问题,使得计算成本大幅增加并且效果不佳。以自编码或变分自编码网络构建的聚类网络改善了聚类效果,但是自编码器提取的特征往往比较差,变分自编码器存在后验崩塌等问题,影响了聚类的结果。为此,本文提出了一种基于混合高斯变分自编码器的聚类网络。方法 使用混合高斯分布作为隐变量的先验分布构建变分自编码器,并以重建误差和隐变量先验与后验分布之间的KL散度(Kullback-Leibler divergence)构造自编码器的目标函数训练自编码网络;以训练获得的编码器对输入数据进行特征提取,结合聚类层构建聚类网络,以编码器隐层特征的软分配分布与软分配概率辅助目标分布之间的KL散度构建目标函数并训练聚类网络;变分自编码器采用卷积神经网络实现。结果 为了验证本文算法的有效性,在基准数据集MNIST(Modified National Institute of Standards and Technology Database)和Fashion-MNIST上评估了该网络的性能,聚类精度(accuracy,ACC)和标准互信息(normalized mutua...     

一种孤立点挖掘的混合核方法

孤立点是不具备数据一般特性的数据对象。支持向量机(SVM)将数据点映射到高维特征空间,通过划分最大间隔的超平面来分离孤立点和正常点。利用支持向量机在处理小样本、高维数及泛化性能强等方面的优势,提出了一种新的基于高斯过程潜变量模型(GPLVM)和支持向量分类的检测模型算法。利用GPLVM提供潜变量到数据空间的平滑概率映射实现数据降维,然后通过SVM交叉验证进行孤立点检测。在KDD99数据集上进行了仿真实验,数值结果表明该算法在保证低误报率的前提下能有效地提高检测率,证明了方法的有效性。     

Unsupervised learning in neural computation

In this article, we consider unsupervised learning from the point of view of applying neural computation on signal and data analysis problems. The article is an introductory survey, concentrating on the main principles and categories of unsupervised learning. In neural computation, there are two classical categories for unsupervised learning methods and models: first, extensions of principal component analysis and factor analysis, and second, learning vector coding or clustering methods that are based on competitive learning. These are covered in this article. The more recent trend in unsupervised learning is to consider this problem in the framework of probabilistic generative models. If it is possible to build and estimate a model that explains the data in terms of some latent variables, key insights may be obtained into the true nature and structure of the data. This approach is also briefly reviewed.     

Nonlinear model and constrained ML for removing back-to-front interferences from recto–verso documents

In this paper, we approach the removal of back-to-front interferences from scans of double-sided documents as a blind source separation problem, and extend our previous linear mixing model to a more effective nonlinear mixing model. We consider the front and back ideal images as two individual patterns overlapped in the observed recto and verso scans, and apply an unsupervised constrained maximum likelihood technique to separate them. Through several real examples, we show that the results obtained by this approach are much better than the ones obtained through data decorrelation or independent component analysis. As compared to approaches based on segmentation/classification, which often aim at cleaning a foreground text by removing all the textured background, one of the advantages of our method is that cleaning does not alter genuine features of the document, such as color or other structures it may contain. This is particularly interesting when the document has a historical importance, since its readability can be improved while maintaining the original appearance.     

Modelling financial time series based on heavy-tailed market microstructure models with scale mixtures of normal distributions

This paper presents a type of heavy-tailed market microstructure models with the scale mixtures of normal distributions (MM-SMN), which include two specific sub-classes, viz. the slash and the Student-t distributions. Under a Bayesian perspective, the Markov Chain Monte Carlo (MCMC) method is constructed to estimate all the parameters and latent variables in the proposed MM-SMN models. Two evaluating indices, namely the deviance information criterion (DIC) and the test of white noise hypothesis on the standardised residual, are used to compare the MM-SMN models with the classic normal market microstructure (MM-N) model and the stochastic volatility models with the scale mixtures of normal distributions (SV-SMN). Empirical studies on daily stock return data show that the MM-SMN models can accommodate possible outliers in the observed returns by use of the mixing latent variable. These results also indicate that the heavy-tailed MM-SMN models have better model fitting than the MM-N model, and the market microstructure model with slash distribution (MM-s) has the best model fitting. Finally, the two evaluating indices indicate that the market microstructure models with three different distributions are superior to the corresponding stochastic volatility models.     

一种任意分布下的隐变量因果结构学习算法

因果发现旨在通过观测数据挖掘变量间的因果关系,在实际应用中需要从观测数据中学习隐变量间的因果结构。现有方法主要利用观测变量间的协方差信息（如四分体约束）或引入非高斯假设（如三分体约束）来解决线性因果模型下的隐变量结构学习问题,但大多限定于分布明确的情况,而实际应用环境往往并不满足这种假设。给出任意分布下隐变量结构的识别性证明,指出在没有混淆因子影响的情况下,两个隐变量的因果方向可识别所需要的最小条件是仅需要其中一个隐变量的噪声服从非高斯分布。在此基础上,针对线性隐变量模型提出一种在任意分布下学习隐变量因果结构的算法,先利用四分体约束方法学习得到隐变量骨架图,再通过枚举骨架图的等价类并测量每一个等价类中的三分体约束来学习因果方向,同时将非高斯约束放宽到尽可能最小的变量子集,从而扩展线性隐变量模型的应用范围。实验结果表明,与MIMBuild和三分体约束方法相比,该算法得到了最佳的F1值,能够在任意分布下学习更多的隐变量因果结构信息,且具有更强的鲁棒性。     

Data augmentation strategies for the Bayesian spatial probit regression model

The well known latent variable representation of the Bayesian probit regression model due to Albert and Chib (1993) allows model fitting to be performed using a simple Gibbs sampler. In addition, various types of dependence among categorical outcomes not explained by covariate information can be accommodated in a straightforward manner as a result of this latent variable representation of the model. One example of this is the spatial probit regression model for spatially-referenced categorical outcomes. In this setting, commonly used covariance structures for describing residual spatial dependence in the normal linear model setting can be imbedded into the probit regression model. Capturing spatial dependence in this way, however, can negatively impact the performance of MCMC model-fitting algorithms, particularly in terms of mixing and sensitivity to starting values. To address these computational issues, we demonstrate how the non-identifiable spatial variance parameter can be used to create data augmentation MCMC algorithms. We compare the performance of several non-collapsed and partially collapsed data augmentation MCMC algorithms through a simulation study and an analysis of land cover data.     

面向不完整多视图聚类的深度互信息最大化方法

多视图聚类是无监督学习领域研究热点之一,近年来涌现出许多优秀的多视图聚类工作,但其中大多数方法均假设各视图是完整的,然而真实场景下数据收集过程极容易发生缺失,造成部分视图不完整。同时,很多方法采取传统机器学习方法（即浅层模型）对数据进行特征学习,这导致模型难以挖掘高维数据内的复杂信息。针对以上问题,本文提出一种面向不完整多视图聚类的深度互信息最大化方法。首先利用深度自编码器挖掘各视图深层次的隐含特征,并通过最大化潜在表示间的互信息来学习各视图间的一致性知识。然后,对于不完整视图中的缺失数据,利用多视图的公共潜在表示进行补全。此外,本文采用一种自步学习策略对网络进行微调,从易到难地学习数据集中的样本,得到更加宜于聚类的特征表示。最后,在多个真实数据集上进行实验,验证了本文方法的有效性。     

非齐次隐马尔可夫因子模型期望最大化算法

潜变量模型在刻画因子之间的相互关系以及因子与观测变量之间的关联性时具有重要作用。在实际应用中,观测数据往往呈现出时序变异、多峰、偏态等特性,因此将经典的潜变量模型延伸到非齐次隐马尔可夫潜变量模型,并且为避免对完全数据的积分计算,将期望最大化（expectation-maximization,EM）算法引入到似然函数的计算上;采用Akaike信息准则和Bayes信息准则选择合适的模型,提出了相应的统计计算和检验方法,有效解决了隐马尔可夫模型中的最大估算似然函数问题;最后选择心理．健康数据进行了实验,实验结果表明该方法是有效的。     

基于数据非随机缺失机制的推荐系统托攻击探测

协同过滤推荐系统极易受到托攻击的侵害. 开发托攻击探测技术已成为保障推荐系统可靠性与鲁棒性的关键. 本文以数据非随机缺失机制为依托,对导致评分缺失的潜在因素进行解析, 并在概率产生模型框架内将这些潜在因素与Dirichlet过程相融合, 提出了用于托攻击探测的缺失评分潜在因素分析(Latent factor analysis for missing ratings, LFAMR)模型. 实验表明,与现有探测技术相比, LFAMR具备更强的普适性和无监督性, 即使缺乏系统相关先验知识,仍可有效探测各种常见托攻击.     

Multivariate information bottleneck

The information bottleneck (IB) method is an unsupervised model independent data organization technique. Given a joint distribution, p(X, Y), this method constructs a new variable, T, that extracts partitions, or clusters, over the values of X that are informative about Y. Algorithms that are motivated by the IB method have already been applied to text classification, gene expression, neural code, and spectral analysis. Here, we introduce a general principled framework for multivariate extensions of the IB method. This allows us to consider multiple systems of data partitions that are interrelated. Our approach utilizes Bayesian networks for specifying the systems of clusters and which information terms should be maintained. We show that this construction provides insights about bottleneck variations and enables us to characterize the solutions of these variations. We also present four different algorithmic approaches that allow us to construct solutions in practice and apply them to several real-world problems.     

Dimensionality reduction of clustered data sets

We present a novel probabilistic latent variable model to perform linear dimensionality reduction on data sets which contain clusters. We prove that the maximum likelihood solution of the model is an unsupervised generalisation of linear discriminant analysis. This provides a completely new approach to one of the most established and widely used classification algorithms. The performance of the model is then demonstrated on a number of real and artificial data sets.