受限玻尔兹曼机研究综述

概率图模型是目前机器学习研究的热点,基于概率图模型构造的生成模型已广泛应用于图像和语音处理等领域.受限玻尔兹曼机（restricted Boltzmann machines,简称RBMs）是一种概率无向图,在建模数据分布方面有重要的研究价值,RBMs既可以结合卷积算子构造深度判别模型,为深度网络提供统计力学的理论支持,也可以结合有向图构建生成模型,提供具有多峰分布的先验信息.主要综述了以RBMs为基础的概率图模型的相关研究.首先介绍了基于RBMs的机器学习模型的基本概念和训练算法,并讨论了基于极大似然估计的各训练算法的联系,比较了各算法的log似然损失;其次,综述了RBMs模型最新的研究进展,包括在目标函数中引入对抗损失和W距离,并构造基于RBMs先验的变分自编码模型（variational autoencoders,简称VAEs）、基于对抗损失的RBMs模型,并讨论了各实值RBMs模型之间的联系和区别;最后,综述了以RBMs为基础的模型在深度学习中的应用,并讨论了神经网络和RBMs模型在研究中存在的问题及未来的研究方向.     

受限玻尔兹曼机的稀疏化特征学习

受限玻尔兹曼机(RBM)作为深度学习算法的一种基础模型被广泛应用,但传统RBM算法没有充分考虑数据的稀疏化特征学习,使得算法性能受数据集的稀疏性影响较大。提出一种RBM稀疏化特征学习方法(sRBM),通过归一化的输入数据均值确定数据集的稀疏系数,将稀疏系数大于阈值的稠密数据集自动转化为稀疏数据集,在不损失信息量的情况下实现输入数据的稀疏化。在手写字符数据集和自然图像数据集上的实验结果表明,sRBM通过输入数据稀疏化有效提升了RBM的稀疏化特征学习性能。     

融合标签的实值条件受限波尔兹曼机推荐算法

针对推荐算法中数据的稀疏性难题,把用户标签融合至实值条件受限玻尔兹曼机(real-valued conditional restricted Boltzmann machine,R_CRBM)模型,利用R_CRBM强大的拟合任意离散分布的能力,预测出用户对未交互商品的评分缺失值。具体来说,首先提出显层单元为实值的R_CRBM模型,接着运用文本分类中的TF-IDF算法预测出用户对所应用过的标签的喜爱度,与标签基因数据相乘得到用户对商品的预测评分,融合至用户历史评分数据中。R_CRBM条件层在原有评分/未评分{0,1}向量中,融入用户标签/未标签{0,1}向量。通过真实数据集进行对比分析,实验结果表明提出的方法在一定程度上提升了推荐的准确性。     

基于受限玻尔兹曼机的疲劳脑电特性分析

为了准确提取和分类视觉疲劳所引起的脑电特征,以此提醒过度用眼的工作人员及时休息,提出了多通道受限玻尔兹曼机算法和卷积神经网络(CNN)算法结合的深度学习混合模型,利用该模型对枕叶区10个通道的脑电信号进行自动提取内在特征和分类。在基于SSVEP的视觉疲劳脑电数据集上进行评估,深度学习混合模型的平均准确率达到88. 63%,比传统的特征提取和分类方法高10%。实验结果证明了深度学习混合模型取得的分类效果较好,并且克服了传统手动提取特征方法不全面的不足,对疲劳脑电的研究具有现实的意义。     

基于云计算的受限玻尔兹曼机推荐算法研究

数据的指数级增长及算法本身的复杂性使受限玻尔兹曼机面临着计算效率的问题。在详细分析受限玻尔兹曼机的基础上,将受限玻尔兹曼机与Hadoop平台的并行计算架构相结合,提出基于云平台的受限玻尔兹曼机推荐算法。该算法通过复制机制解决数据相关性问题,并将传统的受限玻尔兹曼机过程分解为若干个Hadoop任务的循环,实现并行计算。实验结果表明,与在传统平台上的实现相比,基于Hadoop并行架构的受限玻尔兹曼机推荐算法在大体量数据集的条件下可大幅提高推荐计算效率。     

Learning Invariant Features Using Subspace Restricted Boltzmann Machine

The subspace restricted Boltzmann machine (subspaceRBM) is a third-order Boltzmann machine where multiplicative interactions are between one visible and two hidden units. There are two kinds of hidden units, namely, gate units and subspace units. The subspace units reflect variations of a pattern in data and the gate unit is responsible for activating the subspace units. Additionally, the gate unit can be seen as a pooling feature. We evaluate the behavior of subspaceRBM through experiments with MNIST digit recognition task and Caltech 101 Silhouettes image corpora, measuring cross-entropy reconstruction error and classification error.     

Experiments of Fast Learning with High Order Boltzmann Machines

This work reports the results obtained with the application of High Order Boltzmann Machines without hidden units to construct classifiers for some problems that represent different learning paradigms. The Boltzmann Machine weight updating algorithm remains the same even when some of the units can take values in a discrete set or in a continuous interval. The absence of hidden units and the restriction to classification problems allows for the estimation of the connection statistics, without the computational cost involved in the application of simulated annealing. In this setting, the learning process can be sped up several orders of magnitude with no appreciable loss of quality of the results obtained.     

Maximum Matching on Boltzmann Machines

The Boltzmann machine is one of widely used neural network models used to cope with difficult combinatorial optimisation problems. It has been used to find near optimum solutions to such hard problems as graph partitioning and the Travelling Salesman problem. However, very little is known about the time complexity of solving combinatorial optimisation problems on Boltzmann machines. This issue is important because it will help us better understand the power of Boltzmann machines in dealing with hard problems. This paper studies the time complexity of maximum matching in a graph on Boltzmann machines. It is shown that some widely-used Boltzmann machines cannot find a maximum matching in average time polynomial in the number of nodes of the graph although there are conventional deterministic algorithms which solve the problem in polynomial time. On the other hand, this paper also shows that a simpler model of Boltzmann machines, with the 'temperature' parameter fixed at some constant, can find a near maximum matching in polynomial average time.     

Boltzmann Machines Reduction by High-Order Decimation

Decimation is a common technique in statistical physics that is used in the context of Boltzmann machines (BMs) to drastically reduce the computational cost at the learning stage. Decimation allows to analytically evaluate quantities that should otherwise be statistically estimated by means of Monte Carlo (MC) simulations. However, in its original formulation, this method could only be applied to restricted topologies corresponding to sparsely connected neural networks. In this brief, we present a generalization of the decimation process and prove that it can be used on any BM, regardless of its topology and connectivity. We solve the Monk problem with this algorithm and show that it performs as well as the best classification methods currently available.     

Massively Parallel Probabilistic Reasoning with Boltzmann Machines

We present a method for mapping a given Bayesian network to a Boltzmann machine architecture, in the sense that the the updating process of the resulting Boltzmann machine model probably converges to a state which can be mapped back to a maximum a posteriori (MAP) probability state in the probability distribution represented by the Bayesian network. The Boltzmann machine model can be implemented efficiently on massively parallel hardware, since the resulting structure can be divided into two separate clusters where all the nodes in one cluster can be updated simultaneously. This means that the proposed mapping can be used for providing Bayesian network models with a massively parallel probabilistic reasoning module, capable of finding the MAP states in a computationally efficient manner. From the neural network point of view, the mapping from a Bayesian network to a Boltzmann machine can be seen as a method for automatically determining the structure and the connection weights of a Boltzmann machine by incorporating high-level, probabilistic information directly into the neural network architecture, without recourse to a time-consuming and unreliable learning process.     

基于特征提取和极限学习机的软测量方法

针对建模数据存在的高维、共线性等特征,以及常用的基于人工智能的建模方法存在的模型结构难以确定、学习速度慢等缺点,提出了由基于主元分析(PCA)的特征提取和基于优化极限学习机(OELM)的建模算法两部分组成的软测量方法.采用PCA消除输入变量间的共线性并降低输入变量维数,以提取的线性无关的独立变量作为软测量模型的输入,从而简化模型结构.采用集成极限学习机(ELM)与支持向量机(SVM)算法优点的OELM方法作为建模算法,避免了ELM模型的随机性和SVM模型求解的复杂性.将特征提取方法与OELM方法结合后,提高了软测量模型的训练速度和预测性能.采用所述方法,对混凝土抗压强度的软测量问题进行了实验研究,验证了所提方法的有效性.该方法同时可以应用于基于雷达、光电等高维数据的目标识别,具有广阔的应用前景.     

Parallel Coarse Grain Computing of Boltzmann Machines

The resolution of combinatorial optimization problems can greatly benefit from the parallel and distributed processing which is characteristic of neural network paradigms. Nevertheless, the fine grain parallelism of the usual neural models cannot be implemented in an entirely efficient way either in general-purpose multicomputers or in networks of computers, which are nowadays the most common parallel computer architectures. Therefore, we present a parallel implementation of a modified Boltzmann machine where the neurons are distributed among the processors of the multicomputer, which asynchronously compute the evolution of their subset of neurons using values for the other neurons that might not be updated, thus reducing the communication requirements. Several alternatives to allow the processors to work cooperatively are analyzed and their performance detailed. Among the proposed schemes, we have identified one that allows the corresponding Boltzmann Machine to converge to solutions with high quality and which provides a high acceleration over the execution of the Boltzmann machine in uniprocessor computers.     

Learning to Walk by Imitation in Low-Dimensional Subspaces

In this paper, we provide the first demonstration that a humanoid robot can learn to walk directly by imitating a human gait obtained from motion capture (mocap) data without any prior information of its dynamics model. Programming a humanoid robot to perform an action (such as walking) that takes into account the robot's complex dynamics is a challenging problem. Traditional approaches typically require highly accurate prior knowledge of the robot's dynamics and environment in order to devise complex (and often brittle) control algorithms for generating a stable dynamic motion. Training using human mocap is an intuitive and flexible approach to programming a robot, but direct usage of mocap data usually results in dynamically unstable motion. Furthermore, optimization using high-dimensional mocap data in the humanoid full-body joint space is typically intractable. We propose a new approach to tractable imitation-based learning in humanoids without a robot's dynamic model. We represent kinematic information from human mocap in a low-dimensional subspace and map motor commands in this low-dimensional space to sensory feedback to learn a predictive dynamic model. This model is used within an optimization framework to estimate optimal motor commands that satisfy the initial kinematic constraints as best as possible while generating dynamically stable motion. We demonstrate the viability of our approach by providing examples of dynamically stable walking learned from mocap data using both a simulator and a real humanoid robot.     

基于非负矩阵分解的语音深层低维特征提取方法

作为一种基于深层神经网络提取的低维特征,瓶颈特征在连续语音识别中取得了很大的成功。然而训练瓶颈结构的深层神经网络时,瓶颈层的存在会降低网络输出层的帧准确率,进而反过来影响该特征的性能。针对这一问题,本文基于非负矩阵分解算法,提出一种利用不包含瓶颈层的深层神经网络提取低维特征的方法。该方法利用半非负矩阵分解和凸非负矩阵分解算法对隐含层权值矩阵分解得到基矩阵,将其作为新的特征层权值矩阵,然后在该层不设置偏移向量的情况下,通过数据前向传播提取新型特征。实验表明,该特征具有较为稳定的规律,且适用于不同的识别任务和网络结构。当使用训练数据充足的语料进行实验时,该特征表现出同瓶颈特征几乎相同的识别性能;而在低资源环境下,基于该特征识别系统的识别率明显优于深层神经网络混合识别系统和瓶颈特征识别系统。     

基于权值不确定性的玻尔兹曼机算法

受限制的玻尔兹曼机（RBM）是一种无向图模型.基于RBM的深度学习模型包括深度置信网（DBN）和深度玻尔兹曼机（DBM）等.在神经网络和RBM的训练过程中,过拟合问题是一个比较常见的问题.针对神经网络的训练,权值随机变量（weight random variables）、Dropout方法和早期停止方法已被用于缓解过拟合问题.首先,改变RBM模型中的训练参数,使用随机变量代替传统的实值变量,构建了基于随机权值的受限的波尔兹曼机（weight uncertainty RBM,简称WRBM）,接下来,在WRBM基础上构建了相应的深度模型：Weight uncertainty Deep Belief Network（WDBN）和Weight uncertainty Deep Boltzmann Machine（WDBM）,并且通过实验验证了WDBN和WDBM的有效性.最后,为了更好地建模输入图像,引入基于条件高斯分布的RBM模型,构建了基于spike-and-slab RBM（ssRBM）的深度模型,并通过实验验证了模型的有效性.     

A metaheuristic-driven approach to fine-tune Deep Boltzmann Machines

Deep learning techniques, such as Deep Boltzmann Machines (DBMs), have received considerable attention over the past years due to the outstanding results concerning a variable range of domains. One of the main shortcomings of these techniques involves the choice of their hyperparameters, since they have a significant impact on the final results. This work addresses the issue of fine-tuning hyperparameters of Deep Boltzmann Machines using metaheuristic optimization techniques with different backgrounds, such as swarm intelligence, memory- and evolutionary-based approaches. Experiments conducted in three public datasets for binary image reconstruction showed that metaheuristic techniques can obtain reasonable results.     

融入受限玻尔兹曼机的偏最小二乘优化方法

偏最小二乘法内部采用主成分分析,不能充分表达数据的非线性特征,对非线性数据的预测精度较低。为此,提出一种融合受限玻尔兹曼机与偏最小二乘的分析预测方法。该方法利用受限玻尔兹曼机对特征空间提取非线性结构,将提取的特征成分取代偏最小二乘中的成分,从而得到适应非线性的模型。实验结果表明,融合受限玻尔兹曼机与偏最小二乘法的分析方法能较好地反映数据的非线性特征。     

An Agent-Based Approach for Scheduling Multiple Machines

We present a new agent-based solution approach for the problem of scheduling multiple non-identical machines in the face of sequence dependent setups, job machine restrictions, batch size preferences, fixed costs of assigning jobs to machines and downstream considerations. We consider multiple objectives such as minimizing (weighted) earliness and tardiness, and minimizing job-machine assignment costs. We use an agent-based architecture called Asynchronous Team (A-Team), in which each agent encapsulates a different problem solving strategy and agents cooperate by exchanging results. Computational experiments on large instances of real-world scheduling problems show that the results obtained by this approach are significantly better than any single algorithm or the scheduler alone. This approach has been successfully implemented in an industrial scheduling system.