Exchange Monte Carlo Sampling From Bayesian Posterior for Singular Learning Machines

Many singular learning machines such as neural networks, normal mixtures, Bayesian networks, and hidden Markov models belong to singular learning machines and are widely used in practical information systems. In these learning machines, it is well known that Bayesian learning provides better generalization performance than maximum-likelihood estimation. However, it needs huge computational cost to sample from a Bayesian posterior distribution of a singular learning machine by a conventional Markov chain Monte Carlo (MCMC) method, such as the metropolis algorithm, because of singularities. Recently, the exchange Monte Carlo (MC) method, which is well known as an improved algorithm of MCMC method, has been proposed to apply to Bayesian neural network learning in the literature. In this paper, we propose the idea that the exchange MC method has a better effect on Bayesian learning in singular learning machines than that in regular learning machines, and show its effectiveness by comparing the numerical stochastic complexity with the theoretical one.