Towards adaptive learning with improved convergence of deep belief networks on graphics processing units |
| |
Authors: | Noel Lopes Bernardete Ribeiro |
| |
Affiliation: | 1. CISUC – Center for Informatics and Systems of University of Coimbra, Portugal;2. UDI/IPG – Research Unit, Polytechnic of Guarda, Portugal;3. Department of Informatics Engineering, University of Coimbra, Portugal |
| |
Abstract: | In this paper we focus on two complementary approaches to significantly decrease pre-training time of a deep belief network (DBN). First, we propose an adaptive step size technique to enhance the convergence of the contrastive divergence (CD) algorithm, thereby reducing the number of epochs to train the restricted Boltzmann machine (RBM) that supports the DBN infrastructure. Second, we present a highly scalable graphics processing unit (GPU) parallel implementation of the CD-k algorithm, which boosts notably the training speed. Additionally, extensive experiments are conducted on the MNIST and the HHreco databases. The results suggest that the maximum useful depth of a DBN is related to the number and quality of the training samples. Moreover, it was found that the lower-level layer plays a fundamental role for building successful DBN models. Furthermore, the results contradict the pre-conceived idea that all the layers should be pre-trained. Finally, it is shown that by incorporating multiple back-propagation (MBP) layers, the DBNs generalization capability is remarkably improved. |
| |
Keywords: | Deep learning Deep belief networks Restricted Boltzmann machines Contrastive divergence Adaptive step size GPU computing |
本文献已被 ScienceDirect 等数据库收录! |
|