基于门控联合池化自编码器的通用性文本表征

为了学习文本的语义表征,以往的研究者主要依赖于复杂的循环神经网络(recurrent neural networks,RNNs)和监督式学习方法。该文提出了一种门控联合池化自编码器(gated mean-max AAE)用于学习中英文的文本语义表征。该文的自编码器完全通过多头自注意力机制(multi-head self-attention mechanism)来构建编码器和解码器网络。在编码阶段,提出了均值—最大化(mean-max)联合表征策略,即同时运用平均池化(mean pooling)和最大池化(max pooling)操作来捕获输入文本中多样性的语义信息。为促使联合池化表征可以全面地指导重构过程,解码器采用门控操作进行动态关注。通过在大规模中英文未标注语料上训练模型,获得了高质量的句子编码器。在重构文本段落的实验中,该文模型在实验效果和计算效率上均超越了传统的RNNs模型。将公开训练好的文本编码器,使其可以方便地运用于后续的研究。

Gated Mean-Max Autoencoder for Text Representations

In order to learn distributed representations of text sequences, the previous methods focus on complex recurrent neural networks or supervised learning. In this paper, we propose a gated mean-max autoencoder both for Chinese and English text representations. In our model, we simply rely on the multi-head self-attention mechanism to construct the encoder and decoder. In the encoding we propose a mean-max strategy that applies both mean and max pooling operations over the hidden vectors to capture diverse information of the input. To enable the information to steer the reconstruction process, the decoder employ element-wise gate to select between mean and max representations dynamically. By training our model on a large amount of Chinese and English un-labelled data respectively, we obtain high-quality text encoders for publicl available. Experimental results of reconstructing coherent long texts from the encoded representations demonstrate the superiority of our model over the traditional recurrent neural network, in terms of both performance and complexity.