基于多模态信息融合的时间序列预测模型

针对传统单因子模型无法充分利用时间序列相关信息，以及这些模型对时间序列预测准确性和可靠性较差的问题，提出一种基于多模态信息融合的时间序列预测模型——Skip-Fusion对多模态数据中的文本数据和数值数据进行融合。首先利用BERT（Bidirectional Encoder Representations from Transformers）预训练模型和独热编码对不同类别的文本数据进行编码表示；再使用基于全局注意力机制的预训练模型获得多文本特征融合的单一向量表示；然后将得到的单一向量表示与数值数据按时间顺序对齐；最后通过时间卷积网络（TCN）模型实现文本和数值特征的融合，并通过跳跃连接完成多模态数据的浅层和深层特征的再次融合。在股票价格序列的数据集上进行实验，Skip-Fusion模型的均方根误差（RMSE）和日收益（R）分别为0.492和0.930，均优于现有的单模态模型和多模态融合模型的结果，同时在可决系数（R-Squared）上取得了0.955的拟合优度。实验结果表明，Skip-Fusion模型能够有效进行多模态信息融合并具有较高的预测准确性和可靠性。

Time series prediction model based on multimodal information fusion

Aiming at the problem that traditional single factor methods cannot make full use of the relevant information of time series and has the poor accuracy and reliability of time series prediction， a time series prediction model based on multimodal information fusion，namely Skip-Fusion， was proposed to fuse the text data and numerical data in multimodal data. Firstly， different types of text data were encoded by pre-trained Bidirectional Encoder Representations from Transformers （BERT） model and one-hot encoding. Then， the single vector representation of the multi-text feature fusion was obtained by using the pre-trained model based on global attention mechanism. After that， the obtained single vector representation was aligned with the numerical data in time order. Finally， the fusion of text and numerical features was realized through Temporal Convolutional Network （TCN） model， and the shallow and deep features of multimodal data were fused again through skip connection. Experiments were carried out on the dataset of stock price series， Skip-Fusion model obtains the results of 0.492 and 0.930 on the Root Mean Square Error （RMSE） and daily Return （R） respectively， which are better than the results of the existing single-modal and multimodal fusion models. Experimental results show that Skip-Fusion model obtains the goodness of fit of 0.955 on the R-squared， indicating that Skip-Fusion model can effectively carry out multimodal information fusion and has high accuracy and reliability of prediction.