基于模态相关性学习的细粒度分类

针对单模态细粒度分类方法难以区分图像间细微差异的问题，将多模态融合方法引入到细粒度分类任务中，充分利用多模态数据的相关性和互补性，提出了一种基于模态相关性学习的细粒度分类方法。该方法分为两个阶段，首先考虑到图像和文本数据之间的对应关系，利用它们的匹配程度作为约束来进行模型的预训练；接着，加载上一步得到的网络参数，先提取多模态特征，再利用文本特征指导图像特征的生成；最后，基于融合后的特征进行细粒度分类。该方法在UPMC-Food101、MEP-3M-MEATS和MEP-3M-OUTDOORS数据集上进行训练测试，分别达到91.13%、82.39%和93.17%的准确率。实验结果表明，该方法相对于传统的多模态融合方法具有更好的性能，是一种有效的细粒度分类方法。

Fine-grained classification based on modal correlation learning

To address the problem of difficulty in distinguishing subtle differences between images in single-modal fine-grained classification methods, this paper introduced a multimodal fusion approach into the task of fine-grained classification. By fully utilizing the correlations and complementarity of multimodal data, this paper proposed a modality correlation learning-based fine-grained classification method. The method consisted of two stages. Firstly, considering the correspondence between image and text data, it used their matching degree as a constraint for model pretraining. Subsequently, with the loaded network parameters from the previous step, it first extracted multimodal features, and followed by utilizing text features to guide the generation of image features. Finally, it performed fine-grained classification based on the fused features. The method was trained and tested on the UPMC-Food101, MEP-3M-MEATS, and MEP-3M-OUTDOORS datasets, achieving accuracies of 91.13%, 82.39%, and 93.17%, respectively. Experimental results demonstrate that this method outperforms traditional multimodal fusion methods, making it an effective fine-grained classification approach.