基于跨模态空间匹配的多模态肺部肿块分割网络

现有多模态分割方法通常先对图像进行配准，再对配准后的图像进行分割。对于成像特点差异较大的不同模态，两阶段的结构匹配与分割算法下的分割精度较低。针对该问题，该文提出一种基于跨模态空间匹配的多模态肺部肿块分割网络(MMSASegNet)，其具有模型复杂度低和分割精度高的特点。该模型采用双路残差U型分割网络作为骨干分割网络，以充分提取不同模态输入特征，利用可学习的空间变换网络对其输出的多模态分割掩膜进行空间结构匹配；为实现空间匹配后的多模态特征图融合，形变掩膜和参考掩膜分别与各自模态相同分辨率的特征图进行矩阵相乘，并经特征融合模块，最终实现多模态肺部肿块分割。为提高端到端多模态分割网络的分割性能，采用深度监督学习策略，联合损失函数约束肿块分割、肿块空间匹配和特征融合模块，同时采用多阶段训练以提高不同功能模块的训练效率。实验数据采用T2权重(T2W)磁共振图像和扩散权重磁共振图像(DWI)肺部肿块分割数据集，该方法与其他多模态分割网络相比，DSC (Dice Similarity Coefficient)和HD (Hausdorff Distance)等评价指标均显著提高。

Multi-Modal Pulmonary Mass Segmentation Network Based on Cross-Modal Spatial Alignment

Most of the existing multi-modal segmentation methods are adopted on the co-registered multi-modal images. However, these two-stage algorithms of the segmentation and the registration achieve low segmentation performance on the modalities with remarkable spatial misalignment. To solve this problem, a cross-modal Spatial Alignment based Multi-Modal pulmonary mass Segmentation Network (MMSASegNet) with low model complexity and high segmentation accuracy is proposed. Dual-path Res-UNet is adopted as the backbone segmentation architecture of the proposed network for the better multi-modal feature extraction. Spatial Transformer Networks (STN) is applied to the segmentation masks from two paths to align the spatial information of mass region. In order to realize the multi-modal feature fusion based on the spatial alignment on the region of mass, the deformed mask and the reference mask are matrix-multiplied by the feature maps of each modality respectively. Further, the yielding cross-modality spatially aligned feature maps from multiple modalities are fused and learned through the feature fusion module for the multi-modal mass segmentation. In order to improve the performance of the end-to-end multi-modal segmentation network, deep supervision learning strategy is employed with the joint cost function constraining mass segmentation, mass spatial alignment and feature fusion. Moreover, the multi-stage training strategy is adopted to improve the training efficiency of each module. On the pulmonary mass datasets containing T2-Weighted-MRI（T2W） and Diffusion-Weighted-MRI Images（DWI）, the proposed method achieved improvement on the metrics of Dice Similarity Coefficient (DSC) and Hausdorff Distance (HD).