基于帧间一致性的自监督室内逆渲染

针对目前逆渲染监督学习方法难以获得标签、泛化能力差的问题，提出了一种基于IFC（inter-frame coherence）的自监督训练方法。由于逆渲染问题的不适定性，引入额外的反照率一致性损失和交叉渲染损失强化自监督网络，其主要思想是对连续光照变化的图像序列执行IFC约束。即通过图像帧之间的位姿图和深度图，在相邻帧之间执行图像投影和扭曲；通过这种方法在相邻帧之间建立约束，并使用孪生训练来确保对光度不变量的一致估计。该方法使用完全卷积神经网络从室内视频序列中恢复几何形状、反射率和光照。自监督网络使用没有标签的连续帧图像集合进行训练，通过结合可微分渲染器，使网络以自监督的方式进行学习。通过与其他主流方法的比较，定量和定性实验结果表明提出方法在多个基准上表现更优。

Indoor inverse rendering from video based on inter-frame coherence self-supervision

This paper proposed a self-supervised training method based on inter-frame consistency to solve the problem that the current inverse rendering supervised learning method is challenging to obtain labels and has poor generalization ability. Due to the ill-posed nature of the inverse rendering problem, this paper introduced additional albedo consistency loss and cross-rendering loss to strengthen the self-supervised network, the main idea of which is to enforce inter-frame consistency constraints on image sequences with continuous illumination changes. The method performed image projection and warping between adjacent frames through pose maps and depth maps between image frames. This method established constraints between adjacent frames and used siamese training to ensure photometric invariance consensus estimate. This paper used a fully convolutional neural network to recover geometry, reflectivity, and illumination from indoor video sequences. The method trained the self-supervised network using a collection of unlabeled consecutive frame images and incorporating a differentiable renderer, making the network learn in a self-supervised manner. Compared with other mainstream methods, quantitative and qualitative experimental results show that the proposed method performs better on multiple benchmarks.