A lighting robust fitting approach of 3D morphable model for face reconstruction

Three-dimensional morphable model (3DMM) is a powerful tool for recovering 3D shape and texture from a single facial image. The success of 3DMM relies on two things: an effective optimization strategy and a realistic approach to synthesizing face images. However, most previous methods have focused on developing an optimization strategy under Phong’s synthesis approach. In this paper, we adopt a more realistic synthesis technique that fully considers illumination and reflectance in the 3DMM fitting process. Using the sphere harmonic illumination model (SHIM), our new synthesis approach can account for more lighting factors than Phong’s model. Spatially varying specular reflectance is also introduced into the synthesis process. Under SHIM, the cost function is nearly linear for all parameters, which simplifies the optimization. We apply our new optimization algorithm to determine the shape and texture parameters simultaneously. The accuracy of the recovered shape and texture can be improved significantly by considering the spatially varying specular reflectance. Hence, our algorithm produces an enhanced shape and texture compared with previous SHIM-based methods that recover shape from feature points. Although we use just a single input image in a profile pose, our approach gives plausible results. Experiments on a well-known image database show that, compared to state-of-the-art methods based on Phong’s model, the proposed approach enhances the robustness of the 3DMM fitting results under extreme lighting and profile pose.