Generating Material-Aware 3D Models from Sparse Views

Image-to-3D diffusion models have significantly advanced 3D content generation. However, existing methods often struggle to disentangle material and illumination from coupled appearance, as they primarily focus on modeling geometry and appearance. This paper introduces a novel approach to generate material-aware 3D models from sparse-view images using generative models and efficient pre-integrated rendering. The output of our method is a relightable model that independently models geometry, material, and lighting, enabling downstream tasks to manipulate these components separately. To fully leverage information from limited sparse views, we propose a mixed supervision framework that simultaneously exploits view-consistency via captured views and diffusion prior via generating views. Additionally, a view selection mechanism is proposed to mitigate the degenerated diffusion prior. We adapt an efficient yet powerful pre-integrated rendering pipeline to factorize the scene into a differentiable environment illumination, a spatially varying material field, and an implicit SDF field. Our experiments on both real-world and synthetic datasets demonstrate the effectiveness of our approach in decomposing each component as well as manipulating the illumination. Source codes are available at https://github.com/Sheldonmao/MatSparse3D.