InFusion: Inpainting 3D Gaussians via Learning Depth Completion from Diffusion Prior

3D Gaussians have recently emerged as an efficient representation for novelview synthesis. This work studies its editability with a particular focus onthe inpainting task, which aims to supplement an incomplete set of 3D Gaussianswith additional points for visually harmonious rendering. Compared to 2Dinpainting, the crux of inpainting 3D Gaussians is to figure out therendering-relevant properties of the introduced points, whose optimizationlargely benefits from their initial 3D positions. To this end, we propose toguide the point initialization with an image-conditioned depth completionmodel, which learns to directly restore the depth map based on the observedimage. Such a design allows our model to fill in depth values at an alignedscale with the original depth, and also to harness strong generalizability fromlargescale diffusion prior. Thanks to the more accurate depth completion, ourapproach, dubbed InFusion, surpasses existing alternatives with sufficientlybetter fidelity and efficiency under various complex scenarios. We furtherdemonstrate the effectiveness of InFusion with several practical applications,such as inpainting with user-specific texture or with novel object insertion.

Further reading#

• Access Paper in arXiv.org