Fast 3D-Aware Image Synthesis with Sparse Voxel Grids

Katja Schwarz Axel Sauer Michael Niemeyer Yiyi Liao Andreas Geiger
Max Planck Institute for Intelligent Systems, Zhejiang University and University of Tübingen
[Paper] [Code]

Abstract & Method

State-of-the-art 3D-aware generative models rely on coordinate-based MLPs to parameterize 3D radiance fields. While demonstrating impressive results, querying an MLP for every sample along each ray leads to slow rendering. Motivated by recent results in voxel-based novel view synthesis, we consider a sparse voxel grid representations for fast and 3D-consistent generative modeling. To obtain a compact representation of the scene and allow for scaling to higher voxel resolutions, our model disentangles the foreground object (modeled in 3D) from the background (modeled in 2D).

Responsive image

In contrast to existing approaches, our method requires only a single forward pass to generate a full 3D scene. It hence allows for efficient rendering from arbitrary viewpoints while yielding 3D consistent results with high visual fidelity.

3D-aware Image Synthesis

Sparse Voxel Grids

Background Removal

Latent Interpolation


            author = {Katja Schwarz and Axel Sauer and Michael Niemeyer and Yiyi Liao and Andreas Geiger},
            title = {VoxGRAF: Fast 3D-Aware Image Synthesis with Sparse Voxel Grids},
            journal = {ARXIV},
            year = {2022}