WildFusion

3D-Aware Image Synthesis with Latent Diffusion Models in View Space

While existing 3D-aware generative models achieve high photorealism and 3D-consistent viewpoint control, the vast majority of approaches only consider single-class and aligned data like human faces or cat faces. We identify two main causes for this:

• Existing approaches assume a shared canonical coordinate system to represent 3D objects. As a consequence, they require either poses from an off-the-shelf pose estimator or assume, and sometimes learn to refine, a given pose distribution. In contrast, in-the-wild images typically have no clearly defined canonical camera system and camera poses or pose distributions are not available or very challenging to obtain.
• Most existing 3D-aware generative models are generative adversarial networks (GANs). We find that for complex datasets with challenging pose distributions GANs struggle with distribution coverage and suffer from mode collapse.

We hence propose WildFusion, a new approach to 3D-aware image synthesis based on latent diffusion models (LDMs) that addresses these limitations. We train our approach in view space which removes the need for camera poses and a priori camera pose distributions, unlocking 3D-aware image synthesis on unaligned, diverse datasets. To ensure distribution coverage on more diverse datasets, we build our approach upon latent diffusion models (LDMs) instead of GANs.

Our 3D-aware LDM, called WildFusion, follows LDMs’ two-stage approach: First, we train a powerful 3D-aware autoencoder from large collections of unposed images without multiview supervision that simultaneously performs both compression and enables novel-view synthesis. The autoencoder is trained with pixel-space reconstruction losses on the input views and uses adversarial training to supervise novel views. Note that by using adversarial supervision for the novel views, our autoencoder is trained for novel-view synthesis without the need for multiview supervision. Adding monocular depth cues helps the model learn a faithful 3D representation and further improves novel-view synthesis. In the second stage, we train a diffusion model in the compressed and 3D-aware latent space, which enables us to synthesize novel samples and turns the novel-view synthesis system, i.e., our autoencoder, into a 3D-aware generative model.

Technical Contributions

• We remove the need for posed images and a priori camera pose distributions for 3D-aware image synthesis by modeling instances in view space instead of canonical space.
• We learn a powerful 3D-aware autoencoder from unposed images without multiview supervision that simultaneously performs compression, while inferring a 3D representation suitable for novel-view synthesis.
• We show that our novel 3D-aware LDM, WildFusion, enables high-quality 3D-aware image synthesis with reasonable geometry and strong distribution coverage, achieving state-of-the-art performance in the unposed image training setting, which corresponds to training on in-the-wild image data.

Baseline Comparison

We find that existing GAN-based approaches struggle with very low sample diversity (mode collapse) on multi-modal datasets with complex camera distributions like ImageNet. The results below show samples from WildFusion and 3DGP, the strongest baseline, where each row corresponds to samples of one class. While 3DGP collapses and produces almost identical samples within classes, WildFusion produces diverse, high-quality samples because it builds upon Latent Diffusion Models.

Autoencoder for Compression and Novel View Synthesis

Our 3D-aware autoencoder performs both compression and enables novel-view synthesis. Notably, it is trained from large collections of unposed images without any direct multiview supervision. The learned compressed 3D-aware latent space can then be used to train a latent diffusion model. In addition, we can leverage our autoencoder to more efficiently perform novel view synthesis for a single given image than common GAN-based methods relying on GAN-inversion. We show pairs of input images and synthesized novel views from our autoencoder below.

Generated Samples

We train a latent diffusion model on the compressed 3D-aware latent space of the 3D-aware autoencoder. Our 3D-aware LDM enables high-quality 3D-aware image synthesis with reasonable geometry and strong distribution coverage / high sample diversity.

Interpolation

Using WildFusion, we can interpolate in a semantically meaningful way between two given single images while simultaneously allowing to change the viewpoint. Note that the geometry also changes accordingly. Specifically, we encode two images into latent space, further encode into the diffusion model’s Gaussian prior space (inverse DDIM), interpolate the resulting encodings, and generate the corresponding 3D images along the interpolation path.

Generative Resampling with Different Noise Levels

We can further use WildFusion to perform 3D-aware generative image resampling. Given an image, we forward diffuse its latent encoding for varying numbers of steps and re-generate from the partially noised encodings. Depending on how far we diffuse, we control how strongly the sample adheres to the input image. For the samples below, we gradually increase the number of diffusion steps from left to right.

Citation