Text-guided image generation has progressed rapidly in recent years, inspiring major breakthroughs in text-guided shape generation. Recently, it has been shown that using score distillation, one can successfully text-guide a NeRF model to generate a 3D object. We adapt the score distillation to the publicly available, and computationally efficient, Latent Diffusion Models, which apply the entire diffusion process in a compact latent space of a pretrained autoencoder. As NeRFs operate in image space, a naïve solution for guiding them with latent score distillation would require encoding to the latent space at each guidance step. Instead, we propose to bring the NeRF to the latent space, resulting in a Latent-NeRF. Analyzing our Latent-NeRF, we show that while Text-to-3D models can generate impressive results, they are inherently unconstrained and may lack the ability to guide or enforce a specific 3D structure. To assist and direct the 3D generation, we propose to guide our Latent-NeRF using a Sketch-Shape: an abstract geometry that defines the coarse structure of the desired object. Then, we present means to integrate such a constraint directly into a Latent-NeRF. This unique combination of text and shape guidance allows for increased control over the generation process. We also show that latent score distillation can be successfully applied directly on 3D meshes. This allows for generating high-quality textures on a given geometry. Our experiments validate the power of our different forms of guidance and the efficiency of using latent rendering.
Official Implementation for "Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures".
TL;DR - We explore different ways of introducing shape-guidance for Text-to-3D and present three models: a purely text-guided Latent-NeRF, Latent-NeRF with soft shape guidance for more exact control over the generated shape, and Latent-Paint for texture generation for explicit shapes.
-
27.11.2022- Code release -
14.11.2022- Created initial repo
In the Latent-Paint application, a texture is generated for an explicit mesh directly on its texture map using stable-diffusion as a prior.
Here the geometry is used as a hard constraint where the generation process is tied to the given mesh and its parameterization.
Below we can see the progress of the generation process over the optimization process
To create such results, run the train_latent_paint script. Parameters are handled using pyrallis and can be passed from a config file or the cmd.
python -m scripts.train_latent_paint --config_path demo_configs/latent_paint/goldfish.yamlor alternatively
python -m scripts.train_latent_paint --log.exp_name 2022_11_22_goldfish --guide.text "A goldfish" --guide.shape_path /nfs/private/gal/meshes/blub.objHere we use a simple coarse geometry which we call a SketchShape to guide the generation process.
A SketchShape presents a soft constraint which guides the occupancy of a learned NeRF model but isn't constrained to its exact geometry.
A SketchShape can come in many forms, here are some extruded ones.
To create such results, run the train_latent_nerf script. Parameters are handled using pyrallis and can be passed from a config file or the cmd.
python -m scripts.train_latent_nerf --config_path demo_configs/latent_nerf/lego_man.yamlOr alternatively
python -m scripts.train_latent_nerf --log.exp_name '2022_11_25_lego_man' --guide.text 'a lego man' --guide.shape_path shapes/teddy.obj --render.nerf_type latentHere we apply a text-to-3D without any shape constraint similarly to dreamfusion and stable-dreamfusion.
We directly train the NeRF in latent space, so no encoding into the latent space is required during training.
To create such results, run the train_latent_nerf script. Parameters are handled using pyrallis and can be passed from a config file or the cmd.
python -m scripts.train_latent_nerf --config_path demo_configs/latent_nerf/sand_castle.yamlOr alternatively
python -m scripts.train_latent_nerf --log.exp_name 'sand_castle' --guide.text 'a highly detailed sand castle' --render.nerf_type latentAs our Latent-NeRF is supervised by Stable-Diffusion, we can also use Textual Inversion tokens as part of the input text prompt. This allows conditioning the object generation on specific objects and styles, defined only by input images.
For Textual-Inversion results use the guide.concept_name with a concept from the 🤗 concept library. For example --guide.concept_name=cat-toy and then simply use the corresponding token in your --guide.text
Install the common dependencies from the requirements.txt file
pip install -r requirements.txtFor Latent-NeRF with shape-guidance, additionally install igl
conda install -c conda-forge iglFor Latent-Paint, additionally install kaolin
pip install git+https://github.com/NVIDIAGameWorks/kaolinNote that you also need a 🤗 token for StableDiffusion. First accept conditions for the model you want to use, default one is CompVis/stable-diffusion-v1-4. Then, add a TOKEN file access token to the root folder of this project, or use the huggingface-cli login command
Scripts for training are available in the scripts/ folder, see above or in the demo_configs/ for some actual examples.
Meshes for shape-guidance are available under shapes/
-
Check out the
vis/trainto see the actual rendering used during the optimization. You might want to play around with theguide.mesh_scaleif the object looks too small or too large. -
For
Latent-NeRFwith shape-guidance try changingguide.proximal_surfaceandoptim.lambda_shapeto control the strictness of the guidance
| Path | Description |
|---|---|
| Repository root folder | |
| ├ demo_configs | Configs for running specific experiments |
| ├ scripts | The training scripts |
| ├ shapes | Various shapes to use for shape-guidance |
| ├ src | The actual code for training and evaluation |
| │ ├ latent_nerf | Code for Latent-NeRF training |
| │ │ ├ configs | Config structure for training |
| │ │ ├ models | NeRF models |
| │ │ ├ raymarching | The CUDA ray marching modules |
| │ │ ├ training | The Trainer class and related code |
| │ ├ latent_paint | Code for Latent-Paint training |
| │ │ ├ configs | Config structure for training |
| │ │ ├ models | Textured-Mesh models |
| │ │ ├ training | The Trainer class and related code |
The Latent-NeRF code is heavily based on the stable-dreamfusion project, and the Latent-Paint code borrows from text2mesh.
If you use this code for your research, please cite our paper Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures
@article{metzer2022latent,
title={Latent-NeRF for Shape-Guided Generation of 3D Shapes and Textures},
author={Metzer, Gal and Richardson, Elad and Patashnik, Or and Giryes, Raja and Cohen-Or, Daniel},
journal={arXiv preprint arXiv:2211.07600},
year={2022}
}
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent