Dale Decatur, Itai Lang, Rana Hanocka

University of Chicago

Abstract: We present 3D Highlighter, a technique for localizing semantic regions on a mesh using text as input. A key feature of our system is the ability to interpret “out-of-domain” localizations. Our system demonstrates the ability to reason about where to place non-obviously related concepts on an input 3D shape, such as adding clothing to a bare 3D animal model. Our method contextualizes the text description using a neural field and colors the corresponding region of the shape using a probability-weighted blend. Our neural optimization is guided by a pre-trained CLIP encoder, which bypasses the need for any 3D datasets or 3D annotations. Thus, 3D Highlighter is highly flexible, general, and capable of producing localizations on a myriad of input shapes.

Install and activate the conda environment with the following commands.

conda env create --file 3DHighlighter.yml
conda activate 3DHighlighter

Note: The installation will fail if run on something other than a CUDA GPU machine.

• Python 3.9
• CUDA 11
• 16 GB GPU

Run the scripts below to get example localizations.

# hat on a dog
./demo/run_dog_hat.sh
# shoes on a horse
./demo/run_horse_shoes.sh
# hat on a candle
./demo/run_candle_hat.sh

Due to small non-determinisms in CLIP's backward process and the sensitivity of our optimization, results can vary across different runs even when fully seeded. If the result of the optimization does not match the expected result, try re-running the optimization.

@article{decatur2022highlighter,
    author = {Decatur, Dale and Lang, Itai and Hanocka, Rana},
    title  = {3D Highlighter: Localizing Regions on 3D Shapes via Text Descriptions},
    journal = {arXiv},
    year = {2022}
}