Giter VIP home page Giter VIP logo

informative-drawings's Introduction

Informative Drawings: Learning to generate line drawings that convey geometry and semantics

Setup

Clone this repository

git clone https://github.com/carolineec/informative-drawings.git
cd informative-drawings

Install dependencies

We provide an environment.yml file listing the dependences and to create a conda environment. Our model uses Pytorch 1.7.1

conda env create -f environment.yml
conda activate drawings

Use the following command to install CLIP (only needed for training).

conda activate drawings
pip install git+https://github.com/openai/CLIP.git

Testing

Pre-trained model is available here, place the model weights in checkpoints.

cd checkpoints
unzip model.zip

run pre-trained model on images in --dataroot. Replace examples with the folder path containing your input images.

python test.py --name anime_style --dataroot examples/test

Results will be saved to the results directory by default. You can change the save location by specifying the file path with --results_dir.

Training

We provide a pre-trained network for mapping ImageNet features into depth images here. Place the pre-trained features to depth network in the ./checkpoints/feats2Geom folder.

cd checkpoints/feats2Geom
unzip feats2depth.zip

To train a model with name myexperiment from scratch use the following command.

python train.py --name myexperiment \
--dataroot examples/train/photos \
--depthroot examples/train/depthmaps \
--root2 examples/train/drawings \
--no_flip

Replace the example data examples/train/photos, examples/train/depthmaps, and examples/train/drawings with the paths to the dataset of photographs, depth maps, and line drawings respectively. Corresponding images and depth maps in the file paths specified by --dataroot and --depthroot should have the same file names. You will also need to specify a path to an unaligned dataset of line drawings with --root2. A small example of training data is provided in examples/train.

Because the model can start making grayscale photos after some training, it is recommended to save model checkpoints frequently by adding the flag --save_epoch_freq 1.

Depth Maps

For training, geometry supervision requires depth maps for the dataset of photographs. To produce psuedo-ground truth depth maps we rely on a pretrained model from Boosting Monocular Depth Estimation Models to High-Resolution via Content-Adaptive Multi-Resolution Merging.

Citation

If you find this work useful please use the following citation:

 @article{chan2022drawings,
	      title={Learning to generate line drawings that convey geometry and semantics},
	      author={Chan, Caroline and Durand, Fredo and Isola, Phillip},
	      booktitle={CVPR},
	      year={2022}
	      }

Acknowledgements

Model code adapted from pix2pixHD and pytorch-CycleGAN-and-pix2pix

informative-drawings's People

Contributors

carolineec avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

informative-drawings's Issues

Paper links

Do you have a paper link? the results look very interesting.

Details about dataset

Hi, thanks for the great work. Do you have any plan to release the training dataset? Or how many training samples have you used in each (image, depth, Line)?

OS X - No module named 'model'

I've had it working your over a year. Now I get

Traceback (most recent call last):
  File "/informative-drawings/test.py", line 15, in <module>
    from model import Generator, GlobalGenerator2, InceptionV3
ModuleNotFoundError: No module named 'model'

Is it a simple path issue? Any idea on how to fix it?

Pre-trained model files

Hi,
In the pre-trained model link, there is only one file i.e., generator A file. Can you please share the remaining three files i.e., files for Generator B, Discriminator A and Discriminator B which can help to perform fine tuning

Tweaking setting for better results

I've trained a model with pictures of Egon Schiele. The colors slowly begin to drift into his color scheme. The linework is pretty much not existing. So far it looks more like a color graded image. Are there settings I can tweak to make his lines and watercolor textures more prominent
Luisa Neubauer-hell_out-74
Like this with a mixed in other line draw model but with his variable line widths.
182174589-f1f689d4-b115-411c-af5c-0546b202d44e Kopie
?

Colab Notebook?

This is an amazing method. Is there a colab notebook (especially for video) planned?
Thank you! :)

None Cuda dependent version

Hi,
I have a Mac with a M1 CPU and no access to a Cuda device. Is it possible to run the model without it?
I've tried removing all .cuda() calls in test.py and loading with:map_location=torch.device('cpu')

Running test.py through this error:

python test.py --name anime_style --dataroot examples/test Namespace(name='anime_style', checkpoints_dir='checkpoints', results_dir='results', geom_name='feats2Geom', batchSize=1, dataroot='examples/test', depthroot='', input_nc=3, output_nc=1, geom_nc=3, every_feat=1, num_classes=55, midas=0, ngf=64, n_blocks=3, size=256, cuda=True, n_cpu=8, which_epoch='latest', aspect_ratio=1.0, mode='test', load_size=256, crop_size=256, max_dataset_size=inf, preprocess='resize_and_crop', no_flip=False, norm='instance', predict_depth=0, save_input=0, reconstruct=0, how_many=100) loaded checkpoints/anime_style/netG_A_latest.pth Traceback (most recent call last): File "/Users/jan/Documents/ML/informative-drawings/test.py", line 108, in <module> dataloader = DataLoader(test_data, batch_size=opt.batchSize, shuffle=False) File "/opt/homebrew/Caskroom/miniforge/base/envs/drawings/lib/python3.10/site-packages/torch/utils/data/dataloader.py", line 200, in __init__ if isinstance(dataset, IterableDataset): File "/opt/homebrew/Caskroom/miniforge/base/envs/drawings/lib/python3.10/typing.py", line 1469, in __instancecheck__ raise TypeError("Instance and class checks can only be used with" TypeError: Instance and class checks can only be used with @runtime_checkable protocols

I'm not a deveopler, any idea how I might get it to run?

Is there some way to preserve image size after drawing?

Hi there,
Thanks for your excellent work! I am trying to follow your work and when I want to generate line drawings with different input size, the output image cannot preserve the size.

For example, input size of (390, 500, 3) leads to the output of (392, 500, 3), and input size of (300, 230, 3) leads to the output of (300, 232, 3).

The difference is small but still make some problem to match inputs and outputs in pixels. Is there some way to generate a drawing with the same size of input image?

Ask for Supplementary and Helen Dataset

Dear author,

I find that you have greatly improved the quality of face pen painting synthesis after using Helen dataset instead of Apdrawing dataset. I'm very interested in it. Can you provide the supplemental of the paper and the experimental dataset, as well as the training / test segmentation strategy of the dataset ?

Thanks
Nan

Thank you for great work

Thank you for great work

I am having doubt about output image shape/size?

does it wont return same size as input (720 * 720 )?

Is there a way to upscale ??

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.