please share sflckr vqgan for segmentation maps about taming-transformers HOT 3 CLOSED

Ah, I see. The cond_stage_model.colorize is actually not that important - it is only used for visualization purposes (to make the colorization of logged segmentation maps persistent). The main issue is then that cond_stage_model.colorize is present in the provided weights but not in the model you are instantiating. If you look at

Finally, this shouldn't matter if you use model.params.ckpt_path=logs/2020-11-09T13-31-51_sflckr/checkpoints/last.ckpt (or, equivalently, add ckpt_path: logs/2020-11-09T13-31-51_sflckr/checkpoints/last.ckpt to the model.params section of the config you are using, e.g. in line 5 of sflckr.yaml) instead of -r logs/2020-11-09T13-31-51_sflckr/checkpoints/last.ckpt on the command line, because it will then load the checkpoint non-strictly and simply ignore unexpected keys. See

from taming-transformers.

Hi @ink1,
the vqgan weights of both the first stage (encoding images) and the cond stage (encoding segmentations/depth/...) are actually part of the transformer checkpoints. We added a small script scripts/extract_submodel.py such that you can now extract the sflckr segmentation map vqgan weights with

python scripts/extract_submodel.py logs/2020-11-09T13-31-51_sflckr/checkpoints/last.ckpt sflckr_cond_stage_model.ckpt cond_stage_model

Also added an example config for training a segmentation map cond stage in configs/sflckr_cond_stage.yaml. If you want to finetune the extracted weights you can adjust the data section of that config and then run

python main.py --base configs/sflckr_cond_stage.yaml -t True --gpus 0, model.params.ckpt_path=sflckr_cond_stage_model.ckpt

Best
Patrick

from taming-transformers.

Much appreciated, Patrick! This is all great!
Due to a lack of compute budget I'm trying to leverage your models as much as I can. I do understand that both vqgan models (image and segmentation) are inside of the sflckr checkpoint.
My main problem is when I'm trying to resume from it, I hit
RuntimeError: Error(s) in loading state_dict for Net2NetTransformer:
Unexpected key(s) in state_dict: "cond_stage_model.colorize".
My understanding is that this is the top layer conditionally added to vqgan models.
The condition is the presence of a particular parameter, probably this one
https://github.com/CompVis/taming-transformers/blob/master/configs/sflckr_cond_stage.yaml#L8
But when I tried adding it to sflkr yaml after this line
https://github.com/ink1/taming-transformers/blob/master/configs/sflckr.yaml#L44
it gets rejected. This is why I thought if I load both vqgan models explicitly, I'll be in control.
The funny thing is, on restart after the error above, the sflkr model gets shrunk to under 2 GB (but I provide vqgan image model from its checkpoint) and a subsequent restart works fine. What's even more interesting is that the checkpoints are 4 GB again.
I strongly suspect, I just can't get my configuration straight.
I'm also curious if colorization of segmentation is persistent across runs.
Thanks again @pesser

from taming-transformers.