It seems that when merging CelebA-Mask-HQ masks the order of masks matters. In particular one should first render "skin" and only then "ear".

Compare:
https://github.com/taki0112/SPADE-Tensorflow/blob/master/dataset/spade_celebA/segmap/7.jpg
https://github.com/olegmyrk/SPADE-Tensorflow/blob/develop/dataset/spade_celebA/segmap/7.png

Could you share the script that was used to merge the masks?

I run python main.py --dataset spade_celebA --segmap_ch 3 --phase, but the sved image is dark. something is wrong.

Hi,
thanks for the awesome work you've done!
Is it necessary to use the image_encoder model when training the entire model?
I tried to generate image only with random noise Z on coco-stuff dataset, and I can't get the satisfactory result.
Thanks!

Hi there,

Was wondering whether the version of TF being used could be specified; I see that tf.contrib is being used so it doesn't seem to be TF 2.0, but I am also not having any luck running it on various versions of TF 1.x

I can not load the provided checkpoints successully, So I'm wondering the version of tensorflow-gpu? anybody know?

Hello.

What is the training setting that was used for the Cityscapes example? The default batchsize of this code is 1, while the paper used 32. Also, is did removing the encoder have any degrading effect on the final result?

Thanks!

Hello!
Trying to run on TF 2.0
Traceback (most recent call last):
File "main.py", line 1, in
from SPADE import SPADE
File "/content/SPADE-Tensorflow/SPADE.py", line 7, in
from vgg19_keras import VGGLoss
File "/content/SPADE-Tensorflow/vgg19_keras.py", line 2, in
from keras.applications.vgg19 import preprocess_input
File "/opt/conda/lib/python3.7/site-packages/keras/init.py", line 25, in
from keras import models
File "/opt/conda/lib/python3.7/site-packages/keras/models.py", line 19, in
from keras import backend
File "/opt/conda/lib/python3.7/site-packages/keras/backend.py", line 36, in
from tensorflow.python.eager.context import get_config
ImportError: cannot import name 'get_config' from 'tensorflow.python.eager.context' (/opt/conda/lib/python3.7/site-packages/tensorflow_core/python/eager/context.py)
)
Do you have plans on upgrading to TF 2.0?

There seem to be some tensor shapes mismatch error when increasing batch size > 1. However, the code works fine with Batch size = 1.
Did you try increasing the batch size and train?

can you share the checkpoints for this thanks

The effect produced is different from the original image.E.G There is color difference in the generated picture.
Is it possible to add partial image generation?

Hi,
I tested the model with the segmap_label.txt in the README and the segmap images in ./dataset/spade_celebA/segmap_test:

segmap_label.txt for CelebAMask-HQ
{(0, 0, 0): 0, (0, 0, 255): 1, (255, 0, 0): 2, (150, 30, 150): 3, (255, 65, 255): 4, (150, 80, 0): 5, (170, 120, 65): 6, (125, 125, 125): 7, (255, 255, 0): 8, (0, 255, 255): 9, (255, 150, 0): 10, (255, 225, 120): 11, (255, 125, 125): 12, (200, 100, 100): 13, (0, 255, 0): 14, (0, 150, 80): 15, (215, 175, 125): 16, (220, 180, 210): 17, (125, 125, 255): 18}

but the output images are not correct, I think the reason is the segmap_label.txt does not correspond to the images the project provide, so what is the correct segmap_label?
Thanks!

There are two operations "down_sample_avg" on "x_init" and "seg_map", respectively, in line 239,240 of "SPADE.py".
However, I couldn't find any usages of these, so why did you insert these?

Thank you.

Hi, I am unable to reproduce prediction or training using existing celebA hinge checkpoint.

Here is the stack trace of calling random test with pretrained checkpoint:

python main.py --dataset spade_celebA --segmap_ch 3 --phase random

Everything is runs okay till reading the checkpoints. But then some shapes are reported as mismatched and I am not able to figure out what could the problem be. I am using TensorFlow 1.14.
Do I need to use an older TF version for this to work or are there some additional code modifications that were performed after saving the pretrained checkpoint?

UPDATE: I resolved mismatching shapes lhs shape= [5,5,16,128] rhs shape= [5,5,19,128] as described below. Now new mismatch is lhs shape= [32768,256] rhs shape= [8192,256]

[*] Reading checkpoints...
W1017 12:37:21.855408 139936319534848 deprecation.py:323] From /home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [5,5,16,128] rhs shape= [5,5,19,128]
[[{{node save/Assign_623}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [5,5,16,128] rhs shape= [5,5,19,128]
[[node save/Assign_623 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:537) ]]

Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_623:
generator/spade_resblock_fix_2/spade_2/conv_128/conv2d/kernel/Adam (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:383)

Original stack trace for 'save/Assign_623':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 537, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 538, in random_test
could_load, checkpoint_counter = self.load(self.checkpoint_dir)
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 524, in load
self.saver.restore(self.sess, os.path.join(checkpoint_dir, ckpt_name))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1322, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [5,5,16,128] rhs shape= [5,5,19,128]
[[node save/Assign_623 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:537) ]]

Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_623:
generator/spade_resblock_fix_2/spade_2/conv_128/conv2d/kernel/Adam (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:383)

Original stack trace for 'save/Assign_623':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 537, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

UPDATE: I resolved this issue by reproducing CelebAHQ masks with 19 segmentation labels (instead of 16 labels as originally defined in spade_celebA\segmap_label.txt).

Now I get the following error (with mismatched dimensions 32768 and 8192):

[*] Reading checkpoints...
W1017 13:40:56.285303 139846562793216 deprecation.py:323] From /home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [32768,256] rhs shape= [8192,256]
[[{{node save/Assign_185}}]]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [32768,256] rhs shape= [8192,256]
[[node save/Assign_185 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:539) ]]

Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_185:
encoder/linear_var/kernel/Adam_1 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:385)

Original stack trace for 'save/Assign_185':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 539, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 540, in random_test
could_load, checkpoint_counter = self.load(self.checkpoint_dir)
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 526, in load
self.saver.restore(self.sess, os.path.join(checkpoint_dir, ckpt_name))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1322, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [32768,256] rhs shape= [8192,256]
[[node save/Assign_185 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:539) ]]

Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_185:
encoder/linear_var/kernel/Adam_1 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:385)

Original stack trace for 'save/Assign_185':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 539, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()

Any ideas?

Thank you!

HI,
when I loaded the ckpt, I got an error like this:

Assign requires shapes of both tensors to match. lhs shape= [4,4,19,64] rhs shape= [4,4,22,64]
[[node save/Assign_15 (defined at /home/zhang/PycharmProjects/test/SPADE-Tensorflow/SPADE.py:537) = Assign[T=DT_FLOAT, _class=["loc:@discriminator/ms_0conv_0/conv2d/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](discriminator/ms_0conv_0/conv2d/kernel/Adam, save/RestoreV2/_519)]]
[[{{node save/RestoreV2/_1366}} = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_979_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

Caused by op 'save/Assign_15', defined at:
File "/home/zhang/PycharmProjects/test/SPADE-Tensorflow/main.py", line 125, in
main()
File "/home/zhang/PycharmProjects/test/SPADE-Tensorflow/main.py", line 116, in main
gan.random_test()
File "/home/zhang/PycharmProjects/test/SPADE-Tensorflow/SPADE.py", line 537, in random_test
self.saver = tf.train.Saver()
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1102, in init
self.build()
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1114, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 1151, in _build
build_save=build_save, build_restore=build_restore)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 795, in _build_internal
restore_sequentially, reshape)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 428, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/training/saver.py", line 119, in restore
self.op.get_shape().is_fully_defined())
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/ops/state_ops.py", line 221, in assign
validate_shape=validate_shape)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/ops/gen_state_ops.py", line 61, in assign
use_locking=use_locking, name=name)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/framework/op_def_library.py", line 787, in _apply_op_helper
op_def=op_def)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py", line 488, in new_func
return func(*args, **kwargs)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 3274, in create_op
op_def=op_def)
File "/home/zhang/anaconda2/envs/DFNRMVS2/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 1770, in init
self._traceback = tf_stack.extract_stack()

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [4,4,19,64] rhs shape= [4,4,22,64]
[[node save/Assign_15 (defined at /home/zhang/PycharmProjects/test/SPADE-Tensorflow/SPADE.py:537) = Assign[T=DT_FLOAT, _class=["loc:@discriminator/ms_0conv_0/conv2d/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:GPU:0"](discriminator/ms_0conv_0/conv2d/kernel/Adam, save/RestoreV2/_519)]]
[[{{node save/RestoreV2/_1366}} = _SendT=DT_FLOAT, client_terminated=false, recv_device="/job:localhost/replica:0/task:0/device:GPU:0", send_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device_incarnation=1, tensor_name="edge_979_save/RestoreV2", _device="/job:localhost/replica:0/task:0/device:CPU:0"]]

How could I solve this problem?

It seems that the black color label in the face dataset is segmented as background and the network generates a random background.

Is there a way to exclude this black color label during training for my own dataset? Like, in the official code there is a option for None label. Cause my None label has too different contents I don't want to confuse the network for this label.

안녕하세요. 제가 직접 segmentation을 수행하여 dataset을 만들고 training을 하려고 하는데 제가 None label을 만들어서 검은색으로 마킹을 했는데 training을 할땐 이 label을 제외시키고 트레이닝을 하고싶습니다. None label은 라벨링이 되지 않은 모든 object들이 다 들어있어서 네트워크가 이것 때문에 혼동을 할것 같아서요. 혹시 작성하신 코드 상에서 검은색으로 표기된 label을 제외하고 트레이닝을 할 수 있게 하려면 어떻게 바꿔야할지 알려주실 수 있으신가요?

항상 좋은 코드 감사합니다.

Thank you for sharing! I wonder is the figure in the README produced by your training code? how long did you trained to get that performance?

Hi,

Thanks for all the hard work. I was wondering how one would add support for multi-gpu training. I see that in SPADE.py we do a pre-fetch to the GPU in the build_model(), but I'm somewhat stumped on how multi-gpu would work

Thanks

I am new to this, Please help me to make the spade.py to load the checkpoint. I have downloaded the file and placed it in a folder called checkpoint in the root directory. Thanks in Advance.

Thanks a lot for your wonderful work!

I want to ask a question about VAE. In the decoder, as we use z_sample(x_mean, x_var), we cannot obtain exactly the same reconstruction image due to the variational mechanism.

Is that right?

tqdm
numpy
pillow
scipy
opencv-python
tensorflow-gpu
keras

For example, if the image size is 632, the following error will occur.

ValueError: Dimensions must be equal, but are 38 and 39 for 'generator/spade_resblock_0/spade_1/mul' (op: 'Mul') with input shapes: [1,38,38,1024], [1,39,39,1024].

tried to set resize_256 to 312 and so on.

With 512, these codes worked fine.

What do you think I should do?

Thanks for your code !
I have a few questions:

1. In order to achieve the same results like yours, how many GPUs do you need and also what kind of GPU did you use ?
2. What is your batch_size ?

Hi, Thanks very much for this wonderful repository. I will like to understand how you process the segmentation maps as it is currently not available in the code.

how to use guide test and how to use segmentation guide for facial attribute manipulation on your own provided image outside the dataset?
get this error for guide test
Traceback (most recent call last):
File "main.py", line 125, in
main()
File "main.py", line 120, in main
gan.guide_test()
File "/content/SPADE-Tensorflow/SPADE.py", line 584, in guide_test
style_image = load_style_image(self.guide_img, self.img_width, self.img_height, self.img_ch)
File "/content/SPADE-Tensorflow/utils.py", line 156, in load_style_image
img = cv2.cvtColor(img, cv2.COLOR_BGR2RGB)
cv2.error: OpenCV(3.4.3) /io/opencv/modules/imgproc/src/color.cpp:181: error: (-215:Assertion failed) !_src.empty() in function 'cvtColor'

How should I prepare data and use training arguments if I want to use pix2pixHD form edges to image as in figure 2 in the paper?

Shall I use train_A as images and train_b as the edge images and then use --label_nc 0 and --no-instance?

Hi Junho,

Really a great work and I was very happy to try out your implementation.

However, I notice a potential problem for the path name matching in the Image_data preprocess. Because the result list from glob is not sorted, the order from both image and segmap path lists might be consistent but in my case they are totally messed up. Therefore I changed the code into:

        self.image = sorted(glob(self.img_dataset_path + '/*.*'), key=os.path.basename)
        self.segmap = sorted(glob(self.segmap_dataset_path + '/*.*'), key=os.path.basename)
        self.segmap_test = sorted(glob(self.segmap_test_dataset_path + '/*.*'), key=os.path.basename)

The 2nd issue is that when using gray-scale image, the VGGLoss part raises error complaining shape mismatching. I modified the code in vgg19_keras.py, VGGLoss's call function:

        if x.shape[-1] == 1:
            x = tf.tile(x, [1, 1, 1, 3])
        if y.shape[-1] == 1:
            y = tf.tile(y, [1, 1, 1, 3])

Just repeat the single channel into 3 to resolve the shape problem.

The last one is in the sample imaging saving, simply add

    if img.shape[-1] == 1:
        img = img[..., 0]

after
https://github.com/taki0112/SPADE-Tensorflow/blob/master/utils.py#L208

Thank you again for your nice work, hope those lines would be helpful.

Best,
Yuhua (Bill)

안녕하세요,
항상 좋은 코드 올려주셔서 감사합니다.

이번 spade코드를 공부하다보니 이해가 잘 안되는 부분이 있어서 질문 드립니다.
celebA가 아닌, 다른 데이터셋의 이미지들을 비슷한 형식으로

dataset
탭탭탭image
탭탭탭segmap
탭탭탭segmap_test

위와 같이 배치하면 one-hot vector의 형태로 seg_map이 만들어지는건가요?

코드를 읽다보니 제가 이해하기엔 self.segmap_dataset_path에도 self.img_dataset_path에 있는 같은 이미지를 넣으면 class Image_data의 preprocess함수가 "segmap_label.txt"파일이 없으면 self.segmap_dataset_path 폴더 안에 있는 이미지들을 컬러든 그레이든 (self.segmap 하나하나 읽어서 color_value_dict를 만드는 거 같은데... 이 dictionary만드는 부분이 무슨 원리인지 이해가 잘 안됩니다. 즉, 모든 이미지들을 다 읽어서 비슷한특성을 가지는 "나무","바다" 등등에 label(label이 +=1이 되는 부분(if tuple(x[i, j, :]) not in self.color_value_dict.keys() : 이 구문)도 이해가 어렵네요)을 임의로 부여하는건가요? 그말은 이미지 순서나 종류가 바뀌면 매번 코드를 돌릴때마다 다른 label순서가 될 수도 있다는 건지도 궁금합니다. 뭔가 open을 이용하여 text 파일로 label들을 만들어주는거 같아보여요.
==> 즉, 미리 만들어둔 segmap_label.txt가 없으면 preprocess(self)함수에서 txt파일을 만들어주는 역할을 하는데, 어떤기준으로 픽셀별 label을 부여하는 건지 혼동됩니다.(instance seg.가 아닌, semantic seg.가 맞죠?)
그 이후, convert_from_color_segmentation()함수에서는 만든 text파일과 이미지를 읽어와서 이미지의 각 픽셀별로 text파일에서 만들어진 segmentation label의 규칙에 따라 dictionary value에 대응되면 같은 label로 부여하는 거 같은데 맞나요?

긴글 읽어주셔서 감사합니다:)

Thank you for providing the version of TensorFlow. I would like to ask if the label image and the real image should be consistent during training. When I was training, I found that real.png did not match real_semap.png.

Looking forward to your reply

There seem to be a mismatch of shapes between the checkpoint of the graph you uploaded (the latest one ) and the actual graph in repo.

InvalidArgumentError (see above for traceback): Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:

Assign requires shapes of both tensors to match. lhs shape= [4,4,19,64] rhs shape = [4,4,22,64] [[Node: save/Assign_14 = Assign[T=DT_FLOAT, _class ["loc:@discriminator/ms_0conv_0/conv2d/kernel"], use_locking=true, validate_shape=true, _device="/job:localhost/replica:0/task:0/device:CPU:0" (discriminator/ms_0conv_0/conv2d/kernel, save/RestoreV2:46)]]

Is the checkpoint uploaded recently, trained on same graph or some other graph?

any pretrained models on ade20k or cityscapes?

HI, the provided color label map can generate reasonable result, but the gray label map can not. Anybody knows about this?