Hi, I am unable to reproduce prediction or training using existing celebA hinge checkpoint.
Here is the stack trace of calling random test with pretrained checkpoint:
python main.py --dataset spade_celebA --segmap_ch 3 --phase random
Everything is runs okay till reading the checkpoints. But then some shapes are reported as mismatched and I am not able to figure out what could the problem be. I am using TensorFlow 1.14.
Do I need to use an older TF version for this to work or are there some additional code modifications that were performed after saving the pretrained checkpoint?
UPDATE: I resolved mismatching shapes lhs shape= [5,5,16,128] rhs shape= [5,5,19,128] as described below. Now new mismatch is lhs shape= [32768,256] rhs shape= [8192,256]
[*] Reading checkpoints...
W1017 12:37:21.855408 139936319534848 deprecation.py:323] From /home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [5,5,16,128] rhs shape= [5,5,19,128]
[[{{node save/Assign_623}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [5,5,16,128] rhs shape= [5,5,19,128]
[[node save/Assign_623 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:537) ]]
Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_623:
generator/spade_resblock_fix_2/spade_2/conv_128/conv2d/kernel/Adam (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:383)
Original stack trace for 'save/Assign_623':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 537, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 538, in random_test
could_load, checkpoint_counter = self.load(self.checkpoint_dir)
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 524, in load
self.saver.restore(self.sess, os.path.join(checkpoint_dir, ckpt_name))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1322, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [5,5,16,128] rhs shape= [5,5,19,128]
[[node save/Assign_623 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:537) ]]
Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_623:
generator/spade_resblock_fix_2/spade_2/conv_128/conv2d/kernel/Adam (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:383)
Original stack trace for 'save/Assign_623':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 537, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
UPDATE: I resolved this issue by reproducing CelebAHQ masks with 19 segmentation labels (instead of 16 labels as originally defined in spade_celebA\segmap_label.txt).
Now I get the following error (with mismatched dimensions 32768 and 8192):
[*] Reading checkpoints...
W1017 13:40:56.285303 139846562793216 deprecation.py:323] From /home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.
Instructions for updating:
Use standard file APIs to check for files with this prefix.
Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1356, in _do_call
return fn(*args)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1341, in _run_fn
options, feed_dict, fetch_list, target_list, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1429, in _call_tf_sessionrun
run_metadata)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [32768,256] rhs shape= [8192,256]
[[{{node save/Assign_185}}]]
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1286, in restore
{self.saver_def.filename_tensor_name: save_path})
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 950, in run
run_metadata_ptr)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1173, in _run
feed_dict_tensor, options, run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1350, in _do_run
run_metadata)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/client/session.py", line 1370, in _do_call
raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Assign requires shapes of both tensors to match. lhs shape= [32768,256] rhs shape= [8192,256]
[[node save/Assign_185 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:539) ]]
Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_185:
encoder/linear_var/kernel/Adam_1 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:385)
Original stack trace for 'save/Assign_185':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 539, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 540, in random_test
could_load, checkpoint_counter = self.load(self.checkpoint_dir)
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 526, in load
self.saver.restore(self.sess, os.path.join(checkpoint_dir, ckpt_name))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 1322, in restore
err, "a mismatch between the current graph and the graph")
tensorflow.python.framework.errors_impl.InvalidArgumentError: Restoring from checkpoint failed. This is most likely due to a mismatch between the current graph and the graph from the checkpoint. Please ensure that you have not altered the graph expected based on the checkpoint. Original error:
Assign requires shapes of both tensors to match. lhs shape= [32768,256] rhs shape= [8192,256]
[[node save/Assign_185 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:539) ]]
Errors may have originated from an input operation.
Input Source operations connected to node save/Assign_185:
encoder/linear_var/kernel/Adam_1 (defined at /home/blaz/github/SPADE-Tensorflow/SPADE.py:385)
Original stack trace for 'save/Assign_185':
File "main.py", line 125, in
main()
File "main.py", line 116, in main
gan.random_test()
File "/home/blaz/github/SPADE-Tensorflow/SPADE.py", line 539, in random_test
self.saver = tf.train.Saver()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 825, in init
self.build()
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 837, in build
self._build(self._filename, build_save=True, build_restore=True)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 875, in _build
build_restore=build_restore)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 508, in _build_internal
restore_sequentially, reshape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saver.py", line 350, in _AddRestoreOps
assign_ops.append(saveable.restore(saveable_tensors, shapes))
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/training/saving/saveable_object_util.py", line 72, in restore
self.op.get_shape().is_fully_defined())
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/state_ops.py", line 227, in assign
validate_shape=validate_shape)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/ops/gen_state_ops.py", line 66, in assign
use_locking=use_locking, name=name)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 788, in _apply_op_helper
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/util/deprecation.py", line 507, in new_func
return func(*args, **kwargs)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 3616, in create_op
op_def=op_def)
File "/home/blaz/anaconda2/envs/tf1.14/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2005, in init
self._traceback = tf_stack.extract_stack()
Any ideas?
Thank you!