Epoch 50/50
- 23s - loss: 0.8512 - dice_coef: 1.1176e-04 - dice_coef_rounded: 1.0000 - binary_crossentropy: 0.0086 - val_loss: 0.8512 - val_dice_coef: 1.1405e-04 - val_dice_coef_rounded: 1.0000 - val_binary_crossentropy: 0.0084
steps_per_epoch 19 validation_steps 6
And at the end of the training, I get this strange error. However the unet_weights4.h5 got saved correctly.
None
Traceback (most recent call last):
File "/opt/conda/envs/jupyter_env/bin/pixel_decoder", line 11, in <module>
load_entry_point('pixel-decoder', 'console_scripts', 'pixel_decoder')()
File "/example/pixel_decoder/main.py", line 75, in cli
main(args.pop('command'), **args)
File "/example/pixel_decoder/main.py", line 56, in main
train(**kwargs)
File "/example/pixel_decoder/train.py", line 62, in train
metrics=[dice_coef, dice_coef_rounded, metrics.binary_crossentropy])
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/engine/training.py", line 830, in compile
sample_weight, mask)
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/engine/training.py", line 429, in weighted
score_array = fn(y_true, y_pred)
File "/example/pixel_decoder/loss.py", line 28, in dice_logloss3
return binary_crossentropy(y_true, y_pred) * 0.15 + dice_coef_loss(y_true, y_pred) * 0.85
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/losses.py", line 77, in binary_crossentropy
return K.mean(K.binary_crossentropy(y_true, y_pred), axis=-1)
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/keras/backend/tensorflow_backend.py", line 3068, in binary_crossentropy
output = tf.clip_by_value(output, _epsilon, 1 - _epsilon)
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/ops/clip_ops.py", line 61, in clip_by_value
[t, clip_value_min, clip_value_max]) as name:
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5770, in __enter__
g = _get_graph_from_inputs(self._values)
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5428, in _get_graph_from_inputs
_assert_same_graph(original_graph_element, graph_element)
File "/opt/conda/envs/jupyter_env/lib/python3.6/site-packages/tensorflow/python/framework/ops.py", line 5364, in _assert_same_graph
original_item))
ValueError: Tensor("loss/conv2d_11_loss/Const:0", shape=(), dtype=float32) must be from the same graph as Tensor("conv2d_11/Sigmoid:0", shape=(?, 256, 256, 1), dtype=float32).
I attach the log file of the training.