Hi Marko,

I run through the code and prediction script. But all of the predicted mask image all look like black.

I compute the max value and min value, it equals to 0.00170872779563 and
0.0 respectively.

Are there anything wrong with this.

`img_array = np.load('/data/imgs_mask_test.npy')

print(img_array.max())

print(img_array.min())`

Thanks.

Hi, Marko,

Thanks for sharing your codes. I have several questions about the method you use.

The data

Why do use load the mask images as a grayscale image with [0, 255] data instead of converting them to binary mast with [0, 1] as the ground truth which can be interpreted as the probability of being the foreground area. That would be more natural for segmentation.

The loss

Using the official evaluation function as the target function is a bit of indirect for supervising the network. Have you considered formulate the problem as segmentation as use binary cross-entropy loss function? That would be natural to train. In other hand, add a C constant lead to another different target function.

Best regards.

P.S In case you didn't see this message, I also post on the Kaggle forum. If you see it, please reply there.

I am a bit confused about the network structure when comparing it with what was originally proposed in the paper (Fig. 1). At the very bottom of the unet, the definition is as follows:

conv4 = Convolution2D(256, 3, 3, activation='relu', border_mode='same')(pool3)
conv4 = Convolution2D(256, 3, 3, activation='relu', border_mode='same')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)
conv5 = Convolution2D(512, 3, 3, activation='relu', border_mode='same')(pool4)
conv5 = Convolution2D(512, 3, 3, activation='relu', border_mode='same')(conv5)
up6 = merge([UpSampling2D(size=(2, 2))(conv5), conv4], mode='concat', concat_axis=1)

So conv5 has dimension of (N, 512, H, W);
"UpSampling2D(size=(2, 2))(conv5)": (N, 512, 2H, 2W)
conv4 has dimension of (N, 256, 2H, 2W);
"merge" actually merges (N, 512, 2H, 2W) with (N, 256, 2H, 2W), which results in dimension of (N, 768, 2H, 2W)...

As I understand, the merged two sets of features are supposed to have the same dimension, instead of one has 512 features and the other has 256 features. I wonder if I misunderstood something. Thanks!

Do we can train this network by ASUS laptop 3Ghz cpu, 8GB RAM and cuda compatibility 2.1 in a good time?

Line 150 in train.py should probably be:
imsave(os.path.join(pred_dir,str(image_id)+'_pred.png'), image)

I did the process steps as you said.
How can I perform testing?
what does submission.csv file' mean?
I could not create test mask.

@jocicmarko If my target images have multiple classes (i.e. 0, 1, 2, etc.) do I have to modify anything on the last last layer of the NN?

Hi,

I think in the paper, we are required to performed a "learned upsampling" or a "convolution transpose".

However, the implementation here uses UpSampling2D.
https://github.com/jocicmarko/ultrasound-nerve-segmentation/blob/master/train.py#L51

Any comments on this?

Rahul.V.

Hi, thanks so much for releasing this code, it's already helped a bunch in helping me go from very basic MNIST models to something more sophisticated.

I'm trying to run your code now but it dies when trying to compile the model. The nvidia card I'm using has very little memory at this point, do you think that's the actual error? The simpler auto-encoder model for the MNIST dataset from http://blog.keras.io/building-autoencoders-in-keras.html works, even when I substitute Adam.

From my reading of the trace, there's the main error, but that's ultimately coming off the dice_coeff functions, though I don't really understand why they would.

I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 755M
major: 3 minor: 0 memoryClockRate (GHz) 1.0195
pciBusID 0000:01:00.0
Total memory: 2.00GiB
Free memory: 1.65GiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:806] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 755M, pci bus id: 0000:01:00.0)

Loading and preprocessing train data...

Creating and compiling model...

ValueError Traceback (most recent call last)
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in merge_with(self, other)
562 try:
--> 563 self.assert_same_rank(other)
564 new_dims = []

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in assert_same_rank(self, other)
608 raise ValueError(
--> 609 "Shapes %s and %s must have the same rank" % (self, other))
610

ValueError: Shapes (?,) and (?, ?) must have the same rank

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in with_rank(self, rank)
638 try:
--> 639 return self.merge_with(unknown_shape(ndims=rank))
640 except ValueError:

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in merge_with(self, other)
569 raise ValueError("Shapes %s and %s are not compatible" %
--> 570 (self, other))
571

ValueError: Shapes (?,) and (?, ?) are not compatible

During handling of the above exception, another exception occurred:

ValueError Traceback (most recent call last)
/notebooks/ultrasound-nerve-segmentation/train.py in ()
134
135 if name == 'main':
--> 136 train_and_predict()

/notebooks/ultrasound-nerve-segmentation/train.py in train_and_predict()
102 print('Creating and compiling model...')
103 print('-'*30)
--> 104 model = get_unet()
105 model_checkpoint = ModelCheckpoint('unet.hdf5', monitor='loss', save_best_only=True)
106

/notebooks/ultrasound-nerve-segmentation/train.py in get_unet()
68 model = Model(input=inputs, output=conv10)
69
---> 70 model.compile(optimizer=Adam(lr=1e-5), loss=dice_coef_loss, metrics=[dice_coef])
71
72 return model

/usr/local/lib/python3.4/dist-packages/keras/engine/training.py in compile(self, optimizer, loss, metrics, loss_weights, sample_weight_mode, **kwargs)
587 loss_weight = loss_weights_list[i]
588 output_loss = weighted_loss(y_true, y_pred,
--> 589 sample_weight, mask)
590 if len(self.outputs) > 1:
591 self.metrics.append(output_loss)

/usr/local/lib/python3.4/dist-packages/keras/engine/training.py in weighted(y_true, y_pred, weights, mask)
309 def weighted(y_true, y_pred, weights, mask=None):
310 # score_array has ndim >= 2
--> 311 score_array = fn(y_true, y_pred)
312 if mask is not None:
313 # Cast the mask to floatX to avoid float64 upcasting in theano

/notebooks/ultrasound-nerve-segmentation/train.py in dice_coef_loss(y_true, y_pred)
24
25 def dice_coef_loss(y_true, y_pred):
---> 26 return -dice_coef(y_true, y_pred)
27
28

/notebooks/ultrasound-nerve-segmentation/train.py in dice_coef(y_true, y_pred)
20 y_true_f = K.flatten(y_true)
21 y_pred_f = K.flatten(y_pred)
---> 22 return (2. * K.dot(y_true_f, K.transpose(y_pred_f)) + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)
23
24

/usr/local/lib/python3.4/dist-packages/keras/backend/tensorflow_backend.py in dot(x, y)
212 yt = tf.reshape(tf.transpose(y, perm=y_permute_dim), [y_shape[-2], -1])
213 return tf.reshape(tf.matmul(xt, yt), x_shape[:-1] + y_shape[:-2] + y_shape[-1:])
--> 214 out = tf.matmul(x, y)
215 return out
216

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/math_ops.py in matmul(a, b, transpose_a, transpose_b, a_is_sparse, b_is_sparse, name)
1207 transpose_a=transpose_a,
1208 transpose_b=transpose_b,
-> 1209 name=name)
1210
1211 sparse_matmul = gen_math_ops._sparse_mat_mul

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/gen_math_ops.py in _mat_mul(a, b, transpose_a, transpose_b, name)
1176 """
1177 result = _op_def_lib.apply_op("MatMul", a=a, b=b, transpose_a=transpose_a,
-> 1178 transpose_b=transpose_b, name=name)
1179 return result
1180

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/op_def_library.py in apply_op(self, op_type_name, name, **keywords)
702 op = g.create_op(op_type_name, inputs, output_types, name=scope,
703 input_types=input_types, attrs=attr_protos,
--> 704 op_def=op_def)
705 outputs = op.outputs
706 return _Restructure(ops.convert_n_to_tensor(outputs),

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in create_op(self, op_type, inputs, dtypes, input_types, name, attrs, op_def, compute_shapes, compute_device)
2260 original_op=self._default_original_op, op_def=op_def)
2261 if compute_shapes:
-> 2262 set_shapes_for_outputs(ret)
2263 self._add_op(ret)
2264 self._record_op_seen_by_control_dependencies(ret)

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/ops.py in set_shapes_for_outputs(op)
1700 raise RuntimeError("No shape function registered for standard op: %s"
1701 % op.type)
-> 1702 shapes = shape_func(op)
1703 if shapes is None:
1704 raise RuntimeError(

/usr/local/lib/python3.4/dist-packages/tensorflow/python/ops/common_shapes.py in matmul_shape(op)
84 def matmul_shape(op):
85 """Shape function for a MatMul op."""
---> 86 a_shape = op.inputs[0].get_shape().with_rank(2)
87 transpose_a = op.get_attr("transpose_a")
88 b_shape = op.inputs[1].get_shape().with_rank(2)

/usr/local/lib/python3.4/dist-packages/tensorflow/python/framework/tensor_shape.py in with_rank(self, rank)
639 return self.merge_with(unknown_shape(ndims=rank))
640 except ValueError:
--> 641 raise ValueError("Shape %s must have rank %d" % (self, rank))
642
643 def with_rank_at_least(self, rank):

ValueError: Shape (?,) must have rank 2

Hi Marko,

Regarding the the main optimization step model.fit(imgs_train, imgs_mask_train, batch_size=32, nb_epoch=20, verbose=1, shuffle=True, callbacks=[model_checkpoint])

How many iterations within each epoch will it be included? Here, we have about 5635 pairs of training images and training mask images. If batch_size=32, should it be 5635/32 iterations?
I did not find a place where this parameter of training_iters are setup. Will it be handled automatically by model.fit?

Besides, how do you setup the dropout rate in this training process? Will it be helpful?

Thank you very much.

Hi there,

Thanks for posting the code. I noticed in your calculation of the DICE coefficient, you smoothed the result by 1 i.e.

def dice_coef(y_true, y_pred):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)


def dice_coef_loss(y_true, y_pred):
    return -dice_coef(y_true, y_pred)

My understanding is that

My questions: 1) why did you add the smoothing value? How does it help during training?

2.) why did you take the negative of the coefficient in Line 19?

Pardon my questions if the answer seems self-evident as I am not a user of Keras at all but it seems to me that you computed something different from the original dice coefficient. I might be wrong so I would appreciate your explanation. Thank you!

Hi Marko,

May I ask you a generic question regarding u-net for semantic segmentation?

I noticed some u-net implementation involves so-called offset variable to make up the size change between the input image and generated image after layers of conv-pool-unpool-deconv,etc.

However, it seems to me that we can control the output layer size by specifying the output layer size in conv2d_transpose in Tensorflow. Therefore, we can always control the output size, and remove the necessity of using off-set variable. Is this thought right?

In this example posted by https://github.com/NVIDIA/DIGITS/tree/master/examples/semantic-segmentation

They also state the usage of “offset variable”: You might also be curious to understand the need for the score layer which is a Crop type of layer. This layer is added to crop its input so as to compensate the shifts induced by the successive convolution and pooling layers down the network. The offset value of 18 will work for any dataset however that offset value needs to be recomputed if the topology of the network changes.

Any insight on this kind of “offset variable” in semantic segmentation problem is highly appreciated.

I have my own training sets,but the background are too large.
Also,I know the background is black(pixel=0).
When I am calculating the dice,I want to 'remove' the background(Don't calculate it into dice).
I try this:for i in y_true_f.eval():
But it didn't work~

Do we can train this network bu a 3Ghz cpu, 8GB RAM and cuda compatibility 2.1?

Hi, jocicmarko. While reading your code, I have some question. You scale imgs_mask to [0,1], while you preprocess your imgs like that x-=np.mean x/=np.std.I want to know the reason why you don't preprocess your imgs and imgs_mask like that x-=np.mean x/=np.std.Thank you for your replying

Hi Marko,

I have limited number (around 500~700) images with large size (10241024). I constructed large amount of training set based on sampling from original images. The sampled size is 128128. I trained a model, and then tried to predict over the test images (1024*1024). Previously, I just make prediction over the sub-frame of the test images, and then stitch these predicted masked together.

I heard that there is a way to let trained network can predict any size of images. The trick is replacing inputs = Input((1, img_rows, img_cols)) to inputs = Input((1, None, None)). I tried, but i have error messages as. Do you have any thoughts on this? Or is it possible to let U-net predict images having different size with trained images.

Traceback (most recent call last):
  File "train-unet-v3b.py", line 113, in <module>
    train_and_predict()
  File "train-unet-v3b.py", line 96, in train_and_predict
    model = get_unet()
  File "train-unet-v3b.py", line 54, in get_unet
    up6 = merge([UpSampling2D(size=(2, 2))(conv5), conv4], mode='concat', concat_axis=1)
  File "/development/gtfw/lib/python3.4/site-packages/Keras-1.0.3-py3.4.egg/keras/engine/topology.py", line 485, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/development/gtfw/lib/python3.4/site-packages/Keras-1.0.3-py3.4.egg/keras/engine/topology.py", line 543, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/development/gtfw/lib/python3.4/site-packages/Keras-1.0.3-py3.4.egg/keras/engine/topology.py", line 148, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/development/gtfw/lib/python3.4/site-packages/Keras-1.0.3-py3.4.egg/keras/layers/convolutional.py", line 1043, in call
    self.dim_ordering)
  File "/development/gtfw/lib/python3.4/site-packages/Keras-1.0.3-py3.4.egg/keras/backend/tensorflow_backend.py", line 508, in resize_images
    X.set_shape((None, None, original_shape[2] * height_factor, original_shape[3] * width_factor))
TypeError: unsupported operand type(s) for *: 'NoneType' and 'int'

I noticed that in def train_and_predict() the loading and preprocessing of test data shows as follows:

    print('-'*30)
    print('Loading and preprocessing test data...')
    print('-'*30)

    imgs_test, imgs_id_test = load_test_data()
    imgs_test = preprocess(imgs_test)

    imgs_test = imgs_test.astype('float32')
    imgs_test -= mean
    imgs_test /= std

However vars mean and std are inherited from calculations done from the training data a few lines above:

    imgs_train = imgs_train.astype('float32')
    mean = np.mean(imgs_train)  # mean for data centering
    std = np.std(imgs_train)  # std for data normalization

Is this intentional and if so, what is the benefit/rationale?

Your code works very well, and I have been using and also modified for many projects.
However i just realized today that Dice and IOU are both non-differentiable function.
Then my question is, why it works in your code?
Thanks!

Hello, I tried to run your code to learn deep leaning architecture. But the test_masks which were predicted were the same? Can you please show your test_masks results as images instead of.npy files? Would be very helpful to understand.

Thank you so much in advance.

I am trying to modify the activation layer at the end of the U-Net implementation by doing this:

conv10 = Convolution2D(1, 1, 1, activation='linear')(conv9)
reshapeLayer = Reshape((img_rows*img_cols, 1))(conv10)
softmaxLayer = Activation('softmax')(reshapeLayer)
model = Model(input=inputs, output=softmaxLayer)
model.compile(optimizer=Adam(lr=1e-5), loss='categorical_crossentropy', metrics=[dice_coef])

Unfortunately I keep getting mismatch errors for imgs_mask_train . I tried 3 approaches:

1. Flattening by doing imgs_mask_train = imgs_mask_train.ravel()

2. Doing some transposing and reshaping:

mgs_mask_train = np.transpose(imgs_mask_train,(0,2,3,1)) \Move channel layer at the end
imgs_mask_train = np.reshape(imgs_mask_train, (imgs_mask_train.shape[0],
imgs_mask_train.shape[1]*imgs_mask_train.shape[2], imgs_mask_train.shape[3]))

That didn't work either.
3) Categorical conversion:
imgs_mask_train = to_categorical(imgs_mask_train)

Any suggestions to fix this and do softmax? I am new to Keras and perhaps I am still confused.

Hi Marko,

I am not very clear about the function of preprocess(imgs).

def preprocess(imgs):
     imgs_p = np.ndarray((imgs.shape[0], imgs.shape[1], img_rows, img_cols), dtype=np.uint8)

    for i in range(imgs.shape[0]):

        imgs_p[i, 0] = cv2.resize(imgs[i, 0], (img_cols, img_rows), interpolation=cv2.INTER_CUBIC)

    return imgs_p

At first, you define imgs_p with the shape of (batchsize, channel, img_rows,img_cols). However, in the resize function, you resize the original image to (img_cols, img_rows). It seems to me that it should resize to (img_rows,img_cols).

In fact, if I use the original shape, imgs_p[i, 0] = resize(imgs[i, 0], (img_cols, img_rows)) I got the following error message

Traceback (most recent call last): File "train.py", line 153, in <module> train_and_predict() File "train.py", line 92, in train_and_predict imgs_train = preprocess(imgs_train) File "train.py", line 81, in preprocess imgs_p[i, 0] = resize(imgs[i, 0], (img_cols, img_rows)) ValueError: could not broadcast input array from shape (80,64) into shape (64,80)

Besides, I have a second question.

If the backend of Keras is Tensorflow, then what part do I need to modify to run your code. One thing I can think of is about shape of the training set. In your current implementation, the shape is (size, channel, width,height). I need to re-shape it to (size,width,height, channel). Are other codes that I need to modify?

Thanks,
Huaiyang

Hello Marko,

thank you for your U-NET implementation.
I've used it to design a 3D-UNET. That was really helpful.

What do you think about a dice_coef_loss like:

def dice_coef_loss(y_true, y_pred):
    return 1.0 - dice_coef(y_true, y_pred)

This will return values between [0 , 1] and not negative ones, in comparison to other loss functions. (don't know if it makes any differences at the end)

Greetings
kleinfeld

Hi Marko,

First of all, thanks for sharing your code in this repository. It has been really useful for the project I am doing: segmentation of skin lesions.

I have run all your code, and I already have the "imgs_mask_test.npy" file but I would like to convert this numpy array file into tiff/png/jpeg images. Do you have a code for doing that?

Thanks in advance.

Hi Marko,

Thank you for sharing the code.

I have two questions while reading your code.

1. The input mask images have channel 1, while the output layer(or the prediction) from U-net is of channel 2. How does your code handle this kind of scenario?
   After reading your code, I have not been very clear about the mechanism you use for handling this.

2. I am not very clear about the usages of this function

def preprocess(imgs): imgs_p = np.ndarray((imgs.shape[0], imgs.shape[1], img_rows, img_cols), dtype=np.uint8) for i in range(imgs.shape[0]): imgs_p[i, 0] = cv2.resize(imgs[i, 0], (img_cols, img_rows), interpolation=cv2.INTER_CUBIC) return imgs_p

I can see this code is used to transform the mask images, but I do not know why?

Thank you very much for your help.

hi i tried your code , everything goes well but the predicted masks generated are of low contrast.how to increase the contrast so that i can see the clear prediction
here is the sample prediction

Hi,
I have a basic question about keras implementation. Assuming I train with batch, for example, (batch_size = 64, image_size, image_size), slipped from training, for example (10000, image_size, image_size). My test_data (4000, image_size, image_size).
I would like to know when running model.predict(test_data), keras will split the test_data in to batch 64 to evaluate with iteration, or it evaluates with whole stack?

and if I pass the single images in test_data one by one, is there any changes?

Thank guys
Bests,

When i try to run this model.fit, now i am getting the following Error

model.fit(X_train, Y_train, batch_size= 10, epochs=3, verbose=1)

ValueError: Error when checking target: expected conv2d_60 to have shape (None, 1, 320, 1) but got array with shape (286, 1, 320, 528)

I am implementing Unet in Keras using theano as backend,
X_train - total 286 images
Y_train - total 286 above images but it is segmented by some local thresholding method.

Actual image image size : rows, cols = 327, 532
But i am giving this : rows, cols = 320, 528
because there are 4 MaxPooling2D layers in my topology !! (As it is suggested by Marko Jocić )

Can someone Please help me to resolve this error ?

Dear Jocicmarko,

I am new to deep learning practice. I run you code on all 5000+ training images on a GPU and get following output. I wonder why the dice suddenly drop at Epoch 19. Did I do anything wrong? (I did not change the code so I do not know where I could mess up some key parameters).

Thank you very much!

5635/5635 [==============================] - 212s - loss: -0.0853 - dice_coef: 0.0853
Epoch 2/20
5635/5635 [==============================] - 41s - loss: -0.2196 - dice_coef: 0.2196
Epoch 3/20
5635/5635 [==============================] - 41s - loss: -0.2450 - dice_coef: 0.2450
Epoch 4/20
5635/5635 [==============================] - 41s - loss: -0.2607 - dice_coef: 0.2607
Epoch 5/20
5635/5635 [==============================] - 41s - loss: -0.3141 - dice_coef: 0.3141
Epoch 6/20
5635/5635 [==============================] - 41s - loss: -0.3927 - dice_coef: 0.3927
Epoch 7/20
5635/5635 [==============================] - 41s - loss: -0.4538 - dice_coef: 0.4538
Epoch 8/20
5635/5635 [==============================] - 42s - loss: -0.4979 - dice_coef: 0.4979
Epoch 9/20
5635/5635 [==============================] - 41s - loss: -0.5239 - dice_coef: 0.5239
Epoch 10/20
5635/5635 [==============================] - 42s - loss: -0.5417 - dice_coef: 0.5417
Epoch 11/20
5635/5635 [==============================] - 42s - loss: -0.5480 - dice_coef: 0.5480
Epoch 12/20
5635/5635 [==============================] - 42s - loss: -0.5603 - dice_coef: 0.5603
Epoch 13/20
5635/5635 [==============================] - 42s - loss: -0.5740 - dice_coef: 0.5740
Epoch 14/20
5635/5635 [==============================] - 42s - loss: -0.5829 - dice_coef: 0.5829
Epoch 15/20
5635/5635 [==============================] - 42s - loss: -0.5903 - dice_coef: 0.5903
Epoch 16/20
5635/5635 [==============================] - 42s - loss: -0.5967 - dice_coef: 0.5967
Epoch 17/20
5635/5635 [==============================] - 41s - loss: -0.5966 - dice_coef: 0.5966
Epoch 18/20
5635/5635 [==============================] - 41s - loss: -0.6054 - dice_coef: 0.6054
Epoch 19/20
5635/5635 [==============================] - 41s - loss: -0.0017 - dice_coef: 0.0017
Epoch 20/20
5635/5635 [==============================] - 41s - loss: -0.0011 - dice_coef: 0.0011

I am using the Dice loss function you have stated in my code.
The question I have is that you're using sigmoid as the activation function in the final layer which basically gives you a prediction map in the range [0,1]. But you're comparing it directly with the label map which only contains 2 values 0 and 1. How is it working without rounding?

Moreover, I also noticed you didn't use rounding even during output predictions? Can anyone clarify this a bit?

Hello Marko,

The code you posted is very clean and well-written. Thank you! If I have an image training set with input channel as 3, how can I fit this set to your framework. Looks like the Kaggle set has training set with channel=1.

Huaiyanggongzi

Can dice loss be generalized for multiclass segmentation?

Hi, I tried to use this UNet to segment the image and used Dice as the loss function. The function was like below and I didn't change it.

def dice_coef(y_true, y_pred, smooth=1.):
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred)
    intersection = K.sum(y_true_f * y_pred_f)
    return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)


def dice_coef_loss(y_true, y_pred):
    return -dice_coef(y_true, y_pred)
The complie is like:

model.compile(optimizer=Adam(lr=initial_learning_rate), loss=dice_coef_loss, metrics=[dice_coef])

However, during the training, the log message is like below:

Epoch 1/1000
299s - loss: 6.4668 - dice_coef: 0.2710 - val_loss: 5.3623 - val_dice_coef: 0.5395
Epoch 2/1000
228s - loss: 4.7529 - dice_coef: 0.5754 - val_loss: 4.4627 - val_dice_coef: 0.3444
Epoch 3/1000
229s - loss: 3.7705 - dice_coef: 0.6485 - val_loss: 3.3135 - val_dice_coef: 0.7470
Epoch 4/1000
232s - loss: 3.0341 - dice_coef: 0.7334 - val_loss: 2.7504 - val_dice_coef: 0.7445
Epoch 5/1000
230s - loss: 2.5172 - dice_coef: 0.7546 - val_loss: 2.2995 - val_dice_coef: 0.7674
Epoch 6/1000
230s - loss: 2.0970 - dice_coef: 0.7988 - val_loss: 1.9096 - val_dice_coef: 0.8277

In my opinion, I think the loss should be negative dice_coef. If I'm wrong, how was this loss calculated?
Further, Somethings the loss is equal to negativedice_coef, sometimes it is not.

Thank you.

Hi, marko, thanks for your sharing. I have run you code, and when i run 'python train.py', after it completed all epoch, there is a warning for all images in /pregs: low contrast image. I ignored the warning, after all the file 'imgs_test_mask.npy' has generated. Then i run 'python submission.py', there is no any errors. but final submission.csv only have the images number, no pixel value. can you help me?

I would apply U-Net to RGB pictures (input). Could I understand if is it possible with a small dataset? Is it possible with U-Net and this implementation?

Using Keras 2.0.3, Theano 0.9, Python 3.5.

Images are 256 x 256 grayscale, with binary masks.
Channels first is ensured in Keras back end, appropriate axis=1, using K.set_image_data_format('channels_first') and inputs = Input((1, dim, dim)).

When running the Unet before UpSampling2D was changed to Conv2DTranspose, the model trained without problems.

When running the Unet with Conv2DTranspose, Theano throws the below error.
Is there any advantage to using Conv2DTranspose?

Traceback (most recent call last):
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/theano/compile/function_module.py", line 884, in __call__
    self.fn() if output_subset is None else\
ValueError: GpuCorrMM shape inconsistency:
  bottom shape: 5 256 32 32
  weight shape: 512 256 2 2
  top shape: 5 512 16 16 (expected 5 512 17 17)


During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/.../src/train_binary.py", line 102, in <module>
    train()
  File "/Users/.../src/train_binary.py", line 77, in train
    callbacks=[model_checkpoint, csv_log_cback])
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/keras/engine/training.py", line 1498, in fit
    initial_epoch=initial_epoch)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/keras/engine/training.py", line 1152, in _fit_loop
    outs = f(ins_batch)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/keras/backend/theano_backend.py", line 1158, in __call__
    return self.function(*inputs)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/theano/compile/function_module.py", line 898, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/six-1.10.0-py3.5.egg/six.py", line 685, in reraise
    raise value.with_traceback(tb)
  File "/Library/Frameworks/Python.framework/Versions/3.5/lib/python3.5/site-packages/theano/compile/function_module.py", line 884, in __call__
    self.fn() if output_subset is None else\
ValueError: GpuCorrMM shape inconsistency:
  bottom shape: 5 256 32 32
  weight shape: 512 256 2 2
  top shape: 5 512 16 16 (expected 5 512 17 17)

Apply node that caused the error: GpuCorrMM_gradInputs{half, (2, 2), (1, 1)}(GpuContiguous.0, GpuContiguous.0, TensorConstant{32}, TensorConstant{32})
Toposort index: 242
Inputs types: [CudaNdarrayType(float32, 4D), CudaNdarrayType(float32, 4D), TensorType(int64, scalar), TensorType(int64, scalar)]
Inputs shapes: [(512, 256, 2, 2), (5, 512, 16, 16), (), ()]
Inputs strides: [(1024, 4, 2, 1), (131072, 256, 16, 1), (), ()]
Inputs values: ['not shown', 'not shown', array(32), array(32)]
Outputs clients: [[GpuSubtensor{int64:int64:int8, int64:int64:int8, int64:int64:int8, :int64:}(GpuCorrMM_gradInputs{half, (2, 2), (1, 1)}.0, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1}, ScalarFromTensor.0, ScalarFromTensor.0, Constant{1}, Constant{0}, Constant{8}, Constant{1}, Constant{8})]]

I am trying to segment imbalanced 3 classes in my brain volume (following this paper: https://arxiv.org/pdf/1702.05970.pdf page 7 formula (1) by adding the weight_class.

My label volume (after to_categorical is (batch_size, img_size^2, 3)). But I think keras's class_weight in fit function does not support 3D tensor.
I think I have to adjust the loss function by creating the weight_volume corresponding to to label volume. My question is if I use the dice function, how can I implement this?

I would like to post my failed code when using the cross entropy loss function. It works with simple y_pred and y_true that I defined by myself. But it does not work when it embedded to Unet due to placeholder, nontype at first dimension in Unet. I believe that the code has a lot of trouble, but hopefully it will show I idea.

val = np.array([[[1, 0, 0], [0, 1, 0], [0, 1, 0]], 
                   [[0, 0, 1], [0, 0, 1], [1, 0, 0]], 
                   [[1, 0, 0], [1, 0, 0], [0, 0, 1]]])

val1 = np.array([[[1, 0, 0], [0, 1, 0], [0, 1, 0]], 
                   [[0, 0, 1], [0, 0, 1], [1, 0, 0]], 
                   [[1, 0, 0], [1, 0, 0], [0, 0, 1]]])

kvar = K.variable(value=val, dtype='float32', name='example_var')
kvar1 = K.variable(value=val1, dtype='float32', name='example_var')

def cross_entropy_function(y_true, y_pred):
    # y_true and y_pred has shape (batch_size, img_size*_img_size, 3)
    shape = K.int_shape(y_true)
    weight0 = K.get_value((shape[0]*shape[1] - K.sum(K.flatten(y_true[:,:,0])))
                          /K.sum(K.flatten(y_true[:,:,0]))-1.0)
    weight1 = K.get_value((shape[0]*shape[1] - K.sum(K.flatten(y_true[:,:,1])))
                          /K.sum(K.flatten(y_true[:,:,1]))-1.0)
    weight2 = K.get_value((shape[0]*shape[1] - K.sum(K.flatten(y_true[:,:,2])))
                          /K.sum(K.flatten(y_true[:,:,2]))-1.0)
    weight_y_true = K.get_value(y_true)
    #print weight0, weight1, weight2
    weight_y_true_0 = np.ones(shape = (shape[0],shape[1],1))
    weight_y_true_1 = np.ones(shape = (shape[0],shape[1],1))
    weight_y_true_2 = np.ones(shape = (shape[0],shape[1],1))
    weight_y_true_0[np.where(weight_y_true[:,:,0]==1)] = weight0 
    weight_y_true_1[np.where(weight_y_true[:,:,1]==1)] = weight1 
    weight_y_true_2[np.where(weight_y_true[:,:,2]==1)] = weight2
    A = np.concatenate((weight_y_true_0, weight_y_true_1), 2)
    A = np.concatenate((A, weight_y_true_2), 2)
    class_weight = K.variable(value=A, dtype='float32')

    # calculate weighted cross entropy
    epsilon = tf.constant(value=1e-4)
    y_true_f = K.flatten(y_true)
    y_pred_f = K.flatten(y_pred) 
    class_weight_f = K.flatten(class_weight)
    L = tf.reduce_sum(class_weight_f*
                      (y_true_f*K.log(y_pred_f+ epsilon) + 
                       (1-y_true_f)*K.log(1-y_pred_f+ epsilon)))
    L = tf.reduce_mean(L)
    #print K.get_value(L)
    return L 

def loss(y_true, y_pred):    
    return -cross_entropy_function(y_true, y_pred)
print K.get_value(loss(kvar, kvar1))

Hey,
are the inputsize and the outputsize of the images equal?

Exception Traceback (most recent call last)
in ()
----> 1 up6 = merge([UpSampling2D(size=(2, 2, 2), dim_ordering='default')(conv5), conv4], mode='concat', concat_axis=1)

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in merge(inputs, mode, concat_axis, dot_axes, output_shape, output_mask, name)
1526 node_indices=node_indices,
1527 tensor_indices=tensor_indices,
-> 1528 name=name)
1529 return merge_layer.inbound_nodes[0].output_tensors[0]
1530 else:

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in init(self, layers, mode, concat_axis, dot_axes, output_shape, output_mask, node_indices, tensor_indices, name)
1184 self._arguments_validation(layers, mode,
1185 concat_axis, dot_axes,
-> 1186 node_indices, tensor_indices)
1187 self.built = True
1188 self.add_inbound_node(layers, node_indices, tensor_indices)

/usr/local/lib/python2.7/dist-packages/keras/engine/topology.pyc in _arguments_validation(self, layers, mode, concat_axis, dot_axes, node_indices, tensor_indices)
1251 raise Exception('"concat" mode can only merge layers with matching ' +
1252 'output shapes except for the concat axis. ' +
-> 1253 'Layer shapes: %s' % (input_shapes))
1254
1255 def call(self, inputs, mask=None):

Exception: "concat" mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, 0, 8, 512), (None, 0, 8, 256)]

Hi,
thanks for replying regarding train.py
i changed from python 3.5 to python 2.7
now i am getting following error:

   71     conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)
   72 

---> 73 up7 = concatenate([UpSampling2D(size=(2, 2))(conv6), conv3], axis=3)
74 conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)
75 conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)

ValueError: Concatenate layer requires inputs with matching shapes except for the concat axis. Got inputs shapes: [(None, 0, 104, 256), (None, 0, 105, 128)]

i have used theano backend.

the code is :
`from future import print_function

import os
os.environ['KERAS_BACKEND']='theano'
import keras
from skimage.transform import rotate, resize, rescale
#from skimage.io import imsave
from skimage.io import imsave, imread
import numpy as np
from keras.models import Model
from keras.layers import Input, concatenate, Conv2D, MaxPooling2D, UpSampling2D
from keras.optimizers import Adam
from keras.callbacks import ModelCheckpoint
from keras import backend as K
K.image_dim_ordering = 'th'

#from data import load_train_data, load_test_data

K.set_image_data_format('channels_last') # TF dimension ordering in this code

img_rows = 420
img_cols = 580

smooth = 1.

def load_train_data():
imgs_train = np.load('imgs_train.npy')
imgs_mask_train = np.load('imgs_mask_train.npy')
return imgs_train, imgs_mask_train

def load_test_data():
imgs_test = np.load('imgs_test.npy')
imgs_id = np.load('imgs_id_test.npy')
return imgs_test, imgs_id

def dice_coef(y_true, y_pred):
y_true_f = K.flatten(y_true)
y_pred_f = K.flatten(y_pred)
intersection = K.sum(y_true_f * y_pred_f)
return (2. * intersection + smooth) / (K.sum(y_true_f) + K.sum(y_pred_f) + smooth)

def dice_coef_loss(y_true, y_pred):
return -dice_coef(y_true, y_pred)

def get_unet():
inputs = Input((1, img_rows, img_cols))
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(inputs)
conv1 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv1)
pool1 = MaxPooling2D(pool_size=(2, 2))(conv1)

conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(pool1)
conv2 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv2)
pool2 = MaxPooling2D(pool_size=(2, 2))(conv2)

conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(pool2)
conv3 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv3)
pool3 = MaxPooling2D(pool_size=(2, 2))(conv3)

conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(pool3)
conv4 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv4)
pool4 = MaxPooling2D(pool_size=(2, 2))(conv4)

conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(pool4)
conv5 = Conv2D(512, (3, 3), activation='relu', padding='same')(conv5)

up6 = concatenate([UpSampling2D(size=(2, 2))(conv5), conv4], axis=3)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(up6)
conv6 = Conv2D(256, (3, 3), activation='relu', padding='same')(conv6)

up7 = concatenate([UpSampling2D(size=(2, 2))(conv6), conv3], axis=3)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(up7)
conv7 = Conv2D(128, (3, 3), activation='relu', padding='same')(conv7)

up8 = concatenate([UpSampling2D(size=(2, 2))(conv7), conv2], axis=3)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(up8)
conv8 = Conv2D(64, (3, 3), activation='relu', padding='same')(conv8)

up9 = concatenate([UpSampling2D(size=(2, 2))(conv8), conv1], axis=3)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(up9)
conv9 = Conv2D(32, (3, 3), activation='relu', padding='same')(conv9)

conv10 = Conv2D(1, (1, 1), activation='sigmoid')(conv9)

model = Model(inputs=[inputs], outputs=[conv10])

model.compile(optimizer=Adam(lr=1e-5), loss=dice_coef_loss, metrics=[dice_coef])

return model`

thanks in advance!!

I use 18 images to train the model, and I got loss like this~I do not know how to solve this?Is the training sets too small?

Epoch 2/20
19/19 [==============================] - 3s - loss: -0.0191 - dice_coef: 0.0191
Epoch 3/20
19/19 [==============================] - 3s - loss: -0.0191 - dice_coef: 0.0191
Epoch 4/20
19/19 [==============================] - 4s - loss: -0.0191 - dice_coef: 0.0191
Epoch 5/20
19/19 [==============================] - 4s - loss: -0.0191 - dice_coef: 0.0191
Epoch 6/20
19/19 [==============================] - 4s - loss: -0.0191 - dice_coef: 0.0191
Epoch 7/20
19/19 [==============================] - 3s - loss: -0.0191 - dice_coef: 0.0191
Epoch 8/20
19/19 [==============================] - 4s - loss: -0.0191 - dice_coef: 0.0191
Epoch 9/20
19/19 [==============================] - 4s - loss: -0.0191 - dice_coef: 0.0191
Epoch 10/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 11/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 12/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 13/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 14/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 15/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 16/20
19/19 [==============================] - 3s - loss: -0.0192 - dice_coef: 0.0192
Epoch 17/20
19/19 [==============================] - 4s - loss: -0.0193 - dice_coef: 0.0193
Epoch 18/20
19/19 [==============================] - 4s - loss: -0.0193 - dice_coef: 0.0193
Epoch 19/20
19/19 [==============================] - 3s - loss: -0.0193 - dice_coef: 0.0193
Epoch 20/20
19/19 [==============================] - 4s - loss: -0.0193 - dice_coef: 0.0193

Hi jocicmarko , thanks for sharing the code
i m new born to this deep learning.
while executing train.py , i am getting following warning which is not allowing to complete execution.

C:\Anaconda3\lib\site-packages\skimage\transform_warps.py:84: UserWarning: The default mode, 'constant', will be changed to 'reflect' in skimage 0.15.

python version-3.5
scikit-image-0.13.0

any help is appreciated a lot........!

Hello @jocicmarko ,
Excellent code. I'm trying to reimplement your code.
I notice that in the method of model compile, you set loss function to dice_coef_loss and the metrics is dice_coef.
model.compile(optimizer=Adam(lr=1e-5), loss=dice_coef_loss, metrics=[dice_coef])
If I want to print out both dice coefficient and f1 score, how do I achieve that? Here I define the f1 score function as follow:

def f1score(y_true, y_pred):
    return f1_score(y_true, y_pred, average=None)

Hope you will give me some advices. Looking forward to your reply.
Sincerely,

Hi Marko,
I'am trying to use your code. My images are .png files.
data.py runs without exceptions. but train.py fails in that line 51
up6 = merge([UpSampling2D(size=(2, 2))(conv5), conv4], mode='concat', concat_axis=1)
with that exception message:
Exception: "concat" mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, 512, 42, 78), (None, 256, 43, 79)]
I know the shapes doesn't match, but why?
during competion your code runs well with original ultrasound images(.tiff).
Could you help me please to customize your code that i can train your model on my own images?

In another thread, you were discussing whether U-net can predict images with different sizes. If in the training stage, I am also trying to feed U-net with images of different size, can I do it with the current U-net implementation, (or just use None in the input layer)? Thanks.

Hi,

Thank you for your code. Could you add a license so people can use it? E.g. MIT or GPLv3.

Thanks,
Jeff

Hi, jocicmarko, glad to read your code. I remember you ever resized the image to 64 * 80, but now the image is resized to 96*96 in your code. Could you please tell us why do you change the size of the input image? Does the later size work better?
And the size of the original image is 420 * 580, which the height isn't equal to the weight. This time you make img_rows equal to img_cols. Is there sepcial reason making you change this? Will it lose too much information though there is noise existing?
Hoping to get your reply. Thank you very much!

y_true : 1d array-like, or label indicator array / sparse matrix
Ground truth (correct) target values.
y_pred : 1d array-like, or label indicator array / sparse matrix
Estimated targets as returned by a classifier.

what is y_true and y_pred in this code ?
in this code , how is he reading y_true and y_pred . Where is he defining y_true, y_pred ? I dint find any.

https://github.com/jocicmarko/ultrasound-nerve-segmentation/blob/master/train.py#L19

I tried to print y_true and y_pred , but it says it is not defined.
Can anyone say , where he is defining y_true and y_pred ? !!

@jocicmarko Thank you for sharing your implementation.

I'm using the model for training on 10000+ rgb images and the dice-coef exceeds 1.8000 (loss is obviously negative of that number). Shouldn't 1 be considered as the maximum value can be achieved for the accuracy? Appreciate your thoughts on that.

Hi, I am trying to use your code for another segmentation project, I just want to know if the pixels in the masks that you use for training have different values or each one is labeled with a category.
Thanks,

Hi,

The link you provided for downloading the data is not working.
Can you provide a cloud/drive link to access and download the data?
Please Help!

Thanks!