Hi orobix,

First thanks for sharing your work :) but I have some questions :

1. Can all your data pre-processing steps be used channel-wise with rgb data ? or some transformations shouldn't ? (gamma ?)

2. How did you find the value for the gamma correction for example, did you try with you eyes ? with your model ? is it a standard (for this dataset only ?) ?

3. Why you transform your data to grayscale ? Aren't you loosing information ? Is there any specific reasons ? which ones ?

Hello,

I checked both shapes of conv1 and pool1 and they don't seem to be coherent:
K.int_shape(conv1)
(None, 32, 48, 48)
K.int_shape(pool1)
(None, 16, 24, 48)
It looks like you're doing the maxpooling operation on the feature map.

In retina-unet/src/retinaNN_training.py, don't you think that you should replace the 37th line
pool1 = MaxPooling2D((2, 2))(conv1)
by
pool1 = MaxPooling2D((2, 2),data_format='channels_first')(conv1)?

Eliott

I can't get data from http://www.isi.uu.nl/Research/Databases/DRIVE/ ....
If there any backup?

Hi,

Thanks for making this code available. I wanted to give this a quick look. I am on Mac OS, installed latest Keras and Tensorflow. Tried on python 3.5 after 2to3 for mostly the print statements.

Here is the error I am getting:

Using TensorFlow backend.
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcublas.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcudnn.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcufft.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcuda.1.dylib locally
I tensorflow/stream_executor/dso_loader.cc:108] successfully opened CUDA library libcurand.dylib locally

train images/masks shape:
(20, 1, 565, 565)
train images range (min-max): 0.0 - 1.0
train masks are within 0-1

patches per full image: 9500

train PATCHES images/masks shape:
(190000, 1, 48, 48)
train PATCHES images range (min-max): 0.0 - 1.0
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:892] OS X does not support NUMA - returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:102] Found device 0 with properties:
name: GeForce GT 650M
major: 3 minor: 0 memoryClockRate (GHz) 0.9
pciBusID 0000:01:00.0
Total memory: 1023.69MiB
Free memory: 25.46MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:126] DMA: 0
I tensorflow/core/common_runtime/gpu/gpu_init.cc:136] 0: Y
I tensorflow/core/common_runtime/gpu/gpu_device.cc:838] Creating TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GT 650M, pci bus id: 0000:01:00.0)
Traceback (most recent call last):
File "./src/retinaNN_training.py", line 107, in
model = get_unet(n_ch, patch_height, patch_width) #the U-net model
File "./src/retinaNN_training.py", line 34, in get_unet
conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(inputs)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/keras/engine/topology.py", line 514, in call
self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/keras/engine/topology.py", line 572, in add_inbound_node
Node.create_node(self, inbound_layers, node_indices, tensor_indices)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/keras/engine/topology.py", line 149, in create_node
output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/keras/layers/convolutional.py", line 466, in call
filter_shape=self.W_shape)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/keras/backend/tensorflow_backend.py", line 1579, in conv2d
x = tf.nn.conv2d(x, kernel, strides, padding=padding)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 394, in conv2d
data_format=data_format, name=name)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/op_def_library.py", line 703, in apply_op
op_def=op_def)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 2319, in create_op
set_shapes_for_outputs(ret)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/ops.py", line 1711, in set_shapes_for_outputs
shapes = shape_func(op)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 246, in conv2d_shape
padding)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 184, in get2d_conv_output_size
(row_stride, col_stride), padding_type)
File "/Users/ilknur/anaconda/lib/python3.5/site-packages/tensorflow/python/framework/common_shapes.py", line 149, in get_conv_output_size
"Filter: %r Input: %r" % (filter_size, input_size))
ValueError: Filter must not be larger than the input: Filter: (3, 3) Input: (1, 48)

Hi,
Thanks for sharing the code. I have a general question regarding the type of semantic segmentation task you are trying to solve.

In addition to the retina blood vessel segmentation, another popular segmentation task is discussed in https://www.kaggle.com/c/ultrasound-nerve-segmentation The ground truth labelled area is usually like this one.

It seems to me that, for the retina study, the labelled area is ground truth image is more like a tree structure with a lot of small branches; while for this ultra-sound study, the labelled area is always a single block. Even though both of them belong to the semantic segmentation problem, the characteristics of labelled area are different with each other. My question is that, will the net architecture you designed here be a good fit for Kaggle study? Do you have any specific suggestion, which kind of modification can be made, when adopting your solution for that kaggle problem?

Hi,

I'm curious about how other experiments performed, it would be great to add these results to the readme if possible.
For example, have you tried using the entire image as input instead of small sized batches? This is basically the idea of the U-Net and worked well in other tasks.
Thanks again for open sourcing this!

Hi,
I am going to train the net, but I get the following error:
I am using CUDA7.5, Tesla K40 GPU, Debian.

$/majid/work/retina-unet$ python run_training.py

1. Create directory for the results (if not already existing)
Dir already existing
copy the configuration file in the results folder

2. Run the training on GPU (no nohup)
Using Theano backend.
Using gpu device 0: Tesla K40m (CNMeM is disabled, cuDNN not available)

train images/masks shape:
(20, 1, 565, 565)
train images range (min-max): 0.0 - 1.0
train masks are within 0-1

patches per full image: 9500

train PATCHES images/masks shape:
(190000, 1, 48, 48)
train PATCHES images range (min-max): 0.0 - 1.0
Traceback (most recent call last):
  File "./src/retinaNN_training.py", line 176, in <module>
    model = get_unet(n_ch, patch_height, patch_width)  #the U-net model
  File "./src/retinaNN_training.py", line 34, in get_unet
    conv1 = Convolution2D(32, 3, 3, activation='relu', border_mode='same')(inputs)
  File "/home/azimi/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 569, in __call__
    self.add_inbound_node(inbound_layers, node_indices, tensor_indices)
  File "/home/azimi/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 632, in add_inbound_node
    Node.create_node(self, inbound_layers, node_indices, tensor_indices)
  File "/home/azimi/.local/lib/python2.7/site-packages/keras/engine/topology.py", line 164, in create_node
    output_tensors = to_list(outbound_layer.call(input_tensors[0], mask=input_masks[0]))
  File "/home/azimi/.local/lib/python2.7/site-packages/keras/layers/convolutional.py", line 445, in call
    filter_shape=self.W_shape)
  File "/home/azimi/.local/lib/python2.7/site-packages/keras/backend/theano_backend.py", line 1482, in conv2d
    np_kernel = kernel.eval()
  File "/home/azimi/.local/lib/python2.7/site-packages/theano/gof/graph.py", line 519, in eval
    rval = self._fn_cache[inputs](*args)
  File "/home/azimi/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 886, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/home/azimi/.local/lib/python2.7/site-packages/theano/gof/link.py", line 325, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/home/azimi/.local/lib/python2.7/site-packages/theano/compile/function_module.py", line 873, in __call__
    self.fn() if output_subset is None else\
RuntimeError: Cuda error: k_copy_4d: invalid device function.
Apply node that caused the error: HostFromGpu(GpuDimShuffle{3,2,0,1}.0)
Toposort index: 1
Inputs types: [CudaNdarrayType(float32, 4D)]
Inputs shapes: [(32, 48, 3, 3)]
Inputs strides: [(1, 32, 4608, 1536)]
Inputs values: ['not shown']
Outputs clients: [['output']]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.

The architecture is outdated. Merge layers are no longer used in Keras: keras-team/keras#10377

How can this be fixed?

I try the same configuration file as 'test', but I can not achieve the same score 0.9790 as 'test'.

Why? I tried 2 times, First time get 0.9783, second time I get 0.9784. And I am trying the third time and it is not finish yet. But I guess this time will better than 0.9784, because that 'val_loss' is less than previous times(last improved: Epoch 00080: val_loss improved from 0.09076 to 0.09074)

So... Is the 'validation patches random extract from training datasets' cause the different result

Thank you for your work.
And I have some question:
First, I used GTX1060 6GB for this code, run as 'test' configuration will cost 900 sec per epoch, is that common? And how long you take with TITAN X?
Second, is that 'loss' 'acc' 'val_loss' and 'val_acc' have some related? And some connection with AUC of ROC? (I will very apprecite if you teach me about that 4 data means.)
Last, please let me know DL BBS or funds segmentation with DL BBS, and how I contact you?(Not this 'issue' way...)
: )

Thank you very much and I am so sorry to asking quetions here no about the code.

Thank you.

In order to avoid overfitting on the DRIVE testing dataset, the testing dataset is no longer used for validation.
The validation dataset is now obtained by selecting a fraction (10%) of the training data, and a new training is performed with this new validation scheme. The results are basically the same (AUC ROC from 0.9791 to 0.9790).

Hi,
After execute run_testing.py script, i see following message:
ValueError: CorrMM images and kernel must have the same stack size

Apply node that caused the error: CorrMM{half, (1, 1), (1, 1)}(InplaceDimShuffle{0,2,3,1}.0, Subtensor{::, ::, ::int64, ::int64}.0)
Toposort index: 60
Inputs types: [TensorType(float32, 4D), TensorType(float32, 4D)]
Inputs shapes: [(32L, 16L, 24L, 48L), (3L, 3L, 64L, 32L)]
Inputs strides: [(73728L, 96L, 4L, 1536L), (4L, 12L, -1152L, -36L)]
Inputs values: ['not shown', 'not shown']
Outputs clients: [[Elemwise{Composite{((i0 + i1) + Abs((i0 + i1)))}}[(0, 0)](CorrMM{half, (1, 1), (1, 1)}.0, InplaceDimShuffle{x,0,x,x}.0)]]

keras version is 2.0.8
keras.json
{
"epsilon": 1e-07,
"floatx": "float32",
"image_data_format": "channels_last",
"backend": "theano",
"image_dim_ordering": "th"
}
os win server 2008 r2
python 2.7

I use test_best_weights.h5 and DRIVE database.

hello everyone!
this project works well under linux system, but can not work under win10 system.
there is no any error under win10 system and the result is a totally black picture.
anyone encount the same problem?

Thank you for opensource the code.

I made some changes with your model and use the same training settings as 'test' experiment, which following follow settings:
[training settings]
#number of total patches:
N_subimgs = 190000
#Number of training epochs
N_epochs = 150
And I got a wierd problem, that model, I just called g-model, when training epochs got 110~120 times, It will cause a GPU error, I have training that model for TEN times and every time will appear that error.
I don't know why, maybe there have something wrong with GPU's calculation, dividing by zero? or data overflow?
Fortunately, g-model achieve higher tested score, average is 0.97945 (with 0.9794, 0.9795, 0.9799, 0.9797, 0.9793, 0.9789). Yes, there are all finished with 110~120 epochs and 190000 subimgs, there are all have 'last improved' at 60~70 epochs, after that to the end(when stopped with GPU error), no anything improved. So I don't know that model at that settings can be better or not.
I am training g-model with subimgs=25000, epochs=100 at this moment.

Now I want to ask about 'test' experiment and your source model, you choose that parameters can make the model converge or there are higher parameters can make higher score? (I am feel ashamed to ask that I didn't tested source model with higher parameters)
Or, A higher score with reasonable settings and costs can saying that model is better than another lower score model? by the way, g-model costs 3 times training time as such as the source model.

In other words, I want to ask the evaluation standard of models, score base on the same settings or only final score?

I just found cuDNN 5105 are faster than 5005, source model with 'test' settings, costs 650s per epoch, after use cuDNN 5105, it decreased to 550s per epoch. Theano report a warning but I haven't found any problem. (ps. That problem I said above this was appeared before I update the cuDNN)

Can I use your code in my graduation project? My project name is 'Retina image processing and vessels segmentation' with python implement.

Thank you for reading my question.

I'm new to deep learning, I have this error, it is clearly the target images needs to be in different shape. However, since I'm doing augmentation using (.flow_from_directory) for both the images and the masks. How can I reshape target (mask) to be as same as activation results? Notice I getting generator after augmentation. Any idea?

permute_1 (Permute) (None, 25600, 2) 0 reshape_1[0][0]

activation_1 (Activation) (None, 25600, 2) 0 permute_1[0][0]

your help is greatly appreciated. Thanks.

Hi,
I have some questions about the data output. I applied your model to another data set ,but I want to know how to achieve Multi-classification task,I hope you can give me some suggestions.Moreover，my data set need information of color,and I don't know how to process it,I hope you can give me guidance.
Any insightful suggestions are very appreciate!

Hi, I am going to reproduce the experiments. I use the Keras 1.2, tensorflow version is 1.1.0
I got the error :
TypeError: Expected int32, got list containing Tensors of type '_Message' instead.

It points to line 48 in get_unet,
up1 = merge([UpSampling2D(size=(2, 2))(conv3), conv2], mode='concat', concat_axis=1)
Can anyone help me?

Hi,
I'm going through your code, I see some code which has been deprecated in due course of time.
like here
visualize_util has been changed to vis_utils and plot to plot_model.

I would like to fix such bugs by raising a PR, let me know your consent.

Hi,

First of all, great project thanks for open sourcing it! I need to use this model as a preprocessing step for another task therefore it would save me some time to have the weights available. Is there any chance of posting these?

Best,
Martín

Hi, I am going to test the net, but I get the following error: I am using CUDA8.0, GTX TITAN X GPU, Ubuntu 14.04.
root@1f9eab07a59c:~/laocp/keras/retina-unet-master# python run_testing.py

1. Create directory for the results (if not already existing)

2. Run the prediction on GPU (no nohup)
   Using Theano backend.
   WARNING (theano.sandbox.cuda): The cuda backend is deprecated and will be removed in the next release (v0.10). Please switch to the gpuarray backend. You can get more information about how to switch at this URL:
   https://github.com/Theano/Theano/wiki/Converting-to-the-new-gpu-back-end%28gpuarray%29

Using cuDNN version 5105 on context None
Mapped name None to device cuda: GeForce GTX TITAN X (0000:01:00.0)
Using gpu device 0: GeForce GTX TITAN X (CNMeM is disabled, cuDNN 5105)

the side H is not compatible with the selected stride of 5
img_h 584, patch_h 48, stride_h 5
(img_h - patch_h) MOD stride_h: 1
So the H dim will be padded with additional 4 pixels
the side W is not compatible with the selected stride of 5
img_w 565, patch_w 48, stride_w 5
(img_w - patch_w) MOD stride_w: 2
So the W dim will be padded with additional 3 pixels
new full images shape:
(20, 1, 588, 568)

test images shape:
(20, 1, 588, 568)

test mask shape:
(20, 1, 584, 565)
test images range (min-max): 0.0 - 1.0
test masks are within 0-1

Number of patches on h : 109
Number of patches on w : 105
number of patches per image: 11445, totally for this dataset: 228900

test PATCHES images shape:
(228900, 1, 48, 48)
test PATCHES images range (min-max): 0.0 - 1.0
Traceback (most recent call last):
File "./src/retinaNN_predict.py", line 111, in
model = model_from_json(open(path_experiment+name_experiment +'_architecture.json').read())
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/models.py", line 333, in model_from_json
return layer_module.deserialize(config, custom_objects=custom_objects)
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/layers/init.py", line 46, in deserialize
printable_module_name='layer')
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 140, in deserialize_keras_object
list(custom_objects.items())))
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 2378, in from_config
process_layer(layer_data)
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 2347, in process_layer
custom_objects=custom_objects)
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/layers/init.py", line 46, in deserialize
printable_module_name='layer')
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/utils/generic_utils.py", line 141, in deserialize_keras_object
return cls.from_config(config['config'])
File "/root/laocp/anaconda2/lib/python2.7/site-packages/keras/engine/topology.py", line 1210, in from_config
return cls(**config)
TypeError: init() got an unexpected keyword argument 'input_dtype'.

Any idea on how to resolve this problem? Thx for sharing!!

Traceback (most recent call last):
File "./src/retinaNN_training.py", line 190, in
patches_masks_train = masks_Unet(patches_masks_train) #reduce memory consum
ption
File "./lib\help_functions.py", line 60, in masks_Unet
new_masks = np.empty((masks.shape[0],im_h*im_w,2))
MemoryError

How to fix this?

Hola! @FarajiMehrdad has created a ZenHub account for the orobix organization. ZenHub is the only project management tool integrated natively in GitHub – created specifically for fast-moving, software-driven teams.

How do I use ZenHub?

To get set up with ZenHub, all you have to do is download the browser extension and log in with your GitHub account. Once you do, you’ll get access to ZenHub’s complete feature-set immediately.

What can ZenHub do?

ZenHub adds a series of enhancements directly inside the GitHub UI:

• Real-time, customizable task boards for GitHub issues;
• Multi-Repository burndown charts, estimates, and velocity tracking based on GitHub Milestones;
• Personal to-do lists and task prioritization;
• Time-saving shortcuts – like a quick repo switcher, a “Move issue” button, and much more.

Add ZenHub to GitHub

Still curious? See more ZenHub features or read user reviews. This issue was written by your friendly ZenHub bot, posted by request from @FarajiMehrdad.

change the parameters in keras.json can't solve the problem , maybe is the vision problem? who can tell me! thx a lot!

Can you please explain how did you reach to the following equation

y_ = y - int(img_h/2)  # origin (0,0) shifted to image center
R_inside = 270 - int(patch_h*1.42) #radius is 270 (from DRIVE db docs), minus the patch diagonal (assumed it is a square #this is the limit to contain the full patch in the FOV
radius = np.sqrt((x_*x_)+(y_*y_))

According to me it the equation should be R_inside = 270 - int(patch_h/1.42) #Half of diagonal

At this line it fails to create the file, if dataset directory does not exist

Hi, I am running this model against a given data set. For curiosity, I just test the prediction result using the model weights after finished the first 12 epochs of training.

The prediction shape is (841,4096,2). I found that the values of [:,:,0] are almost all 1 while [:,:,1] are all very small values. Is this a strong signal that the model has not been able to learn anything during the first very few epochs. Thanks.

Hi,
I have some questions about the data augmentation. I have already finished reading the code in your package ,but I didn't find the code about random elastic deformations of the training samples. I found that the pathches were extracted from the images randomly. It's different from the paper: U-Net: Convolutional Networks for Biomedical Image Segmentation.

up1 = concatenate([conv2,up1],axis=1)
up2 = concatenate([conv1,up2], axis=1)

About the parameters, I think axis=0 is right,since we need to enlarge the numbers of channels,and the datamat is channels_first such as (3,576,588) which implicts 3 channels.If axis =1,don't we enlarge the rows on the wrong dimensions?
Be grateful of your help! @

file: /lib/help_functions
function: pred_to_imgs
describe: In line 90, reshape function, shape (pred_images.shape[0],1,48,48)) are hard-coded, I will go wrong when you change patchsize in 'configuration.txt'.
my solution is added patch_height=48,patch_width=48 into that function, and use it to have a flexible shape.

And I don't understand line 363 364 in file /lib/extract_patches.py, is it useless or not?

thank you

Hi,

I used your best weights to predict for a single test image. I revised the code (see attachments) and it generates very noisy image. In the background there are so many false-positives. I applied the same pre-processing steps as you did (by looking at the documentation). I use training values for the standart deviation and mean. The result and the code are at the attachment. Am I doing wrong in somewhere?
I changed the priority for the channel order as "first channel" since I use Keras. Could you please help me figure out this issue?

from keras.models import model_from_json
import cv2
import copy
from matplotlib import pyplot as plt
import numpy as np


def prepocess_data(the_img):
    #normalize
    #std:55.851990770984536, img_mean:83.96562255652191
    img_std = 55.851990770984536 #np.std(the_img)
    img_mean = 83.96562255652191#np.mean(the_img)
    img_normalized = (the_img - img_mean)/img_std
    img_normalized = ((img_normalized-np.min(img_normalized)) / (np.max(img_normalized)-np.min(img_normalized)))*255

    #clahe preporcess
    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8,8))
    clahe_processed = clahe.apply(np.array(img_normalized, dtype=np.uint8))

    #gama correction
    gamma = 1.2
    invGamma = 1.0 / gamma
    table = np.array([((i / 255.0) ** invGamma) * 255 for i in np.arange(0, 256)]).astype("uint8")
    gama_img =  cv2.LUT(np.array(clahe_processed, dtype = np.uint8), table)

    #finally return res
    return gama_img

def predictImage(inImg, theModel):
    stepSize = 24
    windowSize = 48
    prob_threshold = 0.99

    #create zero img
    outImg = np.zeros((inImg.shape[0], inImg.shape[1]), dtype=inImg.dtype)#outImg with zeros
    print inImg.shape
    for x in range(48, inImg.shape[0]-windowSize, stepSize):
        for y in range(48, inImg.shape[1]-windowSize, stepSize):
            #sub image rectangle
            subImg =copy.copy(inImg[x-windowSize//2:x+windowSize//2, y-windowSize//2:y+windowSize//2])
            subImg = prepocess_data(subImg)
            cnnShape = np.reshape(subImg,(1,1,48,48))
            predictions = theModel.predict(cnnShape, batch_size=32, verbose=2) #(1,2304,2)
            #reshape out to img
            positives = (predictions[0,:,1] > prob_threshold).astype(inImg.dtype)
            positives = np.reshape(positives,(48,48))
            outImg[x-windowSize//2:x+windowSize//2, y-windowSize//2:y+windowSize//2] = positives
    return outImg


baseFolder = './test/'
#load model
jsonFile= open(baseFolder+'test_architecture.json','r')
jsonString = jsonFile.read()
jsonFile.close()
cnnModel = model_from_json(jsonString)
#load models
cnnModel.load_weights(baseFolder+'test_best_weights.h5')
#cnnModel.summary()

#read img
img_path = 'DRIVE/images/01_test.tif' #DRIVE data, test image 1
img = cv2.imread(img_path)
b,g,r = cv2.split(img) # also tried your gray scale version, the results even worse. I use direclty g channel as gray scale

#cv2.imshow('OriginalG',g);
#preprocess
g = prepocess_data(g)

#cv2.waitKey(-1)

print('Please wait, image is predicting...')
outImg = predictImage(g, cnnModel)

cv2.imwrite('predicted.jpg',outImg[:,:]*255 )
plt.subplot(121),plt.imshow(g,'gray'), plt.title('ORIGINAL')
plt.subplot(122),plt.imshow(outImg[:,:],'gray'), plt.title('PREDICTED1')
plt.show()

del cnnModel

when i run 'python run_training.py',the error is:

1. Create directory for the results (if not already existing)
   Dir already existing
   copy the configuration file in the results folder

2. Run the training on GPU with nohup
   nohup: ignoring input and redirecting stderr to stdout

Hi,

I applied your model to another data set. I let the program run 60 epochs, and the loss plot is shown as following. In accordance with the plot, do you think the program has converged? It seems to me that loss change has become flat, but the change of loss is not very significant between epoch 1 and epoch 60.

Any insightful suggestions are very appreciated!

Hey,

I'm curious about your expansion path.
From retina-unet/src/retinaNN_training.py:

 up1 = UpSampling2D(size=(2, 2))(conv3)
 up1 = concatenate([conv2,up1],axis=1)
 conv4 = Conv2D(64, (3, 3), activation='relu', padding='same',data_format='channels_first')(up1)
 conv4 = Dropout(0.2)(conv4)
 conv4 = Conv2D(64, (3, 3), activation='relu', padding='same',data_format='channels_first')(conv4)

I am not 100% sure, but I think there's a 2x2 convolution missing.
From the U-net white paper:

Every step in the expansive path consists of an upsampling of the
feature map followed by a 2x2 convolution (“up-convolution”) that halves the
number of feature channels, a concatenation with the correspondingly cropped
feature map from the contracting path, and two 3x3 convolutions, each followed
by a ReLU.

So:
Upsampling
then 2x2 Convolution
then the concatenation
then 3x3 convolution
then a second 3x3 convolution

I have to admit, I find the wording of the paper a bit confusing, so I wanted to ask you, what you think about this. Did you maybe leave the layer out on purpose? Or just an oversight? I'm currently trying to get a U-net for MRI brain tumour segmentation going, but I'm running out of ideas why it fails (It reproduces the full brain rather than only the tumour area) so in my search I came across your repo and the difference in layer architecture.

Kind regards

Edit:
I believe maybe in Keras 1 and 2 the setup of the layers is a bit different.

Hi,I am running this model on another data set.For this code,I have two questions.

1. In the training process, ./lib help_functions the method of mask_Unet

           if  masks[i,j] == 0:
                new_masks[i,j,0]=1
                new_masks[i,j,1]=0
            else:
                new_masks[i,j,0]=0
                new_masks[i,j,1]=1

why do you always make the value of 'new_masks[i,j,1]' equal to the 'masks[i,j]'s'? Suppose there are 3 classes,

 if  masks[i,j] == 0:
           new_masks[i,j,0]=0
           new_masks[i,j,1]=0
           new_masks[i,j,2]=0
  elif masks[i,j] == 1:
           new_masks[i,j,0]=0
           new_masks[i,j,1]=0
           new_masks[i,j,2]=1
  else:
           new_masks[i,j,0]=0
           new_masks[i,j,1]=1
           new_masks[i,j,2]=0

whether it is correct or not?
2.Correspondingly,in the predict process,./lib help_function the method of pred_to_imgs

if mode=="original":
        for i in range(pred.shape[0]):
            for pix in range(pred.shape[1]):
                pred_images[i,pix]=pred[i,pix,1]

I want to know the reason why pred_images[i,pix]=pred[i,pix,1] instead of pred_images[i,pix]=pred[i,pix,0]. Is it correct that the pred_images[i,pix]=max(pred[i,pix,:]) takes place of pred_images[i,pix]=pred[i,pix,1]?
your help is greatly appreciated. Thanks.

If I run the code directly, everything is right.

one day, I delete these:
#conv10 = Conv2D(2, (1, 1), activation='relu', padding='same')(conv9)
#conv10 = core.Reshape((2,patch_height*patch_width))(conv10)
#conv10 = core.Permute((2,1))(conv10)
....
....
#patches_masks_train = masks_Unet(patches_masks_train) #reduce memory consumption

And when I train the model, I find:
1265s - loss: 14.6464 - acc: 0.2041 - val_loss: 14.2844 - val_acc: 0.2732
Epoch 2/150
Epoch 00001: val_acc did not improve
1425s - loss: 14.1214 - acc: 0.2283 - val_loss: 14.2647 - val_acc: 0.2608
...

Why I must reshape the mask????

nohup: ignoring input and redirecting stderr to stdout

Can this be modified for 3D medical images with loss function dice coefficient?

Did u do some experiments?

And if the g-net is from any paper?(u-net and g-net,which is better?)

I don't really understand the mean of "patches_masks_train = masks_Unet(patches_masks_train)"...
in masks_Unet:
masks = np.reshape(masks,(masks.shape[0],im_him_w))
new_masks = np.empty((masks.shape[0],im_him_w,2))
I think each line in new_masks is a pic ,but what about "2"?

THANKS.

up1 = merge([UpSampling2D(size=(2, 2))(conv3), conv2], mode='concat', concat_axis=1)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/layers.py", line 460, in merge
name=name)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/layers.py", line 111, in init
node_indices, tensor_indices)
File "/usr/local/lib/python2.7/dist-packages/keras/legacy/layers.py", line 191, in _arguments_validation
'Layer shapes: %s' % (input_shapes))
ValueError: "concat" mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, 0, 24, 128), (None, 0, 24, 64)]

my keras version is 2.0.4
and i also change the keras.json file.
I try my best, and can not deal with this problem

Thank you for your dedicated work.

I want to ask why you used "softmax" and "categorical_crossentropy".

I understand this problem as "Multi Label classification", so usually we use
"sigmoid" and "binary cross entropy".

Is not this multi-label problem?

In short, what does the final layer(conv6 = core.Reshape((2,patch_height*patch_width))(conv6))
mean? especailly I wonder about "2".

thank you for any advices in advance.

Hi,
I met this problem when I training the model:
Traceback (most recent call last):
File "./src/retinaNN_training.py", line 107, in
model = get_unet(n_ch, patch_height, patch_width) #the U-net model
File "./src/retinaNN_training.py", line 48, in get_unet
up1 = merge([UpSampling2D(size=(2, 2))(conv3), conv2], mode='concat', concat_axis=1)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/engine/topology.py", line 1539, in merge
name=name)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/engine/topology.py", line 1170, in init
node_indices, tensor_indices)
File "/usr/local/lib/python2.7/dist-packages/Keras-1.1.2-py2.7.egg/keras/engine/topology.py", line 1237, in _arguments_validation
'Layer shapes: %s' % (input_shapes))
Exception: "concat" mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, 0, 24, 128), (None, 0, 24, 64)]

I don't know how to fix it. I'm new to keras, could you please to help me?
Thank you for reading my question.

I've downloaded the DRIVE datasets but when I run this:

python run_testing.py

I get this error:

Traceback (most recent call last):
  File "./src/retinaNN_predict.py", line 111, in <module>
    model = model_from_json(open(path_experiment+name_experiment +'_architecture.json').read())
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/models.py", line 345, in model_from_json
    return layer_module.deserialize(config, custom_objects=custom_objects)
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/layers/__init__.py", line 54, in deserialize
    printable_module_name='layer')
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/utils/generic_utils.py", line 139, in deserialize_keras_object
    list(custom_objects.items())))
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/engine/topology.py", line 2497, in from_config
    process_node(layer, node_data)
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/engine/topology.py", line 2456, in process_node
    layer(input_tensors, **kwargs)
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/engine/topology.py", line 602, in __call__
    output = self.call(inputs, **kwargs)
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/legacy/layers.py", line 212, in call
    return K.concatenate(inputs, axis=self.concat_axis)
  File "/Users/pontikos/Library/Python/2.7/lib/python/site-packages/keras/backend/tensorflow_backend.py", line 1709, in concatenate
    return tf.concat([to_dense(x) for x in tensors], axis)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/array_ops.py", line 1034, in concat
    name=name)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/ops/gen_array_ops.py", line 519, in _concat_v2
    name=name)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 2338, in create_op
    set_shapes_for_outputs(ret)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1719, in set_shapes_for_outputs
    shapes = shape_func(op)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/ops.py", line 1669, in call_with_requiring
    return call_cpp_shape_fn(op, require_shape_fn=True)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 610, in call_cpp_shape_fn
    debug_python_shape_fn, require_shape_fn)
  File "/Users/pontikos/anaconda2/lib/python2.7/site-packages/tensorflow/python/framework/common_shapes.py", line 676, in _call_cpp_shape_fn_impl
    raise ValueError(err.message)
ValueError: Dimension 1 in both shapes must be equal, but are 12 and 24 for 'merge_1/concat' (op: 'ConcatV2') with input shapes: [?,256,24,12], [?,64,24,24], [] and with computed input tensors: input[2] = <1>.

I was running the repo on keras 1 and it worked fine. Now while upgrading keras from version 1 to 2 some incompatibility issues have arisen. Is there any plan to make it compatible with keras 2

Hi there,

After execute prepare_datasets_DRIVE.py script, it shows following message on Ubuntu16.04. Any idea what's the reason behind it? Many thanks.

ground truth and border masks are correctly withih pixel value range 0-255 (black-white)
saving train datasets
Traceback (most recent call last):
File "prepare_datasets_DRIVE.py", line 82, in
write_hdf5(imgs_train, dataset_path + "DRIVE_dataset_imgs_train.hdf5")
File "prepare_datasets_DRIVE.py", line 15, in write_hdf5
with h5py.File(outfile,"w") as f:
File "/usr/lib/python2.7/dist-packages/h5py/_hl/files.py", line 272, in init
fid = make_fid(name, mode, userblock_size, fapl, swmr=swmr)
File "/usr/lib/python2.7/dist-packages/h5py/_hl/files.py", line 98, in make_fid
fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/_objects.c:2577)
File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/_objects.c:2536)
File "h5py/h5f.pyx", line 96, in h5py.h5f.create (/build/h5py-nQFNYZ/h5py-2.6.0/h5py/h5f.c:1984)
IOError: Unable to create file (Unable to open file: name = './drive_datasets_training_testing/drive_dataset_imgs_train.hdf5', errno = 2, error message = 'no such file or directory', flags = 13, o_flags = 242)

patches_masks_train = masks_Unet(patches_masks_train) #reduce memory consumption

Have you defined masks_Unet function ?

Traceback (most recent call last):
File "./src/retinaNN_training.py", line 15, in <module>
from keras.layers import Input, concatenate, Conv2D, MaxPooling2D, UpSampling2D, Reshape, core, Dropout
ImportError: cannot import name concatenate

I got this problem when running python run_training.py. Please help...

Thank you very much!

Traceback (most recent call last):
File "./src/retinaNN_predict.py", line 212, in
precision = float(confusion[1,1])/float(confusion[1,1]+confusion[0,1])
ZeroDivisionError: float division by zero
===============This is my configration.txt===============
[data paths]
path_local = ./DRIVE_datasets_training_testing/
train_imgs_original = DRIVE_dataset_imgs_train.hdf5
train_groundTruth = DRIVE_dataset_groundTruth_train.hdf5
train_border_masks = DRIVE_dataset_borderMasks_train.hdf5
test_imgs_original = DRIVE_dataset_imgs_test.hdf5
test_groundTruth = DRIVE_dataset_groundTruth_test.hdf5
test_border_masks = DRIVE_dataset_borderMasks_test.hdf5

[experiment name]
name = my_test201609112025

[data attributes]

Dimensions of the patches extracted from the full images

patch_height = 48
patch_width = 48

[training settings]

number of total patches:

N_subimgs = 40

if patches are extracted only inside the field of view:

inside_FOV = False

Number of training epochs

N_epochs = 100
batch_size = 32

number of full images for the validation (max 20)

full_images_to_test = 20

if running with nohup

nohup = False

[testing settings]

Choose the model to test: best==epoch with min loss, last==last epoch

best_last = best

number of full images for the test (max 20)

full_images_to_test = 20

How many original-groundTrutuh-prediction images are visualized in each image

N_group_visual = 1

Compute average in the prediction, improve results but require more patches to be predicted

average_mode = False

Only if average_mode==True. Stride of the average, lower value require more patches to be predicted

stride_height = 45
stride_width = 45

if running with nohup

nohup = False

What should I do? Thanks for any help.

======== Evaluate the results =======================
Calculating results only inside the FOV:
y scores pixels: 2429685 (radius 270: 2702703.14==228906), including background around retina: 6599200 (584565==329960)
y true pixels: 2429685 (radius 270: 2702703.14==228906), including background around retina: 6599200 (584565==329960)
Traceback (most recent call last):
File "./src/retinaNN_predict.py", line 183, in
precision, recall, thresholds = precision_recall_curve(y_true, y_scores)
File "/home/liushixin/anaconda3/lib/python3.6/site-packages/sklearn/metrics/ranking.py", line 441, in precision_recall_curve
sample_weight=sample_weight)
File "/home/liushixin/anaconda3/lib/python3.6/site-packages/sklearn/metrics/ranking.py", line 318, in _binary_clf_curve
raise ValueError("{0} format is not supported".format(y_type))
ValueError: continuous format is not supported
anyone can give some advice?

up1 = merge([UpSampling2D(size=(2, 2))(conv3),conv2], mode='concat', concat_axis=1)
This line is causing this error (

Traceback (most recent call last):
File "retinaNN_training.py", line 144, in
model = get_unet(nb_epoch,img_width,img_hight) #the U-net model
File "retinaNN_training.py", line 47, in get_unet
up1 = merge([UpSampling2D(size=(2, 2))(conv3),conv2], mode='concat', concat_axis=1)
File "/share/apps/tensorflow_anaconda2/envs/tensorflow/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/topology.py", line 1680, in merge
name=name)
File "/share/apps/tensorflow_anaconda2/envs/tensorflow/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/topology.py", line 1299, in init
node_indices, tensor_indices)
File "/share/apps/tensorflow_anaconda2/envs/tensorflow/lib/python2.7/site-packages/Keras-1.2.0-py2.7.egg/keras/engine/topology.py", line 1371, in _arguments_validation
'Layer shapes: %s' % (input_shapes))
ValueError: "concat" mode can only merge layers with matching output shapes except for the concat axis. Layer shapes: [(None, 128, 84, 84), (None, 64, 85, 85)]
) Although I changed dim-ordering to "th", and Keras version 1.2.0.
???

hi, what is minimum requirements to run this code?
what is the python version? 3 or 2.7