how about the performance? about espnet HOT 39 CLOSED

You didn’t turn on the decoder flag. you end up training ESPNet-C and not ESPNet.

Please use your current model as pretrained encoder (instead of ours) and train it with decoding flag.

from espnet.

Try batch size of 6 or 8.

from espnet.

yeah I found it
I resized the feature map
I should upsample the model out
it's work now
thank you !

from espnet.

yes

from espnet.

Something is wrong with your evaluation.

We achieve mIOU of 53.3 and 61.4 for ESPNet-C and ESPNet on the Cityscapes validation set. Are you using the evaluation scripts provided by the Cityscapes dataset?

from espnet.

I just using the evaluation scripts provided by the Cityscapes dataset, and the mIOU is 43.5% with your released ESP_C model(p=2, q=8), can you give me some advice? Thansk very much

from espnet.

I am assuming that you are using VisualizeResults.py file in the test folder. This file generates the label images at a resolution of 1024x512.

To use Cityscapes scripts, you need to upsample these generated images by a factor of 2 so that your label image size is the same as the input image i.e. 2048x1024. Could you please tell me how are you up sampling your label images to get to this size?

from espnet.

I just using VisualizeResults.py file with input_size=2048x1024, I wonder is it same with the 1024x512 and then upsampling with factor of 2?

from espnet.

We never trained/tested our models at this high resolution because it will demand enormous resources which are not available on embedded devices. Could you try generating the results at 1024x512 resolution and see if you are able to generate the reported numbers?

Note 1: if you are upsampling segmentation masks, then please use nearest neighbor interpolation.

Note 2: if you are upsampling the feature maps of last layer, then use Bilinear interpolation and then apply softmax to get the final feature maps.

P.S. you can finetune ESPNet models at high resolution. I believe fine tuning at high resolutions will further improve its accuracy.

from espnet.

OK, thanks very much for you real time answers and very useful advices. I will do experiments following your suggestions and then share results later。

from espnet.

May I ask about your mIOU during training? I can get only 0.47 mIOU in splited validation dataset and 0.50mIOU in train dataset during training .

from espnet.

Which scripts are you using to evaluate mIOU?

from espnet.

I have not evaluate the model on test set of cityscapes dataset, the data of miou is from trainlog.

from espnet.

This looks good. Train it for 300 epochs and you should be Okay.

Note: official mIOU metric used for evaluation on the Cityscapes is different than the one which we have in our code. Please evaluate your best model (with min validation loss) on the Cityscapes validation set using their scripts to compare the number reported in paper.

from espnet.

thank you for your quickly answering, I'm waiting for the 300 epochs.

from espnet.

For our numbers on the Cityscapes validation set, please see Table 2(f) in the paper which reports the data for both ESPNet-C as well as ESPNet.

Good luck and thanks for showing interest in our work!

from espnet.

hi, this is my full train log, I use the pretraind encoder in your github to train ESPNet, but It seems that I can't get the good performance as you. Do u have any advices for training? how to tune the hyperparameters?
trainValLog.txt

from espnet.

Hi,

This looks good to me. Please evaluate it using the Cityscapes dataset scripts because they are different as they use weighted mIOU; different from the one we provide. Once you evaluate on that, you will see similar performance.

Note that please use your best model for generating results. That is, the model with least validation loss.

from espnet.

Thank you @sacmehta , I'll try tomorrow.It's really nice of you ^ _ ^

from espnet.

hello, this is my result with bilinear interpolation on val set , And I think it's worse than you

from espnet.

Could you please tell me if you use ESpNet-C or ESPNet?

from espnet.

ESPNet,I use the pretrained ESPnet-c which you provided in the code to train the model

from espnet.

Which configuration?

from espnet.

all hyperparameters are default value in your code. Do you have any advice?

from espnet.

Can you share the command you used to train the model?

from espnet.

python main.py
look at this

from espnet.

sorry I didn't remember whether did I changed the setting during training with the command -decoder=True, but I'll figure it out.

from espnet.

yeah,in the model dict ,there are two up layer. and the folder to contain the result is "results_enc__dec_2_8"
So I think I have turn it on during training with the command line.
Do you have any suggestions?

from espnet.

but the size of parameters is different. I must have done something wrong.
Thank you for your help. I'm going to retrain one.

from espnet.

I believe this is my final result for ESPNet, I have checked the number of "parameters" in train log, and it's right for ESPNet.
May I ask u for any advice for hyperparameters? are you using the default setting in your code to train your model?

from espnet.

hi，this is the evaluated result in validation set on cityscapes val dataset.(with bilinear resize to 2048*1024)
I use your espnet_p2_q8 with decoder provided in the code .
Did you released your best model? or whether I've done something wrong?

from espnet.

Provided model is our best model on the validation set.

Did you resize feature maps or segmentation masks using Bilinear interpolation?

from espnet.

Also, check unique values in the generated segmentation masks. They should be between 0 and number of classes. You can check this using

numpy.unique

from espnet.

Are you able to attain the reported accuracy?

from espnet.

hi, @sacmehta ,have you done something to augment the original dataset, I can only get 0.59 below your 0.61 on cityscapes validation set

from espnet.

No, we didn’t use any additional augmentation.

+/- 2 points deviation is kind of expected. We used a batch size of 12 for ESPNet-c and 6 for ESPNet. What batch size did you use?

from espnet.

same as you, 6 for ESPNet .
thank you very much!

from espnet.

@sacmehta Is it necessary to train 300 epochs? The val loss and mIoU are sluggish after the early epoch.

from espnet.

My result is val mIoU 0.601.

from espnet.