Hi,
Thanks for sharing your work!

In your code:
if (args.lambda_semi > 0 or args.lambda_semi_adv > 0) and i_iter >= args.semi_start_adv:

as far as i understand, you want to check whether to train the generator with unlabeled data(so that lambda_semi or semi_adv should be larger than 0), and also you check if semi_start_adv is smaller than iteration number. If you do that, inside if case would be run just once. I think it should be i_iter <= args.semi_start_adv, so that once starting number of semi-adversarial training more than number of iterations that have been run, generator weights will be updated with adversarial loss and self thought learning?

Am I missing something?

I appreciate your answer, thanks! Ecem

I would like to ask what is the use of utils.metric.py?

Hello,thanks for your work. In your article ,you said your remove the last classification layer and use the Atrous Spatial Pyramid Pooling method in the last layer. But in your deeplab.py, it use the classification_module and no mention to the ASPP method. I can not understand why?Can you replay me?...please

Hello!
During training loss_D value decrease to ~0.2 and segmentation network cant learn nothing. Does it mean that discriminator dominate under generator? what range of loss_d value is normal(for classical gan is 0.69) )? Maybe it's possible to attach some loss graphics?

In line 371 of script train.py:
loss = loss_seg + args.lambda_adv_pred * loss_adv_pred

It seems to me that the loss_adv_pred part doesn't do anything for the training, since you have set require_gradient for the discriminator to False in line 309-311:

# don't accumulate grads in D
for param in model_D.parameters():
param.requires_grad = False

Am I thinking it wrong? If so, could you please tell me about it?

Thanks for sharing.i have got a issure about command parameters.
write thispython evaluate_voc.py --help in terminal,and thers is a optional --num-class

the original code is:NUM_CLASSES = 21,i wonder what will happen if i code NUM_CLASSES = 3 .
what 's the order of the object .is it random or has a default order?

Hello, I want to ask one question related to the loss_adv_p and loss_D. As the log, the loss_adv_p tends to increase while the loss_D tends to decrease. We expect that loss_D will decrease, it is correct; however, how about the loss_adv_p? I found one interesting thing that is args.lambda_adv_pred, you can set it to 0.5, then the loss_adv_p will decrease but the loss loss_D will increase. They looks opposite side Thanks

iter =        0/   20000, loss_seg = 3.045, loss_adv_p = 0.695, loss_D = 0.693, loss_semi = 0.000

and at 180 iters

iter =      180/   20000, loss_seg = 0.490, loss_adv_p = 1.358, loss_D = 0.572, loss_semi = 0.000

I have a question with the choice of mask_T. How did you choose its value? I think 0.2 is too small. Maybe larger value may be better?

I don't understand why you use the VOCGTDataSet instead of VOCDataSet , they choose the same data from voc train dataset :train_ids[:partial_size],,,I think the difference between these two class is whether to pad the image and label,
first:
ignore_mask = (labels.numpy() == 255)
ignore_mask_gt = (labels_gt.numpy() == 255)
then:

train with pred

D_out = interp(model_D(F.softmax(pred)))
loss_D = bce_loss(D_out, make_D_label(pred_label, ignore_mask))

train with gt

D_gt_v = Variable(one_hot(labels_gt)).cuda(args.gpu)
D_out = interp(model_D(D_gt_v))
loss_D = bce_loss(D_out, make_D_label(gt_label, ignore_mask_gt))

I think these two loss_D is already have different D_out,is it necessary to distinguish VOCGTDataSet and VOCDataSet ?

Hey ,
When I try to validate the results , I am getting this error

  output = model(Variable(image, volatile=True).cuda(gpu0))
Traceback (most recent call last):
  File "evaluate_voc.py", line 235, in <module>
    main()
  File "evaluate_voc.py", line 214, in main
    output = model(Variable(image, volatile=True).cuda(gpu0))
  File "/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/summer_interns/jose/AdvSemiSeg/model/deeplab.py", line 167, in forward
    x = self.conv1(x)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/modules/module.py", line 491, in __call__
    result = self.forward(*input, **kwargs)
  File "/home/ubuntu/.local/lib/python2.7/site-packages/torch/nn/modules/conv.py", line 301, in forward
    self.padding, self.dilation, self.groups)
RuntimeError: CUDNN_STATUS_MAPPING_ERROR

hello, can you please tell me what the [ def make_D_label(label, ignore_mask): ]mean in train.py line 175-179.
thanks

It seems loss_semi_adv&&loss_adv_pred alwayes contribute to the network during training,
I understandt that :
loss_adv_pred = bce_loss(D_out, make_D_label(gt_label, ignore_mask))
is using the gt_label,cause the pred is come from labeled data,but
loss_semi_adv = args.lambda_semi_adv * bce_loss(D_out, make_D_label(gt_label, ignore_mask_remain))
why this loss also using the gt_label,the data is come from the trainloader_remain_iter...?

Hello, in your implementation, you first train G network, then train on D network. The code can be summarized as

#=========Train on G=========
 for param in model_D.parameters():
                param.requires_grad = False
loss_seg = loss_calc(pred, labels, args.gpu)
loss_adv_pred = bce_loss(D_out, make_D_label(gt_label, ignore_mask))
loss = loss_seg + args.lambda_adv_pred * loss_adv_pred
loss.backward()
#=======Train on D===========
for param in model_D.parameters():
                param.requires_grad = True
pred = pred.detach()
#-----Fake-------
loss_D = bce_loss(D_out, make_D_label(pred_label, ignore_mask))
loss_D.backward()
#-----Real--------
loss_D = bce_loss(D_out, make_D_label(gt_label, ignore_mask_gt))
loss_D.backward()
optimizer.step()
optimizer_D.step()

However, follow the DCGAN example, I make more simple way that does not requires setting param.requires_grad = False/True. This is my modified

#=======Train on D===========
#-----Real--------
loss_D = bce_loss(D_out, make_D_label(gt_label, ignore_mask_gt))
loss_D.backward()
#-----Fake-------
pred= model(image)#add more here
pred = pred.detach()
loss_D = bce_loss(D_out, make_D_label(pred_label, ignore_mask))
loss_D.backward()
optimizer_D.step()
#=========Train on G=========
loss_seg = loss_calc(pred, labels, args.gpu)
loss_adv_pred = bce_loss(D_out, make_D_label(gt_label, ignore_mask))
loss = loss_seg + args.lambda_adv_pred * loss_adv_pred
loss.backward()
optimizer.step()

In this modification, I trained the D network first with real label and detach for the fake label. We did not use param.requires_grad = False/True because detach did it. The code is enabled to run. However, I did not guarantee it equivalent with your purpose or not "During the training G network, we fix the D network`. Could you give me some comment?

Reference :https://discuss.pytorch.org/t/when-training-gan-why-do-we-not-need-to-zero-grad-discriminator/20439

In the update step of the discriminator (line 208), the generator does not get the data, so the backward step does not calculate any gradients for it.
In line 217 the input to the discriminator is detached as you already observed. Thus the backward call of errD_fake also does not calculate the gradients for the generator.
Before updating the generator (line 225 etc.) the gradients are zeroed, so it looks alright

Hi
Thank you for release the code and answer my question.
how can I train a new advFull model ?
just set PARTIAL_DATA=1？ in train.py

Hi, when I use your code by your README, I find that the model training is fintuning on your well-trained model. When I choose not to fintuning , and want to retrain the data VOC2012. Some wrongs with the loss. The losses don't converge. I want to konw if anything wrong with your code? emmm, I with you to test your code. Thanks.

Is't possible to start training in multi gpu mode? seems that it's not possible.

parser.add_argument("--gpu", type=int, default=0,
help="choose gpu device.")

Hi, I am wondering why you use trainloader_iter and trainloader_gt_iter at the same time during your training process, since they are both refer to the labeled dataset. During the D training, only labels_gt is used from trainloader_gt_iter, what is the difference if using labels from trainloader_gt_iter?

Hi,

Thanks for sharing your code, i find your work very interesting!

I would like to use your work for my research, I have been checking your code, and as far as i understand, you have a function make_D_labels which return a matrix with all values 1 in case they come from GT or 0 when they come from generator.

However, you have an ignore_mask which you use it for assigning some part of D_label matrix to 255 . I do not understand this. Why would you assign 255 to any item of the matrix? Could you explain what i am missing?

Thanks for your help, I appreciate it!

Regards,Ecem

hi, thanks for your work.
when i run train.py, the error occurs : file"train.py",line 375,in main
loss_seg_value+=loss_seg.data.cpu( ).numpy()[0]/args.ier_size
indexError: too many indices fro array

i don't know how to solve it ,please help me

hello, when I try ti run train.py, an error accurs: ImportError: No module named packaging
i use python2.7 pytorch=0.4.0 cv2=3.4.2
can you please tell me how to solve it
thanks very much

Thanks for sharing your work. I used evaluate code and I got the error

    import copy_reg
ModuleNotFoundError: No module named 'copy_reg'

I am using python 3.6 from conda install, Ubuntu 16.04. How can I fix it? Thanks

In "read me", you commented that the mean IOU should be around 68.8% for the pre-trained model for 0.125 set (adv+ semi).

However, in the paper the result is 69.5% other than 68.8%.

Why did you comment that it is around 68.8%? (Actually, the pre-trained weight from you got 69.4% in my computer, not around 68.8%, which is similar with the result of the paper.)

(In addition, when I retrained your code with the batch size of 8 for RTX 2080 Ti.
The result was 68.5%, which is around 68.8%, but I think this is because of the batch size.)

raise RuntimeError('Attempting to deserialize object on a CUDA '
RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.load with map_location=torch.device('cpu') to map your storages to the CPU.

How can i run it on cpu.Hlep me,please.Thank you!

when I train the model on VOC2012, it stops at the 160/20000 .

Traceback (most recent call last):
File "train.py", line 474, in
main()
File "train.py", line 335, in main
_, batch = trainloader_remain_iter.next()
File "/home/zhengkai/anaconda3/envs/tensorflow/lib/python3.5/site-packages/torch/utils/data/dataloader.py", line 572, in next
raise StopIteration
StopIteration

Excuse me, There are one more questions...hope it not bothing you too much...
"The output of D is a probability map with values ranges from 0-1. When the input is unlabeled data, we use the D output map as the indicator for semi-supervised learning."

D_out = interp(model_D(F.softmax(pred)))
D_out_sigmoid = F.sigmoid(D_out).data.cpu().numpy().squeeze(axis=1)
##following is used to draw the confindence map in your papar Figure3
for i in range(args.batch_size):
plt.imshow(D_out_sigmoid[i], cmap='jet')
plt.show()
Can I interpret the D_out_sigmoid is the confidence map you mentioned?
I use VOC_20000.pth and VOC_20000_D.pth(when use these two model the evaluatation mIOU is 0.709) as the parameters for RESTORE_FROM and RESTORE_FROM_D,and tried to get the figure of confidence map,but it looks not matching so well,it looks weird ,can you explain how you get the confidence map in your paper?

想请问一下在训练得到confidence map过程中判别器的输出map大概是怎样的呢，我现在一直找不到分割网络和判别网路的训练平衡点，我做的肝脏病灶分割，分割网络预先训练使其达到一定的分割效果才加的判别器，判别器loss会降到很低，加上判别器分割精度会下降，请问您对我的训练有什么建议吗

Hello, I am planning to use your data to run your network but it fails. My data is in two categories. One is for the background and the other is set to 0 and 1, respectively. For the single channel label map, use your The network has been trained, but the test result output is all 0. May I ask if I need to adjust some parameters, thank you

I want to run the code for "DeepGlobe Land Cover Classification Challenge dataset". Please explain me, what are the changes I need to make in the code.

Thanking you

When I run the evaluation code, I get a mean IoU of 47% as against 68.8% as mentioned. This was done using pretrained model using 1/8 training data. Is there something that I have done wrong or some changes that have to be made to the code before running evaluation? The dataset was VOC2012.

Hello, I'm very sorry. Now that I understand that the gradient is set to false in your code, I've always understood that adv is in D, and it's actually in G. I know that the hyperparameters should be modified based on the data set. Since I misunderstood the "confrontation", I misunderstood your code and I'm sorry. I consulted with others before I discovered my mistake. Thank you for your reply.

Hi! First, I'd like to thank you for the very helpful repo. During training with the same discriminator as yours, I have met some problems which I can't figure out. 1. The discriminator loss of pred and GT are almost unchanged for most of the time. So I am wondering if this phenomenon is normal. Btw, I haven't added semi data yet.

@hfslyc
设置梯度回传为False，这个位置不对。按照你的代码推理，对抗过程应该不能设置梯度为False。

train D

bring back requires_grad

for param in model_D.parameters():
param.requires_grad = True

train with pred

pred = pred.detach()

您可以给出解释么？我在语义分割对抗方面是小白。

Hello, this is not the bug in your code. I just want to ask a question about my code
I used your code in my dataset. I found that both the loss_adv and loss_D tend to decrease. While the correct trend must be loss_adv increases and loss_D decrease. What is my problem? Could you give me some suggestions? Thanks

Hello, first of all, thank you for sharing your work. I have a question in the process of looking at the code.ignore_mask = (y.numpy() == 255) Can you explain this to me? If my label y has a value of 0-9, can it be written like this? I look forward to your reply. Thank you very much.

loss_semi_adv_value += loss_semi_adv.data.cpu().numpy()[0]/args.lambda_semi_adv
IndexError: too many indices for array

It is not a bug. I just want to ask this question to make more clearly understand.
Your discriminator network produces a spatial map by using fully convolution layer.

1. Does the spatial map same as the confident map? Or confident map generates by unlabeled data?
2. Does the value of the spatial map (from D network) in range 0 to 1 or just {0,1}?

I got RuntimeError: CUDNN_STATUS_EXECUTION_FAILED when executing evaluation_voc.py

My conda list is below
python 2.7.17 h9bab390_0
python-dateutil 2.8.1 py_0
pytorch 0.4.0 py27hdf912b8_0
cudatoolkit 9.0 h13b8566_0
cudnn 7.6.4 cuda9.0_0

The weird thing is that train.py, which uses the same modules, works well with the same environment.

P.S. If the version of pytorch is upper than 0.4, the following adjustment is required.
with torch.no_grad():
for index, batch in enumerate(testloader):
if index % 100 == 0:
print('%d processd'%(index))
image, label, size, name = batch
size = size[0].numpy()
output = model(image.cuda(args.gpu))
output = interp(output).cpu().data[0].numpy()

You have three losses: loss_seg, loss_Adv and loss_semi. Each loss used batch size of 10. So, what is batch size of combined loss=loss_seg + alpha *loss_Adv +beta*loss_semi. Is it still 10 or 30? Thanks

@hfslyc

Thanks for your great work. Did you implement the code to reproduce the experiments in Cityscape dataset in the paper? Thanks!

Thanks for sharing your work. I used evaluate code and I got the error.Here is the detail:

Traceback (most recent call last):
File "evaluate_voc.py", line 236, in
main()
File "evaluate_voc.py", line 210, in main
for index, batch in enumerate(testloader):
File "C:\Users\landlord\Anaconda3\envs\aa\lib\site-packages\torch\utils\data\dataloader.py", line 264, in next
batch = self.collate_fn([self.dataset[i] for i in indices])
File "C:\Users\landlord\Anaconda3\envs\aa\lib\site-packages\torch\utils\data\dataloader.py", line 264, in
batch = self.collate_fn([self.dataset[i] for i in indices])
File "E:\AdvSemiSeg-master\dataset\voc_dataset.py", line 58, in getitem
img_h, img_w = label.shape
> AttributeError: 'NoneType' object has no attribute 'shape'

I am using python 3.6 from conda install, win10. How can I fix it? Thanks

Hi,

Thanks again for sharing your code, your work is very interesting.

I see in your paper you mention that the discriminator is only trained with labeled data. However, you assign D_remain argument to True. And in the instructions you give in order to train the network for reproducing your results, you do not set it to False. Am i wrong?

Thanks,Ecem

Hello ,
How can i train my own dataset using custom classes?

File "D:\workspace\AdvSemiSeg\dataset\voc_dataset.py", line 56, in getitem
img_h, img_w = label.shape
AttributeError: 'NoneType' object has no attribute 'shape'

Can you help me? I tried some ways,but it still has errors.Thank you!

In this paper, the discriminator network takes a probability map of size H × W × C and outputs a confidence map of size H × W × 1. I don't understand how to train this discriminator network, what are the labels during the training phase ? As my understanding, when the probability map comes from the ground truth, the label is a map of size H × W × 1 and the values in the map are all 1, in contrast, when the probability map comes from the segmentation network, the label is a map of size H × W × 1 and the values in the map are all 0. Am I right? Hope to get your answer. Thanks so much

Hi there,

First, impressive work!

I'm reading your code but find one uncertain part: what's SEMI_START and SEMI_START_ADV represented for? I know it's used during training semi to determine the #iterations starting training. However in the original code, SEMI_START=5000 and SEMI_START_ADV=0, which means that semi is trained at the very FIRST iteration. Is that a bug since in the paper you mentioned "we start semi-supervised learning after training for 5000 iterations with labeled data"?

Any comments are appreciated!