zbf1991 / rrm Goto Github PK

Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach

Python 23.25% C++ 58.31% SWIG 0.11% C 18.33%

rrm's Introduction

Reliability Does Matter: An End-to-End Weakly Supervised Semantic Segmentation Approach

AAAI 2020 (Spotlight).

The code of the new extended work now is available. In the further, I will try to merge these two works so that making the whole project is more elegant.

Paper of the extended work will come soon.

This project is based on Regularized loss and PSA.

Before Running, build python extension module:

cd wrapper/bilateralfilter
swig -python -c++ bilateralfilter.i
python setup.py install

More details please see here

Download pretrained models to ./netWeights:

Google: due to the coronavirus outbreak in China, I will upload models after I can enter my lab. But you can download “[ilsvrc-cls_rna-a1_cls1000_ep-0001.params]” and “[res38_cls.pth]” from here.

BaiduYun

[ilsvrc-cls_rna-a1_cls1000_ep-0001.params] is an init pretained model.

[res38_cls.pth] is a classification model pretrained on VOC 2012 dataset.

[RRM_final.pth] is the final model (AAAI).

[RRM(attention)_final.pth] is the final model of the new extended work (64.7 mIoU on Pascal Voc 2012 val set).

Training of the New Extended Work:

Training from init model:

you need 4 GPUs and the pretrained model [ilsvrc-cls_rna-a1_cls1000_ep-0001.params]:

python train_from_init(attention).py --voc12_root /your/path/VOCdevkit/VOC2012

Training from a pretrained classification model:

you only need 2 GPU and the pretrained model [res38_cls.pth]

python train_from_cls_weight(attention).py --IMpath /your/path/VOCdevkit/VOC2012/JPEGImages

Training of the AAAI Work:

I suggest that it is better to use the 2nd method due to lower computing costs.

Training from init model:

you need 4 GPUs and the pretrained model [ilsvrc-cls_rna-a1_cls1000_ep-0001.params]:

python train_from_init.py --voc12_root /your/path/VOCdevkit/VOC2012

Training from a pretrained classification model:

you only need 1 GPU and the pretrained model [res38_cls.pth]

python train_from_cls_weight.py --IMpath /your/path/VOCdevkit/VOC2012/JPEGImages

Inferencing of the extended work:

you need 1 GPU and the final model [RRM(attention)_final.pth]:

python infer_RRM.py --IMpath /your/path/VOCdevkit/VOC2012/JPEGImages

Inferencing of the AAAI work:

you need 1 GPU and the final model [RRM_final.pth]:

python infer_RRM.py --IMpath /your/path/VOCdevkit/VOC2012/JPEGImages

rrm's People

Contributors

Stargazers

Watchers

Forkers

zhouleisjtu templeblock godlin-bd cv-ip gcv9htd hengbingwang colorfullywonderfully kenneth-x yangyang117 weixuansun pipizhum

rrm's Issues

Loss NAN

I trained the network with train_cls_weight.py and initialized the model with res38_cls.pth you provided. But the loss encounter NAN, can you help check why ?

dloss weight 1e-07
{'gpu_id': '0', 'LISTpath': 'voc12/train_aug(id).txt', 'IMpath': '/home/test/VOCdevkit/VOC2012/JPEGImages', 'SAVEpath': './output/model_weights', 'batch_size': 4, 'max_step': 20000, 'network': 'network.RRM', 'lr': 0.0007, 'num_workers': 16, 'wt_dec': 1e-05, 'weights': './netWeights/res38_cls.pth', 'session_name': 'RRM_', 'crop_size': 321, 'class_numbers': 20, 'crf_la_value': 4, 'crf_ha_value': 32, 'densecrfloss': 1e-07, 'rloss_scale': 0.5, 'sigma_rgb': 15.0, 'sigma_xy': 100}
Session started:  Sun Sep 27 12:50:05 2020
Iter:    0/20000 Loss:6.9927 imps:1.0 Fin:Mon Sep 28 10:08:25 2020 lr: 0.0007
closs: 0.0044 celoss: 9.4394 dloss: -0.6275
closs: 0.1205 celoss: 5.1730 dloss: -0.3368
closs: 0.1477 celoss: 5.8635 dloss: -0.2407
closs: 0.5365 celoss: 4.6322 dloss: -0.2464
closs: 0.2427 celoss: 3.2945 dloss: -0.3354
closs: 0.0825 celoss: 5.2225 dloss: -1.0677
closs: 0.0979 celoss: 9.1182 dloss: -1.2027
closs: 0.1901 celoss: 8.9809 dloss: -0.4709
closs: 0.3256 celoss: 1.4215 dloss: -0.1852
closs: 0.5821 celoss: 3.6866 dloss: -0.0932
closs: 0.4300 celoss: 3.5157 dloss: -0.2225
closs: 0.5388 celoss: 4.0678 dloss: -0.4153
closs: 0.2153 celoss: 0.7157 dloss: -0.1775
closs: 0.2832 celoss: 6.4929 dloss: -0.2787
closs: 0.3324 celoss: 3.6478 dloss: -0.1633
closs: 0.3553 celoss: 3.6112 dloss: -0.2413
closs: 0.2878 celoss: 2.4347 dloss: -0.2414
closs: 0.2624 celoss: 2.8557 dloss: -0.5394
closs: 0.0829 celoss: 3.8166 dloss: -0.4282
closs: 0.0380 celoss: 1.8013 dloss: -0.3638
closs: 0.0930 celoss: 3.9196 dloss: -0.6067
closs: 0.0150 celoss: 7.1038 dloss: -0.5287
closs: 0.0909 celoss: 0.9512 dloss: -0.4204
closs: 0.0452 celoss: 0.8439 dloss: -0.7635
closs: 0.0099 celoss: 5.4013 dloss: -0.7359
closs: 0.0284 celoss: 0.4117 dloss: -0.9293
closs: 0.0791 celoss: 2.7661 dloss: -0.5141
closs: 0.1172 celoss: 0.9836 dloss: -0.7071
closs: 0.2513 celoss: 2.3509 dloss: -0.4201
closs: 0.0033 celoss: 1.8597 dloss: -0.5371
closs: 0.0681 celoss: 2.0706 dloss: -0.8229
closs: 0.1058 celoss: 1.3337 dloss: -0.8299
closs: 0.0597 celoss: 1.5510 dloss: -1.1047
closs: 0.0299 celoss: 1.5824 dloss: -0.7586
closs: 0.0992 celoss: 2.3143 dloss: -1.0210
closs: 0.0791 celoss: 1.9394 dloss: -0.8417
closs: 0.0222 celoss: 0.4387 dloss: -0.4944
closs: 0.0288 celoss: 0.6625 dloss: -0.5628
closs: 0.0135 celoss: 4.5136 dloss: -0.6194
closs: 0.1020 celoss: 3.7248 dloss: -1.1224
closs: 0.0648 celoss: 1.3399 dloss: -0.3393
closs: 0.2042 celoss: 5.2593 dloss: -0.9651
closs: 0.0111 celoss: 1.0034 dloss: -0.3220
closs: 0.1350 celoss: 1.2016 dloss: -0.3987
closs: 0.1801 celoss: 1.7029 dloss: -0.6153
closs: 0.0074 celoss: 0.4721 dloss: -0.3116
closs: 0.0672 celoss: 0.2015 dloss: -0.8397
closs: 0.0198 celoss: 1.8567 dloss: -0.6717
closs: 0.0164 celoss: 0.7002 dloss: -0.7289
closs: 0.1535 celoss: 0.4099 dloss: -1.0501
Iter:   50/20000 Loss:2.5912 imps:0.5 Fin:Tue Sep 29 06:15:09 2020 lr: 0.0007
closs: 0.0186 celoss: 0.5534 dloss: -0.7522
closs: 0.0935 celoss: 2.2394 dloss: -0.9134
closs: 0.0657 celoss: 0.0391 dloss: -0.9369
closs: 0.0918 celoss: 3.0824 dloss: -1.0096
closs: 0.0200 celoss: 0.2935 dloss: -0.6191
closs: 0.0463 celoss: 0.5819 dloss: -0.7374
closs: 0.0586 celoss: 1.6121 dloss: -0.5797
closs: 0.0387 celoss: 1.6065 dloss: -0.6697
closs: 0.1740 celoss: 1.4451 dloss: -0.8965
closs: 0.0808 celoss: 1.7410 dloss: -0.3401
closs: 0.0106 celoss: 0.4464 dloss: -1.0401
closs: 0.1394 celoss: 0.8560 dloss: -1.4389
closs: 0.0874 celoss: 1.5509 dloss: -0.8592
closs: 0.1644 celoss: 1.1979 dloss: -1.9854
closs: 0.1198 celoss: 0.9197 dloss: -0.9195
closs: 0.1065 celoss: 1.2109 dloss: -0.6373
closs: 0.0588 celoss: 1.0543 dloss: -0.5841
closs: 0.0050 celoss: 1.8805 dloss: -1.6217
closs: 0.2087 celoss: 4.1127 dloss: -0.5923
closs: 0.0425 celoss: 0.4801 dloss: -0.6069
closs: 0.0653 celoss: 0.5786 dloss: -0.2389
closs: 0.2031 celoss: 1.5584 dloss: -0.4824
closs: 0.2697 celoss: 0.9956 dloss: -0.7905
closs: 0.0298 celoss: 1.3744 dloss: -0.5592
closs: 0.0852 celoss: 1.8596 dloss: -0.2178
closs: 0.1423 celoss: 3.2036 dloss: -0.8012
closs: 0.0112 celoss: 0.7400 dloss: -0.7337
closs: 0.0452 celoss: 0.0967 dloss: -0.9006
closs: 0.1238 celoss: 0.9540 dloss: -0.4174
closs: 0.1372 celoss: 0.4022 dloss: -0.4906
closs: 0.0215 celoss: 0.7961 dloss: -0.7226
closs: 0.2029 celoss: 0.7700 dloss: -1.0545
closs: 0.2938 celoss: 1.3062 dloss: -0.4279
closs: 0.1345 celoss: 0.1922 dloss: -0.4158
closs: 0.1429 celoss: 3.1052 dloss: -0.6353
closs: 0.1459 celoss: 1.4670 dloss: -0.8719
closs: 0.0420 celoss: 0.3905 dloss: -1.0492
closs: 0.1515 celoss: 0.7394 dloss: -1.0913
closs: 0.0812 celoss: 2.3438 dloss: -0.7482
closs: 0.0015 celoss: 0.2092 dloss: -0.6132
closs: 0.0264 celoss: 0.9715 dloss: -0.5522
closs: 0.0316 celoss: 0.0637 dloss: -0.7099
closs: 0.2186 celoss: 2.0530 dloss: -0.8735
closs: 0.0712 celoss: 0.9859 dloss: -1.4845
closs: 0.1248 celoss: 0.9471 dloss: -0.4290
closs: 0.1304 celoss: 1.3958 dloss: -0.4913
closs: 0.0504 celoss: 0.7157 dloss: -1.0267
closs: 0.0273 celoss: 1.2975 dloss: -0.4367
closs: 0.0822 celoss: 1.7112 dloss: -0.4826
closs: 0.1391 celoss: 1.1379 dloss: -0.7480
Iter:  100/20000 Loss:0.5579 imps:0.5 Fin:Tue Sep 29 10:22:49 2020 lr: 0.0007
closs: 0.0558 celoss: 1.2818 dloss: -0.6047
closs: 0.0448 celoss: 0.1854 dloss: -0.6605
closs: 0.0711 celoss: 0.7248 dloss: -0.4592
closs: 0.0851 celoss: 1.1428 dloss: -0.6737
closs: 0.0606 celoss: 0.6375 dloss: -1.0760
closs: 0.1295 celoss: 1.7562 dloss: -0.6098
closs: 0.0650 celoss: 0.8115 dloss: -0.5792
closs: 0.0359 celoss: 0.6091 dloss: -0.3923
closs: 0.1426 celoss: 0.7211 dloss: -0.4499
closs: 0.0850 celoss: 0.5728 dloss: -1.3284
closs: 0.0153 celoss: 1.0915 dloss: -0.5173
closs: 0.0204 celoss: 0.6239 dloss: -0.5377
closs: 0.0717 celoss: 0.8700 dloss: -0.6135
closs: 0.1348 celoss: 0.9156 dloss: -0.6275
closs: 0.0953 celoss: 0.6156 dloss: -0.5429
closs: 0.0737 celoss: 1.3502 dloss: -0.3975
closs: 0.1044 celoss: 0.6590 dloss: -0.7249
closs: 0.0416 celoss: 0.1282 dloss: -0.7763
closs: 0.0668 celoss: 0.4131 dloss: -0.8943
closs: 0.1354 celoss: 2.1792 dloss: -0.7357
closs: 0.1223 celoss: 0.3392 dloss: -0.6979
closs: 0.0507 celoss: 1.1406 dloss: -0.7358
closs: 0.0800 celoss: 0.5590 dloss: -1.5224
closs: 0.1624 celoss: 1.3533 dloss: -0.4674
closs: 0.0646 celoss: 0.4244 dloss: -0.6408
closs: 0.0753 celoss: 1.2881 dloss: -1.0271
closs: 0.0161 celoss: 0.0051 dloss: -2.0526
closs: 0.1103 celoss: 2.5632 dloss: -0.7065
closs: 0.0040 celoss: 0.0918 dloss: -0.6101
closs: 0.0128 celoss: 0.6130 dloss: -0.8321
closs: 0.0266 celoss: 0.2074 dloss: -0.7096
closs: 0.0671 celoss: 0.2723 dloss: -0.7433
closs: 0.0234 celoss: 0.3458 dloss: -0.8951
closs: 0.1489 celoss: 0.5624 dloss: -0.7095
closs: 0.0199 celoss: 1.2300 dloss: -1.1305
closs: 0.1348 celoss: 0.8209 dloss: -0.6234
closs: 0.1849 celoss: 1.1255 dloss: -0.5713
closs: 0.0594 celoss: 0.8439 dloss: -0.7159
closs: 0.0013 celoss: 0.0482 dloss: -1.5655
closs: 0.0320 celoss: 0.3643 dloss: -0.6873
closs: 0.0661 celoss: 1.1748 dloss: -0.6629
closs: 0.0166 celoss: 1.2994 dloss: -0.5595
closs: 0.0715 celoss: 0.2884 dloss: -1.0227
closs: 0.0228 celoss: 0.9772 dloss: -0.6697
closs: 0.2057 celoss: 0.2226 dloss: -0.6602
closs: 0.0184 celoss: 1.2756 dloss: -0.8218
closs: 0.0130 celoss: 0.2805 dloss: -0.5967
closs: 0.1203 celoss: 1.2384 dloss: -0.5204
closs: 0.1348 celoss: 0.4711 dloss: -0.6461
closs: 0.0274 celoss: 0.2033 dloss: -0.6084
Iter:  150/20000 Loss:0.0987 imps:0.5 Fin:Tue Sep 29 12:06:49 2020 lr: 0.0007
closs: 0.4298 celoss: 1.5731 dloss: -0.9354
closs: 0.0787 celoss: 3.2816 dloss: -0.7151
closs: 0.0946 celoss: 1.3763 dloss: -0.9236
closs: 0.1766 celoss: 0.9614 dloss: -0.7405
closs: 0.0379 celoss: 2.2669 dloss: -0.2658
closs: 0.1181 celoss: 0.6767 dloss: -0.7966
closs: 0.2176 celoss: 1.7115 dloss: -1.3958
closs: 0.2362 celoss: 0.9105 dloss: -0.9004
closs: 0.1016 celoss: 0.8248 dloss: -0.7804
closs: 0.0429 celoss: 2.7413 dloss: -1.1431
closs: 0.0257 celoss: 0.7514 dloss: -0.6107
closs: 0.5349 celoss: 0.9333 dloss: -0.5871
closs: 0.1494 celoss: 0.2364 dloss: -0.5980
closs: 0.2620 celoss: 0.7616 dloss: -0.5764
closs: 0.0516 celoss: 0.8762 dloss: -0.6489
closs: 0.2253 celoss: 0.2913 dloss: -0.5292
closs: 0.2918 celoss: 2.4652 dloss: -0.7501
closs: 0.3032 celoss: 3.9186 dloss: -0.3560
closs: 0.7368 celoss: 5.9555 dloss: -0.1446
closs: 3.4404 celoss: 28.8456 dloss: -0.2489
closs: 66284.4844 celoss: 1845719.8750 dloss: -0.1338
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan
closs: nan celoss: nan dloss: nan

How can I get the mIOU results?

你好，请问如何得到论文公布的结果？我好像只能得到伪标签图片

Would you tell me where is the function of evaluation in the RRM_infer.py?

Hey, thank you for your work and your codes. It is actually an impressing research.
I just wonder, when I run the infer file, why there is only output of images, but not the final mIoU value? This is kind strange and different for normal WSSS work.
Thank you for your time and waiting for your reply.

跑通了infer_RRM.py后只有结果图并没有显示关于mIOU指标的结果，请问需要如何操作？

The Energy loss

Hi,

Nice work! I was sorry to bother you.

I was just getting started in wsss porblem, I was very puzzled about the Energy loss because of the negative value, could you help me about my question.

Thank you very much.

Our-RRM-ResNet

Hi, nice to meet you. I read your paper and the performance of the model is excellent. But, you did not talk too much about Our-RRM-ResNet(101). Can you talk about the details of it? How did you train this model including learning rate, learning schedule, batch size, and others? Did you use any pre-train weight? Is this deeplabv2 the same as the official one? Sorry, I have so many questions, but I could not solve this problem for a long time. If it is possible, can you release the code?

In addition, I also noticed that you might encounter the same problem(resnet-38) before in the repository of psa. The author of psa/irnet did not talk much about the resnet-38 and resnet-50. The author suggested people use official deeplabv2 which was pretrained on COCO dataset(segmentation). However, I did not want to use it because I care about "Weakly Supervised". Also, the author suggested another GitHub with resnet38 but a few people could reproduce the result from it.

computer crash

when i run the infer_RRM.py and train_from_cls_weight.py, computer will automatically shut down after a short while. i have 1080ti 11G

zero-size array to reduction operation maximum which has no identity

Thanks for your work!
But I got a problem using my own dataset!
It said zero-size array to reduction operation maximum which has no identity that is caused by a empty array in this line bg_score = np.power(1 - np.max(v, axis=0, keepdims=True), alpha) in myTool.py, and I delete the cam_label thresh code if cam_label[i] > 1e-5:, it runs successfully but I got a negative dloss, I don't know whether it is correct?

why I could not use the RRM_10000.pth in PSA project?

Excuse me, maybe this question looks silly.
However, I do not understand why we could the res_38.pth, which is pretrained in psa project, in your framework. But I could not use the infer file in psa to infer the model RRM_10000.pth to test it performance. Thank you and waiting for your answer.

About the dataloader

When will you publish the implementation?

Hello， I have a question.
When will you publish the implementation?
Thank you.

What's the difference between the AAAI'20 work and the extended work?

outputs are all black images

I am gonna ask a really dumb question. I downloaded the pre-train weight and also train it with pascal voc dataset. But the output from infer_RRM.py didn't give me color decoded and segmented images but a complete black images. I guess it might caused by the weight I loaded into the network, or some stupid mistakes. Does anyone have the same problem as I did....?

Unable to reproduce results

Hi,

Thanks for the great work. I'm trying to reproduce your results, but I can't seem to get close to the scores you report in the paper.

I have tried two settings:

The default parameters as given in the code. This results in ~56.2 mIoU without CRF and ~57.6 mIoU with CRF
With parameters as given in the paper (lower learning rate (0.001) and lower image resolution (321)). This results in ~55.4 without CRF and ~57.4 with CRF.

Other than those things, I have not made any changes. Any idea why my scores are much lower? Are there other parameters which I need to set?

Thanks!

About the dense energy loss

Thank you for your repo. Great work.

I'm confused by the difference between your proposed dense energy loss and the dense crf loss from Regularized loss. As I noticed from your code, there is an additional "Gate" which performs as a soft filter as mentioned in the paper (if I understood correctly). Could you please explain more? Thank you.

Can people get the result in the paper if they train it from 'init'?

Hi, nice to meet you.
There are two pretrain weights about your classification network.
One is "ilsvrc-cls_rna-a1_cls1000_ep-0001.params" and another one is "res38_cls.pth".
My question is that can people get the same result in your paper if they train from "ilsvrc-cls_rna-a1_cls1000_ep-0001.params"?

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.