GPU out of memory about attentionmask HOT 9 CLOSED

Your GPU is too small. You can try follow the instructions in the README on reducing memory footprint, however, I highly doubt it will be enough to remove the scales 24, 48 and 96. You probably have to reduce the input image size. You can do this in the configs/myModel.config file by reducing SCALE for training or TEST_SCALE for testing. This will lead, however, to worse results than in the paper.

from attentionmask.

Any recommendations on what to reduce the number to?

from attentionmask.

I lowered both to 700, and got the following error even after 100% completetion

Progress: |██████████████████████████████████████████████████| 100.0% CompleteF0227 14:57:04.590605 8270 syncedmem.hpp:31] Check failed: error == cudaSuccess (29 vs. 0) driver shutting down
*** Check failure stack trace: ***
Aborted (core dumped)

from attentionmask.

This error is normal, it is a Caffe problem and happens during clean up. Your results are correct

from attentionmask.

Right okay. I ran the evaluation command several times, and each time it seems to crash my computer (i.e. it freezes)? Any idea why? Or how I may debug this?

from attentionmask.

python evalCOCO.py attentionMask-8-128 --dataset val2014 --useSegm True --end 5000
loading annotations into memory...
Done (t=4.03s)
creating index...
index created!
Loading and preparing results...
Killed

currently getting this error when it doesn't freeze

from attentionmask.

I am currently under the assumption that when running the evalCOCO.py, it causes immense resource starvation leading to the process either getting "Killed" or the computer freezing. After using the command I analyzed memory usage via the "System Monitor" and noticed that the computer freezes as the memory usage for python hits around 14gb. I am currently running the following specs (added below), and I am assuming that I may be running out of memory. I am currently thinking of assigning more swap memory, but currently unsure what to do. Any input would be appreciated and I was also wondering what hardware + software specs you used when running for the results in the paper. Thanks in advance.

from attentionmask.

Your are right, it's a problem of the CPU memory. It takes a lot of memory to load a json file with 1000 proposals for 5000 images. Therefore, you either have to cut down on the images by using --end in testing and evaluation (e.g. --end 1000 -> testing only the first 1000 images of the dataset) or cut down on the proposals by cutting down the number to 500 or 100. The later can be done by changing the dynamicK variable in utils.py line 90 to 500 or 100. Then you have to rerun testing and evaluation.

from attentionmask.

Right okay, thanks. I am currently trying to run the train.sh script with the default GitHub cloned repository. I am currently encountering the following runtime error:

*** Check failure stack trace: ***
./train.sh: line 26: 15023 Aborted (core dumped) python trainAttentionMask.py 0 attentionMask-8-128 --restore attentionMask-8-128_iter_$STEP_OLD.solverstate --step $SIZE_EPOCH
training done
start validation
WARNING: Logging before InitGoogleLogging() is written to STDERR
W0228 17:59:53.975257 15040 _caffe.cpp:122] DEPRECATION WARNING - deprecated use of Python interface
W0228 17:59:53.975277 15040 _caffe.cpp:123] Use this instead (with the named "weights" parameter):
W0228 17:59:53.975280 15040 _caffe.cpp:125] Net('models/attentionMask-8-128.test.prototxt', 1, weights='params/attentionMask-8-128_iter_880000.caffemodel')
Traceback (most recent call last):
File "testAttentionMask.py", line 58, in
caffe.TEST)
RuntimeError: Could not open file params/attentionMask-8-128_iter_880000.caffemodel
validation done
start evaluation
Traceback (most recent call last):
File "evalCOCO.py", line 42, in
cocoDt = cocoGt.loadRes("results/%s.json" % args.model)
File "build/bdist.linux-x86_64/egg/pycocotools/coco.py", line 317, in loadRes
AssertionError: Results do not correspond to current coco set
evaluation done

from attentionmask.