Unable to get working with GPU: OOM and/or hangs about air HOT 2 CLOSED

Glad to hear you've found my work useful.

This is a new error for me as well. I'm guessing it is caused by that 8GB (100000, 100000) float64 numpy array in compute_overlap.pyx code, which likely somehow gets stored in the GPU memory that has only 6144MB capacity. To make this array smaller, try setting --max_detections 1000 and see if the error disappears. Although with MOB postprocessing, you'd want to use a quite large --max_detections value so all candidate boxes get merged together properly (but the effect might be negligible between 10k and 100k anyhow).

Playing with the other memory options might help too, as you've already tried. Although the detection performance likely degrades a lot with those tiny resized images (150x100), I'd rather touch the --image_tiling_dim option first.

from air.

Another possible solution could be to increase your Docker container memory limit to something like 16 GB. I think that compute_overlap computation should happen in CPU and main memory regardless if GPU is used or not, so it's weird that the GPU memory would be a limiting factor here.

Nevertheless, decreasing that --max_detections parameter likely brings your memory consumption to an acceptable level regardless where the bottleneck is.

from air.