force_fg triggers too many retries exception which kills server about rootpainter3d HOT 4 CLOSED

The offending code is here:

The mechanism here is that the system will retry if it didn't find any foreground, if force_fg is also true, which happens with a certain probability early on in training to avoid sampling too many background only patches early in training.

The problem (I believe) that is causing the server to crash is that the max_retries were for some reason set to 2.

bedb9dc

I've put it back up to 200. I consider it unlikely that you will have 200 randomly sampled images without foreground. If so then I can come up with a different solution.

In general I don't think this retry solution is optimal, but hopefully this is a temporary fix.

Can you please let me know if it is working OK for you now.

from rootpainter3d.

It seems to be training now, although the server output is a somewhat cluttered with exceptions, which might confuse. However, it is okay for me :).

from rootpainter3d.

Actually I agree these error messages aren't great. I will change the code so it is not an exception when foreground is not found.

from rootpainter3d.

Fixed so it retries without exceptions/warnings. Still in another branch though. will close this issue when it is merged into master

dd8362c

from rootpainter3d.