[Improvement] Disallow sendShuffleData if requireBufferId expired about incubator-uniffle HOT 7 CLOSED

Do you have the solution of the problem?

from incubator-uniffle.

Yes, it is be testing in our production environment. I will watch it for a while. If it's OK, I will create a pr

from incubator-uniffle.

Could you share your solution? We can discuss first.

from incubator-uniffle.

Could you share your solution? We can discuss first.

1. In server side, if requireBufferId not found when send data, thrown an exception.
2. In client side, if fail to send data, require buffer again.

from incubator-uniffle.

cc @colinmjj . There seems not be cases in our production environment. But I think the analysis is correct. What do you think?

from incubator-uniffle.

I think @xianjingfeng is right, with current implementation, OOM will happen if requireBufferId was expired in Shuffle Server already, this maybe caused by GC, network problem, high workload in shuffle server etc.
It's better to have the limitation to accept the data with requireBufferId only to avoid such problem.

from incubator-uniffle.

closed by #157

from incubator-uniffle.