Coordinator HA problem about firestorm HOT 6 CLOSED

And how to upgrade coordinator partially to ensure gracefully recommission/decommission coordinator? Do we have some best practice?

from firestorm.

It looks like all coordinators will serve for all spark jobs instead of the only one serve for online(like master/slave). And the spark job will pick one coordinator to get assignment according to quorm sequence.

And the coordinator is only responsible for partition assignment.

If we have two coordinator serve for online, it means when one coordinator dead, the RSS still works. And when all coordinators dead, the existing shuffleIDs still work, and the new-coming shuffle request will fail.

If i am wrong ,feel free to let me know. Anyone could help provide more detailed architecture design info?

Your understanding is right.

from firestorm.

And how to upgrade coordinator partially to ensure gracefully recommission/decommission coordinator? Do we have some best practice?

We can upgrade coordinator one by one. Only one is active, the cluster can provide normal service. And our internal team are working on a k8s operator. The work will be submitted to this repo soon. We're testing the operator.

from firestorm.

If using local disk of shuffle servers to store shuffle data, shuffle server can be deployed to K8s? Or just to deploy coordinators on K8s using operator?

And do we have any plan to control specified spark jobs to communicate with specified coordinator(the one have been upgraded) instead of according to quorm sequence defined in config?

from firestorm.

1.We will deploy shuffle servers which use local disks on K8S.
2.No such plan.

from firestorm.

Close it. Thanks for your explanation. @jerqi

from firestorm.