disaster recovery of etcd cluster about kstone HOT 6 OPEN

We didn't use etcd-operator, we used the built-in kstone-etcd-operator.It has more complete support for persistent storage and better disaster tolerance. In the next version, we will support rebuilding lost-quorum clusters using snapshots, and kstone-dashboard will also support visualization operations.

from kstone.

Hi tangcong, thanks for your reply. 

Based on the official wiki:

https://etcd.io/docs/v3.5/op-guide/runtime-configuration/# restart-cluster-from-majority-failure

When we use a dynamic way to deploy Etcd clusters(That's how it works in K8s), it appears that the only way to rebuild a lost-quorum cluster is to take a snapshot. The problem of snapshots is that if we use a snapshot to recover, we'll definitely lose some data(after the snapshot is taken, some new update are write to etcd, then k8s cluster is down). 
If use the static way to build a etcd cluster, I can simply restart all nodes to rebuild the etcd cluster without using snapshot, do you have some ideas about how to achive it if we use dynamic way?

from kstone.

We support storing data on persistent data disks. If it is not persistent, we can also force a snapshot from the healthy node to rebuild the cluster, so that the probability of losing data is very small. @rayliu419

from kstone.

@tangcong ,

"We support storing data on persistent data disks. If it is not persistent"
I think most of k8s users will use PVC/PV as external storage, so that's not a problem. The problem is even if you have persistent data disks, you can't restore the cluster with these persistent data disks just as static way.

"we can also force a snapshot from the healthy node to rebuild the cluster, so that the probability of losing data is very small."
Yes, we can force the health node to dump snapshot, but here are two problems:

1. Since etcd cluster continue to accept write all the time, you can't snapshot all writes, right?
2. In some cases, such as k8s is down, you don't have a healthy node.

Actually, I don't know why etcd community doesn't have a solution to recover it without using snapshot(maybe they have, but I don't find it). In cloudnative enviorment, it's a critical case.

from kstone.

I think I find a way to recover the data without losing data. Etcd supports --force-new-cluster to reconfigure the cluster, we can find the latest node in the etcd cluster to restart the cluster. If more than half pvc survive, we can recover it in this way.

from kstone.

Yes, I submitted an issue #kstone-io/kstone-etcd-operator#2 at the time, and the plan was implemented in this way. thank you.

from kstone.