Investigate unexpected resource disappearance about kine HOT 7 CLOSED

Interesting to see this popping up here as I'm in the process of trying to find the cause for our resource disappearance.
We however run Kubernetes 1.18 against a normal Etcd cluster and observed secrets, deployments and statefulsets getting terminated in the events stream during higher load on the API server and Etcd. (we replaced worker nodes and the scheduler started reassigning workloads)

That leaves me slightly worried now as that probably rules out our slightly older Etcd version and it might be in the API server 😞

And while I'm here - thx for etcd-io/etcd#11776 as that is currently blocking my test environment 😉

from kine.

@MerlinDMC if you've seen something similar on clusters running stock etcd that makes me feel slightly more optimistic that there isn't some weird issue with kine that's disappearing our resources out from underneath us.

from kine.

yes - but I was just about to try kine as a replacement to get rid of the problem 🤣

from kine.

Can you share any more details on what you were seeing? Was the apiserver actually deleting things from etcd? How were you monitoring the issue?

from kine.

I was monitoring status of the deployments via kubectl get --all-namespaces events -w. At some point during the rollout of our services to new nodes (roughly 1000 pods) I did see terminations for a lot of deployments in the event stream.

I'm not sure yet if the apiserver did issue those deletes or if it did read keys from Etcd and didn't get all the data back. No person however did issue deletes.

I'm trying to get a test scenario up that has the same application load so we can try to forcefully get into the same issue again and collect more data.

from kine.

I have identified an issue that could, under certain circumstances, cause data loss within the kine back-end database, including removal of random resources from the Kubernetes datastore.

Like etcd, kine stores all changes to the MVCC store as rows in a database. We use SQL instead of BoltDB like etc, but both need to be compacted periodically to remove old revisions from the history lest the store grow out of control. Kine does this internally by tracking relationships between resource versions in the prevous_rev column of the database - each version of a resource points to the previous version via this column. There is also a special key with the name compact_rev_key that stores the revision that we have compacted up to, so that we can authoritatively respond with the appropriate error when a client asks for an old version of a resource, and that resource is not found.

In an unfortunate early design choice, kine stores the compact revision in the prev_revision column of the compact_rev_key entry, rather than the value. This means that it needs to be excluded when selecting old resource versions to delete.

Recent changes to improve compaction efficiency by moving the compact logic from go code to pure SQL incorrectly handled the compact_rev_key entry - I excluded it from deletion, instead of excluding the row it refers to, when building the list of revisions that are safe to delete.

If when compaction is triggered the compact_rev_key happens to point at the most recent revision of a resource, that object will be deleted, resulting in its disappearance from the etcd API. Due to the way compaction moves forward through the version history, it is most likely that it will point at a deleted or superseded version and the resource will remain unaffected, but occasionally it will point at a current revision, in which case all revisions of the object will be removed.

from kine.

Validated on all three releases: v1.17.14-rc1+k3s2, v1.18.12-rc2+k3s1, and v1.19.3-rc1+k3s3

• Performed the test steps listed in #62
• Validated standard regression checks on the cluster are passing (create, scale, delete, modify workloads, secrets, etc.)
• ensured compaction is still working proeperly

from kine.