Sometimes the number of zk-nodes suddenly increases significantly even when the bulk write QPS remains unchanged about clickhouse HOT 4 CLOSED

Replication issues/mutations issues/outages of nodes. Monitor system.replication_queue at least count(), replication lag (max_absolute_replica_delay). I saw a spike from 50 thousands znodes to 5 million because of adding a new node to the cluster.

from clickhouse.

Den's explanations are all plausible. I've also seen spikes due to adding a new replica: as it's replicating old data, new (small) parts being ingested in the meantime(.

Check replication status in all replicas and check what nodes are being created / maintained to investigate your specific circumstances.

from clickhouse.

Den's explanations are all plausible. I've also seen spikes due to adding a new replica: as it's replicating old data, new (small) parts being ingested in the meantime(.

Check replication status in all replicas and check what nodes are being created / maintained to investigate your specific circumstances.

@den-crane @Algunenano But I did not add new nodes to the new cluster. How can this be explained?

from clickhouse.

It's just a guess. Since you are not providing any data about the kind of nodes that are being created and not deleted, there is no way to provide you with anything but guesses.

from clickhouse.