"Session moved to another server" error since 24.2 about clickhouse HOT 4 CLOSED

Some of our new discoveries:

• The first DDL after the instance is started will always fail

I suspect it's because of the order in which the instances are started.
In the patch of check_session_valid patch, introduced a session node in zookeeper. It is designed to mange the version of keeper connections. The path of the session node is taken from the UUID of the server.

https://github.com/ClickHouse/ClickHouse/blob/24.3/programs/server/Server.cpp#L896

When startup, create zk connection, including creating a zookeeper object and executing initSession. initSession is used to create a session node or modify the version of the session node.
However, uuid has not been loaded at this time.

Consequences of the problem:

1. The session node is not created or does not have an updated version, which causes the DDL to fail after restarting.

from clickhouse.

This error is expected and is thrown as extra protection to prevent stale sessions from applying changes.
The exception should be self-explanatory and it works like following:

• on some of multi requests, internally append the check request on a znode containing the session id and the current latest (known) version of that znode
• on response
  • if the session that made the multi request is still most up to date session, check will pass with ZOK
  • if the session is not the most up to date session, check will return ZBADVERSION
    • as the ZBADVERSION for a check request on an internal znode does not provide any useful information to the user, we return a more descriptive information with "Session moved to another server"
  • remove the check response from other responses

When you see such error it means the extra protection triggered and as a safety mechanism it decided to ignore the request.
Protection is against rare but scary problems related to correctness (e.g. a data loss could occur during inserts).

from clickhouse.

This could explain the issue
I'll verify and create a fix for it
thanks for catching it!

from clickhouse.

Fix: #64986

from clickhouse.