[NEW] Compacting the output of topology commands for for fragmented clusters about valkey HOT 7 OPEN

Implement CLUSTER SLOTS DENSE client capability.

CLUSTER SLOTS is one of the mostly widely used command by clients for topology discovery. This would be a good feature addition and solve the large output response on fragmented slots. I think we can introduce the CLIENT CAPA <feature> <yes|no> to enable/disable the feature rather than introducing another level of sub command.

from valkey.

I definitely feel that CLUSTER SLOTS had it almost right, and that CLUSTER SHARDS adds functionality that most clients don't need. Since most clients already use CLUSTER SLOTS, adopting this option should be relatively easy. However, note that we still need to sort the replicas in the current CLUSTER SLOTS implementation, not just compact the slots data.

from valkey.

... and if we do want to support DENSE, I think the slot ranges should be represented as a flat multi-range list [start1, end1, start2, end2, ...] rather than one list of starts and another list of ends. It's more intuitive (IMAO). For a single-range shard, it's identical to a non-DENSE reply.

The only reason I suggested the other approach is that it keeps the total number of arguments the same in all cases :) I'm not very convinced it's useful for them to be the same in specific edge cases.

Improve the docs. Let's show an example of CLUSTER SHARDS in RESP3, so you actually see the reply as a map, which is was designed for.

I honestly am not really convinced the map response is ideal for this case anymore. We are basically spending a bunch of bits in the network response to send information the client already knows. It's only useful if a human is reading it ad hoc.

Give it more time. CLUSTER SHARDS is still new. Clients want to support nodes running all still supported Redis OSS versions. The motivation for moving to CLUSTER SHARDS hasn't been strong enough, given they'll need to have a fallback anyway, but once all versions that don't support SHARDS are EoL, the situation will be different.

I don't agree because client developers (see Bar) aren't happy with it. I think we should be opinionated about the API. I think our original approach of moving to a new new command didn't really work as well as we thought it would.

from valkey.

I think we should just do better marketing for CLUSTER SHARDS. It's very little extra information that the clients can simply ignore, so it's no real waste of bandwidth. It's also pretty trivial to parse, given the client can parse RESP, so that can hardly be a real problem.

Of the ideas suggested, I only support implementing defragmentation logic in valkey-cli, or at least making sure it's smart when rebalancing so that in minimizes creating fragmentation.

IMO we should instead focus on this:

• Improve the docs. Let's show an example of CLUSTER SHARDS in RESP3, so you actually see the reply as a map, which is was designed for.
• Explain the benefits, both in the docs of CLUSTER SHARDS and CLUSTER SLOTS.
• Give it more time. CLUSTER SHARDS is still new. Clients want to support nodes running all still supported Redis OSS versions. The motivation for moving to CLUSTER SHARDS hasn't been strong enough, given they'll need to have a fallback anyway, but once all versions that don't support SHARDS are EoL, the situation will be different.
• It may not be a frequent problem. To get very scattered slot distributions, you'd need to scale up multiple times (e.g. add one shard at a time) without paying attention to the slots. It's possible for cluster operators (not only valkey-cli) to be smarter and avoid creating fragmented slot ownershits. Always doubling or halving the number of shards is one way to completely avoid fragmentation, if done properly.

from valkey.

... and if we do want to support DENSE, I think the slot ranges should be represented as a flat multi-range list [start1, end1, start2, end2, ...] rather than one list of starts and another list of ends. It's more intuitive (IMAO). For a single-range shard, it's identical to a non-DENSE reply.

from valkey.

OK, I'm convinced. 👍

from valkey.

I like CLUSTER SLOTS DENSE (or COMPACT) but I do think the slots should be a list of starts and end indices. That's the slot format used in CLUSTER SHARDS and CLUSTER ADDSLOTRANGE.

The idea about two lists that need to be iterated in parallel isn't great.

Another option we could add to CLUSTER SLOTS is NO-REPLICAS, that clients can use if they only care about primaries.

We should introduce these changes togerher with #298 and a possible format for push notifications id to use the same as one line of CLUSTER SLOTS DENSE. Clients can then implement both features at the same time.

from valkey.