Choosing the seed for generated points in the hash space about rabbitmq-consistent-hash-exchange HOT 6 CLOSED

Can you please explain why you find this necessary?

On 23 mar 2016, at 15:03, Bahul Neel Upadhyaya [email protected] wrote:

At present, the points occupied by a consumer in the hash space are generated randomly. I have a use case where I would like to have those to produced consistently from a given seed value; this is to allow me to bind to more than one queue and have messages (with the same routing key) on those queues routed consistently to the same consumer.

One common way of doing this is to provide a seed value (often a string) and then repeatedly hashing to produce a sequence of hashes:

h1 = hash(seed)
h2 = hash(h1)
h3 = hash(h2)
...
hN = hash(hN-1)
Would this be possible maybe my appending an optional suffix to the binding key e.g. routing_key = <<"20,seedValue">>?

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub

from rabbitmq-consistent-hash-exchange.

@michaelklishin Not my bug but I can add our use case for this feature if it helps.

We use the consistent hash exchange with the primary key of a row in a database table as our routing key. We have 2 queues bound and 2 process consuming. This way each process will never clash when trying to update the same row. This allows us to cache the row in memory.

If we want to scale up the number of processes consuming (which we do at busy periods for our data processing) we have to do the following:

1. Pause publishing to the hash exchange
2. Wait for queues to reach 0 messages
3. Bind a new queue
4. Start a new process
5. Unpause publishing to the hash exchange

If the routing key would never change then we would simply have to:

1. Bind a new queue
2. Start a new process

(I must admit I don't understand the underlying implementation here and I'm not clear if @bahulneel's proposed change would allow us to achieve this. Just contributing a use case in the hope it makes things more clear).

from rabbitmq-consistent-hash-exchange.

@michaelklishin after a little thought, I can see how I can achieve the same behaviour I originally required a different way (mostly due to a better understanding of exchanges and queues).

However, I can think of another use-case for which this would be desirable. In some situations, there is some coordination or migration work that needs to occur when moving messages for one session (for example) from one node to another (persistent state, working dataset, etc.). In this case, I would like to bounce a consumer (potentially for a config change) and be sure that it occupies the same region of the hash space as it did before and reduce the cost of rebalancing the message routeing.

from rabbitmq-consistent-hash-exchange.

OK, we will leave this open for consideration. Thank you, @bahulneel and @andrewmichaelsmith!

from rabbitmq-consistent-hash-exchange.

This didn't make it into a pretty significant revamp of this plugin (#38). We might still consider it.

from rabbitmq-consistent-hash-exchange.

This was discussed in #32 and to some extent improved in #45. As of 3.8.5, this plugin will recover the ring consistently (the same ring partition must correspond to the same queue) due to ordered recovery but initial ring formation is still performed sequentially on a "first bind, first served" basis. That is, the ring partitions are assigned in the order of binding and the plugin obviously does not control those.

I think we will settle for this design for some time. Revisiting ring population would be a pretty significant change, and Jump consistent hashing does not really optimize for this case.

from rabbitmq-consistent-hash-exchange.