Comments (6)
Can you please explain why you find this necessary?
On 23 mar 2016, at 15:03, Bahul Neel Upadhyaya [email protected] wrote:
At present, the points occupied by a consumer in the hash space are generated randomly. I have a use case where I would like to have those to produced consistently from a given seed value; this is to allow me to bind to more than one queue and have messages (with the same routing key) on those queues routed consistently to the same consumer.
One common way of doing this is to provide a seed value (often a string) and then repeatedly hashing to produce a sequence of hashes:
h1 = hash(seed)
h2 = hash(h1)
h3 = hash(h2)
...
hN = hash(hN-1)
Would this be possible maybe my appending an optional suffix to the binding key e.g. routing_key = <<"20,seedValue">>?—
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub
from rabbitmq-consistent-hash-exchange.
@michaelklishin Not my bug but I can add our use case for this feature if it helps.
We use the consistent hash exchange with the primary key of a row in a database table as our routing key. We have 2 queues bound and 2 process consuming. This way each process will never clash when trying to update the same row. This allows us to cache the row in memory.
If we want to scale up the number of processes consuming (which we do at busy periods for our data processing) we have to do the following:
- Pause publishing to the hash exchange
- Wait for queues to reach 0 messages
- Bind a new queue
- Start a new process
- Unpause publishing to the hash exchange
If the routing key would never change then we would simply have to:
- Bind a new queue
- Start a new process
(I must admit I don't understand the underlying implementation here and I'm not clear if @bahulneel's proposed change would allow us to achieve this. Just contributing a use case in the hope it makes things more clear).
from rabbitmq-consistent-hash-exchange.
@michaelklishin after a little thought, I can see how I can achieve the same behaviour I originally required a different way (mostly due to a better understanding of exchanges and queues).
However, I can think of another use-case for which this would be desirable. In some situations, there is some coordination or migration work that needs to occur when moving messages for one session (for example) from one node to another (persistent state, working dataset, etc.). In this case, I would like to bounce a consumer (potentially for a config change) and be sure that it occupies the same region of the hash space as it did before and reduce the cost of rebalancing the message routeing.
from rabbitmq-consistent-hash-exchange.
OK, we will leave this open for consideration. Thank you, @bahulneel and @andrewmichaelsmith!
from rabbitmq-consistent-hash-exchange.
This didn't make it into a pretty significant revamp of this plugin (#38). We might still consider it.
from rabbitmq-consistent-hash-exchange.
This was discussed in #32 and to some extent improved in #45. As of 3.8.5
, this plugin will recover the ring consistently (the same ring partition must correspond to the same queue) due to ordered recovery but initial ring formation is still performed sequentially on a "first bind, first served" basis. That is, the ring partitions are assigned in the order of binding and the plugin obviously does not control those.
I think we will settle for this design for some time. Revisiting ring population would be a pretty significant change, and Jump consistent hashing does not really optimize for this case.
from rabbitmq-consistent-hash-exchange.
Related Issues (20)
- Documentation Lacking around Routing Key Implications HOT 4
- Binaries HOT 2
- Move test suite to Common Test
- Handling orphan queues HOT 2
- same message with same routing key goes to different consumer HOT 1
- Routing keys greater than 10 million when the system will lead to memory overflow HOT 3
- Use retries when waiting for Mnesia tables to load HOT 2
- Consistent hash exchange should keep same distribution after server restart HOT 15
- Exchange perfomance HOT 5
- routing to another exchanges HOT 1
- Message routing distribution among queues is not uniformly random HOT 23
- Is header hashing supported? HOT 3
- Queue deletion and binding removal can affect hash ring state consistency HOT 13
- Highly non-uniform distribution of messages over queue when using UUID as routing key HOT 6
- Message Distribution is not uniform (at all) with weights less than 4 HOT 1
- Durable consistent hashing exchange cannot recover their previous state after node restart (3.8.4-specific) HOT 1
- Provide examples in more programming languages HOT 2
- Negative routing key should result in a channel error HOT 3
- Hash on AMQP Message Properties HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rabbitmq-consistent-hash-exchange.