Distributing kestrel about kestrel HOT 8 CLOSED

As far as I understand reading None is not an issue at twitter. AFAIK: they have some (3, I guess) kestrel servers and a lot more workers. Each worker does the following: choose random server, read message, process message, repeat (may be chosen server is used for several iterations since operation like /close/open exists).

Anyway workers outnumber servers and choose server at random, so, eventually, each server would be read without any complicated logic.

from kestrel.

I resolved it to the following:

• Connect to random queue
• Read item
• Continue to read items until queue hits 0, if it hits 0 connect to a NEW queue via round robin.

When all queues are zero, it will simply just bounce between them (which is kind of shitty) so I set a higher /t=/open

from kestrel.

both of those sound fine. we pick servers at random, and if the queue is empty, sleep for a while (exponential backoff) and then try another.

jkalucki has been experimenting with a much more aggressive client in the "grabby-hands" project.

from kestrel.

Would it make sense to select a kestrel server based on the hash of a queue name? It seems to me that would be most efficient, though you may end up with small queues on some kestrel servers and large queues on others.

from kestrel.

having a queue exist on only one (or even a subset) of the kestrels in a cluster would defeat the purpose of having the cluster, though.

from kestrel.

It's been a year since this thread was active. Any news on ways to distribute kestrel? Preferred or best client policies?

from kestrel.

we still generally have clients pick a server at random in ruby, using kestrel-client (https://github.com/twitter/kestrel-client). in java/scala, we have most clients connect to every server at once and launch a timeout-get on each one to minimize latency. those use grabby-hands (https://github.com/jkalucki/grabby-hands).

is that what you meant?

from kestrel.

Yeah, that's exactly what I was wondering. I will look into those clients.

from kestrel.