Dynomite crashes on restart about dynomite HOT 8 CLOSED

When you face these issues, there are a few factors that you should consider:

1- Add more nodes to balance out traffics so each node is not responsible for so much data and traffics
2- Change "hit max allowable limit" to a greater than 200000
3- Change to smaller timeout on peer responses so that the coordinator node can drop those messages.
Then, we can use the warm-up process on the restarted node. We are also working on the data repair
process to help out too (similar to Cassandra repair)

from dynomite.

Should alloc_msg_count ever be decremented? I don't see a decrement in the code so wouldn't this cause all client traffic to be ignored until the server is restarted?

Also, I'm not sure it's related, but I also see all nodes eventually returning bad values. Usually integer zeroes but sometimes string values for operations that should return ints. This only happens when the coordinator does not own the key.

from dynomite.

After considering it, the zeroes issue is most certainly Dynomite responding to requests out of order. We have a lot of keys that have zero values. However, I can reproduce the issue from a client that has not issued any previous operations. If I continually call "GET some_random_nonexistent_key" I essentially see a stream of responses to other operations rather than all nils.

from dynomite.

We already recognized this and will have a fix soon. Thanks.

from dynomite.

Btw, are you using Dyno (our java client)? Then use TokenAware and you won't run into this issue.

from dynomite.

Unfortunately, no. We have Perl and Go clients.

from dynomite.

It's great to hear that you're aware of the bug. Is there an open issue on GH or any more information you can provide? I've been unable to reproduce the problem in a development environment and would appreciate any further details you could provide.

Thanks.

from dynomite.

Hi @ralfonso Dynomite has moved a lot lately. There is a fix in place not just for the out of order thing but also the root cause which was causing the out of order (ce9b818)

If you have a private branch or a fork the above commit should solve your out of order issues and also improve consistency across nodes.

from dynomite.