Occasional Horde Registry Lookup Misses about tictac HOT 3 CLOSED

Hi @Miserlou!

I'm glad the project has been helpful for you while learning Elixir!

I haven't been working on the project lately so I can't say I've seen the issue. Here's a few thoughts:

• I don't think the project was setup for Horde state-handoff. I looked into doing that. This means that when a node goes down (either loses it's cluster connection or a new deploy is rolling out), any existing game servers will be restarted in a new node, but they lose their state.
• There may have been other changes in the Horde project that I don't know about since I created this demo.
• I wonder if the newly started process is crashing.
• There may be a bug in your code between receiving the pid and inserting it into the registry?

I know how frustrating intermittent problems can be! My suggestion is to try and add more logging to try and expose what's going on.

from tictac.

Thanks for your reply, and for your very handy project/blog-post.

I was able to track the issue down. Horde is actually behaving as expected, the nature of the distribution creates a race condition where I could be trying too lookup a registered PID before it has had a chance to propagate to other nodes. I see two obvious solutions to this, one would be to try some kind of exponential backoff for retrying a failed lookup until it reaches a failure threshold, the other - which I'm using now - is to add delta_crdt_options: [sync_interval: 15] to the DistributedSupervior/Registry settings, which sends CRDT syncs ever 15ms rather than the default of 250ms, which obviously creates a lot more network traffic but lowers the length of the race window enough for acceptable performance in my application.

These are trade-offs that anybody implementing a Horde system will have to consider, but I think I'm satisfied with the situation for right now. This isn't an issue, so I'll close the ticket now and hope that anybody else who encounters this issue can stumble upon this. Cheers!

from tictac.

Thanks for sharing your discovery!

from tictac.