unexpected backoff durations when using built-in mqtt bridge when network is disconnected about thin-edge.io HOT 7 CLOSED

Another unexpected behaviour is that the mqtt.bridge.reconnect_policy.initial_interval setting is not treated as a hard minimum value, but it is the value before the jitter is added (see the log entry below which shows a possible value of 17s, which is less than the "initial_interval" of 30s)

Jun 11 13:01:25 96136f6a6861 tedge-mapper[664]: 2024-06-11T13:01:25.329372422Z  INFO tedge_mqtt_bridge: Waiting 17.568652105s until attempting reconnection to cloud broker

from thin-edge.io.

Though I did finally look up the backoff crate's documentation which explains most of the behaviour, however I would question the approach used by the crate.

from thin-edge.io.

The logic for the exponential backoff is straight from the backoff crate. I believe the algorithm used is to keep track of the current interval (initially 30s), and on each iteration, generate a random value within a range of the current interval ± half its value.

According to this, all the values are behaving as expected.

1. 30s -> [15s, 45s] -> 42s
2. 60s -> [30s, 90s] -> 31s
3. 120s -> [60s, 180s] -> 166s
4. 240s -> [120s, 360s] -> 307s
5. 480s -> [240s, 720s] -> 591s
6. 600s (maximum interval) -> [300s, 900s] -> 371s

The range doesn't change after that, and all the values are within the range [300s, 900s]
672s
325s
757s
823s
640s
806s

We can change the randomization factor (currently set to 0.5, hence the ± half the value), and we can also hard bound the minimum value if we want to (although if we're cautious enough to set it to 30s, I'd say we should be cautious enough to add jitter (although we could set the initial interval of the underlying algorithm to be configured initial interval / randomization factor, so the initial_interval setting is actually a hard lower bound on the value).

We could also expose more of the parameters as configuration, but I figured that would likely not prove useful, and would cause unnecessary confusion.

from thin-edge.io.

Yeah the crate's documentation was indeed helpful, so maybe it is more a documentation problem.

Though the backoff isn't exactly exponential for the first few values (not sure if this will be problematic or not)

from thin-edge.io.

we can also hard bound the minimum value if we want to (although if we're cautious enough to set it to 30s, I'd say we should be cautious enough to add jitter

Though there is an alternative approach to calculating the backoff factor, and it is to take a value between [value*2.0, value*2.5]...that way you can enforce such things as minimum values (but with jitter).

from thin-edge.io.

I'll try to get some numbers so that we can compare the algorithms as I found some good materials that should be fairly easy compare:

Based on this write-up, https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/, using the aws-arch-backoff-simulator repository, the current backoff algorithm will be compared with the other algorithms listed in the write-up.

from thin-edge.io.

I'm closing this as I've compared the selected backoff algorithm (including jitter), and it seems to be on-par with the other methods (according to the others mentioned in the above aws blog post). The selected algorith takes slightly longer "to complete", however this might be a good thing when the server is in "overload" and gives a bit more time for the server function to be restored.

from thin-edge.io.