Handling Soft TKOs about mcrouter HOT 3 CLOSED

@jamescarr there are couple of options you can tune to have desirable behavior of handling Soft TKOs.
Let's see what happens with TKO logic step-by-step.

First of all, when is a box marked as "Soft TKO"?
After mcrouter has received >= --timeouts-until-tko timeouts in a row from this box. But only if total number of boxes marked as Soft TKO is less than --maximum-soft-tkos.

What about Hard TKO?
A box is marked as hard TKO after mcrouter has encountered connect error or connect timeout. Even one connect error/timeout is enough, mcrouter doesn't wait for --timeouts-until-tko to mark box as Hard TKO.

What happens after a box is marked as TKO (be it hard or soft)?
Mcrouter starts sending version requests (aka probes) to this box. Every other request sent to this box is immediately discarded.
First probe is sent after --probe-timeout-initial milliseconds since box was marked as TKO.
Interval between consecutive probes exponentially grows until it reaches --probe-timeout-max.

When do we unmark the box marked as TKO?
Once mcrouter gets a successful reply (not error or timeout) on probe request.

How do I check which servers are marked as TKO?
Try stats servers and stats suspect_serversstats commands

Why do we need TKO logic at all?
Usually when some memcached server is slow/down/overloaded, we want to simply send request to a failover destination. Once memcached server is marked TKO, failover becomes is instant, we don't need to do 2 round trips (one to known slow/broken box, one to failover destination) for a request.

I don't want any TKO tracking!
One can disable it with --disable-tko-tracking.

That said, try to tune TKO behavior using options described above. Increasing server timeout/limiting number of soft TKOs/decreasing first probe timeout may help if you see lots of errors caused by TKO behavior.

If you see any weird behavior (e.g. mcrouter incorrectly marks/unmarks box as TKO), please provide some details so we can find the root cause.

from mcrouter.

Hi @alikhtarov, to increase server timeout which options should I use? Because I see that --server-timeout is deprecated:

-t, --server-timeout     server timeout in ms (DEPRECATED try to use cluster-server-timeout and regional-server-timeout) [default: 1000]

But I don't see cluster-server-timeout and regional-server-timeout options. Does this actually mean --cluster-pools-timeout and --regional-pools-timeout instead?

from mcrouter.

Leaving these links just in case someone else also comes over this.

Timeout settings defined in header file ("server-timeout" is followed by more specific timeouts, which, if specified, will override it):

Timeout docs: https://github.com/facebook/mcrouter/wiki/Command-line-options#timeouts

from mcrouter.