High priority / backlog on metric flush about metrictank HOT 5 CLOSED

typically this happens when write queues are not sized correctly.
when the queue fills up it will block ingest. this is by design.
you can either make your queues bigger, or increase write concurrency to cassandra. (may or may not be very helpful, depends on how long it takes for the queue to fill up and drain)
you can look at the write queue panel on the dashboard to see this.

from metrictank.

The write queues are at 5.5 Million each and write concurrency is set to 24.
On metric flush the queues wont exceed 2 million. This might be a bit inaccurate because we only have a resolution of 30sec on metrictanks metrics.

I think there was some kind of formula to calculate the min queue size, but i have to search it in the docs. But i might be wrong with it.

Edit:
Found it:
Does unique series relate to active series idx or tank? https://github.com/grafana/metrictank/blob/v1.0.0/docs/cassandra.md#write-queues

For tank it would fit into the queue size but for idx it won't fit.

Queue Size: (245.500.000) = 132.000.000
Tank: (series * (34+1)) = 47.450.000
Idx: (series * (3*4+1)) = 181.090.000

from metrictank.

if not the queues, the other reason may be that your cpu's get maxed out when it starts sealing all chunks and adding them to the write queue

from metrictank.

It seems you're right, the primary which has to handle more metrics reaches 85% and builds up the backlog.
The second primary which has to process less metrics only has 50% cpu usage and no significant backlog.

Maybe we can reassign some partitions to distribute the metric load more evenly.
I will close this ticket after our measures were successful to confirm that it was the cpu usage.

Thanks for your help

from metrictank.

I could confirm that it was in fact a cpu bottleneck, we've rebalanced the metrics more evenly and even a big flush does not cause backlog anymore.

Thanks for your help

from metrictank.