"Too many simultaneous queries" error after upgrading to 0.12 about graphite-clickhouse HOT 6 CLOSED

Hey hey.

To me it looks a little bit strange. Can you try the master to confirm if the issue is still there?

In any case, the easiest solution would be to increase clickhouse setting max_concurrent_queries, just try 150.

from graphite-clickhouse.

Hey Robert @spoofedpacket, are there any news regarding the issue?

from graphite-clickhouse.

@Felixoid Thanks for the response. I noticed from the logs that the majority of failed queries related to the graphite_index table so I decided to try something:

• The graphite_index table has grown quite large over time. Metrics paths that have stopped updating werent being deleted from the table, so there was about a year's worth of stale data in there.
• It was also partitioned with a YYYYMMDD partition key, which lead to it having over 400 partitions - I'm guessing that would have grown forever if left alone.
• I dropped and recreated the table with a monthly partition key this time and let metrics feed in for a few days
• There's also a cron in place now to purge metrics paths that haven't been updated in 2 weeks, based on this discussion: #83 (comment)
• After this, I started sending queries again for several hours and the error did not reappear

I'm thinking this is something peculiar to my setup, due to the very large index table at the time. Happy to close the issue for now, if it reoccurs I'll try bumping the max_concurrent_queries to 150.

Thanks again!

from graphite-clickhouse.

There's actually another way to solve the issue. You can try to use the index-use-daily = false parameter, it can be more efficient for your setup.

To prevent the writing of daily data, there's a setting in carbon-clickhouse, but I'm unhappy with its name disable-daily-index = true, so it will be changed in the next config refactoring in a couple of weeks

from graphite-clickhouse.

Oh, I didn't know about that option in carbon-clickhouse. I'll give both a try, thanks for the tip!

from graphite-clickhouse.

I'm closing it, feel free to reopen it on new questions

from graphite-clickhouse.