Tags performance about graphite-clickhouse HOT 4 CLOSED

In investigating a performance bottleneck, we have run into this same issue, except without using tags - this is not specific to tags. We have a lot (20 000) different paths , doing
SELECT ... FROM graphite ... WHERE Path IN (SELECT Path FROM graphite_tree WHERE Path LIKE '...')
can be 5 times as fast as doing:
SELECT Path FROM graphite_tree WHERE Path LIKE '...'
and using the results to construct a huge static Path IN query:
SELECT ... FROM graphite ... WHERE Path IN (SELECT Path FROM graphite_tree WHERE Path IN ('abc', 'def', ...)
Apparently the subquery form allows clickhouse to run additional optimizations.

from graphite-clickhouse.

Also came to this issue when doing a seriesByTag that results in a lot of paths. This time the ClickHouse server throws errors because the resulting query (Path IN ( ..) is too big.

from graphite-clickhouse.

Hello, colleagues.

Since the logic of a data fetching is slightly changed after ClickHouse internal aggregation was implemented, now it's necessary to first get the data locally, group metrics by the retention scheme, and only then send specific queries with proper step and aggregate function to the DB.

Although, #100 may address the issue mentioned here since benchmarks look promissing. It would be nice if you could test it and bring your feedback.

from graphite-clickhouse.

I assume this should be solved with #100, but feel free to open it in case of another opinion!

from graphite-clickhouse.