Comments (7)
den-crane
commented on July 16, 2024
????
I don't believe it.
export QUERY="SELECT number from numbers_mt(1000000000) FORMAT RowBinary;"
time clickhouse-client -h 127.0.0.1 --port 9000 <<< $QUERY | wc -c
8000000000
real 0m16.837s
user 0m14.412s
sys 0m3.881s
time curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< $QUERY | wc -c
8000000000
real 0m13.813s
user 0m1.065s
sys 0m6.347s
export QUERY="SELECT 'xxssssssssssssssx', intDiv(number,10000), number%103, number from numbers_mt(100000000) FORMAT RowBinary;"
time clickhouse-client -h 127.0.0.1 --port 9000 <<< $QUERY | wc -c
3500000000
real 0m4.976s
user 0m4.297s
sys 0m1.251s
time curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< $QUERY | wc -c
3500000000
real 0m4.204s
user 0m0.527s
sys 0m2.271s
20.10.1.4751
from graphite-clickhouse.
Felixoid
commented on July 16, 2024
Hey @lexx-bright,
TBH, the outcome of the native protocol doesn't look attractive to me since the #88 is merged. The query you've used in the test is heavy outdated. Here's a comparison of data passed by using row/point, aggregation by metric name with groupArray, and the query implemented in the PR:
from=-7d
maxDataPoints=100
| method | rows | points | data (binary) | time (s) |
|---|---|---|---|---|
| row/point | 9887027 | 9887027 | 556378258 (530M) | 16.486 |
| groupArray | 986 | 9887027 | 158180388 (150M) | 35.498 |
| -Resample | 986 | 98553 | 1421418 (1M) | 13.181 |
note: it's localhost, so no network data transfer overhead
As you see, the amount of data for this particular request was decreased 500 times.
from graphite-clickhouse.
lexx-bright
commented on July 16, 2024
@Felixoid thanks for update, I'll check with new query.
Anyway, the fact that clickhouse performs so differently with TCP and HTTP looks odd.
@den-crane here's more info. Reproduced on my testing environment with 20.10.1.4788 installed.
SHOW CREATE TABLE db003_test.graphite_tagged
┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE db003_test.graphite_tagged (`Date` Date CODEC(DoubleDelta, LZ4), `Tag1` String CODEC(ZSTD(9)), `Path` String CODEC(ZSTD(9)), `Tags` Array(String) CODEC(ZSTD(9)), `Deleted` UInt8 CODEC(DoubleDelta, LZ4), `Version` UInt32 CODEC(DoubleDelta, LZ4)) ENGINE = ReplacingMergeTree(Version) PARTITION BY toYYYYMM(Date) ORDER BY (Tag1, Path, Date) SETTINGS index_granularity = 8192 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
SHOW CREATE TABLE db003_test.points_r1m_avg
┌─statement──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ CREATE TABLE db003_test.points_r1m_avg (`Path` LowCardinality(String), `Value` Float64 CODEC(ZSTD(1)), `Time` UInt32 CODEC(DoubleDelta, LZ4), `Date` Date CODEC(DoubleDelta, LZ4), `Timestamp` UInt32 CODEC(DoubleDelta, LZ4)) ENGINE = GraphiteMergeTree('graphite_rollup_r1m_avg') PARTITION BY toYYYYMM(Date) ORDER BY (Path, Time) SETTINGS index_granularity = 8192 │
└────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
[root@masternode ~]# export QUERY="SELECT Path, Time, Value, Timestamp
> FROM db003_test.points_r1m_avg
> WHERE
> Time >= 1600921652
> AND Time <= 1601526452
> AND Path IN (SELECT Path FROM db003_test.graphite_tagged GROUP BY Path)
> FORMAT RowBinary;"
[root@masternode clickhouse-server]# time curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< $QUERY | wc -c
22340209125
real 0m33.834s
user 0m5.918s
sys 0m47.098s
2020.10.01 02:36:48.542759 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> executeQuery: (from 127.0.0.1:55446) SELECT Path, Time, Value, Timestamp FROM db003_test.points_r1m_avg WHERE Time >= 1600921652 AND Time <= 1601526452 AND Path IN (SELECT Path FROM db003_test.graphite_tagged GROUP BY Path) FORMAT RowBinary;
2020.10.01 02:36:48.543064 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "(Time >= 1600921652) AND (Time <= 1601526452)" moved to PREWHERE
2020.10.01 02:36:48.543271 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.graphite_tagged (SelectExecutor): Key condition: unknown
2020.10.01 02:36:48.543297 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.graphite_tagged (SelectExecutor): MinMax index condition: unknown
2020.10.01 02:36:48.543324 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.graphite_tagged (SelectExecutor): Selected 1 parts by partition key, 1 parts by primary key, 18 marks by primary key, 18 marks to read from 1 ranges
2020.10.01 02:36:48.577047 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.graphite_tagged (SelectExecutor): Key condition: unknown
2020.10.01 02:36:48.577101 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.graphite_tagged (SelectExecutor): MinMax index condition: unknown
2020.10.01 02:36:48.577114 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.graphite_tagged (SelectExecutor): Selected 1 parts by partition key, 1 parts by primary key, 18 marks by primary key, 18 marks to read from 1 ranges
2020.10.01 02:36:48.616358 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.points_r1m_avg (SelectExecutor): Key condition: (column 1 in [1600921652, +inf)), (column 1 in (-inf, 1601526452]), and, (column 0 in 8208-element set), and, (column 1 in [1600921652, +inf)), (column 1 in (-inf, 1601526452]), and, and
2020.10.01 02:36:48.616417 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.points_r1m_avg (SelectExecutor): MinMax index condition: unknown, unknown, and, unknown, and, unknown, unknown, and, and
2020.10.01 02:36:48.637321 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Debug> db003_test.points_r1m_avg (SelectExecutor): Selected 7 parts by partition key, 7 parts by primary key, 6904 marks by primary key, 6904 marks to read from 7 ranges
2020.10.01 02:36:49.861450 [ 8735 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:36:49.861574 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.86 GiB.
2020.10.01 02:36:49.861651 [ 8736 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.86 GiB.
2020.10.01 02:36:49.950584 [ 8716 ] {} <Debug> DNSResolver: Updating DNS cache
2020.10.01 02:36:49.951027 [ 8716 ] {} <Debug> DNSResolver: Updated DNS cache
2020.10.01 02:36:50.152509 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:36:57.363134 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:36:57.364270 [ 8723 ] {} <Debug> system.trace_log (MergerMutator): Selected 6 parts from 202010_1_11_2 to 202010_16_16_0
2020.10.01 02:36:57.364327 [ 8723 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:36:57.364348 [ 8723 ] {} <Debug> system.trace_log (MergerMutator): Merging 6 parts: from 202010_1_11_2 to 202010_16_16_0 into Compact
2020.10.01 02:36:57.364380 [ 8723 ] {} <Debug> system.trace_log (MergerMutator): Selected MergeAlgorithm: Horizontal
2020.10.01 02:36:57.365636 [ 8723 ] {} <Debug> system.trace_log (MergerMutator): Merge sorted 871 rows, containing 10 columns (10 merged, 0 gathered) in 0.001286413 sec., 677076.4909869537 rows/sec., 115.91 MiB/sec.
2020.10.01 02:36:57.660611 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:37:04.864414 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:37:04.951168 [ 8743 ] {} <Debug> DNSResolver: Updating DNS cache
2020.10.01 02:37:04.951438 [ 8743 ] {} <Debug> DNSResolver: Updated DNS cache
2020.10.01 02:37:05.165873 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:37:05.167880 [ 8717 ] {} <Debug> system.metric_log (MergerMutator): Selected 6 parts from 202010_1_31_6 to 202010_36_36_0
2020.10.01 02:37:05.167937 [ 8717 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:37:05.167960 [ 8717 ] {} <Debug> system.metric_log (MergerMutator): Merging 6 parts: from 202010_1_31_6 to 202010_36_36_0 into Compact
2020.10.01 02:37:05.168221 [ 8717 ] {} <Debug> system.metric_log (MergerMutator): Selected MergeAlgorithm: Horizontal
2020.10.01 02:37:05.194132 [ 8717 ] {} <Debug> system.metric_log (MergerMutator): Merge sorted 271 rows, containing 264 columns (264 merged, 0 gathered) in 0.026170765 sec., 10355.066044114492 rows/sec., 20.76 MiB/sec.
2020.10.01 02:37:05.195767 [ 8717 ] {} <Debug> MemoryTracker: Peak memory usage: 4.30 MiB.
2020.10.01 02:37:12.365832 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.86 GiB.
2020.10.01 02:37:12.671975 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.86 GiB.
2020.10.01 02:37:19.867551 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.86 GiB.
2020.10.01 02:37:19.951645 [ 8751 ] {} <Debug> DNSResolver: Updating DNS cache
2020.10.01 02:37:19.952210 [ 8751 ] {} <Debug> DNSResolver: Updated DNS cache
2020.10.01 02:37:20.179412 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.86 GiB.
2020.10.01 02:37:22.357041 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Information> executeQuery: Read 56528894 rows, 16.03 GiB in 33.814214038 sec., 1671749 rows/sec., 485.37 MiB/sec.
2020.10.01 02:37:22.357332 [ 8756 ] {2664ffc3-5304-4825-bcf4-3a5f8e820d6b} <Information> DynamicQueryHandler: Done processing query
[root@masternode clickhouse-server]# time clickhouse-client -h 127.0.0.1 --port 9000 <<< $QUERY | wc -c
22340209125
real 0m20.172s
user 0m6.663s
sys 0m24.558s
2020.10.01 02:34:56.909353 [ 8755 ] {} <Debug> TCPHandler: Connected ClickHouse client version 20.10.0, revision: 54441, user: default.
2020.10.01 02:34:56.920035 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> executeQuery: (from 127.0.0.1:56378) SELECT Path, Time, Value, Timestamp FROM db003_test.points_r1m_avg WHERE Time >= 1600921652 AND Time <= 1601526452 AND Path IN (SELECT Path FROM db003_test.graphite_tagged GROUP BY Path) FORMAT RowBinary;
2020.10.01 02:34:56.920732 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> InterpreterSelectQuery: MergeTreeWhereOptimizer: condition "(Time >= 1600921652) AND (Time <= 1601526452)" moved to PREWHERE
2020.10.01 02:34:56.921198 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.graphite_tagged (SelectExecutor): Key condition: unknown
2020.10.01 02:34:56.921251 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.graphite_tagged (SelectExecutor): MinMax index condition: unknown
2020.10.01 02:34:56.921272 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.graphite_tagged (SelectExecutor): Selected 1 parts by partition key, 1 parts by primary key, 18 marks by primary key, 18 marks to read from 1 ranges
2020.10.01 02:34:56.969683 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.graphite_tagged (SelectExecutor): Key condition: unknown
2020.10.01 02:34:56.969734 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.graphite_tagged (SelectExecutor): MinMax index condition: unknown
2020.10.01 02:34:56.969746 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.graphite_tagged (SelectExecutor): Selected 1 parts by partition key, 1 parts by primary key, 18 marks by primary key, 18 marks to read from 1 ranges
2020.10.01 02:34:57.004877 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.points_r1m_avg (SelectExecutor): Key condition: (column 1 in [1600921652, +inf)), (column 1 in (-inf, 1601526452]), and, (column 0 in 8208-element set), and, (column 1 in [1600921652, +inf)), (column 1 in (-inf, 1601526452]), and, and
2020.10.01 02:34:57.004938 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.points_r1m_avg (SelectExecutor): MinMax index condition: unknown, unknown, and, unknown, and, unknown, unknown, and, and
2020.10.01 02:34:57.020697 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> db003_test.points_r1m_avg (SelectExecutor): Selected 7 parts by partition key, 7 parts by primary key, 6904 marks by primary key, 6904 marks to read from 7 ranges
2020.10.01 02:34:57.349037 [ 8735 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:34:57.351924 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:34:57.353738 [ 8728 ] {} <Debug> system.trace_log (MergerMutator): Selected 6 parts from 202010_1_6_1 to 202010_11_11_0
2020.10.01 02:34:57.353841 [ 8728 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:34:57.353958 [ 8728 ] {} <Debug> system.trace_log (MergerMutator): Merging 6 parts: from 202010_1_6_1 to 202010_11_11_0 into Compact
2020.10.01 02:34:57.354043 [ 8728 ] {} <Debug> system.trace_log (MergerMutator): Selected MergeAlgorithm: Horizontal
2020.10.01 02:34:57.354140 [ 8723 ] {} <Debug> system.query_thread_log (MergerMutator): Selected 6 parts from 202010_1_1_0 to 202010_6_6_0
2020.10.01 02:34:57.354219 [ 8723 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:34:57.354261 [ 8723 ] {} <Debug> system.query_thread_log (MergerMutator): Merging 6 parts: from 202010_1_1_0 to 202010_6_6_0 into Compact
2020.10.01 02:34:57.354362 [ 8723 ] {} <Debug> system.query_thread_log (MergerMutator): Selected MergeAlgorithm: Horizontal
2020.10.01 02:34:57.354437 [ 8736 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:34:57.363524 [ 8718 ] {} <Debug> system.query_log (MergerMutator): Selected 6 parts from 202010_1_1_0 to 202010_6_6_0
2020.10.01 02:34:57.363604 [ 8718 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:34:57.363634 [ 8718 ] {} <Debug> system.query_log (MergerMutator): Merging 6 parts: from 202010_1_1_0 to 202010_6_6_0 into Compact
2020.10.01 02:34:57.363712 [ 8718 ] {} <Debug> system.query_log (MergerMutator): Selected MergeAlgorithm: Horizontal
2020.10.01 02:34:57.365188 [ 8728 ] {} <Debug> system.trace_log (MergerMutator): Merge sorted 585 rows, containing 10 columns (10 merged, 0 gathered) in 0.011300696 sec., 51766.72304077554 rows/sec., 8.93 MiB/sec.
2020.10.01 02:34:57.368759 [ 8723 ] {} <Debug> system.query_thread_log (MergerMutator): Merge sorted 50 rows, containing 39 columns (39 merged, 0 gathered) in 0.014488955 sec., 3450.9044993237953 rows/sec., 4.01 MiB/sec.
2020.10.01 02:34:57.369207 [ 8718 ] {} <Debug> system.query_log (MergerMutator): Merge sorted 7 rows, containing 45 columns (45 merged, 0 gathered) in 0.005571852 sec., 1256.3147764872433 rows/sec., 1.74 MiB/sec.
2020.10.01 02:34:57.497648 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:35:04.853740 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:35:04.941068 [ 8737 ] {} <Debug> DNSResolver: Updating DNS cache
2020.10.01 02:35:04.941632 [ 8737 ] {} <Debug> DNSResolver: Updated DNS cache
2020.10.01 02:35:05.008650 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:35:12.355130 [ 8734 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:35:12.518617 [ 8733 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:35:12.521167 [ 8720 ] {} <Debug> system.metric_log (MergerMutator): Selected 6 parts from 202010_1_16_3 to 202010_21_21_0
2020.10.01 02:35:12.521273 [ 8720 ] {} <Debug> DiskLocal: Reserving 1.00 MiB on disk `default`, having unreserved 10.87 GiB.
2020.10.01 02:35:12.521336 [ 8720 ] {} <Debug> system.metric_log (MergerMutator): Merging 6 parts: from 202010_1_16_3 to 202010_21_21_0 into Compact
2020.10.01 02:35:12.521722 [ 8720 ] {} <Debug> system.metric_log (MergerMutator): Selected MergeAlgorithm: Horizontal
2020.10.01 02:35:12.564678 [ 8720 ] {} <Debug> system.metric_log (MergerMutator): Merge sorted 158 rows, containing 264 columns (264 merged, 0 gathered) in 0.043364386 sec., 3643.5428833236565 rows/sec., 7.30 MiB/sec.
2020.10.01 02:35:12.567665 [ 8720 ] {} <Debug> MemoryTracker: Peak memory usage: 4.30 MiB.
2020.10.01 02:35:16.428355 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Information> executeQuery: Read 56528894 rows, 16.03 GiB in 19.508232557 sec., 2897694 rows/sec., 841.30 MiB/sec.
2020.10.01 02:35:16.428423 [ 8755 ] {5a80e1ed-c2d3-4132-981e-b707d25e15b6} <Debug> MemoryTracker: Peak memory usage (for query): 109.04 MiB.
2020.10.01 02:35:16.430043 [ 8755 ] {} <Debug> MemoryTracker: Peak memory usage (for query): 109.04 MiB.
2020.10.01 02:35:16.430082 [ 8755 ] {} <Information> TCPHandler: Processed in 19.510784629 sec.
2020.10.01 02:35:17.047994 [ 8755 ] {} <Information> TCPHandler: Done processing connection.
from graphite-clickhouse.
Felixoid
commented on July 16, 2024
Side note: when I've tested LowCardinality(String) one year ago, it was MUCH worse than just String CODEC(ZSTD(3)). The memory consumption on the CH side was doubled because of LowCardinality index files. Consider testing the string with compressing, it could save some resources. As well, I use Float64 CODEC(Gorilla, LZ4) for Value column. But I didn't test the ZSTD(1) for it.
Could you run hyperfine to take statistics for a series of runs instead of one? Save your test query to the file query, then it should be something like
hyperfine --warmup 2 "clickhouse-client -h 127.0.0.1 --port 9000 < query"
hyperfine --warmup 2 "curl -s --data-binary @query 'http://127.0.0.1:8123/'"
from graphite-clickhouse.
lexx-bright
commented on July 16, 2024
With old query.
export QUERY="SELECT Path, Time, Value, Timestamp
FROM db003_test.points_r1m_avg
WHERE
Time >= 1600921652
AND Time <= 1601526452
AND Path IN (SELECT Path FROM db003_test.graphite_tagged GROUP BY Path)
FORMAT RowBinary;"
[root@masternode ~]# /root/.cargo/bin/hyperfine --show-output --warmup 2 "curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< \$QUERY | wc -c"
Benchmark #1: curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< $QUERY | wc -c
22340209125
...
Time (mean ± σ): 30.723 s ± 3.160 s [User: 5.834 s, System: 42.959 s]
Range (min … max): 26.086 s … 35.895 s 10 runs
[root@masternode ~]# /root/.cargo/bin/hyperfine --show-output --warmup 2 "clickhouse-client -h 127.0.0.1 --port 9000 <<< \$QUERY | wc -c"
Benchmark #1: clickhouse-client -h 127.0.0.1 --port 9000 <<< $QUERY | wc -c
22340209125
...
Time (mean ± σ): 19.830 s ± 0.984 s [User: 6.452 s, System: 24.190 s]
Range (min … max): 18.111 s … 21.313 s 10 runs
With new query.
export QUERY="SELECT Path, groupArray(Time), groupArray(Value), groupArray(Timestamp)
FROM db003_test.points_r1m_avg
WHERE
Time >= 1600921652
AND Time <= 1601526452
AND (Path IN (SELECT Path FROM db003_test.graphite_tagged GROUP BY Path))
GROUP BY Path FORMAT RowBinary;"
[root@masternode ~]# /root/.cargo/bin/hyperfine --show-output --warmup 2 "curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< \$QUERY | wc -c"
Benchmark #1: curl -s --data-binary @- 'http://127.0.0.1:8123/' <<< $QUERY | wc -c
907626256
...
Time (mean ± σ): 3.262 s ± 0.191 s [User: 261.0 ms, System: 1933.2 ms]
Range (min … max): 3.049 s … 3.647 s 10 runs
[root@masternode ~]# /root/.cargo/bin/hyperfine --show-output --warmup 2 "clickhouse-client -h 127.0.0.1 --port 9000 <<< \$QUERY | wc -c"
Benchmark #1: clickhouse-client -h 127.0.0.1 --port 9000 <<< $QUERY | wc -c
907626256
...
Time (mean ± σ): 3.307 s ± 0.091 s [User: 1.841 s, System: 1.073 s]
Range (min … max): 3.209 s … 3.505 s 10 runs
New query drastically outperforms the old one due to reduced amount of data. But there may be the same performance issue if dataset is much bigger .
from graphite-clickhouse.
Felixoid
commented on July 16, 2024
With the latest master, there isn't possible such thing as big data set. Please, try the query with enabled internal-aggregation = true, as described in #88. Be aware of using a slightly tweaked carbonapi config.
I guess the amount of data should be decreased even more :-j
from graphite-clickhouse.
Felixoid
commented on July 16, 2024
Ok, nothing could be done here, I guess. Internal aggregation should solve most of the performance problems.
from graphite-clickhouse.
Related Issues (20)
- How to let graphite-web's tag works with graphite-clickhouse? HOT 20
- graphite-clickhouse is broken with ClickHouse >21.1 over tls HOT 5
- Tags and grafana multi-value variables
- Carbonapi got unexpected EOF on some queries responce HOT 2
- "Too many simultaneous queries" error after upgrading to 0.12 HOT 6
- Automate packages pushing for autobuilds and releases HOT 5
- Wrong glibc version in Debian Buster HOT 1
- Cannot disable index table HOT 1
- Metrics with NULL values are not processed correctly HOT 4
- Purpose of filter "Path in metrics_list" HOT 4
- empty output in CH because of different timezone in cluster and on carbon-clickhouse/graphite-clichouse/carbonapi node HOT 5
- seriesByTag('') causes "index out of range [0] with length 0" HOT 6
- Unescaped metric name in the response. HOT 5
- Printed default config is invalid HOT 1
- seriesByTag invalid regular expression parsing HOT 4
- Question: Skip Aggregation on the first retention policy HOT 1
- Internal aggregation is using incorrect from and until unix timestamps when maxDataPoint is used HOT 7
- invalid memory address or nil pointer dereference HOT 1
- graphite-clickhouse 0.13+ - not support tree-table? HOT 3
- Release new version? HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from graphite-clickhouse.