slach / clickhouse-flamegraph Goto Github PK
View Code? Open in Web Editor NEWCLI utility for build flamegraph based on system.trace_log
License: MIT License
clickhouse-flamegraph's People
Contributors
![renovate[bot] avatar](https://avatars.githubusercontent.com/in/2740?v=4 "renovate[bot]")
Stargazers
Watchers
Forkers
junneyang shangshujie365 isgasho playprogram111 msaf1980 lingnancfy awesomeleo taiyang-li heenabansal2009 lyndonyang1024 infine8 yusufozturk srikanthccv fuziontech jackywooclickhouse-flamegraph's Issues
Dependency Dashboard
This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.
This repository currently has no open or pending branches.
Detected dependencies
docker-compose
docker-compose.yml
dockerfile
docker/clickhouse-flamegraph/Dockerfile
alpine 3.19
github-actions
.github/workflows/codeql-analysis.yml
actions/checkout v4github/codeql-action v3github/codeql-action v3github/codeql-action v3
gomod
go.mod
go 1.19github.com/araddon/dateparse v0.0.0-20210429162001-6b43995a97de@6b43995a97degithub.com/mailru/go-clickhouse/v2 v2.2.1github.com/pkg/errors v0.9.1github.com/rs/zerolog v1.32.0github.com/urfave/cli/v2 v2.27.2
- Check this box to trigger a request for Renovate to run again on this repository
Index out of range
Build with latest master branch
go version
go version go1.21.6 linux/amd64
And run as
clickhouse-flamegraph --clickhouse-dsn xx@xx/default --query-id 7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84 --flamegraph-script /data1/home/wujianchao/cmd/FlameGraph --output-dir . --trace-type CPU --format svg --debug
But got index out of range
{"level":"info","dsn":"xxxx","time":"2024-04-25T15:10:54+08:00","message":"connected to ClickHouse"}
panic: runtime error: index out of range [0] with length 0
goroutine 1 [running]:
github.com/mailru/go-clickhouse/v2.ensureHavePort({0x0, 0x0})
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/config.go:187 +0x13f
github.com/mailru/go-clickhouse/v2.(*Config).url(0xc000174580, 0xc00023ced8?, 0x0)
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/config.go:85 +0x47
github.com/mailru/go-clickhouse/v2.newConn(0xc000174580)
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/conn.go:68 +0x105
github.com/mailru/go-clickhouse/v2.(*chDriver).Open(0x0?, {0x7ffcb1e6c472?, 0xc00023d150?})
/data1/home/wujianchao/go/pkg/mod/github.com/mailru/go-clickhouse/[email protected]/clickhouse.go:22 +0x3a
database/sql.dsnConnector.Connect(...)
/usr/local/go/src/database/sql/sql.go:758
database/sql.(*DB).conn(0xc000163110, {0x8e9860, 0xbb1b20}, 0x1)
/usr/local/go/src/database/sql/sql.go:1387 +0x717
database/sql.(*DB).query(0x0?, {0x8e9860, 0xbb1b20}, {0x84cd4a, 0x1b}, {0x0, 0x0, 0x0}, 0x98?)
/usr/local/go/src/database/sql/sql.go:1721 +0x57
database/sql.(*DB).QueryContext.func1(0x0?)
/usr/local/go/src/database/sql/sql.go:1704 +0x4f
database/sql.(*DB).retry(0xc00023d3b8?, 0xc00023d380)
/usr/local/go/src/database/sql/sql.go:1538 +0x42
database/sql.(*DB).QueryContext(0x17?, {0x8e9860?, 0xbb1b20?}, {0x84cd4a?, 0x83a9cc?}, {0x0?, 0x0?, 0x1?})
/usr/local/go/src/database/sql/sql.go:1703 +0xc5
database/sql.(*DB).Query(...)
/usr/local/go/src/database/sql/sql.go:1717
main.fetchQuery(0x7ffcb1e6c472?, {0x84cd4a, 0x1b}, {0x0, 0x0, 0x0}, 0xc00023d690)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:586 +0x73
main.checkClickHouseVersion(...)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:398
main.generate(0xc0001486c0)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:263 +0x1ba
main.run(0xc00022c160?)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:199 +0x1e6
github.com/urfave/cli/v2.(*Command).Run(0xc00022c160, 0xc0001486c0, {0xc0000240e0, 0xe, 0xe})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x9dd
github.com/urfave/cli/v2.(*App).RunContext(0xc000226000, {0x8e9860?, 0xbb1b20}, {0xc0000240e0, 0xe, 0xe})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:337 +0x5db
github.com/urfave/cli/v2.(*App).Run(...)
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311
main.main()
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:152 +0x1725
Usability issue + typo.
milovidov@server:~$ clickhouse-flamegraph --verbose --from=2019-11-30T00:00:00+03:00 --to=2019-11-30T01:00:00+03:00
{"level":"info","dsn":"tcp://localhost:9000?database=default","time":"2019-11-30T00:45:47+03:00","message":"conected to clickhouse"}
{"level":"debug","sql":"\nSELECT \n\tquery, query_id \nFROM system.query_log\nWHERE type = 1 AND event_time >= ? AND event_time <= ?\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:45:47+03:00","message":"query OK"}
{"level":"debug","sql":"\nSELECT \n\tquery_id,\n\tcount() AS samples, \n\tarrayStringConcat(arrayReverse(arrayMap(x -> concat( demangle(addressToSymbol(x)), '#', addressToLine(x) ), trace)), ';') AS stack\nFROM system.trace_log\nWHERE event_time >= ? AND event_time <= ?\nGROUP BY query_id, trace\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:45:47+03:00","message":"query OK"}
{"level":"fatal","error":"open clickouse-flamegraphs/1205caf2-5262-4825-85ea-521ffe9a68e2.txt: no such file or directory","stackFile":"clickouse-flamegraphs/1205caf2-5262-4825-85ea-521ffe9a68e2.txt","time":"2019-11-30T00:45:51+03:00"}
milovidov@server:~$ mkdir clickouse-flamegraphs
milovidov@server:~$ clickhouse-flamegraph --verbose --from=2019-11-30T00:00:00+03:00 --to=2019-11-30T01:00:00+03:00
{"level":"info","dsn":"tcp://localhost:9000?database=default","time":"2019-11-30T00:46:57+03:00","message":"conected to clickhouse"}
{"level":"debug","sql":"\nSELECT \n\tquery, query_id \nFROM system.query_log\nWHERE type = 1 AND event_time >= ? AND event_time <= ?\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:46:57+03:00","message":"query OK"}
{"level":"debug","sql":"\nSELECT \n\tquery_id,\n\tcount() AS samples, \n\tarrayStringConcat(arrayReverse(arrayMap(x -> concat( demangle(addressToSymbol(x)), '#', addressToLine(x) ), trace)), ';') AS stack\nFROM system.trace_log\nWHERE event_time >= ? AND event_time <= ?\nGROUP BY query_id, trace\n","sqlArgs":"[2019-11-30 00:00:00 +0300 MSK 2019-11-30 01:00:00 +0300 MSK]","time":"2019-11-30T00:46:57+03:00","message":"query OK"}
{"level":"info","processedFiles":4,"time":"2019-11-30T00:47:03+03:00","message":"done processing"}
- It doesn't work unless I manually create a
clickouse-flamegraphsdirectory. - This directory name contains a typo.
Error when output dir is `.`
Error when output dir is .
clickhouse-flamegraph --clickhouse-dsn http://xxx?allow_introspection_functions=1 --query-id 7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84 --flamegraph-script /data1/home/wujianchao/cmd/FlameGraph --output-dir . --trace-type CPU --format svg --debug --from '2024-04-25 00:00:00' --to '2024-04-26 00:00:00'
{"level":"info","dsn":"http://xxx?allow_introspection_functions=1","time":"2024-04-25T15:38:21+08:00","message":"connected to ClickHouse"}
{"level":"debug","sql":"SELECT version() AS version","sqlArgs":"[]","time":"2024-04-25T15:38:21+08:00","message":"query OK"}
{"level":"debug","sql":"\nSELECT DISTINCT hostName() AS host_name, q.query, q.query_id \nFROM system.query_log AS q\nWHERE event_time >= ? AND event_time <= ? AND query_id IN ('7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84') \n","sqlArgs":"[2024-04-25 00:00:00 +0000 UTC 2024-04-26 00:00:00 +0000 UTC]","time":"2024-04-25T15:38:21+08:00","message":"query OK"}
{"level":"info","sqlFiles":1,"time":"2024-04-25T15:38:21+08:00","message":"write .sql files"}
{"level":"debug","sql":"\nSELECT \n\thostName() AS host_name,\n replaceAll(t.query_id,':','_') AS query_id,\n\ttrace_type,\n\tsum(abs(size)) AS total_size,\n\tcount() AS samples, \n\tconcat(\n\t\tmultiIf( \n\t\t\tposition( toString(trace_type), 'Memory') > 0 AND sum(size) >= 0, 'allocate;',\n\t\t\tposition( toString(trace_type), 'Memory') > 0 AND sum(size) < 0, 'free;',\n\t\t\tconcat( toString(trace_type), ';')\n\t\t),\n\t\tarrayStringConcat(arrayReverse(arrayMap(x -> concat( demangle(addressToSymbol(x)), '#', addressToLine(x) ), trace)), ';')\n\t) AS stack\nFROM system.trace_log AS t ANY LEFT JOIN system.query_log AS q ON q.query_id=t.query_id\nWHERE trace_type IN ('CPU') AND event_time >= ? AND event_time <= ? AND query_id IN ('7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84') \nGROUP BY host_name, query_id, trace_type, trace\nSETTINGS allow_introspection_functions=1\n","sqlArgs":"[2024-04-25 00:00:00 +0000 UTC 2024-04-26 00:00:00 +0000 UTC]","time":"2024-04-25T15:38:21+08:00","message":"query OK"}
panic: runtime error: index out of range [1] with length 1
goroutine 1 [running]:
main.generate(0xc0001ce680)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:349 +0xb45
main.run(0xc000256160?)
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:199 +0x1e6
github.com/urfave/cli/v2.(*Command).Run(0xc000256160, 0xc0001ce680, {0xc0001aa000, 0x12, 0x12})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/command.go:279 +0x9dd
github.com/urfave/cli/v2.(*App).RunContext(0xc00024c000, {0x8e9860?, 0xbb1b20}, {0xc0001aa000, 0x12, 0x12})
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:337 +0x5db
github.com/urfave/cli/v2.(*App).Run(...)
/data1/home/wujianchao/go/pkg/mod/github.com/urfave/cli/[email protected]/app.go:311
main.main()
/data1/home/wujianchao/project/github/clickhouse-flamegraph/main.go:152 +0x1725
The formatted sql in log :
SELECT
hostName() AS host_name,
replaceAll(t.query_id, ':', '_') AS query_id,
trace_type,
sum(abs(size)) AS total_size,
count() AS samples,
concat(
multiIf(
position(toString(trace_type), 'Memory') > 0 AND sum(size) >= 0, 'allocate;',
position(toString(trace_type), 'Memory') > 0 AND sum(size) < 0, 'free;',
concat(toString(trace_type), ';')
),
arrayStringConcat(arrayReverse(arrayMap(x -> concat(demangle(addressToSymbol(x)), '#', addressToLine(x)), trace)), ';')
) AS stack
FROM
system.trace_log AS t
ANY LEFT JOIN system.query_log AS q ON q.query_id = t.query_id
WHERE
trace_type IN ('CPU')
AND event_time >= '2024-04-25 00:00:00'
AND event_time <= '2024-04-26 00:00:00'
AND query_id IN ('7d6b01fa-89f0-4eaf-9d4b-e273a6e9ee84')
GROUP BY
host_name,
query_id,
trace_type,
trace
SETTINGS
allow_introspection_functions = 1
;
And got 79 rows in my ClickHouse instance.
Originally posted by @JackyWoo in #121 (comment)
Usage of clickhouse-flamegraph
I am trying to visualize the system.trace_log via clickhouse-flamegraph. But readme doesn't help much as a first time user to use it.
Steps I followed till now:
-
installed clickhouse-flamegraph.
-
cloned Flamegraph repo by brendenGregg.
-
Desired configuration changed for profiling in config.xml and users.xml
-
CH server is running as a service
-
execute query with clickhouse-client.
-
I have a query id to profile.
-
I executed this query to see if trace_log exists for this Query . yes it exists.
'SELECT
arrayStringConcat(arrayReverse(arrayMap(x -> concat(addressToLine(x), '#', demangle(addressToSymbol(x))), trace)), ';') AS stack,
count() AS samples
FROM system.trace_log
WHERE (query_id = '2c3238dd-9de0-4f4b-82bb-511cfb69c810') AND (event_date = today())
GROUP BY trace
ORDER BY samples DESC
LIMIT 10
FORMAT TabSeparated'
What next , how I can use this query id to visualize system.trace_log in flamegraph.
try add http and tcp support both
Use cases for clickhouse-flamegraph tool
Hi, is it possible to provide some use cases for the clickhouse-flamegraph tool. I still don't know how to use even with the usage in readme. 😅 As a user, what am I suppose to put for command and arguments, what do they even refer to? Where and when is the flamegraph svg generated, will a flamegraph be automatically generated whenever a client query? Or do I need to use the clickhouse-flamegraph tool to perform the query? It would be great if there could be a use case to follow through that helps the user to generate one flamegraph. Thank you! 😆
Max query size exceeded error - when too much query_id in date range
need refactoring main.go logic
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.