danthegoodman1 / bighouse Goto Github PK

We can increase performance substantially if we can instead pre-create machines, and then update their configuration with new env vars (which should boot them as well). This should allow for much faster boot times.

Assuming ipv6 addresses don't change either, this means we can boostrap the cluster super fast.

We will not only need to keep track of machines, but have our own baby scheduling system that can grab them. Can probably do this with UPDATE ... SKIP LOCKED.

We also need to have timeouts for what happens when we don't get all the nodes in time, or emergency "go get more nodes" when we fail to boot some.

Can do single 16 core machine vs 8 or 10+ of them on the ookla data set: aws s3 ls --no-sign-request s3://ookla-open-data/parquet/performance/type=mobile/year=2019/quarter=1/

Record performance in README.

Make sure that we are creating machines all in the same region.

CH container that doesn't have table engines. Maybe leave what's needed for web tables (mergetree) but can definitely remove a ton of table engines like log and such.

Also build without compressing CH so it can boot faster, container image compression makes it useless anyway.

This will greatly improve startup speed.

Queries that run between checking for cluster bootstrapping and running user query. For example setting up a web table: https://github.com/ClickHouse/web-tables-demo

Have option for single one that runs on any node, and a query that will run once on every node.

Discussing with qxip, a generic url(Cluster)-like table function that will fetch the list of files from a remote url based on the query predicate, and then behave as normal url(Cluster) with the resulting file list.

Useful for baking IceDB right into CH via a metadata server (rather than CH querying the meta store directly).

Either CH is taking too long to exit, or we are not getting the signal passed properly (probably this). Need to make it shut down faster to fix slow fly update.

https://clickhouse.com/docs/en/operations/settings/settings#settings-max_query_size so icedb returns are not breaking queries when there are thousands of files, make this an env var

See https://github.com/ClickHouse/web-tables-demo and #4

Run time, rows/s, rows read, data read, etc. should be included in the query response