danthegoodman1 / bighouse Goto Github PK
View Code? Open in Web Editor NEWRunning ClickHouse like it's BigQuery
License: Other
Running ClickHouse like it's BigQuery
License: Other
Can do single 16 core machine vs 8 or 10+ of them on the ookla data set: aws s3 ls --no-sign-request s3://ookla-open-data/parquet/performance/type=mobile/year=2019/quarter=1/
Record performance in README.
Make sure that we are creating machines all in the same region.
We can increase performance substantially if we can instead pre-create machines, and then update their configuration with new env vars (which should boot them as well). This should allow for much faster boot times.
Assuming ipv6 addresses don't change either, this means we can boostrap the cluster super fast.
We will not only need to keep track of machines, but have our own baby scheduling system that can grab them. Can probably do this with UPDATE ... SKIP LOCKED
.
We also need to have timeouts for what happens when we don't get all the nodes in time, or emergency "go get more nodes" when we fail to boot some.
CH container that doesn't have table engines. Maybe leave what's needed for web tables (mergetree) but can definitely remove a ton of table engines like log and such.
Also build without compressing CH so it can boot faster, container image compression makes it useless anyway.
This will greatly improve startup speed.
Run time, rows/s, rows read, data read, etc. should be included in the query response
https://clickhouse.com/docs/en/operations/settings/settings#settings-max_query_size so icedb returns are not breaking queries when there are thousands of files, make this an env var
Either CH is taking too long to exit, or we are not getting the signal passed properly (probably this). Need to make it shut down faster to fix slow fly update.
Discussing with qxip, a generic url(Cluster)-like table function that will fetch the list of files from a remote url based on the query predicate, and then behave as normal url(Cluster) with the resulting file list.
Useful for baking IceDB right into CH via a metadata server (rather than CH querying the meta store directly).
Queries that run between checking for cluster bootstrapping and running user query. For example setting up a web table: https://github.com/ClickHouse/web-tables-demo
Have option for single one that runs on any node, and a query that will run once on every node.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.