Add support for auto-increment and skip_rows about clickhouse HOT 5 OPEN

I would take 1 point every 1000.

You need to put some constraints on which OFFSETS and STEP values you can have.
Because in general case it does mean random access of rows, and clickhouse as block_compressed dbms with sparse index bad fit for that.

from clickhouse.

Hey @UnamedRus. For sure. This is only a suggestion on the API side. I am pretty sure Clickhouse can check the OFFSETS and SKIP_ROWS and fail the query if it is out of bound.

For the sparse compressed dbms, for my use case, there is a garuantee the row is only incrementing by 1, so it won't be sparse.

On my side, I am tracking the number of rows as side metadata, so It won't exceed it. But I am thinking to go and build my own storage mechanism right now as Clickhouse isn't fast enough for my use case.

Or I would be happy to consider contributing pieces if I was guided enough.

from clickhouse.

From my understanding auto-increment requires the UniqueMergeTree to be implemented #41817 and this one requires a built-in RAFT for maintaining the state.

from clickhouse.

Hey @nikitamikhaylov. Thanks for sharing about the RFC. From what I can read, I would still need support for a primary key which isn't shown in the proposed designed.

from clickhouse.

Hey @nikitamikhaylov. Extra information. There is a lot of speed to gain by implementing this natively in Clickhouse and making sampling more flexible, non-deterministic in general.

I have implemented my own storage mechanism and I am getting a 10x speed compared to clickhouse, 1/10 storage space.

Kind regards, T.C

from clickhouse.