Add support for incremental checkpointing and checkpointing during runtime about duckdb HOT 2 CLOSED

Notes from today's discussion

storage

abstract block backends for file/cloud/...
block references in API, checksums on blocks

configurable block size
fsync costs depending on file size
async io?

use direct io O_DIRECT
group commit?

hashing
logical checks? format correct? validation
write-verify before transaction commit on checkpoint
WAL write-verify?

1 MB min for magnetic
32K for flash

from duckdb.

I just wanted to add one more +1 to this feature! I know it is on the roadmap already, just excited for it! I also figured I'd lay out my use case in case it is helpful to see how DuckDB is coming in handy "in the wild". Since SQL is the language I'm best at, I really enjoy using DuckDB rather than fighting with data frame syntax.

I am working with an 80GB data set to start with and my data processing is expanding the size to 400 GB. Each query is using almost all of my RAM (200 of 250GB) so I have to disconnect/reconnect between each query to trigger a checkpoint. That is leading to a full 80-400GB re-checkpoint that slows things down a fair bit (30 minutes runtime for each of 100 queries...). I'm still on version 0.2.2, but I didn't see something in the release notes that would change this behavior quite yet.

Thank you for all of the work you do!! Local big data processing on DuckDB is so close!!

from duckdb.