Consider decoupling the file size from the page size about sqlite-s3vfs HOT 4 CLOSED

For every page write it will require an incremental upload to S3. So if your page size is 1/4 of the block size, you will need 4 writes of the same block, most of which will just be the same data.

tamato uses the maximum block size, but we haven't done any specific performance tuning on that.

from sqlite-s3vfs.

On this, I suspect there are loads of ways of making things more performant, but I think from TaMaTo's point of view, things are fast enough(?), and most ways I (and from what I read above, you?) can think of making it more performant introduces complexity. So at the moment, I'm probably a bit anti, since there would be risk of bugs for no real benefit...

Consider recommending a larger block size (may need testing to find an optimal size, but the 65536 from the docs doesn't seem like a bad start).

And on this, I did consider explicitly recommending a larger block size, but I guess didn't really want to spend time testing, and I wonder if it's dependant on usage patterns. Maybe 4k is better than 64k for some cases, and 64k better for others?

from sqlite-s3vfs.

I'm going to shut this, but here is my un-proposal for "packed blocks":

There's a possibility of having packed-blocks that contain say 16 blocks, for initial data, or any new data that is contiguous and written fast enough that the upload HTTP POST is still open and within some small timeout.

As you point out, on writing to a packed block it gets expensive as the block needs to be split.

from sqlite-s3vfs.

There's no specific need from tamato at this time*, it's more that I enjoy making things faster.

Though it's quite possibly because we haven't been looking, benchmarking is definitely something we will look into at some point.

from sqlite-s3vfs.