Index out of range crash about bolt HOT 8 CLOSED

Here's a slightly nicer repro case:

https://gist.github.com/cespare/caef571fd9c8d3db72c2

Here I closed and reopened the DB from within my code so you only need to run it once.

• Open the DB
• Write 1000 documents
• Close and reopen
• Call db.Buckets() crashes

from bolt.

It looks like when opening a file, it's mmapped at whatever size the file naturally has, and then the Buckets call tries to access a page beyond that point, getting an index out of bounds panic.

Not 100% on what's the best way to fix this. It seems the db size / page count is not stored explicitly in the meta page? Otherwise, could read that to get the right mmap size, when opening. Alternatively, always write to last byte when growing -- but even then, it's not guaranteed to hit the disk immediately.

from bolt.

@cespare @tv42 It was a really dumb bug. While calculating the minimum size I used a wrong intermediate variable. It's fixed with #55.

from bolt.

@cespare by the way, thanks for the reproducible code. It made it really easy to fix the bug.

from bolt.

@benbjohnson Thanks!

from bolt.

@tv42 The meta page doesn't store the last byte written but it does store a high water mark for pages using Meta.pgid. You can calculate the last byte by multiplying that by the Meta.pageSize.

from bolt.

I see what's going on with meta.pgid, I confused that with page.id when reading so I was blind to it. I just went through the meta0 vs meta1 and transaction handling code -- really sweet & simple!

from bolt.

@tv42 I really like the simple approach LMDB used with the "double buffering" meta pages. It's hard to get much simpler than that. :)

I should probably document some of the unexported fields (e.g. meta.pgid, etc). Or maybe better names would help like meta.maxpgid. Actually, I need to fix some of the naming. I tried to Go-ify the names as I ported it over but I didn't touch the lower level pieces.

Maybe the following changes in Meta to start:

type Meta struct {
    ...
    bucketPageID  pageID
    freelistPageID pageID
    maxPageID     pageID
    maxTransactionID    transactionID
}

from bolt.