Short story:
Is it intentional that newRawIterator creates a iterator.NewMergedIterator with strict = true rather than using the supplied options?

Long story:
I was writing from Ruby using https://github.com/DAddYE/leveldb and goleveldb was failing to iterate over it: open /tmp/ldb/000005.sst: no such file or directory.

Reproducible by running the follow ruby code and then the following go code. I was at least hoping to ignore the error by reducing strictness, but, as the short story points out, that didn't work. (It isn't clear to me which, if either, driver is wrong or why they can't talk to each other):

require 'leveldb'
require 'fileutils'

path = '/tmp/ldb/'
FileUtils.rm_r(path) if Dir.exists?(path)
db = LevelDB::DB.new(path)

data = 'a' * 1024
8000.times do |i|
  db.put i.to_s, data
end
db.close

package main

import (
    "fmt"
    "github.com/syndtr/goleveldb/leveldb"
)

func main() {
    path := "/tmp/ldb/"
    db, err := leveldb.OpenFile(path, nil)
    if err != nil {
        panic(err)
    }
    iterator := db.NewIterator(nil, nil)
    count := 0
    for iterator.Next() {
        count++
    }
    iterator.Release()
    fmt.Println(count, iterator.Error())
    db.Close()
}

Using btcd, which currently uses goleveldb on commit 92de35c, if disk space runs out (write /tmp/mainnet/blocks_leveldb/000054.log: no space left on device), btcd can be restarted without problem.

After updating goleveldb to commit 62cc4d4, goleveldb corrupts the database on the same error. When btcd starts back up, it says leveldb: bad record in batch

Hi,

Just stumbled across this. Might be exactly when I'm looking for. Would like to guestimate the status of this project? Alpha, beta, stable, production? Is it used in production as of today?

Also, I guess it is not related to http://code.google.com/p/leveldb/ in any way.

Thanks,
Jens

Hi!

First, thanks for your work on goleveldb!

I'm using goleveldb in smira/aptly (command-line tool). Currently opens db with default flags, which means 'create if missing'. If I delete e.g. CURRENT file from DB and open it, it would trash all DB contents. However if I run recover, it would re-create CURRENT and restored DB contents.

I wonder what is the correct way to open DB so that it won't get trashed if some files are missing (e.g. due to crash)?

In syncthing-android (wrapper for syncthing which uses goleveldb), there is a crash reproducible Nullpointerexception and crash in the goleveldb code.

crash log

original issue

I don't have this myself, but @MStrecke and @Eucyon do, and could probably try a linnux-x86 compatible binary on their device via command line.

I also don't know what to do for a minimal example, but I can try to do what you need (or maybe @calmh can help).

Edit: The same code runs without problems on Android ARMv5 and ARMv7.

Similar to #58

github.com/syndtr/goleveldb/leveldb/storage

../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:68: undefined: newFileLock
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:79: undefined: rename
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:274: undefined: rename
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:316: undefined: rename
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:353: undefined: syncDir
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:439: undefined: rename

I have a pair of crashes that I have seen when using leveldb on conformal's btcd code. I don't have a lot of information past the trace however the interesting thing to note is that btcd is an extremely heavy user that creates millions of records at speeds that are as fast as leveldb will go. At this point in time this code path has run millions of times. Any clues?

panic: interface conversion: interface is nil, not *block.Reader

goroutine 20 [running]:
github.com/syndtr/goleveldb/leveldb/table.(_Reader).getDataIter(0xc22d7d4aa0, 0xc2213dcb60, 0x205b2ad78, 0x0, 0x6, ..
.)
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:203 +0x3e6
github.com/syndtr/goleveldb/leveldb/table.(_Reader).Get(0xc22d7d4aa0, 0xc2213dcb20, 0xe, 0xe, 0x205b2ad78, ...)
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:128 +0x49a
github.com/syndtr/goleveldb/leveldb.(_tOps).get(0xc2100a7c00, 0xc229a53fa0, 0xc2213dcb20, 0xe, 0xe, ...)
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table.go:363 +0x29c
github.com/syndtr/goleveldb/leveldb.(_version).get(0xc2136c9460, 0xc2213dcb20, 0xe, 0xe, 0x0, ...)
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/version.go:144 +0x6ea
github.com/syndtr/goleveldb/leveldb.(_DB).get(0xc21000b780, 0xc21d4e5450, 0x6, 0x6, 0x1ab8e14, ...)
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:354 +0x2e5
github.com/syndtr/goleveldb/leveldb.(_DB).Get(0xc21000b780, 0xc21d4e5450, 0x6, 0x6, 0x0, ...)
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:374 +0x14e
github.com/conformal/btcdb/ldb.(_LevelDb).getBlkByHeight(0xc2100b0d90, 0x34cea, 0x0, 0x0, 0x0, ...)
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/block.go:51 +0x130
github.com/conformal/btcdb/ldb.(_LevelDb).fetchTxDataBySha(0xc2100b0d90, 0xc21731e0a0, 0x0, 0x0, 0x0, ...)
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/dbcache.go:78 +0x204
github.com/conformal/btcdb/ldb.(_LevelDb).FetchTxByShaList(0xc2100b0d90, 0xc216329000, 0x710, 0x7d0, 0x0, ...)
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/dbcache.go:47 +0x167
github.com/conformal/btcchain.(_BlockChain).fetchTxList(0xc2100c0300, 0xc2180ae1c0, 0xc216329000, 0x710, 0x7d0, ...)
/home/marco/git/go/src/github.com/conformal/btcchain/txlookup.go:124 +0x228
github.com/conformal/btcchain.(_BlockChain).fetchInputTransactions(0xc2100c0300, 0xc2180ae1c0, 0xc213756690, 0x0, 0x0
, ...)
/home/marco/git/go/src/github.com/conformal/btcchain/txlookup.go:256 +0x4c6
github.com/conformal/btcchain.(_BlockChain).checkConnectBlock(0xc2100c0300, 0xc2180ae1c0, 0xc213756690, 0x0, 0x0)
/home/marco/git/go/src/github.com/conformal/btcchain/validate.go:739 +0x163
github.com/conformal/btcchain.(_BlockChain).connectBestChain(0xc2100c0300, 0xc2180ae1c0, 0xc213756690, 0x0, 0x0)
/home/marco/git/go/src/github.com/conformal/btcchain/chain.go:893 +0x8b9
github.com/conformal/btcchain.(_BlockChain).maybeAcceptBlock(0xc2100c0300, 0xc213756690, 0x0, 0x0)
/home/marco/git/go/src/github.com/conformal/btcchain/accept.go:158 +0xe12
github.com/conformal/btcchain.(_BlockChain).ProcessBlock(0xc2100c0300, 0xc213756690, 0x0, 0x0)
/home/marco/git/go/src/github.com/conformal/btcchain/process.go:158 +0x9de
main.(_blockManager).handleBlockMsg(0xc210095b40, 0xc22a8f6ed0)
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:215 +0xeb
main.(_blockManager).blockHandler(0xc210095b40)
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:255 +0xc2
created by main.(_blockManager).Start
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:332 +0x128

rename D: \ db \ CURRENT.4 D: \ db \ CURRENT: Cannot create a file when that file already exists.

return on file_storage.go 218 line

I'm using goleveldb to keep an index from certain kinds metadata to entries in my logfiles. I have four leveldbs for each day which are rotated at midnight. After the process is running several days the memory consumption keeps growing and growing.
I'm using a write buffer of 16 Mbytes.

Here's the output from lsof:

process 30008 user 68uW REG 253,2 0 15310869 ./20140527-indexdb.ldb/servicenames/LOCK
process 30008 user 70w REG 253,2 165 15310870 ./20140527-indexdb.ldb/servicenames/LOG
process 30008 user 74w REG 253,2 54 15310871 ./20140527-indexdb.ldb/servicenames/MANIFEST-000000
process 30008 user 82w REG 253,2 8740127 15310873 ./20140527-indexdb.ldb/servicenames/000001.log
process 30008 user 129uW REG 253,2 0 15310875 ./20140527-indexdb.ldb/servicemetadata/LOCK
process 30008 user 130w REG 253,2 165 15310876 ./20140527-indexdb.ldb/servicemetadata/LOG
process 30008 user 131w REG 253,2 54 15310877 ./20140527-indexdb.ldb/servicemetadata/MANIFEST-000000
process 30008 user 134w REG 253,2 361647 15310879 ./20140527-indexdb.ldb/servicemetadata/000001.log
process 30008 user 146uW REG 253,2 0 15310851 ./20140527-indexdb.ldb/base/LOCK
process 30008 user 153w REG 253,2 24820721 15310912 ./20140527-indexdb.ldb/transactions/000002.log
process 30008 user 155w REG 253,2 165 15310852 ./20140527-indexdb.ldb/base/LOG
process 30008 user 159w REG 253,2 54 15310853 ./20140527-indexdb.ldb/base/MANIFEST-000000
process 30008 user 161w REG 253,2 59070 15310855 ./20140527-indexdb.ldb/base/000001.log
process 30008 user 164uW REG 253,2 0 15310857 ./20140527-indexdb.ldb/transactions/LOCK
process 30008 user 171w REG 253,2 365 15310858 ./20140527-indexdb.ldb/transactions/LOG
process 30008 user 175w REG 253,2 128 15310859 ./20140527-indexdb.ldb/transactions/MANIFEST-000000
process 30008 user 192uW REG 253,2 0 15310863 ./20140527-indexdb.ldb/message/LOCK
process 30008 user 193w REG 253,2 166 15310864 ./20140527-indexdb.ldb/message/LOG
process 30008 user 194w REG 253,2 54 15310865 ./20140527-indexdb.ldb/message/MANIFEST-000000
process 30008 user 196w REG 253,2 26185266 15310867 ./20140527-indexdb.ldb/message/000001.log

Here's the output from pprof:

https://github.com/syndtr/goleveldb/blob/master/leveldb/memdb/memdb.go#L376

Random is seeded with a constant. It is the same value always. So the call to randHeight always returns 1, since rnd.Int()%4 is never 0. Is that on purpose?

or should 0xdeadbeef be replaced with something like time.Now().UnixNano()

It looks like I've encountered an issue with goleveldb. I'm using it in https://github.com/smira/aptly.

Short version:

• fill DB with some data
• cleanup db
• remove CURRENT file (simulate crash)
• recover db
• some keys are lost

If we skip cleanup DB step, data is not lost (as it looks).

I'm using go1.3, goleveldb @ ff3719c

All stuff that interacts with goleveldb is isolated in https://github.com/smira/aptly/blob/master/database/leveldb.go

I have all the db states (initial, after cleanup, after recover), they're pretty big (136M each), please let me know if you need them.

In the documentation, there is the following example:

iter := db.NewIterator(nil)
for iter.Next() {
    // Remember that the contents of the returned slice should not be modified, and
    // only valid until the next call to Next.
    key := iter.Key()
    value := iter.Value()
    ...
}
iter.Release()
err = iter.Error()

It appears to me that the preferred way to seek then iterate would be:

iter.Seek(key)
for iter.Next() {
    doSomethingWith(iter.Key(), iter.Value())
}

But after beating my head against this for a little bit and checking other documentation, it skips one! Obviously, this is because Seek seeks the next value, not an arbitrary location in the DB.

So after revealing a bit of confusion on my part: is there a "blessed" way to Seek-then-Iterate?

leveldb/storage/file_storage_windows.go

I was surprised not to find it http://godoc.org/github.com/syndtr/goleveldb I've clicked the "Refresh" links, but still can't see any documentation for the module.

I write a high performance NoSQL named ledisdb which uses cgo + leveldb and works well in production.

Now I want to use pure go leveldb instead so I use your goleveldb here: ledisdb-pure-go. It passes all the tests and benchmark looks well, but I am still unsure to use it in production.

Has someone used it in production yet?

go test -bench .
PASS
BenchmarkDBWrite 200000 9580 ns/op 12.11 MB/s
BenchmarkDBWriteBatch 1000000 2864 ns/op 40.50 MB/s
BenchmarkDBWriteUncompressed 200000 9180 ns/op 12.64
MB/s
BenchmarkDBWriteBatchUncompressed 1000000 3045 ns/op
38.09 MB/s
BenchmarkDBWriteRandom 200000 11085 ns/op 10.46 MB/s
BenchmarkDBWriteRandomSync panic: 0 0 ns/op
runtime error: invalid memory address or nil pointer dereferencetesting: Benchma
rkDBWriteRandomSync left GOMAXPROCS set to 8

[signal 0xc0000005 code=0x1 addr=0x0 pc=0x4503ed]

goroutine 16862 [running]:
github.com/syndtr/goleveldb/leveldb.BenchmarkDBWriteRandomSync(0xc0809ebc30)
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/bench_test.go:299 +0x3
d
testing.(_B).runN(0xc0809ebc30, 0x1)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist417694871/go/src/pkg/testi
ng/benchmark.go:119 +0x89
testing.(_B).launch(0xc0809ebc30)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist417694871/go/src/pkg/testi
ng/benchmark.go:187 +0x79
created by testing.(*B).run
C:/Users/ADMINI~1/AppData/Local/Temp/2/bindist417694871/go/src/pkg/testi
ng/benchmark.go:167 +0x2f

goroutine 1 [running]:
testing.RunBenchmarks(0x624f80, 0x729c00, 0x13, 0x13)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist417694871/go/src/pkg/testi
ng/benchmark.go:288 +0x2d6
testing.Main(0x624f80, 0x72b8a0, 0x42, 0x42, 0x729c00, ...)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist417694871/go/src/pkg/testi
ng/testing.go:373 +0x1b2
main.main()
C:/Users/ChengWei/AppData/Local/Temp/go-build705916663/github.com/syndtr
/goleveldb/leveldb/_test/_testmain.go:211 +0x9d

goroutine 648 [finalizer wait]:

goroutine 16864 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compaction(0xc0810b5900)
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:437 +
0x692
created by github.com/syndtr/goleveldb/leveldb.openDB
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:117 +0x44b

goroutine 16865 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0xc0810b5900)
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/db_write.go:36 +0x137
created by github.com/syndtr/goleveldb/leveldb.openDB
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:118 +0x465

goroutine 16863 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compactionError(0xc0810b5900)
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:115 +
0x1c6
created by github.com/syndtr/goleveldb/leveldb.openDB
c:/gopath/src/github.com/syndtr/goleveldb/leveldb/db.go:114 +0x40f
exit status 2
FAIL github.com/syndtr/goleveldb/leveldb 66.494s

plan9 does not have syscall.{LOCK_UN,LOCK_EX,Flock,LOCK_NB}.

perhaps using the plan 9 lock bit (DMEXCL) would be better.
build log of CGO_ENABLED=0 goxc -os=plan9 follows.

[goxc] 2013/09/13 11:08:13 Working directory: '/home/mischief/code/go/src/github.com/syndtr/goleveldb/leveldb/storage', Config name: 'default'
[goxc] 2013/09/13 11:08:13 'main package' dirs: []
[goxc] 2013/09/13 11:08:13 Running tasks: [go-vet go-test go-install xc codesign copy-resources archive pkg-build rmbin downloads-page]
[goxc:go-vet] 2013/09/13 11:08:13 invoking 'go vet ./...' from '/home/mischief/code/go/src/github.com/syndtr/goleveldb/leveldb/storage'
go tool: no such tool "vet"; to install:
    go get code.google.com/p/go.tools/cmd/vet
[goxc:go-vet] 2013/09/13 11:08:13 'go' returned error: exit status 3
[goxc:go-vet] 2013/09/13 11:08:13 Go-vet failed (goxc just treats this as a warning now)
[goxc:go-vet] 2013/09/13 11:08:13 Task go-vet succeeded
[goxc:go-test] 2013/09/13 11:08:13 invoking 'go test ./... -ldflags "-X main.BUILD_DATE '2013-09-13T11:08:13-07:00' -X main.VERSION 'unknown' "' from '/home/mischief/code/go/src/github.com/syndtr/goleveldb/leveldb/storage'
ok      github.com/syndtr/goleveldb/leveldb/storage 0.002s
[goxc:go-test] 2013/09/13 11:08:13 Task go-test succeeded
[goxc:go-install] 2013/09/13 11:08:13 invoking 'go install -ldflags "-X main.BUILD_DATE '2013-09-13T11:08:13-07:00' -X main.VERSION 'unknown' "' from '/home/mischief/code/go/src/github.com/syndtr/goleveldb/leveldb/storage'
[goxc:go-install] 2013/09/13 11:08:13 Task go-install succeeded
[goxc:xc] 2013/09/13 11:08:13 building for platform plan9_386.
[goxc:xc] 2013/09/13 11:08:13 specified env vars for 'go': [GOOS=plan9 GOARCH=386]
[goxc:xc] 2013/09/13 11:08:13 invoking 'go build -ldflags "-X main.BUILD_DATE '2013-09-13T11:08:13-07:00' -X main.VERSION 'unknown' " -o /home/mischief/code/go/bin/storage-xc/unknown/plan9_386/storage .' from '/home/mischief/code/go/src/github.com/syndtr/goleveldb/leveldb/storage'
# github.com/syndtr/goleveldb/leveldb/storage
./file_storage_unix.go:42: undefined: syscall.LOCK_UN
./file_storage_unix.go:44: undefined: syscall.LOCK_EX
./file_storage_unix.go:46: undefined: syscall.Flock
./file_storage_unix.go:46: undefined: syscall.LOCK_NB
[goxc:xc] 2013/09/13 11:08:13 'go' returned error: exit status 2
[goxc:xc] 2013/09/13 11:08:13 Error: exit status 2
[goxc:xc] 2013/09/13 11:08:13 Have you run `goxc -t` for this platform???
[goxc:xc] 2013/09/13 11:08:13 No successes!
[goxc:xc] 2013/09/13 11:08:13 Stopping after 'xc' failed with error 'exit status 2'

go version
go version devel +d58997478ec6 Mon Apr 08 00:09:35 2013 -0700 windows/386

go test github.com/syndtr/goleveldb/leveldb
--- FAIL: TestCorruptDB_Journal (0.00 seconds)
panic: runtime error: invalid memory address or nil pointer dereference [recover
ed]
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x0 addr=0x0 pc=0x4beafc]

goroutine 4 [running]:
testing.func·004()
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/testi
ng/testing.go:348 +0x104
sync/atomic.AddUint64()
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/sync/
atomic/asm_386.s:69 +0xc
github.com/syndtr/goleveldb/leveldb.(_session).allocFileNum(0x12387510, 0x0, 0x0
)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/session_util.g
o:101 +0x4c
github.com/syndtr/goleveldb/leveldb.(_session).create(0x12387510, 0x0, 0x0)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/session.go:49
+0x38
github.com/syndtr/goleveldb/leveldb.Open(0x1237e510, 0x1237f380, 0x12382240, 0x0
, 0x1237d380, ...)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/db.go:85 +0xbb

github.com/syndtr/goleveldb/leveldb.(_dbHarness).openDB(0x12391e00)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/db_test.go:73
+0x4a
github.com/syndtr/goleveldb/leveldb.(_dbHarness).init(0x12391e00, 0x123821e0, 0x
12382240)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/db_test.go:68
+0x10b
github.com/syndtr/goleveldb/leveldb.newDbCorruptHarness(0x123821e0, 0x0)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/corrupt_test.g
o:32 +0xeb
github.com/syndtr/goleveldb/leveldb.TestCorruptDB_Journal(0x123821e0)
D:/apps/go/GoPath/src/github.com/syndtr/goleveldb/leveldb/corrupt_test.g
o:158 +0x28
testing.tRunner(0x123821e0, 0x665dcc)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/testi
ng/testing.go:353 +0x86
created by testing.RunTests
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/testi
ng/testing.go:433 +0x683

goroutine 1 [chan receive]:
testing.RunTests(0x588238, 0x665dc0, 0x3a, 0x3a, 0x1, ...)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/testi
ng/testing.go:434 +0x69e
testing.Main(0x588238, 0x665dc0, 0x3a, 0x3a, 0x664a60, ...)
C:/Users/ADMINI1/AppData/Local/Temp/2/bindist012168375/go/src/pkg/testi
ng/testing.go:365 +0x68
main.main()
C:/Users/chaiss/AppData/Local/Temp/go-build640285479/github.com/syndtr/g
oleveldb/leveldb/_test/_testmain.go:195 +0x80
FAIL github.com/syndtr/goleveldb/leveldb 0.055s

The documentation only mentions goroutine-safeness to say that an iterator is not. Is everything else (i.e. concurrent Put()s from separate routines are fine etc)? It would be nice if this was explicitly documented somewhere.

Got this error when compiling for solaris:

github.com/syndtr/goleveldb/leveldb/storage

../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:68: undefined: newFileLock
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:79: undefined: rename
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:274: undefined: rename
../../../../../github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:316: undefined: rename

Obtained by using "go build -race" on fedora 18

I am thinking this may be related to issue #26 & #27

==================
WARNING: DATA RACE
Read by goroutine 20:
  encoding/binary.littleEndian.Uint64()
      /home/marco/hg/go/src/pkg/encoding/binary/binary.go:70 +0xd6
  github.com/syndtr/goleveldb/leveldb.iKey.num()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/key.go:85 +0xd9
  github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:23 +0x1d7
  github.com/syndtr/goleveldb/leveldb/block.(*Iterator).Seek()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/block/reader.go:328 +0x3f7
  github.com/syndtr/goleveldb/leveldb/table.(*Reader).Get()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:137 +0x656
  github.com/syndtr/goleveldb/leveldb.(*tOps).get()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table.go:363 +0x2b1
  github.com/syndtr/goleveldb/leveldb.(*version).get()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/version.go:144 +0x860
  github.com/syndtr/goleveldb/leveldb.(*DB).get()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:354 +0x3de
  github.com/syndtr/goleveldb/leveldb.(*DB).get()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:364 +0x48
  github.com/conformal/btcdb/ldb.(*LevelDb).getBlkByHeight()
      /home/marco/git/go/src/github.com/conformal/btcdb/ldb/block.go:51 +0x19c
  github.com/conformal/btcdb/ldb.(*LevelDb).fetchTxDataBySha()
      /home/marco/git/go/src/github.com/conformal/btcdb/ldb/dbcache.go:78 +0x22a
  github.com/conformal/btcdb/ldb.(*LevelDb).FetchTxByShaList()
      /home/marco/git/go/src/github.com/conformal/btcdb/ldb/dbcache.go:47 +0x1a3
  github.com/conformal/btcchain.(*BlockChain).fetchTxList()
      /home/marco/git/go/src/github.com/conformal/btcchain/txlookup.go:124 +0x301
  github.com/conformal/btcchain.(*BlockChain).fetchInputTransactions()
      /home/marco/git/go/src/github.com/conformal/btcchain/txlookup.go:256 +0x6f0
  github.com/conformal/btcchain.(*BlockChain).checkConnectBlock()
      /home/marco/git/go/src/github.com/conformal/btcchain/validate.go:739 +0x1b7
  github.com/conformal/btcchain.(*BlockChain).connectBestChain()
      /home/marco/git/go/src/github.com/conformal/btcchain/chain.go:893 +0xd2b
  github.com/conformal/btcchain.(*BlockChain).maybeAcceptBlock()
      /home/marco/git/go/src/github.com/conformal/btcchain/accept.go:158 +0x11ca
  github.com/conformal/btcchain.(*BlockChain).ProcessBlock()
      /home/marco/git/go/src/github.com/conformal/btcchain/process.go:158 +0xcd5
  main.(*blockManager).handleBlockMsg()
      /home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:215 +0x180
  main.(*blockManager).blockHandler()
      /home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:255 +0xfa

Previous write by goroutine 11:
  runtime.appendslice()
      /home/marco/hg/go/src/pkg/runtime/slice.c:63 +0x0
  github.com/syndtr/goleveldb/leveldb.(*iComparer).Separator()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:42 +0x4b2
  github.com/syndtr/goleveldb/leveldb/table.(*Writer).Add()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table/writer.go:70 +0x1aa
  github.com/syndtr/goleveldb/leveldb.(*tWriter).add()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table.go:453 +0x110
  github.com/syndtr/goleveldb/leveldb.func·014()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:343 +0x91f
  github.com/syndtr/goleveldb/leveldb.(*DB).transact()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0x143
  github.com/syndtr/goleveldb/leveldb.(*DB).doCompaction()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0xfc0
  github.com/syndtr/goleveldb/leveldb.(*DB).compaction()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:459 +0x255

Goroutine 20 (running) created at:
  main.(*blockManager).Start()
      /home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:332 +0x170
  main.(*server).peerHandler()
      /home/marco/git/go/src/github.com/conformal/btcd/server.go:193 +0xad

Goroutine 11 (running) created at:
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:72 +0x366
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  fmt.(*ss).doScanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
  fmt.Fscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:307 +0x3e1
  fmt.(*ss).doScanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
  fmt.Fscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  fmt.(*ss).doScanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
  fmt.Fscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  github.com/syndtr/goleveldb/leveldb.(*session).getFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/session_util.go:44 +0x7f
  github.com/syndtr/goleveldb/leveldb.(*DB).cleanFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_util.go:53 +0x27e
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:69 +0x329
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  fmt.(*ss).doScanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
  fmt.Fscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:307 +0x3e1
  fmt.(*ss).doScanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
  fmt.Fscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  fmt.(*ss).doScanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
  fmt.Fscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
  fmt.Sscanf()
      /home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
  github.com/syndtr/goleveldb/leveldb/storage.(*file).parse()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
  github.com/syndtr/goleveldb/leveldb/storage.(*FileStorage).GetFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
  github.com/syndtr/goleveldb/leveldb.(*session).getFiles()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/session_util.go:44 +0x7f
  github.com/syndtr/goleveldb/leveldb.(*DB).recoverJournal()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:226 +0x2a7
  github.com/syndtr/goleveldb/leveldb.openDB()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:64 +0x2ee
  github.com/syndtr/goleveldb/leveldb.Open()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:103 +0x29a
  github.com/syndtr/goleveldb/leveldb.OpenFile()
      /home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:118 +0x146
  github.com/conformal/btcdb/ldb.openDB()
      /home/marco/git/go/src/github.com/conformal/btcdb/ldb/leveldb.go:160 +0x545
  github.com/conformal/btcdb/ldb.CreateDB()
      /home/marco/git/go/src/github.com/conformal/btcdb/ldb/leveldb.go:173 +0xf4
  github.com/conformal/btcdb.CreateDB()
      /home/marco/git/go/src/github.com/conformal/btcdb/db.go:194 +0x1c5
  main.loadBlockDB()
      /home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:391 +0x47c
  main.btcdMain()
      /home/marco/git/go/src/github.com/conformal/btcd/btcd.go:116 +0x22c
  main.main()
      /home/marco/git/go/src/github.com/conformal/btcd/btcd.go:183 +0x3d
==================

I'm interested in eventually using this library, but at the moment i'm only using the official c leveldb library.

As i'm contemplating this library, i thought it would be useful to point out the fork of leveldb by basho (https://github.com/basho/leveldb) for use in riak. Their fork follows the same principal of google's leveldb (it's based on the same code), but they are making some changes geared towards larger databases (like the ability to take backups of a live database) and other performance improvements.

I'm curious if you are curious were aware of their modifications, and if you are planning to implement some of their changes.

When writing a lot of entries the database get's pretty big fast (I'm doing a lot of appends).

From what I observed a meaningful compaction is only done when (re-) opening a database.

Is there a way to trigger the compaction without closing/opening the database?

CompactRange(leveldb.Range{}) does not seem to have the same effect.

This is another crash seen when using conformal's btcd. As with issue #26 this code path has run millions of times and suddenly blows up. Sorry about the lack of information but I will try to narrow this down if no one has any clues/ideas.

unexpected fault address 0xc20e830000
fatal error: fault
[signal 0xb code=0x1 addr=0xc20e830000 pc=0x588274]

goroutine 12 [running]:
[fp=0x200a7c6a8] runtime.throw(0xa4079c)
/usr/local/go/src/pkg/runtime/panic.c:473 +0x67
[fp=0x200a7c6c0] runtime.sigpanic()
/usr/local/go/src/pkg/runtime/os_openbsd.c:235 +0xe7
[fp=0x200a7c8b8] github.com/syndtr/goleveldb/leveldb.(_version).release_NB(0xc200841700)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/version.go:73 +0x314
[fp=0x200a7c8c8] github.com/syndtr/goleveldb/leveldb.(_session).setVersion(0xc20017f000, 0xc202016e00)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/session_util.go:69 +0x68
[fp=0x200a7c908] github.com/syndtr/goleveldb/leveldb.(_session).commit(0xc20017f000, 0xc2033f6000, 0x0, 0x0)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/session.go:166 +0xda
[fp=0x200a7c948] github.com/syndtr/goleveldb/leveldb.func·015(0x0, 0x0)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:381 +0x8e
[fp=0x200a7c9e8] github.com/syndtr/goleveldb/leveldb.(_DB).transact(0xc20016b600, 0x200a7ce18)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0xab
[fp=0x200a7ce68] github.com/syndtr/goleveldb/leveldb.(_DB).doCompaction(0xc20016b600, 0xc202016380, 0x200a7cf00)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:382 +0xd88
[fp=0x200a7cfb0] github.com/syndtr/goleveldb/leveldb.(_DB).compaction(0xc20016b600)
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:459 +0x181
[fp=0x200a7cfb8] runtime.goexit()
/usr/local/go/src/pkg/runtime/proc.c:1223
created by github.com/syndtr/goleveldb/leveldb.openDB
/axiom/go2/src/github.com/syndtr/goleveldb/leveldb/db.go:72 +0x1de

goroutine 1 [semacquire]:
sync.runtime_Semacquire(0xc20130ad88)
/usr/local/go/src/pkg/runtime/zsema_openbsd_amd64.c:165 +0x2e
sync.(_WaitGroup).Wait(0xc200126060)
/usr/local/go/src/pkg/sync/waitgroup.go:109 +0xf2
main.(_server).WaitForShutdown(0xc200126000)
/axiom/go2/src/github.com/conformal/btcd/server.go:361 +0x2a
main.btcdMain(0x0, 0x0)
/axiom/go2/src/github.com/conformal/btcd/btcd.go:174 +0x912
main.main()
/axiom/go2/src/github.com/conformal/btcd/btcd.go:183 +0x2e

Tip does not run on Windows. Commit 0e05891 causes the windows code to always error out with "The process can not access the file because it is being used by another process." when a db is opened.

Here's the stackdump:

panic: runtime error: slice bounds out of range [recovered]
panic: runtime error: slice bounds out of range

goroutine 50 [running]:
github.com/syndtr/goleveldb/leveldb.func·015()
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:392 +0x172
github.com/syndtr/goleveldb/leveldb.iKey.ukey(0xc2006180df, 0x0, 0xa7, 0x0, 0xa7, ...)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/key.go:73 +0x75
github.com/syndtr/goleveldb/leveldb.(_iComparer).Compare(0xc20016a660, 0xc2006180df, 0x0, 0xa7, 0xc201207720, ...)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/comparer.go:21 +0x85
github.com/syndtr/goleveldb/leveldb/iterator.(_MergedIterator).smallest(0xc200afca50)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:178 +0x158
github.com/syndtr/goleveldb/leveldb/iterator.(_MergedIterator).Next(0xc200afca50, 0xc20061802d)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:113 +0x185
github.com/syndtr/goleveldb/leveldb.func·013(0x0, 0x0)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:247 +0x1cc
github.com/syndtr/goleveldb/leveldb.(_DB).transact(0xc20012d180, 0x6c7a50)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0xaa
github.com/syndtr/goleveldb/leveldb.(_DB).doCompaction(0xc20012d180, 0xc200172540, 0x6c7f00)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0x7e0
github.com/syndtr/goleveldb/leveldb.(_DB).compaction(0xc20012d180)
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x1cb
created by github.com/syndtr/goleveldb/leveldb.open
/Users/sascha/Development/Go/projects/src/github.com/syndtr/goleveldb/leveldb/db.go:70 +0x1ec

Hello,

I saw you fixed #14 so I wanted to test your package and here what I got:

go version
go version devel +d29da2ced72b Mon May 13 10:28:12 2013 +0400 windows/386

Windows 7 x32

Error:

D:\Golibs\os\src\github.com\syndtr\goleveldb\leveldb>go test
fatal error: runtime: cannot map pages in arena address space

goroutine 328 [running]:
[fp=0x16c787c] runtime.throw(0x665341)
        c:/Go/src/pkg/runtime/panic.c:473 +0x65
[fp=0x16c789c] runtime.SysMap(0x3ffa0000, 0x100000)
        c:/Go/src/pkg/runtime/mem_windows.c:70 +0x78
[fp=0x16c78bc] runtime.MHeap_SysAlloc(0x1730000, 0x100000)
        c:/Go/src/pkg/runtime/zmalloc_windows_386.c:360 +0x103
[fp=0x16c78e4] MHeap_Grow(0x1730000, 0x10)
        c:/Go/src/pkg/runtime/mheap.c:224 +0x4f
[fp=0x16c7904] MHeap_AllocLocked(0x1730000, 0x4, 0x2b)
        c:/Go/src/pkg/runtime/mheap.c:108 +0x286
[fp=0x16c7918] runtime.MHeap_Alloc(0x1730000, 0x4, 0x2b, 0x0, 0x1, ...)
        c:/Go/src/pkg/runtime/mheap.c:77 +0x5c
[fp=0x16c7950] MCentral_Grow(0x1b35d4c)
        c:/Go/src/pkg/runtime/mcentral.c:203 +0x81
[fp=0x16c7974] runtime.MCentral_AllocList(0x1b35d4c, 0xc, 0x16c7988)
        c:/Go/src/pkg/runtime/mcentral.c:47 +0x4c
[fp=0x16c7994] runtime.MCache_Alloc(0x15d0000, 0x2b, 0x1400, 0x1)
        c:/Go/src/pkg/runtime/mcache.c:24 +0x72
[fp=0x16c79c4] runtime.mallocgc(0x1400, 0x1, 0x1, 0x1)
        c:/Go/src/pkg/runtime/zmalloc_windows_386.c:47 +0xdf
[fp=0x16c79e4] makeslice1(0x5021e0, 0x13e0, 0x13e0, 0x16c7a18)
        c:/Go/src/pkg/runtime/slice.c:61 +0x7f
[fp=0x16c7a04] runtime.makeslice(0x5021e0, 0x13e0, 0x0, 0x13e0, 0x0, ...)
        c:/Go/src/pkg/runtime/slice.c:34 +0x9c
[fp=0x16c7a58] github.com/syndtr/goleveldb/leveldb/table.(*bInfo).readAll(0x3f9feb80, 0x201818c0, 0x3f9b4a98, 0x20181800, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/block.go:66 +0x74
[fp=0x16c7a90] github.com/syndtr/goleveldb/leveldb/table.(*Reader).getBlock(0x3f9bd0f0, 0x3f9feb80, 0x20181440, 0x3f9b4988, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:171 +0x8e
[fp=0x16c7ac8] github.com/syndtr/goleveldb/leveldb/table.(*Reader).getDataIter(0x3f9bd0f0, 0x3f9feb80, 0x20181440, 0x3f9b4988, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:202 +0x174
[fp=0x16c7afc] github.com/syndtr/goleveldb/leveldb/table.(*indexIter).Get(0x3f9b19c0, 0x0, 0x0, 0x0, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/table/reader.go:234 +0xd0
[fp=0x16c7b14] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).setData(0x3f9b19e0, 0x1)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:138
+0x2d
[fp=0x16c7b20] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).Next(0x3f9b19e0, 0x3ff9d800)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:85 +0x7f
[fp=0x16c7b2c] github.com/syndtr/goleveldb/leveldb/iterator.(*IndexedIterator).Next(0x3f9b17e0, 0x4ba7c0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/indexed_iter.go:84 +0x45
[fp=0x16c7b70] github.com/syndtr/goleveldb/leveldb/iterator.(*MergedIterator).Next(0x3ea83ea0, 0x3fdb03a0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/iterator/merged_iter.go:112 +0x124
[fp=0x16c7c1c] github.com/syndtr/goleveldb/leveldb.func┬Ě015(0x0, 0x0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:247 +0x172
[fp=0x16c7c6c] github.com/syndtr/goleveldb/leveldb.(*DB).transact(0x200bd500, 0x16c7cac)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0x74
[fp=0x16c7f2c] github.com/syndtr/goleveldb/leveldb.(*DB).doCompaction(0x200bd500, 0x3e997480, 0x16c7f00)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0x75d
[fp=0x16c7fd4] github.com/syndtr/goleveldb/leveldb.(*DB).compaction(0x200bd500)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x17b
[fp=0x16c7fd8] runtime.goexit()
        c:/Go/src/pkg/runtime/proc.c:1223
created by github.com/syndtr/goleveldb/leveldb.openDB
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db.go:72 +0x1e6

goroutine 1 [chan receive]:
testing.RunTests(0x58aa7c, 0x666dc0, 0x3c, 0x3c, 0x1, ...)
        c:/Go/src/pkg/testing/testing.go:434 +0x69e
testing.Main(0x58aa7c, 0x666dc0, 0x3c, 0x3c, 0x665a60, ...)
        c:/Go/src/pkg/testing/testing.go:365 +0x68
main.main()
        C:/Users/Dobek/AppData/Local/Temp/go-build124087311/github.com/syndtr/goleveldb/leveldb/_test/_testmain.go:199 +0x80

goroutine 327 [chan send]:
github.com/syndtr/goleveldb/leveldb.(*DB).flush(0x200bd500, 0x0, 0x0, 0x0)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:62 +0x176
github.com/syndtr/goleveldb/leveldb.(*DB).Write(0x200bd500, 0x3ea85f20, 0x204cd010, 0x0, 0x0, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:109 +0x181
github.com/syndtr/goleveldb/leveldb.(*DB).Put(0x200bd500, 0x3ea6ecb0, 0xb, 0xb, 0x3eaa6400, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:162 +0x84
github.com/syndtr/goleveldb/leveldb.(*dbHarness).put(0x204f5480, 0x3ea6eca0, 0xb, 0x21ba8800, 0x3e8, ...)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_test.go:122 +0xb5
github.com/syndtr/goleveldb/leveldb.TestDb_SparseMerge(0x2012b540)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_test.go:928 +0x165
testing.tRunner(0x2012b540, 0x666f4c)
        c:/Go/src/pkg/testing/testing.go:353 +0x86
created by testing.RunTests
        c:/Go/src/pkg/testing/testing.go:433 +0x683

goroutine 329 [chan receive]:
github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0x200bd500)
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db_write.go:25 +0x3b
created by github.com/syndtr/goleveldb/leveldb.openDB
        D:/Golibs/os/src/github.com/syndtr/goleveldb/leveldb/db.go:73 +0x1fb

goroutine 7 [runnable]:
exit status 2
FAIL    github.com/syndtr/goleveldb/leveldb     15.997s

Best regards,
Dobrosław Żybort

It would be really great if there would be an (atomic) Append() method.

I know that I can work around this using Get() and Put() (and additional synchronization), but an atomic Append() - maybe even with a WriteOption CreateIfMissing true/false - would be very, very helpful.

When I run the following

package level

import (
        "github.com/syndtr/goleveldb/leveldb"
        "github.com/syndtr/goleveldb/leveldb/opt"
        "math/rand"
        "testing"
        "time"
        "unsafe"
)

func init() {
        rand.Seed(time.Now().UnixNano())
}

func randBytes(b []byte) {
        for i := 0; i < len(b)/8; i++ {
                *((*int64)(unsafe.Pointer(&b[i*8]))) = rand.Int63()
        }
}

func BenchmarkLeveldb(b *testing.B) {
        b.StopTimer()
        ro := &opt.ReadOptions{}
        wo := &opt.WriteOptions{}
        db, err := leveldb.OpenFile("paty", &opt.Options{Flag: opt.OFCreateIfMissing})
        if err != nil {
                panic(err)
        }
        key := make([]byte, 32)
        b.StartTimer()
        for i := 0; i < b.N; i++ {
                randBytes(key)
                db.Put(key, key, wo)
                db.Get(key, ro)
        }
}

I get

testing: warning: no tests to run
PASS
BenchmarkLeveldb    panic: resource temporarily unavailable

goroutine 3 [running]:
_/home/zond/tmp/leveltest.BenchmarkLeveldb(0xc2000900f0)
    /home/zond/tmp/leveltest/level_test.go:28 +0xfa
testing.(*B).runN(0xc2000900f0, 0x64)
    /usr/local/go/src/pkg/testing/benchmark.go:119 +0x86
testing.(*B).launch(0xc2000900f0)
    /usr/local/go/src/pkg/testing/benchmark.go:204 +0x14c
created by testing.(*B).run
    /usr/local/go/src/pkg/testing/benchmark.go:167 +0x2c

goroutine 1 [runnable]:
testing.(*B).run(0xc2000900f0, 0x3, 0x7f4d0d3d2c08, 0x1, 0x1, ...)
    /usr/local/go/src/pkg/testing/benchmark.go:168 +0x49
testing.RunBenchmarks(0x5b95a8, 0x671f80, 0x1, 0x1)
    /usr/local/go/src/pkg/testing/benchmark.go:300 +0x505
testing.Main(0x5b95a8, 0x679a60, 0x0, 0x0, 0x671f80, ...)
    /usr/local/go/src/pkg/testing/testing.go:373 +0x1af
main.main()
    _/home/zond/tmp/leveltest/_test/_testmain.go:43 +0x9a

goroutine 4 [select]:
github.com/syndtr/goleveldb/leveldb.(*DB).compaction(0xc2000ad000)
    /home/zond/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:411 +0x63f
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/zond/go/src/github.com/syndtr/goleveldb/leveldb/db.go:73 +0x1e9

goroutine 5 [chan receive]:
github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0xc2000ad000)
    /home/zond/go/src/github.com/syndtr/goleveldb/leveldb/db_write.go:25 +0x3c
created by github.com/syndtr/goleveldb/leveldb.openDB
    /home/zond/go/src/github.com/syndtr/goleveldb/leveldb/db.go:74 +0x200

goroutine 6 [runnable]:
exit status 2
FAIL    _/home/zond/tmp/leveltest   0.070s

What am I doing wrong? :O

I'm on Ubuntu 12.04 64-bit and when I try to build, I get this error message
./db_compaction.go:250: closure needs too many variables; runtime will reject it
It has to do with nested functions based on what I found here: https://groups.google.com/forum/#!topic/golang-nuts/tQovQj1j7wo

It looks like there are a few in the call to d.transact() at the affected line, so I'm not sure how to tackle it myself.

I have written a program which reproduces this issue here: https://gist.github.com/davecgh/8376120

This appears related to the size rather than the actual number of items. The test program in the linked gist, for example, fails exactly when it hits 122682 items as it is written (notice from running the program how 122681 items works as expected), but if you change the key value to contain one more _, the number of items before it fails drops to 119366 items since the overall size increased.

This is a significant issue for us in github.com/conformal/btcd since we have to delete large amounts of transaction data on chain switches and this bug ends up returning the deleted data offsets instead of the fact the items are now deleted.

I have a package that is using goleveldb. It seems to have received some bad data that ultimately caused a nil key to get set. Now when the compactor runs, I see the following:

panic: nil iKey [recovered] panic: nil iKey [recovered] panic: nil iKey

On the one hand, I should have curated my data more carefully to prevent nil data from becoming part of it. On the other hand, I think either this should have been prevented programmatically within the leveldb library, and/or leveldb should not be panicking (which leads to the situation where a library brings down my entire app), but instead returning an error (that I can choose to handle in a variety of ways).

The Open and Recover calls seem to accomplish the same thing (opening a database) with separate limitations; the Open calls can create a database but not open a corrupt one, while Recover can open a corrupt database (for specific values of corrupt) but not create one. Assuming one always wants to make a best effort to open a database, is best practice to use Open and retry with Recover if Open failed? Or start with Recover and use Open if the db did not exist?

Also,

The DB must be closed after use, by calling Close method.

Or else? What is the risk/downside of exiting abruptly without close - corruption, or just a non-compact database or similar? Will that be fixed on open, or only on recover?

D:>go version
go version devel +d58997478ec6 Mon Apr 08 00:09:35 2013 -0700 windows/386

D:>go test github.com/syndtr/goleveldb/leveldb/storage
--- FAIL: TestFileStorage_Flock (0.00 seconds)
file_storage_test.go:112: OpenFile(2): expect error
FAIL
FAIL github.com/syndtr/goleveldb/leveldb/storage 0.046s

Hi, I was getting an error running tests on your package and traced it to a call to Grow() for a *bytes.Buffer in randomString() in bench_test.go. It was introduced into your system on this push (4ab1ba6).

In my Go 1.0.3 install and on the Go source code online, it shows Grow() as an unexported method grow(), which therefore fails on my system. Are you using an older form of Go or is there something wrong with my system? Thanks for making this package!

-Bill

When I cd into the leveldb directory and type "go build", I get this error:

./session_util.go:28: method d.file.Num is not an expression, must be called

It looks like Num at line 28 of session_util.go just needs the parenthesis added. I can submit a pull request with the change if it saves you time.

I'm having a trouble with leveldb on windows. Every time I create a database for the first time it panics. Subsequent launches do work.

Code that will show the problem.

package main

import (
    "fmt"
    "github.com/syndtr/goleveldb/leveldb"
)
func main() {
    db, err := leveldb.OpenFile("db", nil)
    if err != nil {
        fmt.Println("Error:", err)
    }
    err = db.Put([]byte("a"), []byte("b"), nil)
    if err != nil {
        fmt.Println("Error put", err)
    }
}

Error

ERROR fsync: The handle is invalid.
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x1 addr=0x0 pc=0x43c4f0]

goroutine 1 [running]:
runtime.panic(0x4ef2c0, 0x67b08f)
        C:/Users/ADMINI~1/AppData/Local/Temp/2/makerelease106211947/go/src/pkg/r
untime/panic.c:266 +0xa6
github.com/syndtr/goleveldb/leveldb.(*DB).isClosed(0x0, 0x0)
        c:/GoDev/src/github.com/syndtr/goleveldb/leveldb/db_state.go:105 +0x50
github.com/syndtr/goleveldb/leveldb.(*DB).ok(0x0, 0x12390608, 0x19)
        c:/GoDev/src/github.com/syndtr/goleveldb/leveldb/db_state.go:110 +0x2a
github.com/syndtr/goleveldb/leveldb.(*DB).Write(0x0, 0x123acd40, 0x0, 0x1, 0x1c2
73b)
        c:/GoDev/src/github.com/syndtr/goleveldb/leveldb/db_write.go:113 +0x5a
github.com/syndtr/goleveldb/leveldb.(*DB).Put(0x0, 0x1c273a, 0x1, 0x1, 0x1c273b,
 ...)
        c:/GoDev/src/github.com/syndtr/goleveldb/leveldb/db_write.go:221 +0x86
main.main()
        C:/GoDev/src/test/test.go:12 +0x1b0

Perhaps related some of the tests seem to be failing. The tests result in an other panic.

I have looked at the code but I couldn't figure out where it was going wrong. I hope you might have an idea how to fix this.

Since the trace is so long i put it here:
https://gist.github.com/marcopeereboom/6994597

this does not happen on linux and bitrig.

Since the documentation does not mention anything about this, is it possible to use different filters on leveldb.Open(...)? The reason I ask is because the bloom filter might need to be enlarged as your database is growing.

Please find an example code below.
Each time I call the leveldb.Open, it returns an error saying:

rename mytest/CURRENT.0 mytest/CURRENT: The process cannot access the file because it is being used by another process.

I tried both 64 and 32 bit go 1.0.3 - same results.
Also does not matter if "mytest" folder was already there, or not.

package main

import (
"github.com/syndtr/goleveldb/leveldb"
"github.com/syndtr/goleveldb/leveldb/opt"
"github.com/syndtr/goleveldb/leveldb/storage"
)

func main() {
stor, err := storage.OpenFile("mytest")
if err != nil {
panic(err.Error())
}
println("storage open", stor)
db, err := leveldb.Open(stor, &opt.Options{Flag: opt.OFCreateIfMissing})
if err != nil {
panic(err.Error())
}
println("db open", db) // this line is never reached on windows
}

btcd (github.com/conformal/btcd) uses goleveldb to store data.

The database is opened with:

        myCache := cache.NewEmptyCache()
        opts := &opt.Options{Flag: flag,
                BlockCache:      myCache,
                MaxOpenFiles:    256,
                CompressionType: opt.NoCompression,
        }

After a while, the database throws an error about a missing data.

I added two fmt.Printfs to reader.go in readRawBlock()'s select case to print the type of Compression being used. I also added spew.Dump(bh) We have a tool to dump data.

(table.blockHandle) {
 offset: (uint64) 1937402,
 length: (uint64) 4827
}
(uint8) 0
No Compression

On a different block:

(table.blockHandle) {
 offset: (uint64) 0,
 length: (uint64) 22302
}
(uint8) 1
Snappy Compression
1381438195854721313 [Warn] unable to locate block sha 00000000000033640c6b34c40b8b95ba4fae759ec7970e546660a3abd9f9314c err snappy: unsupported COPY_4 tag
1381438195854866024 [Info] Invalid block 00000000000033640c6b34c40b8b95ba4fae759ec7970e546660a3abd9f9314c


And another:
(table.blockHandle) {
 offset: (uint64) 1237318,
 length: (uint64) 3572
}
(uint8) 1
Snappy Compression

Yet another:
(table.blockHandle) {
 offset: (uint64) 1119730,
 length: (uint64) 4331
}
(uint8) 0
No Compression

Any idea what is causing the compression to be changed?  Multiple people are seeing this on different machines.  I am the only one getting the COPY_4 tag issue.

Steps to reproduce:

1. Build dvid. If you don't want to use the CMake route, which builds golang from tarball, you can just do:

% go build -a -tags 'purego' dvid.go

2. Download the dvid-test-data repo and follow the instructions in the README. That leads to the error below.

2013/02/26 10:58:27 Processing Block /demo/dvid-test-data/grayscale8/iso.2622.png via blockHandler 0: (13,13,1)
panic: runtime error: slice bounds out of range [recovered]
    panic: runtime error: slice bounds out of range

goroutine 4 [running]:
github.com/syndtr/goleveldb/leveldb._func_014(0x2671fb0, 0x2671100, 0x2671fb8, 0x2671c70)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:392 +0x1ac
----- stack segment boundary -----
github.com/syndtr/goleveldb/leveldb.iKey.ukey(0x4ab868, 0x0, 0xf8400e42b0, 0xa00000002, 0x1, ...)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/key.go:73 +0x75
github.com/syndtr/goleveldb/leveldb.(*iComparer).Compare(0xf8400a8e50, 0xf8400e42b0, 0xa0000000a, 0x4ab868, 0x0, ...)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:21 +0xb2
github.com/syndtr/goleveldb/leveldb.(*session).pickCompaction(0xf8400d9000, 0x0, 0x2671fa0, 0x1)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/session.go:173 +0x4c5
github.com/syndtr/goleveldb/leveldb.(*DB).compaction(0xf8400b4180, 0x0)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:463 +0x157
created by github.com/syndtr/goleveldb/leveldb.open
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db.go:70 +0x1f3

goroutine 1 [chan receive]:
net.(*pollServer).WaitRead(0xf8400fa8c0, 0xf840111000, 0xf8400c48a0, 0x23, 0x1, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:268 +0x73
net.(*netFD).accept(0xf840111000, 0xd19ce, 0x0, 0xf840090390, 0xf8400a9040, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:622 +0x20d
net.(*TCPListener).AcceptTCP(0xf8400f7730, 0xf8400f8a00, 0x0, 0x0, 0x8, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/tcpsock_posix.go:320 +0x71
net.(*TCPListener).Accept(0xf8400f7730, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/tcpsock_posix.go:330 +0x49
net/http.(*Server).Serve(0xf840114040, 0xf840114000, 0xf8400f7730, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:1029 +0x88
net/http.Serve(0xf840114000, 0xf8400f7730, 0x0, 0x0, 0xf840114000, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:994 +0x71
github.com/janelia-flyem/dvid/server.(*Service).ServeRpc(0x4a7958, 0x30081c, 0xe, 0x1bd888, 0xf8400f84c0, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/server/server.go:109 +0x1a8
github.com/janelia-flyem/dvid/server.Serve(0x7fff5fbff857, 0x7fff00000008, 0x0, 0x0, 0x0, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/server/server.go:63 +0x401
main.DoServe(0xf8400a8ce0, 0x5, 0x2f0cec, 0x7672657300000005)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/dvid.go:108 +0xcd
main.DoCommand(0xf8400a8ce0, 0xf8400a8ce0, 0x27b9e0, 0x985b)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/dvid.go:80 +0x159
main.main()
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/dvid.go:62 +0x184

goroutine 2 [syscall]:
created by runtime.main
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/runtime/proc.c:221

goroutine 3 [chan send]:
github.com/syndtr/goleveldb/leveldb.(*DB).flush(0xf8400b4180, 0x0, 0x0, 0x0, 0x10bdff, ...)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db_write.go:62 +0x1b5
github.com/syndtr/goleveldb/leveldb.(*DB).Write(0xf8400b4180, 0xf84013a030, 0xf8404ccc68, 0x0, 0x0, ...)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db_write.go:109 +0x1d3
github.com/syndtr/goleveldb/leveldb.(*DB).Put(0xf8400b4180, 0xf840b7b7a0, 0x2000000011, 0xf841535000, 0x100000001000, ...)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db_write.go:162 +0x94
github.com/janelia-flyem/dvid/keyvalue.(*goLDB).Put(0xf8400d2580, 0xf840b7b7a0, 0x2000000011, 0xf841535000, 0x100000001000, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/keyvalue/leveldbgo.go:236 +0xb4
github.com/janelia-flyem/dvid/datastore.kvdb.putBytes(0xf840092d70, 0xf8400d2580, 0xf840b7b7a0, 0x2000000011, 0xf841535000, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datastore/keyvalue.go:97 +0xe5
github.com/janelia-flyem/dvid/datastore.blockHandler(0x0, 0x0)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datastore/cache.go:147 +0x87f
created by github.com/janelia-flyem/dvid/datastore.init·1
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datastore/cache.go:60 +0xe4

goroutine 5 [chan receive]:
github.com/syndtr/goleveldb/leveldb.(*DB).writeJournal(0xf8400b4180, 0x0)
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db_write.go:25 +0x3c
created by github.com/syndtr/goleveldb/leveldb.open
    /Users/katzw/work/janelia/go/src/github.com/syndtr/goleveldb/leveldb/db.go:71 +0x20a

goroutine 6 [chan receive]:
net.(*pollServer).WaitRead(0xf8400fa8c0, 0xf8400c07e0, 0xf8400c48a0, 0x23, 0x1, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:268 +0x73
net.(*netFD).accept(0xf8400c07e0, 0xd19ce, 0x0, 0xf840090390, 0xf8400a9040, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:622 +0x20d
net.(*TCPListener).AcceptTCP(0xf8400f75e0, 0x2673e58, 0x0, 0x0, 0x18, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/tcpsock_posix.go:320 +0x71
net.(*TCPListener).Accept(0xf8400f75e0, 0x0, 0x0, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/tcpsock_posix.go:330 +0x49
net/http.(*Server).Serve(0xf8400fa200, 0xf840114000, 0xf8400f75e0, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:1029 +0x88
net/http.(*Server).ListenAndServe(0xf8400fa200, 0x6970612f00000005, 0x39411, 0x100000001)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:1019 +0xb6
github.com/janelia-flyem/dvid/server.(*Service).ServeHttp(0x4a7958, 0x3007fc, 0xe, 0x0)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/server/server.go:90 +0x15a
created by github.com/janelia-flyem/dvid/server.Serve
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/server/server.go:60 +0x3d1

goroutine 7 [syscall]:
syscall.Syscall6()
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/syscall/asm_darwin_amd64.s:38 +0x5
syscall.kevent(0xe, 0x0, 0x0, 0xf8400b4308, 0xa, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/syscall/zsyscall_darwin_amd64.go:199 +0x88
syscall.Kevent(0xf80000000e, 0x0, 0x0, 0xf8400b4308, 0xa0000000a, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/syscall/syscall_bsd.go:546 +0xa4
net.(*pollster).WaitFD(0xf8400b4300, 0xf8400fa8c0, 0x0, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd_darwin.go:96 +0x185
net.(*pollServer).Run(0xf8400fa8c0, 0x0)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:236 +0xe4
created by net.newPollServer
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/newpollserver.go:35 +0x382

goroutine 8 [chan receive]:
net.(*pollServer).WaitRead(0xf8400fa8c0, 0xf8400c0870, 0xf8400c48a0, 0x23, 0x1, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:268 +0x73
net.(*netFD).Read(0xf8400c0870, 0xf84011b000, 0x100000001000, 0xffffffff, 0xf840090390, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/fd.go:428 +0x1ec
net.(*TCPConn).Read(0xf8400f7838, 0xf84011b000, 0x100000001000, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/tcpsock_posix.go:87 +0xce
bufio.(*Reader).fill(0xf840114200, 0xf84011a018)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/bufio/bufio.go:77 +0xf0
bufio.(*Reader).Read(0xf840114200, 0xf8400f8830, 0x900000001, 0xf800000001, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/bufio/bufio.go:142 +0x188
encoding/gob.decodeUintReader(0xf8400fccf0, 0xf840114200, 0xf8400f8830, 0x900000009, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/encoding/gob/decode.go:65 +0x87
encoding/gob.(*Decoder).recvMessage(0xf84011a000, 0x12289, 0xf8400fcef8, 0x26758a0)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/encoding/gob/decoder.go:73 +0x43
encoding/gob.(*Decoder).decodeTypeSequence(0xf84011a000, 0xf84011ee00, 0x160, 0x0, 0x2d4ca8, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/encoding/gob/decoder.go:142 +0x57
encoding/gob.(*Decoder).DecodeValue(0xf84011a000, 0x1b3f70, 0xf84011eee0, 0x160, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/encoding/gob/decoder.go:206 +0x124
encoding/gob.(*Decoder).Decode(0xf84011a000, 0x1b3f60, 0xf84011eee0, 0xf84011eee0, 0x4011a000, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/encoding/gob/decoder.go:185 +0x20a
net/rpc.(*gobServerCodec).ReadRequestHeader(0xf8400fcea0, 0xf84011eee0, 0x293cf0, 0x293cf0, 0x100000000, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:371 +0x4e
net/rpc.(*Server).readRequestHeader(0xf84008a600, 0xf840114240, 0xf8400fcea0, 0x0, 0x0, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:522 +0x94
net/rpc.(*Server).readRequest(0xf84008a600, 0xf840114240, 0xf8400fcea0, 0xf840106000, 0xf8400fb980, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:489 +0x90
net/rpc.(*Server).ServeCodec(0xf84008a600, 0xf840114240, 0xf8400fcea0, 0xf840114240)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:408 +0x69
net/rpc.(*Server).ServeConn(0xf84008a600, 0xf840114180, 0xf8400f7838, 0xf840114180)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:400 +0x146
net/rpc.(*Server).ServeHTTP(0xf84008a600, 0xf840114140, 0xf840114100, 0xf840118000, 0xf800000000, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:636 +0x377
net/http.(*ServeMux).ServeHTTP(0xf84008de00, 0xf840114140, 0xf840114100, 0xf840118000, 0xf840114100, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:941 +0xb9
net/http.(*conn).serve(0xf840112d20, 0x0)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:669 +0x621
created by net/http.(*Server).Serve
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/http/server.go:1057 +0x430

goroutine 9 [chan send]:
github.com/janelia-flyem/dvid/datastore.(*VersionService).ProcessBlock(0xf8400d2880, 0xf8403e2cc0, 0xf84013b630, 0xc0000000c, 0xf800000100, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datastore/cache.go:75 +0x1a5
github.com/janelia-flyem/dvid/datatype/grayscale8.(*Datatype).ProcessBlocks(0xf84008c6c0, 0xf8400d2880, 0xf8408b2a01, 0xf84145df00, 0x0, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datatype/grayscale8/grayscale8.go:206 +0x5f2
github.com/janelia-flyem/dvid/datatype/grayscale8.(*Datatype).ServerAdd(0xf84008c6c0, 0xf8400d2880, 0xf840107fa0, 0xf840107fb0, 0xf840114a40, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datatype/grayscale8/grayscale8.go:150 +0x9cf
github.com/janelia-flyem/dvid/datatype/grayscale8.(*Datatype).Do(0xf84008c6c0, 0xf8400d2880, 0xf840107fa0, 0xf840107fb0, 0xf840114a40, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/datatype/grayscale8/grayscale8.go:73 +0x15c
github.com/janelia-flyem/dvid/server.(*Command).datatypeDo(0xf840107fa0, 0xf840114a40, 0x2ed254, 0x706c656800000004, 0xffffffff, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/server/command.go:120 +0x716
github.com/janelia-flyem/dvid/server.(*RpcConnection).Do(0xf8400f7600, 0xf840107fa0, 0xf840114a40, 0x0, 0x0, ...)
    /Users/katzw/work/janelia/go/src/github.com/janelia-flyem/dvid/server/command.go:63 +0x3a8
----- stack segment boundary -----
reflect.Value.call(0x24d708, 0x377c6, 0x130, 0x2e3444, 0x6c6c614300000004, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/reflect/value.go:521 +0x135e
reflect.Value.Call(0x24d708, 0x377c6, 0x130, 0x278eeb8, 0x300000003, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/reflect/value.go:334 +0x85
net/rpc.(*service).call(0xf8400fa800, 0xf84008a600, 0xf8400f7740, 0xf840106000, 0xf8400fb980, ...)
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:352 +0x133
created by net/rpc.(*Server).ServeCodec
    /Users/katzw/buildem/src/golang-1.0.3/src/pkg/net/rpc/server.go:423 +0x39e

de350e4 appears to have broken goleveldb on go1.2 due to a send of a chan struct{} on a chan chan<- struct{}. This is clearly a bug in go1.2, but can be mitigated. Please see this PR for details.

Is there a way to disable the auto db compaction goleveldb does? My db keeps being corrupted, and if it does panic, it is always in db_compaction.go

I want to test filling my DB without goleveldb running compaction at all to see if it still corrupts.

Hi!

I use goleveldb in syncthing (https://github.com/syncthing/syncthing) and have been doing a fair bit of profiling and hacking lately to reduce syncthing's memory footprint. I'm now at the point where goleveldb represents ~70-80 MB of a 100 MB footprint in my benchmarks. To a certain extent this is fair, with caches etc. But a large part of it is temporary heap allocations, in fact pretty much all of them made by snappy.Decode() from readRawBlock(). I made a naive attempt to reuse buffers with a sync.Pool in https://github.com/calmh/goleveldb/tree/reduce-heap, and it makes a dent but it's not huge. Probably the reason it's not very effective is that the pool isn't filled very often, and I'm not sure about the code paths that result in the data buffers becoming unused to hook into.

This isn't a bug report as such, more a humble request to give it some work at some point, or pass me some hints and I'm happy to root around in it. :)

WARNING: DATA RACE
Write by goroutine 11:
runtime.appendslice()
/home/marco/hg/go/src/pkg/runtime/slice.c:63 +0x0
github.com/syndtr/goleveldb/leveldb.(_iComparer).Separator()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:42 +0x4b2
github.com/syndtr/goleveldb/leveldb/table.(_Writer).Add()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table/writer.go:70 +0x1aa
github.com/syndtr/goleveldb/leveldb.(_tWriter).add()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table.go:453 +0x110
github.com/syndtr/goleveldb/leveldb.(_tOps).createFrom()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/table.go:329 +0x287
github.com/syndtr/goleveldb/leveldb.(_cMem).flush()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:87 +0x161
github.com/syndtr/goleveldb/leveldb.func·009()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:170 +0x10a
github.com/syndtr/goleveldb/leveldb.(_DB).transact()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0x143
github.com/syndtr/goleveldb/leveldb.(_DB).memCompaction()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:171 +0x33c
github.com/syndtr/goleveldb/leveldb.func·014()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:256 +0x3f7
github.com/syndtr/goleveldb/leveldb.(_DB).transact()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:136 +0x143
github.com/syndtr/goleveldb/leveldb.(_DB).doCompaction()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:365 +0xfc0
github.com/syndtr/goleveldb/leveldb.(_DB).compaction()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_compaction.go:459 +0x255

Previous read by goroutine 20:
encoding/binary.littleEndian.Uint64()
/home/marco/hg/go/src/pkg/encoding/binary/binary.go:69 +0x41
github.com/syndtr/goleveldb/leveldb.iKey.num()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/key.go:85 +0xd9
github.com/syndtr/goleveldb/leveldb.(_iComparer).Compare()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/comparer.go:23 +0x1d7
github.com/syndtr/goleveldb/leveldb/memdb.(_DB).findGE()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/memdb/memdb.go:168 +0x152
github.com/syndtr/goleveldb/leveldb/memdb.(_DB).Find()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/memdb/memdb.go:137 +0x113
github.com/syndtr/goleveldb/leveldb.func·007()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:330 +0xa0
github.com/syndtr/goleveldb/leveldb.(_DB).get()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:349 +0x36e
github.com/syndtr/goleveldb/leveldb.(_DB).get()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:364 +0x48
github.com/conformal/btcdb/ldb.(_LevelDb).getBlkByHeight()
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/block.go:51 +0x19c
github.com/conformal/btcdb/ldb.(_LevelDb).fetchTxDataBySha()
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/dbcache.go:78 +0x22a
github.com/conformal/btcdb/ldb.(_LevelDb).FetchTxByShaList()
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/dbcache.go:47 +0x1a3
github.com/conformal/btcchain.(_BlockChain).fetchTxList()
/home/marco/git/go/src/github.com/conformal/btcchain/txlookup.go:124 +0x301
github.com/conformal/btcchain.(_BlockChain).fetchInputTransactions()
/home/marco/git/go/src/github.com/conformal/btcchain/txlookup.go:256 +0x6f0
github.com/conformal/btcchain.(_BlockChain).checkConnectBlock()
/home/marco/git/go/src/github.com/conformal/btcchain/validate.go:739 +0x1b7
github.com/conformal/btcchain.(_BlockChain).connectBestChain()
/home/marco/git/go/src/github.com/conformal/btcchain/chain.go:893 +0xd2b
github.com/conformal/btcchain.(_BlockChain).maybeAcceptBlock()
/home/marco/git/go/src/github.com/conformal/btcchain/accept.go:158 +0x11ca
github.com/conformal/btcchain.(_BlockChain).ProcessBlock()
/home/marco/git/go/src/github.com/conformal/btcchain/process.go:158 +0xcd5
main.(_blockManager).handleBlockMsg()
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:215 +0x180
main.(_blockManager).blockHandler()
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:255 +0xfa

Goroutine 11 (running) created at:
github.com/syndtr/goleveldb/leveldb.openDB()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:72 +0x366
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
fmt.(_ss).doScanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
fmt.Fscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:307 +0x3e1
fmt.(_ss).doScanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
fmt.Fscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
fmt.(_ss).doScanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
fmt.Fscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
github.com/syndtr/goleveldb/leveldb.(_session).getFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/session_util.go:44 +0x7f
github.com/syndtr/goleveldb/leveldb.(_DB).cleanFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db_util.go:53 +0x27e
github.com/syndtr/goleveldb/leveldb.openDB()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:69 +0x329
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
fmt.(_ss).doScanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
fmt.Fscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:307 +0x3e1
fmt.(_ss).doScanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
fmt.Fscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
fmt.(_ss).doScanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:1158 +0x4ce
fmt.Fscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:144 +0xf7
fmt.Sscanf()
/home/marco/hg/go/src/pkg/fmt/scan.go:116 +0xe3
github.com/syndtr/goleveldb/leveldb/storage.(_file).parse()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:294 +0x179
github.com/syndtr/goleveldb/leveldb/storage.(_FileStorage).GetFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/storage/file_storage.go:168 +0x282
github.com/syndtr/goleveldb/leveldb.(_session).getFiles()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/session_util.go:44 +0x7f
github.com/syndtr/goleveldb/leveldb.(_DB).recoverJournal()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:226 +0x2a7
github.com/syndtr/goleveldb/leveldb.openDB()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:64 +0x2ee
github.com/syndtr/goleveldb/leveldb.Open()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:103 +0x29a
github.com/syndtr/goleveldb/leveldb.OpenFile()
/home/marco/git/go/src/github.com/syndtr/goleveldb/leveldb/db.go:118 +0x146
github.com/conformal/btcdb/ldb.openDB()
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/leveldb.go:160 +0x545
github.com/conformal/btcdb/ldb.CreateDB()
/home/marco/git/go/src/github.com/conformal/btcdb/ldb/leveldb.go:173 +0xf4
github.com/conformal/btcdb.CreateDB()
/home/marco/git/go/src/github.com/conformal/btcdb/db.go:194 +0x1c5
main.loadBlockDB()
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:391 +0x47c
main.btcdMain()
/home/marco/git/go/src/github.com/conformal/btcd/btcd.go:116 +0x22c
main.main()
/home/marco/git/go/src/github.com/conformal/btcd/btcd.go:183 +0x3d

Goroutine 20 (running) created at:
main.(_blockManager).Start()
/home/marco/git/go/src/github.com/conformal/btcd/blockmanager.go:332 +0x170
main.(_server).peerHandler()

/home/marco/git/go/src/github.com/conformal/btcd/server.go:193 +0xad

go version
go version devel +d58997478ec6 Mon Apr 08 00:09:35 2013 -0700 windows/386

go get github.com/syndtr/goleveldb/leveldb

github.com/syndtr/goleveldb/leveldb/storage

storage\file_storage_unix.go:15: undefined: syscall.LOCK_UN
storage\file_storage_unix.go:17: undefined: syscall.LOCK_EX
storage\file_storage_unix.go:19: undefined: syscall.Flock
storage\file_storage_unix.go:19: undefined: syscall.LOCK_NB
storage\file_storage_windows.go:11: setFileLock redeclared in this block
previous declaration at storage\file_storage_unix.go:14

This issue came out of #8 (comment).

Background: The filter can be changed when reopening a database. If the filter defined at database open has a different name than the filter used when persisting an earlier SSTable, no filter will be used.

Issue: For big data and/or realtime queries, this can have a big impact on performance. Obviously, this will only be an issue until the data for all SSTables has been rewritten/merged to new SSTables. How long time the migration takes, depends on the number of writes made to the database.

Proposed solution: A possible way to work around the above issue would be to introduce opt.Options.oldFilters (or similar) which is a list of old filter that's been used. On database creation, create a lookup map from filter name to filter.Filter instance (also including opt.Options.Filter). So, for new sstables, use opt.Options.Filter and for filter checks, first lookup filter instance O(1) operation and use it if found. This would make it possible to reuse older filters during the migration period and I don't think it would have a significant performance penalty.

Workaround: Iterate through every key and reput it. This will pretty much force LevelDB to create a new Filter for every LevelDB.

According to README initialization is done as follows

desc, err := descriptor.OpenFile("path/to/db")
...
db, err := leveldb.Open(desc, &opt.Options{Flag: opt.OFCreateIfMissing})
...

My question is, what's the use of the desc variable? Would I ever want to use it except at intiialization?

Hello -

using btcd, which uses https://github.com/conformal/btcdb, if I am able to get btcd to open too many file descriptors, i see this in the LOG

09:06:48.372512 table@compaction commited F-1 S-2MiB D·359 T·2.006759154s
09:06:48.372800 table@compaction L2·1 -> L3·11 S·20MiB Q·35196428
09:06:48.625839 table@compaction created L3@42029 N·51164 S·2MiB "\a..x,v35147984":"\v..x,v8665899"
09:06:48.949797 table@compaction created L3@42030 N·51229 S·2MiB "\v..x,v34167170":"\f..x,v10668000"
09:06:50.797969 table@compaction created L3@42031 N·51202 S·2MiB "\f..x,d10667688":"\r..x,v11986250"
09:06:53.980449 table@compaction created L3@42032 N·51280 S·2MiB "\r..x,d11986048":"\x0e..x,v15906385"
09:06:56.275747 table@compaction created L3@42033 N·51221 S·2MiB "\x0e..x,d15906011":"\x0f..x,v34665764"
09:06:56.327011 table@compaction created L3@42034 N·770 S·31KiB "\x0f..x,d34664997":"\x0f..x,v35139856"
09:06:56.329686 table@compaction commited F-6 S-10MiB D·192 T·7.956848756s
09:06:56.329985 table@compaction L2·1 -> L3·11 S·20MiB Q·35196428
09:06:56.340234 table@build error "open /data/btcd/mainnet/blocks_leveldb/042035.sst: too many open files"
09:06:57.344396 table@build error "open /data/btcd/mainnet/blocks_leveldb/042036.sst: too many open files"
09:06:58.346125 table@build error "open /data/btcd/mainnet/blocks_leveldb/042037.sst: too many open files"
09:06:59.353190 table@build error "open /data/btcd/mainnet/blocks_leveldb/042038.sst: too many open files"
09:07:00.390279 table@build error "open /data/btcd/mainnet/blocks_leveldb/042039.sst: too many open files"
09:07:01.391303 table@build error "open /data/btcd/mainnet/blocks_leveldb/042040.sst: too many open files"
09:07:02.392545 table@build error "open /data/btcd/mainnet/blocks_leveldb/042041.sst: too many open files"

....

btcd essentially stalls and I kill the process. The problem is, when I start btcd back up, the database is corrupt. I get "not found" on the open.

open options:

        myCache := cache.NewEmptyCache()
        opts := &opt.Options{
                BlockCache:   myCache,
                MaxOpenFiles: 256,
                Compression:  opt.NoCompression,
        }

Does goleveldb assume that 256 will always be available?