Giter VIP home page Giter VIP logo

grocksdb's Introduction

grocksdb, RocksDB wrapper for Go

Go Report Card Coverage Status godoc

This is a Fork from tecbot/gorocksdb. I respect the author work and community contribution. The LICENSE still remains as upstream.

Why I made a patched clone instead of PR:

  • Supports almost C API (unlike upstream). Catching up with latest version of Rocksdb as promise.
  • This fork contains no defer in codebase (my side project requires as less overhead as possible). This introduces loose convention of how/when to free c-mem, thus break the rule of tecbot/gorocksdb.

Install

Prerequisite

  • librocksdb
  • libsnappy
  • libz
  • liblz4
  • libzstd
  • libbz2 (optional)

Please follow this guide: https://github.com/facebook/rocksdb/blob/master/INSTALL.md to build above libs.

Build

After installing both rocksdb and grocksdb, you can build your app using the following commands:

CGO_CFLAGS="-I/path/to/rocksdb/include" \
CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd" \
  go build

Or just:

go build // if prerequisites are in linker paths

If your rocksdb was linked with bz2:

CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd -lbz2" \
  go build

Usage

See also: doc

API Support

Almost C API, excepts:

  • putv/mergev/deletev/delete_rangev
  • compaction_filter/compaction_filter_factory/compaction_filter_context
  • transactiondb_property_value/transactiondb_property_int
  • optimistictransactiondb_property_value/optimistictransactiondb_property_int

grocksdb's People

Contributors

cuonglm avatar dependabot[bot] avatar gezort avatar juneezee avatar kingster avatar linxgnu avatar muthukrishnan24 avatar nddeluca avatar ptrus avatar yihuang avatar

Stargazers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Watchers

avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar avatar

Forkers

jimenezrick lee87902407 fedoral kstadium iotaledger hiqsociety eftrueboy enefuture zhanglei bxvs888 anjulapaulus murena7 preccybutton diarsaba sumstudio xrpdevs blackinkgj lyrictian gonzojive wu-yy opensourcecontribs yxwei-umn kingster ylsgit taraxa-project rustwizard cwbhhjl zhongqiuwood okx aureliar8 yihuang warrior21st panda-sheep snirs90 nddeluca kava-labs nick13jaremek phial3 bprosnitz aaxiba nibiruchain cessory saeid-a caubechankiu ptrus nananatsu smellthemoon zhiwayzhang

grocksdb's Issues

Iterator works weird with SetIterateUpperBound and 10.000 keys

I can't test iterator with 10.000 keys. I try to run iteration several times on them, using SetIterateUpperBound in ReadOptions.

Expected behavior

Successful logs

/go/src/github.com/fubss/grocksdb-test-2$ GOFLAGS="-count=1" go test -timeout 30s -run ^TestDrop$ github.com/fubss/grocksdb-test-2 -v
=== RUN TestDrop
-------->NewProvider intialization...
-------->Opening DB and checking format...
-------->RocksDB constructing...
-------->Opening DB...
-------->ParanoidChecks is: true
-------->DB was successfully opened in path: [ /tmp/tests/grocksdbtest2 ]
-------->Checkin if DB is empty...
-------->Checking for emptiness has finished
-------->rocks db IsEmpty()=true
-------->Getting key [_f] from RocksDB...
-------->got data []
-------->Checking for db format at path [/tmp/tests/grocksdbtest2]
format is latest, nothing to do provider_test.go:69: expextedSetup_0
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_1
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_2
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_3
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_4
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
-------->WritingBatch.Count()=[20]
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
-------->WritingBatch.Count()=[8404]
-------->Have removed 8404 entries for channel db3 in rocksdb /tmp/tests/grocksdbtest2
-------->WritingBatch.Count()=[1596]
provider_test.go:106: expextedResults_0
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is not Valid
-------->iterator is not valid anymore
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:106: expextedResults_1
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:106: expextedResults_2
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is not Valid
-------->iterator is not valid anymore
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
-------->Closing db...
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->itr.Err()=[error while obtaining db iterator: rocksdb: closed
github.com/fubss/grocksdb-test-2.(*DB).GetIterator
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/helper.go:151
github.com/fubss/grocksdb-test-2.(*DBHandle).GetIterator
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/provider.go:263
github.com/fubss/grocksdb-test-2.(*DBHandle).deleteAll
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/provider.go:168
github.com/fubss/grocksdb-test-2.TestDrop
/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/provider_test.go:115
testing.tRunner
/usr/local/go/src/testing/testing.go:1259
runtime.goexit
/usr/local/go/src/runtime/asm_amd64.s:1581]. Impossible to create an iterator
-------->Error! Closing iterator...
--- PASS: TestDrop (0.39s)
PASS
ok github.com/fubss/grocksdb-test-2 0.406s

Actual behavior

Sometimes test passes, sometimes not. When it doesn't, iterator becomes invalid in unexpected (every time new) place during iteration within 10.000 keys.
itr.Err() returns nil in this case.

Example logs of wrong behavior

/go/src/github.com/fubss/grocksdb-test-2$ GOFLAGS="-count=1" go test -timeout 30s -run ^TestDrop$ github.com/fubss/grocksdb-test-2 -v
=== RUN TestDrop
-------->NewProvider intialization...
-------->Opening DB and checking format...
-------->RocksDB constructing...
-------->Opening DB...
-------->ParanoidChecks is: true
-------->DB was successfully opened in path: [ /tmp/tests/grocksdbtest2 ]
-------->Checkin if DB is empty...
-------->Checking for emptiness has finished
-------->rocks db IsEmpty()=true
-------->Getting key [_f] from RocksDB...
-------->got data []
-------->Checking for db format at path [/tmp/tests/grocksdbtest2]
format is latest, nothing to do provider_test.go:69: expextedSetup_0
endKey is nil
-------->endKey is nil: eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Constructing iterator with sKey=[db1([]byte{0x64, 0x62, 0x31, 0x0})] and eKey=[db1([]byte{0x64, 0x62, 0x31, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db1 ([100 98 49 0]: [db1 ([100 98 49 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_1
endKey is nil
-------->endKey is nil: eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Constructing iterator with sKey=[db2([]byte{0x64, 0x62, 0x32, 0x0})] and eKey=[db2([]byte{0x64, 0x62, 0x32, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db2 ([100 98 50 0]: [db2 ([100 98 50 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_2
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_3
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
-------->iterator is not valid anymore
provider_test.go:69: expextedSetup_4
endKey is nil
-------->endKey is nil: eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Constructing iterator with sKey=[db3([]byte{0x64, 0x62, 0x33, 0x0})] and eKey=[db3([]byte{0x64, 0x62, 0x33, 0x1})]
-------->Getting new RocksDB Iterator... for start key: [db3 ([100 98 51 0]: [db3 ([100 98 51 1])]
-------->if-case: endKey!=nil, UpperBound set
-------->PurgeOnIterCleanup = true
-------->ni is not Valid, err=[]
-------->itr is Valid
provider_test.go:125: Iterator error is: [%!s()]
provider_test.go:129:
Error Trace: provider_test.go:129
provider_test.go:72
Error: Not equal:
expected: 9999
actual : 6356
Test: TestDrop
--- FAIL: TestDrop (0.27s)
FAIL
FAIL github.com/fubss/grocksdb-test-2 0.280s
FAIL
/go/src/github.com/fubss/grocksdb-test-2$

Steps to reproduce the behavior

  1. go get my test git repo
  2. run GOFLAGS="-count=1" go test -timeout 30s -run ^TestDrop$ github.com/fubss/grocksdb-test-2 -v

Additional information

I'm using Ubuntu 20 LTS.

go env

GO111MODULE=""
GOARCH="amd64"
GOBIN=""
GOCACHE="/home/ivanl/.cache/go-build"
GOENV="/home/ivanl/.config/go/env"
GOEXE=""
GOEXPERIMENT=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOINSECURE=""
GOMODCACHE="/home/ivanl/go/pkg/mod"
GONOPROXY=""
GONOSUMDB=""
GOOS="linux"
GOPATH="/home/ivanl/go"
GOPRIVATE=""
GOPROXY="https://proxy.golang.org,direct"
GOROOT="/usr/local/go"
GOSUMDB="sum.golang.org"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GOVCS=""
GOVERSION="go1.17.3"
GCCGO="gccgo"
AR="ar"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD="/home/ivanl/go/src/github.com/fubss/grocksdb-test-2/go.mod"
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build2854272429=/tmp/go-build -gno-record-gcc-switches"

my RocksDB LOG file

2021/12/15-16:23:43.954005 7f39f45b2ac0 [/db_impl/db_impl_open.cc:184] file size check will be skipped during open.
2021/12/15-16:23:43.954290 7f39f45b2ac0 RocksDB version: 6.22.1
2021/12/15-16:23:43.954304 7f39f45b2ac0 Git sha 51b540921dd7495c9cf2265eb58942dad1f2ef72
2021/12/15-16:23:43.954307 7f39f45b2ac0 Compile date 2021-12-04 14:52:07
2021/12/15-16:23:43.954316 7f39f45b2ac0 DB SUMMARY
2021/12/15-16:23:43.954317 7f39f45b2ac0 DB Session ID: 6449I48JEUTH264QRR4L
2021/12/15-16:23:43.954347 7f39f45b2ac0 SST files in /tmp/tests/grocksdbtest2 dir, Total Num: 0, files:
2021/12/15-16:23:43.954351 7f39f45b2ac0 Write Ahead Log file in /tmp/tests/grocksdbtest2:
2021/12/15-16:23:43.954353 7f39f45b2ac0 Options.error_if_exists: 0
2021/12/15-16:23:43.954355 7f39f45b2ac0 Options.create_if_missing: 1
2021/12/15-16:23:43.954357 7f39f45b2ac0 Options.paranoid_checks: 0
2021/12/15-16:23:43.954359 7f39f45b2ac0 Options.flush_verify_memtable_count: 1
2021/12/15-16:23:43.954360 7f39f45b2ac0 Options.track_and_verify_wals_in_manifest: 0
2021/12/15-16:23:43.954362 7f39f45b2ac0 Options.env: 0x7f39f55571c0
2021/12/15-16:23:43.954364 7f39f45b2ac0 Options.fs: Posix File System
2021/12/15-16:23:43.954366 7f39f45b2ac0 Options.info_log: 0x18aa860
2021/12/15-16:23:43.954368 7f39f45b2ac0 Options.max_file_opening_threads: 16
2021/12/15-16:23:43.954369 7f39f45b2ac0 Options.statistics: (nil)
2021/12/15-16:23:43.954371 7f39f45b2ac0 Options.use_fsync: 0
2021/12/15-16:23:43.954373 7f39f45b2ac0 Options.max_log_file_size: 0
2021/12/15-16:23:43.954374 7f39f45b2ac0 Options.max_manifest_file_size: 1073741824
2021/12/15-16:23:43.954376 7f39f45b2ac0 Options.log_file_time_to_roll: 0
2021/12/15-16:23:43.954378 7f39f45b2ac0 Options.keep_log_file_num: 1000
2021/12/15-16:23:43.954379 7f39f45b2ac0 Options.recycle_log_file_num: 0
2021/12/15-16:23:43.954381 7f39f45b2ac0 Options.allow_fallocate: 1
2021/12/15-16:23:43.954382 7f39f45b2ac0 Options.allow_mmap_reads: 0
2021/12/15-16:23:43.954384 7f39f45b2ac0 Options.allow_mmap_writes: 0
2021/12/15-16:23:43.954385 7f39f45b2ac0 Options.use_direct_reads: 0
2021/12/15-16:23:43.954387 7f39f45b2ac0 Options.use_direct_io_for_flush_and_compaction: 0
2021/12/15-16:23:43.954388 7f39f45b2ac0 Options.create_missing_column_families: 0
2021/12/15-16:23:43.954390 7f39f45b2ac0 Options.db_log_dir:
2021/12/15-16:23:43.954391 7f39f45b2ac0 Options.wal_dir: /tmp/tests/grocksdbtest2
2021/12/15-16:23:43.954393 7f39f45b2ac0 Options.table_cache_numshardbits: 6
2021/12/15-16:23:43.954394 7f39f45b2ac0 Options.WAL_ttl_seconds: 0
2021/12/15-16:23:43.954396 7f39f45b2ac0 Options.WAL_size_limit_MB: 0
2021/12/15-16:23:43.954397 7f39f45b2ac0 Options.max_write_batch_group_size_bytes: 1048576
2021/12/15-16:23:43.954399 7f39f45b2ac0 Options.manifest_preallocation_size: 4194304
2021/12/15-16:23:43.954401 7f39f45b2ac0 Options.is_fd_close_on_exec: 1
2021/12/15-16:23:43.954402 7f39f45b2ac0 Options.advise_random_on_open: 1
2021/12/15-16:23:43.954404 7f39f45b2ac0 Options.db_write_buffer_size: 0
2021/12/15-16:23:43.954406 7f39f45b2ac0 Options.write_buffer_manager: 0x1898340
2021/12/15-16:23:43.954407 7f39f45b2ac0 Options.access_hint_on_compaction_start: 1
2021/12/15-16:23:43.954408 7f39f45b2ac0 Options.new_table_reader_for_compaction_inputs: 0
2021/12/15-16:23:43.954410 7f39f45b2ac0 Options.random_access_max_buffer_size: 1048576
2021/12/15-16:23:43.954411 7f39f45b2ac0 Options.use_adaptive_mutex: 0
2021/12/15-16:23:43.954413 7f39f45b2ac0 Options.rate_limiter: (nil)
2021/12/15-16:23:43.954433 7f39f45b2ac0 Options.sst_file_manager.rate_bytes_per_sec: 0
2021/12/15-16:23:43.954435 7f39f45b2ac0 Options.wal_recovery_mode: 2
2021/12/15-16:23:43.954436 7f39f45b2ac0 Options.enable_thread_tracking: 0
2021/12/15-16:23:43.954438 7f39f45b2ac0 Options.enable_pipelined_write: 0
2021/12/15-16:23:43.954439 7f39f45b2ac0 Options.unordered_write: 0
2021/12/15-16:23:43.954441 7f39f45b2ac0 Options.allow_concurrent_memtable_write: 1
2021/12/15-16:23:43.954442 7f39f45b2ac0 Options.enable_write_thread_adaptive_yield: 1
2021/12/15-16:23:43.954444 7f39f45b2ac0 Options.write_thread_max_yield_usec: 100
2021/12/15-16:23:43.954446 7f39f45b2ac0 Options.write_thread_slow_yield_usec: 3
2021/12/15-16:23:43.954448 7f39f45b2ac0 Options.row_cache: None
2021/12/15-16:23:43.954449 7f39f45b2ac0 Options.wal_filter: None
2021/12/15-16:23:43.954451 7f39f45b2ac0 Options.avoid_flush_during_recovery: 0
2021/12/15-16:23:43.954452 7f39f45b2ac0 Options.allow_ingest_behind: 0
2021/12/15-16:23:43.954454 7f39f45b2ac0 Options.preserve_deletes: 0
2021/12/15-16:23:43.954456 7f39f45b2ac0 Options.two_write_queues: 0
2021/12/15-16:23:43.954457 7f39f45b2ac0 Options.manual_wal_flush: 0
2021/12/15-16:23:43.954459 7f39f45b2ac0 Options.atomic_flush: 0
2021/12/15-16:23:43.954460 7f39f45b2ac0 Options.avoid_unnecessary_blocking_io: 0
2021/12/15-16:23:43.954462 7f39f45b2ac0 Options.persist_stats_to_disk: 0
2021/12/15-16:23:43.954463 7f39f45b2ac0 Options.write_dbid_to_manifest: 0
2021/12/15-16:23:43.954465 7f39f45b2ac0 Options.log_readahead_size: 0
2021/12/15-16:23:43.954466 7f39f45b2ac0 Options.file_checksum_gen_factory: Unknown
2021/12/15-16:23:43.954468 7f39f45b2ac0 Options.best_efforts_recovery: 0
2021/12/15-16:23:43.954469 7f39f45b2ac0 Options.max_bgerror_resume_count: 2147483647
2021/12/15-16:23:43.954471 7f39f45b2ac0 Options.bgerror_resume_retry_interval: 1000000
2021/12/15-16:23:43.954473 7f39f45b2ac0 Options.allow_data_in_errors: 0
2021/12/15-16:23:43.954474 7f39f45b2ac0 Options.db_host_id: hostname
2021/12/15-16:23:43.954476 7f39f45b2ac0 Options.max_background_jobs: 2
2021/12/15-16:23:43.954478 7f39f45b2ac0 Options.max_background_compactions: -1
2021/12/15-16:23:43.954480 7f39f45b2ac0 Options.max_subcompactions: 1
2021/12/15-16:23:43.954481 7f39f45b2ac0 Options.avoid_flush_during_shutdown: 0
2021/12/15-16:23:43.954483 7f39f45b2ac0 Options.writable_file_max_buffer_size: 1048576
2021/12/15-16:23:43.954484 7f39f45b2ac0 Options.delayed_write_rate : 16777216
2021/12/15-16:23:43.954486 7f39f45b2ac0 Options.max_total_wal_size: 0
2021/12/15-16:23:43.954488 7f39f45b2ac0 Options.delete_obsolete_files_period_micros: 21600000000
2021/12/15-16:23:43.954489 7f39f45b2ac0 Options.stats_dump_period_sec: 600
2021/12/15-16:23:43.954491 7f39f45b2ac0 Options.stats_persist_period_sec: 600
2021/12/15-16:23:43.954492 7f39f45b2ac0 Options.stats_history_buffer_size: 1048576
2021/12/15-16:23:43.954493 7f39f45b2ac0 Options.max_open_files: -1
2021/12/15-16:23:43.954495 7f39f45b2ac0 Options.bytes_per_sync: 0
2021/12/15-16:23:43.954496 7f39f45b2ac0 Options.wal_bytes_per_sync: 0
2021/12/15-16:23:43.954498 7f39f45b2ac0 Options.strict_bytes_per_sync: 0
2021/12/15-16:23:43.954499 7f39f45b2ac0 Options.compaction_readahead_size: 0
2021/12/15-16:23:43.954501 7f39f45b2ac0 Options.max_background_flushes: -1
2021/12/15-16:23:43.954502 7f39f45b2ac0 Compression algorithms supported:
2021/12/15-16:23:43.954514 7f39f45b2ac0 kZSTDNotFinalCompression supported: 1
2021/12/15-16:23:43.954518 7f39f45b2ac0 kZSTD supported: 1
2021/12/15-16:23:43.954520 7f39f45b2ac0 kXpressCompression supported: 0
2021/12/15-16:23:43.954521 7f39f45b2ac0 kLZ4HCCompression supported: 1
2021/12/15-16:23:43.954523 7f39f45b2ac0 kLZ4Compression supported: 1
2021/12/15-16:23:43.954525 7f39f45b2ac0 kBZip2Compression supported: 1
2021/12/15-16:23:43.954526 7f39f45b2ac0 kZlibCompression supported: 1
2021/12/15-16:23:43.954528 7f39f45b2ac0 kSnappyCompression supported: 1
2021/12/15-16:23:43.954543 7f39f45b2ac0 Fast CRC32 supported: Supported on x86
2021/12/15-16:23:43.955343 7f39f45b2ac0 [/db_impl/db_impl_open.cc:285] Creating manifest 1
2021/12/15-16:23:43.957575 7f39f45b2ac0 [/version_set.cc:4577] Recovering from manifest file: /tmp/tests/grocksdbtest2/MANIFEST-000001
2021/12/15-16:23:43.957766 7f39f45b2ac0 [/column_family.cc:600] --------------- Options for column family [default]:
2021/12/15-16:23:43.957770 7f39f45b2ac0 Options.comparator: leveldb.BytewiseComparator
2021/12/15-16:23:43.957771 7f39f45b2ac0 Options.merge_operator: None
2021/12/15-16:23:43.957772 7f39f45b2ac0 Options.compaction_filter: None
2021/12/15-16:23:43.957772 7f39f45b2ac0 Options.compaction_filter_factory: None
2021/12/15-16:23:43.957773 7f39f45b2ac0 Options.sst_partitioner_factory: None
2021/12/15-16:23:43.957774 7f39f45b2ac0 Options.memtable_factory: SkipListFactory
2021/12/15-16:23:43.957775 7f39f45b2ac0 Options.table_factory: BlockBasedTable
2021/12/15-16:23:43.957802 7f39f45b2ac0 table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x18a6af0)
cache_index_and_filter_blocks: 0
cache_index_and_filter_blocks_with_high_priority: 1
pin_l0_filter_and_index_blocks_in_cache: 0
pin_top_level_index_and_filter: 1
index_type: 0
data_block_index_type: 0
index_shortening: 1
data_block_hash_table_util_ratio: 0.750000
hash_index_allow_collision: 1
checksum: 1
no_block_cache: 0
block_cache: 0x18a6b40
block_cache_name: LRUCache
block_cache_options:
capacity : 8388608
num_shard_bits : 4
strict_capacity_limit : 0
memory_allocator : None
high_pri_pool_ratio: 0.000
block_cache_compressed: (nil)
persistent_cache: (nil)
block_size: 4096
block_size_deviation: 10
block_restart_interval: 16
index_block_restart_interval: 1
metadata_block_size: 4096
partition_filters: 0
use_delta_encoding: 1
filter_policy: nullptr
whole_key_filtering: 1
verify_compression: 0
read_amp_bytes_per_bit: 0
format_version: 5
enable_index_compression: 1
block_align: 0
max_auto_readahead_size: 262144
prepopulate_block_cache: 0
2021/12/15-16:23:43.957804 7f39f45b2ac0 Options.write_buffer_size: 67108864
2021/12/15-16:23:43.957804 7f39f45b2ac0 Options.max_write_buffer_number: 2
2021/12/15-16:23:43.957807 7f39f45b2ac0 Options.compression: Snappy
2021/12/15-16:23:43.957808 7f39f45b2ac0 Options.bottommost_compression: Disabled
2021/12/15-16:23:43.957809 7f39f45b2ac0 Options.prefix_extractor: nullptr
2021/12/15-16:23:43.957809 7f39f45b2ac0 Options.memtable_insert_with_hint_prefix_extractor: nullptr
2021/12/15-16:23:43.957810 7f39f45b2ac0 Options.num_levels: 7
2021/12/15-16:23:43.957811 7f39f45b2ac0 Options.min_write_buffer_number_to_merge: 1
2021/12/15-16:23:43.957812 7f39f45b2ac0 Options.max_write_buffer_number_to_maintain: 0
2021/12/15-16:23:43.957812 7f39f45b2ac0 Options.max_write_buffer_size_to_maintain: 0
2021/12/15-16:23:43.957813 7f39f45b2ac0 Options.bottommost_compression_opts.window_bits: -14
2021/12/15-16:23:43.957814 7f39f45b2ac0 Options.bottommost_compression_opts.level: 32767
2021/12/15-16:23:43.957815 7f39f45b2ac0 Options.bottommost_compression_opts.strategy: 0
2021/12/15-16:23:43.957816 7f39f45b2ac0 Options.bottommost_compression_opts.max_dict_bytes: 0
2021/12/15-16:23:43.957816 7f39f45b2ac0 Options.bottommost_compression_opts.zstd_max_train_bytes: 0
2021/12/15-16:23:43.957831 7f39f45b2ac0 Options.bottommost_compression_opts.parallel_threads: 1
2021/12/15-16:23:43.957832 7f39f45b2ac0 Options.bottommost_compression_opts.enabled: false
2021/12/15-16:23:43.957833 7f39f45b2ac0 Options.bottommost_compression_opts.max_dict_buffer_bytes: 0
2021/12/15-16:23:43.957834 7f39f45b2ac0 Options.compression_opts.window_bits: -14
2021/12/15-16:23:43.957835 7f39f45b2ac0 Options.compression_opts.level: 32767
2021/12/15-16:23:43.957835 7f39f45b2ac0 Options.compression_opts.strategy: 0
2021/12/15-16:23:43.957836 7f39f45b2ac0 Options.compression_opts.max_dict_bytes: 0
2021/12/15-16:23:43.957837 7f39f45b2ac0 Options.compression_opts.zstd_max_train_bytes: 0
2021/12/15-16:23:43.957838 7f39f45b2ac0 Options.compression_opts.parallel_threads: 1
2021/12/15-16:23:43.957838 7f39f45b2ac0 Options.compression_opts.enabled: false
2021/12/15-16:23:43.957839 7f39f45b2ac0 Options.compression_opts.max_dict_buffer_bytes: 0
2021/12/15-16:23:43.957840 7f39f45b2ac0 Options.level0_file_num_compaction_trigger: 4
2021/12/15-16:23:43.957840 7f39f45b2ac0 Options.level0_slowdown_writes_trigger: 20
2021/12/15-16:23:43.957841 7f39f45b2ac0 Options.level0_stop_writes_trigger: 36
2021/12/15-16:23:43.957842 7f39f45b2ac0 Options.target_file_size_base: 67108864
2021/12/15-16:23:43.957843 7f39f45b2ac0 Options.target_file_size_multiplier: 1
2021/12/15-16:23:43.957843 7f39f45b2ac0 Options.max_bytes_for_level_base: 268435456
2021/12/15-16:23:43.957844 7f39f45b2ac0 Options.level_compaction_dynamic_level_bytes: 0
2021/12/15-16:23:43.957845 7f39f45b2ac0 Options.max_bytes_for_level_multiplier: 10.000000
2021/12/15-16:23:43.957847 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[0]: 1
2021/12/15-16:23:43.957848 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[1]: 1
2021/12/15-16:23:43.957848 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[2]: 1
2021/12/15-16:23:43.957849 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[3]: 1
2021/12/15-16:23:43.957850 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[4]: 1
2021/12/15-16:23:43.957850 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[5]: 1
2021/12/15-16:23:43.957851 7f39f45b2ac0 Options.max_bytes_for_level_multiplier_addtl[6]: 1
2021/12/15-16:23:43.957852 7f39f45b2ac0 Options.max_sequential_skip_in_iterations: 8
2021/12/15-16:23:43.957852 7f39f45b2ac0 Options.max_compaction_bytes: 1677721600
2021/12/15-16:23:43.957853 7f39f45b2ac0 Options.arena_block_size: 1048576
2021/12/15-16:23:43.957854 7f39f45b2ac0 Options.soft_pending_compaction_bytes_limit: 68719476736
2021/12/15-16:23:43.957855 7f39f45b2ac0 Options.hard_pending_compaction_bytes_limit: 274877906944
2021/12/15-16:23:43.957855 7f39f45b2ac0 Options.rate_limit_delay_max_milliseconds: 100
2021/12/15-16:23:43.957856 7f39f45b2ac0 Options.disable_auto_compactions: 0
2021/12/15-16:23:43.957857 7f39f45b2ac0 Options.compaction_style: kCompactionStyleLevel
2021/12/15-16:23:43.957858 7f39f45b2ac0 Options.compaction_pri: kMinOverlappingRatio
2021/12/15-16:23:43.957859 7f39f45b2ac0 Options.compaction_options_universal.size_ratio: 1
2021/12/15-16:23:43.957860 7f39f45b2ac0 Options.compaction_options_universal.min_merge_width: 2
2021/12/15-16:23:43.957861 7f39f45b2ac0 Options.compaction_options_universal.max_merge_width: 4294967295
2021/12/15-16:23:43.957861 7f39f45b2ac0 Options.compaction_options_universal.max_size_amplification_percent: 200
2021/12/15-16:23:43.957862 7f39f45b2ac0 Options.compaction_options_universal.compression_size_percent: -1
2021/12/15-16:23:43.957863 7f39f45b2ac0 Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize
2021/12/15-16:23:43.957863 7f39f45b2ac0 Options.compaction_options_fifo.max_table_files_size: 1073741824
2021/12/15-16:23:43.957872 7f39f45b2ac0 Options.compaction_options_fifo.allow_compaction: 0
2021/12/15-16:23:43.957875 7f39f45b2ac0 Options.table_properties_collectors:
2021/12/15-16:23:43.957876 7f39f45b2ac0 Options.inplace_update_support: 0
2021/12/15-16:23:43.957877 7f39f45b2ac0 Options.inplace_update_num_locks: 10000
2021/12/15-16:23:43.957877 7f39f45b2ac0 Options.memtable_prefix_bloom_size_ratio: 0.000000
2021/12/15-16:23:43.957879 7f39f45b2ac0 Options.memtable_whole_key_filtering: 0
2021/12/15-16:23:43.957879 7f39f45b2ac0 Options.memtable_huge_page_size: 0
2021/12/15-16:23:43.957880 7f39f45b2ac0 Options.bloom_locality: 0
2021/12/15-16:23:43.957881 7f39f45b2ac0 Options.max_successive_merges: 0
2021/12/15-16:23:43.957881 7f39f45b2ac0 Options.optimize_filters_for_hits: 0
2021/12/15-16:23:43.957882 7f39f45b2ac0 Options.paranoid_file_checks: 0
2021/12/15-16:23:43.957883 7f39f45b2ac0 Options.force_consistency_checks: 1
2021/12/15-16:23:43.957883 7f39f45b2ac0 Options.report_bg_io_stats: 0
2021/12/15-16:23:43.957884 7f39f45b2ac0 Options.ttl: 2592000
2021/12/15-16:23:43.957885 7f39f45b2ac0 Options.periodic_compaction_seconds: 0
2021/12/15-16:23:43.957886 7f39f45b2ac0 Options.enable_blob_files: false
2021/12/15-16:23:43.957886 7f39f45b2ac0 Options.min_blob_size: 0
2021/12/15-16:23:43.957887 7f39f45b2ac0 Options.blob_file_size: 268435456
2021/12/15-16:23:43.957888 7f39f45b2ac0 Options.blob_compression_type: NoCompression
2021/12/15-16:23:43.957889 7f39f45b2ac0 Options.enable_blob_garbage_collection: false
2021/12/15-16:23:43.957889 7f39f45b2ac0 Options.blob_garbage_collection_age_cutoff: 0.250000
2021/12/15-16:23:43.960088 7f39f45b2ac0 [/version_set.cc:4617] Recovered from manifest file:/tmp/tests/grocksdbtest2/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0
2021/12/15-16:23:43.960094 7f39f45b2ac0 [/version_set.cc:4632] Column family [default] (ID 0), log number is 0
2021/12/15-16:23:43.960191 7f39f45b2ac0 [/version_set.cc:4123] Creating manifest 4
2021/12/15-16:23:43.962209 7f39f45b2ac0 [DEBUG] [/db_impl/db_impl_files.cc:334] [JOB 1] Delete /tmp/tests/grocksdbtest2/MANIFEST-000001 type=3 #1 -- OK
2021/12/15-16:23:43.964586 7f39f45b2ac0 [/db_impl/db_impl_open.cc:1759] SstFileManager instance 0x18a9140

  • I managed to find out that the problem in the lines with SetIterateUpperBound. When I comment them, Iterator works stable. But I want to use all iterator features with upper bound setting.
  • When I set exactly upper bound equal the last key name, iterator seems also working fine.

I'm new in DBs, golang, github, sorry if anything wrong.

failed to compile: library not found for -lsnappy

Hi. Many thanks wholeheartedly for maintaining this repo, as the original Gorocksdb is no longer compatible with the newer version of RocksDB.
I've encountered the following issue during compiling.

/opt/homebrew/cellar/go/1.20.5/libexec/pkg/tool/darwin_arm64/link: running cc failed: exit status 1
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)

Environment: MacOS Ventura with Apple Silicon
Both RocksDB and its dependencies (including snappy) are installed via Homebrew.

Any assistance is greatly appreciated.

opening more than 11 column families results in error 

                    cfNames = []string{"default", "m", "v", "c", "e", "x", "k", "l", "s", "h", "z", "bxb"}
                    //cfNames = []string{"default", "m", "v", "c", "e", "x", "k", "l", "s", "h", "z", "b", "p"}
                    dbr[i], cfh, err = gorocksdb.OpenDbColumnFamilies(opts[i], defaultDBDir+"/d/"+ks, cfNames, []*gorocksdb.Options{opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i]})


fatal error: unexpected signal during runtime execution
[signal SIGSEGV: segmentation violation code=0x1 addr=0x478 pc=0xc41992]

runtime stack:
runtime.throw(0x123210e, 0x2a)
        /usr/local/go/src/runtime/panic.go:1117 +0x72
runtime.sigpanic()
        /usr/local/go/src/runtime/signal_unix.go:718 +0x2e5

goroutine 1 [syscall, locked to thread]:
runtime.cgocall(0x9062fc, 0xc000207478, 0xc000010588)
        /usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc000207448 sp=0xc000207410 pc=0x79123b
git.a.com/a/grocksdb._Cfunc_rocksdb_open_column_families(0x7f4bb919fa00, 0x7f4bb903f260, 0xc, 0xc00005a480, 0xc00001e120, 0xc00001e180, 0xc000010588, 0x0)
        _cgo_gotypes.go:2705 +0x49 fp=0xc000207478 sp=0xc000207448 pc=0x874c29
git.a.com/a/grocksdb.OpenDbColumnFamilies.func1(0xc000059900, 0x7f4bb903f260, 0xc, 0xc0002075d0, 0xc0002075b8, 0xc0002075e8, 0xc000010588, 0x8)
        /root/go/pkg/mod/git.a.com/a/[email protected]/db.go:158 +0x229 fp=0xc0002074e8 sp=0xc000207478 pc=0x878b49
git.a.com/a/grocksdb.OpenDbColumnFamilies(0xc000059900, 0xc000016c80, 0x8, 0xc000168540, 0xc, 0xc, 0xc000207878, 0xc, 0xc, 0x0, ...)
        /root/go/pkg/mod/git.a.com/a/[email protected]/db.go:158 +0x285 fp=0xc000207610 sp=0xc0002074e8 pc=0x875f85

go test failed

I met this bug when I start the go test to run the inner unit test I got the error told me that I have no linked with Lz4 or other Lib, Why cause this error ?

do i hv to value.Free() if i declared but didnt use it?

sorry to ask something here. great work. using grocksdb now. will credit your work.

func something() {
var value *grocksdb.Slice

     if abc == 1 {

          value = dbr.GetCF(rodb,.......)
          value.Free() 
    }

//value.Free() <-- is this needed here? coz it's not been used
}

TransactionDB is not safe-concurrent on writes!

Hi @linxGnu,

When I write into the same rocksdb file with 2 gorutines, i have this unexpected panic.

// configure
func (db *DB) init() error {
	cache := grocksdb.NewLRUCache(8 * common.MiB)

	blockBaseTableOptions := grocksdb.NewDefaultBlockBasedTableOptions()
	blockBaseTableOptions.SetBlockCache(cache)

	opts := grocksdb.NewDefaultOptions()
	opts.SetCreateIfMissing(true)
	opts.SetBlockBasedTableFactory(blockBaseTableOptions)
	opts.SetWriteBufferSize(4 * common.MiB)

	txDBOpts := grocksdb.NewDefaultTransactionDBOptions()

	rdb, err := grocksdb.OpenTransactionDb(opts, txDBOpts, db.path)
	if err != nil {
		return errors.Wrap(err, "failed to open db")
	}

	db.db = rdb
	db.ro = grocksdb.NewDefaultReadOptions()
	db.wo = grocksdb.NewDefaultWriteOptions()
	db.txOpts = grocksdb.NewDefaultTransactionOptions()

	db.ro.SetFillCache(true)
	db.wo.SetSync(db.fsync)

	return nil
}
// test writes
func run(database *rocksdb.DB, chRenewTx chan struct{}) {
	for i := 0; i < expectedTotalMessages; i++ {
		for j := 0; j < 5; j++ {
			key := []byte("key-" + strconv.Itoa(i) + "-" + strconv.Itoa(j))
			value := []byte("value-" + strconv.Itoa(i) + "-" + strconv.Itoa(j))

			if _, err := database.Get(key); err != nil {
				panic(err)
			}

			if err := database.Set(key, value); err != nil {
				panic(err)
			}

			if i > 0 && (i+j)%messagesToCommit == 0 {
				chRenewTx <- struct{}{}
			}
		}
	}
}

func main() {
	database, err := rocksdbwrapper.New("./.db_data", false)
	if err != nil {
		panic(err)
	}

	if err := database.Begin(); err != nil {
		panic(err)
	}

	chRenewTx := make(chan struct{}, 1000)

	go func() {
		for range chRenewTx {
			// fmt.Println("Renewing tx...")
			if err := database.RenewTransaction(); err != nil {
				panic(err)
			}
		}
	}()

	now := time.Now()

	go run(database, chRenewTx) // If i disable this gorutine, does not panic!
	run(database, chRenewTx)

	since := time.Since(now)

	fmt.Println("Time:", since)
	fmt.Println("Rate:", float64(expectedTotalMessages)/since.Seconds())
}
main(11175,0x700008e78000) malloc: *** error for object 0x5028c00: pointer being freed was not allocated
main(11175,0x700008e78000) malloc: *** set a breakpoint in malloc_error_break to debug
SIGABRT: abort
PC=0x7fff2061092e m=3 sigcode=0

goroutine 0 [idle]:
runtime: unknown pc 0x7fff2061092e
stack: frame={sp:0x700008e77ad8, fp:0x0} stack=[0x700008df8350,0x700008e77f50)
0000700008e779d8:  00007fff2048dba3  0000000000000000 
0000700008e779e8:  0000700008e77a38  0000700008e77a30 
0000700008e779f8:  0000000004ff8000  0000700008e78000 
0000700008e77a08:  00007ffeefbff465  00007fff2048db2e 
0000700008e77a18:  0000000000000050  0000700008e77b00 
0000700008e77a28:  00007fff20683627  0000003000000018 
0000700008e77a38:  0000700008e77b10  0000700008e77a50 
0000700008e77a48:  00007fff2068376d  0000000000000000 
0000700008e77a58:  0000000000000000  00007fff2048db2e 
0000700008e77a68:  0000000004ff804b  0000000000000000 
0000700008e77a78:  0000000000000000  0000700008e77c68 
0000700008e77a88:  0000700008e77af8  0000000004ff7028 
0000700008e77a98:  0000000000000050  00007fff809382a0 
0000700008e77aa8:  00007fff8096bae8  00007fff2054d896 
0000700008e77ab8:  0000000000000000  0000700008e77b00 
0000700008e77ac8:  00007fff205502c6  0000000000000000 
0000700008e77ad8: <00007fff2063f5bd  0000000000000003 
0000700008e77ae8:  0000000004ff7028  0000700008e77b20 
0000700008e77af8:  0000000000000000  0000700008e77b40 
0000700008e77b08:  00007fff20594411  0000000004ff7028 
0000700008e77b18:  00007fff2060b829  00007ffffffff9df 
0000700008e77b28:  ffffffff04ff7000  0000000004ff7000 
0000700008e77b38:  0000000000000000  0000700008e77ba0 
0000700008e77b48:  00007fff204741f5  000000002d004b10 
0000700008e77b58:  0000700008e78000  0000700008e77c60 
0000700008e77b68:  0000000000000000  00007fff2048b93d 
0000700008e77b78:  00000000000017fd  00000000000017fd 
0000700008e77b88:  0000000000003000  000000002c830040 
0000700008e77b98:  000000002e00c200  0000700008e77c80 
0000700008e77ba8:  00007fff2047734a  0000000400000000 
0000700008e77bb8:  0000000004dc7380  0000000005028c00 
0000700008e77bc8:  0000000000001400  0000000005000000 
runtime: unknown pc 0x7fff2061092e
stack: frame={sp:0x700008e77ad8, fp:0x0} stack=[0x700008df8350,0x700008e77f50)
0000700008e779d8:  00007fff2048dba3  0000000000000000 
0000700008e779e8:  0000700008e77a38  0000700008e77a30 
0000700008e779f8:  0000000004ff8000  0000700008e78000 
0000700008e77a08:  00007ffeefbff465  00007fff2048db2e 
0000700008e77a18:  0000000000000050  0000700008e77b00 
0000700008e77a28:  00007fff20683627  0000003000000018 
0000700008e77a38:  0000700008e77b10  0000700008e77a50 
0000700008e77a48:  00007fff2068376d  0000000000000000 
0000700008e77a58:  0000000000000000  00007fff2048db2e 
0000700008e77a68:  0000000004ff804b  0000000000000000 
0000700008e77a78:  0000000000000000  0000700008e77c68 
0000700008e77a88:  0000700008e77af8  0000000004ff7028 
0000700008e77a98:  0000000000000050  00007fff809382a0 
0000700008e77aa8:  00007fff8096bae8  00007fff2054d896 
0000700008e77ab8:  0000000000000000  0000700008e77b00 
0000700008e77ac8:  00007fff205502c6  0000000000000000 
0000700008e77ad8: <00007fff2063f5bd  0000000000000003 
0000700008e77ae8:  0000000004ff7028  0000700008e77b20 
0000700008e77af8:  0000000000000000  0000700008e77b40 
0000700008e77b08:  00007fff20594411  0000000004ff7028 
0000700008e77b18:  00007fff2060b829  00007ffffffff9df 
0000700008e77b28:  ffffffff04ff7000  0000000004ff7000 
0000700008e77b38:  0000000000000000  0000700008e77ba0 
0000700008e77b48:  00007fff204741f5  000000002d004b10 
0000700008e77b58:  0000700008e78000  0000700008e77c60 
0000700008e77b68:  0000000000000000  00007fff2048b93d 
0000700008e77b78:  00000000000017fd  00000000000017fd 
0000700008e77b88:  0000000000003000  000000002c830040 
0000700008e77b98:  000000002e00c200  0000700008e77c80 
0000700008e77ba8:  00007fff2047734a  0000000400000000 
0000700008e77bb8:  0000000004dc7380  0000000005028c00 
0000700008e77bc8:  0000000000001400  0000000005000000 

goroutine 7 [syscall]:
runtime.cgocall(0x4161b70, 0xc00005ed80, 0xc0000109d8)
        /usr/local/go/src/runtime/cgocall.go:154 +0x5b fp=0xc00005ed50 sp=0xc00005ed18 pc=0x400445b
github.com/linxGnu/grocksdb._Cfunc_rocksdb_transaction_put(0xc50a970, 0xc000173528, 0x8, 0xc000173530, 0xa, 0xc0000109d8)
        _cgo_gotypes.go:8175 +0x45 fp=0xc00005ed80 sp=0xc00005ed50 pc=0x4153625
github.com/linxGnu/grocksdb.(*Transaction).Put.func2(0xc0001820a0, 0xc000173528, 0xc000173528, 0x8, 0x8, 0xc000173530, 0xc000173530, 0xa, 0x10, 0xc0000109d8)
        /usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/transaction.go:128 +0xcb fp=0xc00005edc8 sp=0xc00005ed80 pc=0x415674b
github.com/linxGnu/grocksdb.(*Transaction).Put(0xc0001820a0, 0xc000173528, 0x8, 0x8, 0xc000173530, 0xa, 0x10, 0x0, 0x0)
        /usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/transaction.go:128 +0x105 fp=0xc00005ee40 sp=0xc00005edc8 pc=0x4155865
gitlab.com/etoshi/backend/normalizer-r2d2/pkg/db/providers/rocksdb.(*DB).Set(0xc00002c540, 0xc000173528, 0x8, 0x8, 0xc000173530, 0xa, 0x10, 0x0, 0x0)
        /Users/sergio/Code/backend/normalizer-r2d2/pkg/db/providers/rocksdb/rocksdb.go:160 +0x1dd fp=0xc00005eec8 sp=0xc00005ee40 pc=0x4159d7d
main.run(0xc00002c540, 0xc00007c240)
        /Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:28 +0x2d5 fp=0xc00005efd0 sp=0xc00005eec8 pc=0x415a6b5
runtime.goexit()
        /usr/local/go/src/runtime/asm_amd64.s:1371 +0x1 fp=0xc00005efd8 sp=0xc00005efd0 pc=0x406a401
created by main.main
        /Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:62 +0x133

goroutine 1 [runnable]:
github.com/linxGnu/grocksdb._Cfunc_rocksdb_free(0xc510d30)
        _cgo_gotypes.go:2661 +0x3c
github.com/linxGnu/grocksdb.(*Slice).Free.func1(0xc000600000)
        /usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/slice.go:53 +0x55
github.com/linxGnu/grocksdb.(*Slice).Free(0xc000600000)
        /usr/local/godeps/pkg/mod/github.com/linx!gnu/[email protected]/slice.go:53 +0x3b
gitlab.com/etoshi/backend/normalizer-r2d2/pkg/db/providers/rocksdb.(*DB).get(0xc00002c540, 0xc000580000, 0x8, 0x8, 0x4051925, 0xc000580010, 0xc00016fe58, 0xa, 0x10)
        /Users/sergio/Code/backend/normalizer-r2d2/pkg/db/providers/rocksdb/rocksdb.go:205 +0x165
gitlab.com/etoshi/backend/normalizer-r2d2/pkg/db/providers/rocksdb.(*DB).Get(0xc00002c540, 0xc000580000, 0x8, 0x8, 0x0, 0x0, 0x0, 0x0, 0x0)
        /Users/sergio/Code/backend/normalizer-r2d2/pkg/db/providers/rocksdb/rocksdb.go:214 +0xaf
main.run(0xc00002c540, 0xc00007c240)
        /Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:24 +0x269
main.main()
        /Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:63 +0x14b

goroutine 6 [chan receive]:
main.main.func1(0xc00007c240, 0xc00002c540)
        /Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:52 +0x35
created by main.main
        /Users/sergio/Code/backend/normalizer-r2d2/_examples/foo/main.go:51 +0xe5

rax    0x0
rbx    0x700008e78000
rcx    0x700008e77ad8
rdx    0x0
rdi    0x903
rsi    0x6
rbp    0x700008e77b00
rsp    0x700008e77ad8
r8     0x0
r9     0x0
r10    0x700008e78000
r11    0x246
r12    0x903
r13    0x50
r14    0x6
r15    0x16
rip    0x7fff2061092e
rflags 0x246
cs     0x7
fs     0x0
gs     0x0
exit status 2

cannot use _Ctype_ulong(s) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration

v1.7.7 rocksdb v7.5.3
can not build in windows

[email protected]\cache.go:79:74: cannot use _Ctype_ulong(s) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration
[email protected]\options.go:1569:96: cannot use _Ctype_ulong(windowSize) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration
[email protected]\options.go:1569:117: cannot use _Ctype_ulong(numDelsTrigger) (value of type _Ctype_ulong) as type _Ctype_ulonglong in variable declaration

`options.SetPrefixExtractor(grocksdb.NewNoopPrefixTransform())` maybe coredump

code:

package main

import (
	"os"

	"github.com/linxGnu/grocksdb"
)

func main() {
	dir, err := os.MkdirTemp(".", "t-")
	CheckErr(err)

	options := grocksdb.NewDefaultOptions()
	options.SetCreateIfMissing(true)
	options.SetPrefixExtractor(grocksdb.NewNoopPrefixTransform())

	db, err := grocksdb.OpenDb(options, dir)
	CheckErr(err)

	db.Close()
	options.Destroy()
}

func CheckErr(err error) {
	if err != nil {
		panic(err)
	}
}

result:

  • ubuntu 20.04/22.04 amd64: ok
  • oracle linux 7.9 amd64: core dump

ld: library not found for -lrocksdb

enviroment:
Environment: MacOS Ventura with Apple Silicon
Both RocksDB(7.7.3) and its dependencies (including snappy) are installed via Homebrew.
using grocksdb 1.7.12

compile command:

CGO_CFLAGS="-I/opt/homebrew/Cellar/rocksdb/7.7.3/include" \
CGO_LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/7.7.3 -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd" \
  go build

got this error

/usr/local/go/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
ld: library not found for -lrocksdb
clang: error: linker command failed with exit code 1 (use -v to see invocation)

any suggesstions?

Shared lib work in ubuntu but not in docker based alpine

Same code and dockerfile, github.com/tecbot/gorocksdb works, grocksdb not. Does it support alpine?

Development environment works:

Ubuntu 20.04.1 Linux m-pc 5.4.0-58-generic #64-Ubuntu SMP Wed Dec 9 08:16:25 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
RocksDB 6.14.6 build from source use PORTABLE=1 make shared_lib
github.com/linxGnu/grocksdb v1.6.28(with mod)

Dockerfile not:

FROM registry.cn-shanghai.aliyuncs.com/xm69/alpine-rocksdb-apk AS app
WORKDIR /home
COPY code .
ENV GOPROXY=https://goproxy.cn
RUN set -eux && \
  #设置源
  echo "http://mirrors.ustc.edu.cn/alpine/edge/main/" > /etc/apk/repositories && \
  echo "http://mirrors.ustc.edu.cn/alpine/edge/community/" >> /etc/apk/repositories && \
  echo "http://mirrors.ustc.edu.cn/alpine/edge/testing/" >> /etc/apk/repositories && \
  apk update && \
  #安装RocksDB
  apk add --allow-untrusted /rocksdb.apk /rocksdb-dev.apk && \
  #打印编译环境
  echo "编译环境：$(uname -a)" && \
  #安装Golang环境
  apk add build-base go && \
  #下载并校验Golang模块
  go mod download && go mod verify && \
  #编译Golang应用
  go build -o /home/main

registry.cn-shanghai.aliyuncs.com/xm69/alpine-rocksdb-apk

If use Static lib, build failure. Without -tags build ok, but can not run.

Error pthread set mutex attr: Invalid argument SIGABRT: abort PC=0x7f4cf3f92a71 m=0 sigcode=18446744073709551610

goroutine 0 [idle]:
runtime: unknown pc 0x7f4cf3f92a71
stack: frame={sp:0x7ffe49c331b8, fp:0x0} stack=[0x7ffe49c15480,0x7ffe49c344c0)
00007ffe49c330b8: 000000000000000a 0000000000000000
00007ffe49c330c8: 00007ffe49c33100 0000000000002710
00007ffe49c330d8: 0000000000000000 0000000000000000
00007ffe49c330e8: 00000000009a8000 0000000000000000
00007ffe49c330f8: 0000000000000000 0000000000000000
00007ffe49c33108: 00007f4cf3f70eac 0000000000800000 <crypto/tls.(*Config).Clone+32>
00007ffe49c33118: 0000000000000380 000000000000001b
00007ffe49c33128: 000000000000001b 0000001400000007
00007ffe49c33138: 00000000022f2c90 0000000000000004
00007ffe49c33148: 00007f4cf3f70eac 00000000f3fe0d88
00007ffe49c33158: 0000000000000020 0000000000000000
00007ffe49c33168: 00007f4c00000000 0000000000000026
00007ffe49c33178: 00000000022f46b0 0000000000000001
00007ffe49c33188: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33198: 00000000022f4328 00000000013d59f0
00007ffe49c331a8: 00000000013d7c40 00000000022f4328
00007ffe49c331b8: <00007f4cf3f92bcc 0000000000000000
00007ffe49c331c8: 0000003000000010 00007ffe49c332a0
00007ffe49c331d8: 00007ffe49c331e0 00000000f3fe37c4
00007ffe49c331e8: 000000000000001f 000000000101dbcb
00007ffe49c331f8: 00007f4cf3fb1834 0000000000000016
00007ffe49c33208: 00000000022f4320 0000000000000001
00007ffe49c33218: 0000000000000040 0000000000000001
00007ffe49c33228: 0000000000000001 0000000000000040
00007ffe49c33238: 00000000022f4310 0000000000000008
00007ffe49c33248: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33258: 00007f4cf3f6a713 00007f4cf3fe0a80
00007ffe49c33268: 00007f4cf3f71539 00007ffe49c33370
00007ffe49c33278: 0000000000b11713 00007ffe49c33310
00007ffe49c33288: 0000000000000120 000000000101dbcb
00007ffe49c33298: 0000000000517a38 0000000000000016
00007ffe49c332a8: 00007ffe49c332c0 00007ffe49c332e0
runtime: unknown pc 0x7f4cf3f92a71
stack: frame={sp:0x7ffe49c331b8, fp:0x0} stack=[0x7ffe49c15480,0x7ffe49c344c0)
00007ffe49c330b8: 000000000000000a 0000000000000000
00007ffe49c330c8: 00007ffe49c33100 0000000000002710
00007ffe49c330d8: 0000000000000000 0000000000000000
00007ffe49c330e8: 00000000009a8000 0000000000000000
00007ffe49c330f8: 0000000000000000 0000000000000000
00007ffe49c33108: 00007f4cf3f70eac 0000000000800000 <crypto/tls.(*Config).Clone+32>
00007ffe49c33118: 0000000000000380 000000000000001b
00007ffe49c33128: 000000000000001b 0000001400000007
00007ffe49c33138: 00000000022f2c90 0000000000000004
00007ffe49c33148: 00007f4cf3f70eac 00000000f3fe0d88
00007ffe49c33158: 0000000000000020 0000000000000000
00007ffe49c33168: 00007f4c00000000 0000000000000026
00007ffe49c33178: 00000000022f46b0 0000000000000001
00007ffe49c33188: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33198: 00000000022f4328 00000000013d59f0
00007ffe49c331a8: 00000000013d7c40 00000000022f4328
00007ffe49c331b8: <00007f4cf3f92bcc 0000000000000000
00007ffe49c331c8: 0000003000000010 00007ffe49c332a0
00007ffe49c331d8: 00007ffe49c331e0 00000000f3fe37c4
00007ffe49c331e8: 000000000000001f 000000000101dbcb
00007ffe49c331f8: 00007f4cf3fb1834 0000000000000016
00007ffe49c33208: 00000000022f4320 0000000000000001
00007ffe49c33218: 0000000000000040 0000000000000001
00007ffe49c33228: 0000000000000001 0000000000000040
00007ffe49c33238: 00000000022f4310 0000000000000008
00007ffe49c33248: 000000000101dbcb 00007ffe49c332b0
00007ffe49c33258: 00007f4cf3f6a713 00007f4cf3fe0a80
00007ffe49c33268: 00007f4cf3f71539 00007ffe49c33370
00007ffe49c33278: 0000000000b11713 00007ffe49c33310
00007ffe49c33288: 0000000000000120 000000000101dbcb
00007ffe49c33298: 0000000000517a38 0000000000000016
00007ffe49c332a8: 00007ffe49c332c0 00007ffe49c332e0

goroutine 1 [syscall]:
runtime.cgocall(0x93e930, 0xc0000cfd88, 0xc0000b6580)
/usr/lib/go/src/runtime/cgocall.go:133 +0x5b fp=0xc0000cfd58 sp=0xc0000cfd20 pc=0x53d37b
github.com/linxGnu/grocksdb._Cfunc_rocksdb_open(0x22d06a0, 0x22d2120, 0xc0000b6580, 0x0)
_cgo_gotypes.go:3395 +0x4e fp=0xc0000cfd88 sp=0xc0000cfd58 pc=0x7528ae
github.com/linxGnu/grocksdb.OpenDb.func1(0xc0000afa80, 0x22d2120, 0xc0000b6580, 0xc0000ec000)
/root/go/pkg/mod/github.com/linx!gnu/[email protected]/db.go:37 +0xa7 fp=0xc0000cfdc0 sp=0xc0000cfd88 pc=0x775347
github.com/linxGnu/grocksdb.OpenDb(0xc0000afa80, 0xc0000b4d80, 0xc, 0xc0000b4d80, 0xc, 0x0)
/root/go/pkg/mod/github.com/linx!gnu/[email protected]/db.go:37 +0x7a fp=0xc0000cfe20 sp=0xc0000cfdc0 pc=0x762e3a
main.main()
/home/main.go:673 +0x870 fp=0xc0000cff88 sp=0xc0000cfe20 pc=0x93bff0
runtime.main()
/usr/lib/go/src/runtime/proc.go:204 +0x209 fp=0xc0000cffe0 sp=0xc0000cff88 pc=0x571a69
runtime.goexit()
/usr/lib/go/src/runtime/asm_amd64.s:1374 +0x1 fp=0xc0000cffe8 sp=0xc0000cffe0 pc=0x5a3461

rax 0x0
rbx 0x0
rcx 0x7f4cf3f92a71
rdx 0x0
rdi 0x2
rsi 0x7ffe49c331c0
rbp 0x7ffe49c331c0
rsp 0x7ffe49c331b8
r8 0x7f4cf3fe00c0
r9 0x7f4cf3fdd933
r10 0x8
r11 0x246
r12 0x22f4328
r13 0x13d59f0
r14 0x13d7c40
r15 0x22d26f8
rip 0x7f4cf3f92a71
rflags 0x246
cs 0x33
fs 0x0
gs 0x0

Include library files in repository

I have a recommendation on how we can improve the build and stability of this library. Our RocksDB library files (librocksdb.a) depend on the operating system's RocksDB files, which can cause inconsistencies. For instance, if you have Ubuntu 23.04 on your local machine with version librocksdb7.8 and GitHub Action's ubuntu-latest has version librocksdb6.11, they will both be incompatible with specific versions of grocksdb. We have two options: we can change the grocksdb library to avoid giving errors based on the build and version particularities of the currently installed library, or we can have each version of the library directly usable and have the same library version. This method has been successfully used in other libraries like confluent-kafka-go. You can check out the libraries at https://github.com/confluentinc/confluent-kafka-go/tree/master/kafka/librdkafka_vendor.

1.6.34 build scripts.

Build is failing and package ignores -tags builtin_shared and defaults to only ever using the prebuilt libs in ./dist

Why are binaries included in the SOURCE package?

The libs included are compiled for a system that has AVX2 support, meaning that when used on any system that doesn't have AVX2, we get "Illegal Instruction" errors.

curious question, mind enlightening regarding batching?

for batching writes (PUT), do I put the PutCF before the mutex lock? (faster) or after the mutex lock? can i do it before the mutex lock? the second version. is it ok?

first version

                b0 := grocksdb.NewWriteBatch()

                mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
                mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
                b0.PutCF(cfhTK[((*i.(*[][]uint8))[2][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[5], (*i.(*[][]uint8))[6])
                b0.PutCF(cfhTK[((*i.(*[][]uint8))[3][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[6][5:], (*i.(*[][]uint8))[7])
                dbr[((*i.(*[][]uint8))[1][0])].Write(wodb[((*i.(*[][]uint8))[1][0])], b0)
                mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()
                mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()

                b0.Destroy()

second version

                b0 := grocksdb.NewWriteBatch()
                b0.PutCF(cfhTK[((*i.(*[][]uint8))[2][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[5], (*i.(*[][]uint8))[6])
                b0.PutCF(cfhTK[((*i.(*[][]uint8))[3][0])+((((*i.(*[][]uint8))[4][0]))%3)], (*i.(*[][]uint8))[6][5:], (*i.(*[][]uint8))[7])

                mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
                mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Lock()
                dbr[((*i.(*[][]uint8))[1][0])].Write(wodb[((*i.(*[][]uint8))[1][0])], b0)
                mutexCfhTK[((*i.(*[][]uint8))[3][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()
                mutexCfhTK[((*i.(*[][]uint8))[2][0])+(((*i.(*[][]uint8))[4][0])%3)].Unlock()

                b0.Destroy()

Support for Eventlistener?

Hello!

First I really like to thank you for your awesome work.
This is outstanding!

We will switch our open source project after months of trying badger, boltdb, pebbledb to your lib I guess.
For our usecase rocksdb it much faster with less CPU and Memory usage.

The only thing I can't find is a way to register the EventListener:
https://github.com/facebook/rocksdb/blob/29e8f6a698acf0d00d88f21a2692fcdd974a74e5/include/rocksdb/listener.h#L358

Do I miss something, or is there no way yet?

Especially the compaction events would be nice for our usecase, to know when compaction is running in the background:

https://github.com/facebook/rocksdb/blob/29e8f6a698acf0d00d88f21a2692fcdd974a74e5/include/rocksdb/listener.h#L400
https://github.com/facebook/rocksdb/blob/29e8f6a698acf0d00d88f21a2692fcdd974a74e5/include/rocksdb/listener.h#L415

Can you give me a hint how to implement this if it is missing?
Thank you for your help!

can u verify opening 12 column families and not crashing?

cfName is 12 items, if i created 11, it will work, but at 12, it will crash. can u try?

dbr, cfh, err = gorocksdb.OpenDbColumnFamilies(opts[i], "/d/a", cfNames, []*gorocksdb.Options{opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i], opts[i]})

crashed at this last line

// OpenDbColumnFamilies opens a database with the specified column families.
func OpenDbColumnFamilies(
        opts *Options,
        name string,
        cfNames []string,
        cfOpts []*Options,
) (db *DB, cfHandles []*ColumnFamilyHandle, err error) {
        numColumnFamilies := len(cfNames)
        if numColumnFamilies != len(cfOpts) {
                err = ErrColumnFamilyMustMatch
                return
        }

        cName := C.CString(name)}

// OpenDbColumnFamilies opens a database with the specified column families.
func OpenDbColumnFamilies(
        opts *Options,
        name string,
        cfNames []string,
        cfOpts []*Options,
) (db *DB, cfHandles []*ColumnFamilyHandle, err error) {
        numColumnFamilies := len(cfNames)
        if numColumnFamilies != len(cfOpts) {
                err = ErrColumnFamilyMustMatch
                return
        }

        cName := C.CString(name)
        cNames := make([]*C.char, numColumnFamilies)
        for i, s := range cfNames {
                cNames[i] = C.CString(s)
        }

        cOpts := make([]*C.rocksdb_options_t, numColumnFamilies)
        for i, o := range cfOpts {
                cOpts[i] = o.c
        }

        cHandles := make([]*C.rocksdb_column_family_handle_t, numColumnFamilies)

        var cErr *C.char
        _db := C.rocksdb_open_column_families(
                opts.c,
                cName,
                C.int(numColumnFamilies),
                &cNames[0],
                &cOpts[0],
                &cHandles[0],
                &cErr,

        cNames := make([]*C.char, numColumnFamilies)
        for i, s := range cfNames {
                cNames[i] = C.CString(s)
        }

        cOpts := make([]*C.rocksdb_options_t, numColumnFamilies)
        for i, o := range cfOpts {
                cOpts[i] = o.c
        }

        cHandles := make([]*C.rocksdb_column_family_handle_t, numColumnFamilies)

        var cErr *C.char
        _db := C.rocksdb_open_column_families(
                opts.c,
                cName,
                C.int(numColumnFamilies),
                &cNames[0],
                &cOpts[0],
                &cHandles[0],
                &cErr,

Error: Library Not Found

I am runnig this comand as per Documentation:

CGO_CFLAGS="-I/opt/homebrew/Cellar/rocksdb/8.5.4/include" \
CGO_LDFLAGS="-L/opt/homebrew/Cellar/rocksdb/8.5.4 -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd" \
  go build

But Geeting this error

/usr/local/go/pkg/tool/darwin_arm64/link: running clang failed: exit status 1
ld: library not found for -lsnappy
clang: error: linker command failed with exit code 1 (use -v to see invocation)

What is the alternate to this ?

Documentation fix: Add -lbz2 link flag

On ubuntu22, I had to add -lbz2 to the CGO_LDFLAGS to successfully run go build

I'd suggest to add it in the Readme build flags:

CGO_CFLAGS="-I/path/to/rocksdb/include" \
CGO_LDFLAGS="-L/path/to/rocksdb -lrocksdb -lstdc++ -lm -lz -lsnappy -llz4 -lzstd -lbz2" \
  go build

Thanks fro your work

what is the this : Default column family not specified

how do i specify default column family name?

                                handleVer = value.Data()[6:14]
                                handleVerStr = strconv.FormatUint(my_func.BytesToUint64(handleVer),10)

                                var cfHandles []*gorocksdb.ColumnFamilyHandle
                                dbM, cfHandles, err = gorocksdb.OpenDbColumnFamilies(optsM, defaultDBDir+"/"+metaDB, []string{handleVerStr}, []*gorocksdb.Options{optsM})
                                if err != nil {
                                        fmt.Printf("%v = %s\n",err,handleVerStr) // failed here
                                }
                                cfHandle = cfHandles[0]

Documentation improvments: When should Destroy be called for db option types ?

HI,

First thank you for maintaining this binding.

I have a small question regarding where one should call Destroy() on option objects.

I have the following code

opts := grocksdb.NewDefaultOptions()
tableOpts := grocksdb.NewDefaultBlockBasedTableOptions()
cache := grocksdb.NewLRUCache(8 * 1024 * 1024)
tableOpts.SetBlockCache(cache)
opts.SetBlockBasedTableFactory(tableOpts)

db := grocksdb.OpenDb(opts ,"foo.rocksdb")
// ...
db.Close()

At which point in this code can I safely call Destroy() on opts, tableOpts and cache ?
Should I wait after the db is closed or can I destroy them after the db is opened ?

Compiler error on arm64 / rpi

I´m trying to compile the project mentioned in #28 on a raspberry Pi4 running ubuntu 20.10 arm64.

The compiler interrupts with

/root/go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:44:13: could not determine kind of name for C.rocksdb_backup_engine_open_opts /root/go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:136:2: could not determine kind of name for C.rocksdb_backup_engine_restore_db_from_backup

Do you have any idea how to get around this?

support a non-hacky way to override link flags

I have more situations that need to customize the link flags, for example recently I cross compiled the app to mingwW64 with the help of nixpkgs, but there I have to link with -lrocksdb-shared.
currently there's a way to suppress the builtin link flags, which is the testing build tag, it works perfectly right now, it's just the name feels hacky, shall we support sth like grocksdb_no_link build tag, where there's no builtin link flags, user can supply with CGO_LDFLAGS env var?

Is it safe enough to use refGoBytes()

Hi, in the Get() function, it uses refGoBytes() to convert []bytes to C.char, involving the use of unsafe.Pointer. Is it possible that a key has been garbage collected, posing a risk of cKey pointing to null?

installed latest rocksdb

CGO_ENABLED=1 CGO_CFLAGS="-I/usr/local/include" CGO_LDFLAGS="-L/usr/local/lib -lrocksdb -lz -lsnappy -llz4 -lzstd -lbz2" go build rocskdbmain.go

# command-line-arguments
/usr/local/go/pkg/tool/linux_amd64/link: running gcc failed: exit status 1
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::CreateIOUring()':
/usr/local/src/rocksdb/./env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::Poll(std::vector<void*, std::allocator<void*> >&, unsigned long)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::AbortIO(std::vector<void*, std::allocator<void*> >&)':
/usr/local/src/rocksdb/env/fs_posix.cc:1109: undefined reference to `io_uring_get_sqe'
/usr/bin/ld: /usr/local/src/rocksdb/env/fs_posix.cc:1118: undefined reference to `io_uring_submit'
/usr/bin/ld: /usr/local/lib/librocksdb.a(fs_posix.o): in function `rocksdb::(anonymous namespace)::PosixFileSystem::AbortIO(std::vector<void*, std::allocator<void*> >&)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::ReadAsync(rocksdb::FSReadRequest&, rocksdb::IOOptions const&, std::function<void (rocksdb::FSReadRequest const&, void*)>, void*, void**, std::function<void (void*)>*, rocksdb::IODebugContext*)':
/usr/local/src/rocksdb/env/io_posix.cc:901: undefined reference to `io_uring_get_sqe'
/usr/bin/ld: /usr/local/src/rocksdb/env/io_posix.cc:910: undefined reference to `io_uring_submit'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::CreateIOUring()':
/usr/local/src/rocksdb/./env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/usr/local/src/rocksdb/env/io_posix.cc:674: undefined reference to `io_uring_get_sqe'
/usr/bin/ld: /usr/local/src/rocksdb/env/io_posix.cc:684: undefined reference to `io_uring_submit_and_wait'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::CreateIOUring()':
/usr/local/src/rocksdb/./env/io_posix.h:272: undefined reference to `io_uring_queue_init'
/usr/bin/ld: /usr/local/lib/librocksdb.a(io_posix.o): in function `rocksdb::PosixRandomAccessFile::MultiRead(rocksdb::FSReadRequest*, unsigned long, rocksdb::IOOptions const&, rocksdb::IODebugContext*)':
/usr/include/liburing.h:745: undefined reference to `__io_uring_get_cqe'
collect2: error: ld returned 1 exit status

DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=22.04
DISTRIB_CODENAME=jammy
DISTRIB_DESCRIPTION="Ubuntu 22.04.1 LTS"

Linux host 5.15.0-73-generic #80-Ubuntu SMP

Add rocksdb sources to the go package

Hi y'all,

I am trying to use this package for my go project and I found its installation/setup procedure quite complicated. By default the package gives following compilation error:

➜  rocksdb go build
# github.com/linxGnu/grocksdb
../../go/pkg/mod/github.com/linx!gnu/[email protected]/array.go:4:11: fatal error: rocksdb/c.h: No such file or directory
    4 | // #include "rocksdb/c.h"
      |           ^~~~~~~~~~~~~
compilation terminated.

To use grocksdb bindings additional steps are required to compile rocksdb and its dependencies.

A better solution would be to let cgo compile the rocksdb sources. Similar to how sqlite3 go bindings do. cgo just picks *.c files stored together with go sources then compiles and automatically links it.

This requires an amalgamation file (i.e. a single file that contains all project sources) to be provided by rocksdb project (like sqlite does). It turns out rocksdb has a tool for that https://github.com/facebook/rocksdb/blob/main/build_tools/amalgamate.py that aim to achieve what we need.

Incorrect binding for MultiGetWithCF

How to reproduce: 
package main

import (
	"fmt"

	"github.com/linxGnu/grocksdb"
)

const dbPath = "/tmp/rocksdb"

var allCFs = []string{"default", "cf1", "cf2"}

var keys = [][]byte{[]byte("k1"), []byte("k2")}
var values = [][]byte{[]byte("v1"), []byte("v2")}

func main() {
	opts := grocksdb.NewDefaultOptions()
	opts.SetCreateIfMissing(true)
	opts.SetCreateIfMissingColumnFamilies(true)

	dbOpts := grocksdb.NewDefaultTransactionDBOptions()
	perTableOpts := make([]*grocksdb.Options, len(allCFs))
	for i := range perTableOpts {
		perTableOpts[i] = grocksdb.NewDefaultOptions()
		perTableOpts[i].SetAtomicFlush(true)
	}

	db, cfs, err := grocksdb.OpenTransactionDbColumnFamilies(
		opts,
		dbOpts,
		dbPath,
		allCFs,
		perTableOpts,
	)

	if err != nil {
		panic(fmt.Sprintf("failed to open db : %s", err.Error()))
	}

	for i := range keys {
		err = db.PutCF(grocksdb.NewDefaultWriteOptions(), cfs[1], keys[i], values[i])
		if err != nil {
			panic(fmt.Sprintf("failed to insert key : %s", err.Error()))
		}
	}

	data, err := db.MultiGetWithCF(grocksdb.NewDefaultReadOptions(), cfs[1], keys...)
	if err != nil {
		panic(fmt.Sprintf("failed to run MultiGetWithCF : %s", err.Error()))
	}
	for i := range data {
		fmt.Printf("fetched data[%d] is : %s\n", i, data[i].Data())
	}
	// Moreover if you specify only one non-default CF call to MultiGetWithCF will cause segfault
	// This can be easily checked by removing `cf2` from `allCFs`
}

The reason of such behaviour is that here we have only one CF. However in rocksdb lib array of CFs is expected

Fix looks quite simple 
	cfs := make(ColumnFamilyHandles, len(keys))
	for i := range keys {
		cfs[i] = cf
	}

	C.rocksdb_transactiondb_multi_get_cf(
		transaction.c,
		opts.c,
		cfs.toCSlice().c(),
		C.size_t(len(keys)),
		cKeys.c(),
		cKeySizes.c(),
		vals.c(),
		valSizes.c(),
		rocksErrs.c(),
	)

Do you accept PRs? :)

Note: there are several such places in the lib

(*WriteBatchIterator).decodeVarint unfortunately doesn’t correctly detect invalid varint sequences and can consume the entire buffer

This is a bug that I reported in July 2022 originally at tecbot/gorocksdb#221 and it is still relevant to this repo:

Courtesy of the Cosmos Network Security, I discovered this code while auditing supply chain dependencies.
If we examine this code in

grocksdb/write_batch.go

Lines 344 to 362 in 254f68b

func (iter *WriteBatchIterator) decodeVarint() uint64 {
var n int
var x uint64
for shift := uint(0); shift < 64 && n < len(iter.data); shift += 7 {
b := uint64(iter.data[n])
n++
x |= (b & 0x7F) << shift
if (b & 0x80) == 0 {
iter.data = iter.data[n:]
return x
}
}
if n == len(iter.data) {
iter.err = io.ErrShortBuffer
} else {
iter.err = errors.New("malformed varint")
}
return 0
}

and then extract it and run the following https://go.dev/play/p/MXpanwJKofY or inlined below:

package main

import (
	"errors"
	"fmt"
	"io"
)

type wbi struct {
	data []byte
	err  error
}

func (iter *wbi) decodeVarint() uint64 {
	var n int
	var x uint64
	for shift := uint(0); shift < 64 && n < len(iter.data); shift += 7 {
		b := uint64(iter.data[n])
		n++
		x |= (b & 0x7F) << shift
		if (b & 0x80) == 0 {
			iter.data = iter.data[n:]
			return x
		}
	}
	if n == len(iter.data) {
		iter.err = io.ErrShortBuffer
	} else {
		iter.err = errors.New("malformed varint")
	}
	return 0
}

func main() {
	// The 10th byte here is invalid and should be reported.
	// Please see https://github.com/golang/go/issues/41185
	b := []byte{0xd7, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f}
	wb := &wbi{data: b}
	v := wb.decodeVarint()
	println(v)
	fmt.Printf("%v\n", wb.err) // This should have reported an error.
}

which prints

18446744073709551575
<nil>

Fix

The fix entails just using the Go standard library's code in which I fixed this bug in March 2021 in the Go standard library per golang/go@aafad20

with a demo at https://go.dev/play/p/kYGBjzh24Qx which should properly print malformed varint

package main

import (
	"encoding/binary"
	"errors"
	"fmt"
	"io"
)

type wbi struct {
	data []byte
	err  error
}

func (iter *wbi) decodeVarint() uint64 {
	var n int
	var x uint64
	for shift := uint(0); shift < 64 && n < len(iter.data); shift += 7 {
		b := uint64(iter.data[n])
		n++
		x |= (b & 0x7F) << shift
		if (b & 0x80) == 0 {
			iter.data = iter.data[n:]
			return x
		}
	}
	if n == len(iter.data) {
		iter.err = io.ErrShortBuffer
	} else {
		iter.err = errors.New("malformed varint")
	}
	return 0
}

func (iter *wbi) decodeVarint_good() (x uint64) {
	v, n := binary.Varint(iter.data)
	if n == len(iter.data) {
		iter.err = io.ErrShortBuffer
	} else if n < 0 {
		iter.err = errors.New("malformed varint")
	} else {
		x = uint64(v)
	}
	return x
}

func main() {
	// The 10th byte here is invalid and should be reported.
	// Please see https://github.com/golang/go/issues/41185
	b := []byte{0xd7, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0xff, 0x7f}
	wb := &wbi{data: b}
	v := wb.decodeVarint_good()
	println(v)
	fmt.Printf("%v\n", wb.err) // This should have reported an error.
}

which correctly prints

0
malformed varint

/cc @elias-orijtech @julienrbrt

could not determine kind of name for C.rocksdb_backup_engine_open_opts

I got these errors when running go build.

# github.com/linxGnu/grocksdb
../../go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:44:13: could not determine kind of name for C.rocksdb_backup_engine_open_opts
../../go/pkg/mod/github.com/linx!gnu/[email protected]/backup.go:136:2: could not determine kind of name for C.rocksdb_backup_engine_restore_db_from_backup

I am using Ubuntu 22.04 LTS, librocksdb-dev (6.11.4-3) from apt and [email protected].

Any help would be appreciated!

how do u do v:=value.Data() , value.Free() without losing "v" value?

how do u do
v:=value.Data()
value.Free()

fmt.Printf("v = %v\n",v) // <- from here onwards, "v" is lost. how to prevent v from being lost?
this shld be mentioned as part of the caveats of this module coz it took me some time to realise this was the case and it was not obvious

        value, _ := db.Get(ro, k)
        //v := value.Data()
        b := make([]byte,value.Size())
        copy(b,value.Data())
        //defer value.Free()
        //fmt.Printf("gv = %v = %v = %v\n",v,value.Data(),b)
        value.Free()
        //fmt.Printf("gv1 = %v = %v = %v\n",v,value.Data(),b)
        return b

Custom Comparator in Go

Comparator interface needs Native method

Native() *C.rocksdb_comparator_t

Cgo translates C types into equivalent unexported Go types. Because the translations are unexported, a Go package should not expose C types in its exported API: a C type used in one Go package is different from the same C type used in another.

Because of this, it is not possible to implement this interface.

Help for dummies? ./cache.go:25:12: could not determine kind of name for C.rocksdb_cache_create_hyper_clock

Hi. I have gorocksdb as a dependency for other project and experiencing compiling issues that are difficult to pinpoint.

System: Ubuntu 20

This folder has all the includes
echo $CGO_CFLAGS
-I/include/rocksdb/

This is the rocksdb repository with compiled lib
echo $CGO_LDFLAGS
-L/root/dev/rocksdb -lrocksdb -lstdc++ -lm -lz -lbz2 -lsnappy -llz4 -lzstd

Getting:

go build

github.com/linxGnu/grocksdb

./cache.go:25:12: could not determine kind of name for C.rocksdb_cache_create_hyper_clock
./cache.go:31:12: could not determine kind of name for C.rocksdb_cache_create_hyper_clock_opts
./cache.go:148:6: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_create
./cache.go:204:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_destroy
./cache.go:154:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_capacity
./cache.go:189:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_estimated_entry_charge
./cache.go:199:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_memory_allocator
./cache.go:194:2: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_set_num_shard_bits
./cache.go:142:5: could not determine kind of name for C.rocksdb_hyper_clock_cache_options_t

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.