Giter VIP home page Giter VIP logo

shardingdb's Introduction

shardingdb

ShardingDB is an open-source, sharded database enhancing LevelDB with concurrent reads/writes support. It significantly improves performance, boosting PutData by 60x and GetData by 7x, making it an ideal drop-in replacement for LevelDB.

Requirements

  • Requires at least go1.14 or newer.

How to use

1. Resharding

1.0 Build the resharding tool

make
cd bin

1.1 Migrate data from LevelDB to new shardingdb

For example, if you have 1 LevelDB data and want to migrate it to 3 shardingdb data, print summary log(1), you can run the following command:

./resharding -i /data1 -o /newfolder1,/newfolder2,/newfolder3 -l 1

1.2 Add sharding db

For example, if you have 1 LevelDB data and want to add 2 more LevelDB folders to shardingdb, print no log(0), you can run the following command:

```bash
./resharding -i /data1 -o /data1,/data2,/data3 

For example, if you have 3 LevelDB data and want to add 1 more LevelDB folder to shardingdb, print detail log(2), you can run the following command:

```bash
./resharding -i /data1,/data2,/data3 -o /data1,/data2,/data3,/data4 -l 2

2. Code example

2.0 Get the package

go get github.com/studyzy/shardingdb

2.1 Import the package

import "github.com/studyzy/shardingdb"

2.2 Use shardingdb

inputPathList := []string{"/data1", "/data2"}
sdb, err := shardingdb.OpenFile(inputPathList, nil)
sdb.Put([]byte("key"), []byte("value"), nil)
sdb.Get([]byte("key"), nil)
...

2.3 Another example

db1, err := leveldb.OpenFile(getTempDir(), nil)
if err != nil {
    t.Fatal(err)
}
db2, err := leveldb.OpenFile(getTempDir(), nil)
if err != nil {
    t.Fatal(err)
}
// Create a new sharding db
sdb, err := shardingdb.NewShardingDb(shardingdb.WithDbHandles(db1,db2), shardingdb.WithShardingFunc(MurmurSharding))
...

Performance Benchmark

Environment

  • Intel(R) Xeon(R) Platinum 8255C CPU @ 2.50GHz * 10 Core
  • 40GB RAM
  • 3 SSD: /data, /data1, /

generate data by command:

go test -timeout 60m -run "TestCompareDbPerformance"

Test case: total 1000000 key-value pairs, 100 go routines, 100 key-value pairs per batch. result means the time cost(second) of the whole operation.

1. PutData

Data Size LevelDB ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(encrypt 3 folders)
100B 2.27 0.659 0.581 0.953
200B 4.45 1.07 0.683 1.9
500B 15.3 3.36 1.49 6.4
1KB 48.9 9.42 3.74 17.69
10KB 1117 351 123 308

2. GetData

Data Size LevelDB ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(encrypt 3 folders)
100B 2.23 1.25 1.02 1.86
200B 3.09 1.42 1.27 2.24
500B 4.17 1.91 1.62 3.73
1KB 7.97 2.37 2.26 4.53
10KB 12.75 9.54 11.03 13.85

3. GetData not found

Data Size LevelDB ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(encrypt 3 folders)
100B 2.14 1.36 0.87 1.43
200B 2.07 1.47 0.9 1.6
500B 2.05 1.51 0.93 1.81
1KB 2.35 1.64 0.891 2.28
10KB 8.68 5.56 2.48 7.75

4. DeleteData

Data Size LevelDB ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(encrypt 3 folders)
100B 3.82 2.76 1.02 1.72
200B 3.81 1.71 1.02 1.74
500B 3.85 1.76 1.05 1.69
1KB 3.84 1.72 1.04 1.74
10KB 3.844 1.78 1.06 1.76

5. Iterator

Data Size LevelDB ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(encrypt 3 folders)
100B 0.133 0.184 0.222 0.18
200B 0.151 0.246 0.246 0.191
500B 0.282 0.351 0.41 0.344
1KB 0.514 0.419 0.472 0.541
10KB 2.46 2.39 1.96 2.3

6. Sharding count compare

run command:

go test  -timeout 60m -run "TestCompareShardingCountPerformance"

6.1 PutData

Data Size ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(9 folders) ShardingDB(30 folders) ShardingDB(60 folders)
100B 0.659 0.581 0.506 0.564 0.728
200B 1.07 0.683 0.624 0.685 0.782
500B 3.36 1.49 1.20 1.18 1.21
1KB 9.42 3.74 2.33 1.92 1.96
10KB 351 123 54 26 18.2

6.2 GetData

Data Size ShardingDB(3 folders) ShardingDB(6 folders) ShardingDB(9 folders) ShardingDB(30 folders) ShardingDB(60 folders)
100B 1.25 1.02 1.03 0.343 0.366
200B 1.42 1.27 1.01 0.66 0.373
500B 1.91 1.62 1.21 0.96 1.34
1KB 2.37 2.26 1.83 1.18 1.19
10KB 9.54 11.03 7.67 4.8 3.4

Most interfaces are the same as goleveldb. For my interface definition, please refer to DbHandle.

shardingdb's People

Contributors

studyzy avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.