Giter VIP home page Giter VIP logo

lmdbjni's Introduction

LMDB JNI

Build Status Coverage Status Coverity Scan Build Status

LMDB JNI provide a Java API to LMDB which is an ultra-fast, ultra-compact key-value embedded data store developed by Symas for the OpenLDAP Project. It uses memory-mapped files, so it has the read performance of a pure in-memory database while still offering the persistence of standard disk-based databases. Transactional with full ACID semantics and crash-proof by design. No corruption. No startup time. Zero-config cache tuning. No dependencies. Proven in production applications.

LMDB JNI is available for 64 bit Linux, OSX, Windows and Android.

About this fork

This is a fork of https://github.com/deephacks/lmdbjni at version 0.4.5 to

  • Simplify building for Android with gradle
  • Make it compatible with Android API 8 (by using Java subset under JDK 1.5)

The native lmdb library has been updated to Dec 2016, after merging with lmdbjni version 0.4.7.

To build the android library, install gradle (tested on Gradle 2.2) and Android NDK (tested on version r10d), set the environment variable NDK_HOME to point to the ndk directory (without trailing slash), and issue gradle build at the project directory. The build process does not need autoconf / automake tools.

The compiled jar file will be stored at lmdbjni-android-hea/build/libs, and the native library .so files are under respective arch directories. Copy the tree under the libs directory to your Android project libs directory for use in your project. (For Android Studio, the native library directories with the .so files should be placed under src/main/jniLibs instead of under libs.)

To control the target platform and architecture, edit the build.gradle file at the project directory and change the lines with

ndkAppPlatform = "android-8"
ndkAppAbi = "armeabi armeabi-v7a x86"

Documentation

Presentations

Benchmarks

  • [In-Memory Microbenchmark] (http://symas.com/mdb/inmem), June 2014

    Multithreaded read performance for a purely in-memory database.

  • In-Memory Microbenchmark (Scaling/NUMA), September 2014

    Same as above showing performance improvements with numactl --interleave=all enabled.

  • On-Disk Microbenchmark, November 2014

    Multithreaded read performance for a database that is over 5 times larger than the size of RAM.

  • [RxLMDB benchmarks] (https://github.com/deephacks/RxLMDB), July 2015

    Benchmarks using RxJava and LMDB comparing zero copy, various serialization mechanisms, parallel and skip scans.

  • LMDB JNI Microbenchmark, February 2015 (source)

    Row scanning speed per second compared with the Java ports of RocksDB, LevelDB and MapDB. Mongodb is difficult to setup in JMH but de.flapdoodle.embed.mongo indicate that it is around 50x slower than lmdb_zero_copy.

    Benchmark                    Mode  Cnt         Score         Error  Units
    Iteration.leveldb           thrpt   10   6965637.351 ±  784589.894  ops/s
    Iteration.lmdb_buffer_copy  thrpt   10   3157796.643 ±  265830.424  ops/s
    Iteration.lmdb_zero_copy    thrpt   10  16372428.882 ± 1812316.504  ops/s
    Iteration.mapdb             thrpt   10   1358748.670 ±   87502.413  ops/s
    Iteration.rocksdb           thrpt   10   1311441.804 ±  176129.883  ops/s
  • LMDB JNI microbenchmark, February 2016

    Random gets on a database with 370 million entries of 30GiB on a non-SSD drive. Keys 29 bytes and values 8 bytes. The target machine was busy serving traffic and this was the memory usage before executing the test.

    $ free -m
                 total       used       free     shared    buffers     cached
    Mem:         32126      31890        235          0         55       9476
    -/+ buffers/cache:      22359       9767
    Swap:         7627       2350       5277
    

    Percentiles measured in nanoseconds using HdrHistogram.

     min       0.50        .90        0.99      0.999     0.9999        max
    5376      10367      12991      30335      51967      83967      83967
    4608      10175      12799      34559      84991     946175     946175
    3568      10239      12991      33791      70655     107007     107007

### Maven

*Windows, Android and OSX support has been discontinued in lmdbjni 0.4.7 (LMDB 0.9.19) and onward. Users can still build releases at their own convenience, but no artifacts will be published to Maven Central.*

*Please refer to [lmdbjava](https://github.com/lmdbjava/lmdbjava).*

```xml
<!-- required java classes -->

<dependency>
  <groupId>org.deephacks.lmdbjni</groupId>
  <artifactId>lmdbjni</artifactId>
  <version>${lmdbjni.version}</version>
</dependency>

<!-- prebuilt liblmdb platform packages -->

<dependency>
  <groupId>org.deephacks.lmdbjni</groupId>
  <artifactId>lmdbjni-linux64</artifactId>
  <version>${lmdbjni.version}</version>
</dependency>

<dependency>
  <groupId>org.deephacks.lmdbjni</groupId>
  <artifactId>lmdbjni-osx64</artifactId>
  <version>${lmdbjni.version}</version>
</dependency>

<dependency>
  <groupId>org.deephacks.lmdbjni</groupId>
  <artifactId>lmdbjni-win64</artifactId>
  <version>${lmdbjni.version}</version>
</dependency>

<!-- Android 5.0 (API level 21) 64-bit ARM -->
<dependency>
  <groupId>org.deephacks.lmdbjni</groupId>
  <artifactId>lmdbjni-android</artifactId>
  <version>${lmdbjni.version}</version>
</dependency>

Usage

Recommended package imports.

 import org.fusesource.lmdbjni.*;
 import static org.fusesource.lmdbjni.Constants.*;

Opening and closing the database.

 try (Env env = new Env("/tmp/mydb")) {
   try (Database db = env.openDatabase()) {
     ... // use the db
   }
 }

Putting, getting, and deleting key/values.

 db.put(bytes("Tampa"), bytes("rocks"));
 String value = string(db.get(bytes("Tampa")));
 db.delete(bytes("Tampa"));

Iterating and seeking key/values forward and backward.

Transaction tx = env.createReadTransaction();
try (EntryIterator it = db.iterate(tx)) {
  for (Entry next : it.iterable()) {
  }
}

try (EntryIterator it = db.iterateBackward(tx)) {
  for (Entry next : it.iterable()) {
  }
}

byte[] key = bytes("London");
try (EntryIterator it = db.seek(tx, key)) {
  for (Entry next : it.iterable()) {
  }
}

try (EntryIterator it = db.seekBackward(tx, key))) {
  for (Entry next : it.iterable()) {
  }
}
tx.abort();

Performing transactional updates.

 try (Transaction tx = env.createWriteTransaction()) {
   db.delete(tx, bytes("Denver"));
   db.put(tx, bytes("Tampa"), bytes("green"));
   db.put(tx, bytes("London"), bytes("red"));
   tx.commit();  // if commit is not called, the transaction is aborted
 }

Working against a snapshot view of the database using cursors.

 // create a read-only transaction...
 try (Transaction tx = env.createReadTransaction()) {
   
   // All read operations will now use the same 
   // consistent view of the data.
   ... = db.openCursor(tx);
   ... = db.get(tx, bytes("Tampa"));
 }

A cursor in a write-transaction can be closed before its transaction ends, and will otherwise be closed when its transaction ends. A cursor must not be used after its transaction is closed. Both these try blocks are unsafe and may SIGSEGV.

 try (Transaction tx = env.createWriteTransaction();
      Cursor cursor = db.openCursor(tx)) {
   ...
   tx.commit();
 }

 try (Transaction tx = env.createWriteTransaction();
      EntryIterator it = db.iterate(tx)) {
   ...
   tx.commit();
 }

A cursor in a read-only transaction must be closed explicitly, before or after its transaction ends. Both these try blocks are safe.

 try (Transaction tx = env.createReadTransaction();
      Cursor cursor = db.openCursor(tx)) {
 }

 try (Transaction tx = env.createReadTransaction();
      EntryIterator it = db.iterate(tx)) {
 }

Set a custom key comparison function for a database.

 db.setComparator(tx, new Comparator<byte[]>() {
      @Override
      public int compare(byte[] key1, byte[] key2) {
        // do compare
      }
    });

Atomic hot backup.

 env.copy(backupPath);

Using a memory pool to make native memory allocations more efficient:

 Env.pushMemoryPool(1024 * 512);
 try {
     // .. work with the DB in here, 
 } finally {
     Env.popMemoryPool();
 }

Zero copy usage

The safest (and least efficient) approach for interacting with LMDB JNI is using buffer copy as shown above. BufferCursor is a more efficient, zero copy mode. This mode is not available on Android.

There are also methods that give access to DirectBuffer, but users should avoid interacting directly with these and use the BufferCursor API instead. Otherwise take extra care of buffer memory address+size and byte ordering. Mistakes may lead to SIGSEGV or unpredictable key ordering etc.

 // read only
 try (Transaction tx = env.createReadTransaction(); 
      BufferCursor cursor = db.bufferCursor(tx)) {
   // iterate from first item and forwards
   if (cursor.first()) {
     do {
       // read a position in buffer
       cursor.keyByte(0);
       cursor.valByte(0);
     } while (cursor.next());
   }

   // iterate from last item and backwards
   if (cursor.last()) {
     do {
       // copy entire buffer
       cursor.keyBytes();
       cursor.valBytes();
     } while (cursor.prev());
   }
   
   // find entry matching exactly the provided key
   cursor.keyWriteBytes(bytes("Paris"));
   if (cursor.seekKey()) {
     // read utf-8 string from position until NULL byte
     cursor.valUtf8(0);
   }

   // find first key greater than or equal to specified key.
   cursor.keyWriteBytes(bytes("London"));
   if (cursor.seekRange()) {
     // read utf-8 string from position until NULL byte
     cursor.keyUtf8(0);
     cursor.valUtf8(0);
   }
 }
 
 // open for write
 try (Transaction tx = env.createWriteTransaction()) {
   // cursors must close before write transactions!
   try (BufferCursor cursor = db.bufferCursor(tx)) {
     if (cursor.first()) {
       // write utf-8 ending with NULL byte
       cursor.keyWriteUtf8("England");
       cursor.valWriteUtf8("London");
       // overwrite existing item if any. Data is not written
       // into database before this operation is called and
       // no updates are visible outside this transaction until
       // the transaction is committed
       cursor.overwrite();
       cursor.first();
       // delete current cursor position
       cursor.delete();
     }
   }
   // commit changes or try-with-resources will auto-abort
   tx.commit();
 } 

License

This project is licensed under the Apache License, Version 2.0 but the binary jar it produces also includes liblmdb library of the OpenLDAP project which is licensed under the The OpenLDAP Public License.

lmdbjni's People

Contributors

krisskross avatar phraktle avatar chirino avatar headuck avatar batterseapower avatar slaunay avatar benalexau avatar raybellis avatar mikeg-red5hift avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.