anthonyu / keptcollections Goto Github PK

Distributed Java Collections for ZooKeeper

Home Page: https://github.com/anthonyu/KeptCollections

Java 100.00%

keptcollections's Introduction

KeptCollections

What is KeptCollections?

KeptCollections is a library of drop-in replacements for the data structures in the Java Collections framework. KeptCollections uses Apache ZooKeeper as a backing store, thus making its data structures distributed and scalable.

Changes made to a KeptCollection by one node are seen by all other nodes within milliseconds, allowing for easy communication between nodes in a computing cluster.

KeptCollections now supports Java generics, thanks to Gaurav Sharma. Classes that implement Serializable can be used as values in any of the collections. Classes that are serializable as strings (i.e. ints, doubles, etc.) will still be stored as strings on the ZooKeeper.

Why KeptCollections?

Implementing a distributed data structure from scratch is difficult. ZooKeeper programming is less hard, but still no walk in the park. Conversely, KeptCollections implements the well known Java Collections APIs, so they can be dropped into new or existing Java code and easily make those applications distributed.

How does one use KeptCollections?

The only difference between the KeptCollections collections and the JDK collections are the names of the classes and their constructors.

For instance, where a Map from Java Collections could be instantiated like:

Map<String, String> map = new HashMap<String, String>();

KeptCollections is instead instantiated like:

Zookeeper zk = new ZooKeeper("localhost:2181", 20000, watcher);

Map<String, String> map =
  new KeptMap(zk, "/mymap", Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);

Both are accessed the same way:

map.put("key", "value");

String value = map.get("key");

See the JavaDoc help for more information.

How stable are the collections?

The KeptSet, KeptMap and KeptLock implementations have seen several years of usage in a production environment at two different cloud computing companies.

The other implementations are similar in code base and well unit tested, but have not seen production usage.

Which Collections interfaces are implemented?

Set

Map

ConcurrentMap

Collection

List

Queue

BlockingQueue

Lock (not from Java Collections, but it sort of fits here regardless).

Which interfaces will (likely) be implemented in the future?

SortedSet

SortedMap

A few of the java.concurrent interfaces, but not those, mainly lock and barrier related, that are already implemented by Menagerie [https://github.com/openUtility/menagerie]

Where can I get a Jar?

Use Maven:

<dependency>
  <groupId>net.killa.kept</groupId>
  <artifactId>KeptCollections</artifactId>
  <version>1.0.0</version>
</dependency>

keptcollections's People

Contributors

Stargazers

Watchers

Forkers

gsharma thaingo cchanyi wjw465150 chenld chyun sirithink tempbottle anddegs marco-scavuzzo desperado1992 starlit2006 ramy-ahmed sonnyit vijayeluri nangal

keptcollections's Issues

An infinite loop?

At KeptMap,the variable j is always 0?

   int j = 0;
    while (true) {
        // get the old value and its version
        byte[] oldval = this.keeper.getData(path, false, stat);

        try {
        // set the new value
        this.keeper.setData(path, value.getBytes(),
            stat.getVersion());

        // return the old value
        return new String(oldval);
        } catch (KeeperException.BadVersionException f) {
        if (j > 10)
            throw f;

        // someone updated it in between, try again
        KeptMap.LOG
            .debug("caught bad version attempting to update, retrying");

        Thread.sleep(50);
        }
    }

Add support for non-String collection types

It would be very helpful for users if we can add support for non-String types in all collections.

More relaxed implementation for KeptBlockingQueue

More relaxed implementation for KeptBlockingQueue
As I understand, the current implementation of KeptBlockingQueue is asynchronously reread the whole queue form ZK after each change of queue is performed. Despite the fact that ZK can join some events, the number of rereads is huge, and, moreover, the asymptotic on transfer data is O(N*N) and that is very painful if we start to process non empty queue.
It seems, the implementation can't be radically improved, due to luck of some features in ZK API (getFirstChild, for example) but it possible to make a more "relaxed" implementation, that is will be not fully conform to java Collections API, but will be have better performance.
For example, if we sacrifice the size() method accuracy, we can process BlockingQueue by batches, and perform new read from ZK only when we process all previously read query items. That is can radically
improve performance, at least, for low-consumer-number scenario.

Race condition

The update in KeptMap has a race condition:


if (this.map.containsKey(key)) {
String oldval = this.get(key);
this.keeper.setData(path, value.getBytes(), -1);
return oldval;
} else {
this.keeper.create(path, value.getBytes(), this.acl, this.createMode);
return null;
}

can fail easily in the presence of simultaneous updates. It is better to do the create, catch NodeExists and then do an update and return. This can be wrapped in a loop to catch the complementary race condition versus deletions.

Publish to a maven repo?

Any chance that Kept Collections will be available in a maven repo?

For that matter, is Kept Collections still alive?

the second infinite loop

And the i variable is always 0 too.

private String removeUnsynchronized(Object key) throws InterruptedException, KeeperException {

String path = this.znode + '/' + key;

try {
Stat stat = new Stat();
int i = 0;

while (true) {
try {
    byte[] oldval = this.keeper.getData(path, false, stat);

    this.keeper.delete(path, stat.getVersion());

    return new String(oldval);
} catch (KeeperException.BadVersionException e) {
    if (i > 10)
    throw e;

    // someone updated it in between, try again
    KeptMap.LOG
        .debug("caught bad version attempting to update, retrying");

    Thread.sleep(50);
}
}

} catch (KeeperException.NoNodeException e) {
return null;
}