pablosmedina / ckite Goto Github PK

what dos j mean here?
ckite / src / main / scala / ckite / statemachine / j /

Thanks,

When running sbt, I get the following error related to an unresolved dependency:

How do you manage to build the project?

Thanks.

[info] Loading global plugins from /home/dell/.sbt/0.13/plugins
[info] Loading project definition from /home/dell/projects/github/ckite/project
[info] Updating {file:/home/dell/projects/github/ckite/project/}ckite-build...
[info] Resolving org.apache.thrift#libthrift;0.5.0-1 ...
[warn] 	module not found: org.apache.thrift#libthrift;0.5.0-1
[warn] ==== typesafe-ivy-releases: tried
[warn]   https://repo.typesafe.com/typesafe/ivy-releases/org.apache.thrift/libthrift/0.5.0-1/ivys/ivy.xml
[warn] ==== sbt-plugin-releases: tried
[warn]   https://repo.scala-sbt.org/scalasbt/sbt-plugin-releases/org.apache.thrift/libthrift/0.5.0-1/ivys/ivy.xml
[warn] ==== local: tried
[warn]   /home/dell/.ivy2/local/org.apache.thrift/libthrift/0.5.0-1/ivys/ivy.xml
[warn] ==== local-preloaded-ivy: tried
[warn]   /home/dell/.sbt/preloaded/org.apache.thrift/libthrift/0.5.0-1/ivys/ivy.xml
[warn] ==== local-preloaded: tried
[warn]   file:////home/dell/.sbt/preloaded/org/apache/thrift/libthrift/0.5.0-1/libthrift-0.5.0-1.pom
[warn] ==== public: tried
[warn]   https://repo1.maven.org/maven2/org/apache/thrift/libthrift/0.5.0-1/libthrift-0.5.0-1.pom
[warn] ==== twitter-repo: tried
[warn]   http://maven.twttr.com/org/apache/thrift/libthrift/0.5.0-1/libthrift-0.5.0-1.pom
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	::          UNRESOLVED DEPENDENCIES         ::
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 	:: org.apache.thrift#libthrift;0.5.0-1: not found
[warn] 	::::::::::::::::::::::::::::::::::::::::::::::
[warn] 
[warn] 	Note: Unresolved dependencies path:
[warn] 		org.apache.thrift:libthrift:0.5.0-1
[warn] 		  +- com.twitter:scrooge-generator_2.10:3.20.0
[warn] 		  +- com.twitter:scrooge-sbt-plugin:3.20.0 (scalaVersion=2.10, sbtVersion=0.13) (/home/dell/projects/github/ckite/project/plugins.sbt#L7-8)
[warn] 		  +- default:ckite-build:0.1-SNAPSHOT (scalaVersion=2.10, sbtVersion=0.13)
sbt.ResolveException: unresolved dependency: org.apache.thrift#libthrift;0.5.0-1: not found

Implement Log compaction trough snapshots

The readme mentions that you can get a value from a local read, but the method shown is private:

//alternatively you can read from its local state machine allowing possible stale values
val value = ckite.readLocal(Get("key1")) 

CKiteClient.scala:23

  private[ckite] def readLocal[T](readCommand: ReadCommand[T]): Future[T] = raft.onLocalReadReceived(readCommand)

Can you make this method public?

Hi Pablo -

It seems that the builder always does a file lookup (perhaps ConfigFactor.load.getConfig("ckite")) when fetching the listen-address.

I've tried to supply a different value in the Builder as you do in the examples with no luck. I've also tried something like this before constructing the builder:

val config = originalConfig.withValue("ckite.finagle.listen-address", ConfigValueFactory.fromAnyRef(myListenAddress))

If I remove the value from my config file, I get an exception.

Ultimately, I need to be set this on each of my nodes at runtime since I can only make a single deployment package for all of my nodes.

Thanks for the help,
John

1. Reduce memory footprint during snapshotting
2. Avoid copy (ByteBuffers and FileChannels)
3. Invert control

Implement Leader election as specified in Raft paper

Write simple Getting started instructions for Scala including:

1. repo location (sonatype for now, will be maven central)
2. artifact coordinates for sbt
3. start a simple cluster

Expose useful metrics to monitor it in runtime. Some ideas:

1. elections count
2. elections time (maybe a histogram or simple avg)
3. write time (histogram)
4. read time (histogram)
5. leader uptime
6. followers last ack
7. log size
8. snapshot size
9. log compactions and time

When I try to run the example, I get this error:

error com.typesafe.config.ConfigException$Null: hardcoded value: Configuration key 'ckite.cluster.localBinding' is set to null but expected STRING.

$ uname -va
Linux bill-ubuntu 3.13.0-24-generic #47-Ubuntu SMP Fri May 2 23:30:00 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux
$ sbt --version
sbt launcher version 0.13.5
$ scala -version
Scala code runner version 2.9.2 -- Copyright 2002-2011, LAMP/EPFL
$ git log
commit 8f790d0
Author: Pablo Medina [email protected]
Date: Wed Apr 30 00:59:05 2014 -0300

StateMachine in charge of lastAppliedIndex persistence

[...]
$ sbt run -Dport=9091 -Dmembers=localhost:9092,localhost:9093 -DdataDir=/tmp/ckite/member1
[info] Loading project definition from /home/bill/software/ckite/project
[info] Set current project to ckite (in build file:/home/bill/software/ckite/)
[info] Compiling 72 Scala sources to /home/bill/software/ckite/target/classes...
[warn] there were 9 deprecation warning(s); re-run with -deprecation for details
[warn] there were 52 feature warning(s); re-run with -feature for details
[warn] two warnings found
[info] Running ckite.example.KVStoreBootstrap
SLF4J: Failed to load class "org.slf4j.impl.StaticLoggerBinder".
SLF4J: Defaulting to no-operation (NOP) logger implementation
SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further details.
error com.typesafe.config.ConfigException$Null: hardcoded value: Configuration key 'ckite.cluster.localBinding' is set to null but expected STRING
com.typesafe.config.ConfigException$Null: hardcoded value: Configuration key 'ckite.cluster.localBinding' is set to null but expected STRING
at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:121)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:136)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:142)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:150)
at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:155)
at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:197)
at ckite.Configuration.localBinding(Configuration.scala:67)
at ckite.Cluster.(Cluster.scala:51)
at ckite.CKiteBuilder.build(CKiteBuilder.scala:76)
at ckite.example.KVStoreBootstrap$delayedInit$body.apply(KVStoreBootstrap.scala:12)
at scala.Function0$class.apply$mcV$sp(Function0.scala:40)
at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.App$$anonfun$main$1.apply(App.scala:71)
at scala.collection.immutable.List.foreach(List.scala:318)
at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:32)
at scala.App$class.main(App.scala:71)
at ckite.example.KVStoreBootstrap$.main(KVStoreBootstrap.scala:4)
at ckite.example.KVStoreBootstrap.main(KVStoreBootstrap.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
[trace] Stack trace suppressed: run last compile:run for the full output.
java.lang.RuntimeException: Nonzero exit code: 1
at scala.sys.package$.error(package.scala:27)
[trace] Stack trace suppressed: run last compile:run for the full output.
error Nonzero exit code: 1
[error] Total time: 36 s, completed May 28, 2014 9:29:39 AM

Hi Pablo -

We've been using ckite with great success so far. Today, I tried using MapDB for persistence and ran into a problem: looks like two methods that are called during startup are not yet implemented (or perhaps I'm looking in the wrong place).

My builder:

val ckite = CKiteBuilder()
.stateMachine(new KVStore())
.rpc(FinagleThriftRpc)
.bootstrap(bootstrap)
.members(nodes.asScala)
.storage(new MapDBStorage(dataDir))
.build

And "io.ckite" % "ckite-mapdb" % "0.2.0", "org.mapdb" % "mapdb" % "0.9.13" are in my build.sbt libraryDependencies.

I see the dataDir get populated with 'log', 'snapshots', and 'state' as expected, but RLog looks like it is trying to reload a snapshot on startup, which eventually calls ckite.mapdb.MapDBStorage.deserializeSnapshot.

I'm looking at the src here: https://github.com/pablosmedina/ckite/blob/master/ckite-mapdb/src/main/scala/ckite/mapdb/MapDBStorage.scala, and it looks like the method isn't implemented.

Is this a bug or am I missing something?

Here's the trace:

[key-value-store-system-akka.actor.default-dispatcher-3] INFO - 16:39:55,327 - witter.finagle.Init$$anonfun$1#apply$mcV$sp: Finagle version 6.27.0 (rev=65df1512aadc8b79871a6dcafbbd210fd088b055) built at 20150727-172532
2015-09-18 16:39:55 INFO Raft:18 - Starting CKite localhost:9091...
2015-09-18 16:39:55 INFO RLog:156 - Initializing RLog...
[ERROR] [09/18/2015 16:39:55.913] [key-value-store-system-akka.actor.default-dispatcher-3] [LocalActorRefProvider(akka://key-value-store-system)] guardian failed, shutting down system
scala.NotImplementedError: an implementation is missing
at scala.Predef$.$qmark$qmark$qmark(Predef.scala:225)
at ckite.mapdb.MapDBStorage.deserializeSnapshot(MapDBStorage.scala:41)
at ckite.mapdb.MapDBStorage.retrieveLatestSnapshot(MapDBStorage.scala:25)
at ckite.rlog.SnapshotManager.latestSnapshot(SnapshotManager.scala:76)
at ckite.RLog.reloadSnapshot(RLog.scala:172)
at ckite.RLog.replay(RLog.scala:164)
at ckite.RLog.initialize(RLog.scala:158)
at ckite.Raft.initializeLog(Raft.scala:29)
at ckite.Raft.start(Raft.scala:19)
at ckite.CKiteClient.start(CKiteClient.scala:35)
at . . .

Thanks for the help and I'm happy to test a fix.

John

Implement Log replication following rules in Raft paper

Write simple Getting started instructions for Java including:

1. repo location (sonatype for now, will be maven central)
2. artifact coordinates for Maven
3. start a simple cluster

Implement joint consensus to be able to add or remove members