twitter / chill Goto Github PK
View Code? Open in Web Editor NEWScala extensions for the Kryo serialization library
Home Page: https://twitter.com/scalding
License: Apache License 2.0
Scala extensions for the Kryo serialization library
Home Page: https://twitter.com/scalding
License: Apache License 2.0
Hi,
I've got poor performance from chill until I rewrite the KryoInjection class in order to Reuse the same Ouput and Input buffer and avoid kryo object creation for each call.
see https://gist.github.com/jcheype/5362926
Best regards.
The version has been released for Scala 2.9.2, but not for 2.10.x.
I'm not using Scala 2.9.x any more, but once you have released for 2.9.2, how about 2.9.3?
Chill has just added support for Scala 2.10?
Currently, README says:
Current version is 0.2.0. groupid="com.twitter" artifact="chill_2.9.2"
It should also say something about Scala 2.10.
ironically, Kryo cannot be serialized.
We need to fix this before the next release.
Just in case it got missed in the comment of my prior issue, I've created a new issue here.
The following code causes a NPE using the develop branch (checked out a few days ago) with scala 2.10:
object Test {
def apply() = {
println("Test applied")
}
}
object Main extends Application {
import com.twitter.chill._
override def main(args: Array[String]) {
val TestAsVar = Test
val bytes = KryoInjection(() => {
println("Anon. fn")
Test()
})
val x = KryoInjection.invert(bytes)
println(x)
x.get match {
case f : Function0[Unit] => f()
}
}
}
The call to Test() causes a NPE. The interesting thing is: if I replace "Test()" with "TestAsVar()", then it works.
Registering these will prevent their names from being written into the stream, which could be a big win.
scala> Map(1->2).getClass
res124: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map1
scala> Map(1->2, 2->4).getClass
res125: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map2
scala> Map(1->2, 2->4, 4->5).getClass
res126: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map3
scala> Map(1->2, 2->4, 4->5, 5->6).getClass
res127: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map4
scala> Map(1->2, 2->4, 4->5, 5->6, 6->7).getClass
res128: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.HashMap$HashTrieMap
scala> Set().getClass
res132: java.lang.Class[_ <: scala.collection.immutable.Set[Nothing]] = class scala.collection.immutable.Set$EmptySet$
scala> Set(1).getClass
res133: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set1
scala> Set(1,2).getClass
res134: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set2
scala> Set(1,2,3).getClass
res135: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set3
scala> Set(1,2,3,4).getClass
res136: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set4
scala> Set(1,2,3,4,5).getClass
res137: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.HashSet$HashTrieSet
Looks like fields serializer fails on that one.
We need some strict unit tests, and we need to make it pass.
Right now, I'm thinking this is not actually working like we expect.
We need to decompile a bunch of examples and verify that it is doing what we expect. Currently, I don't believe that it is (maybe for spark's REPL it works, but in our use case, I don't think it is).
Currently README only mentions MeatLocker.
Should I add an example like this to README (I will send a pull request):
import scala.util.Try
import com.twitter.chill.KryoBijection
object SeriDeseri {
def serialize(ref: AnyRef): Array[Byte] = KryoBijection(ref)
def deserialize(bytes: Array[Byte]) = Try { KryoBijection.invert(bytes) }
}
Make a serializer that just let's protobuf messages do their thing. Make it easy to register for any subclass of Message.
This code causes a NPE for me, with scala 2.10.1 and master:
object Foo {
def Bar = 1
}
object Main extends Application {
override def main(args: Array[String]) {
val bytes = KryoInjection(() => {
println(Foo.Bar)
})
val x = KryoInjection.invert(bytes)
val f = x.get match {
case f : Function0[Unit] => f()
}
}
Any idea what I am doing wrong here?
subclasses of TBase should just use thrift to serialize, not fields.
It could be worth trying to test with the current upcoming trunk version of Kryo. It contains significant perormance improvements including Unsafe-based serialization and ability to serialize directly into off-heap memory with almost a native speed if required. To use these new features, you only need to use new new IO classes (like UnsafeInput and UnsafeOutput) instead of standrad Input/Output classes.
IMHO, these features could be very useful in general and for Big Data apps in particular (Hadoop, Cassandra, Storm, etc).
BTW, after I added Unsafe-serialization to Kryo, some other well-known frameworks included similar features. The list includes: Avro (I added it there myself), Hazelcast (they reused may ideas and adpated it to their APIs). MapDB could be the next one.
Expected performance improvements are rather significant: You often see 3-4X speedups. If your classes contain arrays of native types (ints, floats, doubles) you can expect even 10x performance improvements.
This is using latest chill. See the following:
import com.twitter.chill._
import scala.collection._
val s = List(Some(mutable.HashMap(1->1, 2->2)), None, Some(mutable.HashMap(3->4)))
KryoBijection.invert(KryoBijection(s))
The above incorrectly produces:
res4: java.lang.Object = List(Some(Map(3 -> 4)), None, Some(Map(3 -> 4)))
Weird regex error. See below:
import com.twitter.chill._
val r = """\bhilarious""".r
r.findFirstIn("oh boy this is hilarious").isDefined
true
val chilled = KryoInjection.invert(KryoInjection(r)).get.asInstanceOf[Regex]
chilled.findFirstIn("oh boy this is hilarious").isDefined
java.lang.IndexOutOfBoundsException: No group 0
at java.util.regex.Matcher.group(Matcher.java:470)
at java.util.regex.Matcher.group(Matcher.java:428)
at scala.util.matching.Regex.findFirstIn(Regex.scala:213)
at .<init>(<console>:13)
at .<clinit>(<console>)
at .<init>(<console>:11)
at .<clinit>(<console>)
at $print(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
This would allow MeatLocker instances accessed with "get" before serialization to be survive a serialization round trip.
Caused by: java.lang.NoSuchMethodError: scala.Enumeration$Val: method <init>()V not found
at scala.Enumeration$ValConstructorAccess.newInstance(Unknown Source)
at com.esotericsoftware.kryo.Kryo$1.newInstance(Kryo.java:978)
at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1027)
at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:228)
at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:217)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:718)
at com.twitter.chill.Tuple3Serializer.read(TupleSerializers.scala:60)
they are common enough that we shouldn't use the Fields there.
None is not needed as it is supported by the singleton serializer.
Kryo 2.17 (08-Jul-2012) is being used.
Latest version is 2.21 (28-Feb-2013): http://search.maven.org/#browse|1975274176
Do this to make release and Kryo version management easy. Keep the code java-only te keep it usable for non-scala projects.
http://www.scala-lang.org/api/current/scala/collection/JavaConverters$.html
We've seen some problems with List(1,2,3).asJava being serialized correctly.
TODO: write some tests, if any fail, fix them with custom serializers.
I think this is serialized okay now, but we need to add tests and I think if we register this it can be free (it is a singleton, but a Java class).
I don't know if this is due to some hidden state on my machine (I nuked everything I could think of), but adding scalaVersion := "2.9.3",
to the build gives me the below (and it is super frustrating).
From my irc plea for help:
[10:30am] oscarboykin: first it warns that it can't find source:
[10:30am] oscarboykin: [warn] :: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[10:30am] oscarboykin: [warn] :: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
[10:30am] oscarboykin: but then this becomes an error:
[10:30am] oscarboykin: [info] Done updating.
[10:30am] oscarboykin: sbt.ResolveException: download failed: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[10:30am] oscarboykin: download failed: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
[10:30am] oscarboykin: at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
[10:31am] oscarboykin: meanwhile, 1) the source jars are there: http://oss.sonatype.org/content/repositories/releases/org/scala-tools/testing/specs_2.9.3/1.6.9/
[10:31am] oscarboykin: 2) I don't know why there are needed anyway
[10:32am] oscarboykin: but they are not suffixed with .src as sbt seems to be looking for:
[10:32am] oscarboykin: http://oss.sonatype.org/content/repositories/releases/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[10:32am] oscarboykin: they are .jars.
[info] Loading project definition from /Users/oscarb/workspace/chill/project
[info] Set current project to chill-all (in build file:/Users/oscarb/workspace/chill/)
[info] Updating {file:/Users/oscarb/workspace/chill/}chill...
[info] Resolving org.ow2.asm#asm-tree;4.0 ...
[warn] [FAILED ] org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src: (0ms)
[warn] ==== local: tried
[warn] /Users/oscarb/.ivy2/local/org.scalacheck/scalacheck_2.9.3/1.10.0/srcs/scalacheck_2.9.3-sources.src
[warn] ==== sonatype-snapshots: tried
[warn] http://oss.sonatype.org/content/repositories/snapshots/org/scalacheck/scalacheck_2.9.3/1.10.0/scalacheck_2.9.3-1.10.0-sources.src
[warn] ==== sonatype-releases: tried
[warn] http://oss.sonatype.org/content/repositories/releases/org/scalacheck/scalacheck_2.9.3/1.10.0/scalacheck_2.9.3-1.10.0-sources.src
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/scalacheck/scalacheck_2.9.3/1.10.0/scalacheck_2.9.3-1.10.0-sources.src
[warn] [FAILED ] org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src: (0ms)
[warn] ==== local: tried
[warn] /Users/oscarb/.ivy2/local/org.scala-tools.testing/specs_2.9.3/1.6.9/srcs/specs_2.9.3-sources.src
[warn] ==== sonatype-snapshots: tried
[warn] http://oss.sonatype.org/content/repositories/snapshots/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[warn] ==== sonatype-releases: tried
[warn] http://oss.sonatype.org/content/repositories/releases/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[warn] ==== public: tried
[warn] http://repo1.maven.org/maven2/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: FAILED DOWNLOADS ::
[warn] :: ^ see resolution messages for details ^ ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[warn] :: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[info] Updating {file:/Users/oscarb/workspace/chill/}chill-java...
[info] Resolving com.esotericsoftware.kryo#kryo;2.21 ...
[info] Updating {file:/Users/oscarb/workspace/chill/}chill-all...
[info] Resolving org.scala-tools.testing#specs_2.9.3;1.6.9 ...
[info] Done updating.
[info] Resolving org.objenesis#objenesis;1.2 ...
[info] Updating {file:/Users/oscarb/workspace/chill/}chill-hadoop...
[info] Resolving org.scala-tools.testing#specs_2.9.3;1.6.9 ...
[info] Done updating.
[info] Resolving log4j#log4j;1.2.17 ...
[info] Done updating.
sbt.ResolveException: download failed: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
download failed: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:105)
at sbt.IvySbt.liftedTree1$1(Ivy.scala:52)
at sbt.IvySbt.action$1(Ivy.scala:52)
at sbt.IvySbt$$anon$3.call(Ivy.scala:61)
at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
at xsbt.boot.Locks$GlobalLock.withChannelRetries$1(Locks.scala:81)
at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
at xsbt.boot.Using$.withResource(Using.scala:11)
at xsbt.boot.Using$.apply(Using.scala:10)
at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
at xsbt.boot.Locks$GlobalLock.liftedTree1$1(Locks.scala:52)
at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
at xsbt.boot.Locks$.apply0(Locks.scala:31)
at xsbt.boot.Locks$.apply(Locks.scala:28)
at sbt.IvySbt.withDefaultLogger(Ivy.scala:61)
at sbt.IvySbt.withIvy(Ivy.scala:102)
at sbt.IvySbt.withIvy(Ivy.scala:98)
at sbt.IvySbt$Module.withModule(Ivy.scala:117)
at sbt.IvyActions$.update(IvyActions.scala:121)
at sbt.Classpaths$$anonfun$work$1$1.apply(Defaults.scala:955)
at sbt.Classpaths$$anonfun$work$1$1.apply(Defaults.scala:953)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$58.apply(Defaults.scala:976)
at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$58.apply(Defaults.scala:974)
at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:978)
at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:973)
at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
at sbt.Classpaths$.cachedUpdate(Defaults.scala:981)
at sbt.Classpaths$$anonfun$47.apply(Defaults.scala:858)
at sbt.Classpaths$$anonfun$47.apply(Defaults.scala:855)
at sbt.Scoped$$anonfun$hf10$1.apply(Structure.scala:586)
at sbt.Scoped$$anonfun$hf10$1.apply(Structure.scala:586)
at scala.Function1$$anonfun$compose$1.apply(Function1.scala:49)
at sbt.Scoped$Reduced$$anonfun$combine$1$$anonfun$apply$12.apply(Structure.scala:311)
at sbt.Scoped$Reduced$$anonfun$combine$1$$anonfun$apply$12.apply(Structure.scala:311)
at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:41)
at sbt.std.Transform$$anon$5.work(System.scala:71)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:232)
at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:232)
at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
at sbt.Execute.work(Execute.scala:238)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:232)
at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:232)
at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:722)
[error] (chill/*:update) sbt.ResolveException: download failed: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[error] download failed: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
When doing a roundtrip over a mutable Map of mutable Sets, the result is not the same. It seems that it reuses each mutable.Set as the basis for the next.
This happens with Kryo 2.21 and Chill 0.2.3
Example:
val obj = mutable.Map(4 -> mutable.Set("house1", "house2"), 1 -> mutable.Set("name3", "name4", "name1", "name2"), 0 -> mutable.Set(1, 2, 3, 4))
MeatLocker(obj).copy // => Map(4 -> Set(house1, house2), 1 -> Set(name3, house1, house2, name4, name1, name2), 0 -> Set(name3, house1, 1, house2, 2, 3, name4, name1, 4, name2))
I'm not sure how to configure the Hadoop config using the new system.
The specific instance I ran into was MapLike.DefaultValuesIterable, but I bet there are other similar classes that also don't serialize.
scala> import com.twitter.chill._
import com.twitter.chill._
scala> import scala.collection._
import scala.collection._
scala> val a = mutable.ArrayBuffer(1,2,3,4,5)
a: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 2, 3, 4, 5)
scala> KryoInjection(a)
res0: Array[Byte] = Array(1, 0, 115, 99, 97, 108, 97, 46, 99, 111, 108, 108, 101, 99, 116, 105, 111, 110, 46, 109, 117, 116, 97, 98, 108, 101, 46, 65, 114, 114, 97, 121, 66, 117, 102, 102, 101, -14, 1, 5, 2, 2, 2, 4, 2, 6, 2, 8, 2, 10)
scala> KryoInjection.invert(KryoInjection(a))
res1: Option[java.lang.Object] = Some(Vector(1, 2, 3, 4, 5))
Currently, Chill only supports Array[Byte].
Kryo supports InputStream:
http://kryo.googlecode.com/svn/api/v2/com/esotericsoftware/kryo/io/Input.html
It is costly to create a kryo instance and right now the code is doing it on every open. Cascading calls open after almost every entry in the tuple, so this could kill performance.
We do this because we think (fear? have been burned?) by Kryo being mutable. What we want is an immutable Kryo instance that is only used for serialization and will not change any tokens or register any new ones.
@caniszczyk Can you hook up @ryanlecompte ? He has help review and contributed code. Additionally he helped get chill merged in Spark.
Ryan: FYI our rule is: never merge code you write yourself. Always ping another committer. Also, only merge when travis is green (obviously).
inner case classes seem to capture references to outer even if they are not used.
If we can find some cheap way to detect this, it would really help scalding and possibly spark.
It seems that Chill cannot serialize and deserialize Scala continuations.
For scrooge generated objects, make sure to use scrooge's thrift serialization.
Explain how to use the library. Probably just something like: KryoSerializer.registerAll(k) on a k: Kryo which might have been generated from (new KryoSerializer { }).getKryo
This is (I think) wrongly saying that scala.collection.Map is immutable, when in fact that class may or may not be mutable.
I don't know if I use chill as it supposed to be used, but here are some trivial failing cases that I found?
test("int list serialize/deserialize") {
val listObj = List(
List(List(1)),
List(List(1))
)
val serializedObject = KryoBijection.serialize(listObj)
val deserializedObj = KryoBijection.deserialize[List[List[Int]]](serializedObject)
assert(listObj === deserializedObj)
}
Results in
List(List(List(1)), List(List(1))) did not equal List(1, List(1), List(1, List(1)))
test("int list container serialize/deserialize") {
val listObj = List(
ListContainer(List(1)),
ListContainer(List(1))
)
val serializedObject = KryoBijection.serialize(listObj)
val deserializedObj = KryoBijection.deserialize[List[ListContainer]](serializedObject)
assert(listObj === deserializedObj)
}
case class ListContainer(ints: List[Int])
Results in
List(ListContainer(List(1)), ListContainer(List(1))) did not equal List(1, ListContainer(List(1)))
Java only module to serialize the classes in guava.
See: #60
Set is also special cased for small sets.
It would be great if KryoInjection didn't use allCatch but instead just caught Exception
s. Deserializing potentially large objects is a good opportunity for OOMEs.
https://github.com/scala/scala/blob/master/src/library/scala/util/control/Exception.scala#L150
Hi guys,
We at Spark ran into a problem using Chill to serialize WrappedArray. See the following code. It looks like the problem is with a WrappedArray for specialized tuples. If I replace the ints with string (and thus no specialized Tuple2), the exception goes away.
This is becoming a major problem for the Spark project because anything with primitive WrappedArray would fail.
scala> val ser = new spark.KryoSerializer
ser: spark.KryoSerializer = spark.KryoSerializer@7798ce82
scala> ser.newInstance.deserialize[Array[(Int, Int)]]( ser.newInstance.serialize(Array((1, 2), (2, 3))) )
res12: Array[(Int, Int)] = Array((1,2), (2,3))
scala> ser.newInstance.deserialize[Seq[(Int, String)]]( ser.newInstance.serialize(Array((1, "2"), (2, "3")).toSeq) )
res20: Seq[(Int, String)] = WrappedArray((1,2), (2,3))
scala> ser.newInstance.deserialize[Seq[(Int, Int)]]( ser.newInstance.serialize(Array((1, 2), (2, 3)).toSeq) )
java.lang.ArrayIndexOutOfBoundsException: -2
at java.util.ArrayList.get(ArrayList.java:324)
at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:773)
at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:624)
at com.twitter.chill.WrappedArraySerializer$$anonfun1.apply(WrappedArraySerializer.scala:44)
at com.twitter.chill.WrappedArraySerializer$$anonfun1.apply(WrappedArraySerializer.scala:43)
at scala.collection.immutable.Range.foreach(Range.scala:81)
at com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:43)
at com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:21)
at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
at spark.KryoSerializerInstance.deserialize(KryoSerializer.scala:75)
at <init>(<console>:13)
at <init>(<console>:18)
at <init>(<console>:20)
at <init>(<console>:22)
at <init>(<console>:24)
at .<init>(<console>:28)
at .<clinit>(<console>)
at .<init>(<console>:11)
at .<clinit>(<console>)
at $export(<console>)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)
at spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:890)
at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
at java.lang.Thread.run(Thread.java:680)
Meatlocker probably benefits from using references and not repeatedly serializing the objects in a graph.
There are a bunch of similar classes about building and setting up Kryo given Map[String, String]
(see below).
Can we add a base class that gets most of the common cases, put that in chill-java and leverage it in KryoRegistrationHelper.scala.
There is some serious code smell going on here. Ideally we could get storm to depend on chill-java and share this java-only code.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.