twitter / chill Goto Github PK

View Code? Open in Web Editor NEW

609.0 609.0 156.0 1.22 MB

Scala extensions for the Kryo serialization library

Home Page: https://twitter.com/scalding

License: Apache License 2.0

Scala 58.62% Java 41.38%

chill's People

Contributors

Stargazers

Watchers

Forkers

strategist922 turp1twin rjhall azymnis ryanlecompte romix daikeshi daisyzhou mhooo tsdeng aniket486 giapdangle 2m mansurashraf testvidya11 pwendell iopenstack uzh aquto banno adriaanm phosphene rxin namjoshi rjkernick emaxerrno agemooij mohangadm slmnhq alisheikh skuehn mbbeaubi is00hcw zakattacktwitter carrot-garden rmetzger colinmarc mhaulrich kabitionno he-pin pinguo-liguo nonvirtualthunk figpope lianhuiwang ramalhoruben danielhopkins scrupal alekseyig nkhuyu sriramkrishnan shanfei toddwan lirenhao themodernlife jiyouyou125 mindis mikeaddison93 storiainc ravwojdyla joshrosen rameshraj wanghuatao chermenin bedeedidiong dasbipulkumar anivaniuk ttim ganeshrajulinaro rugby110 mix-sloboda fkowal weibingo piyushnarang jcdavis andrewsmartin jinyu0310 sportsbitenews mail2gauravkumar ckchow testbench-cs-imbus janpychou ranahasan mthomassen raboof kumar-asista fengzanfeng similove drangons tarunk2 vikumar-ciena jcroig skonto jackblie lambdaml coreyoconnor smarter bitpusherllc scala-steward cmoh regadas

chill's Issues

Performance issue

Hi,

I've got poor performance from chill until I rewrite the KryoInjection class in order to Reuse the same Ouput and Input buffer and avoid kryo object creation for each call.

see https://gist.github.com/jcheype/5362926

Best regards.

Release Chill 0.2.1 for Scala 2.10.x

The version has been released for Scala 2.9.2, but not for 2.10.x.

I'm not using Scala 2.9.x any more, but once you have released for 2.9.2, how about 2.9.3?

Update README to reflect that Chill supports Scala 2.10

Chill has just added support for Scala 2.10?

Currently, README says:
Current version is 0.2.0. groupid="com.twitter" artifact="chill_2.9.2"

It should also say something about Scala 2.10.

KryoInjection is not serializable

ironically, Kryo cannot be serialized.

We need to fix this before the next release.

add ReadableConfig, use it in creating ConfiguredInstantiators

NPE on deserialized fn referencing a singleton

Just in case it got missed in the comment of my prior issue, I've created a new issue here.

The following code causes a NPE using the develop branch (checked out a few days ago) with scala 2.10:

object Test {
    def apply() = {
        println("Test applied")
    }

}

object Main extends Application {
    import com.twitter.chill._
    override def main(args: Array[String]) {
        val TestAsVar = Test
        val bytes = KryoInjection(() => {
                                      println("Anon. fn")
                                      Test()
                                  })
        val x = KryoInjection.invert(bytes)
        println(x)
        x.get match {
            case f : Function0[Unit] => f()
        }
    }
}

The call to Test() causes a NPE. The interesting thing is: if I replace "Test()" with "TestAsVar()", then it works.

Register actual Map/Set Instances

Registering these will prevent their names from being written into the stream, which could be a big win.

scala> Map(1->2).getClass
res124: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map1

scala> Map(1->2, 2->4).getClass
res125: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map2

scala> Map(1->2, 2->4, 4->5).getClass
res126: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map3

scala> Map(1->2, 2->4, 4->5, 5->6).getClass
res127: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.Map$Map4

scala> Map(1->2, 2->4, 4->5, 5->6, 6->7).getClass
res128: java.lang.Class[_ <: scala.collection.immutable.Map[Int,Int]] = class scala.collection.immutable.HashMap$HashTrieMap

scala> Set().getClass
res132: java.lang.Class[_ <: scala.collection.immutable.Set[Nothing]] = class scala.collection.immutable.Set$EmptySet$

scala> Set(1).getClass
res133: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set1

scala> Set(1,2).getClass
res134: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set2

scala> Set(1,2,3).getClass
res135: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set3

scala> Set(1,2,3,4).getClass
res136: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.Set$Set4

scala> Set(1,2,3,4,5).getClass
res137: java.lang.Class[_ <: scala.collection.immutable.Set[Int]] = class scala.collection.immutable.HashSet$HashTrieSet

Add "How To Contribute to Chill" section to Wiki

Add a serializer,
add to the appropriate test,
add to KryoSerializer.scala

Add SimpleDateFormat to chill-java

Looks like fields serializer fails on that one.

Map$Map3 having some issues when registration is required

http://stackoverflow.com/questions/16806927/registering-serializers-in-storm-config

Add a chill-akka module from the scalafish code

https://github.com/azymnis/scalafish/tree/master/src/main/scala/org/zymnis/scalafish/serialization

ClosureCleaner is pretty broken

We need some strict unit tests, and we need to make it pass.

Right now, I'm thinking this is not actually working like we expect.

We need to decompile a bunch of examples and verify that it is doing what we expect. Currently, I don't believe that it is (maybe for spark's REPL it works, but in our use case, I don't think it is).

Add example about serializing to bytes and deserializing from bytes

Currently README only mentions MeatLocker.
Should I add an example like this to README (I will send a pull request):

import scala.util.Try
import com.twitter.chill.KryoBijection

object SeriDeseri {
  def serialize(ref: AnyRef): Array[Byte] = KryoBijection(ref)
  def deserialize(bytes: Array[Byte]) = Try { KryoBijection.invert(bytes) }
}

chill protobuf

Make a serializer that just let's protobuf messages do their thing. Make it easy to register for any subclass of Message.

chill thrift

subclasses of TBase should just use thrift to serialize, not fields.

Improve performance by using the Unsafe-based serialization supportted by trunk version of Kryo

It could be worth trying to test with the current upcoming trunk version of Kryo. It contains significant perormance improvements including Unsafe-based serialization and ability to serialize directly into off-heap memory with almost a native speed if required. To use these new features, you only need to use new new IO classes (like UnsafeInput and UnsafeOutput) instead of standrad Input/Output classes.

IMHO, these features could be very useful in general and for Big Data apps in particular (Hadoop, Cassandra, Storm, etc).

BTW, after I added Unsafe-serialization to Kryo, some other well-known frameworks included similar features. The list includes: Avro (I added it there myself), Hazelcast (they reused may ideas and adpated it to their APIs). MapDB could be the next one.

Expected performance improvements are rather significant: You often see 3-4X speedups. If your classes contain arrays of native types (ints, floats, doubles) you can expect even 10x performance improvements.

KryoBijection not deserializing complex List properly

This is using latest chill. See the following:

import com.twitter.chill._
import scala.collection._

val s = List(Some(mutable.HashMap(1->1, 2->2)), None, Some(mutable.HashMap(3->4)))
KryoBijection.invert(KryoBijection(s))

The above incorrectly produces:

res4: java.lang.Object = List(Some(Map(3 -> 4)), None, Some(Map(3 -> 4)))

Regex marshal/unmarshal failure

Weird regex error. See below:

import com.twitter.chill._

val r = """\bhilarious""".r

r.findFirstIn("oh boy this is hilarious").isDefined
true

val chilled = KryoInjection.invert(KryoInjection(r)).get.asInstanceOf[Regex]

chilled.findFirstIn("oh boy this is hilarious").isDefined

java.lang.IndexOutOfBoundsException: No group 0
    at java.util.regex.Matcher.group(Matcher.java:470)
    at java.util.regex.Matcher.group(Matcher.java:428)
    at scala.util.matching.Regex.findFirstIn(Regex.scala:213)
    at .<init>(<console>:13)
    at .<clinit>(<console>)
    at .<init>(<console>:11)
    at .<clinit>(<console>)
    at $print(<console>)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)

MeatLocker should reference the @transient v until serialization

This would allow MeatLocker instances accessed with "get" before serialization to be survive a serialization round trip.

Chill should to support scala Enumerations

Caused by: java.lang.NoSuchMethodError: scala.Enumeration$Val: method <init>()V not found
    at scala.Enumeration$ValConstructorAccess.newInstance(Unknown Source)
    at com.esotericsoftware.kryo.Kryo$1.newInstance(Kryo.java:978)
    at com.esotericsoftware.kryo.Kryo.newInstance(Kryo.java:1027)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.create(FieldSerializer.java:228)
    at com.esotericsoftware.kryo.serializers.FieldSerializer.read(FieldSerializer.java:217)
    at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:718)
    at com.twitter.chill.Tuple3Serializer.read(TupleSerializers.scala:60)

Add serializers for Some, Left, Right

they are common enough that we shouldn't use the Fields there.

None is not needed as it is supported by the singleton serializer.

Update Kryo to the latest version

Kryo 2.17 (08-Jul-2012) is being used.
Latest version is 2.21 (28-Feb-2013): http://search.maven.org/#browse|1975274176

Move the cascading-kryo module here as chill-hadoop / chill-cascading

Do this to make release and Kryo version management easy. Keep the code java-only te keep it usable for non-scala projects.

Handle AsJava/AsScala objects

http://www.scala-lang.org/api/current/scala/collection/JavaConverters$.html

We've seen some problems with List(1,2,3).asJava being serialized correctly.

TODO: write some tests, if any fail, fix them with custom serializers.

Add scala.runtime.BoxedUnit

I think this is serialized okay now, but we need to add tests and I think if we register this it can be free (it is a singleton, but a Java class).

Can't upgrade to 2.9.3

I don't know if this is due to some hidden state on my machine (I nuked everything I could think of), but adding scalaVersion := "2.9.3", to the build gives me the below (and it is super frustrating).

From my irc plea for help:

[10:30am] oscarboykin: first it warns that it can't find source:
[10:30am] oscarboykin: [warn] :: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[10:30am] oscarboykin: [warn] :: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
[10:30am] oscarboykin: but then this becomes an error:
[10:30am] oscarboykin: [info] Done updating.
[10:30am] oscarboykin: sbt.ResolveException: download failed: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[10:30am] oscarboykin: download failed: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
[10:30am] oscarboykin:  at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
[10:31am] oscarboykin: meanwhile, 1) the source jars are there: http://oss.sonatype.org/content/repositories/releases/org/scala-tools/testing/specs_2.9.3/1.6.9/
[10:31am] oscarboykin: 2) I don't know why there are needed anyway
[10:32am] oscarboykin: but they are not suffixed with .src as sbt seems to be looking for:
[10:32am] oscarboykin: http://oss.sonatype.org/content/repositories/releases/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[10:32am] oscarboykin: they are .jars.

Actual output:

[info] Loading project definition from /Users/oscarb/workspace/chill/project
[info] Set current project to chill-all (in build file:/Users/oscarb/workspace/chill/)
[info] Updating {file:/Users/oscarb/workspace/chill/}chill...
[info] Resolving org.ow2.asm#asm-tree;4.0 ...
[warn]  [FAILED     ] org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src:  (0ms)
[warn] ==== local: tried
[warn]   /Users/oscarb/.ivy2/local/org.scalacheck/scalacheck_2.9.3/1.10.0/srcs/scalacheck_2.9.3-sources.src
[warn] ==== sonatype-snapshots: tried
[warn]   http://oss.sonatype.org/content/repositories/snapshots/org/scalacheck/scalacheck_2.9.3/1.10.0/scalacheck_2.9.3-1.10.0-sources.src
[warn] ==== sonatype-releases: tried
[warn]   http://oss.sonatype.org/content/repositories/releases/org/scalacheck/scalacheck_2.9.3/1.10.0/scalacheck_2.9.3-1.10.0-sources.src
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/org/scalacheck/scalacheck_2.9.3/1.10.0/scalacheck_2.9.3-1.10.0-sources.src
[warn]  [FAILED     ] org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src:  (0ms)
[warn] ==== local: tried
[warn]   /Users/oscarb/.ivy2/local/org.scala-tools.testing/specs_2.9.3/1.6.9/srcs/specs_2.9.3-sources.src
[warn] ==== sonatype-snapshots: tried
[warn]   http://oss.sonatype.org/content/repositories/snapshots/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[warn] ==== sonatype-releases: tried
[warn]   http://oss.sonatype.org/content/repositories/releases/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[warn] ==== public: tried
[warn]   http://repo1.maven.org/maven2/org/scala-tools/testing/specs_2.9.3/1.6.9/specs_2.9.3-1.6.9-sources.src
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  ::              FAILED DOWNLOADS            ::
[warn]  :: ^ see resolution messages for details  ^ ::
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[warn]  :: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[warn]  :: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
[warn]  ::::::::::::::::::::::::::::::::::::::::::::::
[info] Updating {file:/Users/oscarb/workspace/chill/}chill-java...
[info] Resolving com.esotericsoftware.kryo#kryo;2.21 ...
[info] Updating {file:/Users/oscarb/workspace/chill/}chill-all...
[info] Resolving org.scala-tools.testing#specs_2.9.3;1.6.9 ...
[info] Done updating.
[info] Resolving org.objenesis#objenesis;1.2 ...
[info] Updating {file:/Users/oscarb/workspace/chill/}chill-hadoop...
[info] Resolving org.scala-tools.testing#specs_2.9.3;1.6.9 ...
[info] Done updating.
[info] Resolving log4j#log4j;1.2.17 ...
[info] Done updating.
sbt.ResolveException: download failed: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
download failed: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src
    at sbt.IvyActions$.sbt$IvyActions$$resolve(IvyActions.scala:214)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:122)
    at sbt.IvyActions$$anonfun$update$1.apply(IvyActions.scala:121)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
    at sbt.IvySbt$Module$$anonfun$withModule$1.apply(Ivy.scala:117)
    at sbt.IvySbt$$anonfun$withIvy$1.apply(Ivy.scala:105)
    at sbt.IvySbt.liftedTree1$1(Ivy.scala:52)
    at sbt.IvySbt.action$1(Ivy.scala:52)
    at sbt.IvySbt$$anon$3.call(Ivy.scala:61)
    at xsbt.boot.Locks$GlobalLock.withChannel$1(Locks.scala:98)
    at xsbt.boot.Locks$GlobalLock.withChannelRetries$1(Locks.scala:81)
    at xsbt.boot.Locks$GlobalLock$$anonfun$withFileLock$1.apply(Locks.scala:102)
    at xsbt.boot.Using$.withResource(Using.scala:11)
    at xsbt.boot.Using$.apply(Using.scala:10)
    at xsbt.boot.Locks$GlobalLock.ignoringDeadlockAvoided(Locks.scala:62)
    at xsbt.boot.Locks$GlobalLock.liftedTree1$1(Locks.scala:52)
    at xsbt.boot.Locks$GlobalLock.withLock(Locks.scala:52)
    at xsbt.boot.Locks$.apply0(Locks.scala:31)
    at xsbt.boot.Locks$.apply(Locks.scala:28)
    at sbt.IvySbt.withDefaultLogger(Ivy.scala:61)
    at sbt.IvySbt.withIvy(Ivy.scala:102)
    at sbt.IvySbt.withIvy(Ivy.scala:98)
    at sbt.IvySbt$Module.withModule(Ivy.scala:117)
    at sbt.IvyActions$.update(IvyActions.scala:121)
    at sbt.Classpaths$$anonfun$work$1$1.apply(Defaults.scala:955)
    at sbt.Classpaths$$anonfun$work$1$1.apply(Defaults.scala:953)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$58.apply(Defaults.scala:976)
    at sbt.Classpaths$$anonfun$doWork$1$1$$anonfun$58.apply(Defaults.scala:974)
    at sbt.Tracked$$anonfun$lastOutput$1.apply(Tracked.scala:35)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:978)
    at sbt.Classpaths$$anonfun$doWork$1$1.apply(Defaults.scala:973)
    at sbt.Tracked$$anonfun$inputChanged$1.apply(Tracked.scala:45)
    at sbt.Classpaths$.cachedUpdate(Defaults.scala:981)
    at sbt.Classpaths$$anonfun$47.apply(Defaults.scala:858)
    at sbt.Classpaths$$anonfun$47.apply(Defaults.scala:855)
    at sbt.Scoped$$anonfun$hf10$1.apply(Structure.scala:586)
    at sbt.Scoped$$anonfun$hf10$1.apply(Structure.scala:586)
    at scala.Function1$$anonfun$compose$1.apply(Function1.scala:49)
    at sbt.Scoped$Reduced$$anonfun$combine$1$$anonfun$apply$12.apply(Structure.scala:311)
    at sbt.Scoped$Reduced$$anonfun$combine$1$$anonfun$apply$12.apply(Structure.scala:311)
    at sbt.$tilde$greater$$anonfun$$u2219$1.apply(TypeFunctions.scala:41)
    at sbt.std.Transform$$anon$5.work(System.scala:71)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:232)
    at sbt.Execute$$anonfun$submit$1$$anonfun$apply$1.apply(Execute.scala:232)
    at sbt.ErrorHandling$.wideConvert(ErrorHandling.scala:18)
    at sbt.Execute.work(Execute.scala:238)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:232)
    at sbt.Execute$$anonfun$submit$1.apply(Execute.scala:232)
    at sbt.ConcurrentRestrictions$$anon$4$$anonfun$1.apply(ConcurrentRestrictions.scala:160)
    at sbt.CompletionService$$anon$2.call(CompletionService.scala:30)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
    at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
    at java.util.concurrent.FutureTask.run(FutureTask.java:166)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
    at java.lang.Thread.run(Thread.java:722)
[error] (chill/*:update) sbt.ResolveException: download failed: org.scalacheck#scalacheck_2.9.3;1.10.0!scalacheck_2.9.3.src
[error] download failed: org.scala-tools.testing#specs_2.9.3;1.6.9!specs_2.9.3.src

Wrong SerDe of mutable.Map[Int, mutable.Set[AnyRef]]

When doing a roundtrip over a mutable Map of mutable Sets, the result is not the same. It seems that it reuses each mutable.Set as the basis for the next.

This happens with Kryo 2.21 and Chill 0.2.3

Example:

val obj = mutable.Map(4 -> mutable.Set("house1", "house2"), 1 -> mutable.Set("name3", "name4", "name1", "name2"), 0 -> mutable.Set(1, 2, 3, 4))

MeatLocker(obj).copy // => Map(4 -> Set(house1, house2), 1 -> Set(name3, house1, house2, name4, name1, name2), 0 -> Set(name3, house1, 1, house2, 2, 3, name4, name1, 4, name2))

Add in a cross-build against 2.10

Move README over from Cascading.Kryo

I'm not sure how to configure the Hadoop config using the new system.

ClosureCleaner.apply should return the cleaned input

Handle iterables

The specific instance I ran into was MapLike.DefaultValuesIterable, but I bet there are other similar classes that also don't serialize.

mutale.ArrayBuffer gets incorrectly deserialized as a Vector

scala> import com.twitter.chill._
import com.twitter.chill._

scala> import scala.collection._
import scala.collection._

scala> val a = mutable.ArrayBuffer(1,2,3,4,5)
a: scala.collection.mutable.ArrayBuffer[Int] = ArrayBuffer(1, 2, 3, 4, 5)

scala> KryoInjection(a)
res0: Array[Byte] = Array(1, 0, 115, 99, 97, 108, 97, 46, 99, 111, 108, 108, 101, 99, 116, 105, 111, 110, 46, 109, 117, 116, 97, 98, 108, 101, 46, 65, 114, 114, 97, 121, 66, 117, 102, 102, 101, -14, 1, 5, 2, 2, 2, 4, 2, 6, 2, 8, 2, 10)

scala> KryoInjection.invert(KryoInjection(a))
res1: Option[java.lang.Object] = Some(Vector(1, 2, 3, 4, 5))

Add support for deserializing InputStream and ByteBuffer

Currently, Chill only supports Array[Byte].

Kryo supports InputStream:
http://kryo.googlecode.com/svn/api/v2/com/esotericsoftware/kryo/io/Input.html

In chill-hadoop check Kryo/Buffer creation.

It is costly to create a kryo instance and right now the code is doing it on every open. Cascading calls open after almost every entry in the tuple, so this could kill performance.

We do this because we think (fear? have been burned?) by Kryo being mutable. What we want is an immutable Kryo instance that is only used for serialization and will not change any tokens or register any new ones.

Give Ryan LeCompte commit access

@caniszczyk Can you hook up @ryanlecompte ? He has help review and contributed code. Additionally he helped get chill merged in Spark.

Ryan: FYI our rule is: never merge code you write yourself. Always ping another committer. Also, only merge when travis is green (obviously).

Handle references correctly

https://code.google.com/p/kryo/issues/detail?id=98

Deal with inner case classes

inner case classes seem to capture references to outer even if they are not used.

If we can find some cheap way to detect this, it would really help scalding and possibly spark.

Add support for Scala continuations

It seems that Chill cannot serialize and deserialize Scala continuations.

chill scrooge

For scrooge generated objects, make sure to use scrooge's thrift serialization.

Update Readme to explain usage

Explain how to use the library. Probably just something like: KryoSerializer.registerAll(k) on a k: Kryo which might have been generated from (new KryoSerializer { }).getKryo

Don't use scala.collection.Map

https://github.com/twitter/chill/blob/develop/chill-scala/src/main/scala/com/twitter/chill/KryoSerializer.scala#L118

This is (I think) wrongly saying that scala.collection.Map is immutable, when in fact that class may or may not be mutable.

Deserialization of list of list or case class that contains list

I don't know if I use chill as it supposed to be used, but here are some trivial failing cases that I found?

test("int list serialize/deserialize") {
    val listObj = List(
      List(List(1)),
      List(List(1))
    )

    val serializedObject = KryoBijection.serialize(listObj)
    val deserializedObj = KryoBijection.deserialize[List[List[Int]]](serializedObject)

    assert(listObj === deserializedObj)
  }

Results in
List(List(List(1)), List(List(1))) did not equal List(1, List(1), List(1, List(1)))

test("int list container serialize/deserialize") {
    val listObj = List(
      ListContainer(List(1)),
      ListContainer(List(1))
    )

    val serializedObject = KryoBijection.serialize(listObj)
    val deserializedObj = KryoBijection.deserialize[List[ListContainer]](serializedObject)

    assert(listObj === deserializedObj)
  }

case class ListContainer(ints: List[Int])

Results in
List(ListContainer(List(1)), ListContainer(List(1))) did not equal List(1, ListContainer(List(1)))

Wanted: chill-guava

Java only module to serialize the classes in guava.

Copy the Map fix for Set

See: #60

Set is also special cased for small sets.

KryoInjection should just catch exceptions

It would be great if KryoInjection didn't use allCatch but instead just caught Exceptions. Deserializing potentially large objects is a good opportunity for OOMEs.

https://github.com/twitter/chill/blob/develop/chill-scala/src/main/scala/com/twitter/chill/KryoBijection.scala#L96

https://github.com/scala/scala/blob/master/src/library/scala/util/control/Exception.scala#L150

issue with specialized Tuple's in WrappedArray

Hi guys,

We at Spark ran into a problem using Chill to serialize WrappedArray. See the following code. It looks like the problem is with a WrappedArray for specialized tuples. If I replace the ints with string (and thus no specialized Tuple2), the exception goes away.

This is becoming a major problem for the Spark project because anything with primitive WrappedArray would fail.

@mateiz, @dlyubimov

scala> val ser = new spark.KryoSerializer
ser: spark.KryoSerializer = spark.KryoSerializer@7798ce82

scala> ser.newInstance.deserialize[Array[(Int, Int)]]( ser.newInstance.serialize(Array((1, 2), (2, 3)))  )
res12: Array[(Int, Int)] = Array((1,2), (2,3))

scala> ser.newInstance.deserialize[Seq[(Int, String)]]( ser.newInstance.serialize(Array((1, "2"), (2, "3")).toSeq)  )
res20: Seq[(Int, String)] = WrappedArray((1,2), (2,3))

scala> ser.newInstance.deserialize[Seq[(Int, Int)]]( ser.newInstance.serialize(Array((1, 2), (2, 3)).toSeq)  )
java.lang.ArrayIndexOutOfBoundsException: -2
  at java.util.ArrayList.get(ArrayList.java:324)
  at com.esotericsoftware.kryo.util.MapReferenceResolver.getReadObject(MapReferenceResolver.java:42)
  at com.esotericsoftware.kryo.Kryo.readReferenceOrNull(Kryo.java:773)
  at com.esotericsoftware.kryo.Kryo.readObject(Kryo.java:624)
  at com.twitter.chill.WrappedArraySerializer$$anonfun1.apply(WrappedArraySerializer.scala:44)
  at com.twitter.chill.WrappedArraySerializer$$anonfun1.apply(WrappedArraySerializer.scala:43)
  at scala.collection.immutable.Range.foreach(Range.scala:81)
  at com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:43)
  at com.twitter.chill.WrappedArraySerializer.read(WrappedArraySerializer.scala:21)
  at com.esotericsoftware.kryo.Kryo.readClassAndObject(Kryo.java:729)
  at spark.KryoSerializerInstance.deserialize(KryoSerializer.scala:75)
  at <init>(<console>:13)
  at <init>(<console>:18)
  at <init>(<console>:20)
  at <init>(<console>:22)
  at <init>(<console>:24)
  at .<init>(<console>:28)
  at .<clinit>(<console>)
  at .<init>(<console>:11)
  at .<clinit>(<console>)
  at $export(<console>)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
  at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
  at java.lang.reflect.Method.invoke(Method.java:597)
  at spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:629)
  at spark.repl.SparkIMain$Request$$anonfun$10.apply(SparkIMain.scala:890)
  at scala.tools.nsc.interpreter.Line$$anonfun$1.apply$mcV$sp(Line.scala:43)
  at scala.tools.nsc.io.package$$anon$2.run(package.scala:25)
  at java.lang.Thread.run(Thread.java:680)

There is some serious code smell going on here. Ideally we could get storm to depend on chill-java and share this java-only code.

https://github.com/twitter/chill/blob/develop/chill-scala/src/main/scala/com/twitter/chill/KryoRegistrationHelper.scala

https://github.com/twitter/chill/blob/develop/chill-hadoop/src/main/java/com/twitter/chill/hadoop/KryoFactory.java

https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/serialization/IKryoFactory.java

https://github.com/nathanmarz/storm/blob/master/storm-core/src/jvm/backtype/storm/serialization/IKryoDecorator.java