Giter VIP home page Giter VIP logo

boopickle's Introduction

BooPickle

Join the chat at https://gitter.im/ochrons/boopickle

Build Status Scala.js

BooPickle is the fastest and most size efficient serialization (aka pickling) library that works on both Scala and Scala.js. It encodes into a binary format instead of the more customary JSON. A binary format brings efficiency gains in both size and speed, at the cost of legibility of the encoded data. BooPickle borrows heavily from both uPickle and Prickle so special thanks to Li Haoyi and Ben Hutchison for those two great libraries!

Features

  • Supports both Scala, Scala Native and Scala.js (no reflection!)
  • Serialization support for all primitives, collections, options, tuples and case classes (including class hierarchies)
  • User-definable custom serializers
  • Transforming serializers to simplify serializing non-case classes
  • Handles references and deduplication of identical objects
  • Very fast
  • Very efficient coding
  • Low memory usage, no intermediate structures needed
  • Zero dependencies
  • Scala 2.12/2.13
  • All modern browsers are supported (not IE9 and below, though)

Getting started

Add following dependency declaration to your Scala project

"io.suzaku" %% "boopickle" % "1.4.0"

On a Scala.js / Scala Native project the dependency looks like this

"io.suzaku" %%% "boopickle" % "1.4.0"

To use it in your code, simply import the Default object contents. All examples in this document assume this import is present.

import boopickle.Default._

To serialize (pickle) something, just call Pickle.intoBytes with your data. This will produce a binary ByteBuffer containing an encoded version of your data.

val data = Seq("Hello", "World!")
val buf = Pickle.intoBytes(data)

And to deserialize (unpickle) the buffer, call Unpickle.fromBytes, specifying the type of your data. BooPickle doesn't encode any type information, so you must use the same types when pickling and unpickling.

val helloWorld = Unpickle[Seq[String]].fromBytes(buf)

Documentation

Read the full documentation

Change history

See a separate changes document

Contributors

BooPickle was created and is maintained by Otto Chrons - [email protected] - Twitter: @ochrons.

Special thanks to Li Haoyi and Ben Hutchison for their pickling libraries, which provided more than inspiration to BooPickle.

Contributors: @japgolly, @FlorianKirmaier, @guersam, @akshaal, @cquiroz, @cornerman, @notxcain, @lolgab, @Philippus

boopickle's People

Contributors

akshaal avatar benhutchison avatar cornerman avatar cquiroz avatar fdietze avatar floriankirmaier avatar fomkin avatar gitter-badger avatar guersam avatar ilinandrii avatar japgolly avatar lihaoyi avatar lolgab avatar mathieuleclaire avatar notxcain avatar ochrons avatar olivierblanvillain avatar philippus avatar scala-steward avatar slandelle avatar tbje avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

boopickle's Issues

Scala.js 1.0.0-M1 support

It would be much appreciated if you could add support for Scala.js 1.0.0-M1.
Thank you very much, love your work!

Won't compile for a combination of nested Seq/case classes

boopickle 1.2.6
This is the minimal test case I've come with:

case class Element(data: Option[Seq[String]])
case class Container(e: Seq[Element])

object Test {
  import boopickle.Default._

  def serializer(c: Container) = Pickle.intoBytes(c)
}

The error is:

Error:(7, 50) Cannot materialize pickler for non-case class: Seq[Element]. If this is a collection, the error can refer to the class inside.
  def serializer(c: Container) = Pickle.intoBytes(c)

Incorrect results when pickling numeric strings in Scala.js

I am pickling numeric strings and I started to notice some were showing up on the server as totally different. After some investigating it seems (?) like 20 digit strings are not pickling/unpickling correctly in Scala.js. See below

import boopickle.Default._
val numbers = Seq(0,1,2,3,4,5,6,7,8,9)
for (_  0 until 10) {
  val s = Array.fill(20)(numbers(Random.nextInt(10))).mkString
  println(s)
  println(Unpickle[String].fromBytes(Pickle.intoBytes(s)))
  println("---------")
}

produces:

58213468898124653887
2873236676995999039
---------
45248643522829592471
8355155375410489239
---------
51496868058242682400
-3843364162885972448
---------
27982541370036899703
-8910946777382203529
---------
27492121753843503624
9045377680133952008
---------
15819678712131102020
-2627065361578449596
---------
50046200309684612602
-5294031911444042246
---------
36950005496042406970
56517348623303738
---------
32035983838405348908
-4857504309013754324
---------
33830692605305806048
-3062795542113297184
---------

Strings of length 19:

9220080788241269241
9220080788241269241
---------
7324346491285683657
7324346491285683657
---------

Strings of length 21:

725745014888300153818
725745014888300153818
---------
562803327751322974960
562803327751322974960
---------

Is it worth it having Pickler & Unpickler separation?

As mentioned here, BooPickle is unlike other serialisation libraries in that the pickled (binary) data contains no general or human-readable information indicative of structure or semantics.

Picklers and Unpicklers must always be in sync or else the protocol fails.

Therefore is there any value in having separate Picklers and Unpicklers instead of just a unified Codec or Protocol? Is there is some value is it enough to justify the downsides? Some of the downsides I'm noticing:

  • Anytime you create a manual pickle, you need to repeat yourself twice. (CompositePickler + CompositeUnpickler, the materializePickler pair, etc.)
  • Easy for pickle/unpickle to get out of sync. (especially as order matters so)

I can understand why it was separated originally but now that the library is being put to use, I'm thinking that this library and usage of it, would be greatly simplified by just having one typeclass instead of two.

Backwards compatibility

Would be great to have the ability to specify field tags to let server speak to let server speak to old clients, like this:

case class Message(@tag(0) field1, @deprecated @tag(1) field2, @tag(2) field3)

Fields marked as @deprecated should be read (if present in bytes) but never written.

Abstract class based sealed hierarchy.

As of cfc5d7a, Boopickle 1.1.0 can automatically generate picklers for a sealed class hierarchy.

It seems to be working when the root is a trait, but not when it's an abstract class:

sealed abstract class Version(val number: Int)
case object V1 extends Version(1)
case object V2 extends Version(2)

import boopickle.Default._

val v: Version = V1
val bytes = Pickle.intoBytes(v)
println(Unpickle[Version].fromBytes(bytes).number)

Cannot materialize pickler for non-case class: Version

Regards

Make EncoderSize non final

Hi @ochrons

Is there any reason why EncoderSize is final (DecoderSize isn't)?
I want to implement my own Int and Long encoding strategies, as I only deal with positive values, and would rather do so by extending existing classes. I currently can do that with the decoder size, but I have to copy-paste the encoder.

Cheers!

nothingPickler = trouble

After chasing down errors that didn't make sense for over 30 min I realised that BasicImplicitPicklers contains a Pickler[Nothing]. Having an instance of any typeclass with Nothing in the hole causes all kinds of problems with Scala's implicit resolution.

I recommend it be removed entirely (what purpose is there supporting runtime failure out-of-the-box?).

Provide an example Play implementation

I'm trying to use boopickle in my Play framework project. However I quickly ran into troubles working with ByteBuffer in Play. I've searched github to find a project that is using boopickle with play framework, but I wasn't able to find anything.

Could you please provide an example? AFAIK Play is the most popular scala framework so this would probably help a lot of potential users of this lib.

Case class with varargs doesn't compile

scala> import boopickle.Default._
import boopickle.Default._

scala> case class Foo(i: Int*)
defined class Foo

scala> Pickle.intoBytes(Foo(1,2,3,4,5))
<console>:16: error: type mismatch;
 found   : Int*
 required: Int
       Pickle.intoBytes(Foo(1,2,3,4,5))

Type Parameters of sealed Traits are not forwarded correctly

This bug is similar too #22

When a Unpickler is generated for a Trait, then the Type Arguments are not correctly forwarded to the Sub-Types.

File1

  sealed trait A1Trait[T]
  case class A1[T](i: Double) extends A1Trait[T]

File2 (because of SI-7046)

boopickle.Unpickler.materializeUnpickler[A1Trait[Int]]

compileroutput (with boopickler 1.0.1-SNAPSHOT):

[error] C:\Users\florian.kirmaier\Dropbox\git\File2.scala:31: cla
ss A1 takes type parameters
[error]   val Bunpickler = boopickle.Unpickler.materializeUnpickler[A1Trait[Int]]
[error]                                                            ^
[error] one error found
[error] (scalaServer/compile:compile) Compilation failed
[error] Total time: 23 s, completed Jun 23, 2015 7:14:11 PM

The function materializeUnpickler generates the following AST:

result: Expr[boopickle.Unpickler[A1Trait[Int]]]({
  implicit object TraitUnpickler$macro$1 extends boopickle.CompositeUnpickler[A1Trait[Int
]] {
    def <init>() = {
      super.<init>();
      ()
    };
    addConcreteType[A1]
  };
  TraitUnpickler$macro$1
})

The type arguments for A1 are completly missing which is the cause of the error.

TransformPickler type-inference

On the topic of type-inference, transformPickler would benefit from currying the arguments so that you would only need to specify one instead of two.

So instead of

transformPickler[Foo, Bar](_.bar, new Foo(_))

you could do

transformPickler((_: Foo).bar)(new Foo(_))

I find that swapping the args around (B=>A)(A=>B) works specially well with case classes as you don't need to specify any types manually.

def transformPickler[A, B: Pickler](ba: B => A)(ab: A => B): Pickler[A] = ...

// Example case class
case class SomeId(value: Long)

// SomeId is translated to SomeId.apply here so compiler infers A & B
val p = transformPickler(SomeId)(_.value)

// confirm type-inference worked
val p2: Pickler[SomeId] = p

Up to you of course :)

Macros fail on path-dependent types

Consider this:

  sealed trait AtomBase {
    sealed trait Atom
    case class Zero(value: String) extends Atom
  }
  trait Atom1 extends AtomBase {
    case class One(value: String) extends Atom
  }
  trait Atom2 extends AtomBase {
    case class Two(value: String) extends Atom
  }
  object Atoms01 extends AtomBase with Atom1
  object Atoms02 extends AtomBase with Atom2

Atoms01 contains only Zero and One; Atoms02 contains only Zero and Two.
This structure is interpreted correctly by Scala. You can add or remove cases from the snippet below to see Scala generates warnings/errors correctly.

  def testExhaustiveness01: Atoms01.Atom => Unit = {
    case _: Atoms01.Zero => ()
    case _: Atoms01.One => ()
  }

  def testExhaustiveness02: Atoms02.Atom => Unit = {
    case _: Atoms02.Zero => ()
    case _: Atoms02.Two => ()
  }

This crashses:

materializePickler[Atoms01.Atom]

It should expect two cases:

  • Atoms01.Zero
  • Atoms01.One

But instead it tries to generate a pickler with cases:

  • AtomBase.this.Zero
  • AtomBase.this.One
  • AtomBase.this.Two

Not wanting to generate double the boilerplate code, instead of using the built-in boopickle macros I'm generating Pickler with Unpicklers using my own macro code I wrote for other things, I'm getting the same error there too. I'll keep playing with it and write back if I get it working. If you know how or beat me to it, save me! I'm tired.

Support unpickling a Seq[ByteBuffer]

Payloads might be transmitted over multiple paquets.

Currently, the only way is unpickle such payload is to copy into a global large ByteBuffer.
It could be possible to save this extra copy, and support unpickling a sequence of ByteBuffers. The only trick is you have still to use an intermediate ByeBuffer when a value is split over 2 consecutive ByteBuffers.

Wrong compiler-error when the inner type can not be serialized

At the following case the error message is realy missliading/wrong.
Boopickle should tell the user, that ATest could not be serialized instead of Seq which is not true.

class ATest

// works fine
boopickle.Unpickle[Seq[String]].fromBytes(e.data.asInstanceOf[ByteBuffer])
// this one is missleading and wrong
boopickle.Unpickle[Seq[ATest]].fromBytes(e.data.asInstanceOf[ByteBuffer])
// this one contains an informative error messages
boopickle.Unpickle[ATest].fromBytes(e.data.asInstanceOf[ByteBuffer]) 
[error] C:\Users\florian.kirmaier\Dropbox\git\Sample.scala:49: The referenced trait [[Seq]] must be sealed. For non-sealed traits, create a pickler with boopickle.CompositePickler. You may also get this error if a pickler for a class in your type hierarchy cannot be found.
[error]     boopickle.Unpickle[Seq[ATest]].fromBytes(e.data.asInstanceOf[ByteBuffer])
[error]                       ^
[error] C:\Users\florian.kirmaier\Dropbox\git\Sample.scala:50: Cannot materialize pickler for non-case class: simplefx.html.SimpleFXElement.ATest
[error]     boopickle.Unpickle[ATest].fromBytes(e.data.asInstanceOf[ByteBuffer])
[error]                       ^
[error] two errors found
[error] (scalajs/compile:compile) Compilation failed
[error] Total time: 8 s, completed Jun 22, 2015 5:17:45 PM

converging with Prickle

What I see is that boopickle is very similar to Prickle in terms of API but uses binary format instead of json one. I wonder is it possible to separate API from the format in a way that by changing one implicit I will be able to switch between json and binary and vice versa?

Provide a way to configure the type of ByteBuffer

Encoder currently uses direct ByteBuffers.

But like with most packing libraries, it's quite common to apply compression after packing.
Sadly, JDK's Deflater still doesn't support being passed a direct ByteBuffer, but only takes a byte array.
This forces users to do an additional array allocation + copy.

It could be great if Encoder could be more configurable, so one can easily override defaults and create his own PickleState.

Null on unpickling class hierarchy

This maybe a bug or just something to be documented. If I unpickle a class on a hierarchy using the parent class boopickle produces a null:

        sealed trait A
        case class B(v: Int) extends A

        val bytes = Pickle.intoBytes(B(1))
        println(Unpickle[B].fromBytes(bytes)) // This produces B(1)
        val bytes2 = Pickle.intoBytes(B(1))
        println(Unpickle[A].fromBytes(bytes2)) // This produces null

A workaround is to mark the pickled object with the root class

        val o: A = B(1)
        val bytes3 = Pickle.intoBytes(o)
        println(Unpickle[A].fromBytes(bytes3)) // This produces B(1)

Problem with huge numeric strings

A string with value "13065136436328809018" currently gets decoded as "-5381607637380742598". It seems that this is due to boopickle's optimizations for numeric strings - I believe the bounds on line Pickler.scala#L154 are too wide.

TransformPickler needlessly complex

I finally just started playing with this lib, and started out importing just BasicImplicitPicklers.

TransformPickler wouldn't work. It turns out I need to import TransformPicklers.

Why?

In place of TransformPickler and the TransformPicklers special conversions, they can be replaced by returning a Pickler[A] with Unpickler[A].

Benefits:

  • Less mem: Creates 1 class instead of 3.
  • Faster compiler time: Less conversions for compiler implicit resolution.
  • Less types for lib consumers to manage. μPickle did this really well: only 3 types: Reader, Writer, type ReadWriter[A] = Reader[A] with Writer[A].

The above benefits could be extended to all places picklers and unpicklers are created in unison. As a library consumer I felt it worked really well :)

Here is a rewrite of TransformPickler that I whipped up.

def xmap[A <: AnyRef, B](ba: B => A)(ab: A => B)(implicit p: Pickler[B], up: Unpickler[B]): Pickler[A] with Unpickler[A] =
  new Pickler[A] with Unpickler[A] {
    override def pickle(obj: A)(implicit state: PickleState): Unit =
      p.pickle(ab(obj))
    override def unpickle(implicit state: UnpickleState): A =
      ba(up.unpickle(state))
}

Example usage:

case class Rev(value: Long)

// Signature change allows better type-inference
implicit val rev = xmap(Rev)(_.value)

// Compiles with no additional implicits
implicitly[Pickler[Rev]]
implicitly[Unpickler[Rev]]

compositePickler.addConcreteType inside class does not compile

Hi, this compiles:

sealed trait F
case object A extends F
implicit def fp = compositePickler[F].addConcreteType[A.type]

while this does not:

class X {
  sealed trait F
  case object A extends F
  implicit def fp = compositePickler[F].addConcreteType[A.type]
}
error:
<macro>:1: error: value X is not a member of object $iw
 Note: implicit method fp is not applicable here because it comes after the application point and it lacks an explicit result type
X.A
                                  ^

Is this expected behavior or a bug?

A Combination of Lists, Case Classes and Tuples does not compile.

A simple combination of Lists, Cases Classes and Tuples does not compile.
I see no reason, why it shouldn't work.
I minimized the example as far as i could.
Note that it compiles when we remove one of the lists or change the tuple-type to a single Double or a Case Class.

  case class A(fills: List[B])
  case class B(stops: List[(Double,Double)])

  boopickle.Unpickle[A].fromBytes(null.asInstanceOf[ByteBuffer])

compileroutput with boopickle 1.0.0:

[error] C:\Users\florian.kirmaier\Dropbox\git\file.scala:32: Can
not materialize pickler for non-case class: scala.collection.immutable.List
[error]   boopickle.Unpickle[A].fromBytes(null.asInstanceOf[ByteBuffer])
[error]                     ^
[error] one error found
[error] (scalaServer/compile:compile) Compilation failed
[error] Total time: 59 s, completed Jun 23, 2015 11:52:35 AM

Boopickle modifies original ByteOrder

Hi,

It's probably quite common to have to mix pickling with writing raw bytes/ints. A typical example is prepending a frame length length.

Being the default on the JVM, and for compatibility with other libraries (Netty in my case), I would like to be able to use BigEndian.

But then, I ran into some issue because Boopickle changes the byte order of the ByteBuffer it's being feed. I believe it happens here.

I was able to work around this by restoring the original byte order before reading raw data, but this sure is tricky. If possible, it would be more convenient if Boopickle could honor the original byte order.

Regards.

Sync source maps and tagging scheme

You've tagged the first release as 0.1.0. Convention on most other projects I looked at (ages ago) is to have a v prefix, so v0.1.0. Accordingly, that's what the source maps are configured to point to. See https://github.com/ochrons/boopickle/blob/master/build.sbt#L56

  1. I think you need to either prefix versions when you tags, or remove the prefix from SBT so they're in sync.
  2. If 0.1.0 is out the door already, you should probably copy or rename the tag so that v0.1.0 exists as it will already be compiled to expect it.

Initial size padding

Hello there,

I have been looking through the code and documentation trying to figure out if there's a method to shave off the initial buffer (which allocates to the default size of 1400) after Pickling an element.

My current solution is just to extract the native ArrayBuffer and just splice before sending it across the wire like so:

  def send(message: Message) = {
    val pickled = Pickle.intoBytes(message)
    val typedBuffer = new TypedArrayBufferOps(pickled).arrayBuffer()

    ws.send(typedBuffer.slice(0, pickled.limit))
}

While it's not too much work, since its possible to Pickle Sequence and other container alike data structures in a single command, technically there shouldn't be any mutations to be applied to the buffer. Is there some other reason for this design choice?

Thanks for reading and apologies in advanced if this isn't considered an issue =)

websockets example

Websockets require ArrayBuffer but I do not understand how I can turn bytebuffer form boopickler into either arraybuffer or blob that websocket wants, example in docs would be rather helpful.

Claim you're the fastest ever? Weaken your claims or prove it.

I find the READE me rather arrogant.

BooPickle is the fastest and most size efficient serialization (aka pickling) library for Scala and Scala.js.

Doubt that it's the fastest library for Scala. There are significantly more serialization libraries than you're benchmarking against. While my experience is limited to Scala Pickling, I can attest that Scala Pickling was benchmarked to be up to 6x faster than Kryo, so it was faster than mainstream Java serialization frameworks as well. We've not done any performance regression testing and have done a lot of refactoring in the past year or so, so it could be that something we did in the past year or so might've killed that lead that we had. But I still seriously doubt your 5/1000/10000 element microbenchmarks and your over the top claims that boopickle is the fastest serialization library in the world. As far as benchmarking serialization goes, these nanobenchmarks will tell you little.

But don't go claiming you're the best at something unless that is indeed true. Maybe this is the scientist in me getting annoyed, but your README reads to me more marketing and less truth. Maybe verify your claims and benchmark every Scala/Scala.js serialization framework with something other than nanobenchmarks, or reduce the over the top muscle-flexing in your README.

Also something I find very suspicious – the design of boopickle very closely resembles Scala Pickling, yet there's no mention of it anywhere. All the way down to à la carte pickling. The resemblance is uncanny. Really odd to me that you'd take a bunch of inspiration seemingly from Pickling, not mention it anywhere, and claim superiority so flagrantly. Not a good way to play along nicely in open source.

Scala 2.12 milestones support

Hi @ochrons

Would it be possible to publish cross-built versions so projects depending on boopickle can give Scala 2.12 milestones a try, please?

Regards

Exception during macro expansion:

I would like to write a generic serializer,

  def toBytes[T](event: T): Array[Byte] = {
    val bb = Pickle.intoBytes[T](event)
    val bytes = bb.array()
    //release buffer to pool
    BufferPool.release(bb)
    bytes
  }

  def fromBytes[T](bytes: Array[Byte]): T = {
    val bb = ByteBuffer.wrap(bytes)
    val event = Unpickle[T].fromBytes(bb)
    BufferPool.release(bb)
    event
  }

however

 I get exception during macro expansion: 
  at boopickle.PicklerMaterializersImpl$.materializePickler(PicklerMaterializersImpl.scala:71)
[error]     val bb = Pickle.intoBytes(event)

 exception during macro expansion: 
    at boopickle.PicklerMaterializersImpl$.materializePickler(PicklerMaterializersImpl.scala:71)
[error]     val event = Unpickle[T].fromBytes(bb)

Can't pickle/unpickle composite Maps

Hi Otto,

Trying to pickle and unpickle composites such as Map[String, Map[String, Int]] doesn't compile.

The workaround I currently use is to wrap the values into a case class like case class MapWrapper(map: Map[String, Int]) and then I can use Map[String, MapWrapper] but that's cumbersome.

Advice?

ByteBuffer to JavaScript bytes

At the section 'Using ByteBuffers in network communication' of the documentation you write:

On the JS side things are a bit more complicated due to the use of JavaScript ArrayBuffer underneath the ByteBuffer.

I would like to see an example!

Performance and size differences - possible regression in 1.2.2

I have observed since upgrading from 1.1.3 an exponential increase in pickling time (~ 1sec vs 1.5 minutes) of a map [String -> case class(simple strings and ints)] with around 350K members using the defaults for pickling/unpickling.

Also the size of the pickle roughly doubled.

Thank you for the pointers on the gitter channel.

Cheers!
Russ

Cannot materialize pickler for non-case class: Int.

I am playing with this library and cannot pickle simple case class. My code looks like this:

import boopickle.Default._

case class Foo(a: Int)

object Main {
  def main(args: Array[String]) {
    val boo = Pickle.intoBytes(Foo(-1))
  }
}

But I get strange error:
Cannot materialize pickler for non-case class: Int. If this is a collection, the error can refer to the class inside.
[error] val booPickle = Pickle.intoBytes(Foo(1))
[error] ^

What am I missing? Thanks for feedback.

Runtime error in JS and JVM: generics

This one crashes on both JS and JVM when calling Pickle.intoBytes[T](Ex(a)):
(Fiddle)

import boopickle.Default._

object data {
  sealed trait T
  case class Ex[A](a:A) extends T

  object T {
    def pickler[A : Pickler] = compositePickler[T].addConcreteType[Ex[A]]
  }
}
import data._

trait Server[A] {
  implicit val aPickler:Pickler[A]
  implicit val tPickler = T.pickler[A]
  def send(a: A) {
    println(Pickle.intoBytes[T](Ex(a)))
  }
}

object ServerImpl extends Server[Int] {
  val aPickler = implicitly[Pickler[Int]]
}

@js.annotation.JSExport
object ScalaFiddle {
  
  ServerImpl.send(5) 
}

Error in JS:

ERROR: Cannot read property 'pickle__O__Lboopickle_PickleState__V' of null

Error on JVM:

@ import $ivy.`me.chrons::boopickle:1.2.5`, boopickle.Default._ 
import $ivy.$                           , boopickle.Default._
@ {
  import boopickle.Default._
  
  object data {
    sealed trait T
    case class Ex[A](a:A) extends T
  
    object T {
      def pickler[A : Pickler] = compositePickler[T].addConcreteType[Ex[A]]
    }
  }
  import data._
  
  trait Server[A] {
    implicit val aPickler:Pickler[A]
    implicit val tPickler = T.pickler[A]
    def send(a: A) {
      println(Pickle.intoBytes[T](Ex(a)))
    }
  }
  
  object ServerImpl extends Server[Int] {
    val aPickler = implicitly[Pickler[Int]]
  }
  
  ServerImpl.send(5)
  } 
java.lang.NullPointerException
  boopickle.PickleState.pickle(Pickler.scala:513)
  $sess.cmd1$data$T$Pickler$macro$1$2$.pickle(cmd1.sc:8)
  $sess.cmd1$data$T$Pickler$macro$1$2$.pickle(cmd1.sc:8)
  boopickle.CompositePickler.pickle(CompositePicklers.scala:28)
  boopickle.PickleImpl$.apply(Default.scala:72)
  boopickle.PickleImpl$.intoBytes(Default.scala:77)
  $sess.cmd1$Server$class.send(cmd1.sc:17)
  $sess.cmd1$ServerImpl$.send(cmd1.sc:21)
  $sess.cmd1$.<init>(cmd1.sc:25)
  $sess.cmd1$.<clinit>(cmd1.sc:-1)

Use a recyclable ByteBuffer for encoding Strings

On my (private, sorry, will try to extract) benchmark, BooPickle is about 10% slower than MessagePack.

I suspect this comes from the way BooPickle packs Strings: standard String#getBytes(Charset).

MessagePack uses a dedicated recycled ByteBuffer and also keep a reference to the CharsetEncoder.
Those could probably be stored in the state, right?

Automatigically handled sealed traits whose implementations are case classes

@ochrons You told me to keep ideas coming, so here's one.

I'll be honest: this is out of my reach and I wouldn't be able to contribute it, so I'll perfectly understand if you decide ditch it.

Quite often, I have some sealed abstraction (trait or abstract class) where all the implementations are case classes. Currently, I have to write Picklers and Unpicklers all the way down to deal with the abstractions, which is simple yet boilerplate.

With macros, it's possible to enumerate the implementations of a sealed trait, here's an example. So, based on this, it could maybe be possible for Boopickle to automatically generate the Picklers and Unpicklers implementations for such hierarchy.

problems with tuples

import boopickle.Default._

sealed trait S
case class PStroke(colour: String) extends S
case class PImage( pair: (Int,String), ops: Seq[S])
case class OptImg(i: Option[PImage])

object TP {

  implicit val picklerS = compositePickler[S].addConcreteType[PStroke]

  def foo(o: OptImg) = {
    Pickle.intoBytes(o)
  }
}

Lovely library, many thanks, but finding some weird stuff. There's no way that I can find to serialize OptImg.. if one takes out 'pair' from PImage, the problem goes away

Compile error when using boopickle with a project that uses scalac "-Xstrict-inference"

When include boopickle 1.0.0 and turn on "-Xstrict-inference", this line:

case class A(s: String)
data: A = A("a")
Pickle.intoBytes(data)

will fail with this error message:

type mismatch;
[error]  found   : Some[A]
[error]  required: Option[Int]
[error]  Pickle.intoBytes(data)

I believe the macro generated by boopickle might contains code that cannot pass through "-Xstrict-inference".

BooPickle doesn't pack negative ints efficiently

The sample below exhibits a 39% larger size with BooPickle than with MessagePack.

import java.io.ByteArrayOutputStream
import boopickle._
import org.msgpack.core.MessagePack

val data = Array(13, -6, 3, 5, -3, -1, 0, -1, 1, -2, -4, 11, -8, 2, 0, -3, 5, -5, 2, -4, 9, -8, 3, -3, 5, -1, -3, 0, 1, 0, 3, -1, -1, 0, 4, 2, -6, -2, 2, 7, -7, 1, -2, -2, 3, 0, -2, 6)
val boopickleSize = Pickle.intoBytes(data).remaining

val out = new ByteArrayOutputStream()
val packer = MessagePack.newDefaultPacker(out)
packer.packArrayHeader(data.size)
for (i <- data)
  packer.packInt(i)
packer.close()
val msgpackSize = out.toByteArray.length

System.out.println(s"boopickle:$boopickleSize msgpack:$msgpackSize diff:${((boopickleSize - msgpackSize).toDouble / msgpackSize * 100).toInt}%")
boopickle:71 msgpack:51 diff:39%

Boopickle still not playing nicely with BIG_ENDIAN

import boopickle._
import java.nio.{ ByteBuffer, ByteOrder }

implicit val state = new PickleState(new Encoder(new HeapByteBufferProvider {
  override protected def allocate(size: Int) = super.allocate(size).order(ByteOrder.BIG_ENDIAN)
}))

val in = 2147483647
val bytes = Pickle.intoBytes(in)
val out = Unpickle[Int].fromBytes(bytes)
if (out != in)
  System.err.println(s"Pickled $in, unpickled $out")

logs "Pickled 2147483647, unpickled -129"

Type Parameters of CaseClasses are not forwarded correctly

When a Unpickler is generated for a CaseClass, then the Type Arguments are not correctly forwarded to the Inner Types.

  case class A1[T](i: A2[T])
  case class A2[T](d: Double)

  val Bunpickler = boopickle.Unpickler.materializeUnpickler[A1[Int]]

compileroutput (with boopickler 1.0.1-SNAPSHOT):

[error] C:\Users\florian.kirmaier\Dropbox\git\File.scala:32: typ
e mismatch;
[error]  found   : A2[T]
[error]  required: A2[Int]
[error]   val Bunpickler = boopickle.Unpickler.materializeUnpickler[A1[Int]]
[error]                                                            ^
[error] one error found

The function materializeUnpickler generates the following AST:

result: {
  implicit object CCUnpickler$macro$1 extends boopickle.Unpickler[A1[Int]] {
    def <init>() = {
      super.<init>();
      ()
    };
    override def unpickle(implicit state: UnpickleState): A1[Int] = {
      val ic = state.dec.readIntCode;
      if (ic.isRight.$amp$amp(ic.right.get.$eq$eq(0)))
        {
          val value = new A1[Int](state.unpickle[A2[T]]);
          state.addIdentityRef(value);
          value
        }
      else
        if (ic.isRight.$amp$amp(ic.right.get.$less(0)))
          state.identityFor[A1[Int]](ic.right.get.unary_$minus)
        else
          throw new IllegalArgumentException("Unknown object coding")
    }
  };
  CCUnpickler$macro$1
}

You can see that the Type Argument T for A2 doesn't get replaced by the inserted Type Argument Int.
This leads to an compilererror at the end.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.