heathermiller / spores Goto Github PK

This test does not compile:

@Test
def nestedPTTCaptureNullary(): Unit = {
  val s = spore{
    delayed {
      spore {
        (_: Unit) => ()
      }
      ()
    }
  }
}

If the outer nullary spore is replaced by Spore[Unit, ...] as below, it compiles:

  @Test
  def nestedPTTCapture(): Unit = {
    val s = spore{
      (_: Unit) => {
        spore {
          (_: Unit) => ()
        }
        ()
      }
    }
  }

In both cases, traverser.traverse(body) (here) is called when checking the outer spore. In both cases, the value of body is a tree that looks like (something alpha-convertible to) this when printed with showRaw:

{
  {
    class anonspore$macro$62 extends AnyRef with scala.spores.Spore[Unit,Unit] { 
      def <init>(): anonspore$macro$62 = {
        anonspore$macro$62.super.<init>();
        ()
      };
      this._className_=(this.getClass().getName());
      def apply(x$macro$61: Unit): Unit = ()
    };
    new anonspore$macro$62()
  };
  ()
}

The difference is that when the traverser goes through the constructor call anonspore$macro$62.super.<init>(), it finds an illegal This reference here in the nullary spore case. This does not happen for (_: Unit) => ...-spores.

spore.pickle() failed to compile when there is just one capture value:

val s = spore {
  val pre = "hello"
  l: List[Int] => l.map(x => s"$pre $x")
}
val d = s.pickle

with this error message:

genPickler is not a valid implicit value for scala.pickling.Pickler[scala.spores.SporeC1[List[Int],List[String]]{type Captured = String; type C1 = String}] because:
[info] hasMatchingSymbol reported error: stepping aside: repeating itself
[info]   val d = s.pickle
[info]             ^

however, it compiled successfully when there are 2 captured values:

val s = spore {
  val pre = "hello"
  val post = "world"
  l: List[Int] => l.map(x => s"$pre $x $post")
}
val d = s.pickle

Given:

class C {
  def m(i: Int): Any = "example " + i
}

package somepackage {
  package nested {
    object TopLevelObject {
      val f = new C
    }
  }
}

The following expression is rejected, because "the fun is not static":

somepackage.nested.TopLevelObject.f.m(x).asInstanceOf[String]

In this case, the problematic "fun" is m.

Expected: casting the result of calling m should be allowed, since m is selected starting from a top-level object.

Hi Heather,

can you just say a few words about

• the current state of SIP-21
• dependency on libraries ( I'n able to build with 2.10.4 and

      libraryDependencies <+= (scalaVersion)("org.scala-lang" % "scala-reflect" % _),
      libraryDependencies ++= (
        if (scalaVersion.value.startsWith("2.10")) List("org.scalamacros" % "quasiquotes" % "2.0.0-M3" cross CrossVersion.full)
        else Nil

• your current version of these libraries you are working with
• a statement of "if" and "when" spores will be integrated to Scala
• current problems / issues

Many thanks for answering so many question in advance,

Martin

Searching for something like "spores scala" shows this repo:

I knew that @jvican had done work here more recently so I had to backtrack through finding this scala-lang blog post to find out I wanted to be in https://github.com/scalacenter/spores instead.

Few different options:

1. Change the repo description and/or README to point to scalacenter/spores
2. Delete this repo - Google will index scalacenter/spores I'm guessing (& re-fork for yourself :D)
3. Fiddle with Google SEO? Not sure how possible that is.

I'm using spores-core 0.2.1.

Motivating example at SIP-21 is use of Futures in Akka, and the following code is given:

    def receive = {
      case Request(data) =>
        Future(spore {
          val from = sender
          val d = data
          () => {
            val result = transform(d)
            from ! Response(result)
          }
       })
    }

with commentary "In this case, the problematic capturing of this is avoided, since the result of this.sender is assigned to the spore's local value from when the spore is created. The spore conformity checking ensures that within the spore's closure, only from and d are used."

But the following code compiles just fine:

    def receive = {
      case Request(data) =>
        Future(spore {
          val d = data
          () => {
            val result = transform(d)
            sender ! Response(result)
          }
       })
    }

... which seems to say that spores just aren't helping, in the exact case where they're supposed to help.

What have I misunderstood?!

It works:

val s: Spore[Unit, Int] = Unit => Int

It doesn't:

val s: Spore[Unit, Int] = () => Unit

Apparently, the macro does not recognize () as Unit, while that's not the case for any Scala code in which you can use both interchangeably. This bug pollutes the user API since she's not able to use this well-known notation when converting from function to spores.

Also, I've got a question regarding to Spore[Unit, T] and NullarySpore[T]. Why is the former needed if you can represent a Function[Unit, T] as a NullarySpore? Isn't it redundant?

Hi,

Nice project!

At my org, it would be convenient for us to have spores that have structural equality in the obvious way. I could imagine that you wouldn't want that to be the default, but would you entertain a PR that created a parallel EqSpore hierarchy (with EqSporeN <: SporeN, of course) that had structural equality?

Since a spore without an environment extends now Spore[T,S] { type Captured = Nothing }, the Excluded check fails always because Nothing is the bottom type. This is a special case that needs to be worked around in the codebase.

@phaller, @heathermiller : Why does example1 die with

  checking x.*(y)...
  checking select x.*
  [error] [Spores]: (symbol) invalid reference to method *
  checking ident y
  [error] one error found

while example2 works?

object A {
  def foo(a:Int, b:Int):Int = {
    a * b * 3
  }
}

object Foo {
  def example1() : Unit = {
    val s = spore {
      val y = 3
      (x: Int) => {
          x * y
      }
    }
  }

  def example2() : Unit = {
    val s = spore {
      val y = 3
      (x: Int) => {
        A.foo(x, y)
      }
    }
  }
}

this throws a java.io.NotSerializableException

sc.parallelize(1 to 5).map(spore{(x: Int) => x * 2}).collect()

I thought this was the intended usage, am I missing something?

This looks like just the library I was hoping for, but I can't seem to find any way to specify a spore that passes sparks closure serializer.

This is with spark 1.6, scala 2.11.6

This is a severe bug.

Spores are very useful, specially when they can reference to variables in their context of creation, declaring them in the spore header. This example reproduces how such spores can't be unpickled.

  def test[T: Pickler: Unpickler](msg: T): Unit = {

    // 1. generate unpickler
    val unpickler = implicitly[Unpickler[T]]

    // 2. pickle value
    val p = msg.pickle

    val pp = p.unpickle[T]
    println("it worked!")

  }

  // Works as expected
  val s = spore[Unit, Int]{Unit => 1}
  test(s)

  // It works because the generated unpickler recognises that this
  // is not a plain spore but an instance of `SporeWithEnv`
  val n1 = 1
  val s2 = spore[Unit, Int]{
    val g = n1
    Unit => g
  }
  test(s2)  

  // Doesn't work because we have a spore with a header
  // and we're casting to `Spore` when it's actually `SporeWithEnv`
  // Therefore, the unpickler is badly generated somehow
  val n2 = 1
  val s3: Spore[Unit,Int] = spore[Unit,Int]{
    val g = n2
    Unit => g
  }
  test(s3)

For the record, this is the output:

[info] it worked!
[info] it worked!
[error] Exception in thread "main" java.lang.ArrayIndexOutOfBoundsException: 98
[error]     at scala.pickling.binary.ByteArrayInput.getByte(BinaryInput.scala:129)
[error]     at scala.pickling.binary.BinaryPickleReader$$anonfun$1.apply(BinaryPickle.scala:177)
[error]     at scala.pickling.binary.BinaryPickleReader$$anonfun$1.apply(BinaryPickle.scala:164)
[error]     at scala.pickling.PickleTools$class.withHints(Tools.scala:521)
[error]     at scala.pickling.binary.BinaryPickleReader.withHints(BinaryPickle.scala:160)
[error]     at scala.pickling.binary.BinaryPickleReader.beginEntry(BinaryPickle.scala:164)
[error]     at wc.SelfDescribingTest$SporeUnpickler$macro$10$2$.unpickle(SelfDescribingTest.scala:41)
[error]     at scala.pickling.Unpickler$class.unpickleEntry(Pickler.scala:79)
[error]     at wc.SelfDescribingTest$SporeUnpickler$macro$10$2$.unpickleEntry(SelfDescribingTest.scala:41)
[error]     at scala.pickling.functions$.unpickle(functions.scala:11)
[error]     at scala.pickling.UnpickleOps.unpickle(Ops.scala:23)
[error]     at wc.SelfDescribingTest$.test(SelfDescribingTest.scala:25)
[error]     at wc.SelfDescribingTest$.delayedEndpoint$wc$SelfDescribingTest$1(SelfDescribingTest.scala:41)
[error]     at wc.SelfDescribingTest$delayedInit$body.apply(SelfDescribingTest.scala:15)
[error]     at scala.Function0$class.apply$mcV$sp(Function0.scala:34)
[error]     at scala.runtime.AbstractFunction0.apply$mcV$sp(AbstractFunction0.scala:12)
[error]     at scala.App$$anonfun$main$1.apply(App.scala:76)
[error]     at scala.App$$anonfun$main$1.apply(App.scala:76)
[error]     at scala.collection.immutable.List.foreach(List.scala:381)
[error]     at scala.collection.generic.TraversableForwarder$class.foreach(TraversableForwarder.scala:35)
[error]     at scala.App$class.main(App.scala:76)
[error]     at wc.SelfDescribingTest$.main(SelfDescribingTest.scala:15)
[error]     at wc.SelfDescribingTest.main(SelfDescribingTest.scala)
java.lang.RuntimeException: Nonzero exit code returned from runner: 1
    at scala.sys.package$.error(package.scala:27)

As you see, we get two it works and then an ArrayIndexOutOfBoundsException.

This bug is very problematic because most of the times the library developers (me included) don't care to expect a subclass of Spore when they want to serialize something. They just expect an object of type Spore and this obviously typechecks since SporeWithEnv <: Spore.

My proposed solution is to force the macros to check if the pickled/unpickled Spore corresponds to a subclass of it. My hunch is that the main problem is in the Unpickler, not the Pickler.

BTW, I'm using last version 0.2.1.

Make release for version 0.2.3 now that everything is working correctly, although not immediately. Maybe wait one or two days until I solve the reported issues in the issue tracker?

Hi,

I saw that you've published a new version - 0.2.4 - so I figured I'd just verify that #43 is fixed as expected.

But it isn't! My toy project still compiles successfully, in spite of the bogus capture.

Was I wrong to expect that this ought to be fixed in this release?

I was finally reading the spores paper from https://infoscience.epfl.ch/record/191239/files/spores_1.pdf (which IIUC is still the up-to-date documentation since it's not subsumed by the SIP).

I like the approach, but one issue made me wonder about typing of spores composition and Excluded, so I thought I'd ask @heathermiller and @phaller.

I might be missing something, also because I've been skimming the paper a bit, but I've looked for explanations in a couple of obvious places and found none.
Also, this issue appears in a part of the type system that is formalized but not proven correct (I've also checked with https://infoscience.epfl.ch/record/191240/files/spores-formally.pdf, and there is no difference).

Take the example in Sec. 2.4. What happens if we have MyActor <: Actor,

type T1 = Spore[Int, String] {
  type Captured = (Int, Util)
  type Excluded = No[Actor]
}
type T2 = Spore[String, Int] {
  type Captured = (MyActor, Int)
  type Excluded = No[Util]
}

then have s1: S1, s2: S2 and compute the type S3 of s3 = s1 compose s2

Applying the rules in the paper, as I understand them, I get the incorrect result:

type T3 = Spore[String, String] {
  type Captured = (Int, Util, MyActor, Int)
  type Excluded = No[Actor] with No[Util]
}

That's because IIUC the paper treats Excluded and Captured as lists, and simply filter elements of excluded to prevent them from appearing literally in Captured:

Excluded(res.type) = {T ∈ Excluded(s1.type) ∪ Excluded(s2.type) | T \not\in
Captured(s1.type), Captured(s2.type)}

(same with the formal type rules T-EComp).

I've wondered whether the use of \not\in actually checks not the set of types specified, but the set of all subtypes of Captured, which would fix the problem. However, that interpretation would apply also to Excluded, and it's not clear how to recover a minimal list of types. Moreover, rule T-ESpore explicitly uses <: to prevent similar problems, while I find no mention of the issue I describe.

It also seems a fix for this case might be simple:

Excluded(res.type) = {T ∈ Excluded(s1.type) ∪ Excluded(s2.type) | \not\exist U \in
Captured(s1.type), Captured(s2.type) (U <: T)}

but I haven't even began to look at the proof to check if it works.

Inside the body of a spore, every call needs to be static (I don't really understand what that means). In practice, the following piece of code raise the error: the invocation of 'l.map[List[Int], List[List[Int]]](((elem: Int) => immutable.this.List.apply[Int](elem)))' is not static.

    val spNotWorking = spore[List[Int], List[Int]] {
      l => l.map(elem => List(elem)).flatten
    }

If instead you create a function doing the same thing, and pass that function inside the spore's header, the error disappear and it works just fine.

    def f(list: List[Int]): List[Int] = {
      list.map(elem => List(elem)).flatten
    }

    val spWorking = spore[List[Int], List[Int]] {
      val lf = f _
      l => lf(l)
    }

Is that a bug or a feature ?

Edit: this happened with Spores 0.2.0.

Spores whose input and output types are not annotated cannot be created. This proposes an enhancement to the spores library so that a snippet like this one works:

val workPlease = spore { (i: Int) => i }

Right now, to allow the previous snippet to work one has to do something like:

val alreadyWorks = spore[Int,Int] { (i: Int) => i }

This would lure a lot of people to use this library and give it a more widespread use since it's quite cumbersome for library developers and clients of the libraries.