scala-js failure since May 20, 2015 about community-build HOT 12 CLOSED

@sjrd Does this look familiar to you?

from community-build.

First failure: http://github.com/scala-js/scala-js/commit/6dbab2c3ebb665efe0c1eee2af33202d6edc42e2

from community-build.

But, that build successfully the day before. Could it be a change in scala/scala?

from community-build.

The diff in the repeatable config:

--- { (Selection)
+++ (clipboard)
@@ -2764,7 +2764,7 @@
                                 "cross-version" : [
                                     "disabled"
                                 ],
-                                "uri" : "https://github.com/scala/scala.git#454a1dcf0c3568ea99f669529b4819782b8b2f00"
+                                "uri" : "https://github.com/scala/scala.git#f635ba91d641d900ffd56ae0f4bbfb8e88bfe2d1"
                             },
                             {
                                 "system" : "sbt",

from community-build.

scala/scala@454a1dc...f635ba9

from community-build.

This is reproducible with:

diff --git a/project/Build.scala b/project/Build.scala
index dffaa16..edc0492 100644
--- a/project/Build.scala
+++ b/project/Build.scala
@@ -47,7 +47,8 @@ object Build extends sbt.Build {
     ScalaJSCrossVersion.binaryScalaJSVersion(previousVersion)

   val commonSettings = Seq(
-      scalaVersion := "2.11.6",
+      scalaVersion := "2.11.7-SNAPSHOT",
+      scalaBinaryVersion := "2.11",
       organization := "org.scala-js",
       version := scalaJSVersion,

// ~/.sbt/0.13/global.sbt
resolvers ++= (
  if (scalaVersion.value.endsWith("-SNAPSHOT"))
    List(
      "pr-scala snapshots" at "http://private-repo.typesafe.com/typesafe/scala-pr-validation-snapshots/",
      Resolver.sonatypeRepo("snapshots")
    )
  else
    Nil
)

from community-build.

Does not seem familiar. I will investigate this in a couple hours when I'm at work, in front of a computer.

from community-build.

Thanks. Would love it if you could share a few notes on how you diagnose this problem, so we can learn a little bit about ScalaJS in the process.

from community-build.

OK, so after having reproduced, this is how I approach such an issue.
First, a coarse dichotomic search to attribute the blame to one of the major components of the pipeline. Basically, the pipeline is as follows:

1. scalac until mixin, included
2. Scala.js compiler plugin -> produces .sjsir files
3. Linker: all the .sjsir files on the classpath are brought together and linked
4. Optimizer: optimizes the linked program
5. Desugar/emitter: transform the IR into a JS AST and print it

So, this is the typical search:

• Verify the linked IR with `testSuite/test:fastOptJS (similar to the Validation performed by the JVM)
  • If the IR verifies, then disable the optimizer.
    • If that succeeds, blame the optimizer (unlikely in this case, because it fails with Rhino anyway, which doesn't use the optimizer in the first place -- but typically we do everything with Node.js)
    • If it still fails, blame Desugar
  • If the IR does not verify, then print some of the IR that does not validate, to validate "manually"
    • If it validates by hand, blame the linker and/or IR checker
    • If it doesn't validate by hand, then inspect the scalac trees after mixin (-Xprint:mixin)
      • If that looks OK, blame the Scala.js compiler plugin
      • If it looks funny, blame scalac (or a macro)

So, in this case, the IR does not validate. The full errors are in this gist:
https://gist.github.com/sjrd/33717376fe91398e2e1d
but I'll concentrate on these:

[error] .../scala/util/hashing/MurmurHash3.scala(91:20:VarRef): Cannot find variable x in scope
[error] .../scala/util/hashing/MurmurHash3.scala(91:20:VarRef): Cannot find variable a in scope
[error] .../scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable x in scope
[error] .../scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable n in scope

because I know that MurmurHash3 is a pretty simple file. So I go to that file and look at lines 91 and 109. They are the lines with foreach in these two methods.

  final def unorderedHash(xs: TraversableOnce[Any], seed: Int): Int = {
    var a, b, n = 0
    var c = 1
    xs foreach { x =>
      val h = x.##
      a += h
      b ^= h
      if (h != 0) c *= h
      n += 1
    }
    var h = seed
    h = mix(h, a)
    h = mix(h, b)
    h = mixLast(h, c)
    finalizeHash(h, n)
  }
  /** Compute a hash that depends on the order of its arguments.
   */
  final def orderedHash(xs: TraversableOnce[Any], seed: Int): Int = {
    var n = 0
    var h = seed
    xs foreach { x =>
      h = mix(h, x.##)
      n += 1
    }
    finalizeHash(h, n)
  }

OK now let's see the IR for orderedHash, which is the simplest, with scalajsp:

> cli/runMain org.scalajs.cli.Scalajsp scalalib/target/scala-2.11/classes/scala/util/hashing/MurmurHash3.sjsir
[info] Running org.scalajs.cli.Scalajsp scalalib/target/scala-2.11/classes/scala/util/hashing/MurmurHash3.sjsir
  def orderedHash__sc_TraversableOnce__I__I(xs: sc_TraversableOnce, seed: int): int = {
    val n: sr_IntRef = mod:sr_IntRef$.create__I__sr_IntRef(0);
    val h: sr_IntRef = mod:sr_IntRef$.create__I__sr_IntRef(seed);
    xs.foreach__F1__V(new sjsr_AnonFunction1().init___sjs_js_Function1((lambda<this, h, x>(this$2: s_util_hashing_MurmurHash3, h: sr_IntRef, x: any, x$2: any) = {
      val x: any = x$2;
      this$2.$$anonfun$2__p1__sr_IntRef__sr_IntRef__O__V(n, h, x);
      (void 0)
    })));
    this.finalizeHash__I__I__I(h.elem$1, n.elem$1)
  }

There is obviously a problem with x in the lambda. A lambda is basically a JavaScript anonymous function, where captures are explicitly stated in the <>. Every capture is an expression from the surrounding scope, and is captured as the corresponding "formal parameter" of the lambda. The remaining formal parameters are the real formal parameters or the generated function (OK, the notation may not be clear).

Here, we refer to x in the expressions captured from the surrounding scope, in <this, h, x>, but x is not declared outside the lambda. Indeed, it is actually declared inside as val x: any = x$2.

So, at this point, I want to have a look at the scalac tree before mixin. Since I don't want to do that for all of the scala lib, I'll just copy the offending snippet in the Helloworld example, which is my sandbox. Since it involves lambdas, I make sure to use the -Ydelambdafy:method when compiling the Helloworld (by default, this flag is used when compiling the library, but not the examples):

> set scalacOptions in helloworld ++= Seq("-Ydelambdafy:method", "-Xprint:mixin")
[info] Defining helloworld/*:scalacOptions
[info] The new value will be used by helloworld/compile:scalacOptions
[info] Reapplying settings...
[info] Set current project to Scala.js (in build file:/localhome/doeraene/projects/scalajs/)
> helloworld/fastOptJS
[info] Updating {file:/localhome/doeraene/projects/scalajs/}helloworld...
[info] Resolving org.eclipse.jetty#jetty-continuation;8.1.16.v20140903 ...
[info] Done updating.
[info] Resolving org.eclipse.jetty#jetty-continuation;8.1.16.v20140903 ...
[info] Compiling 1 Scala source to /localhome/doeraene/projects/scalajs/examples/helloworld/target/scala-2.11/classes...
[[syntax trees at end of                     mixin]] // HelloWorld.scala
package helloworld {
  object HelloWorld extends Object with scalajs.js.JSApp {
    <synthetic> def $js$exported$meth$main(): Object = scala.scalajs.js.JSApp$class.$js$exported$meth$main(HelloWorld.this);
    final def orderedHash(xs: scala.collection.TraversableOnce, seed: Int): Int = {
      var n: runtime.IntRef = scala.runtime.IntRef.create(0);
      var h: runtime.IntRef = scala.runtime.IntRef.create(seed);
      xs.foreach({
        ((x: Object) => HelloWorld.this.$anonfun$1(n, h, x)).$asInstanceOf[Function1]()
      });
      scala.util.hashing.MurmurHash3.finalizeHash(h.elem, n.elem)
    };
    def main(): Unit = {
      HelloWorld.this.orderedHash(immutable.this.Nil, 0);
      ()
    };
    final <artifact> private[this] def $anonfun$1(n$1: runtime.IntRef, h$1: runtime.IntRef, x: Object): Unit = {
      h$1.elem = scala.util.hashing.MurmurHash3.mix(h$1.elem, runtime.this.ScalaRunTime.hash(x));
      n$1.elem = n$1.elem.+(1)
    };
    def <init>(): helloworld.HelloWorld.type = {
      HelloWorld.super.<init>();
      scala.scalajs.js.JSApp$class./*JSApp$class*/$init$(HelloWorld.this);
      ()
    }
  }
}

[info] Fast optimizing /localhome/doeraene/projects/scalajs/examples/helloworld/target/scala-2.11/helloworld-fastopt.js
[error] file:/localhome/doeraene/projects/scalajs/library/src/main/scala/scala/scalajs/runtime/package.scala(67:23:VarRef): Cannot find variable x in scope
[error] file:/localhome/doeraene/projects/scalajs/library/src/main/scala/scala/scalajs/runtime/package.scala(67:23:VarRef): Cannot find variable result in scope
[error] file:/localhome/doeraene/projects/scalajs/examples/helloworld/HelloWorld.scala(15:20:VarRef): Cannot find variable x in scope
[error] file:/localhome/doeraene/projects/scalajs/examples/helloworld/HelloWorld.scala(15:20:VarRef): Cannot find variable n in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable x in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable n in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/collection/TraversableOnce.scala(355:12:VarRef): Cannot find variable x in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/collection/TraversableOnce.scala(355:12:VarRef): Cannot find variable first in scope
[trace] Stack trace suppressed: run last helloworld/compile:fastOptJS for the full output.
[error] (helloworld/compile:fastOptJS) There were 8 IR checking errors.
[error] Total time: 22 s, completed May 20, 2015 10:19:35 AM

OK so the output after mixin seems reasonable (and I can reproduce the linking errors in the sandbox), so I would blame the Scala.js compiler plugin.

Now, I happen to know that the compiler plugin does tricky things with Function nodes emitted by -Ydelambdafy:method. And it is likely that little variations in the scheme are going to break it (I mentioned that on the PR that initially introduced -Ydelambdafy:method ages ago, I think). So let's look at that code:
https://github.com/scala-js/scala-js/blob/a8a926bf80c24d7e42ae45973149d1967c61f88f/compiler/src/main/scala/org/scalajs/core/compiler/GenJSCode.scala#L3747
Oh look! There's even a big comment explaining what we expect to receive from scalac. Let's see if the tree from mixin matches our expectations:

Anonymous functions survive until the backend only under -Ydelambdafy:method and when they do, their body is always of the form EnclosingClass.this.someMethod(arg1, ..., argN, capture1, ..., captureM) where argI are the formal arguments of the lambda, and captureI are local variables or the enclosing def.

and the tree:

      xs.foreach({
        ((x: Object) => HelloWorld.this.$anonfun$1(n, h, x)).$asInstanceOf[Function1]()
      });

Oh ow, big mismatch. The captures have now come to the front, and the formal parameters went to the end. Ouch.

So yes, it came from a change in scalac. But no, it's probably not scalac's fault. The Scala.js compiler plugin should probably be adapted to the new ordering. How to do that right now (while still staying source compatible with older versions of Scala), I don't know yet, but I'll figure it out.

What I really don't understand is what change in scala/scala could possibly have changed the order, in the diff you mentioned. Do you have any idea?

from community-build.

/cc @nicolasstucki The above write-up is also a good reading for you.

from community-build.

Epic writeup, @sjrd. /cc @adriaanm @lrytz @SethTisue

This was the compiler change that change the parameter order: scala/scala@1e7a990

from community-build.

+1 thanks for writing it down so nicely -- helped improve my understanding of the scala.js pipeline!

from community-build.