Comments (12)
@sjrd Does this look familiar to you?
from community-build.
First failure: http://github.com/scala-js/scala-js/commit/6dbab2c3ebb665efe0c1eee2af33202d6edc42e2
from community-build.
But, that build successfully the day before. Could it be a change in scala/scala?
from community-build.
The diff in the repeatable config:
--- { (Selection)
+++ (clipboard)
@@ -2764,7 +2764,7 @@
"cross-version" : [
"disabled"
],
- "uri" : "https://github.com/scala/scala.git#454a1dcf0c3568ea99f669529b4819782b8b2f00"
+ "uri" : "https://github.com/scala/scala.git#f635ba91d641d900ffd56ae0f4bbfb8e88bfe2d1"
},
{
"system" : "sbt",
from community-build.
from community-build.
This is reproducible with:
diff --git a/project/Build.scala b/project/Build.scala
index dffaa16..edc0492 100644
--- a/project/Build.scala
+++ b/project/Build.scala
@@ -47,7 +47,8 @@ object Build extends sbt.Build {
ScalaJSCrossVersion.binaryScalaJSVersion(previousVersion)
val commonSettings = Seq(
- scalaVersion := "2.11.6",
+ scalaVersion := "2.11.7-SNAPSHOT",
+ scalaBinaryVersion := "2.11",
organization := "org.scala-js",
version := scalaJSVersion,
// ~/.sbt/0.13/global.sbt
resolvers ++= (
if (scalaVersion.value.endsWith("-SNAPSHOT"))
List(
"pr-scala snapshots" at "http://private-repo.typesafe.com/typesafe/scala-pr-validation-snapshots/",
Resolver.sonatypeRepo("snapshots")
)
else
Nil
)
from community-build.
Does not seem familiar. I will investigate this in a couple hours when I'm at work, in front of a computer.
from community-build.
Thanks. Would love it if you could share a few notes on how you diagnose this problem, so we can learn a little bit about ScalaJS in the process.
from community-build.
OK, so after having reproduced, this is how I approach such an issue.
First, a coarse dichotomic search to attribute the blame to one of the major components of the pipeline. Basically, the pipeline is as follows:
- scalac until mixin, included
- Scala.js compiler plugin -> produces .sjsir files
- Linker: all the .sjsir files on the classpath are brought together and linked
- Optimizer: optimizes the linked program
- Desugar/emitter: transform the IR into a JS AST and print it
So, this is the typical search:
- Verify the linked IR with `testSuite/test:fastOptJS (similar to the Validation performed by the JVM)
- If the IR verifies, then disable the optimizer.
- If that succeeds, blame the optimizer (unlikely in this case, because it fails with Rhino anyway, which doesn't use the optimizer in the first place -- but typically we do everything with Node.js)
- If it still fails, blame Desugar
- If the IR does not verify, then print some of the IR that does not validate, to validate "manually"
- If it validates by hand, blame the linker and/or IR checker
- If it doesn't validate by hand, then inspect the scalac trees after mixin (
-Xprint:mixin
)- If that looks OK, blame the Scala.js compiler plugin
- If it looks funny, blame scalac (or a macro)
- If the IR verifies, then disable the optimizer.
So, in this case, the IR does not validate. The full errors are in this gist:
https://gist.github.com/sjrd/33717376fe91398e2e1d
but I'll concentrate on these:
[error] .../scala/util/hashing/MurmurHash3.scala(91:20:VarRef): Cannot find variable x in scope
[error] .../scala/util/hashing/MurmurHash3.scala(91:20:VarRef): Cannot find variable a in scope
[error] .../scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable x in scope
[error] .../scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable n in scope
because I know that MurmurHash3
is a pretty simple file. So I go to that file and look at lines 91 and 109. They are the lines with foreach
in these two methods.
final def unorderedHash(xs: TraversableOnce[Any], seed: Int): Int = {
var a, b, n = 0
var c = 1
xs foreach { x =>
val h = x.##
a += h
b ^= h
if (h != 0) c *= h
n += 1
}
var h = seed
h = mix(h, a)
h = mix(h, b)
h = mixLast(h, c)
finalizeHash(h, n)
}
/** Compute a hash that depends on the order of its arguments.
*/
final def orderedHash(xs: TraversableOnce[Any], seed: Int): Int = {
var n = 0
var h = seed
xs foreach { x =>
h = mix(h, x.##)
n += 1
}
finalizeHash(h, n)
}
OK now let's see the IR for orderedHash
, which is the simplest, with scalajsp:
> cli/runMain org.scalajs.cli.Scalajsp scalalib/target/scala-2.11/classes/scala/util/hashing/MurmurHash3.sjsir
[info] Running org.scalajs.cli.Scalajsp scalalib/target/scala-2.11/classes/scala/util/hashing/MurmurHash3.sjsir
def orderedHash__sc_TraversableOnce__I__I(xs: sc_TraversableOnce, seed: int): int = {
val n: sr_IntRef = mod:sr_IntRef$.create__I__sr_IntRef(0);
val h: sr_IntRef = mod:sr_IntRef$.create__I__sr_IntRef(seed);
xs.foreach__F1__V(new sjsr_AnonFunction1().init___sjs_js_Function1((lambda<this, h, x>(this$2: s_util_hashing_MurmurHash3, h: sr_IntRef, x: any, x$2: any) = {
val x: any = x$2;
this$2.$$anonfun$2__p1__sr_IntRef__sr_IntRef__O__V(n, h, x);
(void 0)
})));
this.finalizeHash__I__I__I(h.elem$1, n.elem$1)
}
There is obviously a problem with x
in the lambda. A lambda is basically a JavaScript anonymous function, where captures are explicitly stated in the <>
. Every capture is an expression from the surrounding scope, and is captured as the corresponding "formal parameter" of the lambda. The remaining formal parameters are the real formal parameters or the generated function (OK, the notation may not be clear).
Here, we refer to x
in the expressions captured from the surrounding scope, in <this, h, x>
, but x
is not declared outside the lambda. Indeed, it is actually declared inside as val x: any = x$2
.
So, at this point, I want to have a look at the scalac tree before mixin. Since I don't want to do that for all of the scala lib, I'll just copy the offending snippet in the Helloworld example, which is my sandbox. Since it involves lambdas, I make sure to use the -Ydelambdafy:method
when compiling the Helloworld (by default, this flag is used when compiling the library, but not the examples):
> set scalacOptions in helloworld ++= Seq("-Ydelambdafy:method", "-Xprint:mixin")
[info] Defining helloworld/*:scalacOptions
[info] The new value will be used by helloworld/compile:scalacOptions
[info] Reapplying settings...
[info] Set current project to Scala.js (in build file:/localhome/doeraene/projects/scalajs/)
> helloworld/fastOptJS
[info] Updating {file:/localhome/doeraene/projects/scalajs/}helloworld...
[info] Resolving org.eclipse.jetty#jetty-continuation;8.1.16.v20140903 ...
[info] Done updating.
[info] Resolving org.eclipse.jetty#jetty-continuation;8.1.16.v20140903 ...
[info] Compiling 1 Scala source to /localhome/doeraene/projects/scalajs/examples/helloworld/target/scala-2.11/classes...
[[syntax trees at end of mixin]] // HelloWorld.scala
package helloworld {
object HelloWorld extends Object with scalajs.js.JSApp {
<synthetic> def $js$exported$meth$main(): Object = scala.scalajs.js.JSApp$class.$js$exported$meth$main(HelloWorld.this);
final def orderedHash(xs: scala.collection.TraversableOnce, seed: Int): Int = {
var n: runtime.IntRef = scala.runtime.IntRef.create(0);
var h: runtime.IntRef = scala.runtime.IntRef.create(seed);
xs.foreach({
((x: Object) => HelloWorld.this.$anonfun$1(n, h, x)).$asInstanceOf[Function1]()
});
scala.util.hashing.MurmurHash3.finalizeHash(h.elem, n.elem)
};
def main(): Unit = {
HelloWorld.this.orderedHash(immutable.this.Nil, 0);
()
};
final <artifact> private[this] def $anonfun$1(n$1: runtime.IntRef, h$1: runtime.IntRef, x: Object): Unit = {
h$1.elem = scala.util.hashing.MurmurHash3.mix(h$1.elem, runtime.this.ScalaRunTime.hash(x));
n$1.elem = n$1.elem.+(1)
};
def <init>(): helloworld.HelloWorld.type = {
HelloWorld.super.<init>();
scala.scalajs.js.JSApp$class./*JSApp$class*/$init$(HelloWorld.this);
()
}
}
}
[info] Fast optimizing /localhome/doeraene/projects/scalajs/examples/helloworld/target/scala-2.11/helloworld-fastopt.js
[error] file:/localhome/doeraene/projects/scalajs/library/src/main/scala/scala/scalajs/runtime/package.scala(67:23:VarRef): Cannot find variable x in scope
[error] file:/localhome/doeraene/projects/scalajs/library/src/main/scala/scala/scalajs/runtime/package.scala(67:23:VarRef): Cannot find variable result in scope
[error] file:/localhome/doeraene/projects/scalajs/examples/helloworld/HelloWorld.scala(15:20:VarRef): Cannot find variable x in scope
[error] file:/localhome/doeraene/projects/scalajs/examples/helloworld/HelloWorld.scala(15:20:VarRef): Cannot find variable n in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable x in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/util/hashing/MurmurHash3.scala(109:20:VarRef): Cannot find variable n in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/collection/TraversableOnce.scala(355:12:VarRef): Cannot find variable x in scope
[error] https://raw.githubusercontent.com/scala/scala/v2.11.7-SNAPSHOT/src/library/scala/collection/TraversableOnce.scala(355:12:VarRef): Cannot find variable first in scope
[trace] Stack trace suppressed: run last helloworld/compile:fastOptJS for the full output.
[error] (helloworld/compile:fastOptJS) There were 8 IR checking errors.
[error] Total time: 22 s, completed May 20, 2015 10:19:35 AM
OK so the output after mixin
seems reasonable (and I can reproduce the linking errors in the sandbox), so I would blame the Scala.js compiler plugin.
Now, I happen to know that the compiler plugin does tricky things with Function
nodes emitted by -Ydelambdafy:method
. And it is likely that little variations in the scheme are going to break it (I mentioned that on the PR that initially introduced -Ydelambdafy:method
ages ago, I think). So let's look at that code:
https://github.com/scala-js/scala-js/blob/a8a926bf80c24d7e42ae45973149d1967c61f88f/compiler/src/main/scala/org/scalajs/core/compiler/GenJSCode.scala#L3747
Oh look! There's even a big comment explaining what we expect to receive from scalac. Let's see if the tree from mixin matches our expectations:
Anonymous functions survive until the backend only under
-Ydelambdafy:method
and when they do, their body is always of the formEnclosingClass.this.someMethod(arg1, ..., argN, capture1, ..., captureM)
where argI are the formal arguments of the lambda, and captureI are local variables or the enclosing def.
and the tree:
xs.foreach({
((x: Object) => HelloWorld.this.$anonfun$1(n, h, x)).$asInstanceOf[Function1]()
});
Oh ow, big mismatch. The captures have now come to the front, and the formal parameters went to the end. Ouch.
So yes, it came from a change in scalac. But no, it's probably not scalac's fault. The Scala.js compiler plugin should probably be adapted to the new ordering. How to do that right now (while still staying source compatible with older versions of Scala), I don't know yet, but I'll figure it out.
What I really don't understand is what change in scala/scala could possibly have changed the order, in the diff you mentioned. Do you have any idea?
from community-build.
/cc @nicolasstucki The above write-up is also a good reading for you.
from community-build.
Epic writeup, @sjrd. /cc @adriaanm @lrytz @SethTisue
This was the compiler change that change the parameter order: scala/scala@1e7a990
from community-build.
+1 thanks for writing it down so nicely -- helped improve my understanding of the scala.js pipeline!
from community-build.
Related Issues (20)
- add JDK 17 build HOT 8
- radically simplify the build HOT 6
- Add Cats Effect 3 to the community build HOT 6
- dbuild launcher JARs seem to have gone missing HOT 3
- policy for advancing project SHAs? HOT 1
- Move the JDK 17 build off early-access, onto 17 final HOT 3
- install latest scala-cli on behemoths HOT 2
- Delete and archive the Gitter room HOT 1
- Run the build on JDK 20 HOT 15
- Upgrade to sbt 1.8.0 HOT 1
- Upgrade to JDK 20 final once Temurin is available HOT 2
- fix failing sbt 0.13 builds in 2.12 build HOT 2
- Add JDK 21 to build matrix HOT 22
- add more scalafix projects HOT 12
- scalafmt failing HOT 12
- Scaladoc generation failing in Scalafix repo after recent updates HOT 2
- Add JDK 22-ea to build matrix HOT 4
- Add Akka resolver so I can unfreeze some downstream-from-Akka repos
- Upgrade behemoths to JDK 22 final HOT 1
- Add JDK 23 to matrix HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from community-build.