Rosetta testing method about naga HOT 13 CLOSED

All for giving it a try, and then possibly adjusting as we go along :)

For ref: had some similar challenges working on tests for the Blender glTF exporter. We had issues with buffer views and accesser indices not always being the same, so ended up de-referencing those before comparing test results. Of course we also ended up having blender 2.79 and 2.80 exporting actual vertex data in different order, which kind of broke the idea of comparing those.

from naga.

I did some initial experimentation, you can see here: https://github.com/pjoe/naga/tree/rosetta-experiment/test-data/simple

Still fighting with issues of different expressions ordering

from naga.

Oh, I don't think there is any dependency issue, looks to me like it simply does:

var w: f32 = 1.0;

becomes

Constant(1), // w initialiser 1.0
LocalVariable(1), // w

And then when it reaches the compose, the only thing that isn't already in the expression list is the 0.0 constant.

Starting to make a bit of sense to me :)

from naga.

Managed to make glsl-new use same order (at least for the simple example) in #147.

Check out this: https://github.com/pjoe/naga/blob/rosetta-experiment/test-data/simple/simple-ron.diff (an empty diff). glsl-new now generates identical IR to wgsl 🚀

from naga.

Yup, I'll do a PR next to add the simpel test I've been experimenting with. We can then figure out best structure and setup for running against multiple front and back ends

from naga.

I like this idea ♥

I'm not sure if there are some issues in practice, but guess we just have to try it out:

• Is there always a "canonical" IR, or can things like ordering or different handle IDs differ, while output is still valid? Maybe some of this can be fixed by post-processing IR, e.g. abstracting out handle IDs.

• Will round-trip e.g. WGSL > IR > WGSL always produce the same? Or do we have cases where the IR doesn't capture all details of the input, or again ordering might differ?

from naga.

Good concerns, thank you for evaluating this!
For ordering in canonical IR: we don't have many unoredered sets/maps, and ones we have are likely to be populated with the same elements in the same order. So it could be that we don't need to do anything specific. We'll see ;)

Will round-trip e.g. WGSL > IR > WGSL always produce the same?

That's an interesting concern. It's not clear to me if it couldn't be possible. The main point of this testing method is focusing on IR, so once we fixed IR, we can generated WGSL, and then just make sure that parsing that WGSL produces the same IR.

from naga.

I think I managed to get glsl-new in line, except for expression ordering.

I'm however fighting to understand how wgsl orders expressions :S

For glsl I have:

#version 450 core

layout(location = 0) in vec2 a_pos;
layout(location = 0) out vec4 o_pos;

void main() {
    float w = 1.0;
    o_pos = vec4(a_pos, 0.0, w);
}

which translates to following expression order:

[
Constant(1), // w initialiser 1.0
GlobalVariable(2), // o_pos
GlobalVariable(1), // a_pos
Constant(2), // 0.0
LocalVariable(1), // w
Compose(ty: 2, components: [3, 4, 5,]),
],

However for wgsl I get:

# vertex
[[location 0]] var<in> a_pos : vec2<f32>;
[[location 0]] var<out> o_pos : vec4<f32>;

fn main() -> void {
  var w: f32 = 1.0;
  o_pos = vec4<f32>(a_pos, 0.0, w);
}
entry_point vertex as "main" = main;

and expressions:

[
GlobalVariable(2), // o_pos
GlobalVariable(1), // a_pos
Constant(1), // w initialiser 1.0
LocalVariable(1), // w
Constant(2), // 0.0
Compose(ty: 2, components: [1, 5, 4,]),
],

So I don't understand:

• why is the w initialiser expression inserted later than it's declaration statement ?
• why does the reference to local var w come before the 0.0 constant ?

from naga.

Some insight on the WGSL logic. In the prelude of a function, it registers expressions for all global variables, so that it can access them later. This is unlike GLSL, where you are adding them lazily to the expression list. The benefit here is that if the WGSL parser sees an identifier, it can look it up always in the expression map, or otherwise it's an error. This simplifies the parser, and makes it faster to lookup identifiers.

You might have a concern here: if a function has expressions for all the globals, doesn't it breaks the static analysis of what globals are used? Answer is NO, because the analysis (and we have it in proc) iterates statements, and expressions used from them, recursively. It doesn't care about expressions that are not linked from any statements at the top level.

from naga.

@kvark: okay that explains the first part. Should be possible to do similar for GLSL (always add global vars first when parsing a function definition).

But what about the second one - why the LocalVar(w) before the constant(0.0) ... is that the same thing? Will it also add an expression for a LocalVar as soon as it's defined?

from naga.

I think the second part is a bug in WGSL frontend. Expressions should not depend on ones that go after them.

from naga.

So we can have it all rolling now? yeesss!

from naga.

Thinking about this some more, trying to converge all the frontends into a single IR could be possible, but it's work that no end user needs, it's only a change to ease the testing. Instead of making this a requirement, we should make it a preference, thus allowing SPIR-V frontend to diverge, for example.

Here is an updated test plan. We'll not have separate "tests" for multiple backends/frontends, but rather

• each frontend would have a map of inputs -> IRs. Sharing of IRs in encouraged/preferred (e.g. between GLSL and WGSL).
• each backend would have a map of IRs -> outputs. It doesn't have to use all the IRs, just a representative subset is fine.

We can move the test files (all except RON for IR) into the backend/frontent folders.

from naga.