Are outputs relative to exec root or working directory? about remote-apis HOT 14 CLOSED

All paths are relative to the exec root. You're right, the proto documentation should be fixed! Thank you!

from remote-apis.

All paths are relative to the exec root. You're right, the proto documentation should be fixed! Thank you!

This contradicts the previous clarification in #26

BuildStream, BuildGrid and BuildBarn (buildbarn/bb-remote-execution#18) follow the current proto documentation. I.e., they treat output paths as being relative to working_directory and this change would break existing clients and servers.

from remote-apis.

Bleh! Sorry, Juerg! You're right. I have short memory :-)
Okay, then this should be fixed on the RBE side, and then the SDK...

from remote-apis.

So the conclusion is that the documentation is correct, our RBE implementation is wrong, and the SDK is (consequently) also wrong?

What would the migration path be for clients that rely on the previous behaviour (relative to exec root)?

from remote-apis.

I think that RBE will have to go through a phase where it will infer the client assumption based on where the actual outputs of the action are on the worker, before all its clients are fixed.
@bergsieker @EricBurnett

from remote-apis.

That sounds touchy. Could we be explicit with some sort of RBE-specific migration bit in the protos, clearly marked as RBE only?

from remote-apis.

I don't think it's that touchy. To my knowledge, RBE only has two clients using the feature. Importantly, Bazel is not one of them! So even though the simple migration plan is not formally well defined (what if an action creates two outputs with the same path, one relative to exec root and one relative to work-dir?) I suspect it will work well in practice. And we can quickly fix the two clients, so that the phase doesn't need to last more than a couple of weeks.
And if we do see cases of ambiguity, we can flag those and reevaluate...
@bergsieker and @EricBurnett -- thoughts?

from remote-apis.

I am working on such a client right now, and maybe the Chrome guys are as well. (Unless that's one of the ones you were aware of?)

How can you know whether an action creates outputs relative to the working dir and exec root? After the action completes, do you look for both of them, then adjust the output accordingly?

from remote-apis.

Yes, that is what I'm suggesting. As a temporary fix in RBE, look for both paths, take the one that exists, and log/surface any inconsistencies (such as both exist, or one output exists under working dir only while another under exec root only).

from remote-apis.

@ola-rozenfeld your proposal SGTM. Or depending on how launched these clients are (is the use of our mis-implementation in prod, or just in progress?), we could possibly just make a backwards-incompatible (breaking) change to RBE and accept a short period where these clients don't work, in the interest of getting to the right behaviour sooner.

In either case, I suggest we take this discussion to a RBE bug - sounds like the API is clearly well defined and we've simply mis-implemented it, which is a RBE issue and not a remote-apis issue :).

from remote-apis.

@EricBurnett , assuming discussion moved to an RBE bug, and no changes are needed here, can this be closed?

from remote-apis.

Yes, we've moved this to an internal bug. The published spec is clear, and we'll update our server to match.

from remote-apis.

Also see bazelbuild/bazel#13188; Bazel wasn't changed to match, although it's only assuming the old semantics if an experimental flag is set.

from remote-apis.

The code in Bazel that relied on this was added on Feb 13, 2020.

from remote-apis.