Add properties for files about remote-apis HOT 8 CLOSED

for example, adding mtime is, I think, likely to lead to cache fragmentation from files that are content-identical but produced on different machines via a hermetic process.

I agree that we generally shouldn't add timestamps. The cache fragmentation would only affect Directory protos, not actual file blobs, but it's still not ideal.

However, in BuildStream where we build a whole module in a single Action using a traditional build system (e.g., autotools, cmake), we have a feature to support incremental builds. This currently only works for local builds but we'd like to support it also with remote execution. The way this works is that we use the previous buildtree + the modified sources as input tree for the incremental build. Unfortunately, with traditional build systems it's crucial to properly support file mtime, otherwise too little or too much will be rebuilt.

We won't use file mtime for regular builds, however, for incremental builds the improved user experience is worth the Directory proto cache fragmentation.

I don't expect Bazel or other REAPI clients that work with fine-grained actions to be interested in file mtimes. However, we'd still like to support this for the optional incremental build feature in BuildStream and at least one REAPI server. More clients might be interested in additional node properties, though, e.g., for permissions. That's why we thought it might be sensible to add a string-based extension mechanism that would provide implementations the flexibility for such features. If one or multiple such extensions turn out to be more generally useful, we can consider standardizing them, similar to platform properties.

from remote-apis.

Note that there's a related but less-comprehensive proposal in #54.

I'm not sure that adding the capability to support arbitrary properties is a great idea--for example, adding mtime is, I think, likely to lead to cache fragmentation from files that are content-identical but produced on different machines via a hermetic process. There's clearly a general desire for better permissions handling--at least increased clarity, and possibly more flexibility.

I'm curious to hear what others think of file/node-specific properties--what would be useful vs. overly complex, whether a general mechanism is necessary, etc.

from remote-apis.

from remote-apis.

I'm torn on representing these as general properties vs a small number of known fields. modified-time feels like it should be a standardized and cross-platform field. Permissions are platform-dependent, which argues for general.

I would lean towards a general property usually and allow the server to implement the supported keys. I'm not sure there's an added value to specific field support

from remote-apis.

from remote-apis.

Apologies, I missed your response.

The current use-cases of Directory protos are action inputs and action
outputs, and I think we should outline these explicitly w.r.t to what the
additional metadata should be and how is it to be interpreted (Eric's
proposal
https://docs.google.com/document/d/10ari9WtTTSv9bqB_UU-oe2gBtaAA7HyQgkpP-RFP80c/edit?ts=5d433a83#
adds
other use-cases, and when it goes through, we will need to address these
similarly as well).

That's fine, I'll read the proposal and update here.

• The properties need to be sorted, for canonical Directory
  representation.

Agreed.

• Do we want to make is_executable part of metadata? Logically, it does
  belong there, so will be a bit cleaner, and then we can remove the field in
  v3. On the other hand, it will make the messages referencing executables
  more verbose, particularly if we want to optionally add the properties to
  outputs.

I have no strong feelings on that. On one-hand if file properties are supported then is_executable becomes redundant and could be replaced with a reference to those although I wouldn't want to break functionality.

from remote-apis.

@ola-rozenfeld I've made some updates to #91. Does this address some of your concerns? In terms of what the output should be I intend for properties to be returned as part of the message. In terms of RE, I'd agree that the server should set the properties of the files where possible (and there will need to be an update to #91) for this.

from remote-apis.

I've added a few comments to #91.

• Output metadata: either we say the server will not add any output metadata, or we need to add a message to Command, e.g. OutputMetadata with repeated string property_name, that the server needs to return for every output.

This is still missing if I haven't overlooked anything.

from remote-apis.