tus / tus-v2 Goto Github PK

While reading the Upload Transfer Procedure, I found the following paragraph ambiguous.

Section 4-4

The request MUST include the Upload-Token header field (Section 7.1) which uniquely identifies an upload. The client MUST NOT reuse the token for a different upload.

While it correctly states that the Upload-Token MUST be included, it does not say what the server's response should be in that case. I could not find anywhere else in the document instructions for handling this case. Is this by design or did I miss anything ?

1. If the Upload-Token is always required (which the paragraph suggests), I would probably expect the server to reject the request with status 400.
2. If the Upload-Token is not always required, perhaps to allow the same [http verb, endpoint] combo to be used differently than for a file upload, it might be useful to explicitly state it.

Cheers.

Standardizing on the phrasing "resources allocated for" vs "record of" as they mean the same thing in the draft

Registering header fields has changed a bit recently, see https://www.iana.org/assignments/http-fields/http-fields.xhtml

We probably don't need to do anything right now but should revisit this once we're further along the process

The document defines:

Upload-Token = sf-binary / sf-string / sf-token

but it doesn't really explain why there are three formats. Notionally this could be three different wire encoding of the same value, which seems redundant. There are also possibilities for edge cases to come in, for instance does the same encoding have to be used across different requests?

Furthermore, if the token is generated as suggested and is a byte sequence but sent as a sf-string, you'll probably need to base64 it. That is not documented here and should be if that's the intention.

But, for lack of justification of the options, I'll make an alternative suggestion - just use the single sf-binary and be done with it.

Add note about not supporting parallel chunked uploads.

It's brought up because resumable download supports parallel chunked downloads but we have no intention to support parallel chunked uploads.

Internet-Draft naming conventions enocurage documents to include the name of the working group they are related to. Although the document is not adopted, our best estimation now is that the HTTP WG is the right venue. So let's add httpbis into the various place it should be before publication.

We have multiple mentions of file upload in the description of the Upload Transfer Procedures. I think file upload is fine for examples, but we should rephrase procedures so that they apply more than just to file uploads.

This issue is intended to discuss if and in what form we want to have explicit support or feature detection (we called it service discovery in earlier discussions, but this name is probably ill-suited).

The idea is to allow resumable uploads to be transparently implemented by HTTP clients. This means that application developers just keep using the same API of their HTTP library as they have done in the past with traditional, non-resumable uploads. Once the HTTP library gets updated (e.g. because mobile OS or browsers start implementing resumable uploads), the HTTP library can transparently decide to use resumable uploads without explicit configuration by the application developer. Of course, in order to use resumable uploads, the HTTP library needs to know whether the server supports resumable uploads. If no support is detected, the HTTP library should use the traditional, non-resumable upload technique. We call this process feature detection.

Ideally, the technique used for feature detection meets following criteria (there might not one approach which fits all requirements, so we have to prioritize them):

1. Avoid additional roundtrips by the client, if possible (i.e. an additional HTTP request by the client should be avoided).
2. Be backwards compatible to HTTP/1.1 and existing network infrastructure: This means to avoid using new features in HTTP/2, or features which might require changes to existing network infrastructure (e.g. nginx or HTTP libraries)
3. Conserve the user's privacy (i.e. the feature detection should not leak information other third-parties about which URLs have been connected to)

Following approaches have already been considered:

Include a support statement in the SETTINGS frame. The SETTINGS frame is a HTTP/2 feature and is sent by the server to the client to exchange information about the current connection. The idea was to include an additional statement in this frame, so the client can detect support for resumable uploads without an additional roundtrip. The problem is that this is not compatible with HTTP/1.1. Furthermore, the SETTINGS frame is intended for information about the current connection (not bound to a request/response) and might not be persisted when transmitted through a proxy.

Include a support statement in the DNS record. The client can detect support when resolving a domain name. Of course, DNS is not semantically the correct layer. Also, DNS might not be involved if the record is chached or retrieved from a hosts files.

Send a HTTP request to ask for support. This is the easiest approach where the client sends an OPTIONS request and uses the response to determine if the server indicates support for resumable uploads. An alternative is that the client sends the request to a well-known URL to obtain this response, e.g. /.well-known/resumable-uploads. Of course, while being fully backwards-compatible, it requires an additional roundtrip.

Include a support statement in previous responses. In many cases, the file upload is not the first time that the client connects to the server. Often additional requests are sent beforehand for authentication, data retrieval etc. The responses for those requests can also include a header which indicates support for resumable uploads. There are two options:

• Use the standardized Alt-Svc response header. However, it has been indicated to us that this header might be reworked in the future and could also be semantically different from our intended usage.
• Use a new response header Resumable-Uploads: https://example.org/files/* to indicate under which endpoints support for resumable uploads is available.

Send a 104 intermediate response to indicate support. The clients normally starts a traditional upload and includes a header indicate that it supports resumable uploads (e.g. Upload-Offset: 0). If the server also supports resumable uploads, it will immediately respond with a 104 intermediate response to indicate its support, before further processing the request. This way the client is informed during the upload whether it can resume from possible connection errors or not. While an additional roundtrip is avoided, the problem with that solution is that many HTTP server libraries do not support sending custom 1XX responses and that some proxies may not be able to handle new 1XX status codes correctly.

This issue is intended to track our communicating with the IETF, their working groups etc.

There is an upcoming IETF meeting in March 2022. More information is available at https://www.ietf.org/how/meetings/upcoming/. Our plan is to reach out with a first version of the draft before this deadline.

This list collects existing implementations of the protocol. These are mostly experimental and meant for exploration.

• tusd: tus/tusd#568
• tusdotnet: https://github.com/tusdotnet/tusdotnet/tree/POC/tus2

The example server in this repository at https://github.com/tus/tus-v2/tree/master/examples does not cover the latest changes we have made to it. We should invest some time to make it compliant to our current draft.

The next step is to write a mail to the HTTP WG mailing list (https://lists.w3.org/Archives/Public/ietf-http-wg/). Some content I would like to include:

• Why are resumable uploads useful (similar to the abstract)?
• Explain that this work is related to the tus project.
• Mention that the tus project has seen adoption, so people are interested in standardized upload protocols and ready-to-use software.
• Mention that many services also have proprietary approach (maybe?)
• Mention that tus has been discussed on the mailing list before (maybe?)

This seems a bit up in the air so I held of on a PR.

Which endpoint can send the new header fields is implied but not stated explicity. We can be clearer by stating it like "request header field" for client-only headers, "response header field" for server-only headers, or "request and response header field" if it can be used by both.

Can server skip ahead using Offset Retrieving Procedure? Current thinking is there is no harm in allowing it.

https://tus.github.io/tus-v2/draft-tus-resumable-uploads-protocol.html#section-4-8 currently says:

The client MAY send the metadata of the file using headers such as Content-Type (see Section 8.3 of [HTTP] and Content-Disposition [RFC6266] when starting a new upload. It is OPTIONAL for the client to repeat the metadata when resuming an upload.

There's a PR that tweaks the text but it still leaves content-disposition there, which I find confusing because RFC6266 states that Content-Disposition is a response header (not a request one), We shouldn't be offhandledly recommending breaking HTTP semantic rules. If use of Content-Disposition on requests is a valuable feature for resumable uploads, we probably need to pursue that with more detail.

current wording:

The value of the token SHOULD be a byte sequence
with a minimum of 256-bit (16 byte) cryptographically-secure
random binary data, or a cryptographic token with equivalent
or stronger security properties.

1. There is a typo. Either it's 256-bit (32 byte) or 128-bit (16-byte).
2. I think we should discuss the value in giving a specific recommendation

I think the real goal is that the Upload-Token should contain or encapsulate an identifier that applies to a single upload and is sufficiently unique to the upload server system. One alternative to having the SHOULD with specific requirements could be to give a MUST for system-dependent uniqueness and a 128-bit cryptographically random byte sequence as one example.

If the request body is dynamically generated and the server backtracks to an offset that's no longer available, what should a client do?

If the server backtracks to 0, could the client upload a different thing? (My thinking is no)

There's one mention of "Upload Creation Procedure" in the document, what is it? Perhaps a hangover from some renaming activity?