request does not check Accept-Encoding, Accept-Encoding is not respected about ureq HOT 6 CLOSED

Thanks a lot for the info. This did turn out to be a server-specific issue due to User-Agent 😅

from ureq.

If you send accept-encoding: identity and the server responds with gzip, is it not the server that failed to obey the request header?

ureq does not double check the server fulfils the accept-encoding, and whether it should can be debated, but from your description i'm a bit unclear of whether there's an outright bug here.

It sounds like you describe this:

ureq -> send accept-encoding: identity -> server
ureq <-    sends gzipped content back  <- server

from ureq.

To be clear, there are two separate pairs of headers:

Accept-Encoding: https://tools.ietf.org/html/rfc7231#section-5.3.4
Content-Encoding: https://tools.ietf.org/html/rfc7231#section-3.1.2.2

TE: https://tools.ietf.org/html/rfc7230#section-4.3
Transfer-Encoding: https://tools.ietf.org/html/rfc7230#section-3.3.1

Both Content-Encoding and Transfer-Encoding have a "gzip" option.

Also worth noting:

https://tools.ietf.org/html/rfc7230#page-13

For example, an
origin server might disregard the contents of a received
Accept-Encoding header field if inspection of the User-Agent header
field indicates a specific implementation version that is known to
fail on receipt of certain content codings.

from ureq.

If you send accept-encoding: identity and the server responds with gzip, is it not the server that failed to obey the request header?

I guess the issue is that I am unable to reproduce this behavior with the server in question when I'm testing with curl or wget. I see that it's always respected. When running with ureq, providing the same headers, the response to the GET request is always gzipped and chunked, when I want neither, and when I have specified 'identity' in Accept-Encoding and Transfer-Encoding. I believe additional headers are causing this but haven't yet discovered where they are being added. There doesn't seem to be an easy way to print out the traffic in ureq at the moment, perhaps modifying send_prelude to see exactly what is being sent is an option.

I ended up switching to another solution so I'm not sure I will have time to follow up. Please feel free to close if you think this is a server issue. Thanks for the quick responses!

from ureq.

I guess the issue is that I am unable to reproduce this behavior with the server in question when I'm testing with curl or wget. I see that it's always respected. When running with ureq, providing the same headers, the response to the GET request is always gzipped and chunked

If you want to share the name of the server, and the code you used, I'd be curious to check this out. The main thing that occurs to me is that the server might be detecting certain User-Agent strings and changing behavior based on that; you could try setting the User-Agent to match curl.

I have specified 'identity' in Accept-Encoding and Transfer-Encoding

What happens if you specify neither?

BTW, you probably want to specify identity in TE rather than Transfer-Encoding. And the server is allowed to returned chunked encoding regardless; chunked encoding support is mandatory in HTTP.

from ureq.

from ureq.