Bug: Unicode Normalization not appropriate for ASCII-8BIT about faraday-follow-redirects HOT 7 CLOSED

Follow-Up, see knu/ruby-domain_name#23 (comment) and knu/ruby-domain_name#27

from faraday-follow-redirects.

This fix has been merged and released upstream, see https://github.com/knu/ruby-domain_name/blob/master/CHANGELOG.md. The domain_name gem in v0.6.20240107 now skips Unicode normalization if the input is strictly ASCII only.

from faraday-follow-redirects.

The root cause of the error seems to happen when String#unicode_normalize is called. #unicode_normalize is called by domain_name and domain_name is required by http-cookie, which is the dependency of faraday-cookie_jar.

If you remove faraday-cookie_jar from your middleware stack, this error doesn't happen. Hence, the problem seems to be related to faraday-cookie_jar. I think faraday-follow-redirects is not relevant to the error.

from faraday-follow-redirects.

OK, I misunderstood the situation. response['location'] is already encoded as ASCII-8BIT. I want to look into it further more.

from faraday-follow-redirects.

response['location'] may come from net-http, which is the default Faraday adapter. net-http parses the response headers using #readuntil internally. #readuntil returns ASCII-8BIT encoded string, so response['location'] is encoded as ASCII-8BIT. Maybe, faraday-follow-redirects should consider the case where the Location header is encoded as ASCII-8BIT.

from faraday-follow-redirects.

However, I'm not confident faraday-follow-redirects should encode the URL. faraday-follow-redirects will update env[:url] with response['location'] from net-http here. Should env[:url] always be encoded as UTF-8?

Without faraday-cookie_jar, it's fine to assign an ASCII-8BIT string to env[:url] in almost cases. As far as I commented, this might be the matter of domain_name. The same issue is reported here.

knu/ruby-domain_name#23

from faraday-follow-redirects.

I need to do some more digging to understand how the ruby-domain_name gem is supposed to work and how things exactly interact.

My current thought process is not to encode anything into UTF-8 for this gem:

• URLs by definition cannot be unicode.
• Domains might be encoded in Punycode (RFC3492) but still are always (a subset) ASCII-8bit.
• Same with the rest of the URL. It's actually not even ASCII, but a printable subset.
• Browsers (and possibly other clients) might provide the illusion of the path or arguments being in unicode, but they cannot (see e.g. RFC3986).
• The redirect location from the Location HTTP response header is also ALWAYS in a sub-set of ASCII, see RFC2616 Section 4.2.

Maybe some of the other involved gems do accept UTF-8 encoded domain names for convince, but I'm pretty sure they should always accept and handle ASCII-8BIT too.

from faraday-follow-redirects.