Nil Pointer Deference with gzip/deflate Content about veillance HOT 5 OPEN

http://stackoverflow.com/questions/13130341/reading-gzipped-http-response-in-go

from veillance.

This is a bug, if the content type is deflate it shouldn't be trying gzip it should use zlib. Note thought that there are some servers that are broken an use raw deflate format.

From: http://www.gzip.org/zlib/zlib_faq.html#faq38

"gzip" is the gzip format, and "deflate" is the zlib format. They should probably have called the second one "zlib" instead to avoid confusion with the raw deflate compressed data format. While the HTTP 1.1 RFC 2616 correctly points to the zlib specification in RFC 1950 for the "deflate" transfer encoding, there have been reports of servers and browsers that incorrectly produce or expect raw deflate data per the deflate specficiation in RFC 1951, most notably Microsoft.

from veillance.

golang/go#10377

from veillance.

Neither the Go or Python stdlib url/http handlers deal with the deflate debacle for you. The Python Requests lib does though and it really is a case of try one, fail, try the other (i.e. peek at the bytes).

• http://stackoverflow.com/questions/3947120/does-python-urllib2-automatically-uncompress-gzip-data-fetched-from-webpage
• http://stackoverflow.com/questions/1864191/how-to-deal-with-deflated-response-by-urllib2
• http://stackoverflow.com/questions/1089662/python-inflate-and-deflate-implementations

The python Requests lib implementation is here

And this is a very useful test page http://carsten.codimi.de/gzip.yaws/

from veillance.

What the HTTP 1.1 standard has to say about this:

• https://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html#sec14.3
• https://tools.ietf.org/html/rfc2616#section-3.5
• https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Content-Encoding

from veillance.