html encoded urls about w3lib HOT 6 CLOSED

Shouldn't this be handled at HTML parsing time? E.g. if the raw URL was

"http://example.com?foo=bar&bar=foo"

then it'd be http://example.com?foo=bar&bar=foo after unescaping, we shouldn't transform it further to http://example.com?foo=bar&bar=foo.

from w3lib.

I second @kmike . I don't think it's w3lib's role to decode HTML entity references.
lxml for example correctly decodes the &:

>>> import lxml.html
>>> t = '''<a href="http://example.com?foo=bar&amp;bar=foo">http://example.com?foo=bar&amp;bar=foo</a><br />'''
>>> doc = lxml.html.fromstring(t)
>>> doc.xpath('//@href')
['http://example.com?foo=bar&bar=foo']

@jvanasco , where did you have an issue with this?

from w3lib.

from w3lib.

I agree with @kmike and @redapple here. Even though it is a pain in the ass when this happens, it is not the responsibility of canonicalize_url to handle this.

It should not be fixed, because, albeit weird, http://example.com/?foo=bar&bar=foo is a valid URL different from http://example.com?foo=bar&bar=foo, and so printing a warning may be wrong in itself.

@jvanasco Would you be OK with us closing this issue?

from w3lib.

from w3lib.

Thank you!

from w3lib.