Comments (8)
There is a stalled PR to address that: #25
from w3lib.
Yes @kmike, that PR addressing exactly this issue. Maybe if it is impossible to make general solution for safe symbols then it is worth to give optional control to the user? With some additional parameter in request.meta for example.
from w3lib.
@odinplus I wonder how this site works with Firefox, as according to @redapple's test Firefox doesn't encode |
as well.
from w3lib.
@kmike with Firefox it is answering with same code 400 if there is a pipe symbol in url.
from w3lib.
It seems there is still no consensus between browsers how to handle different characters in URL path (e.g. https://bugzilla.mozilla.org/show_bug.cgi?id=1064700). This means that a website which works in one browser may break in another, and we can't create escaping method which works everywhere. #25 is merged, but we've removed |
handling from it, so #25 does not fix this particular issue. |
handling was removed for these reasons:
- it makes changes smaller, more focused and less controversial;
- Firefox handles
|
the same way as w3lib, so it is not that|
handling is incorrect per se; - Chrome handles
|
differently in path and in query, whilesafe_url_string
doesn't make this distinction currently, using the same set of chars - likely it should though.
I'm not opposed to change the way |
is handled; we can do it in a separate PR.
But even if we do it, we still won't cover all cases, because behavior differs in browsers. @nyov proposed to have an option to specify which browser should we emulate (#25 (comment)). I think it may require work to maintain, because browsers change, so these rules are not set in stone. They already changed between experiments @dangra and @redapple were making.
I wonder if a more future-proof (though less user-friendly) way to tackle this is to fix scrapy/scrapy#833.
from w3lib.
I think Firefox’s approach is the right one in light of https://url.spec.whatwg.org/, which should be considered the latest URL standard.
However, until adoption grows, I wonder if we should, as you @kmike suggest, update safe_url_string
to be “safer”.
from w3lib.
we can't create escaping method which works everywhere
Well, if we focus specifically on the logic of whether or not to escape a given code point, I think escaping it if any major browser escapes it would be a valid, safe approach. In fact, we may want to decide which characters to escape not so much based on what characters web browser escape, but what characters servers out there need escaped. Over-escaping should not be a problem, so aiming to support as many servers as possible by escaping any characters that some server may need escaped may be the safest approach here.
from w3lib.
I think this makes sense.
from w3lib.
Related Issues (20)
- remove_tags not working on html comments HOT 4
- Redirection may not work depending on order of 'content' and 'http-equiv' in meta tag HOT 5
- Fix CI issue on PyPy 3 and the Rust compiler HOT 1
- test_add_or_replace_parameter fails on Python 3.6.13, 3.7.10, 3.8.8, 3.9.2 due to CVE-2021-23336 fix HOT 1
- w3lib.url.safe_url_string incorrectly encode IDNA domain with port HOT 1
- should the canonize_url function convert an apostrophe to %27 HOT 2
- [request] Update pypi release HOT 1
- basic_auth_header uses the wrong flavor of base64
- safe_url_string URL-encodes already-encoded username and password, breaking idempodency HOT 1
- Issue in safe_url_encoding HOT 8
- BOM should take precedence over Content-Type header when detecting the encoding
- safe_url_string handling IPv6 URLs HOT 5
- Function `convert_entity` does not catch `OverflowError`
- Reimplement safe_url_string based on the URL living standard HOT 4
- Location of libw3.a HOT 1
- test_safe_url_string_url regressed on 3.11.4 HOT 1
- canonicalize_url incorrectly handles port when using hostname that requires IDNA encoding HOT 1
- Space at end of query string is trimmed HOT 1
- Missing PyPI description
- canonicalize_url inappropriately "lowercases" credentials (username/password) in url HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from w3lib.