Comments (11)
@youknowone do you have a test case at hand to reproduce the issue?
from httptools.
A request:
b'''GET /ping/ HTTP/1.1\r\nHost: github.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nUser-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4\r\n\r\n'''
I called feed_data each byte by byte and the headers are broken.
Related issue in sanic which uses httptools: sanic-org/sanic#755
They are implmenting their own header fragment buffer.
from httptools.
I'll take a look as soon as I finish working on the next uvloop release.
from httptools.
@yohanboniface feel free to work on this if you have time
from httptools.
@youknowone to make sure I understand the issue: it arises when a request is chunked in a middle of a header field?
So for example, to continue on your data, we'd have this as first chunk:
GET /ping/ HTTP/1.1\r\nHost: github.com\r\nConnection: keep-alive\r\nCache-Control: max-age=0\r\nUpgrade-Insecure-Requests: 1\r\nTransfer-Encoding: chunked\r\nUser-Agent: Mozilla/5.0
And this as second chunk:
(Macintosh; Intel Mac OS X 10_12_5) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.115 Safari/537.36\r\nAccept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,image/apng,*/*;q=0.8\r\nAccept-Encoding: gzip, deflate\r\nAccept-Language: ko-KR,ko;q=0.8,en-US;q=0.6,en;q=0.4\r\n\r\n
And then you'd have an incomplete value for User-Agent
and an invalid header name with the rest of the value?
Is that correct or am I missing something?
edit bah, no, indeed a chunked request is only about the body.
So it's not about the request itself being split in the middle of a header field but that the code implementing httptools chunking it manually before calling feed_data
?
from httptools.
@yohanboniface Yes, your description is correct. User-Agent will be Mozzle/5.0 in that case.
As I know, the chunked body is a spec about logical chunk, not about TCP packet fragment.
A question here: does httptools expect to feed the whole HTTP body (at least "a chunk") at same time? Then it can be a user fault - but still weird.
Because httptools is the parser, I think basically the users can't determine which part of http request is going to httptools or not. For the point of view of user, "end of chunk" of http body and any fragmented packet in http header is not recognizable before putting it into the parser.
I think httptools is the correct place to merge the fragmented tcp packets to avoid double-parsing http request both in httptools and the users.
from httptools.
@youknowone made a quick unittest to reproduce what I've understood of the issue, but… it passes ;)
See #26
Can you please check the unittest and tell me what I'm missing to properly reproduce the issue? thanks :)
from httptools.
Thanks, your test is really helpful.
It seems I need to look into both httptools and sanic.
Give me some time. I am new to both part.
from httptools.
I changed your test a little and it now starts to be broken: #27
from httptools.
I also added a patch to #27. Thanks @yohanboniface, I would never looked into it without your test.
from httptools.
Cool!
I'll let @1st1 do the final review :)
from httptools.
Related Issues (20)
- API change from 0.0.13->0.1.1 HOT 2
- Deprecation warnings in Python 3.10 HOT 3
- Wheel support for linux aarch64 HOT 1
- provide wheel for python3.9 HOT 9
- Note: httptools 0.0.13 from PyPI is incompatible with Python 3.9
- PROXY protocol v1 / v2 support
- [Security] Potential Secret Leak HOT 1
- Silent fail of long url parsing HOT 2
- Git release/tag for 0.2.0 pypi version HOT 2
- Wheels for httptools 0.2.0 on Python 3.10 HOT 1
- CVE-2021-22959 & CVE-2021-22960
- Doesn't build against Python 3.11 HOT 3
- Fail to get wheel for python 3.9.5 when pip install
- llhttp critical CVE's HOT 4
- How to do an early stop? HOT 2
- . HOT 1
- with curl and https HOT 1
- Error installing httptools as a uvicorn dependency HOT 2
- Create wheel for Python 3.12 HOT 1
- Support Cython 3.0+ HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from httptools.