Comments (9)
@tstibbs Excepted, at my discretion.
from cors-anywhere.
I have now imposed some rate limits.
With some exceptions (e.g. localhost because it's used for development and blocking localhost altogether doesn't make sense), you can only make 600 requests per origin per hour. I came to this number by capturing Origin headers for a whole hour, 5 times (in the weekend and weekday, at different times on a given day) (resulting in 1741K requests from 2.9k unique origins, excluding previously blacklisted origins). The data shows that:
- 95% of all sites are not affected by the limit.
- 88% of the requests are dropped (this means that 5% of the sites were using most of the resources of CORS Anywhere)
600 requests per hour is a royal limit. If you need more, host your own instance of CORS Anywhere (see https://github.com/Rob--W/cors-anywhere#demo-server for instructions). If this limit turns out to be too permissive, the limit may be tightened even more.
Here are some statistics based on the data.
Max requests per hour | Origins not affected by the limit (% of all) | Requests from sites affected by the limit (% of all) |
---|---|---|
4000 | 99% | 66% |
3600 | 98% | 68% |
3000 | 98% | 71% |
2000 | 98% | 75% |
1000 | 96% | 83% |
800 | 95% | 85% |
700 | 95% | 86% |
600 (current limit) | 95% | 88% |
500 | 94% | 89% |
400 | 92% | 92% |
300 | 91% | 93% |
200 | 89% | 95% |
100 | 86% | 96% |
90 | 85% | 96.6% |
80 | 84% | 96.9% |
70 | 83% | 97.2% |
60 | 82% | 97.5% |
50 | 80% | 97.8% |
40 | 79% | 98.1% |
30 | 76% | 98.4% |
20 | 71% | 98.8% |
10 | 62% | 99.3% |
9 | 60% | 99.4% |
8 | 59% | 99.4% |
7 | 56% | 99.5% |
6 | 55% | 99.5% |
5 | 50% | 99.6% |
4 | 48% | 99.7% |
3 | 41% | 99.8% |
2 | 37% | 99.8% |
1 | 20% | 99.9% |
0 | 0% | 100.0% |
I captured another 1 hour of requests after setting the limit to 600 per hour, and got the following actual results (excluding blacklisted sites):
- 82 out of the 1348 origins were rate-limited (6%).
- 94k out of 321k requests were accepted (29%) (so 227k were blocked, 71%).
- 49k of all accepted requests were from rate-limited origins (15% of all, 52% of accepted requests)
- 1.5k of all accepted requests were from origins that are allowed to make an unlimited number of requests (0.5% of all requests, 1.6% of accepted requests)
- 43.5k of all accepted requests came from origins that were not affected by the rate limit (13.6%)
These actual findings are within the expectations from the original investigation, so I'll stick to these limits for now.
And here are some more healthy statistics from the past hour (similar to the analysis from https://github.com/Rob--W/cors-anywhere/issues/42#issuecomment-198744800):
Error codes:
70 H12
4 H15
5528 H27
Status codes (including requests that were immediately blacklisted and not visible in the above stats):
101253 200
174 206
1300 301
87 304
3925 400
662592 403 <-- Most of them are blacklisted
3607 404
226238 429 <-- Rate-limited
5528 499
24 500
74 503
(total # of requests in 1 hour: 1004802)
from cors-anywhere.
Are file:///
origins affected by the limits or are they excepted?
from cors-anywhere.
While developing on localhost, I accidentally wrote an infinitely-looping script that sent out multiple requests through your heroku demo... I noticed immediately in the console log, so I closed the browser tab that the site was running in and changed the lines that was causing this to happen.
Thought this took care of it, but a few minutes later my app stopped working altogether. May I know if reaching the rate limit results in a permanent ban or a temporary one that expires after a set amount of time? Honest mistake.
from cors-anywhere.
@erikksuzuki The ban is automatic and automatically expires. I do occasionally add extra restrictions when I spot persistent abuse patterns, but in your case, you probably have access again by now.
Note: When developing on localhost, I suggest that you don't use CORS Anywhere as a proxy, but self-host CORS Anywhere (see README) or use other methods (e.g. configuring CORS correctly or using browser extensions that automatically add CORS headers). That will result in faster response times, and also prevent you from being blocked by mistake.
from cors-anywhere.
@Rob--W I made the same mistake as erikksuzuki did. How much time must go by before the ban expires? Thank you.
from cors-anywhere.
The current rate limiting rules are documented in #164.
Note that you can easily self-host CORS Anywhere to not have the restrictions. To learn how to set it up, see https://github.com/Rob--W/cors-anywhere#demo-server
from cors-anywhere.
I totally understand your reasons for imposing the limits, but could this be conveyed in a bit friendlier manner?
Below message
GET http://127.0.0.1:3000/mytest.html 404 (Not Found)
is the only thing I see in my Chrome console when my JS hangs out on a fetch against Heroku CORS proxy. It took me 2 hours of googling and reading possible reasons to understand that I seem to be making one too many requests per hour.
Could an error message perhaps be returned as a custom HTTP error code or a message in a body since this public instance of CORS proxy is anyway not intended for production use?
from cors-anywhere.
GET http://127.0.0.1:3000/mytest.html 404 (Not Found)
It took me 2 hours of googling and reading possible reasons to understand that I seem to be making one too many requests per hour.
Not true. If you're getting hit by the rate limit, the error is 429 Too Many Requests
, which is indeed a good description of what happens.
404 either means:
- server replied with 404
- you had sent a request that doesn't resemble a valid host name <-- This is the case in your example, if
127.0.0.1:3000
is your private CORS Anywhere instance. If it is not, then your comment has nothing to do with CORS Anywhere. - the connection to the server failed without any specific reason
from cors-anywhere.
Related Issues (20)
- Does not work in Deno (TypeError on every request) HOT 1
- 400 Bad Request HOT 1
- Update HTTP-Proxy to 1.18.1 to resolve https://github.com/advisories/GHSA-6x33-pw7p-hmpq HOT 1
- Redirects ruining GET parameters using axios HOT 2
- how to block specific final link not origin
- cors-anywhere doesn't work with youtube m3u8 files HOT 1
- Error creating app on Heroku website HOT 2
- API domain name resolution problem HOT 1
- performance problem, cors-anywhere doesn't use outbound HTTP keep-alive HOT 1
- s anywhere
- Question about whitelisting HOT 1
- getting an Uncaught (in promise) SyntaxError HOT 1
- Authorization error HOT 1
- Whitelisting subdomain doesn't work as expected
- see the CI test #271 #273
- cors anywhere not working on vercel HOT 1
- add option to add auth header for sites like reddit HOT 1
- Siteye kurulum
- live radio use case: Missing required request header. Must specify one of: origin,x-requested-with HOT 4
- Not found because of proxy error: AggregateError
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from cors-anywhere.