Comments (18)
OK, browsing the internet a little bit more and reading stuff, I came to the conclusion, that multiple running haproxy
instances might not be a problem, since the older
ones should finish themselves once the last open connection gets closed.
But then there might be another issue for this observed behavior:
As soon as there is a change with respect to services running and a new haproxy config
file is generated and reloaded by haproxy
I get HTTP connection errors for a very short time period (~1sec). And this is across services/apps, e.g., when scaling/deploying a service_a
I also get connection errors for a running service_b
.
Any ideas?
from panteras.
ad. to HAproxy multiple processes - this is a normal behavior.
ad. 2'nd question - we are not having this problem at all, till now, HAproxy reload never caused any issue yet.
Do you have problem with every running service, when HAproxy reloads ?
Do you have problem with services when scaling down only ? or also when scaling up ?
from panteras.
I just setup a very simple node.js
based service (just echo'ing the HTTP header
s that were passed to it) for testing the fail-over behavior of HAProxy
.
This service gets stressed by another node.js
based script, which runs 10 requests in parallel in a loop hitting the echo
service from above.
As soon as the HAProxy config
file gets changed (due to the deployment of a new service, scaling an existing service up or down, ...) I get some failing requests reported, e.g.:
...
stress-test 4: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 32
stress-test 5: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 32
stress-test 3: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 32
stress-test 6: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 31
stress-test 8: request error: Error: read ECONNRESET, duration: 27
stress-test 7: request error: Error: read ECONNRESET, duration: 27
stress-test 9: request error: Error: read ECONNRESET, duration: 27
stress-test 1: request error: Error: read ECONNRESET, duration: 30
stress-test 0: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 29
stress-test 2: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 32
stress-test 3: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 30
...
or sometimes connection timeout like this:
...
stress-test 0: remote address: ::ffff:10.50.80.218, service host: 1088cc6d638d, duration: 27
stress-test 0: remote address: ::ffff:10.50.80.218, service host: 4382a069d20c, duration: 30
stress-test 0: remote address: ::ffff:10.50.80.218, service host: 6e97e38e6d33, duration: 34
stress-test 2: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 3: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 9: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 5: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 8: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 6: request error: Error: connect ETIMEDOUT, duration: 21000
stress-test 4: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 1: request error: Error: connect ETIMEDOUT, duration: 21000
stress-test 7: request error: Error: connect ETIMEDOUT, duration: 20099
stress-test 0: remote address: ::ffff:10.50.80.218, service host: 980e24577611, duration: 28
stress-test 2: remote address: ::ffff:10.50.80.218, service host: 0e0a26b73ff8, duration: 28
stress-test 3: remote address: ::ffff:10.50.80.218, service host: 1088cc6d638d, duration: 30
...
from panteras.
I think this is kind of HAproxy design problem that for few millisecond port is unavailable - specially exposed if you test it without keep alive - you should rather simulate browser behavior and check
apache ab
(from apache2-utils package) that supports keep alive
and then:
ab -n 10000 -c 10 -k http://echo.service.my_company.com
this will give you more reliable results.
We are considering to migrate to different Load Balancer. One of our internal developer wrote a golang consul based router, we are waiting for OS it (It need some time to Open Source any internal tool)
from panteras.
if this is still bothering you, change in generate_yml.sh
for that
HAPROXY_RELOAD_COMMAND="iptables -I INPUT -p tcp --dport 80 --syn -j DROP; sleep 1; /usr/sbin/haproxy -p /tmp/haproxy.pid -f /etc/haproxy/haproxy.cfg -sf $(pidof /usr/sbin/hapro
xy); iptables -D INPUT -p tcp --dport 80 --syn -j DROP || true"
I just saw that this option cannot be overwritten from external ENV variable , man man man
from panteras.
more info :
http://inside.unbounce.com/product-dev/haproxy-reloads/
http://engineeringblog.yelp.com/2015/04/true-zero-downtime-haproxy-reloads.html
from panteras.
Thx for the pointers; will have a closer look tomorrow in the office.
I hoped/thought that HAProxy could handle those cases out-of-the-box... :-(
Do you have experience with nginx? Can it handle that reload case more gracefully?
from panteras.
Yes we have experience with nginx, apache and varnish.
Nginx could be a better choice, varnish also (but varnish has implemented it since version 4)
The solution with iptables should also do the trick with HAproxy, especially the 1st method (from the first link) is better, but much more complicated. The 2'nd one with tc
and nl-qdisc-add
could be easy to implement.
And worth mention: there is no faster LB than HAproxy.
IMHO all those methods doesn't fully feet to consul cluster - because you always need to use consul-template "glue code" thats why some of my colleagues decided to write its own router.
from panteras.
I have just tested solution with nl-qdisc-add
that buffers all the stuff for milliseconds and it works really good,
I think I can adapt this into PanteraS - this will definitely satisfy you.
from panteras.
Yes, definitely! :-)
Thank you for all your suggestions and hard work!
from panteras.
fixed
from panteras.
thx
from panteras.
This change didn't fix the issue for me; I am observing the exact same errors as noted above... :-(
from panteras.
Yea, I forgot to include one part, I was wondering where to put it, and forgot about it.
from panteras.
oh man this is not working as suspected, depends on buffer size it can even slow down a lot.
I start to think to implement the best option with two HAproxy deamons and iptables switch.
from panteras.
I have a PoC using iptables switching between two HAproxy,
and this one is really working fine. I have well tested it,
it works much better than queuing/buffering.
I will push changes tomorrow - I have to make some cosmetic changes.
from panteras.
Kindly please verify if this is working fine
from panteras.
This seems to work now and also with my keepalived
PR (#68)...
Thanks for that.
from panteras.
Related Issues (20)
- maxconn configuration HOT 2
- haproxy 100% cpu usage HOT 22
- Struggling to use HTTPS / Enable HAPROXY_SSL HOT 5
- Checkpointing not working HOT 22
- HAProxy SSL Passthrough HOT 3
- Accessing consul services inside container HOT 2
- How to use ssl proxy properly and is fabio ready for replacing haproxy ? HOT 3
- Upgrade version and impossible to build image `/bin/sh: 1: locale-gen: not found` HOT 1
- [Question] Recommendation and details on how to migrate web service into Fabio. HOT 2
- [Feature] Set env JAVA_HOME within Panteras HOT 4
- Help upgrade marathon HOT 2
- [feedback] Chronos/Netdata HOT 6
- How to use verdaccio behind the HA proxy loadbalancer
- Where is consul template file for HAproxy? HOT 2
- evaluate traefik to replace fabio HOT 1
- Marathon crash, /migration-in-progress, where is zkCli/zookeepercli in panteras ? HOT 7
- g HOT 1
- Things to consider when joing a slave into a mesosphere?
- How can I get marathon to restart all services after reboot ? HOT 2
- How to disable panteras http/80 page on Panteras v3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from panteras.