Connection between Peers Fails under higher Loads about zerotierone HOT 7 OPEN

Interesting. Some questions:

(1) How many streams is Veeam creating and using during these congestion incidents?
(2) When the backup job seems to be failing is the underlying ZT link still working? This is important because the Veeam job may fail because of its own internal timing logic when a ZT link is congested and that's different than if a ZT link it is unresponsive.

Try stopping the job when it appears congested and try to send some data between the two ZT nodes by some other method like iperf

from zerotierone.

As soon as i cease the Backup Job, it takes about 10 seconds and the link starts working again, so it is only temporarily failing. I'll report later on some test results.

from zerotierone.

1. The default setting of Veeam is 5 Streams, which is enough to produce the issue.
2. When the Backupjob is failing, the underlying ZT link is also down. We did pings between the two Hosts during a running Backup Job and you can see how the Ping time first gets worse, than the Pings start failing completely and only after this the Backup Job fails because of a connection loss.
3. If we run the same job directly it works like a charm and we did also test it with a Tailscale network, which works without problems too.

I'll try to reproduce a similar behaviour without Veeam later. I'm not sure if Veeam is doing anything special, but it's definitely taking down a ZT Connection completely 😁

from zerotierone.

Ok thanks for the info, but when the backup job seems to be failing the only way we can tell if the ZT link is actually down is if we cease the job and then try to use the link with something else. Otherwise a ping is just another packet competing for a scarce resource that it probably won't get. Can you try that?

I'll try to do some saturation testing on my end.

from zerotierone.

We've seen something similar. If you make 2 small, single CPU VMs and then iperf between them, they fall over.

Capping the bandwidth on the zt interface to some lower number seemed to avoid it. You need to experiment a little to find the best limit. I'm not familiar with how to do it on Windows. It might be a little more tricky if it's incoming traffic. Maybe Veeam has configuration options for that.

You also might be able to run multiple instances of ZeroTier on your server. 1 per CPU core?
We don't have instructions for that that I'm aware of. Someone should do that.

Sorry to interrupt. Just adding a little more context.

from zerotierone.

I'm not able to reproduce the issue between the two same hosts with iperf and multiple connections only. Also in a test environment with 2 vCPUs and 50 iperf connections I'm getting 1.38 Gbit/s without big congestion problems.

Somehow Veeam is triggering something that ZT doesn't like, it seems to be related to simultaneous streams but not only to this. Wireguard or Tailscale are not affected in a similar way. Could it be related to ZTs MTU of 2800, Jumbo Frames, Layer 2?

Sounds stupid, but to me it seems as if the ZT connection is saying to Veeam: Bring it on, I have a lot more bandwidth, I'll handle it, but then drops everything because the actual link is extremly congested and queuing is no longer possible. As if Veeam can't detect how congested the link already is.

Any idea how to test or debug it further?

from zerotierone.

Is there anything new about this issue? We're still facing it

from zerotierone.