UDP Round Robin Only going to single server about seesaw HOT 7 CLOSED

could you paste what ipvsadm reports?

sudo ipvsadm -Ln

from seesaw.

Hi Yuan,

I ran a test using just netcat as shown below to run five requests

echo {"version": "2.1","host":"Request - 4 -test.test.local","short_message":"Helo from seesaw","full_message":"Request 4 here\n\n Load Balancer Testing","level":1,"_user_id":9001,"_user_id":"ped","_some_env_var":"bar"} | ncat -u -w 1 logsvr.<domain fqdn>l 12201

ipvsadm output

admin@lb1:/var/log/seesaw$ sudo ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
UDP  172.16.10.165:12201 rr
  -> 172.16.10.61:12201            Route   1      0          3
  -> 172.16.10.62:12201            Route   1      0          2
dadmin@lb1:/var/log/seesaw$

I can confirm that test message are routing in a round robin style.

The problem is application related. Our logging agent sends via UDP to the VServer VIP and it seems to lock to a single SIEM back end.

When i send single JSON request as above it works perfectly, the requests are unaltered as I am running tcpdump on backend servers to check for that.

Any ideas why?

from seesaw.

It's because connections are tracked and your logging agent is always using the same client side udp port. you can see the connections on the seesaw instance:

sudo ipvsadm -Lnc

from seesaw.

Any ideas why?

By default IPVS will track flows and persist each flow (source IP, destination IP, protocol, source port, destination port) to a single backend - often spreading a flow across multiple backends will result in unwanted behaviour/problems (in the case of UDP think video streaming, VoIP calls, etc). As noted, it is likely that your application is using the same source port for all traffic, resulting in what appears to be a single flow (hence being sent to a single backend).

In cases where it is known that the UDP traffic is per packet and that you can safely send each UDP datagram to a different backend (think DNS requests), you can enable "one packet scheduling" (or OPS). This can be enabled on a Seesaw vserver_entry by adding one_packet: true to the configuration.

from seesaw.

For our use case, we are using in front of our SIEM solution to provide log balancing for SIEM boxes behind.

As each agent send UDP or log source to the load balancer, the behaviour you described suits my use case. Also with firewallslogs, this is a s a requirement for us as it would mean that we could have large volume of firewall logs sending to a single backend.

If I dont use the one packet option, is there a load balancing protocol to use the least saturated backend server with UDP?

Cheers

from seesaw.

I have this working now.

I will post a blog post about how I got it working for others here shortly.

from seesaw.

I have enabled one_packet:true in my vserver_entry as per #102 my log sources for my zeek box still will not round robin correctly.

The output of ipvsadm -L -stats shows the following

sudo ipvsadm -L --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port               Conns   InPkts  OutPkts  InBytes OutBytes
  -> RemoteAddress:Port
UDP  logsvr.test.local:12205           0      116        0    93117        0
  -> clust-01.test.local:122        0       50        0    40502        0
  -> clust-02.test.local:122        0       66        0    52615        0
UDP  172.16.4.166:12210                  0      388        0   209490        0
  -> clust-01.test.local:122        0      388        0   209490        0
  -> clust-02.test.local:122        0        0        0        0        0

The zeek logs are on port 12210 which are being routed to clust-01.test.local whilst other logs are on 12205 which appear to balance correctly.

Relevant vserver_entry

vserver: <
  name: "zeek.test.local"
  entry_address: <
    fqdn: "logsvr.test.local."
    ipv4: "172.16.10.166/24"
    status: PRODUCTION
   >
  rp: "[email protected]"
  vserver_entry: <
    protocol: UDP
    one_packet: true
    port: 12210
    scheduler: RR
    mode: NAT
    healthcheck: <
      type: TCP
      port: 32999
      tls_verify: false
      interval: 10
      timeout: 5
      retries: 3
    >
  >
  backend: <
    host: <
      fqdn: "clust-01.test.local."
      ipv4: "172.16.10.21/24"
      status: PRODUCTION
    >
    weight: 1
  >
  backend: <
    host: <
      fqdn: "clust-02.test.local."
      ipv4: "172.16.10.22/24"
      status: PRODUCTION
    >
    weight: 1
  >
>

I thought that once one_packet: true is enabled that IPVS would not track client source UDP ports.

Any ideas why?

from seesaw.