You shall set your dpdk env for high performance.

from dpdk-ans.

How do you set DPDK env for high performance?

from dpdk-ans.

Refer to dpdk doc . for example: http://fast.dpdk.org/doc/perf/DPDK_17_08_Intel_NIC_performance_report.pdf

from dpdk-ans.

Thanks for the pointer and quick response. Really appreciate the support.

I reserved Hugepage size of 1G, disabled interrupts as mentioned still the rate seems to be same and not improving. For regular iperf command without packet size constraint, I get around 7Gbps which is 70% subscribed.

But for packet size 64B, 1.5Mbps just does not make sense

from dpdk-ans.

It seems parameter(-l) isn't the packet size, it is the buffer length.
-l,--len #[KMG] length of buffer to read or write
(default 128 KB forTCP, 8 KB for UDP)

from dpdk-ans.

I removed -l, now its just -M option. Now , its around 1.5Gbps.Still way too low. Without, -M option I get around 7Gbps. Trying to get more Packets/Per second.

from dpdk-ans.

On Linux I get with command (iperf3 -V -c 10.0.0.2 -M128 -Z -l64 -t10 -4 -Z -i 1 )

root@srv12:~/dpdk-iperf# ip netns exec ns1 iperf3 -V -c 10.0.0.2 -M128 -Z -l64 -t10 -4 -Z -i 1 │- - - - - - - - - - - - - - - - - - - - - - - - -
iperf 3.0.11 │[ ID] Interval Transfer Bandwidth Retr
Linux srv12 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux │[ 5] 0.00-30.04 sec 768 MBytes 214 Mbits/sec 50796 sender
Time: Tue, 19 Dec 2017 19:22:55 GMT │[ 5] 0.00-30.04 sec 764 MBytes 213 Mbits/sec receiver
Connecting to host 10.0.0.2, port 5201 │-----------------------------------------------------------
Cookie: srv12.1513711375.273758.0acf63c543f2 │Server listening on 5201
TCP MSS: 128 │-----------------------------------------------------------
[ 4] local 10.0.0.3 port 54492 connected to 10.0.0.2 port 5201 │Accepted connection from 10.0.0.3, port 54490
Starting Test: protocol: TCP, 1 streams, 64 byte blocks, omitting 0 seconds, 10 second test │[ 5] local 10.0.0.2 port 5201 connected to 10.0.0.3 port 54492
[ ID] Interval Transfer Bandwidth Retr Cwnd │[ ID] Interval Transfer Bandwidth
[ 4] 0.00-1.00 sec 23.4 MBytes 196 Mbits/sec 3438 11.7 KBytes │[ 5] 0.00-1.00 sec 22.6 MBytes 190 Mbits/sec
[ 4] 1.00-2.00 sec 25.8 MBytes 216 Mbits/sec 2709 45.7 KBytes │[ 5] 1.00-2.00 sec 25.0 MBytes 210 Mbits/sec
[ 4] 2.00-3.00 sec 27.4 MBytes 230 Mbits/sec 663 29.6 KBytes │[ 5] 2.00-3.00 sec 27.1 MBytes 227 Mbits/sec
[ 4] 3.00-4.00 sec 28.3 MBytes 238 Mbits/sec 869 28.4 KBytes │[ 5] 3.00-4.00 sec 23.7 MBytes 198 Mbits/sec
[ 4] 4.00-5.00 sec 23.0 MBytes 193 Mbits/sec 2270 25.3 KBytes │[ 5] 4.00-5.00 sec 25.8 MBytes 217 Mbits/sec
[ 4] 5.00-6.00 sec 25.2 MBytes 211 Mbits/sec 815 120 KBytes │[ 5] 5.00-6.00 sec 28.0 MBytes 235 Mbits/sec
[ 4] 6.00-7.00 sec 27.7 MBytes 233 Mbits/sec 2392 29.0 KBytes │[ 5] 6.00-7.00 sec 24.5 MBytes 205 Mbits/sec
[ 4] 7.00-8.00 sec 22.3 MBytes 187 Mbits/sec 1019 33.9 KBytes │[ 5] 7.00-8.00 sec 25.5 MBytes 214 Mbits/sec
[ 4] 8.00-9.00 sec 25.4 MBytes 213 Mbits/sec 1127 19.9 KBytes │[ 5] 8.00-9.00 sec 23.3 MBytes 195 Mbits/sec
[ 4] 9.00-10.00 sec 31.7 MBytes 266 Mbits/sec 1632 44.0 KBytes │[ 5] 9.00-10.00 sec 29.7 MBytes 249 Mbits/sec
│[ 5] 10.00-10.04 sec 2.32 MBytes 490 Mbits/sec
Test Complete. Summary Results: │-
[ ID] Interval Transfer Bandwidth Retr │[ ID] Interval Transfer Bandwidth Retr
[ 4] 0.00-10.00 sec 260 MBytes 218 Mbits/sec 16934 sender │[ 5] 0.00-10.04 sec 260 MBytes 217 Mbits/sec 16934 sender
[ 4] 0.00-10.00 sec 257 MBytes 216 Mbits/sec receiver │[ 5] 0.00-10.04 sec 257 MBytes 215 Mbits/sec receiver
CPU Utilization: local/sender 54.8% (3.4%u/51.4%s), remote/receiver 49.8% (3.6%u/46.2%s) │-----------------------------------------------------------

iperf Done. │--------------------------------------------------------

But, iperf dpdk I get the following

root@srv12:~/dpdk-iperf# ./iperf3 -c 10.0.0.2 -M128 -Z -l64 │[2007] 4.00-5.00 sec 355 KBytes 2.91 Mbits/sec
Connecting to host 10.0.0.2, port 5201 │iperf3: getsockopt - Success
[ 5] local 10.0.0.5 port 39508 connected to 10.0.0.2 port 5201 │[2007] 5.00-6.00 sec 162 KBytes 1.33 Mbits/sec
[ ID] Interval Transfer Bandwidth Retr Cwnd │iperf3: getsockopt - Success
[ 5] 0.00-1.00 sec 2.44 MBytes 20.3 Mbits/sec 151 8.38 KBytes │[2007] 6.00-7.00 sec 146 KBytes 1.19 Mbits/sec
[ 5] 1.00-2.50 sec 652 KBytes 3.57 Mbits/sec 186 6.00 KBytes │iperf3: getsockopt - Success
[ 5] 2.50-3.50 sec 142 KBytes 1.16 Mbits/sec 84 5.62 KBytes │[2007] 7.00-8.00 sec 303 KBytes 2.48 Mbits/sec
[ 5] 3.50-4.25 sec 143 KBytes 1.56 Mbits/sec 78 5.50 KBytes │iperf3: getsockopt - Success
[ 5] 4.25-5.50 sec 358 KBytes 2.35 Mbits/sec 179 7.00 KBytes │[2007] 8.00-9.00 sec 141 KBytes 1.15 Mbits/sec
[ 5] 5.50-6.25 sec 164 KBytes 1.78 Mbits/sec 93 6.00 KBytes │iperf3: getsockopt - Success
[ 5] 6.25-7.01 sec 146 KBytes 1.59 Mbits/sec 85 5.62 KBytes │[2007] 9.00-10.00 sec 152 KBytes 1.25 Mbits/sec
[ 5] 7.01-8.50 sec 302 KBytes 1.66 Mbits/sec 163 5.75 KBytes │iperf3: getsockopt - Success
[ 5] 8.50-9.50 sec 142 KBytes 1.16 Mbits/sec 74 5.12 KBytes │[2007] 10.00-10.26 sec 4.06 KBytes 130 Kbits/sec
An unknown state was sent by the client, ignoring it. │- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.50-10.25 sec 152 KBytes 1.65 Mbits/sec 84 5.62 KBytes │[ ID] Interval Transfer Bandwidth │[2007] 0.00-10.26 sec 0.00 Bytes 0.00 bits/sec sender
[ ID] Interval Transfer Bandwidth Retr │[2007] 0.00-10.26 sec 4.55 MBytes 3.72 Mbits/sec receiver
[ 5] 0.00-10.25 sec 4.58 MBytes 3.75 Mbits/sec 1177 sender │An unknown state was sent by the client, ignoring it.
[ 5] 0.00-10.25 sec 4.55 MBytes 3.72 Mbits/sec receiver
iperf Done. │---------------------------------------------------

I have set following kernel option to tune DPDK
default_hugepagesz=1G hugepagesz=1G hugepages=16 isolcpus=1-11,22-33 nohz_full=1-11,22-33 rcu_nocbs=1-11,22-33

Still, am no where close to performance expectation.

from dpdk-ans.

Ok, After core pinning, it improved but still short of linux

I am running, ANS on lcore 2
Use the following command ./build/ans -c 0x4 -n 4 -- -p 0x1 --config="(0,0,2)"

And I run Iperf as this
root@srv12:~/dpdk-iperf# ./iperf3 -c 10.0.0.2 -N -M128 -l64 -Z -4 -A 2 │[2293] 4.00-5.00 sec 10.6 MBytes 88.9 Mbits/sec
Connecting to host 10.0.0.2, port 5201 │iperf3: getsockopt - Success
[ 5] local 10.0.0.5 port 39716 connected to 10.0.0.2 port 5201 │[2293] 5.00-6.00 sec 10.3 MBytes 86.4 Mbits/sec
[ ID] Interval Transfer Bandwidth Retr Cwnd │iperf3: getsockopt - Success
[ 5] 0.00-1.00 sec 5.70 MBytes 47.8 Mbits/sec 0 70.1 KBytes │[2293] 6.00-7.00 sec 11.2 MBytes 94.0 Mbits/sec
[ 5] 1.00-2.01 sec 10.8 MBytes 89.7 Mbits/sec 0 70.1 KBytes │iperf3: getsockopt - Success
[ 5] 2.01-3.01 sec 10.8 MBytes 90.4 Mbits/sec 0 70.1 KBytes │[2293] 7.00-8.00 sec 10.3 MBytes 86.2 Mbits/sec
[ 5] 3.01-4.04 sec 11.0 MBytes 89.5 Mbits/sec 0 70.1 KBytes │iperf3: getsockopt - Success
[ 5] 4.04-5.00 sec 10.4 MBytes 90.7 Mbits/sec 0 70.1 KBytes │[2293] 8.00-9.01 sec 10.8 MBytes 89.3 Mbits/sec
[ 5] 5.00-6.01 sec 10.4 MBytes 86.6 Mbits/sec 0 70.1 KBytes │iperf3: getsockopt - Success
[ 5] 6.01-7.04 sec 11.2 MBytes 91.5 Mbits/sec 0 70.1 KBytes │[2293] 9.01-10.00 sec 10.7 MBytes 90.5 Mbits/sec
[ 5] 7.04-8.00 sec 10.2 MBytes 88.2 Mbits/sec 0 70.1 KBytes │iperf3: getsockopt - Success
[ 5] 8.00-9.02 sec 10.8 MBytes 89.6 Mbits/sec 0 70.1 KBytes │[2293] 10.00-10.07 sec 538 KBytes 64.8 Mbits/sec
An unknown state was sent by the client, ignoring it. │- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.02-10.04 sec 10.8 MBytes 88.9 Mbits/sec 0 70.1 KBytes │[ ID] Interval Transfer Bandwidth

• • • • • • • • • •                                                                                    │[2293]   0.00-10.07  sec  0.00 Bytes  0.00 bits/sec                  sender
                    

[ ID] Interval Transfer Bandwidth Retr │[2293] 0.00-10.07 sec 102 MBytes 85.0 Mbits/sec receiver
[ 5] 0.00-10.04 sec 102 MBytes 85.3 Mbits/sec 0 sender │An unknown state was sent by the client, ignoring it.
[ 5] 0.00-10.04 sec 102 MBytes 85.3 Mbits/sec receiver │-----------------------------------------------------------
│Server listening on 5201
iperf Done.

Note that,, this is 64 core machine.
I am running iperf client and server on same machine. Server is bound to dpdk interface where are client is bound to linux.

When I run, iperf3 client, server and ANS, performance is the max but it is still 90Mbps max.

Where are linux is giving me 260Mbps.

from dpdk-ans.

We can chat over slack to discuss about this issue.

from dpdk-ans.

Why you want the tcp mss as 128? for ethernet, it is 1460 at least.
-M, --set-mss # set TCP/SCTP maximum segment size (MTU - 40 bytes)

Please try it with -M -l parameter with the latest version. I git push new feature last night.

from dpdk-ans.

Idea is to generate maximum number of packets to max out CPU processing.

I used regular Iperf as client and DPDK_IPERF as Server, then performance was much better, around 10% higher then regular linux.

Something to do with dpdk_iperf when running as client is an issue.

Anything new with that?

from dpdk-ans.

Tried the latest one, I see a connection reset when trying to connect. I reverted for now.

Can you fix this?

from dpdk-ans.

For tcp is streaming connection, tcp will merge some small packets into a segment if remote RX window is enough.
If you want to test the maximum number of packets, you can try udp. we had tested udp TX throughput, please refer to below doc.
https://github.com/ansyun/dpdk-ans/blob/master/doc/guides/ans_performance_report.pdf

from dpdk-ans.

For the latest version . we improve socket layer performance.
After git pull the latest version, you shall recompile ans and dpdk-iperf

from dpdk-ans.

Thanks, seems to work after re-compilation.

With native iperf client and dpdk iperf , I get good numbers

iperf# iperf3 -c 10.0.0.2 -Z -M128 -l64 -Z │iperf3: getsockopt - Resource temporarily unavailable
Connecting to host 10.0.0.2, port 5201 │[2024] 7.00-8.00 sec 30.7 MBytes 258 Mbits/sec
[ 4] local 10.0.0.5 port 40160 connected to 10.0.0.2 port 5201 │iperf3: getsockopt - Resource temporarily unavailable
[ ID] Interval Transfer Bandwidth Retr Cwnd │[2024] 8.00-9.00 sec 30.7 MBytes 257 Mbits/sec
[ 4] 0.00-1.00 sec 7.67 MBytes 64.4 Mbits/sec 66 70.1 KBytes │iperf3: getsockopt - Success
[ 4] 1.00-2.00 sec 30.1 MBytes 253 Mbits/sec 0 70.1 KBytes │[2024] 9.00-9.99 sec 30.5 MBytes 258 Mbits/sec
[ 4] 2.00-3.00 sec 26.8 MBytes 225 Mbits/sec 0 70.1 KBytes │- - - - - - - - - - - - - - - - - - - - - - - - -
[ 4] 3.00-4.00 sec 28.5 MBytes 239 Mbits/sec 0 70.1 KBytes │[ ID] Interval Transfer Bandwidth
[ 4] 4.00-5.00 sec 30.9 MBytes 260 Mbits/sec 0 70.1 KBytes │[2024] 0.00-9.99 sec 0.00 Bytes 0.00 bits/sec sender
[ 4] 5.00-6.00 sec 30.9 MBytes 259 Mbits/sec 0 70.1 KBytes │[2024] 0.00-9.99 sec 278 MBytes 233 Mbits/sec receiver
[ 4] 6.00-7.00 sec 30.9 MBytes 259 Mbits/sec 0 70.1 KBytes │iperf3: error - unable to receive results:
[ 4] 7.00-8.00 sec 30.7 MBytes 258 Mbits/sec 0 70.1 KBytes │-----------------------------------------------------------
[ 4] 8.00-9.00 sec 30.7 MBytes 257 Mbits/sec 0 70.1 KBytes │Server listening on 5201
iperf3: error - unable to receive results:

But, with compiled iperf3 client, I get roughly 1/3rd of these numbers and that too if both dpdk iperf client and server are pinned to the same core which is strange.

Here. I pinned dpdk compiled iperf3 to Core 2 (-A 2) then I get decent (not as much as regular iperf as above)
root@srv12:~/dpdk-iperf# ./iperf3 -c 10.0.0.2 -N -M128 -l64 -Z -4 -A 2 │[2067] 9.00-10.00 sec 803 KBytes 6.58 Mbits/sec
Connecting to host 10.0.0.2, port 5201 │iperf3: getsockopt - Success
[ 5] local 10.0.0.5 port 40188 connected to 10.0.0.2 port 5201 │[2067] 10.00-11.00 sec 448 Bytes 3.58 Kbits/sec
[ ID] Interval Transfer Bandwidth Retr Cwnd │iperf3: getsockopt - Success
[ 5] 0.00-1.03 sec 7.21 MBytes 58.9 Mbits/sec 0 70.4 KBytes │[2067] 11.00-12.00 sec 512 Bytes 4.10 Kbits/sec
[ 5] 1.03-2.01 sec 10.3 MBytes 87.7 Mbits/sec 0 70.4 KBytes │iperf3: getsockopt - Success
[ 5] 2.01-3.00 sec 10.4 MBytes 88.5 Mbits/sec 0 70.4 KBytes │[2067] 12.00-13.00 sec 448 Bytes 3.58 Kbits/sec
[ 5] 3.00-4.01 sec 9.96 MBytes 82.9 Mbits/sec 0 70.4 KBytes │iperf3: getsockopt - Success
[ 5] 4.01-5.02 sec 10.8 MBytes 90.0 Mbits/sec 0 70.4 KBytes │[2067] 13.00-14.01 sec 640 Bytes 5.08 Kbits/sec
[ 5] 5.02-6.01 sec 10.4 MBytes 87.6 Mbits/sec 0 70.4 KBytes │iperf3: getsockopt - Success
[ 5] 6.01-7.00 sec 10.4 MBytes 88.3 Mbits/sec 0 70.4 KBytes │[2067] 14.01-15.00 sec 576 Bytes 4.63 Kbits/sec
[ 5] 7.00-8.01 sec 9.98 MBytes 83.1 Mbits/sec 0 70.4 KBytes │iperf3: getsockopt - Success
[ 5] 8.01-9.02 sec 11.1 MBytes 92.1 Mbits/sec 2 25.8 KBytes │[2067] 15.00-15.91 sec 41.4 KBytes 376 Kbits/sec
An unknown state was sent by the client, ignoring it. │- - - - - - - - - - - - - - - - - - - - - - - - -
[ 5] 9.02-15.88 sec 586 KBytes 700 Kbits/sec 797 56.9 KBytes │[ ID] Interval Transfer Bandwidth

• • • • • • • • • •                                                                                    │[2067]   0.00-15.91  sec  0.00 Bytes  0.00 bits/sec                  sender
                    

[ ID] Interval Transfer Bandwidth Retr │[2067] 0.00-15.91 sec 90.9 MBytes 47.9 Mbits/sec receiver
[ 5] 0.00-15.88 sec 91.1 MBytes 48.2 Mbits/sec 799 sender │An unknown state was sent by the client, ignoring it.
[ 5] 0.00-15.88 sec 90.9 MBytes 48.0 Mbits/sec receiver │-----------------------------------------------------------
│Server listening on 5201
iperf Done.

It is strange that I get this performance only when both dpdk_iperf3 and compiled iperf3 are attached to same core.
I start dpdk_iperf using following command
./dpdk_iperf3 -s --bind 10.0.0.2 -A 2
Note, dpdk_iperf3 server , compiled iperf3 running as client and ANS stack are all on same core and get decent performance, else its pathetic as specified in previous posts.

from dpdk-ans.

Iperf3 versions.

Native Ubuntu 3.1 version, this works with good rate.
root@srv12:~/dpdk-iperf# iperf3 --version
iperf 3.1.3
Linux srv12 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting,

Compiled version, does not have good performance.
root@srv12:~/dpdk-iperf# ./iperf3 --version
iperf 3.1
Linux srv12 4.4.0-104-generic #127-Ubuntu SMP Mon Dec 11 12:16:42 UTC 2017 x86_64
Optional features available: CPU affinity setting, IPv6 flow label, TCP congestion algorithm setting,

from dpdk-ans.

You shall use "# ./iperf3 -c 10.0.0.2" command to test. No any meaning for “-M128 -l64 ” parameters.
ans shall pin one lcore, dpdk-iperf can pin another core, or pin any lcore. iperf client can pin any lcore or another lcore.

from dpdk-ans.

Hi, I am trying to maximize traffic generation based on cores.

Currently
I run ANS on Core 1 and Core 2.
I run Iperf Server on Core 3 and 4, I start to 2 dpdk_iperf server listening on same TCP port(5200), one binds to Core 1 ANS port and other to Core 2 ANS port. dpdk_iperf application itself is running on core 2 and Core 3. Note, Iperf is using same TCP port (5200)
Because of RSS, I have to start as many application instances as how many ANS stack are running as connection can land on any core.

Now, for a single IPerf connection, it can fall on Core 0 or Core 1 ANS and I good line rate.

But, I want to run multiple Server instances.

So, I started 2 more iperf instances running on port 5201.
But, now rate drops by 60-80%.

I can understand as each ANS is busy polling packets for 2 applications running on 5200 and 5201 and hence its slower.

Is there a way to get around this?

from dpdk-ans.

How do you judge rate drop? please copy and share the test output for us analysis.

from dpdk-ans.

Setup : Run ANS on Core 1 and Core 2 and dpdk_iperf Server on Core 3 and 4, listening on same TCP port(5200)

Iperf client Cmd:
iperf3 -c 10.0.0.2 -M128 -l64 -4 -A 23 -p 5201 -N -t 50

Results: This is excellent.

[ 4] local 10.0.0.5 port 45854 connected to 10.0.0.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 60.5 MBytes 507 Mbits/sec 0 18.6 KBytes
[ 4] 1.00-2.00 sec 68.2 MBytes 572 Mbits/sec 0 18.6 KBytes
[ 4] 2.00-3.00 sec 68.1 MBytes 571 Mbits/sec 0 20.5 KBytes
[ 4] 3.00-4.00 sec 68.5 MBytes 575 Mbits/sec 0 22.8 KBytes
[ 4] 4.00-5.00 sec 74.4 MBytes 624 Mbits/sec 0 22.8 KBytes
[ 4] 5.00-6.00 sec 73.8 MBytes 619 Mbits/sec 0 23.9 KBytes
[ 4] 6.00-7.00 sec 74.1 MBytes 621 Mbits/sec 0 23.9 KBytes
[ 4] 7.00-8.00 sec 74.3 MBytes 623 Mbits/sec 0 23.9 KBytes
[ 4] 8.00-9.00 sec 68.7 MBytes 577 Mbits/sec 0 36.4 KBytes
[ 4] 9.00-10.00 sec 68.7 MBytes 577 Mbits/sec 0 36.4 KBytes
[ 4] 10.00-11.00 sec 68.7 MBytes 576 Mbits/sec 0 36.4 KBytes
[ 4] 11.00-12.00 sec 68.6 MBytes 575 Mbits/sec 0 36.4 KBytes
[

Setup : Run ANS on Core 1 and Core 2 and dpdk_iperf Server on Core 3 and 4, listening on same TCP port(5200) and dpdk_iperf Server on Core 5 and 6, listening on same TCP port(5201)

Iperf client Cmd:
iperf3 -c 10.0.0.2 -M128 -l64 -4 -A 23 -p 5201 -N -t 50

Results: This is not as good.

[ 4] local 10.0.0.5 port 45864 connected to 10.0.0.2 port 5201
[ ID] Interval Transfer Bandwidth Retr Cwnd
[ 4] 0.00-1.00 sec 19.1 MBytes 160 Mbits/sec 0 63.2 KBytes
[ 4] 1.00-2.00 sec 20.6 MBytes 173 Mbits/sec 0 63.2 KBytes
[ 4] 2.00-3.00 sec 22.6 MBytes 190 Mbits/sec 0 63.2 KBytes
[ 4] 3.00-4.00 sec 19.3 MBytes 162 Mbits/sec 0 63.2 KBytes
[ 4] 4.00-5.00 sec 25.9 MBytes 218 Mbits/sec 0 63.2 KBytes
[ 4] 5.00-6.00 sec 17.6 MBytes 148 Mbits/sec 0 63.2 KBytes
[ 4] 6.00-7.00 sec 21.4 MBytes 179 Mbits/sec 0 63.2 KBytes
[ 4] 7.00-8.00 sec 30.6 MBytes 256 Mbits/sec 0 63.2 KBytes
[ 4] 8.00-9.00 sec 20.9 MBytes 176 Mbits/sec 0 63.2 KBytes
[ 4] 9.00-10.00 sec 23.0 MBytes 193 Mbits/sec 0 63.2 KBytes
[ 4] 10.00-11.00 sec 21.2 MBytes 178 Mbits/sec 0 63.2 KBytes
[ 4] 11.00-12.00 sec 21.2 MBytes 178 Mbits/sec 0 63.2 KBytes
[ 4] 12.00-13.00 sec 21.5 MBytes 180 Mbits/sec 0 63.2 KBytes
[ 4] 13.00-14.00 sec 19.6 MBytes 164 Mbits/sec 0 63.2 KBytes
[ 4] 14.00-15.00 sec 21.1 MBytes 177 Mbits/sec 0 63.2 KBytes
[ 4] 15.00-16.00 sec 20.3 MBytes 170 Mbits/sec 0 63.2 KBytes

So for the same command, rate dropped by 70% or so.

from dpdk-ans.

Any update on this issue ?

from dpdk-ans.

No idea for that, you may capture the packets to analysis.

from dpdk-ans.

I am trying to see whether performance would be better with RSS disable where packets will be received on one CORE and forwarded to other core. What are the changes I have to make for this? You mentioned I have to use Software RSS, but am not sure about the changes.

from dpdk-ans.

You can change code in ans_main.c, core1 receive packet from NIC, and then distribute to core2 and core3,
core2 and core3 run ans code to handle the packets.

from dpdk-ans.

from dpdk-ans.

Could you describe your requirement more detail? I don't know if understand your requirement correctly. -:)

from dpdk-ans.

from dpdk-ans.

1. You can forward packet by dpdk-ring from core 1 to core x. core x only receive the packet from the dpdk-ring, not NIC.
2. core x can send packet to NIC directly.
3. You shall not start more ans instances because one NIC only beyond to one ANS.
4. By the way, you can pull the latest ans code to try, there are some improvement.
   =========
   Maybe you can configure flow on your NIC to distribute traffic to different core.
   For example, dst IP 10.0.0.2 traffic go to core 1, dst IP 10.0.0.3 traffic go to core 2.

from dpdk-ans.

from dpdk-ans.

Yes, via dpdk, refer to dpdk ether lib

from dpdk-ans.

Thanks for the pointer, I was able to add Ntuple filter to redirect and now all traffic is getting distributed as expected, but its not scaling up.

It seems I can run only 6 dpdk iperf servers, but beyond that I am not able to.

I start ANS stack on 10 cores using following command

./build/ans -c 0x7FE --base-virtaddr=0x2aaa4a90000 -n 48 -- -p 0x1 --config=
"(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6),(0,6,7),(0,7,8),(0,8,9),(0,9,10)" [Cores 1-10]

I try to start 10 instances of dpdk iperf running on different cores [Cores 12-22]
Only, 6 of them start but 7th and onwards crash in anssock_init.

Below is the Back trace

Core was generated by `./dpdk_iperf3 -s -p 5206 --bind 10.0.0.2 -A 18'.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 0x0000000000420d73 in anssock_init ()
[Current thread is 1 (Thread 0x7fc2297528c0 (LWP 426))]
(gdb) bt
#0 0x0000000000420d73 in anssock_init ()
#1 0x000000000041df47 in ans_mod_init () at src/ans_module.c:172
#2 0x000000000040c288 in main (argc=8, argv=0x7ffe5f5762d8) at src/main.c:112

anssock_init is library, so I can't debug further.

Please let me know if there is a work around.

from dpdk-ans.

You can enable ansock debug log in ans_mod_init() by anssock_enable_log().
Shall call anssock_enable_log() before anssock_init().

By the way, could you share the code "Ntuple filter to redirect“? send it [email protected]. maybe we can add a cli to configure flow redirect.

from dpdk-ans.

Sure, I will try that also will send sample code to add Ntuple redirect.

Right now, problem is scaling.

For a single flow, using 1 dpdk-ans core and dpdk-iperf code, I can get around 1M packets/sec.
But, when I enable 2 core, it only goes up 1.2M packets/sec, not a great increase.
Wondering whether any Nic settings such as Rx descriptors need to increased.

DPDK version that ans depends on today, does not have rte_eth_dev_adjust_nb_rx_tx_desc where I could have increased the maximum number of receive descriptors, am assuming it is set to 512 though Nic supports 4096 of them.

Any other pointer?

from dpdk-ans.

How to test for 2 core? share detail steps.

please use the latest dpdp-ans.

from dpdk-ans.

Ok, With latest it seems better, I am trying to get max rate when dpdk-ans is running on 10 cores and dpdk-iperf on 5.

When I run 1 flow at a time, I get around 1 M Packets/sec individually which is excellent.
When I run, 2 flows (2 dpdk-iperf severs involved), I get around 1.7 M Packets/sec
When I run, 4 flows (4 dpdk-iperf severs involved), I get around 3 M Packets/sec max but fluctates between 2.3 - 3Mpps

When I run, 5 flows (5 dpdk-iperf severs involved), I get around 3 M Packets/sec fluctuates from 2.3 -3Mpps.
So, adding more cores after 4 does not seem to increase the throughput.

In summary, If I run each dpdk_iperf sessions one at a time, I get around 1Mpps.
But, If I run 5 of them in parallel, it does not seem to go beyond 3, wondering whether Nic is bottleneck.

I get following errors on console of dpdk_iperf server.
iperf3: getsockopt - Resource temporarily unavailable

Does number of memory channels has any effect ?
sudo ./build/ans -c 0x7FE -n 40

Thanks for the help.

from dpdk-ans.

1. how to run dpdk-iperf server, what is the commands?
2. how to run ans? command? what is the IP in ans, which flow redirect to which core?
3. how to run dpdk-iperf client? what is the commands? how many dpdk-iperf client is running? how to calc the Packets/sec? what is the client IP?
4. what your test network figure?

from dpdk-ans.

Setup: 2 Interfaces, 64

cores
Intf0: DPDK driver
Intf1: Regular linux driver.
Connected by loopback

0. Apply the patch ans_tuple.diff.txt for changes related ntuple filtering based on DST port of iperf server.
   ans_ntuple.diff.txt
1. Starting ANS sudo ./build/ans -c 0x7E -n 32 -- -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6)". This will Run ANS cores 1-6
2. Run iperf.sh.txt to start 5 server instances on different ports with command
   ./iperf.sh server 5 8 ==> 5 is number of instances, 8 is start core ID. 5200 is the starting PORT ID.
3. Configure Intf1 with same subnet , ifconfig ens15f1 10.0.0.5/24
4. Run iperf.sh.txt to start 5 clients instances to connect to 5 server instances
   ./iperf.sh client 5 16 10.0.0.2 ===> 5 is number of instances, 16 is start core ID. 5200 is the starting PORT ID to connect. Note that, this client standard linux iperf.
   5.Use pps.sh.txt to measure Packets per second in Millions.

Let me know if you need more information.

from dpdk-ans.

Actually, with Zero copy option on iperf , It seems to much better. With 6 cores am able to push lot more now the only issue is , am not able to start more that 6 dpdk_iperf instances.
I tried to enable log, but nothing is displayed, it cores before any log is being printed. Can you try in your setup? Should be easy to try it.

from dpdk-ans.

Ok, it is clear for your test ENV.

1. Can you dump the packet on Intf1 to get the packets size, then can calc the throughput. 3m * 8 * packet_size. check if reach NIC line rate.

2. For more than 6 dpdk_iperf issue, maybe it related to virtual memory address mapping.
   you may try below command and stop unused application in you ENV.
   ./build/ans -c 0x7E -n 32 --base-virtaddr=0x2aaa2aa0000 -- -p 0x1 --config="(0,0,1),(0,1,2),(0,2,3),(0,3,4),(0,4,5),(0,5,6)"

from dpdk-ans.

from dpdk-ans.

Ok, i will try it in my ENV.

from dpdk-ans.

No problem in my ENV. I also run dpdk_iperf3(port 5000 - 5008) by manual.
./iperf.sh server 6 2
root@ubuntu:~/src/dpdk-iperf# ps -ef |grep iperf
root 3463 2107 10 19:01 pts/2 00:00:46 ./dpdk_iperf3 -s -p 5000 --bind 10.0.0.2
root 3476 3225 7 19:02 pts/3 00:00:29 ./dpdk_iperf3 -s -p 5001 --bind 10.0.0.2
root 3483 3307 6 19:03 pts/4 00:00:24 ./dpdk_iperf3 -s -p 5002 --bind 10.0.0.2
root 3490 3389 6 19:03 pts/5 00:00:23 ./dpdk_iperf3 -s -p 5003 --bind 10.0.0.2
root 3580 3560 6 19:03 pts/6 00:00:20 ./dpdk_iperf3 -s -p 5004 --bind 10.0.0.2
root 3713 3669 5 19:04 pts/7 00:00:16 ./dpdk_iperf3 -s -p 5005 --bind 10.0.0.2
root 3801 3781 5 19:04 pts/8 00:00:15 ./dpdk_iperf3 -s -p 5006 --bind 10.0.0.2
root 3888 3868 5 19:04 pts/9 00:00:14 ./dpdk_iperf3 -s -p 5007 --bind 10.0.0.2
root 3977 3957 5 19:05 pts/10 00:00:13 ./dpdk_iperf3 -s -p 5008 --bind 10.0.0.2
root 4106 1 4 19:08 pts/11 00:00:02 ./dpdk_iperf3 -s -p 5200 --bind 10.0.0.2 -A 2
root 4111 1 3 19:08 pts/11 00:00:02 ./dpdk_iperf3 -s -p 5201 --bind 10.0.0.2 -A 3
root 4114 1 3 19:08 pts/11 00:00:02 ./dpdk_iperf3 -s -p 5202 --bind 10.0.0.2 -A 4
root 4117 1 3 19:08 pts/11 00:00:01 ./dpdk_iperf3 -s -p 5203 --bind 10.0.0.2 -A 5
root 4133 1 3 19:08 pts/11 00:00:01 ./dpdk_iperf3 -s -p 5204 --bind 10.0.0.2 -A 6
root 4136 1 3 19:08 pts/11 00:00:01 ./dpdk_iperf3 -s -p 5205 --bind 10.0.0.2 -A 7

from dpdk-ans.

from dpdk-ans.

Yes, 8 seems to work. May be its 10, but before that after 6 rate drops, from the client side there are lot of retries which indicates NIC might be dropping packets.
Its a 10Gbps link, I am barely touching 50% utilization.

One thing might be number of Rx descriptors that are being used. In newer DPDK version, there is an API to set the Max RX descriptors (rte_eth_dev_adjust_nb_rx_tx_desc) but DPDK that ANS requires does not support this.

Do you know how to set max RX descriptors?

Typically via ethool we can see the max rx descriptors, but not sure how to check with this DPDK version.
ethtool -g ens15f0
Ring parameters for ens15f0:
Pre-set maximums:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 4096
Current hardware settings:
RX: 4096
RX Mini: 0
RX Jumbo: 0
TX: 512

from dpdk-ans.

Ok, after changing ANS_RX_DESC_DEFAULT to 512, seems to help.

Now, I am not able to scale beyond 8 instances. I thought it has to with with MAX_NB_SOCKETS sets to 8, but that did not help.

from dpdk-ans.

So, basically dpdk_iperf cores if I start beyond 8 instances. You should be able to see in your setup too.

from dpdk-ans.

Please run dpdk-iperf server by manual, if any output in the screen?

anssock app id:

from dpdk-ans.

You shall add all dpdk-iperf client throughput to check if reach the line rate of the NIC.

from dpdk-ans.

This is the output I see , when I start ANS on 10 cores, I see core on the 7th dpdk-iperf init itself. Wondering whether some mem alloc fails and asserts?

EAL: Detected 64 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
EAL: This may cause issues with mapping memory into secondary processes
EAL: Analysing 16 files
EAL: Mapped segment 0 of size 0x200000000
EAL: Mapped segment 1 of size 0x200000000
EAL: TSC frequency is ~2099996 KHz
EAL: Master lcore 0 is ready (tid=cbfb58c0;cpuset=[0])
USER8: LCORE[-1] anssock any lcore id 0xffffffff
USER8: LCORE[16] anssock app id: 11898
USER8: LCORE[16] anssock app name: dpdk_iperf3
USER8: LCORE[16] anssock app bind lcoreId: 7
USER8: LCORE[16] anssock app bind ringId: 6
USER8: LCORE[16] anssock app lcoreId: 16
./iperf.sh: line 33: 11898 Segmentation fault (core dumped) ./dpdk_iperf3 -s -p $port --bind 10.0.0.2 -A $core

from dpdk-ans.

Right now, Starting ANS on 8 cores and dpdk-iperf on 8 cores, seems stable.
When I start ANS on 10 cores and start dpdk-perf on 8 Cores, cores on initialization of 7 dpdk-iperf

from dpdk-ans.

Go to /dpdk-17.05.2/usertools directory and run "# ./cpu_layout.py" to show your cpu layout.
Maybe this issue related to numa.

from dpdk-ans.

This is what I see.

root @ ~/dpdk-17.05-rc2 - [] # usertools/cpu_layout.py

Core and Socket Information (as reported by '/proc/cpuinfo')

cores = [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15]
sockets = [0, 1]

    Socket 0        Socket 1       
    --------        --------       

Core 0 [0, 32] [16, 48]
Core 1 [1, 33] [17, 49]
Core 2 [2, 34] [18, 50]
Core 3 [3, 35] [19, 51]
Core 4 [4, 36] [20, 52]
Core 5 [5, 37] [21, 53]
Core 6 [6, 38] [22, 54]
Core 7 [7, 39] [23, 55]
Core 8 [8, 40] [24, 56]
Core 9 [9, 41] [25, 57]
Core 10 [10, 42] [26, 58]
Core 11 [11, 43] [27, 59]
Core 12 [12, 44] [28, 60]
Core 13 [13, 45] [29, 61]
Core 14 [14, 46] [30, 62]
Core 15 [15, 47] [31, 63]

from dpdk-ans.

If dpdk-iperf run on core 16, it will crash because no reserved memory pool on that numa.
I will fix it. thanks..
it will get better performance if application and ans are in the same CPU SOCKET, otherwise performance reduce much.

from dpdk-ans.

Ok, thanks. Is it an easy fix?

from dpdk-ans.

Yes, after moving to same socket performance is really good, thanks for the pointer.

Let me know if you have the fix to scale up more.

from dpdk-ans.

I will fix the issue at this weekend.
NIC is PIN to CPU socket based on hardware design, so NIC, ANS and application shall be in the same CPU socket. For other CPU socket, can run another group(NIC + ANS + application).

from dpdk-ans.

Sure. So the fix willlook into cpu layout to bind instead of binding serially?

from dpdk-ans.

already fixed the issue, please verify it.
Thanks.

from dpdk-ans.

Thanks for the fix, seems good now.

from dpdk-ans.