This requires patching the ixgbevf driver to support ZC, we already started working on this some time ago but need to complete it. We will try to allocate more time on the roadmap for this as soon as possible.

from pf_ring.

Thank you so much! I really appreciate your help! :)

On Fri, Jun 19, 2015 at 10:33 AM, Alfredo Cardigliano <
[email protected]> wrote:

This requires patching the ixgbevf driver to support ZC, we already
started working on this some time ago but need to complete it. We will try
to allocate more time on the roadmap for this as soon as possible.

—
Reply to this email directly or view it on GitHub
#15 (comment).

Sincerely yours, Pavel Odintsov

from pf_ring.

I'm also very much interested in ixgbevf ZC driver.

from pf_ring.

Very interested in this as well.

from pf_ring.

Any update on this?

from pf_ring.

Not yet, we are busy with other projects with higher priority. sorry.

from pf_ring.

@cardigliano: Is this supported now ?

from pf_ring.

Hi Alfredo,

Does the VF support is included latest PF_RING? I do see some commits related to it, please do confirm.

Regards,
Jessel.

from pf_ring.

@jcmathews this is still in progress, I will update this issue as soon as it is ready

from pf_ring.

Thanks

from pf_ring.

Support for ixgbe-vf has been added to ZC and it is available on latest dev branch. This is currently under testing, if you want to start playing with it you are welcome.

from pf_ring.

Hi Alfredo,

Do you have configuration details to hook vf's by using qemu cmd line(but not by libvirt)? Any script changes to examples_zc/kvm/host/vm-boot.sh

Regards,
Jessel.

from pf_ring.

@jcmathews I do not have a sample configuration for Qemu atm, I will work on that and update the documentation asap. (however you should be able to find instructions on google as it's a generic sriov configuration)

from pf_ring.

@cardigliano
Qemu v2.5 have some issue w.r.t to IOMMU, when using argument pci-assign.
Qemu v2.12 will boot up using vfio-pci, but ipc will not work since ivshmem-doorbell is not in place.
Generic SRIO configuration unbinds existing driver(say ixgbevf) and hook vf's using vfio-pci. Do you have idea about some other configuration.

from pf_ring.

@jcmathews could you provide more details about the configuration/processing pattern you want to achieve? Virtual Functions are not supposed to be used with IPC/ivshmem as they are two differnt ways of achieving the same, bringing some traffic to the VM

from pf_ring.

@cardigliano :
i have 2 10G cards (X540 & 82599ES) cards,
i will receive the traffic from X540, then pass it to VM's via multiple QMP sockets. The VM's does some packet processing and sent back traffic to 82599 virtual functions.
For this case, my infrastructure look forward to use QMP sockets for traffic distribution towards VM and SRIOV based configuration for sending out the traffic to outside world.

from pf_ring.

Got it. Could you comment more about "Generic SRIO configuration unbinds existing driver(say ixgbevf) and hook vf's using vfio-pci"? VF can be either on the host or assigned to a VM (unbinding it from the host), I guess you need the latter for sending from the VM, what is the problem with that?

from pf_ring.

@cardigliano :

Sorry for late reply.

The below step are from
https://www.linux-kvm.org/page/10G_NIC_performance:_VFIO_vs_virtio

Assign the VF to a guest
Unbind from igbvf driver and Bind to VFIO driver

unbind from previous driver (take igbvf device for example)
echo <vf_BDF> > /sys/bus/pci/device/<vf_BDF>/driver/unbind
lspci -s <vf_BDF> -n //to get its number
//it will return like below
0a:13.3 0200: 8086:1520 (rev 01)
//8086 1520 is its numeric number
bind to vfio-pci driver
echo 8086 1520 > /sys/bus/pci/drivers/vfio-pci/new_id
Now you can see this device is bound to vfio-pci driver
lspci -s <vf_BDF> -k

Create guest with direct passthrough via VFIO framework
qemu-kvm -m 16G -smp 8 -net none -device vfio-pci,host=81:10.0 -drive file=/var/lib/libvirt/images/rhel7.1.img,if=virtio -nographic
'-net none' tells qemu not emulate network devices
'-device vfio-pci,host=' designate a vfio-pci device and the device's host BDF

Problem 1:
Our qemu version 2.5 has issues with registering VFs and throws error as follows,
qemu-system-x86_64: -device pci-assign,host=08:10.3: No IOMMU found. Unable to assign device "(null)"
However if 2.12 is used, this error will not come.

Problem 2:
ivshmem to ivshmem-doorbell is not in place, so Qemu 2.12 version will not be compatible with PFRING.

Is above mentioned registration correct for VF's or any other way you used to use?

from pf_ring.

Hi @jcmathews, we added instructions for assigning a VF without virsh at http://www.ntop.org/guides/pf_ring/vm/sriov.html (which is what you are doing).
Could you confirm that this is working for you on Qemu 2.12 and you are able to use the ixgbevf on the VM? (besides the fact that ivshmem is not working on this Qemu version)
Thank you.

from pf_ring.

I got some errors as follows

WARNING: "ixgbevf_hv_set_rar_vf" [/root/PF_RING/drivers/intel/ixgbevf/ixgbevf-4.5.1-zc/src/ixgbevf.ko] undefined!
WARNING: "ixgbevf_hv_reset_hw_vf" [/root/PF_RING/drivers/intel/ixgbevf/ixgbevf-4.5.1-zc/src/ixgbevf.ko] undefined!

Kernel Version Used : 3.10.0-327.el7.x86_64 #1 SMP Thu Oct 29 17:29:29 EDT 2015 x86_64 x86_64 x86_64 GNU/Linux
RHEL7

#if IS_ENABLED(CONFIG_PCI_HYPERV) not enabled in VM. while compiling it throws error.

from pf_ring.

Strange this is working for me on a VM with 3.10.0-862.14.4.el7.x86_64, I am investigating..
Please provide cat /boot/config-3.10.0-327.el7.x86_64 | grep CONFIG_PCI_HYPERV

from pf_ring.

config-3.10.0-327.el7.x86_64.zip
CONFIG_PCI_HYPERV Not found

from pf_ring.

I pushed a patch to handle that case, please try compiling latest code from github:
cd PF_RING/drivers/intel/ixgbevf/ixgbevf-4.5.1-zc && make

from pf_ring.

48.253334] ixgbevf: Intel(R) 10GbE PCI Express Virtual Function Driver - version 4.5.1
[ 48.255431] Copyright(c) 1999 - 2018 Intel Corporation.
[ 48.322206] ixgbevf 0000:00:04.0: MAC address not assigned by administrator.
[ 48.324320] ixgbevf 0000:00:04.0: Assigning random MAC address
[ 48.327969] ixgbevf 0000:00:04.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 48.331626] ixgbevf 0000:00:04.0: 12:64:14:f9:18:65
[ 48.333524] ixgbevf: eth1: ixgbevf_probe: Intel(R) 82599 Virtual Function
[ 48.334782] ixgbevf: eth1: ixgbevf_probe: GRO is enabled
[ 48.336058] ixgbevf: eth1: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
[ 48.348392] [PF_RING] removing dev=ens4 ifindex=5 (2)
[ 48.352923] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready.

zsend if failing to send packets out.

[root@localhost examples_zc]# ./zsend -i zc:ens4 -r 2 -l 256
#########################################################################

ERROR: You do not seem to have a valid PF_RING ZC 7.3.0.181122

license for ens4 (MAC 12:64:14:F9:18:65) [Intel 10 Gbit ixgbe VF]

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################

We're now working in demo mode with packet capture and

transmission limited to 5 minutes

#########################################################################
#########################################################################

ERROR: You do not seem to have a valid PF_RING ZC 7.3.0.181122

license for ens4 (MAC 12:64:14:F9:18:65) [Intel 10 Gbit ixgbe VF]

ERROR: Please get one at http://shop.ntop.org/.

#########################################################################
Sending packets to zc:ens4
Estimated CPU freq: 1745089500 Hz
Rate set to 2.00 Gbit/s, 64-byte packets, 2976190 pps

Absolute Stats: 4'096 pkts - 1'146'880 bytes

=========================
Absolute Stats: 4'096 pkts - 1'146'880 bytes
Actual Stats: 0.00 pps - 0.00 Gbps [0 bytes / 1.0 sec]

My host kernel cmd line is not having these "msi=1 pci=assign-busses", its an RHEL7 machine.
VF's are created.
75: ens3f1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN mode DEFAULT qlen 1000
link/ether 14:02:ec:96:ed:65 brd ff:ff:ff:ff:ff:ff
vf 0 MAC 00:01:02:03:04:06, spoof checking on, link-state auto, trust off, query_rss off
vf 1 MAC 12:64:14:f9:18:65, spoof checking on, link-state auto, trust off, query_rss off

from pf_ring.

@jcmathews could you try with pfsend (which is using a different memory allocation method) to figure out if this is related to hugepages?
Did you also check pfcount to receive traffic?

from pf_ring.

Host Logs:
[340246.523724] ixgbe 0000:08:00.1 ens3f1: 1934439 Spoofed packets detected
[340248.528037] ixgbe 0000:08:00.1 ens3f1: 2287641 Spoofed packets detected
[340250.531983] ixgbe 0000:08:00.1 ens3f1: 2287730 Spoofed packets detected
[340250.532131] ixgbe 0000:08:00.1 ens3f1: initiating reset due to lost link with pending Tx work
[340252.536104] ixgbe 0000:08:00.1 ens3f1: Reset adapter
[340254.801276] ixgbe 0000:08:00.1 ens3f1: detected SFP+: 5

VM Logs:

[ 196.394348] Copyright(c) 1999 - 2018 Intel Corporation.
[ 196.460808] ixgbevf 0000:00:04.0: MAC address not assigned by administrator.
[ 196.463247] ixgbevf 0000:00:04.0: Assigning random MAC address
[ 196.466332] ixgbevf 0000:00:04.0: Multiqueue Disabled: Rx Queue count = 1, Tx Queue count = 1
[ 196.468624] ixgbevf 0000:00:04.0: 5a:27:c9:bb:f6:a7
[ 196.469158] ixgbevf: eth1: ixgbevf_probe: Intel(R) 82599 Virtual Function
[ 196.470125] ixgbevf: eth1: ixgbevf_probe: GRO is enabled
[ 196.471265] ixgbevf: eth1: ixgbevf_probe: Intel(R) 10GbE PCI Express Virtual Function Driver
[ 196.505308] [PF_RING] removing dev=ens4 ifindex=6 (2)
[ 196.511071] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[ 196.529759] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[ 196.531917] IPv6: ADDRCONF(NETDEV_UP): ens4: link is not ready
[ 198.619689] load_driver.sh (14458): drop_caches: 3
[ 243.393594] bash (14613): drop_caches: 3
[ 277.957133] Bits 55-60 of /proc/PID/pagemap entries are about to stop being page-shift some time soon. See the linux/Documentation/vm.

Pfsend:
Using PF_RING v.7.3.0
Estimated CPU freq: 2656320000 Hz
Rate set to 2.00 Gbit/s, 260-byte packets, 892857.14 pps
Dumping statistics on /proc/net/pf_ring/stats/15070-ens4.2
TX rate: [current 1'093'826.08 pps/2.45 Gbps][average 1'093'826.08 pps/2.45 Gbps][total 1'093'895.00 pkts]
TX rate: [current 1'141'535.75 pps/2.56 Gbps][average 1'117'681.79 pps/2.50 Gbps][total 2'235'586.00 pkts]
TX rate: [current 1'141'567.48 pps/2.56 Gbps][average 1'125'644.56 pps/2.52 Gbps][total 3'377'456.00 pkts]
TX rate: [current 1'141'538.66 pps/2.56 Gbps][average 1'129'618.16 pps/2.53 Gbps][total 4'519'199.00 pkts]
TX rate: [current 1'141'561.76 pps/2.56 Gbps][average 1'132'007.06 pps/2.54 Gbps][total 5'661'053.00 pkts]
TX rate: [current 1'141'545.48 pps/2.56 Gbps][average 1'133'596.76 pps/2.54 Gbps][total 6'802'772.00 pkts]
TX rate: [current 1'141'555.41 pps/2.56 Gbps][average 1'134'733.63 pps/2.54 Gbps][total 7'944'437.00 pkts]
TX rate: [current 880'062.71 pps/1.97 Gbps][average 1'102'901.38 pps/2.47 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 980'362.18 pps/2.20 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 882'330.56 pps/1.98 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 802'122.55 pps/1.80 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 735'281.58 pps/1.65 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 678'716.90 pps/1.52 Gbps][total 8'824'593.00 pkts]
TX rate: [current 0.00 pps/0.00 Gbps][average 630'239.11 pps/1.41 Gbps][total 8'824'593.00 pkts]

Packets started flowing out and then immediately stopped. Retries also didnt work

from pf_ring.

@jcmathews are you reforging the MAC address to match what is configured for the VF? It seems the adapter detects spoofed packets. Please note pfsend supports that:
-M Reforge source MAC (format AA:BB:CC:DD:EE:FF)
-m Reforge destination MAC (format AA:BB:CC:DD:EE:FF)

from pf_ring.

@cardigliano
Irrespective of mac address, pfsend should be able to send packets down to adapter. Here i am able to send some packets at start(only once after VM is up) then it stop's sending. Say my case is to work in promiscuous mode here. Am i missing something?

kernel cmd line is not having these "msi=1 pci=assign-busses. Do you see any impacts?

from pf_ring.

@jcmathews please note that spoof check is a generic feature the adapter provide (not related to ZC), you can disable it with:
ip link set vf 0 spoofchk off
Please let me know if it works for you.

from pf_ring.

is there a way this zc working on OVH public clouds?
I don't have KVM management access since public clouds are all guest VMs.
nfnetlink_queue 24576 0
nfnetlink_log 20480 0
nfnetlink 16384 2 nfnetlink_queue,nfnetlink_log
virtio_net 49152 0

from pf_ring.