Giter VIP home page Giter VIP logo

openwrt-mlnx-ofed's Introduction

Openwrt-mlnx-ofed

  • 本项目为OpenWRT Package,在编译过程中选中以集成最新版本MLNX_OFED驱动。将默认提供MLNX_OFED 4.9版本驱动,同时支持CX3,CX4及之后的Mellanox网卡。本项目开发目的如下:

  • a) 增加IB网络支持

  • 通过安装本驱动,可以令你的OpenWRT路由器同时支持Infiniband和ETH网络,并能利用ipoib在ib网络上实现tcpip通信。本源码包正确编译运行后,输入
ip address
  • 即可在输出中看到“ib0” ipoib网络适配器。
  • b) 在IB子网与ETH子网间通信
  • 实例网络拓扑如下:外网通过eth接入wan端口,eth子网从lan端口接出,而ib子网从ib0端口输出。
  • 需要先在IB子网的一台机器上开启opensm服务程序(暂不支持Openwrt),然后给ib0适配器配置ip地址,并将lan与ib0置于防火墙的同一区域下。这时ib适配器可与lan侧的eth设备通信,同理也能和wan侧通信。ib子网将与原有的eth网络通过openwrt三层功能连在一起,从而自由地访问eth网络。
  • c) 增强SRIOV兼容性
  • 在虚拟机宿主开启SRIOV功能,并将VF直通进入OpenWRT虚拟机,能达到极佳的网络性能。但是OpenWRT集成的Mellanox系列网卡驱动存在问题,会导致OpenWRT系统在这种应用场景下无限重启,使用本源码包替代原有的mlx4,mlx5驱动,可解决这一问题。

使用方法:

  1. lede/package$下运行 或者openwrt/package$下运行
git clone https://github.com/allegro0132/Openwrt-mlnx-ofed.git
  1. make menuconfig 并在编译过程中选择"mlnx_ofed",取消选择"kmod-mlx5, kmod-mlx4"

  2. 一点优化(可选) 由于compat与mlx_compat存在冲突,ath系列和rx2x00系列驱动无法安装,并会在开机时产生如下报错

kmodloader: 15 modules could not be probed

可依如下次序取消选择这些不兼容的kmod,让开机界面清爽一些:

kmod-rt2800-usb
kmod-rt2x00-lib
kmod-ath9k
kmod-ath9k-htc
kmod-ath5k
kmod-ath10k
kmod-ath
  • 说明
  • 若直接使用Mellanox提供的源码编译,会在mlx5_core的编译过程中报错。我简单的修改了源码,使其能正常编译,但目前尚不明确改版驱动是否会导致问题。
 CC [M]  /lede/build_dir/target-x86_64_musl/linux-x86_64/mlnx_ofed-4.9-2.2.4.0/mlnx-ofed-kernel-4.9/drivers/net/ethernet/mellanox/mlx5/core/en_tc.o
In file included from /lede/build_dir/target-x86_64_musl/linux-x86_64/mlnx_ofed-4.9-2.2.4.0/mlnx-ofed-kernel-4.9/drivers/net/ethernet/mellanox/mlx5/core/en_tc.c:50:
/lede/build_dir/target-x86_64_musl/linux-x86_64/mlnx_ofed-4.9-2.2.4.0/mlnx-ofed-kernel-4.9/include/net/tc_act/tc_mirred.h: In function 'to_mirred_compat':
/lede/build_dir/target-x86_64_musl/linux-x86_64/mlnx_ofed-4.9-2.2.4.0/mlnx-ofed-kernel-4.9/include/net/tc_act/tc_mirred.h:51:24: error: dereferencing pointer to incomplete type 'const struct tc_action_ops'
 if (!a->ops || !a->ops->dump || !is_tcf_mirred_compat(a))
                       ^~
scripts/Makefile.build:261: recipe for target '/lede/build_dir/target-x86_64_musl/linux-x86_64/mlnx_ofed-4.9-2.2.4.0/mlnx-ofed-kernel-4.9/drivers/net/ethernet/mellanox/mlx5/core/en_tc.o' failed
  • 我在这里直接去掉
|| !a->ops->dump
  • 具体的改动已经注释在"mlnx-ofed-kernel-4.9/drivers/net/ethernet/mellanox/mlx5/core/en_tc.o"中,还望大家不吝赐教,带来更加优雅的解决方案。

openwrt-mlnx-ofed's People

Contributors

allegro0132 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar

openwrt-mlnx-ofed's Issues

请教HW offloading

非常感谢你的贡献!!

请教一下:

在虚拟机宿主开启SRIOV功能,并将VF直通进入OpenWRT虚拟机,能达到极佳的网络性能。但是OpenWRT集成的Mellanox系列网卡驱动存在问题,会导致OpenWRT系统在这种应用场景下无限重启,使用本源码包替代原有的mlx4,mlx5驱动,可解决这一问题。

  • 对于如下ConnectX-4 Lx支持的特性,不知道您的驱动一样支持嘛?

CPU Offloads
– RDMA over Converged Ethernet (RoCE)
– TCP/UDP/IP stateless offload
– LSO, LRO, checksum offload
– RSS (can be done on encapsulated packet), TSS, VLAN insertion/stripping, Receive flow steering
– Intelligent interrupt coalescence

最后,其实我想使用OpenWRT的SQM功能,没查到ConnectX-4 Lx是否支持offloading到硬件上处理SQM。。。

感谢赐教!!

New builds?

Hello. Thank you for awesome work.

Can you please make new build based on 6.1 kernel please?
Or use some stable release instead of snapahot?
Because updating system breakes itself when there are ton of differences compared to current snapshot. Worst is difference in kernel version i guess.

Maybe you can also use github actions to automate your builds.

Thanx.

可否指导一下怎么编译Qlogic QL41x6x的驱动呢?

适用于网络适配器的 Qlogic Linux 驱动的Linux源代码在下边,且是可以在Ubuntu 20.10正常使用的。

sudo lspci | grep Ethernet
03:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
0b:00.0 Ethernet controller: QLogic Corp. FastLinQ QL41000 Series Gigabit Ethernet Controller (SR-IOV VF) (rev 02)

https://www.dell.com/support/home/zh-cn/drivers/driversdetails?driverid=rtmtk

但是我不懂怎么改成OpenWRT可以编译的Makefile,Kconfig文件。。。可否指导一下?

官方Linux驱动在这里:
https://driverdownloads.qlogic.com/QLogicDriverDownloads_UI/Defaultnewsearch.aspx

image

关于iommu_groups

这个貌似是你写的? https://zhuanlan.zhihu.com/p/356437308

再综合如下两篇材料,

https://www.thomas-krenn.com/en/wiki/Enable_Proxmox_PCIe_Passthrough
http://khmel.org/proxmox-debian-10-kvm-enabling-sr-iov-for-mellanox-infiniband-cards.html

我发现一个不知道怎么解的问题,就是iommu_groups都是同一个,这应该是不对的。您知道问题出在那儿吗?
目前环境 PVE 7.02,网卡和你的是一样的,firmware version: 14.31.1014
主板:Asrock C246 WSI
CPU: i5-8400T (edit: 查了一圈,貌似最大的问题出现在这些消费类的CPU并不支持ACS,也就是不能够将一个物理的网卡的各个VF独立到不同的IOMMU Group, 不知道你的CPU是什么呢?)
PCIE7 是从CPU出来的

image

IOMMU Group 1 01:00.0 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
IOMMU Group 1 01:00.1 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx] [15b3:1015]
IOMMU Group 1 01:00.2 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
IOMMU Group 1 01:00.3 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
IOMMU Group 1 01:00.4 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
IOMMU Group 1 01:00.5 Ethernet controller [0200]: Mellanox Technologies MT27710 Family [ConnectX-4 Lx Virtual Function] [15b3:1016]
root@asrock:~# lspci -s 01:00 -v
01:00.0 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
        Flags: bus master, fast devsel, latency 0, IRQ 16, IOMMU group 1
        Memory at a8000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at a6b00000 [disabled] [size=1M]
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [48] Vital Product Data
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [1c0] Secondary PCI Express
        Capabilities: [230] Access Control Services
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core

01:00.1 Ethernet controller: Mellanox Technologies MT27710 Family [ConnectX-4 Lx]
        Subsystem: Mellanox Technologies Stand-up ConnectX-4 Lx EN, 25GbE dual-port SFP28, PCIe3.0 x8, MCX4121A-ACAT
        Flags: bus master, fast devsel, latency 0, IRQ 17, IOMMU group 1
        Memory at aa000000 (64-bit, prefetchable) [size=32M]
        Expansion ROM at a6a00000 [disabled] [size=1M]
        Capabilities: [60] Express Endpoint, MSI 00
        Capabilities: [48] Vital Product Data
        Capabilities: [9c] MSI-X: Enable+ Count=64 Masked-
        Capabilities: [c0] Vendor Specific Information: Len=18 <?>
        Capabilities: [40] Power Management version 3
        Capabilities: [100] Advanced Error Reporting
        Capabilities: [150] Alternative Routing-ID Interpretation (ARI)
        Capabilities: [180] Single Root I/O Virtualization (SR-IOV)
        Capabilities: [230] Access Control Services
        Kernel driver in use: mlx5_core
        Kernel modules: mlx5_core
root@asrock:~# lsmod | grep vfio
vfio_pci               57344  0
vfio_virqfd            16384  1 vfio_pci
irqbypass              16384  2 vfio_pci,kvm
vfio_iommu_type1       36864  0
vfio                   36864  2 vfio_iommu_type1,vfio_pci
root@asrock:~# dmesg |grep -e DMAR -e IOMMU
[    0.019183] ACPI: DMAR 0x000000007A47DCB0 0000C8 (v01 INTEL  EDK2     00000002      01000013)
[    0.019269] ACPI: Reserving DMAR table memory at [mem 0x7a47dcb0-0x7a47dd77]
[    0.086967] DMAR: IOMMU enabled
[    0.221486] DMAR: Host address width 39
[    0.221488] DMAR: DRHD base: 0x000000fed90000 flags: 0x0
[    0.221496] DMAR: dmar0: reg_base_addr fed90000 ver 1:0 cap 1c0000c40660462 ecap 19e2ff0505e
[    0.221500] DMAR: DRHD base: 0x000000fed91000 flags: 0x1
[    0.221505] DMAR: dmar1: reg_base_addr fed91000 ver 1:0 cap d2008c40660462 ecap f050da
[    0.221508] DMAR: RMRR base: 0x000000794b9000 end: 0x000000794d8fff
[    0.221510] DMAR: RMRR base: 0x0000007f000000 end: 0x0000008f7fffff
[    0.221512] DMAR: RMRR base: 0x00000079d18000 end: 0x00000079d97fff
[    0.221514] DMAR-IR: IOAPIC id 2 under DRHD base  0xfed91000 IOMMU 1
[    0.221517] DMAR-IR: HPET id 0 under DRHD base 0xfed91000
[    0.221519] DMAR-IR: Queued invalidation will be enabled to support x2apic and Intr-remapping.
[    0.224118] DMAR-IR: Enabled IRQ remapping in x2apic mode
[    2.087938] DMAR: No ATSR found
[    2.087940] DMAR: dmar0: Using Queued invalidation
[    2.087946] DMAR: dmar1: Using Queued invalidation
[    2.088902] DMAR: Intel(R) Virtualization Technology for Directed I/O

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.