Giter VIP home page Giter VIP logo

litepcie's Introduction

                                  __   _ __      ___  _________
                                 / /  (_) /____ / _ \/ ___/  _/__
                                / /__/ / __/ -_) ___/ /___/ // -_)
                               /____/_/\__/\__/_/   \___/___/\__/

                               Copyright 2015-2022 / EnjoyDigital

                            A small footprint and configurable PCIe core
                                     powered by Migen & LiteX

License

[> Intro

LitePCIe provides a small footprint and configurable PCIe core.

LitePCIe is part of LiteX libraries whose aims are to lower entry level of complex FPGA cores by providing simple, elegant and efficient implementations of components used in today's SoC such as Ethernet, SATA, PCIe, SDRAM Controller...

Using Migen to describe the HDL allows the core to be highly and easily configurable.

LitePCIe can be used as LiteX library or can be integrated with your standard design flow by generating the verilog rtl that you will use as a standard core.

[> Features

PHY:

  • Xilinx Ultrascale(+) (up to PCIe Gen3 X16).
  • Xilinx 7-Series (up to PCIe Gen2 X8).
  • Intel Cyclone5 (up to PCIe Gen2 X4).
  • 64/128/256/512-bit datapath.
  • Clock domain crossing.

Core:

  • TLP layer.
  • Reordering.
  • MSI (Single, Multi-vector)/MSI-X.
  • Crossbar.

Frontend:

  • DMA (with Scatter-Gather).
  • MMAP (AXI/Wishbone Slave/Master).
  • PTM (on Xilinx 7-Series/Gen2 X1 for now).

Software:

  • Linux Driver (MMAP and DMA).

[> FPGA Proven

LitePCIe is already used in commercial and open-source designs:

[> Possible improvements

  • add standardized interfaces (AXI, Avalon-ST)
  • add Intel Stratix support
  • add Lattice support
  • add more documentation
  • ... See below Support and consulting :)

If you want to support these features, please contact us at florent [AT] enjoy-digital.fr.

[> Getting started

  1. Install Python 3.6+ and FPGA vendor's development tools.
  2. Install LiteX and the cores by following the LiteX's wiki installation guide.
  3. You can find examples of integration of the core with LiteX in LiteX-Boards and in the examples directory.

[> Tests

Unit tests are available in ./test/. To run all the unit tests:

$ ./setup.py test

Tests can also be run individually:

$ python3 -m unittest test.test_name

[> License

LitePCIe is released under the very permissive two-clause BSD license. Under the terms of this license, you are authorized to use LitePCIe for closed-source proprietary designs. Even though we do not require you to do so, those things are awesome, so please do them if possible:

  • tell us that you are using LitePCIe
  • cite LitePCIe in publications related to research it has helped
  • send us feedback and suggestions for improvements
  • send us bug reports when something goes wrong
  • send us the modifications and improvements you have done to LitePCIe.

[> Support and consulting

We love open-source hardware and like sharing our designs with others.

LitePCIe is developed and maintained by EnjoyDigital.

If you would like to know more about LitePCIe or if you are already a happy user and would like to extend it for your needs, EnjoyDigital can provide standard commercial support as well as consulting services.

So feel free to contact us, we'd love to work with you! (and eventually shorten the list of the possible improvements :)

[> Contact

E-mail: florent [AT] enjoy-digital.fr

litepcie's People

Contributors

a3f avatar enjoy-digital avatar felixheld avatar johnsel avatar maleadt avatar mithro avatar sergachev avatar sjkelly avatar smunaut avatar timkpaine avatar tongchen126 avatar trabucayre avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

litepcie's Issues

Add Power9 support

Quick test on Power9 with PCIe X4 / Gen2 report issue:
EEH: PHB#0 failure detected, location: N/A
Probably related to the kernel driver.

Using litepcie with LiteFury (SQRL CLE-101)

In #38, @enjoy-digital 's recommendation was to try this out with the SQRL CLE-215. I got a LiteFury, which according to litex-boards is equivalent to the CLE-101, its little sibling.

I would like to get it running as PCIe endpoint and read the LiteFury's memory via PCIe (analogously to the standard LiteFury demo).

I am following the steps outlined in https://github.com/litex-hub/litex-boards/blob/master/litex_boards/targets/sqrl_acorn.py#L10:

# disable SATA, otherwise Vivado fails to satisfy the routing
python3 -m litex_boards.targets.sqrl_acorn --driver --variant=cle-101 --with-pcie --build --load
echo 1 > $pcidevice/remove
echo 1 > /sys/bus/pci/rescan

I see the BIOS boot up over UART, but the CLE-101 doesn't reappear in lspci output and the litepcie driver won't bind. Removing and rescanning worked with the Vivado example for LiteFury, so I'd assume it not to be the problem. I can't test with --flash, it reports Error: Unknown flash device (ID 0x00000000) (Flashing the LiteFury example via Vivado to SPI NOR worked though).

So my questions are:

  • Did I miss any obvious steps?
  • Do I need a specific Vivado version?
  • Did you test PCIe with CLE-101? Is there maybe some PCIe specific configuration missing?
  • Any other pointers how to further debug this?

Thanks!

clock domain separation of litepcie

thanks in advance
the structure of litepcie is the way the it wont let you to separate the pcie dma and its related modules along TLP handler and endpoint form soc clock domain it will only let you to do CDC after endpoint which result in bandwidth loss
for instance if I use a 250mhz 64 bandwidth endpoint then I have to CDC it to CPU clock of 100 mhz that means more that half of
DMA BW is dropped also there is no chance in increasing the soc cpu clock
as you know clock frequency of CPU can be low as 100mhz but I need to have a high performance high throughput pcie dma winch is separate form CPU clock domain .
that means it has to have a CDC for its control form CPU and has a CDC over its CSR register
but the problem here is that will causes a lot more resource utilization you have to CDC its wishbone and then use a CSR on its own clock that means it would be better to have a separate wishbone and CSR from SOC and then have CDC connection form soc or to soc (also I dont see any need for connection to soc only master of pcie lite (that is the PC) has to have access to these CSR so soc cpu dose not need any connection to read or write for example dma buffer (there can be use case although ))
in summery for now there is no possible way to have a high BW pcielite with soc enabled
even increasing data wdith wont solve the issue for higher speed US and higher BW at the end the Clock frequncy limitation of
CPU and interlock of the structure between CPU and pcielite will result in lower clock secltion to make evry thing work
please find a solution to that or separate pcielite from soc cpu clock doamin properly
thanks

Issues with UltraScale+ PCIe IP naming

Using Vivado 2019.2, it seems like the PCIe block name has changed from pcie4c_uscale_plus to pcie4_uscale_plus, so the build fails during the IP generation until this is manually changed (still waiting for the build to finish so I haven't tested functionality yet).

Building userspace software without a crossover UART

I found while working on the Alveo U250 that the user software build fails when a normal, non-crossover UART is used; as CSR_UART_BASE is defined but CSR_UART_XOVER_RXEMPTY_ADDR isn't:

gcc -c -O2 -Wall -g -I../kernel -MMD -o litepcie_util.o litepcie_util.c
litepcie_util.c: In function ‘uart_test’:
litepcie_util.c:458:33: error: CSR_UART_XOVER_RXEMPTY_ADDR’ undeclared (first use in this function); did you mean ‘CSR_UART_RXEMPTY_ADDR’?
  458 |         if ((litepcie_readl(fd, CSR_UART_XOVER_RXEMPTY_ADDR) & 0x1) == 0) {
      |                                 ^~~~~~~~~~~~~~~~~~~~~~~~~~~
      |                                 CSR_UART_RXEMPTY_ADDR
litepcie_util.c:458:33: note: each undeclared identifier is reported only once for each function it appears in
litepcie_util.c:459:45: error: CSR_UART_XOVER_RXTX_ADDR’ undeclared (first use in this function); did you mean ‘CSR_UART_RXTX_ADDR’?
  459 |             printf("%c", litepcie_readl(fd, CSR_UART_XOVER_RXTX_ADDR) & 0xff);
      |                                             ^~~~~~~~~~~~~~~~~~~~~~~~
      |                                             CSR_UART_RXTX_ADDR
make: *** [Makefile:20: litepcie_util.o] Error 1

I think the #ifdef needs to be changed to something more specific.

LitePCIeDMALoopback behavior

Hi Sir,
Could you describe the behavior of LitePCIeDMALoopback class in more details ?
It means that the data transferred by some RX buffer descriptor in DMA READER table will go directly to the destination of the corresponding TX buffer descriptor in DMA WRITER table?

Thanks.
Jimmy

conflict IRQ

Hi guys:

I tried to run the example but ran into following errors. Any suggestion?

Problem

[shaolin@shaolin example_designs]$ ./make.py all
Traceback (most recent call last):
  File "./make.py", line 92, in <module>
    soc = top_class(platform, **top_kwargs)
  File "/home/shaolin/Lattice/litepcie/example_designs/targets/dma.py", line 61, in __init__
    with_timer=False
  File "/usr/local/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/soc/integration/soc_core.py", line 176, in __init__
    interrupt, mod_name, interrupt_rmap[interrupt]))

My configuration

  1. python 3.5
  2. litex version
[shaolin@shaolin litex]$ git show
commit 2ecd1b06668ff4d896e9462f2e967de783fb9792
Merge: c83ae98 0ac3530
Author: enjoy-digital <[email protected]>
Date:   Fri Jan 26 01:58:37 2018 +0100

    Merge pull request #61 from PaulSchulz/master
  1. litedram version
[shaolin@shaolin litedram]$ git show
commit ec9ad2fc3905e9dae86b5f3c2bf243dd5ee5087f
Author: Florent Kermarrec <[email protected]>
Date:   Wed Jan 31 09:32:21 2018 +0100

    frontend/dma: add description of fifo_buffered parameter
  1. litepcie version
[shaolin@shaolin litepcie]$ git show
commit 7b5b8063b3beaf76a12dcdda87dfcf91b1b79707
Author: Florent Kermarrec <[email protected]>
Date:   Sun Jan 28 03:04:49 2018 +0100

    core/tlp/depacketizer: simplify using NextValue

Move Ultrascale(+) TLP adaption code from Verilog to Migen

The AXI streams exposed by the on Ultrascale (+) PHYs are not standardized TLPs. To support the Ultrascale (+) PHYs in LitePCIe (that operates on standardized TLPs), the AXI streams have been adapted for both downstream/upstream directions in pcie_us(p)_support.v files. To simplify the code/integration and avoid too much code duplication, it would be good to move this code to Migen which would also ease supporting various data widths.

USP PCIe x8 dma_test hanging

When trying to use a x8 link width, the dma_test hangs. Everything seems to be fine with a x4 core.

So for the x4 core, I instanciate it with :

            self.submodules.pcie_phy = USPPCIEPHY(platform, platform.request("pcie_x4"),
                speed = "gen3",
                data_width = 128,
                bar0_size  = 0x20000)

And for x8 core, I do it like this :

            self.submodules.pcie_phy = USPPCIEPHY(platform, platform.request("pcie_x8"),
                speed = "gen3",
                data_width = 256,
                bar0_size  = 0x20000)

The litepcie module loads fine and detects the card :

[  134.322424] litepcie: loading out-of-tree module taints kernel.
[  134.322526] litepcie: module verification failed: signature and/or required key missing - tainting kernel
[  134.323251] litepcie 0000:01:00.0: \x1b[1m[Probing device]\x1b[0m
[  134.323274] litepcie 0000:01:00.0: enabling device (0000 -> 0002)
[  134.323495] litepcie 0000:01:00.0: Version LiteX SoC on Xilinx ADRV2CRR-FMC 2022-03-02 09:26:12
[  134.323570] litepcie 0000:01:00.0: 1 MSI IRQs allocated.
[  134.323603] litepcie 0000:01:00.0: Creating /dev/litepcie0

And the info and scratch_test also work, but not dma_test :

root@asuka /home/tnt # ./driver/user/litepcie_util info
FPGA identification: LiteX SoC on Xilinx ADRV2CRR-FMC 2022-03-02 09:26:12
root@asuka /home/tnt # ./driver/user/litepcie_util scratch_test
Write 0x12345678 to scratch register:
Read: 0x12345678
Write 0xdeadbeef to scratch register:
Read: 0xdeadbeef
root@asuka /home/tnt # ./driver/user/litepcie_util dma_test

[hangs forever...]

Support for Xilinx soft PCIe PHY?

Hello,

I'm wondering if parts of this project can be used to control the Xilinx soft PCIe PHY (PG239)? If I'm right, the provided examples only use the hardened-in-silicon IPs from Xilinx.

Any experience with that core, or do you known some other resources for that?

Thanks :)

Is there anyway to specify the install dir of the FPGA tools?

Hey guys:
Nice work but I still ran into problem after I installed the Xilinx tools:

-bash-4.1$ ./make.py all

      __   _ __      ___  _________
     / /  (_) /____ / _ \/ ___/  _/__
    / /__/ / __/ -_) ___/ /___/ // -_)
   /____/_/\__/\__/_/   \___/___/\__/

  A small footprint and configurable PCIe
          core powered by Migen
====== Building options: ======
Platform:  kc705
Target:    dma
Subtarget: PCIeDMASoC
System Clk: 125.0 MHz
===============================
Traceback (most recent call last):
  File "./make.py", line 154, in <module>
    vns = platform.build(soc, build_name=build_name, **build_kwargs)
  File "/homes/shx089/tools/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/build/xilinx/platform.py", line 33, in build
    return self.toolchain.build(self, *args, **kwargs)
  File "/homes/shx089/tools/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/build/xilinx/vivado.py", line 204, in build
    _run_vivado(build_name, toolchain_path, source)
  File "/homes/shx089/tools/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/build/xilinx/vivado.py", line 65, in _run_vivado
    settings = common.settings(vivado_path, "Vivado", ver, first="name")
  File "/homes/shx089/tools/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/build/xilinx/common.py", line 51, in settings
    ver = max(vers)
  File "/homes/shx089/tools/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/build/tools.py", line 34, in versions
    for n in os.listdir(path):
FileNotFoundError: [Errno 2] No such file or directory: '/opt/Xilinx/Vivado

I installed the vivado in a dir other than /opt/Xilinx/Vivado. How can I change that? I there any parameter that I can override on command line?

Thanks in advance.

Add PCIe <-> Wishbone function?

How about adding this PCIe <-> Wishbone bridge to litepcie? Basically, it can transfer data between the Host PCIe Bus and SoC's wishbone bus. Code is here. The LiteWishbone2PCIeDMA/LitePCIe2WishboneDMA receives control from PCIe Driver and LiteWishbone2PCIeDMANative/LitePCIe2WishboneDMANative receives control from internal circuit.

You can see how to start a dma transaction from PCIe to Wishbone here by using LitePCIe2WishboneDMA started by Host PCIe driver. Or starting a dma transaction by pull up the start signal using LitePCIe2WishboneDMANative.

The stability has been tested over 24 hours as I used it to build a Gigabyte PCIe Ethernet Adapter.

In my local repo, I placed this code under 'litepcie/frontend' but it may also serve as a demo under 'example' folder in the first place.

there is no packet separation capability for dma

thanks in advance
I need to indicate start and end of packet for DMA
but there is no indication in scatter gather table for sart and end of packet
requesting addition of RX/TX SOF/EOF to the descriptors.
(tlast wont reach user I have to use escape coding to separate my packets or have a packet base transmission and use their header length to seperate data packets and create which it should be produced by DMA scatter gather descriptor )

flawed DMA driver

thanks in advance
your dma on fpga side is capable of accepting different descriptors with different length of data and address
but on the software side it assumed that only with the length in byte of 8192 and 256 number of descriptors is used
and there is no possible way to DMA in either MMAP and or copy with length larger or other that the 8192
the structure is that the kernel driver is always update descriptors with predefined length and if I read or write anything other that
8192 length it either wont or drop some of it
the proper usage was to create and update descriptor based on the length taken form user space request
but in its interrupt handler it always update with a fixed value
and also addition of read with the capability that uses buffered endpoint data length to read the last renaming data in buffer would be nice (to read exactly the same amount of data that already exist (for the reading of last packet or blocking control word data through dma ))

PCIe bus never comes alive

I am trying to use LitePCie for my project with the Trenz TE0712 board. The target and platform files are not there in the litex_boards, so I made my own files mainly following the sqrl_acron board. Here is a minimal file example_test.py I created that includes only PCIe in the design. This board uses PCIe_x1 and 64 bit data width.

from litex.build.generic_platform import *
from litex.build.xilinx import XilinxPlatform, VivadoProgrammer
from litex.build.openocd import OpenOCD

import os
import argparse
import sys

from migen import *

# from litex_boards.platforms import acorn

from litex.soc.interconnect.csr import *
from litex.soc.integration.soc_core import *
from litex.soc.integration.builder import *

from litex.soc.cores.clock import *
from litex.soc.cores.led import LedChaser

from litedram.modules import MT41K256M16
from litedram.phy import s7ddrphy

from litepcie.phy.s7pciephy import S7PCIEPHY
from litepcie.core import LitePCIeEndpoint, LitePCIeMSI
from litepcie.frontend.dma import LitePCIeDMA
from litepcie.frontend.wishbone import LitePCIeWishboneBridge
from litepcie.software import generate_litepcie_software


# IOs ----------------------------------------------------------------------------------------------
_io = [
    # clk / rst
    ("clk200", 0,
        Subsignal("p", Pins("H4"), IOStandard("DIFF_SSTL15")),
        Subsignal("n", Pins("G4"), IOStandard("DIFF_SSTL15"))
    ),

    # Leds
    ("sys_led",  0, Pins("U22"), IOStandard("LVCMOS33")),
    ("led2",  0, Pins("W22"), IOStandard("LVCMOS33")),
    
    # SPIFlash
    ("flash_cs_n", 0, Pins("T19"), IOStandard("LVCMOS33")),
    ("flash", 0,
        Subsignal("mosi", Pins("P22")),
        Subsignal("miso", Pins("R22")),
        Subsignal("wp",   Pins("P21")),
        Subsignal("hold", Pins("R21")),
        IOStandard("LVCMOS33"),
    ),

    # pcie
    ("pcie_x1", 0,
        Subsignal("rst_n", Pins("M17"), IOStandard("LVCMOS33"), Misc("PULLUP=TRUE")),
        Subsignal("clk_p", Pins("F10")),
        Subsignal("clk_n", Pins("E10")),
        Subsignal("rx_p",  Pins("B8")),
        Subsignal("rx_n",  Pins("A8")),
        Subsignal("tx_p",  Pins("B4")),
        Subsignal("tx_n",  Pins("A4")),
    ),

    # dram
    ("ddram", 0,
        Subsignal("a",       Pins(
            "J1 P6 N5 N3 G1 M3 N2 J5",
            "L1 P2 L4 P5 K2 M1 M5"),
            IOStandard("SSTL15")),
        Subsignal("ba",      Pins("P4 H5 H2"), IOStandard("SSTL15")),
        Subsignal("ras_n",   Pins("M6"),  IOStandard("SSTL15")),
        Subsignal("cas_n",   Pins("M2"), IOStandard("SSTL15")),
        Subsignal("we_n",    Pins("J2"),  IOStandard("SSTL15")),
        Subsignal("cs_n",    Pins("K1"), IOStandard("SSTL15")),
        Subsignal("dm",      Pins(
            "W2 Y7 V4 V5"),
            IOStandard("SSTL15")),
        Subsignal("dq",      Pins(
            "T1 U3 U2 U1 Y2 W1 Y1 V2",
            "V7 W9 AB7 AA8 AB8 AB6 Y8 Y9",
            "AB1 AB5 AB3 AA1 Y4 AA5 AB2 W4",
            "T4 U6 T6 AA6 Y6 T5 U5 R6"),
            IOStandard("SSTL15")),
        Subsignal("dqs_p",   Pins("R3 V9 Y3 W6"),
            IOStandard("DIFF_SSTL15")),
        Subsignal("dqs_n",   Pins("R2 V8 AA3 W5"),
            IOStandard("DIFF_SSTL15")),
        Subsignal("clk_p",   Pins("R1"), IOStandard("DIFF_SSTL15")),
        Subsignal("clk_n",   Pins("P1"), IOStandard("DIFF_SSTL15")),
        Subsignal("cke",     Pins("L3"), IOStandard("SSTL15")),
        Subsignal("odt",     Pins("K3"),  IOStandard("SSTL15")),
        Subsignal("reset_n", Pins("H3"),  IOStandard("LVCMOS15")),
        Misc("SLEW=FAST"),
    ),
]

_serial_io = [
    # Serial adapter on P2.
    ("serial", 0,
        Subsignal("tx", Pins("U18")),
        Subsignal("rx", Pins("P16")),
        Misc("SLEW=FAST"),
        IOStandard("LVCMOS33"),
    ),
]

# Platform -----------------------------------------------------------------------------------------
class Platform(XilinxPlatform):
    default_clk_name   = "clk200"
    default_clk_period = 1e9/200e6

    def __init__(self):
        XilinxPlatform.__init__(self, "xc7a200tfbg484-2", _io, toolchain="vivado")
        self.add_platform_command("set_property INTERNAL_VREF 0.750 [get_iobanks 34]")

        self.toolchain.bitstream_commands = [
            "set_property BITSTREAM.CONFIG.SPI_BUSWIDTH 4 [current_design]",
            "set_property BITSTREAM.CONFIG.CONFIGRATE 66 [current_design]",
            "set_property BITSTREAM.GENERAL.COMPRESS TRUE [current_design]",
            "set_property CFGBVS VCCO [current_design]",
            "set_property CONFIG_VOLTAGE 3.3 [current_design]",
        ]

        self.toolchain.additional_commands = [
            # Non-Multiboot SPI-Flash bitstream generation.
            "write_cfgmem -force -format bin -interface spix4 -size 16 -loadbit \"up 0x0 {build_name}.bit\" -file {build_name}.bin",

            # Multiboot SPI-Flash Operational bitstream generation.
            "set_property BITSTREAM.CONFIG.TIMER_CFG 0x0001fbd0 [current_design]",
            "set_property BITSTREAM.CONFIG.CONFIGFALLBACK Enable [current_design]",
            "write_bitstream -force {build_name}_operational.bit ",
            "write_cfgmem -force -format bin -interface spix4 -size 16 -loadbit \"up 0x0 {build_name}_operational.bit\" -file {build_name}_operational.bin",

            # Multiboot SPI-Flash Fallback bitstream generation.
            "set_property BITSTREAM.CONFIG.NEXT_CONFIG_ADDR 0x00400000 [current_design]",
            "write_bitstream -force {build_name}_fallback.bit ",
            "write_cfgmem -force -format bin -interface spix4 -size 16 -loadbit \"up 0x0 {build_name}_fallback.bit\" -file {build_name}_fallback.bin"
        ]

    def create_programmer(self, name='openocd'):
        if name == 'openocd':
            return OpenOCD("openocd_xc7_ft232.cfg", "bscan_spi_xc7a200t.bit")
        elif name == 'vivado':
            # TODO: some board versions may have s25fl128s
            return VivadoProgrammer(flash_part='s25fl256sxxxxxx0-spi-x1_x2_x4')

    def do_finalize(self, fragment):
        XilinxPlatform.do_finalize(self, fragment)
        self.add_period_constraint(self.lookup_request("clk200", loose=True), 1e9/200e6)


# CRG ----------------------------------------------------------------------------------------------

class CRG(Module):
    def __init__(self, platform, sys_clk_freq):
        self.rst = Signal()

        self.clock_domains.cd_sys       = ClockDomain()
        self.clock_domains.cd_sys4x     = ClockDomain(reset_less=True)
        self.clock_domains.cd_sys4x_dqs = ClockDomain(reset_less=True)
        self.clock_domains.cd_idelay    = ClockDomain()

        # Clk/Rst
        clk200 = platform.request("clk200")

        # PLL
        self.submodules.pll = pll = S7PLL()
        self.comb += pll.reset.eq(self.rst)
        pll.register_clkin(clk200, 200e6)
        pll.create_clkout(self.cd_sys,       sys_clk_freq)
        pll.create_clkout(self.cd_sys4x,     4*sys_clk_freq)
        pll.create_clkout(self.cd_sys4x_dqs, 4*sys_clk_freq, phase=90)
        pll.create_clkout(self.cd_idelay,    200e6)

        self.submodules.idelayctrl = S7IDELAYCTRL(self.cd_idelay)

# PCIeSoC -----------------------------------------------------------------------------------------

class PCIeSoC(SoCCore):
    def __init__(self, sys_clk_freq = int(100e6), **kwargs):
        sys_clk_freq = int(100e6)

        platform = Platform()
        # SoCCore ----------------------------------------------------------------------------------
        SoCCore.__init__(self, platform, sys_clk_freq,
            ident          = "LiteX SoC on TE0712",
            **kwargs)

        # CRG --------------------------------------------------------------------------------------
        self.submodules.crg = CRG(platform, sys_clk_freq)

        # PCIe -------------------------------------------------------------------------------------
        # PHY
        self.submodules.pcie_phy = S7PCIEPHY(platform, platform.request("pcie_x1"),
            data_width = 64,
            bar0_size  = 0x20000)
        self.add_pcie(phy=self.pcie_phy, ndmas=1)
        # FIXME: Apply it to all targets (integrate it in LitePCIe?).
        platform.add_period_constraint(self.crg.cd_sys.clk, 1e9/sys_clk_freq)
        platform.toolchain.pre_placement_commands.add("set_clock_groups -group [get_clocks {sys_clk}] -group [get_clocks userclk1] -asynchronous", sys_clk=self.crg.cd_sys.clk)
        platform.toolchain.pre_placement_commands.add("set_clock_groups -group [get_clocks {sys_clk}] -group [get_clocks clk_125mhz] -asynchronous", sys_clk=self.crg.cd_sys.clk)
        platform.toolchain.pre_placement_commands.add("set_clock_groups -group [get_clocks {sys_clk}] -group [get_clocks clk_250mhz] -asynchronous", sys_clk=self.crg.cd_sys.clk)
        platform.toolchain.pre_placement_commands.add("set_clock_groups -group [get_clocks clk_125mhz] -group [get_clocks clk_250mhz] -asynchronous")
        

# Create simple LED module (fpga description)
class Blink(Module):
    def __init__(self, blink_freq, sys_clk_freq, led):
        counter = Signal(32)
        # synchronous assignments
        self.sync += [
            counter.eq(counter + 1),
            If(counter == int((sys_clk_freq/blink_freq)/2 - 1),
                counter.eq(0),
                led.eq(~led)
            )
        ]
        # combinatorial assignements
        self.comb += []

class LEDModule(Module):
    def __init__(self, platform):
        pass
        # submodules
        self.submodules += Blink(1, 100e6, platform.request("sys_led"))
        self.submodules += Blink(2, 100e6, platform.request("led2"))
                
# Build --------------------------------------------------------------------------------------------

def main():
    parser = argparse.ArgumentParser(description="LiteX SoC on TE0712")
    parser.add_argument("--build",  action="store_true", help="Build bitstream")
    parser.add_argument("--driver", action="store_true", help="Generate LitePCIe driver")
    parser.add_argument("--load",   action="store_true", help="Load bitstream")
    parser.add_argument("--flash",  action="store_true", help="Flash bitstream")
    parser.add_argument("--sys-clk-freq",    default=100e6,       help="System clock frequency.")

    builder_args(parser)
    soc_core_args(parser)
    args = parser.parse_args()

    soc      = PCIeSoC(
        sys_clk_freq = int(float(args.sys_clk_freq)), 
        **soc_core_argdict(args))
    builder  = Builder(soc, **builder_argdict(args))
    builder.build(run=args.build)
    
    if args.driver:
        generate_litepcie_software(soc, os.path.join(builder.output_dir, "driver"))

    if args.load:
        prog = soc.platform.create_programmer()
        prog.load_bitstream(os.path.join(builder.gateware_dir, soc.build_name + ".bit"))

    if args.flash:
        prog = soc.platform.create_programmer()
        prog.load_bitstream(os.path.join(builder.gateware_dir, soc.build_name + ".bin"))
        
    # platform = Platform()
    # module = LEDModule(platform)
    # platform.build(module)

if __name__ == "__main__":
    main()

I can build the module using ./example_test.py --uart-name=crossover --build --driver --cpu-type None and it builds and generates the bitstream without any major problem. However, when I program the FPGA after the CPU reset, the PCIe Phy never shows up during boot. Below is the stdout during boot. The lspci returns nothing.

[    3.599568] imx6q-pcie 33800000.pcie: 33800000.pcie supply epdev_on not found, using dummy regulator
[    3.609235] OF: PCI: host bridge /pcie@0x33800000 ranges:
[    3.614659] OF: PCI:   No bus range found for /pcie@0x33800000, using [bus 00-ff]
[    3.622155] OF: PCI:    IO 0x1ff80000..0x1ff8ffff -> 0x00000000
[    3.628086] OF: PCI:   MEM 0x18000000..0x1fefffff -> 0x18000000
[    3.634493] imx6q-pcie 33800000.pcie: Initialize PHY with EXT REfCLK!.
[    3.641335] imx6q-pcie 33800000.pcie: PHY Initialization End!.
[    3.647502] imx6q-pcie 33800000.pcie: pcie phy pll is locked.
[    3.898406] imx6q-pcie 33800000.pcie: phy link never came up
[    3.904082] imx6q-pcie 33800000.pcie: failed to initialize host
[    3.910008] imx6q-pcie 33800000.pcie: unable to add pcie port.
[    3.916162] imx6q-pcie: probe of 33800000.pcie failed with error -110
[    3.924816] input: gpio-keys as /devices/platform/gpio-keys/input/input1
[    3.932480] rtc-m41t80 3-0068: Oscillator failure, data is invalid.
[    3.938762] rtc-m41t80 3-0068: hctosys: unable to read the hardware clock

I know my PCIe lines are right as I use the same pin configurations in a separate Vivado project and I can successfully connect to the Linux CPU through PCIe. Below is the same output when PCIe is recognized correctly.

[    3.600366] imx6q-pcie 33800000.pcie: 33800000.pcie supply epdev_on not found, using dummy regulator
[    3.610221] OF: PCI: host bridge /pcie@0x33800000 ranges:
[    3.615671] OF: PCI:   No bus range found for /pcie@0x33800000, using [bus 00-ff]
[    3.623186] OF: PCI:    IO 0x1ff80000..0x1ff8ffff -> 0x00000000
[    3.629139] OF: PCI:   MEM 0x18000000..0x1fefffff -> 0x18000000
[    3.635846] imx6q-pcie 33800000.pcie: Initialize PHY with EXT REfCLK!.
[    3.642702] imx6q-pcie 33800000.pcie: PHY Initialization End!.
[    3.648878] imx6q-pcie 33800000.pcie: pcie phy pll is locked.
[    3.709523] imx6q-pcie 33800000.pcie: Link up, Gen2
[    3.715866] imx6q-pcie 33800000.pcie: PCI host bridge to bus 0000:00
[    3.722262] pci_bus 0000:00: root bus resource [bus 00-ff]
[    3.727778] pci_bus 0000:00: root bus resource [io  0x0000-0xffff]
[    3.733983] pci_bus 0000:00: root bus resource [mem 0x18000000-0x1fefffff]
[    3.741536] pci 0000:01:00.0: enabling Extended Tags
[    3.758650] pci 0000:00:00.0: BAR 0: assigned [mem 0x18000000-0x180fffff 64bit]
[    3.765979] pci 0000:00:00.0: BAR 14: assigned [mem 0x18100000-0x181fffff]
[    3.772866] pci 0000:00:00.0: BAR 6: assigned [mem 0x18200000-0x1820ffff pref]
[    3.780099] pci 0000:01:00.0: BAR 0: assigned [mem 0x18100000-0x1810ffff]
[    3.786911] pci 0000:00:00.0: PCI bridge to [bus 01-ff]
[    3.792146] pci 0000:00:00.0:   bridge window [mem 0x18100000-0x181fffff]
[    3.799363] pcieport 0000:00:00.0: Signaling PME with IRQ 226
[    3.805205] pcieport 0000:00:00.0: AER enabled with IRQ 226
[    3.811129] serial 0000:01:00.0: enabling device (0000 -> 0002)

I am using NXP i.MX8 processor on the Linux side. What am I missing here? I am completely new to LitePCIe.

DMA Loopback test fails on Acorn CLE-101

Built using:
$ ./sqrl_acorn.py --variant cle-101 --with-pcie --uart-name crossover --build

With Vivado 2021.2

$ ./litepcie_util dma_test
Unable to find DMA RX_DELAY (min errors: 2048/2048), exiting.

litepcie_util scratch register test and info both work.

DMA ADDR of the Device

Could anyone give me some intuition about the DMA, like the address of the device memory it is transferring from? I want to transfer data between the main memory of my development board(KC705) and the host PC memory, is it doable by litepcie?

DMA tlast and tkeep missing

Thanks in advance for you effort .
unfortunately there is no connection form pcie endpoint DMA to user area for tlast of packets and also tkeep
If I wanted to use 128 bit width data then there is chance that I read 5 64bit then the last one will be transfred in in 128 bit width field
but its tkeep will be zero (regrading to xilinx pcie endpoint doc ). as you know pcie transfer with the word width of 32bit (if i am not wrong) but xilinx endpoint in 128bit mode has to output 128 bit for that matter it uses tkeep to indicate the unregistered data .
but in your IP tkeep and tlast has been drop from TLP handler to DMA and there is no tkeep after dma and also no tlast

Non zero-copy dma_test errors after recent DMA refactor

After #65, non zero-copy dma_test occasionally results in errors:

+ ./user/litepcie_util dma_test
DMA_SPEED(Gbps) TX_BUFFERS      RX_BUFFERS      DIFF    ERRORS
          7.03       21569           21441         128  127116032
          7.02       43105           42977         128        0
          7.02       64641           64513         128        0
          7.03       86209           86081         128        0
          7.02      107745          107617         128  121705984
          7.02      129281          129153         128        0
          7.03      150849          150721         128        0
          7.02      172385          172257         128        0
          7.03      193953          193825         128  87809568
          7.02      215489          215361         128        0

FWIW, I tried with the various error checking PRs I have created here, and it isn't a read or ioctl that's silently failing.
cc @sergachev

Linux kernel space software does not compile

Hi,

I'm getting this error:

make -C /lib/modules/4.20.0-042000-generic/build M=/home/nanortemis/Tools/litex/litex/litepcie/litepcie/software/linux/kernel modules
make[1]: Entering directory '/usr/src/linux-headers-4.20.0-042000-generic'
  CC [M]  /home/nanortemis/Tools/litex/litex/litepcie/litepcie/software/linux/kernel/main.o
/home/nanortemis/Tools/litex/litex/litepcie/litepcie/software/linux/kernel/main.c: In function ‘litepcie_dma_wait’:
/home/nanortemis/Tools/litex/litex/litepcie/litepcie/software/linux/kernel/main.c:292:13: error: implicit declaration of function ‘signal_pending’; did you mean ‘timer_pending’? [-Werror=implicit-function-declaration]
         if (signal_pending(current)) {
             ^~~~~~~~~~~~~~
             timer_pending
cc1: some warnings being treated as errors
make[2]: *** [scripts/Makefile.build:292: /home/nanortemis/Tools/litex/litex/litepcie/litepcie/software/linux/kernel/main.o] Error 1
make[1]: *** [Makefile:1563: _module_/home/nanortemis/Tools/litex/litex/litepcie/litepcie/software/linux/kernel] Error 2
make[1]: Leaving directory '/usr/src/linux-headers-4.20.0-042000-generic'
make: *** [Makefile:12: litepcie.ko] Error 2

a quick solution can be found here: diederikdehaas/rtl8812AU#75 (comment)

will litepcie be helpful for developing PCIE VIP

For developing inhouse PCIE VIP, will this litepcie project be helpful as a start point? I assume the generated PCIE IP could be at least the BFM part, and on top we can build up SV UVM layers? Thanks,

Example Design

HI,

Can you please add some example design for Pcie artix 7 family, like led blink or bram. i want to learn how pcie core communicates with the application logic.

A LitePCIeDMA class question

Hi Sir,
By enjoy-digital/litex#983,

I realize that the Host is just advertised by separate DMAReader/DMAWriter IRQs of the progress/status on the FPGA.

Then, how does the host ensure that writes are done ahead of FPGA reads (DMA Reader, Host --> FPGA) and reads are done after FPGA writes (DMAWriter, FPGA --> Host) ?

Since the host seems to have no way to let FPGA to follow its write or wait for its read.

Please advise. Many thanks.
Jimmy

TLP Packetizer/Depacketizer: Create unit-tests and avoid code duplication

The TLP Packetizers/Depacketizers are using specific classes for the 64/128/256/512-bit datapaths. It should be possible to simplify the code by creating a generic Packetizer/Depacketizer that would support all the supported data widths. Before doing this, unit-tests for the current implementations should be created and used to validate the new generic module.

Linux user space software does not compile

Hi,

I'm getting this error:

gcc -c -O2 -Wall -g -I../kernel -MMD -o litepcie_util.o litepcie_util.c
litepcie_util.c: In function ‘dump_version’:
litepcie_util.c:211:46: error: ‘CSR_IDENTIFIER_SYSID_ADDR’ undeclared (first use in this function); did you mean ‘CSR_IDENTIFIER_MEM_BASE’?
     printf("sysid=0x%x\n", litepcie_readl(s, CSR_IDENTIFIER_SYSID_ADDR));
                                              ^~~~~~~~~~~~~~~~~~~~~~~~~
                                              CSR_IDENTIFIER_MEM_BASE
litepcie_util.c:211:46: note: each undeclared identifier is reported only once for each function it appears in
litepcie_util.c:212:48: error: ‘CSR_IDENTIFIER_FREQUENCY_ADDR’ undeclared (first use in this function); did you mean ‘CSR_IDENTIFIER_MEM_BASE’?
     printf("frequency=%d\n", litepcie_readl(s, CSR_IDENTIFIER_FREQUENCY_ADDR));
                                                ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~
                                                CSR_IDENTIFIER_MEM_BASE
make: *** [Makefile:17: litepcie_util.o] Error 1

CSR_IDENTIFIER_SYSID_ADDR and CSR_IDENTIFIER_FREQUENCY_ADDR are not defined.

Queries about litepcie

Hi!

While I was searching for a xilinx FPGA solution for PCI Express, I stumbled upon litepcie. Since there is very little documentation is available about the litepcie, I am writing my queries here. Other users are also welcomed to use this thread/issue to discuss their queries about litepcie.

As stated on litepcie page, that it supports several xilinx devices such as ultrascale plus, ultrascale, 7 series etc. This leaves room for a lot of uncertainty. Such as, many ultrascale plus devices have Ultrascale+ PCI Express Integrated Block which include ZCU106 Board(part:xczu7ev-ffvc1156-2) whereas ZCU102 Board(part:xczu9eg-ffvc1156-2) does not have any integrated PCIe block and it only uses the PCI PHY 1.0.

So my point is, there are different version of PHY in different devices of the same category(e.g. ultrascale+). Now I am confused that which one is supported by litepcie? My best guess is that litepcie is build only for Ultrascale+ PCI Express Integrated Block so it will work on ZCU106 board but it will not work on ZCU102 board because there is no Integrated Block there.

And also I want to know if there are any video/pdf tutorials about litepcie?

One more thing: Under the getting started heading at litepcie page, step 3 needs more description.
TODO: add/describe examples
What does this step 3 means?

Please clear the query.
Bilal

Possible roadmap for the project for PCIE Gen3?

Would it be possible for the lead developer to share a possible guideline doc for upgrading this library to support PCIE Gen3?

Interested in contributing to the development and I have started reading the source code as well, but it would be nice if the lead developer could share their insight.

Upto the developer though, if possible!

kernel build result in error

thanks in advance
due to incomplete description of struct type for liteuart.c build result in error
I had to modify the make file and remove liteuart to be built
there is a missing struct type called struct xa_limit limit;
and also macro XA_LIMIT

Interrupt vector conflict for IRQ 0

Hi,
I tried to run litepcie following the Readme step by step, on a Fedora 25 machine.
After recursively clone litex (and litedram), the make.py shows this error:

./make.py all Traceback (most recent call last): File "./make.py", line 92, in <module> soc = top_class(platform, **top_kwargs) File "/home/dh/Documents/litepcie/example_designs/targets/dma.py", line 61, in __init__ with_timer=False File "/usr/lib/python3.5/site-packages/litex-0.1-py3.5.egg/litex/soc/integration/soc_core.py", line 169, in __init__ interrupt, mod_name, interrupt_rmap[interrupt])) AssertionError: Interrupt vector conflict for IRQ 0, user defined nmi conflicts with SoC inbuilt dma_writer

I'm using default values of litepcie/example_design/targets ... is really the soc_core having conflicts with DMA example?

Allow MMAP access to BAR0 without loading kernel driver

Once the kernel driver is loaded, it's possible to access BAR0 to access the wishbone bridge of the SoC to access registers or create a virtual console:

# Create bridge:
# --------------
# sudo litex_server --pcie --pcie-bar=/sys/bus/pci/devices/0000\:01\:00.0/resource0

# Use scripts:
# -------------
# ./test_regs.py
# ./litescope_wishbone.py (--help to see preconfigured triggers)

# Use console:
# litex_crossover_uart --base-address=-0x82000000 (will create /dev/pts/X)
# litex_term /dev/ptsX

If possible, we should allow this without loading the kernel driver and understand what needs to be enabled.

Pre-built bitstream for NeTV2 with PCIe core?

Hi @enjoy-digital , @cgmAether opened an issue on the Alphamax repo (AlphamaxMedia/netv2-fpga#26) with a simple request that maybe you might have a quick resolution for.

@cgmAether is looking for any bitstream that loads onto the NeTV2 and allows it to enumerate on the PCIe bus:

I don't even care if the NetV2 successfully writes or reads from its memory, although that is preferred, as long as it completes the transactions and does not hang the PCIe Bus. I'm simply looking for a NetV2 bitstream which acts as a PCIe stub.

I don't know the details but it seems the goal is to do some characterization of EMI interference of PCIe, so a minimally functional link is all that's needed.

Do you happen to have a bitstream handy that might fit the bill? Not asking you to compile or write anything, just asking if you have an old build hanging out that might have a pre-built bitstream that could do the trick for this use case. Thanks and sorry for the bother!

litecpie dma test result in error

thanks in advance
after building both fpga and kernel I tested the dma which resulted in error
it will show error in dma test if I use it without any change to configs
if I change its width(lite_util config (its 16bit width by default )) to 32 bit its gets better but still encounter error
or if I change it to zero copy it gets resolved
and also I have tested the old version for last year which had no zero copy
in that version also there was moment which in dma test it outputs too many errors .

Segmentation fault when running litepcie_util dma_loopback_test

Hi,

I'm getting a segmentation fault when running litepcie_util dma_loopback_test.
I'm trying to run this test on the Xilinx KC705 eval board.

lspci output

01:00.0 Memory controller: Xilinx Corporation Device 7022 (rev 01)
        Subsystem: Xilinx Corporation Device 0007
        Control: I/O+ Mem+ BusMaster+ SpecCycle- MemWINV- VGASnoop- ParErr- Stepping- SERR- FastB2B- DisINTx-
        Status: Cap+ 66MHz- UDF- FastB2B- ParErr- DEVSEL=fast >TAbort- <TAbort- <MAbort- >SERR- <PERR- INTx-
        Latency: 0, Cache Line Size: 64 bytes
        Region 0: Memory at f7c00000 (32-bit, non-prefetchable) [size=1M]
        Capabilities: [40] Power Management version 3
                Flags: PMEClk- DSI- D1- D2- AuxCurrent=0mA PME(D0+,D1+,D2+,D3hot+,D3cold-)
                Status: D0 NoSoftRst+ PME-Enable- DSel=0 DScale=0 PME-
        Capabilities: [48] MSI: Enable- Count=1/1 Maskable- 64bit-
                Address: 00000000  Data: 0000
        Capabilities: [60] Express (v2) Endpoint, MSI 00
                DevCap: MaxPayload 512 bytes, PhantFunc 0, Latency L0s <64ns, L1 unlimited
                        ExtTag- AttnBtn- AttnInd- PwrInd- RBE+ FLReset- SlotPowerLimit 75.000W
                DevCtl: Report errors: Correctable- Non-Fatal- Fatal- Unsupported-
                        RlxdOrd- ExtTag- PhantFunc- AuxPwr- NoSnoop+
                        MaxPayload 128 bytes, MaxReadReq 512 bytes
                DevSta: CorrErr- UncorrErr- FatalErr- UnsuppReq- AuxPwr- TransPend-
                LnkCap: Port #0, Speed 5GT/s, Width x2, ASPM L0s, Exit Latency L0s unlimited, L1 unlimited
                        ClockPM- Surprise- LLActRep- BwNot- ASPMOptComp-
                LnkCtl: ASPM Disabled; RCB 64 bytes Disabled- CommClk-
                        ExtSynch- ClockPM- AutWidDis- BWInt- AutBWInt-
                LnkSta: Speed 5GT/s, Width x1, TrErr- Train- SlotClk- DLActive- BWMgmt- ABWMgmt-
                DevCap2: Completion Timeout: Range B, TimeoutDis-, LTR-, OBFF Not Supported
                DevCtl2: Completion Timeout: 50us to 50ms, TimeoutDis-, LTR-, OBFF Disabled
                LnkCtl2: Target Link Speed: 5GT/s, EnterCompliance- SpeedDis-
                         Transmit Margin: Normal Operating Range, EnterModifiedCompliance- ComplianceSOS-
                         Compliance De-emphasis: -6dB
                LnkSta2: Current De-emphasis Level: -6dB, EqualizationComplete-, EqualizationPhase1-
                         EqualizationPhase2-, EqualizationPhase3-, LinkEqualizationRequest-
        Capabilities: [100 v1] Device Serial Number 00-00-00-01-01-00-0a-35

dmesg output after calling ./init.sh

[  514.864422] litepcie: loading out-of-tree module taints kernel.
[  514.864484] litepcie: module verification failed: signature and/or required key missing - tainting kernel
[  514.864868] litepcie Probing device
[  514.865949] litepcie Assigned to minor 0

output after calling ./litepcie_util dma_loopback_test

./litepcie_util dma_loopback_test
Segmentation fault (core dumped)

dmesg

[  674.303697] show_signal_msg: 35 callbacks suppressed
[  674.303699] litepcie_util[28008]: segfault at 7fa8a4c89000 ip 00005640ca618450 sp 00007ffdaffffa00 error 6 in litepcie_util[5640ca618000+1000]
[  674.303705] Code: 01 44 24 14 c1 ee 02 48 63 c9 74 33 69 54 24 0c 76 59 41 31 44 8d 46 ff 48 8d 04 0f 4a 8d 7c 87 04 48 01 f9 83 c2 01 0f 1f 00 <89> 10 48 83 c0 04 81 c2 76 59 41 31 48 39 c1 75 ef 01 74 24 0c 8b

License on the PCIE Verilog files?

So in the source code, the Verilog for PCIE is present. The license says it contains confidential Xilinx property. Now my question, does that still allow the user to download and use the litepcie library?

Should there be a mention of this in the License? Sorry if the question is trivial, I am still learning the fine points around here.

PCIe issues on ADRV2CRR-FMC

Issue

Trying to bring up PCIe (gen3 4x and gen3 8x) on this board yielded some unexpected issues and it took some time to find a sequence that works.

I'm documenting here the observations, the theory about what I think the problems are and workarounds.

Test Setup

First description of the setup :

  • ADRV2CRR-FMC carrier with ADRV9009-ZU11EG plugged in
  • Asus H510-K motherboard with Intel i3-10105
    • PCIe1 is the 16x slot which is connected directly to the CPU
    • PCIe3 is the 1x slot which comes from the H510 chipset
  • The "PCIe extender" I refer to in some tests below uses a USB3 A-A cable abused to transport PCIe signals from a 1x pcie->usb3 stub inside the computer to an external PCIe 16x slot (mechanical 16x, electrically it's 1x).
  • The "external PCIe switch" I also refer to in some tests below is based on PI7C9X2G404. It has three 1x downstream slots and is connected via the same method as the extender above.

Initial Observation ( Feb 22 )

  • Card plugged in mobo PCIe1 (16x)

    • Only detected if bitstream is loaded after system is booted and trigger rescan
    • Not detected if booted with bitstream loaded
    • Doesn't actual work ( trying to show ID report \xff.... )
    • Link speed reported as 2.5GT/s (gen1)
    • Link width is correct
  • Card plugged in mobo PCIe3 (1x) via a 'usb cable extender'

    • Needs to limit link speed to gen2. In gen3, tons of errors are reported.
    • Card detected fine and ID report is correct, dma_test works -> It works !
    • Link speeds reported as 5GT/s (expected, limited on purpose in bios)
    • Link width is 1x (expected ... that slot it 1x only)
  • Card plugged in an external PCIe switch, the PCIe switch connected to mobo PCIe3

    • PCIe switch shows up correcty with expected link speed
    • Card behavior is the same as when it's plugged in PCIe1 directly (the mobo x16 slot): i.e. not working
    • (note: The PCIe presence jumper needs to be set to 1x or the pcie switch doesn't see the card)
  • Card plugged in an external PCIe switch, the PCIe switch connected to mobo PCIe1

    • Same behavior as if switch plugged in PCIe1
  • Card plugged in mobo PCIe1 (16x) via a 'usb cable extender' (limiting to 1x)

    • Same behavior as direct connection expect link width is 1x

Theories about problems ( Feb 23 )

Following more testing the next day, I think there are several problems and that's why the symptoms are weird and the different cases results make little sense.

  • If the card isn't detected at boot, the bios seems to not bother to configure the PCIe root port. So I have to manually write the LinkControl register in linux to set it up properly to get gen3 support and get it to train properly.

  • When doing a pcie rescan, even if it detects the device ... linux is dumb as a brick and will not configure the memory zone through the upstream switches/root ports, they remain [disabled]. Solution for that is to do a delete of the pcie root port where the card is plugged, and then do a rescan. When linux will re-add the bridge, it will then properly configure it for the downstream devices.

  • For some reason, once the PCIe core has trained once ... it cannot go through a reboot cycle, that will lock it up. So if I configure it before the machine is started, then boot, it's fine. Or if I boot with the card unconfigured and do it all once in linux, that works too. But if I get the card up and try a warm boot, it will never be seen ever again.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.