tcp-acceleration-service / tas Goto Github PK

I have built dpdk and I can see the ports with testpmd. The documentations tells us to use, vfio-pci for intel cards. With Mellanox cards I get a bind error for vfio-pci. I tried to bind it to igb_uio, TAS is not able to find the Ethernet cards. I get the following error.

No ethernet devices
network init failed

Sometimes I get the following error. I searched the repo for this output string and I found the place it occurs. However what would be the main reason on why this happens?

util_create_shmsiszed: mmap failed: Cannot allocate memory
mapping flexnic dma memory failed

Note:
OFED Version: MLNX_OFED_LINUX-5.0-2.1.8.0
DPDK Version:19.11.2
Thanks
Vineeth

The following assertion inside the uxsocket_receive function fails

assert(cmsg->cmsg_len == CMSG_LEN(sizeof(int)));

cmsg->cmsg_len is set to 8, whereas CMSG_LEN(sizeof(int)) returns 20.
Any idea as to why this may be happening?

Causes TX traffic to be forwarded unnecessarily between cores: https://github.com/tcp-acceleration-service/tas/blob/master/tas/fast/fast_flows.c#L117

Expected Behavior

TAS should send/receive packets without waiting indefinitely

Current Behavior

TAS sometimes fails to send (or receive) data sent by the last call to send(). This causes the receiver to wait indefinitely even after the sender has stopped.

Steps to Reproduce

The bug is non-deterministic and may happen at the server or the client, however it can be reproduced fairly reliably using the following server and client in this repo

1. Compile using -Ofast and -march=native
2. Run TAS on both the server and client
3. Server:

   LD_PRELOAD=<path-to-libtas_interpose.so>  ./server <server-ip> <server-port

4. Client:

   LD_PRELOAD=<path-to-libtas_interpose.so>  ./client<server-ip> <server-port

Context (Environment)

The bug was discovered when this software RDMA stack was attempted over TAS on two machines equipped with the 10G Intel 82599 NICs. All performance benchmarks get blocked indefinitely on TAS. They run fine on the regular kernel TCP stack.

Is it possible to add the benchmarks into the repo as well? Thanks

I am not able to run iperf on top of TAS. When ever I try and run iperf on top of TAS, the program stalls and there is no output on the screen. Although, I am able to run the benchmarking repo code just fine.

Hey,

Are there any documentations or scripts on how to reproduce the experiments in the paper?

Thanks,
Alireza

Hello,

I would like to use 1G huge pages instead of using 2M huge pages that are described in the documentation.

There is a function called shm_preinit which tries to grab some pages before DPDK does anything. I'm wondering why it doesn't work with 1G page size, however, it works properly with the 2M page size.

I run this command sudo tas/tas --ip-addr=10.0.0.1/24 --fp-cores-max=1 and get the following error:

util_create_shmsiszed: ftruncate failed: Invalid argument
mapping flexnic internal memory failed

Has it designed to work with the 2M page sizes? It doesn't make sense to me though. Is there anything I'm missing?

Thanks,
Alireza

every time test flexkvs, failed at starting
server will close connection right now when recv, maybe the newest test code not pushed to github
when server recv data from connection, the max msg size is 128, but for request body, it need sizeof(*req) + ntohl(req->request.bodylen) at least

static inline int conn_recv(ssctx_t sc, int epfd, struct connection *c)
{
    ssize_t res;

    /* read until request is complete (or no more data) */
    do {
        res = ss_read(sc, c->fd, c->rx_buf + c->rx_len, MAX_MSGSIZE -
                c->rx_len);
        if (res == 0) {
            fprintf(stderr, "Closing connection on EOF error: %d count: %d\n", errno, MAX_MSGSIZE - c->rx_len);
            conn_close(sc, c);
            return -1;
        } else if (res < 0 && errno == EAGAIN) {
            /* request is not complete yet */
            conn_upepoll(sc, epfd, c, 0);
            return 1;
        } else if (res < 0) {
            perror("Closing connection on read error");
            conn_close(sc, c);
            return -1;
        }

        c->rx_len += res;
    } while (!conn_req_iscomp(c));

    return 0;
}

static inline int conn_req_iscomp(struct connection *c)
{
    protocol_binary_request_header *req =
            (protocol_binary_request_header *) (c->rx_buf + c->rx_off);
    uint16_t avail = c->rx_len - c->rx_off;
    if (avail < sizeof(*req)) {
        return 0;
    }

    printf("---Function: %s ---line: %d---avail: %d bodylen: %d\n", __FUNCTION__, __LINE__,avail, ntohl(req->request.bodylen));
    if (avail < sizeof(*req) + ntohl(req->request.bodylen)) {
        return 0;
    }
    return 1;
}

when start tas with multi queue virtio_net, it start failed
[root@localhost tas]# ./tas --ip-addr=192.168.122.123/24 --fp-no-xsumoffload
Warning: NIC does not support all requested RSS hash functions.
virtio_dev_configure(): Unsupported Rx multi queue mode 1
Port0 dev_configure = -22
rte_eth_dev_configure failed
network init failed

looks like the default port_conf->rxmode not change to ETH_MQ_RX_NONE when using virtio_net

but virtio_dev_configure() check failed if it's not ETH_MQ_RX_NONE

static int
virtio_dev_configure(struct rte_eth_dev *dev)
{
......
	const struct rte_eth_rxmode *rxmode = &dev->data->dev_conf.rxmode;
	const struct rte_eth_txmode *txmode = &dev->data->dev_conf.txmode;
	struct virtio_hw *hw = dev->data->dev_private;
	uint32_t ether_hdr_len = RTE_ETHER_HDR_LEN + VLAN_TAG_LEN +
		hw->vtnet_hdr_size;
	uint64_t rx_offloads = rxmode->offloads;
	uint64_t tx_offloads = txmode->offloads;
	uint64_t req_features;
	int ret;

	PMD_INIT_LOG(DEBUG, "configure");
	req_features = VIRTIO_PMD_DEFAULT_GUEST_FEATURES;

	if (rxmode->mq_mode != ETH_MQ_RX_NONE) {
		PMD_DRV_LOG(ERR,
			"Unsupported Rx multi queue mode %d",
			rxmode->mq_mode);
		return -EINVAL;
	}

	if (txmode->mq_mode != ETH_MQ_TX_NONE) {
		PMD_DRV_LOG(ERR,
			"Unsupported Tx multi queue mode %d",
			txmode->mq_mode);
		return -EINVAL;
	}
......

BTW, my dpdk is 19.11

commit b9b10ddb4292f2bc5524ae9f427a2795514eff02 (HEAD -> v19.11, origin/main, origin/HEAD, main)
Author: Thomas Monjalon <[email protected]>
Date:   Wed Aug 12 11:15:30 2020 +0200

    switch default git branch name to main

    The default git branch of the main DPDK repository has been renamed
    from master to main.

    Signed-off-by: Thomas Monjalon <[email protected]>
    Acked-by: Bruce Richardson <[email protected]>

commit da816fb1ff3435eb6f39e62ee4765930ef0a160e
Author: Thomas Monjalon <[email protected]>
Date:   Wed Aug 12 09:49:00 2020 +0200

    maintainers: remove QoS and pipeline repositories

    The git trees dpdk-next-qos and dpdk-next-pipeline were created
    to share the load of patches merging.
    It has been decided in the Technical Board that the load is not big
    enough to justify keeping these repositories.

    The patches for ethdev TM and MTR will be managed in dpdk-next-net.
    The sched and meter libraries will be managed in the main tree.
    The packet framework will be managed in the main tree as well.

    Signed-off-by: Thomas Monjalon <[email protected]>
    Acked-by: Bruce Richardson <[email protected]>

commit 3eecaba90a995ec30b502cded921c8a49e50cffb
Author: Ray Kinsella <[email protected]>
Date:   Wed Aug 12 12:26:02 2020 +0100

    doc: describe process for new ABI versions

    Added a section describing new ABI versions, this provides pointers to
    the relevant amended rules that apply during the abi breakage window.
    Also remove the large note at the head of the ABI policy describing the
    ABI stability process that has taken place over the previous year.

    Signed-off-by: Ray Kinsella <[email protected]>

commit 460cb707911e8209efecd52bd7748648f6754c4c
Author: Adam Dybkowski <[email protected]>
Date:   Mon Jul 27 11:41:13 2020 +0200

    doc: announce renaming in crypto scheduler API

    This patch adds a deprecation notice about upcoming changes
    in public API of the Scheduler PMD.

    Signed-off-by: Adam Dybkowski <[email protected]>
    Acked-by: Fan Zhang <[email protected]>
    Acked-by: Thomas Monjalon <[email protected]>
    Acked-by: Akhil Goyal <[email protected]>

We have a client connecting to a TAS server. The application send a bunch of SYN packets to the TAS server, but TAS never responds with a SYN-ACK, and hence the connection never succeeds.

I am very interested in TAS. Are there any data centers using TAS?

Looks like nginx (and possibly other applications) call listen on a socket more than once. Not exactly sure why, or what the sockets spec says should happen, but clearly this works on Linux so we should see if we can also handle this reasonably too.

Hi Antoine @FreakyPenguin

Here is my output on TAS.

I wonder what drops mean on the first line. It seems the number of drop on the fast path is constantly zero kdrops (the second line). Looking at the code, the first line is reported by the slow path. I wonder what those numbers mean. Does it mean that I have packet loss? If yes, why is Fast Path not reporting anything?

My latency also is not really high given this high value of drop rate reported by the slow path. I wonder if I am interpreting drops there mistakenly.

Thanks,
Alireza