ix-project / ix Goto Github PK

View Code? Open in Web Editor NEW

126.0 19.0 39.0 469 KB

IX a protected dataplane operating system project

Home Page: http://ix-project.github.io/ix/

License: Other

Makefile 1.23% C 95.41% Python 1.60% C++ 1.77%

ix's Introduction

Overview

IX a protected dataplane operating system project from Stanford and EPFL. It provides event-driven applications with:

low-latency (including at the tail)
high-throughput
efficient use of resources (to support workload consolidation and energy proportionality)

IX is licensed under an MIT-style license. See LICENSE.

Requirements

IX requires Intel DPDK and a supported Intel NIC:

Intel 82599
Intel X520
Intel X540

For more details, check the requirements page

Setup Instructions

There is currently no binary distribution of IX. You will therefore have to compile it from source. Additionally, you will need to fetch and compile the source dependencies:

fetch the dependencies:
```
git submodule update --init
```

build the dependecies:

sudo chmod +r /boot/System.map-`uname -r`
make -sj64 -C deps/dune
make -sj64 -C deps/pcidma
make -sj64 -C deps/dpdk config T=x86_64-native-linuxapp-gcc
make -sj64 -C deps/dpdk

build IX:

sudo apt-get install libconfig-dev libnuma-dev
make -sj64

The resulting executable files are cp/ixcp.py for the IX control plane and dp/ix for the IX dataplane kernel.

Set up the environment:

cp ix.conf.sample ix.conf
# modify at least host_addr, gateway_addr, devices, and cpu
sudo sh -c 'for i in /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages; do echo 4096 > $i; done'
sudo insmod deps/dune/kern/dune.ko
sudo insmod deps/pcidma/pcidma.ko
sudo modprobe uio
sudo insmod deps/dpdk/build/kmod/igb_uio.ko
sudo ifdown INTERFACE
sudo deps/dpdk/tools/dpdk_nic_bind.py -b igb_uio PCI_ADDRESS

run the IX TCP echo server and check that it works. Make sure that at least the "Device" field in ix.conf matches your network card PCI bus address (try lspci | grep -i eth and look for the virtual adapter):

sudo ./dp/ix -- ./apps/echoserver 4

Then, try from another Linux host:

echo 123 | nc -vv <IP> <PORT>

You should see the following output:

Connection to <IP> <PORT> port [tcp/*] succeeded!
123

ix's People

Contributors

Stargazers

Watchers

ix's Issues

Question about the location of LWIP (userspace or kernelspace)

Hello, I have read your paper about IX - Data plane operating system
and was wondering whether IX project uses TCP stack in userspace or kernel space.

IX won't TX

Dear IX developers,

I'd like to reproduce experiments from IX and Zygos paper in my own environment, but I've encountered following issues.
Here are some details about the environment and problems.

General environment
CPU: Intel Xeon E5-2690
RAM: 120GB
Kernel: 4.4.0-104-generic
DPDK version: 16.04 (installed following the guides)
10GbE NIC: Intel 82599ES
40GbE NIC: Intel XL710
10GbE Multi-port TX

Only the first NIC listed in the configuration can RX / TX packets.
Rest of the NICs can RX packets, but it cannot TX packets out to wire.

40GbE TX

40GbE NIC can RX packets, but it won't TX packets out to wire

Regards,
Ilwoo

How to make the `echolient` app multi-threading?

How to make the echolient app multi-threading?

We tried

	for (i = 0; i < nthreads; i++) {
		if (pthread_create(&tid, NULL, pp_main, NULL)) {
			fprintf(stderr, "failed to spawn thread %d\n", i);
			exit(-1);
		}
	}

	pp_main(NULL);

where pp_main consists of the original init process in main. The app reports:

echoclient: mempool.c:169: mempool_create_datastore: Assertion `mds->magic == 0' failed.
fatal exception 13, code 0 - dying...
dune: --- Begin Trap Dump ---
....

Here is the output of dmesg:

[23626.981318] vmx: created VCPU (VPID 1)
[23627.685779] pcidma 0000:06:00.0: claimed
[23628.022886] vmx: enabled EPT A/D bits
[23628.022909] vmx: created VCPU (VPID 2)
[23628.047588] vmx: stopping VCPU (VPID 2)
[23637.006998] vmx: stopping VCPU (VPID 1)
[23637.013837] pcidma 0000:06:00.0: released
[23637.013957] vmx: destroying VCPU (VPID 2)
[23637.018546] vmx: destroying VCPU (VPID 1)

IX crashed on startup

I have been trying to run ix these days but IX just crashed on startup.

Here is the crash output:

CPU 00| 00:49:26 <4>: init: barrier after al CPU initialization
CPU 00| 00:49:26 <4>: init: module hw SUCESS
CPU 00| 00:49:26 <5>: sandbox: env = 'LC_TIME=zh_CN.UTF-8'
CPU 00| 00:49:26 <4>: sandbox: loading segment - va 0x000000000, len 25150
CPU 00| 00:49:26 <4>: sandbox: loading segment - va 0x000225bc0, len 15a8
CPU 00| 00:49:26 <5>: sandbox: entry addr is 6f000cc0
CPU 00| 00:49:26 <5>: sandbox: umm_mmap: addr:7eff801000, len:7ff000, end:7f00000000
dune: exit due to EPT violation

I have tried different system and machines, but all of them didn't work. Mainly I use Ubuntu 16.04 or Ubuntu 15.10 with released kernel or 4.4.x from kernel.org.

I try to trace and figure out the reason of this bug. But it makes things more obscure.
here is the dmsg:

I notice the crash occurs at setup_argument function at sandbox/main.c. When access arg_ptr's content, the vmx_do_ept_fault is invoked. But I think this ept fault should happen when data_ptr is accessed earlier and it points to the same page as art_prt do.

Thank you very much.

Ethernet device initialization error

After commit 6ba626c I get the following error:

ix: drivers/common.c:194: driver_init: Assertion 'port < rte_eth_dev_count()' failed. Aborted

where rte_eth_dev_count is 0.

Make eth_rx_idle_wait driver independent

eth_rx_idle_wait should be driver independent and accessed through an entry of ix_eth_dev_ops
https://github.com/ix-project/ix/blob/master/dp/drivers/ixgbe.c#L379

issue PR: #11

Minor documentation inconsistencies

Hi! The top-level README.md file says to use git submodule update --init to fetch the project's dependencies, whereas (at least) the Requirements and Getting-started wiki pages point to using the deps/fetch-deps.sh script (which is not on the repo any more).

How to increase connection concurrency?

In the paper of IX, it is said that the ix can support up to 250,000 concurrent connections. But I failed to set up more than 64511 connections, and found that the later SYN packets are not responded. It doesn't work to modify the parameter pp_conn_pool_entries in echoserver.c or increase working cores.
So I wonder how can I increase the max connection concurrency?

Cannot compile IX

I successfully compiled all IX dependencies (with minor modifications to DPDK to compile with gcc 6), but I get an error when linking the IX binary:

user@machine:~/ix$ make -j64
make -C dp
make[1]: Entering directory '/home/user/ix/dp'
gcc -T ix.ld -o ix core/ethdev.o core/ethfg.o core/ethqueue.o core/cfg.o core/control_plane.o core/cpu.o core/init.o core/log.o core/mbuf.o core/mem.o core/mempool.o core/page.o core/pci.o core/utimer.o core/syscall.o core/timer.o core/vm.o core/dpdk.o drivers/ixgbe.o drivers/i40e.o drivers/common.o lwip/inet_chksum.o lwip/ip4_addr.o lwip/memp_min.o lwip/misc.o lwip/pbuf.o net/arp.o net/dump.o net/icmp.o net/ip.o net/net.o net/tcp.o net/tcp_in.o net/tcp_out.o net/tcp_api.o net/udp.o sandbox/main.o sandbox/umm.o sandbox/trap.o sandbox/exec.o ../deps/dune/libdune/libdune.a -Wl,-whole-archive ../deps/dpdk/build/lib/librte_pmd_ixgbe.a -Wl,-no-whole-archive -Wl,-whole-archive ../deps/dpdk/build/lib/librte_pmd_i40e.a -Wl,-no-whole-archive ../deps/dpdk/build/lib/libethdev.a ../deps/dpdk/build/lib/librte_mbuf.a ../deps/dpdk/build/lib/librte_eal.a ../deps/dpdk/build/lib/librte_mempool.a ../deps/dpdk/build/lib/librte_ring.a -lrt -lpthread -lm -lnuma -ldl -lconfig
/usr/bin/ld: ../deps/dune/libdune/libdune.a(entry.o): relocation R_X86_64_32S against .text' can not be used when making a shared object; recompile with -fPIC /usr/bin/ld: ../deps/dune/libdune/libdune.a(dune.o): relocation R_X86_64_32S against symbol dune_syscall_handler' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: Nonrepresentable section on output
collect2: error: ld returned 1 exit status
Makefile:83: recipe for target 'ix' failed
make[1]: *** [ix] Error 1
make[1]: Leaving directory '/home/user/ix/dp'
Makefile:30: recipe for target 'dp' failed
make: *** [dp] Error 2

I recompiled each object in libdune.a with "-fPIC" added to CFLAGS (and also created a similar rule for the .S extension), but the result is still the same. I believe this may be related to an upgrade to gcc 6.2, since I recall compiling it in Ubuntu 16.04 with gcc 5 and did not have this issue. I am using kernel version 4.4.0-51-generic.

Any ideas?

Can not allocate a guest-physically contiguous set of 2MB pages

I want to run IX on SLES 11 sp1 whose kernel version is 2.6.32, and I have modified some kernel code to support Dune and DPDK.
When I start the IX (v1.0), I got this error "page: failed to get machine address for 0x4000000000", the detail is in below:

03:10:11 <4>: init: starting IX
03:10:11 <4>: init: cpu phase
03:10:11 <4>: cpu: detected 12 cores
03:10:11 <4>: init: module CPU SUCESS
03:10:11 <4>: init: module Dune SUCESS
03:10:11 <4>: timer: detected 2399 ticks per US
03:10:11 <4>: init: module timer SUCESS
mempool_datastore: pending_pkt pages: 1 elem_len: 56 nostraddle:0 chunk_size:128 num_chunks:4292
mempool_datastore: arp pages: 3 elem_len: 80 nostraddle:0 chunk_size:128 num_chunks:4614
03:10:11 <4>: init: module net SUCESS
03:10:11 <4>: using config :'./ix.conf'
03:10:11 <4>: no static arp entries defined in config
03:10:11 <4>: net: using the following configuration:
03:10:11 <4>: host IP: 172.18.11.234
03:10:11 <4>: broadcast IP: 172.18.255.255
03:10:11 <4>: gateway IP: 192.168.0.254
03:10:11 <4>: subnet mask: 255.255.0.0
03:10:11 <4>: init: module cfg SUCESS
03:10:11 <4>: init: module cp SUCESS
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 1 on socket 0
EAL: Detected lcore 2 as core 2 on socket 0
EAL: Detected lcore 3 as core 8 on socket 0
EAL: Detected lcore 4 as core 9 on socket 0
EAL: Detected lcore 5 as core 10 on socket 0
EAL: Detected lcore 6 as core 0 on socket 1
EAL: Detected lcore 7 as core 1 on socket 1
EAL: Detected lcore 8 as core 2 on socket 1
EAL: Detected lcore 9 as core 8 on socket 1
EAL: Detected lcore 10 as core 9 on socket 1
EAL: Detected lcore 11 as core 10 on socket 1
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: WARNING: TSC frequency estimated roughly - clock timings may be less accurate.
EAL: TSC frequency is ~2400128 KHz
EAL: Master lcore 0 is ready (tid=f523da40;cpuset=[0])
EAL: lcore 3 is ready (tid=e8437710;cpuset=[3])
EAL: lcore 4 is ready (tid=e7c36710;cpuset=[4])
EAL: lcore 7 is ready (tid=e6433710;cpuset=[7])
EAL: lcore 2 is ready (tid=e8c38710;cpuset=[2])
EAL: lcore 1 is ready (tid=e9439710;cpuset=[1])
EAL: lcore 10 is ready (tid=e4c30710;cpuset=[10])
EAL: lcore 5 is ready (tid=e7435710;cpuset=[5])
EAL: lcore 6 is ready (tid=e6c34710;cpuset=[6])
EAL: lcore 11 is ready (tid=e442f710;cpuset=[11])
EAL: lcore 9 is ready (tid=e5431710;cpuset=[9])
EAL: lcore 8 is ready (tid=e5c32710;cpuset=[8])
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
03:10:15 <4>: init: module dpdk SUCESS
03:10:15 <4>: sched_setaffinity ret: 0
03:10:15 <4>: cpu_init_percpu pcpu: -10485760
03:10:15 <4>: dune_enter_ex
CPU 00| 20:22:24 <4>: cpu: started core 0, numa node 0
CPU 00| 08:00:00 <4>: init: module firstcpu SUCESS
CPU 00| 14:28:15 <2>: page: failed to get machine address for 0x4000000000
ix: core/mempool.c:217: mempool_create_datastore: Assertion `mds->buf' failed.
Aborted

I find this error happens at page_alloc_contig_on_node in dp/core/page.c.

Any ideas? Thanks!

"ixev: ran out of receive memory" when testing the echoserver application

Hi,

I successfully installed and ran IX on Debian jessie with the default 3.16.36 kernel. The echoserver application seems to work fine.

However, when I tried to load the echoserver using nc 192.168.40.1 1234 < /dev/zero > /dev/null, the echoserver immediately becomes unresponsive (after sending a bunch of TCP responses, it stops responding to any TCP or ICMP queries) and prints this in its log:

# ./dp/ix -- ./apps/echoserver 4
...
ixev: ran out of receive memory
CPU 00| 15:55:10 <2>: init: failed to start sandbox

I then have to kill the process (^C) and launch it again.

I used the installation and configuration method from the README, on a Intel 82599ES NIC.

I tried to increase the value in /sys/devices/system/node/node*/hugepages/hugepages-2048kB/nr_hugepages to 16000, but that did not seem to have an effect (except that it reserved 64 GB of RAM on the system)

support for mlx4 driver?

I am interested in this because I use the m510 machines on CloudLab (http://docs.cloudlab.us/hardware.html#%28part._cloudlab-utah%29) regularly in research and these machines have Mellanox ConnectX-3 10Gb Ethernet cards.

I would also like to know more about the efforts required to support a new type of NIC. For example, I notice that the sizes of dp/drivers/{ixgbe,i40e}.c are pretty small.

Thanks!

Can I deploy more than one IX instance on one machine?

Hi
As the title, can I deploy more than one IX instance on one machine? If do, how to deploy it?

driver init problem

When I run the IX TCP echo server, this problem happened.

ubuntu@ubuntu:~/ix$ sudo ./dp/ix -- ./apps/echoserver 4
06:47:00 <4>: init: starting IX
06:47:00 <4>: init: cpu phase
06:47:00 <4>: init: module Dune SUCESS
06:47:00 <4>: cpu: detected 2 cores
mempool_datastore: runners pages: 1 elem_len: 24 nostraddle:0 chunk_size:128 num_chunks:4682 [0x3fffe00000-0x4000000000]
06:47:00 <4>: init: module CPU SUCESS
06:47:00 <4>: timer: detected 2711 ticks per US
06:47:00 <4>: init: module timer SUCESS
mempool_datastore: pending_pkt pages: 1 elem_len: 56 nostraddle:0 chunk_size:128 num_chunks:4292 [0x3fffc00000-0x3fffe00000]
mempool_datastore: arp pages: 3 elem_len: 80 nostraddle:0 chunk_size:128 num_chunks:4614 [0x3fff600000-0x3fffc00000]
06:47:00 <4>: init: module net SUCESS
06:47:00 <4>: using config :'./ix.conf'
06:47:00 <4>: no static arp entries defined in config
06:47:00 <4>: net: using the following configuration:
06:47:00 <4>: host IP: 192.168.0.21
06:47:00 <4>: broadcast IP: 192.168.0.255
06:47:00 <4>: gateway IP: 192.168.0.1
06:47:00 <4>: subnet mask: 255.255.255.0
06:47:00 <4>: init: module cfg SUCESS
06:47:00 <4>: init: module cp SUCESS
06:47:00 <4>: pci: created device 0000:02:01.0, NUMA node -1
06:47:00 <4>: pci: IOMEM - base fd5c0000, len 20000
06:47:00 <4>: pci: IOMEM - base fdff0000, len 10000
06:47:00 <4>: init: module pci SUCESS
EAL: Detected lcore 0 as core 0 on socket 0
EAL: Detected lcore 1 as core 0 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 2 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: Module /sys/module/vfio_pci not found! error 2 (No such file or directory)
EAL: VFIO modules not loaded, skipping VFIO support...
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f77bfc00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f77bf800000 (size = 0x200000)
EAL: Ask a virtual area of 0x12e00000 bytes
EAL: Virtual area found at 0x7f77ac800000 (size = 0x12e00000)
EAL: Ask a virtual area of 0x6800000 bytes
EAL: Virtual area found at 0x7f77a5e00000 (size = 0x6800000)
EAL: Ask a virtual area of 0x4a00000 bytes
EAL: Virtual area found at 0x7f77a1200000 (size = 0x4a00000)
EAL: Ask a virtual area of 0x8000000 bytes
EAL: Virtual area found at 0x7f7799000000 (size = 0x8000000)
EAL: Ask a virtual area of 0x2e00000 bytes
EAL: Virtual area found at 0x7f7796000000 (size = 0x2e00000)
EAL: Ask a virtual area of 0x2c00000 bytes
EAL: Virtual area found at 0x7f7793200000 (size = 0x2c00000)
EAL: Ask a virtual area of 0x1a00000 bytes
EAL: Virtual area found at 0x7f7791600000 (size = 0x1a00000)
EAL: Ask a virtual area of 0x4800000 bytes
EAL: Virtual area found at 0x7f778cc00000 (size = 0x4800000)
EAL: Ask a virtual area of 0x2000000 bytes
EAL: Virtual area found at 0x7f778aa00000 (size = 0x2000000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f778a400000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7f7789e00000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f7789a00000 (size = 0x200000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7f7789600000 (size = 0x200000)
EAL: Requesting 74 pages of size 2MB from socket 0
EAL: TSC frequency is ~2712014 KHz
EAL: Master lcore 0 is ready (tid=c246b940;cpuset=[0])
06:47:47 <4>: init: module dpdk SUCESS
ix: drivers/common.c:194: driver_init: Assertion `port < rte_eth_dev_count()' failed.

ubuntu@ubuntu:~$ lspci | grep -i eth
02:01.0 Ethernet controller: Intel Corporation 82545EM Gigabit Ethernet Controller (Copper) (rev 01)

I use Intel e1000 NIC. Can IX support it with some changes? thanks.

Profiling in IX

How would we profile applications running on top of IX and the IX code itself using profilers like oprofile and perf? We did try using perf to record events against the IX pid but the recorded samples show no symbols related to our application code or IX code.

dune: exit due to EPT violation

Hello, I want to run IX on SLES 11 sp1 whose kernel version is 2.6.32, and I have modified some kernel code to support Dune and DPDK.
When I start the IX (v1.0), I got some error, the detail is in below:

log in stdout:
15:01:47 <4>: init: starting IX
15:01:47 <4>: init: cpu phase
15:01:47 <4>: cpu: detected 12 cores
15:01:47 <4>: init: module CPU SUCESS
15:01:47 <4>: init: module Dune SUCESS
15:01:47 <4>: timer: detected 2399 ticks per US
15:01:47 <4>: init: module timer SUCESS
mempool_datastore: pending_pkt pages: 1 elem_len: 56 nostraddle:0 chunk_size:128 num_chunks:4292
mempool_datastore: arp pages: 3 elem_len: 80 nostraddle:0 chunk_size:128 num_chunks:4614
15:01:47 <4>: init: module net SUCESS
15:01:47 <4>: using config :'./ix.conf'
15:01:47 <4>: no static arp entries defined in config
15:01:47 <4>: net: using the following configuration:
15:01:47 <4>: host IP: 172.18.11.234
15:01:47 <4>: broadcast IP: 172.18.255.255
15:01:47 <4>: gateway IP: 172.18.0.254
15:01:47 <4>: subnet mask: 255.255.0.0
15:01:47 <4>: init: module cfg SUCESS
15:01:47 <4>: init: module cp SUCESS
EAL: Detected lcore 0 as core 0 on socket 1
EAL: Detected lcore 1 as core 1 on socket 1
EAL: Detected lcore 2 as core 2 on socket 1
EAL: Detected lcore 3 as core 8 on socket 1
EAL: Detected lcore 4 as core 9 on socket 1
EAL: Detected lcore 5 as core 10 on socket 1
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 8 on socket 0
EAL: Detected lcore 10 as core 9 on socket 0
EAL: Detected lcore 11 as core 10 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: Setting up physically contiguous memory...
EAL: WARNING: TSC frequency estimated roughly - clock timings may be less accurate.
EAL: TSC frequency is ~2399761 KHz
EAL: WARNING: Master core has no memory on local socket!
EAL: Master lcore 0 is ready (tid=865e6a20;cpuset=[0])
EAL: lcore 3 is ready (tid=797e3710;cpuset=[3])
EAL: lcore 8 is ready (tid=76fde710;cpuset=[8])
EAL: lcore 1 is ready (tid=7a7e5710;cpuset=[1])
EAL: lcore 5 is ready (tid=787e1710;cpuset=[5])
EAL: lcore 6 is ready (tid=77fe0710;cpuset=[6])
EAL: lcore 7 is ready (tid=777df710;cpuset=[7])
EAL: lcore 10 is ready (tid=75fdc710;cpuset=[10])
EAL: lcore 2 is ready (tid=79fe4710;cpuset=[2])
EAL: lcore 11 is ready (tid=757db710;cpuset=[11])
EAL: lcore 9 is ready (tid=767dd710;cpuset=[9])
EAL: lcore 4 is ready (tid=78fe2710;cpuset=[4])
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:02:00.1 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: PCI memory mapped at 0x7f6974ddb000
EAL: PCI memory mapped at 0x7f6974dd7000
PMD: eth_ixgbe_dev_init(): MAC: 2, PHY: 2
PMD: eth_ixgbe_dev_init(): port 0 vendorID=0x8086 deviceID=0x151c
15:01:49 <4>: init: module dpdk SUCESS
CPU 00| 15:01:49 <4>: cpu: started core 0, numa node 1
CPU 00| 15:01:49 <4>: init: module firstcpu SUCESS
mempool_datastore: mbuf pages: 793 elem_len:2112 nostraddle:1 chunk_size:128 num_chunks:46145
CPU 00| 15:01:49 <4>: init: module mbuf SUCESS
mempool_datastore: pbuf pages: 18 elem_len: 48 nostraddle:0 chunk_size:128 num_chunks:46144
mempool_datastore: pbuf_payload pages: 512 elem_len:4096 nostraddle:0 chunk_size:128 num_chunks:42048
mempool_datastore: tcp_pcb pages: 34 elem_len: 272 nostraddle:0 chunk_size:128 num_chunks:42048
mempool_datastore: tcp_seg pages: 4 elem_len: 32 nostraddle:0 chunk_size:128 num_chunks:42048
CPU 00| 15:01:50 <4>: init: module memp SUCESS
mempool_datastore: pcb pages: 16 elem_len: 64 nostraddle:0 chunk_size:128 num_chunks:44096
mempool_datastore: ip pages: 4 elem_len: 16 nostraddle:1 chunk_size:128 num_chunks:44096
CPU 00| 15:01:50 <4>: init: module tcpapi SUCESS
CPU 00| 15:01:50 <4>: pci: created device 0000:02:00.0, NUMA node -1
CPU 00| 15:01:50 <4>: pci: IOMEM - base fb200000, len 200000
CPU 00| 15:01:50 <4>: pci: IOMEM - base fb6f8000, len 4000
EAL: PCI device 0000:02:00.0 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
dune: exit due to EPT violation

and log in dmesg:
[301887.422325] R ffffffff81002df0
[301889.480035] vmx: created VCPU (VPID 1)
[301890.057894] pcidma 0000:02:00.0: setting latency timer to 64
[301890.057898] pcidma 0000:02:00.0: claimed
[301890.058223] ept: failed to get user page 4060
[301890.058225] vmx: page fault failure GPA: 0x4060, GVA: 0x4060
[301890.058226] vmx: --- Begin VCPU Dump ---
[301890.058228] vmx: CPU 0 VPID 1
[301890.058229] vmx: RIP 0x000000007006346e RFLAGS 0x00010206
[301890.058230] vmx: RAX 0x0000000000000001 RCX 0x0000000000000000
[301890.058231] vmx: RDX 0x0000000000000030 RBX 0x0000000000004060
[301890.058233] vmx: RSP 0x00007fff1a949680 RBP 0x000000001a949679
[301890.058234] vmx: RSI 0x00007fff1a9496a0 RDI 0x00007f6984030340
[301890.058235] vmx: R8 0xfefeffffffffffff R9 0xfefeff302d2f3931
[301890.058237] vmx: R10 0x0000000000000000 R11 0x0000000000000000
[301890.058238] vmx: R12 0x00007fff1a9496a0 R13 0x0000000070298120
[301890.058239] vmx: R14 0x0000000000000001 R15 0x0000000000010282
[301890.058240] vmx: Dumping Stack Contents...
[301890.058242] vmx: RSP+0 0x0000000000000001
[301890.058243] vmx: RSP+8 0x0000000070298120
[301890.058244] vmx: RSP+16 0x00000000709e4540
[301890.058245] vmx: RSP+24 0x00000000700636e3
[301890.058246] vmx: RSP+32 0x000000302e303a32
[301890.058247] vmx: RSP+40 0xffffffffffffffff
[301890.058248] vmx: RSP+48 0x00000000700833ff
[301890.058249] vmx: RSP+56 0x00000000700833fe
[301890.058250] vmx: RSP+64 0x00000000709e4540
[301890.058251] vmx: RSP+72 0x0000000070298120
[301890.058252] vmx: RSP+80 0x00007fff1a94a2b0
[301890.058253] vmx: RSP+88 0x0000000000000000
[301890.058254] vmx: --- End VCPU Dump ---
[301890.058255] vmx: stopping VCPU (VPID 1)
[301890.142055] vmx: destroying VCPU (VPID 1)
[301890.142355] pcidma 0000:02:00.0: released

Could you help me? Thanks!

IX app hangs up in 'logk' function

Hi,

We have tried IX on our server. The settings are:

CPU: E2630
NIC: Intel 82599ES 10Gbps
OS: Ubuntu 14.04.1, Linux 3.13.0

The program hangs-up at time(&ts); (in function logk). The ourput is:

07:44:40 <4>: init: starting IX
07:44:40 <4>: init: cpu phase
07:44:40 <4>: cpu: detected 32 cores
07:44:40 <4>: init: module CPU        SUCESS
07:44:40 <4>: init: module Dune       SUCESS
07:44:41 <4>: timer: detected 2394 ticks per US
07:44:41 <4>: init: module timer      SUCESS
mempool_datastore: pending_pkt     pages:   1 elem_len:  56 nostraddle:0 chunk_size:128 num_chunks:4292
mempool_datastore: arp             pages:   3 elem_len:  80 nostraddle:0 chunk_size:128 num_chunks:4614
07:44:41 <4>: init: module net        SUCESS
07:44:41 <4>: using config :'./ix.conf'
07:44:41 <4>: no static arp entries defined in config
07:44:41 <4>: net: using the following configuration:
07:44:41 <4>:   host IP:        192.168.40.1
07:44:41 <4>:   broadcast IP:   192.168.255.255
07:44:41 <4>:   gateway IP:     192.168.1.1
07:44:41 <4>:   subnet mask:    255.255.0.0
07:44:41 <4>: init: module cfg        SUCESS
07:44:41 <4>: init: module cp         SUCESS
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
07:44:45 <4>: init: module dpdk       SUCESS

Now, it hangs up. We run it step by step and find that time(&ts); (in function logk) is the cause.

Then we comment out time(&ts);, and the program runs correctly until showing:

03:23:08 <4>: init: starting IX
03:23:08 <4>: init: cpu phase
01:09:19 <4>: cpu: detected 32 cores
16:37:12 <4>: init: module CPU        SUCESS
16:14:13 <4>: init: module Dune       SUCESS
04:09:52 <4>: timer: detected 2394 ticks per US
16:14:13 <4>: init: module timer      SUCESS
mempool_datastore: pending_pkt     pages:   1 elem_len:  56 nostraddle:0 chunk_size:128 num_chunks:4292
mempool_datastore: arp             pages:   3 elem_len:  80 nostraddle:0 chunk_size:128 num_chunks:4614
16:14:13 <4>: init: module net        SUCESS
01:43:52 <4>: using config :'./ix.conf'
11:52:51 <4>: no static arp entries defined in config
18:00:00 <4>: net: using the following configuration:
18:00:00 <4>:   host IP:        192.168.40.1
18:00:00 <4>:   broadcast IP:   192.168.255.255
18:00:00 <4>:   gateway IP:     192.168.1.1
18:00:00 <4>:   subnet mask:    255.255.0.0
22:39:12 <4>: init: module cfg        SUCESS
11:47:12 <4>: init: module cp         SUCESS
EAL: Detected 32 lcore(s)
EAL: Probing VFIO support...
EAL: PCI device 0000:06:00.0 on NUMA socket 0
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
EAL: PCI device 0000:06:00.1 on NUMA socket 0
EAL:   probe driver: 8086:10fb rte_ixgbe_pmd
19:23:28 <4>: init: module dpdk       SUCESS
CPU 02| 16:37:04 <4>: cpu: started core 2, numa node 0
CPU 02| 01:44:36 <4>: init: module firstcpu   SUCESS
mempool_datastore: mbuf            pages: 793 elem_len:2112 nostraddle:1 chunk_size:128 num_chunks:46145
CPU 02| 01:44:36 <4>: init: module mbuf       SUCESS
mempool_datastore: pbuf            pages:  18 elem_len:  48 nostraddle:0 chunk_size:128 num_chunks:46144
mempool_datastore: pbuf_payload    pages: 512 elem_len:4096 nostraddle:0 chunk_size:128 num_chunks:42048
mempool_datastore: tcp_pcb         pages:  34 elem_len: 272 nostraddle:0 chunk_size:128 num_chunks:42048
mempool_datastore: tcp_seg         pages:   4 elem_len:  32 nostraddle:0 chunk_size:128 num_chunks:42048
CPU 02| 01:44:36 <4>: init: module memp       SUCESS
mempool_datastore: pcb             pages:  16 elem_len:  64 nostraddle:0 chunk_size:128 num_chunks:44096
mempool_datastore: ip              pages:   4 elem_len:  16 nostraddle:1 chunk_size:128 num_chunks:44096
CPU 02| 01:44:36 <4>: init: module tcpapi     SUCESS
CPU 02| 00:33:36 <4>: pci: created device 0000:06:00.0, NUMA node 0
CPU 02| 00:33:36 <4>: pci:      IOMEM - base c7d80000, len 80000
CPU 02| 00:33:36 <4>: pci:      IOMEM - base c7e04000, len 4000
CPU 02| 18:00:00 <4>: init: module ethdev     SUCESS
CPU 02| 18:00:00 <4>: init: percpu phase 2
CPU 02| 18:00:00 <4>: init: module timer      on 2: SUCESS
CPU 02| 18:00:00 <4>: init: module mbuf       on 2: SUCESS
CPU 02| 18:00:00 <4>: init: module memp       on 2: SUCESS
CPU 02| 18:00:00 <4>: init: module tcpapi     on 2: SUCESS
CPU 02| 18:00:00 <4>: init: module migration  on 2: SUCESS
CPU 02| 18:00:00 <4>: init: module syscall    on 2: SUCESS
CPU 02| 18:00:00 <4>: init: module init-net   on 2: SUCESS
CPU 02| 18:00:00 <4>: init: CPU 2 ready
init:CPU 2 ready
CPU 02| 22:54:40 <4>: eth: started an ethernet device
CPU 02| 22:54:40 <4>: eth:      MAC address: 90:E2:BA:26:F0:60
CPU 02| 22:54:40 <4>: eth:      link up - speed 10000 Mbps, half-duplex

CPU 02| 18:00:00 <4>: init: barrier after al CPU initialization
dune: exit due to EPT violation

Do you know what causes the failure of Dune/IX?
Many thanks.

Initialize the i40e DPDK driver without huge pages

See comment here: #6 (diff)

About the static mem address

Dear authors,

I tested ix on my testbed (with Intel(R) Xeon(R) CPU E5-2650 v3 @ 2.30GHz, 128GB memory, Intel x520 10Gbps). The mempool cannot be initialized for "pending_pkt".
The call trace is as follows:
arp_init->mempool_create_datastore->mem_alloc_pages->__mem_alloc_pages->mmap (return MAP_FAILED)

I guess if fails because the base mem (MEM_PHYS_BASE_ADDR - PGSIZE_2MB * nr) is not suitable for my server. How should I modify the mem addr for my testbed? For example:
#define MEM_PHYS_BASE_ADDR 0x4000000000

Many thanks!

Echoclient change

I have changed the code of the echoclient so that it prints out results every 10 seconds instead of waiting for input. Is this something you would be interested to merge into master?

There are some errors when I start the ix

I have met some errors when I start the IX, and I can't fix it.

I can't write 4096 in huge-pages just because the memory isn't enough. If IX need 2G, why should we write 4096 in the 2048-hugepage?
There is something wrong with:
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.1 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped

My environment is: Ubuntu 16.04 LTS Intel Corporation 82599 10 Gigabit TN Network Connection (rev 01)

Here is the log:
20:34:47 <4>: init: starting IX
20:34:47 <4>: init: cpu phase
20:34:47 <4>: cpu: detected 12 cores
20:34:47 <4>: init: module CPU SUCESS
20:34:47 <4>: init: module Dune SUCESS
20:34:48 <4>: timer: detected 2400 ticks per US
20:34:48 <4>: init: module timer SUCESS
mempool_datastore: pending_pkt pages: 1 elem_len: 56 nostraddle:0 chunk_size:128 num_chun
ks:4292mempool_datastore: arp pages: 3 elem_len: 80 nostraddle:0 chunk_size:128 num_chun
ks:461420:34:48 <4>: init: module net SUCESS
20:34:48 <4>: using config :'./ix.conf'
20:34:48 <4>: no static arp entries defined in config
20:34:48 <4>: net: using the following configuration:
20:34:48 <4>: host IP: 172.18.53.10
20:34:48 <4>: broadcast IP: 172.18.255.255
20:34:48 <4>: gateway IP: 172.18.0.254
20:34:48 <4>: subnet mask: 255.255.0.0
20:34:48 <4>: init: module cfg SUCESS
20:34:48 <4>: init: module cp SUCESS
EAL: Detected lcore 0 as core 0 on socket 1
EAL: Detected lcore 1 as core 1 on socket 1
EAL: Detected lcore 2 as core 2 on socket 1
EAL: Detected lcore 3 as core 8 on socket 1
EAL: Detected lcore 4 as core 9 on socket 1
EAL: Detected lcore 5 as core 10 on socket 1
EAL: Detected lcore 6 as core 0 on socket 0
EAL: Detected lcore 7 as core 1 on socket 0
EAL: Detected lcore 8 as core 2 on socket 0
EAL: Detected lcore 9 as core 8 on socket 0
EAL: Detected lcore 10 as core 9 on socket 0
EAL: Detected lcore 11 as core 10 on socket 0
EAL: Support maximum 128 logical core(s) by configuration.
EAL: Detected 12 lcore(s)
EAL: No free hugepages reported in hugepages-1048576kB
EAL: Probing VFIO support...
EAL: IOMMU type 1 (Type 1) is supported
EAL: IOMMU type 8 (No-IOMMU) is not supported
EAL: VFIO support initialized
EAL: Setting up physically contiguous memory...
EAL: Ask a virtual area of 0x30400000 bytes
EAL: Virtual area found at 0x7fce2f800000 (size = 0x30400000)
EAL: Ask a virtual area of 0x81400000 bytes
EAL: Virtual area found at 0x7fcdae200000 (size = 0x81400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fcdade00000 (size = 0x200000)
EAL: Ask a virtual area of 0x12a800000 bytes
EAL: Virtual area found at 0x7fcc83400000 (size = 0x12a800000)
EAL: Ask a virtual area of 0x1e7400000 bytes
EAL: Virtual area found at 0x7fca9be00000 (size = 0x1e7400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fca9ba00000 (size = 0x200000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fca9b400000 (size = 0x400000)
EAL: Ask a virtual area of 0x400000 bytes
EAL: Virtual area found at 0x7fca9ae00000 (size = 0x400000)
EAL: Ask a virtual area of 0x200000 bytes
EAL: Virtual area found at 0x7fca9aa00000 (size = 0x200000)
EAL: Requesting 37 pages of size 2MB from socket 0
EAL: Requesting 37 pages of size 2MB from socket 1
EAL: TSC frequency is ~2400085 KHz
EAL: Master lcore 0 is ready (tid=655158c0;cpuset=[0])
EAL: lcore 11 is ready (tid=5d4f5700;cpuset=[11])
EAL: lcore 8 is ready (tid=5ecf8700;cpuset=[8])
EAL: lcore 6 is ready (tid=5fcfa700;cpuset=[6])
EAL: lcore 9 is ready (tid=5e4f7700;cpuset=[9])
EAL: lcore 1 is ready (tid=624ff700;cpuset=[1])
EAL: lcore 3 is ready (tid=614fd700;cpuset=[3])
EAL: lcore 2 is ready (tid=61cfe700;cpuset=[2])
EAL: lcore 7 is ready (tid=5f4f9700;cpuset=[7])
EAL: lcore 4 is ready (tid=60cfc700;cpuset=[4])
EAL: lcore 5 is ready (tid=604fb700;cpuset=[5])
EAL: lcore 10 is ready (tid=5dcf6700;cpuset=[10])
EAL: PCI device 0000:03:00.0 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
EAL: PCI device 0000:03:00.1 on NUMA socket -1
EAL: probe driver: 8086:151c rte_ixgbe_pmd
EAL: Not managed by a supported kernel driver, skipped
20:34:52 <4>: init: module dpdk SUCESS
CPU 06| 20:34:52 <4>: cpu: started core 6, numa node 0
fatal exception 13, code 0 - dying...
dune: --- Begin Trap Dump ---
dune: RIP 0x000000007000d7f5
dune: CS 0x10 SS 0x18
dune: ERR 0x00000000 RFLAGS 0x00010246
dune: RAX 0x0000000000000000 RCX 0x0000000000000606
dune: RDX 0x0000000000000000 RBX 0x0000000000000000
dune: RSP 0x00007ffdf2d24eb0 RBP 0x00000000700b3242
dune: RSI 0x0000000000000037 RDI 0x00007ffdf2d24390
dune: R8 0x0000000000000000 R9 0x0000000000000037
dune: R10 0x0000000000000037 R11 0x0000000000000000
dune: R12 0x0000000000000004 R13 0x00007ffdf2d24fc8
dune: R14 0x0000000000000000 R15 0x0000000000000000
dune: Dumping Stack Contents...
dune: RSP+0 0x00000000700b33c8
dune: RSP+8 0x0000000070009cca
dune: RSP+16 0x00000000700b25c0
dune: RSP+24 0x0000000000000000
dune: RSP+32 0x00000000700b25c0
dune: RSP+40 0x000000007000a0d0
dune: RSP+48 0x00007ffdf2d24fc0
dune: RSP+56 0x00007fce64523830
dune: RSP+64 0xffffffffff000000
dune: RSP+72 0x00007ffdf2d24fc8
dune: RSP+80 0x0000000400000000
dune: RSP+88 0x0000000070009c70
dune: code before IP 77 b7 3a 00 e8 f6 fa ff ff 85 c0 89 c3 75 3a b9 06 06 00 00
dune: code at IP 0f 32 89 c1 66 0f ef c0 81 e1 00 1f 00 00 b8 01 00 00 00 48
dune: --- End Trap Dump ---

Simpler and more flexible physical device assignment instructions

Both the wiki and readme tell the user to remove the ixgbe device to make it available to IX. That's not always possible if we have other NICs that require the same driver on the host.

Instead, it's better to rebind a device's driver. That can be accomplished through DPDK's dpdk-devbind.pyscript (http://dpdk.org/doc/guides/tools/devbind.html).

Build IX error

When I run make -sj64, the following errors occur:
In file included from ../deps/dpdk/build/include/rte_ether.h:50:0,
from ../deps/dpdk/build/include/rte_ethdev.h:185,
from drivers/i40e.c:35:
../deps/dpdk/build/include/rte_memcpy.h:870:2: error: incompatible type for argument 2 of ‘_mm_storeu_si128’
MOVEUNALIGNED_LEFT47(dst, src, n, srcofs);
^
In file included from /usr/lib/gcc/x86_64-linux-gnu/4.8/include/xmmintrin.h:1246:0,
from /usr/lib/gcc/x86_64-linux-gnu/4.8/include/x86intrin.h:34,
from ../deps/dpdk/build/include/rte_vect.h:67,
from ../deps/dpdk/build/include/rte_memcpy.h:46,
from ../deps/dpdk/build/include/rte_ether.h:50,
from ../deps/dpdk/build/include/rte_ethdev.h:185,
from drivers/i40e.c:35:
/usr/lib/gcc/x86_64-linux-gnu/4.8/include/emmintrin.h:700:1: note: expected ‘__m128i’ but argument is of type ‘int’
_mm_storeu_si128 (__m128i *__P, __m128i __B)
^
make[1]: *** [core/dpdk.o] Error 1
make[1]: *** Waiting for unfinished jobs....
make[1]: *** [drivers/common.o] Error 1
make[1]: *** [drivers/ixgbe.o] Error 1
make[1]: *** [drivers/i40e.o] Error 1
make: *** [dp] Error 2

I use VMware Workstation 15 Player and Ubuntu 14.04.

dubuntu@ubuntu:~/ix$ cat /proc/cpuinfo
processor : 0
vendor_id : GenuineIntel
cpu family : 6
model : 142
model name : Intel(R) Core(TM) i5-7200U CPU @ 2.50GHz
stepping : 9
microcode : 0xde
cpu MHz : 2712.002
cache size : 3072 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 22
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon nopl xtopology tsc_reliable nonstop_tsc pni pclmulqdq vmx ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp kaiser tpr_shadow vnmi ept vpid fsgsbase tsc_adjust bmi1 avx2 smep bmi2 invpcid mpx rdseed adx smap clflushopt xsaveopt xsavec arat flush_l1d arch_capabilities
bugs : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf
bogomips : 5424.00
clflush size : 64
cache_alignment : 64
address sizes : 43 bits physical, 48 bits virtual
power management:

ix hang while initiating sandbox

From:"George Prekas" [email protected]
Sent Time:2017-09-14 23:26:29 (Thursday)
To: [email protected]
Cc: [email protected]
Subject: Re: ix hang while initiating sandbox

Hi Yang,
Can you open an issue so that we can properly track this problem?

On 14/09/2017 12:58, Xinggao Yang wrote:
I don't know if you are still answering the questions about ix.

My ix hanged up like this: https://gist.github.com/XingGaoY/61f75e8853b466f7c881baf6c9815d4f

I did everything strictly as the guideline, and I find a similar closed request on github (#4) so I changed GPA_MAP_SIZE to 1<<31 to solve my previous problem below: https://gist.github.com/XingGaoY/d2f063b94f2d3a64e273c109cca81e94

Here is my cpuinfo: https://gist.github.com/XingGaoY/0b9f6fe2964d644d55769f8403c2b758

And os: Ubuntu 15.10 and Linux kernel 4.4.0

And the dmesg if you need it: https://gist.github.com/XingGaoY/7d860d9387f5f81991f3fcaa6bc717cd