scylladb / dpdk Goto Github PK

Mirror of Data Plane Development Kit, git://dpdk.org/dpdk (http://dpdk.org)

License: GNU General Public License v2.0

Makefile 0.87% C 31.84% Python 0.13% C++ 67.09% Shell 0.06% Objective-C 0.01%

dpdk's Introduction

Scylla

What is Scylla?

Scylla is the real-time big data database that is API-compatible with Apache Cassandra and Amazon DynamoDB. Scylla embraces a shared-nothing approach that increases throughput and storage capacity to realize order-of-magnitude performance improvements and reduce hardware costs.

For more information, please see the ScyllaDB web site.

Build Prerequisites

Scylla is fairly fussy about its build environment, requiring very recent versions of the C++23 compiler and of many libraries to build. The document HACKING.md includes detailed information on building and developing Scylla, but to get Scylla building quickly on (almost) any build machine, Scylla offers a frozen toolchain, This is a pre-configured Docker image which includes recent versions of all the required compilers, libraries and build tools. Using the frozen toolchain allows you to avoid changing anything in your build machine to meet Scylla's requirements - you just need to meet the frozen toolchain's prerequisites (mostly, Docker or Podman being available).

Building Scylla

Building Scylla with the frozen toolchain dbuild is as easy as:

$ git submodule update --init --force --recursive
$ ./tools/toolchain/dbuild ./configure.py
$ ./tools/toolchain/dbuild ninja build/release/scylla

For further information, please see:

Developer documentation for more information on building Scylla.
Build documentation on how to build Scylla binaries, tests, and packages.
Docker image build documentation for information on how to build Docker images.

Running Scylla

To start Scylla server, run:

$ ./tools/toolchain/dbuild ./build/release/scylla --workdir tmp --smp 1 --developer-mode 1

This will start a Scylla node with one CPU core allocated to it and data files stored in the tmp directory. The --developer-mode is needed to disable the various checks Scylla performs at startup to ensure the machine is configured for maximum performance (not relevant on development workstations). Please note that you need to run Scylla with dbuild if you built it with the frozen toolchain.

For more run options, run:

$ ./tools/toolchain/dbuild ./build/release/scylla --help

Testing

See test.py manual.

Scylla APIs and compatibility

By default, Scylla is compatible with Apache Cassandra and its API - CQL. There is also support for the API of Amazon DynamoDB™, which needs to be enabled and configured in order to be used. For more information on how to enable the DynamoDB™ API in Scylla, and the current compatibility of this feature as well as Scylla-specific extensions, see Alternator and Getting started with Alternator.

Documentation

Documentation can be found here. Seastar documentation can be found here. User documentation can be found here.

Training

Training material and online courses can be found at Scylla University. The courses are free, self-paced and include hands-on examples. They cover a variety of topics including Scylla data modeling, administration, architecture, basic NoSQL concepts, using drivers for application development, Scylla setup, failover, compactions, multi-datacenters and how Scylla integrates with third-party applications.

Contributing to Scylla

If you want to report a bug or submit a pull request or a patch, please read the contribution guidelines.

If you are a developer working on Scylla, please read the developer guidelines.

Contact

The community forum and Slack channel are for users to discuss configuration, management, and operations of the ScyllaDB open source.
The developers mailing list is for developers and people interested in following the development of ScyllaDB to discuss technical topics.

dpdk's People

Contributors

Stargazers

Watchers

Forkers

tempbottle ezhangle arkadiuszsz smartmircoarray liucongpei earthliling georgeaaabbbccc wheelcomplex talalbutt datatonic cloudxtreme iluxa devboxos xubingyue chyyuu linearregression live-for-dream jbakamovic demond2 leptons-bosons mkarimawan yedan2010 romaerzhang windofthesky b2ornot2b akinswin smatsushi scslab victor-smirnov wycharry rootfs conseweb hjwsm1989 zmyer mkarthi92 sourcestudynotes maxwell-guo syuu1228 liujun01203 ioselection rjmcguire glommer gumi-presentation-by-dzh liudong16 evilmcjerkface hippobaro gleb-cloudius ferryworld yong311408 kingwolfofsky peerbelt-build citrullin shuixianmu yuya008 krassif zwphit wangyx0055 njlr xopenlee fucai-liu duanjp8617 pattern-inc zhengbinphp zhaodiankui skyformat99 ljx0305 anakli jamesbucher dotnwat markw77 gaopengju cpv-project melor upcloud yuzhongbin duanshisan dabing1993 charmingcow becoinc femiagbabiaka unixian97 jeis2497052 ufosky-cpp skarlsson bwlee bym520zy cremlscent lieight sherlockhsu moosuchot crixalis2013 tchaikov doitliao rafie thirstycrow binbowang1987 sriduth liupsl garryqian buckaroo-pm

dpdk's Issues

Compilation issue with dpdk

Getting the below compilation issue.

[root@vm7 dpdk-master]# make config T=x86_64-native-linuxapp-gcc && make
make[3]: Nothing to be done for `depdirs'.
Configuration done
== Build lib
== Build lib/librte_compat
== Build lib/librte_eal
== Build lib/librte_eal/common
== Build lib/librte_eal/linuxapp
== Build lib/librte_eal/linuxapp/igb_uio
CC [M] /home/pradeep/code/dpdk-master/build/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o
In file included from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/kernel.h:11,
from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/cache.h:4,
from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/time.h:4,
from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/stat.h:18,
from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/sysfs.h:20,
from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/kobject.h:21,
from /usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/device.h:17,
from /home/pradeep/code/dpdk-master/build/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.c:27:
/usr/src/kernels/3.10.0-693.17.1.el7.x86_64/include/linux/log2.h:22:1: error: ignoring attribute ‘noreturn’ because it conflicts with attribute ‘const’ [-Werror=attributes]
int ____ilog2_NaN(void);
^~~
cc1: all warnings being treated as errors
make[8]: *** [/home/pradeep/code/dpdk-master/build/build/lib/librte_eal/linuxapp/igb_uio/igb_uio.o] Error 1
make[7]: *** [module/home/pradeep/code/dpdk-master/build/build/lib/librte_eal/linuxapp/igb_uio] Error 2
make[6]: *** [sub-make] Error 2
make[5]: *** [igb_uio.ko] Error 2
make[4]: *** [igb_uio] Error 2
make[3]: *** [linuxapp] Error 2
make[2]: *** [librte_eal] Error 2
make[1]: *** [lib] Error 2
make: *** [all] Error 2

The latest version of DPDK is 19.02. Which version is seastar's DPDK customized based on?

compilation broken on >=glibc-2.25

Users trying to compile scylladb with dpdk enabled will fail if their version of glibc is >=2.25 with the following error:

dpdk/lib/librte_eal/linuxapp/eal/eal_pci_uio.c:134:8: error: implicit declaration of function ‘makedev’ [-Werror=implicit-function-declaration]
  dev = makedev(major, minor);
        ^
dpdk/lib/librte_eal/linuxapp/eal/eal_pci_uio.c:134:2: error: nested extern declaration of ‘makedev’ [-Werror=nested-externs]
  dev = makedev(major, minor);
  ^
cc1: all warnings being treated as errors

This is due to the following change in glibc:

* The inclusion of <sys/sysmacros.h> by <sys/types.h> is deprecated.  This
  means that in a future release, the macros “major”, “minor”, and “makedev”
  will only be available from <sys/sysmacros.h>.

The fix is to #include <sys/sysmacros.h> in the dpdk/lib/librte_eal/linuxapp/eal/eal_pci_uio.c file :)

I'm not submitting a PR because I've seen that you plan on upgrading your copy of dpdk wrt gcc-6 support, but if you want one just ask ;)

Meanwhile I will maintain a patch on Gentoo packaging for this until this issue is closed.

dpdk-pdump: vdev creation failed: Driver can not attach the device

when running the pdump getting the following error
my system config
hardware= dell poweredge r730
CPU= dual socket (xeon e5 2697a)
OS=CentOS 7
dpdk version=16.11.1
interface getting traffic=port0
NIC=intel x710 DA/SFP+
hugepage size=1G
number of hugepage=16
error copy

[root@localhost pdump]# ./dpdk-pdump -- --pdump 'port=0,queue=*,rx-dev=test.pcap'
EAL: Detected 64 lcore(s)
EAL: Probing VFIO support...
EAL: cannot connect to primary process!
EAL: VFIO support could not be initialized
EAL: WARNING: Address Space Layout Randomization (ASLR) is enabled in the kernel.
EAL: This may cause issues with mapping memory into secondary processes
EAL: PCI device 0000:83:00.0 on NUMA socket 1
EAL: probe driver: 8086:1572 net_i40e
EAL: PCI device 0000:83:00.1 on NUMA socket 1
EAL: probe driver: 8086:1572 net_i40e
EAL: no driver found for net_pcap_rx_0
EAL: Driver cannot attach the device (net_pcap_rx_0)
EAL: Error - exiting with code: 1
Cause: vdev creation failed:create_mp_ring_vdev:736

Use quick sort in sort_by_physaddr maybe better

Sort the hugepg_tbl by physical address (lower addresses first on x86,
- higher address first on powerpc). We use a slow algorithm, but we won't
- have millions of pages, and this is only done at init time.
  */
  But we can be better by other sort function.