Giter VIP home page Giter VIP logo

yostealth / liburing Goto Github PK

View Code? Open in Web Editor NEW
92.0 92.0 3.0 558 KB

Liburing is Python + Cython wrapper around C Liburing, which is a helper to setup and tear-down io_uring instances.

Home Page: https://pypi.org/project/liburing/

License: Creative Commons Zero v1.0 Universal

Python 22.93% Cython 76.97% C 0.10%
async cython file futex io io-uring liburing python python3 socket statx syscall

liburing's People

Contributors

mike96265 avatar yostealth avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

liburing's Issues

io_uring (liburing) python binding asyncio examples request

Hello,
Its so happy to find this library released as the first io_uring api stuff for Python.

As a newbie for the io_uring and underlayer Kernel IO API, too many details are required to learn it.
Could you provide some asyncio style based python examples for understanding ?

p.s
python-3.6.8 with pip-9.0.3 install liburing at newest released Centos (version 8.1.1911), will case datestamp ModuleNotFoundError about "file liburing/setup.py", line 3, thus pip-3 install liburing will fail if not manually run pip-3 install datestamp first.

p.s.s

libs/liburing seems should git clone from https://github.com/axboe/liburing which is empty, and case the pip install liburing process failure.

unless clone this repo to local directory and overwriting libs/liburing by https://github.com/axboe/liburing, and then pip install path/to/this/repo works.

at least works for me :} .

OSError: [Errno 9] Bad file descriptor when running Simple File Example

Hi,
I get the following error when I run the Simple File Example.

Traceback (most recent call last):
  File "test.py", line 74, in <module>
    main()
  File "test.py", line 58, in main
    fd = open(ring, cqes, 'liburing-test-file.txt', os.O_CREAT | os.O_RDWR)
  File "test.py", line 12, in open
    return _submit_and_wait(ring, cqes)  # returns fd
  File "test.py", line 44, in _submit_and_wait
    result = trap_error(cqe.res)  # auto raise appropriate exception if failed
  File "/home/ubuntu/.local/lib/python3.8/site-packages/liburing/wrapper.py", line 27, in trap_error
    raise OSError(-no, strerror(-no))
OSError: [Errno 9] Bad file descriptor 

Could you help me out?

Distributor ID: Ubuntu
Description:    Ubuntu 20.04.2 LTS
Release:        20.04
Codename:       focal
5.4.0-1038-aws

Request for examples

As the liburing concepts are fairly new, it's difficult for beginners (like me) to do even the most basic of the tasks, like reading or reading. So, it is helpful if a couple simple examples on reading and writing can be added.
Thanks.

"void * user_data" param of Python layer "io_uring_prep_cancel"

Hi,

an additional CFFI transfer is required to suit for the clang layer void *user_data define ?

# python
liburing.io_uring_prep_cancel(sqe, ???, 0)  # to cancel a socket ACCEPT sqe
                                            # e.g sqe.user_data = 1, how to transfer it to type "void *" 
// clang
static inline void io_uring_prep_cancel(struct io_uring_sqe *sqe, void *user_data, int flags);

Initialization fails on kernel 5.10

Environment info

  • Kernel version: 5.10.16-051016-generic
  • glibc version: ldd (Ubuntu GLIBC 2.31-0ubuntu9.2) 2.31
  • Python version: 3.8.5 (CPython)
  • Library version: master (i.e. 17ae331)

Actual vs. expected behavior

Actual behavior

When attempting to issue import liburing (e.g. in an interactive Python session) I get:

>>> import liburing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dm/liburing-test/.venv/lib/python3.8/site-packages/liburing/__init__.py", line 17, in <module>
    del lib.STATX_ATTR_VERITY
AttributeError: C attribute cannot be deleted

Expected behavior

AttributeError is not raised.

Steps to reproduce

python3 -m venv .venv
source .venv/bin/activate
python3 -m pip install --upgrade git+https://github.com/YoSTEALTH/Liburing
python3
>>> import liburing
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/dm/liburing-test/.venv/lib/python3.8/site-packages/liburing/__init__.py", line 17, in <module>
    del lib.STATX_ATTR_VERITY
AttributeError: C attribute cannot be deleted

Note: When I remove the two lines del lib.STATX_ATTR_VERITY and del lib.STATX_ATTR_DAX from liburing/__init__.py I can use the library just fine.

Wrapper for asyncio

Hello,

Thank you for your great library and I'm looking forward to new Cython version.

I'm interested in io_uring, and now I've been writing a asyncio wrapper for this library and file operation API.
https://github.com/tree4096/uring_file (forked from https://github.com/qweeze/uring_file)

Can I contribute my code on your library or community of io_uring developers?

After implemeting some optimization, I achieved relatively better performance than other async file I/O libs such as aiofiles, aiofile, and anyio.
The results of performance test as below.

Test code

https://github.com/tree4096/uring_file/blob/master/performance_test.py

Writing Test

Library Time [s] Process Time [s]
anyio 10.69 15.62
aiofiles 4.47 6.19
uring_file ⭐ 1.92 1.88
aiofile* 7.51 15.08

Reading Test

Library Time [s] Process Time [s]
anyio 0.482 0.816
aiofiles 0.349 0.623
uring_file ⭐ 0.118 0.118
aiofile* 0.473 0.828

*aiofile has Linux AIO mode, but does not work on Ubuntu due to the bug. So it run on C threading mode.

Environment

OS: Ubuntu 20.04.4 LTS on WSL2
Kernel ver.: 5.10.102.1-microsoft-standard-WSL2
CPU: AMD Ryzen 7 5800U
SSD: MTFDHBA1T0TDV-1AZ1AABHA
Python ver.: 3.10.5

My library provides simple and efficient APIs to access the io_uring like a database session and the file I/O APIs are written over them.
Here is the sample code.

# Create new Uring
uring = uring_file.Uring(sq_size=8, cq_size=64, session_sq_size=4)

# Get SQE and submit (Open file)
async with uring.session() as session:
    sqe = session.get_sqe()
    sqe.prep_openat(liburing.AT_FDCWD, b"hello.txt", os.O_RDONLY, 0o644)

# Get result of CQE
fd = sqe.result()
print("FD:", fd)

# Close file
async with uring.session() as session:
    # Raw SQE object can be accessed from the _sqe property
    liburing.io_uring_prep_close(session.get_sqe()._sqe, fd)
    # Same expression as session.get_sqe().prep_close(fd)

# Get default Uring
default_uring = uring_file.get_default_uring()

The SQEs of the sessions are automatically merged and submitted together under high frequency access, and I call it "auto bulk submit".
The auto bulk submit is realized by controlling the task scheduling of asyncio in a little hacky way and it makes large performance improvement.
And the overflow protection for the CQ is also implemented by using Semaphore.

I'm not sure because I'm just a beginner but I think these APIs may be useful for developing the io_uring applications on Python.

If you are interested in my library, could you give me some comments or advice?

pip install fails 2021.3.10 ‘_NSIG’ undeclared

Hello there, I've been trying to install this library from pypi without success.

TL;DR

  gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./libs/liburing/src/include -I/home/julien/.pyenv/versions/3.10.6/envs/onioncorn/include -I/home/julien/.pyenv/versions/3.10.6/include/python3.10 -c ./libs/liburing/src/syscall.c -o build/temp.linux-x86_64-cpython-310/./libs/liburing/src/syscall.o
  ./libs/liburing/src/syscall.c: In function ‘__sys_io_uring_enter’:
  ./libs/liburing/src/syscall.c:62:41: error: ‘_NSIG’ undeclared (first use in this function)
     62 |                                         _NSIG / 8);
        |                                         ^~~~~
  ./libs/liburing/src/syscall.c:62:41: note: each undeclared identifier is reported only once for each function it appears in
  ./libs/liburing/src/syscall.c:63:1: warning: control reaches end of non-void function [-Wreturn-type]
     63 | }
        | ^
  error: command '/usr/bin/gcc' failed with exit code 1

Kernel

$ uname -r
5.15.0-67-generic

System dependencies (build-essentials are installed too), I don't think they are necessary as you bundle liburing (the C lib) inside of the source install am I right?

$ apt search liburing
i   liburing-dev - Linux kernel io_uring access library - development files                                         
i A liburing2    - Linux kernel io_uring access library - shared library      

Python/Pip version

$ pip -V
pip 23.0.1 from ... (python 3.10)

Full installation message

$ pip install liburing
Collecting liburing
  Using cached liburing-2021.3.10.tar.gz (35 kB)
  Installing build dependencies ... done
  Getting requirements to build wheel ... done
  Preparing metadata (pyproject.toml) ... done
Collecting cffi
  Using cached cffi-1.15.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (441 kB)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Building wheels for collected packages: liburing
  Building wheel for liburing (pyproject.toml) ... error
  error: subprocess-exited-with-error
  
  × Building wheel for liburing (pyproject.toml) did not run successfully.
  │ exit code: 1
  ╰─> [78 lines of output]
      <string>:11: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
      prefix                        /usr
      includedir                    /usr/include
      libdir                        /usr/lib
      libdevdir                     /usr/lib
      relativelibdir
      mandir                        /usr/man
      datadir                       /usr/share
      __kernel_rwf_t                yes
      __kernel_timespec             yes
      open_how                      no
      statx                         yes
      C++                           yes
      has_ucontext                  yes
      CC                            gcc
      CXX                           g++
      running bdist_wheel
      running build
      running build_py
      creating build
      creating build/lib.linux-x86_64-cpython-310
      creating build/lib.linux-x86_64-cpython-310/liburing
      copying liburing/interface.py -> build/lib.linux-x86_64-cpython-310/liburing
      copying liburing/helper.py -> build/lib.linux-x86_64-cpython-310/liburing
      copying liburing/wrapper.py -> build/lib.linux-x86_64-cpython-310/liburing
      copying liburing/__init__.py -> build/lib.linux-x86_64-cpython-310/liburing
      running egg_info
      writing liburing.egg-info/PKG-INFO
      writing dependency_links to liburing.egg-info/dependency_links.txt
      writing requirements to liburing.egg-info/requires.txt
      writing top-level names to liburing.egg-info/top_level.txt
      reading manifest file 'liburing.egg-info/SOURCES.txt'
      reading manifest template 'MANIFEST.in'
      warning: no previously-included files found matching 'libs/liburing/src/include/liburing/compat.h'
      adding license file 'LICENSE.txt'
      writing manifest file 'liburing.egg-info/SOURCES.txt'
      running build_ext
      generating cffi module 'build/temp.linux-x86_64-cpython-310/liburing._liburing.c'
      creating build/temp.linux-x86_64-cpython-310
      building 'liburing._liburing' extension
      creating build/temp.linux-x86_64-cpython-310/libs
      creating build/temp.linux-x86_64-cpython-310/libs/liburing
      creating build/temp.linux-x86_64-cpython-310/libs/liburing/src
      creating build/temp.linux-x86_64-cpython-310/build
      creating build/temp.linux-x86_64-cpython-310/build/temp.linux-x86_64-cpython-310
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./libs/liburing/src/include -I/home/julien/.pyenv/versions/3.10.6/envs/onioncorn/include -I/home/julien/.pyenv/versions/3.10.6/include/python3.10 -c ./libs/liburing/src/queue.c -o build/temp.linux-x86_64-cpython-310/./libs/liburing/src/queue.o
      ./libs/liburing/src/queue.c: In function ‘_io_uring_get_cqe’:
      ./libs/liburing/src/queue.c:122:23: warning: implicit declaration of function ‘__sys_io_uring_enter2’; did you mean ‘SYS_io_uring_enter’? [-Wimplicit-function-declaration]
        122 |                 ret = __sys_io_uring_enter2(ring->ring_fd, data->submit,
            |                       ^~~~~~~~~~~~~~~~~~~~~
            |                       SYS_io_uring_enter
      ./libs/liburing/src/queue.c: In function ‘io_uring_peek_batch_cqe’:
      ./libs/liburing/src/queue.c:183:17: warning: implicit declaration of function ‘__sys_io_uring_enter’; did you mean ‘SYS_io_uring_enter’? [-Wimplicit-function-declaration]
        183 |                 __sys_io_uring_enter(ring->ring_fd, 0, 0,
            |                 ^~~~~~~~~~~~~~~~~~~~
            |                 SYS_io_uring_enter
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./libs/liburing/src/include -I/home/julien/.pyenv/versions/3.10.6/envs/onioncorn/include -I/home/julien/.pyenv/versions/3.10.6/include/python3.10 -c ./libs/liburing/src/register.c -o build/temp.linux-x86_64-cpython-310/./libs/liburing/src/register.o
      ./libs/liburing/src/register.c: In function ‘io_uring_register_buffers’:
      ./libs/liburing/src/register.c:22:15: warning: implicit declaration of function ‘__sys_io_uring_register’; did you mean ‘SYS_io_uring_register’? [-Wimplicit-function-declaration]
         22 |         ret = __sys_io_uring_register(ring->ring_fd, IORING_REGISTER_BUFFERS,
            |               ^~~~~~~~~~~~~~~~~~~~~~~
            |               SYS_io_uring_register
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./libs/liburing/src/include -I/home/julien/.pyenv/versions/3.10.6/envs/onioncorn/include -I/home/julien/.pyenv/versions/3.10.6/include/python3.10 -c ./libs/liburing/src/setup.c -o build/temp.linux-x86_64-cpython-310/./libs/liburing/src/setup.o
      ./libs/liburing/src/setup.c: In function ‘io_uring_queue_init_params’:
      ./libs/liburing/src/setup.c:143:14: warning: implicit declaration of function ‘__sys_io_uring_setup’; did you mean ‘SYS_io_uring_setup’? [-Wimplicit-function-declaration]
        143 |         fd = __sys_io_uring_setup(entries, p);
            |              ^~~~~~~~~~~~~~~~~~~~
            |              SYS_io_uring_setup
      gcc -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -fPIC -I./libs/liburing/src/include -I/home/julien/.pyenv/versions/3.10.6/envs/onioncorn/include -I/home/julien/.pyenv/versions/3.10.6/include/python3.10 -c ./libs/liburing/src/syscall.c -o build/temp.linux-x86_64-cpython-310/./libs/liburing/src/syscall.o
      ./libs/liburing/src/syscall.c: In function ‘__sys_io_uring_enter’:
      ./libs/liburing/src/syscall.c:62:41: error: ‘_NSIG’ undeclared (first use in this function)
         62 |                                         _NSIG / 8);
            |                                         ^~~~~
      ./libs/liburing/src/syscall.c:62:41: note: each undeclared identifier is reported only once for each function it appears in
      ./libs/liburing/src/syscall.c:63:1: warning: control reaches end of non-void function [-Wreturn-type]
         63 | }
            | ^
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for liburing
Failed to build liburing
ERROR: Could not build wheels for liburing, which is required to install pyproject.toml-based projects

Install error on linux 5.4

When I try to install, I get the following error:

pip install --upgrade install git+https://github.com/YoSTEALTH/Liburing
Defaulting to user installation because normal site-packages is not writeable
Collecting git+https://github.com/YoSTEALTH/Liburing
  Cloning https://github.com/YoSTEALTH/Liburing to /tmp/pip-req-build-wn4o9acz
  Running command git clone -q https://github.com/YoSTEALTH/Liburing /tmp/pip-req-build-wn4o9acz
  Resolved https://github.com/YoSTEALTH/Liburing to commit 9c3d8faf7b564a3b189b2bf0ca7dfe3c3cb7e1ac
  Running command git submodule update --init --recursive -q
  Installing build dependencies ... done
  Getting requirements to build wheel ... error
  ERROR: Command errored out with exit status 1:
   command: /usr/bin/python3 /home/mjs/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpl2jd4zxj
       cwd: /tmp/pip-req-build-wn4o9acz
  Complete output (20 lines):
  Traceback (most recent call last):
    File "/home/mjs/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 349, in <module>
      main()
    File "/home/mjs/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 331, in main
      json_out['return_val'] = hook(**hook_input['kwargs'])
    File "/home/mjs/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py", line 117, in get_requires_for_build_wheel
      return hook(config_settings)
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 154, in get_requires_for_build_wheel
      return self._get_build_requires(
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 135, in _get_build_requires
      self.run_setup()
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 258, in run_setup
      super(_BuildMetaLegacyBackend,
    File "/usr/local/lib/python3.8/dist-packages/setuptools/build_meta.py", line 150, in run_setup
      exec(compile(code, __file__, 'exec'), locals())
    File "setup.py", line 3, in <module>
      from liburing import skip_os
    File "/tmp/pip-req-build-wn4o9acz/liburing/__init__.py", line 1, in <module>
      from ._liburing import ffi, lib  # noqa
  ModuleNotFoundError: No module named 'liburing._liburing'
  ----------------------------------------
WARNING: Discarding git+https://github.com/YoSTEALTH/Liburing. Command errored out with exit status 1: /usr/bin/python3 /home/mjs/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpl2jd4zxj Check the logs for full command output.
ERROR: Command errored out with exit status 1: /usr/bin/python3 /home/mjs/.local/lib/python3.8/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmpl2jd4zxj Check the logs for full command output.

and my OS info:

uname -r   
5.4.0-1043-gcp

incomplete requires list and kernel version check suggestion

Wow, Simple File Example is awesome and friendly.

  • requires list may add python3-devel (package name for Fedora, other Linux distros may different) or cause build/temp.linux-x86_64-3.7/liburing._liburing.c:22:12: fatal error: pyconfig.h: No such file or directory

  • pip installing progress not check kernel version 5.5+ as condition or cause build/temp.linux-x86_64-3.7/liburing._liburing.c:5664:12: error: ‘STATX_ATTR_VERITY’ undeclared (first use in this function)

Unable to install the package via pip

What I did?

ran the command pip install liburing

Output

ERROR: Command errored out with exit status 1:
command: /usr/bin/python /usr/lib/python3.8/site-packages/pep517/_in_process.py build_wheel /tmp/tmpct4dcl0c
cwd: /tmp/pip-install-a88v40ja/liburing
Complete output (40 lines):
running bdist_wheel
running build
running build_py
creating build
creating build/lib.linux-x86_64-3.8
creating build/lib.linux-x86_64-3.8/liburing
copying liburing/wrapper.py -> build/lib.linux-x86_64-3.8/liburing
copying liburing/interface.py -> build/lib.linux-x86_64-3.8/liburing
copying liburing/helper.py -> build/lib.linux-x86_64-3.8/liburing
copying liburing/init.py -> build/lib.linux-x86_64-3.8/liburing
running build_ext
generating cffi module 'build/temp.linux-x86_64-3.8/liburing._liburing.c'
creating build/temp.linux-x86_64-3.8
building 'liburing._liburing' extension
creating build/temp.linux-x86_64-3.8/build
creating build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8
creating build/temp.linux-x86_64-3.8/src
gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O3 -Wall -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -march=x86-64 -mtune=generic -O3 -pipe -fno-plt -fPIC -Isrc/include -I/usr/include/python3.8 -c build/temp.linux-x86_64-3.8/liburing._liburing.c -o build/temp.linux-x86_64-3.8/build/temp.linux-x86_64-3.8/liburing._liburing.o
In file included from src/include/liburing.h:17,
from build/temp.linux-x86_64-3.8/liburing._liburing.c:542:
src/include/custom.h:4:8: error: redefinition of ‘struct __kernel_timespec’
4 | struct __kernel_timespec {
| ^~~~~~~~~~~~~~~~~
In file included from /usr/include/liburing/compat.h:5,
from src/include/liburing.h:14,
from build/temp.linux-x86_64-3.8/liburing._liburing.c:542:
/usr/include/linux/time_types.h:7:8: note: originally defined here
7 | struct __kernel_timespec {
| ^~~~~~~~~~~~~~~~~
In file included from src/include/liburing.h:17,
from build/temp.linux-x86_64-3.8/liburing._liburing.c:542:
src/include/custom.h:9:8: error: redefinition of ‘struct open_how’
9 | struct open_how {
| ^~~~~~~~
In file included from src/include/liburing.h:14,
from build/temp.linux-x86_64-3.8/liburing._liburing.c:542:
/usr/include/liburing/compat.h:9:8: note: originally defined here
9 | struct open_how {
| ^~~~~~~~
error: command 'gcc' failed with exit status 1


ERROR: Failed building wheel for liburing

OS

Arch linux 5.5.7-arch1-1

Additional

As the liburing files were not present initially, i had to install them using yay -S liburing

How to create sockaddr for io_uring_prep_connect?

Hello and thank you for making this library. I am trying to use the io_uring_prep_connect function which requires sockaddr* and socklen_t. I see the structs for sockaddr and sockaddr_in src/liburing/lib/socket.pxd and corresponding empty classes in src/liburing/socket.pyx but I can't figure out how to use them. How can I create a sockaddr variable to pass to io_uring_prep_connect?

Core dump on probe and bad file descriptor on read.

installed using the latest version via GitHub.

Python 3.8.10 (default, Jun  2 2021, 10:49:15) 
[GCC 9.4.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import liburing
>>> probe = liburing.probe()
[1]    30282 segmentation fault (core dumped)  python3

I tried to run the full example program to try to catch the trace:

Exception has occurred: OSError
[Errno 9] Bad file descriptor
  File "/home/mjs/test_liburing.py", line 44, in _submit_and_wait
    result = trap_error(cqe.res)  # auto raise appropriate exception if failed
  File "/home/mjs/test_liburing.py", line 12, in open
    return _submit_and_wait(ring, cqes)  # returns fd
  File "/home/mjs/test_liburing.py", line 58, in main
    fd = open(ring, cqes, '~/liburing-test-file.txt', os.O_CREAT | os.O_RDWR)
  File "/home/mjs/test_liburing.py", line 74, in <module>
    main()

PS: I noticed that liburing updated to version 2.1 few hours ago while this is using version 2.0.

Polling mode

Hello! I've been trying to make example code work with IORING_SETUP_SQPOLL but something goes wrong.
Could you have a look pls?

from liburing import *

ring = io_uring()
cqes = io_uring_cqes()
io_uring_queue_init(8, ring, IORING_SETUP_SQPOLL)

path = os.path.abspath('/tmp/liburing-test-file.txt').encode()
sqe = io_uring_get_sqe(ring)
io_uring_prep_openat(sqe, -1, path, os.O_CREAT | os.O_RDWR, 0o660)
io_uring_submit(ring)

io_uring_wait_cqe(ring, cqes)
cqe = cqes[0]
result = trap_error(cqe.res)  # fails with OSError: [Errno 22] Invalid argument

splice from pipe to socket is not allowed?

As a simply test, just call a cli wget http://127.0.0.1:8188/.

Sorry to submit issue here but instead of axboe/liburing, a clang demo is not easy for me.

# based on https://github.com/YoSTEALTH/Liburing/blob/master/test/splice_test.py
# logic: only change the fd_out from a file descriptor to a socket descriptor.
import os
import socket
import liburing

def test_socket_using_splice(tmpdir):
    fd_in = os.open(os.path.join(tmpdir, '1.txt'), os.O_RDWR | os.O_CREAT, 0o660)

    bind = ("127.0.0.1", 8188)
    server = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, True)
    server.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEPORT, True)
    # server.setblocking(False)
    server.bind(bind)
    server.listen(32)

    server.listen()

    conn, addr = server.accept()
    fd_out = conn.fileno()

    flags = liburing.SPLICE_F_MOVE | liburing.SPLICE_F_MORE
    data = b'hello world'
    buf_size = len(data)
    os.write(fd_in, data)

    r, w = os.pipe()

    ring = liburing.io_uring()
    cqes = liburing.io_uring_cqes()

    try:
        # initialization
        assert liburing.io_uring_queue_init(2, ring, 0) == 0

        # read from file "1.txt"
        sqe = liburing.io_uring_get_sqe(ring)
        liburing.io_uring_prep_splice(sqe, fd_in, 0, w, -1, buf_size, flags)
        sqe.user_data = 1

        # chain top and bottom sqe
        sqe.flags |= liburing.IOSQE_IO_LINK

        # write to file "2.txt"
        sqe = liburing.io_uring_get_sqe(ring)
        liburing.io_uring_prep_splice(sqe, r, -1, fd_out, 0, buf_size, flags)
        sqe.user_data = 2

        # submit both
        assert liburing.io_uring_submit(ring) == 2

        # wait for ``2`` entry to complete using single syscall
        assert liburing.io_uring_wait_cqes(ring, cqes, 2) == 0
        cqe = cqes[0]
        assert cqe.res == buf_size
        assert cqe.user_data == 1
        liburing.io_uring_cqe_seen(ring, cqe)

        # re-uses the same resources from above?!
        assert liburing.io_uring_wait_cqes(ring, cqes, 2) == 0
        cqe = cqes[0]
        ###########################################################################################
        #############################
        assert cqe.res == buf_size  # I've got cqe.res -22, means "EINVAL 22 Invalid argument"
        #############################
        ###########################################################################################
        assert cqe.user_data == 2
        liburing.io_uring_cqe_seen(ring, cqe)
        assert os.read(fd_out, buf_size) == data
    finally:
        server.shutdown(socket.SHUT_RDWR)
        server.close()
        os.close(fd_in)
        liburing.io_uring_queue_exit(ring)


if "__main__"  == __name__:
    test_socket_using_splice("/dev/shm")

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.