mosquito / caio Goto Github PK

View Code? Open in Web Editor NEW

55.0 4.0 10.0 1.32 MB

Linux AIO c python bindings

License: Apache License 2.0

Python 35.71% C 63.12% Makefile 0.70% Shell 0.47%

linux aio eventfd linux-aio asyncio non-blocking system kernel

caio's Introduction

Python wrapper for AIO

Warning

Native Linux aio implementation supports since 4.18 kernel version.

Python bindings for Linux AIO API and simple asyncio wrapper.

Example

import asyncio
from caio import AsyncioContext

loop = asyncio.get_event_loop()

async def main():
    # max_requests=128 by default
    ctx = AsyncioContext(max_requests=128)

    with open("test.file", "wb+") as fp:
        fd = fp.fileno()

        # Execute one write operation
        await ctx.write(b"Hello world", fd, offset=0)

        # Execute one read operation
        print(await ctx.read(32, fd, offset=0))

        # Execute one fdsync operation
        await ctx.fdsync(fd)

        op1 = ctx.write(b"Hello from ", fd, offset=0)
        op2 = ctx.write(b"async world", fd, offset=11)

        await asyncio.gather(op1, op2)

        print(await ctx.read(32, fd, offset=0))
        # Hello from async world


loop.run_until_complete(main())

Troubleshooting

The linux implementation works normal for modern linux kernel versions and file systems. So you may have problems specific for your environment. It's not a bug and might be resolved some ways:

Upgrade the kernel
Use compatible file system
Use threads based or pure python implementation.

The caio since version 0.7.0 contains some ways to do this.

1. In runtime use the environment variable CAIO_IMPL with possible values:

linux - use native linux kernels aio mechanism
thread - use thread based implementation written in C
python - use pure python implementation

2. File default_implementation located near __init__.py in caio installation path. It's useful for distros package maintainers. This file might contains comments (lines starts with # symbol) and the first line should be one of linux thread or python.

Previous versions allows direct import of the target implementation.

caio's People

Contributors

Stargazers

Watchers

Forkers

atugushev ei-grad gaod dizballanze shaunc tree4096 h4l e7dal arpitjain799 itsmehdi97

caio's Issues

PyPI sdist doesnt contain LICENSE

Failed to write more than 2Gb to file

Long story short

Failed to write to file larger than 2Gb

Expected behavior

Succsessfully write and read files larger than 2Gb

Actual behavior

Write to file with offset larger than 2^31 results in exception:

Traceback (most recent call last):
  File "/tmp/check_write_2GB.py", line 21, in <module>
    asyncio.run(main())
  File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "/tmp/check_write_2GB.py", line 18, in main
    await save_data_to_file(dst)
  File "/tmp/check_write_2GB.py", line 11, in save_data_to_file
    await dst_write(data)
  File "/home/aamaslov/.local/lib/python3.8/site-packages/aiofile/utils.py", line 217, in write
    await operation
  File "/home/aamaslov/.local/lib/python3.8/site-packages/aiofile/aio.py", line 242, in write_bytes
    res = await self.__context.write(
  File "/home/aamaslov/.local/lib/python3.8/site-packages/caio/asyncio_base.py", line 88, in submit
    return op.get_value()
SystemError: Invalid argument

Steps to reproduce

Python script check_write_2GB.py:

import asyncio
import aiofile


async def save_data_to_file(dst):
    length = 8 * 1024
    data = b'0' * length
    dst_write = dst.write
    limit = 2049  * 1024 * 1024

    while limit > 0:
        await dst_write(data)
        limit -= len(data)


async def main():
    import sys
    async with aiofile.async_open(sys.argv[1], 'wb') as dst:
        await save_data_to_file(dst)


asyncio.run(main())

~ python3 check_write_2GB.py

Environment info

Kernel version: Linux hostname 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
File system: ext4

I have been produced this problem with implementations:

export CAIO_IMPL=linux - Native linux implementation
export CAIO_IMPL=thread - Thread implementation
export CAIO_IMPL=python - Pure Python implementation

Additional info

strace gives this one for succsessfull writes:
[pid 42589] pwrite64(7, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192, 2147475456) = 8192
And this for failed one:
[pid 42592] pwrite64(7, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192, -2147483648) = -1 EINVAL (Invalid argument)

Looks like offset variable overflows in https://github.com/mosquito/caio/blob/master/caio/thread_aio.c#L159

CPython optimize flags cause readline to hang

Long story short

Read line coroutine stalls if Python interpreter is started with optimization flags

Expected behavior

Program works the same regardless of optimization flags

Actual behavior

Program stalls

Steps to reproduce

Given really simple example:

import asyncio
from aiofile import async_open
async def main() :
    async with async_open("/etc/passwd", "rb") as fp:
        print("File opened")
        print(await fp.readline())
asyncio.run(main())

The program will hang at fp.readline() if started with Python 3 interpreter using optimize flag -O or -OO

Environment info

Kernel version: Linux lauri-x13 5.15.0-37-generic #39-Ubuntu SMP Wed Jun 1 19:16:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
File system: ZFS

I have been produced this problem with implementations:

[yes] export CAIO_IMPL=linux - Native linux implementation
[yes] export CAIO_IMPL=thread - Thread implementation
[yes] export CAIO_IMPL=python - Pure Python implementation

Additional info

I also tried inside Docker image with python:alpine on XFS and the case is same there

Trouble porting my code to caio

Hi, I am having some trouble using caio in my project. I am pretty new to async so I may be doing something stupid 😅

I tried moving my code to async since it can be a bit IO intensive and lower delays do make a difference in my application.

WIP patch: FFY00/python-uhid@b2b49aa
Code: https://github.com/FFY00/python-uhid/tree/async

I am testing with the following snippet:

#!/usr/bin/env python

import uhid

async def create(n = 0):
    return await uhid.UHIDDevice.initialize(
            0x0000, n, f'Test Device {n}', [
            # Generic mouse report descriptor
            0x05, 0x01,  # Usage Page (Generic Desktop)        0
            0x09, 0x02,  # Usage (Mouse)                       2
            0xa1, 0x01,  # Collection (Application)            4
            0x09, 0x02,  # .Usage (Mouse)                      6
            0xa1, 0x02,  # .Collection (Logical)               8
            0x09, 0x01,  # ..Usage (Pointer)                   10
            0xa1, 0x00,  # ..Collection (Physical)             12
            0x05, 0x09,  # ...Usage Page (Button)              14
            0x19, 0x01,  # ...Usage Minimum (1)                16
            0x29, 0x03,  # ...Usage Maximum (3)                18
            0x15, 0x00,  # ...Logical Minimum (0)              20
            0x25, 0x01,  # ...Logical Maximum (1)              22
            0x75, 0x01,  # ...Report Size (1)                  24
            0x95, 0x03,  # ...Report Count (3)                 26
            0x81, 0x02,  # ...Input (Data,Var,Abs)             28
            0x75, 0x05,  # ...Report Size (5)                  30
            0x95, 0x01,  # ...Report Count (1)                 32
            0x81, 0x03,  # ...Input (Cnst,Var,Abs)             34
            0x05, 0x01,  # ...Usage Page (Generic Desktop)     36
            0x09, 0x30,  # ...Usage (X)                        38
            0x09, 0x31,  # ...Usage (Y)                        40
            0x15, 0x81,  # ...Logical Minimum (-127)           42
            0x25, 0x7f,  # ...Logical Maximum (127)            44
            0x75, 0x08,  # ...Report Size (8)                  46
            0x95, 0x02,  # ...Report Count (2)                 48
            0x81, 0x06,  # ...Input (Data,Var,Rel)             50
            0xc0,        # ..End Collection                    52
            0xc0,        # .End Collection                     53
            0xc0,        # End Collection                      54
        ],
    )


async def main():
    wait = []
    for i in range(5):
        wait.append(create(i))

    await asyncio.gather(*wait)


import asyncio
loop = asyncio.get_event_loop()
#loop.run_until_complete(main())
loop.run_until_complete(create())

But I get the following error:

Traceback (most recent call last):
  File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 43, in step
    self.context.submit(*operations.keys())
SystemError: (22, 'Invalid argument')

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "./test.py", line 56, in <module>
    loop.run_until_complete(create())
  File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
    return future.result()
  File "./test.py", line 9, in create
    return await uhid.UHIDDevice.initialize(
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 239, in initialize
    await device.create()
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 316, in create
    await self._uhid.create(
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 220, in create
    await self._send_event(_EventType.UHID_CREATE2, _Create2Req(
  File "/home/anubis/git/python-uhid/uhid/__init__.py", line 186, in _send_event
    n = await self._ctx.write(bytes(data), self._uhid, offset=0)
  File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 87, in submit
    await future
  File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 48, in step
    self.context.submit(operation)
SystemError: (22, 'Invalid argument')

Do you have any idea of what may be happening here?

Mac build fails due to fdatasync

Long story short

I'm able to build this manually using python setup.py bdist_wheel.
However, when using Nix I get this error:

$ nix build nixpkgs#python310Packages.caio
...
caio/thread_aio.c:173:22: error: call to undeclared function 'fdatasync'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
            result = fdatasync(fileno);
                     ^
...

From what I've been reading it seems like Mac OS doesn't have fdatasync, e.g. my editor using clangd can't find the definition.

On the other hand I tried adding -Wimplicit-function-declaration to setup.py, but don't see any extra warnings.

Expected behavior

Build should succeed.

Actual behavior

Build fails.

Steps to reproduce

See above (nix build nixpkgs#python310Packages.caio).

Environment info

Kernel version: Darwin HOSTNAME 22.6.0 Darwin Kernel Version 22.6.0: Tue Nov 7 21:40:08 PST 2023; root:xnu-8796.141.3.702.9~2/RELEASE_ARM64_T6000 arm64
File system: APFS

I have been produced this problem with implementations:

export CAIO_IMPL=linux - Native linux implementation
export CAIO_IMPL=thread - Thread implementation
export CAIO_IMPL=python - Pure Python implementation

Additional info

The nixpkgs commit I'm using is b06ff4bf8f4ad900fe0c2a61fc2946edc3a84be7.
Also caio version is 0.9.13.

i586 Bad address

I am packaging this for openSUSE.

Then on i586 only, I get an error

[   37s] ============================= test session starts ==============================
[   37s] platform linux -- Python 3.6.12, pytest-6.2.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3.6
[   37s] cachedir: .pytest_cache
[   37s] rootdir: /home/abuild/rpmbuild/BUILD/caio-0.6.3
[   37s] plugins: aiomisc-12.0.0
[   37s] collecting ... collected 9 items
[   37s] 
[   37s] tests/test_aio_context.py::test_aio_context[caio.linux_aio] PASSED       [ 11%]
[   37s] tests/test_aio_context.py::test_aio_context[caio.thread_aio] PASSED      [ 22%]
[   37s] tests/test_aio_context.py::test_aio_context[caio.python_aio] PASSED      [ 33%]
[   37s] tests/test_asyncio_adapter.py::test_adapter[caio.linux_aio_asyncio] FAILED [ 44%]
[   38s] tests/test_asyncio_adapter.py::test_adapter[caio.thread_aio_asyncio] PASSED [ 55%]
[   38s] tests/test_asyncio_adapter.py::test_adapter[caio.python_aio_asyncio] PASSED [ 66%]
[   38s] tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.linux_aio_asyncio] PASSED [ 77%]
[   38s] tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.thread_aio_asyncio] PASSED [ 88%]
[   38s] tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.python_aio_asyncio] PASSED [100%]
[   38s] 
[   38s] =================================== FAILURES ===================================
[   38s] _____________________ test_adapter[caio.linux_aio_asyncio] _____________________
[   38s] 
[   38s] first_operation = <aio.AIOOperation at 0xf6714d40: mode="read", fd=15, offset=0, buffer=0x(nil)>
[   38s] first_future = <Future finished exception=SystemError(14, 'Bad address')>
[   38s] 
[   38s]     def step(first_operation, first_future):
[   38s]         operations = {first_operation: first_future}
[   38s]     
[   38s]         while True:
[   38s]             try:
[   38s]                 operation, future = self.operations_queue.get_nowait()
[   38s]                 operations[operation] = future
[   38s]             except asyncio.QueueEmpty:
[   38s]                 break
[   38s]     
[   38s]         try:
[   38s]             # Fast call
[   38s] >           self.context.submit(*operations.keys())
[   38s] E           SystemError: (14, 'Bad address')
[   38s] 
[   38s] ../../BUILDROOT/python-caio-0.6.3-8.1.i386/usr/lib/python3.6/site-packages/caio/asyncio_base.py:43: SystemError
[   38s] 
[   38s] During handling of the above exception, another exception occurred:
[   38s] 
[   38s] tmp_path = PosixPath('/tmp/pytest-of-abuild/pytest-0/test_adapter_caio_linux_aio_as0')
[   38s] async_context_maker = <class 'caio.linux_aio_asyncio.AsyncioContext'>
[   38s] 
[   38s]     @aiomisc.timeout(5)
[   38s]     async def test_adapter(tmp_path, async_context_maker):
[   38s]         async with async_context_maker() as context:
[   38s]             with open(str(tmp_path / "temp.bin"), "wb+") as fp:
[   38s]                 fd = fp.fileno()
[   38s]     
[   38s] >               assert await context.read(32, fd, 0) == b""
[   38s] 
[   38s] tests/test_asyncio_adapter.py:14: 
[   38s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   38s] ../../BUILDROOT/python-caio-0.6.3-8.1.i386/usr/lib/python3.6/site-packages/caio/asyncio_base.py:87: in submit
[   38s]     await future
[   38s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
[   38s] 
[   38s] first_operation = <aio.AIOOperation at 0xf6714d40: mode="read", fd=15, offset=0, buffer=0x(nil)>
[   38s] first_future = <Future finished exception=SystemError(14, 'Bad address')>
[   38s] 
[   38s]     def step(first_operation, first_future):
[   38s]         operations = {first_operation: first_future}
[   38s]     
[   38s]         while True:
[   38s]             try:
[   38s]                 operation, future = self.operations_queue.get_nowait()
[   38s]                 operations[operation] = future
[   38s]             except asyncio.QueueEmpty:
[   38s]                 break
[   38s]     
[   38s]         try:
[   38s]             # Fast call
[   38s]             self.context.submit(*operations.keys())
[   38s]         except Exception:
[   38s]             # Fallback
[   38s]             for operation, future in operations.items():
[   38s]                 try:
[   38s] >                   self.context.submit(operation)
[   38s] E                   SystemError: (14, 'Bad address')
[   38s] 
[   38s] ../../BUILDROOT/python-caio-0.6.3-8.1.i386/usr/lib/python3.6/site-packages/caio/asyncio_base.py:48: SystemError
[   38s] =========================== short test summary info ============================
[   38s] FAILED tests/test_asyncio_adapter.py::test_adapter[caio.linux_aio_asyncio] - ...
[   38s] ========================= 1 failed, 8 passed in 0.42s ==========================

https://build.opensuse.org/package/show/home:jayvdb:py-new/python-caio

Missing loop fixture with aiomisc 17

Long story short

loop fixture missing with aiomisc 17

Expected behavior

Tests should succeed.

Actual behavior

caio> ==================================== ERRORS ====================================
caio> __ ERROR at setup of test_operations_cancel_cleanly[caio.python_aio_asyncio] ___
caio> file /build/source/tests/test_asyncio_adapter.py, line 65
caio>   @aiomisc.timeout(3)
caio>   async def test_operations_cancel_cleanly(
caio>       tmp_path, async_context_maker, asyncio_exception_handler
caio>   ):
caio>       async with async_context_maker() as context:
caio>           with open(str(tmp_path / "temp.bin"), "wb+") as fp:
caio>               fd = fp.fileno()
caio> 
caio>               await context.write(b"\x00", fd, 1024**2 - 1)
caio>               assert os.stat(fd).st_size == 1024**2
caio> 
caio>               for _ in range(50):
caio>                   reads = [
caio>                       asyncio.create_task(context.read(2**16, fd, 2**16 * i))
caio>                       for i in range(16)
caio>                   ]
caio>                   _, pending = await asyncio.wait(
caio>                       reads, return_when=asyncio.FIRST_COMPLETED
caio>                   )
caio>                   for read in pending:
caio>                       read.cancel()
caio>                   if pending:
caio>                       await asyncio.wait(pending)
caio>                   asyncio_exception_handler.assert_not_called()
caio> file /build/source/tests/test_asyncio_adapter.py, line 54
caio>   @pytest.fixture
caio>   async def asyncio_exception_handler(loop):
caio>       handler = Mock(
caio>           side_effect=lambda _loop, ctx: _loop.default_exception_handler(ctx)
caio>       )
caio>       current_handler = loop.get_exception_handler()
caio>       loop.set_exception_handler(handler=handler)
caio>       yield handler
caio>       loop.set_exception_handler(current_handler)
caio> E       fixture 'loop' not found
caio> >       available fixtures: async_context_maker, asyncio_exception_handler, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, context_maker, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
caio> >       use 'pytest --fixtures [testpath]' for help on them.
caio> 
caio> /build/source/tests/test_asyncio_adapter.py:54
caio> =============================== warnings summary ===============================
caio> tests/test_asyncio_adapter.py::test_adapter[caio.python_aio_asyncio]
caio> tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.python_aio_asyncio]
caio>   /nix/store/ma77c9gm2zbikjjgf3i6hs3rp9i13f2r-python3.10-pytest-7.2.1/lib/python3.10/site-packages/_pytest/python.py:184: PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
caio>   You need to install a suitable plugin for your async framework, for example:
caio>     - anyio
caio>     - pytest-asyncio
caio>     - pytest-tornasync
caio>     - pytest-trio
caio>     - pytest-twisted
caio>     warnings.warn(PytestUnhandledCoroutineWarning(msg.format(nodeid)))
caio> 
caio> -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
caio> =========================== short test summary info ============================
caio> ERROR tests/test_asyncio_adapter.py::test_operations_cancel_cleanly[caio.python_aio_asyncio]

Steps to reproduce

Install aiomisc~=17
Run the test suite

Environment info

platform linux -- Python 3.10.10, pytest-7.2.1, pluggy-1.0.0

and aiomisc 17.0.6

Additional info

linux_aio : ImportError: Error on io_setup with code 22

Long story short

Trying to run the CAIO_IMPL=linux version unsuccessfully. I see the required 'linux_aio' and 'linux_aio_asynio' are present, along with 'inux_aio.cpython-38-x86_64-linux-gnu.so'. Trying to import 'linux_aio' produces this error:

ImportError: Error on io_setup with code 22

Expected behavior

Expected the linux version to work

Actual behavior

I get the erro message above.

Steps to reproduce

Environment info

Linux deepdell 5.15.0-43-generic #46~20.04.1-Ubuntu SMP Thu Jul 14 15:20:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
File system: ext4

I have been produced this problem with implementations:

[] export CAIO_IMPL=linux - Native linux implementation

Additional info

Ubuntu 20.04 LTS. I've tried installing the packages libaio1, libaio-dev, build-essential, same error.

mosquito / caio Goto Github PK

caio's Introduction

Python wrapper for AIO

Example

Troubleshooting

caio's People

Contributors

Stargazers

Watchers

Forkers

caio's Issues

Long story short

Expected behavior

Actual behavior

Steps to reproduce

Environment info

Additional info

Long story short

Expected behavior

Actual behavior

Steps to reproduce

Environment info

Additional info

Long story short

Expected behavior

Actual behavior

Steps to reproduce

Environment info

Additional info

Long story short

Expected behavior

Actual behavior

Steps to reproduce

Environment info

Additional info

Long story short

Expected behavior

Actual behavior

Steps to reproduce

Environment info

Additional info

Recommend Projects

Recommend Topics

Recommend Org