mosquito / caio Goto Github PK
View Code? Open in Web Editor NEWLinux AIO c python bindings
License: Apache License 2.0
Linux AIO c python bindings
License: Apache License 2.0
Failed to write to file larger than 2Gb
Succsessfully write and read files larger than 2Gb
Write to file with offset larger than 2^31 results in exception:
Traceback (most recent call last):
File "/tmp/check_write_2GB.py", line 21, in <module>
asyncio.run(main())
File "/usr/lib/python3.8/asyncio/runners.py", line 43, in run
return loop.run_until_complete(main)
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "/tmp/check_write_2GB.py", line 18, in main
await save_data_to_file(dst)
File "/tmp/check_write_2GB.py", line 11, in save_data_to_file
await dst_write(data)
File "/home/aamaslov/.local/lib/python3.8/site-packages/aiofile/utils.py", line 217, in write
await operation
File "/home/aamaslov/.local/lib/python3.8/site-packages/aiofile/aio.py", line 242, in write_bytes
res = await self.__context.write(
File "/home/aamaslov/.local/lib/python3.8/site-packages/caio/asyncio_base.py", line 88, in submit
return op.get_value()
SystemError: Invalid argument
Python script check_write_2GB.py:
import asyncio
import aiofile
async def save_data_to_file(dst):
length = 8 * 1024
data = b'0' * length
dst_write = dst.write
limit = 2049 * 1024 * 1024
while limit > 0:
await dst_write(data)
limit -= len(data)
async def main():
import sys
async with aiofile.async_open(sys.argv[1], 'wb') as dst:
await save_data_to_file(dst)
asyncio.run(main())
~ python3 check_write_2GB.py
Kernel version: Linux hostname 5.4.0-72-generic #80-Ubuntu SMP Mon Apr 12 17:35:00 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
File system: ext4
I have been produced this problem with implementations:
export CAIO_IMPL=linux
- Native linux implementationexport CAIO_IMPL=thread
- Thread implementationexport CAIO_IMPL=python
- Pure Python implementationstrace gives this one for succsessfull writes:
[pid 42589] pwrite64(7, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192, 2147475456) = 8192
And this for failed one:
[pid 42592] pwrite64(7, "\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0\0"..., 8192, -2147483648) = -1 EINVAL (Invalid argument)
Looks like offset variable overflows in https://github.com/mosquito/caio/blob/master/caio/thread_aio.c#L159
Hi, I am having some trouble using caio in my project. I am pretty new to async so I may be doing something stupid ๐
I tried moving my code to async since it can be a bit IO intensive and lower delays do make a difference in my application.
WIP patch: FFY00/python-uhid@b2b49aa
Code: https://github.com/FFY00/python-uhid/tree/async
I am testing with the following snippet:
#!/usr/bin/env python
import uhid
async def create(n = 0):
return await uhid.UHIDDevice.initialize(
0x0000, n, f'Test Device {n}', [
# Generic mouse report descriptor
0x05, 0x01, # Usage Page (Generic Desktop) 0
0x09, 0x02, # Usage (Mouse) 2
0xa1, 0x01, # Collection (Application) 4
0x09, 0x02, # .Usage (Mouse) 6
0xa1, 0x02, # .Collection (Logical) 8
0x09, 0x01, # ..Usage (Pointer) 10
0xa1, 0x00, # ..Collection (Physical) 12
0x05, 0x09, # ...Usage Page (Button) 14
0x19, 0x01, # ...Usage Minimum (1) 16
0x29, 0x03, # ...Usage Maximum (3) 18
0x15, 0x00, # ...Logical Minimum (0) 20
0x25, 0x01, # ...Logical Maximum (1) 22
0x75, 0x01, # ...Report Size (1) 24
0x95, 0x03, # ...Report Count (3) 26
0x81, 0x02, # ...Input (Data,Var,Abs) 28
0x75, 0x05, # ...Report Size (5) 30
0x95, 0x01, # ...Report Count (1) 32
0x81, 0x03, # ...Input (Cnst,Var,Abs) 34
0x05, 0x01, # ...Usage Page (Generic Desktop) 36
0x09, 0x30, # ...Usage (X) 38
0x09, 0x31, # ...Usage (Y) 40
0x15, 0x81, # ...Logical Minimum (-127) 42
0x25, 0x7f, # ...Logical Maximum (127) 44
0x75, 0x08, # ...Report Size (8) 46
0x95, 0x02, # ...Report Count (2) 48
0x81, 0x06, # ...Input (Data,Var,Rel) 50
0xc0, # ..End Collection 52
0xc0, # .End Collection 53
0xc0, # End Collection 54
],
)
async def main():
wait = []
for i in range(5):
wait.append(create(i))
await asyncio.gather(*wait)
import asyncio
loop = asyncio.get_event_loop()
#loop.run_until_complete(main())
loop.run_until_complete(create())
But I get the following error:
Traceback (most recent call last):
File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 43, in step
self.context.submit(*operations.keys())
SystemError: (22, 'Invalid argument')
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "./test.py", line 56, in <module>
loop.run_until_complete(create())
File "/usr/lib/python3.8/asyncio/base_events.py", line 616, in run_until_complete
return future.result()
File "./test.py", line 9, in create
return await uhid.UHIDDevice.initialize(
File "/home/anubis/git/python-uhid/uhid/__init__.py", line 239, in initialize
await device.create()
File "/home/anubis/git/python-uhid/uhid/__init__.py", line 316, in create
await self._uhid.create(
File "/home/anubis/git/python-uhid/uhid/__init__.py", line 220, in create
await self._send_event(_EventType.UHID_CREATE2, _Create2Req(
File "/home/anubis/git/python-uhid/uhid/__init__.py", line 186, in _send_event
n = await self._ctx.write(bytes(data), self._uhid, offset=0)
File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 87, in submit
await future
File "/home/anubis/.virtualenvs/caio/lib/python3.8/site-packages/caio/asyncio_base.py", line 48, in step
self.context.submit(operation)
SystemError: (22, 'Invalid argument')
Do you have any idea of what may be happening here?
I am packaging this for openSUSE.
Then on i586 only, I get an error
[ 37s] ============================= test session starts ==============================
[ 37s] platform linux -- Python 3.6.12, pytest-6.2.2, py-1.9.0, pluggy-0.13.1 -- /usr/bin/python3.6
[ 37s] cachedir: .pytest_cache
[ 37s] rootdir: /home/abuild/rpmbuild/BUILD/caio-0.6.3
[ 37s] plugins: aiomisc-12.0.0
[ 37s] collecting ... collected 9 items
[ 37s]
[ 37s] tests/test_aio_context.py::test_aio_context[caio.linux_aio] PASSED [ 11%]
[ 37s] tests/test_aio_context.py::test_aio_context[caio.thread_aio] PASSED [ 22%]
[ 37s] tests/test_aio_context.py::test_aio_context[caio.python_aio] PASSED [ 33%]
[ 37s] tests/test_asyncio_adapter.py::test_adapter[caio.linux_aio_asyncio] FAILED [ 44%]
[ 38s] tests/test_asyncio_adapter.py::test_adapter[caio.thread_aio_asyncio] PASSED [ 55%]
[ 38s] tests/test_asyncio_adapter.py::test_adapter[caio.python_aio_asyncio] PASSED [ 66%]
[ 38s] tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.linux_aio_asyncio] PASSED [ 77%]
[ 38s] tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.thread_aio_asyncio] PASSED [ 88%]
[ 38s] tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.python_aio_asyncio] PASSED [100%]
[ 38s]
[ 38s] =================================== FAILURES ===================================
[ 38s] _____________________ test_adapter[caio.linux_aio_asyncio] _____________________
[ 38s]
[ 38s] first_operation = <aio.AIOOperation at 0xf6714d40: mode="read", fd=15, offset=0, buffer=0x(nil)>
[ 38s] first_future = <Future finished exception=SystemError(14, 'Bad address')>
[ 38s]
[ 38s] def step(first_operation, first_future):
[ 38s] operations = {first_operation: first_future}
[ 38s]
[ 38s] while True:
[ 38s] try:
[ 38s] operation, future = self.operations_queue.get_nowait()
[ 38s] operations[operation] = future
[ 38s] except asyncio.QueueEmpty:
[ 38s] break
[ 38s]
[ 38s] try:
[ 38s] # Fast call
[ 38s] > self.context.submit(*operations.keys())
[ 38s] E SystemError: (14, 'Bad address')
[ 38s]
[ 38s] ../../BUILDROOT/python-caio-0.6.3-8.1.i386/usr/lib/python3.6/site-packages/caio/asyncio_base.py:43: SystemError
[ 38s]
[ 38s] During handling of the above exception, another exception occurred:
[ 38s]
[ 38s] tmp_path = PosixPath('/tmp/pytest-of-abuild/pytest-0/test_adapter_caio_linux_aio_as0')
[ 38s] async_context_maker = <class 'caio.linux_aio_asyncio.AsyncioContext'>
[ 38s]
[ 38s] @aiomisc.timeout(5)
[ 38s] async def test_adapter(tmp_path, async_context_maker):
[ 38s] async with async_context_maker() as context:
[ 38s] with open(str(tmp_path / "temp.bin"), "wb+") as fp:
[ 38s] fd = fp.fileno()
[ 38s]
[ 38s] > assert await context.read(32, fd, 0) == b""
[ 38s]
[ 38s] tests/test_asyncio_adapter.py:14:
[ 38s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 38s] ../../BUILDROOT/python-caio-0.6.3-8.1.i386/usr/lib/python3.6/site-packages/caio/asyncio_base.py:87: in submit
[ 38s] await future
[ 38s] _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
[ 38s]
[ 38s] first_operation = <aio.AIOOperation at 0xf6714d40: mode="read", fd=15, offset=0, buffer=0x(nil)>
[ 38s] first_future = <Future finished exception=SystemError(14, 'Bad address')>
[ 38s]
[ 38s] def step(first_operation, first_future):
[ 38s] operations = {first_operation: first_future}
[ 38s]
[ 38s] while True:
[ 38s] try:
[ 38s] operation, future = self.operations_queue.get_nowait()
[ 38s] operations[operation] = future
[ 38s] except asyncio.QueueEmpty:
[ 38s] break
[ 38s]
[ 38s] try:
[ 38s] # Fast call
[ 38s] self.context.submit(*operations.keys())
[ 38s] except Exception:
[ 38s] # Fallback
[ 38s] for operation, future in operations.items():
[ 38s] try:
[ 38s] > self.context.submit(operation)
[ 38s] E SystemError: (14, 'Bad address')
[ 38s]
[ 38s] ../../BUILDROOT/python-caio-0.6.3-8.1.i386/usr/lib/python3.6/site-packages/caio/asyncio_base.py:48: SystemError
[ 38s] =========================== short test summary info ============================
[ 38s] FAILED tests/test_asyncio_adapter.py::test_adapter[caio.linux_aio_asyncio] - ...
[ 38s] ========================= 1 failed, 8 passed in 0.42s ==========================
https://build.opensuse.org/package/show/home:jayvdb:py-new/python-caio
Read line coroutine stalls if Python interpreter is started with optimization flags
Program works the same regardless of optimization flags
Program stalls
Given really simple example:
import asyncio
from aiofile import async_open
async def main() :
async with async_open("/etc/passwd", "rb") as fp:
print("File opened")
print(await fp.readline())
asyncio.run(main())
The program will hang at fp.readline()
if started with Python 3 interpreter using optimize flag -O
or -OO
Kernel version: Linux lauri-x13 5.15.0-37-generic #39-Ubuntu SMP Wed Jun 1 19:16:45 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
File system: ZFS
I have been produced this problem with implementations:
export CAIO_IMPL=linux
- Native linux implementationexport CAIO_IMPL=thread
- Thread implementationexport CAIO_IMPL=python
- Pure Python implementationI also tried inside Docker image with python:alpine
on XFS and the case is same there
loop
fixture missing with aiomisc 17
Tests should succeed.
caio> ==================================== ERRORS ====================================
caio> __ ERROR at setup of test_operations_cancel_cleanly[caio.python_aio_asyncio] ___
caio> file /build/source/tests/test_asyncio_adapter.py, line 65
caio> @aiomisc.timeout(3)
caio> async def test_operations_cancel_cleanly(
caio> tmp_path, async_context_maker, asyncio_exception_handler
caio> ):
caio> async with async_context_maker() as context:
caio> with open(str(tmp_path / "temp.bin"), "wb+") as fp:
caio> fd = fp.fileno()
caio>
caio> await context.write(b"\x00", fd, 1024**2 - 1)
caio> assert os.stat(fd).st_size == 1024**2
caio>
caio> for _ in range(50):
caio> reads = [
caio> asyncio.create_task(context.read(2**16, fd, 2**16 * i))
caio> for i in range(16)
caio> ]
caio> _, pending = await asyncio.wait(
caio> reads, return_when=asyncio.FIRST_COMPLETED
caio> )
caio> for read in pending:
caio> read.cancel()
caio> if pending:
caio> await asyncio.wait(pending)
caio> asyncio_exception_handler.assert_not_called()
caio> file /build/source/tests/test_asyncio_adapter.py, line 54
caio> @pytest.fixture
caio> async def asyncio_exception_handler(loop):
caio> handler = Mock(
caio> side_effect=lambda _loop, ctx: _loop.default_exception_handler(ctx)
caio> )
caio> current_handler = loop.get_exception_handler()
caio> loop.set_exception_handler(handler=handler)
caio> yield handler
caio> loop.set_exception_handler(current_handler)
caio> E fixture 'loop' not found
caio> > available fixtures: async_context_maker, asyncio_exception_handler, cache, capfd, capfdbinary, caplog, capsys, capsysbinary, context_maker, doctest_namespace, monkeypatch, pytestconfig, record_property, record_testsuite_property, record_xml_attribute, recwarn, tmp_path, tmp_path_factory, tmpdir, tmpdir_factory
caio> > use 'pytest --fixtures [testpath]' for help on them.
caio>
caio> /build/source/tests/test_asyncio_adapter.py:54
caio> =============================== warnings summary ===============================
caio> tests/test_asyncio_adapter.py::test_adapter[caio.python_aio_asyncio]
caio> tests/test_asyncio_adapter.py::test_bad_file_descritor[caio.python_aio_asyncio]
caio> /nix/store/ma77c9gm2zbikjjgf3i6hs3rp9i13f2r-python3.10-pytest-7.2.1/lib/python3.10/site-packages/_pytest/python.py:184: PytestUnhandledCoroutineWarning: async def functions are not natively supported and have been skipped.
caio> You need to install a suitable plugin for your async framework, for example:
caio> - anyio
caio> - pytest-asyncio
caio> - pytest-tornasync
caio> - pytest-trio
caio> - pytest-twisted
caio> warnings.warn(PytestUnhandledCoroutineWarning(msg.format(nodeid)))
caio>
caio> -- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
caio> =========================== short test summary info ============================
caio> ERROR tests/test_asyncio_adapter.py::test_operations_cancel_cleanly[caio.python_aio_asyncio]
platform linux -- Python 3.10.10, pytest-7.2.1, pluggy-1.0.0
and aiomisc 17.0.6
Trying to run the CAIO_IMPL=linux version unsuccessfully. I see the required 'linux_aio' and 'linux_aio_asynio' are present, along with 'inux_aio.cpython-38-x86_64-linux-gnu.so'. Trying to import 'linux_aio' produces this error:
ImportError: Error on io_setup with code 22
Expected the linux version to work
I get the erro message above.
Linux deepdell 5.15.0-43-generic #46~20.04.1-Ubuntu SMP Thu Jul 14 15:20:17 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
File system: ext4
I have been produced this problem with implementations:
export CAIO_IMPL=linux
- Native linux implementationUbuntu 20.04 LTS. I've tried installing the packages libaio1, libaio-dev, build-essential, same error.
I'm able to build this manually using python setup.py bdist_wheel
.
However, when using Nix I get this error:
$ nix build nixpkgs#python310Packages.caio
...
caio/thread_aio.c:173:22: error: call to undeclared function 'fdatasync'; ISO C99 and later do not support implicit function declarations [-Wimplicit-function-declaration]
result = fdatasync(fileno);
^
...
From what I've been reading it seems like Mac OS doesn't have fdatasync
, e.g. my editor using clangd
can't find the definition.
On the other hand I tried adding -Wimplicit-function-declaration
to setup.py
, but don't see any extra warnings.
Build should succeed.
Build fails.
See above (nix build nixpkgs#python310Packages.caio
).
Kernel version: Darwin HOSTNAME 22.6.0 Darwin Kernel Version 22.6.0: Tue Nov 7 21:40:08 PST 2023; root:xnu-8796.141.3.702.9~2/RELEASE_ARM64_T6000 arm64
File system: APFS
I have been produced this problem with implementations:
export CAIO_IMPL=linux
- Native linux implementationexport CAIO_IMPL=thread
- Thread implementationexport CAIO_IMPL=python
- Pure Python implementationThe nixpkgs
commit I'm using is b06ff4bf8f4ad900fe0c2a61fc2946edc3a84be7
.
Also caio
version is 0.9.13
.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.