antocuni / capnpy Goto Github PK

One cool featured that would be nice to have would be if capnpy would automatically generate Python type annotations.

Relevant links:

• http://mypy-lang.org/
• https://www.python.org/dev/peps/pep-0484/
• https://www.python.org/dev/peps/pep-0561/

At the moment the only way I can see to check if a capnpy object admits checking equality is to try and then catch the TypeError raised by _key. Is there/could there be a better way to do this?

Namedtuple has the method _asdict() and pycapnp also has to_dict(). I think we should also add something equivalent to capnpy, which promptly converts a struct into OrderedDict.

I have two schemas:

• schema_base.capnp

@0xf84075a2d437788c;

enum TestEnum {
	abc @0;
	def @1;
}

• schema_child.capnp

@0xb11cd9d47eecc202;

using Base = import "/schema_base.capnp";

struct TestEnumUser {
	test @0 :Base.TestEnum;
}

where schema_child imports schema_base and makes use of the defined TestEnum.

Trying to initalise a TestEnumUser then raises an AttributeError:

$ python
>>> import schema_base
>>> import schema_child
>>> schema_child.TestEnumUser(test=schema_base.TestEnum.abc)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "capnpy/struct_.py", line 92, in capnpy.struct_.Struct.__repr__ (capnpy/struct_.c:4079)
  File "schema_child.pyx", line 78, in schema_child.TestEnumUser.shortrepr (/tmp/capnpy-pyx-15/home/darius/src/libtest/schema_child.c:2298)
  File "schema_child.pyx", line 80, in schema_child.TestEnumUser.shortrepr (/tmp/capnpy-pyx-15/home/darius/src/libtest/schema_child.c:2231)
  File "schema_child.pyx", line 63, in schema_child.TestEnumUser.test.__get__ (/tmp/capnpy-pyx-15/home/darius/src/libtest/schema_child.c:1876)
AttributeError: type object 'schema_base.TestEnum' has no attribute '_new

capnpy.load(f, payload_type): load a message from a file-like object
Should have a maximum message size parameter or internal check to avoid hanging in case a response message is unreasonably large.

test_strhash

def test_strhash():
    expected_hash_empty_string = 0
    if IS_PYPY:
        expected_hash_empty_string = -2
    #
    h = _hash.strhash

> assert h('', 0, 0) == hash('') == expected_hash_empty_string
E assert -1 == -2
E + where -1 = hash('')

is_pypy

[wleslie@dirac:~/src/capnpy]
18:15:38 $ pypy
Python 2.7.10 (7e8df3df96417c16c2d55b41352ec82c9c69c978, Jun 24 2016, 11:49:45)
[PyPy 5.3.1 with GCC 4.9.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>>> hash('')
-1
>>>>

struct Foo {
	bar :group $Py.nullable {
		isNull @1 :Int8;
		value @2 :Float64;
	}
}

would generate

    @property
    def bar(self):
        g = self._bar
        if g.is_null:
            return None
        return g.value

    def __init__(self, bar=(0, 0.0,)):
        _buf = Foo.__new(bar)
        self._init_from_buffer(_buf, 0, 2, 0)

    @staticmethod
    def __new(bar=(0, 0.0,)):
        builder = _SegmentBuilder()
        pos = builder.allocate(16)
        if bar is None:
            bar_is_null = 1
            bar_value = 0
        else:
            bar_is_null = 0
            bar_value = bar
        builder.write_int8(pos + 0, bar_is_null)
        builder.write_float64(pos + 8, bar_value)
        return builder.as_string()

    def shortrepr(self):
        parts = []
        parts.append("bar = %s" % self.bar.shortrepr())
        return "(%s)" % ", ".join(parts)

The default value of bar is 0, 0.0 which doesn't match the expectation of the following code.
The default should be either None or 0.0.

        if bar is None:
            bar_is_null = 1
            bar_value = 0
        else:
            bar_is_null = 0
            bar_value = bar

As a result,

Foo()
# TypeError: a float is required

shortrepr doesn't work correctly. Because self.bar returns float or None here which do not have shortrepr

Both C++ capnp and pycapnp seem to support relative imports just fine (e.g. c++.capnp located next to the schema and imported as import "c++.capnp") -- wonder if there's a reason for disallowing that explicitly?

Currently the __extends__() method for a Capnpy struct only supports simple classes with no inheritance. Attempting to use a class with inheritance results in a struct with only the methods defined on the class directly available to the struct; any inherited methods are lost.

The method should support classes with inheritance in a sensible way. All methods present on the class should be available on the struct. This will allow developers to avoid duplication of code when they need to share methods across these classes.

The current API assumes that schema is saved in a file, but in my case the capnp schema is fetched from some db as a str. Maybe an argument can be add to load_schema() to support this usage?

Hi,

In Python 3.9 I get the above error when importing capnpy:

$ python
Python 3.9.2 (default, Mar  8 2021, 11:23:50) 
[GCC 9.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import capnpy
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/tmp/venv3.9.2/lib/python3.9/site-packages/capnpy/__init__.py", line 4, in <module>
    from capnpy.compiler.compiler import DynamicCompiler
  File "/tmp/venv3.9.2/lib/python3.9/site-packages/capnpy/compiler/compiler.py", line 11, in <module>
    from capnpy import schema
  File "/tmp/venv3.9.2/lib/python3.9/site-packages/capnpy/schema.py", line 7, in <module>
    from capnpy.struct_ import Struct as _Struct
  File "capnpy/struct_.py", line 1, in init capnpy.struct_
  File "capnpy/type.pxd", line 1, in init capnpy.blob
  File "capnpy/type.py", line 33, in init capnpy.type
  File "capnpy/type.py", line 29, in capnpy.type.Types._make
TypeError: can't set attributes of built-in/extension type 'type'
>>>

If you declare a constant as :Text in the schema, the compiled pyx file doesn't quote the Python string, resulting in a Cython compilation error.

Failing test case: https://github.com/benmoran/capnpy/blob/text-constants/capnpy/testing/compiler/test_compiler.py

Following the "equality and hashing" guide: http://capnpy.readthedocs.io/en/latest/usage.html?highlight=new_#equality-and-hashing

I'd like to compare Structs containing Union fields for equality. I annotate the Struct with $Py.key("*"). But at runtime, the _key() method tries to get all the members of the union, which can never succeed.

capnp installed via latest homebrew:

$ capnp --version
Cap'n Proto version (unknown)

(it's actually v0.6.1)

In this case, it's impossible to load the schema since _capnp_check_version() fails.

If setup() for myapp includes

capnpy_schemas=['myapp/schema.capnp'],

and I run python setup.py sdist, then in myapp.egg-info/SOURCES.txt I have

/absolute/path/to/myapp/schema.c

which breaks installation on non-unix platforms because distutils.util.convert_path throws ValueError for absolute paths.

I think this would be fixed if capnpy.compiler.distutils.capnpify emitted Extension objects with relative (rather than absolute) paths to the schemas.

I was wondering if packed encoding/decoding was something feasible to do with capnpy? (like write_packed / read_packed in pycapnp).

struct A {
    s @0 :Int64;
    x @1 :Int64;
    y @2 :Int64;
    z @3 :Text;
    xyz @4: Void $Py.group("x, y, z");
}

So that A.xyz would be the same as C.xyz in terms of decoding

struct C {
    s @0 :Int64;
    xyz :group $Py.key("x, y, z") {
        x @1 :Int64;
        y @2 :Int64;
        z @3 :Text;
    }
}

Just opening to track this... Any special changes needed for Python 3.12?

Will try to build it myself soon.

Would it be possible for 'loads' to allow objects that support the buffer interface?
https://docs.python.org/2/library/functions.html#buffer

I've been having some decoding errors on the Raspberry Pi using CPython. loads() of any struct ends up returning None without any error.

The python implementation works fine on the platform with PyPy, but in this case I can only really use CPython 3 (unrelated, can't get pypy3 running on arm6), so I need to figure out a way around it and I'd prefer to use capnpy over pycapnp :).

My initial thought is the C code is written specifically for linux/x86, but unfortunately I don't have enough knowledge to find the issue.

I've tested it with both a Raspberry Pi Zero (arm6) and Raspberry Pi 3 (arm7).

If I set a Text field to a str, that works fine.

If I set a Text field to a unicode, I get TypeError: Expected bytes, got unicode. Capnproto Text fields are defined as UTF-8 encoded byte strings, so the conversion is unambiguous and should be performed automatically by the library.

Arguably fields should come out as unicode, too, but doing so will result in slower and non-backwards-compatible code, so if that's supported it should probably come with an option to disable it.

We should either not check the bounds at all, or propagate the exception if we do. Or ideally, have a flag to enable bounds checking.

cdef checkbound(int size, Py_ssize_t length, int offset):

would have to become

cdef int checkbound(int size, Py_ssize_t length, int offset) except -1:

And the same goes for unpack_*, struct_.pxd:_get_end and whoever else.

We've got quite a few traceback samples in capnpy.enum.__init__, probably because deserializing an enum creates a new object instead of reusing some interned object:

(Pdb++) currency == Currency.gbp
True
(Pdb++) currency is Currency.gbp
False

I have 2 schemas

a.capnp

@0xa4dc0279aaa4beb2;

struct A {
}

b.capnp

@0x9961190ba822e7bf;
using A = import "/a.capnp";

struct B {
    field @0: List(A.A);
}

~/capnp_test$ python -m capnpy compile b.capnp

Error compiling Cython file:
------------------------------------------------------------
...
    
    
    property field:
        def __get__(self):
            # no union check
            return self._read_list(0, __a_capnp.A_list_item_type)
                                              ^
------------------------------------------------------------

/tmp/capnpy-pyx-22/foo/capnp_test/b.pyx:57:47: undeclared name not builtin: __a_capnp
/tmp/capnpy-pyx-22/foo/capnp_test/b.c:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
 #error Do not use this file, it is the result of a failed Cython compilation.
  ^

It works if I do

struct B {
    field @0: A.A;
}

For the next release, can we please have a CPython 3.7 wheel?

A typical use case for capnpy is the following chains of requirements (lib, app are Python packages):

capnpy (from app)
capnpy (from lib->app)

i.e. lib contains a .capnproto schema (and so has capnpy as a compile-time requirement) and app makes use of the compiled schema provided by lib (and so has both capnpy and lib as run-time requirements).

Schema compilation can be slow, so it is desirable to publish a wheel for lib. If so, then compile-time != run-time in general, and the versions of capnpy used by lib and app may be different, but util.check_version insists that these versions be identical. This means that new releases of capnpy will break the existing wheel(s) for lib, and require a new versions to be built and published with the latest capnpy.

Pinning the capnpy version in lib unfortunately is not a solution, as pip is too dumb to notice the conflict; it first installs the unconstrained version requested by app, and later happily reports "Requirement already satisfied" when being asked to install the pinned version required by lib. A possible workaround would be to omit capnpy from the requirements of app altogether, and rely on the lib requirement to bring in the pinned version, but this seems like bad practice.

Possible solutions I can see:

1. Provide compatibility (between minor releases), such that app and lib can both specify capnpy<N+1.0.0. This really just reduces the frequency of breakages, but may be good enough.

2. Bundle the "incompatible" parts of the capnpy run-time library into wheels at compile-time. I have no idea how hard this would be!

capnpy.loads(bytes(4), MyMessage) 

will raise ValueError, and

capnpy.loads(bytes(0), MyMessage) 

raises EOFError. This is a useful distinction as the first case is a protocol violation, and second is just the end of the input.

However

capnpy.loads(bytes(n), MyMessage) 

will also raise EOFError for any n in (1,2,3), which is incorrect as all three of these cases are also protocol violations and so should instead raise ValueError.

Here's another mysterious failing test, which is not picking up the extensions that capnpy provides. Is this because you're running the tests with a capnpy installed, or something? Note the stderr.

[wleslie@dirac:~/src/capnpy]
 18:28:45 $ pypy -m pytest capnpy/testing/compiler/test_distutils.py 
============================ test session starts =============================
platform linux2 -- Python 2.7.10[pypy-5.3.1-final], pytest-2.9.2, py-1.4.31, pluggy-0.3.1
rootdir: /home/wleslie/src/capnpy, inifile: 
collected 8 items 

capnpy/testing/compiler/test_distutils.py .s.s.sFs

================================== FAILURES ==================================
____________________ TestSetup.test_setuptools_build[py] _____________________

self = 
monkeypatch = <_pytest.monkeypatch.monkeypatch instance at 0x0000000000cca1a0>
ROOT = local('/home/wleslie/src/capnpy')

    def test_setuptools_build(self, monkeypatch, ROOT):
        self.write("example.capnp", """
            @0xbf5147cbbecf40c1;
            struct Point {
                x @0: Int64;
                y @1: Int64;
            }
            """)
        self.write("setup.py", """
            import sys
            sys.path.insert(0, '{root}')
            from setuptools import setup
    
            setup(name='foo',
                  version='1.0',
                  capnpy_options=dict(pyx={pyx}),
                  capnpy_schemas=['example.capnp'],
                  )
            """, root=ROOT, pyx=self.pyx)
        #
        monkeypatch.chdir(self.tmpdir)
        ret = os.system('%s setup.py build_ext --inplace' % sys.executable)
        assert ret == 0
        if self.pyx:
            outfile = self.tmpdir.join('example.so')
        else:
            outfile = self.tmpdir.join('example.py')
        #
>       assert outfile.check(file=True)
E       assert (file=True)
E        +  where  = local('/tmp/pytest-of-wleslie/pytest-6/test_setuptools_build_py_0/example.py').check

/home/wleslie/src/capnpy/capnpy/testing/compiler/test_distutils.py:126: AssertionError
---------------------------- Captured stdout call ----------------------------
running build_ext
---------------------------- Captured stderr call ----------------------------
/home/wleslie/pypy53/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'capnpy_options'
  warnings.warn(msg)
/home/wleslie/pypy53/lib-python/2.7/distutils/dist.py:267: UserWarning: Unknown distribution option: 'capnpy_schemas'
  warnings.warn(msg)
=============== 1 failed, 3 passed, 4 skipped in 0.74 seconds ================
 

The current assumption sometimes fails while dumping messages, producing corrupt results.

When I compile the schema file, cause the error in title.

cmd: python -m capnpy compile MDBase.capnp

platform: win10

py: 3.6

Currently capnpy is de facto unmaintained: I didn't do anything for the past year and I don't have enough time/resources/interest to continue its development.

If anyone is interested to become a maintaner, please ping me and I'll give all necessary permissions.