Use the same mechanism as mkdocs does for their own docs:

• Python hooks https://github.com/squidfunk/mkdocs-material/blob/9888723f86b20224197811fb647bfc4a79ce5a9f/src/.overrides/hooks/shortcodes.py#L36
• Reference from markdown: https://github.com/squidfunk/mkdocs-material/blob/9888723f86b20224197811fb647bfc4a79ce5a9f/docs/plugins/meta.md?plain=1#L128

FYI,

I just happened to be testing pydust on a windows on a windows11 machine and sysconfig.get_config_var("LDLIBRARY") returns None even if Python3.11 is installed in the standard system-wide location: C:\Program Files\Python3.11

This issue lists Renovate updates and detected dependencies. Read the Dependency Dashboard docs to learn more.

Open

These updates have all been created already. Click a checkbox below to force a retry/rebase of any.

• Update dependency ruff to v0.4.2
• Update dependency ziglang to ^0.12.0
• Update dependency black to v24
• Click on this checkbox to rebase all open PRs at once

Detected dependencies

• Check this box to trigger a request for Renovate to run again on this repository

Currently a PyTuple.create only supports converting from a Zig tuple struct.

When wrapping/unwrapping, it's the trampoline that decides which type a value should be converted into. In some cases, e.g. for call arguments, we know we want an args tuple and kwargs dict. It is then ambiguous whether an empty Zig struct should be converted to a PyTuple or PyDict.

Similarly, we would never convert anything into a PySet. And there is an ambiguity in how to handle []u8 as PyString or PyBytes.

The proposal suggests the trampoline has a default target for each type, but all types can be explicitly constructed from any compatible object using their create function (possibly rename to new?)

Our wish list for the upstream Zig project!

See https://github.com/ziglang/zig/wiki/Third-Party-Tracking-Issues-(what-is-important-to-other-people%3F) for similar issues from other projects.

For Loop Array Construction

This would be a really nice shorthand for a lot of our comptime work.

const foo: [10]X = blk: {
    const foo_: [10]X = undefined;
    for (0..10) |i| {
      foo_[i] = x(i);
    }
};

// Becomes
const foo: [10]X = for(0..10) |i| x(i);

I don't know if this is possible, but in Zig debug mode it would be great when raising a Python exception to insert a bunch of stack frames into the Python traceback corresponding to the Zig frames.

I believe this is because ZLS only evaluates Options steps when it parses the build.zig. Currently the discovery of our Python include/library paths are wrapped up inside a Pydust step, which means ZLS isn't finding it.

I wonder if eagerly shelling out to Python to get this information is worth doing?

Hi,

Really cool project, I'm very pleased to discover it given that I am trying to learn zig.

I'd be interested to explore alternative ways to use ziggy-pydust.

I'm thinking that a single python script (akin to Django's manage.py but self-contained) could drive project creation, project mgmt tasks.py and virtualenv creation. This could be an alternative to using poetry.

I would also like to be able to use my builtin installation of zig v12 rather than having to use v11 if possible.

Having text_signature is not sufficient to get ides to offer code completion. Most of them will not load modules to offer suggestions and only parse them. SInce there's nothing to parse for a native module there's no type hints available.

However, since we generate text_signature we can create a simple python script that would load the build library code and use inspect module to produce type stub files. Additionally we need to include py.typed file in the generated package.

If/When https://github.com/python/cpython/pull/101872/files merges we could offer type hints and not only argument hints.

We should skip generation of stub file if there exists a *.pyi file with the name of the module

Currently we link libpython in order to run Zig tests that spawn python.
We then launch these Zig tests from Pytest in order to run all tests together.

This proposal is to invert the control flow by building a custom Zig test runner that gets built as a shared library. This library is then loaded into Pytest as a native extension and executed that way.

This means we can use Pythons that have been statically linked to libpython, and our startup time is probably faster. We should probably still fork/exec in order to isolate segfaults though.

https://stackoverflow.com/questions/68490194/defining-a-python-enum-in-a-c-extension-am-i-doing-this-right/69281896#69281896

Currently we try to import the class from its containing module, but we fail to navigate a hierarchy of nested classes

Often we call into CPython and use catch return null. At these call sites we could enrich the traceback to include Zig stack frames the same way we do for Python exceptions that we raise ourselves with e.g. py.ValueError.raise

Right now we generate the function with nonsensical signature and unclear if a function like this can even be called.

Probably something like:

const Subclass = py.class(struct {
    pub const __mro__ = &.{ "collections.abc.Sized" };
});

Both

const gil = py.gil();
defer gil.release();

const nogil = py.nogil();
defer nogil.acquire();

pub const Range = py.class("Range", struct {
    
    pub const Iterator = py.class("Iterator", struct {
        ...
    });

    ...
});

Currently fails with pydust/src/pydust.zig:201:5: error: Class has no associated module

Update the iterator example and pydust template after this :)

There is an error with this repository's Renovate configuration that needs to be fixed. As a precaution, Renovate will stop PRs until it is resolved.

Error type: Cannot find preset's package (local>fulcrum-so/renovate-config)

While playing around with pydust in my recent PR, I was thinking about whether it would be possible to reduce pydust's dependencies to zero (outside of python and a zig compiler in PATH):

Pydust's current dependencies are as follows (please correct me if I'm wrong here):

1. ziglang: pip installable zig compiler (redundant if zig compiler is in your system PATH)
2. setuptools: presumably required for the pyproject.toml build system integration
3. black: provides stub generation: eg fib.pyi, ..
4. pydantic: provides type-checking, serialization / deserialization and more (dataclasses on steroids)

If one has an objective of making a zero-dependency pydust variant, one could consider dropping ziglang and setuptools and make stub_generation optionally available only if black is installed. As for the use of pydantic, it's only used in the two config classes: config.ExtModule and config.ToolPydust and could conceivably be dropped for some stdlib-only-dependent code (dataclasses?) .

While it is debatable why one would want a zero-dependency project, could this be considered as an objective by the project owners? Or are the above dependencies strategically important for the project's roadmap?

help with debugging:

• __str__
• __repr__

basic math:

• __add__
• __sub__
• __mul__
• __eql__

reference:
https://docs.python.org/3/reference/datamodel.html#emulating-numeric-types

Currently we allow the user to initialize module state in a __new__() !Self function.

It would align better with our class instantiation semantics if instead we used a __init__(self: *Self) !void function.

e.g. py.isinstance(foo, *const SomeClass)

Also on ziggy-pydust-template

We can probably automatically implement tp_traverse and tp_clear for most user types.
Essentially, they must traverse or clear all internal strong references to PyObjects.

See:

• https://pyo3.rs/v0.6.0-alpha.2/print.html#garbage-collector-integration
• https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_clear
• https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_traverse

Most urgent use-case is for __new__ method to be defined, but to ignore / allow all subclasses to decide their own args/kwargs.

One proposal I quite like is "verbs" for new references, and "nouns" for borrowed references.

e.g.

py.getSelf(...) // Returns new reference to PyType object
py.type_(...) /// Returns borrowed reference.

py.init initially existed because it did more complicated object initialisation, such as calling tp_new implementations.

Now however it acts more like allocator.create. So I propose adding a py.alloc(comptime T: type) !*T that passes back the allocated, but un-initialized, struct. This enables use-cases where fields of the struct contain pointers to other fields in the struct.

Now we have fixed the caching issues, we don't need this anymore

I'm thinking something like:

py.class(struct {
   const Self = @This();
   pub using namespace py.zig(struct {
     pub fn myZigOnlyFactoryFunction(...) Self {
         return py.init(Self, ...);
     }
   });
});

I tried to walk through the steps to try it from a conda environment, but ran into an error, on mac:

poetry run pytest
============================================== test session starts ===============================================
platform darwin -- Python 3.11.5, pytest-7.4.2, pluggy-1.3.0
rootdir: /private/var/tmp/poetry-test/poetry-demo
plugins: ziggy-pydust-0.2.12
collected 1 item / 1 error

# ,,,

.../opt/miniconda3/envs/poetry.3.11/lib/python3.11/site-packages/_pytest/runner.py:341: in from_call
    result: Optional[TResult] = func()
.../opt/miniconda3/envs/poetry.3.11/lib/python3.11/site-packages/_pytest/runner.py:372: in <lambda>
    call = CallInfo.from_call(lambda: list(collector.collect()), "collect")
.../opt/miniconda3/envs/poetry.3.11/lib/python3.11/site-packages/pydust/pytest_plugin.py:62: in collect
    h = TestProtocol.Header.unpack(proc.stdout)
.../opt/miniconda3/envs/poetry.3.11/lib/python3.11/site-packages/pydust/pytest_plugin.py:230: in unpack
    (tag, bytes_len) = struct.unpack("<II", buffer.read(8))
E   struct.error: unpack requires a buffer of 8 bytes
------------------------------------------------ Captured stderr -------------------------------------------------
dyld[32874]: Library not loaded: @rpath/libpython3.11.dylib
  Referenced from: <98F79F69-BB8F-365C-8D5A-E3C2161D139D> /private/var/tmp/poetry-test/poetry-demo/zig-out/bin/hello.test.bin
  Reason: tried: '/usr/local/lib/libpython3.11.dylib' (no such file), '/usr/lib/libpython3.11.dylib' (no such file, not in dyld cache)

Steps:

conda create -n poetry.3.11 python=3.11
conda activate --stack poetry.3.11
python -m pip install poetry pytest
poetry new poetry-demo
cd !$
poetry add -G dev ziggy-pydust
# add files and stuff
poetry run pytest

Found this when bumping template. I think it must be ordering issue that I didn't notice before

PyBUF_SIMPLE is a flag used to denote a buffer is a pure 1-dimensional contiguous unsigned u8 array.

This is a pretty common use-case, so we should make it close to trivial to map back and forth to Zig []u8.

In fact, we could support quite a lot of the 1-dimensional buffer protocol using comptime.

• []u8 -> writable simple
• []u32 -> writable, check format string, check alignment, etc.
• []const u32 -> readonly, etc.

[build-system]
requires = ["poetry-core", "ziggy-pydust==0.5.*"]

We should check ourselves that the build requires specifies the correct dependency.

Currently, we expose underlying buffer bytes as slices of a specific type. We ignore the shape, and this works only for simple arrays (see https://docs.python.org/3/c-api/buffer.html#complex-arrays)

The Zig code that runs under pytest seems to require running twice to actually compile?

This may be related to self_managed issue

We can either:

• Build a way to take a dependency on Pydust as a Zig module (incl. an addPythonLibrary build step)
• Take build.zig-like arguments in the pyproject.toml and hook them up. Though there's only so far this would get you. Sufficiently complex projects would still want more control of build.zig

@vitalnodo

Calling a class method is unexpectedly much slower (10-20x) than calling a function. This is not observed with comparable Cython code, so not an issue with Python class in C extensions.

pydust: calling function is fast (compiled with "zig build -Doptimize=ReleaseFast")

$ python -m timeit -s 'from fibonacci._lib import get1' 'get1()'
5000000 loops, best of 5: 37.7 nsec per loop

$ python -m timeit -s 'from fibonacci._lib import get1' 'get1()'
5000000 loops, best of 5: 46.1 nsec per loop

Calling class method is slow:

$ python -m timeit -s 'from fibonacci._lib import Class; s = Class(10)' 's.get()'
200000 loops, best of 5: 981 nsec per loop

$ python -m timeit -s 'from fibonacci._lib import Class; s = Class(10)' 's.get()'
200000 loops, best of 5: 922 nsec per loop

Code: diff against fresh checkout of ziggy-pydust-template:

$ git diff
diff --git a/src/fib.zig b/src/fib.zig
index 3fde4f7..c32e91c 100644
--- a/src/fib.zig
+++ b/src/fib.zig
@@ -80,6 +80,28 @@ pub const FibonacciIterator = py.class(struct {
     }
 });
 
+pub const Class = py.class(struct {
+    const Self = @This();
+
+    i: u64,
+
+    pub fn __new__(args: struct { i: u64 }) !Self {
+        return .{ .i = args.i };
+    }
+
+    pub fn get(self: *Self) u64 {
+        return self.i;
+    }
+});
+
+pub fn get1() u64 {
+    return 10;
+}

Same thing done with Cython - same performance for function and method call:

$ python -m timeit -s 'from testclass import get1' 'get1()'
5000000 loops, best of 5: 39.3 nsec per loop

$ python -m timeit -s 'from testclass import get1' 'get1()'
5000000 loops, best of 5: 44.1 nsec per loop

$ python -m timeit -s 'from testclass import Class; s = Class(10)' 's.get()'
5000000 loops, best of 5: 44.5 nsec per loop

$ python -m timeit -s 'from testclass import Class; s = Class(10)' 's.get()'
5000000 loops, best of 5: 47 nsec per loop

$ cat testclass.pyx 
cdef class Class:
    cdef int i

    def __init__(self, i):
        self.i = i

    cpdef get(self):
        return self.i

cpdef get1():
    return 10

$ cat setup.py 
from setuptools import setup
from Cython.Build import cythonize

setup(
    ext_modules = cythonize("testclass.pyx")
)

$ python setup.py build_ext --inplace

Right now it's possible to instantiate a class via py.init but that returns py.PyObject. How does a user who creates a class instance supposed to remember type of it. The call returns py.PyObject but where should the type be stored?

We should instead look for zig-out/bin/*.test.bin files.