dgrunwald / rust-cpython Goto Github PK

View Code? Open in Web Editor NEW

1.8K 1.8K 135.0 5.1 MB

Rust <-> Python bindings

License: MIT License

Makefile 0.28% Rust 96.09% Python 3.62%

rust-cpython's People

Contributors

Stargazers

Watchers

Forkers

novocaine sunnyone trofimander gentoo90 tempbottle eneas193933 reaperhulk millerjs ihrwein knsd sciyoshi reeze cocuh expobrain therustmonk cmyr tethyssvensson naufraghi pashinin jturner314 flier fiedzia wdv4758h ohanar techcable rowhit fhoehle mkeeler eijebong robbym pombredanne ostrosco rochacbruno markbt indygreg pganssle asoderman rminderhoud github-notables mkatychev tcztzy xorpse whirlscape dushistov purew kaceybolman softprops madhupnetfundu tsnoam gracinet quark-zju derekdreery swissvoc hoangpq yosh fanzeyi mersinvald bastosvinicius hywan bccrisan alphare codefather-labs isgasho ejmg ahmed-masud slpcat wickerwaka yuja songroom2016 shadowkun yssource tailhook binh-vu tchigher ccliuyang cemelo ryefccd c4tcr4sh lausek jruiss masa-koz rustyforks vmray justinaustin hardipinders dmgolembiowski fredlahde age-rs chop0 victor-iyi tdyas 0kalekale i-spark ryancinsight accelleratorcloud stuhood pantsbuild blu3pr1n7 simonsapin durhamg

rust-cpython's Issues

Smarter running of python interpreter version in build scripts.

Currently build scripts of python3-sys and python27-sys try to run the right Python version with following commands:

python
python{major_version}
python{major_version}.{minor_version}

However if one of them fails, because there is no such Python in the path then the build script panics and does not try next commands.

For example I have Python 3.4 and 3.5 installed. I want to build Python extension library for Python 3.4. Python 3.5 is the usual one and would like to call it justpython in the path. But I can't call Python 3.4 python3.4 in the path, because it would cause panic in step 2. So I have to call Python 3.4 just python3, which could lead into thinking that python in the path means Python 2.x.

On windows all Python versions seem to be named python by default and I usually use py launcher to run the right one. Maybe we could use something like that after trying just python.

Add in subclasses. (Python 3.x)

I know that subclasses used to be in the bindings but I would like it readded for those who want to rewrite their python code that subclasses another class in the bindings and document it with a subclass example. It would be absolutely cool to be able to take this and turn it to rust code for python 3:

# coding=utf-8
"""
The MIT License (MIT)

Copyright (c) 2016 AraHaan

Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the "Software"),
to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
"""
from importlib.machinery import FileFinder
from . import api
import sys


class ExtensionImporter(object):
    """Base Importer Class."""
    def __init__(self, extension_list):
        self.extensions = extension_list
        self.path = None

    def find_loader(self, name, path):
        """Filds some loaders for importing the module."""
        str(name)  # use name even though we are not using it for anything.
        self.path = path
        return self

    def load_module(self, fullname):
        """Loads a module into sys.modules."""
        if fullname in sys.modules:
            return sys.modules[fullname]
        return None


class JsonImporter(ExtensionImporter):
    """JSON File Importer Class."""
    def __init__(self):
        super(JsonImporter, self).__init__(['.json'])

    def load_module(self, fullname):
        """Loads modules found with the extension"""
        premodule = super(JsonImporter, self).load_module(fullname)
        api.import_json(premodule, self.path, fullname)


class PycxImporter(ExtensionImporter):
    """PYCX File Importer Class."""
    def __init__(self):
        super(PycxImporter, self).__init__(['.pycx'])

    def load_module(self, fullname):
        """
        Loads modules found with the pycx excention.

        Info: pycx files are compressed python source code files.

        Order of processing (to import):
            zlib decompression
            base64 decode.
        """
        premodule = super(PycxImporter, self).load_module(fullname)
        api.import_pycx(premodule, self.path, fullname)


extension_importers = [JsonImporter(), PycxImporter()]
hook_list = []
for importer in extension_importers:
    hook_list.append((importer.find_loader, importer.extensions))

sys.path_hooks.insert(0, FileFinder.path_hook(*hook_list))
sys.path_importer_cache.clear()

(And yes I add in doc strings even to the top of the module and the file uses Unix line encodings).

This also gives me the perfect opportunity to also learn rust at the same time.

For now I have to make a backend file with the actual Classes and then figure out how to do this in rust:

# coding=utf-8
"""
The MIT License (MIT)

Copyright (c) 2016 AraHaan

Permission is hereby granted, free of charge, to any person obtaining a
copy of this software and associated documentation files (the "Software"),
to deal in the Software without restriction, including without limitation
the rights to use, copy, modify, merge, publish, distribute, sublicense,
and/or sell copies of the Software, and to permit persons to whom the
Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in
all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS
OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER
DEALINGS IN THE SOFTWARE.
"""
from importlib.machinery import FileFinder
from ._pycx_backend import *
import sys

extension_importers = [JsonImporter(), PycxImporter()]
hook_list = []
for importer in extension_importers:
    hook_list.append((importer.find_loader, importer.extensions))

sys.path_hooks.insert(0, FileFinder.path_hook(*hook_list))
sys.path_importer_cache.clear()

It would be nice if I could actually have the Classes implemented in rust with the subclasses and stuff instead of making a backend for it.

Cannot build against python 3 without specifying a minor version

python3-sys/build.rs expects that a specific minor version is specified.

However, the use of the PEP 384 limited ABI allows us to build against an arbitrary python3 version -- there shouldn't be a need for the user of rust-cpython to manually specify which python 3 version is installed.

Reconsider ExtractPyObject

I'm not happy with the current state of ExtractPyObject. In many cases, it involves unnecessary operations on the PyObject refcount due to the two-step operation.

I think it might more sense to separate extraction of values and references into two traits, and decide which one to use based on the syntactic form of the parameter type (handling the case distinction within the py_argparse!() macro instead of within the trait system).

Build fails when only Python3.5 is installed on system

I am using OS X 10.10.5, Rust 1.15.0-nightly (daf8c1dfc 2016-12-05), and python 3.5.1 :: Anaconda 2.5.0 (x86_64).

When I try to compile python3-sys v0.1.2, required by cpython v0.0.5, I get a panic during the buildscript because it uses python2 syntax. Here is the error:

λ RUST_BACKTRACE=1 cargo build --verbose
   Compiling interpolate_idents v0.1.3
   Compiling cpython v0.0.5
   Compiling libc v0.2.18
   Compiling regex-syntax v0.3.9
   Compiling rustc-serialize v0.3.22
   Compiling num-traits v0.1.36
   Compiling winapi v0.2.8
   Compiling abort_on_panic v1.0.0
     Running `rustc --crate-name interpolate_idents /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/interpolate_idents-0.1.3/src/lib.rs --crate-type dylib -C prefer-dynamic -g -C metadata=7526d6b21bb6cca7 -C extra-filename=-7526d6b21bb6cca7 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name build_script_build /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/cpython-0.0.5/build.rs --crate-type bin -g --cfg feature=\"python3-sys\" --cfg feature=\"default\" -C metadata=e3782fff4e36a13c -C extra-filename=-e3782fff4e36a13c --out-dir /Users/nm46057/projects/py-rs/target/debug/build/cpython-e3782fff4e36a13c --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name libc /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/libc-0.2.18/src/lib.rs --crate-type lib -g --cfg feature=\"use_std\" --cfg feature=\"default\" -C metadata=8b2d5d804c9dc299 -C extra-filename=-8b2d5d804c9dc299 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name regex_syntax /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-syntax-0.3.9/src/lib.rs --crate-type lib -g -C metadata=841541a8a23ec4a8 -C extra-filename=-841541a8a23ec4a8 --out-dir/Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name abort_on_panic /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/abort_on_panic-1.0.0/src/lib.rs --crate-type lib -g -C metadata=5e3b3dc826cd958b -C extra-filename=-5e3b3dc826cd958b --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name rustc_serialize /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/rustc-serialize-0.3.22/src/lib.rs --crate-type lib -g -C metadata=2ad29e886085060a -C extra-filename=-2ad29e886085060a --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name winapi /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-0.2.8/src/lib.rs --crate-type lib -g -C metadata=113b8ec38b5aa687 -C extra-filename=-113b8ec38b5aa687 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
     Running `rustc --crate-name num_traits /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-traits-0.1.36/src/lib.rs --crate-type lib -g -C metadata=689e3341ca1ddd47 -C extra-filename=-689e3341ca1ddd47 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
   Compiling winapi-build v0.1.1
     Running `rustc --crate-name build /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/winapi-build-0.1.1/src/lib.rs --crate-type lib -g -C metadata=a0b636a9c55720c5 -C extra-filename=-a0b636a9c55720c5 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
   Compiling utf8-ranges v0.1.3
     Running `rustc --crate-name utf8_ranges /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/utf8-ranges-0.1.3/src/lib.rs --crate-type lib -g -C metadata=6f2dfd8aacd530ad -C extra-filename=-6f2dfd8aacd530ad --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --cap-lints allow`
   Compiling kernel32-sys v0.2.2
     Running `rustc --crate-name build_script_build /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/kernel32-sys-0.2.2/build.rs --crate-type bin -g -C metadata=053635ea8b43b5f6 -C extra-filename=-053635ea8b43b5f6 --out-dir /Users/nm46057/projects/py-rs/target/debug/build/kernel32-sys-053635ea8b43b5f6 --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern build=/Users/nm46057/projects/py-rs/target/debug/deps/libbuild-a0b636a9c55720c5.rlib --cap-lints allow`
   Compiling memchr v0.1.11
   Compiling rand v0.3.15
     Running `rustc --crate-name memchr /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/memchr-0.1.11/src/lib.rs --crate-type lib -g -C metadata=cba4653e30250fb3 -C extra-filename=-cba4653e30250fb3 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern libc=/Users/nm46057/projects/py-rs/target/debug/deps/liblibc-8b2d5d804c9dc299.rlib --cap-lints allow`
     Running `rustc --crate-name rand /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/rand-0.3.15/src/lib.rs --crate-type lib -g -C metadata=f29022bd1a2db7a0 -C extra-filename=-f29022bd1a2db7a0 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern libc=/Users/nm46057/projects/py-rs/target/debug/deps/liblibc-8b2d5d804c9dc299.rlib --cap-lints allow`
     Running `/Users/nm46057/projects/py-rs/target/debug/build/kernel32-sys-053635ea8b43b5f6/build-script-build`
     Running `rustc --crate-name kernel32 /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/kernel32-sys-0.2.2/src/lib.rs --crate-type lib -g -C metadata=3d00101c01f927ac -C extra-filename=-3d00101c01f927ac --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern winapi=/Users/nm46057/projects/py-rs/target/debug/deps/libwinapi-113b8ec38b5aa687.rlib --cap-lints allow`
   Compiling aho-corasick v0.5.3
     Running `rustc --crate-name aho_corasick /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/aho-corasick-0.5.3/src/lib.rs --crate-type lib -g -C metadata=c2e64ef9624713be -C extra-filename=-c2e64ef9624713be --out-dir/Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern memchr=/Users/nm46057/projects/py-rs/target/debug/deps/libmemchr-cba4653e30250fb3.rlib --cap-lints allow`
   Compiling num-integer v0.1.32
     Running `rustc --crate-name num_integer /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-integer-0.1.32/src/lib.rs --crate-type lib -g -C metadata=0235e78e2e4fa152 -C extra-filename=-0235e78e2e4fa152 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern num_traits=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_traits-689e3341ca1ddd47.rlib--cap-lints allow`
   Compiling thread-id v2.0.0
     Running `rustc --crate-name thread_id /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/thread-id-2.0.0/src/lib.rs --crate-type lib -g -C metadata=e6d2dcfb0353967d -C extra-filename=-e6d2dcfb0353967d --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern libc=/Users/nm46057/projects/py-rs/target/debug/deps/liblibc-8b2d5d804c9dc299.rlib --extern kernel32=/Users/nm46057/projects/py-rs/target/debug/deps/libkernel32-3d00101c01f927ac.rlib --cap-lints allow`
   Compiling thread_local v0.2.7
     Running `rustc --crate-name thread_local /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/thread_local-0.2.7/src/lib.rs --crate-type lib -g -C metadata=9cc3dd7e12dfefab -C extra-filename=-9cc3dd7e12dfefab --out-dir/Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern thread_id=/Users/nm46057/projects/py-rs/target/debug/deps/libthread_id-e6d2dcfb0353967d.rlib --cap-lints allow`
   Compiling num-iter v0.1.32
     Running `rustc --crate-name num_iter /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-iter-0.1.32/src/lib.rs --crate-type lib -g -C metadata=85ba68ca8c8abd82 -C extra-filename=-85ba68ca8c8abd82 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern num_integer=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_integer-0235e78e2e4fa152.rlib --extern num_traits=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_traits-689e3341ca1ddd47.rlib --cap-lints allow`
   Compiling regex v0.1.80
     Running `rustc --crate-name regex /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/regex-0.1.80/src/lib.rs --crate-type lib -g -C metadata=bf64488252a40cc1 -C extra-filename=-bf64488252a40cc1 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern thread_local=/Users/nm46057/projects/py-rs/target/debug/deps/libthread_local-9cc3dd7e12dfefab.rlib --extern aho_corasick=/Users/nm46057/projects/py-rs/target/debug/deps/libaho_corasick-c2e64ef9624713be.rlib --extern utf8_ranges=/Users/nm46057/projects/py-rs/target/debug/deps/libutf8_ranges-6f2dfd8aacd530ad.rlib --extern memchr=/Users/nm46057/projects/py-rs/target/debug/deps/libmemchr-cba4653e30250fb3.rlib --extern regex_syntax=/Users/nm46057/projects/py-rs/target/debug/deps/libregex_syntax-841541a8a23ec4a8.rlib --cap-lints allow`
   Compiling num-bigint v0.1.35
   Compiling num-complex v0.1.35
     Running `rustc --crate-name num_bigint /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-bigint-0.1.35/src/lib.rs --crate-type lib -g --cfg feature=\"rustc-serialize\" --cfg feature=\"rand\" --cfg feature=\"default\" -C metadata=7765ff86e058a1ff -C extra-filename=-7765ff86e058a1ff --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern num_integer=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_integer-0235e78e2e4fa152.rlib --extern num_traits=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_traits-689e3341ca1ddd47.rlib --extern rustc_serialize=/Users/nm46057/projects/py-rs/target/debug/deps/librustc_serialize-2ad29e886085060a.rlib --extern rand=/Users/nm46057/projects/py-rs/target/debug/deps/librand-f29022bd1a2db7a0.rlib --cap-lints allow`
     Running `rustc --crate-name num_complex /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-complex-0.1.35/src/lib.rs --crate-type lib -g --cfg feature=\"rustc-serialize\" --cfg feature=\"default\" -C metadata=432c41d0742df634 -C extra-filename=-432c41d0742df634 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern num_traits=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_traits-689e3341ca1ddd47.rlib --extern rustc_serialize=/Users/nm46057/projects/py-rs/target/debug/deps/librustc_serialize-2ad29e886085060a.rlib --cap-lints allow`
   Compiling num-rational v0.1.35
     Running `rustc --crate-name num_rational /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-rational-0.1.35/src/lib.rs --crate-type lib -g --cfg feature=\"rustc-serialize\" --cfg feature=\"bigint\" --cfg feature=\"default\" --cfg feature=\"num-bigint\" -C metadata=207209b7108d94ab -C extra-filename=-207209b7108d94ab --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern num_integer=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_integer-0235e78e2e4fa152.rlib --extern num_traits=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_traits-689e3341ca1ddd47.rlib --extern rustc_serialize=/Users/nm46057/projects/py-rs/target/debug/deps/librustc_serialize-2ad29e886085060a.rlib --extern num_bigint=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_bigint-7765ff86e058a1ff.rlib --cap-lints allow`
   Compiling num v0.1.36
     Running `rustc --crate-name num /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/num-0.1.36/src/lib.rs --crate-type lib -g --cfg feature=\"num-complex\" --cfg feature=\"rustc-serialize\" --cfg feature=\"rational\" --cfg feature=\"bigint\" --cfg feature=\"complex\" --cfg feature=\"num-rational\" --cfg feature=\"default\" --cfg feature=\"num-bigint\" -C metadata=79763f04cdc9f157 -C extra-filename=-79763f04cdc9f157 --out-dir /Users/nm46057/projects/py-rs/target/debug/deps --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --externnum_integer=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_integer-0235e78e2e4fa152.rlib --extern num_iter=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_iter-85ba68ca8c8abd82.rlib --extern num_traits=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_traits-689e3341ca1ddd47.rlib --extern num_rational=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_rational-207209b7108d94ab.rlib --extern num_complex=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_complex-432c41d0742df634.rlib --extern num_bigint=/Users/nm46057/projects/py-rs/target/debug/deps/libnum_bigint-7765ff86e058a1ff.rlib --cap-lints allow`
   Compiling python3-sys v0.1.2
     Running `rustc --crate-name build_script_build /Users/nm46057/.cargo/registry/src/github.com-1ecc6299db9ec823/python3-sys-0.1.2/build.rs --crate-type bin -g --cfg feature=\"python-3\" --cfg feature=\"default\" -C metadata=11a899b63d7350d2 -C extra-filename=-11a899b63d7350d2 --out-dir /Users/nm46057/projects/py-rs/target/debug/build/python3-sys-11a899b63d7350d2 --emit=dep-info,link -L dependency=/Users/nm46057/projects/py-rs/target/debug/deps --extern regex=/Users/nm46057/projects/py-rs/target/debug/deps/libregex-bf64488252a40cc1.rlib --cap-lints allow`
     Running `/Users/nm46057/projects/py-rs/target/debug/build/python3-sys-11a899b63d7350d2/build-script-build`
error: failed to run custom build command for `python3-sys v0.1.2`
process didn't exit successfully: `/Users/nm46057/projects/py-rs/target/debug/build/python3-sys-11a899b63d7350d2/build-script-build` (exit code: 101)
--- stderr
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: "python script failed with stderr:\n\n  File\"<string>\", line 1\n    import MacOS; print MacOS.linkmodel;\n                            ^\nSyntaxError: invalid syntax\n"', ../src/libcore/result.rs:837
stack backtrace:
   1:        0x10530cc2a - std::sys::imp::backtrace::tracing::imp::write::hbea47d9dd19b523c
   2:        0x10530fe0f - std::panicking::default_hook::{{closure}}::h6875a2976258b020
   3:        0x10530fab7 - std::panicking::default_hook::h88ffbc5922643264
   4:        0x105310276 - std::panicking::rust_panic_with_hook::hc790e47d4ecc86cd
   5:        0x105310114 - std::panicking::begin_panic::hc066339e2fdc17d1
   6:        0x105310032 - std::panicking::begin_panic_fmt::h5912b2d2df332044
   7:        0x10530ff97 - rust_begin_unwind
   8:        0x105337520 - core::panicking::panic_fmt::h561c5ee168a3d2cb
   9:        0x1051ce84c - core::result::unwrap_failed::haf85f8825186c457
  10:        0x1051c67d4 - <core::result::Result<T, E>>::unwrap::h840ae8dd3db3331d
  11:        0x1051e3381 - build_script_build::get_macos_linkmodel::had26beefebe98bad
  12:        0x1051e34a8 - build_script_build::get_rustc_link_lib::hbbc2880fcef76db6
  13:        0x1051e5f71 - build_script_build::configure_from_path::hd6798e73fa40bfb4
  14:        0x1051e6e4b - build_script_build::main::h7cd0436253b36ed0
  15:        0x10531106a - __rust_maybe_catch_panic
  16:        0x105310616 - std::rt::lang_start::h5d71a3afaaa4b2ff
  17:        0x1051e7e29 - main

And the relevant code appears to be in build.rs, it assumes that python2 will be installed as the python executable. Seeing as this crate holds bindings for python3, it seems that it should maybe be possible to build it using python3.

How can i use this function in the rust-cpython?

The example code in Python is :
import requests
print(requests.get("http://www.google.com").text)
So i tried to write in Rust:
let requests=py.import("requests").unwrap();
let requests_get=requests.get(py,"get").call(py,("http://www.google.com",),None).unwrap().get(py,"text");
Unfortunately,i failed to get the page source..
let requests_get=requests.get(py,"get").call(py,("http://www.google.com",),None).unwrap()
return the <Response [200]>,so i can not get the text of the response.
I have searched in the doc but find nothing...so how can i fix this problem?

There should be a way to reset the interpreter state between unit tests

I have a unit test in which I want to make sure that a python module is not available by default, and is available when I prepend an entry to sys.path. Right now, I have have each of these checks in their own #[test] method, so that it's easier to see which of the checks fail if either do. However, sometimes the test that prepends to sys.path runs before the other test, which makes it fail since the state of sys.path is preserved.

There's easy workarounds for this for me: I can combine the tests to ensure that they run in the other order; I can remove the directory I add to sys.path after the test; there's probably many others. But it's not intuitive that there's state that persists between unit tests.

My suggestion is to either somehow reset the interpreter when the gil is dropped inside a test function (not sure if it's possible to check that), or to provide a way to reset the interpreter by hand. These suggestions assume that the current behavior of preserving state between calls to Python::acquire_gil is intended, which it seems to be.

Build fails with interpolate_idents

I've tried building the example on 1.2 stable and 1.4 nightly, both error with the following:

% cargo build
   Compiling interpolate_idents v0.0.3
   Compiling cpython v0.0.2
/home/foo/.multirust/toolchains/nightly/cargo/registry/src/
  github.com-0a35038f75765ae4/interpolate_idents-0.0.3/src/lib.rs:36:48: 
  36:67 error: mismatched types:
 expected `&str`,
    found `syntax::parse::token::InternedString`
(expected &-ptr,
    found struct `syntax::parse::token::InternedString`) [E0308]
/home/foo/.multirust/toolchains/nightly/cargo/registry/src/
  github.com-0a35038f75765ae4/interpolate_idents-0.0.3/src/lib.rs:36

            new_ident.push_str(ident.name.as_str());
            ^~~~~~~~~~~~~~~~~~~

note: in expansion of for loop expansion
/home/foo/.multirust/toolchains/nightly/cargo/registry/src/
  github.com-0a35038f75765ae4/interpolate_idents-0.0.3/src/lib.rs:29:17: 
  40:18 note: expansion site
/home/foo/.multirust/toolchains/nightly/cargo/registry/src/
  github.com-0a35038f75765ae4/interpolate_idents-0.0.3/src/lib.rs:36:48: 
  36:67 help: run `rustc --explain E0308` to see a detailed explanation
error: aborting due to previous error
Could not compile `interpolate_idents`.

Not sure what to do to fix this, since it appears to be an issue with interpolate_idents (one of this repo's dependencies). But thought at the very least it would be good to report it.

Consider catching panics instead of aborting

Rust-cpython currently uses abort_on_panic to prevent panics from unwinding into C code.
We cannot use unwind::try because that function cannot be nested, but we cannot exclude nesting since python and rust code can call each other.

However, it seems that Rust plans to add a safe way to catch panics (thread::catch_panic). Once the safety aspects of catch_panic (in particular regarding TLS) are worked out, we should consider catching panics instead of aborting.

Please release a new version to crates.io

The last released version 0.0.4 does no longer compiler correctly. Something like 0.0.5 or 0.1.0 would be nice, so rust-cpython can be made a dependency of other crates.io packages :)

rust-cpython fails to build on Mac

Compiling python27-sys v0.0.6
failed to run custom build command for python27-sys v0.0.6
Process didn't exit successfully: target/debug/build/python27-sys-c6f0449b2a9ec4bf/build-script-build (exit code: 101)
--- stderr
thread '

' panicked at 'called Result::unwrap() on an Err value: "\"pkg-config\" \"--libs\" \"--cflags\" \"python-2.7\" did not exit successfully: exit code: 1\n--- stderr\nPackage python-2.7 was not found in the pkg-config search path.\nPerhaps you should add the directory containing `python-2.7.pc'\nto the PKG_CONFIG_PATH environment variable\nNo package 'python-2.7' found\n"', /Users/rustbuild/src/rust-buildbot/slave/stable-dist-rustc-mac/build/src/libcore/result.rs:729

PyList::iter() not working correctly for map (and possibly others)

When calling list.iter(py).map(|value| {...}) it doesn't seem to actually yield the item for map. By that I mean it would never even map at all but rather just skip the loop section entirely as if the list was empty. Switching to for value in list.iter(py) {...} worked correctly.

Using PyType in pattern matching

Is there a better way to execute specific code based on the of the PyObject? At the moment the only solution I fond is this one:

fn from_python_to_yaml(py: Python, obj: &PyObject) {
    let py_type = obj.get_type(py);
    let type_name = py_type.name(py);

    match type_name.borrow() {
        "list"      => { ... },
        "int"       => { ... },
        "str"       => { ... },
        "unicode"   => { ... },
        "dict"      => { ... },
        "float"     => { ... },
        "bool"      => { ... },
        "NoneType"  => { ... },
        _           => { println!("Unimplemented {:?}", type_name); }
    }
}

but is not optimal from the point of view of the performance and readability.

conversion.rs:111:78: 113:6 error: cannot infer an appropriate lifetime for lifetime parameter `'python` due to conflicting requirements

rustc 1.4.0-nightly (f05b22efb 2015-08-15)

➜  src git:(master) cargo build
   Compiling cpython v0.0.3 (file:///Users/jsalter/Documents/dev/rust-cpython)
conversion.rs:111:78: 113:6 error: cannot infer an appropriate lifetime for lifetime parameter `'python` due to conflicting requirements
conversion.rs:111     fn extract(&&ref obj: &'prepared Self::Prepared) -> PyResult<'python, T> {
conversion.rs:112         Ok(try!(obj.clone().cast_into()))
conversion.rs:113     }
conversion.rs:111:5: 113:6 help: consider using an explicit lifetime parameter as shown: fn extract(&&ref obj: &'prepared Self::Prepared) -> PyResult<'prepared, T>
conversion.rs:111     fn extract(&&ref obj: &'prepared Self::Prepared) -> PyResult<'python, T> {
conversion.rs:112         Ok(try!(obj.clone().cast_into()))
conversion.rs:113     }
objects/list.rs:150:83: 157:6 error: cannot infer an appropriate lifetime for lifetime parameter `'python` due to conflicting requirements
objects/list.rs:150     fn extract(&&ref obj: &'prepared Self::Prepared) -> PyResult<'python, Vec<T>> {
objects/list.rs:151         let list = try!(obj.cast_as::<PyList>());
objects/list.rs:152         let mut v = Vec::with_capacity(list.len());
objects/list.rs:153         for i in 0 .. list.len() {
objects/list.rs:154             v.push(try!(list.get_item(i).extract::<T>()));
objects/list.rs:155         }
                    ...
objects/list.rs:150:5: 157:6 help: consider using an explicit lifetime parameter as shown: fn extract(&&ref obj: &'prepared Self::Prepared)
 -> PyResult<'prepared, Vec<T>>
objects/list.rs:150     fn extract(&&ref obj: &'prepared Self::Prepared) -> PyResult<'python, Vec<T>> {
objects/list.rs:151         let list = try!(obj.cast_as::<PyList>());
objects/list.rs:152         let mut v = Vec::with_capacity(list.len());
objects/list.rs:153         for i in 0 .. list.len() {
objects/list.rs:154             v.push(try!(list.get_item(i).extract::<T>()));
objects/list.rs:155         }
                    ...
error: aborting due to 2 previous errors
Could not compile `cpython`.

Next release stabilization

First of all, thanks for this great library!

I started to use it for creating Python parser bindings for syslog-ng. It seems like I used a work in progress version from your lib, as there were several types and macros removed I relied on (TypeBuilder, py_method).

I fixed some minor bugs (documentation, ::std::mem::transmute) and when I wanted to send a PR I realized that there was a major refactor in cpython in the weekend.

Could you tell me when you want to release a new version from your library? Is the py_class macro stable to use? Can I wrap a Rust object into a Python class with it (by using |py, obj|)?

Thank you very much for your awesome work!

Support for mapping protocol and set objects

We have support for the sequence, iterator and object protocol right now, but don't have support for the mapping protocol. Would be a nice addition to have in the future.

On the concrete object layer adding direct support for set objects would be a nice addition too (we have dictionaries, but not sets: https://docs.python.org/3.5/c-api/set.html).

I'm writing a wrapper which generates bindings automatically between rust and python leveraging type annotations and typing module for python 3.5 and I think sets are the only container type missing right now, as well as the mapping ABC (which would be supported adding mapping protocol).

Python3.5 change from m_reload to m_slot in PyModuleDef

Since Version 3.5 the member "m_reload" of the PyModuleDef changed to "m_slots". I think this is currently not the case for the Rust bindings.

PyModuleDef_Slot* m_slots¶

An array of slot definitions for multi-phase initialization, terminated by a {0, NULL} entry. When using single-phase initialization, m_slots must be NULL.

Changed in version 3.5: Prior to version 3.5, this member was always set to NULL, and was defined as:

inquiry m_reload¶

https://docs.python.org/3/c-api/module.html?highlight=pymoduledef#c.PyModuleDef.m_slots

Document Cargo.toml features

When looking at the README, I almost concluded that rust-cpython would auto-detect the installed major Python version and didn't allow specifying which to use when 2.x and 3.x were both found.

Furthermore, once I did figure out that it uses features to select them, I still couldn't find them mentioned anywhere in the documentation. That's not exactly ergonomic.

...and, as someone who uses Python heavily but has almost no experience with C or C++, I think the description of the extension-module feature would benefit from going into a bit more detail.

(eg. I think an "extension module" is a "compiled module to be loaded into a Python program using import), but I'm not feeling very confident beyond a vague memory of seeing the term when I binge-read the entire distutils section of the Python manual and then poked around in the distutils code in order to figure out how to accomplish certain tasks... definitely not something many potential users will have done.)

Docs improvement: Construct PyErr

Hi,

I am experimenting with the new buffer support with numpy. I am so stoked! This is going to be amazing!

I did get come across a confucian, how do I make a TypeError so I can return from my fn?

Also Is there anything I can do to help with the buffer support?

Guide for how to build extensions in rust

rust-cpython can be used to build extensions as well as to embed python in a rust process, so we should document that and make it easy for people.

To that goal, I have started on some distutils helpers for building and packaging extensions written as rust crates over at https://github.com/novocaine/rust-python-ext. I have used the example hello world code from rust-cpython as a start.

These helpers don't depend on rust-cpython, but I'd like to know your thoughts on either just merging them into this repo (so rust-cpython is 'batteries included'), or maybe the example in this repo could just depend and/or recommend them. I don't mind either way.

Converting PyInt/PyLong to c_long

How do I successfully convert a PyInt or PyLong to a c_long? I see that PyInt implements value(py) but PyLong does not. All of my code uses PyInt but it seems when compiling it thinks I want PyLong instead and so I cannot call value(py) on my PyInt.

/* 
error: no method named `value` found for type `cpython::PyLong` in the current scope
  --> rust/src/stats/mod.rs:62:10
   |
62 |     date.value(py)
   |          ^^^^^
*/ 
pub fn parse_date(py: Python, date: PyInt) -> c_long {
    date.value(py)
}


/*
error: no method named `extract` found for type `cpython::PyLong` in the current scope
  --> rust/src/stats/mod.rs:62:10
   |
62 |     date.extract(py).unwrap()
   |          ^^^^^^^
   |
   = note: found the following associated functions; to be used as methods, functions must have a `self` parameter
note: candidate #1 is defined in the trait `cpython::FromPyObject`
  --> rust/src/stats/mod.rs:62:10
   |
62 |     date.extract(py).unwrap()
   |          ^^^^^^^
*/
pub fn parse_date(py: Python, date: PyInt) -> c_long {
    date.extract(py).unwrap()
}

Example program

I'm running into trouble trying to get the first example program on this project's main page to work for me. It builds OK, but running it does nothing but crash. I don't even know where to begin troubleshooting this:

cargo run
Running `target\debug\yay_python.exe`
error: Process didn't exit successfully: `target\debug\yay_python.exe` (exit code: 3221225477)

I think it is failing at let sys = py.import("sys").unwrap(); because the result is identical when everything after that line is commented out.

My environment:
Windows
Rust 1.12.1 gnu 32 bit
Python 3.4.4 32 bit

What could I be doing wrong?

Windows support

I actually started writing rust-cpython on Windows, but had to give up due to strange issues when linking against the official python binaries. Now that the x86_64-pc-windows-msvc target exists, we should give it another try.

Not able to define extension inside a module

First of all, this is a really good crate, thank you very much for your work! While trying it I ran into the following bug:

Defining a Rust extension for Python works fine as long as everything is in the top level module, but the following code

#[macro_use]
extern crate cpython;

pub mod python {
    use cpython::{Python, PyResult, PyObject};

    py_module_initializer!(example, initpylum, PyInit_pylum, |py, m| {
        try!(m.add(py, "run", py_fn!(run())));
        Ok(())
    });

    fn run(py: Python) -> PyResult<PyObject> {
        println!("Rust says: Hello Python!");
        Ok(py.None())
    }
}

Fails to compile with

<cpython macros>:10:1: 11:40 error: failed to resolve. Use of undeclared type or module `std::mem` [E0433]
[...]
<cpython macros>:10:1: 11:40 error: unresolved name `std::mem::transmute` [E0425]
[...]
error: aborting due to 2 previous errors

Adding an use std; in the python module fix this error.

cargo build requires python3-sys 0.1.2

hi there, on the travis build you used your local copy of python3-sys so it passed the build, but when you fetch the crate with cargo it downloads python3-sys 0.1.1 and it doesn't build properly.

i had to download the crate and use a local copy to make it work (with current nightly build as of 11/11/2015) as well as that one that passed the build (07/11/2015). you may want to update the python3-sys / python27 crates on crate.io

may as well ask if you have any plans for stabilization (what's stopping it, I guess the stabilization of the libc library itself?)

p.s: i guess is the same for python 2.7 version, haven't tried though

PyObject<'p> and Send

When defining a python extension type in rust, it's possible for python code to retain a reference to instances of that type, and later use them on another (python) thread.
Thus, python extension types require that their contents are Send. (and also 'static, but we could potentially allow 'p if we make the 'p lifetime invariant and use a higher-order lifetime when acquiring the GIL)

The problem: PyObject itself is not Send, so it is not possible to create python extension objects that contain references to other python objects :(

So, why is PyObject not Send? The python runtime requires that python objects are only used while the global interpreter lock (GIL) is held. But it's no problem to hold a reference to a python object without having the GIL acquired -- as long as you don't access the python object until re-acquiring the GIL.
Currently, a PyObject<'p> is a *mut ffi::PyObject with two invariants:
a) The PyObject owns one reference to the python object (will call Py_DECREF() in the Drop impl)
b) The GIL is held for at least the lifetime 'p. (i.e. PyObject<'p> contains a Python<'p>)

Python<'p> represents "the current thread holds the GIL for lifetime 'p", and thus is fundamentally not Send. We could attempt to remove the Python<'p> from PyObject<'p>. This would require the user to explicitly pass in the Python<'p> for every operation on the PyObject. But on the other hand, it means we don't need a lifetime on PyObject (and transitively, PyResult etc.), so overall it should be an ergonomics win. It would open up some possibilities for a safe API for temporarily releasing the GIL during a long pure-rust computation (Py_BEGIN_ALLOW_THREADS).

But there's a big issue with this approach: the Drop impl. Calling Py_DECREF() requires holding the GIL, so we wouldn't be able to do that in Drop -- we'd have to provide an explicit fn release(self, Python), and any call to Drop would leak a reference count.
Forgetting to call release() would be a serious footgun.

This becomes less of an issue with static analysis to find missing release() calls: humpty_dumpty might be able to help; and hopefully Rust will get linear types in the future.

So, do we adopt this change? When writing python extension in rust, I'd say yes, it's worth it. But for embedding python in a rust program, trading a few lifetime annotations for a serious footgun is not exactly a good trade-off.

Maybe we could make Drop (re-)acquire the GIL? Locking the GIL recursively is allowed. That would reduce the footgun from a ref leak to a performance problem: instead of a simple decrement (which is getting inlined into the rust code), Drop would involve two calls into python library, both of which read from TLS and then do some cheap arithmetic+branching. Nothing too expensive, so this might be a workable approach (considering you can prevent the performance loss by explicitly calling release()).

link problem in sys.version example

When building the first example (sys.version) from the Usage section I get the following link error,
= note: /usr/bin/ld: cannot find -lpython3.4m
Do I need to specify more dependencies other then cpython = "0.1" ?
This is using rust-1.14, linux

Inadequate CI

Travis is pretty inadequate for a project of this type where what's really required is a selection of build slaves for the supported platforms and python versions.

It seems like currently travis builds on linux x64 only, with python 2.7.

Raising this issue in case anyone knows of a free hosted CI platform that supports other platform types (32 bit, OS X, Windows), has a cunning free solution to the problem, or is willing to donate an instance to the problem.

OS:

Linux
OS X
Windows

Platform:

Building library without python in system.

If i try to use this lib in my desktop env, it`s executing python code with the interpreter, that installed in system. But i want to use another way. I want to use a "portable" build of cpython in .so/.dll, that independent from system python. Is it realistic? If yes, how can i do it?

Using decorators or getting a function's decorator?

Is it currently possible to retrieve the decorator/decorator's data of a Python 3 function when interfacing Python code in Rust?

Support array.array('f', ...) -> Vec<f32>

rust-cpython can convert Python lists to Rust vectors, but it can't yet convert arrays created by the array module. It also doesn't support numpy arrays, which is probably a bigger project, but also valuable goal.

For example, I have an example where I am writing a dot product function:

mod dot {
pub fn dot(a : &Vec<f32>, b : &Vec<f32>) -> f32 {                               
    let mut out = 0.;                                                           
    for (x, y) in a.iter().zip(b.iter()) {                                      
        out += x*y;                                                             
    }                                                                           
    out                                                                         
}                                                                               

} // mod 
fn dot<'p>(py: Python<'p>, args: &PyTuple<'p>) -> PyResult<'p, f32> {           
    if args.len() < 2 {
// TODO: write a macro for making PyErr so it's less painful.                                                         
        let msg = "dot takes two arrays of float";                              
        let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
        return Err(pyerr);                                                      
    }                                                                           
    let arg0 = match args.get_item(0).extract::<Vec<f32>>() {                   
        Ok(x) => x,                                                             
        Err(_) => {                                                             
            let msg = "Could not convert argument 0 to array of float";         
            let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
            return Err(pyerr);                                                  
        }                                                                       
    };                                                                          

    let arg1 = match args.get_item(1).extract::<Vec<f32>>() {                   
        Ok(x) => x,                                                             
        Err(_) => {                                                             
            let msg = "Could not convert argument 0 to array of float";         
            let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
            return Err(pyerr);                                                  
        }                                                                       
    };                                                                          

    if arg0.len() != arg1.len() {                                               
            let msg = "arg0 and arg1 must be the same length";                  
            let pyerr = PyErr::new_lazy_init(py.get_type::<exc::ValueError>(), Some(msg.to_py_object(py).into_object()));
            return Err(pyerr);                                                  
    }                                                                           

    Ok(dot::dot(&arg0, &arg1))                                                  
}  

py_module_initializer!(librust_python_example, |_py, m| {                       
    try!(m.add("__doc__", "Dot product"));                                                     
    try!(m.add("dot", py_fn!(dot)));                                            
    Ok(())                                                                      
});

It works pretty well for a toy example:

In [1]: def dot(a, b):
    return sum(map(lambda x: x[0]*x[1], zip(a,b)))
   ...: 

In [2]: import numpy as np

In [3]: import librust_python_example as r

In [4]: timeit dot([0,2,3], [1,2,3])
100000 loops, best of 3: 1.86 µs per loop

In [5]: timeit np.dot([0,2,3], [1,2,3])
100000 loops, best of 3: 3.89 µs per loop

In [6]: timeit r.dot([0,2,3], [1,2,3])
1000000 loops, best of 3: 920 ns per loop

However, it won't convert numpy arrays or array.arrays:

In [8]: r.dot(np.arange(3), np.arange(3))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-8-0e9a2e3217bb> in <module>()
----> 1 r.dot(np.arange(3), np.arange(3))

ValueError: Could not convert argument 0 to array of float

In [9]: import array
In [10]: r.dot(array.array('f', [0,2,3]), array.array('f', [1,2,3]))
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-10-3eab6b6f9184> in <module>()
----> 1 r.dot(array.array('f', [0,2,3]), array.array('f', [1,2,3]))

ValueError: Could not convert argument 0 to array of float

Missing docs

Just thought I'd let you know that the published docs for some reason do not document all the method you have implemented (http://dgrunwald.github.io/rust-cpython/doc/cpython/struct.PyDict.html, for example). I worry that you may be selling your crate short there!

How to convert Vec<u8> to bytes instead of list

Hi,

I'd like to convert a Vec<u8> to a Python object. In the current implementation, if I use .to_py_object(), I receive a Python list. I'd rather prefer bytes and IIRC in the "past" this is what rust-cpython used to return.

The only way I found so far is to explicitly create a PyBytes object which in my case leads to duplicate code -- see below.

Here is an example:

        let py_dict = PyDict::new(py);
        //let py_bytes: PyBytes = PyBytes::new(py, &self.data); // Will give you bytes
        let py_bytes = self.data.to_py_object(py); // Will give you list
        let _ = py_dict.set_item(py, "data", py_bytes);
        py_dict

A full running example can be found in this GitHub repository: https://github.com/lukaspustina/rust-cpython-exp-bytes.

Cheers,
Lukas

PS Thanks for the hard work to create this library.

Allow defining python extension types

Basically https://docs.python.org/2/extending/newtypes.html, but from safe Rust code.

I'm currently working on this.

py_class! macro support for:

Flexible parameter lists

Add support for:

Optional parameters
*args
**kwargs

run_python_script_try_interpreter and virtualenv

In 2904330, run_python_script_try_interpreter now tries to locate more version specifically named python interpreters to help it locate the correct interpreter.

The problem with this is that although it helps locate say 'python3' on many popular linux distros and os x, it doesn't work correctly with virtualenv, which only adds a link for 'python', not pythonX or pythonX.X.

So for example:

jsalter@sydxplanssh ~ $ which python
/usr/bin/python
jsalter@sydxplanssh ~ $ virtualenv testenv
jsalter@sydxplanssh ~ $ source testenv/bin/activate
(testenv)jsalter@sydxplanssh ~ $ which python
/home/jsalter/testenv/bin/python
(testenv)jsalter@sydxplanssh ~ $ which python2.7
/usr/bin/python2.7

virtualenv sets an environment variable VIRTUAL_ENV - maybe we could check if that is present and if it is, just run 'python'?

Add python 3.6 support.

Python 3.6 should hopefully be released officially as the final version soon it is currently in rc1 state. This means this should allow use of this in 3.6 for those who use rust to help speed up their code. This is something really cool that even I am considering it for my iterators I had to implement in my bot to look through messages (using coroutines) and add them to a number of messages to delete. With a filter and if it was coded in rust on that part it should help it a bit.

like an example:

    opt = ctx.message.content[len(self.bot.bot_prefix + "prune "):].strip()
    num = 1
    if opt:
        try:
            num = int(opt)
        except Exception as e:
            str(e)
            return

(note ctx is passed to a function that provides context on what the message was)
Also import hooks, what if someone was to somehow write one that uses importlib to generate an import hook like so in python:

from importlib.machinery import FileFinder
import zlib
import json
import base64
import sys


class ExtensionImporter(object):
    """Base Importer Class."""
    def __init__(self, extension_list):
        self.extensions = extension_list
        self.path = None

    def find_loader(self, name, path):
        """Filds some loaders for importing the module."""
        str(name)  # use name even though we are not using it for anything.
        self.path = path
        return self

    def load_module(self, fullname):
        """Loads a module into sys.modules."""
        if fullname in sys.modules:
            return sys.modules[fullname]
        return None


class JsonImporter(ExtensionImporter):
    """JSON File Importer Class."""
    def __init__(self):
        super(JsonImporter, self).__init__(['.json'])

    def load_module(self, fullname):
        """Loads modules found with the extension"""
        premodule = super(JsonImporter, self).load_module(fullname)
        if premodule is None:
            try:
                with open(self.path) as f:
                    module = json.load(f)
                    sys.modules[fullname] = module
                    return module
            except FileNotFoundError:
                raise ImportError('Could not import the json file {0}'.format(fullname))


class PycxImporter(ExtensionImporter):
    """PYCX File Importer Class."""
    def __init__(self):
        super(PycxImporter, self).__init__(['.pycx'])

    def load_module(self, fullname):
        """
        Loads modules found with the pycx excention.

        Info: pycx files are compressed python source code files.

        Order of processing (to import):
            zlib decompression
            base64 decode.
        """
        premodule = super(PycxImporter, self).load_module(fullname)
        if premodule is None:
            try:
                with open(self.path, 'rb') as f:
                    filedata = f[len(b'PYCX'):].strip()
                    try:
                        decczfiledata = zlib.decompress(filedata)
                        decoded_data = base64.b64decode(decczfiledata)
                    except zlib.error as ex:
                        raise ImportError('Could not import {0} because of exception:\n{1}.'.format(fullname, str(ex)))
                    # assume utf8 encoding. utf8 is standard python script encoding anyways.
                    module = decoded_data.decode("utf-8")
                    sys.modules[fullname] = module
                    return module
            except FileNotFoundError:
                raise ImportError('Could not import {0}.'.format(fullname))


extension_importers = [JsonImporter(), PycxImporter()]
hook_list = []
for importer in extension_importers:
    hook_list.append((importer.find_loader, importer.extensions))

sys.path_hooks.insert(0, FileFinder.path_hook(*hook_list))
sys.path_importer_cache.clear()

Import hooks like these in python would be absolutely awesome to be able to code in rust instead of needing to cythonize them as cython makes code awful to read and makes the code confusing when it is cythonized.

Although then again it would be nice if rust had classes similar to python as C, C++, even Visual Basic has classes.

How do I add a class to py_module_initializer?

We have a function export example:

py_module_initializer!(hello, inithello, PyInit_hello, |py, m| {
    m.add(py, "__doc__", "Module documentation string")?;
    m.add(py, "run", py_fn!(py, run()))?;
    Ok(())
});

I try to create and export a class. How can I do this? I tried py_class! but had error:

try!(m.add(py, "Cls", py_class!(py, Cls)));


error: no rules expected the token `py`

Can't compile `py_module_initializer!` macro

I can't compile the example for py_module_initializer!. The error states that the arguments for the python 2&3 init functions names are unexpected.

src/lib.rs:

#[macro_use] extern crate cpython;
use cpython::{Python, PyResult, PyObject};

py_module_initializer!(example, initexample, PyInit_example, |py, m| {
    try!(m.add(py, "__doc__", "Module documentation string"));
    try!(m.add(py, "run", py_fn!(py, run())));
    Ok(())
});

fn run(py: Python) -> PyResult<PyObject> {
    println!("Rust says: Hello Python!");
    Ok(py.None())
}

Cargo.toml:

[package]
name = "example"
version = "0.1.0"
authors = ["Andreas Linz <[email protected]>"]

[lib]
name = "example"
crate-type = ["dylib"]

[dependencies]
cpython = "^0.0.5"

$ rustup run nightly -- cargo build --verbose:

rustup run nightly -- cargo build --verbose
       Fresh regex-syntax v0.3.4
       Fresh winapi-build v0.1.1
       Fresh rustc-serialize v0.3.19
       Fresh abort_on_panic v1.0.0
       Fresh winapi v0.2.8
       Fresh num-traits v0.1.34
       Fresh libc v0.2.15
       Fresh interpolate_idents v0.1.2
       Fresh num-integer v0.1.32
       Fresh rand v0.3.14
       Fresh num-complex v0.1.33
       Fresh kernel32-sys v0.2.2
       Fresh memchr v0.1.11
       Fresh num-iter v0.1.32
       Fresh num-bigint v0.1.33
       Fresh thread-id v2.0.0
       Fresh aho-corasick v0.5.2
       Fresh utf8-ranges v0.1.3
       Fresh num-rational v0.1.32
       Fresh thread_local v0.2.6
       Fresh num v0.1.34
       Fresh regex v0.1.73
       Fresh python3-sys v0.1.2
       Fresh cpython v0.0.5
   Compiling example v0.1.0 (file:///tmp/test/example)
     Running `rustc src/lib.rs --crate-name example --crate-type dylib -g -C metadata=8ae2483402690d84 --out-dir /tmp/test/example/target/debug/deps --emit=dep-info,link -L dependency=/tmp/test/example/target/debug/deps --extern cpython=/tmp/test/example/target/debug/deps/libcpython-0781db0248e50d1b.rlib -L native=/usr/lib`
src/lib.rs:5:33: 5:44 error: no rules expected the token `initexample` 
src/lib.rs:5 py_module_initializer!(example, initexample, PyInit_example, |py, m| {
                                             ^~~~~~~~~~~
error: Could not compile `example`.

Caused by:
  Process didn't exit successfully: `rustc src/lib.rs --crate-name example --crate-type dylib -g -C metadata=8ae2483402690d84 --out-dir /tmp/test/example/target/debug/deps --emit=dep-info,link -L dependency=/tmp/test/example/target/debug/deps --extern cpython=/tmp/test/example/target/debug/deps/libcpython-0781db0248e50d1b.rlib -L native=/usr/lib` (exit code: 101)

When I call the macro with the first and last argument only (py_module_initializer!(example, |py, m| {), I get the following error:

rustup run nightly -- cargo build --verbose
       Fresh utf8-ranges v0.1.3
       Fresh winapi-build v0.1.1
       Fresh num-traits v0.1.34
       Fresh winapi v0.2.8
       Fresh rustc-serialize v0.3.19
       Fresh num-integer v0.1.32
       Fresh regex-syntax v0.3.4
       Fresh num-iter v0.1.32
       Fresh num-complex v0.1.33
       Fresh interpolate_idents v0.1.2
       Fresh kernel32-sys v0.2.2
       Fresh libc v0.2.15
       Fresh abort_on_panic v1.0.0
       Fresh rand v0.3.14
       Fresh thread-id v2.0.0
       Fresh memchr v0.1.11
       Fresh num-bigint v0.1.33
       Fresh thread_local v0.2.6
       Fresh aho-corasick v0.5.2
       Fresh num-rational v0.1.32
       Fresh regex v0.1.73
       Fresh num v0.1.34
       Fresh python3-sys v0.1.2
       Fresh cpython v0.0.5
   Compiling example v0.1.0 (file:///tmp/test/example)
     Running `rustc src/lib.rs --crate-name example --crate-type dylib -g -C metadata=8ae2483402690d84 --out-dir /tmp/test/example/target/debug/deps --emit=dep-info,link -L dependency=/tmp/test/example/target/debug/deps --extern cpython=/tmp/test/example/target/debug/deps/libcpython-0781db0248e50d1b.rlib -L native=/usr/lib`
<cpython macros>:2:1: 2:19 error: macro undefined: 'interpolate_idents!' 
<cpython macros>:2 interpolate_idents ! {
                   ^~~~~~~~~~~~~~~~~~
src/lib.rs:5:1: 9:4 note: in this expansion of py_module_initializer! (defined in <cpython macros>)
error: aborting due to previous error 
error: Could not compile `example`.

Caused by:
  Process didn't exit successfully: `rustc src/lib.rs --crate-name example --crate-type dylib -g -C metadata=8ae2483402690d84 --out-dir /tmp/test/example/target/debug/deps --emit=dep-info,link -L dependency=/tmp/test/example/target/debug/deps --extern cpython=/tmp/test/example/target/debug/deps/libcpython-0781db0248e50d1b.rlib -L native=/usr/lib` (exit code: 101)

abort when importing modules in Python 2 inside a virtualenv

If I attempt to compile a module for Python 2, and import it in a python running out of a virtualenv, I get an abort:

% env/bin/python2 -c 'import _foo'
Fatal Python error: PyThreadState_Get: no current thread
zsh: abort      env/bin/python2 -c 'import _foo'

but the system Python works just fine:

% python2 -c 'import _foo'
%

They should be mostly the same:

% python2
Python 2.7.12 (default, Jun 29 2016, 14:04:44)
[GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on darwin
% env/bin/python2
Python 2.7.10 (default, Jul 14 2015, 19:46:27)
[GCC 4.2.1 Compatible Apple LLVM 6.0 (clang-600.0.39)] on darwin

(this is an virtualenv created with virtualenv env; I don't see these issues in Python 3, with a python3 running inside a pyvenv venv virtual environment.)

Fatal Python error: PyThreadState_Get: no current thread

I'm trying to get a hello world example running but have been unsuccessful so far. My code is here for reference https://github.com/justinhalf/py-rust-hello-world

I'm able to cargo build but when I try to import in python, I get:

PYTHONPATH=target/debug/ python -c "import libhello_world"
Fatal Python error: PyThreadState_Get: no current thread
Abort trap: 6

After I compiled the rust code, I didn't get a .so file. I'm working around that for now by doing (cd target/debug/ && ln -s libhello_world.dylib libhello_world.so). I also get a compile failure unless I export PKG_CONFIG_PATH=/Users/justin/anaconda/envs/rust-python/lib/pkgconfig/.

I tried some googling and it seems like it could be related to the rust code being linked against the wrong python libraries. I am running anaconda and using the rust-python environment that I pointed the PKG_CONFIG_PATH variable at. I'm using rust-nightly and am on OSX.

I'm a bit stuck at this point. Any suggestions would be greatly appreciated.

pylong

PyObject::extract::<Vec<i32>>() cannot be called

src/objects/list.rs:252:37: 252:58 error: the requirement `for<'s>  : 's` is not satisfied (`expected bound lifetime parameter 's, found concrete lifetime`) [E0279]
src/objects/list.rs:252         let v2 = list.into_object().extract::<Vec<i32>>().unwrap();

I've looked into this issue, but I don't think it can be solved at the moment -- in the end, all my attempts run into the problem that I need either 'source : 'prepared, and such a constraint is incompatible with the use of the HRTB on PyObject::extract.

I've even tried to get rid of the 'source lifetime on ExtractPyObject (see extract_vec branch); but that only moved the problem to the 'python : 'prepared constraint.

Maybe the problem will finally be solvable if I also get rid of the 'python lifetime (#15).

This is how I'm accessing Py_buffer objects [question]

This is how I'm accessing Py_buffer objects.
It's definitely working but I feel uncomfortable using the _detail module because the code clearly says it should not be used directly.

fn get_pybuffer(po:&PyObject) -> Option<_detail::ffi::Py_buffer>
{
    let ptr = po.as_ptr();
    unsafe {
        let mut buff: _detail::ffi::Py_buffer = mem::zeroed();
        match (*(*(*ptr).ob_type).tp_as_buffer).bf_getbuffer {
            Some(buffproc) => {
                buffproc(ptr, &mut buff as *mut _detail::ffi::Py_buffer, _detail::ffi::PyBUF_STRIDES | _detail::ffi::PyBUF_FORMAT | _detail::ffi::PyBUF_WRITABLE);
                Some(buff)
            },
            None => None
        }
    }
}

My plan is to create something similar to the buffer_info object from pybind11 but if there's a better way please let me know. I'm also wondering if I should just call PyObject_GetBuffer directly as opposed to all that .ob_type.tp_as_buffer.bf_getbuffer

PS: I'm relatively new to rust so, please bear with me if I'm missing something obvious.

pub py_class!

I found this:

rust-cpython/src/py_class/py_class_impl3.rs

Line 43 in 623056d

struct $class { _unsafe_inner: $crate::PyObject }

So there is no way to write py_class in some file and then import it in another? Because now I have:

error: struct `MyType` is private
  --> src/lib.rs:28:5
   |
28 | use self::module::MyType;
   |     ^^^^^^^^^^^^^^^^^^^^^^

when py_class!(class MyType |py| { is written in module/mod.rs file.

Or did I miss something and there is a way to create public class?

python 3 build support

Just FYI I've started working on this over at https://github.com/novocaine/rust-cpython.

I think we can parametrise the python27-sys build.rs so it can be identical across python3-sys and python27-sys, with the parameters being picked up by enabled features in the pythonX-sys/Cargo.toml. The python 3 Cargo.toml needs to have python version features to indicate which python 3 the user is trying to target via pkg-config.

I think there should be something in the doco about how the build works to stop people from having to read the build.rs, but not sure whether to put it in the README.md or in the generated docs.

Lifetimes in py_class!

Sorry if I don't know either Rust or rust-cpython but how can I use py_class!with lifetime params inside it?
I want a variable s to live as long as class MyType.

py_class!(class MyType |py| {
    data s: Into<Cow<'a, str>>;
}

I wanted to write the whole class manually like

pub struct MyType<'a> {...}

But then I saw that even a macros was generated by Python script and I ran away crying...

Typo in README

On Mac OS, you will need to rename the output from *.dynlib to *.so

Shouldn't this be "dylib", not "dynlib"?

Put `py: Python` argument at beginning or end of argument list?

Now that #15 is adding a py: Python argument to almost every function, we should pick a consistent position for that argument.

Beginning: fn(Python, ...)
End: fn(..., Python)

Calls seem more readable to me in the beginning position:
some_method(py, nested_call(py, long, argument, list))
vs.
some_method(nested_call(long, argument, list, py), py)

If we want the non-Copy Python token, and use the beginning position, nested calls will fail to compile with Rust's current reborrowing semantics. This might or might not be fixed in a future Rust version.

However, there are good reasons for Python to remain Copy, so I think the beginning position is the correct choice.

Implement Display and Debug traits

These traits are extremely useful and would be great for debugging issues.