go-python / gopy Goto Github PK

View Code? Open in Web Editor NEW

1.9K 48.0 111.0 4.31 MB

gopy generates a CPython extension module from a go package.

License: BSD 3-Clause "New" or "Revised" License

Go 87.64% Python 11.68% Dockerfile 0.04% Makefile 0.64%

go golang python python-2 python-3 pypy cffi ctypes

gopy's Introduction

go-python

Naive go bindings towards the C-API of CPython-2.

this package provides a go package named "python" under which most of the PyXYZ functions and macros of the public C-API of CPython have been exposed.

theoretically, you should be able to just look at:

http://docs.python.org/c-api/index.html

and know what to type in your go program.

this package also provides an executable "go-python" which just loads "python" and then call python.Py_Main(os.Args). the rational being that under such an executable, go based extensions for C-Python would be easier to implement (as this usually means calling into go from C through some rather convoluted functions hops)

Install

With Go 1 and the go tool, cgo packages can't pass anymore additional CGO_CFLAGS from external programs (except pkg-config) to the "fake" #cgo preprocessor directive.

go-python now uses pkg-config to get the correct location of headers and libraries. Unfortunately, the naming convention for the pkg-config package is not standardised across distributions and OSes, so you may have to edit the cgoflags.go file accordingly.

 $ go get github.com/sbinet/go-python

If go get + pkg-config failed:

 $ cd go-python
 $ edit cgoflags.go
 $ make VERBOSE=1

Note: you'll need the proper header and python development environment. On Debian, you'll need to install the python-all-dev package

Documentation

Is available on godoc:

http://godoc.org/github.com/sbinet/go-python

Example:

package main

import "fmt"
import "github.com/sbinet/go-python"

func init() {
   err := python.Initialize()
   if err != nil {
          panic(err.Error())
   } 
}

func main() {
 	 gostr := "foo" 
	 pystr := python.PyString_FromString(gostr)
	 str := python.PyString_AsString(pystr)
	 fmt.Println("hello [", str, "]")
}

$ go run ./main.go
hello [ foo ]

TODO:

fix handling of integers (I did a poor job at making sure everything was ok)
add CPython unit-tests
do not expose C.FILE pointer and replace it with os.File in "go-python" API
provide an easy way to extend go-python with go based extensions
think about the need (or not) to translate CPython exceptions into go panic/recover mechanism
use SWIG to automatically wrap the whole CPython api ?

gopy's People

Contributors

Stargazers

Watchers

Forkers

justinfx ericsnowcurrently vkolev kellrott kaleidicforks lhartung ashrafulratul readthecodes ryszard pombredanne mbeacom sublimetech senmumu corona10 mewbak zhangdong046 freddiev4 elerch mingrammer lifei themushrr00m devopsmi initialed85 jackrobison pythonai anshuman23 jonike rajgomase24 abourget karlmutch rootshaxor usexuelang klemenswinter ondra baifengbai dzonerzy codelingobot lfzark glycerine fionafibration hoangpq goki sdementen friendlydan isgasho tomoris cosnicolaou 0to1 awesomegolang keithcat1 tedlee1024 renesugar krzemienski lhw362950217 ezc rcoreilly raychorn ziux seemethere org-mars hhuangwx jdmeyer3 sbarikdev juicestus chslink mrttog astrelsky crayonzx cricket-org leemit hy9be hakwolf pyalex feast-dev python-repository-hub theodoree zhangjyr abichinger alexjx rudderlabs almogbaku kjahed cnpryer bidaya0 noamk-cr psha- soren-n theradius cwing2019 bjia56 mlange-42 603713921 4thel00z zmingdi vmarkovtsev achillesxu iq-scm djun foxever acimnotes

gopy's Issues

support comma-err idiom

commar-error idiom should be translated into proper python exceptions when err != nil

wrap top-level go.consts

and implement via a dedicated pyobject with getter but no setter.
one may loose precision... (use python's math/big equivalent?)

bind: support for all builtin types

Currently rune, uintptr, and unsafe.Pointer are not supported.

The package

package types
var X rune

fails with

$ go build && gopy bind --lang=py2 .
2015/08/31 21:18:05 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-283258537
# _/private/var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-283258537
./types.c:150:2: error: use of undeclared identifier 'GoRune'
./types.c:152:2: error: use of undeclared identifier 'c_gopy_ret'
./types.c:154:29: error: use of undeclared identifier 'cgopy_cnv_c2py_rune'
./types.c:161:2: error: use of undeclared identifier 'GoRune'
./types.c:163:36: error: use of undeclared identifier 'cgopy_cnv_py2c_rune'
./types.c:168:23: error: use of undeclared identifier 'c_X'
2015/08/31 21:18:06 error dispatching command: exit status 2

The package

package types
var X uintptr

fails with

$ go build && gopy bind --lang=py2 .
panic: builtin type not already known [uintptr]!

goroutine 1 [running]:
github.com/go-python/gopy/bind.(*symtab).addType(0x8205cf000, 0x8820776a40, 0x820621ef0, 0x8820776318, 0x53dc00)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:363 +0x7c9
github.com/go-python/gopy/bind.(*symtab).addSymbol(0x8205cf000, 0x8820776a40, 0x820621ef0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:308 +0x456
github.com/go-python/gopy/bind.(*Package).process(0x820648dc0, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/package.go:179 +0x26dd
github.com/go-python/gopy/bind.NewPackage(0x820621e50, 0x820661880, 0x14, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/package.go:44 +0x390
main.newPackageFrom(0x8205e80a0, 0x1, 0x1, 0x820648c80, 0x820662000, 0x8205e80a0, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:209 +0x3bf
main.newPackage(0x7fff5fbffa5d, 0x1, 0x8205d2064, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:179 +0x638
main.gopyRunCmdBind(0x8206327e0, 0x8205ca2f0, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/cmd_bind.go:73 +0x4eb
github.com/gonuts/commander.(*Command).Dispatch(0x8206327e0, 0x8205ca2f0, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:221 +0x7e2
github.com/gonuts/commander.(*Command).Dispatch(0x820632900, 0x8205ca2d0, 0x3, 0x3, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:187 +0x1f9
main.main()
    /Users/phil/code/go/src/github.com/go-python/gopy/main.go:38 +0x1fe

The package

package types
import "unsafe"
var X unsafe.Pointer

fails with

$ go build && gopy bind --lang=py2 .
panic: builtin type not already known [unsafe.Pointer]!

goroutine 1 [running]:
github.com/go-python/gopy/bind.(*symtab).addType(0x8205cf180, 0x8820776b70, 0x820621ef0, 0x8820776318, 0x53dcc0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:363 +0x7c9
github.com/go-python/gopy/bind.(*symtab).addSymbol(0x8205cf180, 0x8820776b70, 0x820621ef0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:308 +0x456
github.com/go-python/gopy/bind.(*Package).process(0x820648e60, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/package.go:179 +0x26dd
github.com/go-python/gopy/bind.NewPackage(0x820621e50, 0x820661880, 0x14, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/package.go:44 +0x390
main.newPackageFrom(0x8205e80b0, 0x1, 0x1, 0x820648d20, 0x820662000, 0x8205e80b0, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:209 +0x3bf
main.newPackage(0x7fff5fbffa5d, 0x1, 0x8205d2064, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:179 +0x638
main.gopyRunCmdBind(0x8206326c0, 0x8205ca2f0, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/cmd_bind.go:73 +0x4eb
github.com/gonuts/commander.(*Command).Dispatch(0x8206326c0, 0x8205ca2f0, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:221 +0x7e2
github.com/gonuts/commander.(*Command).Dispatch(0x8206327e0, 0x8205ca2d0, 0x3, 0x3, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:187 +0x1f9
main.main()
    /Users/phil/code/go/src/github.com/go-python/gopy/main.go:38 +0x1fe

Probably the best thing to hand a Python program that asks for an unsafe.Pointer is a ctypes.c_void_p.

support for kwargs calls

gopy should perhaps generate code with support for kwargs.

consider:

package pkg
func Hello(arg1, arg2 string) string {...}

should be callable from python like so:

import pkg
pkg.Hello("1","2")
pkg.Hello(arg1="1", arg2="2")

os.File, io.Reader and io.Writer pythonizations

structs containing *os.File or functions/methods returning *os.File (or equivalent interface) should expose a python object appearing as file-like and implementing the context protocol.
ditto for io.Reader and io.Writer.

bind: support for private fields

The following package

package private

type Test struct {
    Public int
    private int
}

fails with

$ gopy bind --lang=py2 .
2015/08/31 10:15:15 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-571736582
# _/private/var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-571736582
./private.c:224:7: warning: implicit declaration of function 'cpy_func_private_Test_setter_2' is invalid in C99 [-Wimplicit-function-declaration]
# _/private/var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-571736582
Undefined symbols for architecture x86_64:
  "_cpy_func_private_Test_setter_2", referenced from:
      _cpy_func_private_Test_init in private.o
ld: symbol(s) not found for architecture x86_64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
2015/08/31 10:15:17 error dispatching command: exit status 2

bind: support for (unexported) typed consts, vars and funcs

right now:

package consts
type kind int
const (
  Kind kind = 0
)

results in:

sh> gopy bind ./_examples/consts
./consts.go:144: cannot refer to unexported name consts.kind
./consts.go:150: cannot refer to unexported name consts.kind

ditto for (exported) vars of unexported type and (exported) funcs returning values of unexported types.

gopy bind performances

on my desktop, generating+binding the C-extention module takes:

sh> time gopy bind github.com/go-python/gopy/_examples/simple
2015/08/06 17:55:13 work: /home/binet/dev/go/root/tmp/gopy-129374957
runtime
sync/atomic
errors
math
unicode/utf8
github.com/go-python/gopy/_examples/simple
sync
io
syscall
strconv
time
reflect
os
fmt
runtime/cgo
_/home/binet/dev/go/root/tmp/gopy-129374957

real    0m5.098s
user    0m8.107s
sys 0m0.707s

the current breakdown is:

sh> time gopy gen -lang=go github.com/go-python/gopy/_examples/simple

real    0m0.076s
user    0m0.047s
sys 0m0.017s

sh> time gopy gen -lang=py2 github.com/go-python/gopy/_examples/simple

real    0m0.004s
user    0m0.003s
sys 0m0.000s

sh> time go build -v -buildmode=c-shared -o simple.so .
runtime
errors
sync/atomic
math
github.com/go-python/gopy/_examples/simple
unicode/utf8
sync
io
syscall
strconv
time
reflect
os
fmt
runtime/cgo
github.com/go-python/gopy/tbench

real    0m5.035s
user    0m8.220s
sys 0m0.640s

we should investigate where one could optimize...

bind: refactor codegen

refactor codegen.

introduce bind.Struct to collect all methods + ctor-funcs related to a go.Struct
introduce bind.Func for all go.Funcs (but the ctor-funcs)
introduce bind.Type to tie the 2?
introduce bind.Convert to handle (codegen) conversion b/w types?
lift all go.values to pointers so handling is easier in the cgo-code?
introduce bind.Block to model a basic block/scope
attach DeclVar(name, type), Call(recv,args), Return{Results []Value} functions and types to operate on a bind.Block

type Block struct {
 blocks []*Block // sub-blocks
 instrs []Instruction
}

// Instruction is a statement that consumes values and performs a computation.
type Instruction interface {
  Operands(rands []*Value) []*Value
}

// Value is an expression that yields a value
type Value interface {
  Name() string
  Type() types.Type
}

type DeclVar struct {
  n string
  t types.Type
}

type Call struct {
  recv Value
  args []*Value 
}

// ...

another possibility is to build a full AST as Cython does (here)
(need some more design legwork)

bind: support slices as method arguments

The package

package slice_arg

type T struct {
}

func (t T) Slice(s []float64) {
}

fails with

$ go build && gopy bind --lang=py2 .
2015/08/31 10:29:15 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-628775343
# _/private/var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-628775343
./slice_arg.go:164: cannot use s (type cgo_type_0x1114782197) as type []float64 in argument to (*slice_arg.T)(unsafe.Pointer(self)).Slice
2015/08/31 10:29:15 error dispatching command: exit status 2

The following two packages both successfully compile:

package slice_arg

type T struct {
}

func Slice(t T, s []float64) {
}

package slice_arg

type T struct {
}

func (t T) Slice(x float64) {                                            
}

gopy: dont crash interpreter when reloading a C-extension module

while waiting for #27 to be fixed, gopy.load should prevent a user to reload a C-extension module.

wrap top-level go.values

gopy should generate wrappers for top-level go.values.
needs proper support for go.structs.

go.slices support for sq_inplace_concat

ie: go's append (and python's __iadd__)

gopy: support for empty packages

right now, attempting to wrap an empty package, results in:

sh> gopy bind ./_examples/empty
./empty.go:18: imported and not used: "github.com/go-python/gopy/_examples/empty"

we should support this (as importing empty packages may be needed for their side-effects)

bind: unify struct and named-types handling

right now, the codegen path for types.Struct and types.Named is different.
but the only difference (IIRC) is that types.Struct may have fields (which we need to wrap via getters and setters.)

bind: support for embedding

The package

package inheritance

type A struct {
}

func (a A) Method() {
}

type B struct {
    A
}

fails with

$ go build && gopy bind --lang=py2 . && python test.py 
2015/08/31 13:54:17 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-602654280
# _/private/var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-602654280
./inheritance.c:482:26: error: use of undeclared identifier 'cpy_func_inheritance_B_Method'
2015/08/31 13:54:18 error dispatching command: exit status 2

Similarly,

package inheritance

type A struct {
    AField int
}

type B struct {
    A
}

will successfully bind, but the python code

from inheritance import *

b = B()
print b.AField

fails with the expected attribute error:

$ go build && gopy bind --lang=py2 . && python test.py 
2015/08/31 13:52:52 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-277918738
Meow
Traceback (most recent call last):
  File "test.py", line 5, in <module>
    print b.AField
AttributeError: 'inheritance.B' object has no attribute 'AField'

bind: collect arrays and slices into type factory functions

right now, slices and arrays defined like so from go:

package hi
var (
 Array [2]int
 Slice []int
)

appear like so on python:

>>> import gopy
>>> hi = gopy.load("some/path/hi")
>>> dir(hi)
['GetArray', 'SetArray', 'GetSlice', 'SetSlice', '[2]int', '[]int']

one should collect the type(s) [X]int under a module-level function available like so:

# module hi
def array(size, type):
  return _go_array_types[(size,type)]

and useable like so:

import hi
arr = hi.array(2, int)()

instead of like so:

import hi
arr = getattr(hi, '[2]int')()

ditto for slices.

Doc says Java, I think you mean Python

https://godoc.org/github.com/go-python/gopy-gen

gopy: python module facade

consider creating a gopy python module facade to auto-(generate+compile+import) any go package from the python prompt (or python script)
or inject a python import hook?

e.g. (tentative api):

import gopy
hi = getattr(gopy, "github.com/go-python/gopy/_examples/hi")
hi = gopy.Import("github.com/go-python/gopy/_examples/hi")

bind: support for empty structs

right now, wrapping:

package structs

type S struct {}

results in:

./structs.c: In function ‘cpy_func_structs_S_init’:
./structs.c:1152:60: error: expected expression before ‘)’ token
  if (!PyArg_ParseTupleAndKeywords(args, kwds, "|", kwlist, )) {

bind: decide on handling of builtins

right now, builtin types (float32, int, ...) are passed by value between go, cgo and cpython.

it worked ok so far, until we try to handle named types of these builtins.
e.g.:

type Float float32
func (f Float) Value() float32 { return float32(f) }

we could still handle them by value if it were not for the possibility (on the Go side) to attach methods to them, so, on the python side, we'd need a proper cpy_type_mypkg_FloatType with the equivalent methods)
(note: we could still have that but wrap the Float by value. but what about ref-counting?)

the most regular (and less surprising) option is to:

pass them by pointer
inject cgo_incref calls
create a cpy_type_mypkg_Float cpython type
implement the tp_as_number protocol (when sensible)

go.Stringer should be wrapped as `str`

bind: wrap methods of named types

bind: use golang.org/x/mobile/bind/seq for cross-language call

gopy currently (mostly) sends pointers to go values back and forth between cgo and go.
on the cpython side, such types are (usually) defined like so:

typedef void *cgo_type_pkg_MyType;

struct cpy_type_pkg_MyType {
  PyObject_HEAD
  cgo_type_pkg_MyType cgopy; /* pointer to pkg.MyType */
};

under the rules currently defined at golang/go#12416, this will no longer work as we store a pointer to go memory, possibly containing itself pointers to go memory.

so we'll probably have to bite the bullet and move to the (de)serialization protocol of golang.org/x/mobile/bind/seq.

go.slices + go.arrays support for sq_slice, sq_ass_slice

support for sq_slice (ie: sub-slicing)
support for sq_ass_slice

bind: support for nested slices

The package

package nested_slice

func NestedSlice(s [][]float64) {
}

fails with

$ go build && gopy bind --lang=py2 .
panic: could not look-up "[]float64"!


goroutine 1 [running]:
github.com/go-python/gopy/bind.(*symtab).addSliceType(0x8205c5100, 0x820617ea0, 0x882076cad8, 0x820666000, 0x882076caa8, 0x8205c6d80, 0x808, 0x8205c5160, 0x18, 0x8205c6f50, ...)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:483 +0xacc
github.com/go-python/gopy/bind.(*symtab).addType(0x8205c5100, 0x882076cad8, 0x820666000, 0x882076caa8, 0x8205c6d80)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:366 +0x2aa
github.com/go-python/gopy/bind.(*symtab).processTuple(0x8205c5100, 0x8205c4de0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:337 +0x149
github.com/go-python/gopy/bind.(*symtab).addSymbol(0x8205c5100, 0x882076c408, 0x820617f40)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/symtab.go:317 +0xceb
github.com/go-python/gopy/bind.(*Package).process(0x82063edc0, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/package.go:179 +0x26a4
github.com/go-python/gopy/bind.NewPackage(0x820617ea0, 0x820657880, 0x19, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/package.go:44 +0x390
main.newPackageFrom(0x8205de0a8, 0x1, 0x1, 0x82063ec80, 0x820658000, 0x8205de0a8, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:197 +0x3bf
main.newPackage(0x7fff5fbffa5d, 0x1, 0x8205c0144, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:167 +0x638
main.gopyRunCmdBind(0x820628a20, 0x8205c0330, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/cmd_bind.go:73 +0x4eb
github.com/gonuts/commander.(*Command).Dispatch(0x820628a20, 0x8205c0330, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:221 +0x7e2
github.com/gonuts/commander.(*Command).Dispatch(0x820628b40, 0x8205c0310, 0x3, 0x3, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:187 +0x1f9
main.main()
    /Users/phil/code/go/src/github.com/go-python/gopy/main.go:38 +0x1fe

bind: implement buffer protocol for structs

implementing the (new) buffer protocol for structs would allow to share the underlying data (w/o) copies to e.g. a sqlite db from/to python.

gopy: support for cgo-based packages

right now, a package importing "C" can not be (directly) wrapped:

sh> gopy bind ./_examples/cpkg
2015/08/10 09:26:25 _examples/cpkg/cpkg.go:13:8: could not import C (the loader doesn't cgo-process ad hoc packages like "github.com/go-python/gopy/_examples/cpkg"; see Go issue 11627)
2015/08/10 09:26:25 _examples/cpkg/cpkg.go:25:10: undeclared name: C
2015/08/10 09:26:25 _examples/cpkg/cpkg.go:26:8: undeclared name: C
2015/08/10 09:26:25 _examples/cpkg/cpkg.go:27:2: undeclared name: C
2015/08/10 09:26:25 couldn't load packages due to errors: github.com/go-python/gopy/_examples/cpkg
2015/08/10 09:26:25 error dispatching command: gopy-bind: go/build.Import failed with path="./_examples/cpkg": couldn't load packages due to errors: github.com/go-python/gopy/_examples/cpkg

this is because of golang/go#11627.

bind: support for typed constants and named vars

right now:

package consts
type Kind int
const (
  Kind1 Kind = 0
)

results in:

sh> gopy bind ../_examples/consts
./consts.go:172: cannot use consts.Kind1 (type consts.Kind) as type cgo_type_consts_Kind in return argument

dittor for vars.

bind: support funcs taking pointers

right now:

package pointers
type S struct {
    Value int
}

// Inc increments the value of s
func Inc(s *S) {
    s.Value++
}

results in:

>>> import pointers as p
>>> s = p.S(2)
>>> p.Inc(s)
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xb code=0x1 addr=0x2 pc=0x7f1cc7c47fe7]

goroutine 17 [running, locked to thread]:
github.com/go-python/gopy/_examples/pointers.Inc(0x2)
    github.com/go-python/gopy/_examples/pointers/pointers.go:18 +0x7a
main.cgo_func_pointers_Inc(0xc82000a340)
    tmp/gopy-586713227/pointers.go:151 +0x26

goroutine 18 [syscall, locked to thread]:
runtime.goexit()
    go/src/runtime/asm_amd64.s:1696 +0x1
Aborted (core dumped)

that's because of the following cgo code:

//export cgo_func_pointers_Inc
// cgo_func_pointers_Inc wraps pointers.Inc
func cgo_func_pointers_Inc(s cgo_type_pointers_S) () {
    pointers.Inc(*(**pointers.S)(unsafe.Pointer(s)))
}

which should, instead, read:

//export cgo_func_pointers_Inc
// cgo_func_pointers_Inc wraps pointers.Inc
func cgo_func_pointers_Inc(s cgo_type_pointers_S) () {
    pointers.Inc((*pointers.S)(unsafe.Pointer(s)))
}

support for slices and arrays

go.slices and go.arrays should be wrapped and their python counterpart should implement the sequence protocol.
slices and arrays should probably also implement the buffer protocol.

distribution difference: python2.pc vs. python2.7.pc

Testing on an Amazon Linux development instance, I noticed it puts Python 2.7's package configuration in python2.7.pc, whereas Ubuntu calls it python2.pc. I was able to bind the hi example just by changing the python2 in gengo.go to python2.7. I'm guessing Amazon Linux inherits this quirk from its more-widely-used upstreams (CentOS/RHEL), but haven't checked that.

gopy could test for the existence of /usr/lib64/python2.7.pc or /usr/lib/python2.7.pc, and, if one of those is present, put python2.7 instead of python2 in the #cgo pkg-config line in the generated go file.

(It might help some new users if gopy suggested how to install Python headers and pkg-config if they appeared to be missing after a failed build. Some gopy users may be building their first extension.)

gopy: consider support of a CFFI backend

prompted by: https://www.reddit.com/r/Python/comments/3ig3ww/building_python_modules_with_go_15_xpost_from/cugc9ri

it could be interesting to have a CFFI backend for pypy's benefit.
maybe simpler to just tap into cppyy ?

see https://github.com/jbuberel/buildmodeshared/tree/master/gofrompython for a CFFI-based example

go.structs -- wrap fields

(exposed) fields of go structs should be wrapped via python descriptors (tp_getset in python2)

bind: support for variadic arguments

The package

package variadic

func Variadic(s ...float64) {
}

fails with

go build && gopy bind --lang=py2 .
2015/08/31 10:36:03 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-208401189
# _/private/var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-208401189
./variadic.go:159: cannot use *(*[]float64)(unsafe.Pointer(s)) (type []float64) as type float64 in argument to variadic.Variadic
2015/08/31 10:36:03 error dispatching command: exit status 2

handling of packages using other packages' types or values

consider:

// pkg1/pk1.go
package pkg1

const C = 42
var V = 42
type T struct {}

and:

// pkg2/pkg2.go
package pkg2

import ( 
    "github.com/some/pkg1"
)

func NewT() pkg1.T {...}

what should gopy do?

pull in the whole pkg1 package and wrap it as part of pkg2.so ?
wrap pkg2.NewT and return a PyObject/PyDict? an opaque PyCObject ?
wrap pkg2.NewT and return a PyCapsule ?
bail out ?
something else ?

https://docs.python.org/2/c-api/capsule.html#capsules

wrap go.maps (tp_as_sequence, tp_as_dict)

go.maps should be wrapped and their python counterpart should implement the sequence and dict protocols.

bind: support for go.interfaces

right now, the following package:

package iface

type Iface interface {
    F()
}

fails with:

sh> gopy bind ./_examples/iface
panic: unhandled named-type: [*types.TypeName]
&types.Named{obj:(*types.TypeName)(0xc8200b84b0), underlying:(*types.Interface)(0xc8200b8500), methods:[]*types.Func(nil)}


goroutine 1 [running]:
github.com/go-python/gopy/bind.(*symtab).addType(0xc820011400, 0x7f3629be9348, 0xc8200b84b0, 0x7f3629be9538, 0xc82000b470)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/bind/symtab.go:360 +0xdc4
github.com/go-python/gopy/bind.(*symtab).addSymbol(0xc820011400, 0x7f3629be9348, 0xc8200b84b0)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/bind/symtab.go:291 +0x9d8
github.com/go-python/gopy/bind.(*Package).process(0xc8200af560, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/bind/package.go:176 +0x12bf
github.com/go-python/gopy/bind.NewPackage(0xc8200b8410, 0xc8200bc0e0, 0x29, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/bind/package.go:42 +0x388
main.newPackageFrom(0xc8200300b0, 0x1, 0x1, 0xc82009cc80, 0xc8200b6000, 0xc8200300b0, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/gen.go:191 +0x24c
main.newPackage(0x7fff034ca8c8, 0x11, 0xc820012044, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/gen.go:164 +0x2f9
main.gopyRunCmdBind(0xc820016900, 0xc82000a380, 0x1, 0x1, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/cmd_bind.go:73 +0x4eb
github.com/gonuts/commander.(*Command).Dispatch(0xc820016900, 0xc82000a380, 0x1, 0x1, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/gonuts/commander/commands.go:221 +0x7e2
github.com/gonuts/commander.(*Command).Dispatch(0xc820016a20, 0xc82000a370, 0x2, 0x2, 0x0, 0x0)
    /home/binet/dev/go/root/path/src/github.com/gonuts/commander/commands.go:187 +0x1f9
main.main()
    /home/binet/dev/go/root/path/src/github.com/go-python/gopy/main.go:38 +0x1fe

bind: support for private methods

The package

package private_method

type T int

func (t T) PublicMethod() {
}

func (t T) privateMethod() {
}

fails with

$ gopy bind --lang=py2 .
2015/08/31 10:19:12 work: /var/folders/j0/l297p36x68dfmcrh07jptj200000gn/T/gopy-501072551
panic: gopy: could not find symbol for [*types.Signature] (&types.Signature{scope:(*types.Scope)(0x82067e410), recv:(*types.Var)(0x82067e460), params:(*types.Tuple)(nil), results:(*types.Tuple)(nil), variadic:false}) (privateMethod || (scratch/pytest/private_method.T).privateMethod)

goroutine 1 [running]:
github.com/go-python/gopy/bind.(*goGen).genTypeMethods(0x8206a35a8, 0x820677960)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/gengo.go:798 +0x44f
github.com/go-python/gopy/bind.(*goGen).genType(0x8206a35a8, 0x820677960)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/gengo.go:665 +0x2010
github.com/go-python/gopy/bind.(*goGen).gen(0x8206a35a8, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/gengo.go:153 +0x85c
github.com/go-python/gopy/bind.GenGo(0x88207a8140, 0x820648150, 0x82064c3c0, 0x82065efa0, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/bind/bind.go:64 +0x194
main.genPkg(0x82064d140, 0x3f, 0x82065efa0, 0x39ae08, 0x2, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/gen.go:65 +0x43a
main.gopyRunCmdBind(0x8206367e0, 0x8205c0370, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/go-python/gopy/cmd_bind.go:113 +0xd93
github.com/gonuts/commander.(*Command).Dispatch(0x8206367e0, 0x8205c0370, 0x1, 0x1, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:221 +0x7e2
github.com/gonuts/commander.(*Command).Dispatch(0x820636900, 0x8205c0350, 0x3, 0x3, 0x0, 0x0)
    /Users/phil/code/go/src/github.com/gonuts/commander/commands.go:187 +0x1f9
main.main()
    /Users/phil/code/go/src/github.com/go-python/gopy/main.go:38 +0x1fe```

go.structs with multiple constructors

gopy should generate the proper dispatch code for structs with multiple constructor functions

it's actually a bit more complicated than that.
consider:

type S struct {i int}
func NewS1() S { return S{i:1} }
func NewS2() S { return S{i:2} }

there is no way to distinguish between the 2 at the python level.

better just expose the ctors at the py-module level?

or attach them at the class-level like so?

class S(object):
  @staticmethod
  def NewS1(): ...

  @staticmethod
  def NewS2(): ...

(but then, collisions may happen)

bind: support for ctors for named types

consider:

package named
type T float32
type TT T

there is currently no way to create a value v (with some specific value) of type T (or TT) from python.

Limit code scanning to package/elements to be exported

I've created an example (https://github.com/kellrott/gopy/tree/wrapper/_examples/wrapper) with two packages, wrapper and wrapper_python.
wrapper has several features that gopy doesn't support (pointers to structures, channels) and is imported by wrapper_python, which simplifies and hides the interaction with those elements.
When running the command to bind wrapper_python

gopy bind github.com/go-python/gopy/_examples/wrapper/python

I get the error:

panic: could not look-up "*github.com/go-python/gopy/_examples/wrapper.RealStruct"!


goroutine 1 [running]:
github.com/go-python/gopy/bind.(*symtab).addMapType(0x823df3ce0, 0x8205d6320, 0x8820774278, 0x8205d65f0, 0x8820774218, 0x8205d0660, 0x88, 0x823e5bf40, 0x4a, 0x823e5bef0, ...)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:497 +0xa35
github.com/go-python/gopy/bind.(*symtab).addType(0x823df3ce0, 0x8820774278, 0x8205d65f0, 0x8820774218, 0x8205d0660)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:440 +0x326
github.com/go-python/gopy/bind.(*symtab).addStructType(0x823df3ce0, 0x820647bd0, 0x8820774278, 0x823e5b7c0, 0x88207b8510, 0x8205cc8a0, 0x1108, 0x823e5be00, 0x45, 0x823f50a80, ...)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:575 +0x27a
github.com/go-python/gopy/bind.(*symtab).addType(0x823df3ce0, 0x8820774278, 0x823e5b7c0, 0x88207b8510, 0x8205cc8a0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:383 +0xfd6
github.com/go-python/gopy/bind.(*symtab).addPointerType(0x823df3ce0, 0x820647bd0, 0x8820774278, 0x823e5b7c0, 0x8820774248, 0x823e41fb0, 0x8, 0x823e5bdb0, 0x46, 0x823f50980, ...)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:653 +0x14c
github.com/go-python/gopy/bind.(*symtab).addType(0x823df3ce0, 0x8820774278, 0x823e5b7c0, 0x8820774248, 0x823e41fb0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:437 +0x5ed
github.com/go-python/gopy/bind.(*symtab).addStructType(0x823df3ce0, 0x820647bd0, 0x88207b8320, 0x823e5b680, 0x88207b8510, 0x823e573e0, 0x1108, 0x823df3dc0, 0x1c, 0x823cca480, ...)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:575 +0x27a
github.com/go-python/gopy/bind.(*symtab).addType(0x823df3ce0, 0x88207b8320, 0x823e5b680, 0x88207b8510, 0x823e573e0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:383 +0xfd6
github.com/go-python/gopy/bind.(*symtab).addSymbol(0x823df3ce0, 0x88207b8320, 0x823e5b680)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/symtab.go:329 +0xa23
github.com/go-python/gopy/bind.(*Package).process(0x823db3a40, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/package.go:179 +0x26dd
github.com/go-python/gopy/bind.NewPackage(0x820647bd0, 0x820a24460, 0x32, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/bind/package.go:44 +0x390
main.newPackageFrom(0x8206440d0, 0x1, 0x1, 0x820676dc0, 0x820690000, 0x8206440d0, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/gen.go:209 +0x3bf
main.newPackage(0x7fff5fbffca4, 0x32, 0x8205cc064, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/gen.go:179 +0x638
main.gopyRunCmdBind(0x82065ca20, 0x82063c170, 0x1, 0x1, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/cmd_bind.go:73 +0x4eb
github.com/gonuts/commander.(*Command).Dispatch(0x82065ca20, 0x82063c170, 0x1, 0x1, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/gonuts/commander/commands.go:221 +0x7e2
github.com/gonuts/commander.(*Command).Dispatch(0x82065cb40, 0x82063c160, 0x2, 0x2, 0x0, 0x0)
    /Users/ellrott/workspaces/golang/src/github.com/gonuts/commander/commands.go:187 +0x1f9
main.main()
    /Users/ellrott/workspaces/golang/src/github.com/go-python/gopy/main.go:38 +0x1fe

There shouldn't be any need to create symbols for wrapper.RealStruct, it is never used in any of the wrapper_python function signature or public structure elements. It is a private element in wrapper_python.WrapperStruct, but that shouldn't be accessible in the python environment.

bind: support for ctors for named types (arrays, slices)

consider:

package named

type Slice []float32
type Array [2]float32

values of type Slice and Array can't be initialized via __init__(seq).

[build failed] too many errors

$ go get github.com/go-python/gopy
# github.com/go-python/gopy/bind
/usr/src/go/src/github.com/go-python/gopy/bind/gencpy_struct.go:386: cannot use m (type *types.Func) as type *types.Package in argument to types.ObjectString
/usr/src/go/src/github.com/go-python/gopy/bind/gencpy_type.go:363: cannot use m (type *types.Func) as type *types.Package in argument to types.ObjectString
/usr/src/go/src/github.com/go-python/gopy/bind/gencpy_type.go:385: cannot use m (type *types.Func) as type *types.Package in argument to types.ObjectString
/usr/src/go/src/github.com/go-python/gopy/bind/gengo.go:245: cannot use arg.GoType() (type types.Type) as type *types.Package in argument to types.TypeString
/usr/src/go/src/github.com/go-python/gopy/bind/gengo.go:246: cannot use func literal (type func(*types.Package) string) as type types.Type in argument to types.TypeString:
    func(*types.Package) string does not implement types.Type (missing String method)
/usr/src/go/src/github.com/go-python/gopy/bind/gengo.go:791: cannot use m (type *types.Func) as type *types.Package in argument to types.ObjectString
/usr/src/go/src/github.com/go-python/gopy/bind/package.go:288: cannot use m (type *types.Func) as type *types.Package in argument to types.ObjectString
/usr/src/go/src/github.com/go-python/gopy/bind/symtab.go:173: cannot use s.GoType() (type types.Type) as type *types.Package in argument to types.TypeString
/usr/src/go/src/github.com/go-python/gopy/bind/symtab.go:174: cannot use func literal (type func(*types.Package) string) as type types.Type in argument to types.TypeString:
    func(*types.Package) string does not implement types.Type (missing String method)
/usr/src/go/src/github.com/go-python/gopy/bind/symtab.go:263: cannot use t (type types.Type) as type *types.Package in argument to types.TypeString
/usr/src/go/src/github.com/go-python/gopy/bind/symtab.go:263: too many errors


$ go version
go version go1.5 darwin/amd64

$ uname -a
Darwin gotlium.local 13.4.0 Darwin Kernel Version 13.4.0: Wed Mar 18 16:20:14 PDT 2015; root:xnu-2422.115.14~1/RELEASE_X86_64 x86_64

>>> import gopy
>>> hi = gopy.load("github.com/go-python/gopy/_examples/hi")
[...]
>>> print hi
<module 'gopy.hi' from '/home/binet/dev/go/root/path/src/github.com/go-python/gopy/hi.so'>

>>> hi.Hello("you")
hello you from go

>>> hi = gopy.load("github.com/go-python/gopy/_examples/hi")
[...]
Program received signal SIGSEGV, Segmentation fault.
0x00007ffff43ca4c0 in runtime.sigtrampgo (sig=17, info=0x7fffffffd3f0, ctx=0x7fffffffd2c0) at /home/binet/dev/go/root/go/src/runtime/signal_linux.go:93
93      setg(g.m.gsignal)
(gdb) bt
#0  0x00007ffff43ca4c0 in runtime.sigtrampgo (sig=17, info=0x7fffffffd3f0, ctx=0x7fffffffd2c0) at /home/binet/dev/go/root/go/src/runtime/signal_linux.go:93
#1  0x00007ffff43e853b in runtime.sigtramp () at /home/binet/dev/go/root/go/src/runtime/sys_linux_amd64.s:234
#2  0x00007ffff43e8540 in runtime.sigtramp () at /home/binet/dev/go/root/go/src/runtime/sys_linux_amd64.s:235
#3  0x0000000000000001 in ?? ()
#4  0x0000000000000000 in ?? ()

bind: special handling of []byte

go's []byte should probably get a special treatment and be pythonized as bytearray.

gen: refactor gencpy code for py2/py3

eventually, gopy should support generation of python C-extensions for both CPython2 and CPython3 APIs.

sh> gopy gen -lang=py2|python2 my/pkg
sh> gopy gen -lang=py3|python3 my/pkg

this should also be available from the gopy bind command.

bind: wrap go.func values

right now, this fails:

package funcs

type Func func()

type S1 struct {
    F Func
}

type S2 struct {
    F func()
}

like so:

sh> gopy bind ./_examples/funcs
2015/08/07 09:07:03 work: /home/binet/dev/go/root/tmp/gopy-798639975
github.com/go-python/gopy/_examples/funcs
_/home/binet/dev/go/root/tmp/gopy-798639975
# _/home/binet/dev/go/root/tmp/gopy-798639975
./funcs.go:161: cannot use ret.F (type funcs.Func) as type cgo_type_0x2908729187 in return argument
./funcs.go:166: cannot use v (type cgo_type_0x2908729187) as type funcs.Func in assignment
./funcs.go:191: cannot use ret.F (type func()) as type cgo_type_0x2173681852 in return argument
./funcs.go:196: cannot use v (type cgo_type_0x2173681852) as type func() in assignment
2015/08/07 09:07:04 error dispatching command: exit status 2

go-python / gopy Goto Github PK

gopy's Introduction

go-python

Install

Documentation

Example:

TODO:

gopy's People

Contributors

Stargazers

Watchers

Forkers

gopy's Issues

Recommend Projects

Recommend Topics

Recommend Org