juliacn / py2jl Goto Github PK

Python to Julia transpiler.

License: MIT License

Python 84.11% Julia 15.89%

py2jl's Introduction

Python To Julia Transpiler

This project aims at providing a Py2Jl transpiler targeting the compatibility level 1 and 2 mentioned in the section #CPython Compatibility Level, while the methods to support libraries like numpy, scipy or pytorch will be given in the documentation.

This project is in its early stage, and now we have only partly achieved the level 1 compatibility. This means that you can already write performant Julia code using pure Python, but you might not succeed in transforming any existing Python codebase to Julia with this tool.

Usage

python -m Py2Jl input.py output.jl

To run the generated Julia code, you should add the package Py2JlRuntime to your environment (e.g., pkg> dev runtime-support/Py2JlRuntime). The package Py2JlRuntime is included in the runtime-support folder.

CPython Compatibility Level

Level 1: trivial transformation for seemingly similar code correspondence. With this level, Python code transpiled to Julia usually does not work the same.
Level 2: semantics-driven transformation that respects the Python languages semantics and behaviours. With this level, Pure Python code transpiled to Julia strictly works the same.
Level 3: the transpiler respects the original execution model of CPython. such as some unusual functions in the inspect module, sys._getframe. This level is usually forcing the transpiler to use the same or similar object models as the original CPython, which has negative performance implications.
Level 4: the transpiler respects the original object memory layout of CPython. This means the transpiler becomes a replication of the original CPython, which is too boring. This level also fully supports C-extensions, but it is not the only approach.

py2jl's People

Contributors

Stargazers

Watchers

Forkers

anqurvanillapy pallharaldsson mf523 visr kskyten mysterytrader strogo hannahashemi anoop-qasolve hongliliu zicheng111 moelf

py2jl's Issues

Transpile numpy to Julia

This might be an ambitious but super useful goal of this package. What do you think?

A lot of scientific python packages are made with numpy or in numpy style, if this is done, then a lot of other package in python eco-system can be accelerate in a more elegant way.

Small fix

I added PallHaraldsson@4d6e673 after you merged PR.

Can you merge this, or do you need another PR? I'm not sure how/if you or I can convert a commit to a PR or have to do manually?

Status of the package?

@thautwarm do you want to keep this as a JuliaCN supported project or transfer to your own? I'm making a slide about JuliaCN for JuliaCon2019, so I need to decide whether to include it or not.

ERROR: LoadError: UndefVarError: installed not defined

Julia Version 1.4.0-DEV.254
Commit 3b1c1fdde3 (2019-10-04 01:47 UTC)
Platform Info:
OS: Windows (x86_64-w64-mingw32)
CPU: Intel(R) Core(TM) i5-6200U CPU @ 2.30GHz
WORD_SIZE: 64
LIBM: libopenlibm
LLVM: libLLVM-6.0.1 (ORCJIT, skylake)

using Py2Jl

ERROR: LoadError: LoadError: LoadError: UndefVarError: installed not defined
Stacktrace:
[1] getproperty(::Module, ::Symbol) at .\Base.jl:26
[2] top-level scope at C:\Users\zsz.julia\packages\Py2Jl\H6uW5\src\ASTGen.jl:13
[3] eval(::Module, ::Any) at .\boot.jl:330
....

check out Transpiler.jl

Maybe you want to check this:

https://github.com/SimonDanisch/Transpiler.jl

Package doesn't work with even demo.jl

I'm looking into debugging this, I thought it might be the new improved PyCall not working with this package, but even forcing the rev. you used didn't work:

[I doubt it should matter, but maybe me using Julia Version 1.2.0-DEV.55 is the problem? At least Julia 1.1.1 (or 1.0.x) should work, see next comment. ~~Have not yet checked 1.0.x~~]

julia> py2jl"""
       def sum_by(f, seq):
           s = 0
           for e in seq:
               s = s + f(e)
           return s
       result = sum_by(lambda x: x * 10, [1, 2, 3])
       """
ERROR: LoadError: "class: Module, attributes: Symbol[:class, :body]."
Stacktrace:
 [1] (::getfield(Py2Jl.ASTGen, Symbol("##12#122")){Nothing,getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:392
 [2] to_ast(::Nothing, ::Dict{Symbol,Any}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:398
 [3] py2jl(::Nothing, ::String) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:43
 [4] @py2jl_str(::LineNumberNode, ::Module, ::Any) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:51
in expression starting at REPL[5]:1

(v1.2) pkg> st
Status ~/.julia/environments/v1.2/Project.toml
[537997a7] AbstractPlotting v0.9.5
[6e4b80f9] BenchmarkTools v0.4.2
[a93c6f00] DataFrames v0.17.1
[864edb3b] DataStructures v0.15.0
[e9467ef8] GLMakie v0.0.4
[54eb57ff] InteractiveCodeSearch v0.3.0
[6042db11] JDBC v0.4.1
[682c06a0] JSON v0.21.0
[d8e11817] MLStyle v0.3.0
[ee78f7c6] Makie v0.9.1
[dbd62bd0] MakieGallery v0.0.4
[c020b1a1] NaturalSort v0.0.1+ #fix_for_1_0_0 (http://github.com/endrebak/NaturalSort.jl)
[9b87118b] PackageCompiler v0.6.3
[91a5bcdd] Plots v0.26.0
[e53ffe30] Py2Jl v0.1.0 #master (https://github.com/JuliaCN/Py2Jl.jl)
[438e738f] PyCall v1.91.2+ #1309efc (https://github.com/JuliaPy/PyCall.jl.git)
[d330b81b] PyPlot v2.7.0
[bf110322] Python v0.1.0 #master (https://github.com/229668880/Python.jl)
[1a8c2f83] Query v0.11.0
[e6cf234a] RandomNumbers v1.2.0
[f2b01f46] Roots v0.8.1
[3646fa90] ScikitLearn v0.5.0
[992d4aef] Showoff v0.2.1
[98e33af6] SimpleDirectMediaLayer v0.1.0
[276daf66] SpecialFunctions v0.7.2
[65254759] StatsMakie v0.0.2
[9a3f8284] Random

Feature request: a modular way of implementing external packages

It would be very useful to be able to transpile code that imports other packages. I think these package specific translations should go to their own Julia packages (e.g P2JNumpy.jl for #2). To do this, we need to come up with a way of registering the implemented translations in Py2Jl somehow. Any ideas how to do this @thautwarm?

Can't install the package

I tried installing the package on both MacOS (Apple M1) and Linux (Ubuntu), and I get the same issue on both. Please see below.

julia> import Pkg; Pkg.add("Py2Jl")^C

(@v1.4) pkg> add Py2Jl
ERROR: The following package names could not be resolved:
 * Py2Jl (not found in project, manifest or registry)

julia> import Pkg; Pkg.add("Py2Jl")
ERROR: The following package names could not be resolved:
 * Py2Jl (not found in project, manifest or registry)

Stacktrace:
 [1] pkgerror(::String) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:53
 [2] ensure_resolved(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; registry::Bool) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/Types.jl:802
 [3] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}; preserve::Pkg.Types.PreserveLevel, platform::Pkg.BinaryPlatforms.MacOS, kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:148
 [4] add(::Pkg.Types.Context, ::Array{Pkg.Types.PackageSpec,1}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:112
 [5] #add#27 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:109 [inlined]
 [6] add at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:109 [inlined]
 [7] #add#24 at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:107 [inlined]
 [8] add at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:107 [inlined]
 [9] add(::String; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:106
 [10] add(::String) at /Users/julia/buildbot/worker/package_macos64/build/usr/share/julia/stdlib/v1.4/Pkg/src/API.jl:106
 [11] top-level scope at REPL[4]:1

E.g. while loop not working

E.g. what doesn't work (and predictably "unexpected EOF while parsing" with one line shorter, and different error with, from the original, "while True:" ..):

julia> py2jl"""
       def __BOND_repl():
           SENTINEL = 1
           while SENTINEL>1:
             SENTINEL = 2
       """
ERROR: LoadError: "class: Compare, attributes: Symbol[:left, :comparators, :class, :lineno, :colno, :ops]."
Stacktrace:
 [1] (::getfield(Py2Jl.ASTGen, Symbol("##12#122")){Nothing,getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:392
 [2] ##Py2Jl.ASTGen 90#478 at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/MLStyle.jl:282 [inlined]
 [3] #12 at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:282 [inlined]
 [4] (::getfield(Py2Jl.ASTGen, Symbol("##12#122")){Nothing,getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at ./none:0
 [5] (::getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}})(::Array{Dict{Symbol,Any},1}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:187
 [6] macro expansion at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:240 [inlined]
 [7] ##Py2Jl.ASTGen 124#512 at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/MLStyle.jl:229 [inlined]
 [8] (::getfield(Py2Jl.ASTGen, Symbol("###Py2Jl.ASTGen 124#512#152")){String,Dict{Symbol,Any},getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at ./none:0
 [9] (::getfield(Py2Jl.ASTGen, Symbol("##12#122")){Nothing,getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at ./none:0
 [10] (::getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}})(::Array{Dict{Symbol,Any},1}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:187
 [11] ##Py2Jl.ASTGen 156#544 at /home/qwerty/.julia/packages/MLStyle/ksIXg/src/MLStyle.jl:197 [inlined]
 [12] #12 at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:197 [inlined]
 [13] (::getfield(Py2Jl.ASTGen, Symbol("##12#122")){Nothing,getfield(Py2Jl.ASTGen, Symbol("#trans_block#121")){getfield(Py2Jl.ASTGen, Symbol("##7#114")){Nothing}}})(::Dict{Symbol,Any}) at ./none:0
 [14] to_ast(::Nothing, ::Dict{Symbol,Any}) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/ASTGen.jl:398
 [15] py2jl(::Nothing, ::String) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:43
 [16] @py2jl_str(::LineNumberNode, ::Module, ::Any) at /home/qwerty/.julia/packages/Py2Jl/G6hOG/src/Py2Jl.jl:51
in expression starting at REPL[5]:1

Package didn't work, but I got it do work with a downgrade hack

Should it be good with this fix?

(@v1.6) pkg> add [email protected]
Updating `~/.julia/environments/v1.6/Project.toml`
  [d8e11817] ↓ MLStyle v0.4.0 ⇒ v0.3.1


julia> using MLStyle.Infras

julia> @time using Py2Jl
[ Info: Precompiling Py2Jl [e53ffe30-a369-11e8-316f-1171168155ae]
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:560
 11.723012 seconds (3.68 M allocations: 185.802 MiB, 0.42% gc time)


before I got:

julia> @time using Py2Jl
[ Info: Precompiling Py2Jl [e53ffe30-a369-11e8-316f-1171168155ae]
┌ Warning: Pkg.installed() is deprecated
└ @ Pkg /buildworker/worker/package_linux64/build/usr/share/julia/stdlib/v1.6/Pkg/src/Pkg.jl:560
ERROR: LoadError: LoadError: UndefVarError: Infras not defined
Stacktrace:
 [1] include at ./Base.jl:369 [inlined]
 [2] include(::String) at /home/pharaldsson_sym/.julia/packages/Py2Jl/PeoUh/src/Py2Jl.jl:1
 [3] top-level scope at /home/pharaldsson_sym/.julia/packages/Py2Jl/PeoUh/src/Py2Jl.jl:5
 [4] include(::Module, ::String) at ./Base.jl:369
 [5] top-level scope at none:2
 [6] eval at ./boot.jl:331 [inlined]
 [7] eval(::Expr) at ./client.jl:446
 [8] top-level scope at ./none:3
in expression starting at /home/pharaldsson_sym/.julia/packages/Py2Jl/PeoUh/src/ASTGen.jl:1
in expression starting at /home/pharaldsson_sym/.julia/packages/Py2Jl/PeoUh/src/Py2Jl.jl:1
ERROR: Failed to precompile Py2Jl [e53ffe30-a369-11e8-316f-1171168155ae] to /home/pharaldsson_sym/.julia/compiled/v1.6/Py2Jl/hBweI_nzYII.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1328
 [3] _require(::Base.PkgId) at ./loading.jl:1030
 [4] require(::Base.PkgId) at ./loading.jl:928
 [5] require(::Module, ::Symbol) at ./loading.jl:923
 [6] top-level scope at timing.jl:174

NotImplementedError

I'm trying to transpile this program:

https://github.com/hanabi1224/Programming-Language-Benchmarks/blob/main/bench/algorithm/coro-prime-sieve/1.py

Assuming I should use python3, and after installing some dependencies, I at least get it started:

$ python3 -m Py2Jl ../../py.py jl.jl
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 196, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/lib/python3.10/runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "/home/pharaldsson/Py2Jl/Py2Jl.jl/Py2Jl/__main__.py", line 14, in <module>
    wise(py2jl)()  # type: ignore
  File "/home/pharaldsson/.local/lib/python3.10/site-packages/wisepy2.py", line 271, in parse_arg
    return handle(cmd_args)
  File "/home/pharaldsson/.local/lib/python3.10/site-packages/wisepy2.py", line 244, in parse_arg
    return fn(*args, **kwargs)
  File "/home/pharaldsson/Py2Jl/Py2Jl.jl/Py2Jl/__main__.py", line 10, in py2jl
    doc = Compiler(code, out).create_module()
  File "/home/pharaldsson/Py2Jl/Py2Jl.jl/Py2Jl/__init__.py", line 349, in create_module
    return pd.vsep([arg.x for arg in self.transform_stmt_list(self.node.body)])
  File "/home/pharaldsson/Py2Jl/Py2Jl.jl/Py2Jl/__init__.py", line 534, in transform_stmt_list
    result.append(self.transform_stmt(x))
  File "/home/pharaldsson/Py2Jl/Py2Jl.jl/Py2Jl/__init__.py", line 517, in transform_stmt
    return f(self, x)
  File "/home/pharaldsson/Py2Jl/Py2Jl.jl/Py2Jl/__init__.py", line 944, in transform_Import
    raise NotImplementedError
NotImplementedError

Neither did https://github.com/hanabi1224/Programming-Language-Benchmarks/blob/main/bench/algorithm/merkletrees/1.py

but I did get their hello world program to work after I simplified it a bit (it did a bit more), to transpile, so I guess I'm doing right. The output was a bit ugly though...