fluxml / gym.jl Goto Github PK

Hi. I just wanted to get an opinion on porting openai/gym's foundational classes (Space, Env, etc) to Gym.jl and then building additional environments atop these base classes. My reasoning is that for someone who is coming from using openai/gym, transitioning to Gym.jl will be effortless, because they'd already be familiar with underlying structure of the package.

Box space with different datatypes are compared to be equal if range is same.
MWE:

julia> box1 = Box(0, 255, (1,), Float32)
Box(Float32[0.0], Float32[255.0], (1,), Float32)

julia> box2 = Box(0, 255, (1,), UInt8)
Box(UInt8[0x00], UInt8[0xff], (1,), UInt8)

julia> sample(box1)
1-element Array{Float32,1}:
 168.51285

julia> sample(box2)
1-element Array{UInt8,1}:
 0x7d

julia> box1 == box2
true    # <--- This should be false!!

Maybe we should make Box(0, 255, (1,), UInt8) != Box(0, 255, (1,), UInt16) and Box(0, 255, (1,), UInt8) != Box(0, 255, (1,), Int8) as the return data type may cause typeistability down the line.

This just happened. I ran ] add https://github.com/FluxML/Gym.jl as described on Readme, and then I ran using Gym, and I got this:

julia> using Gym
[ Info: Precompiling Gym [56b9baea-2481-11e9-37ae-75904354ad8c]
┌ Warning: Package Gym does not have Random in its dependencies:
│ - If you have Gym checked out for development and have
│   added Random as a dependency but haven't updated your primary
│   environment's manifest file, try `Pkg.resolve()`.
│ - Otherwise you may need to report an issue with Gym
└ Loading Random into Gym from project dependency, future warnings for Gym are suppressed.
WARNING: local variable N conflicts with a static parameter in Type at .julia/packages/Gym/sJrGC/src/Spaces/tuple-space.jl:13.
ERROR: LoadError: LoadError: LoadError: ArgumentError: Package Gym does not have DataStructures in its dependencies:
- If you have Gym checked out for development and have
  added DataStructures as a dependency but haven't updated your primary
  environment's manifest file, try `Pkg.resolve()`.
- Otherwise you may need to report an issue with Gym
Stacktrace:
 [1] require(::Module, ::Symbol) at ./loading.jl:836
 [2] include at ./boot.jl:317 [inlined]
 [3] include_relative(::Module, ::String) at ./loading.jl:1044
 [4] include at ./sysimg.jl:29 [inlined]
 [5] include(::String) at .julia/packages/Gym/sJrGC/src/Spaces/Space.jl:1
 [6] top-level scope at none:0
 [7] include at ./boot.jl:317 [inlined]
 [8] include_relative(::Module, ::String) at ./loading.jl:1044
 [9] include at ./sysimg.jl:29 [inlined]
 [10] include(::String) at .julia/packages/Gym/sJrGC/src/Gym.jl:1
 [11] top-level scope at none:0
 [12] include at ./boot.jl:317 [inlined]
 [13] include_relative(::Module, ::String) at ./loading.jl:1044
 [14] include(::Module, ::String) at ./sysimg.jl:29
 [15] top-level scope at none:2
 [16] eval at ./boot.jl:319 [inlined]
 [17] eval(::Expr) at ./client.jl:393
 [18] top-level scope at ./none:3
in expression starting at .julia/packages/Gym/sJrGC/src/Spaces/dict-space.jl:1
in expression starting at .julia/packages/Gym/sJrGC/src/Spaces/Space.jl:13
in expression starting at .julia/packages/Gym/sJrGC/src/Gym.jl:11
ERROR: Failed to precompile Gym [56b9baea-2481-11e9-37ae-75904354ad8c] to .julia/compiled/v1.0/Gym/Z6zPi.ji.
Stacktrace:
 [1] error(::String) at ./error.jl:33
 [2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1203
 [3] _require(::Base.PkgId) at ./loading.jl:960
 [4] require(::Base.PkgId) at ./loading.jl:858
 [5] require(::Module, ::Symbol) at ./loading.jl:853

EDIT:

julia> versioninfo()
Julia Version 1.0.3
Commit 099e826241 (2018-12-18 01:34 UTC)
Platform Info:
  OS: Linux (x86_64-pc-linux-gnu)
  CPU: Intel(R) Pentium(R) 3558U @ 1.70GHz
  WORD_SIZE: 64
  LIBM: libopenlibm
  LLVM: libLLVM-6.0.0 (ORCJIT, haswell)

Let me know if more information would help you.

Can we bump the version with each environment category that is added?

Milestones can be -
v0.2.0 - Algorithmic env
v0.3.0 - Classic control
v0.4.0 - Atari env and so on..

Makes progress tracking easier

Currently, the environments provide a simple state space. For example, the cart pole environment provides a position and angle. This is fine and appropriate for getting up and running, but any real environment (e.g. Atari) will not provide such precisely encoded information about the current state.

Would you be amenable to returning raw RGB arrays as a rendering "mode"? So, instead of using WebIO, I can choose to render in RGB array mode. In which case, the environment returns an RGB array representing the current game screen to me, and I can choose to plot that or not. I feel that we should develop the infrastructure for this model of rendering now with the simpler environments. This way it is ready as we move to more complex environments that require such support.

Moreover, I believe the Plots package is fairly powerful, so we should encourage users to utilize that for displaying animations. WebIO is great, but more difficult to get working in varying user setups. Plots would be more straight forward and require less up-keep/debug for the maintainers of this repo.

I am interested to contribute towards gym environments purely in julia (https://julialang.org/jsoc/gsoc/flux/). What are the next steps towards development of Gym.jl?

Hello,
I would like to know what you think about having some standalone implementations as functions that take in the environment and other parameters and return the trained policy.

Here an example of how this could look like for deep Q learning:

policy = deepq.learn(env, network=q_function_approximator, lr=learning_rate, epsilon=exploration_rate, buffer_size=buffer_size)

I think this would make it easier to quickly get started with deep reinforcement learning with Flux

PendulumEnv is calculating clamped torque, but using unclamped torque in subsequent calculations. i.e. we calculate:

v = clamp.(u, -env.max_torque, env.max_torque)

but we don't use v in any of the following lines, and we use u directly.

There is another project with this name that is registered in the General registry https://github.com/ozanarkancan/Gym.jl . Given that a google search finds this repo first, it may be worth mentioning the status of this registry, and referencing the other package in the README. Not sure how actively the other project is maintained, however.

Can we get travis for the gym?

Who is currently maintaining this package? I have an interest in seeing it through, and I have been helping/following JuliaReinforcementLearning/CommonRLInterface.jl. If no one is actively maintaining it, can I have write access?

I'll be doing my development in an alternate branch and PR everything for the community to review. That way any stability isn't broken (even though this package is not registered or marked as stable).

Currently, the DiscreteSpace is defined as {1, ..., n} (as it should be), but the lines in CartPole.jl that map {1, 2} --> {-1, 1} are commented out. Additionally, the assertion is commented out. Is there a reason for this? Someone has already written the code to transfer the step! logic to a {1, ..., n} action space, so why aren't we using it?

If there is a reason, can we settle what the standard action space should be?

Hello! I ran julia> ] add https://github.com/FluxML/Gym.jl from terminal and then tried using Gym and got the following error

ERROR: LoadError: UndefVarError: Tracker not defined
Stacktrace:
[1] include at ./boot.jl:328 [inlined]
[2] include_relative(::Module, ::String) at ./loading.jl:1105
[3] include(::Module, ::String) at ./Base.jl:31
[4] top-level scope at none:2
[5] eval at ./boot.jl:330 [inlined]
[6] eval(::Expr) at ./client.jl:425
[7] top-level scope at ./none:3
in expression starting at /Users/christywarden/.julia/packages/Gym/ODARK/src/Gym.jl:4
ERROR: Failed to precompile Gym [56b9baea-2481-11e9-37ae-75904354ad8c] to /Users/christywarden/.julia/compiled/v1.3/Gym/Z6zPi_B9rLB.ji.
Stacktrace:
[1] error(::String) at ./error.jl:33
[2] compilecache(::Base.PkgId, ::String) at ./loading.jl:1283
[3] _require(::Base.PkgId) at ./loading.jl:1024
[4] require(::Base.PkgId) at ./loading.jl:922
[5] require(::Module, ::Symbol) at ./loading.jl:917

Is the use of Tracker out of date? Any help would be greatly appreciated

I figured you guys have probably planned this for sometime in the future, but I thought I'd do whoever ends up adding environments to this gym a solid and with that in mind, I propose adding a registry to the gym.

A registry to contain all the environments and their various configurations. Users can also add their own environments and specs and then use something like make to call that environment.

This does two things: 1) Makes the project more manageable and scalable 2) If you want to work on CartPole, loading Pendulum doesn't make sense, if it can be avoided.

Comments/Concerns?

I think we should decide on a name (Since "Gym.jl" has already been taken) and add this package to the official Julia Registry.

While working on the module, I discovered that calling the WebIO's display function gives an error.

display(ctx.s)
ERROR: UndefVarError: HandlerFunction not defined
Stacktrace:
 [1] #WebIOServer#82(::String, ::Int64, ::Bool, ::Bool, ::String, ::Base.DevNull, ::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}, ::Type, ::typeof(Atom.routepages)) at /Users/kartikeygupta/.julia/packages/WebIO/Rk8wc/src/providers/generic_http.jl:110
 [2] Type at ./none:0 [inlined]
 [3] setup_server at /Users/kartikeygupta/.julia/packages/Atom/y9pNA/src/display/webio.jl:76 [inlined]
 [4] show(::IOContext{Base.GenericIOBuffer{Array{UInt8,1}}}, ::MIME{Symbol("application/prs.juno.plotpane+html")}, ::WebIO.Scope) at /Users/kartikeygupta/.julia/packages/Atom/y9pNA/src/display/webio.jl:68
 [5] show(::IOContext{Base.GenericIOBuffer{Array{UInt8,1}}}, ::String, ::WebIO.Scope) at ./multimedia.jl:79
 [6] displayinplotpane(::WebIO.Scope) at /Users/kartikeygupta/.julia/packages/Atom/y9pNA/src/display/showdisplay.jl:41
 [7] display(::Atom.JunoDisplay, ::WebIO.Scope) at /Users/kartikeygupta/.julia/packages/Atom/y9pNA/src/display/showdisplay.jl:102
 [8] display(::Any) at ./multimedia.jl:287
 [9] top-level scope at none:0

julia> using WebSockets

julia> HTTP
ERROR: UndefVarError: HTTP not defined

julia> using WebSockets: HTTP

julia> HTTP
HTTP

julia> typeof(HTTP)
Module

julia> HTTP.HandlerFunction
ERROR: UndefVarError: HandlerFunction not defined

Is this error reproducible?