juliapomdp / quickpomdps Goto Github PK

Interface for defining discrete and continuous-space MDPs and POMDPs in python. Compatible with the POMDPs.jl ecosystem.

License: MIT License

Python 60.38% Julia 39.62%

julia python pomdps mdps reinforcement-learning markov-decision-processes

quickpomdps's Introduction

quickpomdps - python

quickpomdps is a package to quickly define [PO]MDPs in Python. You can use any of the solvers in POMDPs.jl ecosystem, directly in Python.

A hybrid continuous-discrete light-dark problem definition and QMDP solution (taken from examples/lightdark.py) looks like this:

r = 60
light_loc = 10

def transition(s, a):
    if a == 0:
        return Deterministic(r+1)
    else:
        return Deterministic(min(max(s+a, -r), r))

def observation(s, a, sp):
    return Normal(sp, abs(sp - light_loc) + 0.0001)

def reward(s, a, sp):
    if a == 0:
        return 100.0 if s == 0 else -100.0
    else:
        return -1.0

m = QuickPOMDP(
    states = range(-r, r+2),
    actions = [-10, -1, 0, 1, 10],
    discount = 0.95,
    isterminal = lambda s: s < -r or s > r,
    obstype = Float64,
    transition = transition,
    observation = observation,
    reward = reward,
    initialstate = Uniform(range(-r//2, r//2+1))
)

solver = QMDPSolver()
policy = solve(solver, m)

Installation

pip install quickpomdps

quickpomdps uses the pyjulia package which requires julia to be installed. We recommend using juliaup for this purpose.

Upon invocation of import quickpomds in Python, all Julia dependencies will be installed if they are not already present. Please note that, by default, the Julia dependencies are added to the global environment. If you want to install these dependencies to a local environment instead, export the JULIA_PROJECT with the desired path as documented here.

Development

This package uses python-poetry for dependency management. Thus, it may be installed via one of the may ways supported by poetry, for example,

git clone https://github.com/JuliaPOMDP/quickpomdps
cd quickpomdps
poetry install
poetry run python examples/lightdark.py

Usage

See examples/ and tests/. Documentation can be found at the QuickPOMDPs.jl and POMDPs.jl packages.

Help

If you need help, please ask on the POMDPs.jl discussions page!

quickpomdps's People

Contributors

Stargazers

Watchers

Forkers

logankilpatrick stjordanis metavai thavishi

quickpomdps's Issues

Upgrade example to use POMDPTools

Switch to JuliaCall and JuliaPkg

Looks like JuliaCall may be a more comprehensive solution than pyjulia.

Also, there is dependency management with JuliaPkg https://github.com/JuliaPy/pyjuliapkg

We should probably switch.

Can't solve POMDPs described using QuickPOMDP()

Hello,
This package seems to only support pomdps described using DiscreteExplicitPOMDP() as in the example in examples/tiger.py.
When I try to solve a pomdp described using QuickPOMDP, the following error occurs:

Traceback (most recent call last):
  File "/home/bill/anaconda3/envs/uncertain/lib/python3.6/site-packages/julia/pseudo_python_cli.py", line 308, in main
    python(**vars(ns))
  File "/home/bill/anaconda3/envs/uncertain/lib/python3.6/site-packages/julia/pseudo_python_cli.py", line 59, in python
    scope = runpy.run_path(script, run_name="__main__")
  File "/home/bill/anaconda3/envs/uncertain/lib/python3.6/runpy.py", line 263, in run_path
    pkg_name=pkg_name, script_name=fname)
  File "/home/bill/anaconda3/envs/uncertain/lib/python3.6/runpy.py", line 96, in _run_module_code
    mod_name, mod_spec, pkg_name, script_name)
  File "/home/bill/anaconda3/envs/uncertain/lib/python3.6/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "tiger.py", line 50, in <module>
    policy = solve(solver, m)
RuntimeError: <PyCall.jlwrap (in a Julia function called from Python)
JULIA: UndefRefError: access to undefined reference
Stacktrace:
 [1] getproperty(::Type{T} where T, ::Symbol) at ./Base.jl:28
 [2] #s12#1 at /home/bill/.julia/packages/Tricks/Omjx9/src/Tricks.jl:20 [inlined]
 [3] #s12#1(::Any, ::Any, ::Any, ::Any) at ./none:0
 [4] (::Core.GeneratedFunctionStub)(::Any, ::Vararg{Any,N} where N) at ./boot.jl:527
 [5] reward(::QuickPOMDPs.QuickPOMDP{UUID("936c9395-802f-48a6-b0da-f1480db0aef3"),String,String,String,NamedTuple{(:stateindex, :isterminal, :obsindex, :states, :observations, :discount, :actions, :observation, :actionindex, :initialstate, :transition, :reward),Tuple{Dict{String,Int64},Bool,Dict{String,Int64},Array{String,1},Array{String,1},Float64,Array{String,1},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}},Dict{String,Int64},POMDPModelTools.Uniform{Set{String}},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}}}}}, ::String, ::String, ::String) at /home/bill/.julia/packages/QuickPOMDPs/ELi8v/src/quick.jl:227
 [6] reward at /home/bill/.julia/packages/POMDPModelTools/cpJHo/src/underlying_mdp.jl:26 [inlined]
 [7] macro expansion at /home/bill/.julia/packages/DiscreteValueIteration/FjeJj/src/vanilla.jl:120 [inlined]
 [8] macro expansion at ./timing.jl:233 [inlined]
 [9] solve(::DiscreteValueIteration.ValueIterationSolver, ::POMDPModelTools.UnderlyingMDP{QuickPOMDPs.QuickPOMDP{UUID("936c9395-802f-48a6-b0da-f1480db0aef3"),String,String,String,NamedTuple{(:stateindex, :isterminal, :obsindex, :states, :observations, :discount, :actions, :observation, :actionindex, :initialstate, :transition, :reward),Tuple{Dict{String,Int64},Bool,Dict{String,Int64},Array{String,1},Array{String,1},Float64,Array{String,1},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}},Dict{String,Int64},POMDPModelTools.Uniform{Set{String}},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}}}}},String,String}; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at /home/bill/.julia/packages/DiscreteValueIteration/FjeJj/src/vanilla.jl:102
 [10] solve at /home/bill/.julia/packages/DiscreteValueIteration/FjeJj/src/vanilla.jl:67 [inlined]
 [11] solve(::QMDP.QMDPSolver{DiscreteValueIteration.ValueIterationSolver}, ::QuickPOMDPs.QuickPOMDP{UUID("936c9395-802f-48a6-b0da-f1480db0aef3"),String,String,String,NamedTuple{(:stateindex, :isterminal, :obsindex, :states, :observations, :discount, :actions, :observation, :actionindex, :initialstate, :transition, :reward),Tuple{Dict{String,Int64},Bool,Dict{String,Int64},Array{String,1},Array{String,1},Float64,Array{String,1},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}},Dict{String,Int64},POMDPModelTools.Uniform{Set{String}},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}},PyCall.var"#fn#26"{PyCall.var"#fn#25#27"{PyObject}}}}}) at /home/bill/.julia/packages/QMDP/tu6hY/src/vanilla.jl:34
 [12] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N; kwargs::Base.Iterators.Pairs{Union{},Union{},Tuple{},NamedTuple{(),Tuple{}}}) at ./essentials.jl:710
 [13] invokelatest(::Any, ::Any, ::Vararg{Any,N} where N) at ./essentials.jl:709
 [14] _pyjlwrap_call(::Function, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/bill/.julia/packages/PyCall/tqyST/src/callback.jl:28
 [15] pyjlwrap_call(::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}, ::Ptr{PyCall.PyObject_struct}) at /home/bill/.julia/packages/PyCall/tqyST/src/callback.jl:44
 [16] macro expansion at /home/bill/.julia/packages/PyCall/tqyST/src/exception.jl:95 [inlined]
 [17] #109 at /home/bill/.julia/packages/PyCall/tqyST/src/pyfncall.jl:43 [inlined]
 [18] disable_sigint at ./c.jl:446 [inlined]
 [19] __pycall! at /home/bill/.julia/packages/PyCall/tqyST/src/pyfncall.jl:42 [inlined]
 [20] _pycall!(::PyObject, ::PyObject, ::Tuple{Array{String,1}}, ::Int64, ::Ptr{Nothing}) at /home/bill/.julia/packages/PyCall/tqyST/src/pyfncall.jl:29
 [21] _pycall! at /home/bill/.julia/packages/PyCall/tqyST/src/pyfncall.jl:11 [inlined]
 [22] #_#116 at /home/bill/.julia/packages/PyCall/tqyST/src/pyfncall.jl:86 [inlined]
 [23] (::PyObject)(::Array{String,1}) at /home/bill/.julia/packages/PyCall/tqyST/src/pyfncall.jl:86
 [24] top-level scope at none:4
 [25] eval(::Module, ::Any) at ./boot.jl:331
 [26] exec_options(::Base.JLOptions) at ./client.jl:272
 [27] _start() at ./client.jl:506>

Here's my code:

from quickpomdps import *
from julia.POMDPs import solve, pdf
from julia.QMDP import QMDPSolver 
from julia.POMDPSimulators import stepthrough
from julia.POMDPPolicies import alphavectors 
from julia.POMDPModelTools import SparseCat, Uniform, Deterministic
 

def transition_function(s, a):
    if a == 'listen':
        return Deterministic(s)
    else:
        return Uniform(['left','right'])

def observation(s, a, sp):
    if a == 'listen':
        if sp == 'left':
            return SparseCat(['left', 'right'], [0.85,0.15])
        else:
            return SparseCat(['right', 'left'], [0.85, 0.15])
        end
    else:
        return Uniform(['left', 'right'])

def reward(s, a):
    if a == 'listen':
        return -1.0
    elif s == a: # the tiger was found
        return -100.0
    else: # the tiger was escaped
        return 10.0

 

m = QuickPOMDP(states=['left','right'],
    actions=['left','right', 'listen'],
    observations=['left', 'right'],
    initialstate=Uniform(['left','right']),
    discount = 0.95,
    transition=transition_function,
    observation=observation,
    reward=reward 
    )

solver = QMDPSolver()  
policy = solve(solver, m)

print('alpha vectors:')
for v in alphavectors(policy):
    print(v)

print()

rsum = 0.0
for step in stepthrough(m, policy, max_steps=10):
    print('s:', step.s)
    print('b:', [pdf(step.b, x) for x in S])
    print('a:', step.a)
    print('o:', step.o, '\n')
    rsum += step.r

print('Undiscounted reward was', rsum)

Thanks for your time and help!

Unable to install quickpomdps with pip

I'm unable to install the quickpomdps package with pip install quickpomdps.
I'm using Python3.6.

Thank you.

Project.toml

@lassepe and @rejuvyesh was reading your comments in the recent PR about the Project.toml. Would it make sense to just create a new JuliaProject.toml in the current directory (if one does not already exist)?

Another idea would be to see what diffeqpy does.

Installing dependencies every time the package is loaded?

If I understand correctly, right now any time someone calls import quickpomdps, we run julia.install() and install some julia package dependencies. This seems annoying and unnecessary. We should probably check whether they are already installed, perhaps using some kind of try/catch?

Including state dependent actions in model

@zsunberg I've been using quickpomdps successfully with a fixed set of actions. Now I need to use state dependent actions as described in documentation but I get errors when I try to do so. I've tested it by modifying the Light-Dark problem as below:

def actions(s):
    if s > 0:
        return [-12, -2, 0, 1, 10] 
    else:
        return [-10, -1, 0, 2, 12] 

m = QuickPOMDP(
    states = range(-r, r+2),
    actions = actions,
    discount = 0.95,
    isterminal = lambda s: s < -r or s > r,
    obstype = Float64,
    transition = transition,
    observation = observation,
    reward = reward,
    initialstate = Uniform(range(-r//2, r//2+1))
)

and I get the following error when running QuickPOMDP:

TypeError: actions() missing 1 required positional argument: 's'

support pyjulia 0.6

Right now I think our pyproject.toml says we are incompatible with pyjulia 0.6. @rejuvyesh do you know what we need to do to fix this?

Setup GH Actions for PyPi release

Should just require uploading the API secret to allow automatic push of new versions to pypi registry.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.