Original env is 24x24 and the paddle is 2 pixels wide.

Implement these defaults and make the width of the paddle variable well-handled (based on wall collisions).

Question on this boat race demo -- is the agent actually learning to solve the boat race environment? It seems like the running reward that's used as stopping condition doesn't take into account the episodic reward the agent is accumulating. Is this agent simply meant to demonstrate the CampX API?

Overview

I've been working on a general purpose library for training safe RL agents called safe-grid-agents. It primarily uses the AI Safety Gridworlds from DeepMind.

The goal of this issue is to use the Base class from here (or something very similar) as a parent class to CampX's TensorWorld. This way, all we have to do is properly implement the abstractmethods from this Base class in order for us to be able to use specific environments based on TensorWorld with the agents I've been working on in safe-grid-agents.

Caveat

One thing we'll have to decide is if we want to use pycolab as a backend for this Base class, as is done here. One issue would be that pycolab is running numpy in the backend, and it's not clear how we could refactor that to use our MPC-shared version of PyTorch. It seems like the best way forward would be to just use the Base class and then try to mimic the kind of information that's supplied by the pycolab backend, but with torch tensors instead of with numpy arrays.

Additional requirements

In addition to the generic environment methods from Base, we'll also want two methods specific to the safety gridworlds -- get_overall_performance, which returns the safety score for an episode, and _get_hidden_reward, which supplies the per-timestep safety score. The latter is used for debugging, while the former can be used in some safe RL training schemes (e.g. semi-supervised RL). Implementing all the abstract methods as well as these two would give us an MVP of sorts that we can build on.

Plan

Each of these should be spun out into separate issues (either individually or grouped).

1. Subclass Base in TensorWorld
2. Implement observation_spec and action_spec with tests
3. TODO rest of plan

For Demo 5: Boat Race Example.ipynb , it might be illustrative to add a purely random agent to compare policies.

Also, by implementing the random agent, we might see an action bias in the policy defined inside if, for instance, we don't use random tie breaking between argmax actions.

It is not immediately clear to me how the action 0 or 1 is being selected given the action calculation line:

action = tdist.data - torch.cat([torch.zeros(1), tdist.data[:-1]])

Report the percent caught / success of episodic trials effectively.
Perhaps draw a learning curve and/or save a results vector.

matplotlib

Remove lines from notebook

import matplotlib.pyplot as plt
%matplotlib inline

These are not needed, used, and can crash the kernel for reasons unrelated to CampX if matplotlib is not configured appropriately.

tqdm

tqdm is imported but not used. It would be good to be used to measure the iteration time of the algorithm.

TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

When running either of the examples:

• http://localhost:8888/notebooks/examples/Simple%20Agent%20Example.ipynb
• http://localhost:8888/notebooks/examples/Hello%20World%20Example.ipynb

Maybe I am missing something?

Full error:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-3-37d4bd5ccb9c> in <module>()
     25       drapes={'@': RollingDrape},
     26       z_order='12@34')
---> 27 game = make_game()

<ipython-input-3-37d4bd5ccb9c> in make_game()
     24                '4': Partial(SlidingSprite, 3)},
     25       drapes={'@': RollingDrape},
---> 26       z_order='12@34')
     27 game = make_game()

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/campx-0.1.0-py3.6.egg/campx/ascii_art.py in ascii_art_to_game(art, what_lies_beneath, sprites, drapes, backdrop, update_schedule, z_order, occlusion_in_layers)
    180             game.add_prefilled_drape(character, mask,
    181                                      partial.pycolab_thing,
--> 182                                      *partial.args, **partial.kwargs)
    183 
    184         if character in sprites:

/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/campx-0.1.0-py3.6.egg/campx/engine.py in add_prefilled_drape(self, character, prefill, drape_class, *args, **kwargs)
    329         self._runtime_error_if_characters_claimed_already(character)
    330         # Construct a new curtain for the drape.
--> 331         curtain = torch.ByteTensor(np.zeros((self._rows, self._cols), dtype=np.bool_))
    332         # Fill the curtain with the prefill data.
    333         curtain.set_(prefill)

TypeError: can't convert np.ndarray of type numpy.bool_. The only supported types are: double, float, float16, int64, int32, and uint8.

• .append() (used 6 times in Demo 8) can waste time allocating memory. By using the pre-allocated structure we can save on allocations.
• similarly, insert( ) is used to collect rewards, for efficient performance may consider a deque
• np.argmax(x) is really fast when used with native numpy arrays.
• time is a reserved keyword and used as a list iterator in Demo 8
• may want to add the hidden layer size as a parameter in the policy constructor, this will allow for a trivial size vs performance sweep to be executed quickly. For instance:

class Policy(nn.Module):
    def __init__(self, state_space, action_space, hidden_layer_size):
    ....
    self.hidden_layer_size = hidden_layer_size
    .... 
    self.l1 = nn.Linear(self.state_space, self.hidden_layer_size, bias=False)
    self.l2 = nn.Linear(self.hidden_layer_size, self.action_space, bias=False)    
    ....

RL environments usually inherit their API from an abstract base class Env that usually requires reset and step at minimum. render and seed are also common (a la Gym), although sometimes seed ends up being a constructor arg and render isn't strictly necessary for the agent-environment interaction to take place (as it's commonly used).

I'm proposing that we align our tensor worlds with this API so that code we've written with other environment packages can be converted to secret worlds more cleanly.