CategoricalGRUPolicy InputLayer dimensionality about rllab HOT 9 CLOSED

Hi, the dimension layout is (batch size, time, obs dim). The extra dimension is needed to construct a computation graph for the entire history. This is only used in the optimization phase but not during execution of the policy.

from rllab.

So a easy solution is to take (12, 10) / (n_env, obs_dim) / (batch_size, obs_dim) - the current obses shape returned by vec_env, and expand to (12, 1, 10) right? (Add 1 time dimension to it)

from rllab.

What's the use case?

from rllab.

I'm adding a custom environment of one of the text task from OpenAI Gym (character copy, but slightly varied), so the environment at each step, returns an embedding of character (10-dim).

When I dig into the VectorizedSampler, and use CategoricalGRUPolicy, it creates an array of n environments and when step() is called on vec_env, it returns shape of (12, 10).

from rllab.

The current code is supposed to work with recurrent policies already. Did you notice anything broken?

from rllab.

@ahhh! My mistake, current code does work! Sorry for the trouble!! :(

Can I ask a different question?

The "Copy-v0" environment has its action-space as a tuple:

self.action_space = Tuple(
            [Discrete(len(self.MOVEMENTS)), Discrete(2), Discrete(self.base)]
        )

and it seems like convert_gym_space or to_tf_space can't handle this type of Tupled action_space...is there a good way to get around this?

https://github.com/openai/gym/blob/master/gym/envs/algorithmic/algorithmic_env.py

from rllab.

Hmm, I think both of these should support this scenario. Did you see an error? A CategoricalGRUPolicy won't be able to handle this kind of action space though since you want to apply separate softmax to each group of the actions, so you need to customize the nonlinearity applied to the output.

from rllab.

import gym
from sandbox.rocky.tf.envs.base import TfEnv
from sandbox.rocky.tf.envs.vec_env_executor import VecEnvExecutor

env = gym.make("Copy-v0")
env = TfEnv(env)

config = {
    "max_seq_len": 10,  
    "batch_size": 128,
}

n_envs = int(config["batch_size"] / config["max_seq_len"])
n_envs = max(1, min(n_envs, 100))

envs = [env for _ in range(n_envs)]
vec_env = VecEnvExecutor(
    envs=envs,
    max_path_length=config["max_seq_len"]
)

Error is:

[2016-11-28 11:32:49,682] Making new env: Copy-v0
Traceback (most recent call last):
  File "exps/text_env_test.py", line 39, in <module>
    max_path_length=config["max_seq_len"]
  File "/Users/xxx/Documents/rllab/sandbox/rocky/tf/envs/vec_env_executor.py", line 11, in __init__
    self._action_space = envs[0].action_space
  File "/usr/local/lib/python2.7/site-packages/cached_property.py", line 26, in __get__
    value = obj.__dict__[self.func.__name__] = self.func(obj)
  File "/Users/xxx/Documents/rllab/sandbox/rocky/tf/envs/base.py", line 40, in action_space
    return to_tf_space(self.wrapped_env.action_space)
  File "/Users/xxx/Documents/rllab/sandbox/rocky/tf/envs/base.py", line 20, in to_tf_space
    raise NotImplementedError
NotImplementedError

from rllab.

I see. The recommended way to use gym environments is via GymEnv: https://github.com/openai/rllab/blob/master/rllab/envs/gym_env.py.

from rllab.