FeUdal Networks for Hierarchical Reinforcement Learning
https://arxiv.org/pdf/1703.01161.pdf

gym.error.Error: env.monitor has been deprecated as of 12/23/2016. Remove your call to env.monitor.start(directory) and instead wrap your env with env = gym.wrappers.Monitor(env, directory) to record data.

Trying this two different q_functions:

(non recurrent)

class QFunction(chainer.Chain, StateQFunction):

        def __init__(self, n_input_channels=3, n_actions = 4, bias=0.1):
            self.n_actions = n_actions
            self.n_input_channels = n_input_channels
            conv_layers = chainer.ChainList(
                L.Convolution2D(n_input_channels, 32, 8, stride=4, bias=bias),
                L.Convolution2D(32, 64, 4, stride=2, bias=bias),
                L.Convolution2D(64, 64, 3, stride=1, bias=bias),
                L.Convolution2D(64, 128, 7, stride=1, bias=bias)
                )

            lin_layer = L.Linear(128, 128)                     

            a_stream = MLP(128,n_actions,[2])
            v_stream = MLP(128,1,[2])

            super().__init__(conv_layers=conv_layers, lin_layer=lin_layer, a_stream=a_stream,v_stream=v_stream)

        def __call__(self, x, test=False):
            """
            Args:
                x (ndarray or chainer.Variable): An observation
                test (bool): a flag indicating whether it is in test mode
            """
            h = x
            for l in self.conv_layers:
                h = F.relu(l(h))
            h = self.lin_layer(h)

            batch_size = x.shape[0]
            ya = self.a_stream(h, test=test)
            mean = F.reshape(F.sum(ya,axis=1) / self.n_actions, (batch_size,1))
            ya, mean = F.broadcast(ya,mean)
            ya -= mean

            ys = self.v_stream(h,test=test)
            
            ya,ys = F.broadcast(ya,ys)
            q = ya+ys
            return chainerrl.action_value.DiscreteActionValue(q)

(recurrent)

class QFunctionRecurrent(chainer.Chain, StateQFunction):

        def __init__(self, n_input_channels=3, n_actions = 4, bias=0.1):
            self.n_actions = n_actions
            self.n_input_channels = n_input_channels
            conv_layers = chainer.ChainList(
                L.Convolution2D(n_input_channels, 32, 8, stride=4, bias=bias),
                L.Convolution2D(32, 64, 4, stride=2, bias=bias),
                L.Convolution2D(64, 64, 3, stride=1, bias=bias),
                L.Convolution2D(64, 128, 7, stride=1, bias=bias)
                )

            lstm_layer = L.LSTM(128, 128)                     

            a_stream = MLP(128,n_actions,[2])
            v_stream = MLP(128,1,[2])

            super().__init__(conv_layers=conv_layers, lstm_layer=lstm_layer, a_stream=a_stream,v_stream=v_stream)

        def __call__(self, x, test=False):
            """
            Args:
                x (ndarray or chainer.Variable): An observation
                test (bool): a flag indicating whether it is in test mode
            """
            h = x
            for l in self.conv_layers:
                h = F.relu(l(h))
            h = self.lstm_layer(h)

            batch_size = x.shape[0]
            ya = self.a_stream(h, test=test)
            mean = F.reshape(F.sum(ya,axis=1) / self.n_actions, (batch_size,1))
            ya, mean = F.broadcast(ya,mean)
            ya -= mean

            ys = self.v_stream(h,test=test)
            
            ya,ys = F.broadcast(ya,ys)
            q = ya+ys
            return chainerrl.action_value.DiscreteActionValue(q)

I found that for the non-recurrent version the loss is not zero and the agent will eventually master the gym environment provided.

However, changing nothing else than adding an lstm layer and setting episodic_replay to True the average_loss will become 0 all the time and the agents is not able to learn to better interact with its environent.

First, I thought that this was due to some kind of rounding issues so I set the minibatch_size=1, episodic_update_len = 1 (assuming that one episodic replay will now only containg one time step) but still no changes.

I wonder if this is some kind of bug or (which I think is more likely) an error on my side.

Any help is very much appreciated!

Since gym has introduced its own interface to modify envs using gym.Wrapper, I think it is better to use it in ChainerRL instead of directly modifying methods as in env_modifiers.

Got this error while trying to run train_ddpg_gym.py:
init() got an unexpected keyword argument 'update_interval'

https://arxiv.org/abs/1703.01988

There is already an implementation by @ISakony, though it is not tested against Atari. https://github.com/ISakony/NEC_chainerrl_CartPole-v0

I'm curious about whether ChainerRL can support PyTorch as an additional NN backend. Its interface is similar to Chainer's, but I'm not sure how easy it would be to support both. Any suggestions and opinions are welcome.

Interpolated Policy Gradient: Merging On-Policy and Off-Policy Gradient Estimation for Deep RL

https://arxiv.org/pdf/1706.00387.pdf

Are there any example codes of ACER in the continuous action spaces, using the MuJoCo environments?

I think (Async) PCL would improve if 1 Prioritized Replay Buffer is shared among all the agents.

In chainerrl.experiments.train_agent, statistical information is reported via print per episode during the training loop. However, this sometimes looks so verbose and I want to suppress these messages, but currently there is no good way to do so. Adding some option that enables/disables these prints might be beneficial.

Have you tested PCL on Atari Environments?

gym now complains:

DEPRECATION WARNING: env.spec.timestep_limit has been deprecated. Replace your call to env.spec.timestep_limit with env.spec.tags.get('wrapper_config.TimeLimit.max_episode_steps'). This change was made 12/28/2016 and is included in version 0.7.0

This is due to chainer/chainer#2394.

Until this issue is fixed, please use chainer==1.20.0.1 to run tests.

Maybe TicTacToe or other easy-to-implement games?

Would it be possible to provide examples for stateful models for q learning and a3c?

In ChainerRL, to use user-defined recurrent models, you need to make sure they implement chainerrl.recurent.Recurrent interface, otherwise they won't be treated as recurrent models.

When your model's recurrent-ness comes from chainer.links.LSTM, all you have to do is inheriting chainer.recurrent.RecurrentChainMixin.

This kind of information is missing in the document.

On python 3.4, random.sample don't accept collections.deque, so I got such error.

Traceback (most recent call last):
  File "quickstart.py", line 111, in <module>
    action = agent.act_and_train(obs, reward)
  File "/opt/rl/lib/python3.4/site-packages/chainerrl/agents/dqn.py", line 340, in act_and_train
    self.replay_updator.update_if_necessary(self.t)
  File "/opt/rl/lib/python3.4/site-packages/chainerrl/replay_buffer.py", line 194, in update_if_necessary
    transitions = self.replay_buffer.sample(self.batchsize)
  File "/opt/rl/lib/python3.4/site-packages/chainerrl/replay_buffer.py", line 42, in sample
    return random.sample(self.memory, n)
  File "/opt/rl/lib/python3.4/random.py", line 311, in sample
    raise TypeError("Population must be a sequence or set.  For dicts, use list(d).")
TypeError: Population must be a sequence or set.  For dicts, use list(d).

python 2.7 works fine, and meybe 3.5+.

It will be great to add performance evaluation and comparisons of algorithms available in ChainerRL.

I think I need a more simplified example with detailed comments.

Travis CI failed on examples/gym/train_ddpg_gym.py:

Traceback (most recent call last):
  File "examples/gym/train_ddpg_gym.py", line 173, in <module>
    main()
  File "examples/gym/train_ddpg_gym.py", line 170, in main
    max_episode_len=timestep_limit)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/experiments/train_agent.py", line 144, in train_agent_with_evaluation
    logger=logger)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/experiments/train_agent.py", line 52, in train_agent
    action = agent.act_and_train(obs, r)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/agents/ddpg.py", line 314, in act_and_train
    self.replay_updater.update_if_necessary(self.t)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/replay_buffer.py", line 327, in update_if_necessary
    self.update_func(transitions)
  File "/home/travis/build/pfnet/chainerrl/chainerrl/agents/ddpg.py", line 246, in update
    self.actor_optimizer.update(lambda: self.compute_actor_loss(batch))
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/chainer/optimizer.py", line 416, in update
    loss.backward()
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/chainer/variable.py", line 398, in backward
    gxs = func.backward(in_data, out_grad)
  File "/home/travis/virtualenv/python2.7.9/lib/python2.7/site-packages/chainer/functions/connection/linear.py", line 59, in backward
    gW = gy.T.dot(x).astype(W.dtype, copy=False)
ValueError: On entry to SGEMV parameter number 8 had an illegal value

This may be the same issue as chainer/chainer#2744

https://arxiv.org/abs/1703.03864

Does it become more_deterministic / less_stable ?

See section 5.1 for new more performant update to PCL: https://arxiv.org/pdf/1702.08892.pdf

The Reactor: A Sample-Efficient Actor-Critic Architecture
https://arxiv.org/abs/1704.04651

Neural Fictitious Self Play

https://arxiv.org/abs/1603.01121

because they are clearly not DQN and their current code structures are difficult to understand.

Is there an existing method to visualize models?

Human-readability of the name of the subdirectory for an experiment might be improved. The current imprementation (time_str = datetime.datetime.now().strftime('%Y%m%d%H%M%S%f') in chainerrl/experiments/prepare_output_dir.py) produces e.g. 21120903182945898662. How about

• strftime('%Y%m%d-%H%M%S-%f') (e.g. 21120903-182945-898662), or
• the basic format in ISO 8601 (e.g. 21120903T182945.898662+0900)?

Currently, chainerrl.experiments.train_agent_async always decays the learning rate linearly to zero. This kind of behavior should be customizable.

Hi, I've defined my own OpenAI gym and have specified my actions in the environment as follows:

`
self.actions = ["NOOP", "LEFT", "RIGHT", "FIRE", "CLOAK"]

self.action_space = spaces.Discrete(len(self.actions))
`

When I try my environment against the 'train_dqn_gym.py' example I can see from my debug output that the training is correctly resulting in trying a variety of different actions.

However, with both 'train_a3c_gym.py' and 'train_acer_gym.py' the action value provided to my step value is always 0 (NOOP) - it never tries any other action.

Have I coded something wrong in my environment? I would appreciate any tips on how to investigate my issue further.

https://arxiv.org/pdf/1703.01703.pdf

DQN.update has code updating average_loss:
https://github.com/pfnet/chainerrl/blob/03c2ff975e1fca64c67d6ad1a1daaa9638d04e66/chainerrl/agents/dqn.py#L232

However, DQN.update_from_episode doesn't have such code:
https://github.com/pfnet/chainerrl/blob/03c2ff975e1fca64c67d6ad1a1daaa9638d04e66/chainerrl/agents/dqn.py#L287

This keeps average_loss zero. This problem is reported in #82.

Current examples don't specify in what configuration they work well, except newer ones (train_pcl_gym.py and train_reinforce_gym.py). Such instructions are important because users can easily confirm that the implementations actually work.

• ale/train_a3c_ale.py
• ale/train_acer_ale.py
• ale/train_dqn_ale.py
• ale/train_nsq_ale.py
• gym/train_a3c_gym.py
• gym/train_acer_gym.py
• gym/train_ddpg_gym.py
• gym/train_dqn_gym.py
• gym/train_pcl_gym.py
• gym/train_reinforce_gym.py

Currently, chainerrl.experiments.prepare_output_dir assume that the current directory is version-controlled by git, but it should not.

A simple REINFORCE implementation that doesn't require a value function would be helpful.

Now that atari_py is not required, this part should be updated: http://chainerrl.readthedocs.io/en/latest/install.html#for-windows-users

There are requirements that just backports py3 features to py2:

• funcsigs
• futures
• statistics
• fastcache

They should be skipped for py3.

Any plans to add openai roboschool for robot-simulation?

I think the examples of gym interface show, the agent and q_functions expect the observation and action to be a Box or a Discrete for each. Is it correct?

If so, how should I use the observation and action other typed, especially a Tuple?
Do I need to modify the environment as to return a Box, or do I have another choice?

Thanks,

https://arxiv.org/abs/1702.08892

recently I got a windows 10 computer, successfully installed bash on ubuntu for windows, cuda, cudnn, chainer and chainerrl. But to run the example, I got the following error. Any suggestions?

(py2env) neil@DESKTOP-C22605O:~/chainerrl$ xvfb-run -s "-screen 0 1400x900x24" python examples/gym/train_dqn_gym.py
Output files are saved in dqn_out/20170324141722891586
INFO:gym.envs.registration:Making new env: Pendulum-v0
Traceback (most recent call last):
File "examples/gym/train_dqn_gym.py", line 179, in
main()
File "examples/gym/train_dqn_gym.py", line 154, in main
episodic_update=args.episodic_replay, episodic_update_len=16)
File "/home/neil/py2env/local/lib/python2.7/site-packages/chainerrl/agents/dqn.py", line 115, in init
cuda.get_device(gpu).use()
File "cupy/cuda/device.pyx", line 75, in cupy.cuda.device.Device.use (cupy/cuda/device.cpp:2083)
File "cupy/cuda/device.pyx", line 81, in cupy.cuda.device.Device.use (cupy/cuda/device.cpp:2035)
File "cupy/cuda/runtime.pyx", line 178, in cupy.cuda.runtime.setDevice (cupy/cuda/runtime.cpp:2915)
File "cupy/cuda/runtime.pyx", line 130, in cupy.cuda.runtime.check_status (cupy/cuda/runtime.cpp:2241)
cupy.cuda.runtime.CUDARuntimeError: cudaErrorUnknown: unknown error