Support for MultiDiscrete action spaces about pytorch-a2c-ppo-acktr-gail HOT 5 CLOSED

@timmeinhardt - You have to use the Bernoulli Distribution and therefore can't be based off of the Categorical distribution.
You can use this piece of code
You have to add this in the model.py file

from distributions import Bernoulli
...
class Policy(nn.Module):
...
    elif action_space.__class__.__name__ == "MultiBinary":
        num_outputs = action_space.shape[0]
        self.dist = Bernoulli(self.base.output_size, num_outputs)

You have to add this in the distributions.py file

FixedBernoulli = torch.distributions.Bernoulli
log_prob_bernoulli = FixedBernoulli.log_prob
FixedBernoulli.log_probs = lambda self, actions: log_prob_bernoulli(self, actions).sum(-1, keepdim = True)
entropy = FixedBernoulli.entropy
FixedBernoulli.entropy = lambda self: entropy(self).sum(-1)
class Bernoulli(nn.Module):
    def __init__(self, num_inputs, num_outputs):
        super(Bernoulli, self).__init__()

        init_ = lambda m: init(m,
              nn.init.orthogonal_,
              lambda x: nn.init.constant_(x, 0),
              gain=0.01)

        self.linear = init_(nn.Linear(num_inputs, num_outputs))

    def forward(self, x):
        x = self.linear(x)
        return FixedBernoulli(logits = x)

from pytorch-a2c-ppo-acktr-gail.

There are no plans currently to add support for MultiDiscrete actions.

But I will gladly accept a PL!

I'm not sure whether there are any blocking factors but MultiDiscrete actions are clearly useful so I can help to fix any problems.

from pytorch-a2c-ppo-acktr-gail.

@codeislife99 Thank you for this!
Do you know how to implement FixedBernoulli.mode?

from pytorch-a2c-ppo-acktr-gail.

@ShaniGam. For implementing a Bernoulli mode you have to choose your own threshold probability for each action. During testing, it would be better to just sample instead of using mode. If it's a stochastic environment then mode would not even be close to what you want.

from pytorch-a2c-ppo-acktr-gail.

We now implemented a Bernoulli mode in form of a threshold. However, @codeislife99 's concerns should be considerer when using the Bernoulli distribution.

from pytorch-a2c-ppo-acktr-gail.