Critic encoders as shared modules ? about maac HOT 3 CLOSED

Dear Shariq,

I have the same question as jeanibarz.

Also, in figure 1, the 2nd MLP layer receives the output of the 1st MLP, however, in your code, I assume the 2nd MLP layer is called self.critics:

The input of self.critics is an encoding of only observations (called s_encoding) plus other values, rather than the output of 1st MLP layer:

Is my understanding correct?

Thanks!

from maac.

By analizing the code, I intuitively let to this conclusion...

Each agent has a unique encoding function, state_encoder, which contrary to which is said in the paper, the embedding is based only based on the state, e_i:

However, as it is explained in the paper, we also need the embedding of other agents too, e_j, which is done with the self.critics_encoder which it takes both state and action of agents, and that is shared:

Lately, in the forward function, the key and value outputs are discarded for current agent:

Thus, by looking only to the code, I would say that the Figure 1 is not completely correct @jeanibarz

Moreover, @yuchen-x, I think you are right, the 2nd MLP is called self.critics. The input of these second MLP is correct, as the s_encodings refer to those e_i, values and the *other_all_values refer to the x_i:

This is, state_encoder refers to the first MLP, and its output would be e_i, which only takes into account the state.
self.citics_encoder takes both state and action into accounts, and is used to get the e_j.

This is not any true evidence, just my self conclusion after taking a look to the code and the paper explanation.

from maac.

Hi all, sorry for the confusion. The deviation of the code from the figure is described in the section of the paper entitled "Multi-Agent Advantage Function." Essentially, we want to calculate a Q-value for each possible action such that we can compute an advantage function, so we remove actions from the input and feed the state alone to a separate encoder from which we compute queries. I guess my intention was that the simplified version (with no advantage function) was easier to understand visually in a figure, but I realize how that can be misleading. Hope this clears things up!

from maac.