Why is the action mean bounded before applying tanh? about dreamer HOT 4 CLOSED

Hi @xlnwel, thanks for your questions. Could you please move your question about the RSSM into a second Github issue? That way, it'll be easier for others who might have similar questions.

Regarding the action distribution, the reason the mean is bounded to [-5, +5] before being transformed by the tanh is to avoid numerical instabilities. For computing log-probabilities, we need to invert the tanh and this becomes difficult in highly saturated regions. By the way, SAC also bounds the mean using tanh but doesn't scale up by 5 afterwards, which limits its action range a bit too much. Check out the link in the code to play around with the tanh normal distribution: https://www.desmos.com/calculator/rcmcf5jwe7

I'm closing the issue for now, but please follow up if this doesn't answer your question.

from dreamer.

Sorry if I am not understanding your former answer correctly, but why we need SampleDist? I especially don't understand what argmax in SampleDist.mode() is doing.

tf.gather(sample, tf.argmax(logprob))[0]

from dreamer.

Many tfd.distributions methods such as entropy become invalid after tfd.TransformedDistribution and SampleDist can be used to approximate those stats. The line you cited just found the sample with the maximum logprob.

from dreamer.

@xlnwel Thank you! I see.

from dreamer.