beanie00 / lesson Goto Github PK

Hi,

Thanks for this great job. I'm using your method as a baseline and have a question about the learning curves in Fig 2. Is the performance the success rate or some normalized score? I did not find some explanation in the paper.

Thanks.

The clip in the softmax call makes the probabilities have a sum != 1, this means that later the sampling will be biased

For low temperatures 1/ww it's likely that many of the probabilities go to 0 and get bumped up to 0.05, with 4 actions if you have 3 of them to zero, you get sums up to ~1.15

[1, 0, 0, 0], becomes [0.95, 0.05, 0.05, 0.05] but the sampling will only consider the first two (because np.random.rand only goes as high as 0.999..) so the distribution becomes equivalent to [0.95, 0.05, 0.00, 0.00].

My suggestion is either re-normalize the clipped probability or just normalize after the clipping.

Also consider using a implementation that subtract the max to avoid numerical issues that could come from extreme Q or ww parameters.

Hi,

I find the Intrinsic reward coefficient is set to a pretty small value, like 0.01 and 0.001. Does this mean the intrinsic reward has a tiny contibute to the learning?

Thanks.

In the appendix (table 1) you mention a task dependent parameter "Temperature" but there is no "tau" or "temperature" in the code, only softmax_ww.

Is this the parameter you tuned?

The default parameter is 50 which is not in the paper range, would it be softmax_ww = 1/tau?