In this line in run_gai.py file why do you iterate over 2 times for a single agent's trajectory?

I noticed you used the whole batch of expert trajectories when training discriminator. Is there an exact reason not to use batches?

Is there a way to render the model while testing & training?

What does this formula mean in ppo? Thanks

What is happening
In this line ??

After this when we run the training graph by feeding obs into placeholders what happen to old policy? does it use assigned parameters or it use previously initialized parameters? I think it uses previously initialized params that happening in this line . Any way why you need to run assign op before training op ?

in this line you have used is to get self.act_probs by dividing the fully connected layer inputs by temp value.

In this line you have added tf.summary.scalar('discriminator', loss) to the summary. But when I observe the tensorboard there are only ppo losses. I think there's a problem with this line. Here the tf.summary.merge_all() only merges ppo scalars.

Please correct this line. I think it should be (-loss)

Hi, thanks for sharing this. Quick question, here where you are collecting rewards

Hi can you merge this in a efficient way to the code? Or can you give me some advice where should I load the pre-retrained weights to the policy network ?

Next Question - What do you think about training GAIL with high dimensional data ? Let's say with images ?

Nice implementation! but I find some inconsistency in your discriminator.py
If you check the formula 17 of paper General Adversarial Imitation Learning
it should be 1-expert_probability, but in line 35 in your discriminator.py, it is 1-policy_probability.
Just want to assure with you :)

Hi Yusuke San:
I really admire your coding skills.
I have reviewed another GAIL which is written by TRPO. After reading your GAIL codes, I find a common set about parameter 'stochastic'. In run_ppo or run_gail, you use 'stochastic = True' in 'Policy.act( )'. But you use 'stochastic = False' in 'Policy.act( )' in test_policy.
So why you use STOCHASTIC policy when training but DETERMINISTIC policy when testing?
Thanks!

I've heard by adding entropy to the loss we can encourage the RL agent to explore more. I tried to train this PPO-Gail method with high dim inputs. At the beginning of training, almost all actions have the same probability. After some training, some actions get higher probability (in the direction of getting more rewards), and entropy is reduced over time.

If I want this how can I add entropy?

I am talking about this line. If I am correct this is the total loss according to PPO. How to optimize the critic network, which is the value function neural network?

In this line inside the ppo.py you have implemented get_grad. But seems like you didn't use it. Is there any reason?

Is there any special reason you didn't use dropout and batch normalization or it's just because input dimensions are relatively small?

Hi, in ppo.pywould you mind explaining the implementation of the loss function, can you please explain which loss equation of the ppo paper has been implemented? Also in regard to gail paper, where the equation 18 of the gail paper has been implemented? thanks for your response,

HI,
When you optimizing the discriminator to output probabilities(later take it as a reward) for each [state, action] tuple, you consider the whole batch.

By doing this don't we loos the sequential behavior of actions? Also we loose start to end connection of a trajectory. Because the neural network will only take one pair into account in a single time step.

What about using an LSTM or RNN?

Also I would like to know what is self.gaes in ppo.py file.

Hi ,
I think it is useful to visualize gradients in discriminator. Instead of using Adam optimizer in the given way I used,

optimizer = tf.train.AdamOptimizer() grads = optimizer.compute_gradients(cross_entropy) train_step = optimizer.apply_gradients(grads) for index, grad in enumerate(grads): tf.summary.histogram("{}-grad".format(grads[index][1].name), grads[index])