I have some trouble to reproduce result about neural-rewriter HOT 3 CLOSED

From my understanding of your colab and your email, at least some of the model checkpoints in your later training process should have a good performance if you evaluate it.

When you perform the training process, the rewriting traces are randomly sampled according to the learned distribution, and due to the randomness, they may not be optimal and even not good occasionally. Also, I manually increase the magnitude of the training losses in the dumped log file, so they may not seem stable by the values themselves.

To evaluate the model, you may adjust your command in the following way:

run_jsp.py --eval --load_model /path/to/ckpt --max_reduce_steps 100

The README includes the description of some flags above. For jsp with a larger size, it would be beneficial to allow a larger number of rewriting steps during evaluation to achieve better performance.

If you have more detailed questions, feel free to reply to our email thread for further clarification.

from neural-rewriter.

Thank you for your clarification. I think now I know what I missed -- the reward you used for the jsp task is the slowdown (In fact, it is minus slowdown so we can minimize the average slowdown), which confused me a lot previously because the logs and printouts are not the minus one.

It might also be confusing for others since we expect the "reward" is something we want to maximize.

Thank you for your time again. I really appreciate!

from neural-rewriter.

Oh I see, yes your understanding is correct, sorry for the confusion!

from neural-rewriter.