Inability to reproduce paper results about clrs HOT 5 CLOSED

Hi Cameron,

Thank you for your interest in our work!
As you rightfully noted, some of our final chosen hyperparameters did not propagate to the public GitHub's run file, and this caused a bit of a discrepancy. Sorry for the inconvenience! We are already preparing a new commit to fix that.

In the meantime, I think the key hyperparameter to change from your setting is hint_mode, which should be encoded_decoded_nodiff. You already figured out the other important hyperparameter to change (hint_teacher_forcing_noise).

Further, you can think of both pgn and pgn_mask as PGNs ("mask" is a hyperparameter for the PGN, masking out possible predictions for the edge targets to follow the graph's edges. Sometimes this is a perfect inductive bias, sometimes it is very wrong.). What we did in the paper is, per-task, report the better result out of those two in the "PGN" column.

The mean reduction patch only affects processors that use the mean aggregation, which we never use in our official experiments, as the max aggregator was always superior.

I hope this is helpful. If you have any other issues, please don't hesitate to contact us.

Thanks,
Petar

from clrs.

To follow up on this, PR #94 integrates these hyperparameters into the main codebase.

from clrs.

Thank you for the quick response! I was able to replicate the paper results much more closely with the new specifications.

from clrs.

Hello, I just wanted to confirm that the paper settings for GAT was number of heads = 1, head size = 128?

from clrs.

Hi Cameron, I am not completely sure at this time, but what we report as "GAT" is actually the maximum performance out of gat, gat_full, gatv2, and gatv2_full, and I think also we swept number of heads between [1, 4, 8].

Basically, the best performance we were able to get out of all of these GAT variants, we reported as "GAT", due to a reduced amount of horizontal space.

from clrs.