ozyyshr / dgm Goto Github PK

Hi,

I follow the readme steps and get the following results after training:

Best dev
{'dev_0a_combined': 24.71,
'dev_0b_macro_accuracy': 55.76,
'dev_0c_micro_accuracy': 44.32,
'dev_0d_confmat': [[322, 301, 180, 1],
[293, 299, 170, 4],
[111, 130, 247, 74],
[0, 0, 0, 138]],
'dev_1a_entail_macro_accuracy': 44.66,
'dev_1b_entail_micro_accuracy': 66.29,
'global_step': 3000}

I guess 55.76 and 44.32 are the final numbers? They seem far away from the numbers reported in Table 1 of the paper. Do you have any idea why this happens? I guess it's because hyperparameter issue (I am following section 3.2 Decision Making in readme)

Thanks!

Hi,
Thanks for your great job!
I have followed the instruction of readme, but changed the DGL version with various versions. During the experimental stages, even with the same seed and same experimental environment, the results still have reproducibility problems. As seen in DGL instructions, after the version of 0.5, DGL by default chooses deterministic implementations for better reproducibility.
I wonder what makes this repo has reproducibility problems?
Thanks again.

1. Where is the fix_questions.py file?
2. What is the path to all the preprocessing scripts?
3. Where is segedu/all_vocabulary.pickle?

Thanks.