cogcomp / jointconstrainedlearning Goto Github PK

Hi
Thank you for making your code and trained models public! I found them very useful, but I am having a little difficulty in replicating the numbers reported in the paper. Would appreciate it if you could help me with it and clarify some doubts that I have:

1. I am getting only 0.71 micro F1-score on MATRES Test set using the pre-trained weights shared in the repo (0104_3.pt) and the number reported in the paper is 0.788 micro F1 score. The 0.71 score is even less than the 0.735 F1 reported with Single Task Learning in Table 5. Could you please confirm if 0104_3.pt is indeed the best model for MATRES?
2. Also, the best model suggested for MATRES in predict.py (0104_3.pt) corresponds to just a fine-tuned version of Roberta (roberta_mlp) and not the LSTM one. Also, the LSTM model is not using the POS Tags, as stated in the paper: it only gets embedding from Roberta model, calculated in the function exp.my_func.
3. In addition, the architecture shared in the paper has common sense features encoded through an additional MLP. But I couldn't find these common-sense features in any of the two models in the model.py module. Is there a separate module for it, which I am missing?
4. Lastly, I couldn't find the Global Inference (ILP) formulation in the code. Could you please point me towards it?

Any help/pointers would be really appreciated!

Thanks,

I find the code about the calculating of precision, recall, and f1 in metric.py maybe wrong.
In your way, the calculation results of precision and recall are the same, except that the divisor of precision contains CM[3][0:3].sum().
The correct code might be:
P = 1.0 * (CM[0][0] + CM[1][1] + CM[2][2]) / (CM.sum(axis=0)[0:3].sum())
R = 1.0 * (CM[0][0] + CM[1][1] + CM[2][2]) / (CM.sum(axis=1)[0:3].sum())
I want to know if it looks like this, thanks.