Hi, I think the process_label function is not correct (https://github.com/ZeyuDai/paragraph-level_implicit_discourse_relation_classification/blob/master/run_discourse_parsing.py#L142). Removing some labels will improve the accuracy and F1.
Here is an example:
pred = np.array([[1,0,0,0], [0,0,0,1]])
target = np.array([[1,0,1,0], [1,0,0,1]])
Then accuracy_score(target, pred) will return 0 and precision_recall_fscore_support(target, pred, average="macro") will return (0.5, 0.375, 0.41666666666666663, None).
After your processing, target will be changed to target = np.array([[1,0,0,0], [1,0,0,0]]). Then accuracy_score(target, pred) will return 1.0 and precision_recall_fscore_support(target, pred, average="macro") will return (0.5, 0.5, 0.5, None).
When I remove this processing function, the accuracy decreases around 3% and F1 decreases more than 1%.
This task is a multilabel classification problem. The standard evaluation tool can be found at https://github.com/attapol/conll15st/blob/master/scorer.py .