Float in words_indexes about wire57 HOT 2 CLOSED

Thank you for your interest.

You are right. In the dataset provided, a token usually has a corresponding word index that is the position of the token in the tokenized source sentence. Sometimes, a token can have an index that is a list[int, float] instead of merely an int.

The latter list is an undocumented and unusable characteristic of the dataset. It appears when a given token occurs more than once in the original sentence, e.g. "Tokyo" in a source sentence with multiple "Tokyo"s. In this case, we have resorted to a heuristic that attempts to guess which occurrence of "Tokyo" we are talking about and produces a list (probable_token_index, confidence_score). Neither value can be used readily. Worse, they can be unreliable, i.e. refer to the wrong words. These list indices are not used during evaluation, so this does not invalidate results. Should you wish to have access to such indices, we invite you to finish implementing this feature correctly.

As for the code referenced by googledoc_manual_OIE_loader.load_WiRe_annotations() , we unfortunately cannot release it.

Thank you for pointing this out.

from wire57.

Thank you for your response!

from wire57.