Question about type_edges in structural attention about lattice HOT 6 CLOSED

Hi Paul,

If you want to construct a complete attention flow, then yes. You can use type_edge to mask any specific attention flow from one type of nodes to another type of nodes.

Best regards,
Fei

from lattice.

Hi Paul,

First of all, thank you for the great work!

Thank you!

Is type_edges fixed?

Yes.

If it is, then why?

We use type_edges to create the attention mask based on token types. (a,b) means there are attention flows from a to b. As written in the docstring, we assign 1&2 for metadata tokens and 3 for cell tokens. (We assign type ids during preprocessing.) So the default type_edges will mask attention flows between all cell tokens. And the attention between cell tokens in the same row/column is unmasked later.

Best regards,
Fei

from lattice.

Thank you for kindly answering my question! :)

I have one more quick question.
Does each value of type_ids, row_ids and col_ids indicate each token or each character of tokens?
It seems counting all of the characters of text (one of type_ids in train.csv has 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1~~).
Am I understanding correctly?

Thank you once again! :)

Hope you have a great one!

Best,
Paul

from lattice.

Hi Paul,

Yes, in train.csv, type_ids, row_ids, and col_ids are character-level, which are assigned during preprocessing.
These character-level ids are then mapped to token-level after tokenization at the beginning of training.

Best regards,
Fei

from lattice.

Thank you for quick response! :)
Also, if there are two token types (type_ids), then should type_edge also be changed like ((1, 1), (2, 2), (1, 2), (2, 1))?
This might be depended on the what type_ids are, but I'm just asking!

I'm new to this kind of task with using PLMs.

Sorry for bothering and thank you once again for your kindness!

Best,
Paul

from lattice.

Thank you!
Hope you have a great one! :)

Best,
Paul

from lattice.