Prevents shortcut about chain-of-hindsight HOT 6 CLOSED

Thanks for your interest. FCM masking can be enabled by setting fcm_min_ratio=0 and fcm_max_ratio=0.15. It is recommended for large models.

from chain-of-hindsight.

Ah, that's where it was. Thanks for the reply! Also, in the paper, only ascending sorting was used, but in the code, it looks like mix of ascending order and descending order is implemented.

Did I understand it correctly??

from chain-of-hindsight.

Also, if I understood it correctly, it seems like that the code concatenate all the data and slice it by the chunk_size.
https://github.com/lhao499/CoH/blob/3949417c638834d9213f6395196db1a802e61d8c/coh/data/hf_data.py#L54

If there is a data like [I, am, a, boy, is, worse, than, I, am, a, girl] and the chunk_size is 6, then the data will be sliced to [I, am, a, boy, is, worse], and [than, I, am, a, girl].
What this means is that, "I am a girl" part is only conditioned to "than".
Did I understand the code correctly??

from chain-of-hindsight.

For the first question, yes both ascending order and descending order are used to increase diversity, we will make this more clear.
For the second question, splitting by chunk_size has better compute efficiency compared with splitting by sentence and adding padding masks. Using padding may have better results, but we did not try it.

from chain-of-hindsight.

Thank you for the reply! So, to get the performance reported on the paper, did you use both ascending order and descending order?

from chain-of-hindsight.

I apologize for the lateness. Yes, both ascending order and descending order were used.
In our more recent Koala chatbot project, we also compared and found using both options worked better.

from chain-of-hindsight.