Reformer for Squad and other QA about reformer-pytorch HOT 4 CLOSED

@hugokitano Hi! I am working with non-language use cases, but for Squad, I think you should be able to encode the entire context and query, separated with some delimiter token, and pass it all into the Reformer?

from reformer-pytorch.

Interesting, kind've like the "full Reformer sequence → sequence" example in the ReadMe? And that means I will have to pass the outputs of the Reformer through a decoder to get the final answer?

from reformer-pytorch.

@hugokitano I think you can frame it many ways, and with enough parameters, it probably won't matter. T5 treated almost all language tasks as a sequence to sequence and got SOTA just by sheer scale. Otherwise, in the past, people have gotten it to work on one stack of attention like BERT

from reformer-pytorch.

closing due to inactivity

from reformer-pytorch.