Is there anyway to train "big data" using transformer? about texar HOT 1 OPEN

By "Transformer" do you mean the example code under examples/ or the transformer modules in the library?

The transformer modules are independent of how you manage training data (either in memory or disk), as long as you pass to it a data minibatch each iteration.

The transformer example code does load the whole training data into the memory beforehand (code here). To avoid this, you may want to use Texar data module that loads data sequentially. Here is an example use of the Texar data module.

from texar.