TransformerX is a Python library for building transformer-based models.

It comes with multiple building blocks and layers you need for creating your model.

Join TensorOps community on Discord

Follow TensorOps Twitter

Install the latest version of TransformerX:

$ pip install transformerx

This example implements a French to English translation model.

Check out these blog posts on Medium:

• The Map Of Transformers

• Transformers in Action: Attention Is All You Need

• Rethinking Thinking: How Do Attention Mechanisms Actually Work?

Note: The data_loader and training modules are still under development and you may want to use your own training and input pipeline. However, the layers package is the core component and will remain the same (you can integrate it with Tensorflow already 🔜 Pytorch and JAX).

from transformerx.data_loader import BaseDataset
from transformerx.training import Transformer, Trainer
from transformerx.layers import TransformerEncoder, TransformerDecoder

depth, n_blocks, dropout = 256, 2, 0.2
ffn_num_hiddens, num_heads = 64, 4
key_size, query_size, value_size = 256, 256, 256

data = BaseDataset(batch_size=128)
norm_shape = [2]
encoder = TransformerEncoder(
    len(data.src_vocab),
    depth,
    norm_shape,
    ffn_num_hiddens,
    num_heads,
    n_blocks,
    dropout,
)
decoder = TransformerDecoder(
    len(data.tgt_vocab),
    depth,
    norm_shape,
    ffn_num_hiddens,
    num_heads,
    n_blocks,
    dropout,
)
model = Transformer(encoder, decoder, tgt_pad=data.tgt_vocab["<pad>"], lr=0.001)
trainer = Trainer(max_epochs=2, gradient_clip_val=1)
trainer.fit(model, data)

• Support CPU/GPU
• Vectorized operations
• Standard API

• Support Tensorflow
• Extensive documentation
  • Documentation
  • Examples (Colab, etc.)
  • Add more contents to the README.md
• Support Pytorch and JAX
• More layers
  • Attention layers
  • Residual and residual gate layers
  • Embedding layers

Contributions are what make the open source community such an amazing place to learn, inspire, and create. Any contributions you make are greatly appreciated.

In case you want to get ideas or just work on a ready-to-solve issue, please check out issues with the label issue list. Here is a list of issue lists

If you have a suggestion that would make this better, please fork the repo and create a pull request. You can also simply open an issue with the tag "enhancement" or "bug".

1. Fork the Project
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

Released under the Apache 2.0 license -> To be changed to MIT license after the first stable release

Copyright © 2021-2023 TensorOps Developers
Soran Ghaderi ([email protected])
follow me on   


Taleb Zarhesh ([email protected])
follow me on