Giter VIP home page Giter VIP logo

beto's Introduction

** This is work in progress **

BETO: Spanish BERT

BETO is a BERT model trained on a big Spanish corpus. BETO is of size similar to a BERT-Base and was trained with the Whole Word Masking technique. Below you find Tensorflow and Pytorch checkpoints for the uncased and cased versions, as well as some results for Spanish benchmarks comparing BETO with Multilingual BERT as well as other (not BERT-based) models.

Download

BETO uncased tensorflow_weights pytorch_weights vocab, config
BETO cased tensorflow_weights pytorch_weights vocab, config

All models use a vocabulary of about 31k BPE subwords constructed using SentencePiece and were trained for 2M steps.

Benchmarks

The following table shows some BETO results in the Spanish version of every task. We compare BETO (cased and uncased) with the Best Multilingual BERT results that we found in the literature (as of October 2019). The table also shows some alternative methods for the same tasks (not necessarily BERT-based methods). References for all methods can be found here.

Task BETO-cased BETO-uncased Best Multilingual BERT Other results
POS 98.97 98.44 97.10 [2] 98.91 [6], 96.71 [3]
NER-C 88.43 82.67 87.38 [2] 87.18 [3]
MLDoc 95.60 96.12 95.70 [2] 88.75 [4]
PAWS-X 89.05 89.55 90.70 [8]
XNLI 82.01 80.15 78.50 [2] 80.80 [5], 77.80 [1], 73.15 [4]

Example of use

For further details on how to use BETO you can visit the 🤗Huggingface Transformers library, starting by the Quickstart section. BETO models can be accessed simply as 'dccuchile/bert-base-spanish-wwm-cased' and 'dccuchile/bert-base-spanish-wwm-uncased' by using the Transformers library. An example on how to download and use the models in this page can be found in this colab notebook. (We will soon add a more detailed step-by-step tutorial in Spanish for newcommers 😉)

Acknowledgments

We thank Adereso for kindly providing support for traininig BETO-uncased, and the Millennium Institute for Foundational Research on Data that provided support for training BETO-cased. Also thanks to Google for helping us with the TensorFlow Research Cloud program.

Citation

Spanish Pre-Trained BERT Model and Evaluation Data

To cite this resource in a publication please use the following:

@inproceedings{CaneteCFP2020,
  title={Spanish Pre-Trained BERT Model and Evaluation Data},
  author={Cañete, José and Chaperon, Gabriel and Fuentes, Rodrigo and Ho, Jou-Hui and Kang, Hojin and Pérez, Jorge},
  booktitle={PML4DC at ICLR 2020},
  year={2020}
}

References

beto's People

Contributors

jorgeperezrojas avatar josecannete avatar gchaperon avatar rodrigofz avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.