Comments (7)
Hi again,
I tried the following code in version 2.4.1
tokenizer = AutoTokenizer.from_pretrained("dccuchile/bert-base-spanish-wwm-cased") tokenizer.tokenize('Hola, Qué pasó aquí compañero?')
And worked as expected.
pd: the following won't work because using AutoTokenizer and AutoModels depends on the name of the model. The model in this case should have 'bert' in it and its not the case with 'dccuchile/cased'.
tokenizer = AutoTokenizer.from_pretrained("dccuchile/cased")
tokenizer.tokenize('Hola, Qué pasó aquí compañero?')
from beto.
Hello @alexvaca0,
Good question! I have two things to say about it:
- You're right, transformers library don't deal with accents in the form we expect. I did a PR to face that problem (huggingface/transformers#2333) that works but it probably needs a major refactor.
- But, even though we included accents when constructing the vocabulary, we didn't in the step of create the pretraining data. That is a serious problem because BETO (uncased) never saw accents in training.
We are evaluating the way to attend that problem. For now I recommend you to only use BETO-cased that doesn't have this problem or to try BETO-uncased (tokenizing with accents using the PR) only when your fine-tuning task has a big dataset of train.
from beto.
First of all, thank you very much for your quick response, I really appreciate that.
Then, if we use the code of that pull request the tokenizer won't remove accents when tokenizing new texts? We are currently using BETO-cased, which, as I understood from your answer, was trained with texts in which accents were included, am I right?
from beto.
If you are working with BETO-Cased you don't need the code in the PR.
The following code should work:
tokenizer = BertTokenizer.from_pretrained("dccuchile/cased", do_lower_case=False) tokenizer.tokenize('Hola, Qué pasó aquí compañero?')
['Hola', ',', 'Qué', 'pasó', 'aquí', 'compañero', '?']
from beto.
We are accessing your model through AutoTokenizer, AutoModel etc; is there any difference compared to calling it through BertTokenizer, BertModel etc?
from beto.
Maybe it has to do with the version of transformers; which one are you using?
from beto.
Hey how can i use this model? Do i have to download the cased file and the work exact same as bert works by loading the file i already downloaded?
from beto.
Related Issues (20)
- Reproducir el test de POS HOT 1
- Relevant punctuation tokens missing in the vocabulary (IMHO) HOT 1
- How to translate with pretrained model HOT 1
- Usar beto con pipeline HOT 2
- hi! any hints on how to use it for question answering in spanish? HOT 2
- Training requirements HOT 1
- Config file missed max_len HOT 1
- Qué cambios se hicieron en el último release del modelo? HOT 2
- Finetuning on custom spanish dataset HOT 3
- Pedida de opinión en el uso de BETO HOT 2
- BETO model size HOT 2
- GLUES benchmark
- Limpieza de texto
- Script for running the tasks
- License for BETO?
- Soporte para uso de Beto
- AutoModel bug (super easy fix) HOT 1
- No se puede cargar en Tensorflow HOT 6
- Gestionar [UNK] HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from beto.