Comments (3)
There's functionality to support handling special cases in arbitrary, user-defined ways. But, I haven't wired this up properly yet.
The tokenizer reads a data file that looks like this: https://github.com/honnibal/spaCy/blob/master/lang_data/en/specials.json
The key form maps to a list of tokens. You can specify the form, lemma, POS tag etc for each token.
So, you can do any sort of non-concatenative morphology you want. You can do:
"sup": [{"F": "what"}, {"F": "is"}, {"F": "up"}]
If you want.
But, as I said, this isn't set up in a nice way yet. You can find the specials.json file spaCy is currently loading under the spacy/en/data file, which is probably installed into your virtualenv. You can find the directory with
python -e import spacy.en; print spacy.en.__file__
from spacy.
That is very helpful. Thanks for your reply will be sure to mention you and your help in the paper! Keep up the awesome work!
from spacy.
This thread has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
from spacy.
Related Issues (20)
- Fail to train openai-community / gpt2 model for custom NER on SpaCy framework HOT 1
- Summary HOT 1
- Sharding Warning HOT 1
- nlp.pipe() with multiple processes on Windows VSCode HOT 2
- `Spacy` has inconsistency when dividing sentences HOT 5
- Incorrect detection of sentence boundaries, if last sentence missing eos symbol for trf model
- Enable override of existing custom pipe HOT 1
- Check that filter_spans input is a Span HOT 3
- Tokenizer Incorrectly Splitting "M1M" HOT 1
- Version incompatibility between Spacy, Cuda, Pytorch and Python HOT 3
- Accessing private transformer models HOT 1
- Problems converting Doc object to/from json HOT 1
- The word transitions to the wrong prototype HOT 1
- Fuzzy Matching not working HOT 1
- Unable to finetune transformer based ner model after initial tuning
- Undesired whitespace normalization of Korean text
- Suggestion: Normalize or Translate the parsing labels for German and English dependency labelling
- Code example discrepancy for `Span.lemma_` in API docs HOT 1
- Signature docs error in API docs for `MorphAnalysis.__contains__` HOT 2
- Import broken python 3.9 HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from spacy.