Comments (7)
Was gb1900 model created using the latest version of DeezyMatch? My guess is that there is an inconsistency between gb1900 model architecture and the current GRU net. I could reproduce the error (on gb1900 model), but the following commands work:
- create a new model:
python DeezyMatch.py -i input_dfm.yaml -d dataset/dataset-string-similarity_test.txt -m test001_for_finetune
- Fine-tune the model:
python DeezyMatch.py -i ./models/test001_for_finetune/input_dfm.yaml -d /home/mariona/githubCode/DeezyMatch/dataset/ocr_test.txt -f test001_for_finetune -m FT_test001_for_finetune
from deezymatch.
@kasra-hosseini no prob, i'll take care now of the -f flag for absolute paths
@mcollardanuy I think it's because it overfits like crazy on the training set with only 100 training instances.
I'll fix the absolute paths, but it would be good to test it across resources that do not share completely the vocabulary. I'll train again the gb1900 model and then try to fine-tune it on ocr and let you know how it goes
from deezymatch.
@kasra-hosseini @mcollardanuy when fine-tuning across datasets I get a bug and it might be the bug we were expecting.
To reproduce it in the VM you need.
-
to be in
15-check-vocab
or in develop after the merge of the new PR -
have a model created on
gb1900 test
-
run this
python DeezyMatch.py -i input_dfm.yaml -d /home/mariona/githubCode/DeezyMatch/dataset/ocr_test.txt -f gb1900 -n 100 -m finetuned_model
from deezymatch.
btw, would it be possible to change the -f flag such that it also accepts absolute paths?
e.g., the following command does not work now (see -f flag)
python DeezyMatch.py -i ./models/test001_for_finetune/input_dfm.yaml -d /home/mariona/githubCode/DeezyMatch/dataset/ocr_test.txt -f /datadrive/khosseini/DeezyMatch/models/test001_for_finetune -m FT_test001_for_finetune
from deezymatch.
Hi! I have also just tried it (in the 15-check-vocab
branch), with Kasra's command from the previous comment, using:
- the model trained on WikiGaz (
./wikigaz_contlayers_001/wikigaz_contlayers_001
) - and fine-tuned:
- on 100 lines of OCR dataset (
/dataset/ocr_test.txt
) and - on all the Greek alphabet trainval dataset (
./dataset/wikigaz_el_trainval.txt
)
- on 100 lines of OCR dataset (
It all worked with no errors, but is it correct that we get such high numbers in the training and such a difference in the validation?
from deezymatch.
@fedenanni Can we close this?
from deezymatch.
yes i think so
from deezymatch.
Related Issues (20)
- Improve documentation on DeezyMatch installation HOT 1
- Improve documentation on generating train/valid/test datasets HOT 4
- Test Tokenizer does not handle n-gram HOT 9
- Add specific datasets for the different DM functions and adapt test notebooks HOT 2
- Add option to extend the vocabulary when fine-tuning a model
- Add post-processing filter to candidate ranking with maximum string length difference allowed HOT 1
- Linting HOT 1
- Add OCR tutorial for DH2022
- Add Heritage Gazetteer of Libya tutorial for DH2022
- Query/Candidate matching on-the-fly
- Allow disabling cosine similarity in candidate ranking
- [Tutorials] Issue with pytorch GPU
- Scaling tests
- KeyError: 'general' | Can't train a model
- pip install deezymatch HOT 1
- What kinds of pretrained models are supported by Deezymatch? HOT 1
- Example datasets and input doesn't reflect expected..
- Fix ranking metric documentation in candidateRanker HOT 1
- Fix the hardcoded multiplier in candidateRanker HOT 1
- Improve documentation on string normalization HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from deezymatch.