Code for extracting a probability distribution ...
- prepare environment: python3.7 + pytorch 1.3 + pandas. A conda environment is adviced.
- download the pretrained model for English
- download the training data based on wikipedia on this page for English and place the 4 files inside a folder (e.g. "English")
- execute the script:
python find_probs_each_word.py --data English/ --checkpoint model.pt --corpus sample.text --output sample.probs
- Code adapted from: https://github.com/facebookresearch/colorlessgreenRNNs/
- Training of the model was not done by me. I adapted and changed this file
- Link to the license: https://creativecommons.org/licenses/by-nc/4.0/