This is the source code for our paper:
An Empirical Comparison of Unsupervised Constituency Parsing Methods
To reproduce our results or evaluate your own method(s), you need to follow the steps:
- Install the dependencies
- Download the datasets
- Pre-process the data
- Tune the model
- (optional) Post-process the punctuation
- Evaluate the prediction
It's recommended to create a new conda or virtual environment.
- Python 3.6
- nltk==3.4
- tqdm==4.31.1
- bayesian-optimization==1.0.1
- For English, we use the Penn Treebank V3
- For Japanese, we use the Keyaki Treebank 1.1
After downloading the dataset(s), extract PTB to data/english
and KTB to data/japanese
. Run tree -L 3 data/
, you will probably see:
data/
├── english
│ └── package
│ ├── README.ALL
│ ├── README1.1ST
│ └── treebank_3
└── japanese
└── KeyakiTreebank
├── LICENSE
├── README
├── acknowledgements.html
├── closed
├── metadata
├── scripts
└── treebank
After preparing the datasets, you can run the preprocess.py
script to clean the data:
python preprocess.py --path_to_raw_ptb data/english/package/treebank_3/parsed/mrg/wsj --path_to_raw_ktb data/japanese/KeyakiTreebank-1.1/treebank
By default, the processed datasets will be saved into data/cleaned_datasets
.
We use the Bayesian Optimization algorithm to tune the hyperparameters of the models. And our implementation is based on the bayesian-optimization
package.
You can use the add_punct.py
script to add punctuation back to your predicted parse trees:
python add_punct.py --ref parses.ref --raw parses.pred -o parses.punct_pp
Here parses.ref
is a gold file and parses.pred
is the prediction file without punctuation.
For evaluation, we suggest using the evalb program. Here we provide the exectuable evalb program along with the parameter file, see evalb
and evalb.prm
.
If you want to reproduce the results in our paper, you can use the evaluate.py
script:
python evaluate.py --gold_file /path/to/gold_file --pred_file /path/to/prediction_file
or with a length constraint:
python evaluate.py --gold_file /path/to/gold_file --pred_file /path/to/prediction_file --len_limit 10
Our experiments are based on the following implementation: