This is the source code for our paper:
An Empirical Comparison of Unsupervised Constituency Parsing Methods

To reproduce our results or evaluate your own method(s), you need to follow the steps:

1. Install the dependencies
2. Download the datasets
3. Pre-process the data
4. Tune the model
5. (optional) Post-process the punctuation
6. Evaluate the prediction

It's recommended to create a new conda or virtual environment.

• Python 3.6
• nltk==3.4
• tqdm==4.31.1
• bayesian-optimization==1.0.1

• For English, we use the Penn Treebank V3
• For Japanese, we use the Keyaki Treebank 1.1

After downloading the dataset(s), extract PTB to data/english and KTB to data/japanese. Run tree -L 3 data/, you will probably see:

data/
├── english
│   └── package
│       ├── README.ALL
│       ├── README1.1ST
│       └── treebank_3
└── japanese
    └── KeyakiTreebank
        ├── LICENSE
        ├── README
        ├── acknowledgements.html
        ├── closed
        ├── metadata
        ├── scripts
        └── treebank

After preparing the datasets, you can run the preprocess.py script to clean the data:

python preprocess.py --path_to_raw_ptb data/english/package/treebank_3/parsed/mrg/wsj --path_to_raw_ktb data/japanese/KeyakiTreebank-1.1/treebank

By default, the processed datasets will be saved into data/cleaned_datasets.

We use the Bayesian Optimization algorithm to tune the hyperparameters of the models. And our implementation is based on the bayesian-optimization package.

You can use the add_punct.py script to add punctuation back to your predicted parse trees:

python add_punct.py --ref parses.ref --raw parses.pred -o parses.punct_pp

Here parses.ref is a gold file and parses.pred is the prediction file without punctuation.

For evaluation, we suggest using the evalb program. Here we provide the exectuable evalb program along with the parameter file, see evalb and evalb.prm.

If you want to reproduce the results in our paper, you can use the evaluate.py script:

python evaluate.py --gold_file /path/to/gold_file --pred_file /path/to/prediction_file

or with a length constraint:

python evaluate.py --gold_file /path/to/gold_file --pred_file /path/to/prediction_file --len_limit 10

Our experiments are based on the following implementation:

• PRPN
• URNNG
• DIORA
• CCM
• CCL