Use CKY algorithm to parse sentences
The project aims to use CKY algorithm to parse a sentence into corresponding parsing tree.
The original data is fetched from WSJ treebank.
The raw_input has one sentence per line.
The tree generated follows the json encoding scheme.
For example:
raw_input: I love doggie.
tree: ["S", ["NP", ["NP+PRON" I], ["VP", ["V", love], ["NP", doggie]]]
The file parser.py contains all the code
count_cfg_freq.py is a handy tool to count unary rule, binary rule and nonterms.
There are two ways to call parser.py
''' python parser.py inputFile outputFile '''
By calling parser.py in such a fashion, the program will take a inputFile(which is a file contains trees for training sentences) and a outputFile(modified trees for training sentences) as input.
The program will count the unary rules of the inputFile, and replace the infrequent words(frequency < 5) into "_RARE" and output to outputFile
''' python parser.py parse train_data test_data predict_file '''
The train_data contains trees for training sentences.
The test_data contains sentences for test purpose.
The predict_file is the directory of the output.
The program will first generate a count file using train_data. Then use the CKY algorithm to parse the test_data into trees, then output to predict_file
The program is compatible with train_data that undergoes markov verticalization. But the run time will be significantly longer.
Trained on 5322 sentences/trees. Test on 244 sentences.
Trained on 5322 sentences/trees. Test on 244 sentences.