Phrase-At-Scale provides a fast and easy way to discover phrases from large text corpora using PySpark. Here's an example of phrases extracted from a review dataset:

• Discover most common phrases in your text
• Size of discovered phrases can be arbitrary (typically: bigrams and trigrams)
• Adjust configuration to control quality of phrases
• Can be used in languages other than English
• Can be run locally using multiple threads, or in parallel on multiple machines
• Annotate your corpora with the phrases discovered

To re-run phrase discovery using the default dataset:

1. Install Spark

2. Clone this repo and move into its top-level directory.

   git clone [email protected]:kavgan/phrase-at-scale.git
   

3. Run the spark job:

   <your_path_to_spark>/bin/spark-submit --master local[200] --driver-memory 4G phrase_generator.py 
   

This will use settings (including input data files) as specified in config.py.

4. You should be able to monitor the progress of your job at http://localhost:4040/

Notes:

• The above command runs the job on master and uses the specified number of threads within local[num_of_threads].
• This job outputs 2 files:
  1. the list of phrases under top-opinrank-phrases.txt
  2. the annotated corpora under data/tagged-data/

To change configuration, just edit the config.py file.

The default configuration uses a subset of the OpinRank dataset, consisting of about 255,000 hotel reviews. You can use the following to cite the dataset:

@article{ganesan2012opinion,
  title={Opinion-based entity ranking},
  author={Ganesan, Kavita and Zhai, ChengXiang},
  journal={Information retrieval},
  volume={15},
  number={2},
  pages={116--150},
  year={2012},
  publisher={Springer} 
}

This repository is maintained by Kavita Ganesan. Please send me an e-mail or open a GitHub issue if you have questions.