This repository contains the code used in the paper "Learning Python Code Suggestion with a Sparse Pointer Network"

• Python 3.5+
• Git 1.7+
• Tensorflow 0.9+ (Tested on 0.9 - 0.11)
• Github3.py
• GitPython
• An account at Github

To recreate the corpus used in the paper, run:

python3 github-scraper/scraper.py --mode=recreate --outdir=<PATH-TO-OUTPUT-DIR> --dbfile=data/cloned_repos.dat --githubuser=<GITHUB USERNAME>

Where outdir is the path on your local machine where the repos will be cloned. You will be prompted for your Github password.

To obtain a fresh corpus based on a new search of Github, using the same criteria as the paper, run:

python3 github-scraper/scraper.py --mode=new --outdir=<PATH-TO-OUTPUT-DIR> --dbfile=cloned_repos.dat --githubuser=<GITHUB USERNAME>

Note that you may interrupt the process and continue where it left off later by providing the same dbfile.

There are a number of other parameters that allow you to create your own custom corpus, specifying the programming language or search term used to query Github amongst others. Run python3 github-scraper/scraper.py -h for more information

Linux/Mac OS: Run the following command in your output directory to remove non Python files

find . -type f ! -name "*.py" -delete

Run the following command to normalise all files with a .py extension by providing the output directory of step 1 as the path. The normalised files will be written to a new directory with "normalised" appended to the path.

python3 github-scraper/normalisation.py --path=<PATH TO DOWNLOADED CORPUS>

Files which can't be parsed as valid Python3 will be ignored. The list of successfully processed files is written to PATH/processed.txt which also allows for the normalisation to continue if interrupted.

To use the same train/dev/test split as used in the paper, copy the files train_files.txt, valid_files.txt and test_files.txt from the data directory into the downloaded corpus and normalised corpus directories.

To generate a new split, run the following command which generates the list of train files (train_files.txt), validation files (valid_files.txt) and test files (test_files.txt) in the ratio 0.5/0.2/0.3. Use the normalised path from the previous step. This will ensure that the list of files is available in both the normalised and unnormalised data sets.

python3 github-scraper/processFiles.py --path=<PATH TO NORMALISED CORPUS>

Then copy the 3 generated lists to the original un-normalised path.