This project is part of a submission to an ICLR 2019 workshop, RLGM Representation Learning on Graphs and Manifolds. The manuscript can be found on arXiv:1905.00075. Our primary purpose is to develop a set of tools to standardize and facilitate use of the arXiv as a dataset. Due to licensing and distribution issues, our work is primarily a set of scripts which builds the dataset from various public data sources. There are additional cleaning, organization, and aggregation functions that it performs as well.

This project is under development as we try to best fit the needs of the community. We have adopted semver and as such will denote major releases with the first numeral in the tagged version.

Currently, the project grabs or generates:

• Article metadata -- title, authors string, category, doi, abstract, submitter
• PDFs -- all PDFs available through arXiv bulk download
• Plain text -- PDFs converted to UTF-8 encoded plain text
• Citation graph -- intra-arXiv citation graph between arXiv IDs only (generated from plain text)
• Author string parsing -- convert metadata author strings into standardized list of name, affiliations

We are able to host certain generated portions of this dataset as released snapshots. The iterations can be found under the releases tab: Releases. However, the rest of it must be generated locally.

The requirements to generate the datasets from this repository vary from dataset to dataset, but the requirements for all is:

Thank you for your interest in contributing! We welcome many forms of input -- GitHub issues, email, pull requests to name a few. Let us know how if there are ways we can better present and preserve the dataset.

If you wish to cite the dataset in your own work, a suggested bibtex entry is:

@misc{clement2019arxiv,
    title={On the Use of ArXiv as a Dataset},
    author={Colin B. Clement and Matthew Bierbaum and Kevin P. O'Keeffe and Alexander A. Alemi},
    year={2019},
    eprint={1905.00075},
    archivePrefix={arXiv},
    primaryClass={cs.IR}
}

Install the required system packages (or use an alternative Python distribution of your choice). For Debian / Ubuntu / similar:

sudo apt install python3 python3-pip python3-virtualenv poppler-utils

Download the code and prepare the python environment:

git clone https://github.com/mattbierbaum/arxiv-public-datasets
cd arxiv-public-datasets

virtualenv venv
. venv/bin/activate

pip3 install -e .
pip3 install -r requirements.txt

Decide where the data should live and modify the config.json file. This directory needs to have adequate space to hold ~ 1TB of pdfs and ~ 70GB of text if you so choose to retrieve them:

cp config.json.example config.json
[set ARXIV_DATA in config.json to your own directory]

The scripts in bin will then create any of the three subdirectories:

$ARXIV_DATA/tarpdfs   # raw pdf files from Amazon AWS bucket
$ARXIV_DATA/fulltext  # .txt from raw .pdf
$ARXIV_DATA/output    # co-citation network, parsed author strings, etc

Run OAI metadata harvester

python bin/metadata.py [OPTIONAL filepath.json.gz]

This will download the entire ArXiv metadata set, saving it as a series of gzip-compressed JSON entries. The default save location is $ARXIV_DATA/arxiv-metadata-oai-<date>.json.gz. This process will take at least 6 hours, as the OAI server only sends 1000 entries every 15 seconds. A resumption token is saved, so the process can be restarted by running again.

Prepare credentials

In addition to the setup above, you need to prepare your AWS credentials for use with boto3, the Python AWS library. A long explanation is available here while the quick method is to:

apt install awscli
aws configure

Bulk download of ArXiv PDFs

This download costs about 100 USD (and is 1.1TB) at the time of writing, as the ArXiv bulk download only allows requester-pays AWS S3 downloads. Ensure that you have at least 1TB of free space in the directory specified in config.json:

python bin/pdfdownload.py [OPTIONAL manifest_file.json.gz]

Bulk PDF conversion

To use our tool for text conversion of all the PDFs from the ArXiv bulk download described above, execute the following. NOTE: if you have not already downloaded the PDFs, this tool will do so. If you have downloaded them, be sure to not change the $ARXIV_DATA so that it will not re-download the tars.

python bin/fulltext.py [OPTIONAL number_of_processes, default cpu_count]

At the time of writing, converting 1.39 million articles requires over 400 core-hours using two Intel Xeon E5-2600 CPUs.

To generate the cocitation network, you first must have the full text. Then, with the directories still set up, run:

python bin/cocitations.py [OPTIONAL number_of_processes, default cpu_count]

The cocitation network will by default be saved in $ARXIV_DATA/output/internal-citations.json.gz.

The OAI metadata from the ArXiv features author strings as submitted by article authors. In order to use them in a principled way, theses strings must be parsed and split. To generate and save these author splittings, run:

python bin/authorsplit.py [OPTIONAL number_of_processes, default cpu_count]

The split author strings will by default be saved in $ARXIV_DATA/output/authors-parsed.json.gz.