Transcripts from EU Parliament meetings from 1996 to 2011. Contains approximately 4.5 GB of text.

Languages: French, Italian, Spanish, Portuguese, Romanian, English, Dutch, German, Danish, Swedish, Bulgarian, Czech, Polish, Slovak, Slovene, Finnish, Hungarian, Estonian, Latvian, Lithuanian, and Greek.

Link: www.statmt.org/europarl/

The entirety of the English language Wikipedia.

Fanfiction.net is the largest repository of fanfiction on the internet.

I think it would be a good idea, where possible, to have separate functions in the pile for downloading a pre-processed version of the dataset from a hosted version (i.e a single wget) and running the entire replication step.

The Stackexchange dataset, for example, requires a large amount of storage for processing, and is pretty slow. It's good that people are able to recreate the pipeline if possible (and the stackexchange data on archive.org is being updated fairly regularly, so will grow over time), but in general, it would be better to host the data somewhere.

There are also some functions that only provide download, and no replication steps. I think these two things should be separated where possible.

So, we should first provide a hosted dataset, then fall back to full downloading and pre-processing, if, say, we can no longer host.

Official description:

A large-scale extendable dataset which generates question and answer pairs from a range of question types at roughly school-level difficulty. It is designed to test the mathematical learning and algebraic reasoning skills of learning models.

Link: https://deepmind.com/research/open-source/mathematics

Having a whole bunch of repositories scattered across GitHub for processing code is no beuno. We should really make a directory in this repo for housing them. If people want to keep theirs off-repo that's fine, but I really don't see why we shouldn't house them here.

I've assigned people who have been loud about this in the past to this issue.

arXiv is a preprint repository containing mathematics, computer science, and physics research papers.

Estimated Size: 75 GB

PUBMED comprises of "more than 30 million citations for biomedical literature from MEDLINE, life science journals, and online books". The data are stored on a FTP, publicly accessible and free for public use. The compressed XML is about 28GB, although much of that is boilerplate. Though the metadata is useful for other purposes, it looks like The-Pile™ would benefit the most from the abstract and titles only.

Example

<ArticleTitle>Side chain packing below the fusion peptide strongly modulates triggering of the Hendra virus F protein.</ArticleTitle>

<AbstractText> Triggering of the Hendra virus fusion (F) protein is required to initiate the conformational changes which drive membrane fusion, but the factors which control triggering remain poorly understood. Mutation of a histidine predicted to lie near the fusion peptide to alanine greatly reduced fusion despite wild-type cell surface expression levels, while asparagine substitution resulted in a moderate restoration in fusion levels. Slowed kinetics of six-helix bundle formation, as judged by sensitivity to heptad repeat B-derived peptides, was observed for all H372 mutants. These data suggest that side chain packing beneath the fusion peptide is an important regulator of Hendra virus F triggering. </AbstractText>

Note that this is different from PMC (PubMed Central). That database contains full text of recent articles. While PubMed abstracts are comprehensive, they do not contain the full text. I'll submit a separate request for that dataset.

I've downloaded the entire set last year so I already have a crawler ready to go.

Official description:

In response to the COVID-19 pandemic, the White House and a coalition of leading research groups have prepared the COVID-19 Open Research Dataset (CORD-19). CORD-19 is a resource of over 200,000 scholarly articles, including over 100,000 with full text, about COVID-19, SARS-CoV-2, and related coronaviruses. This freely available dataset is provided to the global research community to apply recent advances in natural language processing and other AI techniques to generate new insights in support of the ongoing fight against this infectious disease. There is a growing urgency for these approaches because of the rapid acceleration in new coronavirus literature, making it difficult for the medical research community to keep up.

Kaggle URL: https://www.kaggle.com/allen-institute-for-ai/CORD-19-research-challenge

A data set of user-submitted erotic literature.

to assist with data exploration and testing (both ours and that of other people) we should add a “small” flag that causes it to download a small amount of data (size TBD... 10M per data source? --Stella

Low priority

The eye is a platform deicatded to archving any and all kinds of data.

They say they have 140 Tb in total in assorted formats and a good fraction seems to be in text format.

https://the-eye.eu/public/

unfortuatly due to the fact that all of their size estimates seem to be "pending update" it is dificult to give exact estimats on how much of this is textual

URL: https://www.govinfo.gov/help/crecb#about

Size: Estimated 6-8 GB text uncompressed

The Congressional Record is the official record of the proceedings and debates of the United States Congress. It is published daily when Congress is in session. The Congressional Record began publication in 1873 and is still published today.

At the end of each session of Congress, all of the daily editions are collected, re-paginated, and re-indexed into a permanent, bound edition. This permanent edition, referred to as the Congressional Record (Bound Edition), is made up of one volume per session of Congress, with each volume published in multiple parts, each part containing approximately 10 to 20 days of Congressional proceedings. The primary ways in which the bound edition differs from the daily edition are continuous pagination; somewhat edited, revised, and rearranged text; and the dropping of the prefixes H, S, and E before page numbers.

What is available?

Volumes 144 (1998) and prior are made available as digitized versions of the Congressional Record (Bound Edition) created as a result of a partnership between GPO and the Library of Congress. These volumes include all parts of the official printed edition.

There is an API to access the records that seems straight forward, once you get past the idea of collections and packages:

https://api.govinfo.gov/docs/

The data are all in PDF, so it would require some parsing but it looks like the documents are already OCR'd.

Example: https://www.govinfo.gov/content/pkg/CRECB-2001-pt1/pdf/CRECB-2001-pt1.pdf

Preliminary experiments with pdfbox show good extraction. Example:

Mr. BYRD. Yes, exactly, one of which
happens to appear to target a facility
for a district represented by a Member
of the House from Texas. We do not
know what that facility is, but it has
been slipped into this measure.
Mr. SARBANES. I say to the distin-
guished Senator, I was not even aware
of that one. That one has not yet risen
to the level of being covered in these
newspaper stories.
Mr. BYRD. I think that is where I got
a glimmer of it, somewhere in a news-
paper story.

They are in column format, so there will be a lot of words broken by hyphens "distin- guished". I don't think that will be a problem though for the LM.

Additionally, the Congressional Records can be pulled from 1998 forward, but these are already digitized and are on a different API access endpoint.

Official description:

This dataset was collected and prepared by the CALO Project (A Cognitive Assistant that Learns and Organizes). It contains data from about 150 users, mostly senior management of Enron, organized into folders. The corpus contains a total of about 0.5M messages. This data was originally made public, and posted to the web, by the Federal Energy Regulatory Commission during its investigation.

Project URL: https://www.cs.cmu.edu/~./enron/

priority: very high

Multidisciplinary open-access research aggregator.

Size: 1.5 million papers (rough estimate: about 70GiB, give or take)

Overlap with other sets: There is probably some overlap with arXiv and PubMedCentral, but Paperity seems to have a lot of papers from subjects not currently in any of our sets.

Quote from website:

We can provide dumps of Paperity data, full and incremental, for use in external services and applications. For more information please contact us at: services (at) paperity.org.

TODO: contact paperity

https://paperity.org/

Priority: medium

We should consider adding a screenplay dataset in addition to opensubtitles, cause subtitles dont contain the the contextual information like where the actors go, what they do, how they behave, ...

This would contain valuable info about social interactions and situations.

I could write a scraper for several screenplay sites, if this would be liked. :)

priority: medium, shouldnt be too hard since we can use their code

Languages: English, French, Spanish, Arabic, Russian, Chinese (should have translations for all of these)
Date ranges: 1946-2020
Size: 700,000 publications
Link to UN digital library.

Outstanding questions:

• How many of these are downloadable through the portal?
• Are all of the documents available in all languages?
• What is the total corpus size (in bytes) we should expect from this?

bioRxiv is a preprint repository for biology research. It can be downloaded from here.

Language: English
Date ranges: 1951-1998
Size: ~ 5,000 long form documents
Text size: 800 MB

https://www.govinfo.gov/app/collection/czic

The Coastal Zone Information Center (CZIC) collection on this site provides access to nearly 5,000 coastal related documents that GPO received from the National Oceanic and Atmospheric Administration (NOAA) Central Library.

The collection provides almost 30 years of data and information crucial to the understanding of U.S. coastal management and NOAA's mission to sustain healthy coasts. These documents were originally submitted to the NOAA Office of Ocean and Coastal Resource Management (OCRM) by state coastal zone management programs in accordance with the Coastal Zone Management Act (CZMA) of 1972. These historic documents were provided to GPO by the content originator, and digitized for public use. For optimal viewing and for the original look and feel of the original documents, view the PDF versions.

Sample text:

       Begins with an analysis of environmental and cultural dynamics affecting
       prehistoric resources. An evaluation of the resources in each county follows,
       and an examination of the different stresses on prehistoric resources (from
       shore erosion, residential development, wetlands destruction, etc.). The
       management strategy discusses current programs, land use planning at dif-
       ferent levels of government, and a county by county approach. Concludes
       with an appendix on principal existing legislation which concerns prehistoric
       resources.

Priority: medium

Should be already downloaded to hetzner, ask sid for more info

Medium priority

Priority: high

Language: primarily English, with a few thousand works total in German, French, Spanish, Dutch, Portuguese, and Latin
Date ranges: Primarily pre-1923
Size: Unclear. A large number of full length books, so likely > 1GB.

The Biodiversity Heritage Library has a very large collection (~250,000) of pre-OCR'd historical books and documents on natural history topics. https://about.biodiversitylibrary.org/tools-and-services/developer-and-data-tools/

The individual .txt file links are listed in the ItemTextURL column of this TSV (warning: this link leads to a 40+MB file) https://www.biodiversitylibrary.org/data/hosted/item.txt

My primary concern is with the quality of the OCR.

Looks similar to #27:

URL https://www.courtlistener.com/api/bulk-info/

Free Law Project seeks to provide free access to primary legal materials, develop legal research tools, and support academic research on legal corpora. We work diligently with volunteers to expand our efforts at building an open source, open access, legal research ecosystem. Currently Free Law Project sponsors the development of CourtListener, Juriscraper, and RECAP. We currently have 423 courts that can be accessed with our APIs.

For each court there appears to be a file collecting all information on each case heard. A sample download of "Court of Appeals for the First Circuit" with 35K entries is about 500MB. The data seems to be organized with a useful field of "text" or "html", the later of which can be reduced with pandoc. There is definitely overlap with #27, though it's unclear how much. An example:

https://www.courtlistener.com/opinion/4242578/sandquist-v-lebo-automotive-inc/?q=Sandquist%20v.%20Lebo%20Automotive&type=o&order_by=score%20desc&stat_Precedential=on&court=cal

https://cite.case.law/cal-5th/1/233/

According to their site, it looks like they might be a strict subset. By the numbers it looks to be about half the size, "3,676,348: Number of precedential opinions in CourtListener." vs the claimed 6M for case.law. The upshot is that this service is free w/o an account to access all states and can begin parsing immediately.

Description: US caselaw collected by Harvard Law School. Four states are available for download, and people who get "researcher" accounts have access to the entire data set.

Size: 900 MB compressed -> ~4.5 GB uncompressed (open data). Full dataset size unknown.

Language: English

URL: https://case.law/

Already downloaded to hetzner. just need to convert to text.

Medium priority, low hanging fruit

Books extracted from Project Gutenberg 1919, a repository of books published before 1919.

https://www.gutenberg.org/

Priority: high

Priority: low
Difficulty: very high

Language: primarily but not exclusively English
Date ranges: 1600s-2020
Size: +260,000 works indexed and +52,000 available to download via PhilArchive

PhilPapers is an international, interactive academic database of journal articles for professionals and students in philosophy. It is maintained by the Centre for Digital Philosophy at the University of Western Ontario.

I think we could at least add the section of open access works, which should all be downloadable as PDFs from https://philarchive.org/ There may also be other works availabe for download on the main site. In addition, we could probably scrape abstracts for the rest.

An archive of over 800 academic journals on a wide variety of subjects written by and for African scientists. It’s in a mix of languages, mostly African languages or English. The website advertises

The site has 15 132 Issues containing 183 699 Abstracts with 177 532 Full Text Articles for download of which 117 018 are Open Access

However they’re journal count is out-of-date (they say 500) so I suspect that there’ll actually be much more content than these numbers imply.

I suspect that a lot of them will not be duplicative of scientific articles found in other archives because western academia kinda just ignores the scientific output of Africa. It’s very hard to get permission to submit to arXiv as an African, for example, as many countries’ universities are not on the auto-approve list.

Website: https://www.ajol.info/index.php/ajol

A scrape of public libraries provided by the-eye.

Priority: very high

The NIH (National Institutes of Health) provide a record of all the abstracts of publicly funded grants on ExPorter. There are two main URLs:

https://exporter.nih.gov/ExPORTER_Catalog.aspx?sid=0&index=1
https://exporter.nih.gov/CRISP_Catalog.aspx?sid=0&index=1

The later of which contains some overlapping legacy data. The text needs some minimal preprocessing, but is otherwise in good shape. Example:

DESCRIPTION (provided by applicant): Promising results from prophylactic HPV vaccine trials support using these vaccines in cervical cancer prevention programs in the-future. Since vaccine coverage rarely if ever reaches 100%, population-level effectiveness of a prophylactic vaccine designed to prevent a sexually transmitted infection, such as an HPV vaccine, depends not only on the efficacy of the vaccine, but also on the incidence and duration of infection in both men and women. Although much has been learned about the epidemiology of human papillomavirus (HPV) infections in women, little is known about the incidence, determinants, and natural history of HPV infections in men. Research in men has been hampered, in part, by an inability to obtain adequate genital samples for HPV DNA testing. As discussed in this proposal, we developed a sensitive and acceptable method for sample collection and now propose to use this method in a prospective natural history study with the following aims. Among young men, (1) determine the incidence of infection with any type of HPV, oncogenic HPV, specific HPV types including HPV 16 and HPV6/11, and HPV 16 variants; (2) define risk 'factors for incident HPV infection, including lifetime and recent number of sex partners, circumcision status, condom use, frequency of vaginal intercourse, and courtship behavior; and (3) describe the natural history of HPV infection in men as measured by duration and levels of HPV DNA, HPV type-specific seroconversion, duration of antibodies, and development of genital warts. Our long-term goal is development of cost-effective approaches to the prevention of HPV-related cancers.

It's not the largest dataset (estimated about 2 GB compressed?) but it's easy to get and the text is high-quality.

The books3.tar.gz dataset is now statically hosted at this URL:
https://the-eye.eu/public/AI/pile_preliminary_components/books3.tar.gz
Please add this to the datasets list.

https://archive.org/details/stackexchange stackexchange dumps available here in xml format. ~15GB compressed. I would imagine mostly english.

Will take a bit of work to figure out how to parse the XML to raw text (ideally just Question & Top Answer, or Top n Answers?)

Language: predominantly English
Date ranges: 1997-2020
Size: Claims 2M downloadable articles, 800K working papers, 26K books, and 59K chapters

Research Papers in Economics (RePEc) is a collaborative effort of hundreds of volunteers in many countries to enhance the dissemination of research in economics. The heart of the project is a decentralized database of working papers, preprints, journal articles, and software components.

We would be extracting the text components only. From what I've seen, it's PDFs.

http://www.repec.org/

Language: English
Date ranges: 1976-2000 (APS), 2001-2020 (XML)
Size: > 10GB.

Could either do patent applications or patent grants. Most relevant sections are likely "Background", followed by "Summary" / "Brief Description" and then "Detailed Description". May want to exclude claims and discussion of drawings.

https://bulkdata.uspto.gov/

When I try to download the bookcorpus dataset, my connection keeps getting closed, and it eventually gives up:

Connecting to battle.shawwn.com (battle.shawwn.com)|2606:4700:3033::681b:80c6|:443... connected.                                                                                                                   
HTTP request sent, awaiting response... 206 Partial Content                                                                                                                                                        
Length: 2404269430 (2.2G), 2402333974 (2.2G) remaining [application/gzip]                                                                                                                                          
Saving to: ‘books1.tar.gz’                                                                                                                                                                                         
                                                                                                                                                                                                                   
books1.tar.gz                0%[                                      ]   1.95M   221KB/s    in 0.5s                                                                                                               
                                                                                                                                                                                                                   
2020-09-25 11:34:47 (221 KB/s) - Connection closed at byte 2042976. Retrying.                                                                                                                                      
                                                                                                                                                                                                                   
--2020-09-25 11:34:57--  (try:20)  https://battle.shawwn.com/sdb/books1/books1.tar.gz                                                                                                                              
Connecting to battle.shawwn.com (battle.shawwn.com)|2606:4700:3033::681b:80c6|:443... connected.                                                                                                                   
HTTP request sent, awaiting response... 206 Partial Content
Length: 2404269430 (2.2G), 2402226454 (2.2G) remaining [application/gzip]
Saving to: ‘books1.tar.gz’

books1.tar.gz                0%[                                      ]   2.05M   222KB/s    in 0.5s

2020-09-25 11:34:58 (222 KB/s) - Connection closed at byte 2150496. Giving up.

Is anyone else having this problem?

Languages: English, French, Spanish, Arabic, Russian, Chinese (should have translations for all of these)
Date ranges: ~1946-2020
Size: <= 300,000 speeches
Link to speeches in UN digital library.
General Debate speeches from 1970-2018 as a pre-packaged dataset.

Outstanding questions:

How many of these are downloadable through the portal?
Are all of them available in all languages?
What is the total corpus size (in bytes) we should expect from this?