Collections of Indonesian Natural Language Processing dictionaries, corpus, datasets.

• Crawl
  • Kaskus WebText
  • Twitter Puisi
  • Wipedia Links
• Dictionary
  • KBBI
  • Wordlist
• Dump
  • CC-News-ID
  • Javanese Text
  • mC4
  • Twitter
• Multimodal
  • CC3M
  • CC12M
  • YFCC100M OpenAI Subset
• Paraphrase
  • MultiNLI
  • ParaBank
  • ParaNMT
  • PAWS
  • SBERT Paraphrase Data
  • SNLI
• Question Answering
  • SQuAD
  • Mathematics Dataset
• Speech
  • Google TTS
  • Unsupervised
• Summarization
  • Gigaword
  • Reddit TLDR
  • WikiHow
• Translation
  • Europarl
  • EuroPat
  • ParaCrawl

Scraped content from Kaskus with 3 cendol or more.

Poems from various user on Twitter.

Scraped content from Wikipedia external links.

Scraped Kamus Besar Bahasa Indonesia Daring.

105,226 words from Kamus Besar Bahasa Indonesia.

Processed CC-News (CommonCrawl News). Contains News from 2016-2021 from various news outlets.

Extracting Javanese text from available data.

(Hopefully) clean mC4.

Twitter dump from ArchiveTeam Twitter Stream Grab. More data will be released soon!

Translated Conceptual Captions 3M from Google Research.

Translated Conceptual Captions 12M from Google Research, meant to be used for vision-and-language pre-training.

Translated YFCC100M OpenAI Subset used in CLIP dataset ablation.

Translated Multi-Genre Natural Language Inference (MultiNLI). The corpus is modeled on the SNLI corpus, but differs in that covers a range of genres of spoken and written text.

Translated ParaBank v2.0 dataset.

Translated ParaNMT dataset, 5M filtered variant.

Translated PAWS dataset from Google Research.

Various paraphrase datasets compiled by SBERT translated to Indonesian.

The Stanford Natural Language Inference (SNLI) corpus, translated to Indonesian.

Translated SQuAD from Stanford University.

Automatically generated mathematical question and answer pairs. Translated from DeepMind's Mathematics Dataset.

Automatically generated speech using Google Translate TTS (id-ID-Wavenet-A).

A large-scale Indonesian speech corpus for pre-training speech recognition model.

Translated English Gigaword dataset.

4 Million content-summary pairs translated from Webis-TLDR-17.

Replicate WikiHow: A Large Scale Text Summarization Dataset to Indonesian.

A parallel corpus extracted from the European Parliament web site by Philipp Koehn (University of Edinburgh). The main intended use is to aid statistical machine translation research. (Translated to Indonesian using Google Translate)

Parallel corpora of patents from the United States Patent and Trademark Office and from the European Patent Organisation. (Translated to Indonesian using Google Translate)

Massively translated ParaCrawl v.7.1 from OPUS using Google Translate.