llm-jp / llm-jp-corpus Goto Github PK

• Download
• Preprocessing

Maybe because of this: huggingface/datasets#5613

Japanese

• Wikipedia
  • Download
  • Filter and Reformat
  • Tokenize
  • Sample (Use all)
• C4
  • Download
  • Filter and Reformat
  • Tokenize
  • Sample (Use all)

English

• Wikipedia
  • Download
  • Filter and Reformat
  • Tokenize
  • Sample (Use all)
• Pile
  • Download
  • Filter and Reformat
  • Tokenize
  • Sample

Code

• Stack
  • Download
  • Filter and Reformat
  • Tokenize
  • Sample

What

Remove meta information, such as "脚注" (footnotes) and "関連項目" (related articles), from the text.

How

Use the wikiextractor library for text extraction.

• Download
• Preprocessing

Candidates:

• MIT
• Apache 2.0

Why

Ethical filtering based on word pattern matching doesn't work well in Wikipedia. It would be better to exploit the category in Wikipedia for data filtering.

What

The current approach for filtering the Japanese mC4 dataset relies primarily on top-level domains, which is insufficient for eliminating unethical content from the data.

How

Introduce an ethical filtering mechanism.

What

llm-foundry enjoys a benefit when the corpus provides a sequence of token IDs.

How

Include the token_ids field in the resultant corpus.

Usage

import sentencepiece as spm
sp = spm.SentencePieceProcessor(model_file='cc100_ja30K_en20K.symbolRemoved.vocab.reestimated.model')

What

Eliminate low-quality text from the Stack dataset.

How

Apply the filtering rules developed in the RedPajama project.