allenai / bff Goto Github PK

@chris-ha458 has made some great improvements to BFF in the https://github.com/allenai/dolma repo. We should back-port those changes here, especially the ones that have to do with correctness (like the ones involving the choice of hash functions).

Chris' PRs are here:

• allenai/dolma#23
• allenai/dolma#24
• allenai/dolma#31
• allenai/dolma#35
• allenai/dolma#39

They won't apply 1:1, because things have changed in the Dolma repo, but at least the important things should carry over.

One thing that might be worth documenting when we get a chance is that the "bff_duplicate_spans" that are created by the --annotate-only are byte spans rather than character spans as a python person such as myself might first assume.

Hi @dirkgr! Here is a feature that would be very much desirable for decontamination, but I'm not sure how difficult it would be to implement into BFF:

The essential part of the feature would be to remove ngrams instead of paragraphs. This is an approach used by GPT3 and TNLG and other than minHash based decontamination it's the other decontamination approach that has actually been used on LLMs in the literature.

Here is the exact specification from TNLG: "We use n-grams to remove texts that occur in the downstream tasks from the training datasets. When we find an n-gram match between a task document and a training document, we split the training document into two pieces by removing the n-gram along with 200 characters from both of its sides. We also remove any split training document with fewer than 200 characters, or training documents which were split more than 10 times."

I wonder if BFF's bloom filter would be necessary to store all the ngrams from the in a several million document combined eval set? If not I should probably consider building this into the WIMBD tool instead.

Thanks for any thoughts you have on this!

Thanks for sharing the great codes!! They have been very useful for me!

I'm new to Rust and bloom filter and I have one question regarding the deduplication scope in your code -- I saw it runs let bloom_filter = bloom_filter.clone(); for each input file. Does this mean the bloom filter won't be synced across threads, i.e., the deduplication scope isn't global? I also wonder what is the best practice for me to run multi-thread processing if I have a very large pretraining corpus to process?

Appreciate your reply. Thank you!!

thanks for this useful tool.

I would like to know that the modified BFF version mentioned in DataComp-LM paper will be open-sourced?

You should be able to run in parallel by running multiple processes right now, but it might be faster/more accurate to run multiple threads in one process.

This necessitates some way of having each thread write to a different output file though, so it's not totally trivial.