allenai / bff Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
@chris-ha458 has made some great improvements to BFF in the https://github.com/allenai/dolma repo. We should back-port those changes here, especially the ones that have to do with correctness (like the ones involving the choice of hash functions).
Chris' PRs are here:
They won't apply 1:1, because things have changed in the Dolma repo, but at least the important things should carry over.
One thing that might be worth documenting when we get a chance is that the "bff_duplicate_spans" that are created by the --annotate-only
are byte spans rather than character spans as a python person such as myself might first assume.
Hi @dirkgr! Here is a feature that would be very much desirable for decontamination, but I'm not sure how difficult it would be to implement into BFF:
The essential part of the feature would be to remove ngrams instead of paragraphs. This is an approach used by GPT3 and TNLG and other than minHash based decontamination it's the other decontamination approach that has actually been used on LLMs in the literature.
Here is the exact specification from TNLG: "We use n-grams to remove texts that occur in the downstream tasks from the training datasets. When we find an n-gram match between a task document and a training document, we split the training document into two pieces by removing the n-gram along with 200 characters from both of its sides. We also remove any split training document with fewer than 200 characters, or training documents which were split more than 10 times."
I wonder if BFF's bloom filter would be necessary to store all the ngrams from the in a several million document combined eval set? If not I should probably consider building this into the WIMBD tool instead.
Thanks for any thoughts you have on this!
Thanks for sharing the great codes!! They have been very useful for me!
I'm new to Rust and bloom filter and I have one question regarding the deduplication scope in your code -- I saw it runs let bloom_filter = bloom_filter.clone();
for each input file. Does this mean the bloom filter won't be synced across threads, i.e., the deduplication scope isn't global? I also wonder what is the best practice for me to run multi-thread processing if I have a very large pretraining corpus to process?
Appreciate your reply. Thank you!!
thanks for this useful tool.
I would like to know that the modified BFF version mentioned in DataComp-LM paper will be open-sourced?
You should be able to run in parallel by running multiple processes right now, but it might be faster/more accurate to run multiple threads in one process.
This necessitates some way of having each thread write to a different output file though, so it's not totally trivial.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.