oscar-project / documentation Goto Github PK
View Code? Open in Web Editor NEWLicense: Apache License 2.0
License: Apache License 2.0
The documentation should highlight the organization layout and outline the primary repositories, explaining how it all fits together.
For each project, we should have enough information on the README.md so that people could understand what it is about, as well as a link to the "dev documentation" page.
This should be a tiny guide on contributing, both for people interested in coding (Rust and Python) and data validation/sourcing.
This page should serve as a one-stop shop for anything. It is destined to point people towards three documentations:
This simple tutorial should go from getting the data to having something trained on OSCAR, then maybe improved using filters.
OSCAR Schema v2 found in docs/schema/schema-v2.md
shows json files compressed into gz format.
my understanding is that OSCAR-2301 has moved to zstd format with the extension .zst
I think this should change to show files with .zst format or to merely mention that these files are compressed and
that this is just an example choice for compression.
This is also an extension of my study into docs/schema/schema-v2.md
and docs/tools/generation-jeanzay.md
The hash function sha256 is used for the checksum. Althoguh sha256 is considered secure, it is not resistant to length extension attack.
There are several ways to address this, (if it is a concern)
sha384
or sha512/256
. Both are similar in speed since they both first build sha512 (which is still vulnerable to length extension) and truncated and the bottleneck is the hashing step. (SHA512, SHA384,SHA512/256 each use different initialization vectors so you shouldnt try to roll your own btw). Compared to the total size of OSCAR releases, the increased size of sha384 is negligible as is the processing cost. If processing is an issue, other secure hashes such as BLAKE3 could be used. I guess further indepth considerations for hashes should be directed towards Ungoliant which i plan to do.My point is that, atleast such considerations should be made, and it would be helpful for the documentation to communicate that.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.