mxms0 / qsync Goto Github PK
View Code? Open in Web Editor NEWFaster rsync
Faster rsync
Define both in-memory and wire-format structures for file information.
Ideally, wire format would look something like this, with left-most being first (big-endian):
type (bit?) | unique id (bytes) | path length (incl. null) | path (null-terminated) | file size (bytes) | modification time (UTC) | owner? (tbd) | group? (tbd) | permissions (tbd) | content hash |
---|---|---|---|---|---|---|---|---|---|
true for file | xxHash64 | QUIC var-int | UTF-8 bytes | QUIC var-int | QUIC var-int? | tbd? | tbd? | tbd? | xxHash64 (or 0 for unused) |
Wire format for a directory:
type (bit?) | unique id (bytes) | path length (incl. null) | path (null-terminated) | modification time (UTC) | owner? (tbd) | group? (tbd) | permissions (tbd) |
---|---|---|---|---|---|---|---|
false for directory | xxHash64 | QUIC var-int | UTF-8 bytes | QUIC var-int? | tbd? | tbd? | tbd? |
In memory, the structure could be:
struct FILE_DATA {
xxHash64 Id;
std::path Path;
uint64_t Size;
uint64_t ModTime;
tbd Owner;
tbd Group;
tbd Permissions;
};
The file id is not a hash of the file contents, but rather a hash of the file path. This should make it easier to look-up files (by using the id in a hash table) when referring to a specific file.
There is the possibility of a collision if the client (the one sending updated files) is sending two subdirectories which have identically-named sub-subdirectory paths and files. These subdirectories have different absolute paths on the sending filesystem, which is why they do not collide there.
Before sending paths, remove the root (search) path, so the receiving side can write the files under its own path.
If a symlink points to a location under the root search path, fix it up too. If not, just send it as-is?
The way the traversal of files/directories on the client is designed is such that it will send a directory before it sends any of the paths under that directory. This is beneficial because that means a directory will always exist before any subpaths are written to it.
If these are sent out of order it could prevent the server from successfully doing anything.
It also means that folder timestamps need extra work to correct (see #18 for details). Is there a better way?
There's a lot of repeated/boilerplate code to handle partial message buffering in QUIC_STREAM_EVENT_RECEIVE
. Try to unify this into a template function, or simplify the logic, or refactor, so that it's less error prone, and one fix can fix all callers.
Symbolic link support is not complete and only supports files in one place, but supports files and directories in another.
Fix this madness by changing to two kinds of FileInfo::Type::Symlink (in capnp proto file): FileSymlink and DirSymlink.
Then wire up the support for creating symlinks on the server.
The current file transfer model doesn't correctly set the timestamps on directories. What happens is the folder is created and the timestamp is set, but then files and directories are created under it, and those change the timestamp on the directory to the current day and time.
The solution here probably requires refactoring the way directory timestamps are set such that it happens AFTER the last file/directory is written to that directory.
Maybe have a std::stack
where directories are added as they're received, and once all files have been received/written, pop the directories off the stack and correct the timestamps?
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.