nindalf / syncer Goto Github PK
View Code? Open in Web Editor NEWStateful file/device data syncer (mirror of http://git.cypherpunks.ru/cgit.cgi/syncer.git/)
License: GNU General Public License v3.0
Stateful file/device data syncer (mirror of http://git.cypherpunks.ru/cgit.cgi/syncer.git/)
License: GNU General Public License v3.0
syncer -- Fast stateful file/disk data syncer. DESCRIPTION Main purpose of this utility is fast data synchronizing between two hard drives: one is fast (SSD, SATA HDD), another is connected through slow USB interface. Task is to lower amounts of data needed to be transferred. This utility is stateful: it keeps precomputed data hashes in separate statefile and use it to determine if we need to update block of data. # sync from very fast SSD to slow USB connected HDD % ./syncer -src /dev/ada0 -dst /dev/da0 -state state.bin [%%%%%%] # all blocks were transferred to da0 Now we have statefile containing cryptographic hashes of the blocks from source and copy of all read data in destination. Now if we run it again: % ./syncer -src /dev/ada0 -dst /dev/da0 -state state.bin [....%.] # only one block was transferred to da0 Only one modified block was transferred during this session. We read all data from source again, compute hashes and understand what was updated since the last run. Statefile is updated at the end. Utility parallelize hash computations among all found CPUs. It updates statefile atomically (saves data in temporary file and then renames it). You can configure the blocksize: shorter transfers but bigger statefile (it is kept in memory), or larger transfer and smaller statefile. All writes are sequential. syncer is free software: see the file COPYING for copying conditions. INSTALLATION % go get github.com/dchest/blake2b % go build # syncer executable file should be in current directory STATEFILE FORMAT SRC_SIZE || BLK_SIZE || HASH0 || HASH1 || ... SRC_SIZE contains size of the source, when it was firstly read. BLK_SIZE is the blocksize used. Both is 64-bit big-endian unsigned integers. If either size or blocksize differs, then syncer will deny using that statefile as a precaution. HASHx is BLAKE2b-512 hash output, 64 bytes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.