kristapsdz / openrsync Goto Github PK

BSD-licensed implementation of rsync

License: ISC License

Makefile 0.40% C 93.26% Roff 6.34%

openrsync's Introduction

Introduction

This system has been merged into OpenBSD base. If you'd like to contribute to openrsync, please mail your patches to [email protected]. This repository is simply the OpenBSD version plus some glue for portability.

This is an implementation of rsync with a BSD (ISC) license. It's compatible with a modern rsync (3.1.3 is used for testing, but any supporting protocol 27 will do), but accepts only a subset of rsync's command-line arguments.

Its officially-supported operating system is OpenBSD, but it will compile and run on other UNIX systems. See Portability for details.

The canonical documentation for openrsync is its manual pages. See rsync(5) and rsyncd(5) for protocol details or utility documentation in openrsync(1). If you'd like to write your own rsync implementation, the protocol manpages should have all the information required.

The Architecture and Algorithm sections on this page serve to introduce developers to the source code. They are non-canonical.

Project background

openrsync is written as part of the rpki-client(1) project, an RPKI validator for OpenBSD. openrsync was funded by NetNod, IIS.SE, SUNET and 6connect.

Installation

On an up-to-date UNIX system, simply download and run:

% ./configure
% make
# make install

This will install the openrsync utility and manual pages. It's ok to have an installation of rsync at the same time: the two will not collide in any way.

If you upgrade your sources and want to re-install, just run the same. If you'd like to uninstall the sources:

# make uninstall

If you'd like to interact with the openrsync as a server, you can run the following:

% rsync --rsync-path=openrsync src/* dst
% openrsync --rsync-path=openrsync src/* dst

If you'd like openrsync and rsync to interact, it's important to use command-line flags available on both. See openrsync(1) for a listing.

Algorithm

For a robust description of the rsync algorithm, see "The rsync algorithm", by Andrew Tridgell and Paul Mackerras. Andrew Tridgell's PhD thesis, "Efficient Algorithms for Sorting and Synchronization", covers the topics in more detail. This gives a description suitable for delving into the source code.

The rsync algorithm has two components: the sender and the receiver. The sender manages source files; the receiver manages the destination. In the following invocation, first the sender is host remote and the receiver is the localhost, then the opposite.

% openrsync -lrtp remote:foo/bar ~/baz/xyzzy
% openrsync -lrtp ~/foo/bar remote:baz/xyzzy

The algorithm hinges upon a file list of names and metadata (e.g., mode, mtime, etc.) shared between components. The file list describes all source files of the update and is generated by the sender. The sharing is implemented in flist.c.

After sharing this list, both the receiver and sender independently sort the entries by the filenames' lexicographical order. This allows the file list to be sent and received out of order. The ordering preserves a directory-first order, so directories are processed before their contained files. Moreover, once sorted, both sender and receiver may refer to file entries by their position in the sorted array.

After the receiver reads the list, it iterates through each file in the list, passing information to the sender so that the sender may send back instructions to update the file. This is called the "block exchange" and is the maintstay of the rsync algorithm. During the block exchange, the sender waits to receive a request for update or end of sequence message; once a request is received, it scans for new blocks to send to the receiver.

Once the block exchange is complete, the files are all up to date.

The receiver is implemented in receiver.c; the sender, in sender.c. A great deal of the block exchange happens in blocks.c.

Block exchange

The block exchange sequence is different for whether the file is a directory, symbolic link, or regular file.

For symbolic links, the information required by the receiver is already encoded in the file list metadata. The symbolic link is updated to point to the correct target. No update is requested from the sender.

For directories, the directory is created if it does not already exist. No update is requested from the sender.

Regular files are handled as follows. First, the file is checked to see if it's up to date. This happens if the file size and last modification time are the same. If so, no update is requested from the sender.

Otherwise, the receiver examines each file in blocks of a fixed size. See Block sizes for details. (The terminal block may be smaller if the file size is not divisible by the block size.) If the file is empty or does not exist, it will have zero blocks. Each block is hashed twice: first, with a fast Adler-32 type 4-byte hash; second, with a slower MD4 16-byte hash. These hashes are implemented in hash.c. The receiver sends the file's block hashes to the sender.

Once accepted, the sender examines the corresponding file with the given blocks. For each byte in the source file, the sender computes a fast hash given the block size. It then looks for matching fast hashes in the sent block information. If it finds a match, it then computes and checks the slow hash. If no match is found, it continues to the next byte. The matching (and indeed all block operation) is implemented in block.c.

When a match is found, the data prior to the match is first sent as a stream of bytes to the receiver. This is followed by an identifier for the found block, or zero if no more data is forthcoming.

The receiver writes the stream of bytes first, then copies the data in the identified block if one has been specified. This continues until the end of file, at which point the file has been fully reconstituted.

If the file does not exist on the receiver side---the basis case---the entire file is sent as a stream of bytes.

Following this, the whole file is hashed using an MD4 hash. These hashes are then compared; and on success, the algorithm continues to the next file.

Block sizes

The block size algorithm plays a crucial role in the protocol efficiency. In general, the block size is the rounded square root of the total file size. The minimum block size, however, is 700 B. Otherwise, the square root computation is simply sqrt(3) followed by ceil(3)

For reasons unknown, the square root result is rounded up to the nearest multiple of eight.

Architecture

Each openrsync session is divided into a running server and client process. The client openrsync process is executed by the user.

% openrsync -rlpt host:path/to/source dest

The server openrsync is executed on a remote host either on-demand over ssh(1) or as a persistent network daemon. If executed over ssh(1), the server openrsync is distinguished from a client (user-started) openrsync by the --server flag.

Once the client or server openrsync process starts, it examines the command-line arguments to determine whether it's in receiver or sender mode. (The daemon is sent the command-line arguments in a protocol-specific way described in rsyncd(5), but otherwise does the same thing.) The receiver is the destination for files; the sender is the origin. There is always one receiver and one sender.

The server process is explicitly instructed that it is a sender with the --sender command-line flag, otherwise it is a receiver. The client process implicitly determines its status by looking at the files passed on the command line for whether they are local or remote.

openrsync path/to/source host:destination
openrsync host:source path/to/destination

In the first example, the client is the sender: it sends data from itself to the server. In the second, the opposite is true in that it receives data.

The client's command-line files may have any of the following host specifications that determine locality.

local: ../path/to/source ../another
remote server: host:path/to/source :path/to/another
remote daemon: rsync://host/module/path ::another

Host specifications must be consistent: sources must all be local or all be remote on the same host. Both may not be remote. (Aside: it's technically possible to do this. I'm not sure why the GPL rsync is limited to one or the other.)

If the source or destination is on a remote server, the client then fork(2)s and starts the server openrsync on the remote host over ssh(1). The client and the server subsequently communicate over socketpair(2) pipes. If on a remote daemon, the client does not fork, but instead connects to the standalone server with a network socket(2).

The server's command-line, whether passed to an openrsync spawned on-demand over an ssh(1) session or passed to the daemon, differs from the client's.

openrsync --server [--sender] . files...

The files given are either the single destination directory when in receiver mode, or the list of sources when in sender mode. The standalone full-stop is a mystery to me.

Locality detection and routing to client and server run-times are handled in main.c. The client for a server is implemented in client.c and the server in server.c. The client for a network daemon is in socket.c. Invocation of the remote server openrsync is managed in child.c.

Once the client and server begin, they start to negotiate the transfer of files over the connected socket. The protocol used is specified in rsync(5). For daemon connections, the rsyncd(5) protocol is also used for handshaking.

The receiver side is managed in receiver.c and the sender in sender.c.

The receiver side technically has two functions: not only must it upload block metadata to the sender, it must also handle data writes as they are sent by the sender. The rsync protocol is designed so that the sender receives block requests and continuously sends data to the receiver.

To accomplish this, the receiver multitasks as the uploader and downloader. These roles are implemented in uploader.c. and downloader.c, respectively. The multitasking takes place by a finite state machine driven by data coming from the sender and files on disc are they are ready to be checksummed and uploaded.

The uploader scans through the list of files and asynchronously opens files to process blocks. While it waits for the files to open, it relinquishes control to the event loop. When files are available, it hashes and checksums blocks and uploads to the sender.

The downloader waits on data from the sender. When data is ready (and prefixed by the file it will update), the downloader asynchronously opens the existing file to perform any block copying. When the file is available for reading, it then continues to read data from the sender and copy from the existing file.

Differences from rsync

The design of rsync involves another mode running alongside the receiver: the generator. This is implemented as another process fork(2)ed from the receiver, and communicating with the receiver and sender.

In openrsync, the generator and receiver are one process, and an event loop is used for speedy responses to read and write requests.

Security

Besides the usual defensive programming, openrsync makes significant use of native security features.

The system operations available to executing code are foremost limited by OpenBSD's pledge(2). The pledges given depend upon the operating mode. For example, the receiver needs write access to the disc---but only when not in dry-run mode (-n). The daemon client needs DNS and network access, but only to a point. pledge(2) allows available resources to be limited over the course of operation.

The second tool is OpenBSD's unveil(2), which limits access to the file-system. This protects against rogue attempts to "break out" of the destination. It's an attractive alternative to chroot(2) because it doesn't require root permissions to execute.

On the receiver side, the file-system is unveil(2)ed at and beneath the destination directory. After the creation of the destination directory, only targets within that directory may be accessed or modified.

Lastly, the MD4 hashs are seeded with arc4random(3) instead of with time(3). (This function is provided on a number of operating systems.) This is only applicable when running openrsync in server mode, as the server generates the seed.

Portability

Many have asked about portability.

The only officially-supported operating system is OpenBSD, as this has considerable security features. openrsync does, however, use oconfigure for compilation on non-OpenBSD systems. This is to encourage porting.

The actual work of porting is matching the security features provided by OpenBSD's pledge(2) and unveil(2). These are critical elements to the functionality of the system. Without them, your system accepts arbitrary data from the public network.

This is possible (I think?) with FreeBSD's Capsicum, but Linux's security facilities are a mess, and will take an expert hand to properly secure.

rsync has specific running modes for the super-user. It also pumps arbitrary data from the network onto your file-system. openrsync is about 10 000 lines of C code: do you trust me not to make mistakes?

openrsync's People

Contributors

Stargazers

Watchers

openrsync's Issues

"undefined reference to `major'" on debian 10 buster

On Debian 10 (Buster), I get the following error

root@rpki0:~/openrsync# ./configure
config.log: writing...
configure.local: no (fully automatic configuration)
arc4random: no
b64_ntop: yes (with -lresolv)
capsicum: no
err: yes 
explicit_bzero: yes 
getprogname: no
INFTIM: no
md5: no
memmem: yes 
memrchr: yes 
memset_s: no
PATH_MAX: yes 
pledge: no
program_invocation_short_name: yes 
reallocarray: no
recallocarray: no
sandbox_init: no
seccomp-filter: yes 
SOCK_NONBLOCK: yes 
strlcat: no
strlcpy: no
strndup: yes 
strnlen: yes 
strtonum: no
sys_queue: no
systrace: no
unveil: no
zlib: no
__progname: yes 
config.h: written
Makefile.configure: written
root@rpki0:~/openrsync# make
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o blocks.o blocks.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o client.o client.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o compats.o compats.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o downloader.o downloader.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o fargs.o fargs.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o flist.o flist.c
In file included from �[01m�[Kflist.c:34�[m�[K:
�[01m�[Kflist.c:�[m�[K In function ‘�[01m�[Kflist_recv�[m�[K’:
�[01m�[Kflist.c:741:4:�[m�[K �[01;35m�[Kwarning: �[m�[Kimplicit declaration of function ‘�[01m�[Kmajor�[m�[K’ [�[01;35m�[K-Wimplicit-function-declaration�[m�[K]
    �[01;35m�[Kmajor�[m�[K(ff->st.rdev), minor(ff->st.rdev));
    �[01;35m�[K^~~~~�[m�[K
�[01m�[Kextern.h:240:45:�[m�[K �[01;36m�[Knote: �[m�[Kin definition of macro ‘�[01m�[KLOG3�[m�[K’
  rsync_log(__FILE__, __LINE__, 2, (_fmt), ##�[01;36m�[K__VA_ARGS__�[m�[K)
                                             �[01;36m�[K^~~~~~~~~~~�[m�[K
�[01m�[Kflist.c:741:24:�[m�[K �[01;35m�[Kwarning: �[m�[Kimplicit declaration of function ‘�[01m�[Kminor�[m�[K’; did you mean ‘�[01m�[Kmknod�[m�[K’? [�[01;35m�[K-Wimplicit-function-declaration�[m�[K]
    major(ff->st.rdev), �[01;35m�[Kminor�[m�[K(ff->st.rdev));
                        �[01;35m�[K^~~~~�[m�[K
�[01m�[Kextern.h:240:45:�[m�[K �[01;36m�[Knote: �[m�[Kin definition of macro ‘�[01m�[KLOG3�[m�[K’
  rsync_log(__FILE__, __LINE__, 2, (_fmt), ##�[01;36m�[K__VA_ARGS__�[m�[K)
                                             �[01;36m�[K^~~~~~~~~~~�[m�[K
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o hash.o hash.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o ids.o ids.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o io.o io.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o log.o log.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o md4.o md4.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o misc.o misc.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o mkpath.o mkpath.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o mktemp.o mktemp.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o receiver.o receiver.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o sender.o sender.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o server.o server.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o session.o session.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o socket.o socket.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o symlinks.o symlinks.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o uploader.o uploader.c
cc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o main.o main.c
cc -o openrsync blocks.o client.o compats.o downloader.o fargs.o flist.o hash.o ids.o io.o log.o md4.o misc.o mkpath.o mktemp.o receiver.o sender.o server.o session.o socket.o symlinks.o uploader.o main.o -lm
/usr/bin/ld: flist.o: in function `flist_recv':
/root/openrsync/flist.c:737: undefined reference to `minor'
/usr/bin/ld: /root/openrsync/flist.c:737: undefined reference to `major'
collect2: error: ld returned 1 exit status
make: *** [Makefile:34: openrsync] Error 1
root@rpki0:~/openrsync#

--delete-delay --delay-updates

Hi, I'm opening this issue just in case you don't know this requirement for snapshot packages mirrors, so you can keep it in mind for future development of openrsync. Due to the way OpenBSD pkg_add works, mirrors are required to use --delete-delay --delay-updates to sync from its source. openrsync will need something equivalent for the use in mirrors.

openrsync on uLinux

Hi,

I've successfully built openrsync for for uLinux. Here is how:

# pwd
/root/openrsync
# ls -lah
total 24K
drwxr-xr-x    2 root     root        4.0K Apr 19 07:08 .
drwxr-xr-x   11 root     root        4.0K Apr 19 06:51 ..
-rw-r--r--    1 root     root         193 Apr 19 07:03 .checksum
-rw-r--r--    1 root     root         156 Apr 19 07:08 .manifest
-rw-r--r--    1 root     root         635 Apr 19 07:06 Pkgfile
-rw-r--r--    1 root     root         674 Apr 19 07:08 configure.patch
# cat *
#!/bin/sh
# shellcheck disable=SC2034

# Description: Utility for incremental file transfers over networks
# URL:         https://rsync.samba.org
# Maintainer:  James Mills, prologic at shortcircuit dot net dot au

name=openrsync
version=20200330
gitver=8b612161bd6c251485725af08d352ae1a7d82eca
release=1
source="https://github.com/kristapsdz/openrsync/archive/$gitver.zip
configure.patch"

build () {
  cd $name-$gitver || exit 1

  patch -i "$SRC"/configure.patch

  ./configure \
    PREFIX=/usr \
    DESTDIR="$PKG"

  make -j "$(nproc)"
  make DESTDIR="$PKG" install

  ln -s $name "$PKG"/usr/bin/rsync

  rm -rf "$PKG"/usr/man
}
--- configure.orig
+++ configure
@@ -89,21 +89,7 @@
 # It does have gcc, so try that instead.
 # Prefer clang, though.

-which ${CC} 2>/dev/null 1>&2 || {
-	echo "${CC} not found: trying clang" 1>&2
-	echo "${CC} not found: trying clang" 1>&3
-	CC=clang
-	which ${CC} 2>/dev/null 1>&2 || {
-		echo "${CC} not found: trying gcc" 1>&2
-		echo "${CC} not found: trying gcc" 1>&3
-		CC=gcc
-		which ${CC} 2>/dev/null 1>&2 || {
-			echo "gcc not found: giving up" 1>&2
-			echo "gcc not found: giving up" 1>&3
-			exit 1
-		}
-	}
-}
+CC=tcc

 #----------------------------------------------------------------------
 # Allow certain variables to be overriden on the command line.
#

Unfortunately I run into some problems "client-side" when doing an rsync from my macOS host to the target server running uLinux with openrsync:

prologic@Jamess-iMac
Sun Apr 19 17:06:35
~/Projects/ulinux.org
 12
$ rsync -avz --delete public/ [email protected]:/var/www/
rsync: -z not supported yet
building file list ... done
404.html
about/index.html
assets/fonts/1506293e.woff
downloader.c:465: error: realloc: Out of memory
receiver.c:424: error: rsync_downloader
server.c:151: error: rsync_receiver
rsync: writefd_unbuffered failed to write 16385 bytes [sender]: Broken pipe (32)
rsync: connection unexpectedly closed (68 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-54/rsync/io.c(453) [sender=2.6.9]

Any ideas what's going on here? is this a problem with rsync on my macOS client or openrsync on the server?
So far I've had to comment out the entire block of code in the ./configure script that tries to detect a CC because it just doesn't work. The CC on uLinux is TinyCC (tcc).

Secondly I run into the following compilation issues:

# make CC=tcc
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o blocks.o blocks.c
In file included from blocks.c:17:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from blocks.c:17:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o client.o client.c
In file included from client.c:17:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from client.c:17:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
tcc -g -W -Wall -Wextra -Wmissing-prototypes -Wstrict-prototypes -Wwrite-strings -Wno-unused-parameter   -c -o compats.o compats.c
In file included from compats.c:1:
config.h:119: warning: #warning No suitable endian.h could be found.
In file included from compats.c:1:
config.h:120: warning: #warning Please e-mail the maintainers with your OS.
compats.c:1568: error: #error No getprogname available.
make: *** [<builtin>: compats.o] Error 1
#

And so I'm contact you as it asks :)

Here is the Pkgfile (so far):

bombing out on assert in ids.c

$ uname -a
OpenBSD xyz.inet6.se 6.4 GENERIC.MP#699 amd64
$ ./openrsync -av rsync://ftp.eu.openbsd.org/OpenBSD/6.4/octeon .
assertion "j < idsz" failed: file "ids.c", line 87, function "idents_assign_uid"
Abort trap (core dumped)
...
(gdb) backtrace
#0 thrkill () at -:3
#1 0x00000ecf667b528e in _libc_abort () at /usr/src/lib/libc/stdlib/abort.c:51
#2 0x00000ecf6676bb92 in _libc___assert2 (file=Variable "file" is not available.
)
at /usr/src/lib/libc/gen/assert.c:52
#3 0x00000ecc859b0dce in idents_assign_uid (sess=0x7f7ffffce678,
fl=0xece9bb3a000, flsz=18, ids=0x0, idsz=0) at ids.c:87
#4 0x00000ecc859ad762 in flist_recv (sess=0x7f7ffffce678, fd=3,
flp=0x7f7ffffce5b8, sz=0x7f7ffffce5a8) at flist.c:771
#5 0x00000ecc859b70c6 in rsync_receiver (sess=0x7f7ffffce678, fdin=3,
fdout=3, root=0xecf5a682970 ".") at receiver.c:214
#6 0x00000ecc859bb8af in rsync_socket (opts=0x7f7ffffceb80, f=0xecefd5fd0c0)
at socket.c:417
#7 0x00000ecc859c0e7f in main (argc=2, argv=0x7f7ffffcf048) at main.c:456

Same for the in-obsd not-yet-linked-to-build rsync(1), but I cloned your version to verify if it was the same error.

alternative to recallocarray?

Hi, i'm trying to porting this project to FreeBSD. I have an only a problem with "recallocarray". Seems not present in FreeBSD. Any ideas?

openrsync: sender mode for remote daemon receivers not yet supported

Is it possible to support the sender mode for a client? This is a basic function. Thanks.

Feature request: --exclude

I often use --exclude when using rsync(1) to back up a home directories. Example:

rsync \ 
    --exclude "vm/*" \
    --exclude "*.core" \
    --exclude ".gimp-2.8/gimpswap.*" \
    -va $HOME myuser@remote:~/backup

In some cases I can emulate this with openrsync(1) by building a list of directories and running rsync on them individually.

openrsync raw device

Hello,

Back in 2008 I requested a command switch for rsync to copy raw device data to a file, vs simply re-creating device nodes. The goal being to stage and backup partitions loaded with hard link snapshots, to off site, remote media.

At that time, with dynamic files properly managed, rsync was used to create hard link snapshot backups, at a high frequency, to a local dedicated partition. This solution perfectly fit the requirements, including nominal resource consumption, and short backup interval. However, duplicating the backup partition to off site media was problematic, due to the very large number of links.

We managed by snapshotting hard links of our monthly backups, to a local disk which was then physically transferred to the remote site. Then on a daily interval, a remote hard link snapshot was created, from newest local one. That gave us one daily remote snapshot backup, but not all the local intervals, even though that would have been nominal additional data.

With a switch to copy device node data, offsite snapshot maintenance could be initialized from a r/w mounted device file (original partition) to an unmounted remote device file. After the initial (dirty) transfer, the remote backup partition would be maintained by re-mounting the local snapshot ro, transferring the (clean) device node data (with a large checksum block-size), then re-mounting the local partition r/w, to resume regular local snapshots.

This process could be repeated at least once per day (possibly more often) to make all of the local snapshots available remotely.

When I made the original feature request, a rsync developer provided a patch, https://lists.samba.org/archive/rsync/2008-March/020573.html but when I followed up later, I learned it wouldn't make it into the regular release cycle because it didn't have a sponsor...

This would be a great feature and I believe simple to implement? Maybe openrsync could support it?

-George

--inplace option

Hi Kristaps,

I hope you will read this on one of your favorite diving places :-). I would like to bring to your attention a "missing" switch which will be worth adding. Let me explain a bit. The easiest way to backup an OpenBSD machine onto an advanced files system like HAMMER, HAMMER2, or ZFS is to just rsync to it. HAMMER history or ZFS snapshot will take care of the rest. However, unless the switch --inplace is used HAMMER will recreate the file and the history will be lost. Please refer for the contest

HAMMER history friendly backup tool

Trying to sync from macos rsync to server with openrsync, but delete in combination with exclude arg not working

If I try and use the command as follows it doesn't work. Syntax error apparently.

rsync -av --rsync-path=openrsync --delete --exclude "test.txt" tmp openbsd.server:tmp/
building file list ... done
openrsync: syntax error in received rules
rsync: connection unexpectedly closed (8 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at /AppleInternal/Library/BuildRoots/11aa8fb2-5f4b-11ee-bc7f-926038f30c31/Library/Caches/com.apple.xbs/Sources/rsync/rsync/io.c(453) [sender=2.6.9]

if I run it either without --delete, but --exclude arg present, it works. It also works if I include --delete, but not use --exclude. Just together it doesn't want to work.

rsync -av --rsync-path=openrsync --exclude "test.txt" tmp openbsd-server:tmp/
building file list ... done

sent 114 bytes received 16 bytes 86.67 bytes/sec
total size is 12 speedup is 0.09

rsync -av --rsync-path=openrsync --delete tmp openbsd-server:tmp/
building file list ... done

sent 118 bytes received 16 bytes 89.33 bytes/sec
total size is 12 speedup is 0.09

--remove-source-files

Is it possible to add the "--remove-source-files" switch from rsync? It is very usefull especially during migrations and backups + probably quite easy to implement...

【help！！】how to use server-client way in openrsync？

I tried using openrsync to sync files the same way I used ssh, and it worked:

openrsync -av [email protected]:/home/liu/binlog .

However, I want to start an openrsync process like rsync, listen to a port; then use ip:pot and username&password on the client side to synchronize data, but it fails. Can you help me to see how this method is used? Thank you very much!

Adding portability to FreeBSD

The attached patch addresses most of the issues which prevent openrsync from compiling on FreeBSD. This adds in the necessary include files and works around OpenBSD-specific features such as pledge and unveil.

There are still some linker issues with hashes, but this patch gets us more than halfway to a build on FreeBSD 11.x.

rsync-freebsd-patch.txt

Port to linux

These are the changes needed to make the program build and run on linux

rain-1@0b2ef0a

mbufs

sorry to bother you, can you tell what i'm doing wrong?
openrsync -ar --del --timeout=10 /var /mnt/root/
where /var is the directory on the ramdisk(mfs), and /mnt/root is the nfs mount point.
in this case, mbufs grow catastrophically- by more than a thousand, and do not return back.
openrsync --rsync-path rsync -ar --del --timeout=10 /var /mnt/root/
in this case, i don't see any problems(and of course no problems with rsync itself without your wrapper). except for one thing- there is no rsync initially in openbsd(yes, there is some problem with this for me).
by the way, if you run the second variant after the first variant, then mbufs return to the state that was before the first variant %\

bidirectional synchronisation

Is it possible (within reasonable bounds) to include support of bi-directional synchronisation, akin to Unison?