jehops / zap Goto Github PK

Hello!

I'm having trouble implementing this script on my home server.
Following along with the blogpost recommended here I reach a snag when I try to replicate more than one set of snapshots.
When I create snapshots with e.g. 'zap snap 1d' and then replicate it with 'zap rep -v' it works just fine. But if i create another snapshot and try to replicate again zfs spits out this error: "cannot receive new filesystem stream: destination has snapshots ..."

I have attached a file here with all the outputs and some information about my pools such as permissions for user zap and the properties zap:snap and snap:name

I have tried this with both version 0.7.1 (from ports) and 0.7.2 (I just copied the script from here) but nothing works.

If I manually try something like: zap@serv1BSD:/usr/home/dano % zfs send -p -i zroot/var/audit@ZAP_serv1BSD_2017-09-02T13:50:10p0200--1w zroot/var/audit@ZAP_serv1BSD_2017-09-02T14:00:44p0200--1d | ssh zap@localhost "zfs recv -Fu -v -d zstore/zroot_bak"

It works, but the output shows some errors about permissions:
receiving incremental stream of zroot/var/audit@ZAP_serv1BSD_2017-09-02T14:00:44p0200--1d into zstore/zroot_bak/var/audit@ZAP_serv1BSD_2017-09-02T14:00:44p0200--1d cannot receive setuid property on zstore/zroot_bak/var/audit: permission denied cannot receive exec property on zstore/zroot_bak/var/audit: permission denied received 312B stream in 1 seconds (312B/sec)

So now I'm at my wits end and don't know what to do, but I'm sure I have managed to misconfigure something.

Thank you for zap!

Problem

Snapshot timestamps are generated using use the strftime format %Y-%m-%dT%H:%M:%S%z with date. No problems yet, Busybox date uses the underlying C library's strftime, and all popular libraries can parse %z when building dates. The problems start when zap's ss_ts function tries to convert the snapshot timestamp back to seconds-since-epoch for processing. This does not work when using Busybox date:

#output of zap with `set -x`
%sev@host: zap rep -v
...
+ ss_ts 2021-07-24T17:38:15-0500
+ echo 2021-07-24T17:38:15-0500
+ sed 's/T/ /'
+ gdate='2021-07-24 17:38:15-0500'
+ date '-d2021-07-24 17:38:15-0500' '+%s'
date: invalid date '2021-07-24T17:38:15-0500'

The reason for this is because Busybox date -d does not include the timezone in any of its recognized list of formats:

%sev@host: date --help
...
Recognized TIME formats:
        @seconds_since_1970
        hh:mm[:ss]
        [YYYY.]MM.DD-hh:mm[:ss]
        YYYY-MM-DD hh:mm[:ss]
        [[[[[YY]YY]MM]DD]hh]mm[.ss]
        'date TIME' form accepts MMDDhhmm[[YY]YY][.ss] instead

It doesn't parse the timezone by default because Busybox date uses the underlying libc strptime, and according to the POSIX specification, strptime does not support parsing timezones or offsets (see IEEE Std 1003.1-2017 Issue 7, Vol. 2, Chap. 3, strptime(), pg. 2065).

Solutions

In order to be compliant with POSIX, the timezone offset should be stripped from the timestamp, both independently converted to seconds, and summed. The stripped timestamp can use date while the offset should be calculated manually.

There is a shortcut that can be taken if the underlying C library supports %z in strptime, for example glibc and EGLIBC. Busybox date accepts the non-standard parameter -D which allows you to specify the format string to parse the date with. Therefor, this issue can be bypassed in such a circumstance by specifying the timezone offset %z in a format string to date -D. Other C libraries that don't support %z like musl libc will still encounter an error. On uClibc, %z is no-op.

Caveats

According to the POSIX specification, strftime %z should always be constructed following the ISO 8601:2004 standard format of [+-]hhmm, so any date implementation that claims compatibility with POSIX strftime or date should use this format. Based on my research I cannot find any instance where this is incorrect, aside from older versions of coreutils (I do not know which). When constructing a date from a format string using coreutils date, older versions defaulted to including : for %z ([+-]hh:mm). In the current version of coreutils this has since been removed from %z and is now accomplished using %:z. In order to maintain backwards compatibility with snapshots made with older coreutils, this format should be accepted.

My time zone is CEST (UTC+2), and running zap 1d thor/home/tobias fails with
cannot create snapshot 'thor/home/tobias@ZAP_2016-04-01T12:54:00+0200--1d': invalid character '+' in name

Hi,

got a chance to try your new fixes and all seems to work ok AFAICT. I however noticed two small problems:

Creating snapshots of datsets that have spaces in their name works fine, but destroying them does not:

zfs create "thinkpad/blabla bla"
zap 1d "thinkpad/blabla bla"
zap -d

fails with

cannot open 'bla@ZAP_2016-04-06T12:31:54p0200--1d': invalid character '@' in pool name

The fix seems to be easy:

diff --git a/zap b/zap
index 5f3aebe..71da9a9 100755
--- a/zap
+++ b/zap
@@ -119,7 +119,7 @@ create () {

 destroy () {
     now_ts=$(date '+%s')
-    for i in $(zfs list -H -t snap -o name); do
+    zfs list -H -t snap -o name | while read i; do
         if echo "$i" | cut -f1 -d'/' | xargs zpool status \
                 | grep -q "DEGRADED\|FAULTED\|OFFLINE\|REMOVED\|UNAVAIL"; then
             warn "zap skipped destroying $i because of pool state!"

Then there are problems with destroying snapshots of the root dataset. zap -d returns:

cannot open 'thinkpad@ZAP_2016-04-06T08:40:00p0200--3d': invalid character '@' in pool name

It's obviously a problem with cut -f1 -d'/', but I don't know how to solve this in a good way.

P.S.: I have a small zap companion script that adds support for ZFS properties: https://gist.github.com/t6/6a0910d85e49423657101c57c892ca8c

When I run:

$ sudo zap snap -d 1w        
WARN: DID NOT snapshot pool/home because of pool state!

I hardwired pool_ok () to always return 0, and the snapshot takes place normally. I suspect something is wrong with the current implementation of pool_ok().

It would be good to allow a backup server to pull snapshots from a remote server, rather than just pushing snapshots to a backup server.

Reasoning is that the server then would not need access to zfs management on the backup server, providing a degree of safety of the backups in case of compromise of the server being backed up.

Attempting zap rep generates an error: invalid option 'c'

apparently the version of zfs in 11.1 doesn't support the 'c' option for zfs send. Is there somewhere I can customize the send options?

/dev/stdout and /dev/stderr are handled on modern Linux by linking to the file descriptor in the proc filesystem at /proc/self/fd/{1,2}, which usually links to your psuedoterminal slave... However, upon switching user with su or sudo (and I imagine doas), not all systems are configured to update permissions to allow the user to write to it. I've read that ConsoleKit might address this, but in my environment (Alpine Linux, Busybox mdev or eudev, Busybox login, OpenRC) it does not happen:

%sev@host: zap -v
WARN: zap has not be sufficiently tested on Linux.
      Feedback and patches are welcome.

0.8.1
%sev@host: sudo -u zfs zap -v
zap: line 172: can't create /dev/stderr: Permission denied
zap: line 630: can't create /dev/stderr: Permission denied
%sev@host/1: sudo -u zfs ls -l /dev/stderr
lrwxrwxrwx    1 root     root            15 Jun 28 10:19 /dev/stderr -> /proc/self/fd/2
%sev@host: sudo -u zfs ls -l /proc/self/fd/2
lrwx------    1 zfs      nogroup         64 Jun 30 11:50 /proc/self/fd/2 -> /dev/pts/0
%sev@host: sudo -u zfs ls -l /dev/pts/0
crw--w----    1 sev      tty       136,   0 Jun 30 11:51 /dev/pts/0

(FYI: proc file descriptor links are actually duplicating the requested descriptor in the kernel when they are opened, they are not real links)

To fix this, POSIX descriptor redirects should be used instead:

index e8253e2..e205a66 100755
--- a/zap
+++ b/zap
@@ -169,7 +169,7 @@ val_dest () {
 }

 warn () {
-  echo "WARN: $*" > /dev/stderr
+  echo "WARN: $*" >&2
 }

 # ==============================================================================
@@ -627,6 +627,6 @@ case $1 in
   snap|snapshot) shift; snap_parse "$@" ;;
   rep|replicate) shift; rep_parse  "$@" ;;
   destroy)       shift; destroy    "$@" ;;
-  -v|-version|--version) echo "$version" > /dev/stderr ;;
+  -v|-version|--version) echo "$version" >&2 ;;
   *)             usage "${0##*/}: missing or unknown subcommand -- $1"; exit 1 ;;
 esac

%sev@host: sudo -u zfs zap -v
WARN: zap has not be sufficiently tested on Linux.
      Feedback and patches are welcome.

0.8.1

Further reading: https://unix.stackexchange.com/questions/38538

I do not have a FreeBSD machine on hand to test with. On FreeBSD, /dev/stdout and /dev/stderr link to /dev/fd/{1,2}.

This allows an admin to get info on snapshots, e.g. prefix=somepkg-1.3.4. I will make a pull req for this soon

I was following along with #16 and one concern I have is using the same command on both the local and remote host. I like the idea of adding to the pipeline (and I will certainly be using mbuffer with it because my performance is terrible over the wire otherwise) but what if the local and remote filters should not or cannot be the same? I think splitting the remote filter into ZAP_FILTER_REMOTE would be the best option, and a value of - or something would retain current functionality where ZAP_FILTER is used for both local and remote, without having to explicitly copy to the other variable yourself.

For example:

• Use different mbuffer buffer sizes on the local and remote machines
• Compress before sending: ZAP_FILTER='lz4 -c -' ZAP_FILTER_REMOTE='lz4 -cd -' zap rep
• Show status on local machine only: ZAP_FILTER=pv zap rep
  (I imagine pv on remote over SSH would not be an issue since a tty is still being allocated, but maybe it isn't available on remote, or I could be wrong and it would cause a problem... I do not use pv myself so I have no idea.)

The echo and print commands for the verbose flag create a lot of bloat and repeated code and it makes the source harder to read. This can be minimized by adding a function that takes a command line as an argument, prints it if we are being verbose, and then runs it. An optional parameter could be used to replace or augment the string to print, since the command line is not always printed verbatim.
If this is something desirable I can submit a PR.

zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-07T19:27:10p0000--1d should be deleted as it's now more than 1D old but I am facing the following errors

/root/zap/zap destroy
or
/root/zap/zap destroy HOSTNAME
both display

WARN: SNAPSHOT zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-07T19:27:10p0000--1d WAS NOT DESTROYED because its expiration time could not be determined.
/root/zap/zap: 162: bc: not found
WARN: SNAPSHOT zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T20:00:01p0000--1d WAS NOT DESTROYED because its expiration time could not be determined.
/root/zap/zap: 162: bc: not found
WARN: SNAPSHOT zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T21:00:01p0000--1d WAS NOT DESTROYED because its expiration time could not be determined.
/root/zap/zap: 162: bc: not found
WARN: SNAPSHOT zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T22:00:01p0000--1d WAS NOT DESTROYED because its expiration time could not be determined.
/root/zap/zap: 162: bc: not found
WARN: SNAPSHOT zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T23:00:01p0000--1d WAS NOT DESTROYED because its expiration time could not be determined.
/root/zap/zap: 162: bc: not found

zfs list -t snapshot

zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-07T19:27:10p0000--1d     6.62M      -     13.7G  -
zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T20:00:01p0000--1d     2.90M      -     13.7G  -
zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T21:00:01p0000--1d     3.18M      -     13.7G  -
zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T22:00:01p0000--1d     5.48M      -     13.7G  -
zfs/backups/vm-105-disk-0@ZAP_HOSTNAME_2023-01-08T23:00:01p0000--1d        0B      -     13.7G  -

Normally one would create recursive snapshots like this:

# zfs snapshot -r mypool@my_recursive_snapshot
# zfs list -t snapshot
NAME                                        USED  AVAIL  REFER  MOUNTPOINT
mypool@my_recursive_snapshot                   0      -   144K  -
mypool/ROOT@my_recursive_snapshot              0      -   144K  -
mypool/ROOT/default@my_recursive_snapshot      0      -   777M  -
mypool/tmp@my_recursive_snapshot               0      -   176K  -
mypool/usr@my_recursive_snapshot               0      -   144K  -
mypool/usr/home@my_recursive_snapshot          0      -   184K  -
mypool/usr/ports@my_recursive_snapshot         0      -   144K  -
mypool/usr/src@my_recursive_snapshot           0      -   144K  -
mypool/var@my_recursive_snapshot               0      -   616K  -
mypool/var/crash@my_recursive_snapshot         0      -   148K  -
mypool/var/log@my_recursive_snapshot           0      -   178K  -
mypool/var/mail@my_recursive_snapshot          0      -   144K  -
mypool/var/newname@new_snapshot_name           0      -  87.5K  -
mypool/var/newname@my_recursive_snapshot       0      -  87.5K  -
mypool/var/tmp@my_recursive_snapshot           0      -   152K  -

Could zap be enhanced so it will make recursive snapshots, or is there a good reason to not to want to have that option (I'm quite new to snapshots).

Thanks.

On two servers I am setting up I need to use a custom ssh port, -p 7000.

Is it possible to specify a custom ssh port wit zap?

I get an error message while replicating that the -c is not supported.
I have version 0.6.5.9-5 of zfs which comes with Debian 9.

Would it be an idea to have more granular control of the TTL of the snapshots? For instance, if you make a snapshot every 15 minutes, but want to keep these for only an hour or four.

Hi @Jehops I like your tool so much, when local replication gets merged in master and I can get it through ports <3 love it. I was wondering if there is a possibility to use another send command.

Now we use this:
zfs send -p

Can we not use :
zfs send -Lcep

It doesn't hurt enabling them by default (from the ZFS manpage).
When one does not have the necessary features the extra flags will do nothing.

I have been doing zap snapshots for a while on my server now, and finally set up a way to do remote replication with zap to another computer on my network with an external HDD plugged in. Both of these machines are running FreeBSD 11.1 and using the current version of zap from ports (0.7.3)

After running zap rep after following the instructions on the blog, I am getting errors such as this on various zfs datasets:

zfs send -I iocage@ZAP_castleDefense_2017-09-30T02:00:00-0700--3w iocage@ZAP_castleDefense_2017-10-01T02:00:00-0700--3w | ssh [email protected] "sh -c 'zfs recv -du -v backups/iocage'"
receiving incremental stream of iocage@ZAP_castleDefense_2017-10-01T02:00:00-0700--3w into backups/iocage@ZAP_castleDefense_2017-10-01T02:00:00-0700--3w
cannot receive incremental stream: most recent snapshot of backups/iocage does not
match incremental source
WARN: Failed to replicate iocage@ZAP_castleDefense_2017-10-01T02:00:00-0700--3w to [email protected]:backups/iocage.
zfs send -I iocage/iocage@ZAP_castleDefense_2017-09-30T02:00:00-0700--3w iocage/iocage@ZAP_castleDefense_2017-10-01T02:00:00-0700--3w | ssh [email protected] "sh -c 'zfs recv -du -v backups/iocage'"

No remote snapshots found. Sending full stream.
zfs send -p iocage/iocage/jails/homeBackup/data@ZAP_castleDefense_2017-09-28T02:00:00-0700--3w | ssh [email protected] "sh -c 'zfs recv -Fu -v -d backups/iocage'"
warning: cannot send 'iocage/iocage/jails/homeBackup/data@ZAP_castleDefense_2017-09-28T02:00:00-0700--3w': permission denied
zfs bookmark iocage/iocage/jails/homeBackup/data@ZAP_castleDefense_2017-09-28T02:00:00-0700--3w iocage/iocage/jails/homeBackup/data#ZAP_castleDefense_2017-09-28T02:00:00-0700--3w
cannot create bookmark 'iocage/iocage/jails/homeBackup/data#ZAP_castleDefense_2017-09-28T02:00:00-0700--3w': unknown error
cannot open 'backups/iocage/iocage/jails/homeBackup/data': dataset does not exist
WARN: Failed to set canmount=noauto for [email protected]:backups/iocage/iocage/jails/homeBackup/data

No remote snapshots found. Sending full stream.
zfs send -p iocage/iocage/jails/minecraft/data@ZAP_castleDefense_2017-09-28T02:00:00-0700--3w | ssh [email protected] "sh -c 'zfs recv -Fu -v -d backups/iocage'"

warning: cannot send 'iocage/iocage/jails/minecraft/data@ZAP_castleDefense_2017-09-28T02:00:00-0700--3w': permission denied

When I set up the permissions as specified on the source machine I only specified the zpool (iocage), and likewise with the remote system I specified the zpool (backups) as well as the dataset below (backups/iocage)

I also mounted my target zpool on an altroot with this:
zpool import -o altroot="/mnt/backupPool" backups

It appears I can see these settings on the zpools. I'm not sure what additional information to provide to solve this, if more is needed let me know and I can provide it