digint / btrbk Goto Github PK

from: http://www.digint.ch/btrbk/doc/readme.html
A link exists to: http://www.digint.ch/btrbk/doc/doc/FAQ.md, which is broken

Add support for btrfs-progs v3.14.1 (stable on Ubuntu 14.10 / Debian wheezy backport).

btrfs-progs 3.14 do not support "btrfs subvolume list -R" flag, which makes it impossible to check send/receive relationship between source and backup subvolume.

Hey,

I like the ability to quickly run btrbk whenever I feel like it, e.g. before an update, experimental work and so forth. In particular, I may run it multiple times a day. At the same time, I don't preserve snapshots (i.e. all snapshot_preserve_* options are set to "0", all target_preserve_* to "all").

The problem is that in this case, the first run will generate the snapshot

SomeSub.YYYYMMDD

The second run will create a new snapshot and delete the old one, so we have in the source directory (as per the documentation)

SomeSub.YYYYMMDD_1

Now if I run btrbk a third time, it fails because the next snapshot it creates has once again the name SomeSub.YYYYMMDD and a subvolume with this name already exists at the target location.

There are several ways this could be fixed, e.g. looking for the highest _N there is in the source folder (taking into account missing folders) and use _(N+1), or using random suffixes. However, my suggestion would be (and it would be much more informative also!) to use not only the date but also the time as the timestamp suffix. Maybe I'm overlooking something, but this would even make it more elegant since you don't have to look up the largest existing _N. Okay, two gotchas could be multiple snapshots within a second (or whatever precision is chosen for the timestamp), and possibly snapshots taken during daylight saving time changes (in particular, duplicate names may arise, and timestamps may not be ordered chronologically, in case this matters). But anyway, just as a thought, maybe this could be made an option.

It would be nice to have the ability to configure BTRBK to send all snapshots that were unable to be sent to a backup volume once the volume appears again. Currently, it'll only send the current day's snapshots, without attempting to re-send previously failed transfers.

I've been using snapper to maintain snapshots on my laptop's to and recover accidentally deleted files. I found btrbk while looking for something that would provide the same onto external hardware, but it seems to me that it fits that requirement AND can replace snapper, since it maintains in-volume snapshots anyway. Is that so or am I missing something?

So this is probably somewhere on the net but i can't seem to find anything with good examples. It would be nice to have an example of how one would go about restoring from an incremental snapshot/backup. Now that i think about it if the original full backup is gone due to retention policy then are any files not modified between it and the last incremental before it was deleted no longer a part of the "backup". The reason i ask is because i'm thinking of switching distros and would like to just use the snapshots/backups i already have to restore my home partition instead of creating a new one.

Cheers!

btrfs fails on a config like this (if symlinked_subvol is a symlink to a real subvolume):

volume /mnt/btr_pool
  subvolume symlinked_subvol

This needs further investigation. For now, the only known workaround is not to use symlinks.

I wanted to use btrbk to create managed local snapshots and not backup to a different btrfs pool. I thought that I should be able to do this by not defining a target.

So I just left the target out of the subvolume I wanted to snapshot. I received this error:

/mnt/btrfs/vm_images
!!! Subvolume "vm_images" aborted: No targets defined for subvolume: /mnt/btrfs/vm_images

Is there any way to do this? Or am I missing something?

It's also fine if this is not how it's intended to be used, but I would think it would be an easy enhancement to add.

So i was in the process of encrypting my entire system and recreating all my btrfs volumes/subvolumes in the process i made a mistake and created my snapshot dir as @backup instead of @BackuPs like was set up originally and defined in btrbk.conf. When i ran dryrun it did not show me any errors:

btrbk -c /etc/btrbk/btrbk.conf dryrun
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.19.2)

    Date:   Thu Jun 11 08:43:35 2015
    Config: /etc/btrbk/btrbk.conf

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/btrfsroot/@Home
+++ /mnt/btrfsroot/@Backups/@Home.20150611
*** /media/Storage3/Backup/@HomeBackup/@Home.20150611

/mnt/btrfsroot/@Storage
+++ /mnt/btrfsroot/@Backups/@Storage.20150611
*** /media/Storage3/Backup/@StorageBackup/@Storage.20150611

NOTE: Dryrun was active, none of the operations above were actually executed!

but then when i ran run it failed(as it should since the @BackuPs subvol is not present):

btrbk -v -c /etc/btrbk/btrbk.conf run
btrbk command line client, version 0.19.2  (Thu Jun 11 08:44:04 2015)
Using configuration: /etc/btrbk/btrbk.conf
Creating subvolume snapshot for: /mnt/btrfsroot/@Home
>>> /mnt/btrfsroot/@Backups/@Home.20150611
ERROR: can't access '/mnt/btrfsroot/@Backups'
ERROR: Failed to create btrfs subvolume snapshot: /mnt/btrfsroot/@Home -> /mnt/btrfsroot/@Backups/@Home.20150611
WARNING: Skipping subvolume section: Failed to create snapshot: /mnt/btrfsroot/@Home -> /mnt/btrfsroot/@Backups/@Home.20150611
Creating subvolume snapshot for: /mnt/btrfsroot/@Storage
>>> /mnt/btrfsroot/@Backups/@Storage.20150611
ERROR: can't access '/mnt/btrfsroot/@Backups'
ERROR: Failed to create btrfs subvolume snapshot: /mnt/btrfsroot/@Storage -> /mnt/btrfsroot/@Backups/@Storage.20150611
WARNING: Skipping subvolume section: Failed to create snapshot: /mnt/btrfsroot/@Storage -> /mnt/btrfsroot/@Backups/@Storage.20150611
Completed within: 0s  (Thu Jun 11 08:44:04 2015)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.19.2)

    Date:   Thu Jun 11 08:44:04 2015
    Config: /etc/btrbk/btrbk.conf

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/mnt/btrfsroot/@Home
!!! Aborted: Failed to create snapshot: /mnt/btrfsroot/@Home -> /mnt/btrfsroot/@Backups/@Home.20150611

/mnt/btrfsroot/@Storage
!!! Aborted: Failed to create snapshot: /mnt/btrfsroot/@Storage -> /mnt/btrfsroot/@Backups/@Storage.20150611

NOTE: Some errors occurred, which may result in missing backups!

So it seems that some part of dryrun is not testing for the presence of the snapshot dir as it should.

btrfs subvol list /mnt/btrfsroot
ID 259 gen 9 top level 5 path @Backup
ID 646 gen 1303 top level 5 path @Home
ID 901 gen 1307 top level 5 path @Storage

I'm running BTRBK as a timer on a systemd-enabled system. I had my back up drive disconnected from my system for a couple of days, and I recently connected it today. Upon running btrbk, I took a look at the executed commands and found that it will attempt to send the first missed snapshot under the current day's name like so:
btrfs send -p /mnt/subvolroot/server/srv/gameservers/tekkit/backups/world.20150323 /mnt/subvolroot/server/srv/gameservers/tekkit/backups/world.20150326

Currently running BTRBK 0.14

Hello,

I just tried out your great backup script and did and Pull based backup from one remote host.

The first time the backup runs fine. But if I did it the second time the error below appears.

please help.

btrbk command line client, version 0.11 (Tue Mar 10 20:45:11 2015)

ERROR: process died unexpectedly (btrbk v0.11)
Please contact the author: Axel Burri <axel@####.##>

Stack Trace:

Use of uninitialized value $& in concatenation (.) or string at /usr/sbin/btrbk line 673, line 108.
at /usr/sbin/btrbk line 97, line 108.
main::ANON('Use of uninitialized value $& in concatenation (.) or string ...') called at /usr/sbin/btrbk line 673
main::btr_subtree('/mnt/btrbk/kerio', 'HASH(0x13ca968)') called at /usr/sbin/btrbk line 1222

I think this may be related to issue #15. In an ubuntu system, I have '@' subvolume for / and @home for /home (from fstab):

UUID=d34c6201-ad16-4e25-81d7-e4d4b6eadcb2 /               btrfs   defaults,compress=lzo,subvol=@ 0       1
UUID=d34c6201-ad16-4e25-81d7-e4d4b6eadcb2 /home           btrfs   defaults,compress=lzo,subvol=@home 0       2

btrbk config is:

snapshot_preserve_daily    14
snapshot_preserve_weekly   0
snapshot_preserve_monthly  0

target_preserve_daily      20
target_preserve_weekly     10
target_preserve_monthly    12

snapshot_dir               btrbk_snapshots

volume /
  subvolume @
    target send-receive    /mnt/backups/andor

  subvolume @home
    target send-receive    /mnt/backups/andor

However, when trying to dryrun, it complains about not finding the subvolumes:

WARNING: Skipping subvolume "//@": Failed to fetch subvolume detail: finding real path for '//@', No such file or directory
WARNING: Skipping subvolume "//@home": Failed to fetch subvolume detail: finding real path for '//@home', No such file or directory
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.19.2)

    Date:   Mon Jun  8 17:54:12 2015
    Config: /etc/btrbk/btrbk.conf

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
//@
!!! Aborted: Failed to fetch subvolume detail: finding real path for '//@', No such file or directory

//@home
!!! Aborted: Failed to fetch subvolume detail: finding real path for '//@home', No such file or directory

NOTE: Some errors occurred, which may result in missing backups!
Please check warning and error messages above.

NOTE: Dryrun was active, none of the operations above were actually executed!

Any ideas?

I hope you don't mind a question rather than a bug here (you might want to consider adding it to the FAQ).

I have been using btrbk for a few months now -- thank you for your efforts. I have just upgraded my debian stretch system and my cron job fails because the "tree" command has been removed.

Can you suggest any way to get similar information with the new commands? I can't find a "list" option which shows a similar approach. "list snapshots" is the closest, but I found the tree formatting to be really useful to see which of my backups was generating the most snapshot activity.

Actually, I don't really need the full "tree" output. I would really like to see, for each source_subvol, how many snapshot_subvol's there are for it, and the name of the most recent one (whether it is up to date or not would be useful as well).

Any suggestion on getting that info would be great.

Yesterday, I messed around with a mainline kernel as there was a bug in BTRFS prohibiting me from converting my setup from RAID0 to raid10. The conversion worked, but I lost power three quarters of the way through and corrupted the file system.
So, I went ahead and re-created it, using the snapshots taken from my backup disk via btrbk.
After a couple of hours of btrfs send/receive, I have a fully functional system again and thought I couldn't be happier, until I attempted to run btrbk to resume my backups.

Upon running, I get this error:

ERROR: process died unexpectedly (btrbk v0.19.2)
Please contact the author: Axel Burri <[email protected]>

Stack Trace:
----------------------------------------
Died at /usr/bin/btrbk line 904, <FILE> line 134.
 at /usr/bin/btrbk line 119, <FILE> line 134.
    main::__ANON__("Died at /usr/bin/btrbk line 904, <FILE> line 134.\x{a}") called at /usr/bin/btrbk line 904
    main::btr_tree(HASH(0x10c6790)) called at /usr/bin/btrbk line 956
    main::vinfo_subvol_list(HASH(0x10c6790)) called at /usr/bin/btrbk line 296
    main::vinfo_root(HASH(0x10c6790)) called at /usr/bin/btrbk line 1679

Running with btrbk -l trace run, the error is printed right after:

### btrfs subvolume list -a -c -u -q -R '/mnt/subvolroot'
... Command output:
ID 258 gen 2054 cgen 7 top level 5 parent_uuid - received_uuid abd4df05-160d-964b-9cd1-d7e97900ab43 uuid 33e0bc86-10f2-1d44-ae52-5bd716662491 path server
ID 295 gen 1929 cgen 52 top level 258 parent_uuid - received_uuid 40b49bc9-9a40-de4a-854e-9c8704f2e55f uuid f7b2eb3a-3b2f-fc4e-8912-a84fc4deed6c path <FS_TREE>/server/etc
ID 296 gen 2054 cgen 55 top level 5 parent_uuid - received_uuid 2286e4be-44fe-5344-9caf-cae657284e15 uuid b379e5bd-5fd4-8146-a5ac-82b2d48c2f3a path desktop
ID 324 gen 1044 cgen 119 top level 335 parent_uuid - received_uuid e32e9aa2-0eeb-bc47-8c85-e256d4c04265 uuid 92709d52-1f6e-0147-b367-0a9a37414ca6 path <FS_TREE>/server/var/lib/machines/guest_os
ID 332 gen 2054 cgen 143 top level 258 parent_uuid - received_uuid b94d353e-6292-7c44-bfa6-ff9944ecf535 uuid 72eae412-c5e4-a241-bbc8-f8b2e39830d5 path <FS_TREE>/server/var/lib/boinc
ID 335 gen 1044 cgen 152 top level 258 parent_uuid - received_uuid bab52f3b-cca5-6d41-b407-13e45fb34424 uuid c16575f9-d75e-874f-b87d-e754858274c3 path <FS_TREE>/server/var/lib/machines
ID 336 gen 1044 cgen 157 top level 296 parent_uuid - received_uuid 10ead242-eb04-234e-bbaa-41ebc1d7bde6 uuid a9776a67-1dd0-9c4f-a203-4b34b2f8841e path <FS_TREE>/desktop/home/[username]/temp
ID 350 gen 1044 cgen 198 top level 258 parent_uuid - received_uuid - uuid 50fb67cb-e42f-ad46-b86e-5811e7c1e7f6 path <FS_TREE>/server/srv/gameservers
ID 351 gen 1044 cgen 202 top level 350 parent_uuid - received_uuid - uuid 112d9f84-d858-724f-bbcf-d3c0a4ab68bf path <FS_TREE>/server/srv/gameservers/tekkit
Command execution successful
Parsed 9 total subvolumes for filesystem at: /mnt/subvolroot
... btr_tree: processing subvolume list of: /mnt/subvolroot

My btrbk config is as follows:

### GLOBALS ###
snapshot_create onchange
snapshot_preserve_daily 7
snapshot_preserve_weekly 4
snapshot_preserve_monthly 2

target_preserve_daily 28
target_preserve_weekly 16
snapshot_preserve_monthly all

btrfs_commit_delete        after
preserve_day_of_week       sunday
resume_missing             yes
incremental                yes

### END GLOBALS ###

### BEGIN LOCAL CONFIG ###

volume /mnt/subvolroot
    snapshot_dir backups
    subvolume  server
        target send-receive    /mnt/backups/Desktop/btrbk/server

    subvolume  server/srv/gameservers/tekkit 
        snapshot_dir server/srv/gameservers/backups
        target_preserve_daily 7
        target send-receive   /mnt/backups/Desktop/btrbk/tekkit

    subvolume server/etc
        target send-receive /mnt/backups/Desktop/btrbk/server

    subvolume  desktop
        target_preserve_weekly 40
        target send-receive    /mnt/backups/Desktop/btrbk/desktop

    subvolume server/var/lib/machines/guest_os
        snapshot_dir server/var/lib/machines/.snapshots
        target send-receive /mnt/backups/Desktop/btrbk/machines

### END LOCAL CONFIG ###

All snapshot directories and subvolumes appear to be accounted for.
I'm not sure how I should proceed.

Hello, I've add some changes to btrbk, to be able to choose ssh cipher for better performance while send/recieve subvolume snapshots via ssh. I have added two ciphers to choose of aes128-ctr and arcfour256. Here are changes:

btrbk.conf:
ssh_cipher                      aes128-ctr

btrbk:
79a80
>    ssh_cipher        => { default => undef,    accept => [ "arcfour256", "aes128-ctr" ] },
238a240
>    my $ssh_cipher      = config_key($config, "ssh_cipher");
241a244
>   push(@ssh_options, '-c', $ssh_cipher) if ($ssh_cipher);

Speed improvements (without compression):

• default cipher ~28MiB/s
• aes128-ctr ~36MiB/s

Cheers!

The following use-case was not considered (thanks Bernd):

volume /
  subvolume @home
    target send-receive    /mnt/btrfs_backup

This will be fixed in refactored btrbk, to be released soon.

(originally by @diraimondo)

I'm wondering if it would be possible to add an option (command-line and/or config) to let btrbk attempt to transfer all the already-created snapshots on source that are missing on target. This skipping any attempt to create further snaps.
This would fit in the following scenario:

• a laptop where I periodically and/or manually create local snaps (like a local time-machine); this could happens even when away from home for many days; the configuration would lack any target and could keep, for example, at most 10 snaps to save space on laptop HD;
• a server at home that frequently, every hour, try connect the laptop (if present). If there exist at least a new snap on laptop that are not present on server, it should be transferred. In this case it would be sufficient to get just the most recent, not all new snaps. (this looks the missing feature).

Thank you for your time.

If the source snapshot directory ("snapshot_dir" in btrbk.conf) is a subvolume instead of a regular directory, btrbk fails to find the snapshots within. As a consequence, it always wants to make a full backup instead of an incremental one.

Linux package maintainers seem to have no consent on where the btrfs binary should reside in. Here is what I found for some distros (thanks Henrik for pointing me to this):

• debian jessie (stable): btrfs-tools-3.17-1.1: /sbin/btrfs
• debian sid (unstable): btrfs-tools-4.0-2: /bin/btrfs
• gentoo: sys-fs/btrfs-progs-4.0: /sbin/btrfs
• arch: btrfs-progs-4.0-2: /usr/bin/btrfs

This is cumbersome, as for security reasons, btrbk sets $PATH="" and always calls /sbin/btrbk, which now fails on some distros with recent btrfs-progs.

Usually it is the task of the package maintainer to fix things like this (i.e. by changing hardcoded paths before installing), but in the debian case, this is not possible since they use different locations for different versions of btrfs-tools.

I see no other option than relaxing the security measures a bit by setting PATH=/sbin:/bin:/usr/sbin:/usr/bin, and call btrfs <subcommand> ... instead of /sbin/btrfs <subcommand> ... from the btrbk executable (as well as in ssh_filter_btrbk.sh). From a security perspective, this should still be ok since an attacker cannot modify the files in either /sbin, /bin, /usr/sbin and /usr/bin (unless he has root privileges).

While btrbk will already create hourly snapshots if it is run every hour via cron, there is no way to manage the expiration of them separately from the daily ones. For example, a configuration which keeps hourly snapshots for 48 hours, but keep dailies for 14 days is not possible.

In order for this to work, several changes are needed:

1. Modify schedule() to add logic to decide which hourlies to delete, similar to daily, monthly, yearly. This implies a new {snapshot,target}_preserve_hourly configuration option, and will break the current preserve logic: currently, setting {snapshot,target}_preserve_daily 14 means "keep ALL
   snapshots/backups for 14 days", no matter how many there are (there might be several snapshots created per day). Introducing a hourly mechanism will change this to "keep only the last daily for 14 days".
2. Append the actual hour number to the snapshot name, e.g. snapname.YYYYMMDD-HH[_N]. This implies a new configuration option snapshot_timestamp_format, as most btrbk users will not
   use this new "hourly" feature and want a simple and consistent subvolume naming. Alternatively, an option hourly_logic yes could automatically enable this alternative timestamp, and would
   also solve the semantics change problem above (while making everything more confusing). Yet another alternative is to use the "otime" (btrfs creation time) internally, which would allow
   to set ANY user-defined timestamp on the subvolume name, since btrbk will simply ignore it (making everything much more confusing, especially if the clocks on several hosts differ).
3. Do not unconditionally create a new snapshot every time it is run, but only if snapshot_preserve_daily is non-zero or this is the first run of the day. This is important so that some
   subvolume sections can be configured to generate hourlies while others do not, in the same .conf file. This could be solved by introducing a snapshot_create onschedule configuration option.

Summing up, this feature introduces more complexity and confusion, while a main design goal of btrbk is "keep things simple and easy to configure".

Any thoughts?

Hi,

It seems that even though I specify not to keep any snapshots on the source, btrbk is keeping snapshots and not deleting them.
I wonder if this is an configuration issue on my side or an bug.

Directory Listing

drwxr-xr-x 1 root root    12 Mar 30  2013 home/
drwxr-xr-x 1 root root    12 Mar 30  2013 home.20150820/
drwxr-xr-x 1 root root    12 Mar 30  2013 home.20150829/
drwxr-xr-x 1 root root    12 Mar 30  2013 home.20150831/
drwxr-xr-x 1 root root    12 Mar 30  2013 home.20150902/
drwxr-xr-x 1 root root    12 Mar 30  2013 home.20150903/
drwxr-xr-x 1 root root    12 Mar 30  2013 home.20150904/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root.20150820/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root.20150829/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root.20150831/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root.20150902/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root.20150903/
drwxr-xr-x 1 root root   186 Jul 25 16:44 root.20150904/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm.20150820/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm.20150829/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm.20150831/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm.20150902/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm.20150903/
drwxrwxr-x 1 root vmware 138 Jul 19 15:14 vm.20150904/

Configuration

% cat /etc/btrbk/btrbk.conf
resume_missing          no

snapshot_preserve_daily     1
snapshot_preserve_weekly    0
snapshot_preserve_monthly   0

target_preserve_daily       1
target_preserve_weekly      1
target_preserve_monthly     0

volume /var/lib/btrfs/r0-ssd
  subvolume home
    target send-receive     /var/lib/btrfs/r1-sata/backups

  subvolume root
    target send-receive     /var/lib/btrfs/r1-sata/backups

  subvolume vm
    target send-receive     /var/lib/btrfs/r1-sata/backups
    target_preserve_weekly  0
    target_preserve_monthly 0

Dry-Run

Using: btrbk -v -l debug -c /etc/btrbk/btrbk.conf dryrun

btrbk command line client, version 0.20.0-dev  (Sat Sep  5 18:03:54 2015)
Using configuration: /etc/btrbk/btrbk.conf
### 'btrfs' 'subvolume' 'show' '/var/lib/btrfs/r0-ssd' 2>&1
Command execution successful
found btrfs root: /var/lib/btrfs/r0-ssd
### 'btrfs' 'subvolume' 'list' '-a' '-c' '-u' '-q' '-R' '/var/lib/btrfs/r0-ssd'
Command execution successful
Parsed 21 total subvolumes for filesystem at: /var/lib/btrfs/r0-ssd
Found 21 subvolume children of: /var/lib/btrfs/r0-ssd
### 'btrfs' 'subvolume' 'show' '/var/lib/btrfs/r1-sata/backups' 2>&1
Command execution successful
Parsed 12 subvolume detail items: /var/lib/btrfs/r1-sata/backups
### 'btrfs' 'subvolume' 'list' '-a' '-c' '-u' '-q' '-R' '/var/lib/btrfs/r1-sata/backups'
Command execution successful
Parsed 12 total subvolumes for filesystem at: /var/lib/btrfs/r1-sata/backups
Found 9 subvolume children of: /var/lib/btrfs/r1-sata/backups
### 'btrfs' 'subvolume' 'show' '/var/lib/btrfs/r1-sata/backups' 2>&1
Command execution successful
Parsed 12 subvolume detail items: /var/lib/btrfs/r1-sata/backups
Found 9 subvolume children of: /var/lib/btrfs/r1-sata/backups
### 'btrfs' 'subvolume' 'show' '/var/lib/btrfs/r1-sata/backups' 2>&1
Command execution successful
Parsed 12 subvolume detail items: /var/lib/btrfs/r1-sata/backups
Found 9 subvolume children of: /var/lib/btrfs/r1-sata/backups
Snapshot creation enabled (snapshot_create=always)
Creating subvolume snapshot for: /var/lib/btrfs/r0-ssd/home
[btrfs] snapshot (ro):
[btrfs]   source: /var/lib/btrfs/r0-ssd/home
[btrfs]   target: /var/lib/btrfs/r0-ssd//home.20150905
>>> /var/lib/btrfs/r0-ssd//home.20150905
### (dryrun) 'btrfs' 'subvolume' 'snapshot' '-r' '/var/lib/btrfs/r0-ssd/home' '/var/lib/btrfs/r0-ssd//home.20150905'
<SNIP>
Creating subvolume backup (send-receive) for: /var/lib/btrfs/r0-ssd/home
Found 6 snapshot children of: /var/lib/btrfs/r0-ssd/home
Found 1 receive targets in "/var/lib/btrfs/r1-sata/backups/" for: /var/lib/btrfs/r0-ssd/home.20150904
Latest common snapshots for: /var/lib/btrfs/r0-ssd/home: src=/var/lib/btrfs/r0-ssd/home.20150904  target=/var/lib/btrfs/r1-sata/backups/home.20150904
Receiving from snapshot: /var/lib/btrfs/r0-ssd//home.20150905
Incremental from parent snapshot: /var/lib/btrfs/r0-ssd/home.20150904
>>> /var/lib/btrfs/r1-sata/backups/home.20150905
[btrfs] send/receive (incremental):
[btrfs]   source: /var/lib/btrfs/r0-ssd//home.20150905
[btrfs]   parent: /var/lib/btrfs/r0-ssd/home.20150904
[btrfs]   target: /var/lib/btrfs/r1-sata/backups
### (dryrun) 'btrfs' 'send' '-p' '/var/lib/btrfs/r0-ssd/home.20150904' '-v' '/var/lib/btrfs/r0-ssd//home.20150905' | 'btrfs' 'receive' '-v' '/var/lib/btrfs/r1-sata/backups/'
<SNIP>
Cleaning backups of subvolume "/var/lib/btrfs/r0-ssd/home": /var/lib/btrfs/r1-sata/backups/home.*
Filter scheme: preserving all within 1 days
Filter scheme: preserving first in week (starting on sunday), for 1 weeks
Filter scheme: preserving last weekly of month, for 0 months
=== /var/lib/btrfs/r1-sata/backups/home.20150829: preserved weekly: 1 weeks ago, +6 days after sunday
=== /var/lib/btrfs/r1-sata/backups/home.20150831: preserved weekly: 0 weeks ago, +1 days after sunday
<<< /var/lib/btrfs/r1-sata/backups/home.20150903
=== /var/lib/btrfs/r1-sata/backups/home.20150904: preserved daily: 1 days ago
[btrfs] delete:
[btrfs]   subvolume: /var/lib/btrfs/r1-sata/backups/home.20150903
### (dryrun) 'btrfs' 'subvolume' 'delete' '/var/lib/btrfs/r1-sata/backups/home.20150903'
Deleted 1 subvolumes in: /var/lib/btrfs/r1-sata/backups/home.*
Cleaning snapshots: /var/lib/btrfs/r0-ssd//home.*
Filter scheme: preserving all within 1 days
Filter scheme: preserving first in week (starting on sunday), for 0 weeks
Filter scheme: preserving last weekly of month, for 0 months
Deleted 0 subvolumes in: /var/lib/btrfs/r0-ssd//home.*
<SNIP>
Completed within: 0s  (Sat Sep  5 18:03:54 2015)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.20.0-dev)

    Date:   Sat Sep  5 18:03:54 2015
    Config: /etc/btrbk/btrbk.conf

Legend:
    ===  up-to-date subvolume (source snapshot)
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
/var/lib/btrfs/r0-ssd/home
+++ /var/lib/btrfs/r0-ssd//home.20150905
>>> /var/lib/btrfs/r1-sata/backups/home.20150905
--- /var/lib/btrfs/r1-sata/backups/home.20150903

/var/lib/btrfs/r0-ssd/root
+++ /var/lib/btrfs/r0-ssd//root.20150905
>>> /var/lib/btrfs/r1-sata/backups/root.20150905
--- /var/lib/btrfs/r1-sata/backups/root.20150903

/var/lib/btrfs/r0-ssd/vm
+++ /var/lib/btrfs/r0-ssd//vm.20150905
*** /var/lib/btrfs/r1-sata/backups/vm.20150905
--- /var/lib/btrfs/r1-sata/backups/vm.20150810

Maybe you can shed some light on this?
Thanks!

hi there :)

first off, thank you for this nifty little tool. makes my backup-workflow easier.

back to the matter:
it would be nice to be able to run a command or a script before and/or after the backup is executed.
i for example do not have mounted my subvolid=0's by default so for now i put btrbk in a shell script where i first mount the subvolid=0 of the disks needed, run the backups/snapshots and then unmount them again. it works but it's yet another wrapper-script flying around...

bonus would be an action on error.

Right now I'm setting identity files via the config and port numbers via ssh_config. It would be nice to have the ability to specify ssh's port number via btrbk's config for servers that use a non-standard port.

I have a USB drive as one target for my snapshots which isn't connected most of the time.
What I would like to do, is connect it once in a while and sync the snapshot (copy missing and delete old). At the moment this means that I have to connect the drive, make sure it is mounted and wait for the cron-job to happen. Or start the backup manually which will result in two snapshots for the same day.
What I'm missing is a new command or a option for the "run" command to prevent btrbk from making new snapshots, just making sure all targets are up to date.
This would also be useful in case a remote server was down during the backup.

Hi,

first of all: Thanks a lot for this great tool! I use it regularly to transfer snapshots to my usb-drive.

While doing so, I tried the following scenario:

• Take a daily snapshot
• Transfer all daily snapshots, whenever my usb-drive is attached (e.g. once per week)

To do so, I have these settings in my config file:
snapshot_create_always yes
resume_missing yes

Mount point of my usb-drive is /mnt/btrfs_bkp. But if I want to run the script without having the disk attached, it will fail, throwing the following error (since the path /mnt/btrfs_bkp exists, but does not contain a btrfs-filesystem):

WARNING: No real path provided by "btrfs subvolume show" for subvolume "/mnt/btrfs_bkp", using: /mnt/btrfs_bkp

ERROR: process died unexpectedly (btrbk v0.17.0)
Please contact the author: *********
Stack Trace:
----------------------------------------
Use of uninitialized value in hash element at /usr/sbin/btrbk line 879, <FILE> line 95.
at /usr/sbin/btrbk line 103, <FILE> line 95.
main::__ANON__("Use of uninitialized value in hash element at /usr/sbin/btrbk"...) called at /usr/sbin/btrbk line 879
main::btr_tree(HASH(0x2b9c100)) called at /usr/sbin/btrbk line 909
main::vinfo_subvol_list(HASH(0x2b9c100)) called at /usr/sbin/btrbk line 262
main::vinfo_root(HASH(0x2b9c100)) called at /usr/sbin/btrbk line 1574

Is there any way to avoid this? --> Creating the snapshots, but only transfering if the destination path constains a btrfs-filesystem?

I left the backup volume disconnected from my laptop for about a week or two, and went to reconnect it today to transfer over the missing snapshots. The most recent backup on the drive is from March 25th, and the laptop has snapshots from April 5th to today. I was expecting btrbk to continue incremental backups using the March backup as the parent source, but instead it started a full backup from April.

Creating subvolume snapshot for: /mnt/subvolroot/rtfs
>>> /mnt/subvolroot/.snapshots/rtfs.20150415
Checking for missing backups of subvolume "/mnt/subvolroot/rtfs" in: /mnt/backups/Laptop/btrbk/
Resuming subvolume backup (send-receive) for: /mnt/subvolroot/.snapshots/rtfs.20150405
Receiving from snapshot: /mnt/subvolroot/.snapshots/rtfs.20150405
No common parent subvolume present, creating full backup
>>> /mnt/backups/Laptop/btrbk/rtfs.20150405
At subvol /mnt/subvolroot/.snapshots/rtfs.20150405
At subvol rtfs.20150405

The laptop snapshot directory

root [/root]$ ls /mnt/subvolroot/.snapshots
boot.20150405  boot.20150410  boot.20150415  etc.20150409  etc.20150414   home.20150410  home.20150415  rtfs.20150410  rtfs.20150415
boot.20150406  boot.20150411  etc.20150405   etc.20150410  etc.20150415   home.20150411  rtfs.20150405  rtfs.20150411
boot.20150407  boot.20150412  etc.20150406   etc.20150411  home.20150405  home.20150412  rtfs.20150406  rtfs.20150412
boot.20150408  boot.20150413  etc.20150407   etc.20150412  home.20150406  home.20150413  rtfs.20150407  rtfs.20150413
boot.20150409  boot.20150414  etc.20150408   etc.20150413  home.20150407  home.20150414  rtfs.20150409  rtfs.20150414

the backup volume directory

root [/root]$ ls /mnt/backups/Laptop/btrbk 
boot.20150323  boot.20150325  etc.20150324  home.20150323  home.20150325  rtfs.20150324  rtfs.20150405
boot.20150324  etc.20150323   etc.20150325  home.20150324  rtfs.20150323  rtfs.20150325

I understand that the backup volume snapshots are relatively old, but shouldn't an incremental send still be possible with the March snapshots as a parent? There is little difference between the March backups and the recent ones (the differences only being a couple of documents that I worked on and some downloads, no updates or significant changes were made).

Hello,

I have enabled quota support and btrfs creates a qgroup with id 0/{subvolid} whenever a subvolume is created. But this qgroup is not deleted when the subvolume is deleted. Could you add an option to delete the correct qgroup when the subvolume is deleted?

File root subvolume on my file system is labelled @ following the ubuntu naming convention (https://help.ubuntu.com/community/btrfs).
When I perform a dryrun using this label I get the error:

btrbk command line client, version 0.12  (Thu Mar 19 09:16:40 2015)
ERROR: Ambiguous file for option "subvolume" in "btrbk-#####.conf" line 83: @
ERROR: Failed to parse configuration file

This also happens for my subvolume labelled @home.

btrbk command line client, version 0.12  (Thu Mar 19 09:17:40 2015)
ERROR: Ambiguous file for option "subvolume" in "btrbk-#####.conf" line 88: @home
ERROR: Failed to parse configuration file

Hi !
I've set up a backup initiated from my computer to my local NAS.
I would like now to make remote backups to have a real backup plan in case of stealing, fire, asteroid...
Since WAN connection is way too much slow compared to gigabit local network, I'd like the NAS to take care of doing the remote backups, since it's online 24/24 and it can do this every nights.

Myhost --> LAN NAS --> Remote NAS

Here is the configuration :

- MyHost

volume /btrfs/pool_hdd
   snapshot_create  always
   resume_missing   yes

   target_preserve_daily      7
   target_preserve_weekly     4
   target_preserve_monthly    12

   subvolume home/alex
        snapshot_dir            home_snapshots/
        snapshot_name           alex.home
        target send-receive     ssh://nas/btrfs/pool_1/backups/alex/alex-pc

- LAN NAS
All backups of alex-pc are located in /btrfs/pool_1/backups/alex/alex-pc
e.g. alex.home.YYYYMMDD, root.YYYMMDD ...

The backups from LAN NAS to remote NAS are last resort backups, so I want to keep only one snapshot on the remote NAS, no need to have history.

And from this point, I failed to see how I can again backup the backups... Since it has timestamps...
How could I get around this ?

Questions

1. The only solution I see is to make a symlink to the latest snapshot on the LAN NAS, which could be done with your new --raw-output and a little scripting.
   Or is there another obvious solution I didn't see ?
2. And if I follow the symlink road, do you recommend taking doing one snapshot again or directly send alex.home.latest ?

--- Solution 1 : snapshoting the latest snap

Directories would be like that :
/btrfs/pool_1/backups/alex/alex-pc
- alex.home.YYYYMMDD
- alex.home.latest

/btrfs/pool_1/backups/alex/alex-pc-sending-remotely
- alex.home.YYYYMMDD newly created snap from alex.home.latest

volume /btrfs/pool_1/backups/alex
   snapshot_create  always
   resume_missing   no

   snapshot_preserve_daily    1
   snapshot_preserve_weekly   0
   snapshot_preserve_monthly  0

   target_preserve_daily      1
   target_preserve_weekly     0
   target_preserve_monthly    0

   subvolume alex-pc/alex.home.latest
        snapshot_dir            alex-pc-sending-remotely/
        snapshot_name           alex.home
        target send-receive     ssh://remote.nas/backups/alex/alex-pc

--- Solution 2 : directly send the latest snapshot

/btrfs/pool_1/backups/alex/alex-pc
- alex.home.YYYYMMDD
- alex.home.latest

volume /btrfs/pool_1/backups/alex
   snapshot_create  no
   resume_missing   no

   target_preserve_daily      1
   target_preserve_weekly     0
   target_preserve_monthly    0

   subvolume alex-pc/alex.home.latest
        target send-receive     ssh://remote.nas/backups/alex/alex-pc

I tend to prefer solution 1 because if I don't do another snapshot and the LAN NAS go down during a certain amount of time,
it could be possible that the common subvolume between LAN NAS and remote NAS get's deleted during the backup of MyHost (following target_preserve policy) ?

I'd really like your thoughts about all this, and if you see a better way to implement this.
It's complicated, don't hesitate to ask me again if I wasn't clear with my explanations and questions...
Thanks a lot !

Hi,

I'm using btrbk 0.16 on Arch Linux. I have source subvolumes named 0, 1, 2, ... which I would like to backup. However, the config file parsing fails when encountering the "subvolume 0" line (however, e.g. "subvolume 1" works):

...
config: adding volume "<path omitted>" to root context
ERROR: Failed to parse configuration file

When I tried to change "0" to "0/" in the config file, the error gets even worse:

...
ERROR: process died unexpectedly (btrbk v0.16)
Please contact the author: Axel Burri <e-mail address omitted>

Stack Trace:
----------------------------------------
Died at /usr/bin/btrbk line 1371, <FILE> line 116.
 at /usr/bin/btrbk line 101, <FILE> line 116.
    main::__ANON__("Died at /usr/bin/btrbk line 1371, <FILE> line 116.\x{a}") called at /usr/bin/btrbk line 1371

Not sure if this is something i am doing of if it's really broken. I know i did a diff before just can remeber on what version. With .16 this is the output of trying to diff 2 sub volumes

btrbk -c /etc/btrbk/btrbk.conf diff "/media/Storage3/Backup/@HomeBackup/@Home.20150417" "/media/Storage3/Backup/@HomeBackup/@Home.20150420"

ERROR: process died unexpectedly (btrbk v0.16)
Please contact the author: Axel Burri [email protected]

Stack Trace:

Died at /usr/bin/btrbk line 204.
at /usr/bin/btrbk line 101.
main::ANON("Died at /usr/bin/btrbk line 204.\x{a}") called at /usr/bin/btrbk line 204
main::config_key(undef, "btrfs_progs_compat") called at /usr/bin/btrbk line 523
main::btr_subvolume_list("/media/Storage3/Backup/@HomeBackup/@Home.20150417", undef, "subvol_only", 0) called at /usr/bin/btrbk line 649
main::btr_tree("/media/Storage3/Backup/@HomeBackup/@Home.20150417") called at /usr/bin/btrbk line 1220

My btrbk.conf

http://pastebin.com/9J9w6ZaU

I'm looking at encrypting my main btrfs pool using dm-crypt/luks and wonder about backups using btrbk(really in general i suppose) if i setup a ecryptfs folder on my backup harddrive ex: /mnt/usbdrive/.backup mounted at /mnt/usbdrive/backup and create a subvol called @homebackup inside this ecryptfs mount would this cause any potential problems to btrbk? In the end i suppose this is more a general "how well does ecryptfs work with btrfs subvols" but i figured this was as good a place as any to ask plus it might help others.

I'm running arch with btrfs-progs 4.1-1. btrbk -l debug -v dryrun produces the following output: http://pastebin.com/nHfAqwvg

I downgraded to btrfs-progs v 3.17.3 and it worked like a charm, as it used to.

It looks as though the format changed from:

/backup/backup-root
Name: backup-root
uuid: 4c791925-6637-7f49-9d94-8c1ee9ff97d4
Parent uuid: -
Creation time: 2015-06-19 19:21:44
Object ID: 585
Generation (Gen): 1292
Gen at creation: 1087
Parent: 5
Top Level: 5
Flags: -
Snapshot(s):
to

/backup/backup-root
Name: backup-root
UUID: 4c791925-6637-7f49-9d94-8c1ee9ff97d4
Parent UUID: -
Received UUID: -
Creation time: 2015-06-19 19:21:44 -0500
Subvolume ID: 585
Generation: 1292
Gen at creation: 1087
Parent ID: 5
Top level ID: 5
Flags: -
Snapshot(s):

I would keep in a unique configuration file all the remote/local subvolume managed by the backup server. Such volumes could be managed using different strategies (once per day, once per week, only sync because snapshots are made remotely using recent introduced features, ...). It would nice if I could restrict on command-line the selected action on a specified subvolume (not all). As long as I understand, it is not possible at the moment: all the subvolumes are involved.

Running btrbk 0.19.3.

Running btrbk via cron sends me an email even if there aren't any errors.
I tried running the command outside of cron gives me:

sudo btrbk -q -c /etc/btrbk/btrbk-######.conf run           
At subvol /mnt/btr_system/btrbk_snapshots/@.20150709_3
At subvol /mnt/btr_system/btrbk_snapshots/@home.20150709_3
At subvol /mnt/btr_system/btrbk_snapshots/@vms.20150709_3

Which is the same output I'm being emailed daily.
I'm trying to run btrfs-tools only from the main repo (3.17) on Kubuntu 15.04 on my desktop. This doesn't happen on my main ubuntu server where I run the latest btrfs-progs.
I guess it might not be a btrfs-tools or btrbk bug and just a problem with my system.
If this isn't an easy fix I can just download and build the latest btrfs-progs, but I thought I should let you know.

There is a new branch adding a feature to create raw backups on non-btrfs targets ("target_raw" branch). This is very useful if you want to make backups to some cloud storage or your backup file server does not support btrfs. This will probably be merged (as an experimental feature) into the next release.

The idea is to do a btrfs send directly to a file, which can then be restored by cat <file> | btrfs receive.

Features so far:

• Create raw backup images on non-btrfs target
• Compression support (gzip, xz, bzip2)
• Encryption support (gpg)

Bugs:

• Target preserve mechanism is disabled, as we cannot simply delete incremental files here. This must be done by hand
• No rotation of incremental backups: in order to be able to delete old backups, a full backup must be triggered from time to time. I have no simple solution for this, maybe something like an "incremental monthly" config would do the trick.
• Creates buggy (zero-sized) output files if gpg fails

btrbk.conf:

raw_target_compress  xz
raw_target_encrypt   gpg
gpg_keyring          /etc/btrbk/gpg/pubring.gpg
gpg_recipient        [email protected]

volume /mnt/btr_pool
  subvolume home
    target raw ssh://myserver.mydomain.com/non-btrfs-directory

This will create encrypted/compressed raw files on myserver.mydomain.com:

• /non-btrfs-directory/home.20150101.<received_uuid>.<parent_uuid>.btrfs.xz.gpg

Hi!

A dryrun with this config file

snapshot_dir               btrbk_snapshots
incremental                yes
snapshot_create always
resume_missing             yes
preserve_day_of_week       monday

snapshot_preserve_daily    14
snapshot_preserve_weekly   8
snapshot_preserve_monthly  all

target_preserve_daily      14
target_preserve_weekly     8
target_preserve_monthly    all

btrfs_commit_delete        after

volume /home
  subvolume user1
    target send-receive    /mnt/btrfs_bkp

fails like this

btrbk command line client, version 0.19.1  (Sat Jun  6 11:09:09 2015)
Using configuration: /foo/bar/btrbk.conf

ERROR: process died unexpectedly (btrbk v0.19.1)
Please contact the author: Axel Burri <[email protected]>

Stack Trace:
----------------------------------------
Use of uninitialized value $key in concatenation (.) or string at /usr/sbin/btrbk line 367, <FILE> line 95.
 at /usr/sbin/btrbk line 119, <FILE> line 95.
    main::__ANON__("Use of uninitialized value \$key in concatenation (.) or strin"...) called at     /usr/sbin/btrbk line 367
    main::check_file("ERROR", HASH(0x1c49170)) called at /usr/sbin/btrbk line 632
main::btrfs_subvolume_detail(HASH(0x1b8bf08)) called at /usr/sbin/btrbk line 291
main::vinfo_root(HASH(0x1b8bf08)) called at /usr/sbin/btrbk line 1728

Do you have any hint?

Running version 0.19.3, attempting to declare subvolumes like "VirtualBox VMs", "BitTorrent Sync", etc. in the config returns "Ambiguous file for option "subvolume""

I tried using double and single quotes, as well as "" to escape the space, and \040, of which none appeared to work.

I'm attempting to substitute my personal bash-script to remotely backup my laptop on a local server using btrfs send/receive feature. In my attempt the backup is started by the server. My problem is related to the cases where the transfer is interrupted for a network problem (or because if go away with the laptop). In this case, if 'resume_missing' is 'yes', btrbk attempts to resume previous snapshot but this fails. This looks a bug (it should delete previous backup).

$ sudo btrbk -l info run      
btrbk command line client, version 0.17.0  (Sat May  9 11:13:13 2015)
Using configuration: /etc/btrbk/btrbk.conf
Creating subvolume snapshot for: {gandalf}/mnt/btrfs-pool/PACMAN
>>> {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509
Creating subvolume backup (send-receive) for: {gandalf}/mnt/btrfs-pool/PACMAN
Receiving from snapshot: {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509
No common parent subvolume present, creating full backup
>>> /mnt/btrfs-backup-pool/BTRBK/PACMAN.20150509
At subvol /mnt/btrfs-pool/snapshots/PACMAN.20150509
At subvol PACMAN.20150509
^C

$ sudo btrbk -l info run           
btrbk command line client, version 0.17.0  (Sat May  9 11:14:03 2015)
Using configuration: /etc/btrbk/btrbk.conf
Creating subvolume snapshot for: {gandalf}/mnt/btrfs-pool/PACMAN
>>> {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509_1
Checking for missing backups of subvolume "{gandalf}/mnt/btrfs-pool/PACMAN" in: /mnt/btrfs-backup-pool/BTRBK/
Resuming subvolume backup (send-receive) for: {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509
Receiving from snapshot: {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509
No common parent subvolume present, creating full backup
>>> /mnt/btrfs-backup-pool/BTRBK/PACMAN.20150509
At subvol /mnt/btrfs-pool/snapshots/PACMAN.20150509
At subvol PACMAN.20150509
ERROR: creating subvolume PACMAN.20150509 failed. File exists
ERROR: Failed to send/receive btrfs subvolume: {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509  -> /mnt/btrfs-backup-pool/BTRBK
ERROR: Error while resuming backups, aborting
Resumed 1 backups
WARNING: Skipping cleanup of snapshots for subvolume "{gandalf}/mnt/btrfs-pool/PACMAN", as at least one target aborted earlier
Completed within: 2s  (Sat May  9 11:14:05 2015)
--------------------------------------------------------------------------------
Backup Summary (btrbk command line client, version 0.17.0)

    Date:   Sat May  9 11:14:03 2015
    Config: /etc/btrbk/btrbk.conf

Legend:
    +++  created subvolume (source snapshot)
    ---  deleted subvolume
    ***  received subvolume (non-incremental)
    >>>  received subvolume (incremental)
--------------------------------------------------------------------------------
{gandalf}/mnt/btrfs-pool/PACMAN
+++ {gandalf}/mnt/btrfs-pool/snapshots/PACMAN.20150509_1
!!! /mnt/btrfs-backup-pool/BTRBK/PACMAN.20150509
!!! Target "/mnt/btrfs-backup-pool/BTRBK" aborted: Failed to send/receive subvolume

NOTE: Some errors occurred, which may result in missing backups!
Please check warning and error messages above.

I'm not really interested in resuming the transfer of all the previous backups: if I use 'resume_missing' as 'no', then btrbk doesn't attempt to resume the previous transfer. It shots another fresh snapshot on the laptop and it tries to transfer it on the sever. This fits well in my scenario but I would like that btrbk removes all the server-located incomplete snapshots: they are left on the server dirtying my server file-system. I can't even distinguish between completed and uncompleted backups.
Note: my bash-scripts manages this problem adding keywords like 'transfering' / 'synced' to the name of the snapshots.

I think it would be a great feature to be able to set custom backup names instead of it always being date created. It could be done with a command line option like -n or in the config file or both.

Hello, i'm trying to setup btrbk with send/recieve, i've simple setup with such config:

volume /mnt/vol1
subvolume  BACKUP
snapshot_preserve_daily     0
snapshot_preserve_weekly    0
snapshot_preserve_monthly   0
target send-receive         ssh://xx.xx.xx.xx/mnt/vol1/BACKUP/
target_preserve_daily       0
target_preserve_weekly      4
target_preserve_monthly     12

On dst server its ok:

/mnt/vol1/BACKUP
+++ /mnt/vol1//BACKUP.20150911_8
>>> {xx.xx.xx.xx}/mnt/vol1/BACKUP/BACKUP.20150911_8
--- {xx.xx.xx.xx}/mnt/vol1/BACKUP/BACKUP.20150911_7

On src server btrbk ignores snapshot_preserve_* settings and number of snapshots keeps growing:

• src server:

root@ol71a:/etc/btrbk# bls
ID 261 gen 94 top level 5 path BACKUP
ID 306 gen 83 top level 5 path BACKUP.20150911
ID 307 gen 84 top level 5 path BACKUP.20150911_1
ID 308 gen 85 top level 5 path BACKUP.20150911_2
ID 309 gen 86 top level 5 path BACKUP.20150911_3
ID 310 gen 87 top level 5 path BACKUP.20150911_4
ID 311 gen 89 top level 5 path BACKUP.20150911_5
ID 312 gen 90 top level 5 path BACKUP.20150911_6
ID 313 gen 92 top level 5 path BACKUP.20150911_7
ID 314 gen 94 top level 5 path BACKUP.20150911_8

• dst server

root@ol71b:/mnt/vol1/BACKUP# bls
ID 257 gen 225 top level 5 path BACKUP
ID 301 gen 193 top level 257 path BACKUP/BACKUP.20150911
ID 309 gen 222 top level 257 path BACKUP/BACKUP.20150911_8

Any advice, thanks.

Upon transferring several snapshots to my backup volume, I received this error message.

At subvol /mnt/subvolroot/.snapshots/rtfs.20150531

ERROR: process died unexpectedly (btrbk v0.19.0)
Please contact the author: Axel Burri <[email protected]>

Stack Trace:
----------------------------------------
Use of uninitialized value in string ne at /usr/bin/btrbk line 2240, <FILE> line 26.
 at /usr/bin/btrbk line 119, <FILE> line 26.
    main::__ANON__("Use of uninitialized value in string ne at /usr/bin/btrbk lin"...) called at /usr/bin/btrbk line 2237

Believing there was something wrong with that particular rtfs snapshot, I deleted the snapshot for 20150531 and re-ran it. The same error came up with the previous snapshot as well, despite being transferred successfully.

Hello, I have set up for BTRBK snapshot and backup a Ubuntu server. I'm not sure if I did it right. I believe that I have a flaw. In particular diie logic to create the Subvolumes was not easy to recognize. I have the home- and Rootpartionionen from Standard "/" after "/mnt/c" reverse hanging. Would I dri can restore? Does the snapshot without error even when the current root partition?

/etc/fstab:
UUID=..... /mnt/c/rootfs btrfs defaults,subvol=@ 0 1
UUID=.... /mnt/c/home btrfs defaults,subvol=@home 0 2
UUID=.... none swap sw 0 0
/dev/sda1 /mnt/c/server btrfs defaults 0 0

/etc/btrbk/backup.conf:
snapshot_preserve_daily 14
snapshot_preserve_weekly 0
snapshot_preserve_monthly 0
target_preserve_daily 20
target_preserve_weekly 10
target_preserve_monthly all

snapshot_dir btrbk_snapshots

volume /mnt/c
subvolume rootfs
target send-receive /mnt/c/server

subvolume home
target send-receive /mnt/c/server

I'm pretty sure i have everything setup right so here is the problem:

config file:

cat /etc/btrbk/btrbk.conf.example

snapshot_dir @BackuPs

snapshot_create_always yes

incremental no

preserve_day_of_week sunday

snapshot_preserve_daily all
snapshot_preserve_weekly 4
snapshot_preserve_monthly 1

target_preserve_daily all
target_preserve_weekly 4
target_preserve_monthly 1

btrfs_commit_delete after

receive_log no

volume /mnt/btrfsroot
subvolume @Home
target send-receive /media/Storage3/@HomeSnaps

btrfs subvolume list /media/Storage3/@HomeSnaps
ID 5683 gen 51069 top level 5 path @HomeSnaps
btrfs subvolume show /media/Storage3/@HomeSnaps
/media/Storage3/@HomeSnaps
Name: @HomeSnaps
uuid: 1bb1ed43-5c5d-7542-9a6e-87b36eae5f48
Parent uuid: -
Creation time: 2015-03-19 17:43:48
Object ID: 5683
Generation (Gen): 51069
Gen at creation: 51069
Parent: 5
Top Level: 5
Flags: -
Snapshot(s):

btrbk -c /etc/btrbk/btrbk.conf.example dryrun
WARNING: Skipping target: Failed to read btrfs subvolume list for "/media/Storage3/@HomeSnaps"

WARNING: Skipping cleanup of snapshots for subvolume "/mnt/btrfsroot/@Home", as at least one target aborted earlier

Backup Summary (btrbk command line client, version 0.13)

Date:   Fri Mar 20 12:24:14 2015
Config: /etc/btrbk/btrbk.conf.example

/mnt/btrfsroot/@Home
+++ /mnt/btrfsroot/@Backups/@Home.20150320
!!! Target "/media/Storage3/@HomeSnaps" aborted: Failed to read btrfs subvolume list for "/media/Storage3/@HomeSnaps"

NOTE: Some errors occurred, which may result in missing backups!
Please check warning and error messages above.

NOTE: Dryrun was active, none of the operations above were actually executed!

i've tried creating the subvol with and without @. At first i was creating it at /media/Storage3/Backup/HomeSnaps but even moving it to the root of the FS i have the same problem. I am using arch linux with kernel 3.19.1-1-ck and btrfs-progs v3.19. Is it something i am doing wrong or a bug?

I'm sure I'm missing something obvious, but I've tried everything and haven't been able to figure it out.

Running btrbk with the backup volume connected works as expected.
Disconnecting the backup volume and running again, I get this output:

WARNING: Skipping target "/mnt/backups/Laptop/btrbk": Failed to fetch subvolume detail
WARNING: Skipping target "/mnt/backups/Laptop/btrbk": Failed to fetch subvolume detail
WARNING: Skipping target "/mnt/backups/Laptop/btrbk": Failed to fetch subvolume detail
WARNING: Skipping target "/mnt/backups/Laptop/btrbk": Failed to fetch subvolume detail

ERROR: process died unexpectedly (btrbk v0.17.0)
Please contact the author: Axel Burri <[email protected]>

Stack Trace:
----------------------------------------
Died at /usr/bin/btrbk line 1725, <FILE> line 97.
 at /usr/bin/btrbk line 103, <FILE> line 97.
    main::__ANON__("Died at /usr/bin/btrbk line 1725, <FILE> line 97.\x{a}") called at /usr/bin/btrbk line 1725

My config file is as follows:

snapshot_dir               .snapshots
incremental                yes
snapshot_create_always     yes
resume_missing             yes
preserve_day_of_week       sunday
snapshot_preserve_daily    14
snapshot_preserve_weekly   4
snapshot_preserve_monthly  0
target_preserve_daily      40
target_preserve_weekly     32
target_preserve_monthly    all
btrfs_commit_delete        after
btrfs_progs_compat         no

volume /mnt/subvolroot
  subvolume rtfs
    target send-receive /mnt/backups/Laptop/btrbk

  subvolume boot
    target send-receive /mnt/backups/Laptop/btrbk

  subvolume home
    target send-receive /mnt/backups/Laptop/btrbk

  subvolume etc
    target send-receive /mnt/backups/Laptop/btrbk

I first noticed the issue with version 0.17.0.

I'd like to seed the remote backup target with snapshot(s) delivered on a USB drive. What I'm thinking is to run 'btrbk' with a temporary target of the USB mount point. I can then move the USB drive to the target server and 'btrfs send | btrfs receive' into the backup volume. It's not clear to me if this will cause the parent relationships to be preserved and incremental backups to happen from then on. Does this work? Is there another way to accomplish the same thing if not?

When I tried a dry run of this by send/receiving the 'root' snapshot to the USB drive and then running btrbk resume targeting it, it seemed to be copying the entire thing again.

I am trying to make snapshots of entire volume. Can the config file be configured for use without subvolumes? Commenting out the subvolume line leads to errors.

Thanks in advance

Hi!
First of all, thank you for this wonderful program ! Very useful and handy !

I open this issue because I've some concerns about security when btrbk operates over ssh with the ssh_filter_btrbk script.

Let me explain you this:

• I have multiple source computers to be backuped to a target server
• If one computer gets compromised, the attacker cannot do whatever he wants thanks to the ssh_filter_btrbk script. But he can access dangerous btrfs command system-wide (root permissions).

For example:

1. Attacker can destroy other computers backups or system partitions (btrfs subvolume delete accepted for all subvolumes)
2. He can extract data from other backups or system (btrfs sub snap and btrfs send)

Maybe I'm paranoid and this kind of attack is highly unlikely but, I think It would be good to restrict the path in which the btrfs command can proceed and if out of this path: reject_and_die !

If you think this is relevant and needs more attention, I've made a modification of the ssh_filter_script to take into consideration the working path.
(as an example, my authorized_keys looks like this :
from="192.168.0.1",command="/usr/local/bin/ssh_filter_btrbk.sh --log --sudo --restrict-path /btrfs/pool_1/backups/pc1")

I'm no programmer, and even less a bash expert... so my modifications are probably bad but
tell me what you think of all this !

• PS1: it uses /bin/bash, I don't know of to do this regular expression (with =~ operator) valid for all shells
• PS2: the regex is awful, but seems to be working (no regex expert neither ...):
  • complicated because we can put multiple subvolume behind btrfs sub delete
  • So different subvolumes are separated with a space (since it's not authorized in the subvolumes name in btrbk, it souldn't be a problem) and the regex verifies that each space separated subvolume begins with the restricted path. And with the reject of ../ in the path, we should be good ?
• PS3: implemented use_sudo because I'm not connecting to remote root user and btrfs command needs these permissions

#!/bin/bash

set -e
set -u

export PATH=/sbin:/bin:/usr/sbin:/usr/bin

enable_log=
restrict_path=
use_sudo=


while [ "$#" -ge 1 ]
do
key="$1"

case $key in
    -l|--log)
    enable_log=1
    ;;

    -p|--restrict-path)
    restrict_path="$2"
    shift # past argument
    ;;

    -s|--sudo)
    use_sudo="sudo"
    ;;

    *)
    ;;
esac
shift

done



log_cmd()
{
    if [ -n "$enable_log" ]; then
        logger -p $1 -t ssh_filter_btrbk.sh "$2 (Name: ${LOGNAME:-<unknown>}; Remote: ${SSH_CLIENT:-<unknown>}; Restrict-path: $restrict_path; $use_sudo): $SSH_ORIGINAL_COMMAND"
    fi
}

reject_and_die()
{
    log_cmd "auth.err" "btrbk REJECT"
    /bin/echo "ERROR: ssh command rejected" 1>&2
    exit 1
}

run_cmd()
{
    log_cmd "auth.info" "btrbk ACCEPT"
    $use_sudo $SSH_ORIGINAL_COMMAND
}


check_path()
{
    if [ -n "$restrict_path" ]; then
    regex="^btrfs[-a-zA-Z0-9 ]* ([ ]*$restrict_path[^ ]*)*[ ]*$"

        if [[ $SSH_ORIGINAL_COMMAND =~ $regex ]]; then
            run_cmd
        else
            reject_and_die
        fi
    else
    run_cmd
    fi
}


case "$SSH_ORIGINAL_COMMAND" in
    *\$*) reject_and_die ;;
    *\&*) reject_and_die ;;
    *\(*) reject_and_die ;;
    *\{*) reject_and_die ;;
    *\;*) reject_and_die ;;
    *\<*) reject_and_die ;;
    *\>*) reject_and_die ;;
    *\`*) reject_and_die ;;
    *\|*) reject_and_die ;;
    *"../"*) reject_and_die;;       #prevent directory traversal
    btrfs\ subvolume\ show\ *)      check_path ;; #run_cmd ;;   # mandatory
    btrfs\ subvolume\ list\ *)      check_path ;; #run_cmd ;;   # mandatory
    btrfs\ subvolume\ snapshot\ *)  check_path ;; #run_cmd ;;   # mandatory if this host is backup source
    btrfs\ send\ *)                 check_path ;; #run_cmd ;;   # mandatory if this host is backup source
    btrfs\ receive\ *)              check_path ;; #run_cmd ;;   # mandatory if this host is backup target
    btrfs\ subvolume\ delete\ *)    check_path ;; #run_cmd ;;   # mandatory if scheduling is active
    btrfs\ subvolume\ find-new\ *)  check_path ;; #run_cmd ;;   # needed for "btrbk diff"
    btrfs\ filesystem\ usage\ *)    check_path ;; #run_cmd ;;   # needed for "btrbk info"
    *) reject_and_die ;;
esac

Have an option flag that pipes the transfer through the pv utility. pv monitors data going through a pipe, alowing things like counting the bytes transfered.

Should be something like

btrfs send $SRC | pv | btrfs receive $DEST

Note that pv has no way of knowing the total amount of data that needs to be tranfered since send/receive doesn't know, it just counts the bytes. (If I am mistakened and there is a way to determine the side of a snapshot then there is a flag to add that.)