nachoparker / btrfs-sync Goto Github PK

View Code? Open in Web Editor NEW

81.0 5.0 20.0 147 KB

Smart and easy sync of BTRFS snapshots, locally or through SSH

License: GNU General Public License v3.0

Shell 100.00%

btrfs synchronization

btrfs-sync's Introduction

btrfs-sync

Smart and easy sync of BTRFS snapshots, locally or through SSH.

Can be used in conjunction with btrfs-snp

# btrfs-sync --verbose --delete /home/user/.snapshots user@server:/media/USBdrive/home-snapshots
* Skip existing '/home/user/.snapshots/monthly_2018-01-09_200102'
* Skip existing '/home/user/.snapshots/monthly_2018-02-08_200102'
* Skip existing '/home/user/.snapshots/weekly_2018-02-09_140102'
* Skip existing '/home/user/.snapshots/weekly_2018-02-16_150102'
* Skip existing '/home/user/.snapshots/weekly_2018-02-23_150102'
* Skip existing '/home/user/.snapshots/weekly_2018-03-02_180102'
* Skip existing '/home/user/.snapshots/daily_2018-03-03_000101'
* Skip existing '/home/user/.snapshots/daily_2018-03-04_080101'
* Skip existing '/home/user/.snapshots/daily_2018-03-05_100102'
* Skip existing '/home/user/.snapshots/daily_2018-03-06_100102'
* Skip existing '/home/user/.snapshots/daily_2018-03-07_110102'
* Synchronizing '/home/user/.snapshots/hourly_2018-03-08_090101' using seed '.snapshots/hourly_2018-03-07_090101'...
time elapsed [0:00:24] | rate [11.1MiB/s] | total size [ 132MiB]
* Synchronizing '/home/user/.snapshots/hourly_2018-03-08_100101' using seed '.snapshots/hourly_2018-03-09_090101'...
time elapsed [0:01:05] | rate [11.1MiB/s] | total size [ 275MiB]
* Deleting non existent snapshots...
Delete subvolume (no-commit): '/media/USBdrive/home-snapshots/hourly_2018-03-08_090101'
Delete subvolume (no-commit): '/media/USBdrive/home-snapshots/hourly_2018-03-08_100101'

Features

Simple syntax
Progress indication
Support for xz or pbzip2 compression in order to save bandwidth.
Retention policy
Automatic incremental synchronization
Cron friendly

Usage

The syntax is similar to that of scp

Usage:
  btrfs-sync [options] <src> [<src>...] [[user@]host:]<dir>

  -k|--keep NUM     keep only last <NUM> sync'ed snapshots
  -d|--delete       delete snapshots in <dst> that don't exist in <src>
  -z|--xz           use xz     compression. Saves bandwidth, but uses one CPU
  -Z|--pbzip2       use pbzip2 compression. Saves bandwidth, but uses all CPUs
  -q|--quiet        don't display progress
  -v|--verbose      display more information
  -h|--help         show usage

<src> can either be a single snapshot, or a folder containing snapshots
<user> requires privileged permissions at <host> for the 'btrfs' command

Examples

Manual

Synchronize snapshots of home to a USB drive

# btrfs-sync /home/user/.snapshots /media/USBdrive/home-snapshots

Synchronize snapshots of home to a USB drive in another machine

# btrfs-sync /home/user/.snapshots user@server:/media/USBdrive/home-snapshots

Synchronize one snapshot of home to a USB drive in another machine

# btrfs-sync /home/user/.snapshots/monthly_2018-02-08_200102 user@server:/media/USBdrive/home-snapshots

Synchronize only monthly snapshots of home to a USB drive in another machine

# btrfs-sync /home/user/.snapshots/monthly_* user@server:/media/USBdrive/home-snapshots

Cron

Daily synchronization over the internet, keep only last 50

cat > /etc/cron.daily/btrfs-sync <<EOF
#!/bin/bash
/usr/local/sbin/btrfs-sync --quiet --keep 50 --xz /home user@host:/path/to/snaps
EOF
chmod +x /etc/cron.daily/btrfs-sync

Daily synchronization in LAN, mirror snapshot directory

cat > /etc/cron.daily/btrfs-sync <<EOF
#!/bin/bash
/usr/local/sbin/btrfs-sync --quiet --delete /home user@host:/path/to/snaps
EOF
chmod +x /etc/cron.daily/btrfs-sync

More at ownyourbits.com

btrfs-sync's People

Contributors

Stargazers

Watchers

Forkers

kenanpelit juripero dalmat cl-jeremy iiiw mikhailnov pagalba-com jan--f jenszo ryan-jq blackfell rababerladuseladim bhzhu203 kovalewvladimir traxanos dcflachs guillaumeseren celluloid-demon jeansibelius

btrfs-sync's Issues

shellcheck errors and warnings

Great little tool, but could you make it shellcheck-clean?

$ shellcheck btrfs-sync

In btrfs-sync line 80:
[[ $? -ne 0 ]] && { echo "error parsing arguments"; exit 1; }
   ^-- SC2181 (style): Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.


In btrfs-sync line 98:
DST="${@: -1}"
    ^--------^ SC2124 (warning): Assigning an array to a string! Assign as array, or use * instead of @ to concatenate.


In btrfs-sync line 101:
[[ "$SRC" =~ : ]] && {
    ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 102:
  NET_SRC="$( sed 's|:.*||' <<<"$SRC" )"
              ^---------------------^ SC2001 (style): See if you can use ${variable//search/replace} instead.
                                ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 103:
  SRC="$( sed 's|.*:||' <<<"$SRC" )"
  ^-^ SC2178 (warning): Variable was used as an array but is now assigned a string.
          ^---------------------^ SC2001 (style): See if you can use ${variable//search/replace} instead.
                            ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 107:
[[ "$SSH_SRC" != "" ]] && SRC_CMD=( ${SSH_SRC[@]} ) || SRC_CMD=( eval )
    ^------^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                    ^-----------^ SC2206 (warning): Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.


In btrfs-sync line 108:
${SRC_CMD[@]} test -x "$SRC" &>/dev/null || {
^-----------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
                       ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 109:
  [[ "$SSH_SRC" != "" ]] && echo "SSH access error to $NET_SRC. Do you have passwordless login setup, and adequate permissions for $SRC?"
      ^------^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                                                                                                                   ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 110:
  [[ "$SSH_SRC" == "" ]] && echo "Access error. Do you have adequate permissions for $SRC?"
      ^------^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                                                                     ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 116:
  NET="$( sed 's|:.*||' <<<"$DST" )"
          ^---------------------^ SC2001 (style): See if you can use ${variable//search/replace} instead.


In btrfs-sync line 117:
  DST="$( sed 's|.*:||' <<<"$DST" )"
          ^---------------------^ SC2001 (style): See if you can use ${variable//search/replace} instead.


In btrfs-sync line 120:
[[ "$SSH" != "" ]] && DST_CMD=( ${SSH[@]} ) || DST_CMD=( eval )
    ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                ^-------^ SC2206 (warning): Quote to prevent word splitting/globbing, or split robustly with mapfile or read -a.


In btrfs-sync line 121:
${DST_CMD[@]} test -x "$DST" &>/dev/null || {
^-----------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.


In btrfs-sync line 122:
  [[ "$SSH" != "" ]] && echo "SSH access error to $NET. Do you have passwordless login setup, and adequate permissions for $DST?"
      ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 123:
  [[ "$SSH" == "" ]] && echo "Access error. Do you have adequate permissions for $DST?"
      ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 135:
${DST_CMD[@]} "pgrep -f btrfs\ receive &>/dev/null" && { echo "btrfs-sync already running at destination"; exit 1; }
^-----------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.


In btrfs-sync line 139:
while read entry; do SRCS+=( "$entry" ); done < <(
      ^--^ SC2162 (info): read without -r will mangle backslashes.


In btrfs-sync line 141:
    for s in "${SRC[@]}"; do
              ^-------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
              ^-------^ SC2145 (error): Argument mixes string and array. Use * or separate argument.
              ^-------^ SC2027 (warning): The surrounding quotes actually unquote this. Remove or escape them.


In btrfs-sync line 157:
    "${SRC_CMD[@]}" type pbzip2 &>/dev/null && \
                                            ^-- SC2015 (info): Note that A && B || C is not if-then-else. C may run when A is true.


In btrfs-sync line 160:
      ZIP=xz PIZ=unxz
             ^-^ SC2178 (warning): Variable was used as an array but is now assigned a string.


In btrfs-sync line 178:
  while read entry; do
        ^--^ SC2162 (info): read without -r will mangle backslashes.


In btrfs-sync line 179:
    DST_UUIDS+=( "$( sed 's=|.*==' <<<"$entry" )" )
                     ^-----------------------^ SC2001 (style): See if you can use ${variable//search/replace} instead.


In btrfs-sync line 180:
    DSTS+=(      "$( sed 's=.*|==' <<<"$entry" )" )
                     ^-----------------------^ SC2001 (style): See if you can use ${variable//search/replace} instead.


In btrfs-sync line 193:
  local SRC="$1"
        ^-^ SC2178 (warning): Variable was used as an array but is now assigned a string.


In btrfs-sync line 200:
    local LIST="$( "${SRC_CMD[@]}" sudo btrfs subvolume list -su "$SRC" )"
          ^--^ SC2155 (warning): Declare and assign separately to avoid masking return values.
                                                                  ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 204:
        local PATH_=$( awk "{ if ( \$14 == \"$id\" ) print \$16       }" <<<"$LIST" )
              ^---^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In btrfs-sync line 205:
        local DATE=$(  awk "{ if ( \$14 == \"$id\" ) print \$11, \$12 }" <<<"$LIST" )
              ^--^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In btrfs-sync line 210:
            local PATHS=( $( "${SRC_CMD[@]}" sudo btrfs su list -u "$SRC" | awk '{ print $11 }' ) )
                          ^-- SC2207 (warning): Prefer mapfile or read -a to split command output (or quote to avoid splitting).
                                                                    ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 212:
              SHOWP="$( "${SRC_CMD[@]}" sudo btrfs su sh "$( dirname "$SRC" )/$( basename "$p" )" 2>/dev/null )"
              ^---^ SC2178 (warning): Variable was used as an array but is now assigned a string.
                                                                      ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 213:
              RXIDP="$( grep 'Received UUID' <<<"$SHOWP" | awk '{ print $3     }' )"
              ^---^ SC2178 (warning): Variable was used as an array but is now assigned a string.
                                                 ^----^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 214:
              DATEP="$( grep 'Creation time' <<<"$SHOWP" | awk '{ print $3, $4 }' )"
              ^---^ SC2178 (warning): Variable was used as an array but is now assigned a string.
                                                 ^----^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 215:
              [[ "$RXIDP" == "" ]] && continue
                  ^----^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 216:
              PATH_RXID["$RXIDP"]="$p"
                         ^----^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 217:
              DATE_RXID["$RXIDP"]="$DATEP"
                         ^----^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                   ^----^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 225:
        [[ "$PATH_" == "" ]] || [[ "$PATH_" == "$( basename "$SRC" )" ]] && continue
                                                             ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 227:
        local SECS=$( date -d "$DATE" +"%s" )
              ^--^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In btrfs-sync line 235:
  local SHOW="$( "${SRC_CMD[@]}" sudo btrfs subvolume show "$SRC" )"
        ^--^ SC2155 (warning): Declare and assign separately to avoid masking return values.
                                                            ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 237:
  local SRC_UUID="$( grep 'UUID:' <<<"$SHOW" | head -1 | awk '{ print $2 }' )"
        ^------^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In btrfs-sync line 240:
  local SRC_RXID="$( grep 'Received UUID' <<<"$SHOW"   | awk '{ print $3 }' )"
        ^------^ SC2155 (warning): Declare and assign separately to avoid masking return values.


In btrfs-sync line 249:
  local SRC="$1"
        ^-^ SC2178 (warning): Variable was used as an array but is now assigned a string.


In btrfs-sync line 250:
  "${SRC_CMD[@]}" test -d "$SRC" || return
                           ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 252:
  exists_at_dst "$SRC" && { echov "* Skip existing '$SRC'"; return 0; }
                 ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                                    ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 254:
  choose_seed "$SRC"  # sets SEED
               ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 258:
    local SEED_PATH="$( dirname "$SRC" )/$( basename $SEED )"
          ^-------^ SC2155 (warning): Declare and assign separately to avoid masking return values.
                                 ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.
                                                     ^---^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    local SEED_PATH="$( dirname "$SRC" )/$( basename "$SEED" )"


In btrfs-sync line 266:
  [[ "$SEED_ARG" != "" ]] && echov -n " using seed '$SEED'"
      ^-------^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 270:
  sudo btrfs send -q ${SEED_ARG[@]} "$SRC" \
                     ^------------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
                                     ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 273:
    | "${DST_CMD[@]}" "${PIZ[@]} | sudo btrfs receive \"$DST\" 2>&1 |(grep -v -e'^At subvol ' -e'^At snapshot '||true)" \
                       ^-------^ SC2145 (error): Argument mixes string and array. Use * or separate argument.


In btrfs-sync line 275:
      "${DST_CMD[@]}" sudo btrfs subvolume delete "$DST"/"$( basename "$SRC" )" 2>/dev/null
                                                                       ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 280:
  DSTS+=("$DST/$( basename "$SRC" )")
                            ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 282:
  SEED_NEXT="$SRC"
             ^--^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 309:
  ${DST_CMD[@]} sudo btrfs subvolume delete "${PRUNE_LIST[@]}" $SILENT
  ^-----------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
                                                               ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
  ${DST_CMD[@]} sudo btrfs subvolume delete "${PRUNE_LIST[@]}" "$SILENT"


In btrfs-sync line 316:
      [[ "$( basename $src )" == "$( basename $dst )" ]] && { FOUND=1; break; }
                      ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.
                                              ^--^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
      [[ "$( basename "$src" )" == "$( basename "$dst" )" ]] && { FOUND=1; break; }


In btrfs-sync line 320:
[[ "$DEL_LIST" != "" ]] && \
    ^-------^ SC2128 (warning): Expanding an array without an index only gives the first element.


In btrfs-sync line 322:
  ${DST_CMD[@]} sudo btrfs subvolume delete "${DEL_LIST[@]}" $SILENT
  ^-----------^ SC2068 (error): Double quote array expansions to avoid re-splitting elements.
                                                             ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
  ${DST_CMD[@]} sudo btrfs subvolume delete "${DEL_LIST[@]}" "$SILENT"

For more information:
  https://www.shellcheck.net/wiki/SC2068 -- Double quote array expansions to ...
  https://www.shellcheck.net/wiki/SC2145 -- Argument mixes string and array. ...
  https://www.shellcheck.net/wiki/SC2027 -- The surrounding quotes actually u...

src check fails

I'm getting a "Host key verification failed." from somewhere in the loop on line 138.

manually running the output from echov "${SRC_CMD[@]}" inserted after line 138 makes a successful ssh connection with no host key error.

sudo btrfs-sync [email protected]:/mnt/bulk /media/jake/Toshiba2TB

Multiple execution at the same time

Hi!

I would like to backup several clients at the same time. This can be useful, for example, if the bandwidth and CPU of the clients is limited, but the backup server and bandwidth is not utilized at all. So I could significantly increase the speed until all snapshots of all clients are transferred.

btrfs-sync/btrfs-sync

Lines 131 to 132 in 6d3c3f0

 pgrep -F /run/btrfs-sync.pid &>/dev/null && { echo "$BIN is already running"; exit 1; } 

 echo $$ > /run/btrfs-sync.pid

In the bash script I found the above line which prevents simultaneous running.

So my question is: Is this a precautionary measure to protect the user, or are there technical reasons why the script should only run once at the same time?

Have a nice weekend
Markus

Picking the wrong seed subvolume

Hi, thank you for the tool.

When I tried this script on my pool (which is managed manually up till now), the script seems to be picking wrong seed subvolume for some reason:

Here is the snapshots from my source:

ID 2298 gen 4173316 cgen 3216800 top level 5 otime 2019-12-19 15:46:11 uuid 2c4e4d62-7d03-df46-9202-df7d73e23c11 path backups/vaio/home-2019-11-06T12:01:07+09:00
ID 2299 gen 4173307 cgen 3216820 top level 5 otime 2019-12-19 15:53:43 uuid 23be286f-a2f5-c449-a228-67c5d51ed0c2 path backups/vaio/root-2019-11-06T12:01:01+09:00
ID 2309 gen 4173307 cgen 3637696 top level 5 otime 2020-01-14 10:32:37 uuid 222da099-8c76-dd4d-ad0e-6fc6202e1041 path backups/vaio/root-2020-01-14T10:29:47+09:00
ID 2311 gen 4173316 cgen 3637736 top level 5 otime 2020-01-14 10:50:31 uuid 587ad756-4870-0e44-b55b-012be6e6198f path backups/vaio/home-2020-01-14T10:48:47+09:00
ID 2312 gen 4173316 cgen 3643276 top level 5 otime 2020-01-15 13:31:21 uuid c455e809-5c91-d844-a4f3-551240da2b28 path backups/vaio/home-2020-01-15T13:29:36+09:00
ID 2313 gen 4173307 cgen 3643281 top level 5 otime 2020-01-15 13:32:03 uuid f4b4453a-c819-3042-819b-c87bfbb7a439 path backups/vaio/root-2020-01-15T13:29:40+09:00
ID 2382 gen 4176937 cgen 4171879 top level 5 otime 2020-03-11 10:44:42 uuid b6209c2d-70ad-3e49-9fa6-96772a8885da path backups/vaio/home-2020-03-11T10:41:16+09:00
ID 2383 gen 4176895 cgen 4171925 top level 5 otime 2020-03-11 11:02:51 uuid b6bc19cc-030d-3f44-8f42-6e882417cb18 path backups/vaio/root-2020-03-11T10:41:11+09:00
ID 2397 gen 4176922 cgen 4176895 top level 5 otime 2020-03-12 14:36:21 uuid c0e1a432-7955-9d42-a579-0436c00df7f5 path backups/vaio/root-2020-03-12T14:34:08+09:00
ID 2398 gen 4176940 cgen 4176937 top level 5 otime 2020-03-12 14:54:18 uuid 2588fc92-f38d-4049-927a-3a53e900e8a6 path backups/vaio/home-2020-03-12T14:34:04+09:00

Here is the snapshots in my target dir:

ID 1290 gen 12632 cgen 1497 top level 5 otime 2019-10-24 15:00:57 uuid 1e18bd16-5a36-ec4f-9134-bea5c7195ee1 path vaio/root-2019-10-24T14:56:54+09:00
ID 1292 gen 13044 cgen 1500 top level 5 otime 2019-10-24 15:02:16 uuid 466412a6-b05e-b645-bd8d-c6df3684d061 path vaio/home-2019-10-24T14:57:00+09:00
ID 2213 gen 12781 cgen 12632 top level 5 otime 2019-11-06 09:40:58 uuid 9d065470-6b12-0e4b-a535-18dd378f6f63 path vaio/root-2019-10-30T16:22:46+09:00
ID 2216 gen 12821 cgen 12637 top level 5 otime 2019-11-06 09:43:09 uuid c977198b-621c-9643-af9b-cce23dd2c1a1 path vaio/home-2019-10-30T16:22:40+09:00
ID 2259 gen 13643 cgen 12781 top level 5 otime 2019-11-06 12:05:58 uuid 5db41aeb-95b7-7040-bb7f-8fbdbd650952 path vaio/root-2019-11-06T12:01:01+09:00
ID 2271 gen 13540 cgen 12821 top level 5 otime 2019-11-06 13:24:47 uuid ae349fc8-3a91-cc43-b579-45556280fa64 path vaio/home-2019-11-06T12:01:07+09:00
ID 2288 gen 13050 cgen 13044 top level 5 otime 2019-12-19 13:44:37 uuid 1f8e2e03-76ef-7b48-83a9-c569262d00ab path vaio/home-2019-09-24T19:39:25+09:00
ID 2467 gen 13563 cgen 13540 top level 5 otime 2020-03-24 11:35:15 uuid b4b527d7-d2f3-b649-8c61-90368a1f2d44 path vaio/home-2020-01-14T10:48:47+09:00
ID 2476 gen 13567 cgen 13563 top level 5 otime 2020-03-24 11:49:21 uuid 85b4f7aa-66a1-cd45-887f-68e37a493157 path vaio/home-2020-01-15T13:29:36+09:00
ID 2477 gen 13639 cgen 13567 top level 5 otime 2020-03-24 11:50:37 uuid 4784a525-d81b-f042-8c11-42202820f3c1 path vaio/home-2020-03-11T10:41:16+09:00
ID 2506 gen 13757 cgen 13639 top level 5 otime 2020-03-24 12:34:56 uuid 973cf412-e1b1-2541-8698-272a1d435ab8 path vaio/home-2020-03-12T14:34:04+09:00
ID 2507 gen 13679 cgen 13643 top level 5 otime 2020-03-24 12:52:48 uuid adba47bc-e74e-c34d-8a15-9d7ca9e3463f path vaio/root-2020-01-14T10:29:47+09:00
ID 2517 gen 13713 cgen 13679 top level 5 otime 2020-03-24 13:21:53 uuid fed8cb25-fdbf-5f40-b564-c399afd2e80e path vaio/root-2020-01-15T13:29:40+09:00
ID 2527 gen 13753 cgen 13713 top level 5 otime 2020-03-24 13:46:39 uuid d5b32711-6e1d-4242-afdd-c8ce25612086 path vaio/root-2020-03-11T10:41:11+09:00

And the output of btrfs-sync -v /pool/backups/vaio/ /mnt/yuyuko/vaio/ is:

* Check source
* Check destination
* Skip existing '/pool/backups/vaio/home-2019-11-06T12:01:07+09:00'
* Skip existing '/pool/backups/vaio/root-2019-11-06T12:01:01+09:00'
* Skip existing '/pool/backups/vaio/root-2020-01-14T10:29:47+09:00'
* Skip existing '/pool/backups/vaio/home-2020-01-14T10:48:47+09:00'
* Skip existing '/pool/backups/vaio/home-2020-01-15T13:29:36+09:00'
* Skip existing '/pool/backups/vaio/root-2020-01-15T13:29:40+09:00'
* Skip existing '/pool/backups/vaio/home-2020-03-11T10:41:16+09:00'
* Skip existing '/pool/backups/vaio/root-2020-03-11T10:41:11+09:00'
* Synchronizing '/pool/backups/vaio/root-2020-03-12T14:34:08+09:00' using seed 'backups/vaio/home-2020-03-12T14:34:04+09:00'...
ERROR: chown o257-2175045-0 failed: No such file or directory
time elapsed [0:00:58] | rate [1.26KiB/s] | total size [73.8KiB]
Delete subvolume (no-commit): '/mnt/yuyuko/vaio/root-2020-03-12T14:34:08+09:00'
* Retrying '/pool/backups/vaio/root-2020-03-12T14:34:08+09:00'...
* Synchronizing '/pool/backups/vaio/root-2020-03-12T14:34:08+09:00' using seed 'backups/vaio/home-2020-03-12T14:34:04+09:00'...
ERROR: chown o257-2175045-0 failed: No such file or directory
time elapsed [0:00:32] | rate [2.23KiB/s] | total size [73.0KiB]
Delete subvolume (no-commit): '/mnt/yuyuko/vaio/root-2020-03-12T14:34:08+09:00'
* Retrying '/pool/backups/vaio/root-2020-03-12T14:34:08+09:00'...
* Synchronizing '/pool/backups/vaio/root-2020-03-12T14:34:08+09:00' using seed 'backups/vaio/home-2020-03-12T14:34:04+09:00'...
ERROR: chown o257-2175045-0 failed: No such file or directory
time elapsed [0:00:33] | rate [2.21KiB/s] | total size [73.0KiB]
Delete subvolume (no-commit): '/mnt/yuyuko/vaio/root-2020-03-12T14:34:08+09:00'
Abort

btrfs-sync decided that home-2020-03-12T14:34:04+09:00 should be the seed for root-2020-03-12T14:34:08+09:00 , this seems to be wrong to me, because if I show the details of root-2020-03-12T14:34:08+09:00:

backups/vaio/root-2020-03-12T14:34:08+09:00
        Name:                   root-2020-03-12T14:34:08+09:00
        UUID:                   c0e1a432-7955-9d42-a579-0436c00df7f5
        Parent UUID:            b6bc19cc-030d-3f44-8f42-6e882417cb18
        Received UUID:          b4a3dba1-9705-3a4c-9ecf-7421043dd64f
        Creation time:          2020-03-12 14:36:21 +0900
        Subvolume ID:           2397
        Generation:             4176922
        Gen at creation:        4176895
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):

It's parent should be b6bc19cc-030d-3f44-8f42-6e882417cb18, which is root-2020-03-11T10:41:11+09:00:

backups/vaio/root-2020-03-11T10:41:11+09:00
        Name:                   root-2020-03-11T10:41:11+09:00
        UUID:                   b6bc19cc-030d-3f44-8f42-6e882417cb18
        Parent UUID:            f4b4453a-c819-3042-819b-c87bfbb7a439
        Received UUID:          334d2b63-c592-aa47-9c79-99f775140950
        Creation time:          2020-03-11 11:02:51 +0900
        Subvolume ID:           2383
        Generation:             4176895
        Gen at creation:        4171925
        Parent ID:              5
        Top level ID:           5
        Flags:                  readonly
        Snapshot(s):
                                backups/vaio/root-2020-03-12T14:34:08+09:00

Is there anything wrong with my usage of the script?

Is there an option for two-way sync?

I would like to synchronize some btrfs-subvolumes in two-way, meaning I have Computer A and Computer B both having "shared" data stored on btrfs-subvolumes. I don't want to sync one-way as on Computer B there may be changes I also want on Computer A instead of overriding them (and vice-versa). Is there some option or any other solution to handle this?

"Remote user requires sudo privileges" error

Hi,

this script used to work like a charm for me, but the latest commit keeps failing with
Remote user requires sudo privileges

I followed Nachos instructions (again, just to be sure).

When on my sending machine I call
ssh -p 22 -o ServerAliveInterval=5 -o ConnectTimeout=1 -o BatchMode=yes btrfs@<dest-host> sudo echo
which is just what the following line does

btrfs-sync/btrfs-sync

Line 139 in 5647f14

 ${DST_CMD[@]} "sudo echo &> /dev/null" || { echo "Remote user requires sudo privileges"; exit 1; } 

I get


We trust you have received the usual lecture from the local System
Administrator. It usually boils down to these three things:

    #1) Respect the privacy of others.
    #2) Think before you type.
    #3) With great power comes great responsibility.

sudo: no tty present and no askpass program specified

On my receiving machine:

sudo cat /etc/sudoers.d/90_btrfs-sync
btrfs ALL=(root:nobody) NOPASSWD:NOEXEC: /bin/btrfs

If I comment the following lines, snapshot syncing works again:

btrfs-sync/btrfs-sync

Lines 135 to 139 in 5647f14

 ## Make sure, the user's remote shell is bash (not sh) 

 ${DST_CMD[@]} "echo \$0 | grep bash > /dev/null" || { echo "Remote user's shell not bash. Shells other than bash are not supported at the moment"; exit 1; } 

 ## Making sure, the remote user has sudo privileges 

 ${DST_CMD[@]} "sudo echo &> /dev/null" || { echo "Remote user requires sudo privileges"; exit 1; }

Snapshots not synced

Hi!
I am using Nextcloudpi on a plain vanilla Debian installation:

root@obelix:/media/nextcloud/ncp-snapshots# uname -a
Linux obelix 4.19.0-21-amd64 #1 SMP Debian 4.19.249-2 (2022-06-30) x86_64 GNU/Linux

I create snapshots on the data drive and would like to sync those snapshots to another drive; but unfortunately all new snapshots won't be synced to the backup drive because btrfs-sync tells me those snaphots are already in the destination:

root@obelix:/media/backup/ncp-snapshots# btrfs-sync -vd  "/media/USBdrive/ncp-snapshots" "/media/backup/ncp-snapshots"
* Check source
* Check destination
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2021-09-23_201704'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2021-10-23_201704'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2021-11-22_191704'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2021-12-22_191705'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-01-21_191704'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-02-20_191704'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-03-22_191704'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-04-21_201703'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-05-21_201703'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-06-20_201703'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-07-20_201702'
* Skip existing '/media/USBdrive/ncp-snapshots/manual_2022-07-30_114309'
* Skip existing '/media/USBdrive/ncp-snapshots/weekly_2022-08-08_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/manual_2022-08-10_220354'
* Skip existing '/media/USBdrive/ncp-snapshots/weekly_2022-08-15_221721'
* Skip existing '/media/USBdrive/ncp-snapshots/monthly_2022-08-19_201703'
* Skip existing '/media/USBdrive/ncp-snapshots/manual_2022-08-22_201833'
* Skip existing '/media/USBdrive/ncp-snapshots/weekly_2022-08-22_231703'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-23_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/manual_2022-08-24_154736'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-24_221702'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-25_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-26_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-27_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-28_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-29_221702'
* Skip existing '/media/USBdrive/ncp-snapshots/daily_2022-08-29_221703'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-29_231702'
* Skip existing '/media/USBdrive/ncp-snapshots/weekly_2022-08-29_231703'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_001702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_011702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_021702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_031702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_041702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_051702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_061702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_071701'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_081702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_091701'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_101702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_111702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_121702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_131702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_141702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_151702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_161702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_171702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_181702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_191702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_201702'
* Skip existing '/media/USBdrive/ncp-snapshots/hourly_2022-08-30_211701'
* root@obelix:/media/backup/ncp-snapshots# ls /media/nextcloud/ncp-snapshots/ -la
insgesamt 16
drwxr-xr-x 1 root     root     2458 Aug 30 21:17 .
drwxr-xr-x 1 root     root      106 Aug 15 12:08 ..
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 daily_2022-08-23_221703
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 daily_2022-08-24_221702
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 daily_2022-08-25_221703
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 daily_2022-08-26_221703
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 daily_2022-08-27_221703
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 daily_2022-08-28_221703
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 daily_2022-08-29_221703
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-29_221702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-29_231702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_001702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_011702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_021702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_031702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_041702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_051702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_061702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_071701
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_081702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_091701
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_101702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_111702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_121702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_131702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_141702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_151702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_161702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_171702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_181702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_191702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_201702
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 hourly_2022-08-30_211701
drwxrwx--- 1 www-data www-data  378 Jul 29 08:55 manual_2022-07-30_114309
drwxrwx--- 1 www-data www-data  378 Aug 10 15:55 manual_2022-08-10_220354
drwxrwx--- 1 www-data www-data  378 Aug 17 12:50 manual_2022-08-22_201833
drwxrwx--- 1 www-data www-data  378 Aug 23 17:30 manual_2022-08-24_154736
drwxrwx--- 1 www-data www-data  400 Jul 12 06:52 monthly_2021-09-23_201704
drwxrwx--- 1 www-data www-data  400 Jul 12 06:56 monthly_2021-10-23_201704
drwxrwx--- 1 www-data www-data  400 Jul 12 07:03 monthly_2021-11-22_191704
drwxrwx--- 1 www-data www-data  400 Jul 12 07:07 monthly_2021-12-22_191705
drwxrwx--- 1 www-data www-data  400 Jul 12 07:22 monthly_2022-01-21_191704
drwxrwx--- 1 www-data www-data  400 Jul 12 07:30 monthly_2022-02-20_191704
drwxrwx--- 1 www-data www-data  400 Jul 12 07:36 monthly_2022-03-22_191704
drwxrwx--- 1 www-data www-data  400 Jul 12 07:44 monthly_2022-04-21_201703
drwxrwx--- 1 www-data www-data  400 Jul 12 07:51 monthly_2022-05-21_201703
drwxrwx--- 1 www-data www-data  400 Jul 12 08:00 monthly_2022-06-20_201703
drwxrwx--- 1 www-data www-data  378 Jul 12 08:09 monthly_2022-07-20_201702
drwxrwx--- 1 www-data www-data  378 Aug 17 12:50 monthly_2022-08-19_201703
drwxrwx--- 1 www-data www-data  378 Aug  4 12:40 weekly_2022-08-08_221703
drwxrwx--- 1 www-data www-data  378 Aug 10 15:55 weekly_2022-08-15_221721
drwxrwx--- 1 www-data www-data  378 Aug 17 12:50 weekly_2022-08-22_231703
drwxrwx--- 1 www-data www-data  378 Aug 29 21:00 weekly_2022-08-29_231703
root@obelix:/media/backup/ncp-snapshots# ls /media/backup/ncp-snapshots/ -la
insgesamt 16
drwxr-xr-x 1 root     root     550 Aug 24 15:50 .
drwxr-xr-x 1 root     root      26 Aug 15 16:26 ..
drwxrwx--- 1 www-data www-data 400 Aug 16 02:46 monthly_2021-09-23_201704
drwxrwx--- 1 www-data www-data 400 Aug 16 02:50 monthly_2021-10-23_201704
drwxrwx--- 1 www-data www-data 400 Aug 16 02:57 monthly_2021-11-22_191704
drwxrwx--- 1 www-data www-data 400 Aug 16 03:02 monthly_2021-12-22_191705
drwxrwx--- 1 www-data www-data 400 Aug 16 21:52 monthly_2022-01-21_191704
drwxrwx--- 1 www-data www-data 400 Aug 16 21:57 monthly_2022-02-20_191704
drwxrwx--- 1 www-data www-data 400 Aug 16 22:02 monthly_2022-03-22_191704
drwxrwx--- 1 www-data www-data 400 Aug 16 22:09 monthly_2022-04-21_201703
drwxrwx--- 1 www-data www-data 400 Aug 16 22:15 monthly_2022-05-21_201703
drwxrwx--- 1 www-data www-data 400 Aug 16 22:23 monthly_2022-06-20_201703
drwxrwx--- 1 www-data www-data 378 Aug 16 22:28 monthly_2022-07-20_201702

What am I doing wrong here? It seems the monthly snapshots are synced, but not all of them. If I e.g. snyc certain snaphots to a new location snaphost sync is doing fine as the snaphosts get synced the first time.

What can I do to narrow down the problem? I've deleted the entire destination directory and started a sync again with no different result.

thanks in advance!
Christian

Keep functionalitity based on week, day, hour and not just on absolute creation order

When using the keep option, how does it work? Could you please extend the documentation. For instance if I have daily backups and weekly backups, I would like to differentiate how keep is leaving stuff, for instance keep all weekly backups for the month but delete the daily ones, or so. This is quite difficult to achieve with the current absolute counter.
Just a suggestion for improvements,

empty stream is not considered valid error when syncing snapshot

I'm using btrfs-snp and btrfs-sync in cron with basically the following:

btrfs-snp /home hourly 24 3600
btrfs-snp /home daily 7 86400
btrfs-snp /home weekly 4 604800
btrfs-snp /home monthly 12 2592000
btrfs-snp / daily 14 86400

btrfs-sync -d /home/.snapshots /path/to/external/drive/home
btrfs-sync -dv /.snapshots /path/to/external/drive/root

When syncing more than one daily snapshot of my root filesystem I'm getting the following error:

* Check source
* Check destination
* Skip existing '/.snapshots/daily_2019-03-04-131702'
* Synchronizing '/.snapshots/daily_2019-03-05-141702' using seed '@/.snapshots/daily_2019-03-04_131702'...
ERROR: not on mount point: /
time elapsed [0:00:00] | rate [0.00 B/s] | total size [0.00 B]
ERROR: empty stream is not considered valid

There are no issues with syncing multiple snapshots of the home filesystem, just root.

I am doing something wrong?

incremental not working

have some problems. i create snapshots with btrfs-snp:

Subvolume                                                         Total  Exclusive  ID
─────────────────────────────────────────────────────────────────────────────────────────
.snapshots/sync_2022-04-01_110631                               6.19TiB    5.45GiB  6893
.snapshots/sync_2022-04-01_110637                               6.18TiB  256.00KiB  6894
.snapshots/sync_2022-04-02_121818                               6.91TiB   92.39GiB  7958
.snapshots/sync_2022-04-03_084334                               6.93TiB    2.37MiB  8243
─────────────────────────────────────────────────────────────────────────────────────────
Total exclusive data                                                           97.85GiB

now i start btrfs-sync

* Check source
* Check destination
* Synchronizing '/backups/.snapshots/sync_2022-04-01_110631'...
time elapsed [18:07:22] | rate [94,8MiB/s] | total size [5,90TiB]TiB]
* Synchronizing '/backups/.snapshots/sync_2022-04-01_110637' using seed '/backups/.snapshots/sync_2022-04-01_110631'...
time elapsed [0:02:55] | rate [93,7MiB/s] | total size [16,1GiB]
* Synchronizing '/backups/.snapshots/sync_2022-04-02_121818' using seed '/backups/.snapshots/sync_2022-04-01_110637'...
time elapsed [3:09:14] | rate [ 105MiB/s] | total size [1,03TiB]

first and second snapshot with 5,90TiB and 16,1GiB looks good. whereby the second snapshot was in my opinion also too large. 256KiB result in 16,1 GiB. the third snapshot has only 100GiB exclusive data, but the sync is still transfer and it now schon bigger 1TiB.

and du you now why the snapshot size is different between the servers?

Subvolume                                                         Total  Exclusive  ID
─────────────────────────────────────────────────────────────────────────────────────────
sync_2022-04-01_110631                                          5.88TiB    5.60GiB  457
sync_2022-04-01_110637                                          5.89TiB   92.58MiB  6521
sync_2022-04-02_121818                                          6.66TiB    1.03TiB  6532
─────────────────────────────────────────────────────────────────────────────────────────
Total exclusive data                                                            1.03TiB

opens multiple SSH connections

I have a local snapshot folder that contains 48 snapshots. Every hour a remote mirror machine connects to this local machine via btrfs-sync, but this opens a new consecutive SSH connection for each snapshot. Is this intended, or a result of a misconfiguration on my part?

Here's my bash script, which lives in /etc/cron.hourly on the mirror machine

#!/bin/bash
/usr/local/bin/btrfs-sync -v [email protected]:/media/SSD1/ncp-snapshots /media/SSD/ncp-snapshots >>/home/mirror/cron_btrfs-sync.out 2>&1

dir containing snapshots must be a subvolume?

Hi, according to the help prompt this command should take a "dir containing snapshots" as source:

<src> can either be a single snapshot, or a folder containing snapshots

but when I try something like:

btrfs-sync /home/.snapshots root@miniserver:/mnt/SERVERONE/_proveBtrfsSync

(where /home/.snapshots is the dir containing my snapshots, created by the snapper tool, it's not a subvolume, just a dir containing snapshot subvolumes)

I get the following errors:

* Synchronizing '/home/.snapshots'...
ERROR: subvolume /home/.snapshots is not read-only
time elapsed [0:00:00] | rate [0.00 B/s] | total size [0.00 B]
ERROR: empty stream is not considered valid
* Retrying '/home/.snapshots'...
* Synchronizing '/home/.snapshots'...
ERROR: subvolume /home/.snapshots is not read-only
time elapsed [0:00:00] | rate [0.00 B/s] | total size [0.00 B]
ERROR: empty stream is not considered valid
* Retrying '/home/.snapshots'...
* Synchronizing '/home/.snapshots'...
ERROR: subvolume /home/.snapshots is not read-only
time elapsed [0:00:00] | rate [0.00 B/s] | total size [0.00 B]
ERROR: empty stream is not considered valid
Abort

What am I doing wrong?

Thank you very much

shouldn't need to be run via sudo

The instructions for setting up btrfs-sync say that you should give the btrfs user sudo permission on both the host and the target machines.
When this is done, the script could all the btrfs commands via sudo rather than requiring that the entire script is run via sudo. Else, what's the point?
If the entire script is to be run via sudo, then of course, you must give sudo permissions for /usr/local/bin/btrfs-sync, rather than the btrfs command
But when I run the script via sudo, it fails after the getopt command in line 77. I removed the > /dev/null to see what is going on, and I found that there is a "getopt: no permission" message, which is strange. What permission would getopt even need?

Also, when I run the script as root, then the ssh command is executed as root as well so that on the target machine, root's pubkey has to be added to ~btrfs/.ssh/authorized_keys

Anyway, I guess the script should be modified so that all btrfs commands are done via sudo, so that the entire script does not have to be run as root.

Check for bash when destination is local fails

When the destination is is local the test for bash over sh fails as:

btrfs-sync /somewhere/local /someotherplace/local
Remote user's shell not bash. Shells other than bash are not supported at the moment

Commenting out the line (

btrfs-sync/btrfs-sync

Line 136 in 5647f14

 ${DST_CMD[@]} "echo \$0 | grep bash > /dev/null" || { echo "Remote user's shell not bash. Shells other than bash are not supported at the moment"; exit 1; } 

${DST_CMD[@]} "echo \$0 | grep bash > /dev/null" || { echo "Remote user's shell not bash. Shells other than bash are not supported at the moment"; exit 1; }

Script fails for some reason on some snapshots.

root@gr7-mail-web:/mnt/snapshots/daily# btrfs-sync -v /mnt/snapshots/daily/@* [email protected]:/mnt/snapshots

Check source
Check destination
Skip existing '/mnt/snapshots/daily/@home_2020-03-07_062501'
Synchronizing '/mnt/snapshots/daily/@_2020-03-07_062501' using seed '@snapshots/daily/@home_2020-03-07_062501'...
ERROR: chown o257-10-0 failed: No such file or directory
time elapsed [0:00:00] | rate [2.65MiB/s] | total size [2.26MiB]
Delete subvolume (no-commit): '/mnt/snapshots/@_2020-03-07_062501'
Retrying '/mnt/snapshots/daily/@_2020-03-07_062501'...
Synchronizing '/mnt/snapshots/daily/@_2020-03-07_062501' using seed '@snapshots/daily/@home_2020-03-07_062501'...
ERROR: chown o257-10-0 failed: No such file or directory
time elapsed [0:00:00] | rate [2.78MiB/s] | total size [2.25MiB]
Delete subvolume (no-commit): '/mnt/snapshots/@_2020-03-07_062501'
Retrying '/mnt/snapshots/daily/@_2020-03-07_062501'...
Synchronizing '/mnt/snapshots/daily/@_2020-03-07_062501' using seed '@snapshots/daily/@home_2020-03-07_062501'...
ERROR: chown o257-10-0 failed: No such file or directory
time elapsed [0:00:00] | rate [2.63MiB/s] | total size [2.25MiB]
Delete subvolume (no-commit): '/mnt/snapshots/@_2020-03-07_062501'
Abort

Potential bug with "detect remote dst argument" logic

Hi, on line 118:

# detect remote dst argument
[[ "$DST" =~ : ]] && {
  NET="$( sed 's|:.*||' <<<"$DST" )"
  DST="$( sed 's|.*:||' <<<"$DST" )"
  SSH=( ssh -p "$PORT" -o ServerAliveInterval=5 -o ConnectTimeout=1 -o BatchMode=yes "$NET" )
}

The $SSH variable is left just with the value "ssh", which then causes the subsequent block of code to throw the error, ""SSH access error to $NET. Do you have passwordless login setup, and adequate permissions for $DST?" when I call btrfs in the following way. I believe my formatting is correct. Any idea what could be wrong?

$ sudo btrfs-sync --verbose -p 22 ~/datadir-snapshots/ed726c [email protected]:/volume12/chainSnapshotBackups/.

All SRC UUIDs are equal, thus nothing is copied.

I'm running nextcloudpi v1.35.0 and use btrfs snapshots for data backups.

However, for some reason, all my backup subvolumes, share the same UUID, therefore the snapshot sync doesn't really work (skipping all subvolumes).

$ for item in /media/cloud/ncp-snapshots/*; do btrfs su sh "$item" | grep -e "Received UUID:" -e "Name:"; done
Name:                   daily_2021-02-25_231703 
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   daily_2021-02-27_001703 
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   daily_2021-02-28_001703 
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   daily_2021-03-01_001703 
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   daily_2021-03-02_011703  
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   daily_2021-03-03_021703  
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   daily_2021-03-04_021703  
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-03_221703 
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-03_231703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_041702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_051702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_061702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_071703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_081703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_101703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_111703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_121703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_131703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_141702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_151702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_161703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_171702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_181702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_191703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_201703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_221702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   hourly_2021-03-04_231702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   manual_2020-10-09_120552
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   monthly_2020-10-09_121710
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   monthly_2020-11-08_131703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   monthly_2020-12-08_141703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   monthly_2021-01-07_141703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   monthly_2021-02-06_151703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   weekly_2021-02-05_181703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   weekly_2021-02-12_181702
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   weekly_2021-02-19_181703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a
Name:                   weekly_2021-02-26_191703
Received UUID:          73a69463-d24d-f846-ab26-e4e941f9190a

/usr/local/bin/btrfs-sync -dv "/media/cloud/ncp-snapshots" "/media/cloud-backup/ncp-snapshots"
* Check source
* Check destination
* Skip existing '/media/cloud/ncp-snapshots/manual_2020-10-09_120552'
* Skip existing '/media/cloud/ncp-snapshots/monthly_2020-10-09_121710'
* Skip existing '/media/cloud/ncp-snapshots/monthly_2020-11-08_131703'
* Skip existing '/media/cloud/ncp-snapshots/monthly_2020-12-08_141703'
* Skip existing '/media/cloud/ncp-snapshots/monthly_2021-01-07_141703'
* Skip existing '/media/cloud/ncp-snapshots/weekly_2021-02-05_181703'
* Skip existing '/media/cloud/ncp-snapshots/monthly_2021-02-06_151703'
* Skip existing '/media/cloud/ncp-snapshots/weekly_2021-02-12_181702'
* Skip existing '/media/cloud/ncp-snapshots/weekly_2021-02-19_181703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-02-25_231703'
* Skip existing '/media/cloud/ncp-snapshots/weekly_2021-02-26_191703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-02-27_001703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-02-28_001703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-03-01_001703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-03-02_011703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-03-03_021703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-03_221703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-03_231703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_001702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_011703'
* Skip existing '/media/cloud/ncp-snapshots/daily_2021-03-04_021703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_021702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_031702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_041702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_051702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_061702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_071703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_081703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_101703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_111703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_121703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_131703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_141702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_151702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_161703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_171702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_181702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_191703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_201703'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_221702'
* Skip existing '/media/cloud/ncp-snapshots/hourly_2021-03-04_231702'

Behavior on broken pipe?

I am currently setting up a remote backup over ssh. For this I use btrfs-sync. However, no stable SSH connection to the remote host can be guaranteed. The question for me now is how the script behaves if during a synchronization process the connection drops and only a part of a snapshot was received. My fear is:

First, the received part might not be set read-only.
Second, upcoming syncs will use the incomplete snapshot as an incremental parent. This would continue the missing data even further into the future, if I am correct.

Am I interpreting the behavior correctly, or are there certain mechanisms that prevent such problems?

account for failed transfers

a failed transfer results in an uncomplete snapshot. See

https://help.nextcloud.com/t/rethink-backup-via-btrfs-sync/

They should be cleaned upon failure

	pgrep -F /run/btrfs-sync.pid &>/dev/null && { echo "$BIN is already running"; exit 1; }
	echo $$ > /run/btrfs-sync.pid

	## Make sure, the user's remote shell is bash (not sh)
	${DST_CMD[@]} "echo \$0 \| grep bash > /dev/null" \|\| { echo "Remote user's shell not bash. Shells other than bash are not supported at the moment"; exit 1; }

	## Making sure, the remote user has sudo privileges
	${DST_CMD[@]} "sudo echo &> /dev/null" \|\| { echo "Remote user requires sudo privileges"; exit 1; }

nachoparker / btrfs-sync Goto Github PK

btrfs-sync's Introduction

btrfs-sync

Features

Usage

Examples

Manual

Cron

btrfs-sync's People

Contributors

Stargazers

Watchers

Forkers

btrfs-sync's Issues

Recommend Projects

Recommend Topics

Recommend Org