• do you transfer via mbuffer -> mbuffer or via ssh?
• what connectTimeout value are you using?
• which znapzend version are you running?
• any errors/warnings in the log file?

from znapzend.

• I am doing mbuffer transfers
• connect timeout to mbuffer is -W60
• znapzend is v 0.14

Regading the logs, I noticed that - for whatever reason the mbuffer/zfs recv combo on dst doesn't seem to return if anything spooky happended to the DST dataset, resp. the snaphots, where zfs will claim, that the dataset on the receiver has been altered, what should not be possible. However, this morning, I encountered, what I deem to be a pre-requisit for this to happen, when I manually triggered a znapzend --runonce against the destination where I already noticed that the snapshot size wasn't 0, which indicated that the dataset itself must have been changed. When I ran znspzend nevertheless, I got this:

root@nfsvmpool01:~# znapzend --debug --runonce=sataTank/Intranet
[Mon Mar 16 08:31:15 2015] [info] refreshing backup plans...
[Mon Mar 16 08:31:16 2015] [info] found a valid backup plan for sataTank/Intranet...
[Mon Mar 16 08:31:16 2015] [debug] snapshot worker for sataTank/Intranet spawned (8107)
[Mon Mar 16 08:31:16 2015] [info] creating recursive snapshot on sataTank/Intranet

zfs snapshot -r sataTank/Intranet@2015-03-16-083116

[Mon Mar 16 08:31:16 2015] [debug] snapshot worker for sataTank/Intranet done (8107)
[Mon Mar 16 08:31:16 2015] [debug] send/receive worker for sataTank/Intranet spawned (8109)
[Mon Mar 16 08:31:16 2015] [info] starting work on backupSet sataTank/Intranet

zfs list -H -r -o name sataTank/Intranet

[Mon Mar 16 08:31:16 2015] [debug] sending snapshots from sataTank/Intranet to [email protected]:sataTank/Intranet

zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet

ssh -o Compression=yes -o CompressionLevel=1 -o Cipher=arcfour -o batchMode=yes -o ConnectTimeout=30 [email protected] zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet

[Mon Mar 16 08:31:17 2015] [debug] receive process on 192.168.1.2 spawned (8114)

zfs send -I sataTank/Intranet@2015-03-14-160000 sataTank/Intranet@2015-03-16-083116|/opt/csw/bin/mbuffer -q -s 128k -W 60 -m 1G -O '192.168.1.2:10002'

ssh -o Compression=yes -o CompressionLevel=1 -o Cipher=arcfour -o batchMode=yes -o ConnectTimeout=30 '[email protected]' '/opt/csw/bin/mbuffer -q -s 128k -W 60 -m 1G -4 -I 10002|zfs recv sataTank/Intranet'

cannot receive incremental stream: destination sataTank/Intranet has been modified
since most recent snapshot
[Mon Mar 16 08:31:21 2015] [debug] receive process on 192.168.1.2 done (8114)
Mojo::Reactor::Poll: Read failed: Event "close" failed: ERROR: executing receive process at /opt/znapzend-0.13.0/bin/../lib/ZnapZend/ZFS.pm line 379.
[Mon Mar 16 08:31:22 2015] [debug] sending snapshots from sataTank/Intranet@2015-03-14-160000 to [email protected]:sataTank/Intranet@2015-03-14-160000

zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet@2015-03-14-160000

ssh -o Compression=yes -o CompressionLevel=1 -o Cipher=arcfour -o batchMode=yes -o ConnectTimeout=30 [email protected] zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet@2015-03-14-160000

[Mon Mar 16 08:31:22 2015] [debug] sending snapshots from sataTank/Intranet@2015-03-16-083116 to [email protected]:sataTank/Intranet@2015-03-16-083116

zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet@2015-03-16-083116

ssh -o Compression=yes -o CompressionLevel=1 -o Cipher=arcfour -o batchMode=yes -o ConnectTimeout=30 [email protected] zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet@2015-03-16-083116

cannot open 'sataTank/Intranet@2015-03-16-083116': dataset does not exist
[Mon Mar 16 08:31:23 2015] [debug] sending snapshots from sataTank/Intranet/IntranetAssets to [email protected]:sataTank/Intranet/IntranetAssets

zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet/IntranetAssets

ssh -o Compression=yes -o CompressionLevel=1 -o Cipher=arcfour -o batchMode=yes -o ConnectTimeout=30 [email protected] zfs list -H -o name -t snapshot -s creation -d 1 sataTank/Intranet/IntranetAssets

[Mon Mar 16 08:31:24 2015] [debug] receive process on 192.168.1.2 spawned (8128)

zfs send -I sataTank/Intranet/IntranetAssets@2015-03-14-160000 sataTank/Intranet/IntranetAssets@2015-03-16-083116|/opt/csw/bin/mbuffer -q -s 128k -W 60 -m 1G -O '192.168.1.2:10002'

ssh -o Compression=yes -o CompressionLevel=1 -o Cipher=arcfour -o batchMode=yes -o ConnectTimeout=30 '[email protected]' '/opt/csw/bin/mbuffer -q -s 128k -W 60 -m 1G -4 -I 10002|zfs recv sataTank/Intranet/IntranetAssets'

cannot receive incremental stream: destination sataTank/Intranet/IntranetAssets has been modified
since most recent snapshot
On the receiver, I checked for active zfs recv/mbuffer processes and got that:

root@nfsvmpool02:/root# date; ps -ef | grep mbuffer
Mon Mar 16 08:35:44 CET 2015
root 1316 1311 0 08:31:22 ? 0:00 bash -c /opt/csw/bin/mbuffer -q -s 128k -W 60 -m 1G -4 -I 10002|zfs recv sataTa
root 1317 1316 13 08:31:22 ? 4:20 /opt/csw/bin/mbuffer -q -s 128k -W 60 -m 1G -4 -I 10002
root 1394 296 0 08:35:44 pts/1 0:00 grep mbuffer

Also, znapzend didn't return, until I pkilled mbuffer on the receiver.

from znapzend.

[Mon Mar 16 08:31:22 2015] [debug] sending snapshots
from sataTank/Intranet@2015-03-14-160000
to [email protected]:sataTank/Intranet@2015-03-14-160000

this looks like you have the listsnapshots property set on statTank. can you post the output of

zpool get listsnapshots sataTank

it is likely that this will cause the stuck recv process...

from znapzend.

That's true… I have listsnapshots=on on every zpool. Can you explain, what the issue with that is?

from znapzend.

by now, znapzend assumes that zfs list -H -r -o name pool/dataset returns datasets only and no snapshots. if listsnapshots is on, this command will return snapshots, too. znapzend will then handle those snapshots as datasets. send will fail immediately, but recv does not get "triggered", that means it'll wait forever...

I'll make a patch to fix the zfs list dataset only assumption.

--> #141

from znapzend.

Great - I have set listsnapshots to off for the time being.

from znapzend.