Practically, it probably won't be a problem. But still, it would be nice to assert that the in-memory buffered log entries per task do not exceed a threshold.

Search for this issue in the codebase.

ref #10

Ideas

• Find sensible max size for in-memory logs
• Serialize logs to JSON []byte, keep level in separate value
• Keep sum of len([]byte) arrays until max size is reached
• Discard log messages lowest debug level first, then oldest

The zrepl documentation should have a section which details any dataset properties that are overridden.

There appears to be at least one, as I noticed the replicated datasets are not mounted.

This will be important for disaster recovery if it is ever needed - what do I need to put back to normal?

This makes sense but is currently not allowed:

  prune:
    policy: grid
    # allow one day of replication lag, then archive
    grid: 1x1d (keep=all) | 24x1h | 40x1d | 6x30d

Hacky workaround in ca1a482

had to remove auto go-generate in 47726ad

For the local control socket, we should require the CLI version and daemon version to be the same.
As long as we use JSON over HTTP, a header like X-ZREPL-VERSION would suffice.

Version information a la git describe --dirty

• version subcommand
• control version subcommand?
• in documentation -> figure out how to do multi-version docs?
• in log message on start INFO

Example output (mind the total_rx field at EOL)

[INFO][hn1][pull][storage/backups/zrepl/pull/hn1/zroot/ROOT/default][storage/backups/zrepl/pull/hn1/zroot/ROOT/default => zroot/ROOT/default]: progress on receive operation total_rx="%!s(uint64=22101986)"

The current recommendation is for zrepl to run as root (until the finer details of what needs to be set to allow it to run as an unprivileged user are discovered and clearly documented).

For remote replication, this is hindered by the fact that sshd does not allow root login by default. To work around this without opening a large security hole, the following should be added to /etc/ssh/sshd_config:
PermitRootLogin forced-commands-only

For more info, users should be directed to the sshd_config(5) man page:
https://man.freebsd.org/sshd_config

Suggest this should be added to both the installation doc page and the tutorial.

The tutorial example for the authorized_keys file appears to be wrong. Rather than "zrepl stdinserver backups.example.com" I think it should be "zrepl stdinserver prod1.example.com"

From GNU/Linux Ubuntu 17.04, following instructions:

./lazy.sh devsetup

returns

Collecting pygobject==3.22.0 (from -r /home/stephane/code/golang/gopath/src/github.com/zrepl/zrepl/docs/requirements.txt (line 17))
  Could not find a version that satisfies the requirement pygobject==3.22.0 (from -r /home/stephane/code/golang/gopath/src/github.com/zrepl/zrepl/docs/requirements.txt (line 17)) (from versions: )
No matching distribution found for pygobject==3.22.0 (from -r /home/stephane/code/golang/gopath/src/github.com/zrepl/zrepl/docs/requirements.txt (line 17))

and pip3 search pygobject says:

pygi-treeview-dnd (0.1.0)  - Workaround that allows PyGObject programs to use the high level TreeView DnD API

Should be fixed for both retention grid & interval specs -> time.ParseDuration is the problem

refs #13

During first replication of many Gigabytes of data, I initially had the interval of the pull job set as 10m, and the first replication would not be finished by the time the second one was called to start. I checked the status many hours later and could see numerous ssh sessions running which led me to believe multiple replication jobs were now running at once (which I dont think should ever happen). I expected that if another replication job was called to start before the previous had finished, the new job would just be cancelled entirely.

I did not look into the state of my replicated data, or if the replications were proceeding ok. It was purely the fact that multiple zrepl ssh sessions were running that led me to believe this was the behaviour.

If source and pulll diverge (replication lag) and source still has a bookmark of the latest state at puller, we don't have a conflict, just a gap in replication. It is better to resume replication then instead of just throwing Diverged errors.

• Diffing & Replication logic support (was always supported)
• Create Bookmarks
• update docs warning about replication lag (still a thing, but mitigated by this feature)
• Find sane default for pruning bookmarks (basically they cost nothing, just keep them around?)

Right now, all zfs send invocations on the sending side are without -p, meaning we do not replicate any properties.

On the receiving side, as a safeguard, we override mountpoint in order to protect ourselves from a malicious sender that is trying to mount over some filesystem on the backup server.

This is more of a theoretical advantage and I'm investigating better solutions, e.g. zfs receive -x all, see https://github.com/zrepl/zrepl/wiki/ZFS-Feature-Support-&-Wishlist#zfs-receive--x-all

Agenda

• Find a way how property replication should work
  • Where to store properties on receiver?
  • Protect against malicious sender (zfs receive -x all)
  • Think about restore procedures
• Implement it
• Document it (refs #23 )

• oneoff script
• build in docker
• document it

When reading the tutorial, I was initially confused about which PC should have the pull_prod1 job defined. It took a white for me to realize the tutorial section title "Configure backups" was referring to the "backups" server (even though it has a box around it, it wasn't immediately obvious). Perhaps this PC could be re-titled "backup_server" to make it more obvious?

It probably wasn't helped by the typo in "Analysis" section, which says the pull job is defined on prod1 (I believe this should have been "backups").

currently, a slow TCP connection will block the log call for retry_interval

additionally, the dialing / name resolution timeout is not set to retry_interval -> if no name resolution then log blocks for ~30s ?

cmd/test.go:69

Maybe an 'interactive' default logger would be nice?

I was initially confused between the interaction between the "interval" parameters and the "grid" pruning parameters. I think I won't be the only one asking these questions, so it should be added to your documentation site. Some questions I had:

• What happens if I define an source interval of 10m, a source grid with 4x15m, and a pull grid with 3x20m?
• What is the difference between the source interval and pull interval? What happens if the source interval is different from the pull interval? I assume it would be normal to have the source interval quite frequent (e.g. 10m), and much less frequent on the pull job (e.g. 24h)?
• For the grid parameters, what is the difference between 1x24d vs 24x1d
• What does the (keep=all) setting do in the grid parameter?
• For the interval parameter, I don't seem to be able to set "1d"?

Where you have incomplete documentation, and the documentation references the examples in the code, it would be handy if "cmd/sampleconf/random/logging.yml" was a hyperlink to the file on GitHub.

otherwise the output log gets mixed up

could extend logger to aquire its mutex

It can be annoying trying to read verbose log files. It would be nice to have a command "zrepl status" which outputs to the command line the status of any jobs currently running (including progress details?), and also "zrepl list" which outputs to the command line the full suite of snapshots (and size details) available on the PC it is being run (whether they are source snapshots from this machine or pull snapshots from other machines)

• Check if zrepl was down for more than X times the snapshot interval length
• -> disable pruning in such a case (switch it to dry run?)
• -> could detect such a case by searching for gaps in the snapshot list -> problem: gotta understand pruning policy (fading inudces gaps)
• -> should work independently of pruning strategy
• Optionally do time-check in zrepl and compare to system time and see how far off we are?

https://rakyll.org/profiler-labels/

Mapping:

jobs:
- name: pull_app-srv
  ...
  type: pull
  mapping: {
    "<":    "storage/backups/app-srv",
  }

Current result:
zroot/var => storage/backups/app-srv/var

Expected result:
zroot/var => storage/backups/app-srv/zroot/var

• readthedocs theme is better than customized docdock theme
• evaluate cross referencing, but it can't be worse than hugo's
• expressive, unified admonitions (docdock has pletora of notice, panels, alert, etc)
• could reference go code easily using third party go domain

for a job, should test if it can connect to the other side, maybe show the remote version

• API
• subcommand

FreeBSD

• 11.X
• 10.3
• 9.X ? (not supported anymore)

ZoL

• 0.6.5.9_4.10.9_1-1 (Arch Linux, ZFS released Feb 3 2017)
• 0.6.4.2_3.16.39-1+deb8u2 (Debian Jessie, ZFS released June 26 2015)
• 0.6.3_3.16.39-1+deb8u2 (Debian Jessie, ZFS released June 12 2014)
  ZFS: Loaded module v0.6.3-1.3, ZFS pool version 5000, ZFS filesystem version 5
• 0.6.x ?
• 0.5 ?

OS X

• 1.6.1, Sierra (2017-02-10)

illumos based distros

• ?

I believe the documentation should explain the zrepl behaviour for what happens if the first replication is not finished at the time that the second replication is called to begin. This needs to cover the pull, push and local scenarios.

This is likely to occur during the first replication of many Gigabytes of data, if the interval of the pull job set as 10m. The first replication would not be finished by the time the second one was called to start. Given that this will be occurring to new users, it is important they are clear on the behaviour they can expect during this first time use.

https://github.com/hashicorp/hcl

https://godoc.org/github.com/hashicorp/hcl#Parse

https://godoc.org/github.com/hashicorp/hcl/hcl/ast#File

• Feature detection
• Use it in autosnapper (queue up snaps + bookmarks, then do all at once)
• Use it in pruner (queue up destroys, fallback to individual destroy if atomically destroying all of them fails)
• Use as replacement for complicated ZFS lists?

After updating to 0.0.1 release, I am getting some errors upon stopping zrepl:

[signal SIGSEGV: segmentation violation code=0x1 addr=0x20 pc=0x72a326]

goroutine 37 [running]:
github.com/zrepl/zrepl/logger.(*Logger).WithError(0xc4201aa080, 0x0, 0x0, 0x0)
        /wrkdirs/usr/ports/sysutils/zrepl/work/src/github.com/zrepl/zrepl/logger/logger.go:105 +0x26
github.com/zrepl/zrepl/cmd.(*ControlJob).JobStart(0xc4200f7060, 0xad5ec0, 0xc4201381b0)
        /wrkdirs/usr/ports/sysutils/zrepl/work/src/github.com/zrepl/zrepl/cmd/config_job_control.go:62 +0x3fc
github.com/zrepl/zrepl/cmd.(*Daemon).Loop.func1(0xad5ec0, 0xc4201381b0, 0xc4201b4000, 0xad2b80, 0xc4200f7060)
        /wrkdirs/usr/ports/sysutils/zrepl/work/src/github.com/zrepl/zrepl/cmd/daemon.go:82 +0x45
created by github.com/zrepl/zrepl/cmd.(*Daemon).Loop
        /wrkdirs/usr/ports/sysutils/zrepl/work/src/github.com/zrepl/zrepl/cmd/daemon.go:81 +0x367

see #10 for initial proposal & discussion