xeyownt / borgcronic Goto Github PK

View Code? Open in Web Editor NEW

0.0 0.0 0.0 145 KB

A cron script to run Borg Backup

License: GNU General Public License v3.0

Shell 98.90% VBScript 1.10%

borgcronic's People

Contributors

Watchers

borgcronic's Issues

tryborg() must not fail on warning

See borg exit code at https://borgbackup.readthedocs.io/en/stable/usage.html#borg-create.

If rc is 1, tryborg should exit with success, and notify the warning in syslog.

Indicate source for each line in borgcronic log

Currently log list all events and signals, but it is not clear from which process it comes.
Reading the log it should be clear, even for non-dev, which lines comes from which process (from borgcronic daemon, borgcronic wakeup, borg itself).

We could use a convention:

Prefix ## from daemon.
Prefix # from wakeup process.
No prefix, from borg.

Logrotate log files on cygwin

Using either logrotate (if it exists), or savelog (idem), or some poorman replacement.

Unrecoverable error: `.../config does not look like a Borg cache`

This is on borg 1.1.0b5.

I get an unrecoverable error on borg create, namely .../config does not look like a Borg cache.
Looking into borgcronic log file, we have (Lines with # are produced by the cron job, lines without are produced by Borg itself):

# vendredi 1 septembre 2017, 12:55:07 (UTC+0200): [WAKEUP] Sending wakeup signal to daemon.
# vendredi 1 septembre 2017, 12:55:07 (UTC+0200): [DAEMON] Waiting for wake-up signal...
####################################################################################################
# vendredi 1 septembre 2017, 12:55:07 (UTC+0200): Starting borgcronic[16805]
# vendredi 1 septembre 2017, 12:55:08 (UTC+0200): Starting 'borg create'
# vendredi 1 septembre 2017, 12:55:08 (UTC+0200): Doing 'borg create'
# vendredi 1 septembre 2017, 12:55:08 (UTC+0200): ## borg create -v --stats -x --compression lz4 ::{hostname}_{now:%Y%m%d_%H%M} /home --exclude root/.cache --exclude *.nobak --exclude */xapiandb --exclude */.thumbnails
## Borg repo: 296935M (311359488000)
## Borg cache: 1215M (1274908672)
====================================================================================================
Synchronizing chunks cache...
Archives: 51, w/ cached Idx: 51, w/ outdated Idx: 2, w/o cached Idx: 2.
Reading cached archive chunk index for computer_20170830_2357 ...
Merging into master chunks index ...
Reading cached archive chunk index for another_computer_20170828_2317 ...
Merging into master chunks index ...
[...]
Reading cached archive chunk index for computer_20170630_0151 ...
Merging into master chunks index ...
Done.
# vendredi 1 septembre 2017, 12:54:53 (UTC+0200): [DAEMON] Waiting for wake-up signal...
# vendredi 1 septembre 2017, 17:26:59 (UTC+0200): [WAKEUP] Sending wakeup signal to daemon.
####################################################################################################
# vendredi 1 septembre 2017, 17:26:59 (UTC+0200): [DAEMON] Waiting for wake-up signal...
# vendredi 1 septembre 2017, 17:26:59 (UTC+0200): Starting borgcronic[4788]
# vendredi 1 septembre 2017, 17:26:59 (UTC+0200): Starting 'borg create'
# vendredi 1 septembre 2017, 17:26:59 (UTC+0200): Doing 'borg create'
# vendredi 1 septembre 2017, 17:26:59 (UTC+0200): ## borg create -v --stats -x --compression lz4 ::{hostname}_{now:%Y%m%d_%H%M} /home --exclude root/.cache --exclude *.nobak --exclude */xapiandb --exclude */.thumbnails
## Borg repo: 297021M (311449477120)
## Borg cache: 1197M (1255690240)
====================================================================================================
Killed stale lock [email protected].
Removed stale exclusive roster lock for pid 17074.
Removed stale exclusive roster lock for pid 17074.
Killed stale lock [email protected].
Removed stale exclusive roster lock for pid 17074.
Removed stale exclusive roster lock for pid 17074.
Local Exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 3482, in main
    exit_code = archiver.run(args)
  File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 3410, in run
    return set_ec(func(args))
  File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 122, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/borg/archiver.py", line 469, in do_create
    lock_wait=self.lock_wait) as cache:
  File "/usr/local/lib/python3.5/dist-packages/borg/cache.py", line 195, in __init__
    self.open(lock_wait=lock_wait)
  File "/usr/local/lib/python3.5/dist-packages/borg/cache.py", line 296, in open
    self.rollback()
  File "/usr/local/lib/python3.5/dist-packages/borg/cache.py", line 388, in rollback
    self._do_open()
  File "/usr/local/lib/python3.5/dist-packages/borg/cache.py", line 283, in _do_open
    self._check_upgrade(config_path)
  File "/usr/local/lib/python3.5/dist-packages/borg/cache.py", line 264, in _check_upgrade
    raise Exception('%s does not look like a Borg cache.' % config_path) from None
Exception: /var/cache/borg/5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03/config does not look like a Borg cache.

Platform: Linux computer 4.4.0-93-generic #116-Ubuntu SMP Fri Aug 11 21:17:51 UTC 2017 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.1.0b5  Python: CPython 3.5.2
PID: 4854  CWD: /
sys.argv: ['/usr/local/bin/borg', 'create', '-v', '--stats', '-x', '--compression', 'lz4', '::{hostname}_{now:%Y%m%d_%H%M}', '/home', '--exclude', 'root/.cache', '--exclude', '*.nobak', '--exclude', '*/xapiandb', '--exclude', '*/.thumbnails']
SSH_ORIGINAL_COMMAND: None

====================================================================================================

Environment variables:

[D] # Borg variables:
[D]     BORG_KEYS_DIR=/var/lib/borg/keys
[D]     BORG_CACHE_DIR=/var/cache/borg
[D]     BORG_REPO=/smb/lacie-cloudbox/borg
[D]     BORG_SECURITY_DIR=/var/lib/borg/security

The repository is stored on a samba share.

We see two executions of borg create, one at 12:55:08, and a second at 17:26:59. The first got interrupted after syncing with the remote, and we see that actually the time on the client went backwards after the interruption (see timestamp vendredi 1 septembre 2017, 12:54:53 after Done. message). In fact, looking into kern.log, we see that the computer was rebooted at 12:54:41. In auth.log, we can see timestamps up to 12:55:01 before they get reset at 12:54:41. The cause of this backward move is unknown.

At the second borg create, we see that Borg cleans up some stale lock, but then fail permanently.
In future executions, we get always the same error (/config does not look like a Borg cache).

Content of /var/cache/borg:

drwx------  3 root root 4096 sep 23 16:16 ./
drwxr-xr-x 25 root root 4096 mai 22 20:12 ../
-rw-r--r--  1 root root    0 sep 23 12:47 borgcronic.success.create
-rw-r--r--  1 root root    0 sep 23 12:48 borgcronic.success.prune
-rw-r--r--  1 root root   29 sep 23 16:16 borgcronic-ts-create
-rw-r--r--  1 root root   29 sep 23 16:08 borgcronic-ts-create.fail
-rw-------  1 root root  194 mai 22 20:12 CACHEDIR.TAG
drwx------  3 root root 4096 sep 23 16:24 5891b5b522d5df086d0ff0b110fbd9d21bb4fc7163af34d08286a2e846f6be03/

Content of /var/cache/borg/5891...:

drwx------ 3 root root 4096 sep 23 16:24 ./
drwx------ 3 root root 4096 sep 23 16:16 ../
-rw------- 1 root root    0 sep  1 17:27 chunks
drwx------ 2 root root 4096 sep  1 12:55 chunks.archive.d/
-rw------- 1 root root    0 sep  1 17:27 config
-rw------- 1 root root    0 sep  1 17:27 files
-rw------- 1 root root   68 mai 22 20:12 README

We see that all files in this folder are empty. Looking further though, in chunks.archive.d, the files there seem ok.

Support Windows platform

Borgcronic should work at least on Cygwin-64/Windows 7.

Run borgcronic through shellcheck

Let's remove all those nasty bugs.

Install borgcronic script in appropriate place

Borgcronic script should be installed in /usr/bin or /usr/local/bin, and entries in /etc/cron.daily and /etc/cron.weekly should actually be symlink to that script.

Add borg environment variables to log

The borgcronic log contains the complete execution line, but miss the value of the borg environment variables.

The log should list all these variables.
Currently these are: BORG_REPO, BORG_CACHE_DIR, BORG_KEYS_DIR, BORG_SECURITY_DIR.

Send repository summary at regular interval

For instance output of borg list.

Send overall list.
Send list for current hostname.

Goal is to get a positive message about last successful backups.

Interval should be configurable in $CONF file.

Make sure borgcronic doesn't postpone other anacron jobs

If borgcronic is installed as anacron daily job (copied in /etc/cron.daily), check what is the impact on the scheduling of other anacron jobs.

For instance, if borgcronic can't get the lock on the borg repository, it should not postpone execution of other anacron jobs.

Create README file

How to INSTALL
How to USE

When done, send a message to Borg list?

Delay devil startup by a fixed + random amount

Fixed 300 + random 900.

Get out of startup critical path since devil will mostly trigger at boot time.
Random delay for basic load spread on the server.

Send mail notifications on errors or warnings

Example of notifications:

Backup failure
Backup failure if last successful too old.
Last backup too old on borgcronic startup.
Content of borgcronic last.

Some settings to consider

Warn by email. Email info configured in $CONF file.
Period configured in $CONF file.
Message must contain time of last succesful backup.

Should we focus on create event only, or also report failure for purge event?

Can't access network share on Windows

On windows, we can use the devil command, which is a simpler daemon that includes its own scheduler. Use the following script to start borgcronic devil on Windows:

Set WshShell = CreateObject("WScript.Shell") 
WshShell.Run "C:\cygwin64\bin\bash.exe --login -c '/usr/local/bin/borgcronic devil'", 0
Set WshShell = Nothing

Name the script /usr/local/bin/borgcronic-devil.vbs and create a shortcut in Startup folder.
This way process is really started as standard user script, and it can access network share normally.

First borgcronic backup suspended because repository is unknown

Borg asks

Warning: Attempting to access a previously unknown unencrypted repository!
Do you want to continue? [yN] Synchronizing chunks cache..

Work-around is to press y and enter. This should be done automatically.

Copy old config or add new config on install, adding a suffix to filename

On install

Either keep current config, and write new default config with extension .new.
Or create new config, and copy old with extension .old.

borgcronic install should pick REPO name from /etc on update

When updating borgcronic, we must pass again BORG_REPO on the command line, even though it is already available in /etc/borgcronic.conf. Instead, if the configuration file is found, the repo path should be read from there.

Don't trigger a backup when calling borgcronic without arguments

This probably breaks expectation. Instead print a basic usage string.

Add /etc/cron.d entry to run borgcronic every 4 or 6 hours

Borgcronic runs fast enough to allow running it more that once a day.

For instance, an entry like

5 */4 * * * perl -le 'sleep rand 900' && /etc/cron.daily/borgcronic

should do it.

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.