padelt / vzdump-plugin-b2 Goto Github PK

View Code? Open in Web Editor NEW

31.0 5.0 14.0 24 KB

Encrypted off-site backup of Proxmox VE vzdump images for $5/month/TB

License: GNU General Public License v3.0

Shell 100.00%

vzdump-plugin-b2's Introduction

Encrypted off-site backup of Proxmox VE vzdump images for $5/month/TB

This plugin to vzdump will hook into after a single VM backup is done. It will

split the .vma.lzo file that vzdump wrote to chunks of configurable size (default is 2 GB, B2 allows up to 5 GB=5*10^9 byte)
encrypt them using a symmetric key (password - no GPG keys used),
upload them to Backblaze B2 to a configurable path (default: hostname) under a configurable bucket in parallel,
delete the local copy of the backup and
remove all remote backups at B2 except the one it just uploaded.

Every step is checksummed and the checksum file is uploaded as well.

After all chunk uploads are finished, the local copy of the backup is deleted. Therefore you only need twice the size of the largest VM's backup file on the host. The backup is read and written multiple times, so using different storage devices for VMs and backup is a good idea.

Preparation

Get a Backblaze account and generate an application key along with a bucket.

Install git and download this repository:

apt-get install -y git
git clone https://github.com/padelt/vzdump-plugin-b2.git /usr/local/bin/vzdump-plugin-b2

Alternatively transfer it manually to the server.

Provide a recent version of jq:

wget -O /usr/local/bin/jq https://github.com/stedolan/jq/releases/download/jq-1.5/jq-linux64
chmod +x /usr/local/bin/jq

Make a copy of upload-b2.config.template and edit it to your parameters. If you put it anywhere else than the filename upload-b2.config in the same directory as vzdump-plugin-upload-b2.sh, also edit that script and make CONFIG_FILE point to it.
In that config file, note GPG_PASSPHRASE_FILE. That should point to a file containing your long, random and secret passphrase. You can use this do generate one:
```
test -r /root/vzdump-passphrase.txt || dd if=/dev/random bs=1 count=48 2>/dev/null | hexdump -v -e '"%02X"' > /root/vzdump-passphrase.txt
```
Remember and save that string - your backups are unusable without it!

Make vzdump aware of the script by adding a line to /etc/vzdump.conf:

echo "script: /usr/local/bin/vzdump-plugin-b2/vzdump-plugin-upload-b2.sh" >> /etc/vzdump.conf

Make available the b2 command-line utility as documented here:

wget -O /usr/local/bin/b2 https://docs.backblaze.com/public/b2_src_code_bundles/b2
chmod +x /usr/local/bin/b2

Testing

A manual backup of a small VM is a good idea to test general functionality. Look at the log output for hints what may have gone wrong.

vzdump-plugin-b2's People

Contributors

Stargazers

Watchers

Forkers

heartshare ahartimvu cloudxtreme michalkotas promolic1 fruitbytes ozerugae jimmiehansson peacedata0 mamprogr nunombispo ilium007 truewinter geminos-dev

vzdump-plugin-b2's Issues

Make encryption optional

I am running into issues with the GPG encryption:

INFO: gpg: problem with the agent: Inappropriate ioctl for device
INFO: gpg: error creating passphrase: Operation cancelled

While this is not a bug in your script, it might be handy to just disable GPG encryption if there is no use case for it (e.g. in my case here).

b2 installation method

I had to install b2 via the official docs: https://www.backblaze.com/b2/docs/quick_command_line.html

The method from the readme didn't work (on Debian Jessie), I'd suggest updating the readme.

Keep multiple versions of backup

If I understood correctly, you will always remove the other versions of the backups. However it might make sense to keep e.g. the last 14 versions, no?

Would you feel like making that into a parameter?

Error when uploading - UnicodeDecodeError

Hello, script seems to be awesome but unfortunately I can't use it because there is some problem with uploading.
Direct invoking b2 upload_file works ok.
I tried to replace xargs to even simpler for loop for upload step but still no luck, same error.

Log from Proxmox 4.x backup:

INFO: starting new backup job: vzdump 1111 --storage local --compress gzip --remove 0 --node n1 --mode snapshot
INFO: Starting Backup of VM 1111 (qemu)
INFO: status = running
INFO: update VM 1111: -lock backup
INFO: backup mode: snapshot
INFO: ionice priority: 7
INFO: creating archive '/var/lib/vz/dump/vzdump-qemu-1111-2016_04_05-18_03_34.vma.gz'
INFO: started backup task '3cea83e9-753d-46ae-b495-66dcee321e55'
INFO: status: 1% (201457664/10737418240), sparse 1% (127156224), duration 3, 67/24 MB/s
INFO: status: 2% (277479424/10737418240), sparse 1% (127238144), duration 6, 25/25 MB/s
INFO: status: 3% (353501184/10737418240), sparse 1% (128651264), duration 9, 25/24 MB/s
INFO: status: 4% (437125120/10737418240), sparse 1% (131043328), duration 12, 27/27 MB/s
INFO: status: 5% (539754496/10737418240), sparse 1% (137904128), duration 15, 34/31 MB/s
INFO: status: 21% (2322464768/10737418240), sparse 17% (1839480832), duration 18, 594/27 MB/s
INFO: status: 33% (3575775232/10737418240), sparse 27% (2984382464), duration 21, 417/36 MB/s
INFO: status: 61% (6629097472/10737418240), sparse 55% (5940805632), duration 24, 1017/32 MB/s
INFO: status: 62% (6712721408/10737418240), sparse 55% (5945184256), duration 27, 27/26 MB/s
INFO: status: 63% (6796345344/10737418240), sparse 55% (5945315328), duration 30, 27/27 MB/s
INFO: status: 100% (10737418240/10737418240), sparse 91% (9850957824), duration 32, 1970/17 MB/s
INFO: transferred 10737 MB in 32 seconds (335 MB/s)
INFO: archive file size: 279MB
INFO: CHECKSUMMING whole tar.
INFO: SPLITTING into chunks sized <=2000000000 byte
INFO: real  0m0.921s
INFO: user  0m0.000s
INFO: sys   0m0.136s
INFO: CHECKSUMMING splits
INFO: Deleting whole file
INFO: ENCRYPTING
INFO: 3.12user 0.16system 0:03.28elapsed 100%CPU (0avgtext+0avgdata 3448maxresident)k
INFO: 32inputs+573200outputs (1major+274minor)pagefaults 0swaps
INFO: Checksumming encrypted splits
INFO: Deleting cleartext splits
INFO: AUTHORIZING AGAINST B2
INFO: Using https://api.backblaze.com
INFO: UPLOADING to B2 with up to 10 parallel uploads.
INFO: Traceback (most recent call last):
INFO:   File "/usr/local/bin/b2", line 1970, in <module>
INFO: Traceback (most recent call last):
INFO:   File "/usr/local/bin/b2", line 1970, in <module>
INFO:     main()
INFO:   File "/usr/local/bin/b2", line 1916, in main
INFO:     main()
INFO:   File "/usr/local/bin/b2", line 1916, in main
INFO:     decoded_argv = decode_sys_argv()
INFO:   File "/usr/local/bin/b2", line 1089, in decode_sys_argv
INFO:     decoded_argv = decode_sys_argv()
INFO:   File "/usr/local/bin/b2", line 1089, in decode_sys_argv
INFO:     return [arg.decode(encoding) for arg in sys.argv]
INFO: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 5: ordinal not in range(128)
INFO:     return [arg.decode(encoding) for arg in sys.argv]
INFO: UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 5: ordinal not in range(128)
INFO: Something went wrong uploading.
ERROR: Backup of VM 1111 failed - command '/usr/local/bin/vzdump-plugin-b2/vzdump-plugin-upload-b2.sh backup-end snapshot 1111' failed: exit code 10
INFO: Backup job finished with errors
TASK ERROR: job errors

Backup Restoration

Hi,
How to restore an uploaded backup?

Thanks.

SysCallError(104, 'ECONNRESET')

Hi, I get the following error when I try to backup, can you help me?

INFO: /storage/vz/dump/vzdump-qemu-906-2016_09_25-20_23_51.vma.lzo.split.000.gpg: 27%|##6 | 1.07G/4.00G [01:29<10:00, 4.87MB/s]
INFO: /storage/vz/dump/vzdump-qemu-906-2016_09_25-20_23_51.vma.lzo.split.000.gpg: 27%|##6 | 1.07G/4.00G [01:29<07:39, 6.36MB/s]
INFO: /storage/vz/dump/vzdump-qemu-906-2016_09_25-20_23_51.vma.lzo.split.000.gpg: 27%|##6 | 1.07G/4.00G [01:29<04:04, 12.0MB/s]
INFO: ERROR: Unknown error: SysCallError(104, 'ECONNRESET')
INFO: Something went wrong uploading.
ERROR: Backup of VM 906 failed - command '/usr/local/bin/vzdump-plugin-b2/vzdump-plugin-upload-b2.sh backup-end snapshot 906' failed: exit code 10
INFO: Backup job finished with errors

padelt / vzdump-plugin-b2 Goto Github PK

vzdump-plugin-b2's Introduction

Encrypted off-site backup of Proxmox VE vzdump images for $5/month/TB

Preparation

Testing

vzdump-plugin-b2's People

Contributors

Stargazers

Watchers

Forkers

vzdump-plugin-b2's Issues

Make encryption optional

b2 installation method

Keep multiple versions of backup

Error when uploading - UnicodeDecodeError

Backup Restoration

SysCallError(104, 'ECONNRESET')

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent