This is my DIY solution for creating incremental backups. You are free to use it, however it is not a one-click solution, you will need to personalize the script to match your own needs.
- Incremental backups
- Encryption of backups symmetrically or asymmetrically
- Filtering of files in .gitignore
- Flexible file list generation via custom
find
arguments - Enumeration of project folders and treatment of each project as a separate backup
- Flexible upload destinations (Google Drive, OneDrive, Mega, FTP, SFTP)
- arch/: Backups ready to be uploaded.
- data/: Data regarding the backups, such as date of last backup and miscellaneous states.
- temp/: Temporary files, will be cleared at each run, if not empty.
- backup.sh: Contains functions to aid the creation of backups. Backup calls may be placed on the end of the file, or in a separate file, if this script is included with the functions exposed:
. backup.sh
. - upload.sh: Enumerates the directory of backups ready to upload, and initiates parallel upload onto each configured service.
- start.sh: Clears temporary files from previous runs, if any, starts the incremental backup and uploads the generated archives at the end.
The destinations to be backed up are listed towards the end of the backup.sh
file. There are two functions that may be of interest when initiating a backup, such as:
- backup name dir [filter] -- Backup a directory.
- name: Name of the backup.
..
will be translated to/
during upload, so the final storage may be nicely structured. E.g.Servers..ncc1701e..home
will be stored on GDrive asServers/ncc1701e/home/date.tar.xz.gpg
. - dir: Directory to be backed up. Before the backup process, the directory will be traversed and any
.gitignore
files will be respected, so your project folders will be cleanly backed up, assuming you have a proper ignore list in each. - filter: Optional. You may specify additional arguments for
find
, such as the exclusion of specific files/directories. E.g.-size -50M
will exclude files bigger than 50 MB.
- name: Name of the backup.
- backup_dev name dir [filter] -- Backup subdirectories within a directory as separate backups.
- name: Name of the backup, subdirectories will be appended with
..
, as such, you will end up withDir/Subdir/date.tar.xz.gpg
style backups. - dir: Directory whose direct descendants to be treated as individual backups, as if they were each listed in the script as a call to
backup()
. - filter: Regular expression to be matched with Bash's
=~
operator on the directory names to be backed up. E.g.(ATL)?Project[0-9]+|(WindowsForms|Console|Wpf|Silverlight|Web)Application[0-9]+
would skip "generic" (where the name was left to the default during creation) Visual Studio projects to be backed up. In order to prevent false-positives, the regex has to match the full directory name, not just parts of it.
- name: Name of the backup, subdirectories will be appended with
The purpose of the last function is to allow differential treatment for directories such as Visual Studio 20../Projects
and IdeaProjects
. Instead of the directories themselves being backed up as a whole, its subdirectories, the projects themselves, will be treated as different backups automatically. The backups will also be named accordingly.
In the upload.sh
file, there is a loop which goes through the archives to be uploaded. You will have to call your own uploader scripts/commands here. Preferably, call them each in a (...)&
, so they fork off and run the upload command in a subshell in the background. This way, you may run multiple uploads of the same file to different services in the same time. The wait
command at the end of the loop will wait for all background processes to finish.
The current file contains commands to call rclone
(which supports various cloud providers, such as Google Drive, OneDrive, etc) megatools
(for mega.nz) and ftp
. It is recommended that you don't put your login credentials into the script, instead use the proper way to persist logins via the recommended way for the tool being used. rclone
uses a session store by default, megatools
supports ~/.megarc
and ftp
can be made to auto-login via ~/.netrc
:
machine storage.rcs-rds.ro
login name
password pass
If you're running the uploader script from within Cygwin, make sure to install the inetutils
package, otherwise Cygwin will use the Windows implementation of the ftp
command, which has slight implementation differences, and may break the script.
After customization the script may be run with different parameters in order to modify run-time behaviour.
backup.sh [-f] [-s] [-v] [-t f|i] [-e n|g|o] [-g recipient] [-o password]
-f
: Forces a backup, even if no files changed since the last one backup. (Not to be used with incremental backups, since it'd just create a bunch of empty archives.)-s
: Performs a dry run, only prints if any files changed since the last backup.-v
: Verbose mode, currently only useful with-s
, where it prints a list of the actual files that have changed.-t f|i
: Specifies backup type, can be:f
: Full backup, archive will contain all the files.i
: Incremental backup, archive will only contain the files that have changed since the last backup.
-e n|g|o
: Specifies encryption type, can be:n
: None, no encryption will be performed. (Not recommended if your backups will be uploaded to public cloud platforms.)g
: GPG, files will be encrypted for recipient specified in global variablegpg_keyid
or argument-g
.o
: OpenSSL, files will be encrypted using AES-256 in CBC mode with the password specified in the global variableopenssl_pass
or argument-o
.
-g ...
: Specifies the public key which will be used to encrypt the archive. The value of the argument can be an email address, a key ID, or a full fingerprint. You should specify a full fingerprint, in order to avoid collisions with short key IDs or collision-based attacks. For more information, consult GPG's documentation, namely the "Encrypting and decrypting documents" section. [Note: It is possible to specify multiple public keys when encrypting with GPG, however the script does not support this at this time through this argument. It should be trivial to implement this functionality for anyone who needs it, though.]-o ...
: Specifies a password which will be used to encrypt the archive. The value of this argument is NOT the password directly, rather a modifier for the password source followed by a value. To specify a password, usepass:
followed by the actual password. As specifying a password like this will make the password visible to anyone requesting a process list, you should use a more secure mode, such asenv:
for an environmental variable orfile:
for a file. For more information, consult OpenSSL's documentation, namely the "Pass phrase arguments" section.