r3cgm / archiver3 Goto Github PK

Consider removing the journaling feature of ext4 if it makes sense to do so. For example, since we (will) have such good data validation going on already, is it worth the performance penalty to have journaling in play?

Add an optional feature to enable the --checksum option when performing the backup rsync. This will perform a verification of files as they are being transferred. However, this feature is expensive, requiring twice as much bandwidth. It is also somewhat superseded by the verify.py utility which will perform this sort of validation anyway. But if you have bandwidth to spare, this could be a good option to have.

Assume 1.4% overhead.

The code assumes that the archive directory is populated with files. Handle the scenario where the archive has no files gracefully.

When using losetup -f to determine the next free loopback device, consider the case where none are available. There is a builtin limit, maybe 8? This limit is possible to extend through mknods or something.

Provide for use of a Yubikey to perform second factor (or even possibly 'only' factor) decryption of volumes.

In practice, the limit of 100 changed files is too few. Bump this to 250 to allow for more realistic numbers of files changed.

Show an ASCII diagram of the ArchiveR3 structure. For example:

container > loopback > encrypt > map > format > mount

Basically, show some sort of hierarchical way for users to understand what the underlying structure is.

When encrypting a volume via tc-play, there is typically no output generated. Since this operation could conceivably take > 24 hours on a 50+ gig file, not having any kind of status update is annoying. Imagine there was a problem that happened 2 hours in and you had no idea... you might wait until 36 hours had passed before grudgingly stopping the process, and you would have wasted a day and a half. Find some way to print progress updates.

For small directories, it is possible to get negative size calculations. Fix this.

             Estimated Consumption: -0.90% 8192/-909262 8K/-909262B

Need to write all output to a log file.

Perform a check when a new archive is being transferred, or when an existing backup is being extended in size, if the remote backup directory is not big enough to handle it.

Currently, specifying --auto does not automatically create the ArchiveR3 /logs and /data dir if they do not exist already.

When determining the size of files in directories, we are not accounting for files which are not accessible (such as those contained in a subdirectory which we do not have permission to enter).

If you hit Ctrl-C...

                           Archive: /mnt/remote/r3cgm/dir/
      Sizing /mnt/remote/r3cgm/dir/: ^CTraceback (most recent call last):

File "./backup.py", line 370, in
backup.main()
File "./backup.py", line 342, in main
rc = self.backup()
File "./backup.py", line 162, in backup
arc_block = dir_size(archive_dir, block_size=512)
File "/home/user/ArchiveR3/ArchiveR3.py", line 94, in dir_size
status_result('UNEXPECTED ERROR' + sys.exc_info()[0], 3)
TypeError: cannot concatenate 'str' and 'type' objects

Consider using the tc-play API instead of the command-line interface. And, if abandoning the CLI, it may be possible to also remove the dependency for expect.

Perform a bandwidth test to see what the upload rate currently is, e.g. 6mbit/s. Take that result and use it to calculate some fractional upload rate that rsync can use, such as 85%.

Also consider a dependent feature where the rsync is periodically terminated and restarted to allow for recalibration of the upload rate. Since partial transfers are preserved, this should not harm the file upload in progress. And this would help with the use case where local non-backup network needs might increase, at the same time that a long sustained backup was in progress and taking all that bandwidth.

Add some sort of locking scheme (could just be a file touched in /tmp) to prevent multiple instances of the backup/validate/restore from stomping on each others' toes, or worse yet causing archive corruption.

Although the encryption and transfer phase of backups need to be performed 'locally' for security reasons, there is no harm in having the creation of the 0'ed out file via dd performed remotely. So for example we can ssh to remote host and run the dd command there. That could yield a 50x speed increase during this phase of the backup. Currently there is a container being dd'ed the old fashioned way, and will take 16 hours to provision a 50 gig container.

Refactor the code such that the ArchiveR3 module features a verbose attribute which can be set during argument processing, rather than making this an option that gets passed around to various subroutines to let them know they should be verbose.

Code is in place to detect if a given archive is close to reaching capacity, but there is no code yet to actually create a new (larger) archive.

SHA-1 is insecure. Although this probably does not matter given the context, it'd be a good idea to go to SHA-3. However, there is no builtin Python lib for SHA-3, so we'll stick with SHA-1 for now. (Confirm?)

Instead of the absolute size of the container itself.

If for example the backup filesystem is mounted in such a way that newly created files become owned by root instead of the normal user executing the backup script, subsequent operations like attempting to mount the encrypted volume might fail.

Otherwise, prompt to install via pip, or at a minimum show the user what command to run in order to make it happen:

pip install hurry.filesize

Need to fix the timing on the expect script internally, such that it does not accidentally echo the password to the console:

spawn sudo tcplay -c -d /dev/loop0 -a whirlpool -b AES-256-XTS
Passphrase:
Repeat passphrase: asoentusnoaehuntoaehuntoahusnoaehu-files.archive

The rsync call produces output which seems to imply the process is forked. Also, attempting to print it out crashes the program. Get a handle on all output and do something useful with it.

Now that a helper utility has been located to properly tokenize the arguments to Popen, convert the existing shell=True calls to shell=False, because that is the recommended way to do it.

Having just switched the ANSI colors to dark (assuming light background), we should not just assume that everyone has the same sort of terminal. Add a config file option to support dark terminals.

By using the system 'expect' tool, there is a race condition during the mounting of the encrypted container (via 'tcplay -m') where control has not yet been passed by the child process to STDIN before the parent python process issues the password. This results in the password being displayed directly on the screen, and also prevents the backup from proceeding. This race condition has been solved by introducing a 50ms delay automatically in the Python pexpect library.

Make sure the package 'pv' (pipeline viewer) is installed.

Make it so that if an existing container is too small for an archive, the reprovisioning of the container happens automatically if the --auto or --reprovision arguments are specified.

Part of the archive container creation process is to use 'dd' to create a file filled with 0's. When an existing container becomes too small and an increase is needed, take the existing archive and pad with 0's rather than scrapping the entire archive and starting over from scratch.

Encrypting containers which are 100's of gigs in size could conceivably take over a day, so perform a benchmark before getting started so that the user has some sense for how long the operation will take. Command-line tests show that encrypting a 50 meg container should provide a reasonable balance between benchmark time and results accuracy.