Giter VIP home page Giter VIP logo

gcloud-backup's Introduction

PROJECT DESCRIPTION

This project first aim was to provide an easy-to-setup and ready-to-deploy script (without any or minimal dependencies) to sync backups to Google's Cloud Storage. The first versions were a simple script with little to no configuration at all, to upload file's to the cloud and remove them when they were old enough. But more and more features were needed and nowadays it's a very complete tool to upload and manage your backup files to Gcloud.

TLDR;

This project provides an easy-to-setup and ready-to-deploy script to sync backups to Google's Cloud Storage.

FEATURES

  • Upload your backups to Gcloud
  • Remove Old Backups (You may define what "old" means to you [in days]) from the cloud
  • AutoClean Old logs of past year's (Logs are important, but why would you want to keep logs forever and ever?. All of this is optional of course.)
  • Mailing feature so you can see what the application is doing or has been doing and you need not to be present at the moment.
    • The script mails every time it uploads a folder, sends an email if any error ocurred and attaches the error logging file for further details and sends a log of what has been deleted when the option -removeOld it's provided.
  • DryRun: Want to test out uploading or removing what you had already in your bucket, do not want to delete your cloud file's without checking if everything looks good? Try -dryRun so you can see what is going to be done without actually doing it.
    • (Important Note: You actually "don't see anything" because everything goes to the log, even when -dryRun is enabled, because this script is intended to run -unattended.)
  • CygWin feature thanks to @HieiOne that stops the Window's Google SDK from limiting your total speed aproximately at 4.5 MiB/s (real bs if you ask me...). So you may use CygWin to use the Linux's Google SDK that works nicely and smoothly as it should.
  • Mount Share (NEW) feature enables you to mount the share internally within the script so to avoid windows scheduled tasks typical error of "Cannot find the device X:...".

FUTURE IDEAS

  • Easy management of saved credentials
  • Write logs to Windows Event System
  • Port to C# .NET Core

PARAMETERS

  • Action parameters:
    • Default (aka no action parameters): --> doUpload.
    • -clean: Executes action --> AutoClean.
    • -removeOld: Executes action --> Remove Old Backups
    • -all: Executes all actions in order --> autoClean, doUpload, removeOldBackups
  • Optional parameters:
    • -dryRun: Performs a test of the action you selected.
    • -unattended: When the script needs user input won't wait for it when this flag is on, so the script will still run. But the warning/error will be logged in the log file. (This might happen for example, if you enabled mailing feature but haven't generated the credentials yet. If you don't -unattended, the script will understand you're attending to the script's behaviour and ask for credentials, instead, if you --unattended the script will log missing credentials to the log and disable temporarily the mailing feature.)
    • -genCreds: Script runs only to create the credentials.
    • -confFile: You may specify a different config file in case you're testing something out or in case you'd like to have different configurations for different servers or remote shares.

HOW TO USE CYGWIN

The CygWin implementation must have the Gcloud SDK configured, plus all the dependencies the SDK has. How to prepare CygWin to work properly:

IMPORTANT! In case you have an old installation in the OS of SDK you have to remove it from the Path (and uninstall it if you wish)

  • Packages for cygwin: wget, curl, gcc-core, python27, python27-devel, python27-pip, python27-setuptools
  • Download the SDK with wget https://dl.google.com/dl/cloudsdk/channels/rapid/downloads/google-cloud-sdk-287.0.0-linux-x86_64.tar.gz
  • Extract with tar -zxvf google-cloud-sdk-287.0.0-linux-x86_64.tar.gz
  • Add to the path with export PATH=$PATH:~/google-cloud-sdk/bin and update the CygWinSDKPath variable (if its different than in home directory)
  • Also install CRCMOD with pip2 install crcmod
  • Run gcloud init to finish with the installation

UPDATE NOTES

Important: as of version 9.1.2b: Log folder name changed to "Logs" instead of GcloudLogs, if you still want to use the old naming system you're free to change it in the Conf file.

Update! Version 9.4b now you can use a Cygwin installation to run the gsutil commands increasing perfomance (Some newer versions of windows showed a big performance decrease with Google's SDF of Windows) and upload speed (steps to do this in the config file)

DISCLAIMER

This software is provided "as is" with absolute no warranty. Consider that this software it's still in the beta version and although it's very stable and has no known problems, if you detect a bug, you either can issue an #issue here at github or you can fix it yourself if you want and create a pull request.

gcloud-backup's People

Contributors

alex13lopez avatar hieione avatar

Stargazers

 avatar  avatar  avatar  avatar

Watchers

 avatar

Forkers

hieione

gcloud-backup's Issues

Write logs to Windows Event System

This might be useful for professional environments with Nagios or any other System Monitor, so we can read those logs directly and monitor gcloud-backup more effectively and precisely.

Indicate Conf File through parameter

It'd be nice to be able to indicate the path of the conf file so we can have several conf files for different backups or for different purposes.

Better format for emails

It would be nice to have som email formatting to represent more what's happening instead of a plain text message. Something like Veeam has:

What we have:
image

What we could have:

image

Creation Date of file from Gcloud instead of file

Right now the date of the files is fetched from the filename, as how is the code atm, it would work for Veeam 10, 9 and possibly 8, but there might be changes with the naming convention from the veeam team, not only that, but this could be open for other backup solutions.

For that I think the best idea would be to have a dictionary with the key being the version and program and the value the filename format.

Slow upload in Windows

Hello there,

I have been detecting slow uploads in different clients with the same S.O , aka Windows 10, after quite a long research I've detected that the only way to fix it (apart from waiting for Google to fix it) is installing Cygwin and running the gsutil commands through Cygwin.

I say Cygwin and not WSL because Cygwin has the ability to connect to Windows Shares and that might be necessary for some scenarios (at the very least in mine).

As an example, a client of mine has a bandwidth of 500 Mb and the uploads are going at 4.5 MiB instead of 50 like it should, once I installed and tested it with Gsutil cp , the speed increased to the expected.

Bug with getting date if the name has numbers

Known bug that happens if the folder you're uploading has numbers, those numbers will interfere and be recognised as a date when they're not.

As I see it, there's 2 ways of solving this:

  1. Get Modified date for each file from Gcloud.
  2. Remove the folder's name of the string of file's path and then get the date as usual.

The 1st option would be the "pro one" but I'm concerned about performance since you have to get each file's date and 2nd one is easier and does not affect performance.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.