Giter VIP home page Giter VIP logo

core's Introduction

Nanbando

CircleCI Scrutinizer Code Quality Code Coverage

Nanbando is a simple application to automate website backups. It provides an elegant way to extend and configure the backup scripts. It was built with modularity, extensibility and simplicity in mind.

Status

This repository will become version 1.0 of Nanbando. It is under heavy development and currently it's APIs and code are not yet stable (pre 1.0).

This is also thought as a replacement for nanbando/core.

Installation

To install the application simply download the executable and move it to the global bin folder.

composer require nanbando/nanbando --dev

Requirements

  • ^7.0
  • symfony ^3.4

For details see composer.json.

core's People

Contributors

luca-rath avatar wachterjohannes avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar

core's Issues

Logging of all things which was done

Implement that with a logging adapter and a process-manager which logs everything. this log file can be written into the local directory and pulled to the server. also a dry-run can be implemented with that.

SSHConnection with rsa-key

The config for password is always present (default in config). The check in ssh-connection is wrong (or false order)

Cleanup command

The cleanup command should clear all the old backups in a configurable way. As an example:

  • Each day for the last week
  • Each week (first backup) for the last month
  • Each Month (first backup) for the last year
  • Each Year (first backup)

Tempdirectory size will have the size of one complete backup

The size will increase while backup is running. If the backup size is 12GB for example the temp-dir will use at least this size. so the minimum free space for a single backup will be the double size of the resulting zip file.

A possible solution could be to implement a FilesystemAdapter with a zip streaming library.

Update doc to fit sulu 1.3 workspace configuration

./bin/console doctrine:phpcr:workspace:export asdf.xml --session default -p /cmf
./bin/console doctrine:phpcr:workspace:export asdf.xml --session default -p /jcr:versions
./bin/console doctrine:phpcr:workspace:export asdf.xml --session live -p /cmf
./bin/console doctrine:phpcr:workspace:export asdf.xml --session live -p /jcr:versions

only interresting for jackrabbit installations.

Add dump-plugin command

This command should print following information:

  • name
  • description
  • dependencies (which cannot be handled by composer)
  • parameters (mandatory, optional and default values)

Presets backup

Plugins should be able to hook into a backup process and extend configuration.

To achieve that the architecture has to be refactored. The whole process should be handled over events.

  • PRE_BACKUP_EVENT:
    • Create destination and source
    • Create and fill system-database (with basic values)
    • (optional - with plugins) The configuration can be extended
  • BACKUP_EVENT: Thrown foreach backup-part
    • Create part-database and write started to database
    • Find plugin
    • Resolve options
    • Call backup from plugin (inclusive exception handling)
    • Write finished, close and write database
  • PRE_BACKUP_EVENT:
    • Write informations to database
    • Close and write system-database
    • Close destination & source

Same for restore.

The preset can hook into the PRE_BACKUP_EVENT. The check command should also be able to extend it. Maybe also call an CHECK_EVENT or a better name which can also be used.

Preset should have a PresetPool hook into the events and hold PresetInterface objects.

This Architecture should enable also other features in the future and make extensions more powerfull.

PHAR package doesn't work with relative local_directory values

When local_directory is defined as relative path in nanbando.json file, i.e.:
"storage": {
"local_directory": "./backups"
}
nanbando backup command throws the following exception:
[League\Flysystem\Exception]
Impossible to create the root directory "./backups".

I've tried cloning nanbando/core and running pure PHP files instead of PHAR and it worked fine:
php ../local-nanbando-copy/bin/nanbando backup

Environment variables

Enable environment variables. One example beside the database would be a environment variable like dev, prod, stage which will be used to generate the path where the backups will be stored. in the restore you should be able to choose also other environments.

Exception handling of plugins

Ask user if backup should be continued and save status in the database. when trying to restore print a warning message.

Process definition for backup-steps

The backup command will have a new options --process files which then only executes the backup-steps which has the process defined. All the other backups will be ignored. The default is all steps.

The backup will have a new database entry with process which will be used in restore command.

{
    "name": "application",
    "backup": {
        "data": {
            "plugin": "directory",
            "process": ["files"],
            "parameter": {
            }
        },
       {
            "plugin": "mysql",
            "process": ["databases"],
            "parameter": {
            }
        }
    }
}

@chirimoya @c00n64 can you add some thoughts?

Update command specific version

The update command should have a (optional) argument where the user can specify which version should be downloaded.

like composer composer self-update 1.2.4

Documentation: Real-life examples (with sketch)

Show how nanbando can be used in real-life examples.

  • How remotes can be used to sync prod-stage
  • How can sync developer the prod in their local dev environment
  • How can ssh-servers be used to backup and download backups from prod-server

Docs: Create ~/.nanbando.yml before running nanbando self-update

Hi,

If you run nanbando self-update (or probably any other nanbando command) without having nanbando.yml in your home directory, you'll get an error:
Fatal error: Uncaught exception 'Symfony\Component\DependencyInjection\Exception\ServiceNotFoundException' with message 'The service "storage" has a dependency on a non-existent service "filesystem.remote".' in phar:///usr/local/bin/nanbando/vendor/symfony/dependency-injection/Compiler/CheckExceptionOnInvalidReferenceBehaviorPass.php on line 58

You could add an additional step to create the file to the docs before running the self-update command or throw an appropriate Exception if the file is not present

I think there are a few possible solutions for this issue:

  • Add an additional step to the docs to make sure that the user adds the file
  • Throw an appropriate Exception if the file is not existent
  • Create the file with default values if it's not there

Sulu Preset

Presets for:

  • Sulu > 1.3 & Jackrabbit & minimal
  • Sulu > 1.3 & Mysql & minimal
  • Sulu > 1.3 & Jackrabbit & standard
  • Sulu < 1.3 & Jackrabbit & standard
  • Sulu > 1.3 & Mysql & standard
  • Sulu < 1.3 & Mysql & standard

Additionally detection of version an options.

Disable auto-completion in `fetch` command

Hi,

I think that the auto-completion in the fetch command is a bit confusing.

Let's assume I have 2 backups.

  1. I enter nanbando fetch
  2. This is displayed:
[0] 11-11-11 2016-11-11
[1] 23-11-25 2016-11-22

Now I want to fetch backup nr. 1 from 2016-11-22. If I type "1", the auto-completion suggests "11-11-11 2016-11-11". If I press enter, it fetches the backup I didn't want to fetch, just because it begins with 1.

selection_172

Simplify configuration

Integrate configuration of flysystem into nanbando.

In this step the oneup/flysystem-bundle can be removed.

Display message when fetching backups

I created a backup on Environment A with a message:
php nanbando.phar backup -m "Automatic backup from build server".

Now I want to apply (restore) this backup on Environment B:
selection_171

The problem is that it's a bit difficult to distinguish the backups now. It would be great if the message of the backup would also be printed in the console.

Also: What exactly is the label argument good for when creating a backup?
This label will be used to generate the filename for the backup.

Is it really used in the filename? or is it "meta information" like the message?

Restore command for directory-plugin

$ ./nanbando.phar restore -vvv
Which backup
  [0] 17-57-37-2016-05-24
 > 0

Backup "test" started will be restored:
 * label:
 * message:
 * started: 2016-05-24T17:57:37+00:00

- src (directory):
  0/28 [>---------------------------]   0% < 1 sec/< 1 sec 12.0 MiB

  [League\Flysystem\FileNotFoundException]
  File not found at path: backup/src/Application/Application.php


Exception trace:
 () at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/vendor/league/flysystem/src/Filesystem.php:381
 League\Flysystem\Filesystem->assertPresent() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/vendor/league/flysystem/src/Filesystem.php:191
 League\Flysystem\Filesystem->readStream() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/src/Core/Plugin/DirectoryPlugin.php:93
 Nanbando\Core\Plugin\DirectoryPlugin->restore() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/src/Core/Nanbando.php:201
 Nanbando\Core\Nanbando->restore() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/src/Bundle/Command/RestoreCommand.php:57
 Nanbando\Bundle\Command\RestoreCommand->execute() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/vendor/symfony/console/Command/Command.php:256
 Symfony\Component\Console\Command\Command->run() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/vendor/symfony/console/Application.php:791
 Symfony\Component\Console\Application->doRunCommand() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/vendor/symfony/console/Application.php:186
 Symfony\Component\Console\Application->doRun() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/vendor/symfony/console/Application.php:117
 Symfony\Component\Console\Application->run() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/bin/nanbando.php:40
 include() at phar:///Users/johannes/Development/nanbando/test/nanbando.phar/bin/nanbando-phar-stub:16
 require() at /Users/johannes/Development/nanbando/test/nanbando.phar:11

restore

$ unzip -l ~/nanbando/local/test/17-57-37-2016-05-24.zip
Archive:  /Users/johannes/nanbando/local/test/17-57-37-2016-05-24.zip
  Length     Date   Time    Name
 --------    ----   ----    ----
        0  05-24-16 19:57   backup/src/Application/
     1588  05-24-16 19:57   backup/src/Application/Application.php
     4006  05-24-16 19:57   backup/src/Application/Kernel.php
        0  05-24-16 19:57   backup/src/Bundle/Command/
      829  05-24-16 19:57   backup/src/Bundle/Command/BackupCommand.php
     1950  05-24-16 19:57   backup/src/Bundle/Command/FetchCommand.php
     2253  05-24-16 19:57   backup/src/Bundle/Command/InfoCommand.php
      649  05-24-16 19:57   backup/src/Bundle/Command/InstallCommand.php
     1180  05-24-16 19:57   backup/src/Bundle/Command/PushCommand.php
     2433  05-24-16 19:57   backup/src/Bundle/Command/RebuildCommand.php
     1959  05-24-16 19:57   backup/src/Bundle/Command/RestoreCommand.php
      612  05-24-16 19:57   backup/src/Bundle/Command/RollbackCommand.php
     1253  05-24-16 19:57   backup/src/Bundle/Command/SelfUpdateCommand.php
      645  05-24-16 19:57   backup/src/Bundle/Command/UpdateCommand.php
        0  05-24-16 19:57   backup/src/Bundle/DependencyInjection/Compiler/
     2479  05-24-16 19:57   backup/src/Bundle/DependencyInjection/Compiler/RegisterKernelListenersPass.php
        0  05-24-16 19:57   backup/src/Bundle/DependencyInjection/
     1489  05-24-16 19:57   backup/src/Bundle/DependencyInjection/Configuration.php
     1958  05-24-16 19:57   backup/src/Bundle/DependencyInjection/NanbandoExtension.php
        0  05-24-16 19:57   backup/src/Bundle/
      649  05-24-16 19:57   backup/src/Bundle/NanbandoBundle.php
        0  05-24-16 19:57   backup/src/Bundle/Resources/config/
     2206  05-24-16 19:57   backup/src/Bundle/Resources/config/services.xml
        0  05-24-16 19:57   backup/src/Core/Backup/
       58  05-24-16 19:57   backup/src/Core/Backup/Backup.php
        0  05-24-16 19:57   backup/src/Core/Config/
     1092  05-24-16 19:57   backup/src/Core/Config/JsonLoader.php
        0  05-24-16 19:57   backup/src/Core/Database/
      687  05-24-16 19:57   backup/src/Core/Database/Database.php
     1135  05-24-16 19:57   backup/src/Core/Database/ReadonlyDatabase.php
        0  05-24-16 19:57   backup/src/Core/Flysystem/
     4563  05-24-16 19:57   backup/src/Core/Flysystem/PrefixAdapter.php
        0  05-24-16 19:57   backup/src/Core/
     7490  05-24-16 19:57   backup/src/Core/Nanbando.php
        0  05-24-16 19:57   backup/src/Core/Plugin/
     2887  05-24-16 19:57   backup/src/Core/Plugin/DirectoryPlugin.php
     1068  05-24-16 19:57   backup/src/Core/Plugin/PluginInterface.php
      421  05-24-16 19:57   backup/src/Core/Plugin/PluginRegistry.php
        0  05-24-16 19:57   backup/src/Core/Symfony/
     1880  05-24-16 19:57   backup/src/Core/Symfony/CollectorCompilerPass.php
        0  05-24-16 19:57   backup/src/Core/Temporary/
     1199  05-24-16 19:57   backup/src/Core/Temporary/TemporaryFileManager.php
        0  05-24-16 19:57   database/backup/
       91  05-24-16 19:57   database/backup/src.json
        0  05-24-16 19:57   database/
      131  05-24-16 19:57   database/system.json
 --------                   -------
    50840                   46 files

Extract system database when pushing to remote storage

When pushing data to a remote storage the system database should be pushed individually (e.g. application/2017-04-03-12-00-00.json) to enable read easily the configuration of the backup (over flysystem).

This feature is needed to enable following features:

  • Cleanup Command #89
  • Incremental Backup #84

Incremental backup

Backup should be able to define "previous" backup. The backup command will have a new options --incremental. In the archive only new file will be saved and a list of deleted files.

This has to be handled in:

  1. Restore: A ChainAdapter which receives a list of adapters in the right order and go thruh them foreach function in AdapterInterface.
  2. Fetch & Get: load also the dependencies
  3. Information: should display all the dependencies

@chirimoya @c00n84 can you add some thoughts?

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.