gobackup / gobackup Goto Github PK

View Code? Open in Web Editor NEW

1.3K 22.0 210.0 658 KB

🗄 CLI tool for backup your databases, files to cloud storages in schedully.

Home Page: https://gobackup.github.io

License: MIT License

Go 91.23% Makefile 0.19% Dockerfile 1.15% Shell 0.89% HTML 0.22% JavaScript 0.21% TypeScript 5.34% SCSS 0.77%

backup go postgresql mysql redis cli

gobackup's Introduction

GoBackup

CLI tool for backup your databases, files to cloud storages in schedully.

GoBackup is a backup tool design for application servers, to backup your databases, files to cloud storages (FTP, SCP, S3, GCS, Aliyun OSS ...) in schedully.

Simple, easy to use, one time setup, run years without any maintenance, low cost (recycle), secure (encrypt compress).

Inspired by backup/backup and replace it for without Ruby dependency.

GoBackup 是一个为中小型服务器而设计备份工具，内置计划任务，帮你定时备份数据库、配置文件，通过导出、打包压缩，最终存储到外部云存储（如：FTP、SCP、S3、GCS、Aliyun OSS ...）。

简单易用，一次性部署后能持续运行数年无需任何维护，存储成本低（回收机制），安全可靠（加密压缩）。

🎈 Ruby China 的应用服务器从 GoBackup 设计之初（2017 年）开始就每日全量备份到外部的云存储，并持续数年一直默默的备份着，完全不需要维护。Aliyun OSS 上，每月成本仅为 0.2 RMB（文件回收周期为 1 个月）。

https://gobackup.github.io

Features

No dependencies.
Multiple Databases source support.
Multiple Storage type support.
Archive paths or files into a tar.
Split large backup file into multiple parts.
Run as daemon to backup in schedully.
Web UI to manage backups.

Databases

MySQL
PostgreSQL
Redis
MongoDB
SQLite
Microsoft SQL Server
InfluxDB
MariaDB
etcd

Storages

Notifier

since: 1.5.0

Send notification when backup has success or failed.

Mail (SMTP)
Webhook
Discord
Slack
Feishu
DingTalk
GitHub (Comment on Issue)
Telegram
AWS SES
Postmark
SendGrid

Installation

curl -sSL https://gobackup.github.io/install | sh

after that, you will get /usr/local/bin/gobackup command.

Install via Homebrew

brew install gobackup

$ gobackup -h
NAME:
   gobackup - Backup your databases, files to FTP / SCP / S3 / GCS and other cloud storages.

USAGE:
   gobackup [global options] command [command options] [arguments...]

VERSION:
   1.3.0

COMMANDS:
   perform
   start    Start as daemon
   run      Run GoBackup
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --help, -h     show help (default: false)
   --version, -v  print the version (default: false)

Configuration

GoBackup will seek config files in:

~/.gobackup/gobackup.yml
/etc/gobackup/gobackup.yml

Example config: gobackup_test.yml

models:
  gitlab_app:
    databases:
      gitlab_db:
        type: postgresql
        database: gitlab_production
        username: gitlab
        password:
      gitlab_redis:
        type: redis
        mode: sync
        rdb_path: /var/db/redis/dump.rdb
        invoke_save: true
    storages:
      s3:
        type: s3
        bucket: my_app_backup
        region: us-east-1
        path: backups
        access_key_id: $S3_ACCESS_KEY_Id
        secret_access_key: $S3_SECRET_ACCESS_KEY
    compress_with:
      type: tgz

Usage

Perform backup

$ gobackup perform

Backup schedule

GoBackup built in a daemon mode, you can use gobackup start to start it.

You can configure the schedule for each models, it will run backup task at the time you set.

For example

Configure your schedule in gobackup.yml

models:
  my_backup:
    before_script: |
      echo "Before script"
    after_script: |
      echo "After script"
    schedule:
      # At 04:05 on Sunday.
      cron: "5 4 * * sun"
    storages:
      local:
        type: local
        path: /path/to/backups
    databases:
      mysql:
        type: mysql
        host: localhost
        port: 3306
        database: my_database
        username: root
        password: password
  other_backup:
    # At 04:05 on every day.
    schedule:
      every: "1day",
      at: "04:05"
    storages:
      local:
        type: local
        path: /path/to/backups
    databases:
      mysql:
        type: mysql
        host: localhost
        port: 3306
        database: my_database
        username: root
        password: password

Start Daemon & Web UI

GoBackup bulit a HTTP Server for Web UI, you can start it by gobackup start.

It also will handle the backup schedule.

$ gobackup start

2023/03/15 23:00:30 [Config] Load config from default path.
Starting API server on port http://127.0.0.1:2703

NOTE: If you wants start without daemon, use gobackup run instead.

Now visit http://127.0.0.1:2703 you can see the Web UI:

Signal handling

GoBackup will handle the following signals:

HUP - Hot reload configuration.
QUIT - Graceful shutdown.

$ ps aux | grep gobackup
jason            20443   0.0  0.1 409232800   8912   ??  Ss    7:47PM   0:00.02 gobackup run

# Reload configuration
$ kill -HUP 20443
# Exit daemon
$ kill -QUIT 20443

Contributing

The DEVELOPMENT document will help you to setup development environment, and guide you how to test them in local.

When you finish your work, please send a PR.

License

MIT

gobackup's People

Contributors

Stargazers

Watchers

Forkers

greatghoul gotoolkits jonike fossabot daniel-007 shiliuxing wozzup disunbow admpub joyitools runningbruce wppurking devopsmi songjiayang allchain slimus rhinoceros subcrew mind029 merifri sp213 lhyfree huiwang85 kongminhao atouhou zsjtoby elenw priestd09 wcasanova phpip tdonovic antuspenskiy golook alexmourapb jianhaiqing justin168 ruhip yehuangcn golandr gaecom jackerzhou cxnky noandrea rimusz-lab lisuiheng boluow griffinqiu mooncreeks thy0108 master2sid2 opsfarmer jkylin waterbolik minlison shenhi233 torns monsterooo abowloflrf dreitier enng0227 qingchensecai liangshiyuan cugblack go-app-cloud delbertwong vbatushev auzty cnopens geekhuyang wb1t trytrytogo gc888 imwildcat careylipu mmuller88 sonnt85 ttys3 wangliyao sharp irc-friends dnnta edwinhzqhei yashodhank mankaihuang iii80 allanpk716 logicall nickgal dystudio polarrwl mygomod wueason mars-leen dance-show gltsc tom2jack ryan817 bryant-rh changchichung toannd96

gobackup's Issues

Postgres asks for password every time

This is my configuration file.

models:
  yizaoyiwan:
    compress_with:
      type: tgz
    store_with:
      type: local
      path: /home/deploy/.gobackup/backups
    databases:
      yizaoyiwan:
        type: postgresql
        database: yizaoyiwan
        host: localhost
        username: postgres
        password: xxxx
    archive:
      includes:
        - /home/deploy/.bashrc

if I manually set PGPASSWORD in my ~/.bashrc, it works and continues to perform the backup.

https://github.com/huacnlee/gobackup/blob/master/database/postgresql.go#L77

/bin/tar: file changed as we read it

deploy@localhost:~/.gobackup$ gobackup perform
2017/10/12 15:16:45 ======== yizaoyiwan ========
2017/10/12 15:16:45 WorkDir: /tmp/gobackup/1507821405731845154/yizaoyiwan

2017/10/12 15:16:45 ------------- Databases -------------
2017/10/12 15:16:45 => database | PostgreSQL: yizaoyiwan
2017/10/12 15:16:45 -> Dumping PostgreSQL...
2017/10/12 15:16:46 dump path: /tmp/gobackup/1507821405731845154/yizaoyiwan/postgresql/yizaoyiwan/yizaoyiwan.sql
2017/10/12 15:16:46
2017/10/12 15:16:46 ------------- Databases -------------

2017/10/12 15:16:46 ------------- Archives -------------
2017/10/12 15:16:46 => includes 1 rules
2017/10/12 15:16:46 ------------- Archives -------------

2017/10/12 15:16:46 ------------ Compressor -------------
2017/10/12 15:16:46 => Compress | tgz
2017/10/12 15:16:46 [debug] /bin/tar zcf /tmp/gobackup/1507821405731845154/yizaoyiwan/2017.10.12.23.16.46.tar.gz yizaoyiwan
2017/10/12 15:16:46 /bin/tar: yizaoyiwan: file changed as we read it

2017/10/12 15:16:46 Cleanup temp dir...

2017/10/12 15:16:46 ======= End yizaoyiwan =======

2017/10/12 15:16:46 /bin/tar: yizaoyiwan: file changed as we read it

Show this error and no backup file created in target folder.

gobackup version 0.5.0
config file same to #7

Add Dropbox support

https://www.dropbox.com/features/cloud-storage

Update install script '0.7.1' - '0.7.2'

gobackup.yml 配置：

models:
  example:
    compress_with:
      type: tgz
    store_with:
      type: oss
      max_retries: 5
      timeout: 300
      threads: 1 (1 .. 100)
    archive:
      includes:
        - /home/git/.ssh/
        - /etc/nginx/nginx.conf
        - /etc/nginx/conf.d
      excludes:
        - /home/ubuntu/.ssh/known_hosts
        - /etc/logrotate.d/syslog

现象：

设置 cronjob 每日备份，gobackup 会在 /tmp/gobackup 目录下生成对应每日备份文件的压缩文件。我设置的对象存储的保存策略，但本地备份的压缩文件不断积累，需要手动清理。

需求：

新增配置，支持备份对象存储后本地文件直接清除。
支持本地缓存文件的最大数量，支持滚动清理。

can gobackup exclude subdirectories ?

Thanks gobackup author，Can gobackup exclude subdirectories ? such as include the path /home/wwwroot/test.cn, exclude the path home/wwwroot/test.cn/upload , the following is my configuration file：

 ## 源码备份
  sourceBackup:
    # 备份打包
    compress_with:
      # 压缩方式
      type: tgz
    # 存储
    store_with:
      # 存储到oss上
      ........
    # 文件备份
    archive:
      # 需要备份的文件、文件夹
      includes:
        - /home/wwwroot/test.cn
      excludes:
        - /home/wwwroot/test.cn/upload

please consider using pigz or gzip with multiple cores and split the backup file

thanks for the wonderful tool .
but during the compress process , gobackup only single core to run gzip , is it possible make this a configurable parameter in configuration file ?
or replace gzip with pigz for use multi cores to compress the backup file ?

and after compress , the backup file is a huge tgz , is it possible to make an option to split the backup file ?

How to dump once and save to multiple places?

I want to backup mysql, and the yml config just has 1 store_with i can set.

How to dump once and save to local and s3?

I don't want to set 2 models in config file.

Thank you.

TLS FTP not working

Cant upload files to ftp with TLS, any solution?

不会自动递归创建临时目录，导致打包失败

不会自动递归创建临时目录，导致打包失败，具体如下

gobackup 版本 0.4.2
配置文件如下：

models:
  nexus:
    store_with:
      type: scp
      host: 17.17.8.1
      path: /Raid/xx/nexus
      private_key: /home/xx/.ssh/id_rsa
      username: xx
    archive:
      includes:
        - /data/nexus

执行结果：

[root@psd008 gobackup]# gobackup perform
2017/10/10 11:59:23 ======== nexus ========
2017/10/10 11:59:23 WorkDir: /tmp/gobackup/1507636763978943088/nexus

2017/10/10 11:59:23 ------------- Databases -------------
2017/10/10 11:59:23 ------------- Databases -------------

2017/10/10 11:59:23 ------------- Archives -------------
2017/10/10 11:59:23 => includes 1 rules
2017/10/10 11:59:23 [debug] /bin/tar -cPf /tmp/gobackup/1507636763978943088/nexus/archive.tar /data/nexus
2017/10/10 11:59:23 ------------- Archives -------------

2017/10/10 11:59:23 ------------ Compressor -------------
2017/10/10 11:59:23 => Compress with Tgz...
2017/10/10 11:59:23 [debug] /bin/tar zcf /tmp/gobackup/2017.10.10.19.59.23.tar.gz nexus
2017/10/10 11:59:23 /bin/tar: nexus: Cannot stat: No such file or directory
/bin/tar: Exiting with failure status due to previous errors

2017/10/10 11:59:23 Cleanup temp dir...

2017/10/10 11:59:23 ======= End nexus =======

阿里云存储，如果 path 内带斜杠，会生成空目录

path 带斜杠，会给阿里云生成，一个斜杠空目录，删也删不掉，不知道是哪边的问题了，阿里云工单工程师也蒙圈了。

经过探索，发现：设置阿里云存储 path: /xxx/ 就会有问题，试验发现设置 path: xxx，就好了

阿里云那边：
默认进到控制台
https://oss.console.aliyun.com/bucket/oss-cn-hangzhou/aaa-backup/
看到的是这样的
一个斜杠目录，点击是无限循环，删又删不掉....

自己改改 url，如下：
https://oss.console.aliyun.com/bucket/oss-cn-hangzhou/aaa-backup/object?path=/
能看到了，还是删不掉，下载不了

ssh: must specify HostKeyCallback

This situation occurs when attempting to use scp storage, with user/password authentication (no private key):

2018/05/19 23:51:38 ------------- Storage --------------
2018/05/19 23:51:38 => Storage | scp
2018/05/19 23:51:38 PrivateKey /root/.ssh/id_rsa
2018/05/19 23:51:38 ssh: must specify HostKeyCallback

ssh: must specify HostKeyCallback

Add Google Cloud Storage

https://github.com/googleapis/google-cloud-go/tree/main/storage

Config path

Need the ability for set config path manually like gobackup --config path_to_condig/gobackup.yml

get an error

solved

Storage to S3

Storage to Aliyun OSS

Support webdav storage

Bash Install 脚本需要兼容 root 用户的场景

not clean tmp dir when exec on root permission and one job store to different destination

Amazon Linux AMI latest
gobackup -v 0.6.1

用于自动备份docker的存储，docker -v指定的目录所有者为root
所以在执行perform时使用sudo来执行
执行完毕后查看tmp/gobackup目录下的临时文件夹和tar打包文件没有被清理，需要手动清理

另，
指定一个任务时，存储目的地貌似只识别最后一项store
我有一项本地备份任务，希望在存储时，同时上传至多个不同的目的地，比如阿里OSS和aws s3。
如果拆分为两个任务，则会打包两次，生成两个压缩包和临时目录
结合上一个root下执行没有清理的问题，磁盘容量消耗非常大

请问有没有什么解决方法？谢谢~

Support restoring

Backup is useless if you can't conveniently restore from the archives. I know it's doable by invoking some shell commands to un-tar the archive and importing database dumps, but that is not very intuitive, let along there's no documentation at all.

I'd suggest adding a restore subcommand to allow users to selectively restore some part of the data.
Another solution would be to generate restore scripts (generic shell scripts) when perform finished.

Problem with run backup

Hello. I have the next problem. I run the script for run backup (centos7)
/usr/local/bin/gobackup perform >> ~/.gobackup/gobackup.log
Backup doesn't run and log files clean.

Maybe you know where I can find debug or etc?

FTP 配置的 host 兼容 ftp:// 前缀的场景

能添加一点编译说明吗？

您好，我对go不大熟悉，我直接下载代码后编译，提示如下错误，麻烦看下能给点编译说明吗，多谢。

Allow mongo backup without password

Mongodump asks for password on CLI and fails every time when password is not provided.

打印日志默认使用0时区

打印的日志默认显示的是0时区。
当前系统是东8区，3点，结果在日志中显示的是19点

compile error on arm

try to compile on arm server. i got this error ,

root@y2t-web:~/go/src/github.com/huacnlee/gobackup# go build

github.com/bramvdbogaerde/go-scp/auth

../../bramvdbogaerde/go-scp/auth/key.go:10: undefined: ssh.HostKeyCallback

gobackup not working with read-only home directory

I'm trying to run gobackup on a webhost with limited permissions. Unfortunately, I don't write permissions for the whole home directory (~), but only for a sub-directory.

gobackup tries to write to a file called ~/.gobackup/cycler/.json, but fails and exits. Here is the log:

2018/11/23` 19:21:19 ------------- Databases -------------
2018/11/23 19:21:19 => database | mysql : wordpress
2018/11/23 19:21:19 -> Dumping MySQL...
2018/11/23 19:21:24 dump path: /tmp/gobackup/1542997279119109937/wordpress/mysql/wordpress
2018/11/23 19:21:24
2018/11/23 19:21:24 ------------- Databases -------------

2018/11/23 19:21:24 ------------- Archives -------------
2018/11/23 19:21:24 => includes 1 rules
2018/11/23 19:21:24 ------------- Archives -------------

2018/11/23 19:21:24 ------------ Compressor -------------
2018/11/23 19:21:24 => Compress | tgz
2018/11/23 19:21:27 -> /tmp/gobackup/2018.11.23.19.21.24.tar.gz
2018/11/23 19:21:27 ------------ Compressor -------------

2018/11/23 19:21:27 ------------- Storage --------------
2018/11/23 19:21:27 => Storage | scp
2018/11/23 19:21:27 PrivateKey /home/www/p455475/.ssh/id_rsa
2018/11/23 19:21:27 -> scp /2018.11.23.19.21.24.tar.gz
2018/11/23 19:21:27 Store successed
2018/11/23 19:21:27 Load cycler.json failed:open /home/www/p455475/.gobackup/cycler/wordpress.json: no such file or directory
2018/11/23 19:21:27 Skip save cycler.json because it not loaded
2018/11/23 19:21:27 ------------- Storage --------------

2018/11/23 19:21:27 Cleanup temp dir:/tmp/gobackup...

2018/11/23 19:21:27 ======= End wordpress =======

Also, it says storing the files via SCP succeeded, but that's not actually true, i.e. no file is created.

支持windows吗？

Add Azure Storage support

https://docs.microsoft.com/en-us/azure/storage/common/storage-introduction

Upload speed limit

For the small Website cases, they are uses VPS may has a low network bandwidth (~2m - 5m).

If gobackup does't limit upload speed, the backup progress will block the network, then the website will not open.

Docker image support?

Hi, do you plan to support Docker image? For example, adding a Dockerfile and publish it to hub.docker.com?

手动执行gobackup perform成功，但是定时任务提示找不到mysqldump

[root@iZwz9ccgakmnlprso2kzxlZ ~]# crontab -l
0 0 * * * /usr/local/bin/gobackup perform >> ~/.gobackup/gobackup.log
[root@iZwz9ccgakmnlprso2kzxlZ ~]# cat ~/.gobackup/gobackup.log
2019/07/10 00:00:01 ======== mysql ========
2019/07/10 00:00:01 WorkDir: /tmp/gobackup/1562688001676054782/mysql

2019/07/10 00:00:01 ------------- Databases -------------
2019/07/10 00:00:01 => database | mysql : master
2019/07/10 00:00:01 -> Dumping MySQL...
2019/07/10 00:00:01 -> Dump error: mysqldump cannot be found
2019/07/10 00:00:01 Cleanup temp dir:/tmp/gobackup...

2019/07/10 00:00:01 ======= End mysql =======

[root@iZwz9ccgakmnlprso2kzxlZ ~]# mysqldump
Usage: mysqldump [OPTIONS] database [tables]
OR mysqldump [OPTIONS] --databases [OPTIONS] DB1 [DB2 DB3...]
OR mysqldump [OPTIONS] --all-databases [OPTIONS]
For more options, use mysqldump --help
[root@iZwz9ccgakmnlprso2kzxlZ ~]#
[root@iZwz9ccgakmnlprso2kzxlZ ~]# mysql --version
mysql Ver 15.1 Distrib 10.1.21-MariaDB, for Linux (x86_64) using readline 5.1
[root@iZwz9ccgakmnlprso2kzxlZ ~]# mysqldump --version
mysqldump Ver 10.16 Distrib 10.1.21-MariaDB, for Linux (x86_64)
[root@iZwz9ccgakmnlprso2kzxlZ ~]#
[root@iZwz9ccgakmnlprso2kzxlZ ~]# gobackup --version
gobackup version 0.8.0

Gobackup not deleting temporary files

Looks like gobackup saves files at e.g. /tmp/gobackup/2020.09.30.18.01.06.tar.gz, but only deletes /tmp/gobackup/1601481601969223442

As a workaround, I've changed my cronjob to /usr/local/bin/gobackup perform >> /var/log/gobackup.log && rm -rf /tmp/gobackup (note the rm command at the end.)

/tmp/gobackup/ ls -la
total 236273360
drwxr-xr-x  2 root root        4096 Sep 30 18:31 .
drwxrwxrwt 12 root root        4096 Sep 30 06:31 ..
-rw-r--r--  1 root root 19797375857 Sep 27 20:13 2020.09.27.19.53.24.tar.gz
-rw-r--r--  1 root root 19832908870 Sep 28 00:20 2020.09.28.00.01.01.tar.gz
-rw-r--r--  1 root root 19872942935 Sep 28 06:20 2020.09.28.06.00.57.tar.gz
-rw-r--r--  1 root root 19961177560 Sep 28 12:22 2020.09.28.12.01.14.tar.gz
-rw-r--r--  1 root root 20056435330 Sep 28 18:21 2020.09.28.18.01.08.tar.gz
-rw-r--r--  1 root root 20233341709 Sep 29 00:21 2020.09.29.00.01.03.tar.gz
-rw-r--r--  1 root root 20292021283 Sep 29 06:22 2020.09.29.06.01.02.tar.gz
-rw-r--r--  1 root root 20317139957 Sep 29 12:21 2020.09.29.12.01.07.tar.gz
-rw-r--r--  1 root root 20685865291 Sep 30 00:21 2020.09.30.00.01.10.tar.gz
-rw-r--r--  1 root root 20742438814 Sep 30 06:25 2020.09.30.06.03.45.tar.gz
-rw-r--r--  1 root root 20947585039 Sep 30 12:21 2020.09.30.12.01.11.tar.gz
-rw-r--r--  1 root root 19204609375 Sep 30 18:20 2020.09.30.18.01.06.tar.gz

2020/09/30 18:00:01 ======== main ========
2020/09/30 18:00:01 WorkDir: /tmp/gobackup/1601481601969223442/main

2020/09/30 18:00:01 ------------- Databases -------------
2020/09/30 18:00:01 => database | redis : redis
2020/09/30 18:00:01 -> Invoke save...
2020/09/30 18:00:01 Copying redis dump to /tmp/gobackup/1601481601969223442/main/redis/redis
2020/09/30 18:01:06
2020/09/30 18:01:06 ------------- Databases -------------

2020/09/30 18:01:06 ------------ Compressor -------------
2020/09/30 18:01:06 => Compress | tgz
2020/09/30 18:20:52 -> /tmp/gobackup/2020.09.30.18.01.06.tar.gz
2020/09/30 18:20:52 ------------ Compressor -------------

2020/09/30 18:20:52 ------------- Storage --------------
2020/09/30 18:20:52 => Storage | s3
2020/09/30 18:20:52 -> S3 Uploading...
2020/09/30 18:23:46 => https://censored.s3.eu-central-1.amazonaws.com/backup%2F2020.09.30.18.01.06.tar.gz
2020/09/30 18:23:46 ------------- Storage --------------

2020/09/30 18:23:46 Cleanup temp dir:/tmp/gobackup/1601481601969223442...

models:
  main:
    databases:
      redis:
        type: redis
        mode: copy
        rdb_path: /var/lib/keydb/dump.rdb
        invoke_save: false
    compress_with:
      type: tgz
    store_with:
      type: s3
      keep: 20
      bucket: censored
      region: eu-central-1
      path: backup
      access_key_id: censored
      secret_access_key: censored

scp out of memory 异常

如果备份文件很大，使用scp时会报内存溢出。

2017/10/11 09:14:56 -> Connecting...
2017/10/11 09:14:57 -> scp /Raid/psdbackup/nexus/2017.10.11.16.52.28.tar.gz
fatal error: runtime: out of memory

runtime stack:
runtime.throw(0xadea51, 0x16)
        /usr/local/go/src/runtime/panic.go:605 +0x95
runtime.sysMap(0xc820400000, 0x400000000, 0xc420001c00, 0xed1b18)
        /usr/local/go/src/runtime/mem_linux.go:216 +0x1d0
runtime.(*mheap).sysAlloc(0xeb88a0, 0x400000000, 0xc420037e20)
        /usr/local/go/src/runtime/malloc.go:470 +0xd7
runtime.(*mheap).grow(0xeb88a0, 0x200000, 0x0)
        /usr/local/go/src/runtime/mheap.go:887 +0x60
runtime.(*mheap).allocSpanLocked(0xeb88a0, 0x200000, 0xed1b28, 0x0)
        /usr/local/go/src/runtime/mheap.go:800 +0x334
runtime.(*mheap).alloc_m(0xeb88a0, 0x200000, 0xffffffffffff0101, 0xc420037ef0)
        /usr/local/go/src/runtime/mheap.go:666 +0x118
runtime.(*mheap).alloc.func1()

同时执行多个任务时 /tmp/gobackup 被删除导致备份失败

当在同一host上面执行 gobackup perform --model model1; gobackup perform --model model2; 当其中一个任务完成就会删除其它任务正在使用的/tmp/gobackup目录，导致其它任务备份失败，比如在crontab当时定义了多个任务的场景。

Feature Request: ExpandEnv

Background

To use gobackup with Kubernetes or other tools, we need to pass credentials dynamically via environment variable.

Currently, the yaml config file is kind of static. We could leverage os.ExpandEnv to enable the support for environment variable.

Unmarshal cycler.json failed:unexpected end of JSON input

When performing "gobackup perform" for the first time ever:

2018/05/19 23:15:15 ------------- Storage --------------
2018/05/19 23:15:15 => Storage | local
2018/05/19 23:15:15 Store successed /root/backups
2018/05/19 23:15:15 Unmarshal cycler.json failed:unexpected end of JSON input
2018/05/19 23:15:15 ------------- Storage --------------

The second time I run "gobackup perform" no longer throws this error, so I guess this is a low priority bug.

阿里云同样的 access_id 和 secret 在 carrierwave-aliyun 能用，gobackup 用报错

2019/01/01 16:18:52 oss: service returned error: StatusCode=400, ErrorCode=InvalidArgument, ErrorMessage=Authorization header is invalid., RequestId=5C0B226C2FF16CF75093E762

gobackup.yml:

# gobackup config example
# -----------------------
# Put this file in follow place:
# ~/.gobackup/gobackup.yml or /etc/gobackup/gobackup.yml
models:
  yfxs:
    compress_with:
      type: tgz
    store_with:
      type: oss
      keep: 10
      endpoint: oss-cn-hangzhou-internal.aliyuncs.com
      bucket: aaa
      accessKeyID: aaa
      accessKeySecret: aaa
      path: /aaa
    databases:
      aaa_production:
        type: postgresql
        host: localhost
        port: 5432
        database: aaa
        username: aaa
        password: aaa
    archive:
      includes:
        - /application/app/aaa/shared/etc/nginx.conf
        - /etc/nginx/nginx.conf
        - /etc/redis/redis.conf
                                                                                                                        30,0-1        All

Split archive

We can run It on windows

good job！
We can run It on windows10 and Windows7：

tested：
1.mysql backup and files backup to oss
2.mysql backup and files backup to oss
3.OpenSSL - aes-256-cbc encrypt

note:
1.win7 need install tar（http://gnuwin32.sourceforge.net/packages/libarchive.htm），win10 not need
2.win7 and win10 need install openssl (http://gnuwin32.sourceforge.net/packages/openssl.htm),if you want aes-256-cbc encrypt

mysql 源增加additional_options配置参数后执行不成功

问题现象

mysql 加上配置additional_options:

additional_options: "--single-transaction --quick"

执行控制台输出 mysqldump: unknown variable 'single-transaction --quick'

初步分析

看代码发现additional_options是直接作为一个cmd arg 整体添加的exec.Command的。而Go的exec.Command 对args部分存在空格的参数支持行为有问题，如exec.Command("ls", "-l -a") 需要拆成exec.Command("ls", "-l", "-a")

建议

将spaceRegexp.Split(command, -1)逻辑应用每个arg, 类似这个逻辑: args = args.flat_map{|arg| spaceRegexp.Split(arg)}

当配置文件不存在archive段时会出现段错误

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x76378a]

gobackup/archive/archive.go:16
includes := model.Archive.GetStringSlice("includes")

超过5g文件不能上传，是没有支持s3的分段上传么？

unknown function: pg_catalog.set_config()

pg_dump: [archiver (db)] query failed: ERROR: unknown function: pg_catalog.set_config()
pg_dump: [archiver (db)] query was: SELECT pg_catalog.set_config('search_path', '', false)

An error occurred when I tried to backup a cockroachdb (postgresql) database

Here's my yml file
models:
localhost:
compress_with:
type: tgz
store_with:
type: local
keep: 20
path: /home/ernanie/backup/cockroach
databases:
bank:
database: bank
type: postgresql
host: localhost
port: 26257
username: root

Database socket connection

Please, allow to specify connection to local unix socket for mysql and redis database.
So user can provide configuration option 'socket', which will be used instead of 'host' and 'port' to connect to database.

Running in alpine docker: strange error: ash: ./gobackup: not found (v1.0)

# ./gobackup
ash: ./gobackup: not found

s3 storage can support `ForcePathStyle ` option ?

Some OSS cloud service supports s3 protocol with ForcePathStyle , eg Qiniu