swisscom / backman Goto Github PK

Backman version 1.27.0

it's not possible to restore a specific backup.
The API should provide /{file name} as parameter.

Thats right now not possible.

Currently using backman v1.30.2 backing up data from our elasticsearch service in internal iAPC.

Backman is taking almost 30min to backup 23MB, now is running since almost 5 hours to backup 600MB and is not yet done.

That's not possible, we have elasticsearch with 4-5GB of data still growing, how long will take to backup those? days?

Is there a way to configure a mysql service as user-defined service as it works for postgres?

It would be great to be able to trigger backups/restore via CF Tasks

This way we could rollout backman with 0 instances, making the solution a bit more cost effective.
Plus, this way we could run multiple operations in parallel without the risk of running out of resources or facing concurrency related issues.

Because of complications with the current environment variable based configuration (see #2) a better solution might be to implement a JSON based configuration?
Either read entirely from an new env var BACKMAN_CONFIG, and/or provided as config.json together with the app during push?

• Investigate using JSON env var BACKMAN_CONFIG.
  • Will it work without escaping issues?
• Investigate fallback loading local file config.json
• Investigate backwards-compatible fallback to parsing old env vars too?

Pros:

• Easier handling of more complex configuration scenarios
• Future proof

Cons:

• Reconfiguring a deployed app on the fly might get more time consuming (adjusting 1 big config env var instead of just multiple more simpler & smaller ones)

• Investigate possibility to use elasticdump for backup/restore of elasticsearch
  • Does it even work together with Swisscom AppCloud Elasticsearch services?
  • Is it easy to add with the apt-buildpack?
  • Can backman do a full backup and restore based on default elasticsearch binding information?

Same as #59 but with a different error message:

2022-01-19 08:24:51 [APP/PROC/WEB/0] OUT level=debug msg="executing elasticsearch backup command: elasticdump --quiet --input=https://shbfBNDYNTEpHEXy:6Vrz%+sMcPbbzcUm@***************.elasticsearch.dev-scapp-console.swisscom.com:443 --output=$"
2022-01-19 08:24:51 [APP/PROC/WEB/0] OUT level=debug msg="upload S3 object [elasticsearch/elasticsearch-log/elasticsearch-log_20220119082451.gz]"
2022-01-19 08:24:54 [APP/PROC/WEB/0] OUT level=error msg="Wed, 19 Jan 2022 07:24:54 GMT | Error Emitted => URI malformed\nWed, 19 Jan 2022 07:24:54 GMT | Error Emitted => URI malformed"
2022-01-19 08:24:54 [APP/PROC/WEB/0] OUT level=error msg="could not backup service [elasticsearch-log]: elasticdump: exit status 1"
2022-01-19 08:24:54 [APP/PROC/WEB/0] OUT level=error msg="requested backup for service [elasticsearch-log] failed: elasticdump: exit status 1"
2022-01-19 08:24:54 [APP/PROC/WEB/0] OUT level=error msg="could not upload service backup [elasticsearch-log] to S3: context canceled"

Our Backman instance has stopped doing backups of our applications due to below error:

ERROR 1153 (08S01) at line 96: Got a packet bigger than 'max_allowed_packet' bytes

Is it possible to add this parameter to the mysqldump command?

mysqldump --max_allowed_packet=512M ...

Add internal state to backman. Should be in-memory only, no external database needed.
backman is deployed in a single instance configuration and needs no persistent data. If the app gets restarted/restaged it would abort and lose any ongoing backup/restore operation anyway.

• Add tracking of backup/restore state & progress internally
• Expose new API endpoints to query for ongoing operations, similar to OSBAPI last_operation
• Use state API to overhaul service.html UI with JS (likely Vue.js)
  • Dynamically load Backups list, instead of server-side rendering html
  • Show progressbar or card with spinner upon triggering a backup/restore
  • Continuously poll state / last_operation
  • Reload Backups upon action completion (backup,restore,delete)

I propose to have toggle to turn off the HTTP-Endpoint to increase security. In our use-case we don't use it on a daily basis since the Web-UI is anyway read-only and it can easily be monitored through prometheus/logs.

The thing which scares me is that anyone with the credentials can directly access the backups and having this feature turned off would definitively improve my sleep :). What you think?

Hi,

Need your help on configuring backman which is running on iAPC cloud foundry to take the backup of the MariaDB database which is running on Telco cloud TCI (sharedtcs.net) open-stack cluster. I need to use cloud foundry so that my backman will continue to run on iAPC. I have created S3 bucket in iAPC. I want to create a custom DB service so that my backman can connect to the DB that is on TCI to take backup and store on s3 bucket in iAPC. What changes are required in manifest.yml file in this scenario and how can I request the connectivity between iAPC and my database in Telco cloud ?

@JamesClonk

It would be nice to have the functionality to restore but from another backend from another database as we encounter numerous database change like from mariadb to mysql and from dedicated mongodb instances to mongodb ent databases and moving those backup files for over 50 databases is a pure pain in the ass

Hi,

We have backman running in iAPC for daily backup of mariadb databases. We have found weird issue that though we have schedule backup for all the databases with same schedule we are not sure its not running the backup at the schedule or its not getting updated in backman UI. Some of the databases said that the last backup was 9 days ago, some says 4 days ago, some says 1 day ago, and some says 10hrs ago. Even though we have daily backup schedule as shown below but was not running daily.
I have attached the screenshot with the backman UI describing more datail.

            "mysql": {
              "schedule": "0 0 1 * * *",
              "timeout": "2h",
              "retention": {
                "days": 90,
                "files": 20
                },
              "disable_column_statistics": true,
              "service_binding": {
                "type": "mysql",
                "provider": "mariadb",
                "host": "((dbhost_dev1))",
                "username": "((dbusername))",
                "uri": "mysql://((dbusername)):((dbpassword))@((dbhost)):3306/mysql",
                "password": "((dbpassword))",
                "database": "mysql",
                "port": 3306
              }
            }

I am also seeing the app crashed events

ucpe-backman-dev at 13.01.2024 01:00:51 App crashed

ucpe-backman-dev at 12.01.2024 01:00:41 App crashed

ucpe-backman-dev at 11.01.2024 01:00:47 App crashed

@JamesClonk
Backman issue.docx

Hi @JamesClonk ,

If I trigger a backup in the Backman UI I can see the elasticdump process in the container.
But there is no data written to S3.

Now, if I copy the elasticdump command from ps x and execute it directly in the shell I can see the output from elasticsearch written to stdout. (with --output=$)

Replacing the below elasticdump command with 'echo foo' I can see data being written to S3.
So it seems that elasticdump does not print to stdout so backman can process it.

command = append(command, "elasticdump")
command = append(command, "--quiet")
command = append(command, fmt.Sprintf("--input=%s", connectstring))
command = append(command, "--output=$")
log.Debugf("executing elasticsearch backup command: %v", strings.Join(command, " "))
cmd := exec.CommandContext(ctx, command[0], command[1:]...)

// capture stdout to pass to gzipping buffer
outPipe, err := cmd.StdoutPipe()

Also pinned the version to 6.11.0 and older in the hope that the issue lies there but no success.
RUN npm install [email protected] -g

Cheers,
Mischa

Honestly, I don't know if it's a bug in the Cloud Foundry service I'm using, but I have this exact case here.

   2023-12-21T13:09:19.25+0100 [APP/PROC/WEB/0] OUT level=error msg="service binding for [elasticsearch] is missing property: [port]"
   2023-12-21T13:09:19.25+0100 [APP/PROC/WEB/0] OUT level=error msg="service binding for [elasticsearch] is not valid"

Cause, the host (in credentials of the service binding) has no port:

"host": "https://xxx.elasticsearch.com",

instead of

"host": "https://xxx.elasticsearch.com:443",

To fix that in the backman if it finds approval, the well-known port in enrichBinding could be set according to the schema/protocol.
If I find the time, I will submit a pull request.

Add https://github.com/swisscom/backman/blob/develop/service/service.go#L139

// set well-known port based on scheme if still missing
if binding.Port == 0 {
	if strings.EqualFold(u.Scheme, "https") {
		binding.Port = 443
	}
	if strings.EqualFold(u.Scheme, "http") {
		binding.Port = 80
	}
}

Workaround: Downgrade to backman version 1.32.2

BR Christian

Instrument backman to add Prometheus /metrics endpoint, exposing various useful metrics.

• add Prometheus endpoint
• add basic backman metrics
• add "per-service" metrics
• document /metrics in README.md

I have set up Backman for a MariaDB service. When I start this backup (manually or cron), I almost always get an "Out of Memory" error.

The Backman service is configured with a memory limit of 512MB. The backup is about 3 MB unzipped, and 900KB zipped.

With this configuration, the backup should be easily done, shouldn't it?

2020-02-11 09:29:30 [APP/PROC/WEB/0] OUT Exit status 137 (out of memory)
2020-02-11 09:29:30 [CELL/SSHD/0] OUT Exit status 0
2020-02-11 09:29:35 [CELL/0] OUT Cell xxx stopping instance yyy
2020-02-11 09:29:35 [CELL/0] OUT Cell xxx destroying container for instance yyy
2020-02-11 09:29:35 [CELL/0] OUT Cell xxx creating container for instance yyy
2020-02-11 09:29:35 [PROXY/0] OUT Exit status 137
2020-02-11 09:29:36 [CELL/0] OUT Cell xxx successfully destroyed container for instance yyy
2020-02-11 09:29:38 [CELL/0] OUT Cell xxx successfully created container for instance yyy
2020-02-11 09:29:38 [CELL/0] OUT Starting health monitoring of container

Hi @JamesClonk ,

While implementing the backman in cloud foundary I have noticed that there is no support for binding secret store in backman application. In my use case I am using the service binding for my database and passing the URI under $BACKMAN_CONFIG env variable but the DB password is visible in plain text under env variable when we run cf env backman which is not the best practice. We need some mechanism to hide the password. Need your help/advice here @JamesClonk

Even when we pass it using CLI as below these passwords are still visible in env variable

cf push -f manifest.yml --var username=backman --var password=my-password --var dbpassword=Super-secret-password

BACKMAN_CONFIG: |
      {
        "s3": {
          "service_label": "dynstrg",
          "bucket_name": "dev_s3"
        },
        "services": {
          "mariadb": {
            "schedule": "0 0 2,18,22 * * *",
            "retention": {
              "days": 90,
              "files": 20
            },
            "service_binding": {
              "type": "mysql",
              "provider": "mariadb",
              "host": "x.x.x.x", 
              "username": "root",
              "uri": "mysql://root:[email protected]:5432/test",
              "password": "Super-secret-password",
              "database": "test",
              "port": 3306
            }
          }
        }
      }

It would be nice to have Redis and/or Redis Enterprise as a supported data source.

Since upgrading from version v1.15.0 to v1.18.0, the mysql db backups(mariadb and mysql) are not working anymore, because of the following error:
OUT level=error msg="mysqldump: Couldn't execute 'SELECT COLUMN_NAME, JSON_EXTRACT(HISTOGRAM, '$.\"number-of-buckets-specified\"') FROM information_schema.COLUMN_STATISTICS WHERE SCHEMA_NAME = 'CFDB_02C85CC4_2FF3_4535_BFC0_F2DDC7D79429' AND TABLE_NAME = 'answers';': Unknown table 'COLUMN_STATISTICS' in information_schema (1109)"

The solution might be adding the option --column-statistics=0 to the mysqldumps. (Source: https://serverfault.com/questions/912162/mysqldump-throws-unknown-table-column-statistics-in-information-schema-1109).

I installed backman v1.30.1 in our internal Swisscom iAPC on multiple spaces/orgs.
Installation/Configuration is exactly the same for everyone but on some we get the following error after starting the backup of our elasticsearch service: Error Emitted => getaddrinfo ENOTFOUND ikjwdvgadry9g9w

Following the entire log:

2022-01-19 08:01:02 [APP/PROC/WEB/0] OUT level=debug msg="executing elasticsearch backup command: elasticdump --quiet --input=https://ikJWDVGAdry9g9WU:qD[1;A4,Jd?XK[qd@***************.elasticsearch.dev-scapp-console.swisscom.com:443 --output=$"
2022-01-19 08:01:02 [APP/PROC/WEB/0] OUT level=debug msg="upload S3 object [elasticsearch/elasticsearch-log/elasticsearch-log_20220119080102.gz]"
2022-01-19 08:01:25 [APP/PROC/WEB/0] OUT level=error msg="Wed, 19 Jan 2022 07:01:25 GMT | Error Emitted => getaddrinfo ENOTFOUND ikjwdvgadry9g9wu\nWed, 19 Jan 2022 07:01:25 GMT | Error Emitted => getaddrinfo ENOTFOUND ikjwdvgadry9g9wu"
2022-01-19 08:01:25 [APP/PROC/WEB/0] OUT level=error msg="could not backup service [elasticsearch-log]: elasticdump: exit status 1"
2022-01-19 08:01:25 [APP/PROC/WEB/0] OUT level=error msg="requested backup for service [elasticsearch-log] failed: elasticdump: exit status 1"
2022-01-19 08:01:25 [APP/PROC/WEB/0] OUT level=error msg="could not upload service backup [elasticsearch-log] to S3: context canceled"

Hello @JamesClonk

Thanks a lot for the cool backup script of DB's running on CF.
Is it possible to add a feature to save only certain collections in a MongoDB?

Regards,
Flavio

Hello,

I use the latest version of backman in the Swisscom iAPC. I've received a notification about the following vulnerabilities in the image. Can these be easily resolved?

CVE-2018-20839, CVE-2020-14039, CVE-2020-14040, CVE-2020-15366, CVE-2020-16845, CVE-2020-24553, CVE-2020-26160, CVE-2020-28366, CVE-2020-28367, CVE-2020-7754, CVE-2020-9794

Cheers,

R

The used client side encryption is only secure if different encryption keys are used for each data stream(which at the current implementation means every file uploaded to s3)
https://github.com/minio/sio/blob/master/DARE.md
As per the documentation an attacker could do the following if the encryption key is reused:

• replace packets with the same sequence number (replace part of one file with part of another file)
• if the nonce is reused (which is 64bits, so becomes increasingly likely to be reused over many files and upload sessions), the attacker could decrypt the package

I would suggest to use the configured encryption key only as a master key to derive the actual encryption key for each file.
This could be accomplished by using a KDF with the master key and s3 object key (filename) as input. Although the recommendation is to not use the filename in case a new version is uploaded and thus resulting in the same key, i think this probably doesn't apply here as the filename contains a timestamp and is thus unique.

This change (with the key derived from filename) could be done relatively easy by just doing adjustments in the upload/download methods as no further key management is necessary.

1. When triggering a restore, you don't see at the end if it was successfull or not, because the info is in a Popout, which is shown 5s.
   But if restore takes 1h and more you will miss this info and have to search in logs
   Whould be nice to have the result of the triggered backup permanent on screen (till you navigate away from page)
2. Add the version somewhere in the UI

Hi @JamesClonk

First of all, thanks for your work.
Is it possible to add a possibility to restore downloaded backups via the UI? Would be perfect!

Using v2.0.0 and v2.0.1 of backman on iAPC - elastic service backup always fails with the following errors:

10/27/2022, 11:47:06 PM [APP/PROC/WEB/0] OUT level=error msg="could not run elasticdump: exit status 1"
10/27/2022, 11:47:06 PM [APP/PROC/WEB/0] OUT level=error msg="could not backup service [cadc-elastic]: elasticdump: exit status 1"
10/27/2022, 11:47:06 PM [APP/PROC/WEB/0] OUT level=error msg="requested backup for service [cadc-elastic] failed: elasticdump: exit status 1"

After starting a backup I try to get the status of that task, but just receive a 503 and then a 404, until the backup is done.

Requerst:
/api/v1/state/mongodbent/xxx_db
Response:
404 Not Found: Requested route (' DELETED URL..... ') does not exist.

For encrypted connections that require client certificates.

For both mysql and mysqldump it would be at least the following options:

• --ssl
• --ssl-ca=/path/to/ca-cert.pem
• --ssl-cert=/path/to/client-cert.pem
• --ssl-key=/path/to/client-key.pem

It would also be neat if the certificates could be passed as plaintext so that the user doesn't have to bother with mounting them, but it would already be great enough if the configuration above could be applied.

I tried to start backman, but it crashes with error:

./backman: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by ./backman)

We've encountered a problem of identifying databases.

Our setup is as follows. 3 spaces, prod, staging, testing. We have the backman container in prod. Every app in those spaces needs a db. It's always called the same xyz-db. Once we share that db with the prod space it appears as a shared service but once mapped to the container cf seems confused about which app to bind. In the end it displays all of them under the service tab of the container but only one has the label 1 bound app.

Backman seems to face the same problem. Well as of in the user only the name is the identification of the database which would there be a big problem as those 3 services have the same name but are in separate environments

Why would we need only one app? Well, first of all, we don't need to run QA for it nor do we demo it. So it's a product we consume therefor prod. And also to keep those unused resources lower.

Hello @JamesClonk

I'm using your backman Docker repo on Swisscom Cloud Foundry for backing up MongoDB's.
Now I noticed that the backup filename does not use the correct time zone, which was defined when deploying the application by setting "cf set-env backman TZ Europe/Zurich".

backman uses the time zone UTC which is shown in the filename.

Is this a bug of the application or am I setting the time zone of the backmann app wrong?

Thanks for your help :)

Regards,
Flavio

On large DBs the backup does not work - where large is relative. Sometimes a little data + system data is enough to allow the backup to fail.

I also played with the sources and limited the ElasticDump script to single indexes. In this case the backup works fine. Therefore I assume that it is due to the size of the backup.

It is also no timeout issue! I did my tests with 10h timeout.

In order to retrieve easily the history of the triggered backup would be useful to have a this information in the UI in the service under backup with the followin infos:

• date-time started
• how long it took
• status (SUCCESS/FAILURE)
• error message (in case of FAILURE)

Currently any configured cron schedules in BACKMAN_BACKUP_SCHEDULES that contain a comma, for example "0 2,7,12,16,18,22 * * *" fail due to a parsing error.

• Investigate env/config parsing library, can this behaviour by fixed without code changes?
• Investigate using another configuration format and library for defining custom schedules
• backman has to support cron schedules containing commas